OP25 Stops Decoding

[W3AXL]

Hi all

I've got OP25 up and running on my ubuntu box, paired with a simple snd_aloop->darkice setup for private streaming. I'm running in to an issue where OP25, despite showing that it's still tuning and receiving traffic, stops outputting audio. No errors are displayed even with verbosity level 9, it just quits outputting sound altogether. A restart of OP25 fixes the issue but then after a few minutes it does the same thing again.

I'm launching op25 with the following command:

./rx.py --args 'rtl' --calibration=-100 -l http:192.168.0.151:8180 -N 'LNA:42' -S 1200000 -f 857.7375e6 -o 24e3 -O loop0 -x 2 -T trunk.tsv -v 7 -U -w

On the web gui, it's as if nothing has changed. The constellation looks good and it's still tuning to the various frequencies. Just no audio coming out, which I can verify by viewing the output of "arecord -D loop0 -vv".

There was a single time when it worked flawlessly for several hours. I did a system reboot to make sure it would work again and it went right back to the same issue. No clue how I managed to get it to work that one time.

I'd appreciate help on this. It's a headless ubuntu server 18.04LTS. The one thing I think could be happening is the lack of a real hardware soundcard messing with GNURadio's audio sink but I'd imagine that would be an easy fix. Just don't know how to implement it.

Thanks!

 

[W3AXL]

Interesting update: evidently it might not be a problem with OP25 after all. Shutting down the darkice source seemed to make decoding work flawlessly for well over a half hour. I'm guessing it might be either a sample rate issue between OP25 and Darkice or something going on within the snd_aloop module.

I have a sneaking suspicion that the lack of a hardware soundcard on my system might be confusing the alsa driver and causing things to get angry at each other.

I'd love to hear from anyone who knows more about this stuff than I do. If the devs could speculate on what could be causing this strange compatibility error I'd love to know.

 

[boatbod]

If you'd said your hardware was a low power Raspberry Pi3 I would have responded that it seems to be a known intermittent issue with no known cause or solution. The fact that you are apparently using some other hardware makes this all the more curious...

What terminal are you running on this headless machine? I've had most success using the http terminal with headless installations. The traditional curses terminal can get quite quirky when running headless setup with the screen output going nowhere; I had reasonable luck faking it with the "screen" utility, but http is a better overall solution.

Try running "top -H" to see what threads are active when there is voice traffic that is supposed to be decoded. I suspect some sort of deadlock but have never been able to pinpoint it.

 

[W3AXL]

Sorry for taking so long to get back. I somehow got the system working fantastically for almost two weeks with zero issues. Had a power loss a couple days ago and now we're back to the same issue. Nothing has changed in the setup but once again OP25 only outputs audio for a couple minutes before stopping, and once again everything is acting like nothing is wrong.

As for your first question, it's a custom built server with a Xeon on a supermicro board (hence the no onboard soundcard).

As far as I know I'm running the HTTP terminal properly. It works fine and plots fine. Once again this is the startup string I'm using:

./rx.py --args 'rtl' -d -300 -l http:192.168.0.151:8180 -N 'LNA:47' -S 960000 -f 857.7375e6 -o 24e3 -O loop0 -x 2 -T trunk.tsv -v 9 -U -w

I'm almost positive (almost) that it's an OP25 problem. I can leave the darkice instance and the arecord instance running and just restart OP25 and everything will work again for another couple minutes.

Recently it's getting even worse. On the most recent restart tonight I lost voice output in about 15 seconds after launch, and as usual no errors came up anywhere. The last few before that only managed to keep going for a few transmissions before they stopped working too. The only thing I can see occasionally in the OP25 logs is a voice timeout line but it doesn't occur consistently enough for me to say it's part of the problem.

top -H doesn't appear to be significantly different between working decode vs "broken" decode. Always on the top is fir_filter_ccc1 followed by the top program itself and then multiply_cc11 underneath that. Everything else jumps all over the place but during voice decoding, whether broken or not, p25_frame_asse2 is typically towards the top. Again nothing seems different between working vs non working.

I really want this to work. It's super frustrating that it was working for two weeks straight with zero issues and then as soon as I started it back up it went right back to the same problems.

 

[boatbod]

Sorry for taking so long to get back. I somehow got the system working fantastically for almost two weeks with zero issues. Had a power loss a couple days ago and now we're back to the same issue. Nothing has changed in the setup but once again OP25 only outputs audio for a couple minutes before stopping, and once again everything is acting like nothing is wrong.

As for your first question, it's a custom built server with a Xeon on a supermicro board (hence the no onboard soundcard).

As far as I know I'm running the HTTP terminal properly. It works fine and plots fine. Once again this is the startup string I'm using:

./rx.py --args 'rtl' -d -300 -l http:192.168.0.151:8180 -N 'LNA:47' -S 960000 -f 857.7375e6 -o 24e3 -O loop0 -x 2 -T trunk.tsv -v 9 -U -w

I'm almost positive (almost) that it's an OP25 problem. I can leave the darkice instance and the arecord instance running and just restart OP25 and everything will work again for another couple minutes.

Recently it's getting even worse. On the most recent restart tonight I lost voice output in about 15 seconds after launch, and as usual no errors came up anywhere. The last few before that only managed to keep going for a few transmissions before they stopped working too. The only thing I can see occasionally in the OP25 logs is a voice timeout line but it doesn't occur consistently enough for me to say it's part of the problem.

top -H doesn't appear to be significantly different between working decode vs "broken" decode. Always on the top is fir_filter_ccc1 followed by the top program itself and then multiply_cc11 underneath that. Everything else jumps all over the place but during voice decoding, whether broken or not, p25_frame_asse2 is typically towards the top. Again nothing seems different between working vs non working.

I really want this to work. It's super frustrating that it was working for two weeks straight with zero issues and then as soon as I started it back up it went right back to the same problems.

With no error being output, the issue is almost certainly a deadlock or blocking system call that's killing things. The issue is how to determine which thread is getting stuck.

You could try disabling the internal audio player and have the voice frames sent to nc | aplay to see if the freeze still happens. If it doesn't, the problem would have to be in the sockaudio python code vs the c/c++ library.

 

[W3AXL]

I think I might've found a solution, albeit this could just be another fluke. I kept everything else the same and changed the audio device in OP25 from "loop0" to "loopout0" because I figured there might be a conflict between Darkice and OP25 which both use the same device name. This seems to have worked for now as the stream has been going strong for 9 hours.

Not sure if this is the right fix but at least it's progress.

 

[boatbod]

I think I might've found a solution, albeit this could just be another fluke. I kept everything else the same and changed the audio device in OP25 from "loop0" to "loopout0" because I figured there might be a conflict between Darkice and OP25 which both use the same device name. This seems to have worked for now as the stream has been going strong for 9 hours.

Not sure if this is the right fix but at least it's progress.

loop0 is a bi-directional device and it's the same config that I'm using without any issue. I'll be very surprised if it doesn't lock up again somewhen.

 

[W3AXL]

loop0 is a bi-directional device and it's the same config that I'm using without any issue. I'll be very surprised if it doesn't lock up again somewhen.

I also tried piping the audio through UDP but since I'm not super familiar with that approach I gave up before getting too far. Mostly because this seemed to work but if it dies again I'll be giving that option another go.