OP25 Just stopped working...

[ewh01]

So I had the new osmocom version up and running well. Then I started to get random dropouts of the web interface, so I did a restart. After the restart it will go through the app start process, then back to the command prompt. Is there any way I can find out what is causing the error, or do I need to start from scratch (again)?

I know just enough Linux to be dangerous...

Thanks in advance

 

[lwvmobile]

What command are you using to launch with? What we want to look at is the log file, see what is getting put out prior to it stopping. Don't know if you are using something like 2> stderr.2 or similar to log output or if it just goes straight to the terminal, but you'll usually want to look at that for clues as to what happened prior to it stopping.

 

[wgbecks]

Posting your logfile as suggested by @lwvmobile is excellent advice in that it may contain clues as to what's happening. That said, I will tell you that OP25 is very unforgiving of poor USB connections to the SDR. I would verify that your SDR's connection is tight and to eliminate use of any extension cables, or external hub devices.

You sometimes will find a particular USB connector to be lose fitting and may have to move the SDR to another connector and restart op25.
Additionally, don't overlook cheap noisy switch mode power supplies when using SBC's such as a Raspberry Pi or to power and external hub device as these too can contribute to improper or unstable operation.

 

[wgbecks]

Following up on my comments in Post #3, I'd like to amplify on just how volatile op25, and similar applications are that rely on USB data transfers between the host OS and the SDR device. I recently experienced unexpected freeze-ups (crashing) of op25 that occurred anytime from just a few hours to a few days after starting op25 that were finally traced to an intermittent connection at the Mini-USB connector of my Airspy R2 SDR.

I want to give special thanks to @boatbod for his help in diagnosing error messages contained in my logfile associated with these crashes that ultimately lead to pinpointing of the failure mechanism. Further, he responded with a UDEV rule and recovery script that can be implemented to automatically restart op25 when run as a system service in the event of the disruption of communications with SDR device that's a feature especially noteworthy to op25 private and public feed providers.

 

[a417]

Following up on my comments in Post #3, I'd like to amplify on just how volatile op25, and similar applications are that rely on USB data transfers between the host OS and the SDR device.

You couple that with the trivially small default linux usbfs buffer size (debian is currently 16 mb), and it further increases the volatility of the transfer.

cat /sys/module/usbcore/parameters/usbfs_memory_mb
What does this return? Don't worry it's completely safe to run.

 

[wgbecks]

cat /sys/module/usbcore/parameters/usbfs_memory_mb
What does this return? Don't worry it's completely safe to run.

The same abysmal 16 mb defaults found in Raspbian Buster and Bullseye (debian) OS as well and is a value that often needs
to be addressed when running multiple SDR's or SDR's with very high sample rates.

 

[a417]

The same abysmal 16 mb defaults found in Raspbian Buster and Bullseye (debian) OS as well and is a value that often needs
to be addressed when running multiple SDR's or SDR's with very high sample rates.

Agreed, I am up at 2 gb on my heaviest hardware and it is much more stable.

Sorry @wgbecks, i meant to tag to @ewh01 in this and not make it look like an reply to you. Figured you had gone down that road long before...

 

[ewh01]

First - THANKS for the replies!!

How / where do I find the log file?

Here is the start command I've been using without a problem:
./rx.py --args 'rtl=0' -N 'LNA:47' -S 1440000 -f 774.68125e6 -o -200 -l http:192.168.1.78:8080 -w -q 0 -T trunk.tsv -V -2 -U 2> stderr.2

I suppose I should have given you a little more info. I have tried different USB ports. Historically I have used a short USB extension just because of where I have this set up, so I ruled that out by going direct. I typically run this in a 3b, so I tried the SD in a 4b with a different SDR dongle, same thing. I tried this SDR in the 4 running ASD-B, it works.

I have used the same configuration with the boatbod version for over a year without a hitch. I decided to try the osmocom version and it worked for a while....

I have everything shut down at the time, I will run the above "cat" command tomorrow and hopefully I will learn where and how to get hte log file.

Ed

 

[a417]

Ah, I am just now realizing this is on a SBC system...

Ok, let's address the log issue first. The end of your command

<snip> -V -2 -U 2> stderr.2

..redirects errors to a file named 'stderr.2' in the op25 folder. Lets see that.

 

[ewh01]

OK, Attached is the stderr.2 file, and the return on the cat command is 16

 

[KA1RBI]

Thanks for posting this stderr file.

Indeed it does show a crash but looks to be a tributary error, possibly caused by some should-have-been killed but still-existing process that is still bound to the http server TCP port.

That explains why subsequent runs fail, but unfortunately doesn't get to the inital root cause - the first crash.

Would it be possible to capture the stderr.2 from a "primary" failure?

Max

 

[a417]

Thanks for posting this stderr file.
<snip>
Would it be possible to capture the stderr.2 from a "primary" failure?

@ewh01, can you change that single greater than to a double greater than?

Your current command line will automatically create a new log file each time you run it, if you change that > stderr.2 to a >> stderr.2, it will append log file. Not having that overwritten each time will help @KA1RBI figure out what's going on.

Over time, this appended file will get rather large, so beware.

 

[ewh01]

I'll will make the change. How many times should I (try to) run it to get enough info for you to graciously assist?

THANKS MILLIONS!!!

 

[a417]

I'll will make the change. How many times should I (try to) run it to get enough info for you to graciously assist?

THANKS MILLIONS!!!

start from a fresh reboot, and run it a couple of times until it crashes.

The reboot will (pretty much) guarentee a blank slate, and ensure that there's no hanging process or hook issues.

 

[ewh01]

start from a fresh reboot, and run it a couple of times until it crashes.

The reboot will (pretty much) guarentee a blank slate, and ensure that there's no hanging process or hook issues.

Will do. I never actually "runs" anymore, just goes back to the command prompt.
Hopefully you can gather something from the log file.

 

[ewh01]

Here is the log file after several attempts to run. This is one the original configuration that "used to" work.

 

[Outerdog]

What other software is running on this machine?

 

[ewh01]

What other software is running on this machine?

Nothing, I built this from a fresh card.

 

[KA1RBI]

The error messages indicate that at the point in time you're starting rx.py there is something already listening on port 8080. Possibly there is a system startup script that runs it at boot time?

You could start to diagnose it by one or more of the following
1) Try connecting the browser to port 8080 and see what happens
2) Try the command "sudo netstat -anp | grep LISTEN" and post the result
3) Try the command "sudo lsof -n | grep LISTEN" and post the result.

One or the other of the commands in 2&3 might not be installed on your system, but we only need one (or the other)...

Max

 

[ewh01]

Ran #2:

pi@RadioPi:~ $ sudo netstat -anp | grep LISTEN
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 498/vncserver-x11-c
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 504/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 438/cupsd
tcp6 0 0 :::5900 :::* LISTEN 498/vncserver-x11-c
tcp6 0 0 :::22 :::* LISTEN 504/sshd
tcp6 0 0 ::1:631 :::* LISTEN 438/cupsd
unix 2 [ ACC ] STREAM LISTENING 15282 564/systemd /run/user/1000/systemd/private
unix 2 [ ACC ] STREAM LISTENING 16188 564/systemd /run/user/1000/gnupg/S.gpg-agent.extra
unix 2 [ ACC ] STREAM LISTENING 16189 564/systemd /run/user/1000/pulse/native
unix 2 [ ACC ] STREAM LISTENING 16190 564/systemd /run/user/1000/bus
unix 2 [ ACC ] STREAM LISTENING 16191 564/systemd /run/user/1000/gnupg/S.gpg-agent.ssh
unix 2 [ ACC ] STREAM LISTENING 16192 564/systemd /run/user/1000/gnupg/S.dirmngr
unix 2 [ ACC ] STREAM LISTENING 16193 564/systemd /run/user/1000/gnupg/S.gpg-agent
unix 2 [ ACC ] STREAM LISTENING 16194 564/systemd /run/user/1000/gnupg/S.gpg-agent.browser
unix 2 [ ACC ] STREAM LISTENING 18871 673/pcmanfm /run/user/1000/pcmanfm-socket--0
unix 2 [ ACC ] STREAM LISTENING 20656 796/menu-cached /run/user/1000/menu-cached-:0
unix 2 [ ACC ] STREAM LISTENING 23664 1811/lxterminal /run/user/1000/.lxterminal-socket-:0.0
unix 2 [ ACC ] STREAM LISTENING 11441 1/init /run/systemd/private
unix 2 [ ACC ] STREAM LISTENING 17095 522/Xorg @/tmp/.X11-unix/X0
unix 2 [ ACC ] STREAM LISTENING 11449 1/init /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 11465 1/init /run/systemd/fsck.progress
unix 2 [ ACC ] SEQPACKET LISTENING 11477 1/init /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 14413 1/init /run/thd.socket
unix 2 [ ACC ] STREAM LISTENING 14421 1/init /run/avahi-daemon/socket
unix 2 [ ACC ] STREAM LISTENING 15424 1/init /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 15426 1/init /run/cups/cups.sock
unix 2 [ ACC ] STREAM LISTENING 16761 465/dhcpcd /var/run/dhcpcd.sock
unix 2 [ ACC ] STREAM LISTENING 16762 465/dhcpcd /var/run/dhcpcd.unpriv.sock
unix 2 [ ACC ] STREAM LISTENING 21804 498/vncserver-x11-c /tmp/.vnc-0/print.0x1f2_0x6aac970
unix 2 [ ACC ] STREAM LISTENING 17096 522/Xorg /tmp/.X11-unix/X0
unix 2 [ ACC ] STREAM LISTENING 17120 498/vncserver-x11-c /tmp/.vnc-vncservice/vncserver-x11.CtrlComms
unix 2 [ ACC ] STREAM LISTENING 18750 618/ssh-agent /tmp/ssh-xdaQCewbamY1/agent.577
unix 2 [ ACC ] STREAM LISTENING 18220 687/ssh-agent /tmp/ssh-dqSVohfyECSr/agent.683