Broadcastify Calls - Connection Dropping

[rjdj2000]

Anyone else's calls connection dropping out in the morning? Yesterday and today I have gotten 3 separate times that the connection has dropped and reconnected. 1st @ 2:16 AM, 2nd @ 2:54 AM and the 3rd @ 4:50 AM. I know there is a lot of things in-between the keyboard and the endpoint connection so who knows what is up. Just was wondering if anyone else has had the same issue. Also I am using SDRTrunk for this and there is nothing on there about a dropped connection, only an error about uploading a call that it says is a duplicate but I have talked to the creator of it and he says he don't see anything that is of major concern.

 

[DougWare]

Mine is dropping out at random times, but often during the morning.

I see that the DNS record that us used to upload has several IPs in round-robin. It makes me wonder if trunk-recorder is using the first IP in the reply and if something happens it doesn't try another until you restart. NGINX has this behavior as well. I'm still trying to troubleshoot it, but I haven't had the time to get further into it.

What makes it worse is that my "classic" feed with a scanner has started going down 3-4 times a day. I'm on a fiber connection and I'm not seeing any other problems throughout my network, so I'm pretty sure it's not my connection.

Doug

 

[SDWolfman]

I also use SDRTrunk, and I am also experiencing calls connection issues during the morning hours. It started a couple of weeks ago. Oddly, the two feeds I also run off he same SDRTrunk program continue feeding when the calls connection starts to report connection errors.

 

[DougWare]

I'm using trunk-recorder on two Raspberry Pi 4s. I could run it on something powerful enough for a GUI, but I really wanted a CLI/headless interface.

At present I'm running a Pie connected to a BCD325P2 using EzStream for the classic stream. There's a way to stream a classic stream from a folder of trunk-recorder files and I'd like to try that if I can stabilize the trunk-recorder streams so I can free up a scanner & remove a point of failure.

I have enough Linux experience that I can pretty easy script watchdogs and restarts if something fails. From what I can see, I can't find a way using the Calls API to monitor the last call upload time to see if I need to restart the trunk-recorder process. With the classic streams, I just monitor the page for the phrase "Offline", the kill/restart the EzStream process automatically.

 

[dan-dekalb]

When you say “Classic” are you referring to a conventional system rather than a trucked system? In that case, you define the system as conventional in the trunk recorder configuration. You have to set a squelch level, and when a signal is detected, it records and streams via LiquidSoap just like a trucked system.

 

[DougWare]

When you say “Classic” are you referring to a conventional system rather than a trucked system? In that case, you define the system as conventional in the trunk recorder configuration. You have to set a squelch level, and when a signal is detected, it records and streams via LiquidSoap just like a trucked system.

No, sorry for the miscommunication. When I was referring to "classic" I was referring to using a traditional scanner (in this case a P25 Phase 2 scanner) attached to a microphone port on a device streaming an audio stream.

 

[dan-dekalb]

That makes things much clearer. Do you get any error messages when Calls goes down? I could grep my log files and see if I see the same. Moving to LiquidSoap and sending the audio via the trunk-recorder script is pretty easy, and then you don’t miss any audio because the scanner was tuned to one transmission and missed another. Only thing to be concerned about is if it’s a “busy night” the queued audio can get backed up.

 

[DougWare]

I'm really starting to think that my problem may be related to one of my two NooElec v3 sticks going offline, the next time it happens I hope to verify that. The last thing I noticed last time was that it kept switching to SRC 1, never SRC2 and kept scanning for a control channel.

I've really tried this on multiple Pi4s, which seem to run around 50-60% CPU during peaks, with and without USB3 hubs. I've tried this under the latest Raspbian, now I'm on Ubuntu Server because some of the pre-compiled libraries are newer and the build instructions seem to fit the Ubuntu distro better (fdkaac and libfdk-aac-dev are available in Ubuntu's stock repo). I'm not against compiling libraries, it just seems to be what was best for the instructions I read.

 

[DougWare]

I just noticed how poorly the NooElec SDRs fit into the USB port. I assumed it just needed adjusting, so I grabbed my precision screwdriver set, and that's when I realized the enclosures are actually different lengths, poor quality control I guess. I'm about 6 SDRs invested into the NooElec V3s. Can anyone recommend a better SDR that won't break the bank?

Doug

 

[waterbwuk]

Which SDRTrunk version are you on? I was using 0.5.0 Alpha 1 and suddenly it stopping making network connections to BCFY. When I switched to Alpha 3 it started working again. I posted about it on the SDRTrunk Google Group and other people were having lots of network connection issues with Alpha 1 also. Switching to 2/3 seemed to fix it, for now.

 

[dan-dekalb]

How many dongles are you running? I ended up with a 10-port Anker with 5 RTL-SDR dongles plugged in. They seem to draw a lot of power and get very hot when running. I haven’t had an issue with them disconnecting however.

 

[DougWare]

I'm running 2 on 1 Pi4, 2 on another Pi4, and trying to 2 on another Pi3 for two HAM repeaters using rtl_airband.

They do get very hot. I did manage to pull one out of the enclosure and I saw there was a finned heatsink inside that is just pressed up against the outside enclosure case, which is supposed to the the heatsink. If the outside is that hot with poor heat transfer, it must be super hot inside.

Doug

 

[dan-dekalb]

One thing to watch out for is cheep USB hubs leak power back to the Pi. The Pi won’t boot if there’s power on the 5V USB port. I chased the issue for a month thinking it was an issue with my RTL-SDR’s until I figured out that it was the hub. I had to boot the Pi, then plug in the hub or it wouldn’t work. I do suspect that high operating temperatures may be part of the design to control frequency drift. By maintaining a higher temperature internally, the effect of ambient temperature swings represent a smaller percentage swing.

Are the first two for monitoring separate systems? I’m monitoring two P25 systems and an analog on a single PI4.

 

[DougWare]

They are monitoring 2 different systems, and 2 different repeaters for HAM traffic.

I had them all running together on 1 Pi under Raspbian, but I kept getting an error message about needing to set " /bin/echo 0 > /sys/module/usbcore/parameters/usbfs_memory_mb ", even though it was. It would creep in after a few mins and would stop recording. That seems to be related to a USB buffer problem. That was under Raspbian though and I may not have that problem under a Ubuntu with the stock libraries.

 

[dan-dekalb]

I had the same issue. I believe I fixed it by changing the 0 to 1024. It seems like it didn’t like the unlimited value, but a high but actual value stuck.

 

[DougWare]

I'm seeing this in my logs now. It looks like my uploads stopped about 2 hours ago.

[2020-09-07 18:05:49.006277] (info)     - Stopping P25 Recorder Num [0] TG:      40019  Freq: 8.521500e+08      TDMA: false     Slot: 0
[2020-09-07 18:05:49.076013] (error)   [wakeco] TG: 40019       Freq: 8.521500e+08      Broadcastify Metadata Upload Error:
[2020-09-07 18:05:49.244911] (info)   [wakeco]  TG: 40019       Freq: 8.521500e+08      OpenMHz Upload Success - file size: 33392

 

[DougWare]

And I'm seeing the same error on the second Pi running trunk-recorder. Both seem to have stopped uploading at the same time.

[2020-09-07 18:07:57.001991] (info)     - Stopping P25 Recorder Num [0] TG:       1003  Freq: 8.514375e+08      TDMA: false     Slot: 0
[2020-09-07 18:07:57.040993] (error)   [joco]   TG: 1003        Freq: 8.514375e+08      Broadcastify Metadata Upload Error:
[2020-09-07 18:07:57.176183] (info)   [joco]    TG: 1003        Freq: 8.514375e+08      OpenMHz Upload Success - file size: 9432

 

[DougWare]

Well, I believe I caught the underlying event but I'm not sure what's causing it. It went from recording to scanning and never finding a control channel. I really wonder if I need to add all the channels and not just the 4 control channels listed in the DB.

[2020-09-08 15:13:31.003441]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:31.003980]: [wakeco]  TG:      28752  Freq: 8.516250e+08      Ending Recorded Call - Last Update: 4s  Call Elapsed: 7
[2020-09-08 15:13:31.005262]:   - Stopping P25 Recorder Num [1] TG:      28752  Freq: 8.516250e+08      TDMA: false     Slot: 0
[2020-09-08 15:13:31.006234]: [wakeco]  Deleting this call as it has a duration less than minimum duration of 0 TG:      28752Freq: 8.516250e+08      Call Duration: 0s
[2020-09-08 15:13:31.007068]: [wakeco]  TG:      36512  Freq: 8.526500e+08      Ending Recorded Call - Last Update: 4s  Call Elapsed: 4
[2020-09-08 15:13:31.007977]:   - Stopping P25 Recorder Num [2] TG:      36512  Freq: 8.526500e+08      TDMA: false     Slot: 0
[2020-09-08 15:13:31.531983]: [wakeco]  TG: 36512       Freq: 8.526500e+08      Broadcastify Upload Success - file size: 8659
[2020-09-08 15:13:31.739213]: [wakeco]  TG: 36512       Freq: 8.526500e+08      OpenMHz Upload Success - file size: 8659
[2020-09-08 15:13:34.006468]: Pll Phase: 3.14997 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:34.007186]: [wakeco] Retuning to Control Channel: 8.537625e+08
[2020-09-08 15:13:34.011188]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:34.011753]: [wakeco]   Control Channel Message Decode Rate: 0/sec, count:  0
[2020-09-08 15:13:37.008934]: Pll Phase: -4.96739 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:37.009306]: [wakeco] Retuning to Control Channel: 8.535375e+08
[2020-09-08 15:13:37.009446]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:37.009744]: [wakeco]   Control Channel Message Decode Rate: 0/sec, count:  0
[2020-09-08 15:13:40.005889]: Pll Phase: 5.83892 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:40.006357]: [wakeco] Retuning to Control Channel: 8.539625e+08
[2020-09-08 15:13:40.006725]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:40.007069]: [wakeco]   Control Channel Message Decode Rate: 0/sec, count:  0
[2020-09-08 15:13:43.002740]: Pll Phase: -3.08512 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:43.003478]: [wakeco] Retuning to Control Channel: 8.537875e+08
[2020-09-08 15:13:43.004065]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:43.004628]: [wakeco]   Control Channel Message Decode Rate: 0.333333/sec, count:  1
[2020-09-08 15:13:46.000403]: Pll Phase: 3.47647 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:46.000753]: [wakeco] Retuning to Control Channel: 8.531500e+08
[2020-09-08 15:13:46.000901]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:46.001197]: [wakeco]   Control Channel Message Decode Rate: 0/sec, count:  0
[2020-09-08 15:13:49.007202]: Pll Phase: -0.799632 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:49.007679]: [wakeco] Retuning to Control Channel: 8.537625e+08
[2020-09-08 15:13:49.008139]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:49.009612]: [wakeco]   Control Channel Message Decode Rate: 0.333333/sec, count:  1
[2020-09-08 15:13:52.006004]: Pll Phase: 2.92766 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:52.006316]: [wakeco] Retuning to Control Channel: 8.535375e+08
[2020-09-08 15:13:52.006429]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:52.006519]: [wakeco]   Control Channel Message Decode Rate: 0/sec, count:  0
[2020-09-08 15:13:55.001982]: Pll Phase: 1.19821 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:55.002272]: [wakeco] Retuning to Control Channel: 8.539625e+08
[2020-09-08 15:13:55.002360]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:55.002472]: [wakeco]   Control Channel Message Decode Rate: 0/sec, count:  0
[2020-09-08 15:13:58.008247]: Pll Phase: -3.4053 min Freq: -6.283185e-01 Max Freq: 6.283185e-01
[2020-09-08 15:13:58.008629]: [wakeco] Retuning to Control Channel: 8.537875e+08
[2020-09-08 15:13:58.008793]:    - System Source 1 - Min Freq: 8.526338e+08 Max Freq: 8.545538e+08
[2020-09-08 15:13:58.008936]: [wakeco]   Control Channel Message Decode Rate: 0/sec, count:  0