Feed down after network hiccup. Monitor and auto-restart?

[benjamingslade]

We had some power hiccups during a windstorm. My Raspberry Pi stayed up, but I think there was a network hiccup and the connection stopped working. I saw this in the Darkice log:



It looks like it was in a weird state where the TCP/IP connection was up, but not responding properly.

I ran the BCFY shell and restarted the Darkice service/instance, and it was fine. But I was surprised the software wasn't able to recover from this itself.

I was thinking about writing my own shell script, using the systemctl commands from the BCFY shell, to monitor the service status and periodically try to restart the Darkice service/instance if it appears to be down. And maybe, after several Darkice service auto-restarts within a certain time period, reboot the whole Raspberry Pi computer.

Thought/feedback? Maybe someone has already done something similar?

Thanks in advance
Ben
CSX Metropolitan Line - Kensington Live Audio Feed (yes, there's a buzz I'm trying to get rid of)

 

[benjamingslade]

This happened again:

From /home/pi/bcfy/logs/darkice_1.log :

23-May-2021 00:07:08 BufferedSink, new peak: 239 / 441000
23-May-2021 00:07:08 BufferedSink, new peak: 655 / 441000
23-May-2021 00:07:09 BufferedSink, new peak: 1365 / 441000
23-May-2021 00:07:10 BufferedSink, new peak: 2820 / 441000
23-May-2021 00:07:11 BufferedSink, new peak: 5722 / 441000
23-May-2021 00:07:14 BufferedSink, new peak: 11648 / 441000
23-May-2021 00:07:20 BufferedSink, new peak: 23454 / 441000
23-May-2021 00:07:32 BufferedSink, new peak: 46995 / 441000
23-May-2021 00:07:55 BufferedSink, new peak: 94069 / 441000
23-May-2021 00:08:42 BufferedSink, new peak: 188213 / 441000
23-May-2021 00:09:15 TcpSocket :: write, send error 104
23-May-2021 00:09:15 Exception caught in BufferedSink :: write2
23-May-2021 00:09:15 HTTP/1.0 200
23-May-2021 00:11:21 BufferedSink, healed: 0 / 441000
23-May-2021 10:07:21 BufferedSink, new peak: 223 / 441000
23-May-2021 10:07:21 BufferedSink, new peak: 586 / 441000
23-May-2021 10:07:21 BufferedSink, new peak: 1318 / 441000
23-May-2021 10:07:22 BufferedSink, new peak: 2767 / 441000
23-May-2021 10:07:23 BufferedSink, new peak: 5605 / 441000
23-May-2021 10:07:26 BufferedSink, new peak: 11341 / 441000
23-May-2021 10:07:32 BufferedSink, new peak: 22847 / 441000
23-May-2021 10:07:43 BufferedSink, new peak: 45830 / 441000
23-May-2021 10:08:06 BufferedSink, new peak: 91725 / 441000
23-May-2021 10:08:52 BufferedSink, new peak: 183468 / 441000
23-May-2021 10:10:24 BufferedSink, new peak: 367027 / 441000
23-May-2021 10:11:01 BufferedSink :: store, buffer overrun

But the shell tool said the status says "active" for the Darkice instance:

------------------------------------
>>> Configured Darkice Instances <<<
----------------------------------------------------------------------------
Item Status Config Instance
Boot
----------------------------------------------------------------------------
[1] active (running) Configured 1-CSX Metropolitan Line - Kensington
enabled

A restart of this Darkice instance fixed things.

So there's no obvious local check to see if the problem is active (the Darkice service/instance erroneously shows as active)

Maybe a curl call to Broadcastify.com to see if it thinks the service is up and restart the Darkice instance if not. Maybe something like this:

#!/bin/sh
feed_status=`curl --silent https://www.broadcastify.com/listen/feed/35407 | grep '^<td class="online">' | awk -F'"' '{print $2}'`
echo "feed_status=$feed_status"
if [ "$feed_status" == "online" ] ; then
   echo "Status is good"
else
   echo "Need to restart"
   # restart code goes here (maybe just restart the whole Raspberry Pi server?)
fi

Admittedly, grep'ing the results of the curl output may break if they someday change the web page design. To bad there's not an explicit web call to check feed status.

 

[benjamingslade]

It looks like this problem might have a fix described in this posting:

Broadcastify Raspberry Pi Feed Appliance Image v1.0 (BETA)

I've found a pretty big flaw with darkice feeds with this image vs using my own ones configured with darkice for about 5 years that didn't have this major issue. If the internet connection drops, even for 10 seconds or so, the feed stops! I've tested this many ways. If I just reboot my...

forums.radioreference.com