Has anyone else experienced missing archive recordings? My feed has been up all day, but I have a 30 minute segment that is not in the archive. This has happened before and there does not seem to be any discernable pattern to it. To be clear, I am NOT talking about an archive with no audio -- the archive files do not even exist. Of course, it always seems to affect one of the segments that I want to review...
Missing archive recordings
- Thread starter wpappas
- Start date
Looking at your feed - it's been briefly up and down quite a bit over the past 2 weeks. Possible that your internet dropped and caused that archive to not record.... it's pretty common and the most likely explanation.
Thank you for the reply. Most of the "blips" this past week were due to system reboots for software upgrades and configuration changes. It is possible that is what caused the missing segments, but I did not think that any of the feed breaks lined up with that. I will keep a closer watch on it over the coming days to see if another segment disappears.
Hi Lindsay -- I am seeing several more instances of missing archives. At least one yesterday and several today. In all the cases I have looked at my internet connection was solid during the periods dropped. If it was an issue with Asterisk failing to send audio then I would expect to see a blank archive vs. a missing file since I have many such instances during quiet times. Not sure what is going on here.
What is Asterisk?
The software the underlies AllStarLink which is a very widely deployed VoIP network for ham radio.
Ok, got it.
Again, the most likely explanation for archives being missed is that your feed is briefly going down and coming back up. We're not experiencing any issues with the feed archiving service. Because of the way icecast broadcasts are handled and recorded, for a 30 min block to be recorded your feed has to be essentially running non stop without a single hiccup. If your internet connection drops for even 20 secs, it's going to potentially affect the 30 min block.
I'm looking at your logs, your feed in the past few days has come up and down quite a bit. We only log if your feed has been down for more than 15 minutes at a time typically, so you're probably seeing a lot more flapping on your feed than we are logging.
Again, the most likely explanation for archives being missed is that your feed is briefly going down and coming back up. We're not experiencing any issues with the feed archiving service. Because of the way icecast broadcasts are handled and recorded, for a 30 min block to be recorded your feed has to be essentially running non stop without a single hiccup. If your internet connection drops for even 20 secs, it's going to potentially affect the 30 min block.
I'm looking at your logs, your feed in the past few days has come up and down quite a bit. We only log if your feed has been down for more than 15 minutes at a time typically, so you're probably seeing a lot more flapping on your feed than we are logging.
I understand what you are saying, however every single feed interruption that you see was due to deliberate action -- rebooting routers, software upgrades, hardware maintenance, etc. -- so my connection is not nearly as flaky as you may possibly think it is. Furthermore, the interruptions you show in the log above did not actually result in any archive losses so that kind of goes against your theory that a brief interruption would lead to a dropped archive. I have no doubt that your logs are not showing any issues with your feed archiving infrastructure, but something is definitely going on and I would be surprised if I was the only one experiencing this
If there's genuinely something going on with our infrastructure, then I haven't heard so much as a peep from any of the other 7,000+ feed providers.
That's not to say there *couldn't* be an issue. But here's where I'm sitting:
1. I can't see any problem on the surface with our archive infrastructure right now.
2. I haven't had a single report from another feed provider indicating there's an issue.
3. Whenever there *is* a real problem with an archive server, reports flood in .... feed providers come out of the woodwork, and the complaints pile up fast. It's very hard to miss.
Here's the simple reality. A lot of feed providers run gloriously, extravagantly complex home-lab setups ... and they are *constantly* experimenting. DNS, VPNs, failover, multi-homed routing, ad-blocking, VLANs, a Raspberry Pi doing something nobody quite remembers configuring, a pfSense box that achieved sentience sometime last spring. Our feed broadcasting and archiving infrastructure, on the other hand, is about as rock-solid, boring, and architecturally sound as it gets, because it's the exact same infrastructure and architecture running in production for 7,000+ feeds and millions of listeners for almost 15 years.
So the honest math: 99% of the time it's something on your end, and roughly 78.5% of the time I literally cannot diagnose it, because I can't be physically present on your local network to troubleshoot the multi-homed, DNS-rewriting, VPN-tunneled, ad-blocking, triple-failover, super-double-secret home lab you've lovingly assembled and are actively experimenting on. Let's face it, you are an amateur radio operators Tinkering *is* the hobby. Half of us didn't set up a feed so much as build a small research facility that occasionally streams scanner audio.
And yes, our archive infrastructure is designed to be resilient and to keep archiving through a feed outage. But it's not designed to gracefully recover from every scenario where something highly complex on your end decides, mid-stream, that it's no longer playing nicely. Icecast wants one continuous, uninterrupted connection; anything that quietly nudges that stream, even for a moment, can take out a full 30-minute block.
So my honest suggestion: before we assume it's the mothership, take a hard look at what's between your audio source and us. That's almost always where these issues live.
Lindsay -- I feel like I put you on the defensive and that was not my intent at all. My point was not to disparage your infrastructure but to bring forward the idea that perhaps there may be certain configurations that lead to dropped archives and if we knew what those were we could remediate them. I am just a minuscule amateur radio feed provider so I do not expect you to expend a lot of effort for an issue that you are not seeing. Also, missing archives for me is really just an occasional inconvenience and nothing earth shattering. I just wanted you to know that something is going on and it may not be related to a weak internet connection. End of story, roll credits, fade to black.
Hey man, we're all good, trust me. I'm trying to be light hearted here... I'm just emphasizing that I've run this exact exercise about a thousand times over the years, and it has taught me some hard-won humility about where these issues actually live.
Here's how it usually goes. Someone reports a problem broadcasting a feed. We roll up our sleeves and dig in. I'm pulling logs, checking ingest, comparing timestamps, cross-referencing the archive workers, the whole forensic 9 yards. We're 100 hours deep. I've aged visibly. My family has forgotten what I look like. And then, right at the bottom of the ninth inning, the feed provider goes:
"Oh, found it. Turns out I was tunneling the stream through a VPN, over a satellite link, that I was bouncing off the actual moon, and the VPN jitter rate exceeds MP3 streaming tolerances during lunar apogee."
And I'm just sitting here staring at the screen going JFC, WTF, LOL.
So that's all I'm saying. It's not that your issue isn't real. It's that in my experience the culprit is almost always some gloriously creative bit of engineering sitting quietly between the audio source and us, doing something nobody would ever think to suspect until hour 99. If you ever do pin down a specific config that reliably kills a segment, I genuinely want to hear it, because that's the kind of thing we can actually remediate. Until then, my money is on the moon.
Also, maybe try a different broadcast client. I know absolutely nothing about this Asterisk broadcast client. For all I know it's bouncing off the moon,
It's all good on my end as well. The feed "plumbing" is nothing esoteric. No VPN, just a direct internet connection though a router than does NAT -- like most systems I'd guess. I feed my audio to Broadcastify via a simple ffmpg script as this seems to be the common way for people running Linux. I can tell that ffmpg is not losing its connection to Broadcastify because it will fail and restart if it does and I did not see any restarts during the missing archive periods. I guess it could be something to do with how ffmpg is sending the audio frames, but there does not seem to be discernable pattern to it. Just one of those things, I guess.
Similar threads
- Locked
