Broadcastify Calls Node Skew Problems

[blantonl]

UPDATE 7/1/2023 - We are now going to start blocking calls from Broadcastify Calls Nodes that have a node skew of greater than 15 seconds, or less than -15 seconds.

Node Skew represents the difference in time between when a call was captured on a Broadcastify Node and when it was received by the system. Typically, when the skew or time difference is greater than 15 seconds, this indicates that the time set on the machine capturing the call is set incorrectly, or significant network or hardware processing latency exists between the capturing node and the Broadcastify Calls systems.

It is absolutely critical that the time set on your Broadcastify Calls Node is synchronized to a correct accurate time source on the Internet.

Broadcastify Calls Providers can view the current Node Skew for your system by going to your calls management page at:

https://www.broadcastify.com/calls/manage/

Click the "Manage Node" icon for your node:



The current node skew will be listed in the Node Statistics table on the top left.



Calls providers with node skews greater than 10 seconds or less than -10 seconds should verify that the time set on their calls ingest systems are synchronized properly to accurate time sources on the internet.

If time is properly synchronized on the system, and skews are still reporting more than 10 seconds, it is possible that your calls node is not powerful enough to support capturing and uploading the system being monitored. Consideration should be given to upgrading to a more power capturing system.

SDRTrunk ingest nodes can be particularly sensitive to requiring a more power machine if you are capturing a large system or captures multiple systems at the same time.

Some guides below to help you get your time synchronized correctly:

Windows 10

How to force Windows 10 time to synch with a time server?

Linux

How to Install NTP Server and Client(s) on Ubuntu 18.04 LTS

Raspberry Pi

Sync Time from Internet - Raspberry Pi Forums

 

[GTR8000]

 

[jjbllitz]

How are you calculating skew? I'm using Trunk Recorder with a fairly robust system monitoring about 7 frequencies on one tower, and I'm seeing calls rejected with skews calculated at over 60 seconds?????? My machine is synced using NTP and is within a second of atomic time when I check it.

I'm wondering if the fact Trunk Recorder combines several transmissions into a single file before it sends them to Calls is effecting the skew calculation somehow.

My current "average" skew is 5+ seconds and I've always wondered why it seemed that high. Watching the application, when a transmission is complete (actually could be multiple transmissions together), the file shows uploaded almost immediately,

 

[blantonl]

Call skew is the time difference between when the call is captured on your system and when it is received by our system.

We had a problem on friday where we added some metadata to the call rejection error messages, which caused trunk recorder to try to retry sending some duplicate calls, which then probably got blocked as having skews too large.

You should be OK now for your node.

 

[jjbllitz]

Aah. The retries is probably why I was seeing the high skews. I'm not home this week but will check things when I get back.

 

[georgew0819]

I'm seeing this error - Broadcastify calls API upload URL request failed [1 REJECTED-CALL-SKEW-TOO-LONG--86391.006] when trying to upload to node 514. Not really sure how this happened but in order to correct this I rebooted my computer, cleared out the recordings in the SDRTrunk recordings file and restarted SDRTrunk. The skew is now showing 11.618 seconds but I'm still unable to upload to the node. How do I get the node reset so I can resume feeding calls.

 

[blantonl]

I'm seeing this error - Broadcastify calls API upload URL request failed [1 REJECTED-CALL-SKEW-TOO-LONG--86391.006] when trying to upload to node 514. Not really sure how this happened but in order to correct this I rebooted my computer, cleared out the recordings in the SDRTrunk recordings file and restarted SDRTrunk. The skew is now showing 11.618 seconds but I'm still unable to upload to the node. How do I get the node reset so I can resume feeding calls.

We evaluate every call on whether or not it gets rejected or not. There’s no reset to occur. It’s possible you still have aged calls tying to be sent, or the time set on your machine is off

 

[lynchy135]

Call skew is the time difference between when the call is captured on your system and when it is received by our system.

Is captured when the call is finished or when it starts? In particular, if a call is 15 seconds long would we expect it to be rejected because its over skew? I am assuming no, but just want to clarify.

 

[blantonl]

Is captured when the call is finished or when it starts? In particular, if a call is 15 seconds long would we expect it to be rejected because its over skew? I am assuming no, but just want to clarify.

No, the timestamp is when your system finishes capturing the call.

Edit: A call lasting over 30 seconds or more would not be rejected. You can see that obviously for any system that has calls lasting a long time.

 

[fp104]

Hi. I too am having a large number of calls not uploading. My skew time is 9.15 seconds. I am running trunk-recorder on RPi4 with 4 CPU and 8 GB RAM. I was previously using SDRTrunk on Windows but the audio wasn't acceptable. I've checked my time and it is accurate, confirmed with NTP. Is the RPi4 not powerful enough? Other areas I can look at?

Thank you

 

[blantonl]

Hi. I too am having a large number of calls not uploading. My skew time is 9.15 seconds. I am running trunk-recorder on RPi4 with 4 CPU and 8 GB RAM. I was previously using SDRTrunk on Windows but the audio wasn't acceptable. I've checked my time and it is accurate, confirmed with NTP. Is the RPi4 not powerful enough? Other areas I can look at?

Thank you

A lot of duplicates are being rejected for your node. Is there another node that covers that same site? If so, the system is simply voting between the other coverage. Perfectly normal.

Also, yes, an RPi4 can be underpowered for Trunk Recorder except for the smallest of systems or sites

 

[fp104]

A lot of duplicates are being rejected for your node. Is there another node that covers that same site? If so, the system is simply voting between the other coverage. Perfectly normal.

Also, yes, an RPi4 can be underpowered for Trunk Recorder except for the smallest of systems or sites

Thank you for the information. There are a couple of other nodes.

 

[blantonl]

Thank you for the information. There are a couple of other nodes.

Keep it running through. That's the beautify of broadcastify calls - the more nodes the better because remember these node providers are all volunteers anything could take their node offline and there are others to back it up.

Don't be discouraged by rejected duplicate calls.

On the "regular" feeds - we lose and gain about 10-15 feeds a DAY. You could be potentially the only person providing coverage next week. People come and go as volunteers.

 

[fp104]

Keep it running through. That's the beautify of broadcastify calls - the more nodes the better because remember these node providers are all volunteers anything could take their node offline and there are others to back it up.

Don't be discouraged by rejected duplicate calls.

On the "regular" feeds - we lose and gain about 10-15 feeds a DAY. You could be potentially the only person providing coverage next week. People come and go as volunteers.

I'm keeping it running for sure.

 

[slvp25]

I was troubleshooting another issue and I set a "sleep 30" into my upload script (in the assumption that the broadcastify calls api upload would fire simultaneously to the upload script). After I got the other issue resolved and I got the files uploading, every single one of them was rejected for skew too long. Always 34+ seconds.

I then took out the sleep 30, and guess what, the skew was 4 seconds.

The upload script delay WILL AFFECT your skew. Any file conversion, wait/sleep timers, or other blocking of any kind will add to your skew. The broadcastify calls API fires off after the upload script.

This is both good and bad. You can do things during that time before the api fires off, but also you shouldn't delay it any more than necessary. I had some "nice" commands to slow down sox (formerly on an RPi 4b, now it's running on a desktop with a core i7 and I have plenty of cores and cpu to spare so I'll be removing those).

Just some insight for those having skew issues. Another place to check.

 

[nick0909]

I've had trunk-recorder running for months in the current setup with no problems, and a lot longer on a previous computer. Late last night I started getting rejected calls for skew too long in the 31-34 range. My time is synced with us.pool.ntp.org and I am just using the default trunk-recorder uploader. No idea what changed but my node is basically offline now because nearly every call gets rejected. I updated trunk-recorder and restarted it, no change. Any thoughts?

--edit
It was the time sync. I am running trunk-recorder in WSL Ubuntu on Windows. Turns out the WSL clock doesn't always match the Windows clock and can drift. I have now set up WSL to also get time from NTP regularly and things are going smoothly again...

 

[belvdr]

Make sure you use at least 3 NTP servers. Four is recommended.

 

[rjdj2000]

Yesterday I noticed that I wasn't hearing anything (have a browser up at work to listen to my calls stream) so I went to check it and found that the skew was quite high. Went and got the time synced on the pc (have never had any issues until now) and was then able to start listening to current calls again. Could there be something added to notify us on this? If it wasn't for me listening at work and noticing this, it probably wouldn't of been corrected. I know there is a notification if it is down which I have set for 2 hours I believe. Not sure if that would do what i am asking or not if it was set lower.

 

[GTR8000]

The solution is to make sure your PC is regularly syncing the time, not to wait until it's way off to get notified. Lindsay posted links explaining how to make the PC sync more frequently, so start with that.

There is also a lightweight software for Windows called Dimension 4 that will keep the time synced up. It's old software and is no longer maintained, but it works fine even with Windows 11. I suggest finding and installing that for the easiest solution.

 

[rjdj2000]

The solution is to make sure your PC is regularly syncing the time, not to wait until it's way off to get notified. Lindsay posted links explaining how to make the PC sync more frequently, so start with that.

There is also a lightweight software for Windows called Dimension 4 that will keep the time synced up. It's old software and is no longer maintained, but it works fine even with Windows 11. I suggest finding and installing that for the easiest solution.

Thanks for the info. Will look into it.