Broadcastify Calls Duplicate Calls Handing Updates

[blantonl]

TLDR:

We're testing a new algorithm for duplicate calls handling on the following systems:

Allied Radio Matrix for Emergency Response (ARMER) - All Calls Live

Trunked System: Allied Radio Matrix for Emergency Response (ARMER) - All Received Calls

www.broadcastify.com

Michigan's Public Safety Communications System (MPSCS) - All Calls Live

Trunked System: Michigan's Public Safety Communications System (MPSCS) - All Received Calls

www.broadcastify.com

Morris County - All Calls Live

Trunked System: Morris County - All Received Calls

www.broadcastify.com

North Texas Interoperable Radio Network (NTIRN) - All Calls Live

Trunked System: North Texas Interoperable Radio Network (NTIRN) - All Received Calls

www.broadcastify.com

Details

I've written a new duplicate call algorithm that aims to eliminate duplicate calls from multiple nodes on the same trunked radio system. The traditional way which we detect and reject duplicate calls is to reject any calls with the same timestamp that come in +/- 2 seconds. That has somewhat worked ok, but it's been subject to the following problems

• If two different calls for the same talkgroup come in within 2 secs, one is rejected. This would result in "quick" short transmissions being rejected as duplicate
• problems between nodes running different software monitoring the same talkgroups (Trunked Recorder vs SDRTrunk etc) - some send conversations vs individual calls
• Node Skews between different nodes - some nodes operate slower than others due to processing power. Some nodes operate on slower internet connections etc. This causes problems with duplicate detection
• Nodes with incorrect set time.

My solution instead, which I am testing on those above listed systems, is far more complicated but addresses each of these issues above. What we are now doing is building a table, in memory, of all nodes that are contributing calls for each specific talkgroup. We then continuously, dynamically rank those nodes based on the quality of the node. That quality ranking is primarily a function of how low the node skew is, and how long the node has been providing calls for the talkgroup. Then, as calls come in from all the nodes, we assign each talkgroup a node that will exclusively provide calls for that talkgroup, all the while constantly evaluating the health and ranking of all the nodes providing the talkgroup. If the system detects that another node is starting to rank higher, we'll start "reconsidering" the currently assigned node, and after a select amount of considerations we'll switch the node to the better node dynamically if it makes sense.

So far, I'm pleasantly surprised how well the new approach is working. Some concerns I have which I'm trying to suss out are how we miss some calls during a node assignment switchover for talkgroups that aren't extremely active, but aren't dead silent either. I'm also seeing duplicates display on our "All Received Calls" for a trunked system and I'm not sure why, because they don't show up for playlists, individual talkgroups etc. This might be due to some filtering I had previously implemented on those individual pages and now we're doing things differently it very well could be a race condition that I need to account for.

Your feedback is welcome. I'll probably implement this on more systems as we progress.

note that there are almost certainly bugs to iron out, but I’m hopeful this is the right approach for us to take.

 

[GTR8000]

Watching these developments with interest...thanks!

 

[HM1529]

I notice a lot of duplicate traffic on the Bucks County PA TRS when using Calls. I'll welcome the update if/when it gets there.

 

[blantonl]

One of the problems I think that has been exposed with this new algorithm is it seems like we have some SDRTrunk users who are monitoring multiple sites with their setup but uploading as a single site in Broadcastify Calls - which is resulting in us getting 4-5 duplicate uploads from these offending nodes.

I'm not intimately familiar with SDRTRunk. Is this possible? Does SDRTrunk provide any duplicate handling on their end?

 

[mtindor]

One of the problems I think that has been exposed with this new algorithm is it seems like we have some SDRTrunk users who are monitoring multiple sites with their setup but uploading as a single site in Broadcastify Calls - which is resulting in us getting 4-5 duplicate uploads from these offending nodes.

I'm not intimately familiar with SDRTRunk. Is this possible? Does SDRTrunk provide any duplicate handling on their end?

It is possible and is happening I'm sure. @blantonl see attached

 

[GTR8000]

SDRTrunk should be weeding out the dupes before they even leave the PC, but the option must be enabled in the software first, as Mike's post shows. Default is off iirc

@DSheirer

 

[mtindor]

It is possible and is happening I'm sure. @blantonl see attached

Lindsay, keep in mind that even if one users SDRTrunk deduplication, one can conceivably configure SDRTRunk to monitor/stream 10 distinct sites from the same system, a couple DMR sites, etc, all with unique System Names (because you can call each system whatever you want), AND have it stream ALL of those up to one Broadcasify Calls node. Obviously, in worst case scenario, you'd have mixed comms / talkgroups from various disparate systems showing up on that particular call node. That's in addition to the answer to the original question, which is that you certainly can have 5 sites of a particular system set up and instruct them all to stream up to one Broadcastify Calls node. Supposedly, if those 5 sites are configured with the same exact system name, some deduplication can occur.

Basically you set up an Alias List that is a list of the talkgroups on the system. Then, you can assign that alias list to any configured system. Then you can designate which talkgroups out of that configured system should be streamed.

If I have AEP P25 1F6-26.26 configured and Ohio MARCS Site 3.1 configured, in the streaming section of SDRTrunk I could conceivably tell it that I want to stream all AEP P25 talkgroups heard on 1F6-26.26 and all Ohio MARCS talkgroups heard on Site 3.1 to the same Broadcastify Calls node

Mike

 

[mtindor]

SDRTrunk should be weeding out the dupes before they even leave the PC, but the option must be enabled in the software first, as Mike's post shows. Default is off iirc

@DSheirer

Good to know that the handling of dupes if off by default. I wasn't sure what the default was. My screenshot just showed how mine is configured. I guess I must have enabled the handling of dupes manually.

 

[GTR8000]

It's been ages since I've worked with a fresh install, so I could be wrong, but I seem to recall needing to enable it back in the day.

So under normal circumstances, enabling de-dupe in the software, then feeding multiple sites of the same system to a single node should not cause havoc on the BCFY side. Of course if someone wants to be a dick, they can do something extreme like you laid out to cause chaos.

 

[blantonl]

Given the above I'm going to weed out duplicates at the node level in this algorithm. Looks like I'll need to reach out to some of these SDRTrunk users who are presumably capturing 38 different sites and feeding it up as one node without the dup detection feature enabled. 🙄

That's the one danger with a platform for power users supported by power users. They'll do some of the wildest stuff and then be like "what? should I not have done that?"

It's also common for someone to contact me and say something like "My calls node stopped sending calls and is getting errors, nothing has changed on my end."

Me: "Those error logs indicate that something on your internal network is blocking outbound requests to our servers, did you enable a firewall or something?"

Them: "oh yeah, well, I did switch to a new ISP cable modem, and I also installed Starlink as a backup with a new router, and I decided to segment all my radio equipment on a dedicated VLAN tagged specifically for iOT devices, and on that I installed a firewall that blocks x, y, and z, and does packet shaping for maximum performance, but also blocks all outbound Port 80 http access unless I whitelist it - it does an auto-whitelist based on heuristics and the firewall also blocks porn from 8am-5pm but if at night when the port blocker is off (for my "special site that I visit at night") the AI engine might have categorized a broadcastify calls node as malicious due to some of the talkgroup names. I disabled all that and that fixed it. Thanks. "

Me: "🙄"

 

[blantonl]

Also, I've now enabled this alog systemwide across all systems.

 

[GTR8000]

That's the one danger with a platform for power users supported by power users. They'll do some of the wildest stuff and then be like "what? should I not have done that?"

Guilty as charged, but I've had de-dupe enabled since Day 1 at least! 😅

 

[blantonl]

GTR: How are things looking on the systems you monitor, use, feed to etc?

 

[hruskacha]

@blantonl have you ever thought about making Broadcastify Calls ingest Control Channel data on major systems like P25 (Michigan MPSCS)?

Not all trunked networks have usable info in the control channel, but P25 CC data has system level time clock, TGID, RID, patch data, LCN, Frequency, Bandplan, and more.

Imagine the potential if Broadcastify / Radio Reference could build/update their database and algorithms just by sampling CC data from compatibile BCFY Calls nodes?

I have a zip folder with examples of the ~5 log types that SDRTrunk can save to text file. I can provide screenshots of it too, just let me know if you are interested.

 

[GILLIG40]

I’m just a listener. MPSCS sounds very much improved today over last night. Last night some transmissions were being repeated 3-4 times and others were being cut off. None of those issues are present right now. Thank you

 

[Napalm]

I might actually start listening to my own node again if this has fixed the missing short calls issue. I had to switch to OpenMHz just to not having missing short calls.

 

[blantonl]

I might actually start listening to my own node again if this has fixed the missing short calls issue. I had to switch to OpenMHz just to not having missing short calls.

I really really do not recommend listening to an individual node. If someone else provides a node on the same system you run the risk of missing calls because another node out-voted yours.

I am actually going to remove the ability to listen to nodes here soon because it’s going to cause too much confusion.

you should be using playlists to listen on Broadcastify calls

 

[mtindor]

I really really do not recommend listening to an individual node. If someone else provides a node on the same system you run the risk of missing calls because another node out-voted yours.

I am actually going to remove the ability to listen to nodes here soon because it’s going to cause too much confusion.

you should be using playlists to listen on Broadcastify calls

That would sort of be a bummer. I rarely listen to mine, but if I do I'm comparing performance of my feed with another feed of the same system by another user. At some point, if i thought the "other" provider was outvoting my feed all the time, I'd probably just take it down and use it for something else. After all it's not very good if it isn't participating. So I like monitoring my own feed and comparing it to "the other guys" feed. If mine is handling significantly more calls than his, I'm happy. If his are handling majority of calls, then mine is useless (unless his goes down).

Mike

 

[Napalm]

I really really do not recommend listening to an individual node. If someone else provides a node on the same system you run the risk of missing calls because another node out-voted yours.

I am actually going to remove the ability to listen to nodes here soon because it’s going to cause too much confusion.

you should be using playlists to listen on Broadcastify calls

It fixed the issue I was having.

I see your point though. Right now I'm the only guy in town aside from another node (3030) who just uploads three talkgroups to the system. If that changes I'll lose calls.

I'll have to transition "my" listeners onto using playlists soon LOL. It's been easy just giving them a link to my node.

Thanks for your work on this problem, from listening all night it has definitely ended the missing calls.

 

[Napalm]

That would sort of be a bummer. I rarely listen to mine, but if I do I'm comparing performance of my feed with another feed of the same system by another user. At some point, if i thought the "other" provider was outvoting my feed all the time, I'd probably just take it down and use it for something else. After all it's not very good if it isn't participating. So I like monitoring my own feed and comparing it to "the other guys" feed. If mine is handling significantly more calls than his, I'm happy. If his are handling majority of calls, then mine is useless (unless his goes down).

Mike

That's the point though - as long as his is of the same quality as yours, yours or his fills in the missing gaps/downtime.