Beta Testers Wanted: Call Ingest Project

[blantonl]

Hi Folks,

I'm looking for one or two beta testers to help me test a new architecture I'm putting together starting with Trunk Recorder (we'll support others later). I preferably would like to have two Project 25 busy trunking systems that we'll ingest all calls into. The primary testing is for the back end infrastructure (all storage, metadata storage, message queuing etc). I'll soon provide a rudimentary front end interface but I'm now beta testing the back end infrastructure to support billions of calls with tens of thousands of consumers. That back end is put together ready to be tested and fine tuned.

beta testers should:
1) Be fully versed in Trunk Recorder robotastic/trunk-recorder
2) Have a machine ready to upload calls to Broadcastify
3) Be available to assist in troubleshooting their local setup

I'll provide the beta tester an upload script to run, and API key and system entry. All uploaded calls may be deleted, removed, edited etc during the beta period.

This is going to be a really neat project, as it will be tightly integrated with the existing RRDB. In addition to ultimately having a fully featured front end interface to listen to calls live, we'll be able to do things like click on a talkgroup in the RRDB and bring up all calls archived and live, set email alerts for new talkgroups seen on a system and listen to the calls to identify, reporting on talkgroups and call users, dynamically create Broadcastify feeds from multiple systems and talkgroups, among other extremely innovative things. This is going to be an exciting project.

I'm looking for one or two folks that Beta test a busy system so I can fine tune the back end. Email me at lindsay [at] radioreference.com and let me know if you are interested. Tell me which system you can monitor, the hardware you plan to use, and your experience level with trunk recorder.

Thanks!

 

[DSheirer]

Are you going to have an upload API, or will this be locked to trunkrecorder?

It would be easy enough to modify other applications to support if you can expose an endpoint.

Denny

 

[blantonl]

Denny, this will not be locked to any one application. It will support any architecture that can generate individual calls and metadata for each call. Trunked, conventional, etc.

it is a fully functional api that you call, and a one time use upload URL will be returned for that individual call, which would be called by the client to upload the call audio.

 

[maus92]

I could build / or re-use a box that monitors a busy Phase 2 simulcast site. I'd have to learn TrunkRecorder though. Might take a few days to set up.

 

[boatbod]

it is a fully functional api that you call, and a one time use upload URL will be returned for that individual call, which would be called by the client to upload the call audio.

What is your definition of "individual call"? Are we talking about each single radio transmission?

 

[blantonl]

What is your definition of "individual call"? Are we talking about each single radio transmission?

That is correct. For Trunk Recorder, as you know, that might be a number of sequential related calls, but we are processing that metadata on our end.

 

[MtnBiker2005]

You can see what TrunkRecorder does on sites like this one.
https://openmhz.com/systems

 

[NebraskaCoder]

I planned on starting a similar programming project (obviously not linking into RR) but hadn't had time to dive in as I wanted to modify the source code to add features and compatibility. I wish you luck!

 

[krokus]

I could get up to speed on trunk recorder, and provide traffic from an MPSCS site with a zone controller.

 

[TampaTyron]

Nice project. I can see an interesting traffic analysis use case related to enumerating users who are on encrypted talkgroups. TT

 

[blantonl]

Beta is coming along nicely.

Calls System Status - If you look at talkgroups for a system, entries in Yellow are unidentified talkgroups that don't match anything in the RRDB. This is going to be really helpful to ID talkgroups, as I've already use this to do so. See an unidentified talkgroup? You'll be able to go listen to the past history and figure out who it is!

I haven't written any interface yet to listen to the calls, but the back end infrastructure is coming along nicely. Right now we're wrapping up the following on the back end.

1) Finishing up duplicate call handling, which will allow multiple people to all feed the same trunked system, same or different sites, whatever, and we'll keep one entry of a specific call. I could literally see a scenario where multiple people are feeding calls from the same trunked system, backing each other up in cases of outages.

2) Finishing up a "reject calls" function. If you want, you can specify groups not to store at the server level and we'll just reject them - this way providers can just send an entire system up to us and manage what gets stored or not.

3) Planning advanced statistics. Systemwide hourly and daily stats, talkgroup hourly and daily status, src ID hourly and daily status. What talkgroups a src ID has participated on. etc. Lots of good analytical tools coming here.

4) Finalizing the upload scripts and a prerequisite checker script - that will be done today and then I'm bringing aboard two beta testers to start feeding their systems. The systems you currently see are ones I'm personally feeding.

5) Currently only storing 24 hours of audio data - but that will be increased to 90 days when we go to production.

Fun times!

 

[AB5ID]

Any chances of coming up with a menu driven easier to configure package of trunck recorder for those us who are less linux savvy or an sd card image for raspberry pie feed providers?

 

[blantonl]

Any chances of coming up with a menu driven easier to configure package of trunck recorder for those us who are less linux savvy or an sd card image for raspberry pie feed providers?

as the project get's further down the path, we'll definitely be looking for ways to better below equipment to capture systems - up and to prepackaged capture solutions for sale/build/etc

 

[mtindor]

I realize this is all in its infancy, but it would be nice if some of the testers / call ingest providers (and you, Lindsay) would let us know what hardware you are using, what type of receiver device(s), how many devices, and how many simultaneous talkgroups you guys are recording. Of course, really helpful would be details of what minimum hardware specs should be considered.

In my case, I'd love to do this for Stark Co Ohio MARCS site. 17-channel site, split 700/800 mhz. Of course, during any given day, only about half the channels are in use, and quite often by ENC transmissions or public works stuff that would never get recorded.

Anyway, is anyone willing to share system specs on what they are using to record/upload heavily laden trunked sites?

Mike

 

[blantonl]

Some rough stats

1) A "regular" dedicated PC should be powerful enough to ingest and upload a very large busy system with no problem, and could probably do more than one busy system
2) A Raspberry Pi 3 can handle a small P25 site fine
3) A Raspberry Pi 4 can handle a medium size P25 site fine

How many SDR devices you need depends on how the system's channels are distributed in bandwidth.

A single RTL Stick can cover 2MHz of bandwidth
A airspy mini can cover 6 MHz of bandwidth
A airspy regular can cover 10 MHz of bandwidth

I've got a 10 year old Gateway PC I bought at Costco with a Core 2 quad Q8300 Processor and 4 GB of RAM, 3 RTL sticks and 1 Airspy mini... sending 4 "Medium" P25 Trunked Sites and 5 Conventional P25 channels up right now and it rarely sees overruns. I also have a Raspberry Pi 3 that is sending a small single P25 site and it's about at capacity (1 control and 3 voice channels)

A latest and greatest i7 or i9 processor could probably chew through every system you could receive local to you with no problem.

 

[blantonl]

Status update:

1) We brought our first beta tester aboard sending a very large system to us. Working extremely well so far.
2) I wrote a rudimentary interface to listen to individual calls based on the last hour and last call. No "automatic playing" yet, we're still tweaking the back end infrastructure, message queueing and triggers etc. To support billions of calls and 1000's of consumers, I want to make sure we're architecting things right the first time.

Today I'll be working on writing a script to start ingesting a Trunking Recorder setup (NOT trunk-recorder - different). This will allow me to ingest my local EDACS ProVoice system into the infrastructure.

I'll then resume adding additional queries for stats and calls etc.

Stay tuned!!!

 

[blantonl]

The ingest project is coming along nicely. It's now tightly integrated into the RRDB. You can now literally click on a frequency or talkgroup ID in the RRDB and be taken directly to a listing of all calls on that freq or tg over the past 24 hours. If a talkgroup ID or Frequency is displayed in BOLD, you'll be able to review calls for that talkgroup. It will be in BOLD and available if 1) We've received calls in the past 24 hours for that tg/freq and 2) It's tagged with a tag that is within Broadcastify's terms of service (only dispatch etc)

Examples:

https://www.radioreference.com/apps/db/?sid=6643

Montgomery County, MD (P16) Trunking System, Various, Maryland

Montgomery County, MD (P16) Trunking System Profile

www.radioreference.com

San Antonio/Bexar County Trunking System, San Antonio, Texas

San Antonio/Bexar County Trunking System Profile

www.radioreference.com

Greater Austin/Travis Regional Radio System (GATRRS) Trunking System, Austin, Texas

Greater Austin/Travis Regional Radio System (GATRRS) Trunking System Profile

www.radioreference.com

Comal County, Texas (TX) Scanner Frequencies and Radio Frequency Reference

Scanner Frequencies and Radio Frequency Reference for Comal County, Texas (TX)

www.radioreference.com

 

[blantonl]

I think we're ready now to take on a few more ingestors... if you are interested in sending Trunk Recorder traffic to us reach out to me at lindsay at radioreference.com and I'll get you the scripts and the prereqs to get started. It can very easily run alongside any other trunk recorder work you are doing with minimal impact. And the integration with the RRDB is working nicely.

I should have live traffic monitoring and talkgroup playlist building processes ready to go in the next two weeks.

Right now we're storing everything for 24 hours, but my plan is to increase that to 7 days here shortly on the way to a full archive of everything for 90 days.