OP25 Raspberry PI4 + Airspy R2 + OP25 / Trunk Recorder

Status
Not open for further replies.

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
I've been running an Airspy R2 @10 msps on a Raspberry PI 4 with a one-stream OP25 setup with audio sent over UDP to another PC running VLC. The audio sounds great.

I've also run the Airspy R2 @ 2.5 msps with OP25 and it's working great like that as well.

Airspy R2 @ 2.5 msps
- no cores running over 20%
- load average 0.45 - 1.4

Airspy R2 @ 10 msps
- each core running between 25-90%
- load average usually about 3.3

Clearly when one runs at 2.5 msps it's a sacrifice if one needs to monitor a full band. But the performance is so much better on the RPI.

Now, if I attempt to run Trunk Recorder with just one stream, I can't get it to record any audio whatsoever. And, unfortunately, I canno get Trunk Recorder to run the Airspy at 2.5 msps. It indicates there are two sample rates supports, 2.5 msps and 10 msps. If I run at 10 msps it fires up and works, but never records any audio. It tries, and even when it is attempting to record, I get 44-byte audio files. I imagine it's because the PI4 is so overloaded at 10 msps.

If I switch trunk recorder to 2.5 msps on the Airspy, the decode rate is so abysmal that it continuously switches control channel, never captures the system, etc.

So I'm curious if anyone is running Trunk Recorder on an RPI4 with 100% using an Airspy at 2.5 msps. And if so, are you willing to share your config.json ?

Thanks

Mike
 

TippyTurtle

Member
Joined
Dec 10, 2019
Messages
5
I apologize...I'm very new. I have hacked my way through getting trunk-recorder running pretty reasonably...but not for OP25...my system is rebanded 800 motorola smartnet...all analog audio. I am running the whole mess on a single RPI4 and publishing the recordings up on OpenMHZ. I don't know whats up with the CPU, but I get great recordings of about 50 Talkgroups (of 550+ on the Control Channel)...often recording 6 audio channels at a time...almost always 3 or 4 simultaneously during primetime...while running great xscreensavers (never sleep). I should add I have nearly perfect cooling, I have: MazerPi "all heatsink" case. Overheat will choke the RPI cpu clock down to nothing, your cool right?

The system I an listening to is:
EPSCA Simulcast Site Details (Seattle-King County Public Safety (KCERS))
...There are a lot of frequencies NOT in that table that show up, so I have had to adjust my "Centers" a lot to get it all the audio with 3 SDR's. I started with real CRAP SDR's ($7 ebay), I have switched over to the $25 NooElec NESDR, very worth it: No Error/Drift anymore. Some of my cheap SDR's had error like 480000 (480khz) and had to update the setting often because of terrible drift. These NooElec don't seem to drift at all.

The only config help I can suggest is: Are you sure you are centered on the control channel you think you are on? In my area there are so many control channels and my crap SDR's had such bad error, I had the wrong config error the first few times. (The symptom was trunk-recorder trying to record many broadcasts, but only writing 44 byte empty files.) The fact that the published (main) control channel was non-existent and only one of the 4 Alternate control channels actually had data on it had me to be stumped initially. I did a lot of math in a spreadsheet with readings of control channels from the 3 different (crap) SDR's to triangulate what pair (looked at Delta) control channels (of the ~40 that are part of my larger area: KCERS) I was actually looking at. I used gqrx to track this all down.

For what it is worth here is my config:

Code:
{
    "sources": [{
        "center": 851512500,
        "rate": 2048000,
        "squelch": -65,
        "error": 0,
        "digitalRecorders": 4,
    "gain": 10,
        "modulation": "fsk4",
        "analogRecorders": 3,
        "driver": "osmosdr",
        "device": "rtl=41"
         }, {
        "center": 853825000,
        "rate": 2048000,
        "squelch": -65,
        "error": 0,
        "digitalRecorders": 4,
    "gain": 10,
        "modulation": "fsk4",
        "analogRecorders": 3,
        "driver": "osmosdr",
        "device": "rtl=42"
         }, {
        "center": 856812500,
        "rate": 2048000,
        "squelch": -65,
        "error": 0,
        "digitalRecorders": 3,
    "gain": 10,
        "modulation": "fsk4",
        "analogRecorders": 2,
        "driver": "osmosdr",
        "device": "rtl=43"
}],
    "systems": [{
        "control_channels": [852887500,854237500,856787500,851437500],
        "type": "smartnet",
        "bandplan": "800_reband",
        "talkgroupsFile": "KCERS-BellevueTalkGroups.csv",
    "hideEncrypted": false,
        "recordUnknown": false,
    "talkgroupDisplayFormat": "id_tag",
    "hideUnknownTalkgroups": true,
    "minDuration": 0.2,
        "shortName": "kcers2",
    "apiKey": "a0d3458a9cf877589831596f5bef033d1866fb79ff9cd08630c0929c7894a18c",
    "audioArchive": false
    }],
    "uploadServer": "https://api.openmhz.com",
    "defaultMode": "analog",
    "captureDir": "/home/tippyturtle/trunk-recorder/recordings",
    "statusAsString": true,
    "controlWarnRate": 2,
    "logFile": false,
    "frequencyFormat": "mhz"
}


Hope this helps in some way.
Todd
 

TippyTurtle

Member
Joined
Dec 10, 2019
Messages
5
...I should add, sometimes I run 3 analog recorders per SDR (9 total simultaneous recordings) and it is listenable, but the recordings (at times, when 3x recorderings on a single dongle) sound sped up...totally intelligible, but like 2x speed.

I am 50/50 on leaving it at 3 vs 2. I only do 2 on the SDR that has the control channel on it.

Do I want to get nearly 100% of the broadcasts, but sped up at times?...or only get clean recordings for the TG's I give priority?

Todd
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
...I should add, sometimes I run 3 analog recorders per SDR (9 total simultaneous recordings) and it is listenable, but the recordings (at times, when 3x recorderings on a single dongle) sound sped up...totally intelligible, but like 2x speed.

I am 50/50 on leaving it at 3 vs 2. I only do 2 on the SDR that has the control channel on it.

Do I want to get nearly 100% of the broadcasts, but sped up at times?...or only get clean recordings for the TG's I give priority?

Todd
The "speed up" sounds like you are dropping/losing samples.
 

KA1RBI

Member
Joined
Aug 15, 2008
Messages
799
Location
Portage Escarpment
Now, if I attempt to run Trunk Recorder with just one stream, I can't get it to record any audio whatsoever.

(snipped)

If I switch trunk recorder to 2.5 msps on the Airspy, the decode rate is so abysmal

(snipped)

Mike

Mike

Could you grab a "top" display of the system when it's struggling under load ....... The command is
Code:
top -H -p xxx

(change xxx to the process PID of the running OP25 or trunk recorder process). The command "ps ax" should display all active processes including the PID. Also in OP25, the "-p" (pause) option will cause the PID to be displayed at startup.

Once the top window has been displayed you can use the "T" key to sort the threads based on CPU usage. What we are interested in knowing are which threads are the CPU hogs...

Max
 

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
Mike

Could you grab a "top" display of the system when it's struggling under load ....... The command is
Code:
top -H -p xxx

(change xxx to the process PID of the running OP25 or trunk recorder process). The command "ps ax" should display all active processes including the PID. Also in OP25, the "-p" (pause) option will cause the PID to be displayed at startup.

Once the top window has been displayed you can use the "T" key to sort the threads based on CPU usage. What we are interested in knowing are which threads are the CPU hogs...

Max

Hi Max,

Airspy @10 msps

15356 pi 20 0 476324 144196 24824 R 87.0 3.6 1:05.44 fft_filter_ccc3
15387 pi 20 0 476324 144196 24824 R 70.1 3.6 0:48.73 recorder
15368 pi 20 0 476324 144196 24824 R 50.2 3.6 0:14.18 fft_filter_ccc1
15354 pi 20 0 476324 144196 24824 S 16.3 3.6 0:13.47 airspy_source_c

Airspy @ 2.5 msps (and unable to capture control channel)

16171 pi 20 0 1011504 143580 22476 S 16.9 3.6 0:08.38 fft_filter_cc13
16251 pi 20 0 1011504 143580 22476 S 13.9 3.6 0:07.09 recorder
16170 pi 20 0 1011504 143580 22476 S 6.3 3.6 0:02.81 fix_cc4
16169 pi 20 0 1011504 143580 22476 S 5.3 3.6 0:02.63 airspy_source_c
16173 pi 20 0 1011504 143580 22476 S 4.3 3.6 0:02.26 fft_filter_cc13

Just to be clear -- I have no problems with OP25. I just can't get Trunk Recorder running worth a damn on the Airspy @ 2.5 msps, and if I run the Airspy @10 msps its pretty much using up all of the resources with just one digital recorder configured in Trunk Recorder.

It isn't a big deal - I'm building an I5-9400 box right now that I'm going to run this all on. My curiosity was whether any of the 100% success stories that people have been raving about with Trunk Recorder on a PI4 were from people who were actually running Airspy R2s @10 msps LOL.

Mike
 

KA1RBI

Member
Joined
Aug 15, 2008
Messages
799
Location
Portage Escarpment
OK Mike this is interesting... To be able to make sensible comparisons it would be good to get similar numbers for OP25; also, are you able to determine (using top, say) whether trunk recorder CPU utilization really is significantly different than that of OP25 for any given sampling rate?

Max
 

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
OP25 and Airspy @10 msps

6736 pi 20 0 472808 207492 48856 R 62.0 5.2 0:40.67 python
6723 pi 20 0 472808 207492 48856 S 26.3 5.2 0:16.26 fir_filter_ccc1
6721 pi 20 0 472808 207492 48856 R 19.7 5.2 0:13.05 airspy_source_c
6724 pi 20 0 472808 207492 48856 S 15.0 5.2 0:08.15 multiply_cc10
6722 pi 20 0 472808 207492 48856 S 11.7 5.2 0:06.32 fix_cc4
6729 pi 20 0 472808 207492 48856 R 11.3 5.2 0:06.80 gardner_costas1
6726 pi 20 0 472808 207492 48856 S 10.0 5.2 0:06.09 pfb_arb_resamp1
6725 pi 20 0 472808 207492 48856 R 9.3 5.2 0:05.26 fir_filter_ccf1

OP25 and Airspy @2.5 msps

6942 pi 20 0 472552 207860 48972 S 14.0 5.2 0:03.42 python
6929 pi 20 0 472552 207860 48972 S 6.0 5.2 0:01.44 fir_filter_ccc1
6940 pi 20 0 472552 207860 48972 S 6.0 5.2 0:00.82 p25_frame_asse2
6927 pi 20 0 472552 207860 48972 R 4.7 5.2 0:01.17 airspy_source_c
6935 pi 20 0 472552 207860 48972 S 4.0 5.2 0:00.92 gardner_costas1
6930 pi 20 0 472552 207860 48972 R 3.0 5.2 0:00.80 multiply_cc10
6928 pi 20 0 472552 207860 48972 S 2.7 5.2 0:00.58 fix_cc4
6932 pi 20 0 472552 207860 48972 S 2.7 5.2 0:00.64 pfb_arb_resamp1
6931 pi 20 0 472552 207860 48972 S 2.3 5.2 0:00.52 fir_filter_ccf1
6933 pi 20 0 472552 207860 48972 S 2.3 5.2 0:00.55 fir_filter_ccf1

recorder (Trunk Recorder) and Airspy @10 msps

7498 pi 20 0 476324 144500 25132 R 86.0 3.6 0:18.60 fft_filter_ccc3
7526 pi 20 0 476324 144500 25132 R 68.7 3.6 0:14.58 recorder
7510 pi 20 0 476324 144500 25132 R 53.3 3.6 0:08.55 fft_filter_ccc1
7496 pi 20 0 476324 144500 25132 R 17.0 3.6 0:03.78 airspy_source_c
7497 pi 20 0 476324 144500 25132 R 12.3 3.6 0:02.91 fix_cc4
7509 pi 20 0 476324 144500 25132 S 11.0 3.6 0:02.15 copy23
7527 pi 20 0 476324 144500 25132 R 9.0 3.6 0:01.80 recorder
7525 pi 20 0 476324 144500 25132 S 5.7 3.6 0:01.27 optimize_c3
7500 pi 20 0 476324 144500 25132 S 5.0 3.6 0:01.03 fft_filter_ccf3

recorder (Trunk Recorder) and Airspy @2.5 msps

7267 pi 20 0 472804 141604 22456 S 15.6 3.5 0:03.31 fft_filter_ccc3
7295 pi 20 0 472804 141604 22456 R 13.6 3.5 0:02.97 recorder
7265 pi 20 0 472804 141604 22456 S 5.0 3.5 0:01.12 airspy_source_c
7269 pi 20 0 472804 141604 22456 S 4.3 3.5 0:00.97 fft_filter_ccf3
7266 pi 20 0 472804 141604 22456 S 3.0 3.5 0:00.82 fix_cc4
7268 pi 20 0 472804 141604 22456 S 2.0 3.5 0:00.53 rotator_cc34
7278 pi 20 0 472804 141604 22456 S 1.7 3.5 0:00.66 copy23
7296 pi 20 0 472804 141604 22456 S 1.7 3.5 0:00.36 recorder
7270 pi 20 0 472804 141604 22456 S 1.0 3.5 0:00.24 pfb_arb_resamp3

Mike
 

KA1RBI

Member
Joined
Aug 15, 2008
Messages
799
Location
Portage Escarpment
well the trunk recorder running at 2.5 msps doesn't really seem excessive CPU-wise, so unless I'm mis-reading something it's possible the problem might be somewhere else. Also, looks like trunk recorder uses a different front end filter (complex fft vs. complex fir used in OP25). I haven't done any benchmarking of these in comparison to know which one might be preferable as far as CPU consumption is concerned.

Max
 

TippyTurtle

Member
Joined
Dec 10, 2019
Messages
5
So raspberry 4 specific possible issues:
  1. Heat: Is your CPU staying cool? If not it will get throttled.
  2. Amperage: RPI uses a 5v USB C charger. RPI4 needs 3 amps. Each dongle will have it's load too. Make sure your power supply has enough wattage/amperage for your RPI4 and all of it's (USB) accessories. I use a powered USB hub to run my dongles, so they have their own power supply.
  3. Bus(s): What bandwidth do you think you need over USB, Ethernet, etc.? Can the RPI4 facilitate those? (RPI3 there has major issues in some cases because the Ethernet is on the same USB 2.0 bus as everything else...it's a bottleneck. RPI4 has fixed much of this....but maybe not other bottlenecks for what you are trying to do. "Disk" speed might be clogging your USB SDR bus if your main disk is a USB drive. I am running 100% of the sd card which I believe has its own bus. Did you put a USB 2.0 device on a 3.0 hub/bus and expect the 3.0 device to be 3.0 fast?...it won't. It will run at 2.0 speeds usually. Example: SDR 2.0 dongle + 3.0 SSD disk will usually all run at 2.0 speeds.
  4. RAM: I am running a 4gb model and things run great with trunk-recorder with 50 analog TalkGroups (in a 550+ TalkGroup system). I imagine you can get into trouble "Swapping" if you have 1gb...when I say "swapping" RPI is a tricky beast attempting to not kill SD cards; I don't get it all.
  5. "Disk" speed: I am running of the SD card for everything. Therefore I am using a U3 card (30MB/s write speed) vs the more common U1 cards (10MB/s write speed). If you are using a disk over USB see #3...also how fast is it? SDD or 54000rpm spinning drive?
  6. ...I am sure there are more things to consider depending on what you are doing...maybe google raspberry performance tuning/diagnosing.
Loose End: The trunk-recorder install instructions had me run volk_profile which seemed to do a ton of performance testing and tweaking. Maybe it can help you? I haven't had time to go back and google about it. (I realize you are dong OP25 not Trunk-Recorder, but I assume since I can run Trunk-Recorder well on my RPI4, you should be able to run OP25 well as well.)

Side Note: Using chrome in xwindows on the same RPI4 running trunk-recorder does cause a lot of "sped up audio" (aka lost samples) on my raspberry. Chrome seems to be a bit of a dog...LibreOffice doesn't do this. :)

-Todd
 

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
Max,

The focus of my original post was basically "who out there is running Trunk Recorder using an Airspy @ 10 msps on a RPI4 with a claim of everything working 100% flawlessly?" That's all. So far no takers lol.

Of course, I didn't word it like that, but that's what I was wanting to know. Sorry you had to go through that back and forth with me about it. OP25 is running as expected on a PI4. I have no issues with how OP25 performs on a PI4.

Mike
 

KA1RBI

Member
Joined
Aug 15, 2008
Messages
799
Location
Portage Escarpment
Sorry you had to go through that back and forth with me about it. OP25 is running as expected on a PI4. I have no issues with how OP25 performs on a PI4.

Not at all; Trunk Recorder was forked from OP25, so any difference(s) could be quite interesting. Long ago (relatively speaking) when the first R-PI(3) models started to appear I made a significant change to the decimation scheme used in the OP25 demod block with the result that it now uses a two-stage decimation system. Those changes were all at the python level - but it appears trunk recorder has done away with all python; consequently I haven't looked into how it does the demodulation in any detail....

73

Max
 

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
Not at all; Trunk Recorder was forked from OP25, so any difference(s) could be quite interesting. Long ago (relatively speaking) when the first R-PI(3) models started to appear I made a significant change to the decimation scheme used in the OP25 demod block with the result that it now uses a two-stage decimation system. Those changes were all at the python level - but it appears trunk recorder has done away with all python; consequently I haven't looked into how it does the demodulation in any detail....

73

Max

Thanks for that information Max. I wasn't aware that it was forked from OP25. I thought it "made use of" whatever OP25 was already installed. I'm glad OP25 is working just fine though :)

Mike
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
Thanks for that information Max. I wasn't aware that it was forked from OP25. I thought it "made use of" whatever OP25 was already installed. I'm glad OP25 is working just fine though :)

Mike
The author of TrunkRecorder essentially took a snapshot of op25, then modified/integrated it with his application. It is independent of any separate op25 installation which you may have on your system.
 

lukekb

Member
Joined
Sep 4, 2013
Messages
59
This is really interesting! Boatbod and Max are totally right. Trunk Recorder uses a snapshot of OP25 to handle the control channel and voice channel decoding. Instead of using Python to decide to start and stop recordings, I kept everything in C++. I went down the C++ path a long time ago when I was trying to get gr-smartnet to scale and was too lazy to figure out Python. Probably would have been easier to just stick with Python.

I try to keep the snapshot of OP25 up to date with Boatbod's fork. It uses all of the same C++ P25 decoding libraries. It sounds like I need to go update the decimation architecture though because Max made some changes. That code is in Python and I translated that a while ago. That might explain the performance difference. If TR is not efficiently chopping a 10MHz channel down to 12.5KHz, it can really bog down the CPU. I will check it out tonight and post back.

It is awesome that it is working though! I gotta get a RPi4 and try it out with my HackRF.
 

lukekb

Member
Joined
Sep 4, 2013
Messages
59
It does look like Max's code is a bit newer / better than what I have. I am doing a 2 stage decimation process... but I think my numbers maybe screwy and may explode at certain capture rates. I am going update over to the improved code.
 

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
It does look like Max's code is a bit newer / better than what I have. I am doing a 2 stage decimation process... but I think my numbers maybe screwy and may explode at certain capture rates. I am going update over to the improved code.

Just so you know. I switched over to using an I5-9400 for this particular task. So I am not personally messing around on the RPI4 anymore and thus won't be able to tell you if an update improves things.

mike
 

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
I'm running Trunk Recorder + Trunk Player on an Odroid N2 now with Airspy @ 10 msps. All CPU cores nearly maxed out. It's cooking that sucker at about 60 C lol. I need to find a fan assembly for the sucker. It isn't skipping a beat though. I'm monitoring two separate systems, one site each, and converting / storing the audio on the Odroid.

I wish I had ran the same exact setup on the RPI4 _and_ logged the details, to compare the difference in performance between the RPI4 and the Odroid N2. Oh well.

But it's working, miraculously. I wish that somehow processing of the 10 msps data wasn't so taxing, but it is what it is. Really impressive.

Tasks: 183 total, 2 running, 181 sleeping, 0 stopped, 0 zombie
%Cpu0 : 63.4 us, 16.6 sy, 0.0 ni, 20.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 65.3 us, 14.0 sy, 0.0 ni, 20.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu2 : 92.3 us, 5.7 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 90.7 us, 7.0 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 95.0 us, 4.3 sy, 0.0 ni, 0.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 92.0 us, 5.0 sy, 0.0 ni, 2.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 3798992 total, 2654176 free, 777188 used, 367628 buff/cache
KiB Swap: 1048572 total, 1048572 free, 0 used. 2925772 avail Mem

cat /sys/devices/virtual/thermal/thermal_zone0/temp
62900
M
 

lukekb

Member
Joined
Sep 4, 2013
Messages
59
Nice work @mtindor ! I have an ODroid XU4 I should try spinning up. Give the latest version of TR a try. I pushed a change to the master branch last night. I uses the same decimation steps as OP25 now. The math Max used was a lot cleaner so it should reduce the load a little bit.
 

mtindor

OH/WV DB Admin
Database Admin
Joined
Dec 5, 2006
Messages
10,362
Location
Carroll Co OH / EN90LN
Nice work @mtindor ! I have an ODroid XU4 I should try spinning up. Give the latest version of TR a try. I pushed a change to the master branch last night. I uses the same decimation steps as OP25 now. The math Max used was a lot cleaner so it should reduce the load a little bit.

@lukekb Thanks. I will give it a try this weekend and will try and make a note of before / after performance. I appreciate all of the work that the OP25 guys, yourself and Dylan have put into the various packages we are using nowadays.

Mike
 
Status
Not open for further replies.
Top