DSD-FME - Wav files recording for Tier II DMR

[lwvmobile]

That may have been from an older version of the patch. The most recent (up until right now) was actually located in the dsd-fme/patch folder and that would have applied to the current source code without issue.

That being said, I just applied it, made the changes where the filename has no spaces (wasn't sure if its exactly the same format as before, but close enough), and saved a new patch for you to tackle the original issue and also the wav opening/closing issue that was originally addressed way back when the patch was first made.

So, download this zip, put it into the dsd-fme/patch folder and extract it, and then open the terminal to the base of the dsd-fme folder, and run:

git stash save
git apply patch/g_dmr_per_call_wav_file_fixes_w_custom_wav_dir_20240730.patch
cd build
sudo make install

Note, since you already had a patch applied once, we are using git stash save to save those changes (but we won't be able to reapply them). If you had other custom changes, you may need to reapply them manually, often times, stacking multiple patches doesn't work when both patches may address the same parts of code and git doesn't know how to elequently merge them, so manual merge or just redoing portions of work may be necessary if you had any custom changes OTHER than the filename space replacement with underscores.

 

[dolman]

Marvellous. This works like a charm and provides almost everything I need, including filenames with date and time templates. With one exception - if I use it with the default directory, it saves the calls to WAV and everything works great. However, if I start it with the -7 option followed by a custom directory path, it creates two placeholder files in that directory. Then for a call, a wav file is created and deleted when the call ends. Each following call creates a new wav file and deletes it again, and so on.

Do you think it would be possible to look into this? A custom directory is crucial for properly categorizing and sorting incoming calls into appropriate locations for rdio-scanner.

 

[lwvmobile]

I just gave it a go with the patch, and didn't have the issue with the wav files being deleted at the end. What is supposed to happen is that the files are created without the .wav extension, written to, and when they are closed, the .wav extension is appended onto the filename.

My understanding is, and maybe I'm wrong, but pretty sure rdio can be configured or is configured by default to delete .wav files after it ingests them, turning them into mp4 audio files or whatever compressed form they like. Not sure if that can be configured to preserve the .wav files or not, but if rdio is monitoring certain folders, it may be ingesting the wav files and deleting them after conversion and storage to another location.

 

[dolman]

You are perfectly right. It was actually rdio-scanner deleting the files after ingesting them, as I was testing DSD-FME on a directory already configured in rdio dirwatch.

What else can I say? Thank you, thank you, thank you. You're an absolute lifesaver for me.

 

[dolman]

To wrap everything up, I have prepared instructions for those who would like to reproduce my setup. I assume you have installed RTLSDR-Airband, DSD-FME, Pulseaudio, and rdio-scanner, and know how to configure them. Please note, this is valid as of July 31, 2024, and may not work after future changes affecting any of the software in this setup.

1. Create null-sink for Pulseaudio

It will work as a virtual audio cable, directing sound from one application to another. In this case, from RTLSDR-Airband to DSD-FME. To make it permanent and survive between reboots, edit the default.pa file as sudo:

sudo nano /etc/pulse/default.pa

Add the following line at the end of the file. Replace "dmr_test" and "DMR_Test" with whatever names you prefer. This instruction assumes that the sink_name is dmr_test.

load-module module-null-sink sink_name=dmr_test sink_properties=device.description=DMR_Test

2. RTLSDR-Airband configuration

Configure the pulse output for the selected channel in the rtl_airband.conf file with the following entry:

         {
            type = "pulse";
            server = "127.0.0.1";
            sink = "dmr_test";
            continuous = false;
         }

3. DSD-FME configuration

Download the zip file attached to this post. Unzip it and place the g_dmr_per_call_wav_file_fixes_w_custom_wav_dir_20240730.patch file in the main DSD-FME directory. Apply the patch with the following commands when in that directory.

git apply per-call.patch
cd build
sudo make install

Create a directory to store your recordings for rdio-scanner. In this case, it is "/home/user/recordings", then run DSD-FME with the following command:

dsd-fme -i pulse:dmr_test.monitor -7 /home/user/recordings/ -P

4. Configure rdio-scanner

Now, configure dirwatch in rdio-scanner to ingest wav files from "/home/user/recordings/", and you are good to go. Please keep in mind, you don't have to configure the "Mask" field in rdio-scanner to extract metadata. Rdio will ingest them even without it, assigning an appropriate date and time to each call.

 

[dolman]

I've encountered two new roadblocks. lwvmobile, I would much appreciate it if you could take a look when you're available.

1. Running DSD-FME on Boot

I've tried to run DSD-FME as a service to have it start on boot and run in the background by creating a "dmr_decoder.service" file in "/etc/systemd/system" and enabling the service. The file contains the following entries:

[Unit]
Description=Rdio Scanner
After=network.target

[Service]
ExecStart=dsd-fme -i pulse:dmr_test.monitor -7 /home/user/recordings/dmr_test/ -P

[Install]
WantedBy=default.target

Unfortunately, the service doesn't start after reboot and exits with a very strange error: "Err: 6; Connection refused." I don't get it – what does it mean? What kind of connection? I suppose it has something to do with running DSD-FME as root, because the same error appears when I execute DSD-FME from the command line with sudo. I have even tried running the service as another user by adding the "User=" entry to the service configuration file, but the error persists. Is there any workaround to have DSD-FME start at boot in the background?

2. Poor DMR Decoding When Sourced by Pulseaudio Null-Sink

When sourced directly from RTL-SDR, DSD-FME runs smoothly, but when sourced from null-sink, it has a hard time decoding the transmissions. Actually, only the Color Code is correct. Everything else is rubbish, including the voice saved in WAV files. My guesses would be:

a) Null-Sink Configuration - The null-sink might be configured inappropriately with the wrong format or sample rate. Do I have to set any specific sample rate for DSD-FME? Do I have to choose any specific format? As far as I know, available formats are listed on this page. Is it any of them?

b) RTLSDR-Airband Output - RTLSDR-Airband might be sending rubbish to DSD-FME via null-sink. Unfortunately, if this is the case, there's nothing we could do here, and it would be a dead end.

 

[lwvmobile]

"Err: 6; Connection refused."

That is because the user doesn't have permission to use the pulse audio server, either that, or it isn't running. Since the return error string is connection refused, going to assume the server is up and running, just that the root user doesn't have permission to use it. I think this is a similar thing that also happens when people setup OP25 to run as a service, checking out some of those threads on here may shed some light on how to get this to work properly. I didn't have in mind for users to have it start up and run with a system service, so if there is anything there that needs to be done in code on my end, I wouldn't have a clue where to start.

2. Poor DMR Decoding When Sourced by Pulseaudio Null-Sink

That shouldn't be the case at all, usually its the sink/source application that dictates what happens on the null-sink, since its not real hardware. Never had an issue piping audio into it or out of it. Could be what is coming out of rtlsdr-airband, maybe its using a different sample rate, or has some filtering on its output audio? For DMR, what you want going into the sink is 48k/1 s16le audio, but if its a higher rate, like DSDPlus which uses 96k/1 s16le audio, that is also fine, the pa_instance for reading the input will sample it properly still over pulse audio. If its something like 44100 then not sure if that will work or not. I'm more inclined to believe a simpler explanation is the correct one, so either the signal is poor, or noisy, or there is filtering in place coming out of rtlsdr-airband.

When you have both rtlsdr-airband and dsd-fme running, what is the outputs of both of these commands:

pactl list sink-inputs --and-- pactl list source-outputs

 

[dolman]

That is because the user doesn't have permission to use the pulse audio server, either that, or it isn't running. Since the return error string is connection refused, going to assume the server is up and running, just that the root user doesn't have permission to use it. I think this is a similar thing that also happens when people setup OP25 to run as a service, checking out some of those threads on here may shed some light on how to get this to work properly. I didn't have in mind for users to have it start up and run with a system service, so if there is anything there that needs to be done in code on my end, I wouldn't have a clue where to start.

As a first solution, I tried adding root to the pulse-access group, but it didn't work. I will dig deeper into the threads you mentioned. Thanks for the tip.

When you have both rtlsdr-airband and dsd-fme running, what is the outputs of both of these commands:

pactl list sink-inputs --and-- pactl list source-outputs

For pactl list sink-inputs:

Sink Input #1
        Driver: protocol-native.c
        Owner Module: 8
        Client: 33
        Sink: 1
        Sample Specification: float32le 1ch 16000Hz
        Channel Map: mono
        Format: pcm, format.sample_format = "\"float32le\""  format.rate = "16000"  format.channels = "1"  format.channel_map = "\"mono\""
        Corked: no
        Mute: no
        Volume: mono: 65536 / 100% / 0.00 dB
                balance 0.00
        Buffer Latency: 0 usec
        Sink Latency: 3855 usec
        Resample method: speex-float-1
        Properties:
                media.name = "150.000 MHz"
                application.name = "rtl_airband"
                native-protocol.peer = "TCP/IP client from 127.0.0.1:42346"
                native-protocol.version = "35"
                application.process.id = "1366"
                application.process.user = "root"
                application.process.host = "su24p03rs"
                application.process.binary = "rtl_airband"
                application.language = "C"
                application.process.machine_id = "f85e2574263f40b68c57c154938a97ee"
                module-stream-restore.id = "sink-input-by-application-name:rtl_airband"

Sink Input #2
        Driver: protocol-native.c
        Owner Module: 7
        Client: 46
        Sink: 1
        Sample Specification: s16le 2ch 8000Hz
        Channel Map: front-left,front-right
        Format: pcm, format.sample_format = "\"s16le\""  format.rate = "8000"  format.channels = "2"  format.channel_map = "\"front-left,front-right\""
        Corked: no
        Mute: no
        Volume: front-left: 65536 / 100% / 0.00 dB,   front-right: 65536 / 100% / 0.00 dB
                balance 0.00
        Buffer Latency: 0 usec
        Sink Latency: 3829 usec
        Resample method: speex-float-1
        Properties:
                media.name = "AUTO"
                application.name = "DSD-FME"
                native-protocol.peer = "UNIX socket client"
                native-protocol.version = "35"
                application.process.id = "1509"
                application.process.user = "user"
                application.process.host = "su24p03rs"
                application.process.binary = "dsd-fme"
                application.language = "C"
                application.process.machine_id = "f85e2574263f40b68c57c154938a97ee"
                application.process.session_id = "3"
                module-stream-restore.id = "sink-input-by-application-name:DSD-FME"

For pactl list source-outputs:

Source Output #0
        Driver: protocol-native.c
        Owner Module: 7
        Client: 45
        Source: 1
        Sample Specification: s16le 1ch 48000Hz
        Channel Map: mono
        Format: pcm, format.sample_format = "\"s16le\""  format.rate = "48000"  format.channels = "1"  format.channel_map = "\"mono\""
        Corked: no
        Mute: no
        Volume: mono: 65536 / 100% / 0.00 dB
                balance 0.00
        Buffer Latency: 0 usec
        Source Latency: 0 usec
        Resample method: copy
        Properties:
                media.name = "AUTO"
                application.name = "DSD-FME"
                native-protocol.peer = "UNIX socket client"
                native-protocol.version = "35"
                application.process.id = "1509"
                application.process.user = "user"
                application.process.host = "su24p03rs"
                application.process.binary = "dsd-fme"
                application.language = "C"
                application.process.machine_id = "f85e2574263f40b68c57c154938a97ee"
                application.process.session_id = "3"
                module-stream-restore.id = "source-output-by-application-name:DSD-FME"

 

[lwvmobile]

Just looking at this particular section from the output it seems the output audio as float32le 16k1, so not entirely sure if that can be resamples and handled properly with DSD-FME via the pulse audio server. Honestly, never tried it at that rate, and with float to short, so that could very well be an issue. Its probably still worth investigating signal strength and if/how to disable any audio filtering in rtlsdr-airband, if applicable.

Sample Specification: float32le 1ch 16000Hz
Channel Map: mono
Format: pcm, format.sample_format = "\"float32le\""  format.rate = "16000"  format.channels = "1"  format.channel_map = "\"mono\""

Just out of curiosity, was there any specific reason to use rtlsdr-airband and not the built in rtl handling?

 

[dolman]

Its probably still worth investigating signal strength and if/how to disable any audio filtering in rtlsdr-airband, if applicable.

I'm afraid configuration options for Pulseaudio output in rtlsdr-airband are rather limited, at least according to this site. Nevertheless, I will try to get some help on their Github, as I'm almost sure it is the reason why both applications don't talk to each other.

Just out of curiosity, was there any specific reason to use rtlsdr-airband and not the built in rtl handling?

I want to utilize high-sample rate devices as efficiently as possible. I'm currently monitoring about 80 frequencies within a 40MHz spectrum with 1 HackRF and 1 BladeRF (planning to replace the BladeRF with another HackRF, as using BladeRF just for Rx operations in the 2m band is an awful waste of resources). This setup is handled by rtlsdr-airband in the back as the "engine" and rdio-scanner in the front as the UI. I would like to put DSD-FME between them as "middleware" to translate DMR transmissions. In the case of using a regular RTL-SDR directly with DSD-FME, I'm limited to one exclusive frequency per device, which is not efficient at all and would require about 15 (the approximate number of active DMR frequencies in my area) RTL-SDRs for DMR transmissions.

 

[lwvmobile]

Hmmm...not sure if this will work or not, but if your willing to try a little science experiment, we can try to capture three samples at 48k/1 and keep the third. If my brain is working this morning, I think that will allow us to properly sample the 16k/1 coming out of rtlsdr-airband. So, the only thing you'll need to do is open dsd-fme/src/dsd_symbol.c and go down to line 85, and change it so it looks like this.



basically, just copy and paste this over top of what is in there for pulse audio, save, and do a sudo make install after.

// Read the new sample from the input
if(opts->audio_in_type == 0) //pulse audio
{
    for (uint8_t x = 0; x < 3; x++) //read three samples at 48k/1 (48/3 = 16)
        pa_simple_read(opts->pulse_digi_dev_in, &sample, 2, NULL );
}

If this doesn't work properly, then just make sure to disable the additional line added. Keep in mind, this will also break any other input type from pulse audio input that isn't at 16k/1.

 

[dolman]

Unfortunately, it doesn't work. I have replaced the entire // Read the new sample from the input part. DSD-FME doesn't receive any signal from Pulseaudio. I reverted the change back to the original code, and there is some activity. It's rubbish and chaos, but still something, like it was before. After the change, there is no activity at all.

 

[lwvmobile]

Well, maybe we should take a step or two backwards, might be a good idea to take a look at the logs from DSD-FME and find out if there is any particular reason why the audio is all garbled, and so on, see what all kinds of decoder errors are in the log and so on.

 

[dolman]

I have made a few calls with my radio and created two log files. One when DSD-FME is sourced by Pulseaudio, and the other when sourced by RTL SDR for comparison. Same radio, same transmission parameters. Just pure DMR with no encryption. Please find both files attached.

 

[lwvmobile]

Yeah, it looks like there are a substantial amount of errors when using rtlsdr-airband fed into pulseaudio compared to using rtl input directly. The remaining errors may just be due to a slight overload from your own radio to the front end of the SDR. I'm not quite sure where to direct you at this point. I don't have the time to setup and try out rtlsdr-airband for myself at the moment (about to get rimmed by Debbie and all that), so if its not something that can be tweaked signal wise in rtlsdr-airband, I really just think the 16k/1 float audio into the pulse audio server and subsequent resampling handled on that end may be the ultimate culprit to amount of decoding errors. If its not that, then its some sort of audio filtering on their end.

So, what are the alternatives besides using rtlsdr-airband? Is SDR++ an option? Not sure what hardware you are running on nor how many frequencies you are trying to monitor, but SDR++ does allow you to lay down multiple radios on a spectrum and you can feed each of those individually to a session of DSD-FME, or monitor them as analog if there is a mixture of analog and digital.

 

[dolman]

I'm not quite sure where to direct you at this point. I don't have the time to setup and try out rtlsdr-airband for myself at the moment...

Don't bother. You have helped me so much that I couldn't ask for more.

So, what are the alternatives besides using rtlsdr-airband? Is SDR++ an option?

Not really, because SDR++ is a GUI application, and my setup is a completely headless machine managed by SSH. I'm reluctant to install any desktop environment to save system resources.