OP25 Experiments with op25 and liquidsoap

Status
Not open for further replies.

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
I must have too much time on my hands as I spent the better part of today experimenting with liquidsoap to see if there was a way to set up a streaming server without having to pass the audio through the temperamental alsa subsystem prior to streaming it. The good news is I believe I have found an answer, the bad news is that it's rather complicated, so I recommend only going this route if you have some familiarity with linux.

Prerequisites:
Code:
sudo apt-get install liquidsoap liquidsoap-plugin-all
cd ~/op25
git pull
cd ~/op25/op25/gr-op25_repeater/apps

Configuration:
Once you have liquidsoap and the latest op25 changes, you will find a new file called "op25.liq" sitting in the 'apps' directory. If you want to set up a connection to an icecast server you will eventually need to edit this file, however it defaults to capturing the audio data from op25 and simply playing it on the local default sound hardware. I recommend you try it as-is to check that things are working properly.

Setting up op25 to play to liquidsoap requires different command line options as follows:
Remove: "-U" (and "-O" if it exists)
Add: "-V -w"
Code:
./rx.py --nocrypt --args "rtl=0" --gains 'lna:36' -S 960000 -q 0 -d -100 -v 1 -2 -T trunk-test.tsv -V -w 2> stderr.2

Run rx.py in a terminal window like you normally would, but don't expect any audio until you also start the liquidsoap op25.liq script (in a separate terminal window).
Code:
gnorbury@yoga2 ~/op25/op25/gr-op25_repeater/apps $ ./op25.liq
2019/01/01 22:12:48 >>> LOG START
2019/01/01 22:12:48 [protocols.external:3] Found "/usr/bin/wget".
2019/01/01 22:12:48 [main:3] Liquidsoap 1.1.1
2019/01/01 22:12:48 [main:3] Using: graphics=[distributed with Ocaml] pcre=7.0.4 dtools=0.3.1 duppy=0.5.1 duppy.syntax=0.5.1 cry=0.2.2 mm=0.2.1 xmlplaylist=0.1.3 lastfm=0.3.0 ogg=0.4.5 vorbis=0.6.1 opus=0.1.0 speex=0.2.0 mad=0.4.4 flac=0.1.1 flac.ogg=0.1.1 dynlink=[distributed with Ocaml] lame=0.3.2 shine=0.2.0 gstreamer=0.2.0 frei0r=0.1.0 voaacenc=0.1.0 theora=0.3.0 schroedinger=0.1.0 gavl=0.1.5 bjack=0.1.4 alsa=0.2.1 ao=0.2.0 samplerate=0.1.2 taglib=0.3.1 magic=0.7.3 camomile=0.8.4 inotify=1.0 faad=0.3.2 soundtouch=0.1.7 portaudio=0.2.0 pulseaudio=0.1.2 ladspa=0.1.4 dssi=0.1.1 sdl=0.9.1 camlimages=4.2.0 lo=0.1.0 yojson=1.2.3 gd=1.0a5
2019/01/01 22:12:48 [dynamic.loader:3] Could not find dynamic module for fdkaac encoder.
2019/01/01 22:12:48 [dynamic.loader:3] Could not find dynamic module for aacplus encoder.
2019/01/01 22:12:48 [dynamic.loader:2] Loaded plugin file /usr/lib/liquidsoap/1.1.1/plugins/pulseaudio.cmxs.
2019/01/01 22:12:48 [dynamic.loader:2] Loaded plugin file /usr/lib/liquidsoap/1.1.1/plugins/alsa.cmxs.
2019/01/01 22:12:48 [dynamic.loader:2] Loaded plugin file /usr/lib/liquidsoap/1.1.1/plugins/lame.cmxs.
2019/01/01 22:12:48 [dynamic.loader:2] Loaded plugin file /usr/lib/liquidsoap/1.1.1/plugins/cry.cmxs.
audio device: stdout

At this point local audio should be playing properly, much like it would have done before. From here you can edit the appropriate mount point, host, port and password in the op25.liq script and send the stream to icecast (i.e. broadcastify, your local server, ...). The nice part of this is you can decide whether to play the stream locally or not simply by commenting/uncommenting the "out()" statement.
Code:
#!/usr/bin/liquidsoap

# Example liquidsoap streaming from op25 to icecast
# (c) 2019, gnorbury@bondcar.com
#

set("log.stdout", true)
set("log.file", false)
set("log.level", 1)

# Make the native sample rate compatible with op25
set("frame.audio.samplerate", 8000)

input = mksafe(input.external(buffer=0.02, channels=2, samplerate=8000, restart_on_error=false, "./audio.py -s"))


# LOCAL AUDIO
# Uncomment the line below to enable local sound
#
out(input)


# ICECAST STREAMING
# Uncomment to enable output to an icecast server
# Change the "host", "password", and "mount" strings appropriately first!
#
#output.icecast(%mp3(bitrate=16, samplerate=22050, stereo=false), fallible=false, icy_metadata="false", host="localhost", port=8000, mount="mountpoint", password="hackme", mean(input))

Known issues
Shutting down the stream is as simple as hitting "Ctrl-C" in the op25.liq terminal window. There is a small problem with thread termination in the helper application audio.py which causes delayed shutdown of some background processes. Normally this isn't an issue unless you restart the script immediately, at which point you may see "socket.error: [Errno 98] Address already in use" messages.

Unintended consequences
I had to modify audio.py so that it can collect UDP samples and send them to stdout. As a result it is now entirely possible to do the following, much like nc | aplay:
Code:
./audio.py -s | aplay -c 2 -r 8000 -f S16_LE
The consequence of this is that it is now possible to pipe raw audio output from op25 direct to any application that expects to see 2 channel, 8000Hz 16 bit LE samples.
Secondly, I would point out that it's entirely possible to split op25 and liquidsoap onto separate machines simply by sending the udp samples over the network using the "-W" wireshark host option.

Graham
 

KA1RBI

Member
Joined
Aug 15, 2008
Messages
799
Location
Portage Escarpment
Hi Graham

a few thoughts on this....

First, does the alsa loopback method result in a continuous read data stream that even includes silent periods? That is, during periods when our rx is tuned to the control channel awaiting a call there is no decoded UDP audio data flowing and so no sound frame data being written to the alsa sink. What happens in the peer process (darkice etc). during these periods? Does alsa provide continuous silent (zero-data) frames? Or is this audio flow stalled as well?

Second is basically the same question as applied to the liquidsoap alternative. In this case there is no raw audio flowing out of audio.py during silent periods?

Third, it would be nice to integrate HTML5 audio into OP25's http console to allow streaming audio in the browser. The actual HTML and JS to do so is more or less trivial with the actual work happening in the backend. I'm not expert at streaming but as far as I can tell the preferred setup employs mp3 with ogg as fallback. If we focus on mp3 initially what would be needed at server side to implement it?

Max

p.s. thanks for all your efforts
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
Max,

To your first point, op25's sockaudio.py only writes to ALSA when sound is available, but Darkice (or whatever streaming client) reads a continuous pcm stream from ALSA and sends this continuous datastream out to the icecast server (either local or remote). i.e. ALSA fills in the blanks for us. One of the problems we've found particularly with the RPi is that alsa can be buggy on some kernels and is always sensitive to underruns; sometimes it quits working completely for no apparent reason.

With the liquidsoap configuration, op25's audio.py (using sockaudio.py) only pipes data to liquidsoap when audio is actively being decoded. Liquidsoap however "fills in" the silences itself [using "mksafe()"] so the output stream to either a sound subsystem or an icecast server will be one continuous data stream. Conceivably it might be possible to eliminate the mksafe() and allow the pcm stream to start & stop, but this definitely wouldn't work for icecast as it requires the stream to be active in order to be considered "up".

I've often thought about adding html5 audio direct to op25, but essentially it looks like someone would have to implement a complete streaming server (presumably in Python). Being able to install off-the-shelf components such as icecast2, liquidsoap and darkice has it's attractions, but I worry that they may be (a) too resource hungry on low-end hardware, and (b) too complex for the average user to install and configure. Issue (a) is easy to quantify. Issue (b) not so much...

Graham
 

squirrel

Member
Premium Subscriber
Joined
Jan 4, 2006
Messages
110
I have reloaded my Pi over the weekend using this option and I must say the install process was a lot easier and I think the audio sounds better.

I have op25-liq running as a service and when I reboot it will hang for the default timeout (1m30s) trying to shut the service down. I am sure this is due to the thread termination issue you mentioned above. Is there a way that I can shorten this timeout? Functionality is not affected it just makes reboots longer.

Thanks
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
I have reloaded my Pi over the weekend using this option and I must say the install process was a lot easier and I think the audio sounds better.

I have op25-liq running as a service and when I reboot it will hang for the default timeout (1m30s) trying to shut the service down. I am sure this is due to the thread termination issue you mentioned above. Is there a way that I can shorten this timeout? Functionality is not affected it just makes reboots longer.

Thanks
I'm searching for a way to make the shutdown cleaner. Suffice it to say is a tricky problem and I haven't hit upon a good solution yet.

ETA: I worked on it a bit tonight and have come up with a way to improve things. It should now take no more than 5 seconds for audio.py to figure out when the pipe to liquidsoap has broken and it can then shut itself down.

Graham
 
Last edited:

james18211

Member
Joined
Oct 28, 2012
Messages
25
Is this able to handle multiple streams on a single system? I'd love to combine my VMs into a single system and avoid ALSA's BS.
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
Is this able to handle multiple streams on a single system? I'd love to combine my VMs into a single system and avoid ALSA's BS.

LiquidSoap can handle multiple streams, so you'd basically set up two input.external() and two output.icecast() and link them up appropriately.

This is not a complete .liq example, but you should get the gist:
Code:
input_A = mksafe(input.external(buffer=0.02, channels=2, samplerate=8000, restart_on_error=false, "./audio.py -s -u 23456"))
input_B = mksafe(input.external(buffer=0.02, channels=2, samplerate=8000, restart_on_error=false, "./audio.py -s -u 23466"))

output.icecast(%mp3(bitrate=16, samplerate=22050, stereo=false), description="op25_A", url="", fallible=false, icy_metadata="false", host="localhost", port=8000, mount="mountpoint_A", password="hackme_A", mean(input_A))
output.icecast(%mp3(bitrate=16, samplerate=22050, stereo=false), description="op25_B", url="", fallible=false, icy_metadata="false", host="localhost", port=8000, mount="mountpoint_B", password="hackme_B", mean(input_B))

Obviously you'd also have to have two instances of rx.py running in separate terminals with corresponding "-u 23456" and -"u 24566" port numbers to mate up with the two ./audio.py instances started by liquidsoap.
 

quad_track

Member
Joined
Sep 13, 2017
Messages
66
Hi Max, Graham,
Sorry for being completely offtopic here, but have you considered the possibility of integrating DMR repeater linking via the usual protocols into OP25? I realize there is serious overlap with Jonathan Naylor's work here, but here's my thought: a single DMR base only allows two simultaneus voice calls, while with a SDR you can transmit and receive multiple adjacent carriers given enough CPU resources. Is there scope for capacity improvement or is it just overkill?
Currently the alternatives are:
1. Install multiple factory repeaters (duplexer and antennas needed)
2. Same but with MMDVM and possibly one computing platform per TRX hardware
3. Use a SDR and transmit multiple adjacent DMR carriers (one antenna, one duplexer, maybe easier system management)

I am now again within range of a DMR repeater and looking for something to play with. Am I bringing telco world complications into a simple and happy world?

Adrian
 

IQ_imbalance

Member
Premium Subscriber
Joined
Jan 1, 2010
Messages
213
Just did a completely fresh install of stretch and reinstalled op25 and liquidsoap. When i execute ./rx.py as described above and then ./op25.liq I get what appears to be the same output as in post #1 above, but then i get a "audio closing" message and the script exits.

Still no joy w/ -U and -O and the default audio with or without audio.py...could it be i'm just not receiving a strong enough signal to trigger the vox decode? The tuning looks OK from what I can tell from the graphs, but i'm only ~20db over background at best. Still playing around w/ the -q settings but...
Screen Shot 2019-01-17 at 19.54.59.pngScreen Shot 2019-01-17 at 19.55.34.png
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
Just did a completely fresh install of stretch and reinstalled op25 and liquidsoap. When i execute ./rx.py as described above and then ./op25.liq I get what appears to be the same output as in post #1 above, but then i get a "audio closing" message and the script exits.

Still no joy w/ -U and -O and the default audio with or without audio.py...could it be i'm just not receiving a strong enough signal to trigger the vox decode? The tuning looks OK from what I can tell from the graphs, but i'm only ~20db over background at best. Still playing around w/ the -q settings but...
View attachment 67696View attachment 67697
To offer meaningful advice we'll need to see your constellation plot, screen capture of rx.py terminal and whatever is being output in your stderr.2 log along with the exact rx.py command line.

On an unrelated note, if you are seeing "audio closing" it means that audio.py exited for some reason. For that we'd need to see the op25.liq screen output.
 

IQ_imbalance

Member
Premium Subscriber
Joined
Jan 1, 2010
Messages
213
OK, here goes. I'm using ./rx.py --nocrypt --args "rtl=0" --gains 'lna:48' -S 2400000 -q 3 -d 10 -v 5 -2 -T trunk.tsv -V -w 2>stderr.2

If everything looks OK and it's apparently a tuning issue, just let me know and i'll keep playing with the -d value.

Thanks for 1) putting all this together, and 2) entertaining my questions!
 

Attachments

  • Screen Shot 2019-01-17 at 21.56.54.png
    Screen Shot 2019-01-17 at 21.56.54.png
    368.9 KB · Views: 38
  • Screen Shot 2019-01-17 at 22.01.43.png
    Screen Shot 2019-01-17 at 22.01.43.png
    90.1 KB · Views: 41
  • Screen Shot 2019-01-17 at 21.56.30.png
    Screen Shot 2019-01-17 at 21.56.30.png
    264.2 KB · Views: 37

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
OK, here goes. I'm using ./rx.py --nocrypt --args "rtl=0" --gains 'lna:48' -S 2400000 -q 3 -d 10 -v 5 -2 -T trunk.tsv -V -w 2>stderr.2

If everything looks OK and it's apparently a tuning issue, just let me know and i'll keep playing with the -d value.

Thanks for 1) putting all this together, and 2) entertaining my questions!

The "voice timeout" messages are curious and would explain the lack of audio.
Constellation plot looks so-so. The clusters are rather broad, suggesting a weak/noisy signal, but they are clearly visible so some data is being extracted as evidenced by the non-zero TSBK counter and voice update activity.
Log shows phase 2 activity (slot# is present) so I doubt we're dealing with a modulation mismatch (cqpsk/fsk4)

When the system is monitoring the control channel, are you seeing control channel timeouts as well, or are they only occurring on the voice channels? Can you post more of the stderr.2 log please?

Fine tuning offset is rather high. I'd recommend adjusting ppm by +/-1 (as appropriate) and decrease the fine tune.
 

IQ_imbalance

Member
Premium Subscriber
Joined
Jan 1, 2010
Messages
213
Sure thing. I've lowered the fine tune to -q 3 -d 9 and seem to be getting more frequent hits. I was getting a lot of control channel timeouts with the high offsets, but only seeing them occasionally now.

Here's a bit of my last stderr.2:

gr-osmosdr 0.1.4 (0.1.4) gnuradio 3.7.10
built-in source types: file osmosdr fcd rtl rtl_tcp uhd miri hackrf bladerf rfspace airspy soapy redpitaya
Using device #0 Realtek RTL2838UHIDIR SN: 00000001
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
[R82XX] PLL not locked!
Using two-stage decimator for speed=2400000, decim=25/4 if1=96000 if2=24000
Project 25 IMBE Encoder/Decoder Fixed-Point implementation
Developed by Pavel Yazev E-mail: pyazev@gmail.com
Version 1.0 (c) Copyright 2009
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see the file ``LICENSE'' for details.
op25_audio::eek:pen_socket(): enabled udp host(127.0.0.1), wireshark(23456), audio(23456)
p25_frame_assembler_impl: do_imbe[1], do_output[0], do_audio_output[1], do_phase2_tdma[1], do_nocrypt[1]
metadata update not enabled
1547842122.889399 do_metadata state=1: [None] None
1547842156.998839 set tgid=7506, srcaddr=2135530
1547842156.999693 new tgid: 7506 MD NRP Area 5 prio 3
1547842157.000386 new freq: 773.781250
1547842157.000778 voice update: tg(7506), freq(773781250), slot(1), prio(3)
1547842157.001018 do_metadata state=0: [7506] MD NRP Area 5
1547842158.876604 set tgid=7506, srcaddr=2135530
1547842158.878884 set tgid=7506, srcaddr=2135530
1547842158.882918 voice timeout
1547842159.221133 control channel timeout
1547842159.221388 do_metadata state=1: [None] None
1547842159.661181 set tgid=7506, srcaddr=0
1547842159.662351 voice update: tg(7506), freq(773781250), slot(1), prio(3)
1547842159.662729 do_metadata state=0: [7506] MD NRP Area 5
1547842160.753499 voice timeout
1547842161.135939 set tgid=7506, srcaddr=0
1547842161.136312 hold active tg(7506)
1547842161.136502 voice update: tg(7506), freq(773781250), slot(1), prio(3)
1547842161.136671 do_metadata state=0: [7506] MD NRP Area 5
1547842161.205709 set tgid=7506, srcaddr=2135530
1547842162.225403 voice timeout
1547842162.940064 set tgid=7506, srcaddr=0
1547842162.941061 hold active tg(7506)
1547842162.941823 voice update: tg(7506), freq(773781250), slot(1), prio(3)
1547842162.942087 do_metadata state=0: [7506] MD NRP Area 5
1547842164.030911 voice timeout
1547842164.425052 set tgid=7506, srcaddr=0
1547842164.425361 hold active tg(7506)
1547842164.425538 voice update: tg(7506), freq(773781250), slot(1), prio(3)
1547842164.425690 do_metadata state=0: [7506] MD NRP Area 5
1547842165.498524 voice timeout
1547842165.928860 set tgid=7506, srcaddr=0
1547842165.929133 hold active tg(7506)
1547842165.929283 voice update: tg(7506), freq(773781250), slot(1), prio(3)
1547842165.929425 do_metadata state=0: [7506] MD NRP Area 5
1547842167.076558 voice timeout
1547842167.778000 set tgid=7506, srcaddr=0
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
Seems like it's tuning the control channel ok but not being good lock on voice. Symptoms are similar to having incorrect modulation type, but non-qpsk would be unusual for a phase 2 system.

Just for grins, try putting "-D fsk4" on the command line and re-run to see if it makes any difference. I'm not hopeful, but you never know!
What sort of eye do you see if you enable the datascope plot (#4)
 

IQ_imbalance

Member
Premium Subscriber
Joined
Jan 1, 2010
Messages
213
-D fsk4 gave me an error i haven't seen in the log before :
connect_fm_demod() failed test

Here's a datascope plot from today...two things to consider, though: I'm getting more control channel timeouts this AM (but also getting voice update activity and increments in the TSBK count), and (perhaps more worrisome) the convergence points on the datascope plot seem to drift w/ time.
 

Attachments

  • Screen Shot 2019-01-19 at 12.01.12.png
    Screen Shot 2019-01-19 at 12.01.12.png
    588.4 KB · Views: 11

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
That eye diagram looks qpsk but noisy. You can compare with two of mine; the first is the control channel (has two nice well defined eyes) and the second is from an active voice channel (kinda messy). C4FM would be a single wider eye.
 

Attachments

  • Screenshot from 2019-01-19 17-25-16.png
    Screenshot from 2019-01-19 17-25-16.png
    55 KB · Views: 12
  • Screenshot from 2019-01-19 17-25-04.png
    Screenshot from 2019-01-19 17-25-04.png
    58.5 KB · Views: 12

IQ_imbalance

Member
Premium Subscriber
Joined
Jan 1, 2010
Messages
213
good news...getting audio from some of the talkgroups now through audio.py. I modified the control freq. tuning taking the ppm error of this particular rtlsdr into account, and then fine tuned it from there. Still seeing a lot of control channel timeouts and not getting all the talkgroups so probably got a way to go, but now that i know what to look for from the gnuplots it should be easier. Thanks for all the help!
 

boatbod

Member
Joined
Mar 3, 2007
Messages
3,316
Location
Talbot Co, MD
good news...getting audio from some of the talkgroups now through audio.py. I modified the control freq. tuning taking the ppm error of this particular rtlsdr into account, and then fine tuned it from there. Still seeing a lot of control channel timeouts and not getting all the talkgroups so probably got a way to go, but now that i know what to look for from the gnuplots it should be easier. Thanks for all the help!
Tight clusters on the constellation plot is your main goal. Perhaps you need a better antenna?
 

KA1RBI

Member
Joined
Aug 15, 2008
Messages
799
Location
Portage Escarpment
-D fsk4 gave me an error i haven't seen in the log before :
connect_fm_demod() failed test

Here's a datascope plot from today...two things to consider, though: I'm getting more control channel timeouts this AM (but also getting voice update activity and increments in the TSBK count), and (perhaps more worrisome) the convergence points on the datascope plot seem to drift w/ time.

Thanks for posting this eye plot - it's clearly C4FM...
 
Status
Not open for further replies.
Top