Decoding single-channel NXDN

[lwvmobile]

Got it whittled down a little bit more, getting text labels now, but finding they are broken up into multiple parts, so trying to figure out how to figure out which order they go in and how to put it all together.

Here is an example of 'Med 17 Front'

2022-01-18 22:41:18.211598
LABEL HEX = 4d454420
LABEL STR = MED
2022-01-18 22:41:18.850800
LABEL HEX = 31372046
LABEL STR = 17 F
2022-01-18 22:41:19.488182
LABEL HEX = 726f6e74
LABEL STR = ront

and 'Live Oak'

2022-01-18 22:44:22.785680
LABEL HEX = 4c697665
LABEL STR = Live
2022-01-18 22:44:23.425399
LABEL HEX = 204f616b
LABEL STR = Oak

But you don't always get these all in order, or all of them every time. I often get just 'Live' without the 'Oak'. I'm going to presume they come in on other msgtype or lich values. Will have to investigate more at another time.

Here is my current working modifications to multi_rx.py

from datetime import datetime
import codecs #can't even remember if I still need this one for decoding strings or not
rid2 = 0
label2 = 0
##skip a few##

    def process_nxdn_msg(self, s):
        global rid2
        global label2
        if isinstance(s[0], str):    # for python 2/3
            s = [ord(x) for x in s]
        msgtype = chr(s[0])
        lich = s[1]

        now = datetime.now()
        #if msgtype == 'f':
        if lich == 0x41:
            rid = int.from_bytes(s, "big")
            if ((rid & 0xFFFFF0000000000) >> 40) > 0 and rid2 != rid:
                sys.stderr.write ('%s\n' %(now))
                sys.stderr.write ('RID = %d\n' % ((rid&0xFFFFF0000000000)>>40))
                rid2 = rid

        #if msgtype == 'S':
        if lich == 0x57:
            label = int.from_bytes(s, "big")
            if (label & 0xFFFFFFFF) > 0x1FFFFFFF and label2 != label:
                sys.stderr.write ('%s\n' %(now))
                sys.stderr.write ('LABEL HEX = %x\n' % (label&0xFFFFFFFF))
                name = s[8:].decode('utf-8', errors='ignore')
                sys.stderr.write ('LABEL STR = %s\n' % (name))
                label2 = label
        ##continues same as current

 

[KA1RBI]

First, when you start up an NXDN with "destination": "udp://127.0.0.1:23456", does it use 2 UDP ports for each NXDN channel, like DMR does?

I'm not an expert on nxdn, but as far as I know there is no such thing as two separate voice time slots/channel as there is in dmr (or in p25p2/tdma)...

Second, when I attempt to close OP25 with my config file above, it will usually hang and only close when there is an NXDN sync frame coming in.

The closedown has a few bugs - sometimes you need to hit 'q' key followed by ENTER a couple of times. Another bug is as you've noted.

Thanks for this report.

Also - separately, could I have you send me a symbol capture (to contain the various FACCHs and so forth and etc.). - to do so, add the following to one of the "channel" entries in the json:
"log_symbols": "sym-log.dat"

Max

 

[lwvmobile]

I'm not an expert on nxdn, but as far as I know there is no such thing as two separate voice time slots/channel as there is in dmr (or in p25p2/tdma)...

Okay, well, what about audio.py, does that open two UDP ports by default? When I open up it with

./audio.py -u 23456 -x 2

and try to open a second one with

./audio.py -u 23458 -x 2

I get a OSError: [Errno 98] Address already in use error

And I'll get that symbol capture going on one of them, it'll be a lot of dead air with calls mixed in though, so just a heads up. When I get a few calls rolling into it, I'll zip it up and send it to you.

 

[lwvmobile]

Also - separately, could I have you send me a symbol capture (to contain the various FACCHs and so forth and etc.). - to do so, add the following to one of the "channel" entries in the json:
"log_symbols": "sym-log.dat"

Okay, here is a sym-log.dat file, It should have 4 or 5 calls in it.

 

[KA1RBI]

OK thanks lwvmobile - the symbol file shows several FACCH layer 3 messages - after removing the initial 2-byte OP25 header the next byte (actually its low 6 bits) contain the nxdn layer 3 message type - in your trace we see most are either type 0x08 or type 0x3f. The former is a TX_REL and the latter is PROP_FORM (proprietary message - i.e., undocumented). The 0x3f seems to always be sent in lich type 0x51, for some reason. Note that in processing these types of messages the layer 3 message type is determinative, not the lich.

The 08 would be reasonable to add but the 3f is, for me, not something I'd be interested in adding (unless a spec for these "proprietary" message types is in existence)...

 

[lwvmobile]

Yeah, I had to google search and found an NXDN technical manual, I need to brush up on the terminology used by NXDN systems, sadly its a system I have access to and yet understand it the least. OP25 being another thing I understand little about when it comes to the inner workings of. It would take a dummy like me an eternity to go through all the code and figure out how it all works together.

On the process_nxdn_msg(self, s) inside of multi_rx.py, is it designed on only write to stderr when the software is shut down. I originally thought the logging was delayed, but I'm finding it only writes to stderr with the sys.stderr.write when it goes for a proper shutdown. Is that a bug that only happens for process_nxdn_msg(self, s), or does that occur for pretty much everything inside of multi_rx.py? I wasn't sure when you said it was a bug, if you were referring to just the close down, or the writing to stderr.

Anyways, I played with the code a little bit this morning, and I'm sure you're going to yell at me for doing it this way, using global variables and not class variables, I never got past that lesson in Python school. I got the code shaped up like this so far for my personal logging needs:

from datetime import datetime
rid2 = 0
label2 = 0
name_string = ' '

    def process_nxdn_msg(self, s):
        global rid2
        global label2
        global name_string
        if isinstance(s[0], str):    # for python 2/3
            s = [ord(x) for x in s]
        msgtype = chr(s[0])
        lich = s[1]
        now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

        if msgtype == 'f':
            if lich == 0x41:
                rid = int.from_bytes(s, "big")
                if ((rid & 0xFFFFF0000000000) >> 40) > 0 and rid2 != rid:
                    sys.stderr.write ('%s\n' %(now))
                    sys.stderr.write ('RID = %d ' % ((rid&0xFFFFF0000000000)>>40))
                    sys.stderr.write ('%s\n' % (name_string))
                    rid2 = rid
                    name_string = ' '

        if msgtype == 'S':
            label = int.from_bytes(s, "big")
            if (label & 0xFFFFFFFF) > 0x1FFFFFFF:
                name = s[8:].decode('latin-1', errors='ignore')
                if name in name_string:
                    return True;
                else: 
                    name_string += str(name)

        if self.verbosity > 2:
            sys.stderr.write ('process_nxdn_msg %s lich %x\n' % (msgtype, lich))
        if msgtype == 'c':     # CAC type
            ran = s[2] & 0x3f
            msg = cac_message(s[2:])
            if msg['msg_type'] == 'CCH_INFO' and self.verbosity:
                sys.stderr.write ('%-10s %-10s system %d site %d ran %d\n' % (msg['cc1']/1e6, msg['cc2']/1e6, msg['location_id']['system'], msg['location_id']['site'], ran))
            if self.verbosity > 1:
                sys.stderr.write('%s\n' % json.dumps(msg))

and I've been getting pretty good results with it, although I know its not the greatest way to log the RID and Names, just a quick dirty way that's not always consistent, but yeah, its nice for quick reference. This is what it looks like in the stderr log.

2022-01-20 14:12:29
RID = 1006
2022-01-20 14:18:09
RID = 132132
2022-01-20 14:18:09
RID = 132164
2022-01-20 14:18:09
RID = 106
2022-01-20 14:18:21
RID = 1006
2022-01-20 14:18:46
RID = 1000 Live Oak
2022-01-20 14:19:00
RID = 110 MED 18 Front
2022-01-20 14:19:04
RID = 1006 Live Oak
2022-01-20 14:19:24
RID = 1000 Live Oak
2022-01-20 14:19:26
RID = 1006
2022-01-20 14:19:32
RID = 110 MED 18 Front
2022-01-20 14:19:36
RID = 1000 Live Oak
2022-01-20 14:19:48
RID = 1006
2022-01-20 14:20:06
RID = 1000 Live Oak
2022-01-20 14:20:11
RID = 1006
2022-01-20 14:21:38
RID = 1000 Live Oak
2022-01-20 14:21:38
RID = 1006
2022-01-20 14:21:58
RID = 110 MED 18 Front
2022-01-20 14:22:16
RID = 1000 Live Oak
2022-01-20 14:22:22
RID = 1006
2022-01-20 14:22:32
RID = 110 MED 18 F
2022-01-20 14:22:37
RID = 1000 Live Oak
2022-01-20 14:22:44
RID = 1006
2022-01-20 14:22:50
RID = 110 MED 18 Front
2022-01-20 14:22:52
RID = 1000 Live
2022-01-20 14:23:05
RID = 1006
2022-01-20 14:33:01
RID = 230
2022-01-20 14:33:01
RID = 1006
2022-01-20 14:40:20
RID = 110 MED 18 Front
2022-01-20 14:40:22
RID = 1000 Live Oak
2022-01-20 14:40:38
RID = 1006
2022-01-20 14:40:55
RID = 110 MED 18 Front
2022-01-20 14:40:57
RID = 1000 Live
2022-01-20 14:41:00
RID = 1006
2022-01-20 14:41:08
RID = 1 Med 1\00\00\00
2022-01-20 14:41:10
RID = 1000 Live
2022-01-20 14:41:12
RID = 1 Med
2022-01-20 14:41:14
RID = 1000 Live
2022-01-20 14:41:22
RID = 1006
2022-01-20 14:42:26
RID = 110 MED 18 Front
2022-01-20 14:42:27
RID = 1006
2022-01-20 14:42:28
RID = 1000 Live Oak
2022-01-20 14:42:37
RID = 110 MED 18 Front
2022-01-20 14:42:40
RID = 1000 Live Oak
2022-01-20 14:42:49
RID = 1006
2022-01-20 14:43:13
RID = 1000 Live Oak
2022-01-20 14:43:14
RID = 230
2022-01-20 14:43:14
RID = 1006
2022-01-20 14:43:18
RID = 2001
2022-01-20 14:43:18
RID = 1006
2022-01-20 14:43:19
RID = 230
2022-01-20 14:43:20
RID = 106
2022-01-20 14:43:23
RID = 230
2022-01-20 14:43:24
RID = 1006
2022-01-20 14:43:26
RID = 132132
2022-01-20 14:43:26
RID = 132148
2022-01-20 14:43:26
RID = 106
2022-01-20 14:43:29
RID = 1000 Live Oak
2022-01-20 14:43:34
RID = 2001
2022-01-20 14:43:34
RID = 1006
2022-01-20 14:43:36
RID = 230
2022-01-20 14:43:36
RID = 1006

the RID = 1006 comes off of the SREV system which has a little data burst every 30 seconds or so, so it just clogs up the log, but if it weren't for that little sync burst, I'd have to wait forever for OP25 to shutdown properly or risk not having a log at all.

 

[KA1RBI]

Yeah, I had to google search and found an NXDN technical manual

The NXDN tech manual that I have (by design) doesn't cover the "proprietary" extensions. It's titled Nxdn_1A-CAI-v1.3...

I originally thought the logging was delayed, but I'm finding it only writes to stderr with the sys.stderr.write when it goes for a proper shutdown.

Ah - you're correct, this seems to be an "issue" (I think of it more as a "bug") in python3... For decades the implicit contract for stderr (in any language) was that it was supposed to be UNbuffered - for unknown reasons the python3 team decided it should be buffered. When searching for answers all I could seem to find was some very crazy and unnatural-looking code to set it to unbuffered. In OP25 the only concession to this "stupidity" (not ruling out the possibility it could be mine) was to add a command to flush stderr after the completion of initialization, so that at minimum the startup messages could be seen in a timely way... You can do a flush anytime - although doing so millions of times/sec. might be bad for performance...

Anyways, I played with the code a little bit this morning, and I'm sure you're going to yell at me for doing it this way, using global variables

Not the slightest yelling.

I got the code shaped up like this so far for my personal logging needs

Delighted to hear this, you can thank the fact that OP25 is open source and Free Software.

Just try doing that stunt using DSD+ and see how far you get...

Max

 

[mtindor]

You've become quite bitter over time, Max. Kind of disappointing. I appreciate all that you and the OP25 folks do, but I sort of feel like you guys would feel much better about life (in general) if you got that monkey off your back.

 

[lwvmobile]

The NXDN tech manual that I have (by design) doesn't cover the "proprietary" extensions. It's titled Nxdn_1A-CAI-v1.3...

Its a trade secret lol, got to keep it legally distinct by adding a proprietary 'feature' or two so you can brand it as something else.

Ah - you're correct, this seems to be an "issue" (I think of it more as a "bug") in python3... For decades the implicit contract for stderr (in any language) was that it was supposed to be UNbuffered - for unknown reasons the python3 team decided it should be buffered. When searching for answers all I could seem to find was some very crazy and unnatural-looking code to set it to unbuffered. In OP25 the only concession to this "stupidity" (not ruling out the possibility it could be mine) was to add a command to flush stderr after the completion of initialization, so that at minimum the startup messages could be seen in a timely way... You can do a flush anytime - although doing so millions of times/sec. might be bad for performance...

Got it, I just added a flush to the end of my writes, they don't happen frequently enough to impact performance, and I no longer risk losing the data they would print out.

 

[KA1RBI]

You've become quite bitter over time, Max. Kind of disappointing. I appreciate all that you and the OP25 folks do, but I sort of feel like you guys would feel much better about life (in general) if you got that monkey off your back.

LOL

The entire 2nd half of this thread has been filled with messages such as this:

I think I got it figured out. The NXDN voice sounds really good

or

Hang on, think I figured something out. Here is an example that is RID

or

Well, I've been poking around a little bit in the code,

or

I was able to determine that the text labels seem to exist on byte code for lich 0x57

or

Anyways, I played with the code a little bit this morning, and I'm sure you're going to yell at me for doing it this way, using global variables and not class variables, I never got past that lesson in Python school. I got the code shaped up like this so far for my personal logging needs:

Perhaps I'm misinterpreting all this, but what I clearly see is someone with a well-deserved sense of accomplishment and achievement - more to the point - someone EMPOWERED by the ability to delve in, learn from and modify the code! Also I know that 'lwvmobile' is hardly the only one. THAT is something that means more to me than would all of the $ that DSD+ may have collected and/or will collect in the FL operation...

73

 

[lwvmobile]

Well, I managed to work pushing the info to the curses terminal, but had to use some try and exceptions to handle errors stemming from labels that won't decode properly, or are otherwise empty, as they will print to the stderr log okay (but opening the text file will make my text editor complain about the character encoding) but just make terminal.py crash outright otherwise. Its working okay.

terminal.py no likey stuff like this

RID [100] [Sta 1\00\00\00]
2022-01-22 11:06:15

Did want to ask you though, if you have that symbol file still, can you tell me where the RAN value is located in it? Both sites are supposed to be RAN 1 (at least that's what other sources indicate). Trying to find a string of data that's 0b0001 or 0x1 is like trying to find a needle in a haystack if you don't know where to look.

Also, just for fun, here's the current side-by-side comparison

 

[lwvmobile]

Did want to ask you though, if you have that symbol file still, can you tell me where the RAN value is located in it?

Well, maybe I found it, not sure, it seems to check out, but honestly wouldn't know without more samples. I just moved ran = s[2] & 0x3f off of msgtype c and put it on msgtype S. Maybe its not just a coincidence

 

[KA1RBI]

Looks like RAN is part of the Structure (SR) field present as first byte (following 2-byte OP25 header) of 'S' but not 's' messages (and requires ANDing with 0x3f as you've said)...

 

[lwvmobile]

Well, I got it about whipped as good as I would like to for now, but I'll offer just one last update for future reference if any web crawlers or people searching the forum need or want it.

Osmocom OP25 NXDN example configuration and Osmocom OP25 NXDN modification:

The zip file contains the example json file for two NXDN channels using an RTL-SDR dongle, a script file to start the config in multi_rx.py and log it to a log file, modified terminal.py and multi_rx.py, and a screenshot.

The example configuration and start up script will get you going on NXDN with OP25. They haven't changed much really from a previous post, so I will gloss past them.

./multi_rx.py -c suwannee-nxdn-2022.json 2>> suwannee-nxdn.log

The included multi_rx.py has been modded to provide support for NXDN Voice Channel logging to stderr (the example startup script shows writing and appending stderr to a log file with 0 verbosity so it will just log the voice activity and any major errors that occur).

You can run a variation of the following command in a separate terminal in the apps folder if you wish to follow the log in real time.

tail -f suwannee-nxdn.log

The included terminal.py file has been modded to display NXDN voice calls as they occur on a single NXDN channel on the top bar, and for the previous two calls to go to the activity area at the bottom of the curses terminal.



The terminal.py included has one feature disabled to keep it from potentially crashing, but if you wish for the text labels that come on some NXDN systems to be enabled like they are in the log, you can simply navigate terminal.py to line 252 and delete the comment (number sign, pound sign, or hash) and enable the line.

##Caution: enable name_string line below at your own risk, causes complete string to disappear if malformed, or make terminal.py crash randomly
#a0 += '[%12s] ' % (str(msg['name_string']))

CAUTION: Copy and Paste files into the 'apps' folder at your own risk. Make sure to make a backup/rename of your current multi_rx.py and terminal.py prior to pasting mine in.

If anybody in the future does use anything I've included here, please feel free to post a message and make comments on how it works. Its also possible that future updates made to Osmocom OP25 may nullify, break, or supersede these modifications so keep a-breast of any new changes to Osmocom OP25.

The following has only been testing on two seperate NXDN single voice channels so far, so I can't vouch it will work for everything, but if you want to give it a shot, then please do by all means, and report back if it works for you or not. Also, if you have an NXDN control channel as opposed to single voice channels, it may be worth while to use the default terminal and multi_rx first to see if that suits your needs before swapping mine in.

 

[llamatrails]

@lwvmobile

Thanks ever so much for your posting here. I'm new to this, and from your information, I can now monitor the NXDN traffic here in my area.

I'm running this on a Raspberry Pi 4 8GB Buster 64bit. My RTL-SDR v3 dongle is in a metal out building where the discone antenna is outside it above the roof. The dongle is connected to a Beaglebone Black with VirtualHere sending the USB port over the network to my Pi. To further complicate things, the out building and my home are network connected at the moment with a poweline adapter over the AC powerline between them. Yikes !

I've 'fixed' your issues with terminal.py with the following:

# add the following to the list of the other imports at the top of terminal.py
import re

# about line 252 - add the 2 lines after the commented one, it also lengthens the field
                # a0 += '[%12s] ' % (str(msg['name_string']))
                strdecode = re.sub(r"[^A-Za-z0-9 ]+","",str(msg['name_string']))
                a0 += '[%18s] ' % strdecode

# here is a patch

31a32
> import re
252c253,255
<                 #a0 += '[%12s] ' % (str(msg['name_string']))
---
>                 # a0 += '[%12s] ' % (str(msg['name_string']))
>                 strdecode = re.sub(r"[^A-Za-z0-9 ]+","",str(msg['name_string']))
>                 a0 += '[%18s] ' % strdecode

Working display:



Log file as seen with /usr/bin/less showing the control characters. Also have seen many ^@ characters ...



As for the hang when (q)uit in the terminal window, I make use of job control in the shell: CTL-Z to place the process in the background, and then a 'kill -9 %1' to kill it. The %1 references the number 1 in the [1]+ Stopped, or whatever number shows up with the 'jobs' command built in to the shell.

Rick

 

[lwvmobile]

Thanks ever so much for your posting here. I'm new to this, and from your information, I can now monitor the NXDN traffic here in my area.

Cool, glad to see somebody getting some use out of this makeshift mod I made, and making some improvements upon it as well. Its been so long since I looked at this code, it would take me a while to go back into it and remember and relearn what I did last time. Python isn't my native language programming wise, and manipulating strings is definitely not my forte either. I absolutely despise working with strings .