Decoding D-STAR - any success?

[Kasmus]

This is what I get:

Sync:  +D-STAR    o:  699 mod: QPSK g: 25.000000 inlvl: 22% e:==========R====R=====E====R=====R=======R==E========E=======R====E================R===================R=========R======R====R=======R=========R=============R====R======R======R======R========R=====R====R=======R=====R====R====RM====================R====R==================R========E====R====R===========R==========R====R===========R========R====R====R=====================R===E========R===========R=========R===========R============R=================E=======R===================R===============R===========R=====R====R====RM===E=====R=====R====R====RM====R======R=============R============R==========R========R======R==========R=======R===
Sync: no sync

I know there will be no audio but I expected to/from callsigns and other signalling of that kind.

What would a properly decoded D-STAR output look like?

 

[kb1ipd]

As I understand it, D* uses an early version of the AMBE codec that is not entirely to the proper spec of the final codec and for which documentation has proved unobtainable. Therefore, DSD does not support D*. It's been proposed that it could be decoded by maping it out to something like the DV Dongle. I don't know if anyone has actually managed to do that successfully.

 

[woodpecker]

As I understand it, D* uses an early version of the AMBE codec that is not entirely to the proper spec of the final codec and for which documentation has proved unobtainable. Therefore, DSD does not support D*. It's been proposed that it could be decoded by maping it out to something like the DV Dongle. I don't know if anyone has actually managed to do that successfully.

There already is software that will decode D* headers and voice using the DVDongle, it works very well, it would be nice if it could be incorporated into DSD so we only had to use one application though.

I thought the only reason DSD wouldn't decode the D* audio was because the ambe bit interleaving of the 72 byte voice frame was unknown, looking at the code some patterns appear to have been experimented with but all give garbled audio.

 

[r00ty]

I actually took a look at this. I managed to get this to actually decode the header, and slow data. As well as feed in the audio packets (including the first one, which is always missed in this setup). Also to keep feeding them in, and follow each resync (the current test code dumps 200 voice packets regardless of whether they were received, and then exits).

But, it comes down to one of a few possibilities.

1: The mbelib just cannot decode DSTAR ambe.
2: The interleave is proprietary (it's not mentioned at all in the official documentation)
3: There is some fancy bit rotation/inversion going on (could be covered by interleave, but - well it's not a normal tactic employed).

So, not sure how to proceed.

In any case, not sure what the policy would be of publishing the changed code, since I did include code from other projects I found online, to decode FEC, perform interleaving and the bit scrambling routine.

But, I'll include some info for people interested in doing this.

There are three clear sync markers in a DSTAR, well QSO for want of a better word.

111011001010000 - At the very start of a transmission. It's prefixed by a series 64 bits in the repeating pattern "10". So, this is the market for the packet header.

11010001101000 - On the 1st data packet and then every 21st. It's also prefixed by some "10" combos. This is what is currently used for sync.

000100110101111 - On the final data frame (the final data frame is actually a double data, 48 bits in length).

What I did was capture the first one as a new packet type "DSTAR HD" decode the header (this took a lot of research, on how to decode it). But I did after some hours get it decoding the header information and even validating the checksum. All, I have to say with ripped off code. But, no way I was inventing those wheels for curiosity's sake! I also had the header code pass the very first voice packet into mbelib. Then handed back to the sync scanner.

I then modified the DStar data side to pick up the data packet syncs and read the correct amount of voice/data combos between syncs then hand-off again. In this way, it keeps running while keyed and will only send a short amount of gobbledegook on keyoff. I did consider scanning for the end marker. But, I figured that'd be a fair bit of work for something that doesn't as yet decode any audio. So I skipped that.

The decoding of the data, now that's a weird one. The documentation suggests that the same scramble function is used and is initialized at the start of each data sync signal. The truth is it's reset on every 3 octets of data sent. So, a simple XOR with 3 different values on a rolling effect is enough.

What's left is the 72 bits of DV. Which is the one thing so far I don't think anyone can do anything with sadly. I don't see why they would make a different AMBE 2400bps codec. But, it seems that way, or there is some custom bit encoding/interleaving going on.

Anyway if someone wants to add the similar functionality themselves. Here are the links I used to get this far.

http://www.jarl.com/d-star/shogen.pdf - official D-Star document. Many errors in this. But it's got a lot of useful info in.

What does a GMSK D-STAR stream actually look like? - My Ham stuff - clarified a lot of it into plain English for me. It helped a lot.

https://github.com/on1arf/gmsk/tree/a89f8c3326001a2840bc7f86443968838add080e/receiver It has the code I ripped off (as referenced in the above link). I'm not proud. But it did the job.

Like I say, a pointless endeavour. Since the audio still cannot be decoded. But it was a bit of fun for the night.

 

[woodpecker]

I too had looked at the code by on1arf trying to figure out D*, I thought the difference was the de-interleave pattern and the golay coding.

My understanding is as follows, DSD already extracts the voice frames, 3 x 24 bits = 72 bits.

The de-interleave pattern should be as per on1arfs dstar_dv.c source.

The first 2 blocks of 24 bits are golay protected using 24,12,8 (extended golay with parity), mbelib seems to only have a function for 23,12,7 golay (located in ecc.c). The third block of 24 bits is unprotected.

My theory is ambe3600x2250.c could be modified to decode the D* voice by calling a new golay2412 function rather than the golay2312, applying this to the first 2 voice blocks and just copying the third voice block.

I could not translate this into code, I would love to know if my theory would work, can anyone code it?

 

[r00ty]

Looking at the code, it looks like (in a roundabout way) that's what's done.

From what I could read up on the golay technique, the 24,12,8 was actually pretty much the same as 23,12,7. The parity bit presumably being the 24th bit.

Now, since the audio is broken down into 3 blocks of 24 bits. I am guessing the 23,12,7 is just ignoring the parity. The process that I could see was that block A was "degolayed" and then "demodulated". Then block B was "degolayed". Blocks C and D (presumably empty?) were left as is.Then the remaining data is pumped into the mbe playback function and, gobbledegook comes forth from the speakers.

I could be wrong, and perhaps the golay function does need to be re-written for 24,12,8. But, I then find it weird to use a different FEC function for the dstar ambe implementation.

 

[mrscanner2008]

Hi

I found this http://dstarmauricie.netai.net/PDF/D-STAR_Data_Structure.pdf

maybe it could help !

 

[r00ty]

Hi

I found this http://dstarmauricie.netai.net/PDF/D-STAR_Data_Structure.pdf

maybe it could help !

Hi, this is the same as the shogen document I linked. It was quite useful (along with the code I... Borrowed) in decoding the header and data parts (not the document incorrectly says the voice/data segents are 72/24 bytes when in actuality they are 72 and 24 bits). What happens to the 72bit blocks is the mystery part.

We first of all don't know if the ambe code in mbelib can even decode it at all.
Next we don't know if there is some differing interleave pattern (the different possible ways this could be done are silly big numbers, the number of useful ways are less but still a large number of ways I'd not like to have to brute force personally).
Finally, we don't know if there's something different (as per the FEC suggestion above).

For me, even though I have a D-Star radio. I don't like the proprietary nature of it and am personally rooting for codec2 to produce something nice so I/some other operator can create a totally free open standard. Once that's out there, you would get the software modem, and hobby hardware board designs appear. Of course with the info not being a secret, it'd be trivial to add support to dsd for it.

 

[woodpecker]

Looking at the code, it looks like (in a roundabout way) that's what's done.

From what I could read up on the golay technique, the 24,12,8 was actually pretty much the same as 23,12,7. The parity bit presumably being the 24th bit.

Now, since the audio is broken down into 3 blocks of 24 bits. I am guessing the 23,12,7 is just ignoring the parity. The process that I could see was that block A was "degolayed" and then "demodulated". Then block B was "degolayed". Blocks C and D (presumably empty?) were left as is.Then the remaining data is pumped into the mbe playback function and, gobbledegook comes forth from the speakers.

I could be wrong, and perhaps the golay function does need to be re-written for 24,12,8. But, I then find it weird to use a different FEC function for the dstar ambe implementation.

I thought only ambe frame 0 was being passed to the golay function.

This is what I have tried, but doesn't work.

Changed the de-interleave pattern in dstar_const.h to match that in ON1ARFs source.

Assumed that this block of code in dstar.c performs the same de-interleave:-

// voice frame
w = dW;
x = dX;
for (i = 0; i < 72; i++)
{
dibit = getDibit (opts, state);
dstar = dibit + 48;
ambe_fr[*w][*x] = (1 & dibit);
w++;
x++;
}

In ambe3600x2250.c I copied the existing mbe_eccAmbe3600x2250C0 to ecc ambe block 1, ie I did this:-

int
mbe_eccAmbe3600x2250C1 (char ambe_fr[4][24])
{
int j, errs;
char in[23], out[23];

for (j = 0; j < 23; j++)
{
in[j] = ambe_fr[1][j + 1];
}
errs = mbe_golay2312 (in, out);
for (j = 0; j < 23; j++)
{
ambe_fr[1][j + 1] = out[j];
}

return (errs);
}

In the process frame function it already did ecc c0 with this line:-

*errs = mbe_eccAmbe3600x2250C0 (ambe_fr);

So to ecc block c1 I added this:-

*errs += mbe_eccAmbe3600x2250C1 (ambe_fr);

I think this should now run the ecc/golay 23,12 on frames 0 and 1, but I still get R2D2 from a dstar signal.

So either the de-interleave isn't working the same, the ecc/golay function isn't right or maybe the whole codec is different?

Any ideas?

 

[r00ty]

That code does do the interleaving, just by making them a pointer to the de-interleave matrix and iterating through those to pull in the bits in the interleave order back into the block. With regards to the de-interleave pattern ON1ARF uses, I believe that is only used on the 660bit header. I know that's where I used it, to decode the header data (in conjunction with descramble and the FEC decode he describes). For sure the 24bits of data per frame group are NOT interleaved at all.

So, I'm not convinced the interleave should be applied

The code you added was actually exactly what I did. But what I noticed was that in the data decode section, it applies the routine to block 1 (ignoring 0, 2 and 3)

from mbe_eccAmbe3600x2250Data:

// just copy C0
  for (j = 23; j > 11; j--)
    {
      *ambe = ambe_fr[0][j];
      ambe++;
    }

  // ecc and copy C1
  for (j = 0; j < 23; j++)
    {
      gin[j] = ambe_fr[1][j];
    }
  errs = mbe_golay2312 (gin, gout);
  for (j = 22; j > 10; j--)
    {
      *ambe = gout[j];
      ambe++;
    }

  // just copy C2
  for (j = 10; j >= 0; j--)
    {
      *ambe = ambe_fr[2][j];
      ambe++;
    }

  // just copy C3
  for (j = 13; j >= 0; j--)
    {
      *ambe = ambe_fr[3][j];
      ambe++;
    }

So, by adding that code we're effectively processing it twice.

And looking at the process overall it looks like:

from mbe_processAmbe3600x2250Framef:

 *errs = mbe_eccAmbe3600x2250C0 (ambe_fr);
  mbe_demodulateAmbe3600x2250Data (ambe_fr);
  *errs2 = *errs;
  *errs2 += mbe_eccAmbe3600x2250Data (ambe_fr, ambe_d);

  mbe_processAmbe2250Dataf (aout_buf, errs, errs2, err_str, ambe_d, cur_mp, prev_mp, prev_mp_enhanced, uvquality);

So you can see the process is, ungolay block 0 (first 24bits).
Demodulate (this is weird, because as far as I could see the string wasn't passed as a pointer, so does this actually work? Or is a parameter passed as a reference by default in C? I don't use C when programming for work, so I'm not totally up on how that side works).
Then ungolay block 1, pass through 0, 2 and 3.

So, I think it goes back to being either:

1: A yet unknown interleave pattern
2: The mbe library doesn't handle the iteration of AMBE used in dstar
3: some other adjustment to the data before it's fed to the AMBE decoder.

It occurs to me that I am sure I've seen software that can decode D-star using a sound card, and the DV-Dongle. I would hazard the solution could be ascertained by decoding the output of the voice frames (raw) and comparing what is passed to the DV-Dongle (which is effectively an interface to the AMBE-2020 chip). The FEC is done on chip, as far as I understand. So I doubt there would be a difference between that and the other AMBE implementations.

Looking at ON1ARF's code again, it certainly looks to me like the 72bits of AMBE data are combined into 9 byte blocks of "DVTOOL" format data which is sent to the USB dongle. So unless the data is manipulated there before being sent to the AMBE2020 chip. It suggests more of a codec difference.

 

[woodpecker]

That code does do the interleaving, just by making them a pointer to the de-interleave matrix and iterating through those to pull in the bits in the interleave order back into the block. With regards to the de-interleave pattern ON1ARF uses, I believe that is only used on the 660bit header. I know that's where I used it, to decode the header data (in conjunction with descramble and the FEC decode he describes). For sure the 24bits of data per frame group are NOT interleaved at all.

So, I'm not convinced the interleave should be applied.

You could be right, I have also tried the pattern that does no de-interleaving and it still doesn't work.

The code you added was actually exactly what I did. But what I noticed was that in the data decode section, it applies the routine to block 1 (ignoring 0, 2 and 3)

from mbe_eccAmbe3600x2250Data:

// just copy C0
  for (j = 23; j > 11; j--)
    {
      *ambe = ambe_fr[0][j];
      ambe++;
    }

  // ecc and copy C1
  for (j = 0; j < 23; j++)
    {
      gin[j] = ambe_fr[1][j];
    }
  errs = mbe_golay2312 (gin, gout);
  for (j = 22; j > 10; j--)
    {
      *ambe = gout[j];
      ambe++;
    }

  // just copy C2
  for (j = 10; j >= 0; j--)
    {
      *ambe = ambe_fr[2][j];
      ambe++;
    }

  // just copy C3
  for (j = 13; j >= 0; j--)
    {
      *ambe = ambe_fr[3][j];
      ambe++;
    }

So, by adding that code we're effectively processing it twice.

And looking at the process overall it looks like:

from mbe_processAmbe3600x2250Framef:

 *errs = mbe_eccAmbe3600x2250C0 (ambe_fr);
  mbe_demodulateAmbe3600x2250Data (ambe_fr);
  *errs2 = *errs;
  *errs2 += mbe_eccAmbe3600x2250Data (ambe_fr, ambe_d);

  mbe_processAmbe2250Dataf (aout_buf, errs, errs2, err_str, ambe_d, cur_mp, prev_mp, prev_mp_enhanced, uvquality);

I missed that, yes block c1 gets ecc/golay from that function so doing it again is making things worse.

So you can see the process is, ungolay block 0 (first 24bits).
Demodulate (this is weird, because as far as I could see the string wasn't passed as a pointer, so does this actually work? Or is a parameter passed as a reference by default in C? I don't use C when programming for work, so I'm not totally up on how that side works).
Then ungolay block 1, pass through 0, 2 and 3.

I think this works ok because the DMR deocoding uses it which works very well.

So, I think it goes back to being either:

1: A yet unknown interleave pattern
2: The mbe library doesn't handle the iteration of AMBE used in dstar
3: some other adjustment to the data before it's fed to the AMBE decoder.

It occurs to me that I am sure I've seen software that can decode D-star using a sound card, and the DV-Dongle. I would hazard the solution could be ascertained by decoding the output of the voice frames (raw) and comparing what is passed to the DV-Dongle (which is effectively an interface to the AMBE-2020 chip). The FEC is done on chip, as far as I understand. So I doubt there would be a difference between that and the other AMBE implementations.

I have the d-star software that uses the dongle to decode from a soundcard, it works very well but I don't think it will help because everything we need to know is done inside the AMBE2020 inside it.

Looking at ON1ARF's code again, it certainly looks to me like the 72bits of AMBE data are combined into 9 byte blocks of "DVTOOL" format data which is sent to the USB dongle. So unless the data is manipulated there before being sent to the AMBE2020 chip. It suggests more of a codec difference.

I didn't know he was using a dongle, are we looking at the same code, I am looking at the code for extracting DTMF and silence frames to measure performance (dtmf-rcq), a lot of it was originally written by someone else, I thought it was pulling out the DTMF code data from the raw stream, to do that I thought he had completely decoded the stream which is normally only possible inside the chip. Maybe there are too many unknowns to make this work?

Can you share the code to decode the headers and data, I could probably cobble it together but it would take me more hours than I have to spend on it.

 

[r00ty]

I didn't know he was using a dongle, are we looking at the same code, I am looking at the code for extracting DTMF and silence frames to measure performance (dtmf-rcq), a lot of it was originally written by someone else, I thought it was pulling out the DTMF code data from the raw stream, to do that I thought he had completely decoded the stream which is normally only possible inside the chip. Maybe there are too many unknowns to make this work?

I was looking at the gsmk receiver/transmitter projects.

I'm looking at those because when I get some free time I was thinking about using the alpha code from codec2 to produce an implementation of a truly open codec, using a sound card and a simple PTT circuit.

When I get time! Always the way

Can you share the code to decode the headers and data, I could probably cobble it together but it would take me more hours than I have to spend on it.

Sure can, not sure what the implications of publishing this is. So I'll pop up links to only the changed files. But it'll need to wait until I get home Monday night.

 

[groovy]

So close yet so far

A friend and I have spent the past few evenings replicating what woodpecker and r00ty have done. We now have it syncing on both the header and the sync pattern, and decoding the header. (Borrowed some of that from https://github.com/on1arf/gmsk/tree/master/gmskmodem_dstar_raw/gmskmodem/). It reads until it misses multiple syncs, and resyncs if it finds the sync pattern when not expected.

By using the interleave pattern from dstar_dv.c (from ircDDB-mheard) and some careful inspection/guesses, we have no golay errors on the C0/C1 packets (Observation- mbelib doesn't check the parity bit of the first golay, so it is validating a golay 24 as a 23). But I'm still not sure how the interleave for those C2/C3 packets should go- I can see in mbelib that we only look at 11 bits of the first, and 14 of the second. So still no understandable audio. Closer yet still far.

Also in the mix are changes to build on Macs and use of libsndfile to allow 48k .wav files as input- really useful for testing/debugging. We'll try to release what we've done, but some of the Dstar code adds license complication- The code borrowed is GPL licensed.

I'm curious if anybody else has made any progress on this.

 

[groovy]

An interleave table to work with

In case it helps anybody experimenting, here is the interleave table based on the logic from dstar_dv.c. It would be a normal interleave except each byte is reversed, so it looks messy. The C2/C3 numbers are wrong, and the only reason the C0/C1 might be right is a lack of golay errors on either round. The "odd man out" is the 67th bit, which would be the first bit of the C1 block, but isn't since that block is a bit shorter, so it must belong to the C2 or C3 blocks... so many possibilities...

const int dW[72] = {
  0, 0, 2, 3, 1, 1,
  0, 0, 1, 1, 0, 0,

  2, 3, 1, 1, 2, 3,
  1, 1, 0, 0, 2, 3,

  0, 0, 2, 3, 1, 1,
  0, 0, 1, 1, 0, 0,

  2, 3, 1, 1, 2, 3,
  1, 1, 0, 0, 2, 3,

  0, 0, 2, 3, 1, 1,
  0, 0, 1, 1, 0, 0,

  2, 3, 1, 1, 2, 3,
  3, 1, 0, 0, 2, 3
};


const int dX[72] = {
	10, 22, //c0
	11, 12, //c2or3
	10, 22, //c1
	11, 23, //c0
	8,  20, //c1
	9,  21, //c0

	10, 11, //c2orc3
	9,  21, //c1
	8,  9, //c2orc3
	7,  19, //c1
	8,  20, //c0
	9,  10, //c2orc3

	6,  18, //c0
	7,  8, //c2or3
	6,  18, //c1
	7,  19, //c0
	4,  16, //c1
	5,  17, //c0
	6,  7, //c2or3
	5,  17, //c1
        4,  5, //c2or3
        3,  15, //c1
        4,  16, //c0
        5,  6, //c2or3

        2,  14, //c0
        3,  4, //c2or3
        2,  14, //c1
        3,  15, //c0
        0,  12, //c1
        1,  13, //c0

        2,  3, //c2orc3
        1,  13, //c1
        0,  1, //c2orc3
        2,     // c2orc3
        11, //c1
        0,  12, //c0
        1,  0 //c2or3
};

 

[kb9mwr]

https://github.com/dl1bff/ircDDB-mheard/blob/master/ircDDB-mheard.c

This is Michael, DL1BFF's code.
Line 383 - his code handles the bit interleaving and FEC processing

 

[kb9mwr]

As I understand it, D* uses an early version of the AMBE codec that is not entirely to the proper spec of the final codec and for which documentation has proved unobtainable.

A TIA document (TIA-102.BABA-1) (PN-3-3633-AD1) titled "APCO Project 25 Half-Rate Vocoder Addendum" is for AMBE+2 which is similar but not identical to AMBE+ as used in DSTAR.

 

[groovy]

Just an update...

There are other people working on this far more than I am, but from what I understand and what rumors I have heard:

The AMBE+ used by DSTAR is a previous generation from the AMBE+2 documented in TIA-102.BABA-1 and patent US8359197. Based on the mbelib source code, mbelib author was working off of TIA-102.BABA-1. The codec that we therefore have is the newer 2450bps voice/1150 FEC as used by DMR/P25ph2/NXDN/etc; the DSTAR codec is supposedly 2400bps voice /1200bps FEC. The bit allocation within the codec, the quantization tables, and the tone frame formats are all likely different. Should we ever be able to decode DSTAR well, it would have to be with a new codec type entirely within mbelib.

 

[EricCottrell]

Hello,

I finally was able to decode dstar voice. It took some effort, but I do not use mbelib for decoding. I modified dsd to use a USB3000 P25 months ago, so I have the right codecs for the different digital modes.

One of the modifications to dsd was to take out the interleaving on the voice frames of the various modes. This bit interleaving is part of the codec, so it is done by the USB3000. I read that DStar sends the bits as they come out of the AMBE 2000, so I though I just had to undo the interleaving. Well it did not work!

I tried several different interleaves, including reversed bit order, but still it did not work. The comment about reversed bytes gave me an idea. I reversed the bit order in each byte while keeping the same byte order. It worked! All the other digital voice modes I decode with DSD send the MSB of the first byte first, but DStar sends the LSB of the first byte first.

The interleave pattern in the ircDDB-mheard code looks like a 6x12, so I took the standard table and bit reversed the bytes. The results are not quite the same as Groovy's table.

/*
 * 6 x12 byte reversed
 */

const int dW[72] = {
0,0,3,2,1,1,
0,0,1,1,0,0,
3,2,1,1,3,2,
1,1,0,0,3,2,
0,0,3,2,1,1,
0,0,1,1,0,0,
3,2,1,1,3,2,
1,1,0,0,3,2,
0,0,3,2,1,1,
0,0,1,1,0,0,
3,2,1,1,3,3,
2,1,0,0,3,3
};

const int dX[72] = {
10, 22, 11,  9, 10, 22,
11, 23,  8, 20,  9, 21,
10,  8,  9, 21,  8,  6,
 7, 19,  8, 20,  9,  7,
 6, 18,  7,  5,  6, 18,
 7, 19,  4, 16,  5, 17,
 6,  4,  5, 17,  4,  2,
 3, 15,  4, 16,  5,  3,
 2, 14,  3,  1,  2, 14,
 3, 15,  0, 12,  1, 13,
 2,  0,  1, 13,  0, 12,
10, 11,  0, 12,  1, 13
};

73 Eric

 

[groovy]

Eric- I agree on the interleave- As I said I was unsure about the 3/2s, and in the end we ended up at the same place. Until I crashed headlong into other differences in the codec...

On decoding with the USB3000- That is neat- given that device, all of mbelib could probably be replaced. Those run about $300?

On the subject of mbelib decoding dstar- give the people involved some more time, but significant progress is being made. From the recording I most recently heard, the decoding is quite reasonable, with some artifacts that I'm sure will be fixed in the coming days. Stay tuned.

And finally, I don't know if many people know, but there is a much improved implementation of imbe (both decoder as well as encoder!) available as part of the OP25 project. Supposedly the code is significantly more efficient than the code in mbelib. See imbe_vocoder in trunk

 

[AZScanner]

On decoding with the USB3000- That is neat- given that device, all of mbelib could probably be replaced. Those run about $300?

Yes, but at that point, you could almost buy a digital scanner. I like the idea of keeping mbelib around so that people don't have to drop $300+ to monitor a digital radio system.

And finally, I don't know if many people know, but there is a much improved implementation of imbe (both decoder as well as encoder!) available as part of the OP25 project. Supposedly the code is significantly more efficient than the code in mbelib. See imbe_vocoder in trunk

It would be great if the process of getting OP25 working in Windows could be demystified a bit. I can install GNURadio but I have no idea how to go about compiling OP25 to work in it. Is there a tutorial online anywhere that explains how to set this up?

-AZ