Correct, the OMAP CPU also has the DSP core. Unless I am off base here, the DSP also serves as the codec which passes the audio datastream to the DAC. After hysterisis filtering, audio pases analog switches for muting and then to the AF preamp and output.
Test points with a test setup, I would scope the audio chain to see where any distortion becomes a factor.
Unlike some earlier radios, the analog audio in P7200 and up are not very healthy. Overdriven preamp, aged coupling caps can certainly be problematic.
Harris did away with the AF analog processor as found in Orion and M7100 that without argument had the best sound.
That backed up by the 28 watt TDA audio chip.
You can try changing some of the DSP init's and maybe do better, but I would still keep an eye open for early distortins and bad levels.
Just food for thought.
I would be interested what you come up with.
PS: A stand alone ADC codec would make this easier, it will have its own registers to manipulate.
Cheers