The thread that Mike linked has an extensive discussion of this issue.
In order for an OFT system to work successfully, there are several steps that need to be taken.
Enter the correct frequency, and if known, enter the correct RAN.
Be sure that the OFT system is set for 'ID Search' unless you are positive about any TGID being used.
The OFT System must have a Department. Otherwise, the scanner will ignore it. Also, that Department must include a talkgroup. If you do not know what talkgroup is used, then enter a place-keeper talkgroup, such as imply '1'. If you are positive about the TGID, enter it. Note that leaving the OFT system set to use ID Search will also enable you to capture any other talkgroups on the same frequency.
Note that even though NXDN is a narrow signal, I have better results if I set modulation to FM, not 'auto' or 'NFM".
I was the OP previously, and despite my attempts to resolve it, even sending in debug files to UPMan at the time, it was diagnosed as too weak of a signal. The system I was attempting to monitor was a rental car company which was within a mile of my location.
I went as far as taking my SDS100 a block away, identical set up and didn't have any luck. I submitted another debug file from my SDS100, and never received a response back. While a rental car company isn't the most exciting listening, trying to decipher a OFT NXDN system was the goal.
I've only encountered two OFT NXDN systems and had no luck. As hiegtx mentioned, a Department is needed, but I still ran it in ID search and in both NFM and FM mode without any improvements. My "unscientific take-it-for-what-its-worth opinion" is the x36hp/SDS series of scanners have a difficult time locking onto Talkgroup 0 (which seems to be common in the two systems I monitored). The reception was sporadic at best, and when compared to another scanner running in conventional mode, I missed alot of transmissions. I base this on OFT MotoTRBO systems, which use TGID 1 in some instances. In that case, the scanners performed very well.
I have no other suggestions other than what's been mentioned. If we ever see another firmware upgrade, hopefully this will be addressed.