Are any vendor's P25 systems using automated self-test and self-calibration?

[xmo]

In the topic about Motorola's old headquarters (and related nostalgia discussion) elroy asked: "are the current editions of any vendor's P25 systems using an automated self-test and self-calibration system yet?"
-----------

I feel that a discussion about automated self test / calibration is totally unrelated to Motorola nostalgia and interesting enough in its own right to warrant a new topic so I have started this one.

902 posted: "Fixed-end or subscriber? BK uses an automated calibration setup at their factory. Unfortunately, the results of the test setup and the actual device under test sometimes don't mesh on the bench. Not aware of anyone doing that on fixed-end, but there may be over-the-air eye pattern analyses."

902 makes a good point, elroy. Is there anything specific you're looking for?

 

[AM909]

Pinging @902

 

[xmo]

elroy wrote: " I was referring to the fixed site system equipment, incidentally. Seems to me that these days it's very technically feasible for all sites to self-monitor and automatically realign themselves as needed. Embed the measurement systems in the site, provide switching networks to connect the radio equipment to the test equipment, and from there on it's software applications. Anything short of realigning duplexers and combiners should be able to be done via software. "

 

[xmo]

I am working on an answer that explains the current Motorols infrastructure state-of-the-art in this context. Be forewarned, it will be a long read.

 

[xmo]

So, let’s take a look at how a state-of-the-art Motorola regional public safety system takes care of itself and assists the support staff. This system is comprised of several counties, each with its own RF ‘footprint’

Reliable backhaul is the top level. You don’t just plug a few routers into internet connections. At the top level is a regional microwave ring, carrier class, real radios, dishes, and waveguides; dual polarity 2xOC3 BW, fully redundant, with MPLS SARs at each node. This ring connects the DSR master CORE sites, and touches ever county.

Within each county, a similar OC3 carrier class ring connects all sites: dispatch console, prime, and RF, again with MPLS SAR’s at every node. In addition, critical paths such as CORE to CORE, CORE to prime, and CORE to dispatch console have parallel fiber for triple redundancy.

A high level network services software platform automates operations and provides the tools for monitoring, administering, and controlling the entire backhaul network. Administrators are immediately notified of any outages which are automatically compensated for with no disruption of bearer services.

Two types of ‘users’ ride the backhaul. Customer network (CEN) and Motorola (RNI). The CEN carries everything from security cameras, standby generator status / control, CAD, and multi-node IP 911 telephony. Thus, every dispatch location, primary or back-up, is capable of handling the functions needed for any of the connected 911 centers. In a forced evacuation, protocols transfer functions to neighbors until dispatch staff can relocate.

The radio system network (RNI) is totally isolated through epipes, tunnels, and other such network esoterica. The DSR COREs are constantly communicating with each other so that the physical loss of the active one results in the other immediately carrying the full system. I say physical loss because that’s the point of DSR. Otherwise, every component in a master CORE is fully redundant so that any single entity can fail or be replaced with zero effect on users.

Every other Motorola site: console, prime, or RF is also equipped with two of everything – switches, routers, references, controllers, voters, etc. Every network element in the system from the zone CORE process down to the base stations continuously monitors its own health and provides real-time updates to the Unified Event Manager (UEM).

In addition, MOSCAD RTUs monitor and control site environmentals – everything from intrusion, smoke, high/low temp, and room lights, to TTA status. All these also go to the UEM. Based on filter levels, support staff receive immediate notification right to their phones in case of any significant event.

At the far end of all this exists the whole point of a radio system – radios. I suspect that's where elroy's interest lies - we'll take that on next time.

 

[ElroyJetson]

Thanks for moving this over to this subform. I appreciate the assistance.

Yes, being as I'm primarily a hardware based radio kind of technician, it's the actual RF equipment at the trunked sites that I'm mostly inquiring about, as I'm certainly several years behind the times.

I would look at the alignment aspects of a trunked multisite simulcast system as an example. As you'd probably know, multicast systems have very tight specs for keeping the sites and individual repeaters closely matched for frequency and modulation levels, including audio balance. Audio balance and modulation should ideally be kept to within 1 percent variance so as to make the subscriber radio transition from site to site with hardly a trace of on-channel audio interference.

I'm not ignoring the importance of the system at levels above the trunking site RF systems, but they're really beyond MY scope as I've never had any hands on experience with that, or any reasonable prospects of getting that job. (I'm semi retired anyway. )

 

[xmo]

Elroy wrote: “I would look at the alignment aspects of a trunked multisite simulcast system as an example. As you'd probably know, multicast systems have very tight specs for keeping the sites and individual repeaters closely matched for frequency and modulation levels, including audio balance. Audio balance and modulation should ideally be kept to within 1 percent variance so as to make the subscriber radio transition from site to site with hardly a trace of on-channel audio interference.”

First, let’s clear up “audio balance”. There is no “audio” in a P25 infrastructure, only vocoded audio packets. The vocoding takes place at endpoints – subscriber radios, console VPMs or the dynamic transcoder. The entire infrastructure just moves the vocoded audio packets from one place to another.

The job of the base station is to transmit the CAI modulation containing the vocoded audio ‘payload’. As always, there are three critical aspects to this transmission: carrier frequency, launch time, and modulation fidelity.

Carrier frequency and launch time are locked in by redundant GPS references at every site. For specific coverage requirements, launch time can be offset at certain sites by a configuration setting.

With digital systems, modulation fidelity is basically the exact correspondence of each transmitter’s signal to the CAI standard and is principally characterized by error vector magnitude measurement. With Quantar and DSM-II, there were critical measurements to make in the field to ensure the best modulation fidelity. With GTR, modulation fidelity can be verified in the field but not adjusted. The factory sets this in each FRU more accurately than would be possible in the field. Station internal calibration routines then keep this accurate over time and with temperature variations.

So there are no 'alignment aspects' to one of these systems. There is nothing to ‘tweedle’. You can retire your pocket protector.

 

[ElroyJetson]

Even then, unless the architecture of the transceivers has changed RADICALLY in the last few years, the transmitters still have to be aligned, and modulation levels have to be balanced or it distorts the digital modulation. But apparently these new system DO use a fundamentally different archictecture, so that answers my question. I suppose my questions would have been more applicable to previous generations of equipment that were made to operate in either analog or digital modes. I get the idea that today's current generation systems may not even support an analog mode of operation?

Like I said...I'm not current on the technology. I'd like to get a little bit less out of date, if not fully current.

 

[GTR8000]

I get the idea that today's current generation systems may not even support an analog mode of operation?

The GTR 8000 repeaters support both analog and digital (P25 only) natively.

There are countless GTR repeaters in use today transmitting analog signals, with 2.5, 4.0, or 5.0 TX deviation, standalone or simulcast. Or they can do P25, either conventional standalone or simulcast (including LSM for conventional simulcast, although not very common), or trunked FDMA, TDMA, or DDM (Dynamic Dual Mode, i.e. preferring TDMA but will fall back to FDMA based on the capability of affiliated subscribers). The GTR PA is, of course, a linear amplifier. There are even older SmartZone 3600 systems that have had their old Quantars retired, and are running on brand new GTR repeaters.

Most current offerings from the big manufacturers are no different.

 

[xmo]

" The GTR 8000 repeaters support both analog and digital (P25 only) natively. "
--
The GTR is a heck of a radio for any high tier application from mutual aid repeater to paging station to Quantar replacement.

All the Quantar functions of TRC, enhanced wildcard, and SAM are all in there.

 

[PACNWDude]

Thank you for the description xmo, this is better than some of the Motorola instructors I have dealt with over the years, especially with UEM and MOSCAD info. I have sent several people to this site, as sometimes it is better than sitting in a weeklong class. This is one example that gets to the point.

 

[xmo]

elroy wrote: "Even then, unless the architecture of the transceivers has changed RADICALLY in the last few years, the transmitters still have to be aligned, and modulation levels have to be balanced"
--------------------
Things have changed radically. The recommended maintenance section of the GTR ESS manual tells you: "If dust has accumulated on the fan grills, cleaning the fan grills is recommended." That's it. Your toolkit is a soft brush.

Unbelievable.

There is really nothing to ‘align’ in an Astro25 GTR. All that is done at the factory. At commissioning, the Motorola ST’s set the system up so that each station knows its role – how many watts to put out, what its receive distribution system looks like, etc. The STs also document (in an excel workbook) the performance of every RF component at each site so that future verification is easy.

During operation, the system constantly monitors all the stations for proper operation. The system, as with previous trunked versions, monitors every station at each site for failures such as low RF output, and takes channels out of service as necessary. However, today’s systems go well beyond yesterday’s. Every station is part of the system IP address plan. All alarms and events go to the UEM for notification, logging, etc.

From a central point (or elsewhere on the RNI) a technician can take a station out of service, connect to it with CSS, and do anything from checking status to updating firmware. You can turn on a test carrier at a centralized location and then read the over-the-air RSSI seen by the station, thus verifying everything from the antenna on down - all from your desk. Those GTR RSSI readings are spot on, too.

If a module does fail and needs replacement, a junior tech can go to the site to do the physical swap while the senior tech handles all the CSS tasks remotely.

There are two antenna inputs to each station. For phase 2, these provide diversity receive but at all times the system is constantly comparing the two RSSI values. A major difference generates an imbalance alarm. If the primary appears to have failed, the system will automatically switch to the secondary for FDMA calls.

As with previous generations, the system monitors for ‘illegal carriers’. An unwanted signal present for a preset time causes an alarm and temporarily takes the channel out of service – but – today’s systems do more. If the signal is present at two or more sites, the system can use TDOA techniques to provide a geolocation of the interferer.

The problem is that ‘illegal carrier’ monitoring only detects the worst problems such as a stuck transmitter. Most issues are shorter in duration -
IM events or anomalous propagation. You need to know about them. It’s pretty hard for test equipment – or even a base station – to differentiate wanted from unwanted signals on a traffic bearing channel – but – there should never be any signal present at the receiver of an idle channel.

The system knows this, and, if a carrier comes up on an idle channel for more than 2 seconds, this generates an ‘event’ that gets reported to (and logged by) UEM. Now the technician can track these events vs time of day, site, channel, etc and correlate them with other data.

Where does ‘other data’ come from? A Keysight signal intercept and collection system. A Keysight sensor is installed at several key RF sites. Each sensor has two antenna inputs. One is connected to the TTA MCU master receive system and the other has its own dedicated antenna.

Think of a “sensor” as a wideband RTSA in a box connected by the IP network. This system can be configured to monitor specific frequency bands or even a single channel for unwanted signals. I/Q data is recorded for demodulation and analysis. Applications permit listening to analog traffic, rapid identification of known signal formats, or in depth analysis using Keysight’s Pathwave VSA.

Data from interfering signals can be processed through an app to geolocate the source. All this can be configured to run continuously, unattended or to notify support staff of specific events. For further information, see: N6820ES Spectrum Monitoring Surveyor 4D Software

Bonus points if you can look at this VSA screen capture and deduce what is being monitored!

 

[rr60]

This is a fascinating thread. Interesting for me to learn modern design.

The Keysight system takes care of the last mile, the spectrum. The equipment is only as good as the spectrum it uses. Garbage in garbage out.

This thread describes well redundancy and reporting. Thank you @xmo nice job.

In practice, I suspect this best practices modern design, is a rarity. Is this design more and more common place. Or perhaps in the minority?

On the graph, just a wild guess, DTV T-Band TV signal perhaps?😂

 

[902]

elroy wrote: " I was referring to the fixed site system equipment, incidentally. Seems to me that these days it's very technically feasible for all sites to self-monitor and automatically realign themselves as needed. Embed the measurement systems in the site, provide switching networks to connect the radio equipment to the test equipment, and from there on it's software applications. Anything short of realigning duplexers and combiners should be able to be done via software. "

Simulcast Solutions used to make a self-training analog simulcast system that would "train" at a pre-designated time through a separate receiver. The fire system at a former employer was set up to do this at 0300 hrs. every morning, and it would line up the audio phasing so that overlaps are placed in areas where . The master oscillators were Spectracom GPS-disciplined clocks with an Rb fail-over. Those were driving MASTR III simplex base stations on 2 frequencies, an alerting frequency and a routine operations/initial response frequency.

The Sheriff's Motorola system was WPA, which used looser standards and wider bandwidth than actual P25, but was P25 digital protocol at the wider bandwidth, which meant they could use less transmitter sites, but the system was supplanted by the 2013 mandate. That used some sort of pinging scheme through the microwave DS0 and fractional T1s to compensate for phasing and the RF was also sync'ed by Efratom GPSDO/Rb clocks. I'm not sure what the latest Motorola products use, particularly for Phase 2/TDMA, although timing is more critical in a TDMA system and I suppose (but I have never seen referenced) the DMR-like max-usable distance limit applies, too, for Phase 2. That might also be problematic in air ops where simulcast overlaps can't be controlled-for.

XMO, I learned a great deal from your post!

One of the things that came to mind during my first exposure to Motorola PrivacyPlus (and subsequent SmartNet) systems years back was its intolerance of co-channel interference. That just wasn't the "real world" back in the day, although today's systems are engineered for specific coverage (thinking of Regions 8, 30, and 55 "responsible radiation control" percentage of saturation criteria). For VHF and UHF systems, that's still problematic and unrealistic, even with FB8 status and engineering given a K factor excursion away from the normal 4/3 Earth, like during tropo.

If I had to guess, I'd say that vector signal analyzer is monitoring, among other things, for shared-use non-FB8 FB2 repeaters in a given system and using the output as arbitration to protect co- and adjacent channel repeaters as required in 90.187. I can already tell I'm wrong because I zoomed in and could make out an 855 MHz frequency, so??? Power flux density readings from adjacent cellular operations?

 

[xmo]

Sorry the VSA screen capture is fuzzy. One thing you can see is “2FSK”. Also the spectrum view span is around 20 kHz. I thought someone might recognize the spectrum. There’s only one two-way modulation that looks like this.

We have certainly come a long way. I still remember being called out on Christmas eve 1970. I got a panicked call from the police dispatcher “Help! We’re off the air!”

A quick trip to the shop to pick up the service van, then off to the site – at a water tower – is there a colder place on earth?

There I am, standing in the dark, in snow up to my keister, wind and snow swirling around, in front of a Motorola upright, solid state (8560 final) repeater in a ‘weatherproof ‘cabinet, changing out the keying SCRs

Bah..

Today, there’s no panicked call. You get a notification straight to your phone. If it looks serious, you VPN in, RDP your way to anything you want, and decide how serious the issue is. If you decide it actually warrants a trip to the site, you are going to a site with a purpose built shelter, well lit, with redundant HVAC.

You pull up, enter using the tech-on-site protocol, take off your coat, hang it on the provided hook, walk to the lead-lag controller, and press the ‘comfort’ button. In minutes you are at the set-point. You pull a task chair up to the site bench and power up the site computer.

Now you have access to all system and site records, IP plan, engineering drawings, antenna return loss & DTF sweeps, TTA test data, and all the ST commissioning records. You can log into anything on the RNI, assess the situation, and decide if something needs to be replaced. If so, you get a FRU and toolkit out of the site’s spares cabinet.

If you need test equipment, you have the same exact Anritsu LMR Master used by the Motorola STs.

When you are finished, if it’s been a while, you hit the remote start on your fob so the vehicle can warm while you enter your work in the site records. Then, you’re on your way home, the coldest you got was in the 20 foot walk from the shelter back to your vehicle.

You may not be able to remember the last time you used a soldering station, but – life is good at the state-of-the-art!

It is fascinating to watch one of these shelters being ‘set’ – see attached.

 

[xmo]

To clarify how my description relates to the entire regional system – this system - as have many others – grew incrementally. Each entity in this system is responsible for purchasing its own infrastructure and for maintaining it. Some of the systems started many years ago and as upgrades were necessary, opportunity was presented to join with others.

Because of the different history, there will be differences in things like shelters, antennas, etc. However, the backhaul is the same as previously described everywhere, all the entities are using GTR ESS, and all are on the same Motorola system release and same SUA schedule. So all the UEM capabilities, GTR maintenance, diagnostics and performance descriptions pertain to the entire system.

Also, because each entity is responsible for its own maintenance, which optional support features are chosen is up to the needs of each organization. Every optional feature I have discussed is in place at one or more of the member systems.

The daily operations and administration of the entire system is the responsibility of a group of radio people. A management team – department directors - is available for sensitive and policy issues and to keep interlocal agreements update, pay bills, etc – but they don’t get involved in day-to-day unless requested.

Imagine, radio techs running something like this.

Unprecedented.

 

[902]

Today, there’s no panicked call. You get a notification straight to your phone.

Where's the adventure in that?! No rat droppings or dead vermin inside the base station cabinet...

Imagine, radio techs running something like this.

I can't. It would be IT script kiddies who couldn't tell the difference between a radio and a toaster oven.

 

[littona]

Where's the adventure in that?! No rat droppings or dead vermin inside the base station cabinet...

I can't. It would be IT script kiddies who couldn't tell the difference between a radio and a toaster oven.

Or rattlesnakes greeting you at the door!

 

[xmo]

I can't say that I ever had to deal with rattlesnakes - although I suspect that - in Council Bluffs - snow up to your keister is a much more common problem.

I do hear of nightmares when IT owns communications. It makes some sense to mini-mangers (pointy-haired types) when you look at the overall system and it appears to be a giant computer network (in fact - it is)

In our case, 911 directors have had enough bad IT experiences that keeping control over mission critical systems warrants dedicated staff. We reciprocate by ensuring continuity of operations and responding quickly to any management requests. That sets us apart from general IT whose usual response is 'we'll look into that and get back to you'

 

[xmo]

To wrap this up, let’s look at some things the support staff can do to keep the system always performing its best and operational in the event of failures.

In addition to all the Motorola support tools like UEM, affiliation monitor, ATIA viewer, historical reports, and zonewatch, the system has a shared Genesis Genwatch3 ATIA with Subscriber Access Manager, formerly known as CloneWatch.

These tools give system admins real-time understanding of the system performance as well as the ability to drill down to the root cause of any issues that occur.

Any system can be subject to failures that can degrade service. Pre-planned responses can minimize the duration of those problems.

Suppose high VSWR is reported at a site. You can quickly assess the issue right from your desk. You take a channel out of service, key the channel at the suspect site, and compare its OTA level at one of the sensor sites to the data records from commissioning time. A probable defective antenna can thus be identified.

A tech can go to the site to run return loss and DTF tests to compare to site records. Then, the tech can reroute transmit RF to the spare (mutual aid) antenna to restore the site to service until a tower crew can attend to the problem.

Because of the critical nature of site receive, there are multiple action plans to address possible issues.

Excess signal levels at an important site due to close in mobiles or control stations can potentially overload the site receivers and allow IM products to be generated. To preclude this, an automated system constantly monitors the MCU output for level excursions and inserts attenuation as needed until the event has ended. This event is logged in UEM for further analysis by support staff.

Frequent occurrences can be investigated by assigning the KEYSIGHT sensors to monitor the receive band at the site for excursion above the IM threshold level. That data will then show the exact offending frequencies with time stamps. If a high level signal is on a system channel, the system and Genwatch records will identify the exact offending unit.

The Keysight data can geolocate where the unit was when the incident occurred. If necessary, the unit can then be brought in to see if the problem is due to a gain antenna or the transceiver power out having been set too high.

Site receive in multi-channel systems relies on a TTA and multicoupler. Obviously, those components are critical because a failure can affect all channels. In this system, the TTA/MCU is given special attention.

Each TTA/MCU was extensively tested both before and after installation. With two antennas for diversity, one receive antenna or TTA branch can fail with only the small loss of TDMA diversity gain and no impact on FDMA. If that happens, alarms immediately advise staff of the problem.

In addition to the main receive lines, every TTA has a test cable so that comprehensive tests can run at any time. A failed MCU can be quickly replaced with one of several spares. In the event of a TTA failure, replacement spares are on hand. The problem with replacing a TTA is that tower crew scheduling and weather delays could leave a site off-line for an extended period.

That’s unacceptable. To deal with TTA failure, there is a TBA plan. TBA = Tower Bottom Amplifier. A special unit configured with a preselector filter, ultra low-noise amp, and distribution can be connected to the site’s spare / mutual aid antenna and receive service restored quickly with virtually the same reference sensitivity.

Collectively, the triple redundant backhaul, DSR configuration, redundant prime sites, “two-of-everything” site design, on-hand spares, and failure response action plans combine to give this system the state-of-the-art in continuity of operations.

Last, let’s take a quick tour of some of the rest of the pieces that comprise the complete system.

Although every site is equipped with a diesel standby generator, there are also 100 KW G.O.A.T.s - multimode 120/240, 208/277/ 480 with staff trained on rapid deployment and operation. These use camloks to connect to critical console, CORE, prime, and RF sites where a second (manual) transfer switch allows quick deployment.

WAVE service is available to extend key talkgroups to designated manager’s cellphones.

A regional medical communications plan is in place with equipment deployed at every hospital and ER

All subscriber radios are ASK and password protected with all templates in radio management.

The system is equipped with IV&D OTAP.

The system is equipped with unit location where CAD status determines reporting cadence plus there is immediate location on PTT or emergency.

The system is secure capable with DES, AES at all console sites, and a regional master CKR plan with key sharing agreements.

There are also coordinated, consistent, regional ERRCS policies.

If you have been following along and happen to be one of those ARES guys, sitting by the door with your go-kit and your “when all else fails” hat, waiting for “the call” – you have probably realized that it could be a long wait.

Never fear. Our EM people work with the local amateur groups to use them for things like storm spotting, damage assessment teams, parade assistance, etc.

BTW. We have go-kits too.

Portable dispatch consoles (MCC7100 originally) can be deployed anywhere on the RNI, CEN, or through Firstnet, for use at transition sites or by specially trained incident dispatch teams (IDT) at fixed locations or in one of several mobile command post vehicles.

The incident dispatch teams also have laptop MDTs for full access to CAD anywhere they deploy.

Coordinated regional EM departments with COML and COMT have cache radios at multiple locations. They also have portable VHF, UHF, and 800 mutual aid repeaters (PDR) with antennas pre-installed at key RF sites as well as transportable antennas, tripods, coax cables, etc.

There are multi-band APX radios in ‘Gator’ cases that can be deployed as additional assets for any EM related incident or exercise as well as by the IDT people (see photo). These are pre-programmed to operate natively on the system anywhere in the region as well as to operate on neighbor’s systems.

As you can see on the front the unit, you can use a microphone and speaker or a standard dispatch headset. The drawer in the rear contains accessories like AC & DC power cords, mag mount antennas, microphone, headset & footswitch, etc.

Also on the rear panel is an RJ45 I/O connector configured with 4-wire E&M control for connection to gateways or other assets. For example, a cross-over cable connected to an E&M port on a portable repeater can extend a trunked talkgroup out to a VHF, UHF, or 7/800 mutual aid ‘hotspot’. That’s where COML and COMT come in – knowledge of assets and capabilities.

I hope this overview of the state-of-the-art in public safety systems has been informative. I truly believe that there is no finer system in existence anywhere.