(A mathematician here.) What you are requesting is non-trivial and no software that I am aware of supports it. In fact, even ordinary Euclidean geometry has a known problem with determining when a point is inside or outside a polygon (that is exploited to falsely prove all triangles are equilateral, i.e., all sides have the same length). It is one of those problems that sounds easy but is no.
Uniden Sentinel for the 436/536 and SDS100/200 scanners has the ability to define multiple rectangles for monitoring areas, both for the Department and Site levels. Since the downloads from Uniden are based on a fixed radius around a point (as that data comes from RadioReference), the feature is not commonly used - and I personally have not tested it with my SDS100, which has a GPS receiver attached to it for automatic switching.
Rectangles are easy but doing this for generalized polygons are not as easy.
Of course the SDS100 already uses GPS as do many amateur radio HTs to know where it is. Adding INS would only adding allow knowing position when GPS information was not available and would have to use GPS to initialize and occasionally correct as INS drifts. Using the last know GPS position would generally be good enough and making INS just extra cost. I used to occasionally be involved with INS on helicopters when working--trying to explain Kalman filtering to co-workers
Uniden scanners also supports GPS rectangles. Draw enough of rectangles on a map to cover the complex shape you'll need and try to make the form of the rectangles so that they overlap as little as possible. I believe you can do that visually in ProScan by click and drag. Then copy your favorite list for the whole county multiple times and set one rectangle to each copy. Done.
Which is why I suggest that it would have to be something run off a PC/Linux box sending commands to the scanner. Though even rectangles, if I can draw multiple ones per system on Uniden devices, would meet my needs. My local agencies (Pittsburgh/Allegheny Co) are pretty much still all conventional UHF, so I could just set up each zone's list of frequencies as a separate system.Rectangles are easy but doing this for generalized polygons are not as easy.
Point in polygon - Wikipedia
en.wikipedia.org
Conclusion: Add in the required database and computation power needed, isn't going to happen in a scanner you can afford.
If you go through with the effort to break the polygon down into multiple rectangles you can effectively build any shape (well enough for the purpose of controlling a scanner). And yes, you can define multiple rectangles. These can be defined at the department and for trunked systems also for the site.
