Unattended Logging Of Whistler Scanners

Status
Not open for further replies.

SOFA_KING

Member
Joined
Apr 25, 2004
Messages
1,581
Location
SE Florida
It works! And actually completed in much less time than in EZscan. I'll have to compare the two with a stopwatch, but I didn't fall asleep like I usually do with ESscan (laughs).

Unfortunately, I way exceeded the row limit in my old version of Excel (Did newer versions ever expand above that?). If I remove the older data from about 7/2015 to 7/2016, then I can see newer data with the NAC included. After removing unused columns, I get a nice tidy list of all the data needed to analyze trends. And holy cow can it do that when you sort by frequency first, radio ID next, and then date. Great! Having NAC alongside the other data is VERY helpful!

Real magic happens when you sort by radio ID, then frequency, and then date. Now you easily see who uses what frequencies. Wowza! But now you see cross-agency use with the NAC showing (which is not available in EZscan). Crazy!

Thank you Don!

Does anyone have any suggestions on expanding the row limit so I can analyze all of my data? I'm missing out on a lot of user ID's with that limitation.

Phil
 

DonS

Member
Joined
Jun 17, 2003
Messages
4,102
Location
Franktown, CO
It works! And actually completed in much less time than in EZscan. I'll have to compare the two with a stopwatch, but I didn't fall asleep like I usually do with ESscan (laughs).

Was each test (AUParse vs. EZScan) performed immediately after inserting the card into a card reader? That is:
* insert card
* run EZScan
* remove card
* insert card
* run AUParse

If not (if you left the card in the card reader between tests), Windows probably cached data and made the second test artificially fast.

If you did remove the card between runs, then EZScan might be slower because it's allocating memory as it parses files. AUParse doesn't do that. I would only expect such a slowdown with a very large number of recordings. For example, with my ~24000 recordings, the difference in time between AUParse and EZScan is only 4%.

Row limit: Sounds like you're running Excel 2003 or older, with a 64k (2**16) limit. Since Excel 2007, the limit has been 1M (2**20). If you don't want to buy a newer version, OpenOffice supports 1M rows x 1k columns.
 

JSTARS03

Monitor
Premium Subscriber
Joined
Jun 10, 2003
Messages
159
Location
Sunny Florida
Thank you Don, it does work great!

I do not keep the audio files on the SD Card
I move them off, so all my older audio is already
on the hard drive and separated out into the
different folders , but it will be awesome from
this point forward.

Thank you
 

wbloss

Human
Database Admin
Joined
Dec 19, 2002
Messages
1,043
Location
Joplin MO
THANK YOU!!! I downloaded the AUParser and it works fine! While one can view this data in the Audio tab on the Whistler AP and it does sort, there was no way to save. Your parser does that, and I thank You. And yes I would love to purchase a "win500-like" program for the TRX.
Wally
WB0BAV
 

DonS

Member
Joined
Jun 17, 2003
Messages
4,102
Location
Franktown, CO
Here's a more general-purpose version, for those who've moved their recordings off to another location, renamed the files, deleted the ARINDEX.DAT file, etc. It looks for all .AU files in the specified directory and all subdirectories. Instead of showing progress every 100 files, it emits a period ('.') to stderr each time it completes a subdirectory.

http://www.starrsoft.com/Public/scanner/AUParse_DIR.zip

This one is used like this:
AUParse [directory]
where [directory] is an optional parameter specifying the drive and/or directory to search. If not specified, the program searches the current directory.

Examples:
AUParse F: will search the root directory and all subdirectories on drive F:
AUParse "D:\saved recordings" will search the "saved recordings" directory on drive D: and all subdirectories (note the quotes around the name, since it has a space).
AUParse (no parameters) will search the current directory and all subdirectories

EDIT: this one is Windows-specific. It uses a couple of Windows API calls to enumerate files and directories.
 

kc2kth

Member
Joined
Mar 27, 2004
Messages
436
Location
Toms River NJ
Thanks, Don. I really didn't know your relationship or level of involvement with these companies. And you do know I really like WIN500, right? I would surely purchase TRX-2 software from you if you made it.

I just ran a "parse", now about to look at it.

Thanks again!

Phil

+1 on all of this, I was wondering about that involvement too, also a WIN500 customer. Thanks for clearing that up Don!
 

SCPD

QRT
Joined
Feb 24, 2001
Messages
0
Location
Virginia
Thanks Don .. faster than my Uniden program. But anything Uniden pretty much sucks.

I have taken your program, running your app within my Access database, and then sucking in the CSV in and playing back the audio I am interested in.

Nice little app .. does exactly what I do now, just faster doing the header part ( I do it and import the data into the database in one pass).

I think I need a new pc .. only getting 9,000 a second but I think that is my slow card reader, (too cheap to replace it). :D

Edit .. I was able to achieve 17,995 files per second when outputting the CSV file to my SSD drive. So .. a try to speed things up. I am using data from a PSR800 btw. Haha .. I lied, if you run it multiple times, you can get the time above. But .. I got just over 200 files a second the first time.
 
Last edited:

Machria

Member
Premium Subscriber
Joined
Apr 24, 2014
Messages
543
Location
Long Island, NY
No list? Why not? Maybe there should be a list. That is what people are asking for. Logging capabilities. So the closest we get to that is the data captured on the recordings, but the software is slow, and is not giving us any way to export all that logged data so we can better analyze it. That is what we are asking for...an "EZ" way to log COMPLETE data and review it. Is that so hard to comprehend?

So rather than point out my lack of understanding on your design, why not just work on giving us what we ask for? You appear to have this resistance to give us better software and tools, and easy to export features. How many times do people have to keep asking for this? Whistler keeps blowing us off with this locked down, PITA to use EZscan garbage...and is not allowing third party development. Do something about it already. It's been many years now!

If you can't make us a better software application, at least let someone else do it. You could sell more scanners. Drop the pride already. It's a dog. Corporate defender fanboys may tell everyone it's just fine the way it is now, but clearly not everyone feels that way, and it won't change their minds telling them that. Many, like myself, dislike its clunky and limited UI. I honestly can't recommend Whistler scanners because of this software, so I don't...and won't until the software gets better. That is what is holding back sales. You see people express that sentiment time and time again. The TRX hardware is pretty good (although we all know the LSM issue is still an unaddressed problem), and we want to like the product, but the software sucks. Lost sales! It won't change until you accept that and give the customers what they want.

Phil

THAT A BOY!! You tell em! Now THAT is the spirit! We need to keep our foot on the gas with these companies. If we do not bee-otch, moan and complain, they will sit back and let their producs fall short much like the US car manufacturer's did for years!

And this is NOT directed at "Don", I read the rest of the posts and understand the situation. It's directed to the "executives" at Whistler and other scanner manufacturers to get off their butts and modernize the dang things already! ;)


Here's a little app that will parse all the AU files on the scanner's SD card, dumping data in CSV format.
http://www.starrsoft.com/Public/scanner/AUParse.zip

It's a console app. Open a command window (start -> run -> cmd) and cd to the directory where you unzipped the utility. Run it like this:
AUParse <drive>
where <drive> is the drive letter assigned to the SD card, including the colon. For example, if you put the card in a card reader attached to your PC and Windows assigns "drive F:", you'd enter:
AUParse F:

The output will be dumped to the console. To save it as a file, redirect stdout to a file, like so:
AUParse F: > log.csv

The resulting CSV file should be suitable for loading into Excel.

The utility expects the scanner's REC directory (including ARINDEX.DAT and all subdirectories) to be in the root of the specified drive. That is, it's intended to be aimed at the scanner's SD card.

The utility dumps all fields in the .AU header, except the .AU-specific stuff (magic number, etc.). It puts a header line at the top of the output.

Now that is fantastic!! Thank you!

But a couple of quick question for you: Why not just incorprate this into the EZscan software directly with an easy front end? I don't know if you have worked on/designed/developed/helped EZscan or not, but if you did, are you going to work on incorporating this into EZscan? If you didn't, lets request this of Whistler!


No worries.

A couple of things to keep in mind, though: I neither specified the PSR-800 hardware or UI nor wrote the associated PC app. (Heck, even GRE didn't specify that UI, at least not directly). Also, I'm a contractor (have been since I started doing work for GRE 11 years ago - PSR-500) and not a Whistler employee.

I don't know if, in your previous post, you were using the terms "you" and "Whistler" interchangeably or if "you" referred to me, specifically. If the latter, see previous paragraph ;)

This wasn't existing Whistler code that was handed out. It was something I knocked off in an hour in order to test parsing speeds outside the EZ Scan app.

Byte offset 164 in the .AU header? If my quick count is correct, that should be:
unsigned char TsysType

If the recording type is "TGRP", it's the trunked system type ("see _eTrunkingSystemType enum" above).

EDIT... (Well, yeah, I did copy/paste the DCS and CTCSS string data.)

Don,
I would assume you have some sort of "non-compete" with Whistler if you are/were a contractor for them. In that case, have you suggested adding the above type feature to EZscan for them? And if you don't have a non-compete, have you thought about developing your own software to either replace or supliment EZScan? Or is there just not enough money in it to spend the time?

I asked the Proscan guy if he would be supporting Whistler scanners at all, they just about cut my neck off just for asking, as if I had acted like a terrorist or something. LOL Obviosuly a touchy subject for him/them for some reason. I think with the growing popularity of the Whistler line, it would be smart for somebody to develope something.....
 

ProScan

Software Provider
Premium Subscriber
Joined
Jul 2, 2006
Messages
7,409
Location
Ontario, Calif.
I asked the Proscan guy if he would be supporting Whistler scanners at all, they just about cut my neck off just for asking, as if I had acted like a terrorist or something. LOL Obviosuly a touchy subject for him/them for some reason. I think with the growing popularity of the Whistler line, it would be smart for somebody to develope something.....

Excuse me Machria. I don't mean to hijack this but have to interject when I see inaccuracies.

I was or am considering Whistler. I have several GRE/RS models and I don't believe much of a difference in the protocol.

It wasn't the question you asked if I'm going to support Whistler. It's the way you asked the question. Others have suggested, it was a borderline troll.

If you had sent me an email or just simply asked then I would of told you but instead you had to shoot yourself in the foot..
 

Machria

Member
Premium Subscriber
Joined
Apr 24, 2014
Messages
543
Location
Long Island, NY
Excuse me Machria. I don't mean to hijack this but have to interject when I see inaccuracies.

I was or am considering Whistler. I have several GRE/RS models and I don't believe much of a difference in the protocol.

It wasn't the question you asked if I'm going to support Whistler. It's the way you asked the question. Others have suggested, it was a borderline troll.

If you had sent me an email or just simply asked then I would of told you but instead you had to shoot yourself in the foot..

LOL! Your obviosuly VERY sensitive to this. Why the issue? I asked in your thread if you would support the new line of whistler scanners, that's it. I don't have your email, I don't know who you are, nor do I care. This is an open forum for questions and answers, that's why we are here. I still for the life of me, do NOT understand your problem with that.

That asside, me thinks it would be in your best interest to add the new Whistler line to your software as their sales certainly seem to be gaining market share. Glad to see your "considering" it. But I understand you need to weigh the cost/time vs return on investment. I was a developer for 15+ years and then mananged an IT dept for 20+ years, I'm well aware of the process and decision making... But if you don't do it, somebody else is certainly going to, if not Whistler themselves.
 

ProScan

Software Provider
Premium Subscriber
Joined
Jul 2, 2006
Messages
7,409
Location
Ontario, Calif.
LOL! Your obviosuly VERY sensitive to this. Why the issue? I asked in your thread if you would support the new line of whistler scanners, that's it. I don't have your email, I don't know who you are, nor do I care. This is an open forum for questions and answers, that's why we are here. I still for the life of me, do NOT understand your problem with that.

Issue is when you are making suggestions, Don't be rude, others thought you were Trolling. You did the same thing in the Uniden forum with making suggestions and then said the Uniden scanners seems like they were built in 1972. Comments like that negates the suggestions.

I'm just trying to help you out on how to make suggestions. That's all I have to say.
 

SCPD

QRT
Joined
Feb 24, 2001
Messages
0
Location
Virginia
I more or less already have such solution, I just don't have the front end live.

For informational purposes, what exactly are you trying to achieve?

-> Better user experience for listening?

-> A log of all hits that is portable?

-> An easy way to import objects found through search?

-> Something else?

-> All of the above?

I suppose a few people to test my application would be good. Email me if you are interested.

Sorry .. busy playing with Don's program and ran into a gotcha. But re your ? .. the purpose of my Uniden app is to replicate exactly what the Whistler EZScan software does, just for the Uniden Homepatrol and 436/536 scanners.

I am having an issue playing the AU files within my Access database. Wav files are much more common than the AU files and looking for a suitable answer, ie not launching an app to play the file. But .. its only time and all fun.
 

AggieCon

Member
Premium Subscriber
Joined
Nov 25, 2015
Messages
1,448
Location
Texas
Can't have your cake and eat it too

A lot of my older .AU files that were transferred to the local disk via EZ Scan do not have full header information. For example, important things like the frequency. I assume EZ Scan is removing that data when it parses the files to name the directories?

If so, it would be great if that could be modified to leave those files untouched.

Here's why we are stuck with the current Whistler software, based on my understanding:

  • They have some contract with RadioReference, locking down your data; unknown when that expires. Hard facts (i.e. radio frequencies and the entities operating them) do not have copyright protection, but I assume most of you agreed to the EZ Scan user license, which might actually create some legal protection.
  • Whistler contracts Don to update stuff. Either they don't want to pay for more improvements or he thinks it is already perfect.
  • Unlike Uniden, which stores pretty much everything in plaintext, Whistler does not usually release information, so it must be reverse engineered. The exceptions to that include the information they provided last summer and information given via this thread and Don's source code.
While it is possible to make applications for Whistler, it is fairly slow. Just for one example, when creating my audio header program, I spent an hour making arrays of CTCSS and DCS tones that the scanner uses (they use their own set). And now Whistler released the information on that in an accessible form (rather than having to scroll through a small list and transcribe everything).



What I am trying to say is that it is going to be hard for others to justify spending time creating new solutions if Whistler is just going to release free versions of the same type of application. Obviously, Whistler has a better understanding of their product and better access to technical information, so making a short program is a lot easier than someone trying to do so with a 1 pager posted online.


I imagine Whistler would put their name on it if he made a WinTRX type program, but either they don't want to pay for that or he won't do it for free.


And the way some people are treating the folks that make things like UniTrunker, DSD+, and ProScan. Hah. That's some serious motivation to others looking to contribute to the field.
 

eorange

♦Insane Asylum Premium Member♦
Joined
Aug 20, 2003
Messages
2,925
Location
Cleveland, OH
So you're somehow annoyed that Whistler and Don are providing free software?

No one is stopping you from writing your own software, assuming you know how to do it. Years ago I wrote a logger for my PSR-500; I never shared it and just kept it to myself. Most hobbyists who write software just do it because it's fun. So go have fun.
 

SCPD

QRT
Joined
Feb 24, 2001
Messages
0
Location
Virginia
AggieCon .. I know you looked at Don's parsing program. At least in my files .. the frequency information is there in the CSV file and all is good. I am working on finishing my program, having fixed the playback of the AU files now. So .. more work to have it do all the things I want.

The thing .. the cpp file is there, so nothing stopping you from doing what you need to do with the program. So nothing that I can see to stop you from doing just about anything you want with the AU files.

I do agree with the above noter, if you want to do it go for it. It is all part of the fun of the hobby.
 
Last edited:

DonS

Member
Joined
Jun 17, 2003
Messages
4,102
Location
Franktown, CO
A lot of my older .AU files that were transferred to the local disk via EZ Scan do not have full header information. For example, important things like the frequency. I assume EZ Scan is removing that data when it parses the files to name the directories?
You assume incorrectly.

The data wasn't present in older .AU files. For example, the frequency info wasn't added until firmware version 3.3 (May 2016). The TsysType wasn't added until version 3.6 (July 2016).

EZScan isn't removing anything from the .AU files. It copies them to a new directory structure and renames them. Their contents are left intact - including the newer fields like frequency.

Whistler contracts Don to update stuff. Either they don't want to pay for more improvements or he thinks it is already perfect.
As mentioned previously in this thread, Don didn't write the EZScan software. Don isn't responsible for maintaining or updating it. Don doesn't even have that source code.

And now Whistler released the information on that in an accessible form (rather than having to scroll through a small list and transcribe everything).
...
if Whistler is just going to release free versions of the same type of application
Whistler released nothing. I did. Not everything I do in life, including posting on RR forums and writing trivial little utilities for fun, is "Whistler work".

Sure, I was able to copy and paste a couple of strings, but it was nothing that one couldn't copy from the scanner's UI. Or even these two RR pages:
CTCSS (scanners don't have 150.0)
DCS

Believe it or not, my utility wasn't some concerted effort on the part of Whistler and myself to make you waste time. I saw this thread Thursday afternoon. Friday morning, I read a post saying that EZScan's parsing of the files is "slow". I thought "really? let's see". And I wrote some test code. Then I shared it. Then I modified it to work without the ARINDEX.DAT file. And Whistler had absolutely nothing to do with it. Nor did you.
 

AggieCon

Member
Premium Subscriber
Joined
Nov 25, 2015
Messages
1,448
Location
Texas
AggieCon .. I know you looked at Don's parsing program. At least in my files .. the frequency information is there in the CSV file and all is good. I am working on finishing my program, having fixed the playback of the AU files now. So .. more work to have it do all the things I want.

The thing .. the cpp file is there, so nothing stopping you from doing what you need to do with the program. So nothing that I can see to stop you from doing just about anything you want with the AU files.

I do agree with the above noter, if you want to do it go for it. It is all part of the fun of the hobby.

I've had the back-end of my program done for some time now -- long before Don released that source code. Heck, I'm sure you guessed I've been working on this since I asked you about your program for Uniden.

The only thing I lacked was figuring out what was at offset 164. And that didn't make sense before because:

-> I was comparing files created both before and after the switch was introduced.
-> For whatever reason, "0" corresponds to both a trunked system of type "Motorola" and also a conventional frequency with no trunked system type.

This moved that small detail to the back burner of the project. Otherwise, I've been parsing probably everything else in the data files.

Whistler and its agents release marginal documentation, at best. And some of it is contradictory. I guess it makes some people superior by hoarding information, only to finally share it at strategic times.

I guess I missed the release notes that described the changes in the audio headers--or that any changes were made at all.
 

AggieCon

Member
Premium Subscriber
Joined
Nov 25, 2015
Messages
1,448
Location
Texas
The features and functions of the scanner, as well as how to use them?

When was the last time anyone said "hey, go read page ## of the manual, it's in there and easy to follow, you'll be able to get it figured out"???

If Whistler is not going to help all of its customers and the customers end up using forums instead, Whistler and its agents have got to stop the stuff where they belittle those who were wrong about some undocumented feature. If there are problems, errors, or confusion, Whistler needs to take responsibility for incompetency in communicating rather than addressing customers who are helping other customers in a less than cordial attitude.
 
Status
Not open for further replies.
Top