Canada Open Data: TAFL geodatabase

Status
Not open for further replies.

va3pid

Member
Joined
Jul 16, 2013
Messages
5
Location
Toronto, ON
Hi,

After running into TAFL/Spectrum Direct data at work, I was determined to make it a little easier to use. The TAFL files are now licensed as open data, so you can incorporate them into your own projects and data sets.

I've written a script to convert the TAFL database to a SpatiaLite database. SpatiaLite is a version of the free SQLite database system, but it supports spatial queries: you can search for stations near other locations, for example.

Using the data, I drew this visualisation of the whole country's microwave networks: Canada's Microwave Links.

The database might seem a little weird if you've never used GIS before, but it can be used as a regular SQLite database for searching a local copy of the TAFL.

I'm sorry I don't have a pretty website up to demo this, but if anyone wants to build the database, the code's on Github with instructions: scruss/taflmunge.

73 de VA3PID,
Stewart
 

kayn1n32008

ØÆSØ
Joined
Sep 20, 2008
Messages
6,315
Location
Sector 001
Wirelessly posted (Mozilla/5.0 (BlackBerry; U; BlackBerry 9900; en-US) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.1.0.746 Mobile Safari/534.11+)

Neat!
 

va3pid

Member
Joined
Jul 16, 2013
Messages
5
Location
Toronto, ON
Not quite sure yet, wwelles. Would like to have some map-based web-presence that would allow users to query locations, frequencies, see link paths, filter by application … but all of these require a map framework and a much deeper understanding of TAFL than I have.

I have loaded the Ontario p2p links on to uMap (caution, this takes a while to load, as it's rendering all of the data in your browser): Ontario Microwave Links - uMap. Unfortunately, it hides all the the fun fields like frequencies and locations.
 

wwelles

Member
Joined
Jan 8, 2013
Messages
18
Not quite sure yet, wwelles. Would like to have some map-based web-presence that would allow users to query locations, frequencies, see link paths, filter by application … but all of these require a map framework and a much deeper understanding of TAFL than I have.

I have loaded the Ontario p2p links on to uMap (caution, this takes a while to load, as it's rendering all of the data in your browser): Ontario Microwave Links - uMap. Unfortunately, it hides all the the fun fields like frequencies and locations.
Pardon my possible ignorance, but is that different from: TAFL Search with Google Maps Output ?
 

Jay911

.
Feed Provider
Joined
Feb 15, 2002
Messages
9,344
Location
Bragg Creek, Alberta
Most of the "TAFL" searches in the past are done with downloaded text files which have the following shortcomings:

* Not updated except at the end of the month (the "live data" is updated daily)
* Many fields are cut off and don't show the full value (i.e. location limited to 30 characters when the data has many more)
* The person writing the search service may elect to use only the fields he is interested in, so other data that you or I might find useful is discarded
* The person writing the search service may become disinterested in the project after a period of time and stop working on it, leaving users with an out-of-date resource

All four of these are issues I've experienced using the various "TAFL sites" people have made.

These datasets appear to be the monthly-updated dumps, so they're still between 1 and 30 days out of date at any given time, but indeed, what you've been able to do with them is pretty outstanding.

VA3PID - there used to be an app on iOS that could take your location and plot either on a map or as a list, licenses that were nearby. This was awesome for those of us who wonder "what's on that tower over there?". Unfortunately it fell victim to many of the issues I mentioned above. Something like that would be very useful.

I'm no programmer (the last serious work I did was on Turbo Pascal in high school, 25 years ago), but I may take a peek at the github repository. If you have any questions about the TAFL, I may be able to help - I use it and Spectrum Direct every day.

Interesting: Someone posted a comment on the Central region dataset referring to Radio Frequency Map Finder (Canada) - that does, on the desktop, almost exactly what I was commenting on previously (an interactive map of all tower sites in the TAFL). The data is definitely not current, as it shows a frequency on my license which I removed last month - and it only puts up a few fields that are relevant to, I presume, the cellular industry - and the frequency data is "rounded" to the kHz level.
 
Last edited:

va3pid

Member
Joined
Jul 16, 2013
Messages
5
Location
Toronto, ON
Most of the "TAFL" searches in the past are done with downloaded text files which have the following shortcomings:

* Not updated except at the end of the month (the "live data" is updated daily)
* Many fields are cut off and don't show the full value (i.e. location limited to 30 characters when the data has many more)
* The person writing the search service may elect to use only the fields he is interested in, so other data that you or I might find useful is discarded
* The person writing the search service may become disinterested in the project after a period of time and stop working on it, leaving users with an out-of-date resource

All four of these are issues I've experienced using the various "TAFL sites" people have made.
And these are all good concerns.

  1. I'm using the version from the data.gc.ca / donnees.gc.ca Open Data site. It's published under a licence that allows anyone to republish, reuse, remix … The same definitely can't be said for Spectrum Direct and the other ic.gc.ca data files which are under Crown Copyright.
  2. Yes, the cut-off fields are a bit of a pain, but I'd happily swap that for the ability to do wide area, wide frequency searches, which Spectrum Direct doesn't allow.
  3. Since I don't even know yet what I going to be doing with this, I've included all the fields — certainly all the ones that were included by the MS-DOS TAFL program that produced dBaseIII files. I've even included the Mobile Radius of Operation field, which is one of the more horrible kludges I've seen in data.At the very least, I'll be using these data to add obvious radio infrastructure (like VORs) to OpenStreetMap.
  4. I hate being left with orphaned data, too. That's why I provide code first, then will work out what to do with the data later. As long as TAFL is published in its current form, this code should work.

These datasets appear to be the monthly-updated dumps, so they're still between 1 and 30 days out of date at any given time, but indeed, what you've been able to do with them is pretty outstanding.
Thanks! The Canada Open Data site is as much as two months behind, but it does allow almost unlimited use of the data.

VA3PID - there used to be an app on iOS that could take your location and plot either on a map or as a list, licenses that were nearby. This was awesome for those of us who wonder "what's on that tower over there?". Unfortunately it fell victim to many of the issues I mentioned above. Something like that would be very useful.
Since the backend database I'm using (Spatialite) is a minor variant on the database that iOS uses as Core Data, one could write an app. iOS isn't really my thing. The whole database weighs in at around 300 MB, so it would be a big app! Before TAFL appeared on the open data site, such an app would have run into licensing problems.

If you have any questions about the TAFL, I may be able to help - I use it and Spectrum Direct every day.
Thanks; I use it a lot too. I often have to search large areas (> 100 km radius) for all frequencies, and Spectrum Direct can't go that far. P2P microwave links can really mess up a wind farm design, and there's no requirement for a licensee to register a radio path on title, unlike other communications methods.

Someone posted a comment on the Central region dataset referring to Radio Frequency Map Finder (Canada) - that does, on the desktop, almost exactly what I was commenting on previously (an interactive map of all tower sites in the TAFL).
Yes; the City of Toronto links to that on its RFI and health page. Even I noticed sites around me being out of date.

cheers,
Stewart
 

wwelles

Member
Joined
Jan 8, 2013
Messages
18
Since the backend database I'm using (Spatialite) is a minor variant on the database that iOS uses as Core Data, one could write an app. iOS isn't really my thing. The whole database weighs in at around 300 MB, so it would be a big app! Before TAFL appeared on the open data site, such an app would have run into licensing problems.
Probably should've stated earlier, but I'm a Software Dev by profession, however not sure either way if I'd have time to help.

Have you considered hosting a web service with the parsed data that would allow web and smartphone clients to query for a smaller subset of data? This would mean you don't need an offline copy, and would reduce the client size greatly.

I'm not really sure I understand your concerns about using Google Maps to render data? I don't see any ads on the TAFL Search I linked before, or other embedded Google Maps on sites. Although it's possible that's attributed to AdBlock Plus.
 

va3pid

Member
Joined
Jul 16, 2013
Messages
5
Location
Toronto, ON
Have you considered hosting a web service with the parsed data that would allow web and smartphone clients to query for a smaller subset of data? This would mean you don't need an offline copy, and would reduce the client size greatly.
Yes, I am considering that strongly. I'm just looking around for the right framework. If I host it myself, I'm stuck with MySQL Spatial as the backend, which is not very well supported by web mapping frameworks yet.

Using SpatiaLite is not the end of this project; it's just something that gets the data in a more usable intermediate form for me. Also, it's blazingly fast, portable, and needs no admin. Yay!

I'm not really sure I understand your concerns about using Google Maps to render data? I don't see any ads on the TAFL Search I linked before, or other embedded Google Maps on sites. Although it's possible that's attributed to AdBlock Plus.
Google has always maintained the right to inject ads and tracking on any of their services. They can also charge for map API access if the volume reaches a certain level (unlikely here, I know). Also, I contribute to OpenStreetMap, their maps are pretty (unlike Google sameyness), and I know enough people with OSM framework experience who could assist if I get stuck. They've got the whole switch2osm thing to help you use their service.

cheers,
Stewart
 

wwelles

Member
Joined
Jan 8, 2013
Messages
18
Yes, I am considering that strongly. I'm just looking around for the right framework. If I host it myself, I'm stuck with MySQL Spatial as the backend, which is not very well supported by web mapping frameworks yet.

Using SpatiaLite is not the end of this project; it's just something that gets the data in a more usable intermediate form for me. Also, it's blazingly fast, portable, and needs no admin. Yay!
Are you tied to a specific language? Regardless of language, I strongly suggest a JSON based Rest web service. If you indicate the language you'll be using, I may be able to suggest a framework.

Google has always maintained the right to inject ads and tracking on any of their services. They can also charge for map API access if the volume reaches a certain level (unlikely here, I know).
I understand now, thanks.
 

jmorgan123

Member
Joined
Sep 20, 2010
Messages
21
VA3PID - there used to be an app on iOS that could take your location and plot either on a map or as a list, licenses that were nearby. This was awesome for those of us who wonder "what's on that tower over there?". Unfortunately it fell victim to many of the issues I mentioned above. Something like that would be very useful.
Hi Jay911,

Given the shortcomings you noted above, you can use my site (TAFL Search with Google Maps Output) to do this. You do need a data connection on your phone. On the main page, check off "Use current location" and remove all other filters. You can set the max distance to something like 1 km. When you see the results on the map, you can use the little "street view" icon to view the location at street level - a pretty neat feature of Google Maps. You can also click on the red markers for transmitter info, which also now links to the Spectrum Direct license info (where you get any of the extra license details).

Just a little background - my motivation for building the site was to try and find an easy way to program my scanner, plus I have an interest in GIS, MySQL, and PHP. I do read the comments on here and try and update the features. You can always send me a note if there's something you'd like to see.

Jonathan
 
Status
Not open for further replies.
Top