download freqs,

[traperjeff]

Thank you lou........I thought I was doing something wrong, or just didn't know what I was doing....Thanks again..Jeff

 

[DonS]

Don, you need to adjust the software, this will happen again and again as the ° symbol is going to appear over and and over.

Uh-uh. If RR is going to supply something it calls XML, it needs to be well-formed XML.

Sure, Win9x could be modified to handle °, but this is only because a) we know about ° and b) Win9x receives the text directly via HTTP then passes it off to an XML parser.

But:
1. What about other invalid XML? What if something like &blob; appeared? RR should, when it forms the XML, verify that the text it's including is actually valid. If it's not, RR should either escape it, translate it to valid XML, or omit it entirely.
2. What about other apps? If I was using .NET, VB, or some other framework that writes most of my code for me, it's possible that I wouldn't have access to the plain-text XML. Some underlying libraries might receive the HTML body, pass it through an XML parser, then hand me the output. The parser would barf, and I'd never have a chance to fix the offending "XML".

This really needs to happen on RR's side. In my opinion, it isn't realistic to expect multiple apps (existing and not-yet-created) written in multiple languages to handle known and unknown violations of XML's content rules.

 

[loumaag]

Uh-uh. If RR is going to supply something it calls XML, it needs to be well-formed XML.

Sure, Win9x could be modified to handle °, but this is only because a) we know about ° and b) Win9x receives the text directly via HTTP then passes it off to an XML parser.

But:
1. What about other invalid XML? What if something like &blob; appeared? RR should, when it forms the XML, verify that the text it's including is actually valid. If it's not, RR should either escape it, translate it to valid XML, or omit it entirely.
2. What about other apps? If I was using .NET, VB, or some other framework that writes most of my code for me, it's possible that I wouldn't have access to the plain-text XML. Some underlying libraries might receive the HTML body, pass it through an XML parser, then hand me the output. The parser would barf, and I'd never have a chance to fix the offending "XML".

This really needs to happen on RR's side. In my opinion, it isn't realistic to expect multiple apps (existing and not-yet-created) written in multiple languages to handle known and unknown violations of XML's content rules.

Don, you make some good points, but you miss the main point. &blob is not a valid HTML character; &amp is. It is accetable to use it becuase it is valid. &blob is not. The problem isn't on RR's end, it is in your parser, did you use a standard parser library or did you roll your own? A proper XML parser must accept all Unicode Characters according to the W3C Recommendation:

Consequently, XML processors MUST accept any character in the range specified for Char.

Character Range
[2] Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
The mechanism for encoding character code points into bit patterns MAY vary from entity to entity. All XML processors MUST accept the UTF-8 and UTF-16 encodings of Unicode 3.1 [Unicode3]; the mechanisms for signaling which of the two is in use, or for bringing other encodings into play, are discussed later, in 4.3.3 Character Encoding in Entities.

I am pretty sure that " ° " fits in that range. Whether you choose to use or discard the character &amp in your application is up to you, but it since it is valid, your parser has to accept it and not just stop.

 

[DonS]

Don, you make some good points, but you miss the main point. &blob is not a valid HTML character; &amp is.

We're talking about "& d e g ;", not "& a m p ;". In the context of this thread, "& b l o b ;" and "& d e g ;" are equally invalid.

The problem isn't on RR's end, it is in your parser.

No, it's on RR's end. RR is sending the 5-character sequence "& d e g ;". This is not valid XML.

A proper XML parser must accept all Unicode Characters according to the W3C Recommendation:I am pretty sure that " ° " fits in that range.

RR is NOT sending the single-character Unicode ISO/IEC 10646 value for the degree symbol " ° ". It is sending the 5-character sequence "& d e g ;".

What RR needs to send, if it wants to put a degree symbol in the text, is either the Unicode or ISO/IEC 10646 encoding for the degree symbol, or a character reference (e.g. " & # 3 8 ;" would result in a single ampersand).

Whether you choose to use or discard the character &amp in your application is up to you, but it since it is valid, your parser has to accept it and not just stop.

Again, we're not talking about "& a m p ;" in this thread. "& a m p ;" is OK. It's a predefined entity reference that is replaced with a single ampersand character. However, "& d e g ;" is invalid. When the 5-character sequence "& d e g ;" appears in character data, it is, by definition, an entity reference. But there's no 'deg' entity declared previously, so the reference is an error.

If RR wants to send the 5-character sequence "& d e g ;" AND wants to send valid XML, it needs to define an entity earlier in the document. This was described in the initial "web service" thread:

An XML <!ENTITY ... > declaration should take care of that.

Try this ...

<!ENTITY deg CDATA "&#176" -- degree sign, U+00B0 ISOnum -->

 

[loumaag]

We're talking about "& d e g ;", not "& a m p ;". In the context of this thread, "& b l o b ;" and "& d e g ;" are equally invalid.

Sorry about the mistake there Don, I really meant to type ° and somehow got off on &. However, I am sorry to say, you are wrong. ° is valid whereas &blob is not. The length of the character definition is not set at 5 or any other length. For example, whereas ° is 5 characters long and stands for ° (the degrees symbol), © is 6 characters long and stands for © (the copyright symbol). It is a good thing you are not dealing in math or else you would have a little trouble with the 4 character code π for π (greek small Pi).

Don, take a look at this W3C Recommendation page on Character entity references in HTML 4. I think any XML parser has to address those character entities as well as the ones which are also escape characters like &, >, <, etc.

 

[traperjeff]

Ohhhh, Gosh this is so over my head...lol

Jeff

 

[DonS]

--quoting loumaag--
Sorry about the mistake there Don, I really meant to type &deg; and somehow got off on &amp;. However, I am sorry to say, you are wrong. &deg; is valid whereas &blob is not.
--end quote--

In XML (esp. the context of this thread), &deg; is invalid for precisely the same reason that &blob; is invalid: it is an entity reference for which there is no entity declaration.

Take a look at the constraints in section 4.1 (Character and Entity References) of Extensible Markup Language (XML) 1.1 W3C Recommendation. You'll see that an entity must be declared, unless it's one of amp, lt, gt, apos, or quot.

The &deg; sent by RR is an entity reference. The Name deg has not been declared. Bad.

--quoting loumaag--
Don, take a look at this W3C Recommendation page on Character entity references in HTML 4. I think any XML parser has to address those character entities as well as the ones which are also escape characters like &, >, <, etc.
--end quote--

Better yet - take a look at the XML spec I linked above. Neither it nor version 1.0 makes any reference to any HTML specification or recommendation. They certainly don't say that HTML entity names may be used in XML documents without an explicit declaration. Instead, it says the Name given in the entity reference MUST match that in an entity declaration...except that well-formed documents need not declare any of the following entities: amp, lt, gt, apos, quot.. Since the deg entity isn't listed here, it MUST be declared if it is to be used.

If you can find something in the XML specification (EDIT: or any other W3C specification or standard that allows &deg; in an XML document without a declaration) which contradicts my position and bolsters yours, I will, of course, concede. Until then:
1. &deg; is an entity reference
2. deg is not one of the five specified, "pre-defined" entity names in XML (amp, lt, gt, apos, and quot)
3. A document that contains &deg; without explicitly declaring it is either not-well-formed, invalid, or both.

 

[DonS]

Heck, even IE's XML processor can't handle the &deg; entity reference. If I load my cached copy of the OP's Sarasota system's XML into IE 6:

Reference to undefined entity 'deg'. Error processing resource 'file:///C:/Program Files/WinXX/xmldump.xml'. Line 25, Posi...

<site num="001" desc="Sarasota" callsign="WPKY904" county="" location="27&deg; 34.98N 82&deg; 32.18W" neigh="" ct...

(I'm not trying to say that IE has the most robust XML processor around. But, if such a simple thing as an "allowed" HTML entity was to blow it up, I think we would've heard something about it by now.)

 

[blantonl]

I added a DTD, and defined the deg entity.

This should resolve the problem. I'll work on getting the DTD finalized and updated with more as we go along.

Warm regards,

Lindsay

 

[blantonl]

What is interesting is that the Microsoft XML parser still doesn't like the XML from the Sarasota system - it's complaining on the "(" and ")" in the XML.

 

[blantonl]

Ok, here is what I have done (latest update)

I abandonded the idea of the DTD for this fix, doesn't make much sense quite frankly.

I'm parsing out any degree symbols and replacing with the character reference.

In addition, the XML parser error by Microsoft on the Sarasota system was due to some en-dashes in some of the talkgroups descriptions. I removed those - all should be well now.

Warm regards,

Lindsay

 

[traperjeff]

Lindsay, See Look one swipe of the magic wand and every thing is fixed... You are the Man...... Thanks to every one that helped on this...Jeff

 

[loumaag]

Take a look at the constraints in section 4.1 (Character and Entity References) of Extensible Markup Language (XML) 1.1 W3C Recommendation. You'll see that an entity must be declared, unless it's one of amp, lt, gt, apos, or quot.

Whats funny about this is the block of text I quoted in my post of mid-afternoon of 9/15 is from that particular page you supplied the link to. Ah, well, this is certainly not a subject for this thread anyway and I guess Lindsay solved the OP's problem so we can take our different interpretations of what that page says off thread.

 

[DonS]

Whats funny about this is the block of text I quoted in my post of mid-afternoon of 9/15 is from that particular page you supplied the link to.

Yes, the text you quoted previously is from the same document. However, that quoted text merely defines the characters that may be used in an XML document - it doesn't say that every combination of those characters (e.g. &deg;) is valid XML.

Kind of like the ISO C standard that specifies the character set that may be used in valid C source code. While the string "asdjfdkfj" consists solely of characters from that set, it isn't "valid C" unless it appears inside a string literal, is the name of some previously-declared identifier, etc.

It's clear that &deg; consists solely of characters from the specified XML character set. However, the ampersand and semicolon have special meaning in this context: they delimit a character entity reference. Because of this, other parts of the XML "standard" also apply - namely, the part that says the entity name must be defined. Since there's no definition of an entity named deg (in either the document or in XML itself), a reference to that entity name isn't valid XML.

This is why I kept saying that &deg; and &blob; were both invalid XML, and for precisely the same reason. They're both character entity references, but neither of the entity names have been defined.

It seems that the initial confusion stemmed from HTML's pre-defined entity names, and an assumption that XML processors were required to accept the HTML set. I think we've established that this isn't so, and XML documents that make use of such HTML-like entity references, or any other entity references that aren't among the five that XML defines, must explicitly define those entities.

 

[loumaag]

It seems that the initial confusion stemmed from HTML's pre-defined entity names, and an assumption that XML processors were required to accept the HTML set. I think we've established that this isn't so, and XML documents that make use of such HTML-like entity references, or any other entity references that aren't among the five that XML defines, must explicitly define those entities.

I think that you are correct here. I will retract my position.