Isn't the old syntax what Roxen uses to mix different quotes in the same attribute value? I believe it's supposed to work like this:

<foo attr="use ' here"'use " here'/>

I do not really think that was ever an encouraged syntax? &quote; has always been preferably to my knowledge. It has been a while, however.

And it is an issue if you want Parser.HTML to parse actual HTML. Since the 'save a space' seems to be surprisingly common, granted, we only encounter issues in our turbo servers when it is used with "style" or "src", but that happened fairly often, presumable because 'style' has a tendency to contain whitespaces.

Well, that would work, except for this:

p->finish("<t a='b'c='d''e'="f">")->read();

([ /* 3 elements */ "a": "b", "c": "d", "e": "f" ]) (3) Result: "<t a='b'c=d 'e'="f">"

<t a='b'c='d''e'="f">

is clearyly a syntax error. If you interpret 'd''e' as the attribute value of c, then you have a spurious = after it, which is not allowed. If you interpret 'd' as the attribute value if c, then you have a quoted attribute name, which is not allowed.

Yes, I know, the question is if changing that is perhaps more dangerous than changing how the quoting works.

A real html compatible parser would be very nice when parsing actual HTML, but perhaps not 100% compatible with RXML (one somewhat common usecase of Parser.HTML, although HTML parsing is probably more common.)

Changing what? Hasn't it always been a syntax error? Just that the parser doesn't actually report it?

No, Parser.HTML has always _explicitly_ allowed quoted attribute names (it sets the allowed quotes to the same thing for both attribute values and names).

I know this is fairly odd, but so is the more-than-one-quoted-string syntax.

<t "a"='a''b'c'"d> would return ([ "a":"abcd" ])

It will now be ([ "a":"b" "b":"b", "c":"c" "d":"d" ])

Neither is correct.

The only difference in the parser when parsing attribute names and values is that names do not get entities in them parsed. The quoting rules are the same

Really? That doesn't even have matched quotes...

You are free to leave out the last endquote...

I would like something that breaks down an html document to a datastructure, preferable one with tools like searching.

Well, yes, a HTML tokenizer would be useful. HTML5 has a very readable specification.

Parser.HTML is created to allow RXML (or similar) parsing with as little computron usage as possible. What I am using it for mostly is breaking down random HTML documents for data gathering, which isn't the intended use...

Ironically enough it is about 10 times slower than ye olde Opera HTML5 parser at actually parsing html. :)

It is faster to have a simple tokenizer that then outputs tokens that is handled by either a tree generator (as also specified in html5) or somethgin that just calls callbacks for tags (like the current Parser.HTML)

I have seriously considered writing one. But the name 'Parser.HTML' is already taken. :)

Handy reference:

https://html.spec.whatwg.org/multipage/syntax.html#tokenization

Or Parser.HTML5?

Here's one of several real-world examples that I found from our CMS (it's a bit tricky to grep after these constructs...):

#define quote(X) (replace((X)||"", "'", "'"'"'")) [...] "<var name='destname' type='string' size='40' default='" + quote(sbobj->name(id)) + "' />"

A sbobj->name() returning "'foo'" would then produce:

<var ... default=''"'"'foo'"'"'' />

Whether this is a guarantee that we never concatenate two strings with same quote char I don't know, but I'd rather preserve any such backwards compatibility than the very-odd quoted attribute _name_ syntax which I've never seen used.

Eep. That shouldn't be too hard to fix in Parser.HTML though?

There should be a Parser.HTML object that can carry a flag to control the behaviour, if someone wants the new/old behaviour?

Is it possible to use heuristics so that a='b'c=d is detected?

I just called Hedda. He will take a look at it ASAP.