Reland r251495: Re-enable prerender RemovingLink browser tests.

Reland r251495: Re-enable prerender RemovingLink browser tests.

The tests have changed significantly since they were first disabled. Add a
WaitForStop or two for good measure, but leave them as-is for the most part.
They can be disabled again if they still flake. Merge their custom prerender
loader into the main one; it's mostly the same.

In addition, for better test coverage, add a new test which asserts on events
received when a <link rel=prerender> is added for an existing prerender after
that prerender has loaded. Significantly rework the prerender events logic
to allow the test framework to wait on an event being received in the loader.

Fix implementation of set_loader_query_and_fragment to not produce URLs with
two ?s.

Original Review URL: https://codereview.chromium.org/142013004

BUG=167340, 128841
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=252156

[davidben, 2014-02-18 23:25:45]

I'm sort of amazed the original version worked at all. I guess we were unlucky
in our URL counts modulo 3.

----
The following isn't relevant to much, but is amusing and is my way to convince
myself that this is, in fact, the cause of the failure:

So, the original implementation would produce URLs that looked like

prerender_loader.html?replace_text=BASE64BLOB:BASE64BLOB?links_to_insert=2
(note the second ? instead of &)

The test server parses those out using cgi.parse_qs. cgi.parse_qs is not lenient
about ? vs. &:

In [14]: cgi.parse_qs("?a=1&b=2?c=3")
Out[14]: {'?a': ['1'], 'b': ['2?c=3']}

So, test_server.py sees that you want to replace BASE64BLOB with
BASE64BLOB?links_to_insert=2. As it turns out, base64.urlsafe_b64decode is very
very lenient. It skips characters it doesn't understand and such. It does,
however, stop at '=', which is why it doesn't fail on every bot.

BUT if the host + port of the test server is such that there is no base64
padding, links_to_insert gets decoded as well.

In [3]:
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:37601/files/prerender/prerender_page.html")+"?links_to_insert=2")
Out[3]: 'http://127.0.0.1:37601/files/prerender/prerender_page.html'

In [4]:
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:3760/files/prerender/prerender_page.html")+"?links_to_insert=2")
Out[4]:
'http://127.0.0.1:3760/files/prerender/prerender_page.html\x96)\xe4\xb3\xfbh\xfe)\xecz\xbb'

In [5]:
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:376/files/prerender/prerender_page.html")+"?links_to_insert=2")
Out[5]: 'http://127.0.0.1:376/files/prerender/prerender_page.html'

This isn't quiiiiite the same as the listed failed which has:
File not found
prerender/prerender_page.html%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB

But! The HTML files don't specify an encoding, so they get interpreted as
"ISO-8859-1", so each byte Python sends just translates to that codepoint.
Except that's not quite true because what the web calls "ISO-8859-1" is actually
Windows-1252 which is almost the same, but with some control characters swapped
out. The only change being that the first byte, \x96 decodes into U+2013.

%-escaping that to a URL component (which encodes as UTF-8) gives us:
encodeURIComponent("\u2013)\xe4\xb3\xfbh\xfe)\xecz\xbb")
"%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB"

The exact sequence of garbage in the failure. Yay.

[mmenke, 2014-02-19 19:32:40]

On 2014/02/18 23:25:45, David Benjamin wrote:
> I'm sort of amazed the original version worked at all. I guess we were unlucky
> in our URL counts modulo 3.
> 
> ----
> The following isn't relevant to much, but is amusing and is my way to convince
> myself that this is, in fact, the cause of the failure:
> 
> So, the original implementation would produce URLs that looked like
> 
> prerender_loader.html?replace_text=BASE64BLOB:BASE64BLOB?links_to_insert=2
> (note the second ? instead of &)
> 
> The test server parses those out using cgi.parse_qs. cgi.parse_qs is not
lenient
> about ? vs. &:
> 
> In [14]: cgi.parse_qs("?a=1&b=2?c=3")
> Out[14]: {'?a': ['1'], 'b': ['2?c=3']}
> 
> So, test_server.py sees that you want to replace BASE64BLOB with
> BASE64BLOB?links_to_insert=2. As it turns out, base64.urlsafe_b64decode is
very
> very lenient. It skips characters it doesn't understand and such. It does,
> however, stop at '=', which is why it doesn't fail on every bot.
> 
> BUT if the host + port of the test server is such that there is no base64
> padding, links_to_insert gets decoded as well.
> 
> In [3]:
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:37601/files/prerender/prerender_page.html")+"?links_to_insert=2")
> Out[3]: 'http://127.0.0.1:37601/files/prerender/prerender_page.html'
> 
> In [4]:
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:3760/files/prerender/prerender_page.html")+"?links_to_insert=2")
> Out[4]:
>
'http://127.0.0.1:3760/files/prerender/prerender_page.html\x96)\xe4\xb3\xfbh\xfe)\xecz\xbb'
> 
> In [5]:
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:376/files/prerender/prerender_page.html")+"?links_to_insert=2")
> Out[5]: 'http://127.0.0.1:376/files/prerender/prerender_page.html'
> 
> This isn't quiiiiite the same as the listed failed which has:
> File not found
> prerender/prerender_page.html%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB
> 
> But! The HTML files don't specify an encoding, so they get interpreted as
> "ISO-8859-1", so each byte Python sends just translates to that codepoint.
> Except that's not quite true because what the web calls "ISO-8859-1" is
actually
> Windows-1252 which is almost the same, but with some control characters
swapped
> out. The only change being that the first byte, \x96 decodes into U+2013.
> 
> %-escaping that to a URL component (which encodes as UTF-8) gives us:
> encodeURIComponent("\u2013)\xe4\xb3\xfbh\xfe)\xecz\xbb")
> "%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB"
> 
> The exact sequence of garbage in the failure. Yay.

I'm following the bug, but how did the old CL pass the CQ?

[davidben, 2014-02-19 19:36:40]

On 2014/02/19 19:32:40, mmenke wrote:
> On 2014/02/18 23:25:45, David Benjamin wrote:
> > I'm sort of amazed the original version worked at all. I guess we were
unlucky
> > in our URL counts modulo 3.
> > 
> > ----
> > The following isn't relevant to much, but is amusing and is my way to
convince
> > myself that this is, in fact, the cause of the failure:
> > 
> > So, the original implementation would produce URLs that looked like
> > 
> > prerender_loader.html?replace_text=BASE64BLOB:BASE64BLOB?links_to_insert=2
> > (note the second ? instead of &)
> > 
> > The test server parses those out using cgi.parse_qs. cgi.parse_qs is not
> lenient
> > about ? vs. &:
> > 
> > In [14]: cgi.parse_qs("?a=1&b=2?c=3")
> > Out[14]: {'?a': ['1'], 'b': ['2?c=3']}
> > 
> > So, test_server.py sees that you want to replace BASE64BLOB with
> > BASE64BLOB?links_to_insert=2. As it turns out, base64.urlsafe_b64decode is
> very
> > very lenient. It skips characters it doesn't understand and such. It does,
> > however, stop at '=', which is why it doesn't fail on every bot.
> > 
> > BUT if the host + port of the test server is such that there is no base64
> > padding, links_to_insert gets decoded as well.
> > 
> > In [3]:
> >
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:37601/files/prerender/prerender_page.html")+"?links_to_insert=2")
> > Out[3]: 'http://127.0.0.1:37601/files/prerender/prerender_page.html'
> > 
> > In [4]:
> >
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:3760/files/prerender/prerender_page.html")+"?links_to_insert=2")
> > Out[4]:
> >
>
'http://127.0.0.1:3760/files/prerender/prerender_page.html\x96)\xe4\xb3\xfbh\xfe)\xecz\xbb'
> > 
> > In [5]:
> >
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:376/files/prerender/prerender_page.html")+"?links_to_insert=2")
> > Out[5]: 'http://127.0.0.1:376/files/prerender/prerender_page.html'
> > 
> > This isn't quiiiiite the same as the listed failed which has:
> > File not found
> >
prerender/prerender_page.html%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB
> > 
> > But! The HTML files don't specify an encoding, so they get interpreted as
> > "ISO-8859-1", so each byte Python sends just translates to that codepoint.
> > Except that's not quite true because what the web calls "ISO-8859-1" is
> actually
> > Windows-1252 which is almost the same, but with some control characters
> swapped
> > out. The only change being that the first byte, \x96 decodes into U+2013.
> > 
> > %-escaping that to a URL component (which encodes as UTF-8) gives us:
> > encodeURIComponent("\u2013)\xe4\xb3\xfbh\xfe)\xecz\xbb")
> > "%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB"
> > 
> > The exact sequence of garbage in the failure. Yay.
> 
> I'm following the bug, but how did the old CL pass the CQ?

It only happens if the port number has the right number of digits mod 3. Maybe
also hostname differences but 'localhost' and '127.0.0.1' have the same number
of letters (hah). So the port needs to be 4 digits or 1 digit. 1 digit won't
happen, and I guess XP reliably allocated a 4-digit port while the other bots
gave out 5-digit ones?

[mmenke, 2014-02-19 19:40:29]

On 2014/02/19 19:36:40, David Benjamin wrote:
> On 2014/02/19 19:32:40, mmenke wrote:
> > On 2014/02/18 23:25:45, David Benjamin wrote:
> > > I'm sort of amazed the original version worked at all. I guess we were
> unlucky
> > > in our URL counts modulo 3.
> > > 
> > > ----
> > > The following isn't relevant to much, but is amusing and is my way to
> convince
> > > myself that this is, in fact, the cause of the failure:
> > > 
> > > So, the original implementation would produce URLs that looked like
> > > 
> > > prerender_loader.html?replace_text=BASE64BLOB:BASE64BLOB?links_to_insert=2
> > > (note the second ? instead of &)
> > > 
> > > The test server parses those out using cgi.parse_qs. cgi.parse_qs is not
> > lenient
> > > about ? vs. &:
> > > 
> > > In [14]: cgi.parse_qs("?a=1&b=2?c=3")
> > > Out[14]: {'?a': ['1'], 'b': ['2?c=3']}
> > > 
> > > So, test_server.py sees that you want to replace BASE64BLOB with
> > > BASE64BLOB?links_to_insert=2. As it turns out, base64.urlsafe_b64decode is
> > very
> > > very lenient. It skips characters it doesn't understand and such. It does,
> > > however, stop at '=', which is why it doesn't fail on every bot.
> > > 
> > > BUT if the host + port of the test server is such that there is no base64
> > > padding, links_to_insert gets decoded as well.
> > > 
> > > In [3]:
> > >
> >
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:37601/files/prerender/prerender_page.html")+"?links_to_insert=2")
> > > Out[3]: 'http://127.0.0.1:37601/files/prerender/prerender_page.html'
> > > 
> > > In [4]:
> > >
> >
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:3760/files/prerender/prerender_page.html")+"?links_to_insert=2")
> > > Out[4]:
> > >
> >
>
'http://127.0.0.1:3760/files/prerender/prerender_page.html\x96)\xe4\xb3\xfbh\xfe)\xecz\xbb'
> > > 
> > > In [5]:
> > >
> >
>
base64.urlsafe_b64decode(base64.urlsafe_b64encode("http://127.0.0.1:376/files/prerender/prerender_page.html")+"?links_to_insert=2")
> > > Out[5]: 'http://127.0.0.1:376/files/prerender/prerender_page.html'
> > > 
> > > This isn't quiiiiite the same as the listed failed which has:
> > > File not found
> > >
> prerender/prerender_page.html%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB
> > > 
> > > But! The HTML files don't specify an encoding, so they get interpreted as
> > > "ISO-8859-1", so each byte Python sends just translates to that codepoint.
> > > Except that's not quite true because what the web calls "ISO-8859-1" is
> > actually
> > > Windows-1252 which is almost the same, but with some control characters
> > swapped
> > > out. The only change being that the first byte, \x96 decodes into U+2013.
> > > 
> > > %-escaping that to a URL component (which encodes as UTF-8) gives us:
> > > encodeURIComponent("\u2013)\xe4\xb3\xfbh\xfe)\xecz\xbb")
> > > "%E2%80%93)%C3%A4%C2%B3%C3%BBh%C3%BE)%C3%ACz%C2%BB"
> > > 
> > > The exact sequence of garbage in the failure. Yay.
> > 
> > I'm following the bug, but how did the old CL pass the CQ?
> 
> It only happens if the port number has the right number of digits mod 3. Maybe
> also hostname differences but 'localhost' and '127.0.0.1' have the same number
> of letters (hah). So the port needs to be 4 digits or 1 digit. 1 digit won't
> happen, and I guess XP reliably allocated a 4-digit port while the other bots
> gave out 5-digit ones?

Ah, right...randomized port.  I completely forgot about that.  LGTM.

[davidben, 2014-02-19 19:42:15]

The CQ bit was checked by davidben@chromium.org

[commit-bot: I haz the power, 2014-02-19 19:45:05]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/davidben@chromium.org/166273007/1

[commit-bot: I haz the power, 2014-02-20 00:19:22]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/davidben@chromium.org/166273007/1

[commit-bot: I haz the power, 2014-02-20 02:04:30]

Change committed as 252156