Perform tlslite 0.3.8 -> 0.4.6 renames ahead of time.

Perform tlslite 0.3.8 -> 0.4.6 renames ahead of time.

The files all became lowercase which upsets the Windows and Mac bots. (Our
infrastructure can't handle case-only renames on case-insensitive filesystems.)

BUG=115301, 355812
R=mnissler@chromium.org, rsleevi@chromium.org

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=262158

[davidben, 2014-03-31 15:47:13]

And time to start the dominoes. This one's pretty mechanical; I'm just doing the
case-only renames ahead of time.

Try bots for Windows, Mac, and iOS are expected to be red; those bots can't
apply the patch. linux_redux seems to have been broken for some time anyway.

This CL will have to be committed manually. I've verified on a Windows 7 machine
that our git on Windows can checkout across changes that do case-only renames
correctly. I haven't checked if svn works. I'll warn the troopers/sheriffs ahead
of time in case this causes the bots to blow up.

Reasons for doing it this way:

- tlslite upstream renamed all the files from uppercase to lowercase.
apply_patch can't handle those on Windows and Mac.

- While the PYTHONPATH reference to third_party/tlslite is in one place in
src/net, third_party/tlslite is referenced elsewhere (I guess to get a list of
files needed to upload to this or that build slave) including in the separate
tools/build directory. petermayo tells me that coordinating changes between
tools/build/ and src/ will be difficult. So we can't easily move elsewhere from
third_party/tlslite.

- The actual switch from 0.3.8 to 0.4.6 really should have full try bot
coverage.

Doing the renames ahead of time seems the simplest approach here. Yet /another/
possibility that may work is to check it into third_party/tlslite/0.4.6/ or
something and then possibly only the PYTHONPATH reference needs to change. Then
it can be moved back to where it should be...

[Ryan Sleevi, 2014-03-31 19:42:14]

The .pyc change should be looked at from someone in infra w/ a love of Python.

You should file a bug for the tools/build dependency on tlslite - that's a
layering violation that makes testing harder, as you've seen.

https://codereview.chromium.org/211173006/diff/80002/net/test/python_utils.cc
File net/test/python_utils.cc (right):

https://codereview.chromium.org/211173006/diff/80002/net/test/python_utils.cc...
net/test/python_utils.cc:125: python_cmd->AppendArg("-B");
Someone from infra should review this change - as well as the performance
implications for try cycles.

[Ryan Sleevi, 2014-03-31 19:42:42]

Oh, and the mechanical change LGTM - just to be clear, I'm just asking for infra
to check .pyc is good by them.

[davidben, 2014-03-31 20:04:27]

+petermayo for the .pyc thing.

Peter, for context (I think I mentioned this to you while you were helping me
with sorting this out previously?):

tlslite 0.3.8 -> 0.4.6 removes a file util/hmac.py and instead imports the
standard hmac module. But there's a util/hmac.pyc file littered there from
before and if util/bar.py does "import hmac", util/hmac.py takes precedence over
the standard library one (ugh). util/hmac.py is gone, but util/hmac.pyc remains,
so Python kindly imports that one.

After this lands, when landing the actual 0.3.8 -> 0.4.6 roll, I plan to
temporarily put an os.remove(....) at the top of testserver.py to hack around
this. When that's cycled through bots and developers' checkouts, we can remove
it.

In the interest of not making this problem worse and littering checkouts with
even more pyc files, I'm adding -B while doing this rename so we don't get any
new pyc files. Is this reasonable? See Ryan's comment. It could potentially slow
down the try server's startup.

(We may also be able to offset some of the slowdown by patching out all the
imports from tlslite/api.py that we don't need. I recall importing twisted on
bots with it installed was a non-trivial portion of the startup time.)

I've verified that python -B does not get confused if there is a pyc file from
an old version of the py file. (It does, however, still have the pyc with
missing py bug.)

$ echo "import bar" > foo.py
$ echo "print 1" > bar.py
$ python foo.py
1
$ ls
bar.py  bar.pyc  foo.py
$ echo "print 2" > bar.py
$ python -B foo.py
2
$ rm bar.py
$ python -B foo.py
1

[davidben, 2014-04-03 00:37:12]

petermayo: Friendly ping. (See comments #2 and #4.)

[Peter Mayo (wrong one), 2014-04-03 21:16:30]

https://codereview.chromium.org/211173006/diff/80002/net/test/python_utils.cc
File net/test/python_utils.cc (right):

https://codereview.chromium.org/211173006/diff/80002/net/test/python_utils.cc...
net/test/python_utils.cc:125: python_cmd->AppendArg("-B");
On 2014/03/31 19:42:14, Ryan Sleevi wrote:
> Someone from infra should review this change - as well as the performance
> implications for try cycles.

I certainly don't want to see this (independent) change mixed with the rest of
this CL.

I would want to be able to revert it separately should it have surprising
consequences.
I also think we have better ways to solve the particular problem.

[davidben, 2014-04-03 21:29:32]

https://codereview.chromium.org/211173006/diff/80002/net/test/python_utils.cc
File net/test/python_utils.cc (right):

https://codereview.chromium.org/211173006/diff/80002/net/test/python_utils.cc...
net/test/python_utils.cc:125: python_cmd->AppendArg("-B");
On 2014/04/03 21:16:30, petermayo wrote:
> On 2014/03/31 19:42:14, Ryan Sleevi wrote:
> > Someone from infra should review this change - as well as the performance
> > implications for try cycles.
> 
> I certainly don't want to see this (independent) change mixed with the rest of
> this CL.
> 
> I would want to be able to revert it separately should it have surprising
> consequences.
> I also think we have better ways to solve the particular problem.

What are the better ways we have? It's a problem right now for the CL intended
to follow after this one, -B or no. Barring other options, I'm planning on
adding this hack to the top of the test server. What would you suggest instead?

https://codereview.chromium.org/210323002/diff/1260001/net/tools/testserver/t...

Should the -B still happen? If so, it'll want to happen before this CL. The -B
is not going to solve the hmac.pyc problem; both the bots and dev checkouts
already have hmac.pyc files and unless some mechanism to clear them on both as
part of the follow-on CL already exists, I don't see a way around the os.rename.

The goal of the -B is to ensure we have no future instances of this. That's why
it's not actually independent; the point is to do it now when I'm adding (well,
renaming) files so that no new pycs spring up.

I can land it separately and extending my yak chain by one. I can also just drop
it altogether because, like I said, I have to deal with hmac.pyc no matter what.
But not including -B means banking on the better mechanism existing by the time
this problem crops up again.

For what it's worth, depot_tools seems to pretty consistently set
PYTHONDONTWRITEBYTECODE=1 which is the environment variable version of -B. So
there is some precedence.

Preferences?

[davidben, 2014-04-03 21:59:10]

Dropped the -B after offline discussion. Instead, the plan is to pursue dropping
all the pyc files on gclient runhooks.

+mnissler for chrome/browser/policy/OWNERS. Missed that one.

[Mattias Nissler (ping if slow), 2014-04-07 08:21:50]

policy_testserver.py LGTM (feel free to TBR mechanic renames like this)

[davidben, 2014-04-07 18:17:10]

Committed patchset #4 manually as r262158 (presubmit successful).