Added orphaned compiled Python file checks.

Added orphaned compiled Python file checks.

There have been a handful of cases where orphaned compiled Python files (.pyc
files without an accompanying .py file) have been loaded preferentially by
the presubmit scripts, causing weird problems. This adds a check and a mode for
such files, either warning the user or deleting them.

BUG=None
TEST=local
  - Tested against local repository.

R=iannucci@chromium.org, szager@chromium.org

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=292703

[dnj, 2014-09-30 21:31:00]

Shot at a check to ensure errant "pyc" files don't silently ruin presubmit runs.

[dnj, 2014-10-27 19:13:27]

le ping

[szager1, 2014-10-28 17:08:14]

lgtm with nits

https://codereview.chromium.org/621533002/diff/1/tests/presubmit_unittest.py
File tests/presubmit_unittest.py (right):

https://codereview.chromium.org/621533002/diff/1/tests/presubmit_unittest.py#...
tests/presubmit_unittest.py:139: self.UnMock(presubmit.os, 'walk')
Can you group this together with the other presubmit.os mock overrides below? 
Bonus points for using the same mechanism for all of them.

https://codereview.chromium.org/621533002/diff/1/tests/presubmit_unittest.py#...
tests/presubmit_unittest.py:1199:
presubmit.IsCompiledPython(pyc_path).AndReturn(True)
Probably should also test by adding a mock invocation for os.remove and calling
CleanOrphanedCompiledPython with 'True'.

[dnj, 2014-10-28 18:37:35]

https://codereview.chromium.org/621533002/diff/1/tests/presubmit_unittest.py
File tests/presubmit_unittest.py (right):

https://codereview.chromium.org/621533002/diff/1/tests/presubmit_unittest.py#...
tests/presubmit_unittest.py:139: self.UnMock(presubmit.os, 'walk')
On 2014/10/28 17:08:14, szager1 wrote:
> Can you group this together with the other presubmit.os mock overrides below? 
> Bonus points for using the same mechanism for all of them.

Must have been late - didn't realize SuperMoxTestBase already mocks this
function.

https://codereview.chromium.org/621533002/diff/1/tests/presubmit_unittest.py#...
tests/presubmit_unittest.py:1199:
presubmit.IsCompiledPython(pyc_path).AndReturn(True)
On 2014/10/28 17:08:14, szager1 wrote:
> Probably should also test by adding a mock invocation for os.remove and
calling
> CleanOrphanedCompiledPython with 'True'.

I'm actually going to switch gears here - have one "False doesn't remove files"
test and leave the rest test the "True" code path, asserting correct "remove"
invocations.

[dnj, 2014-10-28 19:27:34]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2014-10-28 19:28:03]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/621533002/20001

[commit-bot: I haz the power, 2014-10-28 19:29:56]

Committed patchset #2 (id:20001) as 292703

[Nico, 2014-11-03 23:31:12]

thakis@chromium.org changed reviewers:
+ thakis@chromium.org

[Nico, 2014-11-03 23:31:13]

Uploading has been very very slow for me at times. Every time I ctrl-c it to add
"--bypass-hooks", `git cl upload` has been in this function. I think this is way
to slow to do by default in chrome checkouts. (This was on linux no less, it'll
be even slower on e.g. windows)



ps: stack from just now:
  File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 2892, in
<module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 2878, in
main
    return dispatcher.execute(OptionParser(), argv)
  File "/usr/local/google/home/thakis/src/depot_tools/subcommand.py", line 245,
in execute
    return command(parser, args[1:])
  File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 1837, in
CMDupload
    change=change)
  File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 902, in
RunHook
    rietveld_obj=self.RpcServer())
  File "/usr/local/google/home/thakis/src/depot_tools/presubmit_support.py",
line 1373, in DoPresubmitChecks
    remove=cleanup)
  File "/usr/local/google/home/thakis/src/depot_tools/presubmit_support.py",
line 1462, in CleanOrphanedCompiledPython
    for dirpath, _, filenames in os.walk(path):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 276, in walk
    names = listdir(top)
KeyboardInterrupt

[dnj, 2014-11-04 00:08:28]

On 2014/11/03 23:31:13, Nico wrote:
> Uploading has been very very slow for me at times. Every time I ctrl-c it to
add
> "--bypass-hooks", `git cl upload` has been in this function. I think this is
way
> to slow to do by default in chrome checkouts. (This was on linux no less,
it'll
> be even slower on e.g. windows)
> 
> 
> 
> ps: stack from just now:
>   File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 2892,
in
> <module>
>     sys.exit(main(sys.argv[1:]))
>   File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 2878,
in
> main
>     return dispatcher.execute(OptionParser(), argv)
>   File "/usr/local/google/home/thakis/src/depot_tools/subcommand.py", line
245,
> in execute
>     return command(parser, args[1:])
>   File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 1837,
in
> CMDupload
>     change=change)
>   File "/usr/local/google/home/thakis/src/depot_tools/git_cl.py", line 902, in
> RunHook
>     rietveld_obj=self.RpcServer())
>   File "/usr/local/google/home/thakis/src/depot_tools/presubmit_support.py",
> line 1373, in DoPresubmitChecks
>     remove=cleanup)
>   File "/usr/local/google/home/thakis/src/depot_tools/presubmit_support.py",
> line 1462, in CleanOrphanedCompiledPython
>     for dirpath, _, filenames in os.walk(path):
>   File "/usr/lib/python2.7/os.py", line 294, in walk
>     for x in walk(new_path, topdown, onerror, followlinks):
>   File "/usr/lib/python2.7/os.py", line 294, in walk
>     for x in walk(new_path, topdown, onerror, followlinks):
>   File "/usr/lib/python2.7/os.py", line 294, in walk
>     for x in walk(new_path, topdown, onerror, followlinks):
>   File "/usr/lib/python2.7/os.py", line 294, in walk
>     for x in walk(new_path, topdown, onerror, followlinks):
>   File "/usr/lib/python2.7/os.py", line 294, in walk
>     for x in walk(new_path, topdown, onerror, followlinks):
>   File "/usr/lib/python2.7/os.py", line 294, in walk
>     for x in walk(new_path, topdown, onerror, followlinks):
>   File "/usr/lib/python2.7/os.py", line 276, in walk
>     names = listdir(top)
> KeyboardInterrupt

I can't think of a better way to identify these files off the top of my head,
since they would not be indexed anywhere, so feel free to revert.

[Nico, 2014-11-04 00:11:23]

A revert of this CL (patchset #2 id:20001) has been created in
https://codereview.chromium.org/701683002/ by thakis@chromium.org.

The reason for reverting is: Makes `git cl upload` very slow in chrome
checkouts..