Cached timestamps for toolchain update script

Cached timestamps for toolchain update script

When cached timestamps are available, and toolchain is up-to-date,
reduces the runtime to ~.2s.

R=maruel@chromium.org
TBR=cpu@chromium.org
BUG=323300
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=238217

[M-A Ruel, 2013-11-29 00:33:29]

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
File tools/win/toolchain/get_toolchain_if_necessary.py (right):

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
tools/win/toolchain/get_toolchain_if_necessary.py:22: for name in
sorted(f.lower() for f in files):
file_list = []
for root, dirs, files in os.walk(root):
  file_list.extend(os.path.join(root, f).lower() for f in files))
return sorted(file_list)

much simpler.

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
tools/win/toolchain/get_toolchain_if_necessary.py:49: matches = len(file_list)
== len(timestamps_data['files'])
I wonder about "legitimate" junk files, like thumbs.db, that would slow it down
or worse constantly force a sync.

[scottmg, 2013-11-29 01:54:09]

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
File tools/win/toolchain/get_toolchain_if_necessary.py (right):

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
tools/win/toolchain/get_toolchain_if_necessary.py:22: for name in
sorted(f.lower() for f in files):
On 2013/11/29 00:33:30, M-A Ruel wrote:
> file_list = []
> for root, dirs, files in os.walk(root):
>   file_list.extend(os.path.join(root, f).lower() for f in files))
> return sorted(file_list)
> 
> much simpler.

Interestingly, it results in a different order. Previously, the directories were
sorted and the files were sorted within the directories, but now x\a.txt will
appear before x\b\a.txt followed by x\c.txt.

But anyway... Done, and updated hash.

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
tools/win/toolchain/get_toolchain_if_necessary.py:49: matches = len(file_list)
== len(timestamps_data['files'])
On 2013/11/29 00:33:30, M-A Ruel wrote:
> I wonder about "legitimate" junk files, like thumbs.db, that would slow it
down
> or worse constantly force a sync.

Yeah... If it comes up we could blacklist some files. FWIW, it'll nuke the whole
folder when it reinstalls, so hopefully it sort of self-corrects.

[scottmg, 2013-11-29 01:58:56]

On 2013/11/29 01:54:09, scottmg wrote:
>
https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
> File tools/win/toolchain/get_toolchain_if_necessary.py (right):
> 
>
https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
> tools/win/toolchain/get_toolchain_if_necessary.py:22: for name in
> sorted(f.lower() for f in files):
> On 2013/11/29 00:33:30, M-A Ruel wrote:
> > file_list = []
> > for root, dirs, files in os.walk(root):
> >   file_list.extend(os.path.join(root, f).lower() for f in files))
> > return sorted(file_list)
> > 
> > much simpler.
> 
> Interestingly, it results in a different order. Previously, the directories
were
> sorted and the files were sorted within the directories, but now x\a.txt will
> appear before x\b\a.txt followed by x\c.txt.
> 
> But anyway... Done, and updated hash.
> 
>
https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
> tools/win/toolchain/get_toolchain_if_necessary.py:49: matches = len(file_list)
> == len(timestamps_data['files'])
> On 2013/11/29 00:33:30, M-A Ruel wrote:
> > I wonder about "legitimate" junk files, like thumbs.db, that would slow it
> down
> > or worse constantly force a sync.
> 
> Yeah... If it comes up we could blacklist some files. FWIW, it'll nuke the
whole
> folder when it reinstalls, so hopefully it sort of self-corrects.

It could proactively ignore hidden files I suppose.

[M-A Ruel, 2013-11-29 02:07:28]

On 2013/11/29 01:58:56, scottmg wrote:
> It could proactively ignore hidden files I suppose.

Yes, it's easy to create a desktop.ini file by error. You could use:

import ctypes.wintypes
GetFileAttributes = ctypes.windll.kernel32.GetFileAttributesW
GetFileAttributes.argtypes = (ctypes.wintypes.LPWSTR,)
GetFileAttributes.restype = ctypes.wintypes.DWORD
FILE_ATTRIBUTE_HIDDEN = 0x2
FILE_ATTRIBUTE_SYSTEM = 0x4

def is_hidden(filepath):
  p = GetFileAttributes(filepath)
  return p & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM)

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
File tools/win/toolchain/get_toolchain_if_necessary.py (right):

https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
tools/win/toolchain/get_toolchain_if_necessary.py:22: for name in
sorted(f.lower() for f in files):
On 2013/11/29 01:54:10, scottmg wrote:
> On 2013/11/29 00:33:30, M-A Ruel wrote:
> > file_list = []
> > for root, dirs, files in os.walk(root):
> >   file_list.extend(os.path.join(root, f).lower() for f in files))
> > return sorted(file_list)
> > 
> > much simpler.
> 
> Interestingly, it results in a different order. Previously, the directories
were
> sorted and the files were sorted within the directories, but now x\a.txt will
> appear before x\b\a.txt followed by x\c.txt.
> 
> But anyway... Done, and updated hash.

Ah! I didn't think about that. As you prefer.

[scottmg, 2013-11-29 03:20:45]

On 2013/11/29 02:07:28, M-A Ruel wrote:
> On 2013/11/29 01:58:56, scottmg wrote:
> > It could proactively ignore hidden files I suppose.
> 
> Yes, it's easy to create a desktop.ini file by error. You could use:
> 
> import ctypes.wintypes
> GetFileAttributes = ctypes.windll.kernel32.GetFileAttributesW
> GetFileAttributes.argtypes = (ctypes.wintypes.LPWSTR,)
> GetFileAttributes.restype = ctypes.wintypes.DWORD
> FILE_ATTRIBUTE_HIDDEN = 0x2
> FILE_ATTRIBUTE_SYSTEM = 0x4
> 
> def is_hidden(filepath):
>   p = GetFileAttributes(filepath)
>   return p & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM)

Thanks, done.

> 
>
https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
> File tools/win/toolchain/get_toolchain_if_necessary.py (right):
> 
>
https://codereview.chromium.org/95983002/diff/60001/tools/win/toolchain/get_t...
> tools/win/toolchain/get_toolchain_if_necessary.py:22: for name in
> sorted(f.lower() for f in files):
> On 2013/11/29 01:54:10, scottmg wrote:
> > On 2013/11/29 00:33:30, M-A Ruel wrote:
> > > file_list = []
> > > for root, dirs, files in os.walk(root):
> > >   file_list.extend(os.path.join(root, f).lower() for f in files))
> > > return sorted(file_list)
> > > 
> > > much simpler.
> > 
> > Interestingly, it results in a different order. Previously, the directories
> were
> > sorted and the files were sorted within the directories, but now x\a.txt
will
> > appear before x\b\a.txt followed by x\c.txt.
> > 
> > But anyway... Done, and updated hash.
> 
> Ah! I didn't think about that. As you prefer.

Doesn't make much difference, other than it can't change later. Kept the shorter
code.

[M-A Ruel, 2013-12-02 21:57:08]

lgtm

https://codereview.chromium.org/95983002/diff/120001/tools/win/toolchain/get_...
File tools/win/toolchain/get_toolchain_if_necessary.py (right):

https://codereview.chromium.org/95983002/diff/120001/tools/win/toolchain/get_...
tools/win/toolchain/get_toolchain_if_necessary.py:28: return p &
(FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM)
return bool(p & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM))

https://codereview.chromium.org/95983002/diff/120001/tools/win/toolchain/get_...
tools/win/toolchain/get_toolchain_if_necessary.py:42: def
MakeTimestampsFileName(root):
Make kind of implies creating something. But I don't have a better idea.

[scottmg, 2013-12-02 22:59:46]

https://codereview.chromium.org/95983002/diff/120001/tools/win/toolchain/get_...
File tools/win/toolchain/get_toolchain_if_necessary.py (right):

https://codereview.chromium.org/95983002/diff/120001/tools/win/toolchain/get_...
tools/win/toolchain/get_toolchain_if_necessary.py:28: return p &
(FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM)
On 2013/12/02 21:57:08, M-A Ruel wrote:
> return bool(p & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM))

Done.

https://codereview.chromium.org/95983002/diff/120001/tools/win/toolchain/get_...
tools/win/toolchain/get_toolchain_if_necessary.py:42: def
MakeTimestampsFileName(root):
On 2013/12/02 21:57:08, M-A Ruel wrote:
> Make kind of implies creating something. But I don't have a better idea.

It is creating a file name, I guess.

[scottmg, 2013-12-02 23:01:12]

tbr cpu for third_party sha1 update.

[scottmg, 2013-12-02 23:03:14]

Committed patchset #5 manually as r238217 (presubmit successful).