create_nmf improvements

Several improvements to create_nmf.py

Add -v/--verbose so you can see what the tool is doing.

Fix for StageDependencies so that it treats the main files
in the same special way that _GenerateManifest does.  The
current behavior creates duplicate nested copies of
of the all main (.nexe) files you pass it.

Remove trailing whitespace.

Add a __repr__ for ArchFile so printing them is less opaque.

Remove ErrorOut in favor of Error exception.  Calling 
sys.exit() all over the place makes things like unit testing
harder.  Error exception is now caught in the main entry 
point so the user doesn't see stack traces for errors.

BUG=

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=149557

[binji, 2012-08-01 04:06:46]

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
File native_client_sdk/src/tools/create_nmf.py (right):

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:88: return "ArchFile: %s" % self.path
repr(foo) is usually "<foo ...>"

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:259: if source.endswith('.nexe') and
source in self.main_files:
I don't think we should make any determination based on the filename. The user
can name their nexe whatever they like.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:266: if not os.path.exists(dirname):
IMO, catching the exception is better. Testing for existence doesn't mean
os.makedirs will succeed.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:345: 
style: Two newlines between top-level definitions.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:384: raise Error("No nexe files
specified.  See --help for more info")
maybe just display help in this case?

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:386: remap = {}
I prefer initialization closer to first use.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:413: Trace("Staging dependancies...")
sp: dependencies

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:423: except Error, e:
How does this output differ from the default output when a thrown exception
isn't caught?

[Sam Clegg, 2012-08-01 17:21:03]

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
File native_client_sdk/src/tools/create_nmf.py (right):

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:88: return "ArchFile: %s" % self.path
On 2012/08/01 04:06:46, binji wrote:
> repr(foo) is usually "<foo ...>"

Done.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:259: if source.endswith('.nexe') and
source in self.main_files:
On 2012/08/01 04:06:46, binji wrote:
> I don't think we should make any determination based on the filename. The user
> can name their nexe whatever they like.

I'm only mimicking what _GenerateManifest does.  The job of this function is to
copy files into place to match the manifest right?  So I have to do the same
thing.  You could argue that _GenerateManifest is wrong to use endswith() I
guess.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:345: 
On 2012/08/01 04:06:46, binji wrote:
> style: Two newlines between top-level definitions.

Done.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:384: raise Error("No nexe files
specified.  See --help for more info")
On 2012/08/01 04:06:46, binji wrote:
> maybe just display help in this case?

The convention seems to be to tell the user what he did 
wrong on one line (and sometimes point to the help option)..
unless there really did as for --help.

Try.. ld --bad-arg.. vi --bad-arg .. or even create_mkb.py 
--bad-arg (which is automatically generated by optparse).

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:386: remap = {}
On 2012/08/01 04:06:46, binji wrote:
> I prefer initialization closer to first use.

Ooops.  Didn't mean to move that.

Done.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:413: Trace("Staging dependancies...")
On 2012/08/01 04:06:46, binji wrote:
> sp: dependencies 

Done.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:423: except Error, e:
On 2012/08/01 04:06:46, binji wrote:
> How does this output differ from the default output when a thrown exception
> isn't caught?

When an Error (local class) is thrown it represents an 
error that the user should see.

We catch it here and display it for the user rather than 
letting the interpreter catch it on the way out and display
a backtrace.  This is a direct replacement for code that 
calls sys.exit() after printing an error message (i.e. times
when a backtrace is not wanted/needed/appropriate).

In my experience this is much better to use a local 
exception class than have code calling sys.exit().

It allows for much nicer code reuse and testing.

[binji, 2012-08-01 18:05:20]

lgtm

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
File native_client_sdk/src/tools/create_nmf.py (right):

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:259: if source.endswith('.nexe') and
source in self.main_files:
ok

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:266: if not os.path.exists(dirname):
On 2012/08/01 04:06:46, binji wrote:
> IMO, catching the exception is better. Testing for existence doesn't mean
> os.makedirs will succeed.

opinion on this?

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:384: raise Error("No nexe files
specified.  See --help for more info")
On 2012/08/01 17:21:03, Sam Clegg wrote:
> On 2012/08/01 04:06:46, binji wrote:
> > maybe just display help in this case?
> 
> The convention seems to be to tell the user what he did 
> wrong on one line (and sometimes point to the help option)..
> unless there really did as for --help.
> 
> Try.. ld --bad-arg.. vi --bad-arg .. or even create_mkb.py 
> --bad-arg (which is automatically generated by optparse).

ok

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:423: except Error, e:
sgtm

[Sam Clegg, 2012-08-01 18:26:39]

I also add a --debug/-d option to enable _debug_mode dynamically.

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
File native_client_sdk/src/tools/create_nmf.py (right):

http://codereview.chromium.org/10837028/diff/2001/native_client_sdk/src/tools...
native_client_sdk/src/tools/create_nmf.py:266: if not os.path.exists(dirname):
On 2012/08/01 04:06:46, binji wrote:
> IMO, catching the exception is better. Testing for existence doesn't mean
> os.makedirs will succeed.

Well.. it pretty much guarantees that it won't fail with EEXIST doesn't, which? 
 Which is exactly what the old code does.

I suppose you could argue that EEXISTS could still fire if some other process
makes that directory between the check and the call to makedirs?

Is that what you are getting at?  Some kind of protection against
multi-process/multi-thread race condition?

I would argue that if you have another process messing with the tree while this
script is running it is going to fail
at some point for sure :)    For example, if another process were to delete the
nexe files between GenerateManifest and
StageDependancies.  

On balance I guess the try/catch version is marginally safer,  but I find it
offensive that it takes 5 lines to call mkdir :)   Maybe I'll split it into a
separate function ?

http://stackoverflow.com/questions/600268/mkdir-p-functionality-in-python

[binji, 2012-08-01 22:56:16]

slgtm :)

[commit-bot: I haz the power, 2012-08-01 23:09:12]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/sbc@chromium.org/10837028/4003

[commit-bot: I haz the power, 2012-08-02 01:52:26]

Change committed as 149557