Add a test for escaping of backslashes in list expansion

Add a test for escaping of backslashes in list expansion

shlex.split will eat backslash (commonly found in windows paths).
Committed: http://code.google.com/p/gyp/source/detail?r=1150

[Mark Mentovai, 2011-12-14 17:53:54]

http://codereview.chromium.org/8937016/diff/1/pylib/gyp/input.py
File pylib/gyp/input.py (right):

http://codereview.chromium.org/8937016/diff/1/pylib/gyp/input.py#newcode739
pylib/gyp/input.py:739: output = shlex.split(str(replacement).replace('\\',
'\\\\'))
Are you sure that this is right? Seems perverse. The backslash is very special
and this doesn’t account for all possibilities, like backslashes inside
single-quoted strings.

>>> import shlex
>>> s = "a\\b c 'd\\e'"
>>> shlex.split(s)
['ab', 'c', 'd\\e']
>>> shlex.split(s.replace('\\', '\\\\'))
['a\\b', 'c', 'd\\\\e']

That’s kind of weird, don’t you agree?

[tony, 2011-12-14 18:41:39]

http://codereview.chromium.org/8937016/diff/1/pylib/gyp/input.py
File pylib/gyp/input.py (right):

http://codereview.chromium.org/8937016/diff/1/pylib/gyp/input.py#newcode739
pylib/gyp/input.py:739: output = shlex.split(str(replacement).replace('\\',
'\\\\'))
On 2011/12/14 17:53:54, Mark Mentovai wrote:
> Are you sure that this is right? Seems perverse. The backslash is very special
> and this doesn’t account for all possibilities, like backslashes inside
> single-quoted strings.
> 
> >>> import shlex
> >>> s = "a\\b c 'd\\e'"
> >>> shlex.split(s)
> ['ab', 'c', 'd\\e']
> >>> shlex.split(s.replace('\\', '\\\\'))
> ['a\\b', 'c', 'd\\\\e']
> 
> That’s kind of weird, don’t you agree?

Backslash is only special when being passed to a command line.  This is just
variable expansion.  Shouldn't the special handling of backslash in single
quotes be handled when executing the command?

It's more weird that we escape backslash in list expansion, but not in non-list
expansion.

[tony, 2011-12-14 18:43:26]

Oh, the specific case I'm fixing is where a list of inputs is generated and put
into a variable.  The command outputs Windows style paths with backslashes, but
they lose the backslashes before being stored in the variable.

[Mark Mentovai, 2011-12-14 18:47:40]

I understand that, I’m just pointing out that your proposal fixes one specific
case but breaks another. Can you think up a way to fix this in a generic way
that works well and has defined semantics?

We agreed (a million years ago) that this should use POSIX shell semantics.
Maybe the script that spits out Windows paths should just spit them out with
each argument enclosed within single quotes?

[tony, 2011-12-14 18:51:42]

On 2011/12/14 18:47:40, Mark Mentovai wrote:
> We agreed (a million years ago) that this should use POSIX shell semantics.

I see, so even if the variable isn't passed to a command (e.g., you're trying to
compute the name of a dependency), it should be treated as a shell parameter? 
If that's an explicit goal, then should we also pass non-list cases through
shlex.split?  That'll at least apply the same rules to both.

[Mark Mentovai, 2011-12-14 18:58:25]

No, I don’t think non-lists should go through shlex.split.

When we came up with this, we needed a way to expand a string into list context.
We could have chosen any set of rules, but we chose shlex, because it allows
encoding of arbitrary data, it’s well understood, and a Python implementation
already existed. A simple “split on spaces” sucks because it doesn’t allow you
to encode spaces into the individual list items. The idea was that if you needed
something to expand into list context in GYP, you’d just follow the rules and
use something that behaves properly when fed to shlex. In your case, that means
that the script that produces the strings will need to know it’s coming up with
something that will be fed to shlex, and behave appropriately. Wrapping things
in single quotes is easiest (assuming the input doesn’t have any single quotes
in it already—and if it does, there’s an easy way to escape those.)

[tony, 2011-12-14 19:43:02]

Ok, I've updated the change to just add a unittest so it's more clear that this
is intentional.

[tony, 2011-12-15 18:20:36]

Did you not want me to land this test?

[tony, 2011-12-19 18:45:44]

ping

[tony, 2012-01-03 17:32:31]

Hi Mark, did you want to review this or should I ask someone else?

[Mark Mentovai, 2012-01-12 20:58:53]

LGTM

(I’m finally reaching into the vacation backlog.)