Add the ability to run tests with several sets of VM flags

Add the ability to run tests with several sets of VM flags

The test-option VMOptions that can be placed in .dart test files can now be specified several times like this:

// VMOptions=
// VMOptions=--short_socket_read
// VMOptions=--short_socket_write
// VMOptions=--short_socket_read --short_socket_write

For each time VMOptions is specified a seperate test will be bun with the specified flags.

Used this to run the tests that uses sockets with short read and writes. Fixed a number of bugs in both stream implementation and in tests revealed by this.

R=iposva@google.com, ngeoffray@google.com

Committed: https://code.google.com/p/dart/source/detail?r=388

[Ivan Posva, 2011-10-12 13:53:57]

As we discussed in person, we should instead use an extensible approach. If a
test file has multiple lines with VM options it should execute the test multiple
times each passing one set of options.

Thanks,
-Ivan

[Søren Gjesse, 2011-10-13 07:03:27]

Changed the approach to use several VMOptions lines.

PTAL

[Ivan Posva, 2011-10-13 07:17:33]

LGTM -ip

http://codereview.chromium.org/8226016/diff/10001/tools/testing/architecture.py
File tools/testing/architecture.py (right):

http://codereview.chromium.org/8226016/diff/10001/tools/testing/architecture....
tools/testing/architecture.py:376: return ExeuteCommand(command, verbose)
?

http://codereview.chromium.org/8226016/diff/10001/tools/testing/test_configur...
File tools/testing/test_configuration.py (right):

http://codereview.chromium.org/8226016/diff/10001/tools/testing/test_configur...
tools/testing/test_configuration.py:77: # Look for VM specified as comments in
teh source file. If
Look for VM options specified...
teh -> the

[Søren Gjesse, 2011-10-13 07:28:15]

http://codereview.chromium.org/8226016/diff/10001/tools/testing/architecture.py
File tools/testing/architecture.py (right):

http://codereview.chromium.org/8226016/diff/10001/tools/testing/architecture....
tools/testing/architecture.py:376: return ExeuteCommand(command, verbose)
On 2011/10/13 07:17:34, Ivan Posva wrote:
> ?

Accidental edit.

http://codereview.chromium.org/8226016/diff/10001/tools/testing/test_configur...
File tools/testing/test_configuration.py (right):

http://codereview.chromium.org/8226016/diff/10001/tools/testing/test_configur...
tools/testing/test_configuration.py:77: # Look for VM specified as comments in
teh source file. If
On 2011/10/13 07:17:34, Ivan Posva wrote:
> Look for VM options specified...
> teh -> the

Done.

[ngeoffray, 2011-10-13 07:53:24]

LGTM!

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py
File tools/testing/test_case.py (right):

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py#...
tools/testing/test_case.py:27: def __init__(self, context, path, filename, mode,
arch, vm_options = None):
vm_options = [] ? and then you can remove the null check?

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py#...
tools/testing/test_case.py:38: for vm_option in vm_options:
for flag in vm_options

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py#...
tools/testing/test_case.py:39: if (len(vm_option) > 0):
Maybe you could avoid this len check by doing it before calling this method? It
looks silly to check it on each vm flag.

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_configur...
File tools/testing/test_configuration.py (right):

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_configur...
tools/testing/test_configuration.py:76: if arch in ['ia32', 'x64', 'simarm',
'arm']:
I don't think we want to filter on the arch. The feature is also useful for the
other archs (yes, VMOptions is misleading).

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_configur...
tools/testing/test_configuration.py:81: vm_options =
utils.ParseTestOptionsMultiple(VM_OPTIONS_PATTERN,
vm_options_list = ?

[Søren Gjesse, 2011-10-13 08:57:07]

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py
File tools/testing/test_case.py (right):

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py#...
tools/testing/test_case.py:27: def __init__(self, context, path, filename, mode,
arch, vm_options = None):
On 2011/10/13 07:53:24, ngeoffray wrote:
> vm_options = [] ? and then you can remove the null check?

Done.

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py#...
tools/testing/test_case.py:38: for vm_option in vm_options:
On 2011/10/13 07:53:24, ngeoffray wrote:
> for flag in vm_options

Done.

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_case.py#...
tools/testing/test_case.py:39: if (len(vm_option) > 0):
On 2011/10/13 07:53:24, ngeoffray wrote:
> Maybe you could avoid this len check by doing it before calling this method?
It
> looks silly to check it on each vm flag.

Done.

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_configur...
File tools/testing/test_configuration.py (right):

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_configur...
tools/testing/test_configuration.py:76: if arch in ['ia32', 'x64', 'simarm',
'arm']:
On 2011/10/13 07:53:24, ngeoffray wrote:
> I don't think we want to filter on the arch. The feature is also useful for
the
> other archs (yes, VMOptions is misleading).

Done.

http://codereview.chromium.org/8226016/diff/13001/tools/testing/test_configur...
tools/testing/test_configuration.py:81: vm_options =
utils.ParseTestOptionsMultiple(VM_OPTIONS_PATTERN,
On 2011/10/13 07:53:24, ngeoffray wrote:
> vm_options_list = ?

Done.