Random-walk fuzzer for the Dart VM.

Random-walk fuzzer for the Dart VM.

Start with all the classes and libraries and various interesting values.  Repeatedly choose one as a receiver, construct a message it is likely to understand, send the message, and add the result.

R=groebert@google.com

Committed: https://code.google.com/p/dart/source/detail?r=43615

[groebert, 2015-01-16 09:55:15]

groebert@google.com changed reviewers:
+ groebert@google.com

[groebert, 2015-01-16 09:55:16]

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart
File fuzzer.dart (right):

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode12
fuzzer.dart:12: seed = new DateTime.now().millisecondsSinceEpoch & ((1<<31)-1);
If we execute on multiple machines at the same time, milliseconds will not be a
good source for the seed. Maybe also mix in /dev/urandom?

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode25
fuzzer.dart:25: while(true) {
Better use a pre-defined maximum number of iterations, for example 5M. That way,
you can restart the VM several times with new seeds but won't have to wait too
long to reproduce a bug with a given seed.

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode149
fuzzer.dart:149: if (method.isConstructor) {
Maybe also add a blacklist of known-bad classes/methods so they're skipped
during fuzzing. This could also check that we're not fuzzing anything from the
package fuzzer.

[rmacnak, 2015-02-05 00:38:28]

PTAL.

I've also fixed a few things in the VM so now this fuzzer runs a reasonably long
time before hanging from a blocking IO call or throwing a Dart exception high in
the event loop. Whatever is running this fuzzer should be able to distinguish an
uncaught exception from a crash by checking the exit code.

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart
File fuzzer.dart (right):

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode12
fuzzer.dart:12: seed = new DateTime.now().millisecondsSinceEpoch & ((1<<31)-1);
On 2015/01/16 09:55:16, groebert wrote:
> If we execute on multiple machines at the same time, milliseconds will not be
a
> good source for the seed. Maybe also mix in /dev/urandom?

Added

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode25
fuzzer.dart:25: while(true) {
On 2015/01/16 09:55:16, groebert wrote:
> Better use a pre-defined maximum number of iterations, for example 5M. That
way,
> you can restart the VM several times with new seeds but won't have to wait too
> long to reproduce a bug with a given seed.

Bounded

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode104
fuzzer.dart:104: // TODO: Lists and maps of these values.
Implemented.

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode149
fuzzer.dart:149: if (method.isConstructor) {
On 2015/01/16 09:55:16, groebert wrote:
> Maybe also add a blacklist of known-bad classes/methods so they're skipped
> during fuzzing. This could also check that we're not fuzzing anything from the
> package fuzzer.

Excluded the fuzzer itself and a few functions from dart:io.

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode194
fuzzer.dart:194: // TODO: Implement.
Implemented.

https://codereview.chromium.org/801113003/diff/80001/fuzzer.dart#newcode199
fuzzer.dart:199: // TODO: Chain a bunch of moderately sized arrays, then let go
of them.
Implemented.

[groebert, 2015-02-05 11:03:28]

LGTM.

Once submitted, I can run the fuzzer at scale. I'll start a new thread with you
and asiva on that.

[rmacnak, 2015-02-09 21:13:42]

New patchsets have been uploaded after l-g-t-m from groebert@google.com

[rmacnak, 2015-02-09 21:19:32]

Moved to runtime/tests/vm/dart/random_walk_fuzzer.dart

Note that because it does not end in test.dart this will not be run by our build
bots. Through dart:io it might trash the build bot's setup.

[rmacnak, 2015-02-09 21:32:46]

Committed patchset #9 (id:160001) manually as 43615 (presubmit successful).