Add image-first-tests flag to run-webkit-tests

third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py

 # Copyright (C) 2010 Google Inc. All rights reserved.
 # Copyright (C) 2010 Gabor Rapcsanyi (rgabor@inf.u-szeged.hu), University of Szeged
 # Copyright (C) 2011 Apple Inc. All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
 # modification, are permitted provided that the following conditions are
 # met:
 #
 #     * Redistributions of source code must retain the above copyright
 # notice, this list of conditions and the following disclaimer.
@@ skipping 217 matching lines @@
                       'you might miss new crashes in these tests.')),
             optparse.make_option(
                 '--smoke',
                 action='store_true',
                 help='Run just the SmokeTests'),
             optparse.make_option(
                 '--no-smoke',
                 dest='smoke',
                 action='store_false',
                 help='Do not run just the SmokeTests'),
+            optparse.make_option(
+                '--image-first-tests',
+                action='append',
+                default=[],
+                dest='image_first_tests',
+                help='directories or test that will be only checked for image'
+                ' expectation if it\'s available and fall back to test '
+                'expectation if image expectation is missing (may specify '
+                'multiple times)'),

[Dirk Pranke, 2017/03/20 23:22:27]

Is there a reason to make this a command line argument rather than a hard-coded
list somewhere in the code or in a config file? 

Where and when would we expect these flags to be passed?

[Gleb Lanbin, 2017/03/21 17:31:52]

The initial plan was to add a flag that can be used locally for testing and with
trybots(we need to update trybot JSON recipe for this?). 

So we want to start with the limited set of folders, experiment with it a little
bit and if there are no complaints then send an announcement to blink-dev and
make this testing approach as default. After that we can remove this flag.

If you say that the updating trybot JSON configuration would be problematic then
we can use the "default" option at the line 241 to set all hardcoded values. I
can add a TODO to the flag's description and attach it to the bug so we can
delete this flag later. What do you think?

         ]))
 
     option_group_definitions.append(
         ('Testing Options', [
             optparse.make_option(
                 '--additional-env-var',
                 type='string',
                 action='append',
                 default=[],
                 help=('Passes that environment variable to the tests '
@@ skipping 338 matching lines @@
             _log.debug('\t%s', process)
 
         return run_details
 
     finally:
         printer.cleanup()
 
 if __name__ == '__main__':
     exit_code = main(sys.argv[1:], sys.stdout, sys.stderr)
     sys.exit(exit_code)