Move the layout test scripts into a 'webkitpy' subdirectory in preparation...

Move the layout test scripts into a 'webkitpy' subdirectory in preparation
for upstreaming them to WebKit/WebKitTools/Scripts/webkitpy . This also
involves changing run_webkit_tests.py to run_chromium_webkit_tests.py to
minimize confusion with the existing run-webkit-tests perl script upstream.

There should be no user-visible impact as the existing shell and batch scripts
have been updated for the new paths.

  BUG=none
  TEST=none
  R=eseidel@chromium.org


Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=36739

[Dirk Pranke, 2010-01-21 04:03:12]

take a look and make sure this looks kosher. The try bots seem to be getting
confused by this patch, so I may just have to check it in later tonight to test
it.

[eseidel, 2010-01-21 04:13:36]

I think all the executables would want to stay at the root level, no?

like rebaseline.py and run_chromium_webkit_tests.py?

Also, the webkittens will probably want run- instead of run_ for all
these, but we can always change at a later date.

looks OK to me.  Again, not sure you want to move the executables, but
otherwise OK.

webkitpy is supposed to hide away all the python support junk.
everything directly inside WebKitTools/Scripts is executable.  There
are a few perl modules left there which are soon to be moved to
webkitperl.

-eric

On Wed, Jan 20, 2010 at 8:03 PM,  <dpranke@chromium.org> wrote:
> Reviewers: Eric Seidel (Google),
>
> Message:
> take a look and make sure this looks kosher. The try bots seem to be getting
> confused by this patch, so I may just have to check it in later tonight to
> test
> it.
>
> Description:
> Move the layout test scripts into a 'webkitpy' subdirectory in preparation
> for upstreaming them to WebKit/WebKitTools/Scripts/webkitpy . This also
> involves changing run_webkit_tests.py to run_chromium_webkit_tests.py to
> minimize confusion with the existing run-webkit-tests perl script upstream.
>
> There should be no user-visible impact as the existing shell and batch
> scripts
> have been updated for the new paths.
>
>  BUG=none
>  TEST=none
>  R=eseidel@chromium.org
>
>
> Please review this at http://codereview.chromium.org/545145
>
> SVN Base: svn://chrome-svn/chrome/trunk/src/
>
> Affected files:
>  D     webkit/tools/layout_tests/dedup-tests.py
>  D     webkit/tools/layout_tests/layout_package/apache_http_server.py
>  D     webkit/tools/layout_tests/layout_package/failure.py
>  D     webkit/tools/layout_tests/layout_package/failure_finder.py
>  D     webkit/tools/layout_tests/layout_package/failure_finder_test.py
>  D     webkit/tools/layout_tests/layout_package/html_generator.py
>  D     webkit/tools/layout_tests/layout_package/http_server.bat
>  D     webkit/tools/layout_tests/layout_package/http_server.py
>  D     webkit/tools/layout_tests/layout_package/http_server.sh
>  D     webkit/tools/layout_tests/layout_package/http_server_base.py
>  D
> webkit/tools/layout_tests/layout_package/json_layout_results_generator.py
>  D     webkit/tools/layout_tests/layout_package/json_results_generator.py
>  D     webkit/tools/layout_tests/layout_package/lighttpd.conf
>  D     webkit/tools/layout_tests/layout_package/metered_stream.py
>  D     webkit/tools/layout_tests/layout_package/path_utils.py
>  D     webkit/tools/layout_tests/layout_package/platform_utils.py
>  D     webkit/tools/layout_tests/layout_package/platform_utils_linux.py
>  D     webkit/tools/layout_tests/layout_package/platform_utils_mac.py
>  D     webkit/tools/layout_tests/layout_package/platform_utils_win.py
>  D     webkit/tools/layout_tests/layout_package/test_expectations.py
>  D     webkit/tools/layout_tests/layout_package/test_failures.py
>  D     webkit/tools/layout_tests/layout_package/test_files.py
>  D     webkit/tools/layout_tests/layout_package/test_shell_thread.py
>  D     webkit/tools/layout_tests/layout_package/websocket_server.py
>  M     webkit/tools/layout_tests/rebaseline.bat
>  D     webkit/tools/layout_tests/rebaseline.py
>  M     webkit/tools/layout_tests/rebaseline.sh
>  M     webkit/tools/layout_tests/run_webkit_tests.bat
>  D     webkit/tools/layout_tests/run_webkit_tests.py
>  M     webkit/tools/layout_tests/run_webkit_tests.sh
>  M     webkit/tools/layout_tests/test_output_formatter.bat
>  D     webkit/tools/layout_tests/test_output_formatter.py
>  M     webkit/tools/layout_tests/test_output_formatter.sh
>  D     webkit/tools/layout_tests/test_output_xml_to_json.py
>  D     webkit/tools/layout_tests/test_types/fuzzy_image_diff.py
>  D     webkit/tools/layout_tests/test_types/image_diff.py
>  D     webkit/tools/layout_tests/test_types/test_type_base.py
>  D     webkit/tools/layout_tests/test_types/text_diff.py
>  D     webkit/tools/layout_tests/update_expectations_from_dashboard.py
>  D
> webkit/tools/layout_tests/update_expectations_from_dashboard_unittest.py
>  A  +  webkit/tools/layout_tests/webkitpy/dedup-tests.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/__init__.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/apache_http_server.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/failure.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/failure_finder.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/failure_finder_test.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/html_generator.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/http_server.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/http_server_base.py
>  A  +
>
 webkit/tools/layout_tests/webkitpy/layout_package/json_layout_results_generator.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/json_results_generator.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/metered_stream.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/path_utils.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/platform_utils.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/platform_utils_linux.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/platform_utils_mac.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/platform_utils_win.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/test_expectations.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/test_failures.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/test_files.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/layout_package/test_shell_thread.py
>  A  +  webkit/tools/layout_tests/webkitpy/layout_package/websocket_server.py
>  A  +  webkit/tools/layout_tests/webkitpy/rebaseline.py
>  A  +  webkit/tools/layout_tests/webkitpy/run_chromium_webkit_tests.py
>  A  +  webkit/tools/layout_tests/webkitpy/test_output_formatter.py
>  A  +  webkit/tools/layout_tests/webkitpy/test_output_xml_to_json.py
>  A  +  webkit/tools/layout_tests/webkitpy/test_types/__init__.py
>  A  +  webkit/tools/layout_tests/webkitpy/test_types/fuzzy_image_diff.py
>  A  +  webkit/tools/layout_tests/webkitpy/test_types/image_diff.py
>  A  +  webkit/tools/layout_tests/webkitpy/test_types/test_type_base.py
>  A  +  webkit/tools/layout_tests/webkitpy/test_types/text_diff.py
>  A  +
>  webkit/tools/layout_tests/webkitpy/update_expectations_from_dashboard.py
>  A  +
>
 webkit/tools/layout_tests/webkitpy/update_expectations_from_dashboard_unittest.py
>
>
>

[cjerdonek_webkit.org, 2010-01-21 04:46:37]

Thanks for the forward, Eric.

Just a couple high-level comments:

I can't totally tell from the patch how the supporting files (e.g. the
non-executables) will sit inside webkit.py.  Can/will they all be in a
single subfolder of webkitpy?  In terms of sub-folder naming,
"*_package" is redundant, so a subfolder name like "layout/" is
probably better than something like "layout_package".

Finally, I don't know how many executables there are and what they are
for, but if some of them are not meant to be called by the end-user
(e.g. they are meant to be called internally by the end-user scripts),
then it might be preferable to hide those away somewhere inside
webkitpy as well.  Just a thought.

Thanks,
--Chris


On Wed, Jan 20, 2010 at 8:13 PM, Eric Seidel <eseidel@chromium.org> wrote:
> I think all the executables would want to stay at the root level, no?
>
> like rebaseline.py and run_chromium_webkit_tests.py?
>
> Also, the webkittens will probably want run- instead of run_ for all
> these, but we can always change at a later date.
>
> looks OK to me.  Again, not sure you want to move the executables, but
> otherwise OK.
>
> webkitpy is supposed to hide away all the python support junk.
> everything directly inside WebKitTools/Scripts is executable.  There
> are a few perl modules left there which are soon to be moved to
> webkitperl.
>
> -eric
>
> On Wed, Jan 20, 2010 at 8:03 PM,  <dpranke@chromium.org> wrote:
>> Reviewers: Eric Seidel (Google),
>>
>> Message:
>> take a look and make sure this looks kosher. The try bots seem to be getting

[tkent, 2010-01-21 06:58:22]

I thought the change was ok. But It seems that build bots directly invoke
webkit/tools/layout_tests/run_webkit_test.py as we saw.
So we had better not move it from layout_tests/.

[cjerdonek_webkit.org, 2010-01-21 10:00:22]

Thanks, Dirk.

To help keep things organized, unless the top-level drivers rely
extensively on existing code in webkitpy, I would encourage you to
consider putting all of the python code into a subfolder of webkitpy.
For example--

webkitpy/
    runlayout/ (or whatever you prefer)
        layout/
        test_types/
        <top-level drivers>

The reason is that webkitpy already contains ~35 python files, so the
top-level drivers may get lost within an already long list -- hiding
that they work together.

The style package follows a similar model with an end-user executable
in WebKitTools/Scripts/, and all of the supporting code in its own
webkitpy subfolder called webkitpy/style.

--Chris


On Wed, Jan 20, 2010 at 11:07 PM, Dirk Pranke <dpranke@chromium.org> wrote:
> The high-level executables are .bat and .sh scripts that wrap the
> python scripts; they stay in webkit/tools/layout_tests (and will move
> to WebKitTools/Scripts). *All* of the python code will move to
> Scripts/webkitpy ; there are a few top-level drivers, and then most of
> the code is in two subfolders (test_types and layout_package). You're
> write that the layout_package name is redundant. I can change that
> when I upload the webkit patch. I can also move the test_types dir
> into the layout dir if we'd prefer that approach.
>
> All of the executables are end-user callable.
>
> -- Dirk
>
> On Wed, Jan 20, 2010 at 8:46 PM, Chris Jerdonek <cjerdonek@webkit.org> wrote:
>> Thanks for the forward, Eric.
>>
>> Just a couple high-level comments:
>>
>> I can't totally tell from the patch how the supporting files (e.g. the
>> non-executables) will sit inside webkit.py.  Can/will they all be in a
>> single subfolder of webkitpy?  In terms of sub-folder naming,
>> "*_package" is redundant, so a subfolder name like "layout/" is
>> probably better than something like "layout_package".
>>
>> Finally, I don't know how many executables there are and what they are
>> for, but if some of them are not meant to be called by the end-user
>> (e.g. they are meant to be called internally by the end-user scripts),
>> then it might be preferable to hide those away somewhere inside
>> webkitpy as well.  Just a thought.
>>
>> Thanks,
>> --Chris
>>
>>
>> On Wed, Jan 20, 2010 at 8:13 PM, Eric Seidel <eseidel@chromium.org> wrote:
>>> I think all the executables would want to stay at the root level, no?
>>>
>>> like rebaseline.py and run_chromium_webkit_tests.py?
>>>
>>> Also, the webkittens will probably want run- instead of run_ for all
>>> these, but we can always change at a later date.
>>>
>>> looks OK to me.  Again, not sure you want to move the executables, but
>>> otherwise OK.
>>>
>>> webkitpy is supposed to hide away all the python support junk.
>>> everything directly inside WebKitTools/Scripts is executable.  There
>>> are a few perl modules left there which are soon to be moved to
>>> webkitperl.
>>>
>>> -eric
>>>
>>> On Wed, Jan 20, 2010 at 8:03 PM,  <dpranke@chromium.org> wrote:
>>>> Reviewers: Eric Seidel (Google),
>>>>
>>>> Message:
>>>> take a look and make sure this looks kosher. The try bots seem to be
getting
>>
>

[eseidel, 2010-01-21 20:30:22]

Yeah, Adam and I plan to shove more of webkitpy down into directories
at some point.  webkpy used to just be bugzila-tool support files. :)
There has been a bit of a python explosion as of late.

On Thu, Jan 21, 2010 at 2:00 AM, Chris Jerdonek <cjerdonek@webkit.org> wrote:
> Thanks, Dirk.
>
> To help keep things organized, unless the top-level drivers rely
> extensively on existing code in webkitpy, I would encourage you to
> consider putting all of the python code into a subfolder of webkitpy.
> For example--
>
> webkitpy/
>    runlayout/ (or whatever you prefer)
>        layout/
>        test_types/
>        <top-level drivers>
>
> The reason is that webkitpy already contains ~35 python files, so the
> top-level drivers may get lost within an already long list -- hiding
> that they work together.
>
> The style package follows a similar model with an end-user executable
> in WebKitTools/Scripts/, and all of the supporting code in its own
> webkitpy subfolder called webkitpy/style.
>
> --Chris
>
>
> On Wed, Jan 20, 2010 at 11:07 PM, Dirk Pranke <dpranke@chromium.org> wrote:
>> The high-level executables are .bat and .sh scripts that wrap the
>> python scripts; they stay in webkit/tools/layout_tests (and will move
>> to WebKitTools/Scripts). *All* of the python code will move to
>> Scripts/webkitpy ; there are a few top-level drivers, and then most of
>> the code is in two subfolders (test_types and layout_package). You're
>> write that the layout_package name is redundant. I can change that
>> when I upload the webkit patch. I can also move the test_types dir
>> into the layout dir if we'd prefer that approach.
>>
>> All of the executables are end-user callable.
>>
>> -- Dirk
>>
>> On Wed, Jan 20, 2010 at 8:46 PM, Chris Jerdonek <cjerdonek@webkit.org> wrote:
>>> Thanks for the forward, Eric.
>>>
>>> Just a couple high-level comments:
>>>
>>> I can't totally tell from the patch how the supporting files (e.g. the
>>> non-executables) will sit inside webkit.py.  Can/will they all be in a
>>> single subfolder of webkitpy?  In terms of sub-folder naming,
>>> "*_package" is redundant, so a subfolder name like "layout/" is
>>> probably better than something like "layout_package".
>>>
>>> Finally, I don't know how many executables there are and what they are
>>> for, but if some of them are not meant to be called by the end-user
>>> (e.g. they are meant to be called internally by the end-user scripts),
>>> then it might be preferable to hide those away somewhere inside
>>> webkitpy as well.  Just a thought.
>>>
>>> Thanks,
>>> --Chris
>>>
>>>
>>> On Wed, Jan 20, 2010 at 8:13 PM, Eric Seidel <eseidel@chromium.org> wrote:
>>>> I think all the executables would want to stay at the root level, no?
>>>>
>>>> like rebaseline.py and run_chromium_webkit_tests.py?
>>>>
>>>> Also, the webkittens will probably want run- instead of run_ for all
>>>> these, but we can always change at a later date.
>>>>
>>>> looks OK to me.  Again, not sure you want to move the executables, but
>>>> otherwise OK.
>>>>
>>>> webkitpy is supposed to hide away all the python support junk.
>>>> everything directly inside WebKitTools/Scripts is executable.  There
>>>> are a few perl modules left there which are soon to be moved to
>>>> webkitperl.
>>>>
>>>> -eric
>>>>
>>>> On Wed, Jan 20, 2010 at 8:03 PM,  <dpranke@chromium.org> wrote:
>>>>> Reviewers: Eric Seidel (Google),
>>>>>
>>>>> Message:
>>>>> take a look and make sure this looks kosher. The try bots seem to be
getting
>>>
>>
>