Chromium Code Reviews
Help | Chromium Project | Gerrit Changes | Sign in
(5)

Issue 2644783003: Regenerate MANIFEST.json when WPT tests are run (Closed)

Created:
2 years, 4 months ago by jeffcarp
Modified:
2 years, 2 months ago
Reviewers:
qyearsley, foolip
CC:
blink-reviews, chromium-reviews, Dirk Pranke
Target Ref:
refs/pending/heads/master
Project:
chromium
Visibility:
Public.

Description

Regenerate MANIFEST.json when WPT tests are run BUG=666957 R=qyearsley@chromium.org

Patch Set 1 #

Total comments: 4

Patch Set 2 : Address CL feedback #

Patch Set 3 : Warn if MANIFEST.json was updated by layout test run #

Total comments: 3

Patch Set 4 : Unit test MANIFEST.json regeneration warning #

Patch Set 5 : Redo patch to try to alleviate rebase problems #

Total comments: 4

Patch Set 6 : Simplify MANIFEST.json regeneration, just say the file might have changed #

Total comments: 1

Patch Set 7 : Fix multiline tuple->string #

Patch Set 8 : Rebase #

Patch Set 9 : Fix error introduced in rebase #

Patch Set 10 : Explicitly prepend python.exe on Windows (trying to debug Win tryjob) #

Patch Set 11 : Move MANIFEST.json regeneration up before sharding #

Patch Set 12 : Remove unit test for previous MANIFEST.json regeneration behavior #

Total comments: 2

Patch Set 13 : wip - work on what we discussed in the bug #

Total comments: 7

Patch Set 14 : Regenerate MANIFEST.json from template in run-webkit-tests #

Total comments: 19
Unified diffs Side-by-side diffs Delta from patch set Stats (+68 lines, -4 lines) Patch
A third_party/WebKit/LayoutTests/external/.gitignore View 1 2 3 4 5 6 7 8 9 10 11 12 13 1 chunk +1 line, -0 lines 0 comments Download
A + third_party/WebKit/LayoutTests/external/WPT_BASE_MANIFEST.json View 1 2 3 4 5 6 7 8 9 10 11 12 13 0 chunks +-1 lines, --1 lines 0 comments Download
M third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 3 chunks +11 lines, -0 lines 2 comments Download
M third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 5 chunks +23 lines, -0 lines 9 comments Download
M third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager_unittest.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 1 chunk +29 lines, -0 lines 8 comments Download
M third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 5 chunks +5 lines, -5 lines 0 comments Download

Messages

Total messages: 58 (8 generated)
jeffcarp
I haven't tested this out with a new test yet. Let me know if this ...
2 years, 4 months ago (2017-01-19 23:41:39 UTC) #1
qyearsley
Yeah, I think this is what we want :-D I was kind of concerned about ...
2 years, 4 months ago (2017-01-19 23:56:20 UTC) #2
jeffcarp
On 2017/01/19 at 23:56:20, qyearsley wrote: > Yeah, I think this is what we want ...
2 years, 3 months ago (2017-01-20 22:55:13 UTC) #3
qyearsley
On 2017/01/20 at 22:55:13, jeffcarp wrote: > On 2017/01/19 at 23:56:20, qyearsley wrote: > > ...
2 years, 3 months ago (2017-01-20 23:07:09 UTC) #4
jeffcarp
On 2017/01/20 at 23:07:09, qyearsley wrote: > On 2017/01/20 at 22:55:13, jeffcarp wrote: > > ...
2 years, 3 months ago (2017-01-23 21:21:42 UTC) #5
jeffcarp
https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode750 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:750: _log.warning('MANIFEST.json has been updated to reflect changes in external/wpt!') ...
2 years, 3 months ago (2017-01-23 22:27:51 UTC) #6
qyearsley
On 2017/01/23 at 22:27:51, jeffcarp wrote: > https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py > File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): > > https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode750 ...
2 years, 3 months ago (2017-01-24 01:56:03 UTC) #7
jeffcarp
On 2017/01/24 at 01:56:03, qyearsley wrote: > On 2017/01/23 at 22:27:51, jeffcarp wrote: > > ...
2 years, 3 months ago (2017-01-24 04:03:36 UTC) #8
jeffcarp
https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode750 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:750: _log.warning('MANIFEST.json has been updated to reflect changes in external/wpt!') ...
2 years, 3 months ago (2017-01-24 20:32:39 UTC) #9
jeffcarp
https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode750 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:750: _log.warning('MANIFEST.json has been updated to reflect changes in external/wpt!') ...
2 years, 3 months ago (2017-01-24 20:42:59 UTC) #10
jeffcarp
On 2017/01/24 at 20:42:59, jeffcarp wrote: > https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py > File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): > > https://codereview.chromium.org/2644783003/diff/40001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode750 ...
2 years, 3 months ago (2017-01-26 18:54:04 UTC) #11
qyearsley
On 2017/01/26 at 18:54:04, jeffcarp wrote: > On 2017/01/24 at 20:42:59, jeffcarp wrote: > > ...
2 years, 3 months ago (2017-01-26 19:00:44 UTC) #12
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.org/2644783003/60001
2 years, 3 months ago (2017-01-26 20:25:37 UTC) #14
commit-bot: I haz the power
Try jobs failed on following builders: ios-device on master.tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device/builds/143047) ios-device-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED, ...
2 years, 3 months ago (2017-01-26 20:27:49 UTC) #16
jeffcarp
https://codereview.chromium.org/2644783003/diff/80001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py (right): https://codereview.chromium.org/2644783003/diff/80001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py#newcode336 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py:336: port._generate_manifest = mock_generate_manifest # pylint: disable=W0212 After rebasing I ...
2 years, 3 months ago (2017-01-26 22:26:39 UTC) #17
qyearsley
https://codereview.chromium.org/2644783003/diff/80001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py (right): https://codereview.chromium.org/2644783003/diff/80001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py#newcode298 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py:298: # the result of running the update manifest command. ...
2 years, 3 months ago (2017-01-26 23:08:52 UTC) #18
jeffcarp
On 2017/01/26 at 23:08:52, qyearsley wrote: > https://codereview.chromium.org/2644783003/diff/80001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py > File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py (right): > > https://codereview.chromium.org/2644783003/diff/80001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py#newcode298 ...
2 years, 3 months ago (2017-01-27 23:03:56 UTC) #19
qyearsley
https://codereview.chromium.org/2644783003/diff/100001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): https://codereview.chromium.org/2644783003/diff/100001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode769 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:769: 'with your changes. See https://crbug.com/666957.')) Can you call log ...
2 years, 3 months ago (2017-01-27 23:24:49 UTC) #20
jeffcarp
On 2017/01/27 at 23:24:49, qyearsley wrote: > https://codereview.chromium.org/2644783003/diff/100001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py > File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): > > https://codereview.chromium.org/2644783003/diff/100001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode769 ...
2 years, 3 months ago (2017-01-27 23:50:41 UTC) #21
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.org/2644783003/120001
2 years, 3 months ago (2017-01-27 23:51:31 UTC) #24
commit-bot: I haz the power
Try jobs failed on following builders: win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/372805)
2 years, 3 months ago (2017-01-28 00:38:15 UTC) #26
jeffcarp
On 2017/01/28 at 00:38:15, commit-bot wrote: > Try jobs failed on following builders: > win_chromium_rel_ng ...
2 years, 3 months ago (2017-01-30 03:41:34 UTC) #27
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.org/2644783003/120001
2 years, 3 months ago (2017-01-30 03:41:49 UTC) #29
commit-bot: I haz the power
Try jobs failed on following builders: win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/373067)
2 years, 3 months ago (2017-01-30 04:25:25 UTC) #31
qyearsley
On 2017/01/30 at 04:25:25, commit-bot wrote: > Try jobs failed on following builders: > win_chromium_rel_ng ...
2 years, 3 months ago (2017-02-01 17:58:32 UTC) #32
qyearsley
On 2017/02/01 at 17:58:32, qyearsley wrote: > On 2017/01/30 at 04:25:25, commit-bot wrote: > > ...
2 years, 3 months ago (2017-02-01 18:23:39 UTC) #33
jeffcarp
On 2017/02/01 at 18:23:39, qyearsley wrote: > On 2017/02/01 at 17:58:32, qyearsley wrote: > > ...
2 years, 3 months ago (2017-02-10 02:30:26 UTC) #34
qyearsley
On 2017/02/10 at 02:30:26, jeffcarp wrote: > On 2017/02/01 at 18:23:39, qyearsley wrote: > > ...
2 years, 3 months ago (2017-02-10 17:29:49 UTC) #35
jeffcarp
On 2017/02/10 at 17:29:49, qyearsley wrote: > On 2017/02/10 at 02:30:26, jeffcarp wrote: > > ...
2 years, 3 months ago (2017-02-10 18:17:10 UTC) #36
jeffcarp
On 2017/02/10 at 18:17:10, jeffcarp wrote: > On 2017/02/10 at 17:29:49, qyearsley wrote: > > ...
2 years, 3 months ago (2017-02-10 18:19:03 UTC) #37
jeffcarp
On 2017/02/10 at 18:19:03, jeffcarp wrote: > On 2017/02/10 at 18:17:10, jeffcarp wrote: > > ...
2 years, 3 months ago (2017-02-12 22:39:16 UTC) #38
foolip
Now just missing a test? If you remove MANIFEST.json, this is the LayoutTests/external/.gitignore I tried: ...
2 years, 3 months ago (2017-02-16 07:20:26 UTC) #40
jeffcarp
@qyearsley @foolip I just updated this CL with a new in-progress version we talked about. ...
2 years, 2 months ago (2017-02-21 19:32:11 UTC) #41
jeffcarp
https://codereview.chromium.org/2644783003/diff/240001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py (right): https://codereview.chromium.org/2644783003/diff/240001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py#newcode772 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:772: return None This isn't part of my CL, not ...
2 years, 2 months ago (2017-02-21 19:33:16 UTC) #42
qyearsley
https://codereview.chromium.org/2644783003/diff/240001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py (right): https://codereview.chromium.org/2644783003/diff/240001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#newcode105 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:105: # but to know if we need to generate ...
2 years, 2 months ago (2017-02-21 22:47:57 UTC) #43
foolip
WPT_BASE_MANIFEST.json is too big to show, does it now include itself? And if it doesn't ...
2 years, 2 months ago (2017-02-23 02:51:30 UTC) #44
jeffcarp
This CL is now ready for review after updating based on what we talked about. ...
2 years, 2 months ago (2017-02-28 02:44:10 UTC) #45
jeffcarp
On 2017/02/28 at 02:44:10, jeffcarp wrote: > This CL is now ready for review after ...
2 years, 2 months ago (2017-02-28 02:44:57 UTC) #46
jeffcarp
On 2017/02/28 at 02:44:57, jeffcarp wrote: > On 2017/02/28 at 02:44:10, jeffcarp wrote: > > ...
2 years, 2 months ago (2017-02-28 02:51:00 UTC) #47
foolip
This all LGTM except where not understood, leaving proper and final review to qyearsley@. Improving ...
2 years, 2 months ago (2017-02-28 04:49:17 UTC) #48
qyearsley
https://codereview.chromium.org/2644783003/diff/250001/third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py File third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py (right): https://codereview.chromium.org/2644783003/diff/250001/third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py#newcode86 third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py:86: 'LayoutTests', 'external', 'WPT_BASE_MANIFEST.json') On 2017/02/28 at 04:49:17, foolip wrote: ...
2 years, 2 months ago (2017-02-28 18:54:45 UTC) #49
jeffcarp
On 2017/02/28 at 18:54:45, qyearsley wrote: > https://codereview.chromium.org/2644783003/diff/250001/third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py > File third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py (right): > > https://codereview.chromium.org/2644783003/diff/250001/third_party/WebKit/Tools/Scripts/webkitpy/common/host_mock.py#newcode86 ...
2 years, 2 months ago (2017-02-28 21:18:46 UTC) #50
jeffcarp
On 2017/02/28 at 21:18:46, jeffcarp wrote: > On 2017/02/28 at 18:54:45, qyearsley wrote: > > ...
2 years, 2 months ago (2017-02-28 21:46:06 UTC) #51
qyearsley
On 2017/02/28 at 21:46:06, jeffcarp wrote: > On 2017/02/28 at 21:18:46, jeffcarp wrote: > > ...
2 years, 2 months ago (2017-02-28 21:48:13 UTC) #52
jeffcarp
Publishing my drafts but any subsequent review should be on Gerrit: https://chromium-review.googlesource.com/c/447959 https://codereview.chromium.org/2644783003/diff/250001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py ...
2 years, 2 months ago (2017-02-28 21:57:39 UTC) #53
qyearsley
https://codereview.chromium.org/2644783003/diff/250001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py (right): https://codereview.chromium.org/2644783003/diff/250001/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py#newcode556 third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:556: def _ensure_manifest(self): On 2017/02/28 at 21:57:39, jeffcarp wrote: > ...
2 years, 2 months ago (2017-02-28 22:04:31 UTC) #54
awdf
drive-by comment: Please could you also update the docs at https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md ? I understand with ...
2 years, 2 months ago (2017-03-02 15:56:59 UTC) #55
awdf
On 2017/03/02 15:56:59, awdf wrote: > drive-by comment: Please could you also update the docs ...
2 years, 2 months ago (2017-03-02 16:00:17 UTC) #56
awdf
On 2017/03/02 16:00:17, awdf wrote: > On 2017/03/02 15:56:59, awdf wrote: > > drive-by comment: ...
2 years, 2 months ago (2017-03-02 16:21:45 UTC) #57
jeffcarp
2 years, 2 months ago (2017-03-02 19:30:36 UTC) #58
Message was sent while issue was closed.
On 2017/03/02 at 16:21:45, awdf wrote:
> On 2017/03/02 16:00:17, awdf wrote:
> > On 2017/03/02 15:56:59, awdf wrote:
> > > drive-by comment: Please could you also update the docs at
> > >
> >
https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_plat...
> > > ? 
> > > 
> > > I understand with this CL the command to generate MANIFEST won't be
required,
> > > which will be great!
> > > 
> > > Right now it's additionally confusing because the command in the docs
looks
> > like
> > > it should be run from third_party/WebKit/, (as it starts with
> > > Tools/Scripts/...), but actually needs to be run from src/ to avoid a 
> > > "KeyError: './MANIFEST.json'".
> > 
> > Or rather, it *looks* like it worked from src/, but actually it doesn't! And
I
> > still get a KeyError from third_party/WebKit :(
> > 
> > I'll try a rebase, sorry for the distraction.
> 
> ..deleting the MANIFEST.json and rerunning the script (from
third_party/WebKit/) made it work, for the record.
> 
> Looking forward to when this patch lands and I don't need to worry about
running a script for it to pick up on my tests! :)

Sorry you had to run into that! There's a bug for MANIFEST.json regeneration
being broken on ToT:
https://bugs.chromium.org/p/chromium/issues/detail?id=697207

The new review is here https://chromium-review.googlesource.com/c/447959, going
to try to land it today.

Powered by Google App Engine
This is Rietveld 408576698