Add dedicated cros bootstrap_deps.

Add dedicated cros bootstrap_deps.

This will allow autotest via chromeos-base/chromeos-chrome to bootstrap
telemetry perf and gpu tests.

Also fix gpu_tests/bootstrap_deps as currently the paths are wrong.

BUG=chromium:341333
TEST=Ran "./run_measurement --print-bootstrap-deps-cros" and checked output.

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=252527

[ilja, 2014-02-15 02:37:51]

Please take a look. The idea is for CrOS to pull all of telemetry performance
and gpu tests. But I have some problems with the gpu bootstrapping as it seems
outdated.

https://codereview.chromium.org/166483010/diff/60001/content/test/gpu/gpu_tes...
File content/test/gpu/gpu_tests/bootstrap_deps (left):

https://codereview.chromium.org/166483010/diff/60001/content/test/gpu/gpu_tes...
content/test/gpu/gpu_tests/bootstrap_deps:15:
"src/third_party/webgl_conformance":
I am confused how this is supposed to work as I don't have this directory. So my
best guess is to pull the whole webgl directory which contains
third_party/webgl/src/conformance-suites/1.0.*

[tonyg, 2014-02-16 01:23:13]

dtu just landed a dependencies script which I think should do this and allow us
to remove those hardcoded dependency files. We should figure out if that can be
shared and at minimum he should review this.

[achuithb, 2014-02-18 19:08:22]

I'll defer to sbasi@ and dtu@.

[sbasi1, 2014-02-19 21:49:07]

On 2014/02/18 19:08:22, achuith.bhandarkar wrote:
> I'll defer to sbasi@ and dtu@.

Looking at tonyg's comment we should def have DTU take a look. But if you're
trying to update the files pulled into the telemetry tarball for autotest you
need changes in the chromeos-chrome ebuild and need to verify that the new
deps-telemetry tarball has all the old files plus what you added here. I believe
the change will be a switch from ./run_measurement --print-bootstrap-deps to
./run_measurement --print-bootstrap-deps-cros

[sbasi1, 2014-02-19 21:49:17]

https://codereview.chromium.org/166483010/diff/60001/content/test/gpu/gpu_tes...
File content/test/gpu/gpu_tests/bootstrap_deps (left):

https://codereview.chromium.org/166483010/diff/60001/content/test/gpu/gpu_tes...
content/test/gpu/gpu_tests/bootstrap_deps:15:
"src/third_party/webgl_conformance":
This directory could be in the internal sources if you don't have a full
checkout?
On 2014/02/15 02:37:51, ilja wrote:
> I am confused how this is supposed to work as I don't have this directory. So
my
> best guess is to pull the whole webgl directory which contains
> third_party/webgl/src/conformance-suites/1.0.*

[dtu, 2014-02-19 23:20:54]

On 2014/02/16 01:23:13, tonyg wrote:
> dtu just landed a dependencies script which I think should do this and allow
us
> to remove those hardcoded dependency files. We should figure out if that can
be
> shared and at minimum he should review this.

Yes, there's a script that can find the dependencies! There are still some
dependencies that can't be found automatically, though (in particular JS deps,
and profile data), so the script still reads and uses the hardcoded
bootstrap_deps.

There's some followup work that needs to be done to make it work in this
scenario, though you can still try it to see if it fits your needs.
src$ tools/telemetry_tools/find_dependencies tools/perf/run_benchmark
content/test/gpu/run_gpu_test

[dtu, 2014-02-19 23:21:43]

https://chromiumcodereview.appspot.com/166483010/diff/60001/content/test/gpu/...
File content/test/gpu/gpu_tests/bootstrap_deps (left):

https://chromiumcodereview.appspot.com/166483010/diff/60001/content/test/gpu/...
content/test/gpu/gpu_tests/bootstrap_deps:15:
"src/third_party/webgl_conformance":
On 2014/02/19 21:49:18, sbasi1 wrote:
> This directory could be in the internal sources if you don't have a full
> checkout?
> On 2014/02/15 02:37:51, ilja wrote:
> > I am confused how this is supposed to work as I don't have this directory.
So
> my
> > best guess is to pull the whole webgl directory which contains
> > third_party/webgl/src/conformance-suites/1.0.*
> 

This directory was removed, the new dependency is
src/third_party/webgl/src/sdk/tests. In all likelihood, this didn't work, and
nobody has tested it to make sure it works.

[ilja, 2014-02-20 00:29:32]

On 2014/02/19 23:20:54, dtu wrote:
> On 2014/02/16 01:23:13, tonyg wrote:
> > dtu just landed a dependencies script which I think should do this and allow
> us
> > to remove those hardcoded dependency files. We should figure out if that can
> be
> > shared and at minimum he should review this.
> 
> Yes, there's a script that can find the dependencies! There are still some
> dependencies that can't be found automatically, though (in particular JS deps,
> and profile data), so the script still reads and uses the hardcoded
> bootstrap_deps.
> 
> There's some followup work that needs to be done to make it work in this
> scenario, though you can still try it to see if it fits your needs.
> src$ tools/telemetry_tools/find_dependencies tools/perf/run_benchmark
> content/test/gpu/run_gpu_test

The output format of the script is quite different. It lists each file with full
path instead of relative directories. This means the chromeos-chrome*.ebuild
have to be changed quite a bit to use this output. Nothing impossible. But
running it as you showed me does not output anything in third_party/webgl/ which
means it is missing on the html test pages that I need to fetch.

[ilja, 2014-02-20 00:38:05]

https://codereview.chromium.org/166483010/diff/60001/content/test/gpu/gpu_tes...
File content/test/gpu/gpu_tests/bootstrap_deps (left):

https://codereview.chromium.org/166483010/diff/60001/content/test/gpu/gpu_tes...
content/test/gpu/gpu_tests/bootstrap_deps:15:
"src/third_party/webgl_conformance":
Now that you mention it, this is correct, I am changing it:
https://code.google.com/p/chromium/codesearch#chromium/src/content/test/gpu/g...

[dtu, 2014-02-20 01:33:59]

On 2014/02/20 00:29:32, ilja wrote:
> On 2014/02/19 23:20:54, dtu wrote:
> > On 2014/02/16 01:23:13, tonyg wrote:
> > > dtu just landed a dependencies script which I think should do this and
allow
> > us
> > > to remove those hardcoded dependency files. We should figure out if that
can
> > be
> > > shared and at minimum he should review this.
> > 
> > Yes, there's a script that can find the dependencies! There are still some
> > dependencies that can't be found automatically, though (in particular JS
deps,
> > and profile data), so the script still reads and uses the hardcoded
> > bootstrap_deps.
> > 
> > There's some followup work that needs to be done to make it work in this
> > scenario, though you can still try it to see if it fits your needs.
> > src$ tools/telemetry_tools/find_dependencies tools/perf/run_benchmark
> > content/test/gpu/run_gpu_test
> 
> The output format of the script is quite different. It lists each file with
full
> path instead of relative directories. This means the chromeos-chrome*.ebuild
> have to be changed quite a bit to use this output. Nothing impossible. But
> running it as you showed me does not output anything in third_party/webgl/
which
> means it is missing on the html test pages that I need to fetch.

Why do full paths vs relative directories make a difference?

You can pass the flag --include-page-set-data to include the data files like
those in third_party/webgl, but it's not quite working right at the moment. I
will have a patch up soon.

The primary advantage of having a script to generate the Python deps is that
they don't have to be manually changed any time there's a new dep. That's broken
the ChromeOS bots in the past.

[ilja, 2014-02-20 02:05:44]

On 2014/02/20 01:33:59, dtu wrote:
> On 2014/02/20 00:29:32, ilja wrote:
> > On 2014/02/19 23:20:54, dtu wrote:
> > > On 2014/02/16 01:23:13, tonyg wrote:
> > > > dtu just landed a dependencies script which I think should do this and
> allow
> > > us
> > > > to remove those hardcoded dependency files. We should figure out if that
> can
> > > be
> > > > shared and at minimum he should review this.
> > > 
> > > Yes, there's a script that can find the dependencies! There are still some
> > > dependencies that can't be found automatically, though (in particular JS
> deps,
> > > and profile data), so the script still reads and uses the hardcoded
> > > bootstrap_deps.
> > > 
> > > There's some followup work that needs to be done to make it work in this
> > > scenario, though you can still try it to see if it fits your needs.
> > > src$ tools/telemetry_tools/find_dependencies tools/perf/run_benchmark
> > > content/test/gpu/run_gpu_test
> > 
> > The output format of the script is quite different. It lists each file with
> full
> > path instead of relative directories. This means the chromeos-chrome*.ebuild
> > have to be changed quite a bit to use this output. Nothing impossible. But
> > running it as you showed me does not output anything in third_party/webgl/
> which
> > means it is missing on the html test pages that I need to fetch.
> 
> Why do full paths vs relative directories make a difference?

I will have to strip the root otherwise I can't construct a destination. The
chromeos-chrome.ebuild rsyncs from source directory to a cache directory and it
recursively copies from there to a final destination. It looks pretty
convoluted, but it seems to do it to avoid symlinks and it also excludes .git
and .svn directories. That said I expect a tool that lists every file
individually to be quite a bit slower as each file copy will spawn separate
processes. Probably not too bad.

> You can pass the flag --include-page-set-data to include the data files like
> those in third_party/webgl, but it's not quite working right at the moment. I
> will have a patch up soon.

Yes, it is not working. And this is blocking me (and getting rid of pyauto).

> The primary advantage of having a script to generate the Python deps is that
> they don't have to be manually changed any time there's a new dep. That's
broken
> the ChromeOS bots in the past.

Is anybody else except for ChromeOS going to use your script? Because it appears
that you are just moving the problem from breaking bootstrap_deps to the script
if nothing that can close the CQ uses it. (And ChromeOS can't yet, so who is
going to do run it in the Chromium waterfall?)

[ilja, 2014-02-20 03:17:35]

There is one more potential issue with using per-file vs. per-directory lists.
What I am doing here is merging perf deps and gpu deps. Both of which pull in
telemetry and hence it is listed twice. Obviously we don't want to copy twice.
The directory list is small and fast to clean. I am worried about eliminating
duplicate file entries (which isn't done right now by run_measurement),
especially if we have to call find_dependencies run_benchmark for each benchmark
ChromeOS wants to run (and I assume the list will get large one day). Right now
chromeos-chrome pulls in 7924 unique files which is not terrible but not trivial
either.

[dtu, 2014-02-20 22:48:17]

lgtm

[ilja, 2014-02-20 22:54:36]

Thanks! I filed crbug.com/345450 for tracking the work to switch to the
find_dependencies script.

[ilja, 2014-02-20 22:54:44]

The CQ bit was checked by ihf@chromium.org

[commit-bot: I haz the power, 2014-02-20 22:56:23]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/ihf@chromium.org/166483010/210001

[commit-bot: I haz the power, 2014-02-20 23:11:26]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-02-20 23:11:27]

Try jobs failed on following builders: linux_chromium_rel

[ilja, 2014-02-20 23:28:48]

zmo: may I get an OWNERS review for the path changes in
content/test/gpu/gpu_tests/bootstrap_deps?

[Zhenyao Mo, 2014-02-21 01:27:50]

https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
File content/test/gpu/gpu_tests/bootstrap_deps (right):

https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
content/test/gpu/gpu_tests/bootstrap_deps:16:
"https://src.chromium.org/chrome/trunk/src/third_party/webgl/src/sdk/tests",
One question: this path doesn't actually exist, as webgl isn't directly checked.
Rather, it's pulled through DEPS.

So, will this still work?

[ilja, 2014-02-21 01:31:13]

On 2014/02/21 01:27:50, Zhenyao Mo wrote:
>
https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
> File content/test/gpu/gpu_tests/bootstrap_deps (right):
> 
>
https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
> content/test/gpu/gpu_tests/bootstrap_deps:16:
> "https://src.chromium.org/chrome/trunk/src/third_party/webgl/src/sdk/tests",
> One question: this path doesn't actually exist, as webgl isn't directly
checked.
> Rather, it's pulled through DEPS.
> 
> So, will this still work?

At least on ChromeOS when we run the script to collect the dependencies we have
just compiled the browser and have a full local checkout from where we are
copying. When I tested I got the files I specified/wanted.

[Zhenyao Mo, 2014-02-21 01:38:03]

On 2014/02/21 01:31:13, ilja wrote:
> On 2014/02/21 01:27:50, Zhenyao Mo wrote:
> >
>
https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
> > File content/test/gpu/gpu_tests/bootstrap_deps (right):
> > 
> >
>
https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
> > content/test/gpu/gpu_tests/bootstrap_deps:16:
> > "https://src.chromium.org/chrome/trunk/src/third_party/webgl/src/sdk/tests",
> > One question: this path doesn't actually exist, as webgl isn't directly
> checked.
> > Rather, it's pulled through DEPS.
> > 
> > So, will this still work?
> 
> At least on ChromeOS when we run the script to collect the dependencies we
have
> just compiled the browser and have a full local checkout from where we are
> copying. When I tested I got the files I specified/wanted.

I understand that, but providing a wrong URL still doesn't sound right.  Maybe
just an empty URL and a comment to explain the situation? At least people who
read this file won't get confused.

[ilja, 2014-02-21 01:47:58]

PTAL.

https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
File content/test/gpu/gpu_tests/bootstrap_deps (right):

https://codereview.chromium.org/166483010/diff/210001/content/test/gpu/gpu_te...
content/test/gpu/gpu_tests/bootstrap_deps:16:
"https://src.chromium.org/chrome/trunk/src/third_party/webgl/src/sdk/tests",
On 2014/02/21 01:27:50, Zhenyao Mo wrote:
> One question: this path doesn't actually exist, as webgl isn't directly
checked.
> Rather, it's pulled through DEPS.
> 
> So, will this still work?

Done.

[Zhenyao Mo, 2014-02-21 01:49:28]

content/test/gpu lgtm

[ilja, 2014-02-21 01:52:21]

The CQ bit was checked by ihf@chromium.org

[commit-bot: I haz the power, 2014-02-21 01:54:45]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/ihf@chromium.org/166483010/400002

[commit-bot: I haz the power, 2014-02-21 02:49:42]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-02-21 02:49:43]

Retried try job too often on win for step(s) compile
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=win&number...

[ilja, 2014-02-21 03:22:48]

The CQ bit was checked by ihf@chromium.org

[commit-bot: I haz the power, 2014-02-21 03:23:19]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/ihf@chromium.org/166483010/400002

[commit-bot: I haz the power, 2014-02-21 09:47:47]

Change committed as 252527