[Telemetry] Implement automatic installation of optional openCV dependecy.

[Telemetry] Implement automatic installation of optional openCV dependecy.

BUG=424345

[mthiesse, 2014-10-16 21:26:43]

mthiesse@chromium.org changed reviewers:
+ flackr@chromium.org, tonyg@chromium.org

[mthiesse, 2014-10-16 21:26:44]

Hi Rob, Tony.

PTAL. Not sure if the test I wrote is going to work. Don't know what happens
when I request root on the bots :P

[mthiesse, 2014-10-16 21:30:12]

On 2014/10/16 21:26:44, mthiesse wrote:
> Hi Rob, Tony.
> 
> PTAL. Not sure if the test I wrote is going to work. Don't know what happens
> when I request root on the bots :P

Also I keep forgetting that I can't start trybots myself...

[tonyg, 2014-10-17 01:15:04]

tonyg@chromium.org changed reviewers:
+ dtu@chromium.org

[tonyg, 2014-10-17 01:15:11]

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/bin/linu...
File tools/telemetry/bin/linux/x86_64/cv2.so.sha1 (right):

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/bin/linu...
tools/telemetry/bin/linux/x86_64/cv2.so.sha1:1:
cb83eb5c434ceffbb21be489b4fd127af2f43c78
Do we have a plan for non-linux platforms?

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
File tools/telemetry/telemetry/core/platform/linux_platform_backend.py (right):

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
tools/telemetry/telemetry/core/platform/linux_platform_backend.py:174:
subprocess.check_call(['sudo', 'apt-get', '-q', 'install',
I wonder whether we should instead add these packages to
build/install-build-deps.sh. That'd make the bots and chromium developers happy.
But the question is on standalone telemetry. Dave, we've talked before about
having it pull third-party deps into the zip. Is that feasible?

[mthiesse, 2014-10-17 14:47:20]

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/bin/linu...
File tools/telemetry/bin/linux/x86_64/cv2.so.sha1 (right):

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/bin/linu...
tools/telemetry/bin/linux/x86_64/cv2.so.sha1:1:
cb83eb5c434ceffbb21be489b4fd127af2f43c78
On 2014/10/17 01:15:11, tonyg wrote:
> Do we have a plan for non-linux platforms?

It's definitely possible to add support for mac and windows, are there other
platforms we need to support?

[tonyg, 2014-10-17 16:24:12]

On 2014/10/17 14:47:20, mthiesse wrote:
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/bin/linu...
> File tools/telemetry/bin/linux/x86_64/cv2.so.sha1 (right):
> 
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/bin/linu...
> tools/telemetry/bin/linux/x86_64/cv2.so.sha1:1:
> cb83eb5c434ceffbb21be489b4fd127af2f43c78
> On 2014/10/17 01:15:11, tonyg wrote:
> > Do we have a plan for non-linux platforms?
> 
> It's definitely possible to add support for mac and windows, are there other
> platforms we need to support?

That should cover it :) Android, CrOS, and iOS would just need to run this on
the host platform.

[flackr, 2014-10-21 16:08:16]

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
File tools/telemetry/telemetry/core/platform/linux_platform_backend.py (right):

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
tools/telemetry/telemetry/core/platform/linux_platform_backend.py:102:
self._InstallOpenCV()
As with InstallApplication, raise NotImplementedError if we ask for a module we
don't know how to install.

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
tools/telemetry/telemetry/core/platform/linux_platform_backend.py:174:
subprocess.check_call(['sudo', 'apt-get', '-q', 'install',
On 2014/10/17 01:15:11, tonyg wrote:
> I wonder whether we should instead add these packages to
> build/install-build-deps.sh. That'd make the bots and chromium developers
happy.
> But the question is on standalone telemetry. Dave, we've talked before about
> having it pull third-party deps into the zip. Is that feasible?

We could statically link the external lib dependencies for our opencv, if the
resulting library isn't exorbitantly large. The remaining external deps may be
few enough to package and download as well.

Also, do we need the "-dev" versions of all of these packages? If we're fetching
a prebuilt opencv, not building it, don't we just need the libs which are linked
against.

[mthiesse, 2014-10-21 20:01:10]

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
File tools/telemetry/telemetry/core/platform/linux_platform_backend.py (right):

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
tools/telemetry/telemetry/core/platform/linux_platform_backend.py:102:
self._InstallOpenCV()
On 2014/10/21 16:08:16, flackr wrote:
> As with InstallApplication, raise NotImplementedError if we ask for a module
we
> don't know how to install.

Done.

https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
tools/telemetry/telemetry/core/platform/linux_platform_backend.py:174:
subprocess.check_call(['sudo', 'apt-get', '-q', 'install',
On 2014/10/21 16:08:16, flackr wrote:
> On 2014/10/17 01:15:11, tonyg wrote:
> > I wonder whether we should instead add these packages to
> > build/install-build-deps.sh. That'd make the bots and chromium developers
> happy.
> > But the question is on standalone telemetry. Dave, we've talked before about
> > having it pull third-party deps into the zip. Is that feasible?
> 
> We could statically link the external lib dependencies for our opencv, if the
> resulting library isn't exorbitantly large. The remaining external deps may be
> few enough to package and download as well.
> 
> Also, do we need the "-dev" versions of all of these packages? If we're
fetching
> a prebuilt opencv, not building it, don't we just need the libs which are
linked
> against.

Yes, it's just difficult to figure out exactly what those are. Is there a way to
figure out which packages are used by a binary?

At any rate, after much experimentation, it seems that the 3 packages
libav-tools libswscale2 python-numpy install everything we need.

[flackr, 2014-10-21 20:43:17]

On 2014/10/21 20:01:10, mthiesse wrote:
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
> File tools/telemetry/telemetry/core/platform/linux_platform_backend.py
(right):
> 
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
> tools/telemetry/telemetry/core/platform/linux_platform_backend.py:102:
> self._InstallOpenCV()
> On 2014/10/21 16:08:16, flackr wrote:
> > As with InstallApplication, raise NotImplementedError if we ask for a module
> we
> > don't know how to install.
> 
> Done.
> 
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
> tools/telemetry/telemetry/core/platform/linux_platform_backend.py:174:
> subprocess.check_call(['sudo', 'apt-get', '-q', 'install',
> On 2014/10/21 16:08:16, flackr wrote:
> > On 2014/10/17 01:15:11, tonyg wrote:
> > > I wonder whether we should instead add these packages to
> > > build/install-build-deps.sh. That'd make the bots and chromium developers
> > happy.
> > > But the question is on standalone telemetry. Dave, we've talked before
about
> > > having it pull third-party deps into the zip. Is that feasible?
> > 
> > We could statically link the external lib dependencies for our opencv, if
the
> > resulting library isn't exorbitantly large. The remaining external deps may
be
> > few enough to package and download as well.
> > 
> > Also, do we need the "-dev" versions of all of these packages? If we're
> fetching
> > a prebuilt opencv, not building it, don't we just need the libs which are
> linked
> > against.
> 
> Yes, it's just difficult to figure out exactly what those are. Is there a way
to
> figure out which packages are used by a binary?

Yes :-),
ldd binary

Then you can use apt-file (if it's not obvious from the name) to find which
package provides that file.

> 
> At any rate, after much experimentation, it seems that the 3 packages
> libav-tools libswscale2 python-numpy install everything we need.

[mthiesse, 2014-10-22 15:51:46]

On 2014/10/21 20:43:17, flackr wrote:
> On 2014/10/21 20:01:10, mthiesse wrote:
> >
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
> > File tools/telemetry/telemetry/core/platform/linux_platform_backend.py
> (right):
> > 
> >
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
> > tools/telemetry/telemetry/core/platform/linux_platform_backend.py:102:
> > self._InstallOpenCV()
> > On 2014/10/21 16:08:16, flackr wrote:
> > > As with InstallApplication, raise NotImplementedError if we ask for a
module
> > we
> > > don't know how to install.
> > 
> > Done.
> > 
> >
>
https://codereview.chromium.org/660893002/diff/20001/tools/telemetry/telemetr...
> > tools/telemetry/telemetry/core/platform/linux_platform_backend.py:174:
> > subprocess.check_call(['sudo', 'apt-get', '-q', 'install',
> > On 2014/10/21 16:08:16, flackr wrote:
> > > On 2014/10/17 01:15:11, tonyg wrote:
> > > > I wonder whether we should instead add these packages to
> > > > build/install-build-deps.sh. That'd make the bots and chromium
developers
> > > happy.
> > > > But the question is on standalone telemetry. Dave, we've talked before
> about
> > > > having it pull third-party deps into the zip. Is that feasible?
> > > 
> > > We could statically link the external lib dependencies for our opencv, if
> the
> > > resulting library isn't exorbitantly large. The remaining external deps
may
> be
> > > few enough to package and download as well.
> > > 
> > > Also, do we need the "-dev" versions of all of these packages? If we're
> > fetching
> > > a prebuilt opencv, not building it, don't we just need the libs which are
> > linked
> > > against.
> > 
> > Yes, it's just difficult to figure out exactly what those are. Is there a
way
> to
> > figure out which packages are used by a binary?
> 
> Yes :-),
> ldd binary
> 
> Then you can use apt-file (if it's not obvious from the name) to find which
> package provides that file.
> 
> > 
> > At any rate, after much experimentation, it seems that the 3 packages
> > libav-tools libswscale2 python-numpy install everything we need.

haha well ldd revealed that I'm very wrong, there are many more dependencies
than that. Trying to figure out now which should already be installed, and which
we would have to install.

[mthiesse, 2014-10-23 15:45:24]

Patchset #4 (id:60001) has been deleted

[mthiesse, 2014-10-23 15:57:55]

Well, I'm not sure how I missed it before, but there's a python-opencv package
for ubuntu, so I've switched to using that.

I've added the package to install-build-deps, so that should make the bots
happy, and I expect that the sudo apt-get install python-opencv call will work
well enough for any users of the standalone telemetry library.

PTAL

[tonyg, 2014-10-23 16:56:34]

https://codereview.chromium.org/660893002/diff/100001/build/install-build-dep...
File build/install-build-deps.sh (right):

https://codereview.chromium.org/660893002/diff/100001/build/install-build-dep...
build/install-build-deps.sh:159: telemetry_list="python-opencv"
Please land this change first. I believe we'll have to request Chrome-Labs to
explicitly run this script on the bots before we make the Telemetry change that
depends on it. Also, we (Dave?) will need to figure out a way to pull
third-party deps into the standalone zip.

[mthiesse, 2014-10-28 14:46:43]

On 2014/10/23 16:56:34, tonyg wrote:
>
https://codereview.chromium.org/660893002/diff/100001/build/install-build-dep...
> File build/install-build-deps.sh (right):
> 
>
https://codereview.chromium.org/660893002/diff/100001/build/install-build-dep...
> build/install-build-deps.sh:159: telemetry_list="python-opencv"
> Please land this change first. I believe we'll have to request Chrome-Labs to
> explicitly run this script on the bots before we make the Telemetry change
that
> depends on it. Also, we (Dave?) will need to figure out a way to pull
> third-party deps into the standalone zip.

So if the plan is to add the dependencies to the standalone zip, then there
really isn't any purpose for this CL. Anybody on linux should have opencv
already installed before reaching this code.

Do you agree? Should I just close this?

[tonyg, 2014-10-28 19:11:04]

On 2014/10/28 14:46:43, mthiesse wrote:
> On 2014/10/23 16:56:34, tonyg wrote:
> >
>
https://codereview.chromium.org/660893002/diff/100001/build/install-build-dep...
> > File build/install-build-deps.sh (right):
> > 
> >
>
https://codereview.chromium.org/660893002/diff/100001/build/install-build-dep...
> > build/install-build-deps.sh:159: telemetry_list="python-opencv"
> > Please land this change first. I believe we'll have to request Chrome-Labs
to
> > explicitly run this script on the bots before we make the Telemetry change
> that
> > depends on it. Also, we (Dave?) will need to figure out a way to pull
> > third-party deps into the standalone zip.
> 
> So if the plan is to add the dependencies to the standalone zip, then there
> really isn't any purpose for this CL. Anybody on linux should have opencv
> already installed before reaching this code.
> 
> Do you agree? Should I just close this?

dtu@ could you please confirm?