[telemetry] bitmaptools as a standalone executable

[telemetry] bitmaptools as a standalone executable

The C++ binary implements simple per-pixel algorithms for SpeedIndex
computation. This allows us to achieve near real-time processing without
bringing external dependencies.

The bitmaptools binary needs to be built before it can be used.
The overhead of spawning a child process for each frame is about 3ms.

BUG=323813
TEST=telemetry bitmap_unittest

Original review: https://codereview.chromium.org/136793022
TBR=tonyg,maruel

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=246361

[commit-bot: I haz the power, 2014-01-18 07:53:23]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/szym@chromium.org/131563009/70001

[commit-bot: I haz the power, 2014-01-18 08:55:00]

Retried try job too often on ios_dbg_simulator for step(s) base_unittests,
compile, content_unittests, crypto_unittests, net_unittests, sql_unittests,
ui_unittests, url_unittests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=ios_dbg_si...

[commit-bot: I haz the power, 2014-01-18 16:05:07]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/szym@chromium.org/131563009/70001

[commit-bot: I haz the power, 2014-01-18 16:59:08]

Retried try job too often on ios_dbg_simulator for step(s) base_unittests,
compile, content_unittests, crypto_unittests, net_unittests, sql_unittests,
ui_unittests, url_unittests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=ios_dbg_si...

[commit-bot: I haz the power, 2014-01-18 16:59:54]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/szym@chromium.org/131563009/70001

[tonyg, 2014-01-18 17:08:52]

I patched this in, built bitmaptools and am getting this stack:

Traceback (most recent call last):
  _RunPage at tools/telemetry/telemetry/page/page_runner.py:453
    test.Run(finder_options, page, page_state.tab, results)
  Run at tools/telemetry/telemetry/page/page_test.py:266
    self._test_method(page, tab, results)
  _RunTest at tools/telemetry/telemetry/page/page_measurement.py:55
    self.MeasurePage(page, tab, results)
  MeasurePage at tools/perf/measurements/page_cycler.py:167
    self._speedindex_metric.Stop(page, tab)
  Stop at tools/perf/metrics/speedindex.py:48
    self._impl.Stop(tab)
  Stop at tools/perf/metrics/speedindex.py:162
    for time, bmp in tab.StopVideoCapture()]
  StopVideoCapture at tools/telemetry/telemetry/core/tab.py:198
    bmp, bitmap.WEB_PAGE_TEST_ORANGE)
  _FindHighlightBoundingBox at tools/telemetry/telemetry/core/tab.py:147
    content_box, pixel_count = bmp.GetBoundingBox(color, tolerance=8)
  GetBoundingBox at tools/telemetry/telemetry/core/bitmap.py:261
    return self._PrepareTools().BoundingBox(color, tolerance)
  _PrepareTools at tools/telemetry/telemetry/core/bitmap.py:149
    self._pixels)
  __init__ at tools/telemetry/telemetry/core/bitmap.py:70
    stderr=subprocess.PIPE)
  __init__ at /usr/lib/python2.7/subprocess.py:679
    errread, errwrite)
  _execute_child at /usr/lib/python2.7/subprocess.py:1249
    raise child_exception

Locals:
  args               :
['/usr/local/google/code/chromium/src/out/Release/bitmaptools']
  c2pread            : 31
  c2pwrite           : 32
  child_exception    : OSError(8, 'Exec format error')
  close_fds          : False
  creationflags      : 0
  cwd                : None
  data               : 'cexceptions\nOSError\np0\n(I8\nS\'Exec format
error\'\np1\ntp2\nRp3\n(dp4\nS\'child_traceback\'\np5\nS\'Traceback (most recent
call last):\\n  File "/usr/lib/python2.7/subprocess.py", line 1201, in
_execute_child\\n    os.execvp(executable, args)\\n  File
"/usr/lib/python2.7/os.py", line 344, in execvp\\n    _execvpe(file, args)\\n 
File "/usr/lib/python2.7/os.py", line 368, in _execvpe\\n    func(file,
*argrest)\\nOSError: [Errno 8] Exec format error\\n\'\np6\nsb.'
  env                : None
  errpipe_read       : 35
  errpipe_write      : 36
  errread            : 33
  errwrite           : 34
  executable         :
'/usr/local/google/code/chromium/src/out/Release/bitmaptools'
  fd                 : 33
  gc_was_enabled     : True
  p2cread            : 28
  p2cwrite           : 30
  preexec_fn         : None
  shell              : False
  startupinfo        : None
  universal_newlines : False

OSError: [Errno 8] Exec format error

[commit-bot: I haz the power, 2014-01-18 17:49:55]

Retried try job too often on ios_dbg_simulator for step(s) base_unittests,
compile, content_unittests, crypto_unittests, net_unittests, sql_unittests,
ui_unittests, url_unittests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=ios_dbg_si...

[szym, 2014-01-18 18:35:03]

On Sat, Jan 18, 2014 at 12:08 PM, <tonyg@chromium.org> wrote:

> OSError: [Errno 8] Exec format error


Make sure to rebuild bitmaptools. If problem persists run:
file out/Release/bitmaptools

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[commit-bot: I haz the power, 2014-01-18 20:35:25]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/szym@chromium.org/131563009/70001

[commit-bot: I haz the power, 2014-01-18 21:25:38]

Retried try job too often on ios_dbg_simulator for step(s) base_unittests,
compile, content_unittests, crypto_unittests, net_unittests, sql_unittests,
ui_unittests, url_unittests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=ios_dbg_si...

[commit-bot: I haz the power, 2014-01-18 21:46:05]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/szym@chromium.org/131563009/70001

[commit-bot: I haz the power, 2014-01-18 22:35:39]

Retried try job too often on ios_dbg_simulator for step(s) base_unittests,
compile, content_unittests, crypto_unittests, net_unittests, sql_unittests,
ui_unittests, url_unittests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=ios_dbg_si...

[tonyg, 2014-01-18 22:55:02]

On 2014/01/18 18:35:03, szym wrote:
> On Sat, Jan 18, 2014 at 12:08 PM, <mailto:tonyg@chromium.org> wrote:
> 
> > OSError: [Errno 8] Exec format error
> 
> 
> Make sure to rebuild bitmaptools. If problem persists run:
> file out/Release/bitmaptools
> 
> To unsubscribe from this group and stop receiving emails from it, send an
email
> to mailto:chromium-reviews+unsubscribe@chromium.org.

Yeah, it looks like it built it for the target. We need to build with the host
toolchain.

$ file out/Release/bitmaptools 
out/Release/bitmaptools: ELF 32-bit LSB executable, ARM, version 1 (SYSV),
dynamically linked (uses shared libs), not stripped

I think we should also add it as a dep to the android_tools target. That is what
the android perf bots build and what the Telemetry documentation instructs to
build.

[tonyg, 2014-01-18 23:01:00]

On Sat, Jan 18, 2014 at 2:55 PM,  <tonyg@chromium.org> wrote:
> On 2014/01/18 18:35:03, szym wrote:
>
>> On Sat, Jan 18, 2014 at 12:08 PM, <mailto:tonyg@chromium.org> wrote:
>
>
>> > OSError: [Errno 8] Exec format error
>
>
>
>> Make sure to rebuild bitmaptools. If problem persists run:
>> file out/Release/bitmaptools
>
>
>> To unsubscribe from this group and stop receiving emails from it, send an
>
> email
>>
>> to mailto:chromium-reviews+unsubscribe@chromium.org.
>
>
> Yeah, it looks like it built it for the target. We need to build with the
> host
> toolchain.

Adding 'toolsets': ['host'], to the bitmaptools target gets me up and
running locally.

>
> $ file out/Release/bitmaptools
> out/Release/bitmaptools: ELF 32-bit LSB executable, ARM, version 1 (SYSV),
> dynamically linked (uses shared libs), not stripped
>
> I think we should also add it as a dep to the android_tools target. That is
> what
> the android perf bots build and what the Telemetry documentation instructs
> to
> build.
>
> https://codereview.chromium.org/131563009/

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[szym, 2014-01-19 01:12:34]

I added #host to telemetry.gyp dependency in the All target in all.gyp and
all_android.gyp, but didn't realize there is the 'toolsets' variable.
On Jan 18, 2014 6:00 PM, "Tony Gentilcore" <tonyg@chromium.org> wrote:

> On Sat, Jan 18, 2014 at 2:55 PM,  <tonyg@chromium.org> wrote:
> > On 2014/01/18 18:35:03, szym wrote:
> >
> >> On Sat, Jan 18, 2014 at 12:08 PM, <mailto:tonyg@chromium.org> wrote:
> >
> >
> >> > OSError: [Errno 8] Exec format error
> >
> >
> >
> >> Make sure to rebuild bitmaptools. If problem persists run:
> >> file out/Release/bitmaptools
> >
> >
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >
> > email
> >>
> >> to mailto:chromium-reviews+unsubscribe@chromium.org.
> >
> >
> > Yeah, it looks like it built it for the target. We need to build with the
> > host
> > toolchain.
>
> Adding 'toolsets': ['host'], to the bitmaptools target gets me up and
> running locally.
>
> >
> > $ file out/Release/bitmaptools
> > out/Release/bitmaptools: ELF 32-bit LSB executable, ARM, version 1
> (SYSV),
> > dynamically linked (uses shared libs), not stripped
> >
> > I think we should also add it as a dep to the android_tools target. That
> is
> > what
> > the android perf bots build and what the Telemetry documentation
> instructs
> > to
> > build.
> >
> > https://codereview.chromium.org/131563009/
>

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[szym, 2014-01-22 14:00:55]

However, this was the data posted to the server:

xsrf_token: 7321b53cc85d82c96d24fbed458a66cc
add_as_reviewer: 0
message_only: 1
message: On 2014/01/19 01:12:34, szym wrote:
> >
> > Adding 'toolsets': ['host'], to the bitmaptools target gets me up and
> > running locally.

Done.

> > > I think we should also add it as a dep to the android_tools target. 

android_tools is build for #target, so adding dependency on bitmaptools (which
has toolsets: [host]) is not going to work.

[commit-bot: I haz the power, 2014-01-22 15:41:29]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/szym@chromium.org/131563009/680001

[commit-bot: I haz the power, 2014-01-22 17:50:02]

Change committed as 246361