tracing: Stream processing for some importers

tracing: Stream processing for some importers

This implements stream processing for trace event importer, gzip
importer, and Linux trace event importer.

A tr.useTraceStreams flag is introduced and is disabled by default,
for now, until all importers support stream processing and the
performance is reasonable.

I tested that if I set tr.useTraceStream to true, I can successfully
run trace_processor/bin/process_traces on a ~700M trace file. However,
it is slow. The source of slowness is oboe.js itself. After a little
bit investigation, I think there are some optimizations that can be
done in oboe.js, which I will send in future CLs.

Note that this is still with the assumption that traces fit in memory. If we
want to get rid of this assumption in the future:
- d8 should support reading chunks of data from a file.
- tr.b.TraceStream read methods should return promises.
- oboe.js should support returning just the start and end position of a value
  instead of the value itself, for values that do not fit in memory.

BUG=catapult:#2826

Review-Url: https://codereview.chromium.org/2755943002
Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/cd51f067a7d5d33d695016bd1dc8a139bf736338

[chiniforooshan, 2017-03-16 16:37:40]

chiniforooshan@chromium.org changed reviewers:
+ charliea@chromium.org

[chiniforooshan, 2017-03-16 16:37:40]

PTAL

[charliea (OOO until 10-5), 2017-03-16 18:36:22]

charliea@chromium.org changed reviewers:
+ eakuefner@chromium.org

[eakuefner, 2017-03-17 20:18:14]

Providing a drive-by prereview as a non-expert in importing, this looks great.

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
File tracing/tracing/extras/importer/gzip_importer_test.html (right):

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
tracing/tracing/extras/importer/gzip_importer_test.html:29: function
toString(subtrace) {
This is funky. What is subtrace if it's not a TraceStream? Could we make this
more object-oriented?

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
tracing/tracing/extras/importer/gzip_importer_test.html:30: if (subtrace
instanceof tr.b.TraceStream)
nit: please either put this on the same line, or, if it exceeds max-len, use
braces.

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
File tracing/tracing/extras/importer/linux_perf/ftrace_importer.html (right):

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
tracing/tracing/extras/importer/linux_perf/ftrace_importer.html:224: let header
= incomingEvents instanceof tr.b.TraceStream ?
Can these be const?

[chiniforooshan, 2017-04-04 16:42:00]

Sorry for the delay. I just came back from a 2-week vacation today.

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
File tracing/tracing/extras/importer/gzip_importer_test.html (right):

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
tracing/tracing/extras/importer/gzip_importer_test.html:29: function
toString(subtrace) {
On 2017/03/17 20:18:13, eakuefner wrote:
> This is funky. What is subtrace if it's not a TraceStream? Could we make this
> more object-oriented?

Done.

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
tracing/tracing/extras/importer/gzip_importer_test.html:30: if (subtrace
instanceof tr.b.TraceStream)
On 2017/03/17 20:18:13, eakuefner wrote:
> nit: please either put this on the same line, or, if it exceeds max-len, use
> braces.

Done (applied this to the lines I touched and a few other places that I
noticed).

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
File tracing/tracing/extras/importer/linux_perf/ftrace_importer.html (right):

https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
tracing/tracing/extras/importer/linux_perf/ftrace_importer.html:224: let header
= incomingEvents instanceof tr.b.TraceStream ?
On 2017/03/17 20:18:13, eakuefner wrote:
> Can these be const?

Done.

[nednguyen, 2017-04-04 18:02:00]

On 2017/04/04 16:42:00, chiniforooshan wrote:
> Sorry for the delay. I just came back from a 2-week vacation today.
> 
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> File tracing/tracing/extras/importer/gzip_importer_test.html (right):
> 
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> tracing/tracing/extras/importer/gzip_importer_test.html:29: function
> toString(subtrace) {
> On 2017/03/17 20:18:13, eakuefner wrote:
> > This is funky. What is subtrace if it's not a TraceStream? Could we make
this
> > more object-oriented?
> 
> Done.
> 
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> tracing/tracing/extras/importer/gzip_importer_test.html:30: if (subtrace
> instanceof tr.b.TraceStream)
> On 2017/03/17 20:18:13, eakuefner wrote:
> > nit: please either put this on the same line, or, if it exceeds max-len, use
> > braces.
> 
> Done (applied this to the lines I touched and a few other places that I
> noticed).
> 
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> File tracing/tracing/extras/importer/linux_perf/ftrace_importer.html (right):
> 
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> tracing/tracing/extras/importer/linux_perf/ftrace_importer.html:224: let
header
> = incomingEvents instanceof tr.b.TraceStream ?
> On 2017/03/17 20:18:13, eakuefner wrote:
> > Can these be const?
> 
> Done.

Don't worry about trace fit in memory issue for now. We set an 8GB heap space
for d8, which should be more than enough (I
hope!):https://github.com/catapult-project/catapult/blob/master/tracing/tracing/mre/map_single_trace.py#L102

[nednguyen, 2017-04-04 18:02:29]

Description was changed from

==========
tracing: Stream processing for some importers

This implements stream processing for trace event importer, gzip
importer, and Linux trace event importer.

A tr.useTraceStreams flag is introduced and is disabled by default,
for now, until all importers support stream processing and the
performance is reasonable.

I tested that if I set tr.useTraceStream to true, I can successfully
run trace_processor/bin/process_traces on a ~700M trace file. However,
it is slow. The source of slowness is oboe.js itself. After a little
bit investigation, I think there are some optimizations that can be
done in oboe.js, which I will send in future CLs.

Note that this is still with the assumption that traces fit in memory. If we
want to get rid of this assumption in the future:
- d8 should support reading chunks of data from a file.
- tr.b.TraceStream read methods should return promises.
- oboe.js should support returning just the start and end position of a value
  instead of the value itself, for values that do not fit in memory.

BUG=catapult:#2826
==========

to

==========
tracing: Stream processing for some importers

This implements stream processing for trace event importer, gzip
importer, and Linux trace event importer.

A tr.useTraceStreams flag is introduced and is disabled by default,
for now, until all importers support stream processing and the
performance is reasonable.

I tested that if I set tr.useTraceStream to true, I can successfully
run trace_processor/bin/process_traces on a ~700M trace file. However,
it is slow. The source of slowness is oboe.js itself. After a little
bit investigation, I think there are some optimizations that can be
done in oboe.js, which I will send in future CLs.

Note that this is still with the assumption that traces fit in memory. If we
want to get rid of this assumption in the future:
- d8 should support reading chunks of data from a file.
- tr.b.TraceStream read methods should return promises.
- oboe.js should support returning just the start and end position of a value
  instead of the value itself, for values that do not fit in memory.

BUG=catapult:#2826
==========

[nednguyen, 2017-04-04 18:02:29]

nednguyen@google.com changed reviewers:
+ benjhayden@chromium.org, fmeawad@chromium.org

[nednguyen, 2017-04-04 18:02:55]

On 2017/04/04 18:02:00, nednguyen (slow til 4-10) wrote:
> On 2017/04/04 16:42:00, chiniforooshan wrote:
> > Sorry for the delay. I just came back from a 2-week vacation today.
> > 
> >
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> > File tracing/tracing/extras/importer/gzip_importer_test.html (right):
> > 
> >
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> > tracing/tracing/extras/importer/gzip_importer_test.html:29: function
> > toString(subtrace) {
> > On 2017/03/17 20:18:13, eakuefner wrote:
> > > This is funky. What is subtrace if it's not a TraceStream? Could we make
> this
> > > more object-oriented?
> > 
> > Done.
> > 
> >
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> > tracing/tracing/extras/importer/gzip_importer_test.html:30: if (subtrace
> > instanceof tr.b.TraceStream)
> > On 2017/03/17 20:18:13, eakuefner wrote:
> > > nit: please either put this on the same line, or, if it exceeds max-len,
use
> > > braces.
> > 
> > Done (applied this to the lines I touched and a few other places that I
> > noticed).
> > 
> >
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> > File tracing/tracing/extras/importer/linux_perf/ftrace_importer.html
(right):
> > 
> >
>
https://codereview.chromium.org/2755943002/diff/20001/tracing/tracing/extras/...
> > tracing/tracing/extras/importer/linux_perf/ftrace_importer.html:224: let
> header
> > = incomingEvents instanceof tr.b.TraceStream ?
> > On 2017/03/17 20:18:13, eakuefner wrote:
> > > Can these be const?
> > 
> > Done.
> 
> Don't worry about trace fit in memory issue for now. We set an 8GB heap space
> for d8, which should be more than enough (I
>
hope!):https://github.com/catapult-project/catapult/blob/master/tracing/tracing/mre/map_single_trace.py#L102

Charlie is out, so I add other tracing owners for reviewing.

[chiniforooshan, 2017-04-04 18:19:52]

chiniforooshan@chromium.org changed reviewers:
- benjhayden@chromium.org, fmeawad@chromium.org

[chiniforooshan, 2017-04-04 18:19:53]

Ethan has started reviewing, already. So, I moved others to the CC. But, of
course, comments from everyone is welcome.

[chiniforooshan, 2017-04-04 18:20:17]

chiniforooshan@chromium.org changed reviewers:
- charliea@chromium.org

[benjhayden, 2017-04-04 18:48:14]

benjhayden@chromium.org changed reviewers:
+ benjhayden@chromium.org

[benjhayden, 2017-04-04 18:48:15]

Some nits and a couple questions.

Should there be any changes to the trace_event_importer_test?

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/base/ba...
File tracing/tracing/base/base.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/base/ba...
tracing/tracing/base/base.html:175: tr.enableTraceStreams = false;
Why does this need to be a global instead of plumbed through callers?

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
File tracing/tracing/extras/importer/trace_event_importer.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:208: for (var
subtraceField of SUBTRACE_FIELDS)
Curly braces, please.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1125: if (!unitName)
curly braces

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1127: let unit =
tr.b.TimeDisplayModes[unitName];
const

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1143: for (var
fieldName in container) {
for (const fieldName of Object.keys(container))

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1147: if (fieldName
=== 'metadata') {
if (fieldName !== 'metadata') continue;

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1161: var
importOptions = this.model_.importOptions;
const/let

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1177: if (event.ph ===
'B' || event.ph === 'E') {
switch?

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
File tracing/tracing/importer/simple_line_reader.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
tracing/tracing/importer/simple_line_reader.html:40: get nextLine() {
Should this be an iterator like this?
* [Symbol.iterator]() {
  yield ...
}

[chiniforooshan, 2017-04-07 00:51:06]

Thanks Ben for reviewing. I modified almost all test cases in
trace_event_importer_test to also run on stream inputs, which helped me find and
fix a few bugs either in the code or the test cases.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/base/ba...
File tracing/tracing/base/base.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/base/ba...
tracing/tracing/base/base.html:175: tr.enableTraceStreams = false;
On 2017/04/04 18:48:15, benjhayden wrote:
> Why does this need to be a global instead of plumbed through callers?

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
File tracing/tracing/extras/importer/trace_event_importer.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:208: for (var
subtraceField of SUBTRACE_FIELDS)
On 2017/04/04 18:48:15, benjhayden wrote:
> Curly braces, please.

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1125: if (!unitName)
On 2017/04/04 18:48:15, benjhayden wrote:
> curly braces

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1127: let unit =
tr.b.TimeDisplayModes[unitName];
On 2017/04/04 18:48:15, benjhayden wrote:
> const

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1143: for (var
fieldName in container) {
On 2017/04/04 18:48:15, benjhayden wrote:
> for (const fieldName of Object.keys(container))

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1147: if (fieldName
=== 'metadata') {
On 2017/04/04 18:48:15, benjhayden wrote:
> if (fieldName !== 'metadata') continue;

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1161: var
importOptions = this.model_.importOptions;
On 2017/04/04 18:48:15, benjhayden wrote:
> const/let

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/extras/...
tracing/tracing/extras/importer/trace_event_importer.html:1177: if (event.ph ===
'B' || event.ph === 'E') {
On 2017/04/04 18:48:15, benjhayden wrote:
> switch?

Done.

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
File tracing/tracing/importer/simple_line_reader.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
tracing/tracing/importer/simple_line_reader.html:40: get nextLine() {
On 2017/04/04 18:48:15, benjhayden wrote:
> Should this be an iterator like this?
> * [Symbol.iterator]() {
>   yield ...
> }

I don't see a clean way of turning this into an iterator, given that  here we
need signals like "beginSavingLines", "endSavingLines", "goBack" to change the
iterator behaviour. Do you have an implementation in mind?

[benjhayden, 2017-04-07 16:43:22]

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
File tracing/tracing/importer/simple_line_reader.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
tracing/tracing/importer/simple_line_reader.html:40: get nextLine() {
On 2017/04/07 at 00:51:06, chiniforooshan wrote:
> On 2017/04/04 18:48:15, benjhayden wrote:
> > Should this be an iterator like this?
> > * [Symbol.iterator]() {
> >   yield ...
> > }
> 
> I don't see a clean way of turning this into an iterator, given that  here we
need signals like "beginSavingLines", "endSavingLines", "goBack" to change the
iterator behaviour. Do you have an implementation in mind?

Clients can call those methods while iterating using for..of:
const lineReader = new SimpleLineReader(...);
for (const line of lineReader) {
  lineReader.goBack();
  lineReader.beginSavingLines();
  lineReader.endSavingLines();
}

This would move the while(hasData) loop into SimpleLineReader[Symbol.iterator]
and should let hasData_ be private:

class SimpleLineReader {
  * [Symbol.iterator]() {
    while (this.hasData_) {
      ...
      yield line;
    }
  }
}

[chiniforooshan, 2017-04-10 17:27:13]

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
File tracing/tracing/importer/simple_line_reader.html (right):

https://codereview.chromium.org/2755943002/diff/40001/tracing/tracing/importe...
tracing/tracing/importer/simple_line_reader.html:40: get nextLine() {
On 2017/04/07 16:43:22, benjhayden wrote:
> On 2017/04/07 at 00:51:06, chiniforooshan wrote:
> > On 2017/04/04 18:48:15, benjhayden wrote:
> > > Should this be an iterator like this?
> > > * [Symbol.iterator]() {
> > >   yield ...
> > > }
> > 
> > I don't see a clean way of turning this into an iterator, given that  here
we
> need signals like "beginSavingLines", "endSavingLines", "goBack" to change the
> iterator behaviour. Do you have an implementation in mind?
> 
> Clients can call those methods while iterating using for..of:
> const lineReader = new SimpleLineReader(...);
> for (const line of lineReader) {
>   lineReader.goBack();
>   lineReader.beginSavingLines();
>   lineReader.endSavingLines();
> }
> 
> This would move the while(hasData) loop into SimpleLineReader[Symbol.iterator]
> and should let hasData_ be private:
> 
> class SimpleLineReader {
>   * [Symbol.iterator]() {
>     while (this.hasData_) {
>       ...
>       yield line;
>     }
>   }
> }

Done.

[chiniforooshan, 2017-04-10 17:49:11]

The CQ bit was checked by chiniforooshan@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-10 17:49:20]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-10 17:50:30]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-10 17:50:31]

Dry run: Try jobs failed on following builders:
  Catapult Android Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20An...)
  Catapult Linux Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Li...)
  Catapult Presubmit on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Pr...)

[chiniforooshan, 2017-04-10 21:03:31]

The CQ bit was checked by chiniforooshan@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-10 21:03:41]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-10 21:27:31]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-10 21:27:32]

Dry run: This issue passed the CQ dry run.

[chiniforooshan, 2017-04-13 14:12:02]

ping

[fmeawad, 2017-04-14 16:23:38]

On 2017/04/13 14:12:02, chiniforooshan wrote:
> ping

Since ben and charlie are OOO, I will take a look today.

[eakuefner, 2017-04-14 17:42:07]

eakuefner@chromium.org changed reviewers:
+ fmeawad@chromium.org

[eakuefner, 2017-04-14 17:42:08]

lgtm with one question, but please do wait for fadi to take a look.

https://codereview.chromium.org/2755943002/diff/140001/tracing/tracing/extras...
File tracing/tracing/extras/importer/trace_event_importer_test.html (right):

https://codereview.chromium.org/2755943002/diff/140001/tracing/tracing/extras...
tracing/tracing/extras/importer/trace_event_importer_test.html:6268: //
TODO(chiniforooshan): When the input is a trace stream, it is processed
Wait, what's the TODO here? Testing this for the trace stream case? If you want
to do that by listening for the mentioned event, it's possible to write test
cases that return promises, in which case you could add a listener that resolves
the promise.

[fmeawad, 2017-04-14 22:05:39]

lgtm

[chiniforooshan, 2017-04-21 15:07:33]

https://codereview.chromium.org/2755943002/diff/140001/tracing/tracing/extras...
File tracing/tracing/extras/importer/trace_event_importer_test.html (right):

https://codereview.chromium.org/2755943002/diff/140001/tracing/tracing/extras...
tracing/tracing/extras/importer/trace_event_importer_test.html:6268: //
TODO(chiniforooshan): When the input is a trace stream, it is processed
On 2017/04/14 17:42:08, eakuefner wrote:
> Wait, what's the TODO here? Testing this for the trace stream case? If you
want
> to do that by listening for the mentioned event, it's possible to write test
> cases that return promises, in which case you could add a listener that
resolves
> the promise.

I think we should expect the same behaviour from the event importer for both
input types. I changed the TODO to reflect this.

[chiniforooshan, 2017-04-21 15:07:45]

The CQ bit was checked by chiniforooshan@chromium.org

[chiniforooshan, 2017-04-21 15:07:46]

The patchset sent to the CQ was uploaded after l-g-t-m from
eakuefner@chromium.org, fmeawad@chromium.org
Link to the patchset: https://codereview.chromium.org/2755943002/#ps180001
(title: "review / sync")

[commit-bot: I haz the power, 2017-04-21 15:07:49]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-21 16:09:55]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-21 16:09:56]

Try jobs failed on following builders:
  Catapult Mac Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Ma...)

[chiniforooshan, 2017-04-21 16:21:26]

The CQ bit was checked by chiniforooshan@chromium.org

[commit-bot: I haz the power, 2017-04-21 16:21:36]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-21 17:23:33]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-21 17:23:34]

Try jobs failed on following builders:
  Catapult Mac Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Ma...)

[chiniforooshan, 2017-04-21 20:03:13]

The CQ bit was checked by chiniforooshan@chromium.org

[commit-bot: I haz the power, 2017-04-21 20:03:16]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-21 20:13:34]

CQ is committing da patch.

Bot data: {"patchset_id": 180001, "attempt_start_ts": 1492804993364070,
"parent_rev": "cf5379cbc7095de979bba11519acc126a99532d2", "commit_rev":
"cd51f067a7d5d33d695016bd1dc8a139bf736338"}

[commit-bot: I haz the power, 2017-04-21 20:13:41]

Description was changed from

==========
tracing: Stream processing for some importers

This implements stream processing for trace event importer, gzip
importer, and Linux trace event importer.

A tr.useTraceStreams flag is introduced and is disabled by default,
for now, until all importers support stream processing and the
performance is reasonable.

I tested that if I set tr.useTraceStream to true, I can successfully
run trace_processor/bin/process_traces on a ~700M trace file. However,
it is slow. The source of slowness is oboe.js itself. After a little
bit investigation, I think there are some optimizations that can be
done in oboe.js, which I will send in future CLs.

Note that this is still with the assumption that traces fit in memory. If we
want to get rid of this assumption in the future:
- d8 should support reading chunks of data from a file.
- tr.b.TraceStream read methods should return promises.
- oboe.js should support returning just the start and end position of a value
  instead of the value itself, for values that do not fit in memory.

BUG=catapult:#2826
==========

to

==========
tracing: Stream processing for some importers

This implements stream processing for trace event importer, gzip
importer, and Linux trace event importer.

A tr.useTraceStreams flag is introduced and is disabled by default,
for now, until all importers support stream processing and the
performance is reasonable.

I tested that if I set tr.useTraceStream to true, I can successfully
run trace_processor/bin/process_traces on a ~700M trace file. However,
it is slow. The source of slowness is oboe.js itself. After a little
bit investigation, I think there are some optimizations that can be
done in oboe.js, which I will send in future CLs.

Note that this is still with the assumption that traces fit in memory. If we
want to get rid of this assumption in the future:
- d8 should support reading chunks of data from a file.
- tr.b.TraceStream read methods should return promises.
- oboe.js should support returning just the start and end position of a value
  instead of the value itself, for values that do not fit in memory.

BUG=catapult:#2826

Review-Url: https://codereview.chromium.org/2755943002
Committed:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...
==========

[commit-bot: I haz the power, 2017-04-21 20:13:42]

Committed patchset #10 (id:180001) as
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...