Descriptiontracing: Stream processing for some importers
This implements stream processing for trace event importer, gzip
importer, and Linux trace event importer.
A tr.useTraceStreams flag is introduced and is disabled by default,
for now, until all importers support stream processing and the
performance is reasonable.
I tested that if I set tr.useTraceStream to true, I can successfully
run trace_processor/bin/process_traces on a ~700M trace file. However,
it is slow. The source of slowness is oboe.js itself. After a little
bit investigation, I think there are some optimizations that can be
done in oboe.js, which I will send in future CLs.
Note that this is still with the assumption that traces fit in memory. If we
want to get rid of this assumption in the future:
- d8 should support reading chunks of data from a file.
- tr.b.TraceStream read methods should return promises.
- oboe.js should support returning just the start and end position of a value
instead of the value itself, for values that do not fit in memory.
BUG=catapult:#2826
Review-Url: https://codereview.chromium.org/2755943002
Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/cd51f067a7d5d33d695016bd1dc8a139bf736338
Patch Set 1 #Patch Set 2 : disable trace streams by default #
Total comments: 6
Patch Set 3 : review #
Total comments: 20
Patch Set 4 : review #Patch Set 5 : review #Patch Set 6 : review #Patch Set 7 : nit #Patch Set 8 : sync #
Total comments: 2
Patch Set 9 : review #Patch Set 10 : review / sync #Messages
Total messages: 46 (24 generated)
|