[Telemetry] Change trace_data to hold a list of raw trace data for each trace part

[Telemetry] Change trace_data to hold a list of raw trace data for each trace part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)



Another way to understand the effects of this CL is that we change the 
data we send to trace processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to:
 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119

Review-Url: https://codereview.chromium.org/2619073002
Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/279dbaad26cbbda4a1a08d5348cbc0c8e281f838

[nednguyen, 2017-01-09 20:05:18]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-09 20:05:21]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-09 20:18:31]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-09 20:18:32]

Dry run: Try jobs failed on following builders:
  Catapult Mac Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Ma...)

[nednguyen, 2017-01-09 21:15:56]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-09 21:16:06]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-09 21:27:56]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-09 21:27:56]

Dry run: Try jobs failed on following builders:
  Catapult Windows Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Wi...)

[nednguyen, 2017-01-10 17:02:21]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-10 17:02:28]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-10 17:08:26]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-10 17:08:27]

Dry run: Try jobs failed on following builders:
  Catapult Presubmit on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Pr...)

[nednguyen, 2017-01-10 19:32:18]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-10 19:32:28]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-10 19:49:23]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-10 19:49:24]

Dry run: Try jobs failed on following builders:
  Catapult Linux Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Li...)

[nednguyen, 2017-01-11 15:28:44]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-11 15:28:50]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-11 15:36:31]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-11 15:36:32]

Dry run: Try jobs failed on following builders:
  Catapult Windows Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Wi...)

[nednguyen, 2017-01-11 20:55:57]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-11 20:56:07]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-11 21:03:26]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-11 21:03:27]

Dry run: Try jobs failed on following builders:
  Catapult Windows Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Wi...)

[nednguyen, 2017-01-11 22:37:24]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-11 22:37:33]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-11 22:50:23]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-11 22:50:24]

Dry run: Try jobs failed on following builders:
  Catapult Mac Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Ma...)

[nednguyen, 2017-01-11 23:14:49]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-11 23:14:58]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-11 23:37:37]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-11 23:37:38]

Dry run: This issue passed the CQ dry run.

[nednguyen, 2017-01-12 14:17:40]

Description was changed from

==========
WIP - Change trace_data to hold a list of unpacked raw trace data

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we need to remove the need to unpack the raw
trace data. Previous to this CL, trace data hold one single raw trace 
data for each trace part. Therefore when we have multiple Chrome traces 
in a multi-browser testing case, tracing_controller_backend need to 
merge all the raw trace data into one. As such merging operation 
requires all the traces to be in memory.


BUG=catapult:#3119
==========

[nednguyen, 2017-01-12 14:17:41]

nednguyen@google.com changed reviewers:
+ charliea@chromium.org, perezju@chromium.org

[nednguyen, 2017-01-12 14:17:55]

Patchset #1 (id:1) has been deleted

[nednguyen, 2017-01-12 14:18:03]

Patchset #1 (id:20001) has been deleted

[nednguyen, 2017-01-12 14:18:10]

Patchset #1 (id:40001) has been deleted

[nednguyen, 2017-01-12 14:18:17]

Patchset #1 (id:60001) has been deleted

[nednguyen, 2017-01-12 14:18:26]

Patchset #1 (id:80001) has been deleted

[nednguyen, 2017-01-12 14:18:34]

Patchset #1 (id:100001) has been deleted

[nednguyen, 2017-01-12 14:18:40]

Patchset #1 (id:120001) has been deleted

[nednguyen, 2017-01-12 14:18:47]

Patchset #1 (id:140001) has been deleted

[nednguyen, 2017-01-12 14:21:12]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we need to remove the need to unpack the raw
trace data. Previous to this CL, trace data hold one single raw trace 
data for each trace part. Therefore when we have multiple Chrome traces 
in a multi-browser testing case, tracing_controller_backend need to 
merge all the raw trace data into one. As such merging operation 
requires all the traces to be in memory.


BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we need to remove the need to unpack the raw
trace data. Previous to this CL, trace data hold one single raw trace 
data for each trace part. Therefore when we have multiple Chrome traces 
in a multi-browser testing case, tracing_controller_backend need to 
merge all the raw trace data into one. As such merging operation 
requires all the traces to be in memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

[nednguyen, 2017-01-12 14:21:28]

nednguyen@google.com changed reviewers:
+ rnephew@chromium.org

[nednguyen, 2017-01-12 14:21:39]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-12 14:21:47]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-12 14:23:37]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-12 14:23:38]

Dry run: This issue passed the CQ dry run.

[nednguyen, 2017-01-12 15:50:20]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we need to remove the need to unpack the raw
trace data. Previous to this CL, trace data hold one single raw trace 
data for each trace part. Therefore when we have multiple Chrome traces 
in a multi-browser testing case, tracing_controller_backend need to 
merge all the raw trace data into one. As such merging operation 
requires all the traces to be in memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data. Previous to this CL, trace data hold one single raw trace 
data for each trace part. Therefore when we have multiple Chrome traces 
in a multi-browser testing case, tracing_controller_backend need to 
merge all the raw trace data into one. As such merging operation 
requires all the traces to be in memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

[nednguyen, 2017-01-12 15:51:06]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data. Previous to this CL, trace data hold one single raw trace 
data for each trace part. Therefore when we have multiple Chrome traces 
in a multi-browser testing case, tracing_controller_backend need to 
merge all the raw trace data into one. As such merging operation 
requires all the traces to be in memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single raw trace
data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

[nednguyen, 2017-01-12 15:51:23]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single raw trace
data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

[nednguyen(REVIEW IN OTHER ACC), 2017-01-12 20:30:08]

On 2017/01/12 14:21:29, nednguyen wrote:

Ping folks

[nednguyen(REVIEW IN OTHER ACC), 2017-01-12 20:30:50]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

With this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

[charliea (OOO until 10-5), 2017-01-12 21:10:18]

Sorry! Thought I responded earlier.

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
File telemetry/telemetry/value/trace.py (right):

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
telemetry/telemetry/value/trace.py:69: for traces_list, part in
self._GetTraceParts(trace_data):
High-level question: do you know why it's necessary to combine these? IIUC each
of these is a valid stand-alone trace. Can't we just write each one as a
separate trace, hand that html file over to tracing, and let it import each one
separately and have things still work correctly?

[nednguyen, 2017-01-12 21:29:16]

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
File telemetry/telemetry/value/trace.py (right):

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
telemetry/telemetry/value/trace.py:69: for traces_list, part in
self._GetTraceParts(trace_data):
On 2017/01/12 21:10:17, charliea wrote:
> High-level question: do you know why it's necessary to combine these? IIUC
each
> of these is a valid stand-alone trace. Can't we just write each one as a
> separate trace, hand that html file over to tracing, and let it import each
one
> separately and have things still work correctly?

We have a thread a long ago with a similar question by memory team but I lose
the pointer to it. The basic idea here is Telemetry's architecture is:
1 run ---> 1 set of metrics computed for that run.

So for a multi-browsers test, to make this model consistent, we combine the
traces to avoid: 1 run ---> N set of metrics computed for that run. If we do
this, we need to expand our metric & result system to include the concept of
"trace", which seems to break the layering between testing & metrics.

Besides those conceptual problem of computing metrics on a per traces basic,
another reason why we want to combine data is that because multi-browsers do
share memory (which makes them interesting test cases). By not combining the
tracing data, we lose the view to the total system when trying to compute
metrics.

[perezju, 2017-01-13 10:17:01]

happy with this if others are happy, I've just left a comment.

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/in...
File
telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend_unittest.py
(right):

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/in...
telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend_unittest.py:170:
trace_data.CHROME_TRACE_PART)[0].get('traceEvents', [])
this seems to be a common pattern: GetTracesFor(...)[0], how about adding an
API: GetSingleTraceFor(...) that:
- checks that len(traces) == 1
- returns that single trace

[nednguyen, 2017-01-13 15:53:21]

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/in...
File
telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend_unittest.py
(right):

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/in...
telemetry/telemetry/internal/backends/chrome_inspector/tracing_backend_unittest.py:170:
trace_data.CHROME_TRACE_PART)[0].get('traceEvents', [])
On 2017/01/13 10:17:01, perezju wrote:
> this seems to be a common pattern: GetTracesFor(...)[0], how about adding an
> API: GetSingleTraceFor(...) that:
> - checks that len(traces) == 1
> - returns that single trace

Umm, this is only exists because we have legacy trace parser in Python. I think
we need not make it easier to use the legacy one & transition to JS instead :-)

[charliea (OOO until 10-5), 2017-01-13 16:36:47]

lgtm

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
File telemetry/telemetry/value/trace.py (right):

https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
telemetry/telemetry/value/trace.py:69: for traces_list, part in
self._GetTraceParts(trace_data):
Ned and I talked about this offline, and I was misunderstanding what was
happening. I thought that we were concatenating the traces so that, for two
browsers X and Y, we were outputting something like the following in HTML:

<script>
trace for X
trace for Y
</script>

instead, what we're doing is outputting two <script> tags, like:

<script>
trace for X
</script>
<script>
trace for Y
</script>

which looks great to me.

[nednguyen, 2017-01-13 16:37:44]

On 2017/01/13 16:36:47, charliea wrote:
> lgtm
> 
>
https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
> File telemetry/telemetry/value/trace.py (right):
> 
>
https://codereview.chromium.org/2619073002/diff/160001/telemetry/telemetry/va...
> telemetry/telemetry/value/trace.py:69: for traces_list, part in
> self._GetTraceParts(trace_data):
> Ned and I talked about this offline, and I was misunderstanding what was
> happening. I thought that we were concatenating the traces so that, for two
> browsers X and Y, we were outputting something like the following in HTML:
> 
> <script>
> trace for X
> trace for Y
> </script>
> 
> instead, what we're doing is outputting two <script> tags, like:
> 
> <script>
> trace for X
> </script>
> <script>
> trace for Y
> </script>
> 
> which looks great to me.

Yes, I will put this part to the description for documenting this change as
well.

[nednguyen, 2017-01-13 16:40:29]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

Another way to explain this is that we change the data we send to trace
processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119
==========

[nednguyen, 2017-01-13 16:40:59]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)

Another way to explain this is that we change the data we send to trace
processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)



Another way to understand the effects of this CL is that we change the 
data we send to trace processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119
==========

[nednguyen, 2017-01-13 16:41:14]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)



Another way to understand the effects of this CL is that we change the 
data we send to trace processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)



Another way to understand the effects of this CL is that we change the 
data we send to trace processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to:
 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119
==========

[nednguyen, 2017-01-13 16:41:20]

The CQ bit was checked by nednguyen@google.com

[commit-bot: I haz the power, 2017-01-13 16:41:29]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-13 17:01:49]

CQ is committing da patch.

Bot data: {"patchset_id": 160001, "attempt_start_ts": 1484325680202640,
"parent_rev": "1bcf49e0a7eb1dcd21cd1884385d5f782d79492b", "commit_rev":
"279dbaad26cbbda4a1a08d5348cbc0c8e281f838"}

[commit-bot: I haz the power, 2017-01-13 17:01:53]

Description was changed from

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)



Another way to understand the effects of this CL is that we change the 
data we send to trace processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to:
 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119
==========

to

==========
[Telemetry] Change trace_data to hold a list of raw trace data for each trace
part

Motivation: in order to change trace_data to keep the raw trace data on
disk instead of in memory, we have to remove the need to unpack the raw
trace data up front. Previous to this CL, trace data hold one single 
raw trace data for each trace part. Therefore when we have multiple 
Chrome traces in a multi-browser testing case, 
tracing_controller_backend need to merge all the raw trace data into 
one. And such merging operation requires all the traces to be in 
memory.

The main changes are in trace_data.py.
In addition, with this CL, merging multiple chrome traces is handled by
1) trace_event_importer.py if the trace parsing is done in Python 
(mostly for TBM 1.0)
2) trace2html script (see the change to telemetry/telemetry/value/trace.py)



Another way to understand the effects of this CL is that we change the 
data we send to trace processor written in JS from:

<script>
Combined trace of X & Y (combining this is done in Python)
</script>

to:
 
<script>
Trace of X
</script>
<script>
Trace of Y
</script>

The logic of merging the traces then will be delayed to Javscript
engine which can do this in a much more efficient fashion.

BUG=catapult:#3119

Review-Url: https://codereview.chromium.org/2619073002
Committed:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...
==========

[commit-bot: I haz the power, 2017-01-13 17:01:55]

Committed patchset #1 (id:160001) as
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...