Change ExecLinkWrapper to not buffer all tool output

Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.org
BUG=672182, 672841
Committed: https://crrev.com/81dfeb7be03b9b3b7311ede93908fd8159761446
Cr-Commit-Position: refs/heads/master@{#437696}

[brucedawson, 2016-12-09 20:47:20]

The CQ bit was checked by brucedawson@chromium.org to run a CQ dry run

[brucedawson, 2016-12-09 20:47:59]

Now it's perfect...

[commit-bot: I haz the power, 2016-12-09 20:48:12]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[scottmg, 2016-12-09 20:48:32]

scottmg@chromium.org changed reviewers:
+ scottmg@chromium.org

[scottmg, 2016-12-09 20:48:32]

lgtm

[scottmg, 2016-12-09 20:48:59]

OH NO

My autocomplete been corrupted with a chromium.com though.

[brucedawson, 2016-12-09 20:50:23]

Description was changed from

==========
Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.com
BUG=672182,672841
==========

to

==========
Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.org
BUG=672182,672841
==========

[brucedawson, 2016-12-09 20:50:24]

brucedawson@chromium.org changed reviewers:
- scottmg@chromium.com

[brucedawson, 2016-12-09 20:51:10]

On 2016/12/09 20:48:59, scottmg wrote:
> OH NO
> 
> My autocomplete been corrupted with a http://chromium.com though.

Aw !@#$. That's what I get for using R= for the first time in ever, thus
instantly giving me a reason to forego it. Sorry :-(

[scottmg, 2016-12-09 20:54:33]

On 2016/12/09 20:51:10, brucedawson wrote:
> On 2016/12/09 20:48:59, scottmg wrote:
> > OH NO
> > 
> > My autocomplete been corrupted with a http://chromium.com though.
> 
> Aw !@#$. That's what I get for using R= for the first time in ever, thus
> instantly giving me a reason to forego it. Sorry :-(

I'll survive. :) Actually, I think it's OK, as long as it's only in the R= and
someone doesn't explicitly type it into the web ui it doesn't get added.

[commit-bot: I haz the power, 2016-12-09 21:58:57]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-12-09 21:58:58]

Dry run: This issue passed the CQ dry run.

[brucedawson, 2016-12-09 22:46:27]

The CQ bit was checked by brucedawson@chromium.org

[commit-bot: I haz the power, 2016-12-09 22:47:37]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-12-09 23:39:14]

CQ is committing da patch.

Bot data: {"patchset_id": 1, "attempt_start_ts": 1481323587826400, "parent_rev":
"77b974dc493f72ccaa32b0385d89666fafa21e61", "commit_rev":
"5f6d0b1288a422229626c0e9b207d8ee840ed23c"}

[commit-bot: I haz the power, 2016-12-09 23:39:43]

Description was changed from

==========
Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.org
BUG=672182,672841
==========

to

==========
Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.org
BUG=672182,672841

Review-Url: https://codereview.chromium.org/2568563002
==========

[commit-bot: I haz the power, 2016-12-09 23:39:44]

Committed patchset #1 (id:1)

[commit-bot: I haz the power, 2016-12-12 14:58:30]

Description was changed from

==========
Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.org
BUG=672182,672841

Review-Url: https://codereview.chromium.org/2568563002
==========

to

==========
Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.org
BUG=672182,672841

Committed: https://crrev.com/81dfeb7be03b9b3b7311ede93908fd8159761446
Cr-Commit-Position: refs/heads/master@{#437696}
==========

[commit-bot: I haz the power, 2016-12-12 14:58:31]

Patchset 1 (id:??) landed as
https://crrev.com/81dfeb7be03b9b3b7311ede93908fd8159761446
Cr-Commit-Position: refs/heads/master@{#437696}