Add log and link aliasing to BuildBot.

Add log and link aliasing to BuildBot.

This adds support for log/url aliases to BuildBot, and augments the
annotation engine to extract aliases from specially-formatted STEP_LINK
commands.

BuildBot will now associate and render a log/url's alias alongside it in
the build view.

The BuildBot annotation parser will identify STEP_LINK of the form
"alias-->orig", and add it as an alias for "orig" called "alias" instead
of as a separate URL link.

The template and BuildBot changes are intended to be backwards-compatible
with currently-running BuildBot instances to allow for master restarts
to pick this up gradually. A master that hasn't restarted will render
alias annotations as just more links (e.g., "alias-->orig"), should
aliases be emitted to that master before it has been restarted.

BUG=chromium:595946
TEST=local
  - First up a custom waterfall / recipe engine, ran, looks decent.

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299396

[dnj, 2016-03-18 01:52:45]

dnj@chromium.org changed reviewers:
+ estaab@chromium.org, iannucci@chromium.org

[dnj, 2016-03-18 01:52:46]

PTAL

https://codereview.chromium.org/1812163002/diff/1/masters/master.chromium.per...
File masters/master.chromium.perf.fyi/templates/build.html (right):

https://codereview.chromium.org/1812163002/diff/1/masters/master.chromium.per...
masters/master.chromium.perf.fyi/templates/build.html:156: {% if s.aliases %}
This should protect existing non-restarted waterfalls from not having emitted
the "alias" property to the template engine.

https://codereview.chromium.org/1812163002/diff/1/masters/master.chromium.per...
masters/master.chromium.perf.fyi/templates/build.html:158: <a href="{{ a.url
}}">[{{ a.text }}]</a>
Renders as: link [alias] [alias]

https://codereview.chromium.org/1812163002/diff/1/scripts/master/chromium_ste...
File scripts/master/chromium_step.py (right):

https://codereview.chromium.org/1812163002/diff/1/scripts/master/chromium_ste...
scripts/master/chromium_step.py:1031: parts = link_label.split('-->', 1)
I'm not a huge fan of "-->". If something inspiring comes to mind, feel free to
suggest it.

https://codereview.chromium.org/1812163002/diff/1/third_party/buildbot_8_4p1/...
File third_party/buildbot_8_4p1/buildbot/status/buildstep.py (right):

https://codereview.chromium.org/1812163002/diff/1/third_party/buildbot_8_4p1/...
third_party/buildbot_8_4p1/buildbot/status/buildstep.py:126: return
getattr(self, 'aliases', {}).copy()
I chose this method instead of formally upgrading the pickle version. If an old
build is loaded, it will not have "aliases", and it will return an empty
dictionary.

https://codereview.chromium.org/1812163002/diff/1/third_party/buildbot_8_4p1/...
third_party/buildbot_8_4p1/buildbot/status/buildstep.py:294: text =
str(len(value))
If no text is supplied, the alias will just be a number. This allows "-->stdio"
style aliases.

[dnj, 2016-03-18 01:53:07]

Oh also please pay attention to the templates. I think I got them all but, as
with any pool of generally-identical files, it's easy to get tunnel vision.

[dnj, 2016-03-18 01:59:37]

On 2016/03/18 01:53:07, dnj wrote:
> Oh also please pay attention to the templates. I think I got them all but, as
> with any pool of generally-identical files, it's easy to get tunnel vision.

Might as well link screenshots.
BuildBot view: https://screenshot.googleplex.com/PiDaZZrpZyr
Annotated text: https://screenshot.googleplex.com/ELNXdN7JOet

[iannucci, 2016-03-18 22:21:06]

lgtm

[dnj, 2016-03-18 22:41:40]

The CQ bit was checked by dnj@chromium.org

[commit-bot: I haz the power, 2016-03-18 22:41:46]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1812163002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1812163002/1

[commit-bot: I haz the power, 2016-03-18 22:49:56]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-03-18 22:49:56]

Try jobs failed on following builders:
  Build Presubmit on tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Build%20Presubmit/build...)

[dnj, 2016-03-18 22:56:39]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-03-18 22:56:40]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1812163002/#ps20001
(title: "Update unit tests.")

[commit-bot: I haz the power, 2016-03-18 22:56:48]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1812163002/20001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1812163002/20001

[dnj, 2016-03-18 23:00:55]

The CQ bit was unchecked by dnj@chromium.org

[dnj, 2016-03-18 23:01:03]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-03-18 23:01:04]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/1812163002/#ps40001
(title: "Unused variable.")

[commit-bot: I haz the power, 2016-03-18 23:01:06]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1812163002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1812163002/40001

[commit-bot: I haz the power, 2016-03-18 23:05:01]

Description was changed from

==========
Add log and link aliasing to BuildBot.

This adds support for log/url aliases to BuildBot, and augments the
annotation engine to extract aliases from specially-formatted STEP_LINK
commands.

BuildBot will now associate and render a log/url's alias alongside it in
the build view.

The BuildBot annotation parser will identify STEP_LINK of the form
"alias-->orig", and add it as an alias for "orig" called "alias" instead
of as a separate URL link.

The template and BuildBot changes are intended to be backwards-compatible
with currently-running BuildBot instances to allow for master restarts
to pick this up gradually. A master that hasn't restarted will render
alias annotations as just more links (e.g., "alias-->orig"), should
aliases be emitted to that master before it has been restarted.

BUG=chromium:595946
TEST=local
  - First up a custom waterfall / recipe engine, ran, looks decent.
==========

to

==========
Add log and link aliasing to BuildBot.

This adds support for log/url aliases to BuildBot, and augments the
annotation engine to extract aliases from specially-formatted STEP_LINK
commands.

BuildBot will now associate and render a log/url's alias alongside it in
the build view.

The BuildBot annotation parser will identify STEP_LINK of the form
"alias-->orig", and add it as an alias for "orig" called "alias" instead
of as a separate URL link.

The template and BuildBot changes are intended to be backwards-compatible
with currently-running BuildBot instances to allow for master restarts
to pick this up gradually. A master that hasn't restarted will render
alias annotations as just more links (e.g., "alias-->orig"), should
aliases be emitted to that master before it has been restarted.

BUG=chromium:595946
TEST=local
  - First up a custom waterfall / recipe engine, ran, looks decent.

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=299396
==========

[commit-bot: I haz the power, 2016-03-18 23:05:02]

Committed patchset #3 (id:40001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=299396

[Nico, 2016-03-21 18:40:41]

thakis@chromium.org changed reviewers:
+ thakis@chromium.org

[Nico, 2016-03-21 18:40:42]

I see somewhat frequent "step interrupted"s on
https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang_x64_dbg/
with stacks looking like so:

Traceback (most recent call last):
  File
"/home/chrome-bot/buildbot/build/third_party/twisted_10_2/twisted/spread/pb.py",
line 516, in expressionReceived
    method(*sexp[1:])
  File
"/home/chrome-bot/buildbot/build/third_party/twisted_10_2/twisted/spread/pb.py",
line 828, in proto_message
    self._recvMessage(self.localObjectForID, requestID, objectID, message,
answerRequired, netArgs, netKw)
  File
"/home/chrome-bot/buildbot/build/third_party/twisted_10_2/twisted/spread/pb.py",
line 842, in _recvMessage
    netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
  File
"/home/chrome-bot/buildbot/build/third_party/twisted_10_2/twisted/spread/flavors.py",
line 114, in remoteMessageReceived
    state = method(*args, **kw)
--- <exception caught here> ---
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/process/buildstep.py",
line 176, in remote_update
    self.remoteUpdate(update)
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/process/metrics.py",
line 81, in wrapper
    return func(*args, **kwargs)
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/process/buildstep.py",
line 362, in remoteUpdate
    self.addStdout(update['stdout'])
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/process/buildstep.py",
line 333, in addStdout
    self.logs['stdio'].addStdout(data)
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/status/logfile.py",
line 458, in addStdout
    self.addEntry(STDOUT, text)
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/status/logfile.py",
line 454, in addEntry
    w.logChunk(build, step, self, channel, text)
  File "/home/chrome-bot/buildbot/build/scripts/master/chromium_step.py", line
868, in logChunk
    self.outReceived(line)
  File "/home/chrome-bot/buildbot/build/scripts/master/chromium_step.py", line
878, in outReceived
    self.cursor['log'].addStdout(data)
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/status/logfile.py",
line 458, in addStdout
    self.addEntry(STDOUT, text)
  File
"/home/chrome-bot/buildbot/build/third_party/buildbot_8_4p1/buildbot/status/logfile.py",
line 414, in addEntry
    assert not self.finished
exceptions.AssertionError: 



That has this file on the stack, and this file was changed 2 days ago (and
wasn't before that for months) -- do you think that's related?

Examples:

https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang_x64_db...
https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang_x64_db...
https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang_x64_db...

[dnj, 2016-03-21 18:56:47]

> That has this file on the stack, and this file was changed 2 days ago (and
> wasn't before that for months) -- do you think that's related?

Looking at the stack trace, I don't think it's related.

This patch landed at 16:05, but the master wasn't restarted at 18:20 on the
18th. In the meantime, build #221, which started at 16:03, is exhibiting these
symptoms:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_clang_x64_db...