Add buildbot metadata to compiler_proxy log

Add buildbot metadata to compiler_proxy log

We'd like to analyze compiler proxy log. However, buildbot metadata is
missing, so sometimes hard to determine in which builder the compiler
proxy log is generated.

Append buildbot metadata to compiler_proxy log like ninja log.

Committed: https://chromium.googlesource.com/chromium/tools/build/+/3d1cf68904cab7a23836940d3bdb1f43bee145b8

[shinyak, 2016-09-01 08:45:29]

shinyak@chromium.org changed reviewers:
+ phajdan.jr@chromium.org, tikuta@chromium.org, ukai@chromium.org,
yyanagisawa@chromium.org

[tikuta, 2016-09-01 09:12:54]

lgtm

https://codereview.chromium.org/2302453004/diff/1/scripts/slave/goma_utils.py
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/1/scripts/slave/goma_utils.py...
scripts/slave/goma_utils.py:128: 'env': {},
Why don't you use os.environ.clone()?

[shinyak, 2016-09-01 09:25:38]

https://codereview.chromium.org/2302453004/diff/1/scripts/slave/goma_utils.py
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/1/scripts/slave/goma_utils.py...
scripts/slave/goma_utils.py:128: 'env': {},
On 2016/09/01 09:12:54, tikuta wrote:
> Why don't you use os.environ.clone()?

There is no method clone() in os.environ.
json.dumps(os.environ) also does not work. It's not JSON serializable.
Actually os.environ is not Dict.

[Paweł Hajdan Jr., 2016-09-01 09:42:28]

Isn't os.environ lead to exactly same problems you've hit before?

Why not use api.properties instead?

[shinyak, 2016-09-02 02:35:04]

Hmm, what problems?

This is appending buildbot data (e.g. mastername, buildername, etc.) to
compiler_proxy log
like goma_utils.UploadNinjaLog().

This is not recipe code. Do we have api.properties??

[shinyak, 2016-09-02 02:37:09]

Ah, you're saying something like this patch?
https://codereview.chromium.org/2306523002/

[ukai, 2016-09-02 05:06:32]

lgtm

https://codereview.chromium.org/2302453004/diff/1/scripts/slave/goma_utils.py
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/1/scripts/slave/goma_utils.py...
scripts/slave/goma_utils.py:129: }
sucproc info name?

[shinyak, 2016-09-05 07:04:58]

The CQ bit was checked by shinyak@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-05 07:05:04]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[shinyak, 2016-09-05 07:05:57]

PTAL?

Instead of adding text in compiler_proxy log, I chose to use cloud storage
metadata to put master/slave/builder/clobber/os information.

[commit-bot: I haz the power, 2016-09-05 07:13:13]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-05 07:13:13]

Dry run: Try jobs failed on following builders:
  Build Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/3111fc316b3ef310)

[Yoshisato Yanagisawa, 2016-09-05 08:01:27]

lgtm w/ comment.

[Yoshisato Yanagisawa, 2016-09-05 08:01:39]

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
scripts/slave/goma_utils.py:124: metadata = {
I thought you encode metadata with gzip+base64 and subscriber just read one
field and decode, but maybe ok with this.

[shinyak, 2016-09-06 02:30:33]

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
scripts/slave/goma_utils.py:124: metadata = {
On 2016/09/05 08:01:39, Yoshisato Yanagisawa wrote:
> I thought you encode metadata with gzip+base64 and subscriber just read one
> field and decode, but maybe ok with this.

Hmm, I'm OK with gzip. these name could be too general for metadata?

[shinyak, 2016-09-06 02:36:36]

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
scripts/slave/goma_utils.py:124: metadata = {
Since this is small, I feel it's ok to save as json (without gzip), though...

[Yoshisato Yanagisawa, 2016-09-06 02:50:50]

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
scripts/slave/goma_utils.py:124: metadata = {
Other reviewers may think the other things but I feel metadata keys a bit
confusing because there is no hint in keys to mention who set them, or for
guessing the purpose of them.

I am afraid something bad happens with Windows command line but no need to
compress or base64 encode. (maybe not encoding help us to read the config
easily)

[shinyak, 2016-09-06 02:55:28]

The CQ bit was checked by shinyak@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-06 02:55:38]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[shinyak, 2016-09-06 02:59:22]

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
File scripts/slave/goma_utils.py (right):

https://codereview.chromium.org/2302453004/diff/20001/scripts/slave/goma_util...
scripts/slave/goma_utils.py:124: metadata = {
Since this runs with python subprocess.Popen (finally), it should be ok (I
believe.)

[commit-bot: I haz the power, 2016-09-06 03:02:36]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-06 03:02:37]

Dry run: This issue passed the CQ dry run.

[shinyak, 2016-09-06 05:19:32]

The CQ bit was checked by shinyak@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-06 05:19:42]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[shinyak, 2016-09-06 05:20:24]

yyanagisawa-san, PTAL? If ok, I'd like to submit this.

[Yoshisato Yanagisawa, 2016-09-06 05:23:25]

lgtm

[commit-bot: I haz the power, 2016-09-06 05:23:32]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-06 05:23:33]

Dry run: This issue passed the CQ dry run.

[shinyak, 2016-09-06 06:52:58]

The CQ bit was checked by shinyak@chromium.org

[shinyak, 2016-09-06 06:52:58]

The patchset sent to the CQ was uploaded after l-g-t-m from tikuta@chromium.org,
ukai@chromium.org
Link to the patchset: https://codereview.chromium.org/2302453004/#ps80001
(title: "Use bool for clobber field")

[commit-bot: I haz the power, 2016-09-06 06:53:01]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-06 06:57:37]

Description was changed from

==========
Add buildbot metadata to compiler_proxy log

We'd like to analyze compiler proxy log. However, buildbot metadata is
missing, so sometimes hard to determine in which builder the compiler
proxy log is generated.

Append buildbot metadata to compiler_proxy log like ninja log.
==========

to

==========
Add buildbot metadata to compiler_proxy log

We'd like to analyze compiler proxy log. However, buildbot metadata is
missing, so sometimes hard to determine in which builder the compiler
proxy log is generated.

Append buildbot metadata to compiler_proxy log like ninja log.

Committed:
https://chromium.googlesource.com/chromium/tools/build/+/3d1cf68904cab7a23836...
==========

[commit-bot: I haz the power, 2016-09-06 06:57:38]

Committed patchset #5 (id:80001) as
https://chromium.googlesource.com/chromium/tools/build/+/3d1cf68904cab7a23836...

[shinyak, 2016-09-06 07:14:25]

A revert of this CL (patchset #5 id:80001) has been created in
https://codereview.chromium.org/2318493002/ by shinyak@chromium.org.

The reason for reverting is: gsutil.py -h shows help.
https://uberchromegw.corp.google.com/i/tryserver.chromium.linux/builders/linu....