[infra] Roll clang to match the version used by Flutter

[infra] Roll clang to match the version used by Flutter

Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.

The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.

This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.

As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.

This change is a prerequisite for assembling the SDK in GN
rather than with a python script:

https://codereview.chromium.org/2848943003/

and will be nicer for making prebuilt SDKs for Fuchsia.

R=whesse@google.com

Committed: https://github.com/dart-lang/sdk/commit/cc968ca45b8b1dbbccaf630ef5d903f7925d6d41

[zra, 2017-05-01 06:44:47]

Description was changed from

==========
[infra] Roll clang to match version used by Flutter
==========

to

==========
[infra] Roll clang to match the version used by Flutter

Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.

The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.

This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.

As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.
==========

[zra, 2017-05-01 15:06:33]

Description was changed from

==========
[infra] Roll clang to match the version used by Flutter

Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.

The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.

This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.

As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.
==========

to

==========
[infra] Roll clang to match the version used by Flutter

Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.

The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.

This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.

As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.

This change is a prerequisite for assembling the SDK in GN
rather than with a python script:

https://codereview.chromium.org/2848943003/

and will be nicer for making prebuilt SDKs for Fuchsia.
==========

[zra, 2017-05-01 15:06:49]

zra@google.com changed reviewers:
+ asiva@google.com, whesse@google.com

[zra, 2017-05-03 14:49:35]

ping

[Bill Hesse, 2017-05-03 16:05:34]

LGTM, with comments.

(I see that GN is now downloaded by the new scripts, and I see that the android
pulls down only a trimmed subset, and only on linux and macos).

https://codereview.chromium.org/2854583002/diff/30001/DEPS
File DEPS (left):

https://codereview.chromium.org/2854583002/diff/30001/DEPS#oldcode369
DEPS:369: 'download_from_google_storage',
Is GN now downloaded by the clang or buildtools actions?

https://codereview.chromium.org/2854583002/diff/30001/DEPS
File DEPS (right):

https://codereview.chromium.org/2854583002/diff/30001/DEPS#newcode484
DEPS:484: 'action': ['python', 'sdk/tools/android/download_android_tools.py'],
Is this going to pull down all the NDK and SDK and more in every checkout? 
Before this only happened if OS=android was added at the end of the gclient
file.

If so, explain the tradeoffs and benefits.

https://codereview.chromium.org/2854583002/diff/30001/tools/buildtools/update.py
File tools/buildtools/update.py (right):

https://codereview.chromium.org/2854583002/diff/30001/tools/buildtools/update...
tools/buildtools/update.py:25: path = os.path.join(BUILDTOOLS, 'update.sh')
I don't see tools/buildtools/update.sh being added by this CL, or present
already, or in the DEPS checkout of chromium/buildtools.  Where does it come
from?

[Bill Hesse, 2017-05-03 16:06:48]

https://codereview.chromium.org/2854583002/diff/30001/DEPS
File DEPS (left):

https://codereview.chromium.org/2854583002/diff/30001/DEPS#oldcode369
DEPS:369: 'download_from_google_storage',
On 2017/05/03 16:05:34, Bill Hesse wrote:
> Is GN now downloaded by the clang or buildtools actions?

Yes it is.

https://codereview.chromium.org/2854583002/diff/30001/DEPS
File DEPS (right):

https://codereview.chromium.org/2854583002/diff/30001/DEPS#newcode484
DEPS:484: 'action': ['python', 'sdk/tools/android/download_android_tools.py'],
On 2017/05/03 16:05:34, Bill Hesse wrote:
> Is this going to pull down all the NDK and SDK and more in every checkout? 
> Before this only happened if OS=android was added at the end of the gclient
> file.
> 
> If so, explain the tradeoffs and benefits.

It only downloads a trimmed set, so OK.

[zra, 2017-05-03 17:34:41]

https://codereview.chromium.org/2854583002/diff/30001/DEPS
File DEPS (right):

https://codereview.chromium.org/2854583002/diff/30001/DEPS#newcode484
DEPS:484: 'action': ['python', 'sdk/tools/android/download_android_tools.py'],
On 2017/05/03 16:05:34, Bill Hesse wrote:
> Is this going to pull down all the NDK and SDK and more in every checkout? 
> Before this only happened if OS=android was added at the end of the gclient
> file.
> 
> If so, explain the tradeoffs and benefits.

Improvements:
- Downloads of a stripped-down blob instead of cloning a big git repo.
- Same version that Flutter uses.
- Don't need to edit the .gclient file to grab it.

Possible downsides:
- Pulled even for folks not work on Android issues.
  - However, download is only one-time, and now also works for Mac, so makes
more sense to pull it by default.

https://codereview.chromium.org/2854583002/diff/30001/tools/buildtools/update.py
File tools/buildtools/update.py (right):

https://codereview.chromium.org/2854583002/diff/30001/tools/buildtools/update...
tools/buildtools/update.py:25: path = os.path.join(BUILDTOOLS, 'update.sh')
On 2017/05/03 16:05:34, Bill Hesse wrote:
> I don't see tools/buildtools/update.sh being added by this CL, or present
> already, or in the DEPS checkout of chromium/buildtools.  Where does it come
> from?

It's not in chromium/buildtools. It's in fuchsia's buildtools here:

https://fuchsia.googlesource.com/buildtools/+/master/update.sh

which is what Flutter is using.

[zra, 2017-05-03 21:51:04]

Description was changed from

==========
[infra] Roll clang to match the version used by Flutter

Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.

The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.

This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.

As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.

This change is a prerequisite for assembling the SDK in GN
rather than with a python script:

https://codereview.chromium.org/2848943003/

and will be nicer for making prebuilt SDKs for Fuchsia.
==========

to

==========
[infra] Roll clang to match the version used by Flutter

Flutter uses the same clang toolchain as Fuchsia. This CL puts Dart
on that toolchain, as well. This roll should entail no changes to
glibc version requirements on Linux.

The Fuchsia buildtools distribute clang-format with the toolchain
rather than separately, so this CL introduces forwarding scripts
that are copied to the right place under //buildtools.

This CL sets the default for the number of workers in //tools/gn.py
to 1, as the newer GN version fails occasionally when run
concurrently with other invocations.

As Flutter does, this CL uses the clang toolchain to build for
Android. Thus, it is now possible to build for Android on MacOS.

This change is a prerequisite for assembling the SDK in GN
rather than with a python script:

https://codereview.chromium.org/2848943003/

and will be nicer for making prebuilt SDKs for Fuchsia.

R=whesse@google.com

Committed:
https://github.com/dart-lang/sdk/commit/cc968ca45b8b1dbbccaf630ef5d903f7925d6d41
==========

[zra, 2017-05-03 21:51:05]

Committed patchset #3 (id:30001) manually as
cc968ca45b8b1dbbccaf630ef5d903f7925d6d41 (presubmit successful).

[zra, 2017-05-03 22:59:51]

On 2017/05/03 21:51:05, zra wrote:
> Committed patchset #3 (id:30001) manually as
> cc968ca45b8b1dbbccaf630ef5d903f7925d6d41 (presubmit successful).

I tried landing this and failed. It looks like the updated GN binary doesn't
like being invoked on an output directory that has stuff generated by an older
GN binary.

Bill, is there anyway to make the buildbots clobber their output directory at
the same time as this is landed?

[jamesr1, 2017-05-05 22:29:59]

What platforms + architectures do you want to target using this toolchain?

[zra, 2017-05-06 04:58:39]

On 2017/05/05 22:29:59, jamesr1 wrote:
> What platforms + architectures do you want to target using this toolchain?

Linux x64 (and hopefully ia32 at some point), Mac x64 and ia32, Android arm and
arm64.

phosek and I discussed this change last week, but we can also talk more next
week if needed.