Add --enable-pgo option to build clang with PGO

Add --enable-pgo option to build clang with PGO

With this flag clang would run a multistage pgo build of clang.
As a result clang being built with PGO compiles chromium about 15-20% faster for me. More details could be found in thread https://groups.google.com/a/chromium.org/forum/?utm_source=digest&utm_medium=email#!topic/chromium-dev/A3pWgkaSNrw

BUG=663318

[matthewtff.asm, 2017-04-04 15:19:53]

Description was changed from

==========
Add --enable-pgo option to build clang with PGO

BUG=663318
==========

to

==========
Add --enable-pgo option to build clang with PGO

With this flag clang would run a multistage pgo build of clang.
As a result clang being built with PGO compiles chromium about 15-20% faster for
me. More details could be found in thread
https://groups.google.com/a/chromium.org/forum/?utm_source=digest&utm_medium=...

BUG=663318
==========

[matthewtff.asm, 2017-04-04 15:19:53]

matthewtff@yandex-team.ru changed reviewers:
+ thakis@chromium.org

[matthewtff.asm, 2017-04-04 15:22:18]

Nico, could you take a look at this approach, please?
I'm not sure if I was able to see through code all paths chromium uses to build
llvm tools. Also I was unable to build clang with pgo on windows as such build
requires compiler-rt project, and that one AFAICT cannot be built with clang-cl
for now :(

[Reid Kleckner, 2017-04-04 15:37:29]

rnk@chromium.org changed reviewers:
+ rnk@chromium.org

[Reid Kleckner, 2017-04-04 15:37:30]

Does this include automation of the training data collection process, or is that
still up to the user?

https://codereview.chromium.org/2793343002/diff/1/tools/clang/scripts/update.py
File tools/clang/scripts/update.py (right):

https://codereview.chromium.org/2793343002/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:928: print '--bootstrap makes no sence with
--enable-pgo specified'
s/sence/sense/, here and below

https://codereview.chromium.org/2793343002/diff/1/tools/clang/scripts/update....
tools/clang/scripts/update.py:934: print '--enable-pgo makes no sence on
Windows'
Maybe "--enable-pgo not supported on Windows"?

[matthewtff.asm, 2017-04-04 15:48:36]

On 2017/04/04 15:37:30, Reid Kleckner wrote:
> Does this include automation of the training data collection process, or is
that
> still up to the user?
This includes automation of the training data collection process. As described
in llvm docs:
http://llvm.org/docs/AdvancedBuilds.html#multi-stage-pgo
this build uses sample data from llvm(clang) repo.

>
https://codereview.chromium.org/2793343002/diff/1/tools/clang/scripts/update.py
> File tools/clang/scripts/update.py (right):
> 
>
https://codereview.chromium.org/2793343002/diff/1/tools/clang/scripts/update....
> tools/clang/scripts/update.py:928: print '--bootstrap makes no sence with
> --enable-pgo specified'
> s/sence/sense/, here and below
Done, thanks.

>
https://codereview.chromium.org/2793343002/diff/1/tools/clang/scripts/update....
> tools/clang/scripts/update.py:934: print '--enable-pgo makes no sence on
> Windows'
> Maybe "--enable-pgo not supported on Windows"?
Done

[matthewtff.asm, 2017-04-07 10:23:43]

Gentle ping

[Nico, 2017-04-07 15:58:53]

Sorry, I missed this. Thanks for working on this, this is cool stuff.

Two high-level things:

1. You have a couple small unrelated fixes in this CL – can you send those bits
in separate patches?

2. This forks the behavior of this script up a lot since we now use a manual
bootstrap on Windows but the upstream cmake magic for PGO. Is it possible to use
some upstream cmake thingie for bootstrap on Windows too, so that this is more
unified across platforms?

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/CMakeLists.txt
File tools/clang/CMakeLists.txt (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/CMakeLists....
tools/clang/CMakeLists.txt:61: if (CHROMIUM_TOOLS)
why this change? seems independent of the rest of the patch (?)

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/PGO-stage2....
File tools/clang/PGO-stage2.cmake (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/PGO-stage2....
tools/clang/PGO-stage2.cmake:10: set(ENABLE_LINKER_BUILD_ID ON CACHE BOOL "")
I have a patch to remove this at https://codereview.chromium.org/2784423003/ –
you don't need this.

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/pac...
File tools/clang/scripts/package.py (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/pac...
tools/clang/scripts/package.py:167: help='Build clang with GO enabled.')
Typo GO.

Also, why have a flag for this? Shouldn't this just be the default?

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/pac...
tools/clang/scripts/package.py:192: GsutilArchiveExists(golddir, platform))):
This looks like a good change, but also unrelated.

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
File tools/clang/scripts/update.py (left):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
tools/clang/scripts/update.py:286: f.write('# dirs, so the build artifacts need
to go into a subdirectory.\n')
Also a good change, but also unrelated.

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
File tools/clang/scripts/update.py (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
tools/clang/scripts/update.py:525: os.path.join(LLVM_DIR, 'projects',
'compiler-rt'))
IIRC package.py packages based on blacklists instead of whitelists – have you
checked we don't bundle more stuff due to this being present?

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
tools/clang/scripts/update.py:601: if args.bootstrap and args.lto_gold_plugin:
This is now never true (assuming we default to PGO) since we only pass
--bootstrap on Windows and lto_gold_plugin is only true on Linux.

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
tools/clang/scripts/update.py:619: if not args.enable_pgo:
Why this change?

[matthewtff.asm, 2017-04-11 12:14:43]

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/CMakeLists.txt
File tools/clang/CMakeLists.txt (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/CMakeLists....
tools/clang/CMakeLists.txt:61: if (CHROMIUM_TOOLS)
On 2017/04/07 15:58:53, Nico wrote:
> why this change? seems independent of the rest of the patch (?)

We do not pass -DBOOTSTRAP_CHROMIUM_TOOLS to cmake,
so it fails to build instrumented clang(which is between bootstrapped clang and
stage2 clang).

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/PGO-stage2....
File tools/clang/PGO-stage2.cmake (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/PGO-stage2....
tools/clang/PGO-stage2.cmake:10: set(ENABLE_LINKER_BUILD_ID ON CACHE BOOL "")
On 2017/04/07 15:58:53, Nico wrote:
> I have a patch to remove this at https://codereview.chromium.org/2784423003/ –
> you don't need this.

I'll remove this one, no problems.

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/pac...
File tools/clang/scripts/package.py (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/pac...
tools/clang/scripts/package.py:167: help='Build clang with GO enabled.')
On 2017/04/07 15:58:53, Nico wrote:
> Typo GO.
> 
> Also, why have a flag for this? Shouldn't this just be the default?

I believe it should be default. Made a flag just in case.

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
File tools/clang/scripts/update.py (right):

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
tools/clang/scripts/update.py:525: os.path.join(LLVM_DIR, 'projects',
'compiler-rt'))
On 2017/04/07 15:58:53, Nico wrote:
> IIRC package.py packages based on blacklists instead of whitelists – have
you
> checked we don't bundle more stuff due to this being present?

The only diff I've got is one header file:
lib/clang/5.0.0/include/xray/xray_interface.h

I beleive it's ok to bundle this one since it's size is 3KB

https://codereview.chromium.org/2793343002/diff/20001/tools/clang/scripts/upd...
tools/clang/scripts/update.py:619: if not args.enable_pgo:
On 2017/04/07 15:58:53, Nico wrote:
> Why this change?

We need to build clang with libcxx checked out in tree to get standart include
files. Previously those used to be copied from bootstrap build, but now it's no
longer possible.
Actually 4 minutes are no longer an issue as with PGO build takes much more than
30min :(

[matthewtff.asm, 2017-04-11 12:21:18]

On 2017/04/07 15:58:53, Nico wrote:
> Sorry, I missed this. Thanks for working on this, this is cool stuff.
> 
> Two high-level things:
> 
> 1. You have a couple small unrelated fixes in this CL – can you send
those bits
> in separate patches?
Sent fixes in separate patches, but you already know this.
> 
> 2. This forks the behavior of this script up a lot since we now use a manual
> bootstrap on Windows but the upstream cmake magic for PGO. Is it possible to
use
> some upstream cmake thingie for bootstrap on Windows too, so that this is more
> unified across platforms?
I'll try to spend some time today to find a way to use DistributionExample.cmake
cache
form LLVM/clang to build on windows same way as on linux & mac.

Sorry for two emails in a raw, I'm still unable to find one place to reply in
thread and publish
all my drafts.

[matthewtff.asm, 2017-05-03 17:05:50]

Back again.

I've been able to build clang on windows in two stages using cmake caches with
this changes in clang files:

Diff in llvm/tools/clang:
M       third_party\llvm\tools\clang\cmake\caches\DistributionExample.cmake

M       third_party\llvm\tools\clang\CMakeLists.txt

Index: third_party/llvm/tools/clang/cmake/caches/DistributionExample.cmake

===================================================================

---
third_party/llvm/tools/clang/cmake/caches/DistributionExample.cmake	(revision
298539)

+++ third_party/llvm/tools/clang/cmake/caches/DistributionExample.cmake	(working
copy)

@@ -29,6 +29,13 @@

 
 # Setup the bootstrap build.
 set(CLANG_ENABLE_BOOTSTRAP ON CACHE BOOL "")
-set(CLANG_BOOTSTRAP_CMAKE_ARGS
-  -C ${CMAKE_CURRENT_LIST_DIR}/DistributionExample-stage2.cmake
-  CACHE STRING "")
+
+if(STAGE2_CACHE_FILE)
+  set(CLANG_BOOTSTRAP_CMAKE_ARGS
+    -C ${STAGE2_CACHE_FILE}
+    CACHE STRING "")
+else()
+  set(CLANG_BOOTSTRAP_CMAKE_ARGS
+    -C ${CMAKE_CURRENT_LIST_DIR}/DistributionExample-stage2.cmake
+    CACHE STRING "")
+endif()
Index: third_party/llvm/tools/clang/CMakeLists.txt

===================================================================

--- third_party/llvm/tools/clang/CMakeLists.txt	(revision 298539)

+++ third_party/llvm/tools/clang/CMakeLists.txt	(working copy)

@@ -570,10 +570,17 @@

     add_dependencies(clang-bootstrap-deps compiler-rt)
   endif()
 
+  set(C_COMPILER_NAME "clang")
+  set(CXX_COMPILER_NAME "clang++")
+  if(WIN32)
+    set(C_COMPILER_NAME "clang-cl.exe")
+    set(CXX_COMPILER_NAME "clang-cl.exe")
+  endif()
+
   set(COMPILER_OPTIONS
-    -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++
-    -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang
-    -DCMAKE_ASM_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang)
+    -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/${CXX_COMPILER_NAME}
+    -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/${C_COMPILER_NAME}
+    -DCMAKE_ASM_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/${C_COMPILER_NAME})
 
   if(BOOTSTRAP_LLVM_BUILD_INSTRUMENTED)
     add_dependencies(clang-bootstrap-deps llvm-profdata)


Those look reasonable for me to put them upstream, so first I'll try to land
them there.
Than, I hope, we could proceed with this CL after a clang roll with changes in
upstream.

[Nico, 2017-05-03 17:34:11]

Sounds great!

On Wed, May 3, 2017 at 1:05 PM, <matthewtff@yandex-team.ru> wrote:

> Back again.
>
> I've been able to build clang on windows in two stages using cmake caches
> with
> this changes in clang files:
>
> Diff in llvm/tools/clang:
> M third_party\llvm\tools\clang\cmake\caches\DistributionExample.cmake
>
> M third_party\llvm\tools\clang\CMakeLists.txt
>
> Index: third_party/llvm/tools/clang/cmake/caches/DistributionExample.cmake
>
> ===================================================================
>
> ---
> third_party/llvm/tools/clang/cmake/caches/DistributionExample.cmake
> (revision
> 298539)
>
> +++ third_party/llvm/tools/clang/cmake/caches/DistributionExample.cmake
> (working
> copy)
>
> @@ -29,6 +29,13 @@
>
>
> # Setup the bootstrap build.
> set(CLANG_ENABLE_BOOTSTRAP ON CACHE BOOL "")
> -set(CLANG_BOOTSTRAP_CMAKE_ARGS
> - -C ${CMAKE_CURRENT_LIST_DIR}/DistributionExample-stage2.cmake
> - CACHE STRING "")
> +
> +if(STAGE2_CACHE_FILE)
> + set(CLANG_BOOTSTRAP_CMAKE_ARGS
> + -C ${STAGE2_CACHE_FILE}
> + CACHE STRING "")
> +else()
> + set(CLANG_BOOTSTRAP_CMAKE_ARGS
> + -C ${CMAKE_CURRENT_LIST_DIR}/DistributionExample-stage2.cmake
> + CACHE STRING "")
> +endif()
> Index: third_party/llvm/tools/clang/CMakeLists.txt
>
> ===================================================================
>
> --- third_party/llvm/tools/clang/CMakeLists.txt (revision 298539)
>
> +++ third_party/llvm/tools/clang/CMakeLists.txt (working copy)
>
> @@ -570,10 +570,17 @@
>
> add_dependencies(clang-bootstrap-deps compiler-rt)
> endif()
>
> + set(C_COMPILER_NAME "clang")
> + set(CXX_COMPILER_NAME "clang++")
> + if(WIN32)
> + set(C_COMPILER_NAME "clang-cl.exe")
> + set(CXX_COMPILER_NAME "clang-cl.exe")
> + endif()
> +
> set(COMPILER_OPTIONS
> - -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++
> - -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang
> - -DCMAKE_ASM_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang)
> + -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/${CXX_COMPILER_NAME}
> + -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/${C_COMPILER_NAME}
> + -DCMAKE_ASM_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/${C_COMPILER_NAME})
>
> if(BOOTSTRAP_LLVM_BUILD_INSTRUMENTED)
> add_dependencies(clang-bootstrap-deps llvm-profdata)
>
>
> Those look reasonable for me to put them upstream, so first I'll try to
> land
> them there.
> Than, I hope, we could proceed with this CL after a clang roll with
> changes in
> upstream.
>
>
> https://codereview.chromium.org/2793343002/
>

-- 
You received this message because you are subscribed to the Google Groups
"Chromium-reviews" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[matthewtff.asm, 2017-05-11 13:56:58]

Landed as r302795:
https://github.com/llvm-mirror/clang/commit/c68ab69fbb44fccb8c454872d93883677...

Could you please notify me here as clang rolls past this commit?

[Nico, 2017-05-11 14:04:52]

https://crbug.com/714769 tracks updating clang. We've had some problems
with out distributed compiler service last week and haven't rolled last
week. Hopefully soon, but possibly first to an earlier revision.

On Thu, May 11, 2017 at 9:56 AM, <matthewtff@yandex-team.ru> wrote:

> Landed as r302795:
> https://github.com/llvm-mirror/clang/commit/c68ab69fbb44fccb8c454872d93883
> 677b2a0cb5
>
> Could you please notify me here as clang rolls past this commit?
>
>
> https://codereview.chromium.org/2793343002/
>

-- 
You received this message because you are subscribed to the Google Groups
"Chromium-reviews" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[Nico, 2017-05-19 19:16:14]

Yesterday's clang roll brought us past that commit, and so far it looks
like the roll might stick.

On Thu, May 11, 2017 at 10:04 AM, Nico Weber <thakis@chromium.org> wrote:

> https://crbug.com/714769 tracks updating clang. We've had some problems
> with out distributed compiler service last week and haven't rolled last
> week. Hopefully soon, but possibly first to an earlier revision.
>
> On Thu, May 11, 2017 at 9:56 AM, <matthewtff@yandex-team.ru> wrote:
>
>> Landed as r302795:
>> https://github.com/llvm-mirror/clang/commit/c68ab69fbb44fccb
>> 8c454872d93883677b2a0cb5
>>
>> Could you please notify me here as clang rolls past this commit?
>>
>>
>> https://codereview.chromium.org/2793343002/
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups
"Chromium-reviews" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[matthewtff.asm, 2017-05-19 19:19:22]

Yeah, thanks. I've subscribed to bug you gave, just wasn't sure that roll sticks
given that previous roll was reverted.
But I'm sure you know that kind of this better then me :)

Anyway thanks for notification. I'll update this CL on Monday and check if it
still works.

On 2017/05/19 19:16:14, Nico wrote:
> Yesterday's clang roll brought us past that commit, and so far it looks
> like the roll might stick.
> 
> On Thu, May 11, 2017 at 10:04 AM, Nico Weber <mailto:thakis@chromium.org>
wrote:
> 
> > https://crbug.com/714769 tracks updating clang. We've had some problems
> > with out distributed compiler service last week and haven't rolled last
> > week. Hopefully soon, but possibly first to an earlier revision.
> >
> > On Thu, May 11, 2017 at 9:56 AM, <mailto:matthewtff@yandex-team.ru> wrote:
> >
> >> Landed as r302795:
> >> https://github.com/llvm-mirror/clang/commit/c68ab69fbb44fccb
> >> 8c454872d93883677b2a0cb5
> >>
> >> Could you please notify me here as clang rolls past this commit?
> >>
> >>
> >> https://codereview.chromium.org/2793343002/
> >>
> >
> >
> 
> -- 
> You received this message because you are subscribed to the Google Groups
> "Chromium-reviews" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email
> to mailto:chromium-reviews+unsubscribe@chromium.org.

[matthewtff.asm, 2017-05-23 11:27:56]

So now tools/clang/stage2.cmake plays as a stage-2 cache file for both, PGO and
non-PGO(on windows) build.
Also manual bootstrapping should go away now.

Could you take a look, please?

[matthewtff.asm, 2017-05-25 08:24:13]

On 2017/05/23 11:27:56, matthewtff.asm wrote:
> So now tools/clang/stage2.cmake plays as a stage-2 cache file for both, PGO
and
> non-PGO(on windows) build.
> Also manual bootstrapping should go away now.
> 
> Could you take a look, please?

Guys, ping :)

[Nico, 2017-05-30 02:34:48]

+inglorion fyi (inglorion: maybe there's some way we could make the PGO build of
clang work on Windows?)

matthewtff: We have this long-term goal of a deterministic build of clang, so
that everyone who wanted could run this script locally and get the exact same
binary as the prebuilt one. PGO makes that more or less completely infeasible,
right?

[matthewtff.asm, 2017-05-30 03:02:51]

On 2017/05/30 02:34:48, Nico wrote:
> +inglorion fyi (inglorion: maybe there's some way we could make the PGO build
of
> clang work on Windows?)
> 
> matthewtff: We have this long-term goal of a deterministic build of clang, so
> that everyone who wanted could run this script locally and get the exact same
> binary as the prebuilt one. PGO makes that more or less completely infeasible,
> right?

Why do you say so? It makes build longer, true. But no longer than chromium
build, for certain.
I've ran this script locally lots of time while working on this CL.

[Nico, 2017-05-30 03:05:46]

On 2017/05/30 03:02:51, matthewtff.asm wrote:
> On 2017/05/30 02:34:48, Nico wrote:
> > +inglorion fyi (inglorion: maybe there's some way we could make the PGO
build
> of
> > clang work on Windows?)
> > 
> > matthewtff: We have this long-term goal of a deterministic build of clang,
so
> > that everyone who wanted could run this script locally and get the exact
same
> > binary as the prebuilt one. PGO makes that more or less completely
infeasible,
> > right?
> 
> Why do you say so? It makes build longer, true. But no longer than chromium
> build, for certain.
> I've ran this script locally lots of time while working on this CL.

Oh, I don't mean that the time becomes super long, but that the built clang
binary is less deterministic. In a regular build, you "only" have to make sure
that you don't put build paths in the produced binary, don't put things like
__DATE__ in there, etc. But with PGO, if the profiles are generated locally,
they will be machine-dependent, and then the final binary that uses the profiles
will then also be machine-dependent and usually not independent of the host box.

[matthewtff.asm, 2017-05-30 03:50:35]

On 2017/05/30 03:05:46, Nico wrote:
> On 2017/05/30 03:02:51, matthewtff.asm wrote:
> > On 2017/05/30 02:34:48, Nico wrote:
> > > +inglorion fyi (inglorion: maybe there's some way we could make the PGO
> build
> > of
> > > clang work on Windows?)
> > > 
> > > matthewtff: We have this long-term goal of a deterministic build of clang,
> so
> > > that everyone who wanted could run this script locally and get the exact
> same
> > > binary as the prebuilt one. PGO makes that more or less completely
> infeasible,
> > > right?
> > 
> > Why do you say so? It makes build longer, true. But no longer than chromium
> > build, for certain.
> > I've ran this script locally lots of time while working on this CL.
> 
> Oh, I don't mean that the time becomes super long, but that the built clang
> binary is less deterministic. In a regular build, you "only" have to make sure
> that you don't put build paths in the produced binary, don't put things like
> __DATE__ in there, etc. But with PGO, if the profiles are generated locally,
> they will be machine-dependent, and then the final binary that uses the
profiles
> will then also be machine-dependent and usually not independent of the host
box.

Could you please give an example of host-dependent profile guided optimization?
According to clang docs[1] profiles from instrumented code should give
reproducible results.
But that could mean *on same host*.... Also it mentions profiles combining from
different
hosts, that also lets me think that profiles should be host-independent.

1. https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation

[inglorion, 2017-05-30 18:20:16]

On 2017/05/30 02:34:48, Nico wrote:
> +inglorion fyi (inglorion: maybe there's some way we could make the PGO build
of
> clang work on Windows?)

This should already work. I added support for -fprofile-instr-generate and
-fprofile-instr-use to clang-cl in r288230. These are used by LLVM's
LLVM_BUILD_INSTRUMENTED and LLVM_PROFDATA_FILE to perform PGO instrumented and
optimized builds. If this doesn't work, please let me know.

> matthewtff: We have this long-term goal of a deterministic build of clang, so
> that everyone who wanted could run this script locally and get the exact same
> binary as the prebuilt one. PGO makes that more or less completely infeasible,
> right?

To my knowledge, the same inputs should produce the same outputs. For the PGO
optimized build, the inputs are the sources and the profile data. Different
profile data may result in different outputs, but I don't see how the profile
data generated in the same way using the same source revision would lead to
different optimized outputs. If you have concrete examples, we can figure out
how to deal with those, but I can't think of any off the top of my head.

Note: all of this applies to instrumentation-based profiling. Perhaps you're
thinking of sample-based profiling. I know less about that, but I'm inclined to
assume the sampling makes the collection of profile data non-deterministic.
However, matthewttf is using profile-based instrumentation here, unless I'm
mistaken.

[matthewtff.asm, 2017-05-30 22:55:50]

On 2017/05/30 18:20:16, inglorion wrote:
> On 2017/05/30 02:34:48, Nico wrote:
> > +inglorion fyi (inglorion: maybe there's some way we could make the PGO
build
> of
> > clang work on Windows?)
> 
> This should already work. I added support for -fprofile-instr-generate and
> -fprofile-instr-use to clang-cl in r288230. These are used by LLVM's
> LLVM_BUILD_INSTRUMENTED and LLVM_PROFDATA_FILE to perform PGO instrumented and
> optimized builds. If this doesn't work, please let me know.

I'll give it a try tomorrow. That would be great if it works(didn't work last
time I tried)!

> > matthewtff: We have this long-term goal of a deterministic build of clang,
so
> > that everyone who wanted could run this script locally and get the exact
same
> > binary as the prebuilt one. PGO makes that more or less completely
infeasible,
> > right?
> 
> To my knowledge, the same inputs should produce the same outputs. For the PGO
> optimized build, the inputs are the sources and the profile data. Different
> profile data may result in different outputs, but I don't see how the profile
> data generated in the same way using the same source revision would lead to
> different optimized outputs. If you have concrete examples, we can figure out
> how to deal with those, but I can't think of any off the top of my head.
> 
> Note: all of this applies to instrumentation-based profiling. Perhaps you're
> thinking of sample-based profiling. I know less about that, but I'm inclined
to
> assume the sampling makes the collection of profile data non-deterministic.
> However, matthewttf is using profile-based instrumentation here, unless I'm
> mistaken.

You're totally correct, it's profile-based instrumentation.