upload clang pdbs to chrome's symbol server

tools/clang/scripts/package.py

 #!/usr/bin/env python
 # Copyright (c) 2015 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 """This script will check out llvm and clang, and then package the results up
 to a tgz file."""
 
 import argparse
 import fnmatch
@@ skipping 283 matching lines @@
   objdumpdir = 'llvmobjdump-' + stamp
   shutil.rmtree(objdumpdir, ignore_errors=True)
   os.makedirs(os.path.join(objdumpdir, 'bin'))
   shutil.copy(os.path.join(LLVM_RELEASE_DIR, 'bin', 'llvm-objdump' + exe_ext),
               os.path.join(objdumpdir, 'bin'))
   with tarfile.open(objdumpdir + '.tgz', 'w:gz') as tar:
     tar.add(os.path.join(objdumpdir, 'bin'), arcname='bin',
             filter=PrintTarProgress)
   MaybeUpload(args, objdumpdir, platform)
 
+  # Zip up PDB file on Windows.
+  pdbdir = 'clangpdb-' + stamp

[Nico, 2017/02/21 23:10:11]

I have many questions about this part.

1. The bug says https://chromium-browser-symsrv.commondatastorage.googleapis.com
. This currently puts it in gs://chromium-browser-clang/win/clangpdb-1234.tgz Is
chromium-browser-symsrv better? If so, which exact file name should I use there?

2. As far as I know, the upstream cmake files basically build bin/clang.exe and
then copy it to bin/clang-cl.exe. We only package up clang-cl.exe. However, the
pdb gets created at bin/clang.pdb (due to the upstream setup).
a) Should we copy clang.pdb to clang-cl.pdb before zipping it up?
b) Is the name of a pdb embedded into the pdb?
c) Does it matter that .exe and its .pdb have the same basename (ignoring
extension)?

3. This currently uses tgz -- is that good, or is zip better, or something else
altogether? (All the other archives package.py currently creates are tgz, but it
doesn't matter too much what this one uses as long as python can write it)

clang.pdb is a bit over 400 MB before zipping.

[scottmg, 2017/02/21 23:38:49]

Yes, chromium-browser-symsrv is better because Chrome devs will have that in
their _NT_SYMBOL_PATH, so tools will just work (msvs, windbg, etw, etc.)

If you look at the the official.desktop builder on uberchrome in the "official
symbols" step, that's what you want to do.

That step sneakily does two things, the first is to use Breakpad dump_syms and
upload a text version for the crash server.

The second down near the bottom half of the logs (I just learned this now) is to
push the raw pdbs and the binary into the gs bucket.

The second part is what we'd want to do here for clang.

So, you'd want to not tar it up, and also upload the binary, which allows
debugging crash dumps.

The target paths are finicky. Taking chrome_child.dll as an example from a
recent log https://goto.google.com/drevv

gs://chromium-browser-symsrv/chrome_child.dll/58ACAFB932d5000/chrome_child.dll 

That hex string is a concatenation of two things

dumpbin /headers chrome_child.dll | grep "date stamp"
and
dumpbin /headers chrome_child.dll | grep "size of image"

both of those numbers formatted as hex. There's something fiddly about this that
I don't quite remember, I think the first ones is %08X and the second is %x
(maybe? +Seb and +Bruce who might remember).

Secondarily, the PDB. It goes into something like

gs://chromium-browser-symsrv/chrome_child.dll.pdb/F71B007136FC43EFB957AB08B2E150441/chrome_child.dll.pd_

*That* hex string is the RSDS GUID from the binary (dumpbin /all mybinary.exe |
grep "Format: RSDS") concatenated with the "age" of the RSDS record (which is
the number after the comma following the GUID), so it'll generally be 32+1
characters long, but occasionally longer.

Whew, did you get all that?

+  if sys.platform == 'win32':
+    shutil.rmtree(pdbdir, ignore_errors=True)
+    shutil.copy(os.path.join(LLVM_RELEASE_DIR, 'bin', 'clang.pdb'), pdbdir)
+    with tarfile.open(pdbdir + '.tgz', 'w:gz') as tar:
+      tar.add(pdbdir, filter=PrintTarProgress)
+    MaybeUpload(args, pdbdir, platform)
+
   # FIXME: Warn if the file already exists on the server.
 
 
 if __name__ == '__main__':
   sys.exit(main())