Extend relocation packing to cover arm64.

Extend relocation packing to cover arm64.

Add new delta encoder and signed LEB128 classes to pack relocations with
addends.  Add associated unit tests.

Add packer functions to handle vectors of ELF::Rela structures
(relocations with addends).  Update unit tests.

Templatize elf_file functions to provide versions that handle both
ELF::Rel and ELF::Rela.

Provide 'golden' 64 bit ARM unpacked and packed test data for
elf_file_unittest.  Add gyp targets, actions, and a helper python
script to easily produce these test data files from source using NDK
toolchains.

Tidy terminology in comments, update --help output, and update the
README.TXT.

BUG=385553
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=286503

[rmcilroy, 2014-07-28 10:08:44]

lgtm, thanks!

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/README.TXT (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/README.TXT:8: Packing uses combinations of run length
encoding, delta encoding, and LEB128
nit - s/combinations/a combination

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/README.TXT:35: DT_ANDROID_ARM_REL_OFFSET = DT_LOPROC   
(Processor specific: 0x70000000)
/s/DT_ANDROID_ARM_REL_OFFSET/DT_ANDROID_REL_OFFSET (and for
DT_ANDROID_ARM_REL_SIZE)

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/README.TXT:83: emit R_ARM_RELATIVE relocation with
r_offset = addr, r_addend = addend
R_AARCH64_RELATIVE

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/delta_encoder.cc (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/delta_encoder.cc:29: for (size_t i = 0; i <
relocations.size(); ++i) {
Could you add an assert here that the relocations[i].r_info is as expected.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/delta_encoder.h (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/delta_encoder.h:34: // addends is sometimes small,
four of eight bytes are almost always
"small - four of the eight bytes are almost always unused - and the..."

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/elf_file.cc (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/elf_file.cc:867: hole_size -=
sizeof(relative_relocations[0]);
I'm interested as to why the previous approach didn't work here, were you ending
up with negative hole sizes?  Wouldn't it still provide the same result if you
kept this line the same and only change the hole_size <= 0 below?

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/packer.cc (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/packer.cc:91: packed->insert(packed->begin(),
identifier.begin(), identifier.end());
nit - can you insert the identifier before  sleb128 encoding the relocations?

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/packer.h (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/packer.h:8: // For relative relocations without
addends (32 bit platforms):
optional nit - underline here with '-------...'

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/packer.h:26: // For relative relocations with
addends (64 bit platforms):
ditto

[simonb (inactive), 2014-07-28 12:20:56]

Thanks.  Items addressed in patch set 6.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/README.TXT (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/README.TXT:8: Packing uses combinations of run length
encoding, delta encoding, and LEB128
On 2014/07/28 10:08:44, rmcilroy wrote:
> nit - s/combinations/a combination

Done.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/README.TXT:35: DT_ANDROID_ARM_REL_OFFSET = DT_LOPROC   
(Processor specific: 0x70000000)
On 2014/07/28 10:08:44, rmcilroy wrote:
> /s/DT_ANDROID_ARM_REL_OFFSET/DT_ANDROID_REL_OFFSET (and for
> DT_ANDROID_ARM_REL_SIZE)

Updated in patch set 4 (you're commenting here on a mildly out of date patch
set).

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/README.TXT:83: emit R_ARM_RELATIVE relocation with
r_offset = addr, r_addend = addend
On 2014/07/28 10:08:44, rmcilroy wrote:
> R_AARCH64_RELATIVE

Done.  Good catch, thanks.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/delta_encoder.cc (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/delta_encoder.cc:29: for (size_t i = 0; i <
relocations.size(); ++i) {
On 2014/07/28 10:08:44, rmcilroy wrote:
> Could you add an assert here that the relocations[i].r_info is as expected.

Done.  Also added similar to run_length_encoder.cc.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/delta_encoder.h (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/delta_encoder.h:34: // addends is sometimes small,
four of eight bytes are almost always
On 2014/07/28 10:08:44, rmcilroy wrote:
> "small - four of the eight bytes are almost always unused - and the..."

Reworded for clarity.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/elf_file.cc (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/elf_file.cc:867: hole_size -=
sizeof(relative_relocations[0]);
On 2014/07/28 10:08:44, rmcilroy wrote:
> I'm interested as to why the previous approach didn't work here, were you
ending
> up with negative hole sizes?  Wouldn't it still provide the same result if you
> kept this line the same and only change the hole_size <= 0 below?

We pad using an integral count of R_NONE relocations.  kPreserveAlignment is
256.  An arm32 relocation is 8 bytes and a factor of 256.  However, an arm64
relocation with addend is 24 bytes, and not a factor of 256.

For example, removing 11 arm64 relocations produces a 264 byte unaligned_hole
size.  264-264%256 is 12, but we cannot generate a 12-byte padding using 24-byte
R_ARM_NONE arm64 relocations with addends.

I have removed the checks for hole_size < 0.  This cannot occur -- hole_size is
by definition a whole number of relative relocation sizes, and
hole_size%anything is zero when hole_size, sequentially reduced by sizeof a
relocation, reaches zero.  (The previous approach could be considered brittle;
it relied on kPreserveAlignment always being divisible by the size in bytes of a
relocation.)

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/packer.cc (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/packer.cc:91: packed->insert(packed->begin(),
identifier.begin(), identifier.end());
On 2014/07/28 10:08:44, rmcilroy wrote:
> nit - can you insert the identifier before  sleb128 encoding the relocations?

Done.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
File tools/relocation_packer/src/packer.h (right):

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/packer.h:8: // For relative relocations without
addends (32 bit platforms):
On 2014/07/28 10:08:44, rmcilroy wrote:
> optional nit - underline here with '-------...'

Done.

https://codereview.chromium.org/410933004/diff/40001/tools/relocation_packer/...
tools/relocation_packer/src/packer.h:26: // For relative relocations with
addends (64 bit platforms):
On 2014/07/28 10:08:44, rmcilroy wrote:
> ditto

Done.

[rmcilroy, 2014-07-28 15:33:40]

lgtm

[simonb (inactive), 2014-07-28 17:29:48]

The CQ bit was checked by simonb@chromium.org

[commit-bot: I haz the power, 2014-07-28 17:31:48]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/simonb@chromium.org/410933004/100001

[commit-bot: I haz the power, 2014-07-28 20:37:15]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  android_clang_dbg on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/android_clang_d...)
  chromium_presubmit on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)
  linux_chromium_chromeos_rel on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_rel_swarming on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[commit-bot: I haz the power, 2014-07-28 20:45:43]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-07-28 20:45:44]

Try jobs failed on following builders:
  android_clang_dbg on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/android_clang_d...)
  chromium_presubmit on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[simonb (inactive), 2014-07-30 10:10:20]

Unit test shared libraries landed separately:

https://codereview.chromium.org/427793002/

[simonb (inactive), 2014-07-30 10:11:22]

The CQ bit was checked by simonb@chromium.org

[commit-bot: I haz the power, 2014-07-30 10:12:21]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/simonb@chromium.org/410933004/140001

[commit-bot: I haz the power, 2014-07-30 13:59:49]

Change committed as 286503