[Android] Add 64-bit support to memdump (and memory_inspector parser)

[Android] Add 64-bit support to memdump (and memory_inspector parser)

This change makes memdump able to parse and print out 64 bit addresses.
No change is intended for the 32-bit output.
This also updates the memdump parser in memory_inspector to match the
memdump change and updates its prebuilts.

BUG=340294
Committed: https://crrev.com/d7339a4b9655113f7b7babe05755fb6650ad80ec
Cr-Commit-Position: refs/heads/master@{#293927}

[Primiano Tucci (use gerrit), 2014-09-09 10:17:35]

Patchset #1 (id:1) has been deleted

[Primiano Tucci (use gerrit), 2014-09-09 10:24:54]

primiano@chromium.org changed reviewers:
+ skyostil@chromium.org

[Primiano Tucci (use gerrit), 2014-09-09 10:24:54]

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
File tools/memory_inspector/memory_inspector/core/memory_map.py (right):

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
tools/memory_inspector/memory_inspector/core/memory_map.py:8: PAGE_SIZE = 4096
This will be required (out of this module) soon by next CLs. Putting in this CL
just to avoid solving conflicts later.

[Sami, 2014-09-09 10:44:26]

lgtm with some nits and uneducated questions by yours truly.

https://codereview.chromium.org/550093003/diff/20001/tools/android/memdump/me...
File tools/android/memdump/memdump.cc (right):

https://codereview.chromium.org/550093003/diff/20001/tools/android/memdump/me...
tools/android/memdump/memdump.cc:50: // some hundred ms. when printing large
mmap-ed and non resident regions.
Trailing == most significant bits, hence likely to be zero? Just checking which
way around things are here :)

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
File tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py
(right):

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py:71:
continue  # Sadly, this happened. Probably a kernel bug, see b/17402069.
Should we log a warning?

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py:82:
entry.resident_pages = [ord(c) for c in base64.b64decode(m.group(11))]
Having 11 groups in a regex is probably an argument for using named groups or
using something other than a regex in the first place :) No need to block this
patch on that though.

[Primiano Tucci (use gerrit), 2014-09-09 11:11:39]

https://codereview.chromium.org/550093003/diff/20001/tools/android/memdump/me...
File tools/android/memdump/memdump.cc (right):

https://codereview.chromium.org/550093003/diff/20001/tools/android/memdump/me...
tools/android/memdump/memdump.cc:50: // some hundred ms. when printing large
mmap-ed and non resident regions.
On 2014/09/09 10:44:26, Sami wrote:
> Trailing == most significant bits, hence likely to be zero? Just checking
which
> way around things are here :)

Hmm no, least significant bytes, not bits.
Imagine a region of 32 pages, where pages 0-8,10-16 are resident.
The full bitmap would be:
11111111 11111100 00000000 00000000
The strip here would reduce this to:
11111111 11111100

The most common and interesting case is virtual address space reservations (i.e.
mmap(a_lot, PROT_NONE, ...) where all the bits are 0).

I'll reword the comment here.

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
File tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py
(right):

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py:71:
continue  # Sadly, this happened. Probably a kernel bug, see b/17402069.
On 2014/09/09 10:44:26, Sami wrote:
> Should we log a warning?

Yeah, probably it's a good idea.

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py:82:
entry.resident_pages = [ord(c) for c in base64.b64decode(m.group(11))]
On 2014/09/09 10:44:26, Sami wrote:
> Having 11 groups in a regex is probably an argument for using named groups or
> using something other than a regex in the first place :) No need to block this
> patch on that though.

Hmm suggestions for alternatives? I can't simply tokenize on spaces because the
mapped file name (the part in quotes) can contain spaces, e.g.,
1234-5678 .... "/dev/ashmem (deleted)" [AAAAA=].

[Sami, 2014-09-09 11:16:20]

https://codereview.chromium.org/550093003/diff/20001/tools/android/memdump/me...
File tools/android/memdump/memdump.cc (right):

https://codereview.chromium.org/550093003/diff/20001/tools/android/memdump/me...
tools/android/memdump/memdump.cc:50: // some hundred ms. when printing large
mmap-ed and non resident regions.
On 2014/09/09 11:11:38, Primiano Tucci wrote:
> On 2014/09/09 10:44:26, Sami wrote:
> > Trailing == most significant bits, hence likely to be zero? Just checking
> which
> > way around things are here :)
> 
> Hmm no, least significant bytes, not bits.
> Imagine a region of 32 pages, where pages 0-8,10-16 are resident.
> The full bitmap would be:
> 11111111 11111100 00000000 00000000
> The strip here would reduce this to:
> 11111111 11111100
> 
> The most common and interesting case is virtual address space reservations
(i.e.
> mmap(a_lot, PROT_NONE, ...) where all the bits are 0).
> 
> I'll reword the comment here.

Ah, somehow I was confusing this with addresses instead of a residency bit set.

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
File tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py
(right):

https://codereview.chromium.org/550093003/diff/20001/tools/memory_inspector/m...
tools/memory_inspector/memory_inspector/backends/android/memdump_parser.py:82:
entry.resident_pages = [ord(c) for c in base64.b64decode(m.group(11))]
On 2014/09/09 11:11:39, Primiano Tucci wrote:
> On 2014/09/09 10:44:26, Sami wrote:
> > Having 11 groups in a regex is probably an argument for using named groups
or
> > using something other than a regex in the first place :) No need to block
this
> > patch on that though.
> 
> Hmm suggestions for alternatives? I can't simply tokenize on spaces because
the
> mapped file name (the part in quotes) can contain spaces, e.g.,
> 1234-5678 .... "/dev/ashmem (deleted)" [AAAAA=].

Maybe shlex?

[Primiano Tucci (use gerrit), 2014-09-09 11:53:30]

> Maybe shlex?
Tried a quick patch with shlex (http://pastebin.com/VjiW79NH) but it seems more
than 10x slower than re:

Average while parsing a real world memdump output:
shlex:    1.26585698128 seconds
re (cur): 0.101233005524 seconds

[Sami, 2014-09-09 11:57:13]

On 2014/09/09 11:53:30, Primiano Tucci wrote:
> > Maybe shlex?
> Tried a quick patch with shlex (http://pastebin.com/VjiW79NH) but it seems
more
> than 10x slower than re:
> 
> Average while parsing a real world memdump output:
> shlex:    1.26585698128 seconds
> re (cur): 0.101233005524 seconds

Wow, I guess speed doesn't come for free :)

[Primiano Tucci (use gerrit), 2014-09-09 12:00:20]

The CQ bit was checked by primiano@chromium.org

[commit-bot: I haz the power, 2014-09-09 12:01:18]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/primiano@chromium.org/550093003/60001

[commit-bot: I haz the power, 2014-09-09 13:01:28]

Committed patchset #3 (id:60001) as adf8207d764e5ca26b4337143c1fffca3228941a

[commit-bot: I haz the power, 2014-09-10 03:53:01]

Patchset 3 (id:??) landed as
https://crrev.com/d7339a4b9655113f7b7babe05755fb6650ad80ec
Cr-Commit-Position: refs/heads/master@{#293927}