Support profiling based on linux kernel performance events.

Support profiling based on linux kernel performance events.

Since 2.6.31 perf_events interface has been available in the
kernel. There's a nice tool called "perf" (linux-2.6/tools/perf) that
uses this interface and provides capabilities similar to oprofile. The
simplest form of its usage is just dumping the raw log (trace) of
events generated by the kernel. In this patch I'm adding a script
(tools/ll_prof.py) to build profiles based on perf trace and our code
log. All the heavy-lifting is done by perf. Compared to oprofile agent
this approach does not require recompilation and supports code moving
garbage collections.

Expected usage is documented in the ll_prof's help. Basically one
should run V8 under perf passing --ll-prof flag and then the produced
logs can be analyzed by tools/ll_prof.py.

The new --ll-prof flag enables logging of generated code object
locations and names (like --log-code), and also of their bodies, which
can be later disassembled and annotated by the script.

Committed: http://code.google.com/p/v8/source/detail?r=5663

[mnaganov (inactive), 2010-10-17 19:51:11]

Excellent job, Vitaly!

Having this code in place, we can drop OProfile support (as it's only for linux
too, and requires significant efforts to be set up properly).

http://codereview.chromium.org/3831002/diff/3001/4004
File src/log-utils.cc (right):

http://codereview.chromium.org/3831002/diff/3001/4004#newcode157
src/log-utils.cc:157: char* code_name = new char[name_len +
sizeof(kCodeLogExt)];
ScopedVector<char>?

http://codereview.chromium.org/3831002/diff/3001/4004#newcode158
src/log-utils.cc:158: memcpy(code_name, name, name_len);
OS::StrNCpy?

http://codereview.chromium.org/3831002/diff/3001/4005
File src/log-utils.h (right):

http://codereview.chromium.org/3831002/diff/3001/4005#newcode154
src/log-utils.h:154: static FILE* output_code_handle_;
Please add a comment for this variable.

http://codereview.chromium.org/3831002/diff/3001/4006
File src/log.cc (right):

http://codereview.chromium.org/3831002/diff/3001/4006#newcode1361
src/log.cc:1361: int pos = static_cast<int>(ftell(Log::output_code_handle_));
Should we put ftell into OS interface, like FOpen?

http://codereview.chromium.org/3831002/diff/3001/4006#newcode1362
src/log.cc:1362: fwrite(code->instruction_start(), 1, code->instruction_size(),
and FWrite

http://codereview.chromium.org/3831002/diff/3001/4016
File tools/ll_prof.py (right):

http://codereview.chromium.org/3831002/diff/3001/4016#newcode72
tools/ll_prof.py:72: JS_CODE_HEADER_SIZE = 32
My usual solution to eliminate such dependencies is to include all needed
constants into the log itself and read them by the analyzer.

http://codereview.chromium.org/3831002/diff/3001/4016#newcode218
tools/ll_prof.py:218: command = "%s %s -D -b binary %s %s" % (
You may consider using dd for getting a range of bytes from a file.

http://codereview.chromium.org/3831002/diff/3001/4016#newcode358
tools/ll_prof.py:358: r"code-move,(0x[a-z0-9]+),(0x[a-f0-9]+)")
Why a-z?

http://codereview.chromium.org/3831002/diff/3001/4016#newcode511
tools/ll_prof.py:511: TRACE_HEADER_DESC = Descriptor([
I think it's a good idea to add a link to perf (?) documentation here.

http://codereview.chromium.org/3831002/diff/3001/4016#newcode894
tools/ll_prof.py:894: mmap_info = trace_reader.ReadMmap(header, offset)
So, pprof emits its own modules map, so we don't need to
LogSharedLibrariesAddresses, right?

[Vitaly Repeshko, 2010-10-18 10:48:54]

Thanks for the comments! Please take another look.


-- Vitaly

http://codereview.chromium.org/3831002/diff/3001/4004
File src/log-utils.cc (right):

http://codereview.chromium.org/3831002/diff/3001/4004#newcode157
src/log-utils.cc:157: char* code_name = new char[name_len +
sizeof(kCodeLogExt)];
On 2010/10/17 19:51:12, Michail Naganov wrote:
> ScopedVector<char>?

Done.

http://codereview.chromium.org/3831002/diff/3001/4004#newcode158
src/log-utils.cc:158: memcpy(code_name, name, name_len);
On 2010/10/17 19:51:12, Michail Naganov wrote:
> OS::StrNCpy?

I need the length anyway, so memcpy is easier.

http://codereview.chromium.org/3831002/diff/3001/4005
File src/log-utils.h (right):

http://codereview.chromium.org/3831002/diff/3001/4005#newcode154
src/log-utils.h:154: static FILE* output_code_handle_;
On 2010/10/17 19:51:12, Michail Naganov wrote:
> Please add a comment for this variable.

Done.

http://codereview.chromium.org/3831002/diff/3001/4006
File src/log.cc (right):

http://codereview.chromium.org/3831002/diff/3001/4006#newcode1361
src/log.cc:1361: int pos = static_cast<int>(ftell(Log::output_code_handle_));
On 2010/10/17 19:51:12, Michail Naganov wrote:
> Should we put ftell into OS interface, like FOpen?

Probably not worth it while the standard C functions do the job.

http://codereview.chromium.org/3831002/diff/3001/4006#newcode1362
src/log.cc:1362: fwrite(code->instruction_start(), 1, code->instruction_size(),
On 2010/10/17 19:51:12, Michail Naganov wrote:
> and FWrite

Same here.

http://codereview.chromium.org/3831002/diff/3001/4016
File tools/ll_prof.py (right):

http://codereview.chromium.org/3831002/diff/3001/4016#newcode72
tools/ll_prof.py:72: JS_CODE_HEADER_SIZE = 32
On 2010/10/17 19:51:12, Michail Naganov wrote:
> My usual solution to eliminate such dependencies is to include all needed
> constants into the log itself and read them by the analyzer.

I added code-info log entry that is used to pass the architecture and code
header size.

http://codereview.chromium.org/3831002/diff/3001/4016#newcode218
tools/ll_prof.py:218: command = "%s %s -D -b binary %s %s" % (
On 2010/10/17 19:51:12, Michail Naganov wrote:
> You may consider using dd for getting a range of bytes from a file.

Good idea. Done.

http://codereview.chromium.org/3831002/diff/3001/4016#newcode358
tools/ll_prof.py:358: r"code-move,(0x[a-z0-9]+),(0x[a-f0-9]+)")
On 2010/10/17 19:51:12, Michail Naganov wrote:
> Why a-z?

Fixed (in other places too).

http://codereview.chromium.org/3831002/diff/3001/4016#newcode511
tools/ll_prof.py:511: TRACE_HEADER_DESC = Descriptor([
On 2010/10/17 19:51:12, Michail Naganov wrote:
> I think it's a good idea to add a link to perf (?) documentation here.

I doesn't seem to have an up-to-date site, but the documentation in the kernel
repo is pretty good. Added a link to it.

http://codereview.chromium.org/3831002/diff/3001/4016#newcode894
tools/ll_prof.py:894: mmap_info = trace_reader.ReadMmap(header, offset)
On 2010/10/17 19:51:12, Michail Naganov wrote:
> So, pprof emits its own modules map, so we don't need to
> LogSharedLibrariesAddresses, right?

Right, the kernel sends perf an MMAP event each time the process mmap's
something with PROT_EXEC. This also covers the binary itself and the even the
kernel modules (!).

[mnaganov (inactive), 2010-10-18 12:38:09]

LGTM

Please provide a documentation page in V8 Wiki also.

http://codereview.chromium.org/3831002/diff/3001/4004
File src/log-utils.cc (right):

http://codereview.chromium.org/3831002/diff/3001/4004#newcode158
src/log-utils.cc:158: memcpy(code_name, name, name_len);
On 2010/10/18 10:48:54, Vitaly wrote:
> On 2010/10/17 19:51:12, Michail Naganov wrote:
> > OS::StrNCpy?
> 
> I need the length anyway, so memcpy is easier.
> 

Well, StrNCpy does require length:
void StrNCpy(Vector<char> dest, const char* src, size_t n)

My point is that using string functions for strings provides better clarity,
than using a generic memcpy.

You can fix that before committing.

http://codereview.chromium.org/3831002/diff/10001/11004#newcode182
src/log-utils.cc:182: kDynamicBufferBlockSize, kMaxDynamicBufferSize,
Oh, right, I missed this problem.

http://codereview.chromium.org/3831002/diff/10001/11016
File tools/ll_prof.py (right):

http://codereview.chromium.org/3831002/diff/10001/11016#newcode206
tools/ll_prof.py:206: command = "dd if=%s.code of=%s bs=1 count=%d skip=%d && "
\
Well, I thought you could just pipe them, no? 'dd if=... | objdump ...'

[Vitaly Repeshko, 2010-10-18 12:57:31]

http://codereview.chromium.org/3831002/diff/3001/4004
File src/log-utils.cc (right):

http://codereview.chromium.org/3831002/diff/3001/4004#newcode158
src/log-utils.cc:158: memcpy(code_name, name, name_len);
On 2010/10/18 12:38:09, Michail Naganov wrote:
> On 2010/10/18 10:48:54, Vitaly wrote:
> > On 2010/10/17 19:51:12, Michail Naganov wrote:
> > > OS::StrNCpy?
> > 
> > I need the length anyway, so memcpy is easier.
> > 
> 
> Well, StrNCpy does require length:
> void StrNCpy(Vector<char> dest, const char* src, size_t n)
> 
> My point is that using string functions for strings provides better clarity,
> than using a generic memcpy.
> 
> You can fix that before committing.

Technically, the first memcpy is not a string operation, the terminating NUL
byte is not copied. StrNCpy will require calling Vector::SubVector to append the
suffix. Do you still think it'd be nicer?

http://codereview.chromium.org/3831002/diff/10001/11016
File tools/ll_prof.py (right):

http://codereview.chromium.org/3831002/diff/10001/11016#newcode206
tools/ll_prof.py:206: command = "dd if=%s.code of=%s bs=1 count=%d skip=%d && "
\
On 2010/10/18 12:38:09, Michail Naganov wrote:
> Well, I thought you could just pipe them, no? 'dd if=... | objdump ...'

I had tried this. (Initially I was going to pass the bytes to
process.communicate() as stdin.) Won't work, because objdump has to be able to
seek in its input.

[fschneider, 2010-10-18 13:09:41]

This is really cool! I'm looking forward to trying this out.

some minor drive-by comments:

http://codereview.chromium.org/3831002/diff/10001/11009
File src/platform-linux.cc (right):

http://codereview.chromium.org/3831002/diff/10001/11009#newcode407
src/platform-linux.cc:407: // Linux profiler built into the kernel logs all
mmap's with
The Linux profiler...

http://codereview.chromium.org/3831002/diff/10001/11015
File src/platform.h (right):

http://codereview.chromium.org/3831002/diff/10001/11015#newcode265
src/platform.h:265: // Support for profiler.  Notifies the external profiling
process
Support for _the_ profiler.

http://codereview.chromium.org/3831002/diff/10001/11015#newcode268
src/platform.h:268: // --never-compact) if accurate profiling is desired .
Extra space before .

[Vitaly Repeshko, 2010-10-19 16:44:00]

Mikhail, Florian, thanks for reviewing this! I'm submitting and writing a howto.


-- Vitaly

http://codereview.chromium.org/3831002/diff/10001/11009
File src/platform-linux.cc (right):

http://codereview.chromium.org/3831002/diff/10001/11009#newcode407
src/platform-linux.cc:407: // Linux profiler built into the kernel logs all
mmap's with
On 2010/10/18 13:09:42, fschneider wrote:
> The Linux profiler...

Fixed.

http://codereview.chromium.org/3831002/diff/10001/11015
File src/platform.h (right):

http://codereview.chromium.org/3831002/diff/10001/11015#newcode265
src/platform.h:265: // Support for profiler.  Notifies the external profiling
process
On 2010/10/18 13:09:42, fschneider wrote:
> Support for _the_ profiler.

Fixed (also above).

http://codereview.chromium.org/3831002/diff/10001/11015#newcode268
src/platform.h:268: // --never-compact) if accurate profiling is desired .
On 2010/10/18 13:09:42, fschneider wrote:
> Extra space before .

Fixed.