Add HEAP_PROFILE_TIME_INTERVAL to dump heap profiles periodically in third_party/tcmalloc/chromium.

Add HEAP_PROFILE_TIME_INTERVAL to dump heap profiles periodically in third_party/tcmalloc/chromium.

This patch is originally from http://codereview.chromium.org/7865021/. 


BUG=none
TEST=none

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=115808

[Alexander Potapenko, 2011-12-20 13:26:04]

General comment: this flag does not give you a guaranteed time interval between
two dumps. Are you ok with that?
If not, consider using a SIGPROF handler.

http://codereview.chromium.org/8633006/diff/10001/third_party/tcmalloc/chromi...
File third_party/tcmalloc/chromium/src/heap-profiler.cc (right):

http://codereview.chromium.org/8633006/diff/10001/third_party/tcmalloc/chromi...
third_party/tcmalloc/chromium/src/heap-profiler.cc:111:
EnvToInt64("HEAP_PROFILE_TIME_INTERVAL", 0),
Please name the variable accordingly, i.e. heap_profile_time_interval.

http://codereview.chromium.org/8633006/diff/10001/third_party/tcmalloc/chromi...
third_party/tcmalloc/chromium/src/heap-profiler.cc:293: time(NULL) -
last_dump_time >= FLAGS_heap_profile_sec_interval) {
I think you'd better call time(NULL) only once.

[Dai Mikurube (NOT FULLTIME), 2011-12-21 05:40:56]

Thank you!

Yes, exact time intervals is not necessary.

The goal is long-running heap profiling.  Allocation sizes doesn't provide good
intervals for that, but a little varied intervals are acceptable for this
purpose.

http://codereview.chromium.org/8633006/diff/10001/third_party/tcmalloc/chromi...
File third_party/tcmalloc/chromium/src/heap-profiler.cc (right):

http://codereview.chromium.org/8633006/diff/10001/third_party/tcmalloc/chromi...
third_party/tcmalloc/chromium/src/heap-profiler.cc:111:
EnvToInt64("HEAP_PROFILE_TIME_INTERVAL", 0),
On 2011/12/20 13:26:04, Alexander Potapenko wrote:
> Please name the variable accordingly, i.e. heap_profile_time_interval.

Done.

http://codereview.chromium.org/8633006/diff/10001/third_party/tcmalloc/chromi...
third_party/tcmalloc/chromium/src/heap-profiler.cc:293: time(NULL) -
last_dump_time >= FLAGS_heap_profile_sec_interval) {
On 2011/12/20 13:26:04, Alexander Potapenko wrote:
> I think you'd better call time(NULL) only once.

Done.

[Alexander Potapenko, 2011-12-26 12:18:37]

LGTM

[commit-bot: I haz the power, 2011-12-27 05:00:24]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/dmikurube@chromium.org/8633006/15001

[Dai Mikurube (NOT FULLTIME), 2011-12-27 05:00:40]

Thanks.  Checked the "Commit" box.

[commit-bot: I haz the power, 2011-12-27 06:12:03]

Change committed as 115808

[jar (doing other things), 2011-12-27 16:55:53]

Can you give an example of where this functionalitiy would be useful?

When I do profiling, I generally want to profile between events, such as before
and after loading a specific page, just after startup, during shutdown, around
closing a tab, etc.   It is not clear when I'd just want to watch a trend
(kindred to what is done in a task manager), as I really need the correlation to
events of my chosing.

Comments?

[jar (doing other things), 2011-12-27 17:04:30]

As a more specific, and possibly constructive comment, the memory analysis tools
on Windows that we've built into chrome respond to a specific keystroke.  After
the keystroke, they dump a list of all active memory (things that were allocated
but not freed) along with the call stack that instigated the allocation.  A
second tool analyses this dump, aggregating the results into a blame list,
blaming specfic modules (see on the stacks) for the allocations.

The whole system is based around a user defined signal, such as a keystroke (but
in linux land, it migh be a numeric signal()).  

On 2011/12/27 16:55:53, jar wrote:
> Can you give an example of where this functionalitiy would be useful?
> 
> When I do profiling, I generally want to profile between events, such as
before
> and after loading a specific page, just after startup, during shutdown, around
> closing a tab, etc.   It is not clear when I'd just want to watch a trend
> (kindred to what is done in a task manager), as I really need the correlation
to
> events of my chosing.
> 
> Comments?

[Dai Mikurube (NOT FULLTIME), 2012-01-04 19:26:23]

Sorry, I overlooked the replies.

A typical case of watching a trend is observing long-running and memory
consuming tests.  Memory bloat usually occurs in hours and days in most
scenarios.  Just a snapshot of large consumption doesn't help analysis. 
Watching a trend of such a long-running test will help analyzing memory bloat.

btw, I'm interested in your tool and "specific keystroke" in your tool.  I was
thinking about such user-specified triggers to dump.  SIGQUIT (Ctrl-\) was an
option on Linux, but looked not so good...

[jar (doing other things), 2012-01-04 20:12:26]

The existing memory watcher tools can be built under the "tools" group on MSVC
when you load the chrome.sln.   They include some stuff (on windows) to grab the
keystroke, distribute this notification to all processes, walk the stacks, and
dump the results.

I'm still slightly at a loss for understanding what you're hoping to visualize
with the ongoing snapshots.  I guess you can (if you build the
graphing/visualization tools) see that the eventual bloat was either aquired in
one gulp, or was the result of a slow growth.  Perhaps the distinction will give
you two different paths to follow to find the source.... but it seems much more
valuable to simply look at the bloated (leaked) memory and ask: Where was this
allocated?  During what functions? etc.  The one gotcha is when it is allocated
in a complex system, such as V8, and then there is the question of what was v8
doing (in the uber javascript domain).  For the browser process, I don't think
we have such an element of confusion, and the "blaming" of memory use on code
*seems* more valuable than looking at a trend line.

What use case am I missing?

Thanks,

Jim

On 2012/01/04 19:26:23, Dai Mikurube wrote:
> Sorry, I overlooked the replies.
> 
> A typical case of watching a trend is observing long-running and memory
> consuming tests.  Memory bloat usually occurs in hours and days in most
> scenarios.  Just a snapshot of large consumption doesn't help analysis. 
> Watching a trend of such a long-running test will help analyzing memory bloat.
> 
> btw, I'm interested in your tool and "specific keystroke" in your tool.  I was
> thinking about such user-specified triggers to dump.  SIGQUIT (Ctrl-\) was an
> option on Linux, but looked not so good...

[jar (doing other things), 2012-01-04 22:15:19]

Excellent answer! Thanks!

So what you are after is looking at memory that is not short term, or more
specifically and equivalently, memory that was allocated "a while ago" or
longer.

You currently detect this by looking at a graph, effectively doing low-pass
filtering to see a consistent range at a given point in time... and that sort of
shows the issue.  It smooths out short term variation.

I need to think about this issue, and see if we can get (in a single snapshot)
this same metric (amount of long-term allocated memory).

The memory-watcher tool can certainly do this, as it records when each
allocation took place.  As a result, when it dumps memory, it could partition
the tally at any point in time (i.e., ignore allocations after time t).

That doesn't account for a thrashing process, that is allocating and freeing
rapidly, maintaining a high average, despite a smaller long-term alloc.   ...but
then again... I think such thrashers confuse the trend line in a similar
non-obvious way.

This is an interesting metric to consider.

Thanks,

Jim



On 2012/01/04 20:12:26, jar wrote:
> The existing memory watcher tools can be built under the "tools" group on MSVC
> when you load the chrome.sln.   They include some stuff (on windows) to grab
the
> keystroke, distribute this notification to all processes, walk the stacks, and
> dump the results.
> 
> I'm still slightly at a loss for understanding what you're hoping to visualize
> with the ongoing snapshots.  I guess you can (if you build the
> graphing/visualization tools) see that the eventual bloat was either aquired
in
> one gulp, or was the result of a slow growth.  Perhaps the distinction will
give
> you two different paths to follow to find the source.... but it seems much
more
> valuable to simply look at the bloated (leaked) memory and ask: Where was this
> allocated?  During what functions? etc.  The one gotcha is when it is
allocated
> in a complex system, such as V8, and then there is the question of what was v8
> doing (in the uber javascript domain).  For the browser process, I don't think
> we have such an element of confusion, and the "blaming" of memory use on code
> *seems* more valuable than looking at a trend line.
> 
> What use case am I missing?
> 
> Thanks,
> 
> Jim
> 
> On 2012/01/04 19:26:23, Dai Mikurube wrote:
> > Sorry, I overlooked the replies.
> > 
> > A typical case of watching a trend is observing long-running and memory
> > consuming tests.  Memory bloat usually occurs in hours and days in most
> > scenarios.  Just a snapshot of large consumption doesn't help analysis. 
> > Watching a trend of such a long-running test will help analyzing memory
bloat.
> > 
> > btw, I'm interested in your tool and "specific keystroke" in your tool.  I
was
> > thinking about such user-specified triggers to dump.  SIGQUIT (Ctrl-\) was
an
> > option on Linux, but looked not so good...

[Dai Mikurube (NOT FULLTIME), 2012-01-05 01:58:18]

Thank you for the comment and summary.

Awesome.  Such a single snapshot would be useful.  It has an additional
advantage in time accuracy.  Accuracy in periodical dumping  is bounded by the
period.

A trade-off is storage consumption in memory to store clock time.  2 or 4 bytes
will be required per allocation.  It may be acceptable, but not ignorable.

On 2012/01/04 22:15:19, jar wrote:
> Excellent answer! Thanks!
> 
> So what you are after is looking at memory that is not short term, or more
> specifically and equivalently, memory that was allocated "a while ago" or
> longer.
> 
> You currently detect this by looking at a graph, effectively doing low-pass
> filtering to see a consistent range at a given point in time... and that sort
of
> shows the issue.  It smooths out short term variation.
> 
> I need to think about this issue, and see if we can get (in a single snapshot)
> this same metric (amount of long-term allocated memory).
> 
> The memory-watcher tool can certainly do this, as it records when each
> allocation took place.  As a result, when it dumps memory, it could partition
> the tally at any point in time (i.e., ignore allocations after time t).
> 
> That doesn't account for a thrashing process, that is allocating and freeing
> rapidly, maintaining a high average, despite a smaller long-term alloc.  
...but
> then again... I think such thrashers confuse the trend line in a similar
> non-obvious way.
> 
> This is an interesting metric to consider.
> 
> Thanks,
> 
> Jim
> 
> 
> 
> On 2012/01/04 20:12:26, jar wrote:
> > The existing memory watcher tools can be built under the "tools" group on
MSVC
> > when you load the chrome.sln.   They include some stuff (on windows) to grab
> the
> > keystroke, distribute this notification to all processes, walk the stacks,
and
> > dump the results.
> > 
> > I'm still slightly at a loss for understanding what you're hoping to
visualize
> > with the ongoing snapshots.  I guess you can (if you build the
> > graphing/visualization tools) see that the eventual bloat was either aquired
> in
> > one gulp, or was the result of a slow growth.  Perhaps the distinction will
> give
> > you two different paths to follow to find the source.... but it seems much
> more
> > valuable to simply look at the bloated (leaked) memory and ask: Where was
this
> > allocated?  During what functions? etc.  The one gotcha is when it is
> allocated
> > in a complex system, such as V8, and then there is the question of what was
v8
> > doing (in the uber javascript domain).  For the browser process, I don't
think
> > we have such an element of confusion, and the "blaming" of memory use on
code
> > *seems* more valuable than looking at a trend line.
> > 
> > What use case am I missing?
> > 
> > Thanks,
> > 
> > Jim
> > 
> > On 2012/01/04 19:26:23, Dai Mikurube wrote:
> > > Sorry, I overlooked the replies.
> > > 
> > > A typical case of watching a trend is observing long-running and memory
> > > consuming tests.  Memory bloat usually occurs in hours and days in most
> > > scenarios.  Just a snapshot of large consumption doesn't help analysis. 
> > > Watching a trend of such a long-running test will help analyzing memory
> bloat.
> > > 
> > > btw, I'm interested in your tool and "specific keystroke" in your tool.  I
> was
> > > thinking about such user-specified triggers to dump.  SIGQUIT (Ctrl-\) was
> an
> > > option on Linux, but looked not so good...