[UMA] Use proper C++ objects to serialize tracked_objects across process boundaries.

base/tracked_objects.h

 // Copyright (c) 2012 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 #ifndef BASE_TRACKED_OBJECTS_H_
 #define BASE_TRACKED_OBJECTS_H_
 #pragma once
 
 #include <map>
 #include <set>
 #include <stack>
 #include <string>
 #include <utility>
 #include <vector>
 
 #include "base/base_export.h"
 #include "base/gtest_prod_util.h"
 #include "base/lazy_instance.h"
 #include "base/location.h"
 #include "base/profiler/alternate_timer.h"
 #include "base/profiler/tracked_time.h"
 #include "base/time.h"
 #include "base/synchronization/lock.h"
 #include "base/threading/thread_local_storage.h"
 #include "base/tracking_info.h"
-#include "base/values.h"
 
 // TrackedObjects provides a database of stats about objects (generally Tasks)
 // that are tracked.  Tracking means their birth, death, duration, birth thread,
 // death thread, and birth place are recorded.  This data is carefully spread
 // across a series of objects so that the counts and times can be rapidly
 // updated without (usually) having to lock the data, and hence there is usually
 // very little contention caused by the tracking.  The data can be viewed via
 // the about:profiler URL, with a variety of sorting and filtering choices.
 //
 // These classes serve as the basis of a profiler of sorts for the Tasks system.
@@ skipping 85 matching lines @@
 // very little global performance impact.
 //
 // The above description tries to define the high performance (run time)
 // portions of these classes.  After gathering statistics, calls instigated
 // by visiting about:profiler will assemble and aggregate data for display.  The
 // following data structures are used for producing such displays.  They are
 // not performance critical, and their only major constraint is that they should
 // be able to run concurrently with ongoing augmentation of the birth and death
 // data.
 //
+// This header also exports collection of classes that provide serialized
+// representations of the core tracked_objects:: classes.  These serialized
+// representations are designed for safe transmission of the tracked_objects::
+// data across process boundaries.  Each consists of:
+// (1) a default constructor, to support the IPC serialization macros,
+// (2) a constructor that extracts data from the type being serialized, and
+// (3) the serialized data.
+//
 // For a given birth location, information about births is spread across data
-// structures that are asynchronously changing on various threads.  For display
-// purposes, we need to construct Snapshot instances for each combination of
-// birth thread, death thread, and location, along with the count of such
-// lifetimes.  We gather such data into a Snapshot instances, so that such
-// instances can be sorted and aggregated (and remain frozen during our
-// processing).  Snapshot instances use pointers to constant portions of the
-// birth and death datastructures, but have local (frozen) copies of the actual
-// statistics (birth count, durations, etc. etc.).
+// structures that are asynchronously changing on various threads.  For
+// serialization and display purposes, we need to construct SerializedSnapshot
+// instances for each combination of birth thread, death thread, and location,
+// along with the count of such lifetimes.  We gather such data into a
+// SerializedSnapshot instances, so that such instances can be sorted and
+// aggregated (and remain frozen during our processing).
 //
-// A DataCollector is a container object that holds a set of Snapshots. The
-// statistics in a snapshot are gathered asynhcronously relative to their
-// ongoing updates.  It is possible, though highly unlikely, that stats could be
-// incorrectly recorded by this process (all data is held in 32 bit ints, but we
-// are not atomically collecting all data, so we could have count that does not,
-// for example, match with the number of durations we accumulated).  The
-// advantage to having fast (non-atomic) updates of the data outweighs the
-// minimal risk of a singular corrupt statistic snapshot (only the snapshot
-// could be corrupt, not the underlying and ongoing statistic).  In constrast,
-// pointer data that is accessed during snapshotting is completely invariant,
-// and hence is perfectly acquired (i.e., no potential corruption, and no risk
-// of a bad memory reference).
-//
-// After an array of Snapshots instances are collected into a DataCollector,
-// they need to be prepared for displaying our output.  We currently implement a
-// serialization into a Value hierarchy, which is automatically translated to
-// JSON when supplied to rendering Java Scirpt.
+// The SerializedProcessData struct is a serialized representation of the list

[jar (doing other things), 2012/04/04 17:55:35]

nit:  The use of the word "Serialized..." as a prefix is probably not the best. 
Usually this means you've translated to something like a string, or some serial
data.  In truth, you've put it into a persistent struct that can be copied
around freely.  I'm not sure what the name should be... and need to think
more...

Some plausible words would be:
Snapshot
Frozen

I guess you're using the word Serialized to suggest the mental model that "this
class has the moral equivalent of serialized data, and you can think of it as
such."  It is a bit unexpected that the data then effectively has accessors
(slotted access to members).

[Ilya Sherman, 2012/04/05 02:51:04]

Went with 'Snapshot'

+// of ThreadData objects for a process.  It holds a set of SerializedSnapshots
+// and tracks parent/child relationships for the executed tasks.  The statistics
+// in a snapshot are gathered asynhcronously relative to their ongoing updates.
+// It is possible, though highly unlikely, that stats could be incorrectly
+// recorded by this process (all data is held in 32 bit ints, but we are not
+// atomically collecting all data, so we could have count that does not, for
+// example, match with the number of durations we accumulated).  The advantage
+// to having fast (non-atomic) updates of the data outweighs the minimal risk of
+// a singular corrupt statistic snapshot (only the snapshot could be corrupt,
+// not the underlying and ongoing statistic).  In constrast, pointer data that
+// is accessed during snapshotting is completely invariant, and hence is
+// perfectly acquired (i.e., no potential corruption, and no risk of a bad
+// memory reference).
 //
 // TODO(jar): We can implement a Snapshot system that *tries* to grab the
 // snapshots on the source threads *when* they have MessageLoops available
 // (worker threads don't have message loops generally, and hence gathering from
 // them will continue to be asynchronous).  We had an implementation of this in
 // the past, but the difficulty is dealing with message loops being terminated.
 // We can *try* to spam the available threads via some message loop proxy to
 // achieve this feat, and it *might* be valuable when we are colecting data for
 // upload via UMA (where correctness of data may be more significant than for a
 // single screen of about:profiler).
@@ skipping 20 matching lines @@
 namespace tracked_objects {
 
 //------------------------------------------------------------------------------
 // For a specific thread, and a specific birth place, the collection of all
 // death info (with tallies for each death thread, to prevent access conflicts).
 class ThreadData;
 class BASE_EXPORT BirthOnThread {
  public:
   BirthOnThread(const Location& location, const ThreadData& current);
 
-  const Location location() const;
-  const ThreadData* birth_thread() const;
-
-  // Insert our state (location, and thread name) into the dictionary.
-  // Use the supplied |prefix| in front of "thread_name" and "location"
-  // respectively when defining keys.
-  void ToValue(const std::string& prefix,
-               base::DictionaryValue* dictionary) const;
+  const Location location() const { return location_; }
+  const ThreadData* birth_thread() const { return birth_thread_; }
 
  private:
   // File/lineno of birth.  This defines the essence of the task, as the context
   // of the birth (construction) often tell what the item is for.  This field
   // is const, and hence safe to access from any thread.
   const Location location_;
 
   // The thread that records births into this object.  Only this thread is
   // allowed to update birth_count_ (which changes over time).
   const ThreadData* const birth_thread_;
 
   DISALLOW_COPY_AND_ASSIGN(BirthOnThread);
 };
 
 //------------------------------------------------------------------------------
+// A serialized representation of the BirthOnThread class.
+
+struct BASE_EXPORT SerializedBirthOnThread {
+  SerializedBirthOnThread();
+  explicit SerializedBirthOnThread(const BirthOnThread& birth);
+  ~SerializedBirthOnThread();
+
+  SerializedLocation location;
+  std::string thread_name;
+};
+
+//------------------------------------------------------------------------------
 // A class for accumulating counts of births (without bothering with a map<>).
 
 class BASE_EXPORT Births: public BirthOnThread {
  public:
   Births(const Location& location, const ThreadData& current);
 
   int birth_count() const;
 
-  // When we have a birth we update the count for this BirhPLace.
+  // When we have a birth we update the count for this birthplace.
   void RecordBirth();
 
   // When a birthplace is changed (updated), we need to decrement the counter
   // for the old instance.
   void ForgetBirth();
 
   // Hack to quickly reset all counts to zero.
   void Clear();
 
  private:
@@ skipping 17 matching lines @@
   // threads, we create DeathData stats that tally the number of births without
   // a corresponding death.
   explicit DeathData(int count);
 
   // Update stats for a task destruction (death) that had a Run() time of
   // |duration|, and has had a queueing delay of |queue_duration|.
   void RecordDeath(const DurationInt queue_duration,
                    const DurationInt run_duration,
                    int random_number);
 
-  // Metrics accessors, used only in tests.
+  // Metrics accessors, used only for serialization and in tests.
   int count() const;
   DurationInt run_duration_sum() const;
   DurationInt run_duration_max() const;
   DurationInt run_duration_sample() const;
   DurationInt queue_duration_sum() const;
   DurationInt queue_duration_max() const;
   DurationInt queue_duration_sample() const;
 
-  // Construct a DictionaryValue instance containing all our stats. The caller
-  // assumes ownership of the returned instance.
-  base::DictionaryValue* ToValue() const;
-
   // Reset the max values to zero.
   void ResetMax();
 
   // Reset all tallies to zero. This is used as a hack on realtime data.
   void Clear();
 
  private:
   // Members are ordered from most regularly read and updated, to least
   // frequently used.  This might help a bit with cache lines.
   // Number of runs seen (divisor for calculating averages).
   int count_;
   // Basic tallies, used to compute averages.
   DurationInt run_duration_sum_;
   DurationInt queue_duration_sum_;
   // Max values, used by local visualization routines.  These are often read,
   // but rarely updated.
   DurationInt run_duration_max_;
   DurationInt queue_duration_max_;
-  // Samples, used by by crowd sourcing gatherers.  These are almost never read,
+  // Samples, used by crowd sourcing gatherers.  These are almost never read,
   // and rarely updated.
   DurationInt run_duration_sample_;
   DurationInt queue_duration_sample_;
 };
 
 //------------------------------------------------------------------------------
+// A serialized representation of the DeathData class.
+
+struct BASE_EXPORT SerializedDeathData {
+  SerializedDeathData();
+  explicit SerializedDeathData(const DeathData& death_data);
+  ~SerializedDeathData();
+
+  int count;
+  DurationInt run_duration_sum;
+  DurationInt run_duration_max;
+  DurationInt run_duration_sample;
+  DurationInt queue_duration_sum;
+  DurationInt queue_duration_max;
+  DurationInt queue_duration_sample;
+};
+
+//------------------------------------------------------------------------------
 // A temporary collection of data that can be sorted and summarized.  It is
 // gathered (carefully) from many threads.  Instances are held in arrays and
 // processed, filtered, and rendered.
 // The source of this data was collected on many threads, and is asynchronously
 // changing.  The data in this instance is not asynchronously changing.
 
-class BASE_EXPORT Snapshot {
- public:
-  // When snapshotting a full life cycle set (birth-to-death), use this:
-  Snapshot(const BirthOnThread& birth_on_thread,
-           const ThreadData& death_thread,
-           const DeathData& death_data);
+struct BASE_EXPORT SerializedSnapshot {

[jar (doing other things), 2012/04/04 17:55:35]

nit: The historical name "Snapshot" is probably not good to preserve, given that
you're developing a bunch of snapshot classes now.

As per our discussion, this class shows the bottom level leaf in the data
structure, describing data on a set of "similar" tasks.  It could almost be a
Lifetime, except it lists aggregate data about a whole set of tasks that had the
same birth location and death thread (and Lifetime sounds like a singular run,
not a set of runs of a task).  Perhaps we'd call this collection of "similar
lives" a Family, or we could think of it as as a LineItem or Row in the final
profiler report.  Plausible names are then:

Family
LineItem
TaskGroup
TaskRow
Lifetimes  (note the plural)
Lives

For any of the above names, the word "Snapshot" should probably be a postfix,
such as:
FamilySnapshot
LivesSnapshot

...but Frozen should probably be a prefix:
FrozenFamily
FrozenLives

[Ilya Sherman, 2012/04/05 02:51:04]

Went with just 'TaskSnapshot' -- does that sound ok to you?

+  SerializedSnapshot();
+  SerializedSnapshot(const BirthOnThread& birth,
+                     const DeathData& death_data,
+                     const std::string& death_thread_name);
+  ~SerializedSnapshot();
 
-  // When snapshotting a birth, with no death yet, use this:
-  Snapshot(const BirthOnThread& birth_on_thread, int count);
-
-  // Accessor, that provides default value when there is no death thread.
-  const std::string DeathThreadName() const;
-
-  // Construct a DictionaryValue instance containing all our data recursively.
-  // The caller assumes ownership of the memory in the returned instance.
-  base::DictionaryValue* ToValue() const;
-
- private:
-  const BirthOnThread* birth_;  // Includes Location and birth_thread.
-  const ThreadData* death_thread_;
-  DeathData death_data_;
+  SerializedBirthOnThread birth;
+  SerializedDeathData death_data;
+  std::string death_thread_name;
 };
 
 //------------------------------------------------------------------------------
 // For each thread, we have a ThreadData that stores all tracking info generated
 // on this thread.  This prevents the need for locking as data accumulates.
 // We use ThreadLocalStorage to quickly identfy the current ThreadData context.
 // We also have a linked list of ThreadData instances, and that list is used to
 // harvest data from all existing instances.
 
+struct SerializedProcessData;
 class BASE_EXPORT ThreadData {
  public:
   // Current allowable states of the tracking system.  The states can vary
   // between ACTIVE and DEACTIVATED, but can never go back to UNINITIALIZED.
   enum Status {
     UNINITIALIZED,              // PRistine, link-time state before running.
     DORMANT_DURING_TESTS,       // Only used during testing.
     DEACTIVATED,                // No longer recording profling.
     PROFILING_ACTIVE,           // Recording profiles (no parent-child links).
     PROFILING_CHILDREN_ACTIVE,  // Fully active, recording parent-child links.
@@ skipping 10 matching lines @@
   // set *before* any births on the threads have taken place.  It is generally
   // only used by the message loop, which has a well defined thread name.
   static void InitializeThreadContext(const std::string& suggested_name);
 
   // Using Thread Local Store, find the current instance for collecting data.
   // If an instance does not exist, construct one (and remember it for use on
   // this thread.
   // This may return NULL if the system is disabled for any reason.
   static ThreadData* Get();
 
-  // Constructs a DictionaryValue instance containing all recursive results in
-  // our process.  The caller assumes ownership of the memory in the returned
-  // instance.  During the scavenging, if |reset_max| is true, then the
-  // DeathData instances max-values are reset to zero during this scan.
-  static base::DictionaryValue* ToValue(bool reset_max);
+  // Fills |process_data| with all the recursive results in our process.
+  // During the scavenging, if |reset_max| is true, then the DeathData instances
+  // max-values are reset to zero during this scan.
+  static void ToSerializedProcessData(bool reset_max,
+                                      SerializedProcessData* process_data);
 
   // Finds (or creates) a place to count births from the given location in this
   // thread, and increment that tally.
   // TallyABirthIfActive will returns NULL if the birth cannot be tallied.
   static Births* TallyABirthIfActive(const Location& location);
 
   // Records the end of a timed run of an object.  The |completed_task| contains
   // a pointer to a Births, the time_posted, and a delayed_start_time if any.
   // The |start_of_run| indicates when we started to perform the run of the
   // task.  The delayed_start_time is non-null for tasks that were posted as
@@ skipping 21 matching lines @@
 
   // Record the end of execution in region, generally corresponding to a scope
   // being exited.
   static void TallyRunInAScopedRegionIfTracking(
       const Births* birth,
       const TrackedTime& start_of_run,
       const TrackedTime& end_of_run);
 
   const std::string thread_name() const;
 
-  // Snapshot (under a lock) copies of the maps in each ThreadData instance. For
-  // each set of maps (BirthMap, DeathMap, and ParentChildSet) call the Append()
-  // method of the |target| DataCollector.  If |reset_max| is true, then the max
-  // values in each DeathData instance should be reset during the scan.
-  static void SendAllMaps(bool reset_max, class DataCollector* target);
-
   // Hack: asynchronously clear all birth counts and death tallies data values
   // in all ThreadData instances.  The numerical (zeroing) part is done without
   // use of a locks or atomics exchanges, and may (for int64 values) produce
   // bogus counts VERY rarely.
   static void ResetAllThreadData();
 
   // Initializes all statics if needed (this initialization call should be made
   // while we are single threaded). Returns false if unable to initialize.
   static bool Initialize();
 
@@ skipping 77 matching lines @@
 
 
   // In this thread's data, record a new birth.
   Births* TallyABirth(const Location& location);
 
   // Find a place to record a death on this thread.
   void TallyADeath(const Births& birth,
                    DurationInt queue_duration,
                    DurationInt duration);
 
+  // Snapshot (under a lock) the profiled data for the tasks in each ThreadData
+  // instance.  Also updates the |birth_counts| tally for each task to keep
+  // track of the number of living instances of the task.  If |reset_max| is
+  // true, then the max values in each DeathData instance should be reset during
+  // the scan.
+  static void SerializeAllExecutedTasks(
+      bool reset_max,
+      SerializedProcessData* process_data,
+      std::map<const BirthOnThread*, int>* birth_counts);
+
+  // Snapshots (under a lock) the profiled data for the tasks for this thread
+  // and serializes all of the executed tasks' data -- i.e. the data for the
+  // tasks with with entries in the death_map_ -- into |process_data|.  Also
+  // updates the |birth_counts| tally for each task to keep track of the number
+  // of living instances of the task.  If |reset_max| is true, then the max

[jar (doing other things), 2012/04/04 17:55:35]

Comment should be clear that the count includes only births for which a death
has not yet been recorded.  (or includes all births, including dead ones.. if I
have it backwards).  Comment (probably copied from my words) only says it "keeps
track" and  I wasn't sure if we had to contrast with process_data to get live
counts.

[Ilya Sherman, 2012/04/05 02:51:04]

Done.

+  // values in each DeathData instance should be reset during the scan.
+  void SerializeExecutedTasks(
+      bool reset_max,
+      SerializedProcessData* process_data,
+      std::map<const BirthOnThread*, int>* birth_counts);
+
   // Using our lock, make a copy of the specified maps.  This call may be made
   // on  non-local threads, which necessitate the use of the lock to prevent
   // the map(s) from being reallocaed while they are copied. If |reset_max| is
   // true, then, just after we copy the DeathMap, we will set the max values to
   // zero in the active DeathMap (not the snapshot).
   void SnapshotMaps(bool reset_max,
                     BirthMap* birth_map,
                     DeathMap* death_map,
                     ParentChildSet* parent_child_set);
 
@@ skipping 124 matching lines @@
   // Record of what the incarnation_counter_ was when this instance was created.
   // If the incarnation_counter_ has changed, then we avoid pushing into the
   // pool (this is only critical in tests which go through multiple
   // incarnations).
   int incarnation_count_for_pool_;
 
   DISALLOW_COPY_AND_ASSIGN(ThreadData);
 };
 
 //------------------------------------------------------------------------------
-// DataCollector is a container class for Snapshot and BirthOnThread count
-// items.
+// A serialized representation of a (parent, child) task pair, for tracking
+// hierarchical profiles.
 
-class BASE_EXPORT DataCollector {
+struct BASE_EXPORT SerializedParentChildPair {
  public:
-  typedef std::vector<Snapshot> Collection;
+  SerializedParentChildPair();
+  explicit SerializedParentChildPair(
+      const ThreadData::ParentChildPair& parent_child);
+  ~SerializedParentChildPair();
 
-  // Construct with a list of how many threads should contribute.  This helps us
-  // determine (in the async case) when we are done with all contributions.
-  DataCollector();
-  ~DataCollector();
+  SerializedBirthOnThread parent;
+  SerializedBirthOnThread child;
+};
 
-  // Adds all stats from the indicated thread into our arrays.  Accepts copies
-  // of the birth_map and death_map, so that the data will not change during the
-  // iterations and processing.
-  void Append(const ThreadData &thread_data,
-              const ThreadData::BirthMap& birth_map,
-              const ThreadData::DeathMap& death_map,
-              const ThreadData::ParentChildSet& parent_child_set);
+//------------------------------------------------------------------------------
+// A serialized representation of the list of ThreadData objects for a process.
 
-  // After the accumulation phase, the following accessor is used to process the
-  // data (i.e., sort it, filter it, etc.).
-  Collection* collection();
+struct BASE_EXPORT SerializedProcessData {
+ public:
+  SerializedProcessData();
+  ~SerializedProcessData();
 
-  // Adds entries for all the remaining living objects (objects that have
-  // tallied a birth, but have not yet tallied a matching death, and hence must
-  // be either running, queued up, or being held in limbo for future posting).
-  // This should be called after all known ThreadData instances have been
-  // processed using Append().
-  void AddListOfLivingObjects();
-
-  // Generates a ListValue representation of the vector of snapshots, and
-  // inserts the results into |dictionary|.
-  void ToValue(base::DictionaryValue* dictionary) const;
-
- private:
-  typedef std::map<const BirthOnThread*, int> BirthCount;
-
-  // The array that we collect data into.
-  Collection collection_;
-
-  // The total number of births recorded at each location for which we have not
-  // seen a death count.  This map changes as we do Append() calls, and is later
-  // used by AddListOfLivingObjects() to gather up unaccounted for births.
-  BirthCount global_birth_count_;
-
-  // The complete list of parent-child relationships among tasks.
-  ThreadData::ParentChildSet parent_child_set_;
-
-  DISALLOW_COPY_AND_ASSIGN(DataCollector);
+  std::vector<SerializedSnapshot> snapshots;
+  std::vector<SerializedParentChildPair> descendants;
+  int process_id;
 };
 
 }  // namespace tracked_objects
 
 #endif  // BASE_TRACKED_OBJECTS_H_