Add one-time randomization support for FieldTrial, and the ability to

base/metrics/field_trial.h

 // Copyright (c) 2011 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 // FieldTrial is a class for handling details of statistical experiments
 // performed by actual users in the field (i.e., in a shipped or beta product).
 // All code is called exclusively on the UI thread currently.
 //
 // The simplest example is an experiment to see whether one of two options
 // produces "better" results across our user population.  In that scenario, UMA
 // data is uploaded to aggregate the test results, and this FieldTrial class
 // manages the state of each such experiment (state == which option was
 // pseudo-randomly selected).
 //
 // States are typically generated randomly, either based on a one time
-// randomization (generated randomly once, and then persistently reused in the
-// client during each future run of the program), or by a startup randomization
+// randomization (which will yield the same results, in terms of selecting
+// the client for a field trial or not, for every run of the program on a
+// given machine, with some exceptions), or by a startup randomization
 // (generated each time the application starts up, but held constant during the
 // duration of the process), or by continuous randomization across a run (where
 // the state can be recalculated again and again, many times during a process).
-// Only startup randomization is implemented thus far.
+// Continuous randomization is not yet implemented.
 
 //------------------------------------------------------------------------------
 // Example:  Suppose we have an experiment involving memory, such as determining
 // the impact of some pruning algorithm.
 // We assume that we already have a histogram of memory usage, such as:
 
 //   HISTOGRAM_COUNTS("Memory.RendererTotal", count);
 
 // Somewhere in main thread initialization code, we'd probably define an
 // instance of a FieldTrial, with code such as:
@@ skipping 68 matching lines @@
   // to assign all the remaining probability to a group ('default').
   static const int kDefaultGroupNumber;
 
   // The name is used to register the instance with the FieldTrialList class,
   // and can be used to find the trial (only one trial can be present for each
   // name).
   // Group probabilities that are later supplied must sum to less than or equal
   // to the total_probability. Arguments year, month and day_of_month specify
   // the expiration time. If the build time is after the expiration time then
   // the field trial reverts to the 'default' group.
+  // Using this constructor creates a startup-randomized FieldTrial. If you
+  // want a one-time randomized trial, call UseOneTimeRandomization() right
+  // after construction.
   FieldTrial(const std::string& name, Probability total_probability,
              const std::string& default_group_name, const int year,
              const int month, const int day_of_month);
 
+  // Changes the field trial to use one-time randomization. Must be called
+  // right after construction. |machine_id| must be something that is fairly
+  // diverse, related to the current machine, and does not change, or changes
+  // very infrequently, while running on the same machine. The reason it must
+  // be fairly diverse is that users with the same |machine_id| and the
+  // same log-on user name may end up in the same FieldTrial group.
+  //
+  // A suggested |machine_id| might be the host name, or something fancier
+  // such as the primary network adapter's MAC address or the hard disk's
+  // serial number.
+  //
+  // If an empty |machine_id| is given, the field trial will be disabled.
+  void UseOneTimeRandomization(const std::string& machine_id);

[jar (doing other things), 2011/04/21 01:03:50]

I'm not convinced you need an argument here, and clearly it puts a large burden
on the caller.  Can't we do without??

I'd rather see an interface that emulates persisted coin toss, with one toss per
persistent trial, than to require all coin-tosses on a single machine to always
come out the same way.  

[Jói, 2011/04/21 19:50:33]

Done.

+
+  // Disables this trial, meaning it always determines the default group
+  // has been selected. Must be called right after construction.
+  void Disable();

[jar (doing other things), 2011/04/21 01:03:50]

Traditionally, disabling meant we were not part of the field trial.  That state
is quite distinct from the default case, although we do eventually turn into a
disabled case when our time expires. 

The subtle distinction is visible in MakeName() and the pattern by which that
function is used in histograms.  When we are disabled, the user often has
over-ridden the target state, and hence the user can't be included in a
would-be-random population.

How will this fit in with your planned use case?

[Jói, 2011/04/21 19:50:33]

So I actually took a closer look at this logic.  It seems that group_name() can
never return the empty string, neither before nor after my change, as you are
always in one of the groups of a trial if the trial is registered.  I updated
the documentation to reflect this, including at the top where it shows how to
determine whether to produce a separate histogram name.  Please verify my
findings as you obviously know this code a lot better than I do.

+
   // Establish the name and probability of the next group in this trial.
   // Sometimes, based on construction randomization, this call may cause the
   // provided group to be *THE* group selected for use in this instance.
   // The return value is the group number of the new group.
   int AppendGroup(const std::string& name, Probability group_probability);
 
   // Return the name of the FieldTrial (excluding the group name).
   std::string name() const { return name_; }
 
   // Return the randomly selected group number that was assigned.
   // Return kDefaultGroupNumber if the instance is in the 'default' group.
   // Note that this will force an instance to participate, and make it illegal
-  // to attempt to probabalistically add any other groups to the trial.
+  // to attempt to probabilistically add any other groups to the trial.
   int group();
 
   // If the field trial is not in an experiment, this returns the empty string.
   // if the group's name is empty, a name of "_" concatenated with the group
   // number is used as the group name.
   std::string group_name();
 
   // Return the default group name of the FieldTrial.
   std::string default_group_name() const { return default_group_name_; }
 
-  // Helper function for the most common use: as an argument to specifiy the
+  // Helper function for the most common use: as an argument to specify the
   // name of a HISTOGRAM.  Use the original histogram name as the name_prefix.
   static std::string MakeName(const std::string& name_prefix,
                               const std::string& trial_name);
 
   // Enable benchmarking sets field trials to a common setting.
   static void EnableBenchmarking();
 
  private:
   // Allow tests to access our innards for testing purposes.
   FRIEND_TEST(FieldTrialTest, Registration);
   FRIEND_TEST(FieldTrialTest, AbsoluteProbabilities);
   FRIEND_TEST(FieldTrialTest, RemainingProbability);
   FRIEND_TEST(FieldTrialTest, FiftyFiftyProbability);
   FRIEND_TEST(FieldTrialTest, MiddleProbabilities);
   FRIEND_TEST(FieldTrialTest, OneWinner);
   FRIEND_TEST(FieldTrialTest, DisableProbability);
   FRIEND_TEST(FieldTrialTest, Save);
   FRIEND_TEST(FieldTrialTest, DuplicateRestore);
   FRIEND_TEST(FieldTrialTest, MakeName);
+  FRIEND_TEST(FieldTrialTest, MachineIdToUniformDouble);
+  FRIEND_TEST(FieldTrialTest, ProbabilisticMachineIdToUniformDoubleIsUniform);
+  FRIEND_TEST(FieldTrialTest, UseOneTimeRandomization);
 
   friend class base::FieldTrialList;
 
   friend class RefCounted<FieldTrial>;
 
   virtual ~FieldTrial();
 
   // Returns the group_name. A winner need not have been chosen.
   std::string group_name_internal() const { return group_name_; }
 
   // Get build time.
   static Time GetBuildTime();
 
+  // Calculates a uniformly-distributed double between [0.0, 1.0) given
+  // a |machine_id|.
+  static double MachineIdToUniformDouble(const std::string& machine_id);
+
   // The name of the field trial, as can be found via the FieldTrialList.
   // This is empty of the trial is not in the experiment.
   const std::string name_;
 
   // The maximum sum of all probabilities supplied, which corresponds to 100%.
   // This is the scaling factor used to adjust supplied probabilities.
   const Probability divisor_;
 
   // The name of the default group.
   const std::string default_group_name_;
 
   // The randomly selected probability that is used to select a group (or have
   // the instance not participate).  It is the product of divisor_ and a random
   // number between [0, 1).
-  const Probability random_;
+  Probability random_;
 
   // Sum of the probabilities of all appended groups.
   Probability accumulated_group_probability_;
 
   int next_group_number_;
 
   // The pseudo-randomly assigned group number.
   // This is kNotFinalized if no group has been assigned.
   int group_;
 
   // A textual name for the randomly selected group. If this Trial is not a
   // member of an group, this string is empty.
   std::string group_name_;
 
   // When disable_field_trial_ is true, field trial reverts to the 'default'
   // group.
   bool disable_field_trial_;

[jar (doing other things), 2011/04/21 01:03:50]

My apology for not noticing this in previous reviews (for other author's
checkins):

Flags are most readable when they are positive (so that you don't have to deal
with double negatives in your head).  This should probably be:

enable_field_trial_

It is fine to have a call to Disable()... just this flag name should be a
positive name.

[You can probably tell I didn't define NDEBUG for the C standard!  ;-)  ]

[Jói, 2011/04/21 19:50:33]

Fixed this, and also fixed handling of Disable() when it is called after
initialization (which will be the case in my expected usage).

 
   // When benchmarking is enabled, field trials all revert to the 'default'
   // group.
   static bool enable_benchmarking_;
 
   DISALLOW_COPY_AND_ASSIGN(FieldTrial);
 };
 
 //------------------------------------------------------------------------------
 // Class with a list of all active field trials.  A trial is active if it has
@@ skipping 76 matching lines @@
   base::Lock lock_;
   RegistrationList registered_;
 
   DISALLOW_COPY_AND_ASSIGN(FieldTrialList);
 };
 
 }  // namespace base
 
 #endif  // BASE_METRICS_FIELD_TRIAL_H_