Implement UMA log throttling for cellular connections

components/metrics/data_use_tracker.cc

Index: components/metrics/data_use_tracker.cc
diff --git a/components/metrics/data_use_tracker.cc b/components/metrics/data_use_tracker.cc
new file mode 100644
index 0000000000000000000000000000000000000000..2ab89c751732fa1dbdecbff6791e593610fbb47e
--- /dev/null
+++ b/components/metrics/data_use_tracker.cc
@@ -0,0 +1,183 @@
+// Copyright 2016 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#include "components/metrics/data_use_tracker.h"
+
+#include <string>
+
+#include "base/strings/string_number_conversions.h"
+#include "base/strings/stringprintf.h"
+#include "components/metrics/metrics_pref_names.h"
+#include "components/prefs/scoped_user_pref_update.h"
+#include "components/variations/variations_associated_data.h"
+
+namespace metrics {
+
+namespace {
+
+// This function is for forwarding metrics usage pref changes to the appropriate
+// callback on the appropriate thread.
+void UpdateMetricsUsagePrefs(
+    const UpdateUsagePrefCallbackType& update_on_ui_callback,
+    scoped_refptr<base::SequencedTaskRunner> ui_task_runner,
+    const std::string& service_name,
+    int message_size) {
+  ui_task_runner->PostTask(
+      FROM_HERE, base::Bind(update_on_ui_callback, service_name, message_size));
+}
+
+}  // namespace
+
+DataUseTracker::DataUseTracker(PrefService* local_state)
+    : local_state_(local_state),
+      weak_ptr_factory_(this) {}
+
+DataUseTracker::~DataUseTracker() {}
+
+// static
+void DataUseTracker::RegisterPrefs(PrefRegistrySimple* registry) {
+  registry->RegisterDictionaryPref(metrics::prefs::kUserCellDataUse);
+  registry->RegisterDictionaryPref(metrics::prefs::kUmaCellDataUse);
+}
+
+UpdateUsagePrefCallbackType DataUseTracker::GetDataUseForwardingCallback(
+    scoped_refptr<base::SequencedTaskRunner> ui_task_runner) {
+  DCHECK(ui_task_runner->RunsTasksOnCurrentThread());
+
+  return base::Bind(
+      &UpdateMetricsUsagePrefs,
+      base::Bind(&DataUseTracker::UpdateMetricsUsagePrefsOnUIThread,
+                 weak_ptr_factory_.GetWeakPtr()),
+      ui_task_runner);
+}
+
+bool DataUseTracker::ShouldUploadLogOnCellular(int log_bytes) {
+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  RemoveExpiredEntries();
+
+  int uma_weekly_quota_bytes;
+  if (!GetUmaWeeaklyQuota(&uma_weekly_quota_bytes))
+    return true;
+
+  int uma_total_data_use = ComputeTotalDataUse(prefs::kUmaCellDataUse);
+  int new_uma_total_data_use = log_bytes + uma_total_data_use;
+  // If the new log doesn't increase the total UMA traffic to be above the
+  // allowed quota then the log should be uploaded.
+  if (new_uma_total_data_use <= uma_weekly_quota_bytes)
+    return true;
+
+  double uma_ratio;
+  if (!GetUmaRatio(&uma_ratio))
+    return true;
+
+  int user_total_data_use = ComputeTotalDataUse(prefs::kUserCellDataUse);
+  // If after adding the new log the uma ratio is still under the allowed ratio
+  // then the log should be uploaded and vice versa.
+  return new_uma_total_data_use /
+             static_cast<double>(log_bytes + user_total_data_use) <=
+         uma_ratio;
+}
+
+void DataUseTracker::UpdateMetricsUsagePrefsOnUIThread(
+    const std::string& service_name,
+    int message_size) {
+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  UpdateUsagePref(prefs::kUserCellDataUse, message_size);
+  if (service_name == "UMA")
+    UpdateUsagePref(prefs::kUmaCellDataUse, message_size);

[mmenke, 2016/04/04 17:26:23]

Recording UMA and non-UMA data use through a method in components/metrics with a
magic call from data_use_measurement seems a bit funky.  What if some other
service wants similar data?  Add another daily use_tracker for that component,
and introduce a dependency on that component to data_use_measurement, too?

Wonder if there's some way we could fairly easily standardize this, so any
service could pull their usage from the MetricsService, without the service
depending on them.  And don't think "kUserCellDataUse" should go here at all.

[gayane -on leave until 09-2017, 2016/04/05 14:26:46]

For now we would like to manage to get to M51 which is this Friday, because a
lot of people are waiting for UMA on cellular. For future we are planning to do
the same for IOS which already will require significant refactoring. So we can
include the suggested ideas at that time. 
For now I have added a comment on data_use_measurement to warn other components
which will decide to do the same. 
For kUserCellDataUse, this is a pref which is part of our logic. I don't see why
this doesn't belong here.

[mmenke, 2016/04/05 20:18:32]

Not going to block on this, but my general feeling is if we're going to keep
this around, it's best to do it right the first time than not.  You'll either be
stuck with the designed, which becomes progressively harder and harder to change
over time, or you'll have to go through review yet again, wasting everyone's
time.

+}
+
+void DataUseTracker::UpdateUsagePref(const std::string& pref_name,
+                                     int message_size) {

[mmenke, 2016/04/04 17:26:23]

All this does is write to a pref from another thread?  PrefMember supports that
directly - seems like re-inventing the wheel a bit here.

[gayane -on leave until 09-2017, 2016/04/05 14:26:46]

For using PrefMember we can have pass PrefMember object, which can be modified
on IO thread to data_use_measurment instead of the the current callback. What do
you think?

[mmenke, 2016/04/05 20:18:32]

Could DataUseTracker::GetDataUseForwardingCallback do something like:

scoped_ptr<PrefMember> user_pref(...);  // Create it from local_state_.
user_pre
user_pref->MoveToThread(ui);
... same for uma_pref.

Callback = base::Bind(&UpdatePrefsOnIO, base::Owned(user_pref),
base::Owned(uma_pref));

And then just do them all on the IO thread?  That doesn't actually save on
PostTasks, which I suggested elsewhere, but it does leave it all in PrefMember's
camp, at least.

Anyhow, not a huge savings in complexity, I suppose, since we still need stuff
on the UI thread.

+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  DictionaryPrefUpdate pref_updater(local_state_, pref_name);
+  int todays_traffic = 0;
+  std::string todays_key = GetCurrentMeasurementDateAsString();
+
+  const base::DictionaryValue* user_pref_dict =
+      local_state_->GetDictionary(pref_name);
+  user_pref_dict->GetInteger(todays_key, &todays_traffic);
+  pref_updater->SetInteger(todays_key, todays_traffic + message_size);
+}
+
+void DataUseTracker::RemoveExpiredEntries() {
+  DCHECK(thread_checker_.CalledOnValidThread());
+  RemoveExpiredEntriesForPref(prefs::kUmaCellDataUse);
+  RemoveExpiredEntriesForPref(prefs::kUserCellDataUse);
+}
+
+void DataUseTracker::RemoveExpiredEntriesForPref(const std::string& pref_name) {
+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  const base::DictionaryValue* user_pref_dict =
+      local_state_->GetDictionary(pref_name);
+  const base::Time current_date = GetCurrentMeasurementDate();
+  const base::Time week_ago = current_date - base::TimeDelta::FromDays(7);
+
+  base::DictionaryValue user_pref_new_dict;
+  for (base::DictionaryValue::Iterator it(*user_pref_dict); !it.IsAtEnd();
+       it.Advance()) {
+    base::Time key_date;
+    base::Time::FromUTCString(it.key().c_str(), &key_date);
+    if (key_date > week_ago)
+      user_pref_new_dict.Set(it.key(), it.value().CreateDeepCopy());
+  }
+  local_state_->Set(pref_name, user_pref_new_dict);

[mmenke, 2016/04/04 17:26:23]

DataReductionProxyCompressionStats has its own magic that may or may not do
something similar for recording / aggregating daily stats.  I don't suppose we
could share code?  (Disclaimer:  I haven't dug into it, just notice they're
dealing with similar issues to get daily stats)

[gayane -on leave until 09-2017, 2016/04/05 14:26:46]

I had a quick look but It didn't seem to me similar enough, it is not even using
data_use_measurement component.

[mmenke, 2016/04/05 20:18:32]

Right...which means things would have to move around a bit to share code, which
is preferable to code duplication.  I'd suggest you at least test this with a
lot of cases (Time going backwards a day and a year, various amount of days in
cache).

+}
+
+int DataUseTracker::ComputeTotalDataUse(std::string pref_name) {
+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  int total_data_use = 0;
+  const base::DictionaryValue* pref_dict =
+      local_state_->GetDictionary(pref_name);
+  for (base::DictionaryValue::Iterator it(*pref_dict); !it.IsAtEnd();
+       it.Advance()) {
+    int value = 0;
+    it.value().GetAsInteger(&value);
+    total_data_use += value;
+  }
+  return total_data_use;
+}
+
+bool DataUseTracker::GetUmaWeeaklyQuota(int* uma_weekly_quota_bytes) {

[mmenke, 2016/04/04 17:26:23]

Weeakly -> Weekly

[gayane -on leave until 09-2017, 2016/04/05 14:26:46]

Done.

+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  std::string param_value_str = variations::GetVariationParamValue(
+      "UMA_EnableCellularLogUpload", "Uma_Quota");
+  if (param_value_str.empty())
+    return false;
+
+  base::StringToInt(param_value_str, uma_weekly_quota_bytes);
+  return true;
+}
+
+bool DataUseTracker::GetUmaRatio(double* ratio) {
+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  std::string param_value_str = variations::GetVariationParamValue(
+      "UMA_EnableCellularLogUpload", "Uma_Ratio");
+  if (param_value_str.empty())
+    return false;
+  base::StringToDouble(param_value_str, ratio);
+  return true;
+}
+
+base::Time DataUseTracker::GetCurrentMeasurementDate() {
+  return base::Time::Now().LocalMidnight();
+}
+
+std::string DataUseTracker::GetCurrentMeasurementDateAsString() {
+  DCHECK(thread_checker_.CalledOnValidThread());
+
+  base::Time::Exploded today_exploded;
+  GetCurrentMeasurementDate().LocalExplode(&today_exploded);
+  return base::StringPrintf("%04d-%02d-%02d", today_exploded.year,
+                            today_exploded.month, today_exploded.day_of_month);
+}
+
+}  // namespace metrics