[NTP::SectionOrder] Track category index after it has been moved up.

components/ntp_snippets/content_suggestions_metrics.cc

Index: components/ntp_snippets/content_suggestions_metrics.cc
diff --git a/components/ntp_snippets/content_suggestions_metrics.cc b/components/ntp_snippets/content_suggestions_metrics.cc
index b6c26ed20a829597a842ec91da8efffd00c26ef0..38860864bc854b192923f251d5c8419d9b12ad1b 100644
--- a/components/ntp_snippets/content_suggestions_metrics.cc
+++ b/components/ntp_snippets/content_suggestions_metrics.cc
@@ -55,6 +55,8 @@ const char kHistogramMoreButtonShown[] =
     "NewTabPage.ContentSuggestions.MoreButtonShown";
 const char kHistogramMoreButtonClicked[] =
     "NewTabPage.ContentSuggestions.MoreButtonClicked";
+const char kHistogramMovedUpCategoryNewIndex[] =
+    "NewTabPage.ContentSuggestions.MovedUpCategoryNewIndex";

[Marc Treib, 2017/02/01 13:04:23]

".CategoryMovedUpNewIndex"?

[vitaliii, 2017/02/01 13:30:18]

I like previous one more.

Which category? Moved up.

The new one feels like an action, "category moved up".

Also old one seems more consistent with "OpenedCategoryIndex".

Do you feel other way?

[Marc Treib, 2017/02/01 13:40:46]

Ah, I wasn't aware of OpenedCategoryIndex. I guess I'd have preferred
CategoryOpenedIndex there as well (or actually just CategoryOpened; the "Index"
is specified by the "units" field in the .xml), so that all the category-related
things start with "Category". But since it's too late for that, I guess it
doesn't matter anymore.

[vitaliii, 2017/02/01 14:19:37]

Acknowledged.

 const char kHistogramCategoryDismissed[] =
     "NewTabPage.ContentSuggestions.CategoryDismissed";
 const char kHistogramContentSuggestionsTimeSinceLastBackgroundFetch[] =
@@ -330,6 +332,11 @@ void OnSuggestionTargetVisited(Category category, base::TimeDelta visit_time) {
   base::UmaHistogramLongTimes(name, visit_time);
 }
 
+void OnCategoryMovedUp(int new_index) {
+  UMA_HISTOGRAM_EXACT_LINEAR(kHistogramMovedUpCategoryNewIndex, new_index,
+                             kMaxCategories);

[Marc Treib, 2017/02/01 13:04:23]

Hm. I guess you'd mostly look at the number of samples in the histogram over a
given period of time? Or do you see any use for the actual index values?

I'm also a bit worried that this will be hard to interpret: You'd have to
"normalize" over number of users of course, but also over total usage - if there
are more clicks overall, then there's probably also be more movement in the
categories. Or whenever we add a new category, there'll be more movement...

But I also can't immediately think of a better way to track this. Any ideas?

[vitaliii, 2017/02/01 13:30:18]

Partially true. We are worried that top most categories will wiggle around.
Thus, we could use data for the first 3 positions and ignore lower positions.
Number of users - probably, number of clicks - likely not. The order should
become more stable with time, so number of clicks should not matter. Ideally new
users will introduce some values here, but only before their order settles down.
If this highly depends on clicks, we set the thresholds too low. 
tschumann@ and I cannot find a better way as well, but this seemed good enough
to see how stable the order becomes with time.

[Marc Treib, 2017/02/01 13:40:46]

Alright, fair enough.

[Marc Treib, 2017/02/01 13:42:37]

(About tracking only the top 3 or something: No, if we do care about the index
(at least in some cases), then let's just track the index.)

[vitaliii, 2017/02/01 14:19:37]

Acknowledged.

The mention of top 3 was to explain why we need indices.

+}
+
 void OnMoreButtonShown(Category category, int position) {
   // The "more" card can appear in addition to the actual suggestions, so add
   // one extra bucket to this histogram.