Clarifications to network bug triage process.

net/docs/bug-triage-suggested-workflow.txt

Index: net/docs/bug-triage-suggested-workflow.txt
diff --git a/net/docs/bug-triage-suggested-workflow.txt b/net/docs/bug-triage-suggested-workflow.txt
index d3e7141dc4be95c16de369aec4f43c27e209cad1..0696aec7ecab8303d9bd53253d55913dde12a8c8 100644
--- a/net/docs/bug-triage-suggested-workflow.txt
+++ b/net/docs/bug-triage-suggested-workflow.txt
@@ -1,3 +1,52 @@
+Look for new crashers:
+* Go to go/chromecrash.
+* Choose the platforms and releases for which you will be
+  investigating crashers.  As per bug-triage.txt, this should be the
+  most recent canary, the previous canary (if the most recent is less
+  than a day old), and any of dev/beta/stable that were released in the
+  last couple of days.  These releases should be investigated for all
+  platforms. 
+* For each release, in the "Process Type" frame, click on "browser".
+* At the bottom of the "Magic Signature" frame,  click "limit 1000".
+  Reported crashers are sorted in decreasing order of the number of reports for
+  that crash signature.
+* Search the page for "net::".  
+* For each found signature:
+  * If there is a bug already filed, make sure it is correctly
+    describing the current bug (e.g. not closed, or not describing a
+    long-past issue).  If there is a valid associated bug, ignore this
+    signature. 
+  * Ignore signatures that only occur once, as memory corruption can
+    easily cause one-off failures when the sample size is large
+    enough.
+  * Ignore signatures that only come from a single client ID, as
+    individual machine malware and breakage can also easily cause
+    one-off failures.  
+  * Click on the number of reports field to see details of
+    crash. Ignore it if it doesn't appear to be a network bug. 
+  * Otherwise, file a new bug directly from chromecrash.  Note that
+    this may result in filing bugs for low- and very-low- frequency
+    crashes.  That's ok; the bug tracker is a better tool to figure
+    out whether or not we put resources into those crashes than a snap
+    judgement when filing bugs.

[Ryan Sleevi, 2015/03/02 19:01:09]

There is more needed here - for example, examining third-party modules for signs
of malware (known and unknown)

More importantly, however, the TPMs are already executing this crasher triage as
part of their TPM duties, and already providing a high quality signal/noise
filtering. That is, considering population of distribution to population of
crashes.

[mmenke, 2015/03/02 19:15:13]

I'd take issue with "high quality signal/noise filtering".  I've repeatedly seen
something called a "top-10 crasher" when only one user had 3 or 4 crashes in a
couple hours of Canary data.

I really don't feel like we're doing a good or consistent job of catching
crashers.  We're doing a worse job of investigating crashers, but that's beyond
the scope of this document.  :(

[Randy Smith (Not in Mondays), 2015/03/03 15:08:01]

So I thought about doing that, but I have no clue how to actually write that
section.  Want to suggest some text/pop up your own CL?
Conversation with TPMs in progress; I'll reflect the results in this CL.

[Randy Smith (Not in Mondays), 2015/03/03 15:08:02]

Acknowledged.

+* For each bug you file, include the following information:
+  * The backtrace

[Ryan Sleevi, 2015/03/02 19:01:09]

Do not do this unless you've confirmed with the TPMs on the privacy
implications. This _used_ to be practice and was stopped.

[Randy Smith (Not in Mondays), 2015/03/03 15:08:01]

Conversation in process.

[Randy Smith (Not in Mondays), 2015/03/04 22:22:21]

Refined contexts under which adding a backtrace is appropriate.

+  * The channel in which the bug is seen (canary/dev/beta/stable) and
+    its frequency in that channel.
+  * The frequency of this signature in recent releases.  This
+    information is available by:
+    * Clicking on the signature in the "Magic Signature" list
+    * Clicking "Edit" on the dremel query at the top of the page
+    * Removing the "product.version='X.Y.Z.W' AND" string and clicking
+      "Update".
+    * Clicking "Limit 1000" in the Product Version list in the
+      resulting page (without this, the listing will be restricted to
+      the releases in which the signature is most common, which will
+      often not include the canary/dev release being investigated).
+    * Choose some subset of that list, or all of it, to include in the
+      bug.  Make sure to indicate if there is a defined point in the
+      past before which the signature is not present.
+

[Ryan Sleevi, 2015/03/02 19:01:09]

Please run these last two steps by the TPMs.

[Randy Smith (Not in Mondays), 2015/03/03 15:08:02]

These steps are ok by them (laforge@ specifically).

 Identifying unlabeled network bugs on the tracker:
 * Look at new uncomfirmed bugs since noon PST on the last triager's rotation:
     https://code.google.com/p/chromium/issues/list?can=2&q=status%3Aunconfirmed&sort=-id&num=1000
@@ -20,6 +69,15 @@ Identifying unlabeled network bugs on the tracker:
     label and proceed with further investigation.
 * If non-network causes also seem possible, attach those labels as well.
 
+Request data about recent unassigned Cr-Internals-Network bugs from reporters. 
+* For all bugs in this category marked "Needs-Feedback", go through
+  them and remove that label if the reporter has responded with the
+  requested information.
+* For all bugs in this category not marked "Needs-Feedback", go
+  through those bugs and determine if more information is needed from
+  the reporter to make forward progress on the bug.  IF it is, request

[mmenke, 2015/03/02 19:15:13]

nit:  IF -> If

[Randy Smith (Not in Mondays), 2015/03/03 15:08:02]

Done.

+  that information and mark the bug Needs-Feedback.  
+
 Investigating Cr-Internals-Network bugs:
 * It's recommended that while on triage duty, you subscribe to the
     Cr-Internals-Network label.  To do this, go to
@@ -75,25 +133,22 @@ Investigating Cr-Internals-Network bugs:
     sublabel applies, or only the Cr-Internals-Network-HTTP sublabel applies,
     and there's no clear owner), try to figure out the exact cause.
 
-Look for new crashers:
-* Go to go/chromecrash.
-* For each platform, go to the latest canary.
-* In the "Process Type" frame, click on "browser".
-* At the bottom of the "Magic Signature" frame,  click "limit 1000".
-  Reported crashers are sorted in decreasing order of the number of reports for
-  that crash signature.
-* Search the page for "net::".  Ignore crashes that only occur once, as
-    memory corruption can easily cause one-off failures when the sample size is
-    large enough.
-* Click on the number of reports field to see details of crash. Look at the
-    stack trace to confirm it's a network bug.
-  * If it is, and there's no associated bug filed, file a new bug directly from
-      chromecrash, looking at earlier canaries to determine if it's a recent
-      regression.  Use the most specific label possible.
-* The most recent Canary may not yet have a full day of crashes, so it may be
-    worth looking at more than one version.
-* If there's been a dev, beta, or stable release in the last couple days, should
-    also look at those.
+Monitor UMA histograms and gasper alerts.  For each Gasper alert that
+fires, determine if it's a real alert and file a bug if so.  
+* Don't file if the alert is coincident with a major volume change.
+  The volume at a particular date can be determined by hovering the
+  mouse over the appropriate location on the alert line.
+* Don't file if the alert is on a graph with very low volume (< ~200
+  data points); it's probably noise, and we probably don't care even
+  if it isn't.
+* Don't file if the graph is really noisy (but eyeball it to decide if
+  there is an underlying important shift under the noise).
+* Don't file if the alert is in the "Known Ignorable" list:
+  * SimpleCache on Windows
+  * DiskCache on Android.
+For each Gasper alert, respond to chrome-network-debugging@ with a
+summary of the action you've taken and why, including issue link if an
+issue was filed.  
 
 Investigating crashers:
 * Only investigate crashers that are still occurring, as identified by above