| OLD | NEW |
| 1 # Chrome Network Bug Triage : Suggested Workflow | 1 # Chrome Network Bug Triage : Suggested Workflow |
| 2 | 2 |
| 3 [TOC] | 3 [TOC] |
| 4 | 4 |
| 5 ## Looking for new crashers | |
| 6 | |
| 7 1. Go to [go/chromecrash](https://goto.google.com/chromecrash). | |
| 8 | |
| 9 2. For each platform, look through the releases for which releases to | |
| 10 investigate. As per bug-triage.txt, this should be the most recent canary, | |
| 11 the previous canary (if the most recent is less than a day old), and any of | |
| 12 dev/beta/stable that were released in the last couple of days. | |
| 13 | |
| 14 3. For each release, in the "Process Type" frame, click on "browser". | |
| 15 | |
| 16 4. At the bottom of the "Magic Signature" frame, click "limit 1000". Reported | |
| 17 crashers are sorted in decreasing order of the number of reports for that | |
| 18 crash signature. | |
| 19 | |
| 20 5. Search the page for *"net::"*. | |
| 21 | |
| 22 6. For each found signature: | |
| 23 * If there is a bug already filed, make sure it is correctly describing the | |
| 24 current bug (e.g. not closed, or not describing a long-past issue), and | |
| 25 make sure that if it is a *net* bug, that it is labeled as such. | |
| 26 * Ignore signatures that only occur once, as memory corruption can easily | |
| 27 cause one-off failures when the sample size is large enough. | |
| 28 * Ignore signatures that only come from a single client ID, as individual | |
| 29 machine malware and breakage can also easily cause one-off failures. | |
| 30 * Click on the number of reports field to see details of crash. Ignore it | |
| 31 if it doesn't appear to be a network bug. | |
| 32 * Otherwise, file a new bug directly from chromecrash. Note that this may | |
| 33 result in filing bugs for low- and very-low- frequency crashes. That's | |
| 34 ok; the bug tracker is a better tool to figure out whether or not we put | |
| 35 resources into those crashes than a snap judgement when filing bugs. | |
| 36 * For each bug you file, include the following information: | |
| 37 * The backtrace. Note that the backtrace should not be added to the | |
| 38 bug if Restrict-View-Google isn't set on the bug as it may contain | |
| 39 PII. Filing the bug from the crash reporter should do this | |
| 40 automatically, but check. | |
| 41 * The channel in which the bug is seen (canary/dev/beta/stable), its | |
| 42 frequency in that channel, and its rank among crashers in the | |
| 43 channel. | |
| 44 * The frequency of this signature in recent releases. This information | |
| 45 is available by: | |
| 46 1. Clicking on the signature in the "Magic Signature" list | |
| 47 2. Clicking "Edit" on the dremel query at the top of the page | |
| 48 3. Removing the "product.version='X.Y.Z.W' AND" string and clicking | |
| 49 "Update". | |
| 50 4. Clicking "Limit 1000" in the Product Version list in the | |
| 51 resulting page (without this, the listing will be restricted to | |
| 52 the releases in which the signature is most common, which will | |
| 53 often not include the canary/dev release being investigated). | |
| 54 5. Choose some subset of that list, or all of it, to include in the | |
| 55 bug. Make sure to indicate if there is a defined point in the | |
| 56 past before which the signature is not present. | |
| 57 | |
| 58 ## Identifying unlabeled network bugs on the tracker | 5 ## Identifying unlabeled network bugs on the tracker |
| 59 | 6 |
| 60 * Look at new uncomfirmed bugs since noon PST on the last triager's rotation. | 7 * Look at new uncomfirmed bugs since noon PST on the last triager's rotation. |
| 61 [Use this issue tracker | 8 [Use this issue tracker |
| 62 query](https://code.google.com/p/chromium/issues/list?can=2&q=status%3Aunconfi
rmed&sort=-id&num=1000). | 9 query](https://code.google.com/p/chromium/issues/list?can=2&q=status%3Aunconfi
rmed&sort=-id&num=1000). |
| 63 | 10 |
| 64 * Read the title of the bug. | 11 * Read the title of the bug. |
| 65 | 12 |
| 66 * If a bug looks like it might be network/download/safe-browsing related, | 13 * If a bug looks like it might be network/download/safe-browsing related, |
| 67 middle click (or command-click on OSX) to open it in a new tab. | 14 middle click (or command-click on OSX) to open it in a new tab. |
| 68 | 15 |
| 69 * If a user provides a crash ID for a crasher for a bug that could be | 16 * If a user provides a crash ID for a crasher for a bug that could be |
| 70 net-related, look at the crash stack at | 17 net-related, look at the crash stack at |
| 71 [go/crash](https://goto.google.com/crash), and see if it looks to be network | 18 [go/crash](https://goto.google.com/crash), and see if it looks to be network |
| 72 related. Be sure to check if other bug reports have that stack trace, and | 19 related. Be sure to check if other bug reports have that stack trace, and |
| 73 mark as a dupe if so. Even if the bug isn't network related, paste the stack | 20 mark as a dupe if so. Even if the bug isn't network related, paste the stack |
| 74 trace in the bug, so no one else has to look up the crash stack from the ID. | 21 trace in the bug, so no one else has to look up the crash stack from the ID. |
| 75 * If there's no other information than the crash ID, ask for more details | 22 * If there's just a blank form and a crash ID, just ignore the bug. |
| 76 and add the Needs-Feedback label. | |
| 77 | 23 |
| 78 * If network causes are possible, ask for a net-internals log (If it's not a | 24 * If network causes are possible, ask for a net-internals log (If it's not a |
| 79 browser crash) and attach the most specific internals-network label that's | 25 browser crash) and attach the most specific internals-network label that's |
| 80 applicable. If there isn't an applicable narrower component, a clear owner | 26 applicable. If there isn't an applicable narrower component, a clear owner |
| 81 for the issue, or there are multiple possibilities, attach the | 27 for the issue, or there are multiple possibilities, attach the |
| 82 Internals>Network component and proceed with further investigation. | 28 Internals>Network component and proceed with further investigation. |
| 83 | 29 |
| 84 * If non-network causes also seem possible, attach those components as well. | 30 * If non-network causes also seem possible, attach those components as well. |
| 85 | 31 |
| 86 ## Investigating component=Internals>Network bugs | 32 ## Investigating component=Internals>Network bugs |
| 87 | 33 |
| 88 * It's recommended that while on triage duty, you subscribe to the | 34 * It's recommended that while on triage duty, you subscribe to the |
| 89 Internals>Network component (but not its subcomponents). To do this, go | 35 Internals>Network component (but not its subcomponents). To do this, go |
| 90 to the issue tracker and then click "Saved Queries". | 36 to the issue tracker and then click "Saved Queries". |
| 91 Add a query with these settings: | 37 Add a query with these settings: |
| 92 * Saved query name: Network Bug Triage | 38 * Saved query name: Network Bug Triage |
| 93 * Project: chromium | 39 * Project: chromium |
| 94 * Query: component=Internals>Network | 40 * Query: component=Internals>Network |
| 95 * Subscription options: Notify Immediately | 41 * Subscription options: Notify Immediately |
| 96 | 42 |
| 97 * Look through uncomfirmed and untriaged component=Internals>Network bugs, | 43 * Look through uncomfirmed and untriaged component=Internals>Network bugs, |
| 98 prioritizing those updated within the last week. [Use this issue tracker | 44 prioritizing those updated within the last week. [Use this issue tracker |
| 99 query](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3DIn
ternals%3ENetwork+-status%3AAssigned+-status%3AStarted+-status%3AAvailable&sort=
-modified&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+
Modified&x=m&y=releaseblock&cells=ids). | 45 query](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3DIn
ternals%3ENetwork+status%3AUnconfirmed,Untriaged+-label:Needs-Feedback&sort=-mod
ified). |
| 100 | 46 |
| 101 * If more information is needed from the reporter, ask for it and add the | 47 * If more information is needed from the reporter, ask for it and add the |
| 102 Needs-Feedback label. If the reporter has answered an earlier request for | 48 Needs-Feedback label. |
| 103 information, remove that label. | |
| 104 | 49 |
| 105 * While investigating a new issue, change the status to Untriaged. | 50 * While investigating a new issue, change the status to Untriaged. |
| 106 | 51 |
| 107 * If a bug is a potential security issue (Allows for code execution from remote | 52 * If a bug is a potential security issue (Allows for code execution from remote |
| 108 site, allows crossing security boundaries, unchecked array bounds, etc) mark | 53 site, allows crossing security boundaries, unchecked array bounds, etc) mark |
| 109 it Type-Bug-Security. If it has privacy implication (History, cookies | 54 it Type-Bug-Security. If it has privacy implication (History, cookies |
| 110 discoverable by an entity that shouldn't be able to do so, incognito state | 55 discoverable by an entity that shouldn't be able to do so, incognito state |
| 111 being saved in memory or on disk beyond the lifetime of incognito tabs, etc), | 56 being saved in memory or on disk beyond the lifetime of incognito tabs, etc), |
| 112 mark it with component Privacy. | 57 mark it with component Privacy. |
| 113 | 58 |
| 114 * For bugs that already have a more specific network component, go ahead and | 59 * For bugs that already have a more specific network component, go ahead and |
| 115 remove the Internals>Network component and move on. | 60 remove the Internals>Network component to get them off the next triager's |
| 61 radar and move on. |
| 116 | 62 |
| 117 * Try to figure out if it's really a network bug. See common non-network | 63 * Try to figure out if it's really a network bug. See common non-network |
| 118 components section for description of common components for issues incorrectly | 64 components section for description of common components for issues incorrectly |
| 119 tagged as Internals>Network. | 65 tagged as Internals>Network. |
| 120 | 66 |
| 121 * If it's not, attach appropriate labels/components and go no further. | 67 * If it's not, attach appropriate labels/components and go no further. |
| 122 | 68 |
| 123 * If it may be a network bug, attach additional possibly relevant component if | 69 * If it may be a network bug, attach additional possibly relevant component if |
| 124 any, and continue investigating. Once you either determine it's a | 70 any, and continue investigating. Once you either determine it's a |
| 125 non-network bug, or figure out accurate more specific network components, your | 71 non-network bug, or figure out accurate more specific network components, your |
| (...skipping 28 matching lines...) Expand all Loading... |
| 154 * If you are having trouble with an issue, particularly for help understanding | 100 * If you are having trouble with an issue, particularly for help understanding |
| 155 net-internals logs, email the public net-dev@chromium.org list for help | 101 net-internals logs, email the public net-dev@chromium.org list for help |
| 156 debugging. If it's a crasher, or for some other reason discussion needs to | 102 debugging. If it's a crasher, or for some other reason discussion needs to |
| 157 be done in private, use chrome-network-debugging@google.com. TODO(mmenke): | 103 be done in private, use chrome-network-debugging@google.com. TODO(mmenke): |
| 158 Write up a net-internals tips and tricks docs. | 104 Write up a net-internals tips and tricks docs. |
| 159 | 105 |
| 160 * If it appears to be a bug in the unowned core of the network stack (i.e. no | 106 * If it appears to be a bug in the unowned core of the network stack (i.e. no |
| 161 subcomponent applies, or only the Internals>Network>HTTP subcomponent | 107 subcomponent applies, or only the Internals>Network>HTTP subcomponent |
| 162 applies, and there's no clear owner), try to figure out the exact cause. | 108 applies, and there's no clear owner), try to figure out the exact cause. |
| 163 | 109 |
| 164 ## Monitoring UMA histograms and Chirp/Gasper alerts | 110 ## Investigate UMA notifications |
| 165 | |
| 166 Sign up to chrome-network-debugging@google.com mailing list to receive automated | |
| 167 e-mails about UMA alerts. Chirp is the new alert system, sending automated | |
| 168 e-mails with sender address finch-chirp@google.com. Gasper is the old alert | |
| 169 system, sending automated e-mails with sender address gasper-alerts@google.com. | |
| 170 While Chirp is of higher priority, Gasper is not deprecated yet, so both alerts | |
| 171 should be monitored for the time being. | |
| 172 | 111 |
| 173 For each alert that fires, determine if it's a real alert and file a bug if so. | 112 For each alert that fires, determine if it's a real alert and file a bug if so. |
| 174 | 113 |
| 175 * Don't file if the alert is coincident with a major volume change. The volume | 114 * Don't file if the alert is coincident with a major volume change. The volume |
| 176 at a particular date can be determined by hovering the mouse over the | 115 at a particular date can be determined by hovering the mouse over the |
| 177 appropriate location on the alert line. | 116 appropriate location on the alert line. |
| 178 | 117 |
| 179 * Don't file if the alert is on a graph with very low volume (< ~200 data | 118 * Don't file if the alert is on a graph with very low volume (< ~200 data |
| 180 points); it's probably noise, and we probably don't care even if it isn't. | 119 points); it's probably noise, and we probably don't care even if it isn't. |
| 181 | 120 |
| 182 * Don't file if the graph is really noisy (but eyeball it to decide if there is | 121 * Don't file if the graph is really noisy (but eyeball it to decide if there is |
| 183 an underlying important shift under the noise). | 122 an underlying important shift under the noise). |
| 184 | 123 |
| 185 * Don't file if the alert is in the "Known Ignorable" list: | 124 * Don't file if the alert is in the "Known Ignorable" list: |
| 186 * SimpleCache on Windows | 125 * SimpleCache on Windows |
| 187 * DiskCache on Android. | 126 * DiskCache on Android. |
| 188 | 127 |
| 189 For each alert, respond to chrome-network-debugging@google.com with a summary of | 128 ## Looking for new crashers |
| 190 the action you've taken and why, including issue link if an issue was filed. | 129 |
| 130 1. Go to [go/chromecrash](https://goto.google.com/chromecrash). |
| 131 |
| 132 2. For each platform, look through the releases for which releases to |
| 133 investigate. As per bug-triage.txt, this should be the most recent canary, |
| 134 the previous canary (if the most recent is less than a day old), and any of |
| 135 dev/beta/stable that were released in the last couple of days. |
| 136 |
| 137 3. For each release, in the "Process Type" frame, click on "browser". |
| 138 |
| 139 4. At the bottom of the "Magic Signature" frame, click "limit 1000" (Or reduce |
| 140 the limit to 100 first, as that's all the triager needs to look at). |
| 141 Reported crashers are sorted in decreasing order of the number of reports for |
| 142 that crash signature. |
| 143 |
| 144 5. Search the page for *"net::"*. |
| 145 |
| 146 6. For each found signature: |
| 147 * Ignore signatures that only occur once or twice, as memory corruption can |
| 148 easily cause one-off failures when the sample size is large enough. Also |
| 149 ignore crashers that are not in the top 100 for that platform / release. |
| 150 * If there is a bug already filed, make sure it is correctly describing the |
| 151 current bug (e.g. not closed, or not describing a long-past issue), and |
| 152 make sure that if it is a *net* bug, that it is labeled as such. |
| 153 * Ignore signatures that only come from one or two client IDs, as individual |
| 154 machine malware and breakage can cause one-off failures. |
| 155 * Click on the number of reports field to see details of crash. Ignore it |
| 156 if it doesn't appear to be a network bug. |
| 157 * Otherwise, file a new bug directly from chromecrash. |
| 158 * For each bug you file, include the following information: |
| 159 * The backtrace. Note that the backtrace should not be added to the |
| 160 bug if Restrict-View-Google isn't set on the bug as it may contain |
| 161 PII. Filing the bug from the crash reporter should do this |
| 162 automatically, but check. |
| 163 * The channel in which the bug is seen (canary/dev/beta/stable), and its |
| 164 rank among crashers in the channel. |
| 165 * The frequency of this signature in recent releases. This information |
| 166 is available by: |
| 167 1. Clicking on the signature in the "Magic Signature" list |
| 168 2. Clicking "Edit" on the dremel query at the top of the page |
| 169 3. Removing the "product.version='X.Y.Z.W' AND" string and clicking |
| 170 "Update". |
| 171 4. Clicking "Limit 1000" in the Product Version list in the |
| 172 resulting page (without this, the listing will be restricted to |
| 173 the releases in which the signature is most common, which will |
| 174 often not include the canary/dev release being investigated). |
| 175 5. Choose some subset of that list, or all of it, to include in the |
| 176 bug. Make sure to indicate if there is a defined point in the |
| 177 past before which the signature is not present. |
| 191 | 178 |
| 192 ## Investigating crashers | 179 ## Investigating crashers |
| 193 | 180 |
| 194 * Only investigate crashers that are still occurring, as identified by above | 181 * Only investigate crashers that are still occurring, as identified by above |
| 195 section. If a search on go/crash indicates a crasher is no longer occurring, | 182 section. If a search on go/crash indicates a crasher is no longer occurring, |
| 196 mark it as WontFix. | 183 mark it as WontFix. |
| 197 | 184 |
| 198 * On Windows, you may want to look for weird dlls associated with the crashes. | 185 * On Windows, you may want to look for weird dlls associated with the crashes. |
| 199 This generally needs crashes from a fair number of different users to reach | 186 This generally needs crashes from a fair number of different users to reach |
| 200 any conclusions. | 187 any conclusions. |
| (...skipping 13 matching lines...) Expand all Loading... |
| 214 a crash report, and seeing if there are multiple reports for the same crash. | 201 a crash report, and seeing if there are multiple reports for the same crash. |
| 215 If this is the case, it may be also be malware, or an issue with an unusual | 202 If this is the case, it may be also be malware, or an issue with an unusual |
| 216 system/chrome/network config. | 203 system/chrome/network config. |
| 217 | 204 |
| 218 * Dig through crash reports to figure out when the crash first appeared, and | 205 * Dig through crash reports to figure out when the crash first appeared, and |
| 219 dig through revision history in related files to try and locate a suspect CL. | 206 dig through revision history in related files to try and locate a suspect CL. |
| 220 TODO(mmenke): Add more detail here. | 207 TODO(mmenke): Add more detail here. |
| 221 | 208 |
| 222 * Load crash dumps, try to figure out a cause. See | 209 * Load crash dumps, try to figure out a cause. See |
| 223 http://www.chromium.org/developers/crash-reports for more information | 210 http://www.chromium.org/developers/crash-reports for more information |
| 224 | |
| 225 ## Dealing with old bugs | |
| 226 | |
| 227 * For all network issues (Even those with owners, or a more specific component): | |
| 228 | |
| 229 * If the issue has had the Needs-Feedback label for over a month, verify it | |
| 230 is waiting on feedback from the user. If not, remove the label. | |
| 231 Otherwise, go ahead and mark the issue WontFix due to lack of response | |
| 232 and suggest the user file a new bug if the issue is still present. [Use | |
| 233 this issue tracker query for old Needs-Feedback | |
| 234 issues](https://code.google.com/p/chromium/issues/list?can=2&q=component%3
AInternals>Network%20Needs=Feedback+modified-before%3Atoday-30&sort=-modified). | |
| 235 | |
| 236 * If a bug is over 2 months old, and the underlying problem was never | |
| 237 reproduced or really understood: | |
| 238 * If it's over a year old, go ahead and mark the issue as Archived. | |
| 239 * Otherwise, ask reporters if the issue is still present, and attach | |
| 240 the Needs-Feedback label. | |
| 241 | |
| 242 * Old unconfirmed or untriaged Internals>Network issues can be investigated | |
| 243 just like newer ones. Crashers should generally be given higher priority, | |
| 244 since we can verify if they still occur, and then newer issues, as they're | |
| 245 more likely to still be present, and more likely to have a still responsive | |
| 246 bug reporter. | |
| OLD | NEW |