OLD | NEW |
1 # Chrome Network Bug Triage : Suggested Workflow | 1 # Chrome Network Bug Triage : Suggested Workflow |
2 | 2 |
3 [TOC] | 3 [TOC] |
4 | 4 |
5 ## Looking for new crashers | |
6 | |
7 1. Go to [go/chromecrash](https://goto.google.com/chromecrash). | |
8 | |
9 2. For each platform, look through the releases for which releases to | |
10 investigate. As per bug-triage.txt, this should be the most recent canary, | |
11 the previous canary (if the most recent is less than a day old), and any of | |
12 dev/beta/stable that were released in the last couple of days. | |
13 | |
14 3. For each release, in the "Process Type" frame, click on "browser". | |
15 | |
16 4. At the bottom of the "Magic Signature" frame, click "limit 1000". Reported | |
17 crashers are sorted in decreasing order of the number of reports for that | |
18 crash signature. | |
19 | |
20 5. Search the page for *"net::"*. | |
21 | |
22 6. For each found signature: | |
23 * If there is a bug already filed, make sure it is correctly describing the | |
24 current bug (e.g. not closed, or not describing a long-past issue), and | |
25 make sure that if it is a *net* bug, that it is labeled as such. | |
26 * Ignore signatures that only occur once, as memory corruption can easily | |
27 cause one-off failures when the sample size is large enough. | |
28 * Ignore signatures that only come from a single client ID, as individual | |
29 machine malware and breakage can also easily cause one-off failures. | |
30 * Click on the number of reports field to see details of crash. Ignore it | |
31 if it doesn't appear to be a network bug. | |
32 * Otherwise, file a new bug directly from chromecrash. Note that this may | |
33 result in filing bugs for low- and very-low- frequency crashes. That's | |
34 ok; the bug tracker is a better tool to figure out whether or not we put | |
35 resources into those crashes than a snap judgement when filing bugs. | |
36 * For each bug you file, include the following information: | |
37 * The backtrace. Note that the backtrace should not be added to the | |
38 bug if Restrict-View-Google isn't set on the bug as it may contain | |
39 PII. Filing the bug from the crash reporter should do this | |
40 automatically, but check. | |
41 * The channel in which the bug is seen (canary/dev/beta/stable), its | |
42 frequency in that channel, and its rank among crashers in the | |
43 channel. | |
44 * The frequency of this signature in recent releases. This information | |
45 is available by: | |
46 1. Clicking on the signature in the "Magic Signature" list | |
47 2. Clicking "Edit" on the dremel query at the top of the page | |
48 3. Removing the "product.version='X.Y.Z.W' AND" string and clicking | |
49 "Update". | |
50 4. Clicking "Limit 1000" in the Product Version list in the | |
51 resulting page (without this, the listing will be restricted to | |
52 the releases in which the signature is most common, which will | |
53 often not include the canary/dev release being investigated). | |
54 5. Choose some subset of that list, or all of it, to include in the | |
55 bug. Make sure to indicate if there is a defined point in the | |
56 past before which the signature is not present. | |
57 | |
58 ## Identifying unlabeled network bugs on the tracker | 5 ## Identifying unlabeled network bugs on the tracker |
59 | 6 |
60 * Look at new uncomfirmed bugs since noon PST on the last triager's rotation. | 7 * Look at new uncomfirmed bugs since noon PST on the last triager's rotation. |
61 [Use this issue tracker | 8 [Use this issue tracker |
62 query](https://code.google.com/p/chromium/issues/list?can=2&q=status%3Aunconfi
rmed&sort=-id&num=1000). | 9 query](https://code.google.com/p/chromium/issues/list?can=2&q=status%3Aunconfi
rmed&sort=-id&num=1000). |
63 | 10 |
64 * Read the title of the bug. | 11 * Read the title of the bug. |
65 | 12 |
66 * If a bug looks like it might be network/download/safe-browsing related, | 13 * If a bug looks like it might be network/download/safe-browsing related, |
67 middle click (or command-click on OSX) to open it in a new tab. | 14 middle click (or command-click on OSX) to open it in a new tab. |
68 | 15 |
69 * If a user provides a crash ID for a crasher for a bug that could be | 16 * If a user provides a crash ID for a crasher for a bug that could be |
70 net-related, look at the crash stack at | 17 net-related, look at the crash stack at |
71 [go/crash](https://goto.google.com/crash), and see if it looks to be network | 18 [go/crash](https://goto.google.com/crash), and see if it looks to be network |
72 related. Be sure to check if other bug reports have that stack trace, and | 19 related. Be sure to check if other bug reports have that stack trace, and |
73 mark as a dupe if so. Even if the bug isn't network related, paste the stack | 20 mark as a dupe if so. Even if the bug isn't network related, paste the stack |
74 trace in the bug, so no one else has to look up the crash stack from the ID. | 21 trace in the bug, so no one else has to look up the crash stack from the ID. |
75 * If there's no other information than the crash ID, ask for more details | 22 * If there's just a blank form and a crash ID, just ignore the bug. |
76 and add the Needs-Feedback label. | |
77 | 23 |
78 * If network causes are possible, ask for a net-internals log (If it's not a | 24 * If network causes are possible, ask for a net-internals log (If it's not a |
79 browser crash) and attach the most specific internals-network label that's | 25 browser crash) and attach the most specific internals-network label that's |
80 applicable. If there isn't an applicable narrower component, a clear owner | 26 applicable. If there isn't an applicable narrower component, a clear owner |
81 for the issue, or there are multiple possibilities, attach the | 27 for the issue, or there are multiple possibilities, attach the |
82 Internals>Network component and proceed with further investigation. | 28 Internals>Network component and proceed with further investigation. |
83 | 29 |
84 * If non-network causes also seem possible, attach those components as well. | 30 * If non-network causes also seem possible, attach those components as well. |
85 | 31 |
86 ## Investigating component=Internals>Network bugs | 32 ## Investigating component=Internals>Network bugs |
87 | 33 |
88 * It's recommended that while on triage duty, you subscribe to the | 34 * It's recommended that while on triage duty, you subscribe to the |
89 Internals>Network component (but not its subcomponents). To do this, go | 35 Internals>Network component (but not its subcomponents). To do this, go |
90 to the issue tracker and then click "Saved Queries". | 36 to the issue tracker and then click "Saved Queries". |
91 Add a query with these settings: | 37 Add a query with these settings: |
92 * Saved query name: Network Bug Triage | 38 * Saved query name: Network Bug Triage |
93 * Project: chromium | 39 * Project: chromium |
94 * Query: component=Internals>Network | 40 * Query: component=Internals>Network |
95 * Subscription options: Notify Immediately | 41 * Subscription options: Notify Immediately |
96 | 42 |
97 * Look through uncomfirmed and untriaged component=Internals>Network bugs, | 43 * Look through uncomfirmed and untriaged component=Internals>Network bugs, |
98 prioritizing those updated within the last week. [Use this issue tracker | 44 prioritizing those updated within the last week. [Use this issue tracker |
99 query](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3DIn
ternals%3ENetwork+-status%3AAssigned+-status%3AStarted+-status%3AAvailable&sort=
-modified&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+
Modified&x=m&y=releaseblock&cells=ids). | 45 query](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3DIn
ternals%3ENetwork+status%3AUnconfirmed,Untriaged+-label:Needs-Feedback&sort=-mod
ified). |
100 | 46 |
101 * If more information is needed from the reporter, ask for it and add the | 47 * If more information is needed from the reporter, ask for it and add the |
102 Needs-Feedback label. If the reporter has answered an earlier request for | 48 Needs-Feedback label. |
103 information, remove that label. | |
104 | 49 |
105 * While investigating a new issue, change the status to Untriaged. | 50 * While investigating a new issue, change the status to Untriaged. |
106 | 51 |
107 * If a bug is a potential security issue (Allows for code execution from remote | 52 * If a bug is a potential security issue (Allows for code execution from remote |
108 site, allows crossing security boundaries, unchecked array bounds, etc) mark | 53 site, allows crossing security boundaries, unchecked array bounds, etc) mark |
109 it Type-Bug-Security. If it has privacy implication (History, cookies | 54 it Type-Bug-Security. If it has privacy implication (History, cookies |
110 discoverable by an entity that shouldn't be able to do so, incognito state | 55 discoverable by an entity that shouldn't be able to do so, incognito state |
111 being saved in memory or on disk beyond the lifetime of incognito tabs, etc), | 56 being saved in memory or on disk beyond the lifetime of incognito tabs, etc), |
112 mark it with component Privacy. | 57 mark it with component Privacy. |
113 | 58 |
114 * For bugs that already have a more specific network component, go ahead and | 59 * For bugs that already have a more specific network component, go ahead and |
115 remove the Internals>Network component and move on. | 60 remove the Internals>Network component to get them off the next triager's |
| 61 radar and move on. |
116 | 62 |
117 * Try to figure out if it's really a network bug. See common non-network | 63 * Try to figure out if it's really a network bug. See common non-network |
118 components section for description of common components for issues incorrectly | 64 components section for description of common components for issues incorrectly |
119 tagged as Internals>Network. | 65 tagged as Internals>Network. |
120 | 66 |
121 * If it's not, attach appropriate labels/components and go no further. | 67 * If it's not, attach appropriate labels/components and go no further. |
122 | 68 |
123 * If it may be a network bug, attach additional possibly relevant component if | 69 * If it may be a network bug, attach additional possibly relevant component if |
124 any, and continue investigating. Once you either determine it's a | 70 any, and continue investigating. Once you either determine it's a |
125 non-network bug, or figure out accurate more specific network components, your | 71 non-network bug, or figure out accurate more specific network components, your |
(...skipping 28 matching lines...) Expand all Loading... |
154 * If you are having trouble with an issue, particularly for help understanding | 100 * If you are having trouble with an issue, particularly for help understanding |
155 net-internals logs, email the public net-dev@chromium.org list for help | 101 net-internals logs, email the public net-dev@chromium.org list for help |
156 debugging. If it's a crasher, or for some other reason discussion needs to | 102 debugging. If it's a crasher, or for some other reason discussion needs to |
157 be done in private, use chrome-network-debugging@google.com. TODO(mmenke): | 103 be done in private, use chrome-network-debugging@google.com. TODO(mmenke): |
158 Write up a net-internals tips and tricks docs. | 104 Write up a net-internals tips and tricks docs. |
159 | 105 |
160 * If it appears to be a bug in the unowned core of the network stack (i.e. no | 106 * If it appears to be a bug in the unowned core of the network stack (i.e. no |
161 subcomponent applies, or only the Internals>Network>HTTP subcomponent | 107 subcomponent applies, or only the Internals>Network>HTTP subcomponent |
162 applies, and there's no clear owner), try to figure out the exact cause. | 108 applies, and there's no clear owner), try to figure out the exact cause. |
163 | 109 |
164 ## Monitoring UMA histograms and Chirp/Gasper alerts | 110 ## Investigate UMA notifications |
165 | |
166 Sign up to chrome-network-debugging@google.com mailing list to receive automated | |
167 e-mails about UMA alerts. Chirp is the new alert system, sending automated | |
168 e-mails with sender address finch-chirp@google.com. Gasper is the old alert | |
169 system, sending automated e-mails with sender address gasper-alerts@google.com. | |
170 While Chirp is of higher priority, Gasper is not deprecated yet, so both alerts | |
171 should be monitored for the time being. | |
172 | 111 |
173 For each alert that fires, determine if it's a real alert and file a bug if so. | 112 For each alert that fires, determine if it's a real alert and file a bug if so. |
174 | 113 |
175 * Don't file if the alert is coincident with a major volume change. The volume | 114 * Don't file if the alert is coincident with a major volume change. The volume |
176 at a particular date can be determined by hovering the mouse over the | 115 at a particular date can be determined by hovering the mouse over the |
177 appropriate location on the alert line. | 116 appropriate location on the alert line. |
178 | 117 |
179 * Don't file if the alert is on a graph with very low volume (< ~200 data | 118 * Don't file if the alert is on a graph with very low volume (< ~200 data |
180 points); it's probably noise, and we probably don't care even if it isn't. | 119 points); it's probably noise, and we probably don't care even if it isn't. |
181 | 120 |
182 * Don't file if the graph is really noisy (but eyeball it to decide if there is | 121 * Don't file if the graph is really noisy (but eyeball it to decide if there is |
183 an underlying important shift under the noise). | 122 an underlying important shift under the noise). |
184 | 123 |
185 * Don't file if the alert is in the "Known Ignorable" list: | 124 * Don't file if the alert is in the "Known Ignorable" list: |
186 * SimpleCache on Windows | 125 * SimpleCache on Windows |
187 * DiskCache on Android. | 126 * DiskCache on Android. |
188 | 127 |
189 For each alert, respond to chrome-network-debugging@google.com with a summary of | 128 ## Looking for new crashers |
190 the action you've taken and why, including issue link if an issue was filed. | 129 |
| 130 1. Go to [go/chromecrash](https://goto.google.com/chromecrash). |
| 131 |
| 132 2. For each platform, look through the releases for which releases to |
| 133 investigate. As per bug-triage.txt, this should be the most recent canary, |
| 134 the previous canary (if the most recent is less than a day old), and any of |
| 135 dev/beta/stable that were released in the last couple of days. |
| 136 |
| 137 3. For each release, in the "Process Type" frame, click on "browser". |
| 138 |
| 139 4. At the bottom of the "Magic Signature" frame, click "limit 1000" (Or reduce |
| 140 the limit to 100 first, as that's all the triager needs to look at). |
| 141 Reported crashers are sorted in decreasing order of the number of reports for |
| 142 that crash signature. |
| 143 |
| 144 5. Search the page for *"net::"*. |
| 145 |
| 146 6. For each found signature: |
| 147 * Ignore signatures that only occur once or twice, as memory corruption can |
| 148 easily cause one-off failures when the sample size is large enough. Also |
| 149 ignore crashers that are not in the top 100 for that platform / release. |
| 150 * If there is a bug already filed, make sure it is correctly describing the |
| 151 current bug (e.g. not closed, or not describing a long-past issue), and |
| 152 make sure that if it is a *net* bug, that it is labeled as such. |
| 153 * Ignore signatures that only come from one or two client IDs, as individual |
| 154 machine malware and breakage can cause one-off failures. |
| 155 * Click on the number of reports field to see details of crash. Ignore it |
| 156 if it doesn't appear to be a network bug. |
| 157 * Otherwise, file a new bug directly from chromecrash. |
| 158 * For each bug you file, include the following information: |
| 159 * The backtrace. Note that the backtrace should not be added to the |
| 160 bug if Restrict-View-Google isn't set on the bug as it may contain |
| 161 PII. Filing the bug from the crash reporter should do this |
| 162 automatically, but check. |
| 163 * The channel in which the bug is seen (canary/dev/beta/stable), and its |
| 164 rank among crashers in the channel. |
| 165 * The frequency of this signature in recent releases. This information |
| 166 is available by: |
| 167 1. Clicking on the signature in the "Magic Signature" list |
| 168 2. Clicking "Edit" on the dremel query at the top of the page |
| 169 3. Removing the "product.version='X.Y.Z.W' AND" string and clicking |
| 170 "Update". |
| 171 4. Clicking "Limit 1000" in the Product Version list in the |
| 172 resulting page (without this, the listing will be restricted to |
| 173 the releases in which the signature is most common, which will |
| 174 often not include the canary/dev release being investigated). |
| 175 5. Choose some subset of that list, or all of it, to include in the |
| 176 bug. Make sure to indicate if there is a defined point in the |
| 177 past before which the signature is not present. |
191 | 178 |
192 ## Investigating crashers | 179 ## Investigating crashers |
193 | 180 |
194 * Only investigate crashers that are still occurring, as identified by above | 181 * Only investigate crashers that are still occurring, as identified by above |
195 section. If a search on go/crash indicates a crasher is no longer occurring, | 182 section. If a search on go/crash indicates a crasher is no longer occurring, |
196 mark it as WontFix. | 183 mark it as WontFix. |
197 | 184 |
198 * On Windows, you may want to look for weird dlls associated with the crashes. | 185 * On Windows, you may want to look for weird dlls associated with the crashes. |
199 This generally needs crashes from a fair number of different users to reach | 186 This generally needs crashes from a fair number of different users to reach |
200 any conclusions. | 187 any conclusions. |
(...skipping 13 matching lines...) Expand all Loading... |
214 a crash report, and seeing if there are multiple reports for the same crash. | 201 a crash report, and seeing if there are multiple reports for the same crash. |
215 If this is the case, it may be also be malware, or an issue with an unusual | 202 If this is the case, it may be also be malware, or an issue with an unusual |
216 system/chrome/network config. | 203 system/chrome/network config. |
217 | 204 |
218 * Dig through crash reports to figure out when the crash first appeared, and | 205 * Dig through crash reports to figure out when the crash first appeared, and |
219 dig through revision history in related files to try and locate a suspect CL. | 206 dig through revision history in related files to try and locate a suspect CL. |
220 TODO(mmenke): Add more detail here. | 207 TODO(mmenke): Add more detail here. |
221 | 208 |
222 * Load crash dumps, try to figure out a cause. See | 209 * Load crash dumps, try to figure out a cause. See |
223 http://www.chromium.org/developers/crash-reports for more information | 210 http://www.chromium.org/developers/crash-reports for more information |
224 | |
225 ## Dealing with old bugs | |
226 | |
227 * For all network issues (Even those with owners, or a more specific component): | |
228 | |
229 * If the issue has had the Needs-Feedback label for over a month, verify it | |
230 is waiting on feedback from the user. If not, remove the label. | |
231 Otherwise, go ahead and mark the issue WontFix due to lack of response | |
232 and suggest the user file a new bug if the issue is still present. [Use | |
233 this issue tracker query for old Needs-Feedback | |
234 issues](https://code.google.com/p/chromium/issues/list?can=2&q=component%3
AInternals>Network%20Needs=Feedback+modified-before%3Atoday-30&sort=-modified). | |
235 | |
236 * If a bug is over 2 months old, and the underlying problem was never | |
237 reproduced or really understood: | |
238 * If it's over a year old, go ahead and mark the issue as Archived. | |
239 * Otherwise, ask reporters if the issue is still present, and attach | |
240 the Needs-Feedback label. | |
241 | |
242 * Old unconfirmed or untriaged Internals>Network issues can be investigated | |
243 just like newer ones. Crashers should generally be given higher priority, | |
244 since we can verify if they still occur, and then newer issues, as they're | |
245 more likely to still be present, and more likely to have a still responsive | |
246 bug reporter. | |
OLD | NEW |