OLD | NEW |
---|---|
1 Copyright (c) 2010 The Chromium OS Authors. All rights reserved. | 1 Copyright (c) 2010 The Chromium OS Authors. All rights reserved. |
2 Use of this source code is governed by a BSD-style license that can be | 2 Use of this source code is governed by a BSD-style license that can be |
3 found in the LICENSE file. | 3 found in the LICENSE file. |
4 | 4 |
5 | 5 |
6 This document describes the steps to go through in order to run Chrome OS | 6 This document describes the steps to go through in order to run Chrome OS |
7 hardware qualification on a device under test. | 7 hardware qualification on a device under test. |
8 | 8 |
9 ================================================================================ | 9 ================================================================================ |
10 Glossary | 10 Glossary |
(...skipping 63 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
74 a consistent starting point for the qualification tests and allows | 74 a consistent starting point for the qualification tests and allows |
75 the system to collect cold boot performance metrics. Make sure you | 75 the system to collect cold boot performance metrics. Make sure you |
76 don't boot from USB. | 76 don't boot from USB. |
77 | 77 |
78 | 78 |
79 - Connect the DUT to the network and note its IP address <DUT_IP>. The | 79 - Connect the DUT to the network and note its IP address <DUT_IP>. The |
80 IP address is displayed at the bottom of the network selection | 80 IP address is displayed at the bottom of the network selection |
81 menu. Unless specified explicitly, the test setup works correctly | 81 menu. Unless specified explicitly, the test setup works correctly |
82 through either wireless or wired network connections. | 82 through either wireless or wired network connections. |
83 | 83 |
84 - Unless otherwise noted, all tests can be performed on an AC-powered DUT. | |
85 | |
86 - Go to the Autotest server directory and clean up previous test results. | |
87 | |
88 $ cd $HOME/chromeos-hwqual-TAG/autotest/ | |
89 $ rm -rf results.* | |
90 | |
91 ============================================================================== | |
92 Automated and Semi-Automated Test Runs using utility script call_autoserv.py | |
93 ============================================================================== | |
94 | |
95 - Go back to parent directoy above autotest | |
96 | |
97 $ cd $HOME/chromeos-hwqual-TAG/ | |
98 | |
99 - HWQual tests under .../autotest/client and .../autotest/server directories | |
100 can be executed as following: | |
101 | |
102 $ python call_autoserv.py --ip=<DUT_IP> --test=<HWQual test name> | |
103 | |
104 <HWQual test name> - should be the exact name of the test without 'control.' | |
105 prefixed to the test name. | |
106 | |
107 For example : To call control.auto test use the command as - | |
108 $ python call_autoserv.py --ip=<DUT_IP> --test=auto | |
kmixter1
2011/01/21 20:12:12
Why explicitly call python?
| |
109 | |
110 - The script will do the following - | |
111 * Add testing_rsa. | |
112 * Set up ssh connection to the DUT. | |
113 * Search for the test name under autotest/client and autotest/server | |
114 directories. | |
115 * Trigger the test exection. | |
116 * Save results in unique timestamped directoy and display its location. | |
117 * Run the ./generate_test_report script to display PASS / FAIL result output. | |
118 * In case of any FAIL / ERROR condition, will grab the result logs, tar them | |
119 up and display its location. | |
120 | |
121 ============================================================================== | |
122 Alternate way to run the HWQual tests | |
kmixter1
2011/01/21 20:12:12
I suggest removing this and all the autoserv direc
| |
123 ============================================================================== | |
84 | 124 |
85 - Add the DUT root private key to ssh-agent on the Autotest server: | 125 - Add the DUT root private key to ssh-agent on the Autotest server: |
86 | 126 |
87 $ ssh-add $HOME/chromeos-hwqual-TAG/testing_rsa | 127 $ ssh-add $HOME/chromeos-hwqual-TAG/testing_rsa |
88 | 128 |
89 - If ssh-add fails saying that it cannot connect to your authentication agent, | 129 - If ssh-add fails saying that it cannot connect to your authentication agent, |
90 retry the command after running: | 130 retry the command after running: |
91 | 131 |
92 $ eval `ssh-agent -s` | 132 $ eval `ssh-agent -s` |
93 | 133 |
94 - These commands allow the Autotest server to connect and login as root on the | 134 - These commands allow the Autotest server to connect and login as root on the |
95 DUT. | 135 DUT. |
96 | 136 |
97 | 137 |
98 - Make sure you can ssh as root to the DUT from the Autotest | 138 - Make sure you can ssh as root to the DUT from the Autotest |
99 server. The command below should print 0. | 139 server. The command below should print 0. |
100 | 140 |
101 $ ssh root@<DUT_IP> true; echo $? | 141 $ ssh root@<DUT_IP> true; echo $? |
102 | 142 |
103 ================================================================================ | 143 ================================================================================ |
104 Automated and Semi-Automated Test Runs | 144 Automated and Semi-Automated Test Runs with precondition details |
105 ================================================================================ | 145 ================================================================================ |
106 | 146 |
107 - Unless otherwise noted, all tests can be performed on an AC-powered DUT. | |
108 | |
109 - Go to the Autotest server directory and clean up previous test results. | |
110 | |
111 $ cd $HOME/chromeos-hwqual-TAG/autotest/ | |
112 $ rm -rf results.* | |
113 | |
114 | |
115 - Run the fully automated client-side tests: | 147 - Run the fully automated client-side tests: |
116 | 148 |
117 $ ./server/autoserv -r results.auto -m <DUT_IP> \ | 149 $ ./server/autoserv -r results.auto -m <DUT_IP> \ |
118 -c client/site_tests/suite_HWQual/control.auto | 150 -c client/site_tests/suite_HWQual/control.auto |
119 | 151 |
120 | 152 * Alternate method from outside autotest directory: |
153 $ python call_autoserv.py --ip=<DUT_IP> --test=auto | |
154 | |
121 - Plug high-speed high-capacity storage devices in all USB and SD Card | 155 - Plug high-speed high-capacity storage devices in all USB and SD Card |
122 slots and run the external storage test: | 156 slots and run the external storage test: |
123 | 157 |
124 $ ./server/autoserv -r results.external_devices -m <DUT_IP> \ | 158 $ ./server/autoserv -r results.external_devices -m <DUT_IP> \ |
125 -c client/site_tests/suite_HWQual/control.external_drives | 159 -c client/site_tests/suite_HWQual/control.external_drives |
126 | 160 |
127 | 161 |
162 * Alternate method from outside autotest directory: | |
163 $ python call_autoserv.py --ip=<DUT_IP> --test=external_drives | |
164 | |
128 - Run the approved components test by first following the manual | 165 - Run the approved components test by first following the manual |
129 instructions specified in the control file (control.components) and | 166 instructions specified in the control file (control.components) and |
130 then executing: | 167 then executing: |
131 | 168 |
132 $ ./server/autoserv -r results.components -m <DUT_IP> \ | 169 $ ./server/autoserv -r results.components -m <DUT_IP> \ |
133 -c client/site_tests/suite_HWQual/control.components | 170 -c client/site_tests/suite_HWQual/control.components |
134 | 171 |
172 * Alternate method from outside autotest directory: | |
173 $ python call_autoserv.py --ip=<DUT_IP> --test=components | |
174 | |
135 | 175 |
136 - Run the system suspend/resume stability test: | 176 - Run the system suspend/resume stability test: |
137 | 177 |
138 $ ./server/autoserv -r results.suspend_resume -m <DUT_IP> \ | 178 $ ./server/autoserv -r results.suspend_resume -m <DUT_IP> \ |
139 -c client/site_tests/suite_HWQual/control.suspend_resume | 179 -c client/site_tests/suite_HWQual/control.suspend_resume |
140 | 180 |
181 * Alternate method from outside autotest directory: | |
182 $ python call_autoserv.py --ip=<DUT_IP> --test=suspend_resume | |
183 | |
141 | 184 |
142 - If the DUT has video out ports, run the Video Out semi-automated | 185 - If the DUT has video out ports, run the Video Out semi-automated |
143 test by following the instructions specified in the control file | 186 test by following the instructions specified in the control file |
144 (control.video_out) and then executing: | 187 (control.video_out) and then executing: |
145 | 188 |
146 $ ./server/autoserv -r results.video_out.${PORT} -m <DUT_IP> \ | 189 $ ./server/autoserv -r results.video_out.${PORT} -m <DUT_IP> \ |
147 -c client/site_tests/suite_HWQual/control.video_out | 190 -c client/site_tests/suite_HWQual/control.video_out |
148 | 191 |
192 * Alternate method from outside autotest directory: | |
193 $ python call_autoserv.py --ip=<DUT_IP> --test=video_out | |
194 | |
149 - Where PORT is the name of each video port you are testing. For | 195 - Where PORT is the name of each video port you are testing. For |
150 example, if the DUT has one HDMI and one VGA out port, run: | 196 example, if the DUT has one HDMI and one VGA out port, run: |
151 | 197 |
152 $ ./server/autoserv -r results.video_out.hdmi1 -m <DUT_IP> \ | 198 $ ./server/autoserv -r results.video_out.hdmi1 -m <DUT_IP> \ |
153 -c client/site_tests/suite_HWQual/control.video_out | 199 -c client/site_tests/suite_HWQual/control.video_out |
154 | 200 |
201 * Alternate method from outside autotest directory: | |
202 $ python call_autoserv.py --ip=<DUT_IP> --test=video_out | |
203 | |
155 $ ./server/autoserv -r results.video_out.vga1 -m <DUT_IP> \ | 204 $ ./server/autoserv -r results.video_out.vga1 -m <DUT_IP> \ |
156 -c client/site_tests/suite_HWQual/control.video_out | 205 -c client/site_tests/suite_HWQual/control.video_out |
157 | 206 |
207 * Alternate method from outside autotest directory: | |
208 $ python call_autoserv.py --ip=<DUT_IP> --test=video_out | |
209 | |
158 | 210 |
159 - Run the graphics tearing test: | 211 - Run the graphics tearing test: |
160 | 212 |
161 $ ./server/autoserv -r results.teartest -m <DUT_IP> \ | 213 $ ./server/autoserv -r results.teartest -m <DUT_IP> \ |
162 -c client/site_tests/suite_HWQual/control.teartest | 214 -c client/site_tests/suite_HWQual/control.teartest |
163 | 215 |
216 * Alternate method from outside autotest directory: | |
217 $ python call_autoserv.py --ip=<DUT_IP> --test=teartest | |
218 | |
164 - Run audio test, with built-in speakers and microphone | 219 - Run audio test, with built-in speakers and microphone |
165 | 220 |
166 $ ./server/autoserv -r results.audio -m <DUT_IP> \ | 221 $ ./server/autoserv -r results.audio -m <DUT_IP> \ |
167 -c client/site_tests/suite_HWQual/control.audio | 222 -c client/site_tests/suite_HWQual/control.audio |
168 | 223 |
224 * Alternate method from outside autotest directory: | |
225 $ python call_autoserv.py --ip=<DUT_IP> --test=audio | |
226 | |
169 - Plug-in headphone and microphone, run audio test again | 227 - Plug-in headphone and microphone, run audio test again |
170 | 228 |
171 $ ./server/autoserv -r results.audio_ext -m <DUT_IP> \ | 229 $ ./server/autoserv -r results.audio_ext -m <DUT_IP> \ |
172 -c client/site_tests/suite_HWQual/control.audio | 230 -c client/site_tests/suite_HWQual/control.audio |
173 | 231 |
232 * Alternate method from outside autotest directory: | |
233 $ python call_autoserv.py --ip=<DUT_IP> --test=audio | |
234 | |
174 - Run the Keyboard test : | 235 - Run the Keyboard test : |
175 (Wait several seconds after running the test. Then strike the "Search" key, | 236 (Wait several seconds after running the test. Then strike the "Search" key, |
176 e.g. the key above Left Shift and below Tab) | 237 e.g. the key above Left Shift and below Tab) |
177 | 238 |
178 $ ./server/autoserv -r result.keyboard -m <DUT_IP> \ | 239 $ ./server/autoserv -r result.keyboard -m <DUT_IP> \ |
179 -c client/site_tests/suite_HWQual/control.keyboard | 240 -c client/site_tests/suite_HWQual/control.keyboard |
180 | 241 |
242 * Alternate method from outside autotest directory: | |
243 $ python call_autoserv.py --ip=<DUT_IP> --test=keyboard | |
244 | |
181 - Run the DUT on AC. Probe the AC driver: | 245 - Run the DUT on AC. Probe the AC driver: |
182 | 246 |
183 $ ./server/autoserv -r result.probe_ac -m <DUT_IP> \ | 247 $ ./server/autoserv -r result.probe_ac -m <DUT_IP> \ |
184 -c client/site_tests/suite_HWQual/control.probe_ac | 248 -c client/site_tests/suite_HWQual/control.probe_ac |
185 | 249 |
250 * Alternate method from outside autotest directory: | |
251 $ python call_autoserv.py --ip=<DUT_IP> --test=probe_ac | |
252 | |
186 - Run the DUT on battery. Probe the battery driver: | 253 - Run the DUT on battery. Probe the battery driver: |
187 (If you just unplugged AC, please wait for a second before running | 254 (If you just unplugged AC, please wait for a second before running |
188 the test for kernel updating power status.) | 255 the test for kernel updating power status.) |
189 | 256 |
190 $ ./server/autoserv -r result.probe_bat -m <DUT_IP> \ | 257 $ ./server/autoserv -r result.probe_bat -m <DUT_IP> \ |
191 -c client/site_tests/suite_HWQual/control.probe_bat | 258 -c client/site_tests/suite_HWQual/control.probe_bat |
192 | 259 |
260 * Alternate method from outside autotest directory: | |
261 $ python call_autoserv.py --ip=<DUT_IP> --test=probe_bat | |
262 | |
193 - Run the DUT on AC. Plug a power draw USB dongle in each USB port. | 263 - Run the DUT on AC. Plug a power draw USB dongle in each USB port. |
194 Run the max power draw test: | 264 Run the max power draw test: |
195 | 265 |
196 $ ./server/autoserv -r results.max_power_draw.ac -m <DUT_IP> \ | 266 $ ./server/autoserv -r results.max_power_draw.ac -m <DUT_IP> \ |
197 -c client/site_tests/suite_HWQual/control.max_power_draw | 267 -c client/site_tests/suite_HWQual/control.max_power_draw |
198 | 268 |
269 * Alternate method from outside autotest directory: | |
270 $ python call_autoserv.py --ip=<DUT_IP> --test=max_power_draw | |
271 | |
199 - Run the DUT on battery. Plug a power draw USB dongle in each USB | 272 - Run the DUT on battery. Plug a power draw USB dongle in each USB |
200 port. Run the max power draw test: | 273 port. Run the max power draw test: |
201 | 274 |
202 $ ./server/autoserv -r results.max_power_draw.batt -m <DUT_IP> \ | 275 $ ./server/autoserv -r results.max_power_draw.batt -m <DUT_IP> \ |
203 -c client/site_tests/suite_HWQual/control.max_power_draw | 276 -c client/site_tests/suite_HWQual/control.max_power_draw |
204 | 277 |
278 * Alternate method from outside autotest directory: | |
279 $ python call_autoserv.py --ip=<DUT_IP> --test=max_power_draw | |
280 | |
205 - Run the DUT on AC. Run the power settings test: | 281 - Run the DUT on AC. Run the power settings test: |
206 | 282 |
207 $ ./server/autoserv -r results.power_x86_setting.ac -m <DUT_IP> \ | 283 $ ./server/autoserv -r results.power_x86_setting.ac -m <DUT_IP> \ |
208 -c client/site_tests/suite_HWQual/control.power_x86_settings | 284 -c client/site_tests/suite_HWQual/control.power_x86_settings |
209 | 285 |
286 * Alternate method from outside autotest directory: | |
287 $ python call_autoserv.py --ip=<DUT_IP> --test=power_x86_settings | |
288 | |
210 - Run the DUT on battery. Run the power settings test: | 289 - Run the DUT on battery. Run the power settings test: |
211 | 290 |
212 $ ./server/autoserv -r results.power_x86_setting.batt -m <DUT_IP> \ | 291 $ ./server/autoserv -r results.power_x86_setting.batt -m <DUT_IP> \ |
213 -c client/site_tests/suite_HWQual/control.power_x86_settings | 292 -c client/site_tests/suite_HWQual/control.power_x86_settings |
214 | 293 |
294 * Alternate method from outside autotest directory: | |
295 $ python call_autoserv.py --ip=<DUT_IP> --test=power_x86_settings | |
296 | |
215 - Make sure the remaining battery charge is less than 5%. Note that the test | 297 - Make sure the remaining battery charge is less than 5%. Note that the test |
216 will check and fail quickly if the initial battery charge is more than 5%. | 298 will check and fail quickly if the initial battery charge is more than 5%. |
217 Run the DUT on AC. Run the battery charge test: | 299 Run the DUT on AC. Run the battery charge test: |
218 | 300 |
219 $ ./server/autoserv -r results.battery_charge_time -m <DUT_IP> \ | 301 $ ./server/autoserv -r results.battery_charge_time -m <DUT_IP> \ |
220 -c client/site_tests/suite_HWQual/control.battery_charge_time | 302 -c client/site_tests/suite_HWQual/control.battery_charge_time |
221 | 303 |
304 * Alternate method from outside autotest directory: | |
305 $ python call_autoserv.py --ip=<DUT_IP> --test=battery_charge_time | |
306 | |
222 - Make sure that the battery is fully charged. Note that the test will not | 307 - Make sure that the battery is fully charged. Note that the test will not |
223 check if the battery is fully charged before running. Run the DUT on | 308 check if the battery is fully charged before running. Run the DUT on |
224 battery. Run the battery load test by first following the manual | 309 battery. Run the battery load test by first following the manual |
225 instructions specified in the control file (control.battery_load) | 310 instructions specified in the control file (control.battery_load) |
226 and then executing: | 311 and then executing: |
227 | 312 |
228 $ ./server/autoserv -r results.battery_load -m <DUT_IP> \ | 313 $ ./server/autoserv -r results.battery_load -m <DUT_IP> \ |
229 -c client/site_tests/suite_HWQual/control.battery_load | 314 -c client/site_tests/suite_HWQual/control.battery_load |
230 | 315 |
316 * Alternate method from outside autotest directory: | |
317 $ python call_autoserv.py --ip=<DUT_IP> --test=battery_load | |
318 | |
231 ================================================================================ | 319 ================================================================================ |
232 Reviewing Automated and Semi-Automated Test Results | 320 Reviewing Automated and Semi-Automated Test Results |
233 ================================================================================ | 321 ================================================================================ |
234 | 322 |
235 - Autotest logs progress and performance data in results.* directories | 323 - Autotest logs progress and performance data in results.* directories |
236 specified through the '-r' autoserv option. The easy way to see a summary | 324 specified through the '-r' autoserv option. The easy way to see a summary |
237 of your test results is to use the 'generate_test_report' | 325 of your test results is to use the 'generate_test_report' |
238 script installed under $HOME/chromeos-hwqual-TAG/: | 326 script installed under $HOME/chromeos-hwqual-TAG/: |
239 | 327 |
240 $ ../generate_test_report results.* | 328 $ ../generate_test_report results.* |
(...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
274 | 362 |
275 $ cd $HOME/chromeos-hwqual-TAG/autotest/ | 363 $ cd $HOME/chromeos-hwqual-TAG/autotest/ |
276 $ cp ../manual/testcases.csv . | 364 $ cp ../manual/testcases.csv . |
277 | 365 |
278 - Package all results into a tar file: | 366 - Package all results into a tar file: |
279 | 367 |
280 $ tar cjf chromeos-hwqual-results-TAG-DATE.tar.bz2 results.* testcases.csv | 368 $ tar cjf chromeos-hwqual-results-TAG-DATE.tar.bz2 results.* testcases.csv |
281 | 369 |
282 - Send the resulting chromeos-hwqual-results file to your technical support | 370 - Send the resulting chromeos-hwqual-results file to your technical support |
283 contact. | 371 contact. |
OLD | NEW |