Chromium Code Reviews| OLD | NEW |
|---|---|
| 1 Copyright (c) 2010 The Chromium OS Authors. All rights reserved. | 1 Copyright (c) 2010 The Chromium OS Authors. All rights reserved. |
| 2 Use of this source code is governed by a BSD-style license that can be | 2 Use of this source code is governed by a BSD-style license that can be |
| 3 found in the LICENSE file. | 3 found in the LICENSE file. |
| 4 | 4 |
| 5 | 5 |
| 6 This document describes the steps to go through in order to run Chrome OS | 6 This document describes the steps to go through in order to run Chrome OS |
| 7 hardware qualification on a device under test. | 7 hardware qualification on a device under test. |
| 8 | 8 |
| 9 ================================================================================ | 9 ================================================================================ |
| 10 Glossary | 10 Glossary |
| (...skipping 63 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 74 a consistent starting point for the qualification tests and allows | 74 a consistent starting point for the qualification tests and allows |
| 75 the system to collect cold boot performance metrics. Make sure you | 75 the system to collect cold boot performance metrics. Make sure you |
| 76 don't boot from USB. | 76 don't boot from USB. |
| 77 | 77 |
| 78 | 78 |
| 79 - Connect the DUT to the network and note its IP address <DUT_IP>. The | 79 - Connect the DUT to the network and note its IP address <DUT_IP>. The |
| 80 IP address is displayed at the bottom of the network selection | 80 IP address is displayed at the bottom of the network selection |
| 81 menu. Unless specified explicitly, the test setup works correctly | 81 menu. Unless specified explicitly, the test setup works correctly |
| 82 through either wireless or wired network connections. | 82 through either wireless or wired network connections. |
| 83 | 83 |
| 84 - Unless otherwise noted, all tests can be performed on an AC-powered DUT. | |
| 85 | |
| 86 - Go to the Autotest server directory and clean up previous test results. | |
| 87 | |
| 88 $ cd $HOME/chromeos-hwqual-TAG/autotest/ | |
| 89 $ rm -rf results.* | |
| 90 | |
| 91 ============================================================================== | |
| 92 Automated and Semi-Automated Test Runs using utility script call_autoserv.py | |
| 93 ============================================================================== | |
| 94 | |
| 95 - Go back to parent directoy above autotest | |
| 96 | |
| 97 $ cd $HOME/chromeos-hwqual-TAG/ | |
| 98 | |
| 99 - HWQual tests under .../autotest/client and .../autotest/server directories | |
| 100 can be executed as following: | |
| 101 | |
| 102 $ python call_autoserv.py --ip=<DUT_IP> --test=<HWQual test name> | |
| 103 | |
| 104 <HWQual test name> - should be the exact name of the test without 'control.' | |
| 105 prefixed to the test name. | |
| 106 | |
| 107 For example : To call control.auto test use the command as - | |
| 108 $ python call_autoserv.py --ip=<DUT_IP> --test=auto | |
|
kmixter1
2011/01/21 20:12:12
Why explicitly call python?
| |
| 109 | |
| 110 - The script will do the following - | |
| 111 * Add testing_rsa. | |
| 112 * Set up ssh connection to the DUT. | |
| 113 * Search for the test name under autotest/client and autotest/server | |
| 114 directories. | |
| 115 * Trigger the test exection. | |
| 116 * Save results in unique timestamped directoy and display its location. | |
| 117 * Run the ./generate_test_report script to display PASS / FAIL result output. | |
| 118 * In case of any FAIL / ERROR condition, will grab the result logs, tar them | |
| 119 up and display its location. | |
| 120 | |
| 121 ============================================================================== | |
| 122 Alternate way to run the HWQual tests | |
|
kmixter1
2011/01/21 20:12:12
I suggest removing this and all the autoserv direc
| |
| 123 ============================================================================== | |
| 84 | 124 |
| 85 - Add the DUT root private key to ssh-agent on the Autotest server: | 125 - Add the DUT root private key to ssh-agent on the Autotest server: |
| 86 | 126 |
| 87 $ ssh-add $HOME/chromeos-hwqual-TAG/testing_rsa | 127 $ ssh-add $HOME/chromeos-hwqual-TAG/testing_rsa |
| 88 | 128 |
| 89 - If ssh-add fails saying that it cannot connect to your authentication agent, | 129 - If ssh-add fails saying that it cannot connect to your authentication agent, |
| 90 retry the command after running: | 130 retry the command after running: |
| 91 | 131 |
| 92 $ eval `ssh-agent -s` | 132 $ eval `ssh-agent -s` |
| 93 | 133 |
| 94 - These commands allow the Autotest server to connect and login as root on the | 134 - These commands allow the Autotest server to connect and login as root on the |
| 95 DUT. | 135 DUT. |
| 96 | 136 |
| 97 | 137 |
| 98 - Make sure you can ssh as root to the DUT from the Autotest | 138 - Make sure you can ssh as root to the DUT from the Autotest |
| 99 server. The command below should print 0. | 139 server. The command below should print 0. |
| 100 | 140 |
| 101 $ ssh root@<DUT_IP> true; echo $? | 141 $ ssh root@<DUT_IP> true; echo $? |
| 102 | 142 |
| 103 ================================================================================ | 143 ================================================================================ |
| 104 Automated and Semi-Automated Test Runs | 144 Automated and Semi-Automated Test Runs with precondition details |
| 105 ================================================================================ | 145 ================================================================================ |
| 106 | 146 |
| 107 - Unless otherwise noted, all tests can be performed on an AC-powered DUT. | |
| 108 | |
| 109 - Go to the Autotest server directory and clean up previous test results. | |
| 110 | |
| 111 $ cd $HOME/chromeos-hwqual-TAG/autotest/ | |
| 112 $ rm -rf results.* | |
| 113 | |
| 114 | |
| 115 - Run the fully automated client-side tests: | 147 - Run the fully automated client-side tests: |
| 116 | 148 |
| 117 $ ./server/autoserv -r results.auto -m <DUT_IP> \ | 149 $ ./server/autoserv -r results.auto -m <DUT_IP> \ |
| 118 -c client/site_tests/suite_HWQual/control.auto | 150 -c client/site_tests/suite_HWQual/control.auto |
| 119 | 151 |
| 120 | 152 * Alternate method from outside autotest directory: |
| 153 $ python call_autoserv.py --ip=<DUT_IP> --test=auto | |
| 154 | |
| 121 - Plug high-speed high-capacity storage devices in all USB and SD Card | 155 - Plug high-speed high-capacity storage devices in all USB and SD Card |
| 122 slots and run the external storage test: | 156 slots and run the external storage test: |
| 123 | 157 |
| 124 $ ./server/autoserv -r results.external_devices -m <DUT_IP> \ | 158 $ ./server/autoserv -r results.external_devices -m <DUT_IP> \ |
| 125 -c client/site_tests/suite_HWQual/control.external_drives | 159 -c client/site_tests/suite_HWQual/control.external_drives |
| 126 | 160 |
| 127 | 161 |
| 162 * Alternate method from outside autotest directory: | |
| 163 $ python call_autoserv.py --ip=<DUT_IP> --test=external_drives | |
| 164 | |
| 128 - Run the approved components test by first following the manual | 165 - Run the approved components test by first following the manual |
| 129 instructions specified in the control file (control.components) and | 166 instructions specified in the control file (control.components) and |
| 130 then executing: | 167 then executing: |
| 131 | 168 |
| 132 $ ./server/autoserv -r results.components -m <DUT_IP> \ | 169 $ ./server/autoserv -r results.components -m <DUT_IP> \ |
| 133 -c client/site_tests/suite_HWQual/control.components | 170 -c client/site_tests/suite_HWQual/control.components |
| 134 | 171 |
| 172 * Alternate method from outside autotest directory: | |
| 173 $ python call_autoserv.py --ip=<DUT_IP> --test=components | |
| 174 | |
| 135 | 175 |
| 136 - Run the system suspend/resume stability test: | 176 - Run the system suspend/resume stability test: |
| 137 | 177 |
| 138 $ ./server/autoserv -r results.suspend_resume -m <DUT_IP> \ | 178 $ ./server/autoserv -r results.suspend_resume -m <DUT_IP> \ |
| 139 -c client/site_tests/suite_HWQual/control.suspend_resume | 179 -c client/site_tests/suite_HWQual/control.suspend_resume |
| 140 | 180 |
| 181 * Alternate method from outside autotest directory: | |
| 182 $ python call_autoserv.py --ip=<DUT_IP> --test=suspend_resume | |
| 183 | |
| 141 | 184 |
| 142 - If the DUT has video out ports, run the Video Out semi-automated | 185 - If the DUT has video out ports, run the Video Out semi-automated |
| 143 test by following the instructions specified in the control file | 186 test by following the instructions specified in the control file |
| 144 (control.video_out) and then executing: | 187 (control.video_out) and then executing: |
| 145 | 188 |
| 146 $ ./server/autoserv -r results.video_out.${PORT} -m <DUT_IP> \ | 189 $ ./server/autoserv -r results.video_out.${PORT} -m <DUT_IP> \ |
| 147 -c client/site_tests/suite_HWQual/control.video_out | 190 -c client/site_tests/suite_HWQual/control.video_out |
| 148 | 191 |
| 192 * Alternate method from outside autotest directory: | |
| 193 $ python call_autoserv.py --ip=<DUT_IP> --test=video_out | |
| 194 | |
| 149 - Where PORT is the name of each video port you are testing. For | 195 - Where PORT is the name of each video port you are testing. For |
| 150 example, if the DUT has one HDMI and one VGA out port, run: | 196 example, if the DUT has one HDMI and one VGA out port, run: |
| 151 | 197 |
| 152 $ ./server/autoserv -r results.video_out.hdmi1 -m <DUT_IP> \ | 198 $ ./server/autoserv -r results.video_out.hdmi1 -m <DUT_IP> \ |
| 153 -c client/site_tests/suite_HWQual/control.video_out | 199 -c client/site_tests/suite_HWQual/control.video_out |
| 154 | 200 |
| 201 * Alternate method from outside autotest directory: | |
| 202 $ python call_autoserv.py --ip=<DUT_IP> --test=video_out | |
| 203 | |
| 155 $ ./server/autoserv -r results.video_out.vga1 -m <DUT_IP> \ | 204 $ ./server/autoserv -r results.video_out.vga1 -m <DUT_IP> \ |
| 156 -c client/site_tests/suite_HWQual/control.video_out | 205 -c client/site_tests/suite_HWQual/control.video_out |
| 157 | 206 |
| 207 * Alternate method from outside autotest directory: | |
| 208 $ python call_autoserv.py --ip=<DUT_IP> --test=video_out | |
| 209 | |
| 158 | 210 |
| 159 - Run the graphics tearing test: | 211 - Run the graphics tearing test: |
| 160 | 212 |
| 161 $ ./server/autoserv -r results.teartest -m <DUT_IP> \ | 213 $ ./server/autoserv -r results.teartest -m <DUT_IP> \ |
| 162 -c client/site_tests/suite_HWQual/control.teartest | 214 -c client/site_tests/suite_HWQual/control.teartest |
| 163 | 215 |
| 216 * Alternate method from outside autotest directory: | |
| 217 $ python call_autoserv.py --ip=<DUT_IP> --test=teartest | |
| 218 | |
| 164 - Run audio test, with built-in speakers and microphone | 219 - Run audio test, with built-in speakers and microphone |
| 165 | 220 |
| 166 $ ./server/autoserv -r results.audio -m <DUT_IP> \ | 221 $ ./server/autoserv -r results.audio -m <DUT_IP> \ |
| 167 -c client/site_tests/suite_HWQual/control.audio | 222 -c client/site_tests/suite_HWQual/control.audio |
| 168 | 223 |
| 224 * Alternate method from outside autotest directory: | |
| 225 $ python call_autoserv.py --ip=<DUT_IP> --test=audio | |
| 226 | |
| 169 - Plug-in headphone and microphone, run audio test again | 227 - Plug-in headphone and microphone, run audio test again |
| 170 | 228 |
| 171 $ ./server/autoserv -r results.audio_ext -m <DUT_IP> \ | 229 $ ./server/autoserv -r results.audio_ext -m <DUT_IP> \ |
| 172 -c client/site_tests/suite_HWQual/control.audio | 230 -c client/site_tests/suite_HWQual/control.audio |
| 173 | 231 |
| 232 * Alternate method from outside autotest directory: | |
| 233 $ python call_autoserv.py --ip=<DUT_IP> --test=audio | |
| 234 | |
| 174 - Run the Keyboard test : | 235 - Run the Keyboard test : |
| 175 (Wait several seconds after running the test. Then strike the "Search" key, | 236 (Wait several seconds after running the test. Then strike the "Search" key, |
| 176 e.g. the key above Left Shift and below Tab) | 237 e.g. the key above Left Shift and below Tab) |
| 177 | 238 |
| 178 $ ./server/autoserv -r result.keyboard -m <DUT_IP> \ | 239 $ ./server/autoserv -r result.keyboard -m <DUT_IP> \ |
| 179 -c client/site_tests/suite_HWQual/control.keyboard | 240 -c client/site_tests/suite_HWQual/control.keyboard |
| 180 | 241 |
| 242 * Alternate method from outside autotest directory: | |
| 243 $ python call_autoserv.py --ip=<DUT_IP> --test=keyboard | |
| 244 | |
| 181 - Run the DUT on AC. Probe the AC driver: | 245 - Run the DUT on AC. Probe the AC driver: |
| 182 | 246 |
| 183 $ ./server/autoserv -r result.probe_ac -m <DUT_IP> \ | 247 $ ./server/autoserv -r result.probe_ac -m <DUT_IP> \ |
| 184 -c client/site_tests/suite_HWQual/control.probe_ac | 248 -c client/site_tests/suite_HWQual/control.probe_ac |
| 185 | 249 |
| 250 * Alternate method from outside autotest directory: | |
| 251 $ python call_autoserv.py --ip=<DUT_IP> --test=probe_ac | |
| 252 | |
| 186 - Run the DUT on battery. Probe the battery driver: | 253 - Run the DUT on battery. Probe the battery driver: |
| 187 (If you just unplugged AC, please wait for a second before running | 254 (If you just unplugged AC, please wait for a second before running |
| 188 the test for kernel updating power status.) | 255 the test for kernel updating power status.) |
| 189 | 256 |
| 190 $ ./server/autoserv -r result.probe_bat -m <DUT_IP> \ | 257 $ ./server/autoserv -r result.probe_bat -m <DUT_IP> \ |
| 191 -c client/site_tests/suite_HWQual/control.probe_bat | 258 -c client/site_tests/suite_HWQual/control.probe_bat |
| 192 | 259 |
| 260 * Alternate method from outside autotest directory: | |
| 261 $ python call_autoserv.py --ip=<DUT_IP> --test=probe_bat | |
| 262 | |
| 193 - Run the DUT on AC. Plug a power draw USB dongle in each USB port. | 263 - Run the DUT on AC. Plug a power draw USB dongle in each USB port. |
| 194 Run the max power draw test: | 264 Run the max power draw test: |
| 195 | 265 |
| 196 $ ./server/autoserv -r results.max_power_draw.ac -m <DUT_IP> \ | 266 $ ./server/autoserv -r results.max_power_draw.ac -m <DUT_IP> \ |
| 197 -c client/site_tests/suite_HWQual/control.max_power_draw | 267 -c client/site_tests/suite_HWQual/control.max_power_draw |
| 198 | 268 |
| 269 * Alternate method from outside autotest directory: | |
| 270 $ python call_autoserv.py --ip=<DUT_IP> --test=max_power_draw | |
| 271 | |
| 199 - Run the DUT on battery. Plug a power draw USB dongle in each USB | 272 - Run the DUT on battery. Plug a power draw USB dongle in each USB |
| 200 port. Run the max power draw test: | 273 port. Run the max power draw test: |
| 201 | 274 |
| 202 $ ./server/autoserv -r results.max_power_draw.batt -m <DUT_IP> \ | 275 $ ./server/autoserv -r results.max_power_draw.batt -m <DUT_IP> \ |
| 203 -c client/site_tests/suite_HWQual/control.max_power_draw | 276 -c client/site_tests/suite_HWQual/control.max_power_draw |
| 204 | 277 |
| 278 * Alternate method from outside autotest directory: | |
| 279 $ python call_autoserv.py --ip=<DUT_IP> --test=max_power_draw | |
| 280 | |
| 205 - Run the DUT on AC. Run the power settings test: | 281 - Run the DUT on AC. Run the power settings test: |
| 206 | 282 |
| 207 $ ./server/autoserv -r results.power_x86_setting.ac -m <DUT_IP> \ | 283 $ ./server/autoserv -r results.power_x86_setting.ac -m <DUT_IP> \ |
| 208 -c client/site_tests/suite_HWQual/control.power_x86_settings | 284 -c client/site_tests/suite_HWQual/control.power_x86_settings |
| 209 | 285 |
| 286 * Alternate method from outside autotest directory: | |
| 287 $ python call_autoserv.py --ip=<DUT_IP> --test=power_x86_settings | |
| 288 | |
| 210 - Run the DUT on battery. Run the power settings test: | 289 - Run the DUT on battery. Run the power settings test: |
| 211 | 290 |
| 212 $ ./server/autoserv -r results.power_x86_setting.batt -m <DUT_IP> \ | 291 $ ./server/autoserv -r results.power_x86_setting.batt -m <DUT_IP> \ |
| 213 -c client/site_tests/suite_HWQual/control.power_x86_settings | 292 -c client/site_tests/suite_HWQual/control.power_x86_settings |
| 214 | 293 |
| 294 * Alternate method from outside autotest directory: | |
| 295 $ python call_autoserv.py --ip=<DUT_IP> --test=power_x86_settings | |
| 296 | |
| 215 - Make sure the remaining battery charge is less than 5%. Note that the test | 297 - Make sure the remaining battery charge is less than 5%. Note that the test |
| 216 will check and fail quickly if the initial battery charge is more than 5%. | 298 will check and fail quickly if the initial battery charge is more than 5%. |
| 217 Run the DUT on AC. Run the battery charge test: | 299 Run the DUT on AC. Run the battery charge test: |
| 218 | 300 |
| 219 $ ./server/autoserv -r results.battery_charge_time -m <DUT_IP> \ | 301 $ ./server/autoserv -r results.battery_charge_time -m <DUT_IP> \ |
| 220 -c client/site_tests/suite_HWQual/control.battery_charge_time | 302 -c client/site_tests/suite_HWQual/control.battery_charge_time |
| 221 | 303 |
| 304 * Alternate method from outside autotest directory: | |
| 305 $ python call_autoserv.py --ip=<DUT_IP> --test=battery_charge_time | |
| 306 | |
| 222 - Make sure that the battery is fully charged. Note that the test will not | 307 - Make sure that the battery is fully charged. Note that the test will not |
| 223 check if the battery is fully charged before running. Run the DUT on | 308 check if the battery is fully charged before running. Run the DUT on |
| 224 battery. Run the battery load test by first following the manual | 309 battery. Run the battery load test by first following the manual |
| 225 instructions specified in the control file (control.battery_load) | 310 instructions specified in the control file (control.battery_load) |
| 226 and then executing: | 311 and then executing: |
| 227 | 312 |
| 228 $ ./server/autoserv -r results.battery_load -m <DUT_IP> \ | 313 $ ./server/autoserv -r results.battery_load -m <DUT_IP> \ |
| 229 -c client/site_tests/suite_HWQual/control.battery_load | 314 -c client/site_tests/suite_HWQual/control.battery_load |
| 230 | 315 |
| 316 * Alternate method from outside autotest directory: | |
| 317 $ python call_autoserv.py --ip=<DUT_IP> --test=battery_load | |
| 318 | |
| 231 ================================================================================ | 319 ================================================================================ |
| 232 Reviewing Automated and Semi-Automated Test Results | 320 Reviewing Automated and Semi-Automated Test Results |
| 233 ================================================================================ | 321 ================================================================================ |
| 234 | 322 |
| 235 - Autotest logs progress and performance data in results.* directories | 323 - Autotest logs progress and performance data in results.* directories |
| 236 specified through the '-r' autoserv option. The easy way to see a summary | 324 specified through the '-r' autoserv option. The easy way to see a summary |
| 237 of your test results is to use the 'generate_test_report' | 325 of your test results is to use the 'generate_test_report' |
| 238 script installed under $HOME/chromeos-hwqual-TAG/: | 326 script installed under $HOME/chromeos-hwqual-TAG/: |
| 239 | 327 |
| 240 $ ../generate_test_report results.* | 328 $ ../generate_test_report results.* |
| (...skipping 33 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... | |
| 274 | 362 |
| 275 $ cd $HOME/chromeos-hwqual-TAG/autotest/ | 363 $ cd $HOME/chromeos-hwqual-TAG/autotest/ |
| 276 $ cp ../manual/testcases.csv . | 364 $ cp ../manual/testcases.csv . |
| 277 | 365 |
| 278 - Package all results into a tar file: | 366 - Package all results into a tar file: |
| 279 | 367 |
| 280 $ tar cjf chromeos-hwqual-results-TAG-DATE.tar.bz2 results.* testcases.csv | 368 $ tar cjf chromeos-hwqual-results-TAG-DATE.tar.bz2 results.* testcases.csv |
| 281 | 369 |
| 282 - Send the resulting chromeos-hwqual-results file to your technical support | 370 - Send the resulting chromeos-hwqual-results file to your technical support |
| 283 contact. | 371 contact. |
| OLD | NEW |