Chromium Code Reviews| OLD | NEW |
|---|---|
| (Empty) | |
| 1 # Android Instrumentation Tests | |
| 2 | |
| 3 Instrumentation tests are Java tests based on | |
| 4 [`android.app.Instrumentation`](https://developer.android.com/reference/android/ app/Instrumentation.html). | |
| 5 They run on a device. | |
| 6 | |
| 7 [TOC] | |
| 8 | |
| 9 ## Writing an instrumentation test | |
| 10 | |
| 11 Currently, an instrumentation test is just a JUnit3-style test based on | |
| 12 [android.test.InstrumentationTestCase](https://developer.android.com/reference/a ndroid/test/InstrumentationTestCase.html). | |
| 13 (This will change starting in [Android N](https://en.wikipedia.org/wiki/Android_ Nougat).) | |
| 14 | |
| 15 Writing an instrumentation test case can be simple, e.g. | |
| 16 | |
| 17 ```java | |
| 18 package org.chromium.sample.test; | |
| 19 | |
| 20 import android.test.InstrumentationTestCase; | |
| 21 | |
| 22 public class MyInstrumentationTest extends InstrumentationTestCase { | |
| 23 | |
| 24 // Note that, because it's a JUnit3-style test, the test method *must* | |
| 25 // start with "test". | |
| 26 public void testTheFirst() { | |
| 27 bool writingInstrumentationTestsCanBeEasy = true; | |
| 28 | |
| 29 // InstrumentationTestCase inherits the assert* methods through | |
| 30 // junit.framework.TestCase. | |
| 31 assertTrue(writingInstrumentationTestsCanBeEasy); | |
| 32 } | |
| 33 | |
| 34 public void testTheSecond() { | |
| 35 bool writingInstrumentationTestsIsAlwaysEasy = false; | |
| 36 assertFalse(writingInstrumentationTestsIsAlwaysEasy); | |
| 37 } | |
| 38 } | |
| 39 ``` | |
| 40 | |
| 41 After writing a test, you can run it by: | |
| 42 | |
| 43 - Adding the file to the relevant gn target if the entire file is new. | |
| 44 Typically, the "relevant gn target" is simply the target containing the | |
| 45 other files in the same directory. | |
| 46 - Rebuild. | |
| 47 - Run the test using the process described [here](/testing/android/docs/todo.md ). | |
| 48 | |
| 49 ## Instrumentation test features | |
| 50 | |
| 51 In many cases, Chromium has extended the instrumentation test framework | |
| 52 classes to implement additional features. | |
| 53 | |
| 54 ### Test runners | |
| 55 | |
| 56 [todo](/testing/android/docs/todo.md) | |
| 57 | |
| 58 ### Test cases | |
| 59 | |
| 60 [todo](/testing/android/docs/todo.md) | |
| 61 | |
| 62 ### Annotations | |
| 63 | |
| 64 Instrumentation tests in Chromium use a wide variety of annotations to control | |
| 65 and manipulate test execution. Some of these are implemented in Chromium, while | |
| 66 others are pulled in from outside. They include: | |
| 67 | |
| 68 #### Size annotations | |
| 69 | |
| 70 Size annotations are used primarily by the test runner to determine the length | |
| 71 of time to wait before considering a test hung (i.e., its timeout duration). | |
| 72 | |
| 73 Several of the annotations are Android APIs from | |
| 74 [android.test.suitebuilder.annotation](https://developer.android.com/reference/a ndroid/test/suitebuilder/annotation/package-summary.html) | |
| 75 (prior to [Android N](https://en.wikipedia.org/wiki/Android_Nougat)) or | |
| 76 [android.support.test.filters](https://developer.android.com/reference/android/s upport/test/filters/package-summary.html) | |
| 77 (starting in Android N). These are all fairly self-explanatory: | |
| 78 | |
| 79 - [`@SmallTest`](https://developer.android.com/reference/android/support/test/f ilters/SmallTest.html) (timeout: **1 minute**) | |
| 80 - [`@MediumTest`](https://developer.android.com/reference/android/support/test/ filters/MediumTest.html) (timeout: **3 minutes**) | |
| 81 - [`@LargeTest`](https://developer.android.com/reference/android/support/test/f ilters/LargeTest.html) (timeout: **5 minutes**) | |
| 82 | |
| 83 A few additional size annotations are provided in | |
| 84 [//base](https://chromium.googlesource.com/chromium/src/+/master/base): | |
| 85 | |
| 86 - [`@EnormousTest`](https://chromium.googlesource.com/chromium/src/+/master/bas e/test/android/javatests/src/org/chromium/base/test/util/EnormousTest.java) | |
| 87 (timeout: **10 minutes**) Typically used for tests that require WiFi. | |
| 88 - [`@IntegrationTest`](https://chromium.googlesource.com/chromium/src/+/master/ base/test/android/javatests/src/org/chromium/base/test/util/IntegrationTest.java ) | |
| 89 (timeout: **30 minutes**) Used for tests that run against real services. | |
| 90 - [`@Manual`](https://chromium.googlesource.com/chromium/src/+/master/base/test /android/javatests/src/org/chromium/base/test/util/Manual.java) | |
| 91 (timeout: **10 hours**) Used for manual tests. | |
| 92 | |
| 93 Beware that the timeout duration the corresponds to each of these annotations | |
| 94 is subject to change. | |
| 95 | |
| 96 #### Annotations that disable tests | |
| 97 | |
| 98 There are several annotations that control whether or not a test runs. | |
| 99 Some are conditional, others are not. | |
| 100 | |
| 101 ##### Unconditional disabling | |
| 102 | |
| 103 Two annotations disable a test unless a user specifically asks for tests | |
| 104 annotated with that annotation to be run | |
| 105 (e.g., via passing `-A DisabledTest` to the test runner). | |
| 106 Note that these are both implemented in Chromium. | |
| 107 There are versions of @DisabledTest and @FlakyTest in Android that do not allow users to provide a message field. | |
| 108 | |
| 109 - [`@DisabledTest`](https://chromium.googlesource.com/chromium/src/+/master/bas e/test/android/javatests/src/org/chromium/base/test/util/DisabledTest.java)`(msg = "")` | |
|
mikecase (-- gone --)
2016/08/16 23:08:58
s/msg/message
jbudorick
2016/08/16 23:54:23
Done.
| |
| 110 - [`@FlakyTest`](https://chromium.googlesource.com/chromium/src/+/master/base/t est/android/javatests/src/org/chromium/base/test/util/FlakyTest.java)`(msg = "") ` | |
|
mikecase (-- gone --)
2016/08/16 23:08:58
s/msg/message
jbudorick
2016/08/16 23:54:23
Done.
| |
| 111 | |
| 112 In both cases, `msg` is a message that should describe why the test is | |
|
mikecase (-- gone --)
2016/08/16 23:08:57
s/msg/message
jbudorick
2016/08/16 23:54:23
Done.
| |
| 113 disabled or marked as flaky. It should include a crbug link. | |
| 114 | |
| 115 ##### Conditional disabling | |
| 116 | |
| 117 There are two primary annotation categories that conditionally disable tests: | |
| 118 **@DisableIf** and **Restriction**. The **@DisableIf** annotations are intended | |
|
mikecase (-- gone --)
2016/08/16 23:08:57
nit: change Restriction to @Restriction? To be con
jbudorick
2016/08/16 23:54:23
Done.
| |
| 119 to temporarily disable a test in certain scenarios where it *should* work but | |
| 120 doesn't. In contrast, the **@Restriction** annotation is intended to | |
| 121 permanently limit a test to specific configurations. It signifies that the test | |
| 122 was not, is not, and will not be intended to run beyond those configurations. | |
| 123 In both cases, conditional disabling manifests as a skipped test. | |
| 124 | |
| 125 **@DisableIf.Build** allows for conditional test disabling based on values in | |
| 126 [`android.os.Build`](https://developer.android.com/reference/android/os/Build.ht ml): | |
| 127 | |
| 128 ```java | |
| 129 @DisableIf.Build( | |
| 130 | |
| 131 // Describes why the test is disabled. | |
| 132 message = "", | |
| 133 | |
| 134 // Disables the test on SDK levels that match the given conditions. | |
| 135 // Checks against Build.VERSION.SDK_INT. | |
| 136 sdk_is_greater_than = 0, | |
| 137 sdk_is_less_than = Integer.MAX_VALUE, | |
| 138 | |
| 139 // Disables the test on devices that support the given ABI | |
| 140 // (e.g. "arm64-v8a"). Checks against: | |
| 141 // - Build.SUPPORTED_ABIS on L+ | |
| 142 // - Build.CPU_ABI and Build.CPU_ABI2 otherwise | |
| 143 supported_abis_includes = "", | |
| 144 | |
| 145 // Disables the test on devices with hardware that matches the given | |
| 146 // value. Checks against Build.HARDWARE. | |
| 147 hardware_is = "", | |
| 148 | |
| 149 // Disables the test on devices with product names that contain the | |
| 150 // given value. Checks against Build.PRODUCT. | |
| 151 product_name_includes = "", | |
| 152 | |
| 153 ) | |
| 154 ``` | |
| 155 | |
| 156 **@DisableIf.Device** allows for conditional test disabling based on whether | |
| 157 a device is a phone, a tablet, or a "large tablet" as determined by | |
| 158 [org.chromium.ui.base.DeviceFormFactor](https://chromium.googlesource.com/chromi um/src/+/master/ui/android/java/src/org/chromium/ui/base/DeviceFormFactor.java). | |
| 159 Note that this is currently only available to tests in | |
| 160 [//chrome](https://chromium.googlesource.com/chromium/src/+/master/chrome/) | |
| 161 or code that uses //chrome. | |
| 162 | |
| 163 ```java | |
| 164 @DisableIf.Device( | |
| 165 // Disables the test on devices that match the given type(s) as described | |
| 166 // above. | |
| 167 type = {} | |
| 168 ) | |
| 169 ``` | |
| 170 | |
| 171 **@Restriction** currently allows for conditional test disabling based on device | |
| 172 type, device performance, internet connectivity, whether Google Play Services is | |
| 173 up to date, and whether the build was an official one. | |
| 174 | |
| 175 ```java | |
| 176 @Restriction( | |
| 177 // Possible values include: | |
| 178 // | |
| 179 // base: | |
| 180 // - Restriction.RESTRICTION_TYPE_LOW_END_DEVICE | |
| 181 // Restricts the test to low-end devices as determined by SysUtils.isLowE ndDevice(). | |
| 182 // | |
| 183 // - Restriction.RESTRICTION_TYPE_NON_LOW_END_DEVICE | |
| 184 // Restricts the test to non-low-end devices as determined by SysUtils.is LowEndDevice(). | |
| 185 // | |
| 186 // - Restriction.RESTRICTION_TYPE_INTERNET | |
| 187 // Restricts the test to devices that have an internet connection. | |
| 188 // | |
| 189 // chrome: | |
| 190 // - ChromeRestriction.RESTRICTION_TYPE_GOOGLE_PLAY_SERVICES | |
| 191 // Restricts the test to devices with up-to-date versions of Google Play Services. | |
| 192 // | |
| 193 // - ChromeRestriction.RESTRICTION_TYPE_PHONE | |
| 194 // Restricts the test to phones as determined by DeviceFormFactor. | |
| 195 // | |
| 196 // - ChromeRestriction.RESTRICTION_TYPE_TABLET | |
| 197 // Restricts the test to tablets as determined by DeviceFormFactor. | |
| 198 // | |
| 199 // - ChromeRestriction.RESTRICTION_TYPE_OFFICIAL_BUILD | |
| 200 // Restricts the test to official builds as determined by ChromeVersionIn fo.isOfficialBuild(). | |
| 201 value = {} | |
| 202 ) | |
| 203 ``` | |
| 204 | |
| 205 **@MinAndroidSdkLevel** is similar to @Restriction in purpose in that it's | |
| 206 intended to permanently limit a test to only recent versions of Android. | |
| 207 | |
| 208 ```java | |
| 209 @MinAndroidSdkLevel( | |
| 210 // The minimum SDK level at which this test should be executed. Checks | |
| 211 // against Build.VERSION.SDK_INT. | |
| 212 value = 0 | |
| 213 ) | |
| 214 ``` | |
| 215 | |
| 216 #### Annotations that affect how a test is run | |
|
mikecase (-- gone --)
2016/08/16 23:08:57
super nit:
idk if it was just the markdown viewer
jbudorick
2016/08/16 23:54:23
//tools/md_browser/md_browser.py (which I used loc
jbudorick
2016/08/16 23:55:15
s/is definitely renders/definitely renders/
used
| |
| 217 | |
| 218 Several annotations affect how a test is run in interesting or nontrivial ways. | |
| 219 | |
| 220 **@CommandLineFlags.Add** and **@CommandLineFlags.Remove** manipulate Chrome's | |
| 221 command-line flags on a per-test basis (i.e., the flags handled by | |
| 222 [`org.chromium.base.CommandLine`](https://chromium.googlesource.com/chromium/src /+/master/base/android/java/src/org/chromium/base/CommandLine.java) and | |
| 223 [`base::CommandLine`](https://chromium.googlesource.com/chromium/src/+/master/ba se/command_line.h)). | |
| 224 | |
| 225 ```java | |
| 226 @CommandLineFlags.Add( | |
| 227 // The flags to add to the command line for this test. These can be | |
| 228 // anything and typically should include the leading dashes (e.g. "--foo"). | |
| 229 value = {} | |
| 230 ) | |
| 231 | |
| 232 @CommandLineFlags.Remove( | |
| 233 // The flags to remove from the command line for this test. These can only | |
| 234 // be flags added via @CommandLineFlags.Add. Flags already present in the | |
| 235 // command-line file on the device are only present in the native | |
| 236 // CommandLine object and cannot be manipulated. | |
| 237 value = {} | |
| 238 ) | |
| 239 ``` | |
| 240 | |
| 241 #### Feature annotations | |
| 242 | |
| 243 **@Feature** has been used inconsistently in Chromium to group tests across | |
| 244 test cases according to the feature they're testing. | |
| 245 | |
| 246 ```java | |
| 247 @Feature( | |
| 248 // The features associated with this test. These can be anything. | |
| 249 value = {} | |
| 250 ) | |
| 251 ``` | |
| 252 | |
| 253 @Feature doesn't have an inherent function, but it can be used to filter tests | |
| 254 via the test runner's | |
| 255 [-A/--annotation](/testing/android/docs/todo.md) and | |
| 256 [-E/--exclude-annotation](/testing/android/docs/todo.md) flags. For example, | |
| 257 this would run only the tests with @Feature annotations containing at least | |
| 258 "Sync" in `chrome_public_test_apk`: | |
| 259 | |
| 260 ```bash | |
| 261 out/Debug/bin/run_chrome_public_test_apk -A Feature=Sync | |
| 262 ``` | |
| OLD | NEW |