| OLD | NEW |
| (Empty) | |
| 1 # Android Instrumentation Tests |
| 2 |
| 3 Instrumentation tests are Java tests based on |
| 4 [`android.app.Instrumentation`](https://developer.android.com/reference/android/
app/Instrumentation.html). |
| 5 They run on a device. |
| 6 |
| 7 [TOC] |
| 8 |
| 9 ## Writing an instrumentation test |
| 10 |
| 11 Currently, an instrumentation test is just a JUnit3-style test based on |
| 12 [android.test.InstrumentationTestCase](https://developer.android.com/reference/a
ndroid/test/InstrumentationTestCase.html). |
| 13 (This will change starting in [Android N](https://en.wikipedia.org/wiki/Android_
Nougat).) |
| 14 |
| 15 Writing an instrumentation test case can be simple, e.g. |
| 16 |
| 17 ```java |
| 18 package org.chromium.sample.test; |
| 19 |
| 20 import android.test.InstrumentationTestCase; |
| 21 |
| 22 public class MyInstrumentationTest extends InstrumentationTestCase { |
| 23 |
| 24 // Note that, because it's a JUnit3-style test, the test method *must* |
| 25 // start with "test". |
| 26 public void testTheFirst() { |
| 27 bool writingInstrumentationTestsCanBeEasy = true; |
| 28 |
| 29 // InstrumentationTestCase inherits the assert* methods through |
| 30 // junit.framework.TestCase. |
| 31 assertTrue(writingInstrumentationTestsCanBeEasy); |
| 32 } |
| 33 |
| 34 public void testTheSecond() { |
| 35 bool writingInstrumentationTestsIsAlwaysEasy = false; |
| 36 assertFalse(writingInstrumentationTestsIsAlwaysEasy); |
| 37 } |
| 38 } |
| 39 ``` |
| 40 |
| 41 After writing a test, you can run it by: |
| 42 |
| 43 - Adding the file to the relevant gn target if the entire file is new. |
| 44 Typically, the "relevant gn target" is simply the target containing the |
| 45 other files in the same directory. |
| 46 - Rebuild. |
| 47 - Run the test using the process described [here](/testing/android/docs/todo.md
). |
| 48 |
| 49 ## Instrumentation test features |
| 50 |
| 51 In many cases, Chromium has extended the instrumentation test framework |
| 52 classes to implement additional features. |
| 53 |
| 54 ### Test runners |
| 55 |
| 56 [todo](/testing/android/docs/todo.md) |
| 57 |
| 58 ### Test cases |
| 59 |
| 60 [todo](/testing/android/docs/todo.md) |
| 61 |
| 62 ### Annotations |
| 63 |
| 64 Instrumentation tests in Chromium use a wide variety of annotations to control |
| 65 and manipulate test execution. Some of these are implemented in Chromium, while |
| 66 others are pulled in from outside. They include: |
| 67 |
| 68 #### Size annotations |
| 69 |
| 70 Size annotations are used primarily by the test runner to determine the length |
| 71 of time to wait before considering a test hung (i.e., its timeout duration). |
| 72 |
| 73 Several of the annotations are Android APIs from |
| 74 [android.test.suitebuilder.annotation](https://developer.android.com/reference/a
ndroid/test/suitebuilder/annotation/package-summary.html) |
| 75 (prior to [Android N](https://en.wikipedia.org/wiki/Android_Nougat)) or |
| 76 [android.support.test.filters](https://developer.android.com/reference/android/s
upport/test/filters/package-summary.html) |
| 77 (starting in Android N). These are all fairly self-explanatory: |
| 78 |
| 79 - [`@SmallTest`](https://developer.android.com/reference/android/support/test/f
ilters/SmallTest.html) (timeout: **1 minute**) |
| 80 - [`@MediumTest`](https://developer.android.com/reference/android/support/test/
filters/MediumTest.html) (timeout: **3 minutes**) |
| 81 - [`@LargeTest`](https://developer.android.com/reference/android/support/test/f
ilters/LargeTest.html) (timeout: **5 minutes**) |
| 82 |
| 83 A few additional size annotations are provided in |
| 84 [//base](https://chromium.googlesource.com/chromium/src/+/master/base): |
| 85 |
| 86 - [`@EnormousTest`](https://chromium.googlesource.com/chromium/src/+/master/bas
e/test/android/javatests/src/org/chromium/base/test/util/EnormousTest.java) |
| 87 (timeout: **10 minutes**) Typically used for tests that require WiFi. |
| 88 - [`@IntegrationTest`](https://chromium.googlesource.com/chromium/src/+/master/
base/test/android/javatests/src/org/chromium/base/test/util/IntegrationTest.java
) |
| 89 (timeout: **30 minutes**) Used for tests that run against real services. |
| 90 - [`@Manual`](https://chromium.googlesource.com/chromium/src/+/master/base/test
/android/javatests/src/org/chromium/base/test/util/Manual.java) |
| 91 (timeout: **10 hours**) Used for manual tests. |
| 92 |
| 93 Beware that the timeout durations for these annotations are subject to |
| 94 change, though they rarely do. These values are defined |
| 95 [here](https://chromium.googlesource.com/chromium/src/+/master/build/android/pyl
ib/local/device/local_device_instrumentation_test_run.py#20). |
| 96 |
| 97 #### Annotations that disable tests |
| 98 |
| 99 There are several annotations that control whether or not a test runs. |
| 100 Some are conditional, others are not. |
| 101 |
| 102 ##### Unconditional disabling |
| 103 |
| 104 Two annotations disable a test unless a user specifically asks for tests |
| 105 annotated with that annotation to be run |
| 106 (e.g., via passing `-A DisabledTest` to the test runner). |
| 107 Note that these are both implemented in Chromium. |
| 108 There are versions of @DisabledTest and @FlakyTest in Android that do not allow
users to provide a message field. |
| 109 |
| 110 - [`@DisabledTest`](https://chromium.googlesource.com/chromium/src/+/master/bas
e/test/android/javatests/src/org/chromium/base/test/util/DisabledTest.java)`(mes
sage = "")` |
| 111 - [`@FlakyTest`](https://chromium.googlesource.com/chromium/src/+/master/base/t
est/android/javatests/src/org/chromium/base/test/util/FlakyTest.java)`(message =
"")` |
| 112 |
| 113 In both cases, `message` is a message that should describe why the test is |
| 114 disabled or marked as flaky. It should include a crbug link. |
| 115 |
| 116 ##### Conditional disabling |
| 117 |
| 118 There are two primary annotation categories that conditionally disable tests: |
| 119 **@DisableIf** and **@Restriction**. The **@DisableIf** annotations are intended |
| 120 to temporarily disable a test in certain scenarios where it *should* work but |
| 121 doesn't. In contrast, the **@Restriction** annotation is intended to |
| 122 permanently limit a test to specific configurations. It signifies that the test |
| 123 was not, is not, and will not be intended to run beyond those configurations. |
| 124 In both cases, conditional disabling manifests as a skipped test. |
| 125 |
| 126 **@DisableIf.Build** allows for conditional test disabling based on values in |
| 127 [`android.os.Build`](https://developer.android.com/reference/android/os/Build.ht
ml): |
| 128 |
| 129 ```java |
| 130 @DisableIf.Build( |
| 131 |
| 132 // Describes why the test is disabled. |
| 133 message = "", |
| 134 |
| 135 // Disables the test on SDK levels that match the given conditions. |
| 136 // Checks against Build.VERSION.SDK_INT. |
| 137 sdk_is_greater_than = 0, |
| 138 sdk_is_less_than = Integer.MAX_VALUE, |
| 139 |
| 140 // Disables the test on devices that support the given ABI |
| 141 // (e.g. "arm64-v8a"). Checks against: |
| 142 // - Build.SUPPORTED_ABIS on L+ |
| 143 // - Build.CPU_ABI and Build.CPU_ABI2 otherwise |
| 144 supported_abis_includes = "", |
| 145 |
| 146 // Disables the test on devices with hardware that matches the given |
| 147 // value. Checks against Build.HARDWARE. |
| 148 hardware_is = "", |
| 149 |
| 150 // Disables the test on devices with product names that contain the |
| 151 // given value. Checks against Build.PRODUCT. |
| 152 product_name_includes = "", |
| 153 |
| 154 ) |
| 155 ``` |
| 156 |
| 157 **@DisableIf.Device** allows for conditional test disabling based on whether |
| 158 a device is a phone, a tablet, or a "large tablet" as determined by |
| 159 [org.chromium.ui.base.DeviceFormFactor](https://chromium.googlesource.com/chromi
um/src/+/master/ui/android/java/src/org/chromium/ui/base/DeviceFormFactor.java). |
| 160 Note that this is currently only available to tests in |
| 161 [//chrome](https://chromium.googlesource.com/chromium/src/+/master/chrome/) |
| 162 or code that uses //chrome. |
| 163 |
| 164 ```java |
| 165 @DisableIf.Device( |
| 166 // Disables the test on devices that match the given type(s) as described |
| 167 // above. |
| 168 type = {} |
| 169 ) |
| 170 ``` |
| 171 |
| 172 **@Restriction** currently allows for conditional test disabling based on device |
| 173 type, device performance, internet connectivity, whether Google Play Services is |
| 174 up to date, and whether the build was an official one. |
| 175 |
| 176 ```java |
| 177 @Restriction( |
| 178 // Possible values include: |
| 179 // |
| 180 // base: |
| 181 // - Restriction.RESTRICTION_TYPE_LOW_END_DEVICE |
| 182 // Restricts the test to low-end devices as determined by SysUtils.isLowE
ndDevice(). |
| 183 // |
| 184 // - Restriction.RESTRICTION_TYPE_NON_LOW_END_DEVICE |
| 185 // Restricts the test to non-low-end devices as determined by SysUtils.is
LowEndDevice(). |
| 186 // |
| 187 // - Restriction.RESTRICTION_TYPE_INTERNET |
| 188 // Restricts the test to devices that have an internet connection. |
| 189 // |
| 190 // chrome: |
| 191 // - ChromeRestriction.RESTRICTION_TYPE_GOOGLE_PLAY_SERVICES |
| 192 // Restricts the test to devices with up-to-date versions of Google Play
Services. |
| 193 // |
| 194 // - ChromeRestriction.RESTRICTION_TYPE_PHONE |
| 195 // Restricts the test to phones as determined by DeviceFormFactor. |
| 196 // |
| 197 // - ChromeRestriction.RESTRICTION_TYPE_TABLET |
| 198 // Restricts the test to tablets as determined by DeviceFormFactor. |
| 199 // |
| 200 // - ChromeRestriction.RESTRICTION_TYPE_OFFICIAL_BUILD |
| 201 // Restricts the test to official builds as determined by ChromeVersionIn
fo.isOfficialBuild(). |
| 202 value = {} |
| 203 ) |
| 204 ``` |
| 205 |
| 206 **@MinAndroidSdkLevel** is similar to @Restriction in purpose in that it's |
| 207 intended to permanently limit a test to only recent versions of Android. |
| 208 |
| 209 ```java |
| 210 @MinAndroidSdkLevel( |
| 211 // The minimum SDK level at which this test should be executed. Checks |
| 212 // against Build.VERSION.SDK_INT. |
| 213 value = 0 |
| 214 ) |
| 215 ``` |
| 216 |
| 217 #### Annotations that affect how a test is run |
| 218 |
| 219 Several annotations affect how a test is run in interesting or nontrivial ways. |
| 220 |
| 221 **@CommandLineFlags.Add** and **@CommandLineFlags.Remove** manipulate Chrome's |
| 222 command-line flags on a per-test basis (i.e., the flags handled by |
| 223 [`org.chromium.base.CommandLine`](https://chromium.googlesource.com/chromium/src
/+/master/base/android/java/src/org/chromium/base/CommandLine.java) and |
| 224 [`base::CommandLine`](https://chromium.googlesource.com/chromium/src/+/master/ba
se/command_line.h)). |
| 225 |
| 226 ```java |
| 227 @CommandLineFlags.Add( |
| 228 // The flags to add to the command line for this test. These can be |
| 229 // anything and typically should include the leading dashes (e.g. "--foo"). |
| 230 value = {} |
| 231 ) |
| 232 |
| 233 @CommandLineFlags.Remove( |
| 234 // The flags to remove from the command line for this test. These can only |
| 235 // be flags added via @CommandLineFlags.Add. Flags already present in the |
| 236 // command-line file on the device are only present in the native |
| 237 // CommandLine object and cannot be manipulated. |
| 238 value = {} |
| 239 ) |
| 240 ``` |
| 241 |
| 242 #### Feature annotations |
| 243 |
| 244 **@Feature** has been used inconsistently in Chromium to group tests across |
| 245 test cases according to the feature they're testing. |
| 246 |
| 247 ```java |
| 248 @Feature( |
| 249 // The features associated with this test. These can be anything. |
| 250 value = {} |
| 251 ) |
| 252 ``` |
| 253 |
| 254 @Feature doesn't have an inherent function, but it can be used to filter tests |
| 255 via the test runner's |
| 256 [-A/--annotation](/testing/android/docs/todo.md) and |
| 257 [-E/--exclude-annotation](/testing/android/docs/todo.md) flags. For example, |
| 258 this would run only the tests with @Feature annotations containing at least |
| 259 "Sync" in `chrome_public_test_apk`: |
| 260 |
| 261 ```bash |
| 262 out/Debug/bin/run_chrome_public_test_apk -A Feature=Sync |
| 263 ``` |
| OLD | NEW |