Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(11)

Side by Side Diff: testing/android/docs/instrumentation.md

Issue 2255593002: [Android] Add partial md documentation for instrumentation test features. (Closed) Base URL: https://chromium.googlesource.com/chromium/src.git@master
Patch Set: Created 4 years, 4 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
OLDNEW
(Empty)
1 # Android Instrumentation Tests
2
3 Instrumentation tests are Java tests based on
4 [`android.app.Instrumentation`](https://developer.android.com/reference/android/ app/Instrumentation.html).
5 They run on a device.
6
7 [TOC]
8
9 ## Writing an instrumentation test
10
11 Currently, an instrumentation test is just a JUnit3-style test based on
12 [android.test.InstrumentationTestCase](https://developer.android.com/reference/a ndroid/test/InstrumentationTestCase.html).
13 (This will change starting in [Android N](https://en.wikipedia.org/wiki/Android_ Nougat).)
14
15 Writing an instrumentation test case can be simple, e.g.
16
17 ```java
18 package org.chromium.sample.test;
19
20 import android.test.InstrumentationTestCase;
21
22 public class MyInstrumentationTest extends InstrumentationTestCase {
23
24 // Note that, because it's a JUnit3-style test, the test method *must*
25 // start with "test".
26 public void testTheFirst() {
27 bool writingInstrumentationTestsCanBeEasy = true;
28
29 // InstrumentationTestCase inherits the assert* methods through
30 // junit.framework.TestCase.
31 assertTrue(writingInstrumentationTestsCanBeEasy);
32 }
33
34 public void testTheSecond() {
35 bool writingInstrumentationTestsIsAlwaysEasy = false;
36 assertFalse(writingInstrumentationTestsIsAlwaysEasy);
37 }
38 }
39 ```
40
41 After writing a test, you can run it by:
42
43 - Adding the file to the relevant gn target if the entire file is new.
44 Typically, the "relevant gn target" is simply the target containing the
45 other files in the same directory.
46 - Rebuild.
47 - Run the test using the process described [here](/testing/android/docs/todo.md ).
48
49 ## Instrumentation test features
50
51 In many cases, Chromium has extended the instrumentation test framework
52 classes to implement additional features.
53
54 ### Test runners
55
56 [todo](/testing/android/docs/todo.md)
57
58 ### Test cases
59
60 [todo](/testing/android/docs/todo.md)
61
62 ### Annotations
63
64 Instrumentation tests in Chromium use a wide variety of annotations to control
65 and manipulate test execution. Some of these are implemented in Chromium, while
66 others are pulled in from outside. They include:
67
68 #### Size annotations
69
70 Size annotations are used primarily by the test runner to determine the length
71 of time to wait before considering a test hung (i.e., its timeout duration).
72
73 Several of the annotations are Android APIs from
74 [android.test.suitebuilder.annotation](https://developer.android.com/reference/a ndroid/test/suitebuilder/annotation/package-summary.html)
75 (prior to [Android N](https://en.wikipedia.org/wiki/Android_Nougat)) or
76 [android.support.test.filters](https://developer.android.com/reference/android/s upport/test/filters/package-summary.html)
77 (starting in Android N). These are all fairly self-explanatory:
78
79 - [`@SmallTest`](https://developer.android.com/reference/android/support/test/f ilters/SmallTest.html) (timeout: **1 minute**)
80 - [`@MediumTest`](https://developer.android.com/reference/android/support/test/ filters/MediumTest.html) (timeout: **3 minutes**)
81 - [`@LargeTest`](https://developer.android.com/reference/android/support/test/f ilters/LargeTest.html) (timeout: **5 minutes**)
82
83 A few additional size annotations are provided in
84 [//base](https://chromium.googlesource.com/chromium/src/+/master/base):
85
86 - [`@EnormousTest`](https://chromium.googlesource.com/chromium/src/+/master/bas e/test/android/javatests/src/org/chromium/base/test/util/EnormousTest.java)
87 (timeout: **10 minutes**) Typically used for tests that require WiFi.
88 - [`@IntegrationTest`](https://chromium.googlesource.com/chromium/src/+/master/ base/test/android/javatests/src/org/chromium/base/test/util/IntegrationTest.java )
89 (timeout: **30 minutes**) Used for tests that run against real services.
90 - [`@Manual`](https://chromium.googlesource.com/chromium/src/+/master/base/test /android/javatests/src/org/chromium/base/test/util/Manual.java)
91 (timeout: **10 hours**) Used for manual tests.
92
93 Beware that the timeout duration the corresponds to each of these annotations
94 is subject to change.
95
96 #### Annotations that disable tests
97
98 There are several annotations that control whether or not a test runs.
99 Some are conditional, others are not.
100
101 ##### Unconditional disabling
102
103 Two annotations disable a test unless a user specifically asks for tests
104 annotated with that annotation to be run
105 (e.g., via passing `-A DisabledTest` to the test runner).
106 Note that these are both implemented in Chromium.
107 There are versions of @DisabledTest and @FlakyTest in Android that do not allow users to provide a message field.
108
109 - [`@DisabledTest`](https://chromium.googlesource.com/chromium/src/+/master/bas e/test/android/javatests/src/org/chromium/base/test/util/DisabledTest.java)`(msg = "")`
mikecase (-- gone --) 2016/08/16 23:08:58 s/msg/message
jbudorick 2016/08/16 23:54:23 Done.
110 - [`@FlakyTest`](https://chromium.googlesource.com/chromium/src/+/master/base/t est/android/javatests/src/org/chromium/base/test/util/FlakyTest.java)`(msg = "") `
mikecase (-- gone --) 2016/08/16 23:08:58 s/msg/message
jbudorick 2016/08/16 23:54:23 Done.
111
112 In both cases, `msg` is a message that should describe why the test is
mikecase (-- gone --) 2016/08/16 23:08:57 s/msg/message
jbudorick 2016/08/16 23:54:23 Done.
113 disabled or marked as flaky. It should include a crbug link.
114
115 ##### Conditional disabling
116
117 There are two primary annotation categories that conditionally disable tests:
118 **@DisableIf** and **Restriction**. The **@DisableIf** annotations are intended
mikecase (-- gone --) 2016/08/16 23:08:57 nit: change Restriction to @Restriction? To be con
jbudorick 2016/08/16 23:54:23 Done.
119 to temporarily disable a test in certain scenarios where it *should* work but
120 doesn't. In contrast, the **@Restriction** annotation is intended to
121 permanently limit a test to specific configurations. It signifies that the test
122 was not, is not, and will not be intended to run beyond those configurations.
123 In both cases, conditional disabling manifests as a skipped test.
124
125 **@DisableIf.Build** allows for conditional test disabling based on values in
126 [`android.os.Build`](https://developer.android.com/reference/android/os/Build.ht ml):
127
128 ```java
129 @DisableIf.Build(
130
131 // Describes why the test is disabled.
132 message = "",
133
134 // Disables the test on SDK levels that match the given conditions.
135 // Checks against Build.VERSION.SDK_INT.
136 sdk_is_greater_than = 0,
137 sdk_is_less_than = Integer.MAX_VALUE,
138
139 // Disables the test on devices that support the given ABI
140 // (e.g. "arm64-v8a"). Checks against:
141 // - Build.SUPPORTED_ABIS on L+
142 // - Build.CPU_ABI and Build.CPU_ABI2 otherwise
143 supported_abis_includes = "",
144
145 // Disables the test on devices with hardware that matches the given
146 // value. Checks against Build.HARDWARE.
147 hardware_is = "",
148
149 // Disables the test on devices with product names that contain the
150 // given value. Checks against Build.PRODUCT.
151 product_name_includes = "",
152
153 )
154 ```
155
156 **@DisableIf.Device** allows for conditional test disabling based on whether
157 a device is a phone, a tablet, or a "large tablet" as determined by
158 [org.chromium.ui.base.DeviceFormFactor](https://chromium.googlesource.com/chromi um/src/+/master/ui/android/java/src/org/chromium/ui/base/DeviceFormFactor.java).
159 Note that this is currently only available to tests in
160 [//chrome](https://chromium.googlesource.com/chromium/src/+/master/chrome/)
161 or code that uses //chrome.
162
163 ```java
164 @DisableIf.Device(
165 // Disables the test on devices that match the given type(s) as described
166 // above.
167 type = {}
168 )
169 ```
170
171 **@Restriction** currently allows for conditional test disabling based on device
172 type, device performance, internet connectivity, whether Google Play Services is
173 up to date, and whether the build was an official one.
174
175 ```java
176 @Restriction(
177 // Possible values include:
178 //
179 // base:
180 // - Restriction.RESTRICTION_TYPE_LOW_END_DEVICE
181 // Restricts the test to low-end devices as determined by SysUtils.isLowE ndDevice().
182 //
183 // - Restriction.RESTRICTION_TYPE_NON_LOW_END_DEVICE
184 // Restricts the test to non-low-end devices as determined by SysUtils.is LowEndDevice().
185 //
186 // - Restriction.RESTRICTION_TYPE_INTERNET
187 // Restricts the test to devices that have an internet connection.
188 //
189 // chrome:
190 // - ChromeRestriction.RESTRICTION_TYPE_GOOGLE_PLAY_SERVICES
191 // Restricts the test to devices with up-to-date versions of Google Play Services.
192 //
193 // - ChromeRestriction.RESTRICTION_TYPE_PHONE
194 // Restricts the test to phones as determined by DeviceFormFactor.
195 //
196 // - ChromeRestriction.RESTRICTION_TYPE_TABLET
197 // Restricts the test to tablets as determined by DeviceFormFactor.
198 //
199 // - ChromeRestriction.RESTRICTION_TYPE_OFFICIAL_BUILD
200 // Restricts the test to official builds as determined by ChromeVersionIn fo.isOfficialBuild().
201 value = {}
202 )
203 ```
204
205 **@MinAndroidSdkLevel** is similar to @Restriction in purpose in that it's
206 intended to permanently limit a test to only recent versions of Android.
207
208 ```java
209 @MinAndroidSdkLevel(
210 // The minimum SDK level at which this test should be executed. Checks
211 // against Build.VERSION.SDK_INT.
212 value = 0
213 )
214 ```
215
216 #### Annotations that affect how a test is run
mikecase (-- gone --) 2016/08/16 23:08:57 super nit: idk if it was just the markdown viewer
jbudorick 2016/08/16 23:54:23 //tools/md_browser/md_browser.py (which I used loc
jbudorick 2016/08/16 23:55:15 s/is definitely renders/definitely renders/ used
217
218 Several annotations affect how a test is run in interesting or nontrivial ways.
219
220 **@CommandLineFlags.Add** and **@CommandLineFlags.Remove** manipulate Chrome's
221 command-line flags on a per-test basis (i.e., the flags handled by
222 [`org.chromium.base.CommandLine`](https://chromium.googlesource.com/chromium/src /+/master/base/android/java/src/org/chromium/base/CommandLine.java) and
223 [`base::CommandLine`](https://chromium.googlesource.com/chromium/src/+/master/ba se/command_line.h)).
224
225 ```java
226 @CommandLineFlags.Add(
227 // The flags to add to the command line for this test. These can be
228 // anything and typically should include the leading dashes (e.g. "--foo").
229 value = {}
230 )
231
232 @CommandLineFlags.Remove(
233 // The flags to remove from the command line for this test. These can only
234 // be flags added via @CommandLineFlags.Add. Flags already present in the
235 // command-line file on the device are only present in the native
236 // CommandLine object and cannot be manipulated.
237 value = {}
238 )
239 ```
240
241 #### Feature annotations
242
243 **@Feature** has been used inconsistently in Chromium to group tests across
244 test cases according to the feature they're testing.
245
246 ```java
247 @Feature(
248 // The features associated with this test. These can be anything.
249 value = {}
250 )
251 ```
252
253 @Feature doesn't have an inherent function, but it can be used to filter tests
254 via the test runner's
255 [-A/--annotation](/testing/android/docs/todo.md) and
256 [-E/--exclude-annotation](/testing/android/docs/todo.md) flags. For example,
257 this would run only the tests with @Feature annotations containing at least
258 "Sync" in `chrome_public_test_apk`:
259
260 ```bash
261 out/Debug/bin/run_chrome_public_test_apk -A Feature=Sync
262 ```
OLDNEW

Powered by Google App Engine
This is Rietveld 408576698