Document the Combined Field Trial Testing Configuration Format

testing/variations/README.md

Index: testing/variations/README.md
diff --git a/testing/variations/README.md b/testing/variations/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..0fa298cbe0b2bf24f6f3ace2900df403ffcecb3a
--- /dev/null
+++ b/testing/variations/README.md
@@ -0,0 +1,112 @@
+# Field Trial Testing Configuration
+
+This directory contains the field trial configuration used to ensure testing
+coverage of the experiments defined in `fieldtrial_testing_config.json`.
+
+Note that these apply specifically for Chromium builds. Chrome branded /
+official builds do not use these definitions.
+
+## Config File Format
+
+```json
+{
+    "StudyName": [
+        {
+            "platforms": [Array of Strings of Valid Platforms for These Experiments],
+            "experiments": [
+                {
+                    "//0:": "Comment Line 0. Lines 0-9 are supported.",
+                    "name": "ExperimentName",
+                    "params": {Dictionary of Params},
+                    "enable_features": [Array of Strings of Features],
+                    "disable_features": [Array of Strings of Features]
+                },
+                ...
+            ]
+        },
+        ...
+    ],
+    ...
+}
+```
+
+The config file is a dictionary at the top level mapping a study name to an
+array of *experiment descriptions*. The study name should match the Field Trial
+study name.
+
+### Experiment Descriptions
+
+Each *experiment description* is a dictionary containing both `platform` and
+`experiments`.
+
+`platform` is an array of strings of valid platforms and the
+strings may be `android`, `chromeos`, `ios`, `linux`, `mac`, or `windows`.
+
+`experiments` is an array containing the *experiments*.
+
+The converter uses the platforms array to determine what experiments to include
+for the study. All matching platforms will have their experiments concatenated
+together for the study.
+
+The first available experiment after platform filtering and concatenation is the
+default experiment for Chromium builds. This experiment is also used for perf
+bots and browser tests in the waterfall.

[Alexei Svitkine (slow), 2016/09/21 21:38:17]

I feel like this paragraph should live in the first section of the doc, which
also discusses Chromium builds.

[robliao, 2016/09/21 22:06:09]

Done.

+
+### Experiments
+Each *experiment* is a dictionary that must contain the `name` key, identifying
+the experiment group name which should match the Field Trial experiment group
+name.
+
+The remaining keys, `params`, `enable_features`, and `disable_features` are
+optional.
+
+`params` is a dictionary mapping parameter name to parameter.
+
+`enable_features` and `disable_features` indicate which features should be
+enabled and disabled respectively. As a reminder, as the variations framework
+does not actually fetch the Field Trial definitions from the server for Chromium
+builds, so any feature enabling or disabling must be done here.
+
+#### Comments
+
+Each experiment may have up to 10 lines of comments. The comment key must be of
+the form `//N` where `N` is between 0 and 9.
+
+```json
+{
+    "AStudyWithExperimentComment": [
+        {
+            "platforms": ["chromeos", "linux", "mac", "win"],
+            "experiments": [
+                {
+                    //0: "This is the first comment line.",

[Alexei Svitkine (slow), 2016/09/21 21:38:17]

The syntax seems different here than on line 18. Please make consistent

More generally, this syntax is pretty awful and it would be nice if we could
just support normal // and /* */ comments. We do this for internal experiment
configs, so it would be nice to make consistent.

(If it's something you'd like to look at, we definitely appreciate the help -
but I don't want to make you shave even more yaks unless you want it - so I'm OK
with just adding a TODO() here to support a better comment syntax and we can
look at it in the future.)

[robliao, 2016/09/21 22:06:09]

Made it consistent, although this change would be better tracked as a bug rather
than in docs. http://crbug.com/649152

Agreed the syntax could be better, but that's the cost of using JSON to
represent data.
http://stackoverflow.com/questions/244777/can-comments-be-used-in-json

Compliant JSON doesn't have a way to support comments. If we changed our
serialization format, then maybe we can support comments. 

I was surprised to see this added recently in 
https://codereview.chromium.org/2306473002 while this CL was going through
review.

+                    //1: "This is the second comment line.",
+                    "name": "DesktopExperiment"
+                }
+            ]
+        }
+    ]
+}
+```
+
+### Specifying Different Experiments for Different Platforms
+Simply specify two different experiment configurations under the same study:
+
+```json
+{
+    "DifferentExperimentsPerPlatform": [
+        {
+            "platforms": ["chromeos", "linux", "mac", "win"],
+            "experiments": [{ "name": "DesktopExperiment" }]
+        },
+        {
+            "platforms": ["android", "ios"],
+            "experiments": [{ "name": "MobileExperiment" }]
+        }
+    ]
+}
+```
+
+## Presubmit
+The presubmit tool will ensure that your changes follow the correct ordering and
+format.