[system-health] Add support for disabling individual stories on individual platforms

tools/perf/page_sets/system_health/platforms.py

Index: tools/perf/page_sets/system_health/platforms.py
diff --git a/tools/perf/page_sets/system_health/platforms.py b/tools/perf/page_sets/system_health/platforms.py
index cc6bb5b381a2f703d71945530a3667a4492e4b48..a54fc41c4dd9daf1053a79b6839c9cf0bcca511d 100644
--- a/tools/perf/page_sets/system_health/platforms.py
+++ b/tools/perf/page_sets/system_health/platforms.py
@@ -6,7 +6,10 @@ DESKTOP = 'desktop'
 MOBILE = 'mobile'
 
 ALL_PLATFORMS = frozenset({DESKTOP, MOBILE})
+
+# Use the constants below to mark on which platforms the story has WPR
+# recordings. To disable a story (e.g. because it crashes or takes too long),
+# use @decorators.Disabled instead.
 DESKTOP_ONLY = frozenset({DESKTOP})
 MOBILE_ONLY = frozenset({MOBILE})
-# This is used for disabling a story on all platforms.
 NO_PLATFORMS = frozenset()

[nednguyen, 2016/09/08 19:31:16]

Why do you add this back? I thought your decision was removing this is right :-)

[petrcermak, 2016/09/09 09:34:09]

Good question. The reason is that it needs to be used for stories which don't
have a WPR recording (e.g. FacebookPhotosDesktopStory in browsing_stories.py).

If I just used @decorators.Disabled('all') on the story, the benchmark would
fail because of "missing WPR archive" (even though the story would never
actually run).

Maybe the story_runner logic could be updated to handle such cases properly, but
it might might be difficult in the general case (shared_state.CanRun() can
depend on the current state of things, so it might not be possible to tell
upfront if a story will run; downloading the WPR archives on demand could, on
the other hand, add noise to benchmarks which don't tear down the browser
between individual stories) and is probably not be worth the effort.