🎯 Fail if an instrumentation test is missing size annotation

build/android/pylib/instrumentation/instrumentation_test_instance.py

 # Copyright 2015 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 import collections
 import copy
 import logging
 import os
 import pickle
 import re
@@ skipping 529 matching lines @@
       pickle.dump(pickle_data, pickle_file)
 
   def _FilterTests(self, tests):
 
     def gtest_filter(c, m):
       t = ['%s.%s' % (c['class'].split('.')[-1], m['method'])]
       return (not self._test_filter
               or unittest_util.FilterTestNames(t, self._test_filter))
 
     def annotation_filter(all_annotations):
-      if not self._annotations:
+      if not self._annotations or not all_annotations:

[jbudorick, 2016/04/07 02:30:11]

I don't think this is logically consistent. A test with no annotations
_shouldn't_ match on a non-empty set of desired annotations.

The appropriate fixes would be to either: 
 - support WebRTC providing a completely empty set of desired annotations, or
 - have WebRTC add size annotations to their tests.

[jbudorick, 2016/04/07 02:31:26]

(or 3, remove the default annotations entirely, but that's a change with a large
potential for side effects)

[agrieve, 2016/04/07 02:41:12]

Should we add an exception when we detect a test method with no annotation?
Seems a bit weird that we can detect test methods, and then have no way for them
to not get filtered out. This change special-cases them so that no annotation
means always included it, but I think if we don't want that logic, we should
throw an error if we find an unannotated test (since there is no way to run it).
wdyt?

[jbudorick, 2016/04/07 02:50:18]

Honestly, I think the right answer for this is probably either to have WebRTC
add annotations or to remove the default set and only do annotation-based
filtering if a user explicitly passes one, and I'm leaning toward the latter.

In order to confidently do that, we'd have to figure out which tests in each of
the four chromium suites aren't annotated with any of the default annotations or
the default excluded annotations.

         return True
       return any_annotation_matches(self._annotations, all_annotations)
 
     def excluded_annotation_filter(all_annotations):
       if not self._excluded_annotations:
         return True
       return not any_annotation_matches(self._excluded_annotations,
                                         all_annotations)
 
     def any_annotation_matches(annotations, all_annotations):
@@ skipping 74 matching lines @@
   def GenerateTestResults(
       result_code, result_bundle, statuses, start_ms, duration_ms):
     return GenerateTestResults(result_code, result_bundle, statuses,
                                start_ms, duration_ms)
 
   #override
   def TearDown(self):
     if self._isolate_delegate:
       self._isolate_delegate.Clear()