| 
    
      
  | 
  
 Chromium Code Reviews| 
         Created: 
          4 years, 4 months ago by charliea (OOO until 10-5) Modified: 
          
          
          4 years, 3 months ago CC: 
          
          
          chromium-reviews, telemetry-reviews_chromium.org, nduca Base URL: 
          
          
          https://chromium.googlesource.com/chromium/src.git@master Target Ref: 
          
          
          refs/pending/heads/master Project: 
          
          chromium Visibility: 
          
          
          
        Public.  | 
      
        
  Description[system health] Add power system health benchmarks
Even though the metrics that we have aren't ideal quite yet (total
energy consumed, average power draw), we still want system health
benchmarks running ASAP in order to prevent major regressions.
We discussed having two runs of the system health benchmarks:
memory and non-memory, with memory being isolated due to its high
overhead. At least for the time being, that's not possible with power
benchmarks because we need to decide in the benchmark constructor
whether to enable BattOr tracing, even though we don't have any ability
to know whether a BattOr is attached. Because of that, we're forced to
disable the test on any devices without BattOrs in the @ShouldDisable
predicate.
BUG=637158, 589726
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq
Committed: https://crrev.com/71253911d21dfc93fdeccb6c46e518c2a078a3b9
Cr-Commit-Position: refs/heads/master@{#415561}
   
  Patch Set 1 #
      Total comments: 4
      
     
  
  Patch Set 2 : Made BattOr tracing optional, renamed test suites #Patch Set 3 : Removed "value can be added" predicate #
      Total comments: 2
      
     
  
  Patch Set 4 : Add memory to separate test suite comment #Patch Set 5 : #Patch Set 6 : Record only rail events #Patch Set 7 : Added missing "category_filter" #Patch Set 8 : #Patch Set 9 : #Messages
    Total messages: 63 (39 generated)
     
  
  
 Description was changed from ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. BUG=637158 ========== to ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. BUG=637158 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== 
 charliea@chromium.org changed reviewers: + petrcermak@chromium.org 
 
 Description was changed from ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. BUG=637158 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== to ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== 
 On 2016/08/12 01:35:49, charliea wrote: Wouldn't enabling battor tracing & battor power metrics be resilient to the case of device not having the battor? i.e: it will just output nothing instead of throwing exception? 
 petrcermak@chromium.org changed reviewers: + nednguyen@google.com 
 Looks good overall. I've left you two comments. On 2016/08/12 02:01:54, nednguyen (ooo til 8-13) wrote: > On 2016/08/12 01:35:49, charliea wrote: > > Wouldn't enabling battor tracing & battor power metrics be resilient to the case > of device not having the battor? i.e: it will just output nothing instead of > throwing exception? If this would work, then it would be great. If not, then I guess we have to either modify telemetry (so that the benchmark can determine whether it can use BattOr in its setup), or we'll have a separate power benchmark if necessary. I'll leave this up to Ned. Apart from this, LGTM :-) Thanks, Petr https://codereview.chromium.org/2239053003/diff/1/tools/perf/benchmarks/syste... File tools/perf/benchmarks/system_health.py (right): https://codereview.chromium.org/2239053003/diff/1/tools/perf/benchmarks/syste... tools/perf/benchmarks/system_health.py:59: class _PowerSystemHealthBenchmark(perf_benchmark.PerfBenchmark): Instead of interleaving the classes: _MemorySystemHealthBenchmark _PowerSystemHealthBenchmark DesktopMemorySystemHealthBenchmark ... DesktopPowerSystemHealthBenchmark ... Please keep each group together: _MemorySystemHealthBenchmark DesktopMemorySystemHealthBenchmark ... _PowerSystemHealthBenchmark DesktopPowerSystemHealthBenchmark ... https://codereview.chromium.org/2239053003/diff/1/tools/perf/benchmarks/syste... tools/perf/benchmarks/system_health.py:88: return not _IGNORED_STATS_RE.search(value.name) How many values does your metric generate? If it's only a couple, then you don't need this (we need it in memory because we are producing hundreds of values). 
 On 2016/08/12 09:03:00, petrcermak wrote: > Looks good overall. I've left you two comments. > > On 2016/08/12 02:01:54, nednguyen (ooo til 8-13) wrote: > > On 2016/08/12 01:35:49, charliea wrote: > > > > Wouldn't enabling battor tracing & battor power metrics be resilient to the > case > > of device not having the battor? i.e: it will just output nothing instead of > > throwing exception? > > If this would work, then it would be great. If not, then I guess we have to > either modify telemetry (so that the benchmark can determine whether it can use > BattOr in its setup), or we'll have a separate power benchmark if necessary. > I'll leave this up to Ned. Apart from this, LGTM :-) +1 for this working--I think it'll make having a large number of stories combined with a large number of bots on the waterfall much more manageable. 
 PTAL https://codereview.chromium.org/2239053003/diff/1/tools/perf/benchmarks/syste... File tools/perf/benchmarks/system_health.py (right): https://codereview.chromium.org/2239053003/diff/1/tools/perf/benchmarks/syste... tools/perf/benchmarks/system_health.py:59: class _PowerSystemHealthBenchmark(perf_benchmark.PerfBenchmark): On 2016/08/12 09:03:00, petrcermak wrote: > Instead of interleaving the classes: > > _MemorySystemHealthBenchmark > _PowerSystemHealthBenchmark > DesktopMemorySystemHealthBenchmark > ... > DesktopPowerSystemHealthBenchmark > ... > > Please keep each group together: > > _MemorySystemHealthBenchmark > DesktopMemorySystemHealthBenchmark > ... > _PowerSystemHealthBenchmark > DesktopPowerSystemHealthBenchmark > ... Done. https://codereview.chromium.org/2239053003/diff/1/tools/perf/benchmarks/syste... tools/perf/benchmarks/system_health.py:88: return not _IGNORED_STATS_RE.search(value.name) On 2016/08/12 09:03:00, petrcermak wrote: > How many values does your metric generate? If it's only a couple, then you don't > need this (we need it in memory because we are producing hundreds of values). Done. 
 Please update the patch description and title and add 589726 to BUG=... (so that it posts an update to the overall SH tracking bug). Apart from that, LGTM. Thanks, Petr https://codereview.chromium.org/2239053003/diff/40001/tools/perf/benchmarks/s... File tools/perf/benchmarks/system_health.py (right): https://codereview.chromium.org/2239053003/diff/40001/tools/perf/benchmarks/s... tools/perf/benchmarks/system_health.py:27: 2) Be run in a separate test suite. maybe add "(such as memory)"? 
 Description was changed from ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== to ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== 
 https://codereview.chromium.org/2239053003/diff/40001/tools/perf/benchmarks/s... File tools/perf/benchmarks/system_health.py (right): https://codereview.chromium.org/2239053003/diff/40001/tools/perf/benchmarks/s... tools/perf/benchmarks/system_health.py:27: 2) Be run in a separate test suite. On 2016/08/12 14:35:00, petrcermak wrote: > maybe add "(such as memory)"? Done. 
 The CQ bit was checked by charliea@chromium.org 
 The patchset sent to the CQ was uploaded after l-g-t-m from petrcermak@chromium.org Link to the patchset: https://codereview.chromium.org/2239053003/#ps60001 (title: "Add memory to separate test suite comment") 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Try jobs failed on following builders: mac_retina_perf_cq on master.tryserver.chromium.perf (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_...) 
 The CQ bit was checked by charliea@chromium.org to run a CQ dry run 
 Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Dry run: Try jobs failed on following builders: linux_android_rel_ng on master.tryserver.chromium.android (JOB_FAILED, https://build.chromium.org/p/tryserver.chromium.android/builders/linux_androi...) 
 The CQ bit was checked by nednguyen@google.com to run a CQ dry run 
 Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Dry run: Try jobs failed on following builders: win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...) 
 The CQ bit was checked by charliea@chromium.org 
 The patchset sent to the CQ was uploaded after l-g-t-m from petrcermak@chromium.org Link to the patchset: https://codereview.chromium.org/2239053003/#ps100001 (title: "Record only rail events") 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Try jobs failed on following builders: linux_android_rel_ng on master.tryserver.chromium.android (JOB_FAILED, https://build.chromium.org/p/tryserver.chromium.android/builders/linux_androi...) 
 The CQ bit was checked by charliea@chromium.org 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by charliea@chromium.org 
 The CQ bit was checked by charliea@chromium.org 
 The patchset sent to the CQ was uploaded after l-g-t-m from petrcermak@chromium.org Link to the patchset: https://codereview.chromium.org/2239053003/#ps120001 (title: "Added missing "category_filter"") 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by charliea@chromium.org 
 The CQ bit was checked by charliea@chromium.org 
 The patchset sent to the CQ was uploaded after l-g-t-m from petrcermak@chromium.org Link to the patchset: https://codereview.chromium.org/2239053003/#ps160001 (title: " ") 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Try jobs failed on following builders: mac_retina_perf_cq on master.tryserver.chromium.perf (JOB_TIMED_OUT, no build URL) 
 nednguyen@google.com changed reviewers: + rnephew@chromium.org 
 The CQ bit was checked by charliea@chromium.org 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Try jobs failed on following builders: android_s5_perf_cq on master.tryserver.chromium.perf (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_...) 
 Description was changed from ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== to ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== 
 The CQ bit was checked by charliea@chromium.org 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Your CL can not be processed by CQ because of: * Failed to parse additional trybots 
 Description was changed from ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== to ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== 
 The CQ bit was checked by nednguyen@google.com 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 The CQ bit was unchecked by commit-bot@chromium.org 
 Try jobs failed on following builders: mac_retina_perf_cq on master.tryserver.chromium.perf (JOB_TIMED_OUT, no build URL) 
 The CQ bit was checked by nednguyen@google.com 
 CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or... 
 
            
              
                Message was sent while issue was closed.
              
            
             
          
        Description was changed from ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== to ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== 
 
            
              
                Message was sent while issue was closed.
              
            
             
          
        Committed patchset #9 (id:160001) 
 
            
              
                Message was sent while issue was closed.
              
            
             
          
        Description was changed from ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq ========== to ========== [system health] Add power system health benchmarks Even though the metrics that we have aren't ideal quite yet (total energy consumed, average power draw), we still want system health benchmarks running ASAP in order to prevent major regressions. We discussed having two runs of the system health benchmarks: memory and non-memory, with memory being isolated due to its high overhead. At least for the time being, that's not possible with power benchmarks because we need to decide in the benchmark constructor whether to enable BattOr tracing, even though we don't have any ability to know whether a BattOr is attached. Because of that, we're forced to disable the test on any devices without BattOrs in the @ShouldDisable predicate. BUG=637158,589726 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq Committed: https://crrev.com/71253911d21dfc93fdeccb6c46e518c2a078a3b9 Cr-Commit-Position: refs/heads/master@{#415561} ========== 
 
            
              
                Message was sent while issue was closed.
              
            
             
          
        Patchset 9 (id:??) landed as https://crrev.com/71253911d21dfc93fdeccb6c46e518c2a078a3b9 Cr-Commit-Position: refs/heads/master@{#415561}  | 
    
