Update unit info file for perf dashboard.

tools/perf/unit-info.json

 {
   "_description" : "This file contains info about our performance test units, used by the perf dashboard (http://chromeperf.appspot.com).",
 
   "%": {
     "improvement_direction": "down",
-    "why": "Percent CPU usage"
+    "why": "Percent CPU usage. Used by cpu metric."
   },
   "bytes": {
     "improvement_direction": "down"
   },
   "commit_count": {
     "improvement_direction": "up",
     "why": "layer_tree_host_perftest"
   },
   "count": {
     "improvement_direction": "down",
     "why": "Processes"
   },
   "files": {
     "improvement_direction": "down",
     "why": "Static initializers"
   },
   "fps": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "The faster the better. Used by scirra benchmark."
   },
   "frames": {
     "improvement_direction": "down",
-    "why": "Dropped frames"
+    "why": "Dropped frames."
   },
   "frames-per-second": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "Synonym for fps."
   },
   "frame_count": {
     "improvement_direction": "up",
     "why": "layer_tree_host_perftest"
   },
   "frame_time": {
     "improvement_direction": "down"
   },
   "kb": {
-    "improvement_direction": "down"
+    "improvement_direction": "down",
+    "why": "Synonym for KB, used in memory and io metrics."
   },
   "KB": {
-    "improvement_direction": "down"
+    "improvement_direction": "down",
+    "why": "KB of memory usage. Less memory usage is better. Used in endure."
   },
   "lines": {
     "improvement_direction": "up",
-    "why": "Coverage"
+    "why": "Coverage. More test coverage is better."
   },
   "load": {
     "improvement_direction": "down"
   },
   "MB": {
     "improvement_direction": "down"
   },
   "ms": {
-    "improvement_direction": "down"
+    "improvement_direction": "down",
+    "why": "Used in many Telemetry measurements. Fewer ms of time means faster."
   },
   "milliseconds": {
-    "improvement_direction": "down"
+    "improvement_direction": "down",
+    "why": "Synonym of ms."
   },
   "milliseconds-per-frame": {
     "improvement_direction": "down"
   },
   "objects (bigger is better)": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "Used in spaceport benchmark."
   },
   "ObjectsAt30FPS": {
     "improvement_direction": "up"
   },
   "percent": {
-    "improvement_direction": "down"
+    "improvement_direction": "down",
+    "why": "Synonym for %, used in memory metric for percent fragmentation."
+  },
+  "percentage": {

[tonyg, 2014/01/09 03:47:43]

Wondering if alloy should use a canonical unit instead of adding.

[victorhsieh, 2014/01/09 04:42:11]

No objection to use canonical unit, but in our case, it's a "up" (the higher the
better).

Please let me know how should we handle the transition from our code, if the
decision is to change this.  Thanks.

[qyearsley, 2014/01/09 18:07:38]

Alright, since the improvement direction is different I think it's fine to call
it "percentage" and have a comment that higher is better.

Or, Victor, if you think it makes more sense and you want to change your code to
send the units as "% (bigger is better)" and we could add that here instead.

[victorhsieh, 2014/01/09 18:41:56]

Or "coverage %", whichever is more reusable.

I actually don't know the unit does mean something.  I though it's just a label
until this code review.  So just curious, how is the direction effect the
dashboard?

Also, if we change the unit, would it affect anything, since each data point has
it own unit before?

[qyearsley, 2014/01/09 19:09:29]

"coverage %" makes sense to me.

In the datastore model used by the perf dashboard, the unit information is kept
with the test, not with each data point, even though units are generally
uploaded with each data point. If the dashboard starts receiving points for a
test with a new units string, then it will update the units for that test, which
will retroactively affect all points for that test.

Now, when the dashboard back-end detects an upward or downward jump in the
values for some test, whether this jump is marked as an improvement or a
regression is determined by the units.

The dashboard reads this file (unit-info.json), and it determines the
improvement direction for the test from the units and the information here. So,
for example, for all tests that have the units "%", downward movement is
considered an improvement, because it says so in this file.

(Note: This way of doing things works, but it has a couple small  disadvantages:
1. this file has to be kept updated, and 2. different tests with the same unit
(e.g. "score", "%", "count") might have different improvement direction. In the
future, we might want to consider changing all tests to report improvement
direction in addition to units.)

[victorhsieh, 2014/01/09 19:16:48]

Thanks for the clear explanation.  I'll change the unit from our side once this
is landed.

BTW, how about "coverage%" (remove the white space), just to avoid string
parsing trouble.

On 2014/01/09 19:09:29, qyearsley wrote:

[qyearsley, 2014/01/09 19:30:58]

Sure, no problem. "%coverage" would also be fine.

I'm pretty sure that the space character wouldn't cause an issue anywhere on the
dashboard (units can be any string), but if it might cause an issue somewhere
else, it would be fine to not have it.

By the way, what is cts? And what is "alloy-perf-test/cts%/passed"?

This is a good time to mention: There's also a "test-info.json" and
"trace-info.json" file in the same directory as this file, which contain helpful
little descriptions of test suites and individual trace lines on the graph.
(These work similarly to unit-json and have the same disadvantage -- the
dashboard reads these files and updates test information based on them.)

Right now there are no descriptions for "alloy-perf-test" and "passed" in these
files... could you describe what these things are? I actually don't know
anything about Alloy, and I don't know where the test source code is located
either...

[victorhsieh, 2014/01/09 19:44:42]

Unfortunately I can't mention the detail publicly.  If you need the description
for reference, would you please just mention that it's an internal experimental
project, and maybe put my ldap as contact?

On 2014/01/09 19:30:58, qyearsley wrote:

[qyearsley, 2014/01/09 19:47:13]

Ah, if it's internal that would explain why I couldn't find it in the main
chromium source tree. No problem.

On 2014/01/09 19:44:42, victorhsieh wrote:

+    "improvement_direction": "up",
+    "why": "Used in alloy-perf-test/cts%/passed, looks like higher->better?"
   },
   "points": {
     "improvement_direction": "up",
-    "why": "Synonym for score, used in ChromeOS touchpad tests"
+    "why": "Synonym for score, used in ChromeOS touchpad tests."
   },
   "ports": {
     "improvement_direction": "down"
   },
   "relocs": {
     "improvement_direction": "down"
   },
   "runs/ms": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "Used in dromaeo. Higher runs/ms implies faster execution."

[tonyg, 2014/01/09 03:47:43]

Not sure this one is used in dromaeo. Is it?

[qyearsley, 2014/01/09 18:07:38]

Ah, that's right. It's not used in dromaeo. rileya added this about 2 weeks ago
for "vector math" microbenchmarks. Will change comment.

   },
   "runs/s": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "Used in dromaeo. Higher runs/s implies faster execution."
+  },
+  "runs_per_s": {

[tonyg, 2014/01/09 03:47:43]

Any chance of getting the cros bots to use a canonical unit or should we just
bite the bullet and add this?

[qyearsley, 2014/01/09 18:07:38]

I'm asking quickly why the cros bots use this unit. If it's simple/fast to
change the units they send, then we shall just do that and not add this.

+    "improvement_direction": "up",
+    "why": "Synonym for runs/s, used in dromaeo data sent by cros bots."
   },
   "runs_per_second": {
     "improvement_direction": "up"
   },
   "score": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "Used in a variety of benchmarks where a higher score is better."
   },
   "score_(bigger_is_better)": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "Synonym for score."
   },
   "score (bigger is better)": {
-    "improvement_direction": "up"
+    "improvement_direction": "up",
+    "why": "Synonym for score, used in jsgamebench and dom_perf."
   },
   "sec": {
     "improvement_direction": "down"
   },
   "seconds": {
     "improvement_direction": "down"
   },
   "us": {
     "improvement_direction": "down"
   }
 }
-