Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(324)

Issue 1691893002: Fuzzer now deduplicates on the analysis side instead of the download side (Closed)

Created:
4 years, 10 months ago by kjlubick
Modified:
4 years, 10 months ago
Reviewers:
jcgregorio
CC:
reviews_skia.org
Base URL:
https://skia.googlesource.com/buildbot@metrics
Target Ref:
refs/heads/master
Visibility:
Public.

Description

Fuzzer now deduplicates on the analysis side instead of the download side This will drastically decrease the stored fuzzes and reduce re-analysis time. There were some refactorings in this, e.g. putting the logic to download all fuzzes in common/storage, moving frontend/data to data and putting the logic to download data.FuzzReports in a centralized location. This also has a tiny UI bug that makes details-summary-sk notify when expanded is clicked. Finally, a pipe "|" separated list of all fuzzes is uploaded to GCS when a commit is "closed", i.e. when we move from revision 'abc' to 'def', we store the names of the bad and grey fuzzes in 'abc' to speed up and make more accurate fuzz counting. This will be built upon in a follow-on CL. BUG=skia:4438 Committed: https://skia.googlesource.com/buildbot/+/37adeea89b1ed1b4e9466a849173399b5cf20a44

Patch Set 1 #

Unified diffs Side-by-side diffs Delta from patch set Stats (+328 lines, -3288 lines) Patch
M fuzzer/go/aggregator/aggregator.go View 12 chunks +51 lines, -5 lines 0 comments Download
M fuzzer/go/backend/version_updater.go View 5 chunks +46 lines, -79 lines 0 comments Download
M fuzzer/go/common/storage.go View 2 chunks +85 lines, -1 line 0 comments Download
M fuzzer/go/config/config.go View 1 chunk +1 line, -0 lines 0 comments Download
A + fuzzer/go/data/report.go View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/report_mock.go View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/report_test.go View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/result.go View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/stacktrace.go View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/stacktrace_test.go View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/parse-asan-double.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/parse-asan-single.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/parse-catchsegv-debug.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/parse-catchsegv-edge.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/parse-catchsegv-release.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/0grey_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/0grey_debug.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/0grey_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/0grey_release.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/1bad_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/1bad_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/1grey_debug.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/1grey_release.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/2bad_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/2bad_debug.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/2bad_debug.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/2bad_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/2grey_release.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/3bad_debug.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/3bad_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/3bad_release.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/3grey_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/4bad_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/4bad_debug.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/4bad_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/4bad_release.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/5bad_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/5bad_debug.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/5bad_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/5bad_release.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/6bad_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/6bad_debug.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/6bad_debug.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/6bad_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/6bad_release.dump View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/6bad_release.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/7bad_debug.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/7bad_debug.err View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/7bad_release.asan View 0 chunks +-1 lines, --1 lines 0 comments Download
A + fuzzer/go/data/testdata/stacktrace/7bad_release.err View 0 chunks +-1 lines, --1 lines 0 comments Download
M fuzzer/go/deduplicator/deduplicator.go View 3 chunks +8 lines, -2 lines 0 comments Download
M fuzzer/go/deduplicator/deduplicator_test.go View 1 chunk +1 line, -1 line 0 comments Download
D fuzzer/go/frontend/data/report.go View 1 chunk +0 lines, -465 lines 0 comments Download
D fuzzer/go/frontend/data/report_mock.go View 1 chunk +0 lines, -97 lines 0 comments Download
D fuzzer/go/frontend/data/report_test.go View 1 chunk +0 lines, -177 lines 0 comments Download
D fuzzer/go/frontend/data/result.go View 1 chunk +0 lines, -252 lines 0 comments Download
D fuzzer/go/frontend/data/stacktrace.go View 1 chunk +0 lines, -193 lines 0 comments Download
D fuzzer/go/frontend/data/stacktrace_test.go View 1 chunk +0 lines, -436 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/parse-asan-double.asan View 1 chunk +0 lines, -64 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/parse-asan-single.asan View 1 chunk +0 lines, -52 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/parse-catchsegv-debug.dump View 1 chunk +0 lines, -166 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/parse-catchsegv-edge.dump View 1 chunk +0 lines, -160 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/parse-catchsegv-release.dump View 1 chunk +0 lines, -159 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/0grey_debug.asan View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/0grey_debug.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/0grey_release.asan View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/0grey_release.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/1bad_debug.asan View 1 chunk +0 lines, -52 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/1bad_release.asan View 1 chunk +0 lines, -16 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/1grey_debug.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/1grey_release.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/2bad_debug.asan View 1 chunk +0 lines, -19 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/2bad_debug.dump View 1 chunk +0 lines, -189 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/2bad_debug.err View 1 chunk +0 lines, -2 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/2bad_release.asan View 1 chunk +0 lines, -64 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/2grey_release.err View 1 chunk +0 lines, -3 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/3bad_debug.dump View 1 chunk +0 lines, -162 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/3bad_release.asan View 1 chunk +0 lines, -50 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/3bad_release.dump View 1 chunk +0 lines, -158 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/3grey_debug.asan View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/4bad_debug.asan View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/4bad_debug.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/4bad_release.asan View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/4bad_release.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/5bad_debug.asan View 1 chunk +0 lines, -17 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/5bad_debug.err View 1 chunk +0 lines, -3 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/5bad_release.asan View 1 chunk +0 lines, -17 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/5bad_release.err View 1 chunk +0 lines, -3 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/6bad_debug.asan View 1 chunk +0 lines, -6 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/6bad_debug.dump View 1 chunk +0 lines, -34 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/6bad_debug.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/6bad_release.asan View 1 chunk +0 lines, -6 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/6bad_release.dump View 1 chunk +0 lines, -34 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/6bad_release.err View 1 chunk +0 lines, -1 line 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/7bad_debug.asan View 1 chunk +0 lines, -2 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/7bad_debug.err View 1 chunk +0 lines, -2 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/7bad_release.asan View 1 chunk +0 lines, -2 lines 0 comments Download
D fuzzer/go/frontend/data/testdata/stacktrace/7bad_release.err View 1 chunk +0 lines, -2 lines 0 comments Download
M fuzzer/go/frontend/gsloader/gsloader.go View 8 chunks +17 lines, -157 lines 0 comments Download
M fuzzer/go/frontend/syncer/fuzz_syncer.go View 1 chunk +1 line, -1 line 0 comments Download
M fuzzer/go/fuzzcache/fuzzcache.go View 1 chunk +1 line, -1 line 0 comments Download
M fuzzer/go/fuzzcache/fuzzcache_test.go View 1 chunk +1 line, -1 line 0 comments Download
M fuzzer/go/fuzzer-be/main.go View 3 chunks +18 lines, -6 lines 0 comments Download
M fuzzer/go/fuzzer-fe/main.go View 1 chunk +1 line, -1 line 0 comments Download
M fuzzer/go/generator/afl_generation.go View 1 chunk +4 lines, -0 lines 0 comments Download
A fuzzer/go/storage/download_reports.go View 1 chunk +137 lines, -0 lines 0 comments Download
M fuzzer/res/imp/fuzzer-collapse-file-sk.html View 2 chunks +1 line, -2 lines 0 comments Download
M res/imp/9/details-summary.html View 1 chunk +1 line, -0 lines 0 comments Download

Depends on Patchset:

Dependent Patchsets:

Messages

Total messages: 10 (6 generated)
kjlubick
4 years, 10 months ago (2016-02-11 18:26:49 UTC) #5
jcgregorio
lgtm
4 years, 10 months ago (2016-02-12 14:12:25 UTC) #6
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1691893002/1 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1691893002/1
4 years, 10 months ago (2016-02-12 14:14:39 UTC) #8
commit-bot: I haz the power
4 years, 10 months ago (2016-02-12 14:17:48 UTC) #10
Message was sent while issue was closed.
Committed patchset #1 (id:1) as
https://skia.googlesource.com/buildbot/+/37adeea89b1ed1b4e9466a849173399b5cf2...

Powered by Google App Engine
This is Rietveld 408576698