Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(49)

Side by Side Diff: appengine/findit/crash/crash_pipeline.py

Issue 2432203003: [Predator] Run predator. (Closed)
Patch Set: Rebase. Created 4 years, 1 month ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « appengine/findit/crash/changelist_classifier.py ('k') | appengine/findit/crash/culprit.py » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Copyright 2016 The Chromium Authors. All rights reserved. 1 # Copyright 2016 The Chromium Authors. All rights reserved.
2 # Use of this source code is governed by a BSD-style license that can be 2 # Use of this source code is governed by a BSD-style license that can be
3 # found in the LICENSE file. 3 # found in the LICENSE file.
4 4
5 import copy 5 import copy
6 import json 6 import json
7 import logging 7 import logging
8 8
9 from common import appengine_util 9 from common import appengine_util
10 from common import pubsub_util 10 from common import pubsub_util
11 from common.http_client_appengine import HttpClientAppengine 11 from common.http_client_appengine import HttpClientAppengine
12 from common.pipeline_wrapper import BasePipeline 12 from common.pipeline_wrapper import BasePipeline
13 from common.pipeline_wrapper import pipeline 13 from common.pipeline_wrapper import pipeline
14 from crash import findit_for_chromecrash 14 from crash import findit_for_chromecrash
15 from crash import findit_for_clusterfuzz 15 from crash import findit_for_clusterfuzz
16 from crash.type_enums import CrashClient 16 from crash.type_enums import CrashClient
17 from lib import time_util 17 from lib import time_util
18 from lib.gitiles import gitiles_repository 18 from lib.gitiles import gitiles_repository
19 from model import analysis_status 19 from model import analysis_status
20 20
21 21
22 # TODO(http://crbug.com/659346): this needs complete coverage tests. 22 # TODO(http://crbug.com/659346): write complete coverage tests for this.
23 def FinditForClientID(client_id): 23 def FinditForClientID(client_id, repository): # pragma: no cover
24 """Construct a Findit object from a client id string specifying the class. 24 """Construct a Findit object from a client id string specifying the class.
25 25
26 We cannot pass Findit objects to the various methods in 26 We cannot pass Findit objects to the various methods in
27 ``crash.crash_pipeline``, because they are not JSON serializable. For 27 ``crash.crash_pipeline``, because they are not JSON serializable. For
28 now, we just serialize Findit objects as their ``client_id``, and then 28 now, we just serialize Findit objects as their ``client_id``, and then
29 use this function to reconstruct them. Alas, this means we will lose 29 use this function to reconstruct them. Alas, this means we will lose
30 various other information stored in the Findit object (i.e., stuff that 30 various other information stored in the Findit object (i.e., stuff that
31 comes from CrashConfig); which could lead to some hard-to-diagnose 31 comes from CrashConfig); which could lead to some hard-to-diagnose
32 coherency bugs, since the new Findit object will be based on the 32 coherency bugs, since the new Findit object will be based on the
33 CrashConfig at the time it's constructed, which may be different 33 CrashConfig at the time it's constructed, which may be different
34 than the CrashConfig at the time the previous Findit object was 34 than the CrashConfig at the time the previous Findit object was
35 constructed. In the future we should fix all this to serialize Findit 35 constructed. In the future we should fix all this to serialize Findit
36 objects in a more robust way. 36 objects in a more robust way.
37 """ 37 """
38 assert isinstance(client_id, (str, unicode)), ( 38 assert isinstance(client_id, (str, unicode)), (
39 'FinditForClientID: expected string or unicode, but got %s' 39 'FinditForClientID: expected string or unicode, but got %s'
40 % client_id.__class__.__name__) 40 % client_id.__class__.__name__)
41 # TODO(wrengr): it'd be nice to replace this with a single lookup in 41 # TODO(wrengr): it'd be nice to replace this with a single lookup in
42 # a dict; but that's buggy for some unknown reason. 42 # a dict; but that's buggy for some unknown reason.
43 if client_id == CrashClient.FRACAS: 43 if client_id == CrashClient.FRACAS:
44 cls = findit_for_chromecrash.FinditForFracas 44 cls = findit_for_chromecrash.FinditForFracas
45 elif client_id == CrashClient.CRACAS: # pragma: no cover 45 elif client_id == CrashClient.CRACAS: # pragma: no cover
46 cls = findit_for_chromecrash.FinditForCracas 46 cls = findit_for_chromecrash.FinditForCracas
47 elif client_id == CrashClient.CLUSTERFUZZ: # pragma: no cover 47 elif client_id == CrashClient.CLUSTERFUZZ: # pragma: no cover
48 cls = findit_for_clusterfuzz.FinditForClusterfuzz 48 cls = findit_for_clusterfuzz.FinditForClusterfuzz
49 else: # pragma: no cover 49 else: # pragma: no cover
50 raise ValueError('FinditForClientID: ' 50 raise ValueError('FinditForClientID: '
51 'unknown or unsupported client %s' % client_id) 51 'unknown or unsupported client %s' % client_id)
52 52
53 return cls(gitiles_repository.GitilesRepository( 53 return cls(repository)
54 http_client=HttpClientAppengine()))
55 54
56 55
57 # Some notes about the classes below, for people who are not familiar 56 # Some notes about the classes below, for people who are not familiar
58 # with AppEngine pipelines: 57 # with AppEngine pipelines:
59 # 58 #
60 # The pipeline library is designed in a strange way which requires that 59 # The pipeline library is designed in a strange way which requires that
61 # all the ``__init__`` and ``run`` methods in this file take the exact 60 # all the ``__init__`` and ``run`` methods in this file take the exact
62 # same arguments. This arises from the interaction between a few 61 # same arguments. This arises from the interaction between a few
63 # different constraints. First, for any given pipeline, its ``__init__`` 62 # different constraints. First, for any given pipeline, its ``__init__``
64 # and ``run`` must take the same arguments. Second, all the objects that 63 # and ``run`` must take the same arguments. Second, all the objects that
(...skipping 18 matching lines...) Expand all
83 # 82 #
84 # Moreover, the ``run`` and ``finalized`` methods are executed in separate 83 # Moreover, the ``run`` and ``finalized`` methods are executed in separate
85 # processes, so we'll actually end up reconstructing the ``Findit`` object 84 # processes, so we'll actually end up reconstructing the ``Findit`` object
86 # twice. This also means ``run`` can't store anything in the pipeline 85 # twice. This also means ``run`` can't store anything in the pipeline
87 # object and expect it to still be available in the ``finalized`` method. 86 # object and expect it to still be available in the ``finalized`` method.
88 87
89 class CrashBasePipeline(BasePipeline): 88 class CrashBasePipeline(BasePipeline):
90 def __init__(self, client_id, crash_identifiers): 89 def __init__(self, client_id, crash_identifiers):
91 super(CrashBasePipeline, self).__init__(client_id, crash_identifiers) 90 super(CrashBasePipeline, self).__init__(client_id, crash_identifiers)
92 self._crash_identifiers = crash_identifiers 91 self._crash_identifiers = crash_identifiers
93 self._findit = FinditForClientID(client_id) 92 self._findit = FinditForClientID(
93 client_id,
94 gitiles_repository.GitilesRepository(http_client=HttpClientAppengine()))
94 95
95 @property 96 @property
96 def client_id(self): # pragma: no cover 97 def client_id(self): # pragma: no cover
97 return self._findit.client_id 98 return self._findit.client_id
98 99
99 def run(self, *args, **kwargs): 100 def run(self, *args, **kwargs):
100 raise NotImplementedError() 101 raise NotImplementedError()
101 102
102 103
103 class CrashAnalysisPipeline(CrashBasePipeline): 104 class CrashAnalysisPipeline(CrashBasePipeline):
(...skipping 110 matching lines...) Expand 10 before | Expand all | Expand 10 after
214 N.B., due to the structure of AppEngine pipelines, this method must 215 N.B., due to the structure of AppEngine pipelines, this method must
215 accept the same arguments as are passed to ``__init__``; however, 216 accept the same arguments as are passed to ``__init__``; however,
216 because they were already passed to ``__init__`` there's no use in 217 because they were already passed to ``__init__`` there's no use in
217 recieving them here. Thus, we discard all the arguments to this method 218 recieving them here. Thus, we discard all the arguments to this method
218 (except for ``self``, naturally). 219 (except for ``self``, naturally).
219 """ 220 """
220 run_analysis = yield CrashAnalysisPipeline( 221 run_analysis = yield CrashAnalysisPipeline(
221 self._client_id, self._crash_identifiers) 222 self._client_id, self._crash_identifiers)
222 with pipeline.After(run_analysis): 223 with pipeline.After(run_analysis):
223 yield PublishResultPipeline(self._client_id, self._crash_identifiers) 224 yield PublishResultPipeline(self._client_id, self._crash_identifiers)
OLDNEW
« no previous file with comments | « appengine/findit/crash/changelist_classifier.py ('k') | appengine/findit/crash/culprit.py » ('j') | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698