[Findit] Fix 1500 byte cap on failed_steps_and_tests and output_nodes

[Findit] Fix 1500 byte cap on failed_steps_and_tests and output_nodes

Add created_time support for wf_failure_groups
Disable indexing on failed_steps_and_tests and output_nodes

BUG=633837
Committed: https://chromium.googlesource.com/infra/infra/+/d6897e11508a50d02e7da183a61445c8654ce38a

[josiahk, 2016-08-09 22:45:37]

Description was changed from

==========
[Findit] Fix 1500 byte cap on failed_steps_and_tests and output_nodes

Add created_time support for wf_failure_groups
Disable indexing on failed_steps_and_tests and output_nodes

BUG=633837
==========

to

==========
[Findit] Fix 1500 byte cap on failed_steps_and_tests and output_nodes

Add created_time support for wf_failure_groups
Disable indexing on failed_steps_and_tests and output_nodes

BUG=633837
==========

[josiahk, 2016-08-09 22:45:37]

josiahk@google.com changed reviewers:
+ chanli@chromium.org, lijeffrey@chromium.org, stgao@chromium.org

[josiahk, 2016-08-09 22:50:40]

Hello Jeff and Chan!

This patch addresses the 1500 byte limit mentioned in crbug.com/633837.

This patch also introduces created_time support for wf_failure_group and
_GetMatchingFailureGroups, which should help limit the number of query results
that come back.

For instance, instead of querying ndb for ALL compile wf_failure_groups, we can
query ndb for only the past 24 hours' compile wf_failure_groups.

Thanks!
Josiah

[lijeffrey, 2016-08-09 23:00:01]

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/model/wf_f...
File appengine/findit/model/wf_failure_group.py (right):

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/model/wf_f...
appengine/findit/model/wf_failure_group.py:67: 
nit: remove empty line

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/waterfall/...
File appengine/findit/waterfall/try_job_util.py (right):

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/waterfall/...
appengine/findit/waterfall/try_job_util.py:24: MATCHING_GROUPS_HOURS_AGO = 24
make this seconds, but as

MATCHING_GROUPS_SECONDS_AGO = 24 * 60 * 60  # 24 hours. This gives us finer
grain control over how long ago a build failure can happen to be considered as
part of a group.

also put a TODO before it that this should be moved to config. Since your
internship is up this week put the todo under me i.e. TOOD(lijeffrey): Move this
to config.

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/waterfall/...
appengine/findit/waterfall/try_job_util.py:204: new_group.created_time =
datetime.utcnow()
nit: you should move created time to the first call before updating
build_failure_type

[josiahk, 2016-08-09 23:10:36]

Code review changes.

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/model/wf_f...
File appengine/findit/model/wf_failure_group.py (right):

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/model/wf_f...
appengine/findit/model/wf_failure_group.py:67: 
On 2016/08/09 23:00:00, lijeffrey wrote:
> nit: remove empty line

Done.

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/waterfall/...
File appengine/findit/waterfall/try_job_util.py (right):

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/waterfall/...
appengine/findit/waterfall/try_job_util.py:24: MATCHING_GROUPS_HOURS_AGO = 24
On 2016/08/09 23:00:00, lijeffrey wrote:
> make this seconds, but as
> 
> MATCHING_GROUPS_SECONDS_AGO = 24 * 60 * 60  # 24 hours. This gives us finer
> grain control over how long ago a build failure can happen to be considered as
> part of a group.
> 
> also put a TODO before it that this should be moved to config. Since your
> internship is up this week put the todo under me i.e. TOOD(lijeffrey): Move
this
> to config.

Done! Thanks!

https://codereview.chromium.org/2227223002/diff/1/appengine/findit/waterfall/...
appengine/findit/waterfall/try_job_util.py:204: new_group.created_time =
datetime.utcnow()
On 2016/08/09 23:00:00, lijeffrey wrote:
> nit: you should move created time to the first call before updating
> build_failure_type

Done.

[lijeffrey, 2016-08-10 00:09:54]

lgtm. Feel free to commit once the nit is fixed.

https://codereview.chromium.org/2227223002/diff/20001/appengine/findit/waterf...
File appengine/findit/waterfall/try_job_util.py (right):

https://codereview.chromium.org/2227223002/diff/20001/appengine/findit/waterf...
appengine/findit/waterfall/try_job_util.py:24: MATCHING_GROUPS_SECONDS_AGO = 24
* 60 * 60  # 24 hours
nit: comment ends with .

[josiahk, 2016-08-10 00:18:15]

The CQ bit was checked by josiahk@google.com

[josiahk, 2016-08-10 00:18:15]

The patchset sent to the CQ was uploaded after l-g-t-m from
lijeffrey@chromium.org
Link to the patchset: https://codereview.chromium.org/2227223002/#ps40001
(title: "Added .")

[commit-bot: I haz the power, 2016-08-10 00:18:17]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[josiahk, 2016-08-10 00:28:16]

The CQ bit was unchecked by josiahk@google.com

[josiahk, 2016-08-10 00:38:31]

The CQ bit was checked by josiahk@google.com

[commit-bot: I haz the power, 2016-08-10 00:38:37]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[josiahk, 2016-08-10 00:38:37]

The CQ bit was unchecked by josiahk@google.com

[josiahk, 2016-08-10 00:47:21]

The CQ bit was checked by josiahk@google.com

[commit-bot: I haz the power, 2016-08-10 00:47:27]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-10 00:50:16]

Description was changed from

==========
[Findit] Fix 1500 byte cap on failed_steps_and_tests and output_nodes

Add created_time support for wf_failure_groups
Disable indexing on failed_steps_and_tests and output_nodes

BUG=633837
==========

to

==========
[Findit] Fix 1500 byte cap on failed_steps_and_tests and output_nodes

Add created_time support for wf_failure_groups
Disable indexing on failed_steps_and_tests and output_nodes

BUG=633837

Committed:
https://chromium.googlesource.com/infra/infra/+/d6897e11508a50d02e7da183a6144...
==========

[commit-bot: I haz the power, 2016-08-10 00:50:17]

Committed patchset #3 (id:40001) as
https://chromium.googlesource.com/infra/infra/+/d6897e11508a50d02e7da183a6144...