CQ: don't preserve state between runs

CQ: don't preserve state between runs

This is expected to help with the issue that causes CQ
to commit changes without verification.

BUG=239272

[Isaac (away), 2013-08-21 23:47:41]

Do you have evidence that the commits are happening because of crashes?  What
were the tests that you used for this?

[Paweł Hajdan Jr., 2013-08-21 23:52:28]

On 2013/08/21 23:47:41, Isaac wrote:
> Do you have evidence that the commits are happening because of crashes?

I talked to Robbie, and according to his explanation there are cases where CQ
can put info into saved state that will make it commit without verification.

In other words, it's not fail-safe (in case of errors, assume verification
failed or has yet to be done) but seems to work other way around, i.e. if the
verification info failed to be saved, it's assumed to pass. This sounds
plausible to me. I'd treat this CL as an experiment to test that explanation,
and if it fails to fix the problem it can be reverted and we'll learn something
more.

> What were the tests that you used for this?

Presubmit. Please let me know if I should run something more.

[Isaac (away), 2013-08-21 23:52:52]

+cc some other folks on this CL.

Though I think the direction is probably not correct at this point.

[Isaac (away), 2013-08-21 23:55:28]

On 2013/08/21 23:52:28, Paweł Hajdan Jr. wrote:
> On 2013/08/21 23:47:41, Isaac wrote:
> > Do you have evidence that the commits are happening because of crashes?
> 
> I talked to Robbie, and according to his explanation there are cases where CQ
> can put info into saved state that will make it commit without verification.
> 
> In other words, it's not fail-safe (in case of errors, assume verification
> failed or has yet to be done) but seems to work other way around, i.e. if the
> verification info failed to be saved, it's assumed to pass. This sounds
> plausible to me. I'd treat this CL as an experiment to test that explanation,
> and if it fails to fix the problem it can be reverted and we'll learn
something
> more.

We can't "experiment" with the production system.  You could run locally in
read-only
mode and see what happens though.  But one issue here is that if CQ crashes,
which it does, it will delete all state, and have to redo presubmit on pending
issues
which will essentially mean it is non-responsive for an hour.  So if it crashes
more than once an hour, which sometimes it does, it will stop working.
> 
> > What were the tests that you used for this?
> 
> Presubmit. Please let me know if I should run something more.

That is definitely not enough.