Attempt to repair corrupt enterprise policy force-installed extensions

Attempt to repair corrupt enterprise policy force-installed extensions

For normally installed extensions, when the content verification system
detects corruption we just disable the extension and provide a "Repair"
button on the chrome://extensions page that users can click to manually
reinstall the extension. But for enterprise force-installed extensions
that are supposed to always remain enabled, we weren't sure how to
handle this (since we have had a few bugs over time which resulted in
false-positive corruption detection in various edge cases, and this can
happen as a result of minor disk corruption as well as malicious
changes). So we just let them keep running but recorded an UMA histogram
to measure how often this happens. Looking at the data for this, it
happens very rarely - around 0.5% of total enterprise policy extensions
seem to be affected.

In some recent conversations with Google's own internal IT folks, we
decided we'd like to attempt to automatically repair instead of just
doing nothing, so this CL implements that. It also includes new UMA
stats for measuring the number of repair attempts and successes, as well
as issuing a warning to the browser log which enterprise admins could
have automated systems watching for. Over time we'd like to implement a
separate log facility for these kinds of events that are of interest
just to enterprise admins so that they don't have to turn on the general
browser log.

BUG=447040
Committed: https://crrev.com/56282ab78b4c684e7008f777a81924d823177817
Cr-Commit-Position: refs/heads/master@{#417608}

[asargent_no_longer_on_chrome, 2016-09-01 23:53:05]

The CQ bit was checked by asargent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-01 23:53:31]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-01 23:55:45]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-01 23:55:46]

Dry run: Try jobs failed on following builders:
  mac_chromium_compile_dbg_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_comp...)
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[asargent_no_longer_on_chrome, 2016-09-02 04:20:45]

Patchset #2 (id:20001) has been deleted

[asargent_no_longer_on_chrome, 2016-09-02 04:21:01]

The CQ bit was checked by asargent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-02 04:21:22]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-02 04:33:27]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-02 04:33:29]

Dry run: Try jobs failed on following builders:
  chromeos_daisy_chromium_compile_only_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromeos_daisy_...)

[asargent_no_longer_on_chrome, 2016-09-02 05:50:26]

The CQ bit was checked by asargent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-02 05:50:35]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-02 07:09:41]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-02 07:09:43]

Dry run: This issue passed the CQ dry run.

[asargent_no_longer_on_chrome, 2016-09-02 16:25:20]

Description was changed from

==========
Attempt to repair corrupt enterprise policy force-installed extensions

BUG=447040
==========

to

==========
Attempt to repair corrupt enterprise policy force-installed extensions

For normally installed extensions, when the content verification system
detects corruption we just disable the extension and provide a "Repair"
button on the chrome://extensions page that users can click to manually
reinstall the extension. But for enterprise force-installed extensions
that are supposed to always remain enabled, we weren't sure how to
handle this (since we have had a few bugs over time which resulted in
false-positive corruption detection in various edge cases, and this can
happen as a result of minor disk corruption as well as malicious
changes). So we just let them keep running but recorded an UMA histogram
to measure how often this happens. Looking at the data for this, it
happens very rarely - around 0.5% of total enterprise policy extensions
seem to be affected.

In some recent conversations with Google's own internal IT folks, we
decided we'd like to attempt to automatically repair instead of just
doing nothing, so this CL implements that. It also includes new UMA
stats for measuring the number of repair attempts and successes, as well
as issuing a warning to the browser log which enterprise admins could
have automated systems watching for. Over time we'd like to implement a
separate log facility for these kinds of events that are of interest
just to enterprise admins so that they don't have to turn on the general
browser log.

BUG=447040
==========

[asargent_no_longer_on_chrome, 2016-09-02 16:33:09]

asargent@chromium.org changed reviewers:
+ holte@chromium.org, lazyboy@chromium.org

[asargent_no_longer_on_chrome, 2016-09-02 16:33:09]

lazyboy: please review overall

holte: please review histograms changes in
 tools/metrics/histograms/histograms.xml
 c/b/e/chrome_content_verifier_delegate.cc
 c/b/e/pending_extension_manager.cc

[Steven Holte, 2016-09-02 19:02:07]

histograms lgtm

[lazyboy, 2016-09-02 19:21:53]

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/chromeos...
File chrome/browser/chromeos/extensions/external_cache.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/chromeos...
chrome/browser/chromeos/extensions/external_cache.cc:247: bool keep_if_present =
unrelated to this CL, but generally I'd have expected this code to be in the
following form:
bool value = false;
bool keep_if_present = entry->Get(kKeepIfPresent, &value) && value;
i.e. my q is are we guaranteed to never set this value to false?

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/content_verifier_browsertest.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:314: void
AddResponse(const std::string& extension_id,
ExtensionId instead of std::string for extension ids.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:605: // Test the case
of a corrupt extension that is force-installed by policy and
nit: Tests the ...

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:607:
IN_PROC_BROWSER_TEST_F(ContentVerifierTest, PolicyCorrupted) {
Nice and thorough test!

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:654: if
(part.find("x=") == 0 &&
base::StartWith(part, "x=")

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/extension_service.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/extension_service.cc:206: // Already installed. Skip
this install if the current location has
Add comment to reflect the new checks you've added below.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/pending_extension_manager.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.cc:58: if
(base::ContainsKey(expected_policy_reinstalls_, id)) {
Does ::Remove() always mean that the issue was "resolved"? I see
ExtensionService::OnExtensionInstalled
https://cs.chromium.org/chromium/src/chrome/browser/extensions/extension_serv...
calling this on failure cases too, e.g.
PendingExtensionInfo::ShouldAllowInstall() is false.

Depending on what we consider resolved, this might or might not matter...

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.cc:105: return
expected_policy_reinstalls_.find(id) !=
nit: return base::ContainsKey(...) to be consistent with rest of the code.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/pending_extension_manager.h (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.h:72: // Notify the manager
that we are reinstalling the policy force-installed
nit: Notifies the manager...
Also I'd be more clear in the description and in the function name (which
admittedly becomes ugly) that this is only for policy extension's content
corruption.
(you can probably suggest a better name)
void ExpectPolicyExtensionReinstallDueToCorruption()

The worry is that folks don't misinterpret this as generic reinstall mechanism
while changing the code in the future...

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.h:74: void
ExpectPolicyReinstall(const std::string& id);
nit: use ExtensionId (which is a string) for new extension id types.

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
File extensions/browser/updater/extension_downloader.h (right):

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/extension_downloader.h:83: // potentially concurrent
update checks complete.
Update comment about |is_corrupt_reinstall|

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/extension_downloader.h:186: const
ManifestFetchData::ExtraParams* extra,
Prefer ManifestFetchData::ExtraParams const ref to pointer. I see tests are
passing null, but that could simply become default ctor ExtraParams()

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
File extensions/browser/updater/manifest_fetch_data.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/manifest_fetch_data.cc:113: parts.push_back("cpr");
We should update the this function's documentation.

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
File extensions/browser/updater/manifest_fetch_data.h (right):

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/manifest_fetch_data.h:80: // reinstalling due to
corruption being detected in it.
s/detected in it/detected in the extension
as this is harder to guess whether the policy itself corrupted or the extension
did.

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/manifest_fetch_data.h:81: bool
is_corrupt_policy_reinstall;
(again feel free to suggest something better)
in order to clear the confusion i mentioned above, maybe sth like:
is_policy_reinstall_due_to_corrupt_extension or something that disambiguates
which of policy or extension is corrupt.

[asargent_no_longer_on_chrome, 2016-09-08 23:34:29]

The CQ bit was checked by asargent@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-08 23:35:20]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-09 00:58:47]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-09 00:58:48]

Dry run: This issue passed the CQ dry run.

[asargent_no_longer_on_chrome, 2016-09-09 03:30:44]

All comments addressed. 

In conversation with the Omaha folks, it sounds like it works for them if we
just change the "installsource" parameter that we already normally send to a
value of "reinstall" in this case, so I've changed the CL to reflect that and it
slightly simplified things. 

lazyboy: PTAL

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/chromeos...
File chrome/browser/chromeos/extensions/external_cache.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/chromeos...
chrome/browser/chromeos/extensions/external_cache.cc:247: bool keep_if_present =
On 2016/09/02 19:21:52, lazyboy wrote:
> unrelated to this CL, but generally I'd have expected this code to be in the
> following form:
> bool value = false;
> bool keep_if_present = entry->Get(kKeepIfPresent, &value) && value;
> i.e. my q is are we guaranteed to never set this value to false?

I guess it just means that "keep_if_present" is a special key that connotes a
behavior just by its presence in the dictionary regardless of its value (of true
or false). If the dictionary doesn't have this key, then keep_if_present will be
false.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/content_verifier_browsertest.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:314: void
AddResponse(const std::string& extension_id,
On 2016/09/02 19:21:53, lazyboy wrote:
> ExtensionId instead of std::string for extension ids.

Done.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:605: // Test the case
of a corrupt extension that is force-installed by policy and
On 2016/09/02 19:21:52, lazyboy wrote:
> nit: Tests the ...

Done.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:607:
IN_PROC_BROWSER_TEST_F(ContentVerifierTest, PolicyCorrupted) {
On 2016/09/02 19:21:53, lazyboy wrote:
> Nice and thorough test!

thanks!

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/content_verifier_browsertest.cc:654: if
(part.find("x=") == 0 &&
On 2016/09/02 19:21:53, lazyboy wrote:
> base::StartWith(part, "x=")

Done.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/extension_service.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/extension_service.cc:206: // Already installed. Skip
this install if the current location has
On 2016/09/02 19:21:53, lazyboy wrote:
> Add comment to reflect the new checks you've added below.

Done.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/pending_extension_manager.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.cc:58: if
(base::ContainsKey(expected_policy_reinstalls_, id)) {
On 2016/09/02 19:21:53, lazyboy wrote:
> Does ::Remove() always mean that the issue was "resolved"? I see
> ExtensionService::OnExtensionInstalled
>
https://cs.chromium.org/chromium/src/chrome/browser/extensions/extension_serv...
> calling this on failure cases too, e.g.
> PendingExtensionInfo::ShouldAllowInstall() is false.
> 
> Depending on what we consider resolved, this might or might not matter...

Yeah, I think in that case it will be resolved, because we aren't able to
install the extension.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.cc:105: return
expected_policy_reinstalls_.find(id) !=
On 2016/09/02 19:21:53, lazyboy wrote:
> nit: return base::ContainsKey(...) to be consistent with rest of the code.

Done.

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
File chrome/browser/extensions/pending_extension_manager.h (right):

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.h:72: // Notify the manager
that we are reinstalling the policy force-installed
On 2016/09/02 19:21:53, lazyboy wrote:
> nit: Notifies the manager...
> Also I'd be more clear in the description and in the function name (which
> admittedly becomes ugly) that this is only for policy extension's content
> corruption.
> (you can probably suggest a better name)
> void ExpectPolicyExtensionReinstallDueToCorruption()
> 
> The worry is that folks don't misinterpret this as generic reinstall mechanism
> while changing the code in the future...

Done. Used "ExpectPolicyReinstallForCorruption" ("extension" is implied by the
extensions namespace and location in the codebase)

https://codereview.chromium.org/2299203004/diff/60001/chrome/browser/extensio...
chrome/browser/extensions/pending_extension_manager.h:74: void
ExpectPolicyReinstall(const std::string& id);
On 2016/09/02 19:21:53, lazyboy wrote:
> nit: use ExtensionId (which is a string) for new extension id types.

Done.

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
File extensions/browser/updater/extension_downloader.h (right):

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/extension_downloader.h:83: // potentially concurrent
update checks complete.
On 2016/09/02 19:21:53, lazyboy wrote:
> Update comment about |is_corrupt_reinstall|

Done.

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/extension_downloader.h:186: const
ManifestFetchData::ExtraParams* extra,
On 2016/09/02 19:21:53, lazyboy wrote:
> Prefer ManifestFetchData::ExtraParams const ref to pointer. I see tests are
> passing null, but that could simply become default ctor ExtraParams()

Done.

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
File extensions/browser/updater/manifest_fetch_data.cc (right):

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/manifest_fetch_data.cc:113: parts.push_back("cpr");
On 2016/09/02 19:21:53, lazyboy wrote:
> We should update the this function's documentation.

(changes in this file have been reverted, since we changed to using the
installsource which we already had a way to customize)

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
File extensions/browser/updater/manifest_fetch_data.h (right):

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/manifest_fetch_data.h:80: // reinstalling due to
corruption being detected in it.
On 2016/09/02 19:21:53, lazyboy wrote:
> s/detected in it/detected in the extension
> as this is harder to guess whether the policy itself corrupted or the
extension
> did.

Comment simplified (Note that this struct moved to extension_downloader.h)

https://codereview.chromium.org/2299203004/diff/60001/extensions/browser/upda...
extensions/browser/updater/manifest_fetch_data.h:81: bool
is_corrupt_policy_reinstall;
On 2016/09/02 19:21:53, lazyboy wrote:
> (again feel free to suggest something better)
> in order to clear the confusion i mentioned above, maybe sth like:
> is_policy_reinstall_due_to_corrupt_extension or something that disambiguates
> which of policy or extension is corrupt.

This moved to extension_downloader.h, and I changed this member's name to
"is_corrupt_reinstall" to match the variable usage there.

[lazyboy, 2016-09-09 04:49:34]

lgtm

[asargent_no_longer_on_chrome, 2016-09-09 16:52:33]

The CQ bit was checked by asargent@chromium.org

[asargent_no_longer_on_chrome, 2016-09-09 16:52:33]

The patchset sent to the CQ was uploaded after l-g-t-m from holte@chromium.org
Link to the patchset: https://codereview.chromium.org/2299203004/#ps80001
(title: "switched to using installsource, addressed review comments")

[commit-bot: I haz the power, 2016-09-09 16:53:10]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-09 16:58:21]

Description was changed from

==========
Attempt to repair corrupt enterprise policy force-installed extensions

For normally installed extensions, when the content verification system
detects corruption we just disable the extension and provide a "Repair"
button on the chrome://extensions page that users can click to manually
reinstall the extension. But for enterprise force-installed extensions
that are supposed to always remain enabled, we weren't sure how to
handle this (since we have had a few bugs over time which resulted in
false-positive corruption detection in various edge cases, and this can
happen as a result of minor disk corruption as well as malicious
changes). So we just let them keep running but recorded an UMA histogram
to measure how often this happens. Looking at the data for this, it
happens very rarely - around 0.5% of total enterprise policy extensions
seem to be affected.

In some recent conversations with Google's own internal IT folks, we
decided we'd like to attempt to automatically repair instead of just
doing nothing, so this CL implements that. It also includes new UMA
stats for measuring the number of repair attempts and successes, as well
as issuing a warning to the browser log which enterprise admins could
have automated systems watching for. Over time we'd like to implement a
separate log facility for these kinds of events that are of interest
just to enterprise admins so that they don't have to turn on the general
browser log.

BUG=447040
==========

to

==========
Attempt to repair corrupt enterprise policy force-installed extensions

For normally installed extensions, when the content verification system
detects corruption we just disable the extension and provide a "Repair"
button on the chrome://extensions page that users can click to manually
reinstall the extension. But for enterprise force-installed extensions
that are supposed to always remain enabled, we weren't sure how to
handle this (since we have had a few bugs over time which resulted in
false-positive corruption detection in various edge cases, and this can
happen as a result of minor disk corruption as well as malicious
changes). So we just let them keep running but recorded an UMA histogram
to measure how often this happens. Looking at the data for this, it
happens very rarely - around 0.5% of total enterprise policy extensions
seem to be affected.

In some recent conversations with Google's own internal IT folks, we
decided we'd like to attempt to automatically repair instead of just
doing nothing, so this CL implements that. It also includes new UMA
stats for measuring the number of repair attempts and successes, as well
as issuing a warning to the browser log which enterprise admins could
have automated systems watching for. Over time we'd like to implement a
separate log facility for these kinds of events that are of interest
just to enterprise admins so that they don't have to turn on the general
browser log.

BUG=447040
==========

[commit-bot: I haz the power, 2016-09-09 16:58:22]

Committed patchset #4 (id:80001)

[commit-bot: I haz the power, 2016-09-09 17:00:02]

Description was changed from

==========
Attempt to repair corrupt enterprise policy force-installed extensions

For normally installed extensions, when the content verification system
detects corruption we just disable the extension and provide a "Repair"
button on the chrome://extensions page that users can click to manually
reinstall the extension. But for enterprise force-installed extensions
that are supposed to always remain enabled, we weren't sure how to
handle this (since we have had a few bugs over time which resulted in
false-positive corruption detection in various edge cases, and this can
happen as a result of minor disk corruption as well as malicious
changes). So we just let them keep running but recorded an UMA histogram
to measure how often this happens. Looking at the data for this, it
happens very rarely - around 0.5% of total enterprise policy extensions
seem to be affected.

In some recent conversations with Google's own internal IT folks, we
decided we'd like to attempt to automatically repair instead of just
doing nothing, so this CL implements that. It also includes new UMA
stats for measuring the number of repair attempts and successes, as well
as issuing a warning to the browser log which enterprise admins could
have automated systems watching for. Over time we'd like to implement a
separate log facility for these kinds of events that are of interest
just to enterprise admins so that they don't have to turn on the general
browser log.

BUG=447040
==========

to

==========
Attempt to repair corrupt enterprise policy force-installed extensions

For normally installed extensions, when the content verification system
detects corruption we just disable the extension and provide a "Repair"
button on the chrome://extensions page that users can click to manually
reinstall the extension. But for enterprise force-installed extensions
that are supposed to always remain enabled, we weren't sure how to
handle this (since we have had a few bugs over time which resulted in
false-positive corruption detection in various edge cases, and this can
happen as a result of minor disk corruption as well as malicious
changes). So we just let them keep running but recorded an UMA histogram
to measure how often this happens. Looking at the data for this, it
happens very rarely - around 0.5% of total enterprise policy extensions
seem to be affected.

In some recent conversations with Google's own internal IT folks, we
decided we'd like to attempt to automatically repair instead of just
doing nothing, so this CL implements that. It also includes new UMA
stats for measuring the number of repair attempts and successes, as well
as issuing a warning to the browser log which enterprise admins could
have automated systems watching for. Over time we'd like to implement a
separate log facility for these kinds of events that are of interest
just to enterprise admins so that they don't have to turn on the general
browser log.

BUG=447040

Committed: https://crrev.com/56282ab78b4c684e7008f777a81924d823177817
Cr-Commit-Position: refs/heads/master@{#417608}
==========

[commit-bot: I haz the power, 2016-09-09 17:00:03]

Patchset 4 (id:??) landed as
https://crrev.com/56282ab78b4c684e7008f777a81924d823177817
Cr-Commit-Position: refs/heads/master@{#417608}