[sql] Allow storing mmap status in a VIEW instead of meta table.

[sql] Allow storing mmap status in a VIEW instead of meta table.

Some databases do not use the sql::MetaTable to handle versioning.
The mmap verification pass uses the [meta] table to store progress for
larger databases, and it uses detection of the [meta] table to
distinguish new/empty databases from existing databases.  For
non-[meta] databases, store the status in an MmapStatus view.

BUG=652359
Committed: https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb
Cr-Commit-Position: refs/heads/master@{#430343}

[Scott Hess - ex-Googler, 2016-10-05 19:34:13]

The CQ bit was checked by shess@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-10-05 19:34:49]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-05 20:35:17]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-05 20:35:18]

Dry run: This issue passed the CQ dry run.

[Scott Hess - ex-Googler, 2016-10-06 15:50:52]

The CQ bit was checked by shess@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-10-06 15:51:16]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-06 17:50:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-06 17:50:25]

Dry run: Try jobs failed on following builders:
  win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[Scott Hess - ex-Googler, 2016-10-21 20:45:42]

rebase to get sync fix.

[Scott Hess - ex-Googler, 2016-10-21 20:46:01]

The CQ bit was checked by shess@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-10-21 20:46:39]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-22 01:00:41]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-22 01:00:42]

Dry run: This issue passed the CQ dry run.

[Scott Hess - ex-Googler, 2016-10-28 23:16:43]

Rebase and cleanup

[Scott Hess - ex-Googler, 2016-10-28 23:16:51]

The CQ bit was checked by shess@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-10-28 23:17:11]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-28 23:44:33]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-28 23:44:34]

Dry run: Try jobs failed on following builders:
  ios-simulator on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...)

[Scott Hess - ex-Googler, 2016-10-31 18:09:32]

Debug iOS simulator results.

[Scott Hess - ex-Googler, 2016-10-31 19:24:57]

OK, its an older SQLite on iOS.

[Scott Hess - ex-Googler, 2016-10-31 19:25:01]

The CQ bit was checked by shess@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-10-31 19:25:25]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-31 22:01:10]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-31 22:01:10]

Dry run: This issue passed the CQ dry run.

[Scott Hess - ex-Googler, 2016-11-01 20:53:24]

shess@chromium.org changed reviewers:
+ dskiba@chromium.org

[Scott Hess - ex-Googler, 2016-11-01 20:53:25]

Hey, looks like it's working for everyone just now.  Let's check it in!

[DmitrySkiba, 2016-11-03 19:54:01]

LGTM!

[Scott Hess - ex-Googler, 2016-11-03 21:50:44]

shess@chromium.org changed reviewers:
+ michaeln@chromium.org, pasko@chromium.org, pavely@chromium.org,
rdevlin.cronin@chromium.org

[Scott Hess - ex-Googler, 2016-11-03 21:50:45]

OWNERS:
rdevlin.cronin@chromium.org for activity_database.cc
pasko@chromium.org for predictor_database.cc
pavely@chromium.org for directory_backing_store.cc
michaeln@chromium.org for dom_storage_database.cc

Not going TBR or a blanket owner because this _could_ change behavior of your
storage.  I don't think it will, but I'd hate to be at fault for pushing
something on you.

WRT such changes, you'll get:
 - a VIEW in the schema.  It should not bloat your schema, though.
 - You may start to see http://crbug.com/537742 crashes (*).
 - lower cache usage.
 - comparable or better performance (**).

(*) These users have broken profiles or disks or something, so it's highly
likely they were previously seeing non-crashing broken behavior of various
sorts.  The verification pass in the sql::Connection code attempts to validate
the db before turning this on, to prevent a surge.  I've had soul-searching
discussions with various people on this, and in the end this would have been the
default if SQLite had supported it at release.

(**) Probably not much better, most people imagine mmap will make things
amazing, but in reality page-sized file I/O is already basically mmap+memcpy,
and memcpy is fast.

[pavely, 2016-11-03 23:34:19]

components/sync/syncable/directory_backing_store.cc lgtm

[pasko, 2016-11-04 14:12:02]

chrome/browser/predictors lgtm

FWIW, we can switch to use the sql::MetaTable for database versioning, if
additional consistency from that is valuable. The DB is not used in production
at the moment, and the schema has changed drastically with no means of backwards
compatibility. I guess we cannot do the switch for dom_storage_database so
easily, hence consistency like this is a bit moot?

[Scott Hess - ex-Googler, 2016-11-04 18:04:56]

On 2016/11/04 14:12:02, pasko wrote:
> chrome/browser/predictors lgtm
> 
> FWIW, we can switch to use the sql::MetaTable for database versioning, if
> additional consistency from that is valuable. The DB is not used in production
> at the moment, and the schema has changed drastically with no means of
backwards
> compatibility. I guess we cannot do the switch for dom_storage_database so
> easily, hence consistency like this is a bit moot?

The VIEW needs approximately the CREATE VIEW statement length in sqlite_master,
which already exists, so should not use additional space unless the db has a lot
of tables and indices.  sql::MetaTable needs about the same amount of space in
sqlite_master (for the table schema), plus at least one page for data storage. 
So I think it's probably a better option for cases where the meta table really
isn't necessary.

That said, the only reason this data is used is because the fleet is likely to
have a lot of latent file I/O errors which right now lead to the db not working
correctly, but don't crash anything.  With mmap, some of those cases become
crashes.  If the crashes accumulated organically, they would just be noise like
any other problem of that sort (I/O errors with the executable or resource
mmaps, for instance).  If we switch 1B users from file I/O to mmap, there would
be a surge of crashes as we discovered all those broken files at once.  So the
right long-term solution is probably to start landing code to rewrite those
failing databases in hopes that will clear the I/O error condition, with the
goal of removing this data entirely.

[Devlin, 2016-11-04 19:27:36]

extensions lgtm

[michaeln, 2016-11-04 22:59:02]

r/s lgtm

[Scott Hess - ex-Googler, 2016-11-07 18:22:35]

The CQ bit was checked by shess@chromium.org

[commit-bot: I haz the power, 2016-11-07 18:22:57]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-11-07 19:26:50]

Committed patchset #6 (id:100001)

[commit-bot: I haz the power, 2016-11-07 19:35:16]

Description was changed from

==========
[sql] Allow storing mmap status in a VIEW instead of meta table.

Some databases do not use the sql::MetaTable to handle versioning.
The mmap verification pass uses the [meta] table to store progress for
larger databases, and it uses detection of the [meta] table to
distinguish new/empty databases from existing databases.  For
non-[meta] databases, store the status in an MmapStatus view.

BUG=652359
==========

to

==========
[sql] Allow storing mmap status in a VIEW instead of meta table.

Some databases do not use the sql::MetaTable to handle versioning.
The mmap verification pass uses the [meta] table to store progress for
larger databases, and it uses detection of the [meta] table to
distinguish new/empty databases from existing databases.  For
non-[meta] databases, store the status in an MmapStatus view.

BUG=652359

Committed: https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb
Cr-Commit-Position: refs/heads/master@{#430343}
==========

[commit-bot: I haz the power, 2016-11-07 19:35:17]

Patchset 6 (id:??) landed as
https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb
Cr-Commit-Position: refs/heads/master@{#430343}