Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(30)

Issue 1349863003: [sql] Use memory-mapped I/O for sql::Connection. (Closed)

Created:
5 years, 3 months ago by Scott Hess - ex-Googler
Modified:
5 years, 3 months ago
CC:
chromium-reviews, michaeln
Base URL:
https://chromium.googlesource.com/chromium/src.git@master
Target Ref:
refs/pending/heads/master
Project:
chromium
Visibility:
Public.

Description

[sql] Use memory-mapped I/O for sql::Connection. sql::Connection::Open*() uses PRAGMA mmap_size to enable SQLite's memory-mapped I/O. Additionally instrument to flush dirty pages from the page cache after writes. BUG=489784, 533682 Committed: https://crrev.com/9a1948a4d6d445d5c8e209bdcd1cd050af72060b Cr-Commit-Position: refs/heads/master@{#350362}

Patch Set 1 #

Total comments: 1

Patch Set 2 : style nit, rebase #

Total comments: 15

Patch Set 3 : Wordsmithing, mmap_size smaller, no mmap on iOS, mmap test vs flush, fix sync test failure. #

Total comments: 3

Patch Set 4 : rebase #

Patch Set 5 : name changes, disable mmap for sync. #

Patch Set 6 : overreverted sync fix, needed to keep iOS fix #

Total comments: 2

Patch Set 7 : Rebase #

Patch Set 8 : Speling chaneg. #

Unified diffs Side-by-side diffs Delta from patch set Stats (+211 lines, -0 lines) Patch
M sql/connection.h View 1 2 3 4 3 chunks +20 lines, -0 lines 0 comments Download
M sql/connection.cc View 1 2 3 4 5 chunks +91 lines, -0 lines 0 comments Download
M sql/connection_unittest.cc View 1 2 3 5 2 chunks +78 lines, -0 lines 0 comments Download
M sql/statement.cc View 1 chunk +5 lines, -0 lines 0 comments Download
M sync/syncable/directory_backing_store.cc View 1 2 3 4 5 6 7 1 chunk +4 lines, -0 lines 0 comments Download
M third_party/sqlite/BUILD.gn View 1 chunk +7 lines, -0 lines 0 comments Download
M third_party/sqlite/sqlite.gyp View 1 chunk +6 lines, -0 lines 0 comments Download

Messages

Total messages: 62 (20 generated)
Scott Hess - ex-Googler
I've been procrastinating on this, but I finally sat down and groveled through my myriad ...
5 years, 3 months ago (2015-09-16 22:46:16 UTC) #2
commit-bot: I haz the power
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/1 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/1
5 years, 3 months ago (2015-09-16 23:04:16 UTC) #4
commit-bot: I haz the power
Dry run: Try jobs failed on following builders: ios_dbg_simulator_ninja on tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios_dbg_simulator_ninja/builds/70268) ios_rel_device_ninja on ...
5 years, 3 months ago (2015-09-16 23:08:48 UTC) #6
commit-bot: I haz the power
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/20001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/20001
5 years, 3 months ago (2015-09-16 23:15:08 UTC) #8
commit-bot: I haz the power
Dry run: Try jobs failed on following builders: linux_chromium_chromeos_ozone_rel_ng on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_ozone_rel_ng/builds/63958)
5 years, 3 months ago (2015-09-16 23:48:19 UTC) #10
Scott Hess - ex-Googler
On 2015/09/16 23:48:19, commit-bot: I haz the power wrote: > Dry run: Try jobs failed ...
5 years, 3 months ago (2015-09-17 00:36:13 UTC) #11
rmcilroy
Generally lgtm assuming the tests can be fixed and we are sure the f.Write operations ...
5 years, 3 months ago (2015-09-17 17:17:44 UTC) #12
Scott Hess - ex-Googler
Wordsmithing, mmap_size smaller, no mmap on iOS, mmap test vs flush, fix sync test failure.
5 years, 3 months ago (2015-09-17 21:10:56 UTC) #13
Scott Hess - ex-Googler
OK, made some changes based on comments. https://codereview.chromium.org/1349863003/diff/20001/sql/connection.cc File sql/connection.cc (right): https://codereview.chromium.org/1349863003/diff/20001/sql/connection.cc#newcode511 sql/connection.cc:511: void Connection::ReleaseCacheMemoryIfNeeded(bool ...
5 years, 3 months ago (2015-09-17 21:14:37 UTC) #15
commit-bot: I haz the power
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/40001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/40001
5 years, 3 months ago (2015-09-17 21:15:51 UTC) #16
Scott Hess - ex-Googler
+pvalenzuela@ for the directory_backing_store_corruption_testing.cc change. https://codereview.chromium.org/1349863003/diff/40001/sync/test/directory_backing_store_corruption_testing.cc File sync/test/directory_backing_store_corruption_testing.cc (right): https://codereview.chromium.org/1349863003/diff/40001/sync/test/directory_backing_store_corruption_testing.cc#newcode28 sync/test/directory_backing_store_corruption_testing.cc:28: base::ScopedFILE db_file(base::OpenFile(backing_file_path, "rb+")); "wb" ...
5 years, 3 months ago (2015-09-17 21:18:56 UTC) #18
pval...(no longer on Chromium)
lgtm https://codereview.chromium.org/1349863003/diff/40001/sync/test/directory_backing_store_corruption_testing.cc File sync/test/directory_backing_store_corruption_testing.cc (right): https://codereview.chromium.org/1349863003/diff/40001/sync/test/directory_backing_store_corruption_testing.cc#newcode28 sync/test/directory_backing_store_corruption_testing.cc:28: base::ScopedFILE db_file(base::OpenFile(backing_file_path, "rb+")); On 2015/09/17 21:18:56, Scott Hess ...
5 years, 3 months ago (2015-09-17 21:41:05 UTC) #19
commit-bot: I haz the power
Dry run: Try jobs failed on following builders: ios_dbg_simulator_ninja on tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios_dbg_simulator_ninja/builds/70781)
5 years, 3 months ago (2015-09-17 22:26:36 UTC) #21
Scott Hess - ex-Googler
On 2015/09/17 22:26:36, commit-bot: I haz the power wrote: > Dry run: Try jobs failed ...
5 years, 3 months ago (2015-09-17 22:35:40 UTC) #22
Scott Hess - ex-Googler
https://codereview.chromium.org/1349863003/diff/40001/sync/test/directory_backing_store_corruption_testing.cc File sync/test/directory_backing_store_corruption_testing.cc (right): https://codereview.chromium.org/1349863003/diff/40001/sync/test/directory_backing_store_corruption_testing.cc#newcode28 sync/test/directory_backing_store_corruption_testing.cc:28: base::ScopedFILE db_file(base::OpenFile(backing_file_path, "rb+")); On 2015/09/17 21:41:05, pvalenzuela wrote: > ...
5 years, 3 months ago (2015-09-17 23:07:27 UTC) #23
rmcilroy
Still LGTM once you've fixed the iOS issues. https://codereview.chromium.org/1349863003/diff/20001/sql/connection.cc File sql/connection.cc (right): https://codereview.chromium.org/1349863003/diff/20001/sql/connection.cc#newcode511 sql/connection.cc:511: void ...
5 years, 3 months ago (2015-09-18 11:16:29 UTC) #24
Scott Hess - ex-Googler
I'm probably going to break off the sync testing change into a separate CL, as ...
5 years, 3 months ago (2015-09-18 15:43:41 UTC) #25
rmcilroy
https://codereview.chromium.org/1349863003/diff/20001/sql/connection.cc File sql/connection.cc (right): https://codereview.chromium.org/1349863003/diff/20001/sql/connection.cc#newcode511 sql/connection.cc:511: void Connection::ReleaseCacheMemoryIfNeeded(bool assume_changed) { On 2015/09/18 15:43:41, Scott Hess ...
5 years, 3 months ago (2015-09-18 16:06:32 UTC) #26
Scott Hess - ex-Googler
rebase
5 years, 3 months ago (2015-09-18 22:44:54 UTC) #27
Scott Hess - ex-Googler
name changes, disable mmap for sync.
5 years, 3 months ago (2015-09-18 23:19:08 UTC) #28
commit-bot: I haz the power
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/80001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/80001
5 years, 3 months ago (2015-09-18 23:20:00 UTC) #30
commit-bot: I haz the power
Dry run: Try jobs failed on following builders: ios_dbg_simulator_ninja on tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios_dbg_simulator_ninja/builds/71440)
5 years, 3 months ago (2015-09-19 00:13:53 UTC) #32
Scott Hess - ex-Googler
overreverted sync fix, needed to keep iOS fix
5 years, 3 months ago (2015-09-19 00:23:47 UTC) #33
commit-bot: I haz the power
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/100001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/100001
5 years, 3 months ago (2015-09-19 00:24:41 UTC) #35
commit-bot: I haz the power
Dry run: This issue passed the CQ dry run.
5 years, 3 months ago (2015-09-19 01:41:43 UTC) #37
Scott Hess - ex-Googler
OK, I found that the sync issue is going to be a bit beyond this ...
5 years, 3 months ago (2015-09-21 18:08:57 UTC) #38
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/100001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/100001
5 years, 3 months ago (2015-09-21 18:10:36 UTC) #41
commit-bot: I haz the power
Try jobs failed on following builders: chromium_presubmit on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presubmit/builds/102106)
5 years, 3 months ago (2015-09-21 18:25:27 UTC) #43
Scott Hess - ex-Googler
Nice, Patrick wasn't owner of the new part modified :-). pavely@ as owner for the ...
5 years, 3 months ago (2015-09-21 18:32:30 UTC) #45
pval...(no longer on Chromium)
ah, yeah. I am only owner in the test directory. Let me know if you ...
5 years, 3 months ago (2015-09-22 16:57:33 UTC) #46
Scott Hess - ex-Googler
Rebase
5 years, 3 months ago (2015-09-22 19:55:36 UTC) #47
Scott Hess - ex-Googler
Speling chaneg.
5 years, 3 months ago (2015-09-22 19:56:56 UTC) #48
Scott Hess - ex-Googler
On 2015/09/22 16:57:33, pvalenzuela wrote: > What impact, if any, will this have in non-test ...
5 years, 3 months ago (2015-09-22 19:59:14 UTC) #49
Scott Hess - ex-Googler
btw, patch #6 looks clean except for the presubmit failure. So just waiting approval from ...
5 years, 3 months ago (2015-09-22 20:00:31 UTC) #50
pavely
sync/ lgtm
5 years, 3 months ago (2015-09-23 00:09:12 UTC) #51
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/140001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/140001
5 years, 3 months ago (2015-09-23 00:47:21 UTC) #54
commit-bot: I haz the power
Try jobs failed on following builders: linux_android_rel_ng on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_android_rel_ng/builds/72449)
5 years, 3 months ago (2015-09-23 06:41:10 UTC) #56
Scott Hess - ex-Googler
On 2015/09/23 06:41:10, commit-bot: I haz the power wrote: > Try jobs failed on following ...
5 years, 3 months ago (2015-09-23 17:19:24 UTC) #57
commit-bot: I haz the power
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1349863003/140001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1349863003/140001
5 years, 3 months ago (2015-09-23 17:20:48 UTC) #59
commit-bot: I haz the power
Committed patchset #8 (id:140001)
5 years, 3 months ago (2015-09-23 20:30:20 UTC) #60
commit-bot: I haz the power
Patchset 8 (id:??) landed as https://crrev.com/9a1948a4d6d445d5c8e209bdcd1cd050af72060b Cr-Commit-Position: refs/heads/master@{#350362}
5 years, 3 months ago (2015-09-23 20:35:07 UTC) #61
jbroman
5 years, 3 months ago (2015-09-23 22:37:40 UTC) #62
Message was sent while issue was closed.
A revert of this CL (patchset #8 id:140001) has been created in
https://codereview.chromium.org/1365783002/ by jbroman@chromium.org.

The reason for reverting is: mmap_enabled_ isn't initialized, causing MSAN
failures:

https://storage.googleapis.com/chromium-layout-test-archives/WebKit_Linux_MSA...

STDERR: ==3138==WARNING: MemorySanitizer: use-of-uninitialized-value
STDERR:     #0 0x7fc8068d3a65 in ReleaseCacheMemoryIfNeeded
sql/connection.cc:513:7
STDERR:     #1 0x7fc8068d3a65 in sql::Connection::ExecuteAndReturnErrorCode(char
const*) sql/connection.cc:943:0
STDERR:     #2 0x7fc8068ca454 in
sql::Connection::OpenInternal(std::__1::basic_string<char,
std::__1::char_traits<char>, std::__1::allocator<char> > const&,
sql::Connection::Retry) sql/connection.cc:1275:9
STDERR:     #3 0x7fc8068c845f in sql::Connection::Open(base::FilePath const&)
sql/connection.cc:367:10
STDERR:     #4 0x7fc806b1b868 in storage::QuotaDatabase::LazyOpen(bool)
storage/browser/quota/quota_database.cc:488:14.

Powered by Google App Engine
This is Rietveld 408576698