|
|
Created:
4 years, 3 months ago by Yoav Weiss Modified:
4 years, 3 months ago CC:
chromium-reviews, tyoshino+watch_chromium.org, gavinp+loader_chromium.org, blink-reviews, loading-reviews+fetch_chromium.org Target Ref:
refs/pending/heads/master Project:
chromium Visibility:
Public. |
DescriptionTurn off isLinkPreload() when used
Make sure that isLinkPreload flag is turned off when the resource is used,
making sure that it can be cleared earlier.
BUG=635483
Committed: https://crrev.com/41924d42a6d0ebdd5e2aebc4a5dcca2d5bc11eb6
Cr-Commit-Position: refs/heads/master@{#419283}
Patch Set 1 #Patch Set 2 : build issue #Patch Set 3 : Turn off LinkPreload when resource is used + tests #
Total comments: 1
Patch Set 4 : Added required test && addressed review comments #Patch Set 5 : change approach regarding making sure counter does not get out of sync #Patch Set 6 : remove changes in Resource #
Total comments: 2
Patch Set 7 : remove isPreloaded() changes #
Total comments: 2
Patch Set 8 : Fixed tests #
Total comments: 4
Patch Set 9 : Using a mock resource client #
Messages
Total messages: 67 (41 generated)
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Description was changed from ========== Make Resource::isPreloaded() boolean based BUG= ========== to ========== Make Resource::isPreloaded() boolean based BUG= ==========
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: chromeos_daisy_chromium_compile_only_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/chromeos_daisy_...) linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
Description was changed from ========== Make Resource::isPreloaded() boolean based BUG= ========== to ========== Make Resource::isPreloaded() boolean based As currently implemented, isPreloaded() is based on a counter, which increases when preloadStarted runs and decreases when clearPreloads is run. However, for link preload based resources, I suspect that counter may never get turned off, as they get preloaded several times (preloadScanner for link preload, then link preload), so the counter gets incremented multiple times, but gets decremented once. I suspect that causes link preload based resources to not be removed in pruneDeadResources(), so this issue might be related to memory issues we're seeing in https://crbug.com/635483 BUG=635483 ==========
yoav@yoav.ws changed reviewers: + csharrison@chromium.org, japhet@chromium.org
Hey Nate and Charlie, Can you take a look and tell me if I'm horribly wrong about isPreloaded() and it should stay a boolean? :)
On 2016/09/07 11:18:42, Yoav Weiss wrote: > Hey Nate and Charlie, > > Can you take a look and tell me if I'm horribly wrong about isPreloaded() and it > should stay a boolean? :) that is, should stay a counter...
This l-g-t-m but I'll defer to Nate for final word. If this actually solves the memory issue you reference it would be nice to add a test if possible :)
On 2016/09/07 15:08:20, Charlie Harrison wrote: > This l-g-t-m but I'll defer to Nate for final word. > > If this actually solves the memory issue you reference it would be nice to add a > test if possible :) I thought that stopping the regression on the mem benchmarks might be test enough, but I can try to add a unit test inspecting isPreloaded()
Thanks. Perf dashboard is good but not as good as unit tests which can prevent a regression from landing in the first place :)
On 2016/09/07 15:13:48, Yoav Weiss wrote: > On 2016/09/07 15:08:20, Charlie Harrison wrote: > > This l-g-t-m but I'll defer to Nate for final word. > > > > If this actually solves the memory issue you reference it would be nice to add > a > > test if possible :) > > I thought that stopping the regression on the mem benchmarks might be test > enough, but I can try to add a unit test inspecting isPreloaded() Would you mind testing how this works if multiple ResourceFetchers try to preload the same resource? That's the one place I can think of where this might cause isPreloaded() might now go false too soon.
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Description was changed from ========== Make Resource::isPreloaded() boolean based As currently implemented, isPreloaded() is based on a counter, which increases when preloadStarted runs and decreases when clearPreloads is run. However, for link preload based resources, I suspect that counter may never get turned off, as they get preloaded several times (preloadScanner for link preload, then link preload), so the counter gets incremented multiple times, but gets decremented once. I suspect that causes link preload based resources to not be removed in pruneDeadResources(), so this issue might be related to memory issues we're seeing in https://crbug.com/635483 BUG=635483 ========== to ========== Turn off isLinkPreload() when used and make isPreloaded() boolean Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. Also, as currently implemented, isPreloaded() is based on a counter, which increases when preloadStarted runs and decreases when clearPreloads is run. However, for link preload based resources, I suspect that counter may never get turned off, as they get preloaded several times (preloadScanner for link preload, then link preload), so the counter gets incremented multiple times, but gets decremented once. I suspect that causes link preload based resources to not be removed in pruneDeadResources(). BUG=635483 ==========
On 2016/09/07 15:56:42, Nate Chapin wrote: > On 2016/09/07 15:13:48, Yoav Weiss wrote: > > On 2016/09/07 15:08:20, Charlie Harrison wrote: > > > This l-g-t-m but I'll defer to Nate for final word. > > > > > > If this actually solves the memory issue you reference it would be nice to > add > > a > > > test if possible :) > > > > I thought that stopping the regression on the mem benchmarks might be test > > enough, but I can try to add a unit test inspecting isPreloaded() > > Would you mind testing how this works if multiple ResourceFetchers try to > preload the same resource? That's the one place I can think of where this might > cause isPreloaded() might now go false too soon. Added tests and fixed another issue found along the way, where the isLinkPreloaded flag wasn't turned off even after the discovery of the element that was using it. (which meant that the preload couldn't be cleared, unlike speculative preloads)
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)
Thanks for adding for tests, but you didn't test the case I was concerned about: the only situation in which counter vs bool differs is when multiple different ResourceFetchers try to preload the same Resource. How should we handle that situation? I could see the argument that a Resource should cease to be marked preloaded as soon as it gets Used, but marking it as not preloaded when one of many Documents no longer cares about it doesn't seem right. https://codereview.chromium.org/2319483002/diff/40001/third_party/WebKit/Sour... File third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp (right): https://codereview.chromium.org/2319483002/diff/40001/third_party/WebKit/Sour... third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp:477: resource->setLinkPreload(request.isLinkPreload()); Doesn't this have the potential to mark something as a preload that wasn't previously?
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
> Thanks for adding for tests, but you didn't test the case I was concerned about: > the only situation in which counter vs bool differs is when multiple different > ResourceFetchers try to preload the same Resource. How should we handle that > situation? I could see the argument that a Resource should cease to be marked > preloaded as soon as it gets Used, but marking it as not preloaded when one of > many Documents no longer cares about it doesn't seem right. Looking at isPreloaded() uses, the only case that seems problematic on that front is when it's used for early bailout on detemineRevalidationPolicy() - here there can be a scenario where we turn off isPreloaded after one document DCLed, and the second ResourceFetcher will end up redownloading the resource. Otherwise uses seem safe on that front: * In MemoryCache it's used in order to figure out that resources that finished loading but don't yet have an observer should not be pruned. If one fetcher is using that resource, it would be considered "alive", right? * In Resource.cpp it's used to remove only non-preloaded resources that errored from the MemoryCache (unless the error was a cancellation). I don't get why this is necessary. * in canRequest it's used to suppress reporting on preload requests. I'll see if it's possible to keep the counter while avoiding the current situation where the counter can be incremented multiple times for a single fetcher. > > https://codereview.chromium.org/2319483002/diff/40001/third_party/WebKit/Sour... > File third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp (right): > > https://codereview.chromium.org/2319483002/diff/40001/third_party/WebKit/Sour... > third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp:477: > resource->setLinkPreload(request.isLinkPreload()); > Doesn't this have the potential to mark something as a preload that wasn't > previously? Modified to avoid that
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
On 2016/09/12 07:56:41, Yoav Weiss wrote: > > Thanks for adding for tests, but you didn't test the case I was concerned > about: > > the only situation in which counter vs bool differs is when multiple different > > ResourceFetchers try to preload the same Resource. How should we handle that > > situation? I could see the argument that a Resource should cease to be marked > > preloaded as soon as it gets Used, but marking it as not preloaded when one of > > many Documents no longer cares about it doesn't seem right. > > Looking at isPreloaded() uses, the only case that seems problematic on that > front is when it's used for early bailout on detemineRevalidationPolicy() - here > there can be a scenario where we turn off isPreloaded after one document DCLed, > and the second ResourceFetcher will end up redownloading the resource. > > Otherwise uses seem safe on that front: > * In MemoryCache it's used in order to figure out that resources that finished > loading but don't yet have an observer should not be pruned. If one fetcher is > using that resource, it would be considered "alive", right? > * In Resource.cpp it's used to remove only non-preloaded resources that errored > from the MemoryCache (unless the error was a cancellation). I don't get why this > is necessary. > * in canRequest it's used to suppress reporting on preload requests. > > I'll see if it's possible to keep the counter while avoiding the current > situation where the counter can be incremented multiple times for a single > fetcher. > Changed to a new approach that keeps the counter, but only increments it when ResourceFetcher's m_preloads doesn't contain the resource already. Let me know what you think.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
https://codereview.chromium.org/2319483002/diff/100001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp (right): https://codereview.chromium.org/2319483002/diff/100001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp:876: if (m_preloads->add(resource).isNewEntry) Why doesn't the m_preloads->contains(resource) above catch the case where isNewEntry would be false?
https://codereview.chromium.org/2319483002/diff/100001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp (right): https://codereview.chromium.org/2319483002/diff/100001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/ResourceFetcher.cpp:876: if (m_preloads->add(resource).isNewEntry) On 2016/09/12 20:44:51, Nate Chapin wrote: > Why doesn't the m_preloads->contains(resource) above catch the case where > isNewEntry would be false? You're right. It does.
Description was changed from ========== Turn off isLinkPreload() when used and make isPreloaded() boolean Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. Also, as currently implemented, isPreloaded() is based on a counter, which increases when preloadStarted runs and decreases when clearPreloads is run. However, for link preload based resources, I suspect that counter may never get turned off, as they get preloaded several times (preloadScanner for link preload, then link preload), so the counter gets incremented multiple times, but gets decremented once. I suspect that causes link preload based resources to not be removed in pruneDeadResources(). BUG=635483 ========== to ========== Turn off isLinkPreload() when used Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. BUG=635483 ==========
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
A few points: - Can you update the title description (and wrap to 80 cols)? - Can you document these methods a little bit so the next person who wants to remove the counter will know why its there? https://codereview.chromium.org/2319483002/diff/120001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/ResourceFetcherTest.cpp (right): https://codereview.chromium.org/2319483002/diff/120001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/ResourceFetcherTest.cpp:583: EXPECT_FALSE(memoryCache()->contains(resource)); Why is the resource evicted when it was used by a non-preload?
On 2016/09/12 07:56:41, Yoav Weiss wrote: > * In Resource.cpp it's used to remove only non-preloaded resources that errored > from the MemoryCache (unless the error was a cancellation). I don't get why this > is necessary. I didn't write that code, but I think the rationale is that most of the time we don't want to leave failed loads in the cache, so for non-preloads we just evict on failures. However, we also don't want to do 2 failed loads in the preloaded-then-real-load case, so preload errors get to live on in the cache. The exception is when the preload gets cancelled, since that should be a strong signal that we don't care about the result of the preload and are never going to use it.
On 2016/09/12 21:44:52, Nate Chapin wrote: > On 2016/09/12 07:56:41, Yoav Weiss wrote: > > * In Resource.cpp it's used to remove only non-preloaded resources that > errored > > from the MemoryCache (unless the error was a cancellation). I don't get why > this > > is necessary. > > I didn't write that code, but I think the rationale is that most of the time we > don't want to leave failed loads in the cache, so for non-preloads we just evict > on failures. However, we also don't want to do 2 failed loads in the > preloaded-then-real-load case, so preload errors get to live on in the cache. > The exception is when the preload gets cancelled, since that should be a strong > signal that we don't care about the result of the preload and are never going to > use it. makes sense. Thanks!
Description was changed from ========== Turn off isLinkPreload() when used Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. BUG=635483 ========== to ========== Turn off isLinkPreload() when used Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. BUG=635483 ==========
https://codereview.chromium.org/2319483002/diff/120001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/ResourceFetcherTest.cpp (right): https://codereview.chromium.org/2319483002/diff/120001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/ResourceFetcherTest.cpp:583: EXPECT_FALSE(memoryCache()->contains(resource)); On 2016/09/12 21:32:31, Charlie Harrison wrote: > Why is the resource evicted when it was used by a non-preload? Good catch! The resource is evicted since the resource never changed its PreloadResult from PreloadNotReferenced, since it never had a client/observer added to it. I should change the test to add one and make sure it is not removed.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Fixed both tests. PTAL https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/Resource.h (right): https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/Resource.h:259: void testOnlyWillAddClientOrObserver(PreloadReferencePolicy policy) { willAddClientOrObserver(policy); } Adding this test only method, since adding an actual observer (even a Mock one) resulted in weird crashes around the m_observers HashCountedSet, which I suspect is the result of it not being thread safe.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/Resource.h (right): https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/Resource.h:259: void testOnlyWillAddClientOrObserver(PreloadReferencePolicy policy) { willAddClientOrObserver(policy); } On 2016/09/13 09:33:18, Yoav Weiss wrote: > Adding this test only method, since adding an actual observer (even a Mock one) > resulted in weird crashes around the m_observers HashCountedSet, which I suspect > is the result of it not being thread safe. ...this is all single threaded, isn't it? This explanation sounds suspect to me. Are you sure the observer was getting both added and removed?
On 2016/09/13 22:34:18, Nate Chapin wrote: > https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... > File third_party/WebKit/Source/core/fetch/Resource.h (right): > > https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... > third_party/WebKit/Source/core/fetch/Resource.h:259: void > testOnlyWillAddClientOrObserver(PreloadReferencePolicy policy) { > willAddClientOrObserver(policy); } > On 2016/09/13 09:33:18, Yoav Weiss wrote: > > Adding this test only method, since adding an actual observer (even a Mock > one) > > resulted in weird crashes around the m_observers HashCountedSet, which I > suspect > > is the result of it not being thread safe. > > ...this is all single threaded, isn't it? This explanation sounds suspect to me. > Are you sure the observer was getting both added and removed? Both Mock and real observers crashed upon addition. I can investigate further.
The CQ bit was checked by yoav@yoav.ws to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
On 2016/09/14 05:58:20, Yoav Weiss wrote: > On 2016/09/13 22:34:18, Nate Chapin wrote: > > > https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... > > File third_party/WebKit/Source/core/fetch/Resource.h (right): > > > > > https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... > > third_party/WebKit/Source/core/fetch/Resource.h:259: void > > testOnlyWillAddClientOrObserver(PreloadReferencePolicy policy) { > > willAddClientOrObserver(policy); } > > On 2016/09/13 09:33:18, Yoav Weiss wrote: > > > Adding this test only method, since adding an actual observer (even a Mock > > one) > > > resulted in weird crashes around the m_observers HashCountedSet, which I > > suspect > > > is the result of it not being thread safe. > > > > ...this is all single threaded, isn't it? This explanation sounds suspect to > me. > > Are you sure the observer was getting both added and removed? > > Both Mock and real observers crashed upon addition. I can investigate further. Put back the mock observer and no longer see those crashes. Must've been Using It Wrong™ PTAL?
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
non owner lgtm https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/Resource.h (right): https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/Resource.h:259: void testOnlyWillAddClientOrObserver(PreloadReferencePolicy policy) { willAddClientOrObserver(policy); } On 2016/09/13 09:33:18, Yoav Weiss wrote: > Adding this test only method, since adding an actual observer (even a Mock one) > resulted in weird crashes around the m_observers HashCountedSet, which I suspect > is the result of it not being thread safe. I think a "ForTesting" suffix is more common and I think (?) it gets checked by a presubmit that the method is only used for tests.
https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... File third_party/WebKit/Source/core/fetch/Resource.h (right): https://codereview.chromium.org/2319483002/diff/140001/third_party/WebKit/Sou... third_party/WebKit/Source/core/fetch/Resource.h:259: void testOnlyWillAddClientOrObserver(PreloadReferencePolicy policy) { willAddClientOrObserver(policy); } On 2016/09/16 12:18:47, Charlie Harrison wrote: > On 2016/09/13 09:33:18, Yoav Weiss wrote: > > Adding this test only method, since adding an actual observer (even a Mock > one) > > resulted in weird crashes around the m_observers HashCountedSet, which I > suspect > > is the result of it not being thread safe. > > I think a "ForTesting" suffix is more common and I think (?) it gets checked by > a presubmit that the method is only used for tests. Apologies this was a very out of date draft comment :)
lgtm
The CQ bit was checked by yoav@yoav.ws
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
Message was sent while issue was closed.
Description was changed from ========== Turn off isLinkPreload() when used Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. BUG=635483 ========== to ========== Turn off isLinkPreload() when used Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. BUG=635483 ==========
Message was sent while issue was closed.
Committed patchset #9 (id:160001)
Message was sent while issue was closed.
Description was changed from ========== Turn off isLinkPreload() when used Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. BUG=635483 ========== to ========== Turn off isLinkPreload() when used Make sure that isLinkPreload flag is turned off when the resource is used, making sure that it can be cleared earlier. BUG=635483 Committed: https://crrev.com/41924d42a6d0ebdd5e2aebc4a5dcca2d5bc11eb6 Cr-Commit-Position: refs/heads/master@{#419283} ==========
Message was sent while issue was closed.
Patchset 9 (id:??) landed as https://crrev.com/41924d42a6d0ebdd5e2aebc4a5dcca2d5bc11eb6 Cr-Commit-Position: refs/heads/master@{#419283} |