|
|
Created:
4 years, 3 months ago by Xianzhu Modified:
4 years, 3 months ago CC:
blink-reviews, blink-reviews-html_chromium.org, blink-reviews-paint_chromium.org, chromium-reviews, dglazkov+blink, dshwang, eric.carlson_apple.com, feature-media-reviews_chromium.org, fs, gasubic, mlamouri+watch-blink_chromium.org, nessy, slimming-paint-reviews_chromium.org, vcarbune.chromium Target Ref:
refs/pending/heads/master Project:
chromium Visibility:
Public. |
DescriptionInvalidate timeline periodically when media buffered ranges change
Instead of letting media player or source notify HTMLMediaElement
on buffered range change (which would introduce much complexity
in multiple branches and levels of code), add a timer in MediaControls
which checks change of buffered ranges while the media is loading
in paused state, invalidates paint if the ranges change.
BUG=484288
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Patch Set 1 #Patch Set 2 : - #Patch Set 3 : Synchronize buffered ranges when the current play time changes #
Total comments: 4
Patch Set 4 : - #Patch Set 5 : Simplify #Patch Set 6 : - #Patch Set 7 : Conditionally invalidate #
Messages
Total messages: 28 (13 generated)
Description was changed from ========== Fix under-invalidation of media buffered ranges Instead of letting media player or source notify HTMLMediaElement on buffered range change (which would introduce much complexity in multiple branches and levels of code), add a timer in MediaControls which checks change of buffered ranges while the media is loading, invalidates paint if the ranges change and caches the changed buffered ranges for painting. BUG=484288 TEST=No under-invalidation reported for any layout test ========== to ========== Fix under-invalidation of media buffered ranges Instead of letting media player or source notify HTMLMediaElement on buffered range change (which would introduce much complexity in multiple branches and levels of code), add a timer in MediaControls which checks change of buffered ranges while the media is loading, invalidates paint if the ranges change and caches the changed buffered ranges for painting. BUG=484288 TEST=No under-invalidation reported for any layout test CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2 ==========
The CQ bit was checked by wangxianzhu@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: linux_layout_tests_slimming_paint_v2 on master.tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_layout_te...)
The CQ bit was checked by wangxianzhu@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
wangxianzhu@chromium.org changed reviewers: + chrishtr@chromium.org, liberato@chromium.org
The CQ bit was checked by wangxianzhu@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...
How does it invalidate all the other things in the media controls correctly? Are those easier than buffered length for some reason?
On 2016/08/26 23:14:17, chrishtr wrote: > How does it invalidate all the other things in the media controls correctly? Are > those easier > than buffered length for some reason? Some status changes (e.g. network status change) have notifications from the media player via WebMediaPlayerClient interface for synchronized reactions. Some (e.g. current play time change) use timers similar to the one in this CL to throttle the reactions and visual updates. See the playback progress timer in HTMLMediaElement (https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/html/HTML...). These timers seem mainly for firing events defined by the API spec, and by the way for visual updates. The spec doesn't define any event for buffered ranges changes, and I guess this is the reason that we missed visual updates for them.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)
https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... File third_party/WebKit/Source/core/html/shadow/MediaControls.cpp (right): https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... third_party/WebKit/Source/core/html/shadow/MediaControls.cpp:812: } else if (*m_bufferedRangesForPainting == *newBuffered) { might one just unconditionally invalidate the timeline, and avoid the complexity of buffering and checking for equality? or, to put it another way, how many invalidations does this actually save? it's only the timer case that it matters; setPosition via updateTimelinePosition always invalidates anyway, i think. https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... File third_party/WebKit/Source/core/paint/MediaControlsPainter.cpp (right): https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... third_party/WebKit/Source/core/paint/MediaControlsPainter.cpp:329: TimeRanges* bufferedTimeRanges = mediaElement->mediaControls()->bufferedRangesForPainting(); could one use mediaElement->buffered() here as well? the buffered copy could be half a second stale. or, am i just missing something subtle (or obvious :) )?
https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... File third_party/WebKit/Source/core/html/shadow/MediaControls.cpp (right): https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... third_party/WebKit/Source/core/html/shadow/MediaControls.cpp:812: } else if (*m_bufferedRangesForPainting == *newBuffered) { On 2016/08/29 15:25:34, liberato wrote: > might one just unconditionally invalidate the timeline, and avoid the complexity > of buffering and checking for equality? > > or, to put it another way, how many invalidations does this actually save? it's > only the timer case that it matters; setPosition via updateTimelinePosition > always invalidates anyway, i think. Good question. It's unlikely to save invalidations using the comparison. Changed to unconditional. Along with the change in MediaControlsPainter, the CL is much simplified. https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... File third_party/WebKit/Source/core/paint/MediaControlsPainter.cpp (right): https://codereview.chromium.org/2280123002/diff/40001/third_party/WebKit/Sour... third_party/WebKit/Source/core/paint/MediaControlsPainter.cpp:329: TimeRanges* bufferedTimeRanges = mediaElement->mediaControls()->bufferedRangesForPainting(); On 2016/08/29 15:25:34, liberato wrote: > could one use mediaElement->buffered() here as well? the buffered copy could be > half a second stale. > > or, am i just missing something subtle (or obvious :) )? This is required in the under-invalidation checking mode (for testing only) if we enable caching of MediaSliderPart. In the mode, we force repainting any object even it's not invalidated, and any changed painting must have been invalidated. Using buffered() here will break the requirement. However, using buffered() won't cause any real problem because in we only repaint an object when it's invalidated. Reverted changes in this file, and disable caching in under-invalidation checking mode instead.
On 2016/08/26 at 23:33:21, wangxianzhu wrote: > On 2016/08/26 23:14:17, chrishtr wrote: > > How does it invalidate all the other things in the media controls correctly? Are > > those easier > > than buffered length for some reason? > > Some status changes (e.g. network status change) have notifications from the media player via WebMediaPlayerClient interface for synchronized reactions. > > Some (e.g. current play time change) use timers similar to the one in this CL to throttle the reactions and visual updates. See the playback progress timer in HTMLMediaElement (https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/html/HTML...). These timers seem mainly for firing events defined by the API spec, and by the way for visual updates. The spec doesn't define any event for buffered ranges changes, and I guess this is the reason that we missed visual updates for them. This CL won't actually fix the under-painting problems in general, because there will still be a 0.5sec lag before update. Is that right? If so, this CL will make the UI more responsive but won't allow us to get rid of the cache skipper code fully, or we'll have to make it conditional on slimmingPaintUnderInvalidationCheckingEnabled(). Is it really that hard to plumb this data through? Can it be safely piggybacked on an existing update for one of the other UI changes? e.g. playback progress change. Maybe we could cache the buffered range, and only update it when one of the other parts that is already invalidated changes.
On 2016/08/29 19:34:17, chrishtr wrote: > On 2016/08/26 at 23:33:21, wangxianzhu wrote: > > On 2016/08/26 23:14:17, chrishtr wrote: > > > How does it invalidate all the other things in the media controls correctly? > Are > > > those easier > > > than buffered length for some reason? > > > > Some status changes (e.g. network status change) have notifications from the > media player via WebMediaPlayerClient interface for synchronized reactions. > > > > Some (e.g. current play time change) use timers similar to the one in this CL > to throttle the reactions and visual updates. See the playback progress timer in > HTMLMediaElement > (https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/html/HTML...). > These timers seem mainly for firing events defined by the API spec, and by the > way for visual updates. The spec doesn't define any event for buffered ranges > changes, and I guess this is the reason that we missed visual updates for them. > > This CL won't actually fix the under-painting problems in general, because there > will still be > a 0.5sec lag before update. Is that right? If so, this CL will make the UI more > responsive but > won't allow us to get rid of the cache skipper code fully, or we'll have to make > it conditional > on slimmingPaintUnderInvalidationCheckingEnabled(). > > Is it really that hard to plumb this data through? Can it be safely piggybacked > on an existing update for one of the other UI changes? > e.g. playback progress change. Maybe we could cache the buffered range, and only > update it when one of the other parts that is already > invalidated changes. There are multiple paths and implementations of media player/media source. Plumbing the update will need to change and test all of them. We also don't have real-time update for other changes. Also even if we had the update, we might still need to throttle visual update like the way we throttle the play progress update. We can piggyback on playback progress change when the media is playing, but not when the media is paused. Actually the under-invalidation itself is not the problem, but the unchanged painting for a long time is. Many other paintings for the media player also update periodically. They don't have under-invalidation issue because the painter uses the cached data instead of the real-time data. The patch sets other than the latest one do the same for buffered ranges, but require more code. (The previous patches also have a problem that the real-time current time and duration are used to select the range to display, so they can't avoid all chance of under-invalidations. Getting the cached current time and duration requires even more code.) I created the latest patch because it's simple and but get the same user facing result. Only difference is about under-invalidation which does no real harm to the user.
On 2016/08/29 20:54:06, Xianzhu wrote: > On 2016/08/29 19:34:17, chrishtr wrote: > > On 2016/08/26 at 23:33:21, wangxianzhu wrote: > > > On 2016/08/26 23:14:17, chrishtr wrote: > > > > How does it invalidate all the other things in the media controls > correctly? > > Are > > > > those easier > > > > than buffered length for some reason? > > > > > > Some status changes (e.g. network status change) have notifications from the > > media player via WebMediaPlayerClient interface for synchronized reactions. > > > > > > Some (e.g. current play time change) use timers similar to the one in this > CL > > to throttle the reactions and visual updates. See the playback progress timer > in > > HTMLMediaElement > > > (https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/html/HTML...). > > These timers seem mainly for firing events defined by the API spec, and by the > > way for visual updates. The spec doesn't define any event for buffered ranges > > changes, and I guess this is the reason that we missed visual updates for > them. > > > > This CL won't actually fix the under-painting problems in general, because > there > > will still be > > a 0.5sec lag before update. Is that right? If so, this CL will make the UI > more > > responsive but > > won't allow us to get rid of the cache skipper code fully, or we'll have to > make > > it conditional > > on slimmingPaintUnderInvalidationCheckingEnabled(). > > > > Is it really that hard to plumb this data through? Can it be safely > piggybacked > > on an existing update for one of the other UI changes? > > e.g. playback progress change. Maybe we could cache the buffered range, and > only > > update it when one of the other parts that is already > > invalidated changes. > > > There are multiple paths and implementations of media player/media source. > Plumbing the update will need to change and test all of them. We also don't have > real-time update for other changes. Also even if we had the update, we might > still need to throttle visual update like the way we throttle the play progress > update. > > We can piggyback on playback progress change when the media is playing, but not > when the media is paused. Actually the under-invalidation itself is not the > problem, but the unchanged painting for a long time is. > > Many other paintings for the media player also update periodically. They don't > have under-invalidation issue because the painter uses the cached data instead > of the real-time data. The patch sets other than the latest one do the same for > buffered ranges, but require more code. (The previous patches also have a > problem that the real-time current time and duration are used to select the > range to display, so they can't avoid all chance of under-invalidations. Getting > the cached current time and duration requires even more code.) > > I created the latest patch because it's simple and but get the same user facing > result. Only difference is about under-invalidation which does no real harm to > the user. if it helps, webmediaplayer_android is being deprecated. webmediaplayer_ms might not have any concept of this -- i think that it always returns 0. webmediaplayer_impl doesn't directly understand when the buffered ranges change, but i think that pipeline_impl does. don't know if there are any others floating around. not sure if that makes things any easier. in any case, i also don't know how important the paint under-invalidation issue is that you and chrishtr are discussing. If the outcome of that discussion is "it's not important", then MediaControls lg*m.
On 2016/08/29 20:54:06, Xianzhu wrote: > On 2016/08/29 19:34:17, chrishtr wrote: > > On 2016/08/26 at 23:33:21, wangxianzhu wrote: > > > On 2016/08/26 23:14:17, chrishtr wrote: > > > > How does it invalidate all the other things in the media controls > correctly? > > Are > > > > those easier > > > > than buffered length for some reason? > > > > > > Some status changes (e.g. network status change) have notifications from the > > media player via WebMediaPlayerClient interface for synchronized reactions. > > > > > > Some (e.g. current play time change) use timers similar to the one in this > CL > > to throttle the reactions and visual updates. See the playback progress timer > in > > HTMLMediaElement > > > (https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/html/HTML...). > > These timers seem mainly for firing events defined by the API spec, and by the > > way for visual updates. The spec doesn't define any event for buffered ranges > > changes, and I guess this is the reason that we missed visual updates for > them. > > > > This CL won't actually fix the under-painting problems in general, because > there > > will still be > > a 0.5sec lag before update. Is that right? If so, this CL will make the UI > more > > responsive but > > won't allow us to get rid of the cache skipper code fully, or we'll have to > make > > it conditional > > on slimmingPaintUnderInvalidationCheckingEnabled(). > > > > Is it really that hard to plumb this data through? Can it be safely > piggybacked > > on an existing update for one of the other UI changes? > > e.g. playback progress change. Maybe we could cache the buffered range, and > only > > update it when one of the other parts that is already > > invalidated changes. > > > There are multiple paths and implementations of media player/media source. > Plumbing the update will need to change and test all of them. We also don't have > real-time update for other changes. Also even if we had the update, we might > still need to throttle visual update like the way we throttle the play progress > update. > > We can piggyback on playback progress change when the media is playing, but not > when the media is paused. Actually the under-invalidation itself is not the > problem, but the unchanged painting for a long time is. > > Many other paintings for the media player also update periodically. They don't > have under-invalidation issue because the painter uses the cached data instead > of the real-time data. The patch sets other than the latest one do the same for > buffered ranges, but require more code. (The previous patches also have a > problem that the real-time current time and duration are used to select the > range to display, so they can't avoid all chance of under-invalidations. Getting > the cached current time and duration requires even more code.) > > I created the latest patch because it's simple and but get the same user facing > result. Only difference is about under-invalidation which does no real harm to > the user. if it helps, webmediaplayer_android is being deprecated. webmediaplayer_ms might not have any concept of this -- i think that it always returns 0. webmediaplayer_impl doesn't directly understand when the buffered ranges change, but i think that pipeline_impl does. don't know if there are any others floating around. not sure if that makes things any easier. in any case, i also don't know how important the paint under-invalidation issue is that you and chrishtr are discussing. If the outcome of that discussion is "it's not important", then MediaControls lg*m.
Description was changed from ========== Fix under-invalidation of media buffered ranges Instead of letting media player or source notify HTMLMediaElement on buffered range change (which would introduce much complexity in multiple branches and levels of code), add a timer in MediaControls which checks change of buffered ranges while the media is loading, invalidates paint if the ranges change and caches the changed buffered ranges for painting. BUG=484288 TEST=No under-invalidation reported for any layout test CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2 ========== to ========== Invalidate timeline periodically when media buffered ranges change Instead of letting media player or source notify HTMLMediaElement on buffered range change (which would introduce much complexity in multiple branches and levels of code), add a timer in MediaControls which checks change of buffered ranges while the media is loading in paused state, invalidates paint if the ranges change. BUG=484288 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2 ==========
On 2016/08/29 21:19:41, liberato wrote: > On 2016/08/29 20:54:06, Xianzhu wrote: > > On 2016/08/29 19:34:17, chrishtr wrote: > > > On 2016/08/26 at 23:33:21, wangxianzhu wrote: > > > > On 2016/08/26 23:14:17, chrishtr wrote: > > > > > How does it invalidate all the other things in the media controls > > correctly? > > > Are > > > > > those easier > > > > > than buffered length for some reason? > > > > > > > > Some status changes (e.g. network status change) have notifications from > the > > > media player via WebMediaPlayerClient interface for synchronized reactions. > > > > > > > > Some (e.g. current play time change) use timers similar to the one in this > > CL > > > to throttle the reactions and visual updates. See the playback progress > timer > > in > > > HTMLMediaElement > > > > > > (https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/html/HTML...). > > > These timers seem mainly for firing events defined by the API spec, and by > the > > > way for visual updates. The spec doesn't define any event for buffered > ranges > > > changes, and I guess this is the reason that we missed visual updates for > > them. > > > > > > This CL won't actually fix the under-painting problems in general, because > > there > > > will still be > > > a 0.5sec lag before update. Is that right? If so, this CL will make the UI > > more > > > responsive but > > > won't allow us to get rid of the cache skipper code fully, or we'll have to > > make > > > it conditional > > > on slimmingPaintUnderInvalidationCheckingEnabled(). > > > > > > Is it really that hard to plumb this data through? Can it be safely > > piggybacked > > > on an existing update for one of the other UI changes? > > > e.g. playback progress change. Maybe we could cache the buffered range, and > > only > > > update it when one of the other parts that is already > > > invalidated changes. > > > > > > There are multiple paths and implementations of media player/media source. > > Plumbing the update will need to change and test all of them. We also don't > have > > real-time update for other changes. Also even if we had the update, we might > > still need to throttle visual update like the way we throttle the play > progress > > update. > > > > We can piggyback on playback progress change when the media is playing, but > not > > when the media is paused. Actually the under-invalidation itself is not the > > problem, but the unchanged painting for a long time is. > > > > Many other paintings for the media player also update periodically. They don't > > have under-invalidation issue because the painter uses the cached data instead > > of the real-time data. The patch sets other than the latest one do the same > for > > buffered ranges, but require more code. (The previous patches also have a > > problem that the real-time current time and duration are used to select the > > range to display, so they can't avoid all chance of under-invalidations. > Getting > > the cached current time and duration requires even more code.) > > > > I created the latest patch because it's simple and but get the same user > facing > > result. Only difference is about under-invalidation which does no real harm to > > the user. > > if it helps, webmediaplayer_android is being deprecated. webmediaplayer_ms > might not have any concept of this -- i think that it always returns 0. > > webmediaplayer_impl doesn't directly understand when the buffered ranges change, > but i think that pipeline_impl does. don't know if there are any others > floating around. > > not sure if that makes things any easier. > > in any case, i also don't know how important the paint under-invalidation issue > is that you and chrishtr are discussing. If the outcome of that discussion is > "it's not important", then MediaControls lg*m. I don't think the under-invalidation is important. Actually the latest CL contains two unrelated parts. Separated the BoxPainter.cpp part into https://codereview.chromium.org/2297453002/ for performance, then this CL just fix the problem of buffered range update in paused mode.
liberato@ what do you think about the latest patch? It now doesn't deal with anything about under-invalidation, but just invalidates the timeline periodically when the media is loading in paused mode.
On 2016/08/30 22:16:19, Xianzhu wrote: > liberato@ what do you think about the latest patch? It now doesn't deal with > anything about under-invalidation, but just invalidates the timeline > periodically when the media is loading in paused mode. sorry, didn't realize that there was a new CL with the last message. my only concern is if there's a way to get the element into the loading state while no actual loading is going on. in other words, is there a chance that this will keep repainting the display twice a second when chrome would otherwise be doing nothing? i don't think it happens, but it's worth checking that it does transition out to idle as expected if you haven't already. assuming that it works as expected, lgtm. thanks -fl
On 2016/08/30 22:40:17, liberato wrote: > On 2016/08/30 22:16:19, Xianzhu wrote: > > liberato@ what do you think about the latest patch? It now doesn't deal with > > anything about under-invalidation, but just invalidates the timeline > > periodically when the media is loading in paused mode. > > sorry, didn't realize that there was a new CL with the last message. > > my only concern is if there's a way to get the element into the loading state > while no actual loading is going on. in other words, is there a chance that > this will keep repainting the display twice a second when chrome would otherwise > be doing nothing? > > i don't think it happens, but it's worth checking that it does transition out to > idle as expected if you haven't already. > > assuming that it works as expected, lgtm. > > thanks > -fl I think the concern is really reasonable. In theory even if no real data transferred for a long time, the connection can still be kept alive using no-op or heartbeat packets. I would like to add back conditional invalidation. TimeRanges comparison should be cheap (hundreds of machine instructions I guess for typical TimeRanges objects containing 0, 1 or a few Ranges, which is many orders of magnitude lower than a repaint).
I don't like the pattern of starting extra timers during phases of playback that slow down the whole page when they fire. Also, isn't this feature deprecated, per what liberato@ said earlier? It's a bit hacky, but how about noticing that the the buffered range changed when invalidating a part of the media element for some other reason?
This doesn't seem an important feature though because I haven't seen any bug report about it except the one I filed for under-invalidation. I'd like to just close the issue and this CL. On 2016/08/31 01:01:55, chrishtr wrote: > I don't like the pattern of starting extra timers during phases of playback that > slow down > the whole page when they fire. Also, isn't this feature deprecated, per what > liberato@ said > earlier? It's the android implementation that will be deprecated, not the feature. Some mediaplayer implementations still support the feature. We seem also to support the feature for media sources. > It's a bit hacky, but how about noticing that the the buffered range changed > when invalidating > a part of the media element for some other reason? We already paint the latest buffered range whenever the timeline is invalidated for any reason. The problem is that when the video is paused and is being loaded, the whole media player is still, and the user can't see the buffering progress. It may be not important at all. |