Chromium Code Reviews
|
|
Chromium Code Reviews
Issue
256163005:
Introduce AudioClock to improve playback delay calculations. (Closed)
|
Created:
6 years, 7 months ago by scherkus (not reviewing) Modified:
6 years, 7 months ago Reviewers:
DaleCurtis CC:
chromium-reviews, feature-media-reviews_chromium.org Base URL:
svn://svn.chromium.org/chrome/trunk/src Visibility:
Public. |
DescriptionIntroduce AudioClock to improve playback time calculations.
The previous method for calculating playback time assumed all data
currently buffered contained audio data scaled at the same rate. In
reality, when the playback rate changes we enter a brief period where
audio data scaled at different rates is buffered. The end result was
that the media clock would jump backwards/forwards, introducing playback
jank.
BUG=367343
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=267982
Patch Set 1 #
Total comments: 6
Patch Set 2 : AudioClock #
Total comments: 38
Patch Set 3 : comments #Patch Set 4 : fix eos #
Messages
Total messages: 37 (0 generated)
there's a failing test I'm investigating, but would like to get your feedback on API and whether my assumptions about returning 0 frames in the Render() callback implies https://codereview.chromium.org/256163005/diff/1/media/filters/audio_renderer... File media/filters/audio_renderer_impl.cc (right): https://codereview.chromium.org/256163005/diff/1/media/filters/audio_renderer... media/filters/audio_renderer_impl.cc:587: buffered_audio_tracker_->WroteSilence(requested_frames); alternatively, we can delete buffered_audio_tracker_ and only (re)create it when we know we're in steady state I'm not too sure if we can assume that returning 0 frames is equivalent to writing the requested amount of frames of silence ... https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... File media/filters/buffered_audio_tracker.h (right): https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... media/filters/buffered_audio_tracker.h:29: void WroteAudio(int frames, float playback_rate); I need to think about it some more ... but an alternative API could be something like: void WroteAudio(int frames, int total_frames, float playback_rate); ... and we could get rid of WroteSilence(). That way we could [D]CHECK() that alternating calls to AudioCallbackFired() and WroteAudio() were written, but that seems overly strict
What you're essentially doing is creating a new clock. crogers@ advocated for a similar design for a while, with an additional improvement that you use a FIR or IIR filter on the delay since it can be a little noisy (particularly on Linux). I believe acolwell@ and I chatted about this back on http://crbug.com/138098 too. My previous experiments didn't show noticeable improvements with our old scheduler, so I'm glad your improvements make this viable and that you have testing in place to evaluate this new design! With that said, I think you can take this one step further (seemingly without added complexity) and instead of BufferedAudioTracker rename it to AudioClock() and use it to simplify the current time calculations as well. https://codereview.chromium.org/256163005/diff/1/media/filters/audio_renderer... File media/filters/audio_renderer_impl.cc (right): https://codereview.chromium.org/256163005/diff/1/media/filters/audio_renderer... media/filters/audio_renderer_impl.cc:587: buffered_audio_tracker_->WroteSilence(requested_frames); On 2014/04/29 17:22:06, scherkus wrote: > alternatively, we can delete buffered_audio_tracker_ and only (re)create it when > we know we're in steady state > > I'm not too sure if we can assume that returning 0 frames is equivalent to > writing the requested amount of frames of silence ... I don't think you need to track this. algorithm_ should only be NULL after Stop() which means shutdown is imminent. In general I think that assumption is sound, as it's what the AudioRendererMixer will end up doing. https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... File media/filters/buffered_audio_tracker.h (right): https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... media/filters/buffered_audio_tracker.h:29: void WroteAudio(int frames, float playback_rate); On 2014/04/29 17:22:06, scherkus wrote: > I need to think about it some more ... but an alternative API could be something > like: > > void WroteAudio(int frames, int total_frames, float playback_rate); > > ... and we could get rid of WroteSilence(). That way we could [D]CHECK() that > alternating calls to AudioCallbackFired() and WroteAudio() were written, but > that seems overly strict I like the idea of a single function API. Can't you take that one step further and collapse AudioCallbackFired w/ WroteAudio()?
thanks for the feedback! re "simplify the current time calculations as well" --> are you referring to inside of ARI or elsewhere in the audio system? e.g., do you see AudioClock being used outside of ARI? https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... File media/filters/buffered_audio_tracker.h (right): https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... media/filters/buffered_audio_tracker.h:29: void WroteAudio(int frames, float playback_rate); On 2014/04/29 17:55:54, DaleCurtis wrote: > On 2014/04/29 17:22:06, scherkus wrote: > > I need to think about it some more ... but an alternative API could be > something > > like: > > > > void WroteAudio(int frames, int total_frames, float playback_rate); > > > > ... and we could get rid of WroteSilence(). That way we could [D]CHECK() that > > alternating calls to AudioCallbackFired() and WroteAudio() were written, but > > that seems overly strict > > I like the idea of a single function API. Can't you take that one step further > and collapse AudioCallbackFired w/ WroteAudio()? yeah I played around with that .. the key thing is you need to extract BufferedTime() prior to writing data you end up with something like: base::TimeDelta WroteAudio(int frames, int total_frames, float playback_rate, int delay_frames); I'll try it out and see what it's like
https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... File media/filters/buffered_audio_tracker.h (right): https://codereview.chromium.org/256163005/diff/1/media/filters/buffered_audio... media/filters/buffered_audio_tracker.h:29: void WroteAudio(int frames, float playback_rate); On 2014/04/29 18:19:03, scherkus wrote: > On 2014/04/29 17:55:54, DaleCurtis wrote: > > On 2014/04/29 17:22:06, scherkus wrote: > > > I need to think about it some more ... but an alternative API could be > > something > > > like: > > > > > > void WroteAudio(int frames, int total_frames, float playback_rate); > > > > > > ... and we could get rid of WroteSilence(). That way we could [D]CHECK() > that > > > alternating calls to AudioCallbackFired() and WroteAudio() were written, but > > > that seems overly strict > > > > I like the idea of a single function API. Can't you take that one step > further > > and collapse AudioCallbackFired w/ WroteAudio()? > > yeah I played around with that .. the key thing is you need to extract > BufferedTime() prior to writing data > > you end up with something like: > > base::TimeDelta WroteAudio(int frames, int total_frames, float playback_rate, > int delay_frames); > > I'll try it out and see what it's like Ah, that's a bit awkward then. Merging the Wrote() methods sounds fine then if there's no way to streamline the interface. I'll give it some more thought too.
I mean both. With the right approach, this is a piece of code that could be useful to WebRTC and Cast folks as well. Perhaps even dove tailing with the ongoing TimeTicks/TimeDelta discussion for VideoFrames. OSX and Windows have the concept of reference clocks which can be used to similar effect: http://msdn.microsoft.com/en-us/library/windows/desktop/dd370881.aspx https://developer.apple.com/library/ios/qa/qa1643/_index.html (sorry OSX docs are bad :(
PTAL https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_clock.h File media/filters/audio_clock.h (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.h:34: void WroteSilence(int frames, int delay_frames); went with two separate APIs as it started to look really funky when writing only silence as you get forced to provide playback_rate and timestamp https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... File media/filters/audio_clock_unittest.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock_unittest.cc:75: // by one second until it hits the the slowed down audio. note to self: s/the the/the/ https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... File media/filters/audio_renderer_impl.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:621: now >= earliest_end_time_) { I know more of this code can get cleaned up (e.g., removing earliest_end_time_ completely...) but I'd rather do that as a separate CL https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:654: // delay after we've revamped our audio IPC subsystem. I'm not sure how much of this whole comment block is useful anymore WDYT?
lgtm % one potential error and a bunch of up to you nits. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... File media/filters/audio_clock.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:13: : sample_rate_(sample_rate), last_endpoint_timestamp_(kNoTimestamp()) { Instead of storing sample_rate_ you could store a double of base::Time::kMicrosecondsPerSecond / sample_rate_ as microseconds_per_frame_ for the calculations below. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:23: CHECK_GT(playback_rate, 0); DCHECK_GE(frames, 0) ? Add to WroteSilence() as well? Otherwise you could consider using size_t. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:48: silence += base::TimeDelta::FromMicroseconds( Instead of repeatedly converting this, it'd be better to accumulate frames and then perform a single conversion later when subtracting the value off current_media_time. You avoid error due to truncation this way as well as save a few cycles https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:55: base::TimeDelta current_media_time = buffered_audio_[i].endpoint_timestamp; You could write this as a single return statement. That's more legible to me, but up to you. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:70: base::TimeDelta AudioClock::LastEndpointTimestamp() const { hacker_style() if you want. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:81: // Reached the end before accounting for all of |delay_frames|. This Reflow comment block. Line breaks are off. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:97: // be considered played out and hence discared. discarded https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:98: buffered_audio_[i].frames = delay_frames; Is this right? delay_frames may be 0 now. Shouldn't this be -= ? https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:102: // At this point |i| points at what will be the new head of Reflow comment block. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:114: DCHECK_EQ(playback_rate == 0, endpoint_timestamp == kNoTimestamp()); These DCHECKS are kind of a pain when they fire. I find it easier to have an if (xxx) DCHECK()... which should get optimized out on release. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_clock.h File media/filters/audio_clock.h (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.h:49: int sample_rate_; const https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.h:63: float playback_rate; Looks like this could be const? https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... File media/filters/audio_renderer_impl.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:171: audio_clock_.reset(new AudioClock(audio_parameters_.sample_rate())); Kind of stinks to allocate a whole new object. You could have a Reset() method which just clears buffered_audio_. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:579: audio_clock_->WroteSilence(requested_frames, delay_frames); This should only be true after Stop() has been called, so you don't really need this call. Up to you though. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:621: now >= earliest_end_time_) { On 2014/04/30 19:13:58, scherkus wrote: > I know more of this code can get cleaned up (e.g., removing earliest_end_time_ > completely...) but I'd rather do that as a separate CL I agree. Is it possible to get rid of earliest_end_time_ though? I'd think we still need that. One optimization I thought of in the past, was that if received_end_of_stream_ && wrote silence, to post delayed task the ended cb based on a the media clock time. Experiments in the past showed just using earliest_end_time_ was not accurate enough. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:636: task_runner_->PostTask(FROM_HERE, Really this should be TryPostTask() since PostTask requires acquiring a lock and we don't want to stall this thread. Another CL. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:654: // delay after we've revamped our audio IPC subsystem. On 2014/04/30 19:13:58, scherkus wrote: > I'm not sure how much of this whole comment block is useful anymore > > WDYT? Agreed.
Also, please rewrite the change description to be < 72 chars per line :)
On 2014/04/30 20:37:51, DaleCurtis wrote: > Also, please rewrite the change description to be < 72 chars per line :) ... is this a thing we do these days? Last I checked this is just a nice-to-do-if-you-care-about-these-sorts-of-things
On 2014/04/30 20:55:28, scherkus wrote: > On 2014/04/30 20:37:51, DaleCurtis wrote: > > Also, please rewrite the change description to be < 72 chars per line :) > > ... is this a thing we do these days? Last I checked this is just a > nice-to-do-if-you-care-about-these-sorts-of-things It's definitely not required, but certainly a nice thing to do since we don't auto format this stuff on buildbot or revision listings. Which end up as single line monstrosities. :(
there's still a CHECK() that's getting triggered in some tests going to dig in to find out what's causing it https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... File media/filters/audio_clock.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:13: : sample_rate_(sample_rate), last_endpoint_timestamp_(kNoTimestamp()) { On 2014/04/30 20:36:33, DaleCurtis wrote: > Instead of storing sample_rate_ you could store a double of > base::Time::kMicrosecondsPerSecond / sample_rate_ as microseconds_per_frame_ for > the calculations below. opted to combine the expression in CurrentMediaTimestamp() instead https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:23: CHECK_GT(playback_rate, 0); On 2014/04/30 20:36:33, DaleCurtis wrote: > DCHECK_GE(frames, 0) ? Add to WroteSilence() as well? Otherwise you could > consider using size_t. Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:48: silence += base::TimeDelta::FromMicroseconds( On 2014/04/30 20:36:33, DaleCurtis wrote: > Instead of repeatedly converting this, it'd be better to accumulate frames and > then perform a single conversion later when subtracting the value off > current_media_time. You avoid error due to truncation this way as well as save a > few cycles Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:55: base::TimeDelta current_media_time = buffered_audio_[i].endpoint_timestamp; On 2014/04/30 20:36:33, DaleCurtis wrote: > You could write this as a single return statement. That's more legible to me, > but up to you. Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:70: base::TimeDelta AudioClock::LastEndpointTimestamp() const { On 2014/04/30 20:36:33, DaleCurtis wrote: > hacker_style() if you want. Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:81: // Reached the end before accounting for all of |delay_frames|. This On 2014/04/30 20:36:33, DaleCurtis wrote: > Reflow comment block. Line breaks are off. Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:97: // be considered played out and hence discared. On 2014/04/30 20:36:33, DaleCurtis wrote: > discarded Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:98: buffered_audio_[i].frames = delay_frames; On 2014/04/30 20:36:33, DaleCurtis wrote: > Is this right? delay_frames may be 0 now. Shouldn't this be -= ? Say we have: [20, 20, 20] with a delay of 45, we get here with: i = 0 delay_ms = 5 ... which means we end up with [5, 20, 20] to match the delay_ms of 45 If we had [20, 20, 20, 20, 20] with a delay of 60, we get here with: i = 1 delay_ms = 0 ... which means we would end up with [20, 0, 20, 20, 20]. The check below then increments i by one so that we erase the first two elements to arrive at [20, 20, 20] https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:102: // At this point |i| points at what will be the new head of On 2014/04/30 20:36:33, DaleCurtis wrote: > Reflow comment block. Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:114: DCHECK_EQ(playback_rate == 0, endpoint_timestamp == kNoTimestamp()); On 2014/04/30 20:36:33, DaleCurtis wrote: > These DCHECKS are kind of a pain when they fire. I find it easier to have an if > (xxx) DCHECK()... which should get optimized out on release. Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_clock.h File media/filters/audio_clock.h (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.h:49: int sample_rate_; On 2014/04/30 20:36:33, DaleCurtis wrote: > const Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.h:63: float playback_rate; On 2014/04/30 20:36:33, DaleCurtis wrote: > Looks like this could be const? adding const interferes with the implicit copy operator used when doing operations on std::deque<BufferedAudio> https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... File media/filters/audio_clock_unittest.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock_unittest.cc:75: // by one second until it hits the the slowed down audio. On 2014/04/30 19:13:58, scherkus wrote: > note to self: s/the the/the/ Done. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... File media/filters/audio_renderer_impl.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:171: audio_clock_.reset(new AudioClock(audio_parameters_.sample_rate())); On 2014/04/30 20:36:33, DaleCurtis wrote: > Kind of stinks to allocate a whole new object. You could have a Reset() method > which just clears buffered_audio_. Considering the low frequency of this operation I feel it's fine. This way there's less uncertainty around maintaining a Reset() method. https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:579: audio_clock_->WroteSilence(requested_frames, delay_frames); On 2014/04/30 20:36:33, DaleCurtis wrote: > This should only be true after Stop() has been called, so you don't really need > this call. Up to you though. Will leave it in for now ... I suspect this function will get cleaned up some more https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_rend... media/filters/audio_renderer_impl.cc:654: // delay after we've revamped our audio IPC subsystem. On 2014/04/30 20:36:33, DaleCurtis wrote: > On 2014/04/30 19:13:58, scherkus wrote: > > I'm not sure how much of this whole comment block is useful anymore > > > > WDYT? > > Agreed. Trimmed it down.
Still lgtm. Telemetry tests don't seem to spit out where the crash is actually happening :( I guess that's http://crbug.com/330027 https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... File media/filters/audio_clock.cc (right): https://codereview.chromium.org/256163005/diff/20001/media/filters/audio_cloc... media/filters/audio_clock.cc:98: buffered_audio_[i].frames = delay_frames; On 2014/05/02 19:26:05, scherkus wrote: > On 2014/04/30 20:36:33, DaleCurtis wrote: > > Is this right? delay_frames may be 0 now. Shouldn't this be -= ? > > Say we have: [20, 20, 20] with a delay of 45, we get here with: > i = 0 > delay_ms = 5 > > > ... which means we end up with [5, 20, 20] to match the delay_ms of 45 > > If we had [20, 20, 20, 20, 20] with a delay of 60, we get here with: > i = 1 > delay_ms = 0 > > ... which means we would end up with [20, 0, 20, 20, 20]. The check below then > increments i by one so that we erase the first two elements to arrive at [20, > 20, 20] Oh I see, I had this turned around and was considering buffers < i to be the ones that hadn't played out. I see what you're saying now. Thanks for the explanation.
I have the gpu bot CHECK()s repro'ing locally One of the webm files they use has no audio timestamps so the CHECK() against kNoTimestamp() is firing :\
Weird, all decoded output should be timestamped by the decoders. There are checks in place to prevent kNoTimestamp() from getting through. That's really weird that any buffers seen by ARI that aren't EOS don't have timestamps.
On 2014/05/02 19:56:07, DaleCurtis wrote: > Weird, all decoded output should be timestamped by the decoders. There are > checks in place to prevent kNoTimestamp() from getting through. That's really > weird that any buffers seen by ARI that aren't EOS don't have timestamps. OK so it looks like |algorithm_| never gets any data as this is the first and only buffer we get: [8209:8261:0502/130334:ERROR:audio_renderer_impl.cc(378)] DecodedAudioReady status=0 ts=-9223372036854775808 frames=0 duration=-9223372036854775808 eos=1 ... basically an immediate end of stream. I added a check for frames_buffered() before calling FillBuffer()+WroteAudio() as a sanity check and added a unit test. Can you take a quick look and see whether it makes sense? I felt like checking for frames_buffered() was better than checking for kNoTimestamp() as I'd still be interested in knowing when that CHECK() fails inside of AudioClock (and fixing those bugs)
lgtm. What test file is it that is sending an immediate EOS?
On 2014/05/02 20:22:18, DaleCurtis wrote: > lgtm. What test file is it that is sending an immediate EOS? third_party/webgl_conformance/conformance/resources/red-green.webmvp8.webm
Looks like vorbis audio, but it's all preroll so it's trimmed audio.
The CQ bit was checked by scherkus@chromium.org
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/scherkus@chromium.org/256163005/50001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: linux_chromium_rel on tryserver.chromium
The CQ bit was checked by scherkus@chromium.org
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/scherkus@chromium.org/256163005/50001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: mac_chromium_rel on tryserver.chromium
The CQ bit was checked by scherkus@chromium.org
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/scherkus@chromium.org/256163005/50001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: win_chromium_x64_rel on tryserver.chromium
The CQ bit was checked by scherkus@chromium.org
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/scherkus@chromium.org/256163005/50001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: win_chromium_rel on tryserver.chromium
The CQ bit was checked by scherkus@chromium.org
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/scherkus@chromium.org/256163005/50001
Message was sent while issue was closed.
Change committed as 267982 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
