Chromium Code Reviews
|
|
Chromium Code Reviews
Issue
920523002:
[Telemetry] Reduce memory usage in presence of TraceValues (Closed)
Description[Telemetry] Reduce memory usage in presence of TraceValues
Previously, creating a TraceValue would store the associated trace_data in
memory and defer serialization until needed. In this CL, the trace_data is
serialized right away into a temporary file to prevent it from being stored in
memory.
BUG=457385
Committed: https://crrev.com/fb1593148142ed2728485b12461cfd8d1fb68da9
Cr-Commit-Position: refs/heads/master@{#317106}
Patch Set 1 #
Total comments: 1
Patch Set 2 : Address Ned's comment #
Total comments: 2
Patch Set 3 : New approach #
Total comments: 14
Patch Set 4 : Publish changes to switch machines #Patch Set 5 : Address Ned's comments #
Total comments: 7
Patch Set 6 : Address Ned's comments and fix bad logic #
Total comments: 10
Patch Set 7 : Address Ned's comments #
Total comments: 2
Patch Set 8 : Implement cleaned_up in terms of _temp_file #Patch Set 9 : rebase -_- #
Messages
Total messages: 40 (8 generated)
eakuefner@chromium.org changed reviewers: + nednguyen@google.com
Hi Ned, This patch makes it so that serialization of TraceValues is no longer deferred. Testing locally on my Linux machine, the memory usage hovers around 700MB where it was more like 2+GB before. I'm not able to gather numbers on Android at the moment, but I also compared to a version that doesn't add the TraceValue and the difference appears to only be about 100MB.
nednguyen@google.com changed reviewers: + dyen@chromium.org
Hi David, can you patch this and do your memory benchmark again?
On 2015/02/11 20:49:47, nednguyen wrote: > Hi David, can you patch this and do your memory benchmark again? Thanks for the quick fix! I tested it on my system with my simple test file, and it looks like it is using about ~375MB of ram now. That is still a 5x from the original smoothness test (~75MB) but it's probably good enough for these tests to run on the bots without running out of memory now. As another test I ran a more complicated test (gpu_times.top_25_smooth) and it used about ~1.35GB. That's probably still within the threshold of what will run but if you have any more ideas of how to decrease the memory consumption used that would be good!
On 2015/02/11 23:29:35, David Yen wrote: > On 2015/02/11 20:49:47, nednguyen wrote: > > Hi David, can you patch this and do your memory benchmark again? > > Thanks for the quick fix! > > I tested it on my system with my simple test file, and it looks like it is using > about ~375MB of ram now. That is still a 5x from the original smoothness test > (~75MB) but it's probably good enough for these tests to run on the bots without > running out of memory now. > > As another test I ran a more complicated test (gpu_times.top_25_smooth) and it > used about ~1.35GB. That's probably still within the threshold of what will run > but if you have any more ideas of how to decrease the memory consumption used > that would be good! Probably we need to do some more investigation. The memory consumption shouldn't be that high. +Ethan: Can you try to comment out the line that added trace value in TBM: https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/te... Last time David tried this and the memory consumption gets down to 45 Mb. Maybe this new implementation is still leaking some objects here...
On 2015/02/11 at 23:38:40, nednguyen wrote: > On 2015/02/11 23:29:35, David Yen wrote: > > On 2015/02/11 20:49:47, nednguyen wrote: > > > Hi David, can you patch this and do your memory benchmark again? > > > > Thanks for the quick fix! > > > > I tested it on my system with my simple test file, and it looks like it is using > > about ~375MB of ram now. That is still a 5x from the original smoothness test > > (~75MB) but it's probably good enough for these tests to run on the bots without > > running out of memory now. > > > > As another test I ran a more complicated test (gpu_times.top_25_smooth) and it > > used about ~1.35GB. That's probably still within the threshold of what will run > > but if you have any more ideas of how to decrease the memory consumption used > > that would be good! > > Probably we need to do some more investigation. The memory consumption shouldn't be that high. > > +Ethan: Can you try to comment out the line that added trace value in TBM: https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/te... > > Last time David tried this and the memory consumption gets down to 45 Mb. Maybe this new implementation is still leaking some objects here... Yes, I definitely want to run more numbers to pin this down a little further. I'm going to set up an Android device as well so that my memory usage numbers can be comparable. I think enabling trace collection across the board like this is definitely going to impose some amount of mandatory overhead, and I'm afraid to remove it for now because I think the dashboard's trace functionality may rely on it. If the usage is low enough to proceed though, my inclination would be to land this fix (which seems to give the lion's share of benefit) and then make a new bug to dig into how we can reduce overhead further. WDYT?
On 2015/02/12 19:54:50, eakuefner wrote: > On 2015/02/11 at 23:38:40, nednguyen wrote: > > On 2015/02/11 23:29:35, David Yen wrote: > > > On 2015/02/11 20:49:47, nednguyen wrote: > > > > Hi David, can you patch this and do your memory benchmark again? > > > > > > Thanks for the quick fix! > > > > > > I tested it on my system with my simple test file, and it looks like it is > using > > > about ~375MB of ram now. That is still a 5x from the original smoothness > test > > > (~75MB) but it's probably good enough for these tests to run on the bots > without > > > running out of memory now. > > > > > > As another test I ran a more complicated test (gpu_times.top_25_smooth) and > it > > > used about ~1.35GB. That's probably still within the threshold of what will > run > > > but if you have any more ideas of how to decrease the memory consumption > used > > > that would be good! > > > > Probably we need to do some more investigation. The memory consumption > shouldn't be that high. > > > > +Ethan: Can you try to comment out the line that added trace value in TBM: > https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/te... > > > > Last time David tried this and the memory consumption gets down to 45 Mb. > Maybe this new implementation is still leaking some objects here... > > Yes, I definitely want to run more numbers to pin this down a little further. > I'm going to set up an Android device as well so that my memory usage numbers > can be comparable. I think enabling trace collection across the board like this > is definitely going to impose some amount of mandatory overhead, and I'm afraid > to remove it for now because I think the dashboard's trace functionality may > rely on it. > > If the usage is low enough to proceed though, my inclination would be to land > this fix (which seems to give the lion's share of benefit) and then make a new > bug to dig into how we can reduce overhead further. WDYT? I agree that this fix maybe not enough, but get us to a better state. lgtm
https://codereview.chromium.org/920523002/diff/1/tools/telemetry/telemetry/va... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/1/tools/telemetry/telemetry/va... tools/telemetry/telemetry/value/trace.py:123: os.remove(self._temp_file.GetAbsPath()) Err, this is not good. It mean that if you call UploadToCloud, then the TraceValue is now useless, i.e: Serialize will no longer works.
On 2015/02/12 20:59:21, nednguyen wrote: > https://codereview.chromium.org/920523002/diff/1/tools/telemetry/telemetry/va... > File tools/telemetry/telemetry/value/trace.py (right): > > https://codereview.chromium.org/920523002/diff/1/tools/telemetry/telemetry/va... > tools/telemetry/telemetry/value/trace.py:123: > os.remove(self._temp_file.GetAbsPath()) > Err, this is not good. It mean that if you call UploadToCloud, then the > TraceValue is now useless, i.e: Serialize will no longer works. Maybe we need s.t like "CleanUp()" method for all TraceValue. Then call results.CleanUp() in benchmark.Run's final block.
New patchsets have been uploaded after l-g-t-m from nednguyen@google.com
Ned, PTAL. I've removed the logic that deals with trying to delete the tempfile from the uploading method and instead added a destructor. Worst case, cleanup will definitely take place when Telemetry exits, and meantime that allows users to refer to the tempfile safely for as long as the associated TraceValues are referred to.
On 2015/02/12 at 23:28:37, eakuefner wrote: > Ned, PTAL. I've removed the logic that deals with trying to delete the tempfile from the uploading method and instead added a destructor. Worst case, cleanup will definitely take place when Telemetry exits, and meantime that allows users to refer to the tempfile safely for as long as the associated TraceValues are referred to. Sorry, I spoke too soon; it's still not quite right. I'm going to keep hacking and ping you again.
https://codereview.chromium.org/920523002/diff/20001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/20001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:38: os.remove(self._temp_file.GetAbsPath()) Is this guaranteed to be called at the end of the program even when exceptions are raised?
https://codereview.chromium.org/920523002/diff/20001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/20001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:38: os.remove(self._temp_file.GetAbsPath()) On 2015/02/12 23:49:52, nednguyen wrote: > Is this guaranteed to be called at the end of the program even when exceptions > are raised? I'm not sure, and on reflection I think it would be bad anyway to rely on this implicit behavior even if it works out the right way. I like your original idea of CleanUp so that's what I'm going to try next -- there's already a step when we compile the results where we collect all the TraceValues, so at that time it would be easy enough for us to just call CleanUp on each Trace, and that should also make it so that this will be easy to unit test.
Ned, PTAL. I call page_test_results's CleanUp explicitly in JSON and Chart JSON output formatters but haven't touched any other output formatters, and I'm not sure if it's necessary. Let me know what you think.
On 2015/02/13 at 22:45:26, eakuefner wrote: > Ned, PTAL. I call page_test_results's CleanUp explicitly in JSON and Chart JSON output formatters but haven't touched any other output formatters, and I'm not sure if it's necessary. Let me know what you think. Another thought that I have is, perhaps it would be best to CleanUp at the end of PrintSumamry? We still have this pattern of a PTR object can know about multiple output formatters, so if we leave it the output formatters to clean up, there's the risk that another output formatter may later depend on the already-cleaned-up data. This also makes it so that we don't have to modify each output formatter to do the cleanup.
I expect the clean_up to be called in benchmark.Run() method (https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/te...) instead of those chart json https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... File tools/telemetry/telemetry/results/page_test_results.py (right): https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/results/page_test_results.py:133: v.CleanUp() Remove all the trace value after this. https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/__init__.py (right): https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/__init__.py:63: assert isinstance(important, bool) Rebase? https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:58: """Cleans up tempfile after it is no longer needed.""" If self._temp_file is None: return https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:59: os.remove(self._temp_file.GetAbsPath()) self._temp_file = None https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:96: def AsDict(self): Let's raise exception if temp_file is NOne https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:104: def Serialize(self, dir_path): Ditto https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:111: def UploadToCloud(self, bucket): Ditto
picksi@google.com changed reviewers: + picksi@google.com
FYI A new page set of 'pathological' android sites (that I added) has been crashing benchmarks. If I run: tools/perf/run_benchmark smoothness.pathological_mobile_sites --browser=android-chrome --also-run-disabled-tests --page-filter='latimes' ...On a Nexus7 it hangs, and the log contains a lot of GC'ing, which (probably) implies it is trying to get back memory (NB it works OK on a nexus 5 initially, but will crash when cycling a number of pages). I applied this patch and unfortunately it has made no difference.
On 2015/02/16 14:42:04, picksi wrote: > FYI A new page set of 'pathological' android sites (that I added) has been > crashing benchmarks. > > If I run: > > tools/perf/run_benchmark smoothness.pathological_mobile_sites > --browser=android-chrome --also-run-disabled-tests --page-filter='latimes' > > ...On a Nexus7 it hangs, and the log contains a lot of GC'ing, which (probably) > implies it is trying to get back memory (NB it works OK on a nexus 5 initially, > but will crash when cycling a number of pages). > > I applied this patch and unfortunately it has made no difference. Hi Picksi, this patch is supposed to alleviate the memory usage on the host machine, not on the device. Can you file a different bug for memory issue on device?
With my Sheriff on, is this related to the failure here: http://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf/bu... The log says: D/Zygote ( 181): Process 4453 terminated by signal (11) W/ChildProcessConnection( 4392): onServiceDisconnected (crash or killed by oom): pid=4453
With regard to the above issue, this bot is flaky and has subsequently gone green again, so no immediate panic required!
https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... File tools/telemetry/telemetry/results/page_test_results.py (right): https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/results/page_test_results.py:133: v.CleanUp() On 2015/02/13 at 23:14:54, nednguyen wrote: > Remove all the trace value after this. Done. https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/__init__.py (right): https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/__init__.py:63: assert isinstance(important, bool) On 2015/02/13 at 23:14:54, nednguyen wrote: > Rebase? Done. https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:58: """Cleans up tempfile after it is no longer needed.""" On 2015/02/13 23:14:55, nednguyen wrote: > If self._temp_file is None: return Done. https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:59: os.remove(self._temp_file.GetAbsPath()) On 2015/02/13 23:14:55, nednguyen wrote: > self._temp_file = None Done. https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:96: def AsDict(self): On 2015/02/13 23:14:55, nednguyen wrote: > Let's raise exception if temp_file is NOne Done. https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:104: def Serialize(self, dir_path): On 2015/02/13 23:14:55, nednguyen wrote: > Ditto Done. https://codereview.chromium.org/920523002/diff/40001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:111: def UploadToCloud(self, bucket): On 2015/02/13 23:14:55, nednguyen wrote: > Ditto Done.
Oh, and: On 2015/02/13 at 23:14:55, nednguyen wrote: > I expect the clean_up to be called in benchmark.Run() method (https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/te...) instead of those chart json Done.
https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... File tools/telemetry/telemetry/benchmark.py (right): https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... tools/telemetry/telemetry/benchmark.py:193: results.CleanUp() Why not use the with pattern? https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... File tools/telemetry/telemetry/results/page_test_results.py (right): https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... tools/telemetry/telemetry/results/page_test_results.py:130: def CleanUp(self): Can you add unittest cover for CleanUp to make sure that all the traces have Cleaned Up called & they are removed from results? https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:58: """Cleans up tempfile after it is no longer needed.""" Add: "A cleaned up trace alue cannot be used for further operations. CleanUp() may be called more than once without error. (I borrow these from file.close's docstring).
https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... File tools/telemetry/telemetry/benchmark.py (right): https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... tools/telemetry/telemetry/benchmark.py:193: results.CleanUp() On 2015/02/18 00:17:49, nednguyen wrote: > Why not use the with pattern? Done. I initially thought it might be too much indentation but I agree it looks much better with with. https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... File tools/telemetry/telemetry/results/page_test_results.py (right): https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... tools/telemetry/telemetry/results/page_test_results.py:130: def CleanUp(self): On 2015/02/18 00:17:49, nednguyen wrote: > Can you add unittest cover for CleanUp to make sure that all the traces have > Cleaned Up called & they are removed from results? Done. Two separate tests added. The tests were so good that they caught an error in this method that I have also fixed. https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... tools/telemetry/telemetry/value/trace.py:58: """Cleans up tempfile after it is no longer needed.""" On 2015/02/18 00:17:49, nednguyen wrote: > Add: "A cleaned up trace alue cannot be used for further operations. CleanUp() > may be called more than once without error. (I borrow these from file.close's > docstring). Done.
https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... File tools/telemetry/telemetry/benchmark.py (right): https://codereview.chromium.org/920523002/diff/80001/tools/telemetry/telemetr... tools/telemetry/telemetry/benchmark.py:193: results.CleanUp() On 2015/02/18 18:27:16, eakuefner wrote: > On 2015/02/18 00:17:49, nednguyen wrote: > > Why not use the with pattern? > > Done. I initially thought it might be too much indentation but I agree it looks > much better with with. Not only about the looks but this make sure that CleanUp is called no matter what exception is raised. e.g: UploadTraceFilesToCloud may raise exception. https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... File tools/telemetry/telemetry/results/page_test_results.py (right): https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results.py:136: run.values.remove(v) Why do you change this implementation? https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... File tools/telemetry/telemetry/results/page_test_results_unittest.py (right): https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:193: def test_trace_value(self): Hmh, I will make a patch to clean up all of this underscore. This is not our convention to name methods in telemetry. https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:208: def test_clean_up_cleans_up_trace_values(self): testCleanUpCleansUpAllTraceValues https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:216: results.CleanUp() Err, this is awkward wait to test the cleanup has been called for the trace. Can you just create a bit like trace_cleaned_up_called and verify it? Something like the test below but assertTrue(v0.trace_cleaned_up_called) assertTrue(v1.trace_cleaned_up_called) https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:218: def test_no_traces_left_after_clean_up(self): testNoTracesLeftAfterCleanUp
https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... File tools/telemetry/telemetry/results/page_test_results.py (right): https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results.py:136: run.values.remove(v) On 2015/02/18 18:35:59, nednguyen wrote: > Why do you change this implementation? all_page_specific_values is actually a @property that copies all values out of each page run into a new list which it returns. To truly remove the values we need to look at the page runs in this way. https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... File tools/telemetry/telemetry/results/page_test_results_unittest.py (right): https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:193: def test_trace_value(self): On 2015/02/18 at 18:35:59, nednguyen wrote: > Hmh, I will make a patch to clean up all of this underscore. This is not our convention to name methods in telemetry. OK, I saw your CL, so I'll just make the names of my new tests match. https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:208: def test_clean_up_cleans_up_trace_values(self): On 2015/02/18 at 18:35:59, nednguyen wrote: > testCleanUpCleansUpAllTraceValues Done. https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:216: results.CleanUp() On 2015/02/18 at 18:35:59, nednguyen wrote: > Err, this is awkward wait to test the cleanup has been called for the trace. Can you just create a bit like trace_cleaned_up_called and verify it? > Something like the test below but > assertTrue(v0.trace_cleaned_up_called) > assertTrue(v1.trace_cleaned_up_called) I added cleaned_up to the actual TraceValue and got rid of my test class. https://codereview.chromium.org/920523002/diff/100001/tools/telemetry/telemet... tools/telemetry/telemetry/results/page_test_results_unittest.py:218: def test_no_traces_left_after_clean_up(self): On 2015/02/18 at 18:35:59, nednguyen wrote: > testNoTracesLeftAfterCleanUp Done.
lgtm https://codereview.chromium.org/920523002/diff/120001/tools/telemetry/telemet... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/120001/tools/telemetry/telemet... tools/telemetry/telemetry/value/trace.py:77: return self._cleaned_up Nit: "return self._temp_file is None" instead?
New patchsets have been uploaded after l-g-t-m from nednguyen@google.com
https://codereview.chromium.org/920523002/diff/120001/tools/telemetry/telemet... File tools/telemetry/telemetry/value/trace.py (right): https://codereview.chromium.org/920523002/diff/120001/tools/telemetry/telemet... tools/telemetry/telemetry/value/trace.py:77: return self._cleaned_up On 2015/02/18 22:46:39, nednguyen wrote: > Nit: "return self._temp_file is None" instead? Done.
The CQ bit was checked by eakuefner@chromium.org
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/920523002/140001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: ios_dbg_simulator on tryserver.chromium.mac (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.mac/builders/ios_dbg_simulator...)
The CQ bit was checked by eakuefner@chromium.org
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/920523002/160001
Message was sent while issue was closed.
Committed patchset #9 (id:160001)
Message was sent while issue was closed.
Patchset 9 (id:??) landed as https://crrev.com/fb1593148142ed2728485b12461cfd8d1fb68da9 Cr-Commit-Position: refs/heads/master@{#317106} |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
