|
|
Created:
5 years, 1 month ago by Łukasz Anforowicz Modified:
5 years ago Reviewers:
Charlie Reis CC:
chromium-reviews, site-isolation-reviews_chromium.org Base URL:
https://chromium.googlesource.com/chromium/src.git@master Target Ref:
refs/pending/heads/master Project:
chromium Visibility:
Public. |
DescriptionEnabling extra tests that pass on site-isolation FYI bots.
Tests that pass and can be removed from the exclusion list:
BrowserTest.ClearPendingOnFailUnlessNTP
BrowserTest.WindowOpenClose
DnsProbeBrowserTest.*
InlineLoginUISafeIframeBrowserTest.*
LaunchWebAuthFlowFunctionTest.*
OptionsUIBrowserTest.*
PopupBlockerBrowserTest.*
*.RestoreCrossSiteWithExistingSiteInstance
ZoomControllerBrowserTest.*
*.ProfilesWithoutPagesNotLaunched
Tests that don't exist anymore / don't match the exclusion pattern:
*.NewAvatarMenuEnabledInGuestMode
Test suites that still fail, but where the exclusion can be narrowed down:
- RedirectTest.* [8 tests / only 1 fails]
can be narrowed down into:
RedirectTest.ClientEmptyReferer
- ReferrerPolicyTest.* [32 tests / only 1 fails]
can be narrowed down into:
ReferrerPolicyTest.HttpsRedirect
- SSLUITest.* [37 tests / only 1 fails]
can be narrowed down into:
SSLUITest.TestGoodFrameNavigation
- RestoreOnStartupPolicyTest* [10 tests / only 2 fail]
can be narrowed down into:
RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest*
BUG=417518, 448592, 467125, 456420
Committed: https://crrev.com/5f920780e30c5f0101e45b9187ac498d9cbc4d20
Cr-Commit-Position: refs/heads/master@{#362248}
Patch Set 1 #
Total comments: 2
Patch Set 2 : Backed out accidental PhishingClassifierTest changes. #Patch Set 3 : Have to exclude BrowserTest.WindowOpenClose after all (flaky). #Patch Set 4 : Rebasing... #Messages
Total messages: 19 (7 generated)
lukasza@chromium.org changed reviewers: + creis@chromium.org
Charlie, can you take a look please?
On 2015/11/20 22:39:16, Łukasz Anforowicz wrote: > Charlie, can you take a look please? Thanks-- I love CLs like this! It's great to see many of these passing now (around 100 or so?). Please list 417518 and 448592 as the bug numbers (and possibly 532666). I'll note that I've tried to re-enable many of these in the past and they proved flaky, requiring me to keep them disabled. A few questions on that front below. This LGTM once you put back any tests that are still obviously flaky. (For the rest, we can always put them back on the list if they start flaking.) > Tests that pass and can be removed from the exclusion list: > BrowserTest.ClearPendingOnFailUnlessNTP I've tried to re-enable ClearPendingOnFailUnlessNTP before, and it proved flaky for me. Can you confirm it's not flaky anymore (e.g., by sending a few more try jobs, in parallel if you want)? Also, it was tracked in https://crbug.com/467125. > BrowserTest.WindowOpenClose WindowOpenClose was flaky too. Tracked in https://crbug.com/467126. > DnsProbeBrowserTest.* > InlineLoginUISafeIframeBrowserTest.* > LaunchWebAuthFlowFunctionTest.* > OptionsUIBrowserTest.* OptionsUIBrowserTest was flaky for me. I don't think there's a bug for it. > PopupBlockerBrowserTest.* > *.RestoreCrossSiteWithExistingSiteInstance > ZoomControllerBrowserTest.* > *.ProfilesWithoutPagesNotLaunched ProfilesWithoutPagesNotLaunched was flaky for me. Tracked in https://crbug.com/456420. > > Tests that don't exist anymore / don't match the exclusion pattern: > *.NewAvatarMenuEnabledInGuestMode Looks like that was disabled in https://crbug.com/471953. I'm ok removing it from our exclusion list, since we can disable it again if they re-enable it and it still doesn't work. > Test suites that still fail, but where the exclusion can be narrowed down: > - RedirectTest.* [8 tests / only 1 fails] > can be narrowed down into: > RedirectTest.ClientEmptyReferer > > - ReferrerPolicyTest.* [32 tests / only 1 fails] > can be narrowed down into: > ReferrerPolicyTest.HttpsRedirect > > - SSLUITest.* [37 tests / only 1 fails] > can be narrowed down into: > SSLUITest.TestGoodFrameNavigation > > - RestoreOnStartupPolicyTest* [10 tests / only 2 fail] > can be narrowed down into: > RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest*
https://codereview.chromium.org/1466783002/diff/1/testing/buildbot/chromium.f... File testing/buildbot/chromium.fyi.json (right): https://codereview.chromium.org/1466783002/diff/1/testing/buildbot/chromium.f... testing/buildbot/chromium.fyi.json:5660: "--gtest_filter=-BrowserTest.InterstitialCancelsGuestViewDialogs:BrowserTest.OtherRedirectsDontForkProcess:ChromeRenderProcessHostTest.*:ChromeRenderProcessHostTestWithCommandLine.*:DevToolsExperimentalExtensionTest.*:DevToolsExtensionTest.*:ErrorPageTest.*:ExtensionApiTest.ContentScriptOtherExtensions:ExtensionApiTest.ContentScriptExtensionProcess:ExtensionApiTest.DontInjectContentScriptsInBackgroundPages:ExtensionApiTest.Tabs2:ExtensionApiTest.TabsOnUpdated:ExtensionApiTest.WindowOpenPopupIframe:ExtensionOptionsApiTest.ExtensionCanEmbedOwnOptions:ExtensionViewTest.*:IsolatedAppTest.*:MimeHandlerViewTest.*:*PDFExtensionTest.*:PhishingClassifierTest.TestClassification:PhishingDOMFeatureExtractorTest.*:PrerenderBrowserTest.*:ProcessManagementTest.*:RedirectTest.ClientEmptyReferer:ReferrerPolicyTest.HttpsRedirect:SSLUITest.TestGoodFrameNavigation:WebNavigationApiTest.CrossProcessFragment:WebNavigationApiTest.ServerRedirectSingleProcess:WebNavigationApiTest.CrossProcessHistory:WebViewDPITest.*:WebViewPluginTest.*:WebViewTest.AcceptTouchEvents:WebViewTest.IndexedDBIsolation:WebViewTest.ScreenCoordinates:WebViewTest.ContextMenusAPI_PreventDefault:WebViewTest.TestContextMenu:WebViewTest.NestedGuestContainerBounds:WebViewFocusTest.*:WebViewNewWindowTest.*:WebViewVisibilityTest.*:*.NavigateFromNTPToOptionsInSameTab:*.TabMove:*.WhitelistedExtension:*.NewTabPageURL:*.HomepageLocation:RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest*:PhishingClassifierDelegateTest.*:WebUIWebViewBrowserTest*:WebRtcBrowserTest.RunsAudioVideoWebRTCCallInTwoTabs:SingleProcessTracingBrowserTest.TestMemoryInfra" Hmm, this pattern differs from the Linux pattern. Looks like PhishingClassifierTest.TestClassification was changed here but not above? Was that intentional? We try to keep the two patterns identical if possible.
Description was changed from ========== Enabling extra tests that pass on site-isolation FYI bots. Tests that pass and can be removed from the exclusion list: BrowserTest.ClearPendingOnFailUnlessNTP BrowserTest.WindowOpenClose DnsProbeBrowserTest.* InlineLoginUISafeIframeBrowserTest.* LaunchWebAuthFlowFunctionTest.* OptionsUIBrowserTest.* PopupBlockerBrowserTest.* *.RestoreCrossSiteWithExistingSiteInstance ZoomControllerBrowserTest.* *.ProfilesWithoutPagesNotLaunched Tests that don't exist anymore / don't match the exclusion pattern: *.NewAvatarMenuEnabledInGuestMode Test suites that still fail, but where the exclusion can be narrowed down: - RedirectTest.* [8 tests / only 1 fails] can be narrowed down into: RedirectTest.ClientEmptyReferer - ReferrerPolicyTest.* [32 tests / only 1 fails] can be narrowed down into: ReferrerPolicyTest.HttpsRedirect - SSLUITest.* [37 tests / only 1 fails] can be narrowed down into: SSLUITest.TestGoodFrameNavigation - RestoreOnStartupPolicyTest* [10 tests / only 2 fail] can be narrowed down into: RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest* BUG=417518,448592 ========== to ========== Enabling extra tests that pass on site-isolation FYI bots. Tests that pass and can be removed from the exclusion list: BrowserTest.ClearPendingOnFailUnlessNTP BrowserTest.WindowOpenClose DnsProbeBrowserTest.* InlineLoginUISafeIframeBrowserTest.* LaunchWebAuthFlowFunctionTest.* OptionsUIBrowserTest.* PopupBlockerBrowserTest.* *.RestoreCrossSiteWithExistingSiteInstance ZoomControllerBrowserTest.* *.ProfilesWithoutPagesNotLaunched Tests that don't exist anymore / don't match the exclusion pattern: *.NewAvatarMenuEnabledInGuestMode Test suites that still fail, but where the exclusion can be narrowed down: - RedirectTest.* [8 tests / only 1 fails] can be narrowed down into: RedirectTest.ClientEmptyReferer - ReferrerPolicyTest.* [32 tests / only 1 fails] can be narrowed down into: ReferrerPolicyTest.HttpsRedirect - SSLUITest.* [37 tests / only 1 fails] can be narrowed down into: SSLUITest.TestGoodFrameNavigation - RestoreOnStartupPolicyTest* [10 tests / only 2 fail] can be narrowed down into: RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest* BUG=417518,448592,467125,456420 ==========
On 2015/11/20 23:11:20, Charlie Reis wrote: > On 2015/11/20 22:39:16, Łukasz Anforowicz wrote: > > Charlie, can you take a look please? > > Thanks-- I love CLs like this! It's great to see many of these passing now > (around 100 or so?). Please list 417518 and 448592 as the bug numbers (and > possibly 532666). > > I'll note that I've tried to re-enable many of these in the past and they proved > flaky, requiring me to keep them disabled. A few questions on that front below. > This LGTM once you put back any tests that are still obviously flaky. (For the > rest, we can always put them back on the list if they start flaking.) > > > Tests that pass and can be removed from the exclusion list: > > BrowserTest.ClearPendingOnFailUnlessNTP > > I've tried to re-enable ClearPendingOnFailUnlessNTP before, and it proved flaky > for me. Can you confirm it's not flaky anymore (e.g., by sending a few more try > jobs, in parallel if you want)? Also, it was tracked in > https://crbug.com/467125. I've run 10+ fyi bot tries and they are green. > > > BrowserTest.WindowOpenClose > > WindowOpenClose was flaky too. Tracked in https://crbug.com/467126. Yup - WindowOpenClose failed after a few tries. I've readded the exclusion for this test. > > > DnsProbeBrowserTest.* > > InlineLoginUISafeIframeBrowserTest.* > > LaunchWebAuthFlowFunctionTest.* > > OptionsUIBrowserTest.* > > OptionsUIBrowserTest was flaky for me. I don't think there's a bug for it. > > > PopupBlockerBrowserTest.* > > *.RestoreCrossSiteWithExistingSiteInstance > > ZoomControllerBrowserTest.* > > *.ProfilesWithoutPagesNotLaunched > > ProfilesWithoutPagesNotLaunched was flaky for me. Tracked in > https://crbug.com/456420. > > > > > Tests that don't exist anymore / don't match the exclusion pattern: > > *.NewAvatarMenuEnabledInGuestMode > > Looks like that was disabled in https://crbug.com/471953. I'm ok removing it > from our exclusion list, since we can disable it again if they re-enable it and > it still doesn't work. > > > Test suites that still fail, but where the exclusion can be narrowed down: > > - RedirectTest.* [8 tests / only 1 fails] > > can be narrowed down into: > > RedirectTest.ClientEmptyReferer > > > > - ReferrerPolicyTest.* [32 tests / only 1 fails] > > can be narrowed down into: > > ReferrerPolicyTest.HttpsRedirect > > > > - SSLUITest.* [37 tests / only 1 fails] > > can be narrowed down into: > > SSLUITest.TestGoodFrameNavigation > > > > - RestoreOnStartupPolicyTest* [10 tests / only 2 fail] > > can be narrowed down into: > > RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest*
The CQ bit was checked by lukasza@chromium.org to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1466783002/60001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1466783002/60001
On 2015/11/30 19:37:53, Łukasz Anforowicz wrote: > On 2015/11/20 23:11:20, Charlie Reis wrote: > > On 2015/11/20 22:39:16, Łukasz Anforowicz wrote: > > > Charlie, can you take a look please? > > > > Thanks-- I love CLs like this! It's great to see many of these passing now > > (around 100 or so?). Please list 417518 and 448592 as the bug numbers (and > > possibly 532666). > > > > I'll note that I've tried to re-enable many of these in the past and they > proved > > flaky, requiring me to keep them disabled. A few questions on that front > below. > > This LGTM once you put back any tests that are still obviously flaky. (For > the > > rest, we can always put them back on the list if they start flaking.) > > > > > Tests that pass and can be removed from the exclusion list: > > > BrowserTest.ClearPendingOnFailUnlessNTP > > > > I've tried to re-enable ClearPendingOnFailUnlessNTP before, and it proved > flaky > > for me. Can you confirm it's not flaky anymore (e.g., by sending a few more > try > > jobs, in parallel if you want)? Also, it was tracked in > > https://crbug.com/467125. > > I've run 10+ fyi bot tries and they are green. > > > > > BrowserTest.WindowOpenClose > > > > WindowOpenClose was flaky too. Tracked in https://crbug.com/467126. > > Yup - WindowOpenClose failed after a few tries. I've readded the exclusion for > this test. > > > > > DnsProbeBrowserTest.* > > > InlineLoginUISafeIframeBrowserTest.* > > > LaunchWebAuthFlowFunctionTest.* > > > OptionsUIBrowserTest.* > > > > OptionsUIBrowserTest was flaky for me. I don't think there's a bug for it. > > > > > PopupBlockerBrowserTest.* > > > *.RestoreCrossSiteWithExistingSiteInstance > > > ZoomControllerBrowserTest.* > > > *.ProfilesWithoutPagesNotLaunched > > > > ProfilesWithoutPagesNotLaunched was flaky for me. Tracked in > > https://crbug.com/456420. > > > > > > > > Tests that don't exist anymore / don't match the exclusion pattern: > > > *.NewAvatarMenuEnabledInGuestMode > > > > Looks like that was disabled in https://crbug.com/471953. I'm ok removing it > > from our exclusion list, since we can disable it again if they re-enable it > and > > it still doesn't work. > > > > > Test suites that still fail, but where the exclusion can be narrowed down: > > > - RedirectTest.* [8 tests / only 1 fails] > > > can be narrowed down into: > > > RedirectTest.ClientEmptyReferer > > > > > > - ReferrerPolicyTest.* [32 tests / only 1 fails] > > > can be narrowed down into: > > > ReferrerPolicyTest.HttpsRedirect > > > > > > - SSLUITest.* [37 tests / only 1 fails] > > > can be narrowed down into: > > > SSLUITest.TestGoodFrameNavigation > > > > > > - RestoreOnStartupPolicyTest* [10 tests / only 2 fail] > > > can be narrowed down into: > > > RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest* Driveby comment: I noticed that ExtensionApiTest.Tabs2 and all tests under ExtensionViewTest.* were passing for me locally. Lukasz, could you perhaps check if they work on the bot, and if so, reenable them here as well? Thanks!
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by lukasza@chromium.org
The patchset sent to the CQ was uploaded after l-g-t-m from creis@chromium.org Link to the patchset: https://codereview.chromium.org/1466783002/#ps60001 (title: "Rebasing...")
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1466783002/60001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1466783002/60001
Message was sent while issue was closed.
Committed patchset #4 (id:60001)
Message was sent while issue was closed.
Description was changed from ========== Enabling extra tests that pass on site-isolation FYI bots. Tests that pass and can be removed from the exclusion list: BrowserTest.ClearPendingOnFailUnlessNTP BrowserTest.WindowOpenClose DnsProbeBrowserTest.* InlineLoginUISafeIframeBrowserTest.* LaunchWebAuthFlowFunctionTest.* OptionsUIBrowserTest.* PopupBlockerBrowserTest.* *.RestoreCrossSiteWithExistingSiteInstance ZoomControllerBrowserTest.* *.ProfilesWithoutPagesNotLaunched Tests that don't exist anymore / don't match the exclusion pattern: *.NewAvatarMenuEnabledInGuestMode Test suites that still fail, but where the exclusion can be narrowed down: - RedirectTest.* [8 tests / only 1 fails] can be narrowed down into: RedirectTest.ClientEmptyReferer - ReferrerPolicyTest.* [32 tests / only 1 fails] can be narrowed down into: ReferrerPolicyTest.HttpsRedirect - SSLUITest.* [37 tests / only 1 fails] can be narrowed down into: SSLUITest.TestGoodFrameNavigation - RestoreOnStartupPolicyTest* [10 tests / only 2 fail] can be narrowed down into: RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest* BUG=417518,448592,467125,456420 ========== to ========== Enabling extra tests that pass on site-isolation FYI bots. Tests that pass and can be removed from the exclusion list: BrowserTest.ClearPendingOnFailUnlessNTP BrowserTest.WindowOpenClose DnsProbeBrowserTest.* InlineLoginUISafeIframeBrowserTest.* LaunchWebAuthFlowFunctionTest.* OptionsUIBrowserTest.* PopupBlockerBrowserTest.* *.RestoreCrossSiteWithExistingSiteInstance ZoomControllerBrowserTest.* *.ProfilesWithoutPagesNotLaunched Tests that don't exist anymore / don't match the exclusion pattern: *.NewAvatarMenuEnabledInGuestMode Test suites that still fail, but where the exclusion can be narrowed down: - RedirectTest.* [8 tests / only 1 fails] can be narrowed down into: RedirectTest.ClientEmptyReferer - ReferrerPolicyTest.* [32 tests / only 1 fails] can be narrowed down into: ReferrerPolicyTest.HttpsRedirect - SSLUITest.* [37 tests / only 1 fails] can be narrowed down into: SSLUITest.TestGoodFrameNavigation - RestoreOnStartupPolicyTest* [10 tests / only 2 fail] can be narrowed down into: RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest* BUG=417518,448592,467125,456420 Committed: https://crrev.com/5f920780e30c5f0101e45b9187ac498d9cbc4d20 Cr-Commit-Position: refs/heads/master@{#362248} ==========
Message was sent while issue was closed.
Patchset 4 (id:??) landed as https://crrev.com/5f920780e30c5f0101e45b9187ac498d9cbc4d20 Cr-Commit-Position: refs/heads/master@{#362248}
Message was sent while issue was closed.
Oooops... I forgot to publish my old draft comment... At any rate - this seems much better addressed by Nick's changes to how the gtest filter args can be specified in the json body. https://codereview.chromium.org/1466783002/diff/1/testing/buildbot/chromium.f... File testing/buildbot/chromium.fyi.json (right): https://codereview.chromium.org/1466783002/diff/1/testing/buildbot/chromium.f... testing/buildbot/chromium.fyi.json:5660: "--gtest_filter=-BrowserTest.InterstitialCancelsGuestViewDialogs:BrowserTest.OtherRedirectsDontForkProcess:ChromeRenderProcessHostTest.*:ChromeRenderProcessHostTestWithCommandLine.*:DevToolsExperimentalExtensionTest.*:DevToolsExtensionTest.*:ErrorPageTest.*:ExtensionApiTest.ContentScriptOtherExtensions:ExtensionApiTest.ContentScriptExtensionProcess:ExtensionApiTest.DontInjectContentScriptsInBackgroundPages:ExtensionApiTest.Tabs2:ExtensionApiTest.TabsOnUpdated:ExtensionApiTest.WindowOpenPopupIframe:ExtensionOptionsApiTest.ExtensionCanEmbedOwnOptions:ExtensionViewTest.*:IsolatedAppTest.*:MimeHandlerViewTest.*:*PDFExtensionTest.*:PhishingClassifierTest.TestClassification:PhishingDOMFeatureExtractorTest.*:PrerenderBrowserTest.*:ProcessManagementTest.*:RedirectTest.ClientEmptyReferer:ReferrerPolicyTest.HttpsRedirect:SSLUITest.TestGoodFrameNavigation:WebNavigationApiTest.CrossProcessFragment:WebNavigationApiTest.ServerRedirectSingleProcess:WebNavigationApiTest.CrossProcessHistory:WebViewDPITest.*:WebViewPluginTest.*:WebViewTest.AcceptTouchEvents:WebViewTest.IndexedDBIsolation:WebViewTest.ScreenCoordinates:WebViewTest.ContextMenusAPI_PreventDefault:WebViewTest.TestContextMenu:WebViewTest.NestedGuestContainerBounds:WebViewFocusTest.*:WebViewNewWindowTest.*:WebViewVisibilityTest.*:*.NavigateFromNTPToOptionsInSameTab:*.TabMove:*.WhitelistedExtension:*.NewTabPageURL:*.HomepageLocation:RestoreOnStartupPolicyTestInstance/RestoreOnStartupPolicyTest.RunTest*:PhishingClassifierDelegateTest.*:WebUIWebViewBrowserTest*:WebRtcBrowserTest.RunsAudioVideoWebRTCCallInTwoTabs:SingleProcessTracingBrowserTest.TestMemoryInfra" On 2015/11/20 23:13:59, Charlie Reis wrote: > Hmm, this pattern differs from the Linux pattern. Looks like > PhishingClassifierTest.TestClassification was changed here but not above? > > Was that intentional? We try to keep the two patterns identical if possible. Not intentional / I thought that I backed out of this change. FWIW, this would be a safe change for test coverage / passability because this test suite has only 2 tests and TestClassification is the only failing one. I learned about $ git diff --word-diff-regex=[^:=-]+ origin/master and for a moment wondered whether I should teach [1] git to use this regex here, but it seems that it would involve editing .git/config and I am not sure if this is ok. BTW - even with word-diff I missed this accidental change (when trying to go one-by-one through the word diff to see if there are other accidental changes)... Sigh... --word-diff=porcelain helped. Done - I've reverted this change / made the 2 command lines consistent. [1] The "update" part at the end of https://idnotfound.wordpress.com/2009/05/09/word-by-word-diffs-in-git/
Message was sent while issue was closed.
On 2015/11/30 19:41:20, alexmos wrote: > Driveby comment: I noticed that ExtensionApiTest.Tabs2 and all tests under > ExtensionViewTest.* were passing for me locally. Lukasz, could you perhaps > check if they work on the bot, and if so, reenable them here as well? Thanks! These tests passed locally when run --gtest_repeat=10 times with --isolate-extensions. OTOH, ExtensionApiTest.Tabs2 intermittently (around 40% of time) fails when run with --site-per-process. I'll try creating a separate CL that enables ExtensionViewTest.* and then seeing if the bot is also happy with this. |