Add --override-os-version to telemetry for decorators

Add --override-os-version to telemetry for decorators

We wish to test telemetry versus linux chromeos builds. This leads to running
a linux configuration for telemetry, with some of the chromeos UI software
running within Chrome.

Telemetry already marks failing tests on an OS basis, with some known failures
for chromeos. However it cannot detect this mixed configuration and ends up
running those tests.

This change adds a new command line flag, --override-os-version which allows
for specifying sub versions like 'linux-chromeos'

This can then be used by decorators to filter out failing tests. We will use
this flag in the configurations of trybots running linux-chromeos

TEST=ran ActualPageRunEndToEndTests.testTrafficSettings locally, which is known
to fail, it now filters out in this config
BUG=chromium:758065

[jonross, 2017-08-28 20:30:39]

jonross@chromium.org changed reviewers:
+ achuith@chromium.org

[jonross, 2017-08-28 20:30:40]

Hey Achuith,

Could you take a look at this change to telemetry? Here I add a new command line
flag, which will let us differentiate the linux-chromeos builds from normal
linux ones.

Thanks,
Jon

[achuithb, 2017-08-28 22:21:32]

The CQ bit was checked by achuith@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-08-28 22:21:41]

Dry run: CQ is trying da patch.

Follow status at:
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[achuithb, 2017-08-28 22:22:40]

achuith@chromium.org changed reviewers:
+ nednguyen@google.com

[achuithb, 2017-08-28 22:22:40]

Ned will be back in a few days and he ought to take a look at this.

[achuithb, 2017-08-28 22:24:02]

On 2017/08/28 22:22:40, achuithb wrote:
> Ned will be back in a few days and he ought to take a look at this.

How does this work with the rest of the tests? You also want the chromeos
disabled tests, right?

[commit-bot: I haz the power, 2017-08-28 22:25:08]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-08-28 22:25:09]

Dry run: Try jobs failed on following builders:
  Catapult Presubmit on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Pr...)

[jonross, 2017-08-28 22:38:54]

On 2017/08/28 22:24:02, achuithb wrote:
> On 2017/08/28 22:22:40, achuithb wrote:
> > Ned will be back in a few days and he ought to take a look at this.
> 
> How does this work with the rest of the tests? You also want the chromeos
> disabled tests, right?

This leads to telemetry/telemetry/decorators.py:_PlatformAttributes having a
different set of attributes for matching.

Currently the linux-chromeos config leads to: ['exact', 'linux', 'trusty']
After this change the attributes are:         ['exact', 'linux',
'linux-chromeos']

Which allows for OS Version filtering as seen in page_run_end_to_end_unittest.py
This is similar to how some tests are already disabled for certain versions,
such as 'snowleopard'

Currently there are only 2-3 tests that are disabled for chromeos that are
failing in the linux-chromeos config. I was thinking that if the others are
working we might as well keep the coverage.

[jonross, 2017-08-28 22:40:03]

Description was changed from

==========
Add --override-os-version to telemetry for decorators

We wish to test telemetry versus linux chromeos builds. This leads to running
a linux configuration for telemetry, with some of the chromeos UI software
running within Chrome.

Telemetry already marks failing tests on an OS basis, with some known failures
for chromeos. However it cannot detect this mixed configuration and ends up
running those tests.

This change adds a new command line flag, --override-os-version which allows
for specifying sub versions like 'linux-chromeos'

This can then be used by decorators to filter out failing tests. We will use
this flag in the configurations of trybots running linux-chromeos

TEST=ran ActualPageRunEndToEndTests.testTrafficSettings locally, which is known
to fail, it now filters out in this config
BUG=758065
==========

to

==========
Add --override-os-version to telemetry for decorators

We wish to test telemetry versus linux chromeos builds. This leads to running
a linux configuration for telemetry, with some of the chromeos UI software
running within Chrome.

Telemetry already marks failing tests on an OS basis, with some known failures
for chromeos. However it cannot detect this mixed configuration and ends up
running those tests.

This change adds a new command line flag, --override-os-version which allows
for specifying sub versions like 'linux-chromeos'

This can then be used by decorators to filter out failing tests. We will use
this flag in the configurations of trybots running linux-chromeos

TEST=ran ActualPageRunEndToEndTests.testTrafficSettings locally, which is known
to fail, it now filters out in this config
BUG=chromium:758065
==========

[achuithb, 2017-08-28 22:54:44]

On 2017/08/28 22:38:54, jonross wrote:
> On 2017/08/28 22:24:02, achuithb wrote:
> > On 2017/08/28 22:22:40, achuithb wrote:
> > > Ned will be back in a few days and he ought to take a look at this.
> > 
> > How does this work with the rest of the tests? You also want the chromeos
> > disabled tests, right?
> 
> This leads to telemetry/telemetry/decorators.py:_PlatformAttributes having a
> different set of attributes for matching.
> 
> Currently the linux-chromeos config leads to: ['exact', 'linux', 'trusty']
> After this change the attributes are:         ['exact', 'linux',
> 'linux-chromeos']
> 
> Which allows for OS Version filtering as seen in
page_run_end_to_end_unittest.py
> This is similar to how some tests are already disabled for certain versions,
> such as 'snowleopard'
> 
> Currently there are only 2-3 tests that are disabled for chromeos that are
> failing in the linux-chromeos config. I was thinking that if the others are
> working we might as well keep the coverage.

Great, seems like you have a stronger case if this is already used for other
platforms.

Ideally we'd detect linux-chromeos automatically, but that seems a bit hard.

Anyway, I think we have to wait for Ned on this.

[nednguyen, 2017-08-31 12:56:39]

On 2017/08/28 22:54:44, achuithb wrote:
> On 2017/08/28 22:38:54, jonross wrote:
> > On 2017/08/28 22:24:02, achuithb wrote:
> > > On 2017/08/28 22:22:40, achuithb wrote:
> > > > Ned will be back in a few days and he ought to take a look at this.
> > > 
> > > How does this work with the rest of the tests? You also want the chromeos
> > > disabled tests, right?
> > 
> > This leads to telemetry/telemetry/decorators.py:_PlatformAttributes having a
> > different set of attributes for matching.
> > 
> > Currently the linux-chromeos config leads to: ['exact', 'linux', 'trusty']
> > After this change the attributes are:         ['exact', 'linux',
> > 'linux-chromeos']
> > 
> > Which allows for OS Version filtering as seen in
> page_run_end_to_end_unittest.py
> > This is similar to how some tests are already disabled for certain versions,
> > such as 'snowleopard'
> > 
> > Currently there are only 2-3 tests that are disabled for chromeos that are
> > failing in the linux-chromeos config. I was thinking that if the others are
> > working we might as well keep the coverage.
> 
> Great, seems like you have a stronger case if this is already used for other
> platforms.
> 
> Ideally we'd detect linux-chromeos automatically, but that seems a bit hard.
> 
> Anyway, I think we have to wait for Ned on this.

How does ChromeOS test run on linux? If it's not using cros_platform_backend in
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry...,
nothing would work, right?

[jonross, 2017-08-31 13:23:44]

On 2017/08/31 12:56:39, nednguyen wrote:
> On 2017/08/28 22:54:44, achuithb wrote:
> > On 2017/08/28 22:38:54, jonross wrote:
> > > On 2017/08/28 22:24:02, achuithb wrote:
> > > > On 2017/08/28 22:22:40, achuithb wrote:
> > > > > Ned will be back in a few days and he ought to take a look at this.
> > > > 
> > > > How does this work with the rest of the tests? You also want the
chromeos
> > > > disabled tests, right?
> > > 
> > > This leads to telemetry/telemetry/decorators.py:_PlatformAttributes having
a
> > > different set of attributes for matching.
> > > 
> > > Currently the linux-chromeos config leads to: ['exact', 'linux', 'trusty']
> > > After this change the attributes are:         ['exact', 'linux',
> > > 'linux-chromeos']
> > > 
> > > Which allows for OS Version filtering as seen in
> > page_run_end_to_end_unittest.py
> > > This is similar to how some tests are already disabled for certain
versions,
> > > such as 'snowleopard'
> > > 
> > > Currently there are only 2-3 tests that are disabled for chromeos that are
> > > failing in the linux-chromeos config. I was thinking that if the others
are
> > > working we might as well keep the coverage.
> > 
> > Great, seems like you have a stronger case if this is already used for other
> > platforms.
> > 
> > Ideally we'd detect linux-chromeos automatically, but that seems a bit hard.
> > 
> > Anyway, I think we have to wait for Ned on this.
> 
> How does ChromeOS test run on linux? If it's not using cros_platform_backend
in
>
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry...,
> nothing would work, right?

I spoke with Achuith about this. The cros_platform_backend has certain system
requirements which cannot be met on linux.

So when running linux-chromiumos we will actually want the
linux_platform_backend.

However while in this configuration there are a few tests which are consistently
failing. We are looking to disable those.

[nednguyen, 2017-08-31 15:19:47]

On 2017/08/31 13:23:44, jonross wrote:
> On 2017/08/31 12:56:39, nednguyen wrote:
> > On 2017/08/28 22:54:44, achuithb wrote:
> > > On 2017/08/28 22:38:54, jonross wrote:
> > > > On 2017/08/28 22:24:02, achuithb wrote:
> > > > > On 2017/08/28 22:22:40, achuithb wrote:
> > > > > > Ned will be back in a few days and he ought to take a look at this.
> > > > > 
> > > > > How does this work with the rest of the tests? You also want the
> chromeos
> > > > > disabled tests, right?
> > > > 
> > > > This leads to telemetry/telemetry/decorators.py:_PlatformAttributes
having
> a
> > > > different set of attributes for matching.
> > > > 
> > > > Currently the linux-chromeos config leads to: ['exact', 'linux',
'trusty']
> > > > After this change the attributes are:         ['exact', 'linux',
> > > > 'linux-chromeos']
> > > > 
> > > > Which allows for OS Version filtering as seen in
> > > page_run_end_to_end_unittest.py
> > > > This is similar to how some tests are already disabled for certain
> versions,
> > > > such as 'snowleopard'
> > > > 
> > > > Currently there are only 2-3 tests that are disabled for chromeos that
are
> > > > failing in the linux-chromeos config. I was thinking that if the others
> are
> > > > working we might as well keep the coverage.
> > > 
> > > Great, seems like you have a stronger case if this is already used for
other
> > > platforms.
> > > 
> > > Ideally we'd detect linux-chromeos automatically, but that seems a bit
hard.
> > > 
> > > Anyway, I think we have to wait for Ned on this.
> > 
> > How does ChromeOS test run on linux? If it's not using cros_platform_backend
> in
> >
>
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry...,
> > nothing would work, right?
> 
> I spoke with Achuith about this. The cros_platform_backend has certain system
> requirements which cannot be met on linux.
> 
> So when running linux-chromiumos we will actually want the
> linux_platform_backend.
> 
> However while in this configuration there are a few tests which are
consistently
> failing. We are looking to disable those.

It worths noting that Telemetry unittest suite run a mixed of purely unittest
that do not need the browser & integration tests that actually launch browser &
does stuff. I suspect that without fixing cros_browser_backend to be able to
take linux_platform_backend, the test suite you run will end up having all the
integration tests disabled.
From my experience, the integration tests are the one that catch most bugs in
production.

[jonross, 2017-08-31 16:41:08]

On 2017/08/31 15:19:47, nednguyen wrote:
> On 2017/08/31 13:23:44, jonross wrote:
> > On 2017/08/31 12:56:39, nednguyen wrote:
> > > On 2017/08/28 22:54:44, achuithb wrote:
> > > > On 2017/08/28 22:38:54, jonross wrote:
> > > > > On 2017/08/28 22:24:02, achuithb wrote:
> > > > > > On 2017/08/28 22:22:40, achuithb wrote:
> > > > > > > Ned will be back in a few days and he ought to take a look at
this.
> > > > > > 
> > > > > > How does this work with the rest of the tests? You also want the
> > chromeos
> > > > > > disabled tests, right?
> > > > > 
> > > > > This leads to telemetry/telemetry/decorators.py:_PlatformAttributes
> having
> > a
> > > > > different set of attributes for matching.
> > > > > 
> > > > > Currently the linux-chromeos config leads to: ['exact', 'linux',
> 'trusty']
> > > > > After this change the attributes are:         ['exact', 'linux',
> > > > > 'linux-chromeos']
> > > > > 
> > > > > Which allows for OS Version filtering as seen in
> > > > page_run_end_to_end_unittest.py
> > > > > This is similar to how some tests are already disabled for certain
> > versions,
> > > > > such as 'snowleopard'
> > > > > 
> > > > > Currently there are only 2-3 tests that are disabled for chromeos that
> are
> > > > > failing in the linux-chromeos config. I was thinking that if the
others
> > are
> > > > > working we might as well keep the coverage.
> > > > 
> > > > Great, seems like you have a stronger case if this is already used for
> other
> > > > platforms.
> > > > 
> > > > Ideally we'd detect linux-chromeos automatically, but that seems a bit
> hard.
> > > > 
> > > > Anyway, I think we have to wait for Ned on this.
> > > 
> > > How does ChromeOS test run on linux? If it's not using
cros_platform_backend
> > in
> > >
> >
>
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry...,
> > > nothing would work, right?
> > 
> > I spoke with Achuith about this. The cros_platform_backend has certain
system
> > requirements which cannot be met on linux.
> > 
> > So when running linux-chromiumos we will actually want the
> > linux_platform_backend.
> > 
> > However while in this configuration there are a few tests which are
> consistently
> > failing. We are looking to disable those.
> 
> It worths noting that Telemetry unittest suite run a mixed of purely unittest
> that do not need the browser & integration tests that actually launch browser
&
> does stuff. I suspect that without fixing cros_browser_backend to be able to
> take linux_platform_backend, the test suite you run will end up having all the
> integration tests disabled.
> From my experience, the integration tests are the one that catch most bugs in
> production.

Are there no integration tests ran by linux_platform_backend?

[nednguyen, 2017-08-31 16:42:00]

On 2017/08/31 16:41:08, jonross wrote:
> On 2017/08/31 15:19:47, nednguyen wrote:
> > On 2017/08/31 13:23:44, jonross wrote:
> > > On 2017/08/31 12:56:39, nednguyen wrote:
> > > > On 2017/08/28 22:54:44, achuithb wrote:
> > > > > On 2017/08/28 22:38:54, jonross wrote:
> > > > > > On 2017/08/28 22:24:02, achuithb wrote:
> > > > > > > On 2017/08/28 22:22:40, achuithb wrote:
> > > > > > > > Ned will be back in a few days and he ought to take a look at
> this.
> > > > > > > 
> > > > > > > How does this work with the rest of the tests? You also want the
> > > chromeos
> > > > > > > disabled tests, right?
> > > > > > 
> > > > > > This leads to telemetry/telemetry/decorators.py:_PlatformAttributes
> > having
> > > a
> > > > > > different set of attributes for matching.
> > > > > > 
> > > > > > Currently the linux-chromeos config leads to: ['exact', 'linux',
> > 'trusty']
> > > > > > After this change the attributes are:         ['exact', 'linux',
> > > > > > 'linux-chromeos']
> > > > > > 
> > > > > > Which allows for OS Version filtering as seen in
> > > > > page_run_end_to_end_unittest.py
> > > > > > This is similar to how some tests are already disabled for certain
> > > versions,
> > > > > > such as 'snowleopard'
> > > > > > 
> > > > > > Currently there are only 2-3 tests that are disabled for chromeos
that
> > are
> > > > > > failing in the linux-chromeos config. I was thinking that if the
> others
> > > are
> > > > > > working we might as well keep the coverage.
> > > > > 
> > > > > Great, seems like you have a stronger case if this is already used for
> > other
> > > > > platforms.
> > > > > 
> > > > > Ideally we'd detect linux-chromeos automatically, but that seems a bit
> > hard.
> > > > > 
> > > > > Anyway, I think we have to wait for Ned on this.
> > > > 
> > > > How does ChromeOS test run on linux? If it's not using
> cros_platform_backend
> > > in
> > > >
> > >
> >
>
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry...,
> > > > nothing would work, right?
> > > 
> > > I spoke with Achuith about this. The cros_platform_backend has certain
> system
> > > requirements which cannot be met on linux.
> > > 
> > > So when running linux-chromiumos we will actually want the
> > > linux_platform_backend.
> > > 
> > > However while in this configuration there are a few tests which are
> > consistently
> > > failing. We are looking to disable those.
> > 
> > It worths noting that Telemetry unittest suite run a mixed of purely
unittest
> > that do not need the browser & integration tests that actually launch
browser
> &
> > does stuff. I suspect that without fixing cros_browser_backend to be able to
> > take linux_platform_backend, the test suite you run will end up having all
the
> > integration tests disabled.
> > From my experience, the integration tests are the one that catch most bugs
in
> > production.
> 
> Are there no integration tests ran by linux_platform_backend?

There are. My point is you can only get the working by making sure that
cros_browser_backend can take in linux_platform_backend

[jonross, 2017-08-31 16:56:34]

On 2017/08/31 16:42:00, nednguyen wrote:
> On 2017/08/31 16:41:08, jonross wrote:
> > On 2017/08/31 15:19:47, nednguyen wrote:
> > > On 2017/08/31 13:23:44, jonross wrote:
> > > > On 2017/08/31 12:56:39, nednguyen wrote:
> > > > > On 2017/08/28 22:54:44, achuithb wrote:
> > > > > > On 2017/08/28 22:38:54, jonross wrote:
> > > > > > > On 2017/08/28 22:24:02, achuithb wrote:
> > > > > > > > On 2017/08/28 22:22:40, achuithb wrote:
> > > > > > > > > Ned will be back in a few days and he ought to take a look at
> > this.
> > > > > > > > 
> > > > > > > > How does this work with the rest of the tests? You also want the
> > > > chromeos
> > > > > > > > disabled tests, right?
> > > > > > > 
> > > > > > > This leads to
telemetry/telemetry/decorators.py:_PlatformAttributes
> > > having
> > > > a
> > > > > > > different set of attributes for matching.
> > > > > > > 
> > > > > > > Currently the linux-chromeos config leads to: ['exact', 'linux',
> > > 'trusty']
> > > > > > > After this change the attributes are:         ['exact', 'linux',
> > > > > > > 'linux-chromeos']
> > > > > > > 
> > > > > > > Which allows for OS Version filtering as seen in
> > > > > > page_run_end_to_end_unittest.py
> > > > > > > This is similar to how some tests are already disabled for certain
> > > > versions,
> > > > > > > such as 'snowleopard'
> > > > > > > 
> > > > > > > Currently there are only 2-3 tests that are disabled for chromeos
> that
> > > are
> > > > > > > failing in the linux-chromeos config. I was thinking that if the
> > others
> > > > are
> > > > > > > working we might as well keep the coverage.
> > > > > > 
> > > > > > Great, seems like you have a stronger case if this is already used
for
> > > other
> > > > > > platforms.
> > > > > > 
> > > > > > Ideally we'd detect linux-chromeos automatically, but that seems a
bit
> > > hard.
> > > > > > 
> > > > > > Anyway, I think we have to wait for Ned on this.
> > > > > 
> > > > > How does ChromeOS test run on linux? If it's not using
> > cros_platform_backend
> > > > in
> > > > >
> > > >
> > >
> >
>
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry...,
> > > > > nothing would work, right?
> > > > 
> > > > I spoke with Achuith about this. The cros_platform_backend has certain
> > system
> > > > requirements which cannot be met on linux.
> > > > 
> > > > So when running linux-chromiumos we will actually want the
> > > > linux_platform_backend.
> > > > 
> > > > However while in this configuration there are a few tests which are
> > > consistently
> > > > failing. We are looking to disable those.
> > > 
> > > It worths noting that Telemetry unittest suite run a mixed of purely
> unittest
> > > that do not need the browser & integration tests that actually launch
> browser
> > &
> > > does stuff. I suspect that without fixing cros_browser_backend to be able
to
> > > take linux_platform_backend, the test suite you run will end up having all
> the
> > > integration tests disabled.
> > > From my experience, the integration tests are the one that catch most bugs
> in
> > > production.
> > 
> > Are there no integration tests ran by linux_platform_backend?
> 
> There are. My point is you can only get the working by making sure that
> cros_browser_backend can take in linux_platform_backend

How would I go about finding out which tests only run with cros_browser_backend,
vs those that also run with linux_platform_backend? Are they all denoted with
@decorators.Enabled/Disabled?

Since there are at least some integration tests running in the
linux_platform_backend then that should be a good starting point. Getting
linux-chromoes coverage of those tests on the CQ will be great, as currently
there is no coverage.

We'll still maintain the slower testing cycle of cros_platform_backend. We can
then look at what is needed to cros_browser_backend to accept a
linux_platform_backend, while achuith@ and bccheng@ continue to work on getting
VM and actual devices up on the CQ.

[nednguyen, 2017-08-31 17:02:22]

On 2017/08/31 16:56:34, jonross wrote:
> On 2017/08/31 16:42:00, nednguyen wrote:
> > On 2017/08/31 16:41:08, jonross wrote:
> > > On 2017/08/31 15:19:47, nednguyen wrote:
> > > > On 2017/08/31 13:23:44, jonross wrote:
> > > > > On 2017/08/31 12:56:39, nednguyen wrote:
> > > > > > On 2017/08/28 22:54:44, achuithb wrote:
> > > > > > > On 2017/08/28 22:38:54, jonross wrote:
> > > > > > > > On 2017/08/28 22:24:02, achuithb wrote:
> > > > > > > > > On 2017/08/28 22:22:40, achuithb wrote:
> > > > > > > > > > Ned will be back in a few days and he ought to take a look
at
> > > this.
> > > > > > > > > 
> > > > > > > > > How does this work with the rest of the tests? You also want
the
> > > > > chromeos
> > > > > > > > > disabled tests, right?
> > > > > > > > 
> > > > > > > > This leads to
> telemetry/telemetry/decorators.py:_PlatformAttributes
> > > > having
> > > > > a
> > > > > > > > different set of attributes for matching.
> > > > > > > > 
> > > > > > > > Currently the linux-chromeos config leads to: ['exact', 'linux',
> > > > 'trusty']
> > > > > > > > After this change the attributes are:         ['exact', 'linux',
> > > > > > > > 'linux-chromeos']
> > > > > > > > 
> > > > > > > > Which allows for OS Version filtering as seen in
> > > > > > > page_run_end_to_end_unittest.py
> > > > > > > > This is similar to how some tests are already disabled for
certain
> > > > > versions,
> > > > > > > > such as 'snowleopard'
> > > > > > > > 
> > > > > > > > Currently there are only 2-3 tests that are disabled for
chromeos
> > that
> > > > are
> > > > > > > > failing in the linux-chromeos config. I was thinking that if the
> > > others
> > > > > are
> > > > > > > > working we might as well keep the coverage.
> > > > > > > 
> > > > > > > Great, seems like you have a stronger case if this is already used
> for
> > > > other
> > > > > > > platforms.
> > > > > > > 
> > > > > > > Ideally we'd detect linux-chromeos automatically, but that seems a
> bit
> > > > hard.
> > > > > > > 
> > > > > > > Anyway, I think we have to wait for Ned on this.
> > > > > > 
> > > > > > How does ChromeOS test run on linux? If it's not using
> > > cros_platform_backend
> > > > > in
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry...,
> > > > > > nothing would work, right?
> > > > > 
> > > > > I spoke with Achuith about this. The cros_platform_backend has certain
> > > system
> > > > > requirements which cannot be met on linux.
> > > > > 
> > > > > So when running linux-chromiumos we will actually want the
> > > > > linux_platform_backend.
> > > > > 
> > > > > However while in this configuration there are a few tests which are
> > > > consistently
> > > > > failing. We are looking to disable those.
> > > > 
> > > > It worths noting that Telemetry unittest suite run a mixed of purely
> > unittest
> > > > that do not need the browser & integration tests that actually launch
> > browser
> > > &
> > > > does stuff. I suspect that without fixing cros_browser_backend to be
able
> to
> > > > take linux_platform_backend, the test suite you run will end up having
all
> > the
> > > > integration tests disabled.
> > > > From my experience, the integration tests are the one that catch most
bugs
> > in
> > > > production.
> > > 
> > > Are there no integration tests ran by linux_platform_backend?
> > 
> > There are. My point is you can only get the working by making sure that
> > cros_browser_backend can take in linux_platform_backend
> 
> How would I go about finding out which tests only run with
cros_browser_backend,
> vs those that also run with linux_platform_backend? Are they all denoted with
> @decorators.Enabled/Disabled?
> 
> Since there are at least some integration tests running in the
> linux_platform_backend then that should be a good starting point. Getting
> linux-chromoes coverage of those tests on the CQ will be great, as currently
> there is no coverage.

Those tests should already be run on linux_chromium_rel_ng on CQ, so you are not
adding more coverage by just enabling them for linux-chromeOS
> 
> We'll still maintain the slower testing cycle of cros_platform_backend. We can
> then look at what is needed to cros_browser_backend to accept a
> linux_platform_backend, while achuith@ and bccheng@ continue to work on
getting
> VM and actual devices up on the CQ.

[jonross, 2017-08-31 17:18:53]

> Those tests should already be run on linux_chromium_rel_ng on CQ, so you are
not
> adding more coverage by just enabling them for linux-chromeOS

I disagree. In this CL I expand the disabling of a consistently failing test.
While originally failing on ChromeOS, that it also fails on linux-ChromeOS
points to aspects of the extra UI in ChromeOS having an impact.

We currently lack coverage of this on the CQ.

In the short term we'd gain this coverage of linux-chromium + ChromeOS ui (ash).

We plan to follow up on this my adding versions where the various services of
Chrome run in separate processes. This includes pulling ash out.

This will then allow us to catch performance regressions at CQ time.

[jonross, 2017-08-31 18:56:49]

On 2017/08/31 17:18:53, jonross wrote:
> > Those tests should already be run on linux_chromium_rel_ng on CQ, so you are
> not
> > adding more coverage by just enabling them for linux-chromeOS
> 
> I disagree. In this CL I expand the disabling of a consistently failing test.
> While originally failing on ChromeOS, that it also fails on linux-ChromeOS
> points to aspects of the extra UI in ChromeOS having an impact.
> 
> We currently lack coverage of this on the CQ.
> 
> In the short term we'd gain this coverage of linux-chromium + ChromeOS ui
(ash).
> 
> We plan to follow up on this my adding versions where the various services of
> Chrome run in separate processes. This includes pulling ash out.
> 
> This will then allow us to catch performance regressions at CQ time.

Closing this review as it is no longer needed. The associated bug will be
updated accordingly