Adding SwiftShader to the DEPS file

Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source
and can now be pulled with the Chromium checkout. The size
of the SwiftShader library is roughly 162 Mb, currently.
Note that this will eventually allow us to remove MESA,
which takes approximately 66 Mb, so the net gain should be
under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are
found.

This will not change the state of SwiftShader as a
component in Chrome and it will not be shipped with Chrome,
only downloaded as a component on demand.

Also note that this cl does not enable building
SwiftShader, it only allows the SwiftShader library to be
pulled, to make sure that everyone agrees with this first
step.

BUG=630728
Committed: https://crrev.com/7d27602ad344f04dcb94ce5ead7950a0cfc1cd02
Cr-Commit-Position: refs/heads/master@{#410359}

[sugoi1, 2016-07-21 16:57:45]

Description was changed from

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source and
can now be pulled with the Chromium checkout. The size of the
SwiftShader library is roughly 162 Mb, currently.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are found.

This will not change the state of SwiftShader as a component
in Chrome and it will not be shipped with Chrome, only
downloaded as a component on demand.

Also note that this cl does not enable building SwiftShader,
it only allows the SwiftShader library to be pulled,
to make sure that everyone agrees with this first step.
==========

to

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source and
can now be pulled with the Chromium checkout. The size of the
SwiftShader library is roughly 162 Mb, currently.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are found.

This will not change the state of SwiftShader as a component
in Chrome and it will not be shipped with Chrome, only
downloaded as a component on demand.

Also note that this cl does not enable building SwiftShader,
it only allows the SwiftShader library to be pulled,
to make sure that everyone agrees with this first step.
==========

[sugoi1, 2016-07-21 16:57:45]

sugoi@chromium.org changed reviewers:
+ kbr@chromium.org, tandrii@chromium.org

[sugoi1, 2016-07-22 17:18:15]

@kbr: Please add anyone who should be approving this cl. I'm not certain who
approves adding libraries to chromium.

[Ken Russell (switch to Gerrit), 2016-07-22 18:37:43]

kbr@chromium.org changed reviewers:
+ dglazkov@chromium.org, jam@chromium.org, sky@chromium.org

[Ken Russell (switch to Gerrit), 2016-07-22 18:37:43]

A few things:

- Please file a bug and reference it, and include the background information and
rationale in the bug. (In case this has to be reverted, it will provide a
central location for the commits and reverts.) The scope of the bug can be as
broad as you want -- pull SwiftShader? Build and run it on multiple platforms?
- It would be good to mention in the commit message that the intent is for this
to replace src/third_party/mesa/ for testing purposes, which will save 74 MB, so
the net overhead of SwiftShader will be less than 162 MB.
- It will be beneficial to set up automatic branching for the SwiftShader
repository, so that each Chromium branch generates a corresponding SwiftShader
branch so fixes can be easily cherry-picked to Chrome Beta, Dev, etc. Talk with
shannonwoods@ who has experience setting that up for ANGLE.

LGTM from my standpoint. CC'ing a few OWNERS of src/third_party. I compared to
the pdfium repository, and it looks like SwiftShader has all the necessary
license files and documentation.

[sugoi1, 2016-07-22 20:29:34]

Description was changed from

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source and
can now be pulled with the Chromium checkout. The size of the
SwiftShader library is roughly 162 Mb, currently.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are found.

This will not change the state of SwiftShader as a component
in Chrome and it will not be shipped with Chrome, only
downloaded as a component on demand.

Also note that this cl does not enable building SwiftShader,
it only allows the SwiftShader library to be pulled,
to make sure that everyone agrees with this first step.
==========

to

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source and can now be pulled
with the Chromium checkout. The size of the SwiftShader library is roughly 162
Mb, currently.
Note that this will eventually allow us to remove MESA, which takes
approximately 66 Mb, so the net gain should be under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are found.

This will not change the state of SwiftShader as a component in Chrome and it
will not be shipped with Chrome, only downloaded as a component on demand.

Also note that this cl does not enable building SwiftShader, it only allows the
SwiftShader library to be pulled, to make sure that everyone agrees with this
first step.

BUG=630728
==========

[sugoi1, 2016-07-22 20:46:47]

Description was changed from

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source and can now be pulled
with the Chromium checkout. The size of the SwiftShader library is roughly 162
Mb, currently.
Note that this will eventually allow us to remove MESA, which takes
approximately 66 Mb, so the net gain should be under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are found.

This will not change the state of SwiftShader as a component in Chrome and it
will not be shipped with Chrome, only downloaded as a component on demand.

Also note that this cl does not enable building SwiftShader, it only allows the
SwiftShader library to be pulled, to make sure that everyone agrees with this
first step.

BUG=630728
==========

to

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source
and can now be pulled with the Chromium checkout. The size
of the SwiftShader library is roughly 162 Mb, currently.
Note that this will eventually allow us to remove MESA,
which takes approximately 66 Mb, so the net gain should be
under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are
found.

This will not change the state of SwiftShader as a
component in Chrome and it will not be shipped with Chrome,
only downloaded as a component on demand.

Also note that this cl does not enable building
SwiftShader, it only allows the SwiftShader library to be
pulled, to make sure that everyone agrees with this first
step.

BUG=630728
==========

[jam, 2016-07-22 20:47:59]

I just fetched this. More than half the size is for llvm and powervr sdk. We
already have the former in third_party/, is it possible to use that one? do we
need the powervr sdk? looking through the includes, I didn't find any.

[sugoi1, 2016-07-22 21:13:30]

I think llvm accounts for about 90Mb of the 162Mb, and PowerVR SDK is about
25Mb,
so definitely more than half the size. Unfortunately, SwiftShader relies on a
specific
older version of llvm, so we need that for it to work. We are looking at
replacements
from llvm for SwiftShader's JIT compiler, but that's a long term project and
won't be
completed anytime soon. As for the PowerVR SDK, we use it only for test
purposes,
so it's not an essential part of the library, but is there a way to pull
SwiftShader only
partially? Or are you suggesting we rip PowerVR SDK out of SwiftShader? For the
latter, I'd need capn@'s approval, when he comes back from his paternity leave
in
2 weeks.

On 2016/07/22 20:47:59, jam wrote:
> I just fetched this. More than half the size is for llvm and powervr sdk. We
> already have the former in third_party/, is it possible to use that one? do we
> need the powervr sdk? looking through the includes, I didn't find any.

[Ken Russell (switch to Gerrit), 2016-07-22 22:51:20]

On 2016/07/22 21:13:30, sugoi1 wrote:
> On 2016/07/22 20:47:59, jam wrote:
> > I just fetched this. More than half the size is for llvm and powervr sdk. We
> > already have the former in third_party/, is it possible to use that one? do
we
> > need the powervr sdk? looking through the includes, I didn't find any.
> I think llvm accounts for about 90Mb of the 162Mb, and PowerVR SDK is about
> 25Mb,
> so definitely more than half the size. Unfortunately, SwiftShader relies on a
> specific
> older version of llvm, so we need that for it to work. We are looking at
> replacements
> from llvm for SwiftShader's JIT compiler, but that's a long term project and
> won't be
> completed anytime soon. As for the PowerVR SDK, we use it only for test
> purposes,
> so it's not an essential part of the library, but is there a way to pull
> SwiftShader only
> partially? Or are you suggesting we rip PowerVR SDK out of SwiftShader? For
the
> latter, I'd need capn@'s approval, when he comes back from his paternity leave
> in
> 2 weeks.

You could factor the PowerVR SDK into its own repository and pull it in via DEPS
when SwiftShader is checked out standalone. That would mean developers would
check out SwiftShader via gclient rather than git directly, but this strategy
has worked well for ANGLE. See
https://chromium.googlesource.com/angle/angle/+/master/doc/DevSetup.md and some
of the GYP files in ANGLE. (It looks like the GN files currently only support
checking out ANGLE inside the Chromium workspace, but I think that's because GN
hasn't been fully decoupled from Chromium yet.)

At this point I'm not sure of the advantage of that since you'd still have the
(potentially large) git history of the PowerVR SDK in the SwiftShader git repo.
You might want to consider filing a ticket to delete it and reconstruct the
repo's history.

[jam, 2016-07-23 00:04:34]

ok, i defer to you guys. rubberstamp lgtm

[tandrii(chromium), 2016-07-26 11:20:18]

https://codereview.chromium.org/2168143003/diff/1/DEPS
File DEPS (right):

https://codereview.chromium.org/2168143003/diff/1/DEPS#newcode109
DEPS:109: 'swiftshader.googlesource.com',
Infrastructure needs to deploy credentials for this host. ~2 hours after this
lands https://chromereviews.googleplex.com/478767014/, you can issue CQ dry run
and see what happens.

[sugoi1, 2016-07-27 14:37:40]

The CQ bit was checked by sugoi@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-07-27 14:37:55]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-07-27 15:46:04]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-07-27 15:46:05]

Dry run: Try jobs failed on following builders:
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sugoi1, 2016-08-04 20:40:09]

sugoi@chromium.org changed reviewers:
+ phajdan@google.com

[sugoi1, 2016-08-05 17:36:49]

sugoi@chromium.org changed reviewers:
+ capn@chromium.org

[sugoi, 2016-08-05 18:10:16]

The CQ bit was checked by sugoi@google.com to run a CQ dry run

[commit-bot: I haz the power, 2016-08-05 18:10:57]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-05 18:17:09]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-05 18:17:09]

Dry run: Try jobs failed on following builders:
  linux_chromium_asan_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sugoi, 2016-08-05 22:10:14]

The CQ bit was checked by sugoi@google.com to run a CQ dry run

[commit-bot: I haz the power, 2016-08-05 22:10:38]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-05 23:13:30]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-05 23:13:32]

Dry run: This issue passed the CQ dry run.

[sugoi, 2016-08-06 03:01:39]

sugoi@google.com changed reviewers:
+ phajdan.jr@chromium.org

[sugoi, 2016-08-06 03:02:46]

Adding phajdan for the small checklicenses.py change.

[Paweł Hajdan Jr., 2016-08-08 08:01:58]

LGTM

[capn, 2016-08-08 14:10:06]

The CQ bit was checked by capn@chromium.org

[capn, 2016-08-08 14:10:06]

The patchset sent to the CQ was uploaded after l-g-t-m from kbr@chromium.org,
jam@chromium.org
Link to the patchset: https://codereview.chromium.org/2168143003/#ps20001
(title: "Added license check exception for swiftshader")

[commit-bot: I haz the power, 2016-08-08 14:10:27]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-08 15:13:03]

Description was changed from

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source
and can now be pulled with the Chromium checkout. The size
of the SwiftShader library is roughly 162 Mb, currently.
Note that this will eventually allow us to remove MESA,
which takes approximately 66 Mb, so the net gain should be
under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are
found.

This will not change the state of SwiftShader as a
component in Chrome and it will not be shipped with Chrome,
only downloaded as a component on demand.

Also note that this cl does not enable building
SwiftShader, it only allows the SwiftShader library to be
pulled, to make sure that everyone agrees with this first
step.

BUG=630728
==========

to

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source
and can now be pulled with the Chromium checkout. The size
of the SwiftShader library is roughly 162 Mb, currently.
Note that this will eventually allow us to remove MESA,
which takes approximately 66 Mb, so the net gain should be
under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are
found.

This will not change the state of SwiftShader as a
component in Chrome and it will not be shipped with Chrome,
only downloaded as a component on demand.

Also note that this cl does not enable building
SwiftShader, it only allows the SwiftShader library to be
pulled, to make sure that everyone agrees with this first
step.

BUG=630728
==========

[commit-bot: I haz the power, 2016-08-08 15:13:04]

Committed patchset #2 (id:20001)

[commit-bot: I haz the power, 2016-08-08 15:15:06]

Description was changed from

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source
and can now be pulled with the Chromium checkout. The size
of the SwiftShader library is roughly 162 Mb, currently.
Note that this will eventually allow us to remove MESA,
which takes approximately 66 Mb, so the net gain should be
under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are
found.

This will not change the state of SwiftShader as a
component in Chrome and it will not be shipped with Chrome,
only downloaded as a component on demand.

Also note that this cl does not enable building
SwiftShader, it only allows the SwiftShader library to be
pulled, to make sure that everyone agrees with this first
step.

BUG=630728
==========

to

==========
Adding SwiftShader to the DEPS file

The SwiftShader library has recently been made open source
and can now be pulled with the Chromium checkout. The size
of the SwiftShader library is roughly 162 Mb, currently.
Note that this will eventually allow us to remove MESA,
which takes approximately 66 Mb, so the net gain should be
under 100Mb in total.

Pulling SwiftShader along with Chromium will allow bots to
build SwiftShader, perform tests with the SwiftShader
library and have quick turnaround time when issues are
found.

This will not change the state of SwiftShader as a
component in Chrome and it will not be shipped with Chrome,
only downloaded as a component on demand.

Also note that this cl does not enable building
SwiftShader, it only allows the SwiftShader library to be
pulled, to make sure that everyone agrees with this first
step.

BUG=630728

Committed: https://crrev.com/7d27602ad344f04dcb94ce5ead7950a0cfc1cd02
Cr-Commit-Position: refs/heads/master@{#410359}
==========

[commit-bot: I haz the power, 2016-08-08 15:15:07]

Patchset 2 (id:??) landed as
https://crrev.com/7d27602ad344f04dcb94ce5ead7950a0cfc1cd02
Cr-Commit-Position: refs/heads/master@{#410359}

[stevenjb, 2016-08-08 19:18:05]

On 2016/08/08 15:15:07, commit-bot: I haz the power wrote:
> Patchset 2 (id:??) landed as
> https://crrev.com/7d27602ad344f04dcb94ce5ead7950a0cfc1cd02
> Cr-Commit-Position: refs/heads/master@{#410359}

This is causing problems on Chrome OS. When I do a simple chrome build, a bunch
of untracked files get pulled into third_party/swiftshader, and I get the
following error:

In file included from ../../third_party/swiftshader/include/EGL/egl.h:39:0,
                 from ../../ui/gl/gl_bindings.h:23,
                 from ../../ui/gl/gl_egl_api_implementation.h:12,
                 from ../../ui/gl/init/gl_factory_ozone.cc:13:
../../third_party/swiftshader/include/EGL/eglplatform.h:99:22: fatal error:
X11/Xlib.h: No such file or directory
 #include <X11/Xlib.h>
                      ^
compilation terminated.

[sadrul, 2016-08-08 19:26:17]

On 2016/08/08 15:15:07, commit-bot: I haz the power wrote:
> Patchset 2 (id:??) landed as
> https://crrev.com/7d27602ad344f04dcb94ce5ead7950a0cfc1cd02
> Cr-Commit-Position: refs/heads/master@{#410359}

This is causing build failure in some bots (e.g.
https://build.chromium.org/p/chromium.linux/builders/Blimp%20Linux%20%28dbg%2...),
and locally. This change fixes the local failures: http://pastebin.com/bLB4tC3k
but I can't tell how to put that up for review. Would it be possible to revert
in the meantime?

[stevenjb, 2016-08-08 19:35:52]

On 2016/08/08 19:18:05, stevenjb wrote:
> On 2016/08/08 15:15:07, commit-bot: I haz the power wrote:
> > Patchset 2 (id:??) landed as
> > https://crrev.com/7d27602ad344f04dcb94ce5ead7950a0cfc1cd02
> > Cr-Commit-Position: refs/heads/master@{#410359}
> 
> This is causing problems on Chrome OS. When I do a simple chrome build, a
bunch
> of untracked files get pulled into third_party/swiftshader, and I get the
> following error:
> 
> In file included from ../../third_party/swiftshader/include/EGL/egl.h:39:0,
>                  from ../../ui/gl/gl_bindings.h:23,
>                  from ../../ui/gl/gl_egl_api_implementation.h:12,
>                  from ../../ui/gl/init/gl_factory_ozone.cc:13:
> ../../third_party/swiftshader/include/EGL/eglplatform.h:99:22: fatal error:
> X11/Xlib.h: No such file or directory
>  #include <X11/Xlib.h>
>                       ^
> compilation terminated.

After deleting third_party/swiftshader, then doing:

git checkout master && gclient sync --force

I have a bunch of files/directories that git sees as not part of the repo and
not ignored:

#	third_party/swiftshader/.dir-locals.el
#	third_party/swiftshader/.gitignore
#	third_party/swiftshader/AUTHORS.txt
#	third_party/swiftshader/Android.mk
#	third_party/swiftshader/BUILD.gn
#	third_party/swiftshader/CMakeLists.txt
#	third_party/swiftshader/CONTRIBUTING.txt
#	third_party/swiftshader/CONTRIBUTORS.txt
#	third_party/swiftshader/LICENSE.txt
#	third_party/swiftshader/README.md
#	third_party/swiftshader/docs/
#	third_party/swiftshader/src/
#	third_party/swiftshader/tests/
#	third_party/swiftshader/third_party/

[stevenjb, 2016-08-08 19:36:27]

On 2016/08/08 19:35:52, stevenjb wrote:
> On 2016/08/08 19:18:05, stevenjb wrote:
> > On 2016/08/08 15:15:07, commit-bot: I haz the power wrote:
> > > Patchset 2 (id:??) landed as
> > > https://crrev.com/7d27602ad344f04dcb94ce5ead7950a0cfc1cd02
> > > Cr-Commit-Position: refs/heads/master@{#410359}
> > 
> > This is causing problems on Chrome OS. When I do a simple chrome build, a
> bunch
> > of untracked files get pulled into third_party/swiftshader, and I get the
> > following error:
> > 
> > In file included from ../../third_party/swiftshader/include/EGL/egl.h:39:0,
> >                  from ../../ui/gl/gl_bindings.h:23,
> >                  from ../../ui/gl/gl_egl_api_implementation.h:12,
> >                  from ../../ui/gl/init/gl_factory_ozone.cc:13:
> > ../../third_party/swiftshader/include/EGL/eglplatform.h:99:22: fatal error:
> > X11/Xlib.h: No such file or directory
> >  #include <X11/Xlib.h>
> >                       ^
> > compilation terminated.
> 
> After deleting third_party/swiftshader, then doing:
> 
> git checkout master && gclient sync --force
> 
> I have a bunch of files/directories that git sees as not part of the repo and
> not ignored:
> 
> #	third_party/swiftshader/.dir-locals.el
> #	third_party/swiftshader/.gitignore
> #	third_party/swiftshader/AUTHORS.txt
> #	third_party/swiftshader/Android.mk
> #	third_party/swiftshader/BUILD.gn
> #	third_party/swiftshader/CMakeLists.txt
> #	third_party/swiftshader/CONTRIBUTING.txt
> #	third_party/swiftshader/CONTRIBUTORS.txt
> #	third_party/swiftshader/LICENSE.txt
> #	third_party/swiftshader/README.md
> #	third_party/swiftshader/docs/
> #	third_party/swiftshader/src/
> #	third_party/swiftshader/tests/
> #	third_party/swiftshader/third_party/

Yeah, I think we need to revert. Reverting now.

[stevenjb, 2016-08-08 19:37:51]

A revert of this CL (patchset #2 id:20001) has been created in
https://codereview.chromium.org/2224093002/ by stevenjb@chromium.org.

The reason for reverting is: Failures on chrome os and elsewhere. Example Chrome
OS failure:

https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/veyron_minnie...

Linux failure:
https://build.chromium.org/p/chromium.linux/builders/Blimp%20Linux%20%28dbg%2...
.

[capn, 2016-08-08 20:19:54]

On 2016/08/08 19:37:51, stevenjb wrote:
> A revert of this CL (patchset #2 id:20001) has been created in
> https://codereview.chromium.org/2224093002/ by mailto:stevenjb@chromium.org.
> 
> The reason for reverting is: Failures on chrome os and elsewhere. Example
Chrome
> OS failure:
> 
>
https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/veyron_minnie...
> 
> Linux failure:
>
https://build.chromium.org/p/chromium.linux/builders/Blimp%20Linux%20%28dbg%2...
> .

Thanks for catching that and quickly resolving it.

This CL was just adding new source files (no new targets yet), which nothing was
supposed to depend on.
Fixing the ozone build seems like the wrong action since it shouldn't be using
those headers in the first place.
Instead I think the root of the failures is this:
https://cs.chromium.org/chromium/src/ui/gl/init/BUILD.gn?rcl=0&l=21
That include directory didn't exist any longer, but gets added again with our
DEPS change.

I'll have that line from the BUILD file removed, then try landing this again.