move html5lib code into dart svn repo

move html5lib code into dart svn repo
also gets the tests running

Changes to scripts:
* pkg/pkg.gyp: add 'third_party' to list of folders to search
* tools/publish_pkg.py: update copyright year
* tools/publish_all_pkgs.py: also include pkg/third_party

Changes to html5lib:
* pubspec.yaml -- removed versions
* README.md -- removed some historical notes
* added html5lib.status
* test/browser -- rename browser_tests to browser_test so test framework finds it
* test/parser_test.dart, test/parser_feature_test.dart -- moved dart:io test into parser_test so parser_feature_test can work in browser
* test/support.dart -- now finds the data folder relative to entry point script
* test/support.dart -- rename "pathos" import to "path"

Not changed:
* ./test/run.sh still works for testing, in addition to ./tools/test.dart

R=dgrove@google.com, ricow@google.com, sigmund@google.com

Committed: https://code.google.com/p/dart/source/detail?r=26152

[Jennifer Messerly, 2013-08-10 06:46:25]

Hello Siggi & Rico!

Rico, would you mind reviewing the test.dart and test_options.dart changes?

Siggi, would you mind reviewing html5lib changes?

Thanks!

[Jennifer Messerly, 2013-08-10 06:50:24]

@Siggi a few more notes:
* I've deleted .gitignore locally
* The test data took a while to upload so I checked the size to be sure. It is a
bit less than 2MB. (Eventually we may wish to get this from gclient, since
https://github.com/html5lib has more up to date tests.)

[Jennifer Messerly, 2013-08-10 06:56:46]

@Dan -- could you take a look at tools/publish_all_pkgs and tools/publish_pkg
changes?

[ricow1, 2013-08-12 05:51:17]

The changes to testing scripts LGTM
I have a few other inputs:
  Most files are missing copyright headers and there seems to be some
copyright/license file mismatch.
  What is the analyzer run scripts for? We should just use the normal way of
running the scripts through the analyzer, no need for special scripts in here?
(or is this just for "fast local testing"?)
  Please check this into svn/third_party instead of
svn/branches/bleeding_edge/dart/third_party and add a dependency to the DEPS
file

Are we still doing development of this on git-hub, or is this also moving
development into the dart repo? If we are still doing development on git-hub I
have an alternative solution (I already mentioned this to Dan/Emily on another
thread about a similar import)
We can do a mirror of the git-hub project on:
  chromium.googlesource.com
and then add a dependency to our DEPS file that will check this out into
third_party. Let me know if this is something we want to do.

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
File third_party/pkg/html5lib/README.md (right):

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/README.md:9: the same code will work on the client or
the server.
or the server -> and the server

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
File third_party/pkg/html5lib/html5lib.status (right):

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:3: # BSD-style license that can be
found in the LICENSE file.
There seems to be a mismatch between this statement and the content of the
license file. The license file has just a copyright notice, not the normal bsd
license body

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:10: */*/*/packages/*/*: Skip
this will only be the case if they are actually named _test.dart - do we
actually have any of these - otherwise I would just delete this.
Alternatively, we could just have the status file (and other related stuff
inside the test directory and then eliminate this)

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:20: test/parser_test: Skip
Is this temporary or will we continue depending on IO for these tests? Do we
have a but to track this

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:28: *: Skip
Are we really incapable of running this on arm, the vm people are pretty far in
their implementation. File a bug and let the vm people triage?

[Siggi Cherem (dart-lang), 2013-08-12 16:35:54]

lgtm

https://codereview.chromium.org/22375011/diff/18001/tools/test.dart
File tools/test.dart (right):

https://codereview.chromium.org/22375011/diff/18001/tools/test.dart#newcode69
tools/test.dart:69: new Path('third_party/pkg/html5lib'),
no need to change this now, but I was wondering if we expect to bring more
third_party packages, we might want to do third_party/pkg and somehow make 'pkg'
match both (pkg and third_party/pkg)

[Siggi Cherem (dart-lang), 2013-08-12 16:38:09]

oh, one more thing, do we need to change some gyp files to make the 'packages'
folders work? When you do ./tools/build.py a packages folder is generated in
out/ReleaseIA32/packages, which is used during tests AFAIK. I wonder how we make
it so that html5lib is included there too.

[Jennifer Messerly, 2013-08-12 17:47:37]

On 2013/08/12 16:38:09, Siggi Cherem (dart-lang) wrote:
> oh, one more thing, do we need to change some gyp files to make the 'packages'
> folders work? When you do ./tools/build.py a packages folder is generated in
> out/ReleaseIA32/packages, which is used during tests AFAIK. I wonder how we
make
> it so that html5lib is included there too.

yeah, this already works. It's handled by pkg/pkg.gyp

[Jennifer Messerly, 2013-08-12 17:54:29]

thanks Rico for the thoughtful comments!

On 2013/08/12 05:51:17, ricow1 wrote:
> The changes to testing scripts LGTM
> I have a few other inputs:
>   Most files are missing copyright headers and there seems to be some
> copyright/license file mismatch.

Yeah, this is the style used in the original html5lib code. When I ported it to
Dart ~1yr ago, the advice I got from our open source folks was to keep the style
in the original code. In other words, don't add copyright notices if it didn't
have it already.

>   What is the analyzer run scripts for? We should just use the normal way of
> running the scripts through the analyzer, no need for special scripts in here?
> (or is this just for "fast local testing"?)

Exactly, those are just for "fast local testing" so the package's tests can run
in a standalone environment. I'm not all that attached to it though, personally.
It was just the scripts we had before.

>   Please check this into svn/third_party instead of
> svn/branches/bleeding_edge/dart/third_party and add a dependency to the DEPS
> file

We're trying to primary development location, so I'm not sure how DEPS would
help. Same thing with the source code location -- it needs to be in the same
branch as our other "pkgs". We want to have all of our "Polymer" packages
developed in one place. I would've checked this into "pkg/html5lib" but it is
under a (slightly) different license so that wouldn't be appropriate.

perhaps "pkg/third_party/html5lib" would be more clear?

> 
> Are we still doing development of this on git-hub, or is this also moving
> development into the dart repo? If we are still doing development on git-hub I
> have an alternative solution (I already mentioned this to Dan/Emily on another
> thread about a similar import)
> We can do a mirror of the git-hub project on:
>   http://chromium.googlesource.com
> and then add a dependency to our DEPS file that will check this out into
> third_party. Let me know if this is something we want to do.

for this package, we want to move its primary development location. But this is
a really good option to know about! It might help in other cases.

[dgrove, 2013-08-12 17:54:56]

lgtm

[Jennifer Messerly, 2013-08-12 17:55:12]

I'm uploading a new version now! here are the replies to inline comments for
your reading pleasure :)

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
File third_party/pkg/html5lib/README.md (right):

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/README.md:9: the same code will work on the client or
the server.
On 2013/08/12 05:51:17, ricow1 wrote:
> or the server -> and the server

Done.

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
File third_party/pkg/html5lib/html5lib.status (right):

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:3: # BSD-style license that can be
found in the LICENSE file.
On 2013/08/12 05:51:17, ricow1 wrote:
> There seems to be a mismatch between this statement and the content of the
> license file. The license file has just a copyright notice, not the normal bsd
> license body

good catch! removed.

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:10: */*/*/packages/*/*: Skip
On 2013/08/12 05:51:17, ricow1 wrote:
> this will only be the case if they are actually named _test.dart - do we
> actually have any of these - otherwise I would just delete this.
> Alternatively, we could just have the status file (and other related stuff
> inside the test directory and then eliminate this)

Good point, removed :)

I copied this from "pkg/pkg.status" but now I wonder why we need it in either
place. It might be a historical artifact from when pub used the root folder of
the package instead of the "lib" folder.

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:20: test/parser_test: Skip
On 2013/08/12 05:51:17, ricow1 wrote:
> Is this temporary or will we continue depending on IO for these tests? Do we
> have a but to track this

I think it is permanent. The test cases are discovered by using a directory
listing, which is hard to duplicate in the browser. I'm open to idea though.

(having this lib work in browser is fairly low priority, IMO. If you're in a
browser, you've already got an HTML5 parser built in ;) )

https://codereview.chromium.org/22375011/diff/18001/third_party/pkg/html5lib/...
third_party/pkg/html5lib/html5lib.status:28: *: Skip
On 2013/08/12 05:51:17, ricow1 wrote:
> Are we really incapable of running this on arm, the vm people are pretty far
in
> their implementation. File a bug and let the vm people triage?

Oh, I have no idea, I just copied this from "pkg/pkg.status". We aren't running
*any* package tests under arm or mips right now. I was surprised to learn this
too :)

https://codereview.chromium.org/22375011/diff/18001/tools/test.dart
File tools/test.dart (right):

https://codereview.chromium.org/22375011/diff/18001/tools/test.dart#newcode69
tools/test.dart:69: new Path('third_party/pkg/html5lib'),
On 2013/08/12 16:35:54, Siggi Cherem (dart-lang) wrote:
> no need to change this now, but I was wondering if we expect to bring more
> third_party packages, we might want to do third_party/pkg and somehow make
'pkg'
> match both (pkg and third_party/pkg)

yeah, I had the same thought. it doesn't look like a simple change. maybe if I
move the entire code to pkg/third_party/html5lib, in which case it would "just
work"

[Jennifer Messerly, 2013-08-12 17:59:30]

thanks everyone, PTAL.

What are thoughts on moving here:

    pkg/third_party/html5lib

instead of:

    third_party/pkg/html5lib

Would that path make it more clear that we are developing the code in SVN?

[Siggi Cherem (dart-lang), 2013-08-12 19:56:22]

lgtm, I'm totally ok with either. but I'm curious to hear Rico's opinion.

[ricow1, 2013-08-13 05:42:59]

On 2013/08/12 17:59:30, John Messerly wrote:
> thanks everyone, PTAL.
> 
> What are thoughts on moving here:
> 
>     pkg/third_party/html5lib
I would go with this: This is something that we (as in the dart team) are
actively developing on. The normal third_party is mostly stuff that we have
pulled in from other sources and mainly have in our repo for easy consistent
access (in /third_party, not branches/bleeding_edge/dart/third_party).

> 
> instead of:
> 
>     third_party/pkg/html5lib
> 
> Would that path make it more clear that we are developing the code in SVN?

[ricow1, 2013-08-13 05:46:37]

LGTM  with the move to pkg/third_party/. 
I still think we can do a better job with the copyright/license stuff. There is
a bunch of dart files: Didn't we develop them? There are also still some files
with the normal dart header (the scripts) that reference the LICENSE file which
is the non standard license file from the original code.

[Jennifer Messerly, 2013-08-14 03:05:39]

PTAL, code now moved to "pkg/third_party/html5lib"

@Dan -- tools/publish_all_pkgs.py was changed again
@Rico -- there's a small change to pkg/pkg.gyp now, but I reverted the
"test.dart" changes.
@Siggi -- test changes gone; running the "pkg" suite now runs html5lib too. Yay!

[dgrove, 2013-08-14 03:07:01]

still lgtm!

[Jennifer Messerly, 2013-08-14 03:12:09]

regarding pkg/third_party:

I could add a README something like this?

"Packages in this directory is developed here, in the main Dart SVN repository,
but are ported and thus under a different open source license. See the LICENSE
file in the package for details. For html5lib, copyright headers are not used in
the original source (see https://github.com/html5lib/html5lib-python), so we
keep the same style here."

[Jennifer Messerly, 2013-08-14 03:13:09]

On 2013/08/14 03:12:09, John Messerly wrote:
> regarding pkg/third_party:
> 
> I could add a README something like this?
> 
> "Packages in this directory is developed here, in the main Dart SVN
repository,
> but are ported and thus under a different open source license. See the LICENSE
> file in the package for details. For html5lib, copyright headers are not used
in
> the original source (see https://github.com/html5lib/html5lib-python), so we
> keep the same style here."

Oops, "are" instead of "is" in first sentence.

[ricow1, 2013-08-14 05:29:32]

Gyp changes LGTM

[Siggi Cherem (dart-lang), 2013-08-14 16:17:31]

lgtm!

https://codereview.chromium.org/22375011/diff/201001/pkg/third_party/html5lib...
File pkg/third_party/html5lib/html5lib.status (right):

https://codereview.chromium.org/22375011/diff/201001/pkg/third_party/html5lib...
pkg/third_party/html5lib/html5lib.status:2: # Skip test not runnable via
test.dart
probably don't need this file anymore?

[Jennifer Messerly, 2013-08-14 20:48:28]

thanks everyone! landing...

https://codereview.chromium.org/22375011/diff/201001/pkg/third_party/html5lib...
File pkg/third_party/html5lib/html5lib.status (right):

https://codereview.chromium.org/22375011/diff/201001/pkg/third_party/html5lib...
pkg/third_party/html5lib/html5lib.status:2: # Skip test not runnable via
test.dart
On 2013/08/14 16:17:32, Siggi Cherem (dart-lang) wrote:
> probably don't need this file anymore?

good catch, done.

[Jennifer Messerly, 2013-08-14 21:00:12]

Committed patchset #5 manually as r26152 (presubmit successful).