Add lextab.pyc as an output of the cached_lex_yacc_tables action

Add lextab.pyc as an output of the cached_lex_yacc_tables action

The primary output is lextab.py, but since it is imported, Python also
writes lextab.pyc. Listing it too as an output means it won't be left
around by "ninja -t clean" (and not much else, I think.)

Prior to the patch
  https://codereview.chromium.org/425953002/
the result of not having lextab.pyc listed as an output and running
"ninja -t clean" was a tree that would build correctly, but that would
always rerun all the IDL code generation scripts, and thus never become
"clean".

BUG=397909
Committed: https://src.chromium.org/viewvc/blink?view=rev&revision=180142

[Jens Widell, 2014-08-12 14:28:06]

PTAL

[haraken, 2014-08-12 14:31:33]

LGTM. Thanks for taking care of these subtle build issues.

[Jens Widell, 2014-08-12 14:34:16]

The CQ bit was checked by jl@opera.com

[commit-bot: I haz the power, 2014-08-12 14:34:55]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/jl@opera.com/463063003/1

[Nils Barth (inactive), 2014-08-12 14:35:02]

LGTM, thanks, updated docs:
http://www.chromium.org/developers/generated-files

One last subtle point:
is the .pyc file generated *by this step*,
or is it generated in a further step, when its imported?

If the later, I'm concerned about subtle breakage;
presumably we could fix via explicitly importing after making it?

[Jens Widell, 2014-08-12 14:43:00]

On 2014/08/12 14:35:02, Nils Barth (inactive) wrote:
> One last subtle point:
> is the .pyc file generated *by this step*,
> or is it generated in a further step, when its imported?

It is generated by this step. In blink_idl_parser.py::main(), we first calls
blink_idl_lexer.py::main(), which deletes the files and initializes the lexer,
thus writing lextab.py. Then blink_idl_parser.py::main() initializes the parser,
which initializes another lexer, at which point lextab.py is imported, and
lextab.pyc is written.

I also ran "ninja cached_lex_yacc_tables" while testing this to make sure, BTW.

> If the later, I'm concerned about subtle breakage;

I also tested listing a bogus file as an output, and ninja seemed to happily
ignore it. But yeah, I am/was also a bit concerned about it. And also about the
theoretical possibility that someone decides to run python with -O, and thus
causing it to be lextab.pyo instead. But if we want to handle that, I think we
should do it by modifying PLY to write lextab.pickle file instead.

[Nils Barth (inactive), 2014-08-12 14:47:41]

On 2014/08/12 14:43:00, Jens Lindström wrote:
> On 2014/08/12 14:35:02, Nils Barth (inactive) wrote:
> > One last subtle point:
> > is the .pyc file generated *by this step*,
> > or is it generated in a further step, when its imported?
> 
> It is generated by this step. In blink_idl_parser.py::main(), we first calls
> blink_idl_lexer.py::main(), which deletes the files and initializes the lexer,
> thus writing lextab.py. Then blink_idl_parser.py::main() initializes the
parser,
> which initializes another lexer, at which point lextab.py is imported, and
> lextab.pyc is written.

Ok, I suspected as much, but would be easy to miss...
A comment to that effect would be a nice touch
(either at the relevant import or in blink_idl_parser, or perhaps both),
but that's not necessary.

> I also ran "ninja cached_lex_yacc_tables" while testing this to make sure,
BTW.

> And also about the theoretical possibility that someone decides to run python
with -O,
> and thus causing it to be lextab.pyo instead.

I think we can safely ignore this; I think this doesn't work with PLY (I tried
in the past
and it didn't work without significant hacking), and it's non-standard enough
that one
should expect it to break.

Thanks for your care!

[commit-bot: I haz the power, 2014-08-12 16:14:45]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  win_blink_rel on tryserver.blink
(http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/22292)

[commit-bot: I haz the power, 2014-08-12 17:46:08]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-08-12 17:46:08]

Try jobs failed on following builders:
  win_blink_rel on tryserver.blink
(http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/22292)

[Jens Widell, 2014-08-12 17:48:52]

The CQ bit was checked by jl@opera.com

[commit-bot: I haz the power, 2014-08-12 17:49:32]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/jl@opera.com/463063003/1

[commit-bot: I haz the power, 2014-08-12 18:13:58]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  win_blink_rel on tryserver.blink
(http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/22325)

[commit-bot: I haz the power, 2014-08-12 18:40:13]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-08-12 18:40:15]

Try jobs failed on following builders:
  win_blink_rel on tryserver.blink
(http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/22332)

[Jens Widell, 2014-08-13 06:02:55]

The CQ bit was checked by jl@opera.com

[commit-bot: I haz the power, 2014-08-13 06:03:06]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/jl@opera.com/463063003/1

[commit-bot: I haz the power, 2014-08-13 06:03:47]

Change committed as 180142

[cjhopman, 2014-08-15 01:26:41]

On 2014/08/13 06:03:47, I haz the power (commit-bot) wrote:
> Change committed as 180142

My understanding is that there are various ways that the .pyc file will actually
never be built (this step just generates the file and so the pyc file isn't
actually created until a later step uses lextab.py, not using cpython, ...). In
that case, you can have a normal build never be "clean" with this change.

[Jens Widell, 2014-08-15 06:04:45]

On 2014/08/15 01:26:41, cjhopman wrote:
> On 2014/08/13 06:03:47, I haz the power (commit-bot) wrote:
> > Change committed as 180142
> 
> My understanding is that there are various ways that the .pyc file will
actually
> never be built (this step just generates the file and so the pyc file isn't
> actually created until a later step uses lextab.py, not using cpython, ...).
In
> that case, you can have a normal build never be "clean" with this change.

The generated lextab.py file is always imported by this step, but if that
doesn't result in a lextab.pyc being written, then we're in trouble. The only
ill effect of not listing it as an output that I'm aware of is that it's left
around by "ninja -t clean", which with the earlier fixes should be harmless
anyway, so I guess we should revert this change.

[Jens Widell, 2014-08-15 06:10:17]

A revert of this CL (patchset #1) has been created in
https://codereview.chromium.org/468743003/ by jl@opera.com.

The reason for reverting is: This CL added the assumption that Python will write
the lextab.pyc file. If that's a wrong assumption, which seems to be the case in
some situations, the result is that the cached_lex_yacc_tables target never
becomes clean, which in turn means that all IDL files are recompiled on each
build.

The problem fixed by this CL was cosmetic only, so is not worth the risk..