improve unicode support

improve unicode support

* Allow unicode in log lines
* Allow simulation tests to contain unicode

BUG=619694
TBR=maruel

Committed: https://github.com/luci/recipes-py/commit/2eb32f6462d24eda586ae27d785c75a6e300197c

[martiniss, 2016-06-23 00:05:05]

The CQ bit was checked by martiniss@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-06-23 00:05:13]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/2088343002/1

[martiniss, 2016-06-23 00:05:27]

martiniss@chromium.org changed reviewers:
+ estaab@chromium.org, iannucci@chromium.org, maruel@chromium.org,
phajdan.jr@chromium.org

[martiniss, 2016-06-23 00:05:27]

PTAL

unicode stuff, so it's tricky :(

[commit-bot: I haz the power, 2016-06-23 00:07:23]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-06-23 00:07:24]

Dry run: This issue passed the CQ dry run.

[M-A Ruel, 2016-06-23 13:12:33]

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/third_party/e...
File recipe_engine/third_party/expect_tests/serialize.py (right):

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/third_party/e...
recipe_engine/third_party/expect_tests/serialize.py:32: return unicode(obj)
use obj.decode('utf-8') since I don't think there's a default code page set in
the process running the recipe.

https://codereview.chromium.org/2088343002/diff/1/recipes/engine_tests/unicod...
File recipes/engine_tests/unicode.py (right):

https://codereview.chromium.org/2088343002/diff/1/recipes/engine_tests/unicod...
recipes/engine_tests/unicode.py:14: cmd=['echo', 'hi'],
'echo', '❤ python'

to test unicode in the command arguments too.

[Paweł Hajdan Jr., 2016-06-23 14:51:55]

In general looks good. Just one comment.

I'm fine with anyone else stamping this.

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/step_runner.py
File recipe_engine/step_runner.py (right):

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/step_runner.p...
recipe_engine/step_runner.py:5: import StringIO
Why are we changing from cStringIO to StringIO? The former was supposed to be
faster.

[M-A Ruel, 2016-06-23 15:06:45]

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/step_runner.py
File recipe_engine/step_runner.py (right):

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/step_runner.p...
recipe_engine/step_runner.py:5: import StringIO
On 2016/06/23 14:51:54, Paweł Hajdan Jr. wrote:
> Why are we changing from cStringIO to StringIO? The former was supposed to be
> faster.

I've never seen it being faster in practice, when I did perf testing.

[martiniss, 2016-06-23 17:32:14]

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/step_runner.py
File recipe_engine/step_runner.py (right):

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/step_runner.p...
recipe_engine/step_runner.py:5: import StringIO
On 2016/06/23 at 15:06:45, M-A Ruel wrote:
> On 2016/06/23 14:51:54, Paweł Hajdan Jr. wrote:
> > Why are we changing from cStringIO to StringIO? The former was supposed to
be
> > faster.
> 
> I've never seen it being faster in practice, when I did perf testing.

cStringIO doesn't work with unicode, sadly. :/

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/third_party/e...
File recipe_engine/third_party/expect_tests/serialize.py (right):

https://codereview.chromium.org/2088343002/diff/1/recipe_engine/third_party/e...
recipe_engine/third_party/expect_tests/serialize.py:32: return unicode(obj)
On 2016/06/23 at 13:12:33, M-A Ruel wrote:
> use obj.decode('utf-8') since I don't think there's a default code page set in
the process running the recipe.

Done.

https://codereview.chromium.org/2088343002/diff/1/recipes/engine_tests/unicod...
File recipes/engine_tests/unicode.py (right):

https://codereview.chromium.org/2088343002/diff/1/recipes/engine_tests/unicod...
recipes/engine_tests/unicode.py:14: cmd=['echo', 'hi'],
On 2016/06/23 at 13:12:33, M-A Ruel wrote:
> 'echo', '❤ python'
> 
> to test unicode in the command arguments too.

AFAIK unicode isn't natively supported on windows, which robbie has brought up
every time i've mentioned unicode. So maybe not? I'd like to though....

[martiniss, 2016-06-23 17:54:53]

Description was changed from

==========
improve unicode support

* Allow unicode in log lines
* Allow simulation tests to unicode

BUG=619694
==========

to

==========
improve unicode support

* Allow unicode in log lines
* Allow simulation tests to contain unicode

BUG=619694
==========

[martiniss, 2016-06-24 18:49:26]

ping

any final comments?

[martiniss, 2016-06-24 21:12:56]

Description was changed from

==========
improve unicode support

* Allow unicode in log lines
* Allow simulation tests to contain unicode

BUG=619694
==========

to

==========
improve unicode support

* Allow unicode in log lines
* Allow simulation tests to contain unicode

BUG=619694
TBR=maruel
==========

[martiniss, 2016-06-24 21:13:03]

TBR because builders are broken

[martiniss, 2016-06-24 21:13:05]

The CQ bit was checked by martiniss@chromium.org

[commit-bot: I haz the power, 2016-06-24 21:13:09]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[nodir, 2016-06-24 21:14:19]

nodir@chromium.org changed reviewers:
+ nodir@chromium.org

[nodir, 2016-06-24 21:14:20]

lgtm

[commit-bot: I haz the power, 2016-06-24 21:15:25]

Description was changed from

==========
improve unicode support

* Allow unicode in log lines
* Allow simulation tests to contain unicode

BUG=619694
TBR=maruel
==========

to

==========
improve unicode support

* Allow unicode in log lines
* Allow simulation tests to contain unicode

BUG=619694
TBR=maruel

Committed:
https://github.com/luci/recipes-py/commit/2eb32f6462d24eda586ae27d785c75a6e30...
==========

[commit-bot: I haz the power, 2016-06-24 21:15:26]

Committed patchset #3 (id:40001) as
https://github.com/luci/recipes-py/commit/2eb32f6462d24eda586ae27d785c75a6e30...