Work around unicode issues by re-encoding properties as utf-8

recipes.py

Index: recipes.py
diff --git a/recipes.py b/recipes.py
index 0626c1f356f2ef24611b9eab21b59192bc8d006c..a3214a5414681040b610ab46e01943587547d9d2 100755
--- a/recipes.py
+++ b/recipes.py
@@ -182,6 +182,21 @@ def new_handle_recipe_return(result, result_filename, stream_engine):
   return 0
 
 
+def strip_unicode(obj):

[iannucci, 2017/04/06 18:10:12]

Could we move this function to e.g. util.py and then import+use it from the json
recipe module, so that it's not defined in two places? Then if we have to fix a
bug in it, we don't get diverging behaviors.

+  if isinstance(obj, unicode):
+    return obj.encode('utf-8', 'replace')
+
+  if isinstance(obj, list):
+    return map(strip_unicode, obj)
+
+  if isinstance(obj, dict):
+    new_obj = type(obj)(
+        (strip_unicode(k), strip_unicode(v)) for k, v in obj.iteritems() )
+    return new_obj
+
+  return obj
+
+
 def run(package_deps, args, op_args):
   from recipe_engine import run as recipe_run
   from recipe_engine import loader
@@ -232,6 +247,11 @@ def run(package_deps, args, op_args):
 
   properties['recipe'] = args.recipe
 
+  # Work around unicode issues by re-encoding properties as utf-8.
+  properties = strip_unicode(properties)
+  reload(sys)

[iannucci, 2017/04/06 18:10:12]

Now that I think about this, I would actually do this even earlier, maybe at
line 20 in this file? This needs to apply in all modes of recipe engine
execution, not just run.

+  sys.setdefaultencoding('UTF8')
+
   os.environ['PYTHONUNBUFFERED'] = '1'
   os.environ['PYTHONIOENCODING'] = 'UTF-8'