Added support for properly building and running isolates in recipes.

Added support for properly building and running isolates in recipes.

(Re-land of https://codereview.chromium.org/132853003/ after lab machines
were whitelisted.)

Switched the GPU recipe to run gl_tests via an isolate. Will convert
remaining tests to run via isolates once this is working well on the bots.

In addition to recipes_test, tested by running gpu/build_and_test recipe
locally on Linux. Isolate is properly uploaded to the server during the
build steps and executed via run_isolated.py during the test steps.

BUG=321878
TBR=iannucci@chromium.org

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=244799

[Ken Russell (switch to Gerrit), 2014-01-14 20:32:35]

FYI

[commit-bot: I haz the power, 2014-01-14 20:33:11]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/kbr@chromium.org/137993010/1

[commit-bot: I haz the power, 2014-01-14 20:34:28]

Presubmit check for 137993010-1 failed and returned exit status 1.

Running presubmit commit checks ...

** Presubmit Messages **
--tbr was specified, skipping OWNERS check

** Presubmit ERRORS **
scripts/slave/unittests/recipes_test.py (11.97s) failed
.............................................................................................................................E.E.E.E.E.E...................................
======================================================================
ERROR: test_linux_debug (__main__.RecipeTest_for_download_and_test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/slave/unittests/recipes_test.py", line 154, in test_
IOError: [Errno 2] No such file or directory:
'/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/download_and_test.expected/linux_debug.json'

======================================================================
ERROR: test_mac_debug (__main__.RecipeTest_for_download_and_test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/slave/unittests/recipes_test.py", line 154, in test_
IOError: [Errno 2] No such file or directory:
'/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/download_and_test.expected/mac_debug.json'

======================================================================
ERROR: test_win_debug (__main__.RecipeTest_for_download_and_test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/slave/unittests/recipes_test.py", line 154, in test_
IOError: [Errno 2] No such file or directory:
'/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/download_and_test.expected/win_debug.json'

======================================================================
ERROR: test_linux_debug (__main__.RecipeTest_for_build_and_upload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/slave/unittests/recipes_test.py", line 154, in test_
IOError: [Errno 2] No such file or directory:
'/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/build_and_upload.expected/linux_debug.json'

======================================================================
ERROR: test_mac_debug (__main__.RecipeTest_for_build_and_upload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/slave/unittests/recipes_test.py", line 154, in test_
IOError: [Errno 2] No such file or directory:
'/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/build_and_upload.expected/mac_debug.json'

======================================================================
ERROR: test_win_debug (__main__.RecipeTest_for_build_and_upload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/slave/unittests/recipes_test.py", line 154, in test_
IOError: [Errno 2] No such file or directory:
'/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/build_and_upload.expected/win_debug.json'

----------------------------------------------------------------------
Ran 171 tests in 9.726s

FAILED (errors=6)
Name                                                                            
            Stmts   Miss  Cover   Missing
--------------------------------------------------------------------------------------------------------------------------
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/android/api    
               64      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/android/test_api
               4      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/archive/api    
               29      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/base_android/api
              25      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/chromite/api   
               22      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/chromium/api   
               99      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/chromium_android/api
         156      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/gclient/api    
              112      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/gclient/test_api
              13      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/generator_script/api
          31      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/generator_script/test_api
      5      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/git/api        
               28      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/gpu/api        
               70      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/gsutil/api     
               20      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/isolate/api    
               24      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/isolate/test_api
               7      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/itertools/api  
                9      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/json/api       
               76      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/json/test_api  
               35      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/path/api       
               99      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/path/test_api  
                9      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/platform/api   
               35      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/platform/test_api
             12      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/properties/api 
               26      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/properties/test_api
           22      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/python/api     
               16      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/raw_io/api     
               34      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/repo/api       
               17      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/rietveld/api   
                9      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/step/api       
               30      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/step_history/api
              13      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/test_utils/api 
               43      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/v8/api         
               17      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipe_modules/webrtc/api     
               14      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/android/android_builder
              33      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/android/instrumentation_tests
        29      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/android_webview_aosp  
               22      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/blink_trybot          
               65      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/chromium              
               12      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/chromium_trybot       
              141      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/dart/dartium          
               30      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gatekeeper            
                6      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/build_and_test    
               21      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/build_and_upload  
               11      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/gpu/download_and_test 
               11      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/libvpx_android_unittests
             27      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/oilpan                
               38      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/ozone_ecs             
               22      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/polymer               
               41      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/run_presubmit         
               19      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/v8                    
               15      0   100%   
/b/commit-queue/workdir/tools/build/scripts/slave/recipes/webrtc_android_apk_trybot
            21      0   100%   
--------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                           
             1789      0   100%   


Presubmit checks took 43.0s to calculate.

[commit-bot: I haz the power, 2014-01-14 20:43:40]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/kbr@chromium.org/137993010/50001

[commit-bot: I haz the power, 2014-01-14 20:44:55]

Change committed as 244799

[bajones, 2014-01-14 21:46:50]

It appears that this is causing issues on the bots. All three Windows Release
testers are failing gl_tests with some variation of:

ERROR net(429): Unable to open given url,
https://isolateserver.appspot.com/content-gs/retrieve/default-gzip/a8a26ac05b...,
after 30 attempts.
Exception: 404 Client Error: Not Found

I'm thinking we need to revert this, but I'm going to double check with kbr@
first to make sure there's not a simple fix or other culprit.