Chrome updater test framework

install_test/chrome_checkout.py

+#!/usr/bin/env python
+# Copyright (c) 2012 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Utilities for checking out Chrome source files from SVN. 
+
+Chrome release version number is required for checkout. The version number is
+used to obtain the revision number, which is then used to checkout files from

[Nirnimesh, 2012/08/09 21:32:48]

describe how "The version number is used to obtain the revision number"

describe about the DEPS file

Also declare all the other files it needs from the internet

[nkang, 2012/08/16 23:46:24]

Changed the text so its more descriptive.

+SVN.
+"""
+
+import httplib
+import logging
+import os
+import re
+import socket
+import subprocess
+
+_BASE_SVN_URL = 'svn://svn.chromium.org/chrome'
+_SELENIUM_URL = 'http://selenium.googlecode.com/svn/trunk/py'
+_PYFTPDLIB_URL = 'http://pyftpdlib.googlecode.com/svn/trunk'
+
+
+def _SetConfiguration():
+  """Sets the logging configuration."""
+  log_format = '%(asctime)s - %(levelname)s : %(message)s'
+  logging.basicConfig(level=logging.DEBUG, format=log_format)
+
+
+_SetConfiguration()
+
+
+def _GetContentAndReturnResponse(server, path):

[Nirnimesh, 2012/08/09 21:32:48]

why not just use urllib?

[nkang, 2012/08/16 23:46:24]

I thought about that, as well. But part of this script was written by Anthony
LaForge, including this method. Since he had already written it and it worked, I
decided not to mess with it. But I went ahead and changed it. However, I used
urllib2, since urllib.urlopen has been deprecated as of Python version 2.6.

+  """Issues a GET requst to server and returns the response body.
+
+  Args:
+    server: Host address.
+    path: The path where the file is located.
+
+  Returns:
+    A string containing the response body.
+  """
+  try:

[Nirnimesh, 2012/08/09 21:32:48]

all this try/catch seems unncessary to me. Remove it. If it fails, the right
trace will be thrown anyway

[nkang, 2012/08/16 23:46:24]

Got rid of the try/except block. Also, this method has been completely reworked.
I changed the code so we use urllib2 instead of httplib, so most of the code
below is no longer in existence.

+    connection = httplib.HTTPConnection(server)
+    headers = {'Content-type': 'text/html'}
+    connection.request('GET', path, '', headers)
+    response = connection.getresponse()
+  except socket.gaierror, err:    
+    connection.close()
+    raise err
+  if response.status != 200:
+    connection.close()
+    raise RuntimeError('%s/%s returned the following status code: %d' %
+                       (server, path, response.status))
+  data = response.read() 
+  connection.close()  
+  return data
+
+
+def _GetDeps(version):
+  """Returns contents of DEPS file that corresponds with the version number.
+
+  Args:
+    version: Chrome version number (e.g., 21.0.1136.0).
+  """
+  deps = _GetContentAndReturnResponse(
+      'src.chromium.org',
+      '/viewvc/chrome/releases/%s/DEPS' % version)
+  return deps
+
+
+def _ParseVersion(version_str):
+  """Parses the version string to get the different identifiers.
+
+  Args:
+    version_str: Chrome release version number.
+  """
+  match = re.search(r'((\d+)\.(\d+)\.(\d+)\.(\d+))', version_str)
+  if match:
+    version = {'version': match.group(1),
+               'major': int(match.group(2)),
+               'minor': int(match.group(3)),
+               'build': int(match.group(4)),
+               'patch': int(match.group(5))}
+    return version
+  raise RuntimeError('Invalid version number was specified: %r' % version_str)  
+
+
+def _GetRevisionInfo(version_str, deps):
+  """Gets the revision info by parsing the contents of the DEPS file.
+
+  Args:
+    version_str: A string representing the Chrome version number.
+    deps: A string that contains the contents of corresponding DEPS file.
+
+  Returns:
+    A string that contains pertinent information about the Chrome version.
+  """
+  version = _ParseVersion(version_str)  
+  match = re.search("'src':[\n\r ]+'(.*?)'", deps)

[Nirnimesh, 2012/08/09 21:32:48]

use ' for the outer quotes

[nkang, 2012/08/16 23:46:24]

The reason we (Anthony LaForge originally added them when he wrote this method,
and I continued to use them) use double quotes on the outside is because the
word 'src' has single quotes around it in the DEPS file. If we switched the
double quotes with the single quotes, it wouldn't match. To use single quotes
inside single quotes, we'd have to use escape characters. So I changed it to
following: 
re.search('\'src\':[\n\r ]+\'(.*?)\'', deps)

The other option, of course, is to use triple quotes, but IMHO, we should use
the escape characters instead.

+  if match:
+    match = re.search(r"@(\d+)", match.group(1))

[Nirnimesh, 2012/08/09 21:32:48]

use '

[nkang, 2012/08/16 23:46:24]

Changed to single quotes.

+    if match:
+      version['revision'] = int(match.group(1))
+
+    match = re.search("""['"]src['"].*?:.*?['"]/branches/(.*?)/.*?,""",

[Nirnimesh, 2012/08/09 21:32:48]

this is very confusing. give an example of what it's trying to match

[nkang, 2012/08/16 23:46:24]

Branch releases have the branch number listed in the DEPS file, for example:
'/branches/1230/src@150565'. So, this line of code is trying to match the branch
number listed in DEPS. If no match is found, that means its a trunk.

+                      deps, re.DOTALL | re.IGNORECASE)
+    if match:
+      version['branch'] = match.group(1)
+    else:
+      version['branch'] = 'trunk'
+  return version
+
+
+def _GetRevision(deps, rev_type='selenium'):

[Nirnimesh, 2012/08/09 21:32:48]

do not provide default arg

[nkang, 2012/08/16 23:46:24]

Got rid of the default argument.

+  """Gets selenium/pyftpdlib rev. number by parsing contents of DEPS file.    
+
+  Args:    
+    deps: A string that contains the contents of corresponding DEPS file.
+    rev_type: Type of revision number to look up: 'selenium' or 'pyftpdlib'.
+    
+  Returns:
+    An integer representing the revision number that was requested.
+  """
+  assert(rev_type == 'selenium' or rev_type == 'pyftpdlib')  

[Nirnimesh, 2012/08/09 21:32:48]

remove parens

[nkang, 2012/08/16 23:46:24]

Got rid of the parenthesis. 

+  if rev_type == 'selenium':
+    m = re.search(r'http://selenium\.googlecode\.com/svn/trunk/py@(\d+)',

[Nirnimesh, 2012/08/09 21:32:48]

m -> match

[nkang, 2012/08/16 23:46:24]

d -> done!

+                  deps, re.DOTALL | re.IGNORECASE | re.MULTILINE)
+  elif rev_type == 'pyftpdlib':
+    m = re.search(r'http://pyftpdlib\.googlecode\.com/svn/trunk@(\d+)',
+                  deps, re.DOTALL | re.IGNORECASE | re.MULTILINE)
+  if m:
+    return int(m.group(1))
+  raise RuntimeError('Could not find the revision number in DEPS.')
+
+
+def _SvnCo(path, revision=None, dest=None):
+  """Does a SVN checkout on specified source files.
+
+  Args:
+    path: URL that is to be checked out.
+    revision: Revision number.
+    dest: Destination where the data will be downloaded.
+  """
+  cmd = 'svn co'

[Nirnimesh, 2012/08/09 21:32:48]

make cmd a list, and then remove shell=True from line 153

[nkang, 2012/08/16 23:46:24]

Changed cmd to a list. However, removing shell=True results in the following
error: 

WindowsError: [Error 2] The system cannot find the file specified

I checked the documentation for the subprocess module, and it states that 'args'
can be passed as a string or as a sequence of program arguments. Documentation
further states that 'shell=True' is required if using a string as 'args', unless
the string specifies the name of a program without any arguments. I'm guessing
that's why you told me to convert 'cmd' to a list. However, even after I changed
it to a list, I kept getting the aforementioned error. The only way it works is
if I pass 'shell=True'. I did some more digging and discovered that if
shell=False is used, python uses the CreateProcess API on Windows to launch
another process, and CreateProcess adds a '.exe' to the process name if it
doesn't have an extension, which would explain why we get the 'file not found'
error. Also, I found a similar thread that someone had posted on the Python-Dev
site. It states that if you need to run something that's not a Windows
executable, such as a batch file, you should use the command interpreter (i.e.
shell=True). Since svn is also launched using a batch file (svn.bat), I guess it
makes sense why it doesn't work with shell=False. This also explains why Chrome
works when launched without shell=True but not svn. 

So, for the time being, I've left shell=True in there. I did get rid of the
assert and changed subprocess.Popen to subprocess.check_call.

+  if revision:
+    cmd += ' --revision %d' % revision        
+  cmd += ' %s' % path
+  if dest:
+    cmd += ' %s' % dest  
+  logging.info(cmd)
+  assert(subprocess.Popen(cmd, shell=True).wait() == 0)  

[Nirnimesh, 2012/08/09 21:32:48]

remove outer parens

[Nirnimesh, 2012/08/09 21:32:48]

remove assert. use subprocess.check_call instead

[nkang, 2012/08/16 23:46:24]

Got rid of the assert and replaced it with subprocess.check_call.

[nkang, 2012/08/16 23:46:24]

Got rid of the assert, so this no longer applies.

+
+
+def _IsVersionValid(version):
+  """Checks if the version number has the correct format.
+
+  Args:
+    version: Version number to check.
+
+  Returns:
+    True if 'n.n.n.n' pattern is found in version number, otherwise False.
+  """
+  if type(version) == str:
+    return re.findall('\d+\.\d+\.\d+\.\d+', version) != []

[Nirnimesh, 2012/08/09 21:32:48]

use re.match()

[nkang, 2012/08/16 23:46:24]

Changed re.findall to re.match.

+  return False
+
+
+def CheckOut(version, dest):
+  """Checks out all necessary source files.
+
+  Args:
+    version: Chrome release version number (e.g., 21.0.1136.0).
+    dest: Destination where the checked out files will go.
+  """
+  if not _IsVersionValid(version):      
+    raise RuntimeError('Invalid version number was specified: %r.' % version)
+  if not os.path.isdir(dest):    
+    os.mkdir(dest)        
+  deps = _GetDeps(version)
+  rev_info = _GetRevisionInfo(version, deps)  
+  logging.info(rev_info)
+  # If its a patch, check out the branch.
+  if rev_info['patch']:
+    svn_url_base = _BASE_SVN_URL + '/branches/%s' % rev_info['branch']
+  # If not, check out the trunk.
+  else:
+    svn_url_base = _BASE_SVN_URL + '/trunk'
+  _SvnCo('%s/src/chrome/test/functional' % svn_url_base,

[Nirnimesh, 2012/08/09 21:32:48]

This seems too fragile. if pyauto requires additional dependencies, no one is
going to remember to add them here. Why not use functional.DEPS?

[nkang, 2012/08/16 23:46:24]

Checked with Ken about this and he was adamant that we should not use
functional.DEPS. He said that functional.DEPS contains items we do not need,
particularly the 'data' folder, which is humongous (over 600MB approx). He
further stated that using functional.DEPS would mean that we would 'have to
substitute the branch numbers'. He recommended that we either leave it like this
or create a new DEPS file for these dependencies. Personally I feel that, given
the amount of time that's already been put into this framework, we should just
continue on like this for the time being. Down the line we can create a new DEPS
file. The rest is up to you guys. I am okay with whatever we decide.

+         rev_info['revision'], os.path.join(dest, 'src', 'chrome', 'test',

[Nirnimesh, 2012/08/09 21:32:48]

how do you plan to handle internal deps?

[nkang, 2012/08/16 23:46:24]

Checked with Anantha about this, and she told me to check with Ken. Checked with
Ken regarding this, and he said that we will not be handling internal deps. He
said that maybe down the line we will but not at the moment.

+                                            'functional'))
+  _SvnCo('%s/src/chrome/test/pyautolib' % svn_url_base,
+         rev_info['revision'], os.path.join(dest, 'src', 'chrome', 'test',
+                                            'pyautolib'))
+  _SvnCo('%s/src/third_party/simplejson' % svn_url_base,
+         rev_info['revision'], os.path.join(dest, 'src', 'third_party', 
+                                            'simplejson'))
+  _SvnCo('%s/src/third_party/tlslite' % svn_url_base,
+         rev_info['revision'], os.path.join(dest, 'src', 'third_party', 
+                                            'tlslite'))
+  _SvnCo('%s/src/net/tools/testserver' % svn_url_base,
+         rev_info['revision'], os.path.join(dest, 'src', 'net', 'tools',
+                                            'testserver'))    
+  _SvnCo(_SELENIUM_URL, _GetRevision(deps, 'selenium'),
+         os.path.join(dest, 'src', 'third_party', 'webdriver', 'pylib',
+                      'selenium'))   
+  _SvnCo(_PYFTPDLIB_URL, _GetRevision(deps, 'pyftpdlib'),
+         os.path.join(dest, 'src', 'third_party', 'pyftpdlib'))