Chrome updater test framework

install_test/chrome_checkout.py

+#!/usr/bin/env python
+# Copyright (c) 2012 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+

[kkania, 2012/06/09 00:06:10]

have a description of this module

[nkang, 2012/06/12 23:48:33]

Added a brief description.

+import httplib
+import optparse
+import os
+import re
+import socket
+import subprocess
+import sys
+
+_BASE_SVN_URL = 'svn://svn.chromium.org/chrome'
+_SELENIUM_URL = 'http://selenium.googlecode.com/svn/trunk/py'
+_PYFTPDLIB_URL = 'http://pyftpdlib.googlecode.com/svn/trunk'
+_DEPS_CACHE = {}
+
+def _GetContentAndReturnResponse(server, path):

[kkania, 2012/06/09 00:06:10]

two newlines between module-level definitions, in several places in this file
and in other files too:
http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Blank_Lines

[nkang, 2012/06/12 23:48:33]

Added two blank lines, where needed.

+  """Returns contents of DEPS file, which contains info about source files

[kkania, 2012/06/09 00:06:10]

This func may be involved with the process of getting the DEPS file, but it is
generic and is not tied to DEPS file at all. I could use it to fetch the latest
news feed. Change this comment to reflect the function.

Also, the first line of doc should take no more than 1 line.  See
http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Comments

[nkang, 2012/06/12 23:48:33]

I don't dispute that you could get the latest news feed using this method. It's
just that in your last response you said that 'its just a helper method that
issues a GET request'. So I thought you were trying to imply that it only issued
a GET request and nothing else, and that's clearly not the case; that's all I
was trying to say. I see where you're coming from, though, so I went ahead and
changed the docstring to the following, 'Issues a GET requst to server and
returns the response body'.

+
+  needed.
+
+  Args:
+    server: Host address, in this case the SVN server.
+    path: The URL where the file is located.
+
+  Returns:
+    A string containing the file's contents if successful, otherwise None.
+  """
+  try:
+    conn = httplib.HTTPConnection(server)
+    headers = {'Content-type': 'text/html'}
+    conn.request('GET', path, '', headers)
+    response = conn.getresponse()
+  except socket.gaierror, err:
+    print 'ChromeCheckout._GetContentAndReturnResponse: %s' % err

[kkania, 2012/06/09 00:06:10]

either
1) catch this exception, do conn.close() and raise the exception again (drop the
print) or 
2) just don't catch the exception at all and wait for python to close the socket
automatically

I'm fine with either choice.

[nkang, 2012/06/12 23:48:33]

Got rid of the print statement. I now catch socket.gaierror, close the
connection, and raise the exception again.

+    conn.close()
+    return None
+  if response.status != 200:
+    conn.close()
+    return None

[kkania, 2012/06/09 00:06:10]

throw an exception here instead of returning None, and change the Returns:
comment

[nkang, 2012/06/12 23:48:33]

RuntimeError is now raised if status code in not 200. Also, updated the
'Returns' comment to reflect this change.

+  data = response.read() 
+  conn.close()
+  return data
+
+def _GetDeps(version):
+  """Returns cached version of DEPS if possible, otherwise gets its contents.
+
+  Args:
+    version: Chrome version number (e.g., 21.0.1136.0).
+
+  Returns:
+    string: A string containing the contents of the DEPS file.
+  """
+  if version in _DEPS_CACHE:

[kkania, 2012/06/09 00:06:10]

get rid of the cache, and call getdeps in checkout and pass DEPS to the two
funcs that need it

[nkang, 2012/06/12 23:48:33]

Updated _GetDeps and got rid of _DEPS_CACHE. _GetDeps is now called in the
CheckOut method. The value returned by _GetDeps is then passed onto _GetRevision
and _GetRevisionInfo. Also updated _GetRevision and _GetRevisionInfo, as they do
not call _GetDeps directly now, instead the contents of DEPS are now passed to
them in the form of an argument.

+    return _DEPS_CACHE[version]
+  deps = _GetContentAndReturnResponse('src.chromium.org',
+                                      '/viewvc/chrome/releases/%s/DEPS'
+                                      % version)

[kkania, 2012/06/09 00:06:10]

This would be more readable:
deps = _GetContentAndReturnResponse(
    'src.chromium.org',
    '/viewvc/chrome/releases/%s/DEPS' % version)

[nkang, 2012/06/12 23:48:33]

Done and done!

+  _DEPS_CACHE[version] = deps
+  return deps
+
+def _ParseVersion(version):
+  """Parses the version number to get the different identifiers."""
+  match = re.search(r'((\d+)\.(\d+)\.(\d+)\.(\d+))', version)
+  if match:
+    version = {'version': match.group(1),
+               'major': int(match.group(2)),
+               'minor': int(match.group(3)),
+               'build': int(match.group(4)),
+               'patch': int(match.group(5))}
+    return version
+  return {}

[kkania, 2012/06/09 00:06:10]

throw

[nkang, 2012/06/12 23:48:33]

Instead of returning an empty dictionary, the method now raises a RuntimeError
exception, if it cannot find a match in the version number that was passed.
Also, since we're throwing an exception now, I also updated the two methods that
use _ParseVersion: _GetRevisionInfo and _GetRevision. Removed the 'if' statement
that confirmed that value returned by _ParseVersion wasn't empty from both of
the aforementioned methods.

+
+def _GetRevisionInfo(version_str):
+  """Gets the revision info by parsing the contents of the DEPS file.
+
+  Args:
+    version_str: A string representing the Chrome version number.
+
+  Returns:
+    A string that contains pertinent information about the Chrome version.
+  """
+  version = _ParseVersion(version_str)
+  if version:
+    body = _GetDeps(version['version'])
+    if body:
+      match = re.search("'src':[\n\r ]+'(.*?)'", body)
+      if match:
+        match = re.search(r"@(\d+)", match.group(1))
+        if match:
+          version['revision'] = int(match.group(1))
+
+      match = re.search("""['"]src['"].*?:.*?['"]/branches/(.*?)/.*?,""",
+                        body, re.DOTALL | re.IGNORECASE)
+      if match:
+        version['branch'] = match.group(1)
+      else:
+        version['branch'] = 'trunk'
+  return version
+
+def _GetRevision(version_str, rev_type='selenium'):
+  """Gets selenium/pyftpdlib rev. number by parsing contents of DEPS file.    
+
+  Args:
+    version_str: A string representing the Chrome build number.
+    rev_type: Type of revision number to look up: 'selenium' or 'pyftpdlib'.
+    
+  Returns:
+    An integer representing the revision no. if successful, otherwise None.
+  """
+  assert(rev_type == 'selenium' or rev_type == 'pyftpdlib')
+  version = _ParseVersion(version_str)
+  if version:
+    body = _GetDeps(version_str)
+    if body:
+      if rev_type == 'selenium':
+        m = re.search(r'http://selenium\.googlecode\.com/svn/trunk/py@(\d+)',
+                      body, re.DOTALL | re.IGNORECASE | re.MULTILINE)
+      elif rev_type == 'pyftpdlib':
+        m = re.search(r'http://pyftpdlib\.googlecode\.com/svn/trunk@(\d+)',
+                      body, re.DOTALL | re.IGNORECASE | re.MULTILINE)
+      if m:
+        return int(m.group(1))
+  return None

[kkania, 2012/06/09 00:06:10]

throw and change doc

[nkang, 2012/06/12 23:48:33]

This method now raises a RuntimeError, instead of returning None. Also updated
the docstring to reflect this change.

+
+def _SvnCo(path, revision=None, dest=None):
+  """Does a SVN checkout on specified source files.
+
+  Args:
+    path: URL that is to be checked out.
+    revision: Revision number.
+    dest: Destination where the data will be downloaded.
+  """
+  cmd = 'svn co'
+  if revision:
+    cmd += ' --revision %d' % revision        
+  cmd += ' %s' % path
+  if dest:
+    cmd += ' %s' % dest
+  print cmd    

[kkania, 2012/06/09 00:06:10]

use logging module or drop

[nkang, 2012/06/12 23:48:33]

I have replaced all the print statements with logging. However, the 'logging'
information page on the Python website states that to 'display console output
for ordinary usage of a command line script or program', the best tool for the
job is the 'print' statement. I think 'logging' is good for more complex
scripts, where one could have several different types of messages, warnings,
errors, or general info, etc. In this script, however, we only have two print
statements. I think 'logging' might be a bit of an overkill for our situation.
What do you think? I've already applied the changes. If you agree, I'll revert
the changes. If not, we can leave it as is. Let me know.

+  return subprocess.Popen.wait(subprocess.Popen(cmd, shell=True))

[kkania, 2012/06/09 00:06:10]

can you just do subprocess.Popen(cmd, shell=True).wait() instead?

[nkang, 2012/06/12 23:48:33]

Done.

+
+def _GetPath(par, ch):
+  return (lambda p, c: os.path.join(p, c) if p else c)(par, ch)
+
+def _IsVersionValid(ver):
+  return ((lambda v: True if(type(v) == str and re.findall

[kkania, 2012/06/09 00:06:10]

Make this more readable.

[nkang, 2012/06/12 23:48:33]

Very readable now.

+                             ('\d+\.\d+\.\d+\.\d+', v) != []) else False)(ver))
+
+def CheckOut(version, dest):
+  """Checks out all necessary source files."""

[kkania, 2012/06/09 00:06:10]

Args:

[nkang, 2012/06/12 23:48:33]

Added description about 'Args'.

+  if not _IsVersionValid(version):
+    print 'Invalid version no. was specified.'

[kkania, 2012/06/09 00:06:10]

throw instead

[nkang, 2012/06/12 23:48:33]

Thrown.

+    return -1
+  if not os.path.isdir(dest):
+    try:
+      os.mkdir(dest)
+    except (OSError, IOError):
+      raise RuntimeError('Could not create %s.\r\n%s.' % (dest, err))  
+  rev_info = _GetRevisionInfo(version)
+  print rev_info

[kkania, 2012/06/09 00:06:10]

take a look at the logging module instead, or just drop this

[nkang, 2012/06/12 23:48:33]

See response I left on the logging comment above.

+  # If its a patch, check out the branch.
+  if rev_info['patch']:
+    svn_url_base = _BASE_SVN_URL + '/branches/%s' % rev_info['branch']
+  # If not, check out the trunk.
+  else:
+    svn_url_base = _BASE_SVN_URL + '/trunk'
+  _SvnCo('%s/src/chrome/test/functional' % svn_url_base,
+         rev_info['revision'], _GetPath(dest, 'functional'))
+  _SvnCo('%s/src/chrome/test/pyautolib' % svn_url_base,
+         rev_info['revision'], _GetPath(dest, 'pyautolib'))
+  _SvnCo('%s/src/third_party/simplejson' % svn_url_base,
+         rev_info['revision'], _GetPath(dest, 'simplejson'))
+  _SvnCo('%s/src/third_party/tlslite' % svn_url_base,
+         rev_info['revision'], _GetPath(dest, 'tlslite'))
+  _SvnCo('%s/src/net/tools/testserver' % svn_url_base,
+         rev_info['revision'], _GetPath(dest, 'testserver'))    
+  _SvnCo(_SELENIUM_URL, _GetRevision(version, 'selenium'),
+         _GetPath(dest, 'selenium'))   
+  _SvnCo(_PYFTPDLIB_URL, _GetRevision(version, 'pyftpdlib'),
+         _GetPath(dest, 'pyftpdlib'))
+  return 0

[kkania, 2012/06/09 00:06:10]

don't return anything

[nkang, 2012/06/12 23:48:33]

The return has been returned. Also updated install_test to reflect this change.