Chrome updater test framework

install_test/install_test.py

+#!/usr/bin/env python
+# Copyright (c) 2012 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.

[Nirnimesh, 2012/08/09 21:32:48]

docstring.

Give an example test. Look at the top of pyauto.py

[nkang, 2012/08/16 23:46:24]

Added a docstring and included instructions on how updater tests can utilize
this framework. As for the example test, I included it in the docstring of
InstallTest, similar to how PyAuto.py provides an example in the PyUITest class.

+
+import httplib
+import logging
+import optparse
+import os
+import platform
+import shutil
+import stat
+import sys
+import tempfile
+import unittest
+import urllib
+import urlparse
+
+import chrome_checkout
+import chrome_installer
+from chrome_installer import ChromeInstallation
+
+sys.path.append(os.path.join(os.path.pardir, 'pyautolib'))
+
+# This import should go after sys.path is set appropriately.
+from fetch_prebuilt_pyauto import FetchPrebuilt
+from pyauto_utils import GTestTextTestRunner
+
+_OPTIONS = None

[Nirnimesh, 2012/08/09 21:32:48]

Do not use globals. What you're doing can easily be done without it.

[nkang, 2012/08/16 23:46:24]

I actually did it this way because I was trying to emulate PyAuto. This is
exactly how PyAuto does it. But if you still want me to change it, let me know
and I'll go ahead and change it.

[Nirnimesh, 2012/08/22 07:06:34]

globals should not be used unless absolutely necessary.
pyauto did not have that luxury, but this class might be able to do without it.

[nkang, 2012/08/24 22:45:26]

Per our conversation, I created a 'staticmethod' in InstallTest called
SetOptions. Main, which parses the command line args, calls SetOptions to set
the test options for InstallTest. This will eliminate the need for a global
variable. After implementing this change, I got rid of the global _OPTIONS
variable. 

+
+
+class InstallTest(unittest.TestCase):
+  """Test fixture for tests involving installing/updating Chrome.
+
+  Provides an interface to install or update chrome from within a testcase, and
+  allows users to run pyauto tests using the installed version. User and system
+  level installations are supported, and either one can be used for running the
+  pyauto tests. Pyautolib files are downloaded at runtime and a PyUITest object
+  is created when Chrome is installed or updated. Users can utilize that object
+  to run updater tests. All Updater tests should derive from this class.
+  """
+
+  _build_iterator = None
+  _current_build = ''
+  _current_location = ''
+  _dir_prefix = '__CHRBLD__'

[Nirnimesh, 2012/08/09 21:32:48]

why the mystique name?

[nkang, 2012/08/16 23:46:24]

We have a particular naming scheme for folders where all the dependencies are
downloaded. It is, _dir_prefix + version_number, so, for example, for version
22.0.1230.0, the folder name will be '__CHRBLD__22.0.1230.0'. The dependencies
are downloaded only once, before firing off the first test. We created this
naming scheme, so subsequent tests can use it to locate the corresponding
dependencies folder in the temp directory. Before we were using tempfile to get
random folder names, but those names had to be kept in a global var, so they
wouldn't be lost when a new instance of InstallTest is created by the next test.
This eliminates the need for global variables.

[Nirnimesh, 2012/08/22 07:06:34]

Declare this convention somewhere.

+  _dir_iterator = None
+  _installation = None
+  _installer_name = 'mini_installer.exe'      

[Nirnimesh, 2012/08/09 21:32:48]

So all this is win only? Where did you declare/check it?

[nkang, 2012/08/16 23:46:24]

Updated the docstring so it states that only Windows is supported. As for
checking it, Ken suggested to remove checks that verify if the current platform
is Windows because, per Ken, these scripts will only be run on Windows.

+  _pyauto = None
+  _installers = []
+  _download_dirs = []
+
+  def __init__(self, methodName='runTest'):
+    unittest.TestCase.__init__(self, methodName)    
+    self._platform = self._GetPlatform()
+    self._Initialize()     
+    current_build = ChromeInstallation.GetCurrent()    
+    if current_build:
+      current_build.Uninstall()      
+    for build in self._builds:
+      if not self._DownloadDeps(build):

[Nirnimesh, 2012/08/09 21:32:48]

Do you want the downloading to be done separately for every test?

Remember that unittest module initializes all tests in one go, and then runs
their setup/test/tearDown one by one.

[nkang, 2012/08/16 23:46:24]

No, the downloading will occur only once. This is why we created the mystique
name :)

Each specified build has a corresponding folder in the temp directory that gets
created only once. These folders have a unique name, which ties them to a
particular build. So, for example, let's say you want to test with builds
22.0.1230.0 and 22.0.1231.0. Before the first test can begin, two folders will
get created in the temp directory, the first named __CHRBLD__22.0.1230.0 and the
second named __CHRBLD__22.0.1231.0. One by one, dependencies for each build will
be downloaded into the corresponding folders. The method that downloads all the
dependencies checks if a folder that corresponds with the current build exists
in the temp directory and only downloads if it does not. This will only be the
case for the first testcase. For every subsequent test, the method will check in
the temp directory, determine that the deps folder already exists, and simply do
nothing. We also took into account the fact that something could go wrong during
the
download process. To eliminate the possibility of an incomplete deps folder, we
first download everything into a temp folder with a randomly generated name. If
all download/checkouts are successful, that folder is then renamed using our
naming scheme. 

+        raise RuntimeError('Could not download dependencies.')    
+    self._installer_iter = iter(self._installers)
+    self._dir_iterator = iter(self._download_dirs)        
+
+  def _Initialize(self):
+    """Sets test parameters."""
+    global _OPTIONS
+    self._url = _OPTIONS.url
+    self._builds = _OPTIONS.builds and _OPTIONS.builds.split(',') or []
+    if not self._url or not self._builds:
+      raise RuntimeError('Please specify a valid URL and two Chrome builds.')
+    self._builds.sort()         
+    self._url = self._url.endswith('/') and self._url or self._url + '/'
+    self._dir = os.path.isdir(_OPTIONS.dir) and _OPTIONS.dir or os.getcwd()    
+    self._options = (_OPTIONS.options and _OPTIONS.options.replace(',', ' ')
+                     or '')
+    self._install_type = ('system-level' in self._options and
+                          chrome_installer.InstallationType.SYSTEM or
+                          chrome_installer.InstallationType.USER)
+    self._build_iterator = iter(self._builds)
+    self._current_build = next(self._build_iterator, None)
+
+  def setUp(self):
+    """Called before each unittest to prepare the test fixture."""    
+    self.InstallBuild()
+    self.failIf(self._pyauto == None)

[Nirnimesh, 2012/08/09 21:32:48]

call parent's setup

[nkang, 2012/08/16 23:46:24]

Sorry, I didn't fully understand this comment. The parent class of InstallTest
is unittest.TestCase, and the default implementations of setUp and tearDown do
not do anything. So I didn't quite understand why we should call parent's setup
method here. 

[Nirnimesh, 2012/08/22 07:06:34]

The fact that the parent does nothing is irrelevant. It's good to call the
parent's method unless you explicitly want to override it.

+
+  def tearDown(self):
+    """Called at the end of each unittest to do any test related cleanup."""
+    self._Refresh()
+    self._DeleteBuild()

[Nirnimesh, 2012/08/09 21:32:48]

call parent's tearDown

[nkang, 2012/08/16 23:46:24]

The parent class of InstallTest is unittest.TestCase, and the default
implementations of setUp and tearDown do not do anything. 

+  
+  def _GetPlatform(self):
+    """Returns the platform name."""
+    return ({'Windows': 'win',
+             'Darwin': 'mac',
+             'Linux': 'linux'}).get(platform.system())
+
+  def _Refresh(self):

[Nirnimesh, 2012/08/09 21:32:48]

Refresh? It's not clear from the method name that it does clearing. Use better
name

[nkang, 2012/08/16 23:46:24]

Renamed to _ClearModulesRegistry.

+    """Deletes the PyUITest object and clears the modules registry."""
+    if self._pyauto:
+      del(self._pyauto)

[Nirnimesh, 2012/08/09 21:32:48]

remove parens

[nkang, 2012/08/16 23:46:24]

Done.

+    for module in ['pyauto', 'pyautolib', '_pyautolib']:
+      if module in sys.modules:
+        sys.modules.pop(module)
+
+  def _RemovePaths(self):
+    """Restores the sys.path variable to its original state."""    
+    sys.path = list(frozenset(sys.path))
+    if self._current_location:
+      if self._current_location in sys.path:
+        sys.path.remove(self._current_location)    
+      if os.path.join(self._current_location, 'pyautolib') in sys.path:      
+        sys.path.remove(os.path.join(self._current_location, 'pyautolib'))   
+
+  def _Install(self):
+    """Helper method that installs Chrome and creates a PyUITest object."""   
+    self._pyauto = None    
+    installer_path = next(self._installer_iter, None)
+    if not installer_path:
+      raise RuntimeError('No more builds left to install.')
+    self._installation = chrome_installer.Install(installer_path,
+                                                  self._install_type,
+                                                  self._current_build,
+                                                  self._options)
+    try:
+      import pyauto
+      self._pyauto = pyauto.PyUITest(methodName='runTest',
+                                     browser_path=os.path.dirname(
+                                       self._installation.GetExePath()))
+      self._pyauto.suite_holder = pyauto.PyUITestSuite(['test.py'])

[Nirnimesh, 2012/08/09 21:32:48]

[] would do I think

[nkang, 2012/08/16 23:46:24]

Yes, it did. Got rid of the arbitrary file name.

+      self._pyauto.setUp()      
+    except ImportError, err:
+      logging.log(logging.ERROR, err)        
+      self._pyauto = None
+
+  def InstallBuild(self):
+    self._current_location = next(self._dir_iterator, None)    
+    sys.path.insert(0, self._current_location)
+    sys.path.insert(1, os.path.join(self._current_location, 'pyautolib'))
+    self._Install()
+
+  def _Update(self):
+    """Helper method for updating Chrome."""    
+    self._RemovePaths()
+    self._current_location = next(self._dir_iterator, None)    
+    assert (self._current_location)
+    sys.path.insert(0, self._current_location)
+    sys.path.insert(1, os.path.join(self._current_location, 'pyautolib'))
+    build = next(self._build_iterator, None)    
+    if not build:
+      raise RuntimeError('No more builds left to install. Following builds '
+                         'have already been installed: %r' % self._builds)          
+    self._current_build = build            
+    self._Install()
+
+  def UpdateBuild(self):    
+    """Installs the second Chrome build specified in the command args."""
+    if self._pyauto:
+      self._pyauto.TearDown()
+    self._Refresh()
+    self._Update()
+
+  def _SrcFilesExist(self, root, items):
+    """Checks if specified files/folders exist at specified 'root' folder.
+
+    Args:
+      root: Parent folder where all the source directories reside.
+      items: List of files/folders to be verified for existence in the root.
+
+    Returns:
+      True, if all sub-folders exist in the root, otherwise False.
+    """
+    return all(map(lambda p: os.path.exists(p) and True or False,
+                   [os.path.join(root, path) for path in items]))
+
+  def _CheckoutSourceFiles(self, build, location):
+    """Checks out source files associated with the current build.
+
+    Args:
+      build: Chrome release version number.
+      location: Destination where source files will be saved.
+    Returns:
+      Zero if successful, otherwise a negative value.
+    """
+    try:
+      chrome_checkout.CheckOut(build, location)
+      return True
+    except AssertionError:
+      return False
+
+  def _FetchPrebuiltPyauto(self, url, location):
+    """Fetches the specified Chrome build.
+
+    Args:
+      url: URL where the build is located.
+      location: Location where the build will be downloaded.
+
+    Returns:
+      True if successful, otherwise False.
+    """    
+    fetch_build = FetchPrebuilt(url, location, self._platform)
+    if fetch_build.DoesUrlExist(url):
+      return fetch_build.Run() == 0
+    else:
+      return False
+
+  def _DownloadInstaller(self, url, location):
+    """Downloads the Chrome installer.
+
+    Args:
+      url: URL where the installer is located.
+      location: Location where installer will be downloaded.
+
+    Returns:
+      True if successful, otherwise False.
+    """
+    try:
+      self._Download(self._installer_name, url, location)
+      return True
+    except (IOError, RuntimeError):
+      return False
+
+  def _DownloadDeps(self, build):
+    """Downloads Chrome build, source files, and Chrome installer.
+
+    Args:
+      build: Chrome release build number.
+
+    Returns:
+      True if successful, otherwise False.
+    """    
+    url = '%s%s/%s' % (self._url, build, self._platform)
+    location = os.path.join('%s', '%s%s') % (tempfile.gettempdir(),
+                                             self._dir_prefix, build)    
+    if os.path.isdir(location):
+      self._download_dirs.append(location)
+      self._installers.append(os.path.join(location, self._installer_name))
+      return True 
+    else:      
+      tmpdir = tempfile.mkdtemp()            
+      if self._CheckoutSourceFiles(build, tmpdir):              
+        if self._FetchPrebuiltPyauto(url, tmpdir):          
+          if self._DownloadInstaller(url, tmpdir):            
+            try:              
+              # This is a workaround because rename was causing problems.
+              shutil.copytree(tmpdir, location)
+              # Callback is there because hidden svn files are read-only, so
+              # we need to change their permissions on the fly to delete them.
+              shutil.rmtree(tmpdir, onerror=self._OnError)
+              self._download_dirs.append(location)
+              self._installers.append(os.path.join(location,
+                                                   self._installer_name))
+              return True
+            except(OSError, IOError), err:
+              return False
+    return False
+
+  def _OnError(self, func, path, exc_info):

[Nirnimesh, 2012/08/09 21:32:48]

I've seen this elsewhere. Move this and line 252, 280.. to a helper method.

[nkang, 2012/08/16 23:46:24]

Put line no. 252 in a helper method, and made the callback a nested method
inside the helper method, _DeleteDir.

+    """Callback for shutil.rmtree."""    
+    if not os.access(path, os.W_OK):
+      os.chmod(path, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
+      func(path)
+    
+  def _DeleteBuild(self):
+    """Uninstalls Chrome."""
+    self._installation.Uninstall()          
+    self._build_iterator = iter(self._builds)      
+    self._dir_iterator = iter(self._download_dirs)
+    self._current_build = next(self._build_iterator, None)
+    self._current_location = next(self._dir_iterator, None)
+    self._RemovePaths()    
+
+  def _DeleteDepFiles(self):
+    """Deletes Chrome related files that were downloaded for testing."""
+    for path in self._download_dirs:
+      try:
+        shutil.rmtree(path, onerror=self._OnError)
+      except shutil.Error, err:
+        logging.log(logging.ERROR, err)        
+        return -1
+    return 0
+
+  def _Download(self, dfile, url, dest=None):

[Nirnimesh, 2012/08/09 21:32:48]

I've seen similar function elsewhere. Reuse code.

[nkang, 2012/08/16 23:46:24]

The only other method that's somewhat similar is
chrome_checkout._GetContentsAndReturnResponse. I could probably use that, but
the only problem is, in that method, we don't actually download anything; it all
happens in memory. In order to download the file, we'd still have to write
additional code that calls _GetContentsAndReturnResponse and reads the contents
of the installer into a buffer and then writes that to a file. 

I think you probably meant _DoesURLExist because a similar method also exists in
fetch_prebuilt_pyauto. After checking with Nirnimesh, moved DoesUrlExist from
this class and FetchPrebuilt to pyauto_utils.py.

+    """Downloads a file from the specified URL.
+
+    Args:
+      dfile: Name of the file to download.
+      url: URL where the file is located.
+      dest: Location where file will be downloaded. Default is CWD.            
+
+    Returns:
+      Zero if successful, otherwise a non-zero value.
+    """
+    filename = ((dest and os.path.exists(dest)) and os.path.join(dest, dfile)
+                or os.path.join(tempfile.gettempdir(), dfile))
+    file_url = '%s/%s' % (url, dfile)
+    if not self._DoesUrlExist(file_url):
+      raise RuntimeError('Either the URL or the file name is invalid.')
+    try:
+      d = urllib.urlretrieve(file_url, filename)
+    except IOError, err:
+      raise err
+    return os.path.isfile(d[0])
+
+  def _DoesUrlExist(self, url):
+    """Checks if a URL exists.
+
+    Args:
+      url: URL to be verified.
+
+    Returns:
+      True if the URL exists, otherwise False.
+    """
+    parse = urlparse.urlparse(url)
+    if parse[0] == '' or parse[1] == '':
+      return False
+    try:
+      connection = httplib.HTTPConnection(parse.netloc)
+      connection.request('HEAD', parse.path)
+      response = connection.getresponse()
+    except (socket.error, socket.gaierror):
+      return False
+    finally:
+      connection.close()
+    if response.status == 302 or response.status == 301:
+      return self._DoesUrlExist(response.getheader('location'))
+    return response.status == 200        
+
+
+class Main(object):
+  """Main program for running Updater tests."""
+
+  _tests_file = 'PYAUTO_TESTS'
+  _mod_path = sys.argv[0]
+  _pyauto_doc_url = 'http://dev.chromium.org/developers/testing/pyauto'
+  
+  def __init__(self):    
+    self._ParseArgs()
+    self._Run()    
+
+  def _GetUnitTests(self):

[Nirnimesh, 2012/08/09 21:32:48]

remove 'Unit'

[nkang, 2012/08/16 23:46:24]

Renamed to _GetTests.

+    """Returns a list of unittests from the calling script."""
+    mod_name = [os.path.splitext(os.path.basename(self._mod_path))[0]]
+    if os.path.dirname(self._mod_path) not in sys.path:
+      sys.path.append(os.path.dirname(self._mod_path))
+    return unittest.defaultTestLoader.loadTestsFromNames(mod_name)
+    
+  def _Run(self):
+    """Runs the unit tests."""    
+    tests = self._GetUnitTests()
+    result = GTestTextTestRunner(verbosity=1).run(tests)
+    del(tests)     
+    if not result.wasSuccessful():
+      print >>sys.stderr, ('Tests can be disabled by editing %s. Ref: %s'
+                           % (self._tests_file, self._pyauto_doc_url))
+      sys.exit(1)
+    else:

[Nirnimesh, 2012/08/09 21:32:48]

remove the else part. it's redundant

[nkang, 2012/08/16 23:46:24]

Removed else.

+      sys.exit(0)
+  
+  def _ParseArgs(self):

[Nirnimesh, 2012/08/09 21:32:48]

Move _ParseArgs before _Run

[nkang, 2012/08/16 23:46:24]

Done.

+    """Parses command line arguments."""
+    global _OPTIONS
+    parser = optparse.OptionParser()
+    parser.add_option(
+        '-b', '--builds', type='string', default='', dest='builds',
+        help='Specifies the two (or more) builds needed for testing.')
+    parser.add_option(
+        '-u', '--url', type='string', default='', dest='url',
+        help='Specifies the chrome-master2 url, without the build number.')

[Nirnimesh, 2012/08/09 21:32:48]

do not refer internal URLs/hostname.

[nkang, 2012/08/16 23:46:24]

Removed the part that referred to an internal URL.

+    parser.add_option(
+        '-d', '--dir', type='string', default=os.getcwd(),
+        help='Specifies directory where the installer will be downloaded.')
+    parser.add_option(
+        '-o', '--options', type='string', default='',
+        help='Specifies any additional Chrome options (i.e. --system-level).')
+    opts, args = parser.parse_args()
+    _OPTIONS = opts

[Nirnimesh, 2012/08/09 21:32:48]

verify the sanity of the args here, esp --builds

[Nirnimesh, 2012/08/09 21:32:48]

Before firing off the install tests, it'd be nice to determine if the individual
chrome builds are sane by themselves and can run pyauto test.

[nkang, 2012/08/16 23:46:24]

Wrote a new method called ValidateArgs, which confirms that all specified builds
have a valid version number and valid urls.

[nkang, 2012/08/16 23:46:24]

I am not entirely sure how we can confirm that the builds can run pyauto tests.
I checked with Ken and he recommended that one solution would be to install the
build and run a simple pyauto test with it (maybe test_basic). I don't know if
that's what you had in mind? If not, I'd welcome any suggestions.

[Nirnimesh, 2012/08/22 07:06:34]

Yes, running a test from test_basic would be fine.

[nkang, 2012/08/24 22:45:26]

After taking everything into consideration, I think this would require a lot of
work. First of all, this would require changes to the checkout script, as we
would need to download the 'functional' folder, which contains test_basic.
Second, since we're launching a PyAuto test, we need to come up with a way to
tell PyAuto where the Chrome binary is. InstallTest overcomes this problem by
creating a PyUITest object and passing the browser path to the class constructor
as a parameter. However, in the case of test_basic, since we're just firing off
a PyAuto test, we don't have that luxury. So we would need to figure out how to
tell PyAuto where the Chrome binary is, as, by default, it would expect it to be
in the same place as the PyAuto binaries. I talked to Ken and Anantha about
this, and they suggested that maybe we should just punt this for now. We can
create another CL for this down the line. Also, Anantha is considering using
ChromeDriver instead of PyAuto, so this may no longer even apply, if we decide
to go with ChromeDriver. I think we should hold off on this, at least until
Anantha makes her decision. What do you think?