Update parallel_emerge to retry when we encounter DNS errors.

Update parallel_emerge to retry when we encounter DNS errors.

BUG=chromium-os:7049
TEST=Tested that DNS errors are retried. Also tested that 404 errors are not retried.

Change-Id: I0f38764e7af822f7db554697b9fa84af033bf111

Committed: http://chrome-svn/viewvc/chromeos?view=rev&revision=1a6b812

[Nick Sanders, 2010-09-25 00:15:32]

lgtm

On Fri, Sep 24, 2010 at 11:12 AM, <davidjames@chromium.org> wrote:

> Reviewers: Nick Sanders,
>
> Description:
> Update parallel_emerge to retry when we encounter DNS errors.
>
> BUG=chromium-os:7049
> TEST=Tested that DNS errors are retried. Also tested that 404 errors are
> not
> retried.
>
> Change-Id: I0f38764e7af822f7db554697b9fa84af033bf111
>
> Please review this at http://codereview.chromium.org/3473017/show
>
> SVN Base: http://git.chromium.org/git/crosutils.git
>
> Affected files:
>  M parallel_emerge
>
>
> Index: parallel_emerge
> diff --git a/parallel_emerge b/parallel_emerge
> index
>
03684c2ed0d702685d95c46a0c33e53d46250b44..61e9920efb08e427ec1dd32ad2bba8dccdace3e4
> 100755
> --- a/parallel_emerge
> +++ b/parallel_emerge
> @@ -924,16 +924,38 @@ class DepGraphGenerator(object):
>       if not binhost_url:
>         return {}
>
> -      url = binhost_url + "/Packages"
> +      def retry_urlopen(url, tries=3):
> +        """Open the specified url, retrying if we run into network errors.
>
> -      prebuilt_pkgs = {}
> +        We do not retry for HTTP errors.
> +
> +        Args:
> +          url: The specified url.
> +          tries: The number of times to try.
> +
> +        Returns:
> +          The result of urllib2.urlopen(url).
> +        """
> +        for i in range(tries):
> +          try:
> +            return urllib2.urlopen(url)
> +          except urllib2.HTTPError as e:
> +            raise
> +          except urllib2.URLError as e:
> +            if i + 1 == tries:
> +              raise
> +            else:
> +              print "Cannot GET %s: %s" % (url, e)
> +
> +      url = binhost_url + "/Packages"
>       try:
> -        f = urllib2.urlopen(url)
> +        f = retry_urlopen(url)
>       except urllib2.HTTPError as e:
>         if e.code == 404:
>           return {}
>         else:
>           raise
> +      prebuilt_pkgs = {}
>       for line in f:
>         if line.startswith("CPV: "):
>           pkg = line.replace("CPV: ", "").rstrip()
> @@ -1144,7 +1166,8 @@ class DepGraphGenerator(object):
>         local_pkgs = LocalPackageDatabase()
>         remote_pkgs = {}
>         if "--getbinpkg" in emerge.opts:
> -          remote_pkgs =
> RemotePackageDatabase(emerge.settings["PORTAGE_BINHOST"])
> +          binhost = emerge.settings["PORTAGE_BINHOST"]
> +          remote_pkgs = RemotePackageDatabase(binhost)
>         AutoRebuildDeps(local_pkgs, remote_pkgs, cycles)
>
>     # We need to remove installed packages so that we can use the
> dependency
>
>
>