Implement exponential back-off mechanism and enforce it at the URLRequestHttpJob level.

net/url_request/request_throttler_manager.h

+// Copyright (c) 2010 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#ifndef NET_URL_REQUEST_REQUEST_THROTTLER_MANAGER_H_
+#define NET_URL_REQUEST_REQUEST_THROTTLER_MANAGER_H_
+
+#include <map>
+#include <string>
+
+#include "base/basictypes.h"
+#include "base/scoped_ptr.h"
+#include "base/singleton.h"
+#include "googleurl/src/gurl.h"
+#include "net/url_request/request_throttler_entry.h"
+
+// Class that registers request throttler entries for URLs being accessed in
+// order to supervise traffic. URL requests for HTTP contents should register
+// their URLs in this request throttler manager on each request.
+// RequestThrottlerManager maintains a map of URL IDs to request throttler
+// entries. It creates request throttler entries when new URLs are registered,
+// and does garbage collection from time to time in order to clean out outdated
+// entries. URL ID consists of lowercased scheme, host and path. All URLs
+// converted to the same ID will share the same request throttler entry.
+//
+// NOTE: All usage of the singleton object of this class should be on the same

[Jói, 2010/11/21 19:31:40]

Can we / should we assert on this in debug builds, e.g. keep the first thread ID
seen accessing the class and assert that subsequent usages are from the same
thread?

[yzshen, 2010/11/22 17:35:01]

I tried to do so. However, the unit tests use different threads as the IO
thread.

They won't access the request throttler module concurrently, so writing the
request throttler module in a non-thread-safe way has no problem.  But they
prevent me from adding the checks as you suggested.

I think this is another reason that we don't like RequestThrottlerManager to be
a singleton. I will change this in a separate CL, and add the thread checks.

Does it sound okay?

[Jói, 2010/11/22 21:16:27]

Sounds good.

+// thread.
+class RequestThrottlerManager {
+ public:
+  static RequestThrottlerManager* GetInstance();
+
+  // Must be called for every request, returns the request throttler entry
+  // associated with the URL. The caller must inform this entry of some events.
+  // Please refer to request_throttler_entry_interface.h for further
+  // informations.
+  scoped_refptr<RequestThrottlerEntryInterface> RegisterRequestUrl(
+      const GURL& url);
+
+  // Registers a new entry in this service and overrides the existing entry (if
+  // any) for the URL. The service will hold a reference to the entry.
+  // It is only used by unit tests.
+  void OverrideEntryForTests(const GURL& url, RequestThrottlerEntry* entry);
+
+  // Explicitly erases an entry.
+  // This is useful to remove those entries which have got infinite lifetime and
+  // thus won't be garbage collected.
+  // It is only used by unit tests.
+  void EraseEntryForTests(const GURL& url);
+
+ protected:
+  RequestThrottlerManager();
+  ~RequestThrottlerManager();
+
+  // Method that allows us to transform a URL into an ID that can be used in our
+  // map. Resulting IDs will be lowercase and consist of the scheme, host and
+  // path (without query string, fragment, etc.).
+  std::string GetIdFromUrl(const GURL& url);
+
+  // Method that ensures the map gets cleaned from time to time. The period at
+  // which garbage collecting happens is adjustable with the
+  // kRequestBetweenCollecting constant.
+  void GarbageCollectEntriesIfNecessary();
+  // Method that does the actual work of garbage collecting.
+  void GarbageCollectEntries();
+
+  // Used by tests.
+  int GetNumberOfEntriesForTests() const { return url_entries_.size(); }
+
+ private:
+  friend struct DefaultSingletonTraits<RequestThrottlerManager>;
+
+  // From each URL we generate an ID composed of the scheme, host and path
+  // that allows us to uniquely map an entry to it.
+  typedef std::map<std::string, scoped_refptr<RequestThrottlerEntry> >
+      UrlEntryMap;
+
+  // Maximum number of entries that we are willing to collect in our map.
+  static const unsigned int kMaximumNumberOfEntries;
+  // Number of requests that will be made between garbage collection.
+  static const unsigned int kRequestsBetweenCollecting;
+
+  // Map that contains a list of URL ID and their matching
+  // RequestThrottlerEntry.
+  UrlEntryMap url_entries_;
+
+  // This keeps track of how many requests have been made. Used with
+  // GarbageCollectEntries.
+  unsigned int requests_since_last_gc_;
+
+  DISALLOW_COPY_AND_ASSIGN(RequestThrottlerManager);
+};
+
+#endif  // NET_URL_REQUEST_REQUEST_THROTTLER_MANAGER_H_