Implement exponential back-off mechanism and enforce it at the URLRequestHttpJob level.

net/url_request/request_throttler_manager.h

+// Copyright (c) 2010 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#ifndef NET_URL_REQUEST_REQUEST_THROTTLER_MANAGER_H_
+#define NET_URL_REQUEST_REQUEST_THROTTLER_MANAGER_H_
+
+#include <map>
+#include <string>
+
+#include "base/basictypes.h"
+#include "base/scoped_ptr.h"
+#include "base/singleton.h"
+#include "googleurl/src/gurl.h"
+#include "net/url_request/request_throttler_entry.h"
+
+// Class that registers request throttler entries for URLs being accessed in
+// order to supervise traffic. URL requests for HTTP contents should register
+// their URLs in this request throttler manager on each request.
+// RequestThrottlerManager maintains a map of URL IDs to request throttler
+// entries. It creates request throttler entries when new URLs are registered,
+// and does garbage collection from time to time in order to clean out outdated
+// entries. URL ID consists of lowercased scheme, host and path. All URLs
+// converted to the same ID will share the same request throttler entry.
+//
+// NOTE: All usage of the singleton object of this class should be on the same
+// thread.

[willchan no longer on Chromium, 2010/11/24 00:23:07]

Please use ThreadChecker to enforce this.

[willchan no longer on Chromium, 2010/11/24 03:10:30]

Btw, I would have said NonThreadSafe, except this is a Singleton, which means
it'll be deleted on the main thread, instead of the IO thread.  Eventually, when
you move this to the URLRequestContext, it should be NonThreadSafe, so it gets
created and deleted on the IO thread.

[yzshen, 2010/11/24 09:42:08]

I tried to do so. However, the unit tests use different threads as the IO
thread.

They won't access the request throttler module concurrently, so writing the
request throttler module in a non-thread-safe way has no problem.
But they prevent me from adding the checks.

I think this is another reason that we don't like
RequestThrottlerManager to be a singleton. As we discussed before, I will make
another CL for it.

[yzshen, 2010/11/24 09:42:08]

I tried to use NonThreadSafe and used the LeakySingletonTraits before. However,
as I mentioned above, unit tests use different IO threads.

I would put it in the same CL of moving this to URLRequestContext, if you don't
mind.

+class RequestThrottlerManager {
+ public:
+  static RequestThrottlerManager* GetInstance();
+
+  // Must be called for every request, returns the request throttler entry
+  // associated with the URL. The caller must inform this entry of some events.
+  // Please refer to request_throttler_entry_interface.h for further
+  // informations.
+  scoped_refptr<RequestThrottlerEntryInterface> RegisterRequestUrl(

[willchan no longer on Chromium, 2010/11/24 00:23:07]

You're using ref counted RequestThrottlerEntryInterfaces.  Please document
somewhere the intended ownership semantics of this object.  How are they
supposed to ever get deleted?  It looks like RequestThrottlerManager doesn't
expose any non-test interface for unregistering a RequestThrottlerEntry.  How
will the RequestThrottlerManager ever release its reference?

Oh I see, you use garbage collection.  That's fine.  It doesn't explain why
you're using reference counting.  Let's think about this conceptually, shouldn't
RequestThrottlerManager be owning these entries?  From what I can tell, you're
using reference counting because you don't know who is holding onto the entry
still.  Is that correct?  This sounds suboptimal.  The manager shouldn't try to
garbage collect an entry that is "active", where "active" means that a
URLFetcher or something is holding onto it.  Rather than refcounting the entry,
you should let clients register themselves with the entry.  And make them
unregister themselves from the entry too.  The RequestThrottlerManager should
own the entry and should make sure to check for other references to the entry,
or should cancel their references via base::WeakPtr or something like that.

Please think REAL HARD whenever you use RefCounted*.  It is usually the wrong
thing to use.

+      const GURL& url);
+
+  // Registers a new entry in this service and overrides the existing entry (if
+  // any) for the URL. The service will hold a reference to the entry.
+  // It is only used by unit tests.
+  void OverrideEntryForTests(const GURL& url, RequestThrottlerEntry* entry);
+
+  // Explicitly erases an entry.
+  // This is useful to remove those entries which have got infinite lifetime and
+  // thus won't be garbage collected.
+  // It is only used by unit tests.
+  void EraseEntryForTests(const GURL& url);
+
+ protected:
+  RequestThrottlerManager();
+  ~RequestThrottlerManager();
+
+  // Method that allows us to transform a URL into an ID that can be used in our
+  // map. Resulting IDs will be lowercase and consist of the scheme, host and
+  // path (without query string, fragment, etc.).
+  std::string GetIdFromUrl(const GURL& url);

[willchan no longer on Chromium, 2010/11/24 03:10:30]

const member function.

[yzshen, 2010/11/24 09:42:08]

Done.

+
+  // Method that ensures the map gets cleaned from time to time. The period at
+  // which garbage collecting happens is adjustable with the
+  // kRequestBetweenCollecting constant.
+  void GarbageCollectEntriesIfNecessary();
+  // Method that does the actual work of garbage collecting.
+  void GarbageCollectEntries();
+
+  // Used by tests.
+  int GetNumberOfEntriesForTests() const { return url_entries_.size(); }
+
+ private:
+  friend struct DefaultSingletonTraits<RequestThrottlerManager>;
+
+  // From each URL we generate an ID composed of the scheme, host and path
+  // that allows us to uniquely map an entry to it.
+  typedef std::map<std::string, scoped_refptr<RequestThrottlerEntry> >
+      UrlEntryMap;
+
+  // Maximum number of entries that we are willing to collect in our map.
+  static const unsigned int kMaximumNumberOfEntries;
+  // Number of requests that will be made between garbage collection.
+  static const unsigned int kRequestsBetweenCollecting;
+
+  // Map that contains a list of URL ID and their matching
+  // RequestThrottlerEntry.
+  UrlEntryMap url_entries_;
+
+  // This keeps track of how many requests have been made. Used with
+  // GarbageCollectEntries.
+  unsigned int requests_since_last_gc_;
+
+  DISALLOW_COPY_AND_ASSIGN(RequestThrottlerManager);
+};
+
+#endif  // NET_URL_REQUEST_REQUEST_THROTTLER_MANAGER_H_