Add an opt-out header for HTTP throttling. Never throttle for localhost.

net/url_request/url_request_throttler_entry.h

Index: net/url_request/url_request_throttler_entry.h
diff --git a/net/url_request/url_request_throttler_entry.h b/net/url_request/url_request_throttler_entry.h
index ddea91902acd1b3b70eaed8369f62cae6baa9a87..2330dbc9e67236abc94fb16315ec07cb9492f7fd 100644
--- a/net/url_request/url_request_throttler_entry.h
+++ b/net/url_request/url_request_throttler_entry.h
@@ -1,4 +1,4 @@
-// Copyright (c) 2010 The Chromium Authors. All rights reserved.
+// Copyright (c) 2011 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
@@ -15,6 +15,8 @@
 
 namespace net {
 
+class URLRequestThrottlerManager;
+
 // URLRequestThrottlerEntry represents an entry of URLRequestThrottlerManager.
 // It analyzes requests of a specific URL over some period of time, in order to
 // deduce the back-off time for every request.
@@ -34,6 +36,9 @@ class URLRequestThrottlerEntry : public URLRequestThrottlerEntryInterface {
   // Maximum number of requests allowed in sliding window period.
   static const int kDefaultMaxSendThreshold;
 
+  // Number of initial errors to ignore before starting exponential back-off.
+  static const int kDefaultNumErrorsToIgnore;
+
   // Initial delay for exponential back-off.
   static const int kDefaultInitialBackoffMs;
 
@@ -50,16 +55,26 @@ class URLRequestThrottlerEntry : public URLRequestThrottlerEntryInterface {
   // Time after which the entry is considered outdated.
   static const int kDefaultEntryLifetimeMs;
 
-  // Name of the header that servers can use to ask clients to delay their next
-  // request.
+  // Name of the header that servers can use to ask clients to delay their
+  // next request.
   static const char kRetryHeaderName[];
 
-  URLRequestThrottlerEntry();
+  // Name of the header that sites can use to opt out of exponential back-off
+  // throttling.
+  static const char kExponentialThrottlingHeader[];
+
+  // Value for exponential throttling header that can be used to opt out of
+  // exponential back-off throttling.
+  static const char kExponentialThrottlingDisableValue[];
+
+  // The manager object's lifetime must enclose the lifetime of this object.

[yzshen1, 2011/03/18 22:49:51]

It is unlikely that the entries will outlive the manager. However, since the
entries are ref-counted, it is not easy to prove this, especially after we
change the manager to be a member of URLRequestContext.

For example, to come to this conclusion, we need to do the following reasoning:
1) Only URLRequestHTTPJob or URLFetcher will hold refs to the entries.
2) They won't outlive the corresponding URLRequestContext.
3) The manager lives as long as the URLRequestContext that contains it.
4) As a result, the manager's lifetime will enclose the lifetime of the entries.

There are a lot of implicit dependencies in the middle. I think it indicates
design flaws.

William suggested that we need to make URLRequestThrottlerEntry *not*
ref-counted. I think that is probably a good idea.
What do you think?

[Jói, 2011/03/23 23:38:24]

I'm not sure about making the entries not ref-counted.  It seems we have two
choices:

a) Leave them ref-counted, which lets us have an interface where you do a lookup
for an entry once (via RegisterRequestUrl) and then use that entry for the
period you need it, which normally is short.

b) Remove ref-counting and force a look-up for every usage.  This would get rid
of ref-counting but would force two calculations per request/response of the
url_id, two lookups in the entry map, etc.

For now, I will leave it refcounted but am adding a TODO to revisit.  I've made
the back-pointer to the manager weak, and the manager now asks the entries to
detach the pointer during its destruction.

[yzshen1, 2011/03/24 20:06:37]

Sounds good.

Another thing that might worth considering is to explicitly use WeakPtr<>.

[Jói, 2011/03/24 22:03:45]

Thanks, that's a good suggestion, but seeing as the manager already maintains a
list of all referrers, there was no additional overhead of doing it this way
(normally, the WeakPtr pattern allows you to skip maintaining the list of all
objects that have a reference to you).

+  URLRequestThrottlerEntry(URLRequestThrottlerManager* manager);

[yzshen1, 2011/03/18 22:49:51]

Please use explicit.

[Jói, 2011/03/23 23:38:24]

Done.

 
   // The life span of instances created with this constructor is set to
   // infinite.
   // It is only used by unit tests.
-  URLRequestThrottlerEntry(int sliding_window_period_ms,
+  URLRequestThrottlerEntry(URLRequestThrottlerManager* manager,
+                           int sliding_window_period_ms,
                            int max_send_threshold,
                            int initial_backoff_ms,
                            double multiply_factor,
@@ -70,12 +85,16 @@ class URLRequestThrottlerEntry : public URLRequestThrottlerEntryInterface {
   // collected.
   bool IsEntryOutdated() const;
 
+  // Causes this entry to never reject requests due to back-off.
+  void DisableBackoffThrottling();
+
   // Implementation of URLRequestThrottlerEntryInterface.
   virtual bool IsDuringExponentialBackoff() const;
   virtual int64 ReserveSendingTimeForNextRequest(
       const base::TimeTicks& earliest_time);
   virtual base::TimeTicks GetExponentialBackoffReleaseTime() const;
   virtual void UpdateWithResponse(
+      const std::string& host,
       const URLRequestThrottlerHeaderInterface* response);
   virtual void ReceivedContentWasMalformed();
 
@@ -90,7 +109,11 @@ class URLRequestThrottlerEntry : public URLRequestThrottlerEntryInterface {
   // Used internally to increase release time following a retry-after header.
   void HandleCustomRetryAfter(const std::string& header_value);
 
-  // Retrieves the backoff entry object we're using. Used to enable a
+  // Used internally to handle the opt-out header.
+  void HandleThrottlingHeader(const std::string& header_value,
+                              const std::string& host);
+
+  // Retrieves the back-off entry object we're using. Used to enable a
   // unit testing seam for dependency injection in tests.
   virtual const BackoffEntry* GetBackoffEntry() const;
   virtual BackoffEntry* GetBackoffEntry();
@@ -121,9 +144,15 @@ class URLRequestThrottlerEntry : public URLRequestThrottlerEntryInterface {
   const base::TimeDelta sliding_window_period_;
   const int max_send_threshold_;
 
+  // True if DisableBackoffThrottling() has been called on this object.
+  bool is_backoff_disabled_;
+
   // Access it through GetBackoffEntry() to allow a unit test seam.
   BackoffEntry backoff_entry_;
 
+  // Weak back-reference to the manager object managing us.
+  URLRequestThrottlerManager* manager_;
+
   DISALLOW_COPY_AND_ASSIGN(URLRequestThrottlerEntry);
 };