Implement exponential back-off mechanism and enforce it at the URLRequestHttpJob level.

net/base/run_all_unittests.cc

Index: net/base/run_all_unittests.cc
===================================================================
--- net/base/run_all_unittests.cc	(revision 64189)
+++ net/base/run_all_unittests.cc	(working copy)
@@ -6,6 +6,7 @@
 #include "base/metrics/histogram.h"
 #include "base/nss_util.h"
 #include "net/base/net_test_suite.h"
+#include "net/request_throttler/request_throttler_manager.h"
 #if defined(OS_WIN)
 #include "net/socket/ssl_client_socket_nss_factory.h"
 #endif
@@ -24,5 +25,7 @@
   base::EnsureNSPRInit();
 #endif
 
+  Singleton<RequestThrottlerManager>::get()->DisableThrottlingByDefault();

[joi, 2010/11/09 20:07:17]

I'm a bit wary of this. Since we're trying to make the throttler as unobtrusive
as possible, it would be nice to see that most of the tests pass without doing
anything special. The only tests that should really be affected are tests that
are making requests that fail with e.g. 4xx or 5xx HTTP error codes, and this
would hopefully be a small subset of the tests.

Can we keep the throttler enabled and turn it off (or add entries) for just the
tests needed? If it's causing lots of tests to fail right now I think that's
actually a bad sign.

[yzshen, 2010/11/09 22:32:22]

Yes, tests that deal with HTTP failures are a small subset and probably won't
cause trouble.

What bothers us is the sliding window mechanism. It is possible that automatic
tests run so fast that a lot of requests to fetch the same URL are issued in a
short period.

I met two test cases which tried to send 50 requests in a short period. But it
is possible that the sliding window results in flakiness in more obscure way, so
I explicitly disable it.
(Maybe this is a reason why sliding window should not belong to this level?)

+
   return test_suite.Run();
 }