| OLD | NEW |
| (Empty) |
| 1 .. _advanced: | |
| 2 | |
| 3 Advanced Usage | |
| 4 ============== | |
| 5 | |
| 6 This document covers some of Requests more advanced features. | |
| 7 | |
| 8 .. _session-objects: | |
| 9 | |
| 10 Session Objects | |
| 11 --------------- | |
| 12 | |
| 13 The Session object allows you to persist certain parameters across | |
| 14 requests. It also persists cookies across all requests made from the | |
| 15 Session instance, and will use ``urllib3``'s `connection pooling`_. So if | |
| 16 you're making several requests to the same host, the underlying TCP | |
| 17 connection will be reused, which can result in a significant performance | |
| 18 increase (see `HTTP persistent connection`_). | |
| 19 | |
| 20 A Session object has all the methods of the main Requests API. | |
| 21 | |
| 22 Let's persist some cookies across requests:: | |
| 23 | |
| 24 s = requests.Session() | |
| 25 | |
| 26 s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') | |
| 27 r = s.get('http://httpbin.org/cookies') | |
| 28 | |
| 29 print(r.text) | |
| 30 # '{"cookies": {"sessioncookie": "123456789"}}' | |
| 31 | |
| 32 | |
| 33 Sessions can also be used to provide default data to the request methods. This | |
| 34 is done by providing data to the properties on a Session object:: | |
| 35 | |
| 36 s = requests.Session() | |
| 37 s.auth = ('user', 'pass') | |
| 38 s.headers.update({'x-test': 'true'}) | |
| 39 | |
| 40 # both 'x-test' and 'x-test2' are sent | |
| 41 s.get('http://httpbin.org/headers', headers={'x-test2': 'true'}) | |
| 42 | |
| 43 | |
| 44 Any dictionaries that you pass to a request method will be merged with the | |
| 45 session-level values that are set. The method-level parameters override session | |
| 46 parameters. | |
| 47 | |
| 48 Note, however, that method-level parameters will *not* be persisted across | |
| 49 requests, even if using a session. This example will only send the cookies | |
| 50 with the first request, but not the second:: | |
| 51 | |
| 52 s = requests.Session() | |
| 53 | |
| 54 r = s.get('http://httpbin.org/cookies', cookies={'from-my': 'browser'}) | |
| 55 print(r.text) | |
| 56 # '{"cookies": {"from-my": "browser"}}' | |
| 57 | |
| 58 r = s.get('http://httpbin.org/cookies') | |
| 59 print(r.text) | |
| 60 # '{"cookies": {}}' | |
| 61 | |
| 62 | |
| 63 If you want to manually add cookies to your session, use the | |
| 64 :ref:`Cookie utility functions <api-cookies>` to manipulate | |
| 65 :attr:`Session.cookies <requests.Session.cookies>`. | |
| 66 | |
| 67 Sessions can also be used as context managers:: | |
| 68 | |
| 69 with requests.Session() as s: | |
| 70 s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') | |
| 71 | |
| 72 This will make sure the session is closed as soon as the ``with`` block is | |
| 73 exited, even if unhandled exceptions occurred. | |
| 74 | |
| 75 | |
| 76 .. admonition:: Remove a Value From a Dict Parameter | |
| 77 | |
| 78 Sometimes you'll want to omit session-level keys from a dict parameter. To | |
| 79 do this, you simply set that key's value to ``None`` in the method-level | |
| 80 parameter. It will automatically be omitted. | |
| 81 | |
| 82 All values that are contained within a session are directly available to you. | |
| 83 See the :ref:`Session API Docs <sessionapi>` to learn more. | |
| 84 | |
| 85 .. _request-and-response-objects: | |
| 86 | |
| 87 Request and Response Objects | |
| 88 ---------------------------- | |
| 89 | |
| 90 Whenever a call is made to ``requests.get()`` and friends you are doing two | |
| 91 major things. First, you are constructing a ``Request`` object which will be | |
| 92 sent off to a server to request or query some resource. Second, a ``Response`` | |
| 93 object is generated once ``requests`` gets a response back from the server. | |
| 94 The Response object contains all of the information returned by the server and | |
| 95 also contains the ``Request`` object you created originally. Here is a simple | |
| 96 request to get some very important information from Wikipedia's servers:: | |
| 97 | |
| 98 >>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python') | |
| 99 | |
| 100 If we want to access the headers the server sent back to us, we do this:: | |
| 101 | |
| 102 >>> r.headers | |
| 103 {'content-length': '56170', 'x-content-type-options': 'nosniff', 'x-cache': | |
| 104 'HIT from cp1006.eqiad.wmnet, MISS from cp1010.eqiad.wmnet', 'content-encodi
ng': | |
| 105 'gzip', 'age': '3080', 'content-language': 'en', 'vary': 'Accept-Encoding,Co
okie', | |
| 106 'server': 'Apache', 'last-modified': 'Wed, 13 Jun 2012 01:33:50 GMT', | |
| 107 'connection': 'close', 'cache-control': 'private, s-maxage=0, max-age=0, | |
| 108 must-revalidate', 'date': 'Thu, 14 Jun 2012 12:59:39 GMT', 'content-type': | |
| 109 'text/html; charset=UTF-8', 'x-cache-lookup': 'HIT from cp1006.eqiad.wmnet:3
128, | |
| 110 MISS from cp1010.eqiad.wmnet:80'} | |
| 111 | |
| 112 However, if we want to get the headers we sent the server, we simply access the | |
| 113 request, and then the request's headers:: | |
| 114 | |
| 115 >>> r.request.headers | |
| 116 {'Accept-Encoding': 'identity, deflate, compress, gzip', | |
| 117 'Accept': '*/*', 'User-Agent': 'python-requests/1.2.0'} | |
| 118 | |
| 119 .. _prepared-requests: | |
| 120 | |
| 121 Prepared Requests | |
| 122 ----------------- | |
| 123 | |
| 124 Whenever you receive a :class:`Response <requests.Response>` object | |
| 125 from an API call or a Session call, the ``request`` attribute is actually the | |
| 126 ``PreparedRequest`` that was used. In some cases you may wish to do some extra | |
| 127 work to the body or headers (or anything else really) before sending a | |
| 128 request. The simple recipe for this is the following:: | |
| 129 | |
| 130 from requests import Request, Session | |
| 131 | |
| 132 s = Session() | |
| 133 | |
| 134 req = Request('POST', url, data=data, headers=headers) | |
| 135 prepped = req.prepare() | |
| 136 | |
| 137 # do something with prepped.body | |
| 138 prepped.body = 'No, I want exactly this as the body.' | |
| 139 | |
| 140 # do something with prepped.headers | |
| 141 del prepped.headers['Content-Type'] | |
| 142 | |
| 143 resp = s.send(prepped, | |
| 144 stream=stream, | |
| 145 verify=verify, | |
| 146 proxies=proxies, | |
| 147 cert=cert, | |
| 148 timeout=timeout | |
| 149 ) | |
| 150 | |
| 151 print(resp.status_code) | |
| 152 | |
| 153 Since you are not doing anything special with the ``Request`` object, you | |
| 154 prepare it immediately and modify the ``PreparedRequest`` object. You then | |
| 155 send that with the other parameters you would have sent to ``requests.*`` or | |
| 156 ``Session.*``. | |
| 157 | |
| 158 However, the above code will lose some of the advantages of having a Requests | |
| 159 :class:`Session <requests.Session>` object. In particular, | |
| 160 :class:`Session <requests.Session>`-level state such as cookies will | |
| 161 not get applied to your request. To get a | |
| 162 :class:`PreparedRequest <requests.PreparedRequest>` with that state | |
| 163 applied, replace the call to :meth:`Request.prepare() | |
| 164 <requests.Request.prepare>` with a call to | |
| 165 :meth:`Session.prepare_request() <requests.Session.prepare_request>`, like this:
: | |
| 166 | |
| 167 from requests import Request, Session | |
| 168 | |
| 169 s = Session() | |
| 170 req = Request('GET', url, data=data, headers=headers) | |
| 171 | |
| 172 prepped = s.prepare_request(req) | |
| 173 | |
| 174 # do something with prepped.body | |
| 175 prepped.body = 'Seriously, send exactly these bytes.' | |
| 176 | |
| 177 # do something with prepped.headers | |
| 178 prepped.headers['Keep-Dead'] = 'parrot' | |
| 179 | |
| 180 resp = s.send(prepped, | |
| 181 stream=stream, | |
| 182 verify=verify, | |
| 183 proxies=proxies, | |
| 184 cert=cert, | |
| 185 timeout=timeout | |
| 186 ) | |
| 187 | |
| 188 print(resp.status_code) | |
| 189 | |
| 190 .. _verification: | |
| 191 | |
| 192 SSL Cert Verification | |
| 193 --------------------- | |
| 194 | |
| 195 Requests verifies SSL certificates for HTTPS requests, just like a web browser. | |
| 196 By default, SSL verification is enabled, and requests will throw a SSLError if | |
| 197 it's unable to verify the certificate:: | |
| 198 | |
| 199 >>> requests.get('https://requestb.in') | |
| 200 requests.exceptions.SSLError: hostname 'requestb.in' doesn't match either of
'*.herokuapp.com', 'herokuapp.com' | |
| 201 | |
| 202 I don't have SSL setup on this domain, so it throws an exception. Excellent. Git
Hub does though:: | |
| 203 | |
| 204 >>> requests.get('https://github.com') | |
| 205 <Response [200]> | |
| 206 | |
| 207 You can pass ``verify`` the path to a CA_BUNDLE file or directory with certifica
tes of trusted CAs:: | |
| 208 | |
| 209 >>> requests.get('https://github.com', verify='/path/to/certfile') | |
| 210 | |
| 211 This list of trusted CAs can also be specified through the ``REQUESTS_CA_BUNDLE`
` environment variable. | |
| 212 | |
| 213 Requests can also ignore verifying the SSL certificate if you set ``verify`` to
False. | |
| 214 | |
| 215 :: | |
| 216 | |
| 217 >>> requests.get('https://kennethreitz.com', verify=False) | |
| 218 <Response [200]> | |
| 219 | |
| 220 By default, ``verify`` is set to True. Option ``verify`` only applies to host ce
rts. | |
| 221 | |
| 222 You can also specify a local cert to use as client side certificate, as a single | |
| 223 file (containing the private key and the certificate) or as a tuple of both | |
| 224 file's path:: | |
| 225 | |
| 226 >>> requests.get('https://kennethreitz.com', cert=('/path/client.cert', '/pa
th/client.key')) | |
| 227 <Response [200]> | |
| 228 | |
| 229 If you specify a wrong path or an invalid cert, you'll get a SSLError:: | |
| 230 | |
| 231 >>> requests.get('https://kennethreitz.com', cert='/wrong_path/client.pem') | |
| 232 SSLError: [Errno 336265225] _ssl.c:347: error:140B0009:SSL routines:SSL_CTX_
use_PrivateKey_file:PEM lib | |
| 233 | |
| 234 .. warning:: The private key to your local certificate *must* be unencrypted. | |
| 235 Currently, requests does not support using encrypted keys. | |
| 236 | |
| 237 .. _ca-certificates: | |
| 238 | |
| 239 CA Certificates | |
| 240 --------------- | |
| 241 | |
| 242 By default Requests bundles a set of root CAs that it trusts, sourced from the | |
| 243 `Mozilla trust store`_. However, these are only updated once for each Requests | |
| 244 version. This means that if you pin a Requests version your certificates can | |
| 245 become extremely out of date. | |
| 246 | |
| 247 From Requests version 2.4.0 onwards, Requests will attempt to use certificates | |
| 248 from `certifi`_ if it is present on the system. This allows for users to update | |
| 249 their trusted certificates without having to change the code that runs on their | |
| 250 system. | |
| 251 | |
| 252 For the sake of security we recommend upgrading certifi frequently! | |
| 253 | |
| 254 .. _HTTP persistent connection: https://en.wikipedia.org/wiki/HTTP_persistent_co
nnection | |
| 255 .. _connection pooling: https://urllib3.readthedocs.io/en/latest/pools.html | |
| 256 .. _certifi: http://certifi.io/ | |
| 257 .. _Mozilla trust store: https://hg.mozilla.org/mozilla-central/raw-file/tip/sec
urity/nss/lib/ckfw/builtins/certdata.txt | |
| 258 | |
| 259 .. _body-content-workflow: | |
| 260 | |
| 261 Body Content Workflow | |
| 262 --------------------- | |
| 263 | |
| 264 By default, when you make a request, the body of the response is downloaded | |
| 265 immediately. You can override this behaviour and defer downloading the response | |
| 266 body until you access the :class:`Response.content <requests.Response.content>` | |
| 267 attribute with the ``stream`` parameter:: | |
| 268 | |
| 269 tarball_url = 'https://github.com/kennethreitz/requests/tarball/master' | |
| 270 r = requests.get(tarball_url, stream=True) | |
| 271 | |
| 272 At this point only the response headers have been downloaded and the connection | |
| 273 remains open, hence allowing us to make content retrieval conditional:: | |
| 274 | |
| 275 if int(r.headers['content-length']) < TOO_LONG: | |
| 276 content = r.content | |
| 277 ... | |
| 278 | |
| 279 You can further control the workflow by use of the :class:`Response.iter_content
<requests.Response.iter_content>` | |
| 280 and :class:`Response.iter_lines <requests.Response.iter_lines>` methods. | |
| 281 Alternatively, you can read the undecoded body from the underlying | |
| 282 urllib3 :class:`urllib3.HTTPResponse <urllib3.response.HTTPResponse>` at | |
| 283 :class:`Response.raw <requests.Response.raw>`. | |
| 284 | |
| 285 If you set ``stream`` to ``True`` when making a request, Requests cannot | |
| 286 release the connection back to the pool unless you consume all the data or call | |
| 287 :class:`Response.close <requests.Response.close>`. This can lead to | |
| 288 inefficiency with connections. If you find yourself partially reading request | |
| 289 bodies (or not reading them at all) while using ``stream=True``, you should | |
| 290 consider using ``contextlib.closing`` (`documented here`_), like this:: | |
| 291 | |
| 292 from contextlib import closing | |
| 293 | |
| 294 with closing(requests.get('http://httpbin.org/get', stream=True)) as r: | |
| 295 # Do things with the response here. | |
| 296 | |
| 297 .. _`documented here`: http://docs.python.org/2/library/contextlib.html#contextl
ib.closing | |
| 298 | |
| 299 .. _keep-alive: | |
| 300 | |
| 301 Keep-Alive | |
| 302 ---------- | |
| 303 | |
| 304 Excellent news — thanks to urllib3, keep-alive is 100% automatic within a sessio
n! | |
| 305 Any requests that you make within a session will automatically reuse the appropr
iate | |
| 306 connection! | |
| 307 | |
| 308 Note that connections are only released back to the pool for reuse once all body | |
| 309 data has been read; be sure to either set ``stream`` to ``False`` or read the | |
| 310 ``content`` property of the ``Response`` object. | |
| 311 | |
| 312 .. _streaming-uploads: | |
| 313 | |
| 314 Streaming Uploads | |
| 315 ----------------- | |
| 316 | |
| 317 Requests supports streaming uploads, which allow you to send large streams or | |
| 318 files without reading them into memory. To stream and upload, simply provide a | |
| 319 file-like object for your body:: | |
| 320 | |
| 321 with open('massive-body', 'rb') as f: | |
| 322 requests.post('http://some.url/streamed', data=f) | |
| 323 | |
| 324 .. warning:: It is strongly recommended that you open files in `binary mode`_. | |
| 325 This is because Requests may attempt to provide the | |
| 326 ``Content-Length`` header for you, and if it does this value will | |
| 327 be set to the number of *bytes* in the file. Errors may occur if | |
| 328 you open the file in *text mode*. | |
| 329 | |
| 330 .. _binary mode: https://docs.python.org/2/tutorial/inputoutput.html#reading-and
-writing-files | |
| 331 | |
| 332 | |
| 333 .. _chunk-encoding: | |
| 334 | |
| 335 Chunk-Encoded Requests | |
| 336 ---------------------- | |
| 337 | |
| 338 Requests also supports Chunked transfer encoding for outgoing and incoming reque
sts. | |
| 339 To send a chunk-encoded request, simply provide a generator (or any iterator wit
hout | |
| 340 a length) for your body:: | |
| 341 | |
| 342 def gen(): | |
| 343 yield 'hi' | |
| 344 yield 'there' | |
| 345 | |
| 346 requests.post('http://some.url/chunked', data=gen()) | |
| 347 | |
| 348 For chunked encoded responses, it's best to iterate over the data using | |
| 349 :meth:`Response.iter_content() <requests.models.Response.iter_content>`. In | |
| 350 an ideal situation you'll have set ``stream=True`` on the request, in which | |
| 351 case you can iterate chunk-by-chunk by calling ``iter_content`` with a chunk | |
| 352 size parameter of ``None``. If you want to set a maximum size of the chunk, | |
| 353 you can set a chunk size parameter to any integer. | |
| 354 | |
| 355 | |
| 356 .. _multipart: | |
| 357 | |
| 358 POST Multiple Multipart-Encoded Files | |
| 359 ------------------------------------- | |
| 360 | |
| 361 You can send multiple files in one request. For example, suppose you want to | |
| 362 upload image files to an HTML form with a multiple file field 'images':: | |
| 363 | |
| 364 <input type="file" name="images" multiple="true" required="true"/> | |
| 365 | |
| 366 To do that, just set files to a list of tuples of ``(form_field_name, file_info)
``:: | |
| 367 | |
| 368 >>> url = 'http://httpbin.org/post' | |
| 369 >>> multiple_files = [ | |
| 370 ('images', ('foo.png', open('foo.png', 'rb'), 'image/png')), | |
| 371 ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))] | |
| 372 >>> r = requests.post(url, files=multiple_files) | |
| 373 >>> r.text | |
| 374 { | |
| 375 ... | |
| 376 'files': {'images': 'data:image/png;base64,iVBORw ....'} | |
| 377 'Content-Type': 'multipart/form-data; boundary=3131623adb2043caaeb5538cc7a
a0b3a', | |
| 378 ... | |
| 379 } | |
| 380 | |
| 381 .. warning:: It is strongly recommended that you open files in `binary mode`_. | |
| 382 This is because Requests may attempt to provide the | |
| 383 ``Content-Length`` header for you, and if it does this value will | |
| 384 be set to the number of *bytes* in the file. Errors may occur if | |
| 385 you open the file in *text mode*. | |
| 386 | |
| 387 .. _binary mode: https://docs.python.org/2/tutorial/inputoutput.html#reading-and
-writing-files | |
| 388 | |
| 389 | |
| 390 .. _event-hooks: | |
| 391 | |
| 392 Event Hooks | |
| 393 ----------- | |
| 394 | |
| 395 Requests has a hook system that you can use to manipulate portions of | |
| 396 the request process, or signal event handling. | |
| 397 | |
| 398 Available hooks: | |
| 399 | |
| 400 ``response``: | |
| 401 The response generated from a Request. | |
| 402 | |
| 403 | |
| 404 You can assign a hook function on a per-request basis by passing a | |
| 405 ``{hook_name: callback_function}`` dictionary to the ``hooks`` request | |
| 406 parameter:: | |
| 407 | |
| 408 hooks=dict(response=print_url) | |
| 409 | |
| 410 That ``callback_function`` will receive a chunk of data as its first | |
| 411 argument. | |
| 412 | |
| 413 :: | |
| 414 | |
| 415 def print_url(r, *args, **kwargs): | |
| 416 print(r.url) | |
| 417 | |
| 418 If an error occurs while executing your callback, a warning is given. | |
| 419 | |
| 420 If the callback function returns a value, it is assumed that it is to | |
| 421 replace the data that was passed in. If the function doesn't return | |
| 422 anything, nothing else is effected. | |
| 423 | |
| 424 Let's print some request method arguments at runtime:: | |
| 425 | |
| 426 >>> requests.get('http://httpbin.org', hooks=dict(response=print_url)) | |
| 427 http://httpbin.org | |
| 428 <Response [200]> | |
| 429 | |
| 430 .. _custom-auth: | |
| 431 | |
| 432 Custom Authentication | |
| 433 --------------------- | |
| 434 | |
| 435 Requests allows you to use specify your own authentication mechanism. | |
| 436 | |
| 437 Any callable which is passed as the ``auth`` argument to a request method will | |
| 438 have the opportunity to modify the request before it is dispatched. | |
| 439 | |
| 440 Authentication implementations are subclasses of ``requests.auth.AuthBase``, | |
| 441 and are easy to define. Requests provides two common authentication scheme | |
| 442 implementations in ``requests.auth``: ``HTTPBasicAuth`` and ``HTTPDigestAuth``. | |
| 443 | |
| 444 Let's pretend that we have a web service that will only respond if the | |
| 445 ``X-Pizza`` header is set to a password value. Unlikely, but just go with it. | |
| 446 | |
| 447 :: | |
| 448 | |
| 449 from requests.auth import AuthBase | |
| 450 | |
| 451 class PizzaAuth(AuthBase): | |
| 452 """Attaches HTTP Pizza Authentication to the given Request object.""" | |
| 453 def __init__(self, username): | |
| 454 # setup any auth-related data here | |
| 455 self.username = username | |
| 456 | |
| 457 def __call__(self, r): | |
| 458 # modify and return the request | |
| 459 r.headers['X-Pizza'] = self.username | |
| 460 return r | |
| 461 | |
| 462 Then, we can make a request using our Pizza Auth:: | |
| 463 | |
| 464 >>> requests.get('http://pizzabin.org/admin', auth=PizzaAuth('kenneth')) | |
| 465 <Response [200]> | |
| 466 | |
| 467 .. _streaming-requests: | |
| 468 | |
| 469 Streaming Requests | |
| 470 ------------------ | |
| 471 | |
| 472 With :class:`requests.Response.iter_lines()` you can easily | |
| 473 iterate over streaming APIs such as the `Twitter Streaming | |
| 474 API <https://dev.twitter.com/streaming/overview>`_. Simply | |
| 475 set ``stream`` to ``True`` and iterate over the response with | |
| 476 :class:`~requests.Response.iter_lines()`:: | |
| 477 | |
| 478 import json | |
| 479 import requests | |
| 480 | |
| 481 r = requests.get('http://httpbin.org/stream/20', stream=True) | |
| 482 | |
| 483 for line in r.iter_lines(): | |
| 484 | |
| 485 # filter out keep-alive new lines | |
| 486 if line: | |
| 487 print(json.loads(line)) | |
| 488 | |
| 489 .. warning:: | |
| 490 | |
| 491 :class:`~requests.Response.iter_lines()` is not reentrant safe. | |
| 492 Calling this method multiple times causes some of the received data | |
| 493 being lost. In case you need to call it from multiple places, use | |
| 494 the resulting iterator object instead:: | |
| 495 | |
| 496 lines = r.iter_lines() | |
| 497 # Save the first line for later or just skip it | |
| 498 | |
| 499 first_line = next(lines) | |
| 500 | |
| 501 for line in lines: | |
| 502 print(line) | |
| 503 | |
| 504 .. _proxies: | |
| 505 | |
| 506 Proxies | |
| 507 ------- | |
| 508 | |
| 509 If you need to use a proxy, you can configure individual requests with the | |
| 510 ``proxies`` argument to any request method:: | |
| 511 | |
| 512 import requests | |
| 513 | |
| 514 proxies = { | |
| 515 'http': 'http://10.10.1.10:3128', | |
| 516 'https': 'http://10.10.1.10:1080', | |
| 517 } | |
| 518 | |
| 519 requests.get('http://example.org', proxies=proxies) | |
| 520 | |
| 521 You can also configure proxies by setting the environment variables | |
| 522 ``HTTP_PROXY`` and ``HTTPS_PROXY``. | |
| 523 | |
| 524 :: | |
| 525 | |
| 526 $ export HTTP_PROXY="http://10.10.1.10:3128" | |
| 527 $ export HTTPS_PROXY="http://10.10.1.10:1080" | |
| 528 | |
| 529 $ python | |
| 530 >>> import requests | |
| 531 >>> requests.get('http://example.org') | |
| 532 | |
| 533 To use HTTP Basic Auth with your proxy, use the `http://user:password@host/` syn
tax:: | |
| 534 | |
| 535 proxies = {'http': 'http://user:pass@10.10.1.10:3128/'} | |
| 536 | |
| 537 To give a proxy for a specific scheme and host, use the | |
| 538 `scheme://hostname` form for the key. This will match for | |
| 539 any request to the given scheme and exact hostname. | |
| 540 | |
| 541 :: | |
| 542 | |
| 543 proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'} | |
| 544 | |
| 545 Note that proxy URLs must include the scheme. | |
| 546 | |
| 547 .. _compliance: | |
| 548 | |
| 549 Compliance | |
| 550 ---------- | |
| 551 | |
| 552 Requests is intended to be compliant with all relevant specifications and | |
| 553 RFCs where that compliance will not cause difficulties for users. This | |
| 554 attention to the specification can lead to some behaviour that may seem | |
| 555 unusual to those not familiar with the relevant specification. | |
| 556 | |
| 557 Encodings | |
| 558 ^^^^^^^^^ | |
| 559 | |
| 560 When you receive a response, Requests makes a guess at the encoding to | |
| 561 use for decoding the response when you access the :attr:`Response.text | |
| 562 <requests.Response.text>` attribute. Requests will first check for an | |
| 563 encoding in the HTTP header, and if none is present, will use `chardet | |
| 564 <http://pypi.python.org/pypi/chardet>`_ to attempt to guess the encoding. | |
| 565 | |
| 566 The only time Requests will not do this is if no explicit charset | |
| 567 is present in the HTTP headers **and** the ``Content-Type`` | |
| 568 header contains ``text``. In this situation, `RFC 2616 | |
| 569 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1>`_ specifies | |
| 570 that the default charset must be ``ISO-8859-1``. Requests follows the | |
| 571 specification in this case. If you require a different encoding, you can | |
| 572 manually set the :attr:`Response.encoding <requests.Response.encoding>` | |
| 573 property, or use the raw :attr:`Response.content <requests.Response.content>`. | |
| 574 | |
| 575 .. _http-verbs: | |
| 576 | |
| 577 HTTP Verbs | |
| 578 ---------- | |
| 579 | |
| 580 Requests provides access to almost the full range of HTTP verbs: GET, OPTIONS, | |
| 581 HEAD, POST, PUT, PATCH and DELETE. The following provides detailed examples of | |
| 582 using these various verbs in Requests, using the GitHub API. | |
| 583 | |
| 584 We will begin with the verb most commonly used: GET. HTTP GET is an idempotent | |
| 585 method that returns a resource from a given URL. As a result, it is the verb | |
| 586 you ought to use when attempting to retrieve data from a web location. An | |
| 587 example usage would be attempting to get information about a specific commit | |
| 588 from GitHub. Suppose we wanted commit ``a050faf`` on Requests. We would get it | |
| 589 like so:: | |
| 590 | |
| 591 >>> import requests | |
| 592 >>> r = requests.get('https://api.github.com/repos/kennethreitz/requests/git
/commits/a050faf084662f3a352dd1a941f2c7c9f886d4ad') | |
| 593 | |
| 594 We should confirm that GitHub responded correctly. If it has, we want to work | |
| 595 out what type of content it is. Do this like so:: | |
| 596 | |
| 597 >>> if r.status_code == requests.codes.ok: | |
| 598 ... print(r.headers['content-type']) | |
| 599 ... | |
| 600 application/json; charset=utf-8 | |
| 601 | |
| 602 So, GitHub returns JSON. That's great, we can use the :meth:`r.json | |
| 603 <requests.Response.json>` method to parse it into Python objects. | |
| 604 | |
| 605 :: | |
| 606 | |
| 607 >>> commit_data = r.json() | |
| 608 | |
| 609 >>> print(commit_data.keys()) | |
| 610 [u'committer', u'author', u'url', u'tree', u'sha', u'parents', u'message'] | |
| 611 | |
| 612 >>> print(commit_data[u'committer']) | |
| 613 {u'date': u'2012-05-10T11:10:50-07:00', u'email': u'me@kennethreitz.com', u'
name': u'Kenneth Reitz'} | |
| 614 | |
| 615 >>> print(commit_data[u'message']) | |
| 616 makin' history | |
| 617 | |
| 618 So far, so simple. Well, let's investigate the GitHub API a little bit. Now, | |
| 619 we could look at the documentation, but we might have a little more fun if we | |
| 620 use Requests instead. We can take advantage of the Requests OPTIONS verb to | |
| 621 see what kinds of HTTP methods are supported on the url we just used. | |
| 622 | |
| 623 :: | |
| 624 | |
| 625 >>> verbs = requests.options(r.url) | |
| 626 >>> verbs.status_code | |
| 627 500 | |
| 628 | |
| 629 Uh, what? That's unhelpful! Turns out GitHub, like many API providers, don't | |
| 630 actually implement the OPTIONS method. This is an annoying oversight, but it's | |
| 631 OK, we can just use the boring documentation. If GitHub had correctly | |
| 632 implemented OPTIONS, however, they should return the allowed methods in the | |
| 633 headers, e.g. | |
| 634 | |
| 635 :: | |
| 636 | |
| 637 >>> verbs = requests.options('http://a-good-website.com/api/cats') | |
| 638 >>> print(verbs.headers['allow']) | |
| 639 GET,HEAD,POST,OPTIONS | |
| 640 | |
| 641 Turning to the documentation, we see that the only other method allowed for | |
| 642 commits is POST, which creates a new commit. As we're using the Requests repo, | |
| 643 we should probably avoid making ham-handed POSTS to it. Instead, let's play | |
| 644 with the Issues feature of GitHub. | |
| 645 | |
| 646 This documentation was added in response to Issue #482. Given that this issue | |
| 647 already exists, we will use it as an example. Let's start by getting it. | |
| 648 | |
| 649 :: | |
| 650 | |
| 651 >>> r = requests.get('https://api.github.com/repos/kennethreitz/requests/iss
ues/482') | |
| 652 >>> r.status_code | |
| 653 200 | |
| 654 | |
| 655 >>> issue = json.loads(r.text) | |
| 656 | |
| 657 >>> print(issue[u'title']) | |
| 658 Feature any http verb in docs | |
| 659 | |
| 660 >>> print(issue[u'comments']) | |
| 661 3 | |
| 662 | |
| 663 Cool, we have three comments. Let's take a look at the last of them. | |
| 664 | |
| 665 :: | |
| 666 | |
| 667 >>> r = requests.get(r.url + u'/comments') | |
| 668 >>> r.status_code | |
| 669 200 | |
| 670 | |
| 671 >>> comments = r.json() | |
| 672 | |
| 673 >>> print(comments[0].keys()) | |
| 674 [u'body', u'url', u'created_at', u'updated_at', u'user', u'id'] | |
| 675 | |
| 676 >>> print(comments[2][u'body']) | |
| 677 Probably in the "advanced" section | |
| 678 | |
| 679 Well, that seems like a silly place. Let's post a comment telling the poster | |
| 680 that he's silly. Who is the poster, anyway? | |
| 681 | |
| 682 :: | |
| 683 | |
| 684 >>> print(comments[2][u'user'][u'login']) | |
| 685 kennethreitz | |
| 686 | |
| 687 OK, so let's tell this Kenneth guy that we think this example should go in the | |
| 688 quickstart guide instead. According to the GitHub API doc, the way to do this | |
| 689 is to POST to the thread. Let's do it. | |
| 690 | |
| 691 :: | |
| 692 | |
| 693 >>> body = json.dumps({u"body": u"Sounds great! I'll get right on it!"}) | |
| 694 >>> url = u"https://api.github.com/repos/kennethreitz/requests/issues/482/co
mments" | |
| 695 | |
| 696 >>> r = requests.post(url=url, data=body) | |
| 697 >>> r.status_code | |
| 698 404 | |
| 699 | |
| 700 Huh, that's weird. We probably need to authenticate. That'll be a pain, right? | |
| 701 Wrong. Requests makes it easy to use many forms of authentication, including | |
| 702 the very common Basic Auth. | |
| 703 | |
| 704 :: | |
| 705 | |
| 706 >>> from requests.auth import HTTPBasicAuth | |
| 707 >>> auth = HTTPBasicAuth('fake@example.com', 'not_a_real_password') | |
| 708 | |
| 709 >>> r = requests.post(url=url, data=body, auth=auth) | |
| 710 >>> r.status_code | |
| 711 201 | |
| 712 | |
| 713 >>> content = r.json() | |
| 714 >>> print(content[u'body']) | |
| 715 Sounds great! I'll get right on it. | |
| 716 | |
| 717 Brilliant. Oh, wait, no! I meant to add that it would take me a while, because | |
| 718 I had to go feed my cat. If only I could edit this comment! Happily, GitHub | |
| 719 allows us to use another HTTP verb, PATCH, to edit this comment. Let's do | |
| 720 that. | |
| 721 | |
| 722 :: | |
| 723 | |
| 724 >>> print(content[u"id"]) | |
| 725 5804413 | |
| 726 | |
| 727 >>> body = json.dumps({u"body": u"Sounds great! I'll get right on it once I
feed my cat."}) | |
| 728 >>> url = u"https://api.github.com/repos/kennethreitz/requests/issues/commen
ts/5804413" | |
| 729 | |
| 730 >>> r = requests.patch(url=url, data=body, auth=auth) | |
| 731 >>> r.status_code | |
| 732 200 | |
| 733 | |
| 734 Excellent. Now, just to torture this Kenneth guy, I've decided to let him | |
| 735 sweat and not tell him that I'm working on this. That means I want to delete | |
| 736 this comment. GitHub lets us delete comments using the incredibly aptly named | |
| 737 DELETE method. Let's get rid of it. | |
| 738 | |
| 739 :: | |
| 740 | |
| 741 >>> r = requests.delete(url=url, auth=auth) | |
| 742 >>> r.status_code | |
| 743 204 | |
| 744 >>> r.headers['status'] | |
| 745 '204 No Content' | |
| 746 | |
| 747 Excellent. All gone. The last thing I want to know is how much of my ratelimit | |
| 748 I've used. Let's find out. GitHub sends that information in the headers, so | |
| 749 rather than download the whole page I'll send a HEAD request to get the | |
| 750 headers. | |
| 751 | |
| 752 :: | |
| 753 | |
| 754 >>> r = requests.head(url=url, auth=auth) | |
| 755 >>> print(r.headers) | |
| 756 ... | |
| 757 'x-ratelimit-remaining': '4995' | |
| 758 'x-ratelimit-limit': '5000' | |
| 759 ... | |
| 760 | |
| 761 Excellent. Time to write a Python program that abuses the GitHub API in all | |
| 762 kinds of exciting ways, 4995 more times. | |
| 763 | |
| 764 .. _link-headers: | |
| 765 | |
| 766 Link Headers | |
| 767 ------------ | |
| 768 | |
| 769 Many HTTP APIs feature Link headers. They make APIs more self describing and | |
| 770 discoverable. | |
| 771 | |
| 772 GitHub uses these for `pagination <http://developer.github.com/v3/#pagination>`_ | |
| 773 in their API, for example:: | |
| 774 | |
| 775 >>> url = 'https://api.github.com/users/kennethreitz/repos?page=1&per_page=1
0' | |
| 776 >>> r = requests.head(url=url) | |
| 777 >>> r.headers['link'] | |
| 778 '<https://api.github.com/users/kennethreitz/repos?page=2&per_page=10>; rel="
next", <https://api.github.com/users/kennethreitz/repos?page=6&per_page=10>; rel
="last"' | |
| 779 | |
| 780 Requests will automatically parse these link headers and make them easily consum
able:: | |
| 781 | |
| 782 >>> r.links["next"] | |
| 783 {'url': 'https://api.github.com/users/kennethreitz/repos?page=2&per_page=10'
, 'rel': 'next'} | |
| 784 | |
| 785 >>> r.links["last"] | |
| 786 {'url': 'https://api.github.com/users/kennethreitz/repos?page=7&per_page=10'
, 'rel': 'last'} | |
| 787 | |
| 788 .. _transport-adapters: | |
| 789 | |
| 790 Transport Adapters | |
| 791 ------------------ | |
| 792 | |
| 793 As of v1.0.0, Requests has moved to a modular internal design. Part of the | |
| 794 reason this was done was to implement Transport Adapters, originally | |
| 795 `described here`_. Transport Adapters provide a mechanism to define interaction | |
| 796 methods for an HTTP service. In particular, they allow you to apply per-service | |
| 797 configuration. | |
| 798 | |
| 799 Requests ships with a single Transport Adapter, the :class:`HTTPAdapter | |
| 800 <requests.adapters.HTTPAdapter>`. This adapter provides the default Requests | |
| 801 interaction with HTTP and HTTPS using the powerful `urllib3`_ library. Whenever | |
| 802 a Requests :class:`Session <requests.Session>` is initialized, one of these is | |
| 803 attached to the :class:`Session <requests.Session>` object for HTTP, and one | |
| 804 for HTTPS. | |
| 805 | |
| 806 Requests enables users to create and use their own Transport Adapters that | |
| 807 provide specific functionality. Once created, a Transport Adapter can be | |
| 808 mounted to a Session object, along with an indication of which web services | |
| 809 it should apply to. | |
| 810 | |
| 811 :: | |
| 812 | |
| 813 >>> s = requests.Session() | |
| 814 >>> s.mount('http://www.github.com', MyAdapter()) | |
| 815 | |
| 816 The mount call registers a specific instance of a Transport Adapter to a | |
| 817 prefix. Once mounted, any HTTP request made using that session whose URL starts | |
| 818 with the given prefix will use the given Transport Adapter. | |
| 819 | |
| 820 Many of the details of implementing a Transport Adapter are beyond the scope of | |
| 821 this documentation, but take a look at the next example for a simple SSL use- | |
| 822 case. For more than that, you might look at subclassing | |
| 823 ``requests.adapters.BaseAdapter``. | |
| 824 | |
| 825 Example: Specific SSL Version | |
| 826 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 827 | |
| 828 The Requests team has made a specific choice to use whatever SSL version is | |
| 829 default in the underlying library (`urllib3`_). Normally this is fine, but from | |
| 830 time to time, you might find yourself needing to connect to a service-endpoint | |
| 831 that uses a version that isn't compatible with the default. | |
| 832 | |
| 833 You can use Transport Adapters for this by taking most of the existing | |
| 834 implementation of HTTPAdapter, and adding a parameter *ssl_version* that gets | |
| 835 passed-through to `urllib3`. We'll make a TA that instructs the library to use | |
| 836 SSLv3: | |
| 837 | |
| 838 :: | |
| 839 | |
| 840 import ssl | |
| 841 | |
| 842 from requests.adapters import HTTPAdapter | |
| 843 from requests.packages.urllib3.poolmanager import PoolManager | |
| 844 | |
| 845 | |
| 846 class Ssl3HttpAdapter(HTTPAdapter): | |
| 847 """"Transport adapter" that allows us to use SSLv3.""" | |
| 848 | |
| 849 def init_poolmanager(self, connections, maxsize, block=False): | |
| 850 self.poolmanager = PoolManager( | |
| 851 num_pools=connections, maxsize=maxsize, | |
| 852 block=block, ssl_version=ssl.PROTOCOL_SSLv3) | |
| 853 | |
| 854 .. _`described here`: http://www.kennethreitz.org/essays/the-future-of-python-ht
tp | |
| 855 .. _`urllib3`: https://github.com/shazow/urllib3 | |
| 856 | |
| 857 .. _blocking-or-nonblocking: | |
| 858 | |
| 859 Blocking Or Non-Blocking? | |
| 860 ------------------------- | |
| 861 | |
| 862 With the default Transport Adapter in place, Requests does not provide any kind | |
| 863 of non-blocking IO. The :attr:`Response.content <requests.Response.content>` | |
| 864 property will block until the entire response has been downloaded. If | |
| 865 you require more granularity, the streaming features of the library (see | |
| 866 :ref:`streaming-requests`) allow you to retrieve smaller quantities of the | |
| 867 response at a time. However, these calls will still block. | |
| 868 | |
| 869 If you are concerned about the use of blocking IO, there are lots of projects | |
| 870 out there that combine Requests with one of Python's asynchronicity frameworks. | |
| 871 Two excellent examples are `grequests`_ and `requests-futures`_. | |
| 872 | |
| 873 .. _`grequests`: https://github.com/kennethreitz/grequests | |
| 874 .. _`requests-futures`: https://github.com/ross/requests-futures | |
| 875 | |
| 876 .. _timeouts: | |
| 877 | |
| 878 Timeouts | |
| 879 -------- | |
| 880 | |
| 881 Most requests to external servers should have a timeout attached, in case the | |
| 882 server is not responding in a timely manner. Without a timeout, your code may | |
| 883 hang for minutes or more. | |
| 884 | |
| 885 The **connect** timeout is the number of seconds Requests will wait for your | |
| 886 client to establish a connection to a remote machine (corresponding to the | |
| 887 `connect()`_) call on the socket. It's a good practice to set connect timeouts | |
| 888 to slightly larger than a multiple of 3, which is the default `TCP packet | |
| 889 retransmission window <http://www.hjp.at/doc/rfc/rfc2988.txt>`_. | |
| 890 | |
| 891 Once your client has connected to the server and sent the HTTP request, the | |
| 892 **read** timeout is the number of seconds the client will wait for the server | |
| 893 to send a response. (Specifically, it's the number of seconds that the client | |
| 894 will wait *between* bytes sent from the server. In 99.9% of cases, this is the | |
| 895 time before the server sends the first byte). | |
| 896 | |
| 897 If you specify a single value for the timeout, like this:: | |
| 898 | |
| 899 r = requests.get('https://github.com', timeout=5) | |
| 900 | |
| 901 The timeout value will be applied to both the ``connect`` and the ``read`` | |
| 902 timeouts. Specify a tuple if you would like to set the values separately:: | |
| 903 | |
| 904 r = requests.get('https://github.com', timeout=(3.05, 27)) | |
| 905 | |
| 906 If the remote server is very slow, you can tell Requests to wait forever for | |
| 907 a response, by passing None as a timeout value and then retrieving a cup of | |
| 908 coffee. | |
| 909 | |
| 910 .. code-block:: python | |
| 911 | |
| 912 r = requests.get('https://github.com', timeout=None) | |
| 913 | |
| 914 .. _`connect()`: http://linux.die.net/man/2/connect | |
| OLD | NEW |