Scripts to download files from google storage based on sha1 sums

third_party/gsutil/boto/docs/source/sqs_tut.rst

Index: third_party/gsutil/boto/docs/source/sqs_tut.rst
diff --git a/third_party/gsutil/boto/docs/source/sqs_tut.rst b/third_party/gsutil/boto/docs/source/sqs_tut.rst
new file mode 100644
index 0000000000000000000000000000000000000000..9445de2698256689a10c54a2ea6e54ebe83291e1
--- /dev/null
+++ b/third_party/gsutil/boto/docs/source/sqs_tut.rst
@@ -0,0 +1,246 @@
+.. _sqs_tut:
+
+=======================================
+An Introduction to boto's SQS interface
+=======================================
+
+This tutorial focuses on the boto interface to the Simple Queue Service
+from Amazon Web Services.  This tutorial assumes that you have boto already
+downloaded and installed.
+
+Creating a Connection
+---------------------
+The first step in accessing SQS is to create a connection to the service.
+The recommended method of doing this is as follows::
+
+    >>> import boto.sqs
+    >>> conn = boto.sqs.connect_to_region(
+    ...     "us-east-1",
+    ...     aws_access_key_id='<aws access key'>,
+    ...     aws_secret_access_key='<aws secret key>')
+
+At this point the variable conn will point to an SQSConnection object in the
+US-EAST-1 region. Bear in mind that just as any other AWS service, SQS is
+region-specific. In this example, the AWS access key and AWS secret key are
+passed in to the method explicitely. Alternatively, you can set the environment
+variables:
+
+* ``AWS_ACCESS_KEY_ID`` - Your AWS Access Key ID
+* ``AWS_SECRET_ACCESS_KEY`` - Your AWS Secret Access Key
+
+and then simply call::
+
+    >>> import boto.sqs
+    >>> conn = boto.sqs.connect_to_region("us-east-1")
+
+In either case, conn will point to an SQSConnection object which we will
+use throughout the remainder of this tutorial.
+
+Creating a Queue
+----------------
+Once you have a connection established with SQS, you will probably want to
+create a queue.  In its simplest form, that can be accomplished as follows::
+
+    >>> q = conn.create_queue('myqueue')
+
+The create_queue method will create (and return) the requested queue if it does not
+exist or will return the existing queue if it does.  There is an
+optional parameter to create_queue called visibility_timeout.  This basically
+controls how long a message will remain invisible to other queue readers
+once it has been read (see SQS documentation for more detailed explanation).
+If this is not explicitly specified the queue will be created with whatever
+default value SQS provides (currently 30 seconds).  If you would like to
+specify another value, you could do so like this::
+
+    >>> q = conn.create_queue('myqueue', 120)
+
+This would establish a default visibility timeout for this queue of 120
+seconds.  As you will see later on, this default value for the queue can
+also be overridden each time a message is read from the queue.  If you want
+to check what the default visibility timeout is for a queue::
+
+    >>> q.get_timeout()
+    30
+
+Listing all Queues
+------------------
+
+To retrieve a list of the queues for your account in the current region::
+
+    >>> conn.get_all_queues()
+    [
+        Queue(https://queue.amazonaws.com/411358162645/myqueue),
+        Queue(https://queue.amazonaws.com/411358162645/another_queue),
+        Queue(https://queue.amazonaws.com/411358162645/another_queue2)
+    ]
+
+This will leave you with a list of all of your :py:class:`boto.sqs.queue.Queue`
+instances. Alternatively, if you wanted to only list the queues that started
+with ``'another'``::
+
+    >>> conn.get_all_queues(prefix='another')
+    [
+        Queue(https://queue.amazonaws.com/411358162645/another_queue),
+        Queue(https://queue.amazonaws.com/411358162645/another_queue2)
+    ]
+
+Getting a Queue (by name)
+-------------------------
+If you wish to explicitly retrieve an existing queue and the name of the queue is known,
+you can retrieve the queue as follows::
+
+    >>> my_queue = conn.get_queue('myqueue')
+    Queue(https://queue.amazonaws.com/411358162645/myqueue)
+
+This leaves you with a single :py:class:`boto.sqs.queue.Queue`, which abstracts
+the SQS Queue named 'myqueue'.
+
+Writing Messages
+----------------
+Once you have a queue setup, presumably you will want to write some messages
+to it.  SQS doesn't care what kind of information you store in your messages
+or what format you use to store it.  As long as the amount of data per
+message is less than or equal to 256Kb, SQS won't complain.
+
+So, first we need to create a Message object::
+
+>>> from boto.sqs.message import Message
+>>> m = Message()
+>>> m.set_body('This is my first message.')
+>>> status = q.write(m)
+
+The write method returns a True if everything went well.  If the write
+didn't succeed it will either return a False (meaning SQS simply chose
+not to write the message for some reason) or an exception if there was
+some sort of problem with the request.
+
+Writing Messages (Custom Format)
+--------------------------------
+The technique above will work only if you use boto's default Message payload format;
+however, you may have a lot of specific requirements around the format of
+the message data.  For example, you may want to store one big string or you might
+want to store something that looks more like RFC822 messages or you might want
+to store a binary payload such as pickled Python objects.
+
+The way boto deals with this issue is to define a simple Message object that
+treats the message data as one big string which you can set and get.  If that
+Message object meets your needs, you're good to go.  However, if you need to
+incorporate different behavior in your message or handle different types of
+data you can create your own Message class.  You just need to register that
+class with the boto queue object so that it knows that, when you read a message from the
+queue, it should create one of your message objects rather than the
+default boto Message object.  To register your message class, you would::
+
+>>> import MyMessage
+>>> q.set_message_class(MyMessage)
+>>> m = MyMessage()
+>>> m.set_body('This is my first message.')
+>>> status = q.write(m)
+
+where MyMessage is the class definition for your message class.  Your
+message class should subclass the boto Message because there is a small
+bit of Python magic happening in the __setattr__ method of the boto Message
+class.
+
+Reading Messages
+----------------
+
+So, now we have a message in our queue.  How would we go about reading it?
+Here's one way:
+
+>>> rs = q.get_messages()
+>>> len(rs)
+1
+>>> m = rs[0]
+>>> m.get_body()
+u'This is my first message'
+
+The get_messages method also returns a ResultSet object as described
+above.  In addition to the special attributes that we already talked
+about the ResultSet object also contains any results returned by the
+request.  To get at the results you can treat the ResultSet as a
+sequence object (e.g. a list).  We can check the length (how many results)
+and access particular items within the list using the slice notation
+familiar to Python programmers.
+
+At this point, we have read the message from the queue and SQS will make
+sure that this message remains invisible to other readers of the queue
+until the visibility timeout period for the queue expires.  If you delete
+the message before the timeout period expires then no one else will ever see
+the message again.  However, if you don't delete it (maybe because your reader crashed
+or failed in some way, for example) it will magically reappear in my queue
+for someone else to read.  If you aren't happy with the default visibility
+timeout defined for the queue, you can override it when you read a message:
+
+>>> q.get_messages(visibility_timeout=60)
+
+This means that regardless of what the default visibility timeout is for
+the queue, this message will remain invisible to other readers for 60
+seconds.
+
+The get_messages method can also return more than a single message.  By
+passing a num_messages parameter (defaults to 1) you can control the maximum
+number of messages that will be returned by the method.  To show this
+feature off, first let's load up a few more messages.
+
+>>> for i in range(1, 11):
+...   m = Message()
+...   m.set_body('This is message %d' % i)
+...   q.write(m)
+...
+>>> rs = q.get_messages(10)
+>>> len(rs)
+10
+
+Don't be alarmed if the length of the result set returned by the get_messages
+call is less than 10.  Sometimes it takes some time for new messages to become
+visible in the queue.  Give it a minute or two and they will all show up.
+
+If you want a slightly simpler way to read messages from a queue, you
+can use the read method.  It will either return the message read or
+it will return None if no messages were available.  You can also pass
+a visibility_timeout parameter to read, if you desire:
+
+>>> m = q.read(60)
+>>> m.get_body()
+u'This is my first message'
+
+Deleting Messages and Queues
+----------------------------
+As stated above, messages are never deleted by the queue unless explicitly told to do so.
+To remove a message from a queue:
+
+>>> q.delete_message(m)
+[]
+
+If I want to delete the entire queue, I would use:
+
+>>> conn.delete_queue(q)
+
+However, and this is a good safe guard, this won't succeed unless the queue is empty.
+
+Additional Information
+----------------------
+The above tutorial covers the basic operations of creating queues, writing messages,
+reading messages, deleting messages, and deleting queues.  There are a
+few utility methods in boto that might be useful as well.  For example,
+to count the number of messages in a queue:
+
+>>> q.count()
+10
+
+This can be handy but is command as well as the other two utility methods
+I'll describe in a minute are inefficient and should be used with caution
+on queues with lots of messages (e.g. many hundreds or more).  Similarly,
+you can clear (delete) all messages in a queue with:
+
+>>> q.clear()
+
+Be REAL careful with that one!  Finally, if you want to dump all of the
+messages in a queue to a local file:
+
+>>> q.dump('messages.txt', sep='\n------------------\n')
+
+This will read all of the messages in the queue and write the bodies of
+each of the messages to the file messages.txt.  The option sep argument
+is a separator that will be printed between each message body in the file.