| Index: third_party/gsutil/third_party/pyasn1/doc/codecs.html
|
| diff --git a/third_party/gsutil/third_party/pyasn1/doc/codecs.html b/third_party/gsutil/third_party/pyasn1/doc/codecs.html
|
| new file mode 100644
|
| index 0000000000000000000000000000000000000000..9c2c36ed6f053a16bbdf271e217714d9ba4a1962
|
| --- /dev/null
|
| +++ b/third_party/gsutil/third_party/pyasn1/doc/codecs.html
|
| @@ -0,0 +1,503 @@
|
| +<html>
|
| +<title>
|
| +PyASN1 codecs
|
| +</title>
|
| +<head>
|
| +</head>
|
| +<body>
|
| +<center>
|
| +<table width=60%>
|
| +<tr>
|
| +<td>
|
| +<h3>
|
| +2. PyASN1 Codecs
|
| +</h3>
|
| +
|
| +<p>
|
| +In ASN.1 context,
|
| +<a href=http://en.wikipedia.org/wiki/Codec>codec</a>
|
| +is a program that transforms between concrete data structures and a stream
|
| +of octets, suitable for transmission over the wire. This serialized form of
|
| +data is sometimes called <i>substrate</i> or <i>essence</i>.
|
| +</p>
|
| +
|
| +<p>
|
| +In pyasn1 implementation, substrate takes shape of Python 3 bytes or
|
| +Python 2 string objects.
|
| +</p>
|
| +
|
| +<p>
|
| +One of the properties of a codec is its ability to cope with incomplete
|
| +data and/or substrate what implies codec to be stateful. In other words,
|
| +when decoder runs out of substrate and data item being recovered is still
|
| +incomplete, stateful codec would suspend and complete data item recovery
|
| +whenever the rest of substrate becomes available. Similarly, stateful encoder
|
| +would encode data items in multiple steps waiting for source data to
|
| +arrive. Codec restartability is especially important when application deals
|
| +with large volumes of data and/or runs on low RAM. For an interesting
|
| +discussion on codecs options and design choices, refer to
|
| +<a href=http://directory.apache.org/subprojects/asn1/>Apache ASN.1 project</a>
|
| +.
|
| +</p>
|
| +
|
| +<p>
|
| +As of this writing, codecs implemented in pyasn1 are all stateless, mostly
|
| +to keep the code simple.
|
| +</p>
|
| +
|
| +<p>
|
| +The pyasn1 package currently supports
|
| +<a href=http://en.wikipedia.org/wiki/Basic_encoding_rules>BER</a> codec and
|
| +its variations --
|
| +<a href=http://en.wikipedia.org/wiki/Canonical_encoding_rules>CER</a> and
|
| +<a href=http://en.wikipedia.org/wiki/Distinguished_encoding_rules>DER</a>.
|
| +More ASN.1 codecs are planned for implementation in the future.
|
| +</p>
|
| +
|
| +<a name="2.1"></a>
|
| +<h4>
|
| +2.1 Encoders
|
| +</h4>
|
| +
|
| +<p>
|
| +Encoder is used for transforming pyasn1 value objects into substrate. Only
|
| +pyasn1 value objects could be serialized, attempts to process pyasn1 type
|
| +objects will cause encoder failure.
|
| +</p>
|
| +
|
| +<p>
|
| +The following code will create a pyasn1 Integer object and serialize it with
|
| +BER encoder:
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder
|
| +>>> encoder.encode(univ.Integer(123456))
|
| +b'\x02\x03\x01\xe2@'
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +BER standard also defines a so-called <i>indefinite length</i> encoding form
|
| +which makes large data items processing more memory efficient. It is mostly
|
| +useful when encoder does not have the whole value all at once and the
|
| +length of the value can not be determined at the beginning of encoding.
|
| +</p>
|
| +
|
| +<p>
|
| +<i>Constructed encoding</i> is another feature of BER closely related to the
|
| +indefinite length form. In essence, a large scalar value (such as ASN.1
|
| +character BitString type) could be chopped into smaller chunks by encoder
|
| +and transmitted incrementally to limit memory consumption. Unlike indefinite
|
| +length case, the length of the whole value must be known in advance when
|
| +using constructed, definite length encoding form.
|
| +</p>
|
| +
|
| +<p>
|
| +Since pyasn1 codecs are not restartable, pyasn1 encoder may only encode data
|
| +item all at once. However, even in this case, generating indefinite length
|
| +encoding may help a low-memory receiver, running a restartable decoder,
|
| +to process a large data item.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder
|
| +>>> encoder.encode(
|
| +... univ.OctetString('The quick brown fox jumps over the lazy dog'),
|
| +... defMode=False,
|
| +... maxChunkSize=8
|
| +... )
|
| +b'$\x80\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
|
| +t\x04\x08he lazy \x04\x03dog\x00\x00'
|
| +>>>
|
| +>>> encoder.encode(
|
| +... univ.OctetString('The quick brown fox jumps over the lazy dog'),
|
| +... maxChunkSize=8
|
| +... )
|
| +b'$7\x04\x08The quic\x04\x08k brown \x04\x08fox jump\x04\x08s over \
|
| +t\x04\x08he lazy \x04\x03dog'
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +The <b>defMode</b> encoder parameter disables definite length encoding mode,
|
| +while the optional <b>maxChunkSize</b> parameter specifies desired
|
| +substrate chunk size that influences memory requirements at the decoder's end.
|
| +</p>
|
| +
|
| +<p>
|
| +To use CER or DER encoders one needs to explicitly import and call them - the
|
| +APIs are all compatible.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder as ber_encoder
|
| +>>> from pyasn1.codec.cer import encoder as cer_encoder
|
| +>>> from pyasn1.codec.der import encoder as der_encoder
|
| +>>> ber_encoder.encode(univ.Boolean(True))
|
| +b'\x01\x01\x01'
|
| +>>> cer_encoder.encode(univ.Boolean(True))
|
| +b'\x01\x01\xff'
|
| +>>> der_encoder.encode(univ.Boolean(True))
|
| +b'\x01\x01\xff'
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<a name="2.2"></a>
|
| +<h4>
|
| +2.2 Decoders
|
| +</h4>
|
| +
|
| +<p>
|
| +In the process of decoding, pyasn1 value objects are created and linked to
|
| +each other, based on the information containted in the substrate. Thus,
|
| +the original pyasn1 value object(s) are recovered.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +>>> substrate = encoder.encode(univ.Boolean(True))
|
| +>>> decoder.decode(substrate)
|
| +(Boolean('True(1)'), b'')
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +Commenting on the code snippet above, pyasn1 decoder accepts substrate
|
| +as an argument and returns a tuple of pyasn1 value object (possibly
|
| +a top-level one in case of constructed object) and unprocessed part
|
| +of input substrate.
|
| +</p>
|
| +
|
| +<p>
|
| +All pyasn1 decoders can handle both definite and indefinite length
|
| +encoding modes automatically, explicit switching into one mode
|
| +to another is not required.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +>>> substrate = encoder.encode(
|
| +... univ.OctetString('The quick brown fox jumps over the lazy dog'),
|
| +... defMode=False,
|
| +... maxChunkSize=8
|
| +... )
|
| +>>> decoder.decode(substrate)
|
| +(OctetString(b'The quick brown fox jumps over the lazy dog'), b'')
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +Speaking of BER/CER/DER encoding, in many situations substrate may not contain
|
| +all necessary information needed for complete and accurate ASN.1 values
|
| +recovery. The most obvious cases include implicitly tagged ASN.1 types
|
| +and constrained types.
|
| +</p>
|
| +
|
| +<p>
|
| +As discussed earlier in this handbook, when an ASN.1 type is implicitly
|
| +tagged, previous outermost tag is lost and never appears in substrate.
|
| +If it is the base tag that gets lost, decoder is unable to pick type-specific
|
| +value decoder at its table of built-in types, and therefore recover
|
| +the value part, based only on the information contained in substrate. The
|
| +approach taken by pyasn1 decoder is to use a prototype pyasn1 type object (or
|
| +a set of them) to <i>guide</i> the decoding process by matching [possibly
|
| +incomplete] tags recovered from substrate with those found in prototype pyasn1
|
| +type objects (also called pyasn1 specification object further in this paper).
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.codec.ber import decoder
|
| +>>> decoder.decode(b'\x02\x01\x0c', asn1Spec=univ.Integer())
|
| +Integer(12), b''
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +Decoder would neither modify pyasn1 specification object nor use
|
| +its current values (if it's a pyasn1 value object), but rather use it as
|
| +a hint for choosing proper decoder and as a pattern for creating new objects:
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ, tag
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +>>> i = univ.Integer(12345).subtype(
|
| +... implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
|
| +... )
|
| +>>> substrate = encoder.encode(i)
|
| +>>> substrate
|
| +b'\x9f(\x0209'
|
| +>>> decoder.decode(substrate)
|
| +Traceback (most recent call last):
|
| +...
|
| +pyasn1.error.PyAsn1Error:
|
| + TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
|
| +>>> decoder.decode(substrate, asn1Spec=i)
|
| +(Integer(12345), b'')
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +Notice in the example above, that an attempt to run decoder without passing
|
| +pyasn1 specification object fails because recovered tag does not belong
|
| +to any of the built-in types.
|
| +</p>
|
| +
|
| +<p>
|
| +Another important feature of guided decoder operation is the use of
|
| +values constraints possibly present in pyasn1 specification object.
|
| +To explain this, we will decode a random integer object into generic Integer
|
| +and the constrained one.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ, constraint
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +>>> class DialDigit(univ.Integer):
|
| +... subtypeSpec = constraint.ValueRangeConstraint(0,9)
|
| +>>> substrate = encoder.encode(univ.Integer(13))
|
| +>>> decoder.decode(substrate)
|
| +(Integer(13), b'')
|
| +>>> decoder.decode(substrate, asn1Spec=DialDigit())
|
| +Traceback (most recent call last):
|
| +...
|
| +pyasn1.type.error.ValueConstraintError:
|
| + ValueRangeConstraint(0, 9) failed at: 13
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +Similarily to encoders, to use CER or DER decoders application has to
|
| +explicitly import and call them - all APIs are compatible.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder as ber_encoder
|
| +>>> substrate = ber_encoder.encode(univ.OctetString('http://pyasn1.sf.net'))
|
| +>>>
|
| +>>> from pyasn1.codec.ber import decoder as ber_decoder
|
| +>>> from pyasn1.codec.cer import decoder as cer_decoder
|
| +>>> from pyasn1.codec.der import decoder as der_decoder
|
| +>>>
|
| +>>> ber_decoder.decode(substrate)
|
| +(OctetString(b'http://pyasn1.sf.net'), b'')
|
| +>>> cer_decoder.decode(substrate)
|
| +(OctetString(b'http://pyasn1.sf.net'), b'')
|
| +>>> der_decoder.decode(substrate)
|
| +(OctetString(b'http://pyasn1.sf.net'), b'')
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<a name="2.2.1"></a>
|
| +<h4>
|
| +2.2.1 Decoding untagged types
|
| +</h4>
|
| +
|
| +<p>
|
| +It has already been mentioned, that ASN.1 has two "special case" types:
|
| +CHOICE and ANY. They are different from other types in part of
|
| +tagging - unless these two are additionally tagged, neither of them will
|
| +have their own tag. Therefore these types become invisible in substrate
|
| +and can not be recovered without passing pyasn1 specification object to
|
| +decoder.
|
| +</p>
|
| +
|
| +<p>
|
| +To explain the issue, we will first prepare a Choice object to deal with:
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ, namedtype
|
| +>>> class CodeOrMessage(univ.Choice):
|
| +... componentType = namedtype.NamedTypes(
|
| +... namedtype.NamedType('code', univ.Integer()),
|
| +... namedtype.NamedType('message', univ.OctetString())
|
| +... )
|
| +>>>
|
| +>>> codeOrMessage = CodeOrMessage()
|
| +>>> codeOrMessage.setComponentByName('message', 'my string value')
|
| +>>> print(codeOrMessage.prettyPrint())
|
| +CodeOrMessage:
|
| + message=b'my string value'
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +Let's now encode this Choice object and then decode its substrate
|
| +with and without pyasn1 specification object:
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +>>> substrate = encoder.encode(codeOrMessage)
|
| +>>> substrate
|
| +b'\x04\x0fmy string value'
|
| +>>> encoder.encode(univ.OctetString('my string value'))
|
| +b'\x04\x0fmy string value'
|
| +>>>
|
| +>>> decoder.decode(substrate)
|
| +(OctetString(b'my string value'), b'')
|
| +>>> codeOrMessage, substrate = decoder.decode(substrate, asn1Spec=CodeOrMessage())
|
| +>>> print(codeOrMessage.prettyPrint())
|
| +CodeOrMessage:
|
| + message=b'my string value'
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +First thing to notice in the listing above is that the substrate produced
|
| +for our Choice value object is equivalent to the substrate for an OctetString
|
| +object initialized to the same value. In other words, any information about
|
| +the Choice component is absent in encoding.
|
| +</p>
|
| +
|
| +<p>
|
| +Sure enough, that kind of substrate will decode into an OctetString object,
|
| +unless original Choice type object is passed to decoder to guide the decoding
|
| +process.
|
| +</p>
|
| +
|
| +<p>
|
| +Similarily untagged ANY type behaves differently on decoding phase - when
|
| +decoder bumps into an Any object in pyasn1 specification, it stops decoding
|
| +and puts all the substrate into a new Any value object in form of an octet
|
| +string. Concerned application could then re-run decoder with an additional,
|
| +more exact pyasn1 specification object to recover the contents of Any
|
| +object.
|
| +</p>
|
| +
|
| +<p>
|
| +As it was mentioned elsewhere in this paper, Any type allows for incomplete
|
| +or changing ASN.1 specification to be handled gracefully by decoder and
|
| +applications.
|
| +</p>
|
| +
|
| +<p>
|
| +To illustrate the working of Any type, we'll have to make the stage
|
| +by encoding a pyasn1 object and then putting its substrate into an any
|
| +object.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +>>> innerSubstrate = encoder.encode(univ.Integer(1234))
|
| +>>> innerSubstrate
|
| +b'\x02\x02\x04\xd2'
|
| +>>> any = univ.Any(innerSubstrate)
|
| +>>> any
|
| +Any(b'\x02\x02\x04\xd2')
|
| +>>> substrate = encoder.encode(any)
|
| +>>> substrate
|
| +b'\x02\x02\x04\xd2'
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +As with Choice type encoding, there is no traces of Any type in substrate.
|
| +Obviously, the substrate we are dealing with, will decode into the inner
|
| +[Integer] component, unless pyasn1 specification is given to guide the
|
| +decoder. Continuing previous code:
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +
|
| +>>> decoder.decode(substrate)
|
| +(Integer(1234), b'')
|
| +>>> any, substrate = decoder.decode(substrate, asn1Spec=univ.Any())
|
| +>>> any
|
| +Any(b'\x02\x02\x04\xd2')
|
| +>>> decoder.decode(str(any))
|
| +(Integer(1234), b'')
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +Both CHOICE and ANY types are widely used in practice. Reader is welcome to
|
| +take a look at
|
| +<a href=http://www.cs.auckland.ac.nz/~pgut001/pubs/x509guide.txt>
|
| +ASN.1 specifications of X.509 applications</a> for more information.
|
| +</p>
|
| +
|
| +<a name="2.2.2"></a>
|
| +<h4>
|
| +2.2.2 Ignoring unknown types
|
| +</h4>
|
| +
|
| +<p>
|
| +When dealing with a loosely specified ASN.1 structure, the receiving
|
| +end may not be aware of some types present in the substrate. It may be
|
| +convenient then to turn decoder into a recovery mode. Whilst there, decoder
|
| +will not bail out when hit an unknown tag but rather treat it as an Any
|
| +type.
|
| +</p>
|
| +
|
| +<table bgcolor="lightgray" border=0 width=100%><TR><TD>
|
| +<pre>
|
| +>>> from pyasn1.type import univ, tag
|
| +>>> from pyasn1.codec.ber import encoder, decoder
|
| +>>> taggedInt = univ.Integer(12345).subtype(
|
| +... implicitTag=tag.Tag(tag.tagClassContext, tag.tagFormatSimple, 40)
|
| +... )
|
| +>>> substrate = encoder.encode(taggedInt)
|
| +>>> decoder.decode(substrate)
|
| +Traceback (most recent call last):
|
| +...
|
| +pyasn1.error.PyAsn1Error: TagSet(Tag(tagClass=128, tagFormat=0, tagId=40)) not in asn1Spec
|
| +>>>
|
| +>>> decoder.decode.defaultErrorState = decoder.stDumpRawValue
|
| +>>> decoder.decode(substrate)
|
| +(Any(b'\x9f(\x0209'), '')
|
| +>>>
|
| +</pre>
|
| +</td></tr></table>
|
| +
|
| +<p>
|
| +It's also possible to configure a custom decoder, to handle unknown tags
|
| +found in substrate. This can be done by means of <b>defaultRawDecoder</b>
|
| +attribute holding a reference to type decoder object. Refer to the source
|
| +for API details.
|
| +</p>
|
| +
|
| +<hr>
|
| +
|
| +</td>
|
| +</tr>
|
| +</table>
|
| +</center>
|
| +</body>
|
| +</html>
|
|
|
Chromium Code Reviews
Issue 1377933002:
[catapult] - Copy Telemetry's gsutilz over to third_party. (Closed)
Base URL: https://github.com/catapult-project/catapult.git@master