Added Base64 support in TextEncoder and TextDecoder.

Added Base64 support in TextEncoder and TextDecoder.
Base64 encoding encodes binary strings to text, which is the opposite
of other text encoders, such as UTF-8. To reuse the existing APIs,
use TextDecoder.decode to encode binary strings to Base64 format and
use TextEncoder.encode to decode Base64 to binary string.

BUG=337586

[c.shu, 2014-01-24 16:08:58]

Would you review it? Thanks.

[jsbell, 2014-01-24 22:15:04]

Looks like streaming is going to require a fair bit of work.

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
File LayoutTests/fast/encoding/api/base64.html (right):

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/base64.html:6: description("Verify Base64
encoding/decoding");
Need a streaming encode and streaming decode test.

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/base64.html:9: decoded = null;
No need to initialize vars to null that will be assigned later.

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/base64.html:13: evalAndLog("decoder = new
TextDecoder(" + JSON.stringify('base64') + ")");
Can you add a comment (or debug("...") line) that explains that decode of base64
is converting binary data to base64 string?

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/base64.html:14: if (decoder) {
No need for this if() block.

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/base64.html:22: evalAndLog("encoder = new
TextEncoder(" + JSON.stringify('base64') + ")");
... and can you add a comment about the reverse here, that encode of base64 is
converting base64 string to binary data?

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/base64.html:23: if (encoder) {
No need for this if() block.

https://codereview.chromium.org/143943017/diff/1/LayoutTests/fast/encoding/ap...
LayoutTests/fast/encoding/api/base64.html:25: shouldBeTrue("encoded.length ===
bytes.length");
This and the rest of the test can just use areArraysEqual() from js-test.js,
i.e.:

shouldBeTrue("areArraysEqual(encoded, bytes)");

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
File Source/wtf/text/TextCodecBase64.cpp (right):

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
Source/wtf/text/TextCodecBase64.cpp:31: #include "wtf/text/StringBuffer.h"
Is this include needed?

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
Source/wtf/text/TextCodecBase64.cpp:32: #include "wtf/unicode/CharacterNames.h"
Is this include needed?

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
Source/wtf/text/TextCodecBase64.cpp:44: void
TextCodecBase64::registerEncodingNames(EncodingNameRegistrar registrar)
Does this registration cause base64 to show up in the set of encodings used
elsewhere in Blink, e.g. in the loader? We should have a explicit tests that it
does NOT.

That should cover page loading, XHR (both the request body and response), and
form submissions. Look in LayoutTests/fast/encoding - hopefully there are
examples.

(Correct behavior should be to fall back to the default encoding, which is
probably windows-1252/latin-1 in the tests.)

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
Source/wtf/text/TextCodecBase64.cpp:55: String TextCodecBase64::decode(const
char* bytes, size_t length, bool flush, bool stopOnError, bool& sawError)
Is streaming handled here? It looks like flush is ignored. Do the
wtf/text/Base64.h APIs handle it?

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
Source/wtf/text/TextCodecBase64.cpp:60: CString TextCodecBase64::encode(const
UChar* characters, size_t length, UnencodableHandling)
It doesn't look like the Codec API handles flushing for encode (i.e. string to
binary). For base64 you can imagine this being needed for:

encoder.encode("ZWFzd", {stream:true});
encoder.encode("XJlLg==");

This isn't normally necessary in normal text encoders since a single UChar maps
to a zero or more whole octets, but for base64 a single UChar may map fractions
of an octet.

(Uh.... actually, lone UTF16 surrogates probably have the same issue when
transcoding from JS UTF-16 strings to something else e.g. UTF-8. So this may be
a pre-existing issue where a surrogate pair is split across encode() calls...)

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
File Source/wtf/text/TextCodecBase64.h (right):

https://codereview.chromium.org/143943017/diff/1/Source/wtf/text/TextCodecBas...
Source/wtf/text/TextCodecBase64.h:2: * Copyright (C) 2011 Apple Inc. All rights
reserved.
New file, should use updated copyright/license. 

http://www.chromium.org/blink/coding-style

[jungshik at Google, 2014-01-27 18:24:28]

Is this the right layer/API to add base64 encoder / decoder  at/to ? 
What's the motivation/use case for adding base64 encoder/decoder to
TextEncoder/Decoder?

[c.shu, 2014-01-27 18:30:51]

On 2014/01/27 18:24:28, Jungshik Shin wrote:
> Is this the right layer/API to add base64 encoder / decoder  at/to ? 
> What's the motivation/use case for adding base64 encoder/decoder to
> TextEncoder/Decoder?

Please see some discussion on blink-dev and whatwg as a background of this
patch. Thanks.
https://groups.google.com/a/chromium.org/forum/#!searchin/blink-dev/stringvie...
http://lists.w3.org/Archives/Public/public-whatwg-archive/2013Aug/0161.html

[jungshik at Google, 2014-01-27 18:31:17]

> use TextDecoder.decode to encode binary strings to Base64 format and
> use TextEncoder.encode to decode Base64 to binary string.

The above description is also confusing because 'Decoder encodes' and 'Encoder
decodes'. 

The binary string is a sequence of bytes in the above, right? What's in 'Base64'
format? Is it a sequence of UChars (16-bit) whose 'higher' half is always 0 and
'lower' half is one of Base64 alphabets? Base64 represented that way has a size
multiplier of 8/3 instead of 4/3. It's pretty wasteful. 
    
Does the encoding spec require supporting this?

[jsbell, 2014-01-27 18:39:56]

On 2014/01/27 18:31:17, Jungshik Shin wrote:
> Does the encoding spec require supporting this?

No, and c.shu's patch is definitely an experiment. Do read the links he provided
- you'll see that there's been both substantial pushback on adding base64
support to the Text Encoding API (e.g. from me, since the API verbs are
backwards). But there have also been numerous comments suggesting "oh, it's not
so bad, it'll work", skeptical of the poor usability of the resulting API. (I
had it in an early iteration of the JS polyfill and I think I messed up both the
implementation and the tests the first time around.)

I believe a likely outcome of this effort is a complicated patch and a confusing
API, which is more data that trying to add base64 support to the Text Encoding
API is a bad idea.

[jungshik at Google, 2014-01-27 19:30:11]

On 2014/01/27 18:39:56, jsbell wrote:
> On 2014/01/27 18:31:17, Jungshik Shin wrote:
> > Does the encoding spec require supporting this?
> 
> No, and c.shu's patch is definitely an experiment. Do read the links he
provided
> - you'll see that there's been both substantial pushback on adding base64
> support to the Text Encoding API (e.g. from me, since the API verbs are
> backwards). But there have also been numerous comments suggesting "oh, it's
not
> so bad, it'll work", skeptical of the poor usability of the resulting API. (I
> had it in an early iteration of the JS polyfill and I think I messed up both
the
> implementation and the tests the first time around.)
> 
> I believe a likely outcome of this effort is a complicated patch and a
confusing
> API, which is more data that trying to add base64 support to the Text Encoding
> API is a bad idea.

Glad to hear that we're mostly on the same page. I agree with you on all the
points you made. I strongly object to adding base64 support to the Text Encoding
API.

[c.shu, 2014-01-27 20:30:30]

> Glad to hear that we're mostly on the same page. I agree with you on all the
> points you made. I strongly object to adding base64 support to the Text
Encoding
> API.

Thanks for the feedback. Is there any other approach you suggest to support
base64 encoding?
We have been talking about StringView, atob/btoa and this Text Encoding. It
seems different
people have different opinions. Utilizing Text encoding is most of the people
have agreed upon.

[jungshik at Google, 2014-01-28 22:04:49]

On 2014/01/27 20:30:30, c.shu wrote:
> > Glad to hear that we're mostly on the same page. I agree with you on all the
> > points you made. I strongly object to adding base64 support to the Text
> Encoding
> > API.
> 
> Thanks for the feedback. Is there any other approach you suggest to support
> base64 encoding?
> We have been talking about StringView, atob/btoa and this Text Encoding. It
> seems different
> people have different opinions. Utilizing Text encoding is most of the people
> have agreed upon.

IMHO, base64 and 'character encodings'(dealt with in TextEncoding) are at
different layers/beasts. 'base64' is a way to represent binary (or non-ASCII
byte sequence) in a small set (64) of ASCII-compatible octets for a target/wire
that can't contain/transfer 'binary' or 'non-ASCII bytes'. So, it's a
'transfer-encoding' different from 'character encoding' TextEncoding deals with.
In MIME, CES (character encoding scheme) is specified in Content-Type header
with 'charset' parameter while 'transfer-encoding' is specified in
Content-Transfer-Encoding header (whose value can be base64, quoted-printable,
8bit, 7bit or binary)

To me, StringView seems to be a natural home for base64 encoder/decoder (I
skimmed over the two threads you gave a link to, but I'm not convinced that
TextEncoding API is a better place). [1]  Well, it's your original idea but it's
kinda turned down because StringView is not standardized at all (it's just
mentioned at MDN).

Any update from the EcmaScript standard?  Another good place would be adding the
support to the language. 


[1] I believe that converting base64 to/from JavaScript String is not a good
idea. As you originally proposed, using ArrayBuffer or something similar seems
desirable.

[c.shu, 2014-01-29 15:15:01]

On 2014/01/28 22:04:49, Jungshik Shin wrote:
> On 2014/01/27 20:30:30, c.shu wrote:
> > > Glad to hear that we're mostly on the same page. I agree with you on all
the
> > > points you made. I strongly object to adding base64 support to the Text
> > Encoding
> > > API.
> > 
> > Thanks for the feedback. Is there any other approach you suggest to support
> > base64 encoding?
> > We have been talking about StringView, atob/btoa and this Text Encoding. It
> > seems different
> > people have different opinions. Utilizing Text encoding is most of the
people
> > have agreed upon.
> 
> IMHO, base64 and 'character encodings'(dealt with in TextEncoding) are at
> different layers/beasts. 'base64' is a way to represent binary (or non-ASCII
> byte sequence) in a small set (64) of ASCII-compatible octets for a
target/wire
> that can't contain/transfer 'binary' or 'non-ASCII bytes'. So, it's a
> 'transfer-encoding' different from 'character encoding' TextEncoding deals
with.
> In MIME, CES (character encoding scheme) is specified in Content-Type header
> with 'charset' parameter while 'transfer-encoding' is specified in
> Content-Transfer-Encoding header (whose value can be base64, quoted-printable,
> 8bit, 7bit or binary)
> 
> To me, StringView seems to be a natural home for base64 encoder/decoder (I
> skimmed over the two threads you gave a link to, but I'm not convinced that
> TextEncoding API is a better place). [1]  Well, it's your original idea but
it's
> kinda turned down because StringView is not standardized at all (it's just
> mentioned at MDN).
> 
> Any update from the EcmaScript standard?  Another good place would be adding
the
> support to the language. 
> 
> 
> [1] I believe that converting base64 to/from JavaScript String is not a good
> idea. As you originally proposed, using ArrayBuffer or something similar seems
> desirable.

I agree. Messing up the text encoding with binary encoding, which is targeted
for data
transferring, is probably not a good idea. It also makes the implementation
complicated
and confusing, as I have explored.
So here are the possible approaches:
1. Text Encoder/Decoder
Confusing APIs and complex implementation.
2. Add Base64 APIs in ArrayBuffer
Seems desirable but people would like to keep the interface minimal.
3. Support StringView of ArrayBuffer
All we need is being able to construct a StringView from an existing ArrayBuffer

and take the buffer from a StringView. Base64 encoding itself can be
taken care of by the existing atob/btoa.
Personally, I prefer option 3. But my bottom line is to have something working
as I believe
it's a useful feature for passing data across network or across processes.

[c.shu, 2014-01-29 15:25:29]

> I agree. Messing up the text encoding with binary encoding, which is targeted
> for data
> transferring, is probably not a good idea. It also makes the implementation
> complicated
> and confusing, as I have explored.
> So here are the possible approaches:
> 1. Text Encoder/Decoder
> Confusing APIs and complex implementation.
> 2. Add Base64 APIs in ArrayBuffer
> Seems desirable but people would like to keep the interface minimal.
> 3. Support StringView of ArrayBuffer
> All we need is being able to construct a StringView from an existing
ArrayBuffer
> 
> and take the buffer from a StringView. Base64 encoding itself can be
> taken care of by the existing atob/btoa.
> Personally, I prefer option 3. But my bottom line is to have something working
> as I believe
> it's a useful feature for passing data across network or across processes.

The 4th option is to enhance atob/btoa to take ArrayBuffer as an input.

[jungshik at Google, 2014-01-29 22:46:27]

On 2014/01/29 15:25:29, c.shu wrote:
> > I agree. Messing up the text encoding with binary encoding, which is
targeted
> > for data
> > transferring, is probably not a good idea. It also makes the implementation
> > complicated
> > and confusing, as I have explored.
> > So here are the possible approaches:
> > 1. Text Encoder/Decoder
> > Confusing APIs and complex implementation.
> > 2. Add Base64 APIs in ArrayBuffer
> > Seems desirable but people would like to keep the interface minimal.
> > 3. Support StringView of ArrayBuffer
> > All we need is being able to construct a StringView from an existing
> ArrayBuffer
> > 
> > and take the buffer from a StringView. Base64 encoding itself can be
> > taken care of by the existing atob/btoa.
> > Personally, I prefer option 3. But my bottom line is to have something
working
> > as I believe
> > it's a useful feature for passing data across network or across processes.
> 
> The 4th option is to enhance atob/btoa to take ArrayBuffer as an input.

I'm with you, too. Either 3rd or 4th would be nice (if it's hard to get
consensus on the 2nd).