Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(371)

Unified Diff: sdk/lib/io/data_transformer.dart

Issue 130513003: [ZLIB] Add support for windowBits, memLevel, raw (Closed) Base URL: https://dart.googlecode.com/svn/branches/bleeding_edge/dart
Patch Set: fix dictionary memory management Created 6 years, 11 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View side-by-side diff with in-line comments
Download patch
Index: sdk/lib/io/data_transformer.dart
diff --git a/sdk/lib/io/data_transformer.dart b/sdk/lib/io/data_transformer.dart
index dec5c5df7f5fc3bde60bb76dbf2fb4cd0d1e667c..2e43584878f8fd5d33ab6e9f3f61d7b346ddf565 100644
--- a/sdk/lib/io/data_transformer.dart
+++ b/sdk/lib/io/data_transformer.dart
@@ -5,11 +5,39 @@
part of dart.io;
+/// Minimal value for [windowBits]
+const int ZLIB_MIN_WINDOW_BITS = 8;
Anders Johnsen 2014/02/04 12:30:45 Talking with lrn@ about this, can you create a new
Lasse Reichstein Nielsen 2014/02/04 13:11:57 I think it is better with an abstract ZLib class c
vicb 2014/02/04 19:55:14 This is what I did first before reverting. I'll re
vicb 2014/02/04 19:55:14 Changed for ZLibConst + DEFAULT_
+/// Maximal value for [windowBits]
+const int ZLIB_MAX_WINDOW_BITS = 15;
+/// Default value for [windowBits]
+const int ZLIB_DFT_WINDOW_BITS = 15;
+
+/// Minimal value for [level]
+const int ZLIB_MIN_LEVEL = -1;
+/// Maximal value for [level]
+const int ZLIB_MAX_LEVEL = 9;
+/// Default value for [level]
+const int ZLIB_DFT_LEVEL = 6;
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Maybe DFT -> DEFAULT (if that's what it stands for
+
+/// Minimal value for [memLevel]
+const int ZLIB_MIN_MEM_LEVEL = 1;
+/// Maximal value for [memLevel]
+const int ZLIB_MAX_MEM_LEVEL = 9;
+/// Default value for [memLevel]
+const int ZLIB_DFT_MEM_LEVEL = 8;
+
+/// Compression strategies
+const int ZLIB_STRATEGY_FILTERED = 1;
+const int ZLIB_STRATEGY_HUFFMAN_ONLY = 2;
+const int ZLIB_STRATEGY_RLE = 3;
+const int ZLIB_STRATEGY_FIXED = 4;
+const int ZLIB_STRATEGY_DEFAULT = 0;
+
+
/**
* An instance of the default implementation of the [ZLibCodec].
*/
-const ZLibCodec ZLIB = const ZLibCodec();
-
+final ZLibCodec ZLIB = new ZLibCodec();
Anders Johnsen 2014/02/04 12:30:45 The problem here is that ZLIB is now wrong coding
Lasse Reichstein Nielsen 2014/02/04 13:11:57 I don't *really* mind having ZLIB be a non-const v
vicb 2014/02/04 19:55:14 ZLIB is "almost constant" as the class has no stat
vicb 2014/02/04 19:55:14 done
/**
* The [ZLibCodec] encodes raw bytes to ZLib compressed bytes and decodes ZLib
@@ -17,107 +45,278 @@ const ZLibCodec ZLIB = const ZLibCodec();
*/
class ZLibCodec extends Codec<List<int>, List<int>> {
/**
- * The compression level of the [ZLibCodec].
+ * When true, `GZip` frames will be added to the compressed data.
+ */
+ final bool gzip;
+
+ /**
+ * The compression-[level] can be set in the range of `-1..9`, with `6` being
+ * the default compression level. Levels above `6` will have higher compression
Lasse Reichstein Nielsen 2014/02/04 13:11:57 long line.
vicb 2014/02/04 19:55:14 fixed
+ * rates at the cost of more CPU and memory usage. Levels below `6` will use
+ * less CPU and memory at the cost of lower compression rates.
*/
final int level;
/**
+ * Specifies how much memory should be allocated for the internal compression
+ * state. `1` uses minimum memory but is slow and reduces compression ratio;
+ * `9` uses maximum memory for optimal speed. The default value is `8`.
+ *
+ * The memory requirements for deflate are (in bytes):
+ * (1 << (windowBits + 2)) + (1 << (memLevel + 9))
+ * that is: 128K for windowBits = 15 + 128K for memLevel = 8 (default values)
+ */
+ final int memLevel;
+
+ /**
+ * Tunes the compression algorithm. Use the value STRATEGY_DEFAULT for normal
+ * data, STRATEGY_FILTERED for data produced by a filter (or predictor),
+ * STRATEGY_HUFFMAN_ONLY to force Huffman encoding only (no string match), or
+ * STRATEGY_RLE to limit match distances to one (run-length encoding).
+ */
+ final int strategy;
+
+ /**
+ * Base two logarithm of the window size (the size of the history buffer). It
+ * should be in the range 8..15. Larger values result in better compression at
+ * the expense of memory usage. The default value is 15
+ */
+ final int windowBits;
+
+ /**
+ * When true, deflate generates raw data with no zlib header or trailer, and
+ * will not compute an adler32 check value
+ */
+ final bool raw;
+
+ /**
+ * Initial compression dictionary.
+ *
+ * It should consist of strings (byte sequences) that are likely to be
+ * encountered later in the data to be compressed, with the most commonly used
+ * strings preferably put towards the end of the dictionary. Using a dictionary
+ * is most useful when the data to be compressed is short and can be predicted
+ * with good accuracy; the data can then be compressed better than with the
+ * default empty dictionary.
+ */
+ final List<int> dictionary;
+
+ ZLibCodec({this.level: ZLIB_DFT_LEVEL,
+ this.windowBits: ZLIB_DFT_WINDOW_BITS,
+ this.memLevel: ZLIB_DFT_MEM_LEVEL,
+ this.strategy: ZLIB_STRATEGY_DEFAULT,
+ this.dictionary: null,
+ this.raw: false}) {
+ validateZLibeLevel(level);
+ validateZLibMemLevel(memLevel);
+ validateZLibStrategy(strategy);
+ validateZLibWindowBits(windowBits);
+ }
+
+ /**
* Get a [Converter] for encoding to `ZLib` compressed data.
*/
Converter<List<int>, List<int>> get encoder =>
- new ZLibEncoder(gzip: false, level: level);
+ new ZLibEncoder(gzip: false, level: level, windowBits: windowBits,
+ memLevel: memLevel, strategy: strategy,
+ dictionary: dictionary, raw: raw);
/**
* Get a [Converter] for decoding `ZLib` compressed data.
*/
- Converter<List<int>, List<int>> get decoder => const ZLibDecoder();
-
- /**
- * The compression-[level] can be set in the range of `1..10`, with `6` being
- * the default compression level. Levels above 6 will have higher compression
- * rates at the cost of more CPU and memory usage. Levels below 6 will use
- * less CPU and memory, but at the cost of lower compression rates.
- */
- const ZLibCodec({this.level: 6});
+ Converter<List<int>, List<int>> get decoder => new ZLibDecoder(
+ windowBits: windowBits, dictionary: dictionary, raw: raw);
}
/**
* An instance of the default implementation of the [GZipCodec].
*/
-const GZipCodec GZIP = const GZipCodec();
+final GZipCodec GZIP = new GZipCodec();
/**
- * The [GZipCodec] encodes raw bytes to GZip compressed bytes and decodes GZip
+ * The GZipCodec encodes raw bytes to GZip compressed bytes and decodes GZip
* compressed bytes to raw bytes.
*
- * The difference between [ZLibCodec] and [GZipCodec] is that the [GZipCodec]
- * wraps the `ZLib` compressed bytes in `GZip` frames.
+ * The difference between [ZLibCodec] and GZipCodec is that the GZipCodec wraps
+ * the `ZLib` compressed bytes in `GZip` frames.
*/
class GZipCodec extends Codec<List<int>, List<int>> {
/**
- * The compression level of the [ZLibCodec].
+ * When true, `GZip` frames will be added to the compressed data.
+ */
+ final bool gzip;
+
+ /**
+ * The compression-[level] can be set in the range of `-1..9`, with `6` being
+ * the default compression level. Levels above `6` will have higher compression
Lasse Reichstein Nielsen 2014/02/04 13:11:57 long line.
+ * rates at the cost of more CPU and memory usage. Levels below `6` will use
+ * less CPU and memory at the cost of lower compression rates.
*/
final int level;
/**
+ * Specifies how much memory should be allocated for the internal compression
+ * state. `1` uses minimum memory but is slow and reduces compression ratio;
+ * `9` uses maximum memory for optimal speed. The default value is `8`.
+ *
+ * The memory requirements for deflate are (in bytes):
+ * (1 << (windowBits + 2)) + (1 << (memLevel + 9))
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Indent by four spaces and put an empty line before
+ * that is: 128K for windowBits = 15 + 128K for memLevel = 8 (default values)
+ */
+ final int memLevel;
+
+ /**
+ * Tunes the compression algorithm. Use the value STRATEGY_DEFAULT for normal
+ * data, STRATEGY_FILTERED for data produced by a filter (or predictor),
+ * STRATEGY_HUFFMAN_ONLY to force Huffman encoding only (no string match), or
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Use full names (ZLIB_*) for constants. Use either
+ * STRATEGY_RLE to limit match distances to one (run-length encoding).
+ */
+ final int strategy;
+
+ /**
+ * Base two logarithm of the window size (the size of the history buffer). It
+ * should be in the range 8..15. Larger values result in better compression at
+ * the expense of memory usage. The default value is 15
+ */
+ final int windowBits;
+
+ /**
+ * Initial compression dictionary.
+ *
+ * It should consist of strings (byte sequences) that are likely to be
+ * encountered later in the data to be compressed, with the most commonly used
+ * strings preferably put towards the end of the dictionary. Using a dictionary
+ * is most useful when the data to be compressed is short and can be predicted
+ * with good accuracy; the data can then be compressed better than with the
+ * default empty dictionary.
+ */
+ final List<int> dictionary;
+
+ /**
+ * When true, deflate generates raw data with no zlib header or trailer, and
+ * will not compute an adler32 check value
+ */
+ final bool raw;
+
+ GZipCodec({this.level: ZLIB_DFT_LEVEL,
+ this.windowBits: ZLIB_DFT_WINDOW_BITS,
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Arguably, I'd indent the lines so that "this" matc
vicb 2014/02/04 19:55:14 I prefer to let my IDE decide and not reindent by
+ this.memLevel: ZLIB_DFT_MEM_LEVEL,
+ this.strategy: ZLIB_STRATEGY_DEFAULT,
+ this.dictionary: null,
+ this.raw: false}) {
+ validateZLibeLevel(level);
+ validateZLibMemLevel(memLevel);
+ validateZLibStrategy(strategy);
+ validateZLibWindowBits(windowBits);
+ }
+
+ /**
* Get a [Converter] for encoding to `GZip` compressed data.
*/
Converter<List<int>, List<int>> get encoder =>
- new ZLibEncoder(gzip: true, level: level);
+ new ZLibEncoder(gzip: true, level: level, windowBits: windowBits,
+ memLevel: memLevel, strategy: strategy,
+ dictionary: dictionary, raw: raw);
Lasse Reichstein Nielsen 2014/02/04 13:11:57 You can do the same const default constructor tric
/**
* Get a [Converter] for decoding `GZip` compressed data.
*/
- Converter<List<int>, List<int>> get decoder => const ZLibDecoder();
-
- /**
- * The compression-[level] can be set in the range of `1..10`, with `6` being
- * the default compression level. Levels above 6 will have higher compression
- * rates at the cost of more CPU and memory usage. Levels below 6 will use
- * less CPU and memory, but at the cost of lower compression rates.
- */
- const GZipCodec({this.level: 6});
+ Converter<List<int>, List<int>> get decoder => new ZLibDecoder(
+ windowBits: windowBits, dictionary: dictionary, raw: raw);
}
-
/**
- * The [ZLibEncoder] is the encoder used by [ZLibCodec] and [GZipCodec] to
- * compress data.
+ * The ZLibEncoder encoder is used by [ZLibCodec] and [GZipCodec] to compress
+ * data.
*/
class ZLibEncoder extends Converter<List<int>, List<int>> {
/**
- * If [gzip] is true, `GZip` frames will be added to the compressed data.
+ * When true, `GZip` frames will be added to the compressed data.
*/
final bool gzip;
/**
- * The compression level used by the encoder.
+ * The compression-[level] can be set in the range of `-1..9`, with `6` being
+ * the default compression level. Levels above `6` will have higher compression
Lasse Reichstein Nielsen 2014/02/04 13:11:57 long line.
+ * rates at the cost of more CPU and memory usage. Levels below `6` will use
+ * less CPU and memory at the cost of lower compression rates.
*/
final int level;
/**
- * Create a new [ZLibEncoder] converter. If the [gzip] flag is set, the
- * encoder will wrap the encoded ZLib data in GZip frames.
- */
- const ZLibEncoder({this.gzip: false, this.level: 6});
+ * Specifies how much memory should be allocated for the internal compression
+ * state. `1` uses minimum memory but is slow and reduces compression ratio;
+ * `9` uses maximum memory for optimal speed. The default value is `8`.
+ *
+ * The memory requirements for deflate are (in bytes):
+ * (1 << (windowBits + 2)) + (1 << (memLevel + 9))
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Block-quote this.
vicb 2014/02/04 19:55:14 fixed
+ * that is: 128K for windowBits = 15 + 128K for memLevel = 8 (default values)
+ */
+ final int memLevel;
+
+ /**
+ * Tunes the compression algorithm. Use the value STRATEGY_DEFAULT for normal
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Full constant references [ZLIB_STRATEGY_DEFAULT] (
vicb 2014/02/04 19:55:14 fixed
+ * data, STRATEGY_FILTERED for data produced by a filter (or predictor),
+ * STRATEGY_HUFFMAN_ONLY to force Huffman encoding only (no string match), or
+ * STRATEGY_RLE to limit match distances to one (run-length encoding).
+ */
+ final int strategy;
+
+ /**
+ * Base two logarithm of the window size (the size of the history buffer). It
+ * should be in the range 8..15. Larger values result in better compression at
+ * the expense of memory usage. The default value is 15
+ */
+ final int windowBits;
+ /**
+ * Initial compression dictionary.
+ *
+ * It should consist of strings (byte sequences) that are likely to be
+ * encountered later in the data to be compressed, with the most commonly used
+ * strings preferably put towards the end of the dictionary. Using a dictionary
+ * is most useful when the data to be compressed is short and can be predicted
+ * with good accuracy; the data can then be compressed better than with the
+ * default empty dictionary.
+ */
+ final List<int> dictionary;
+
+
+ /**
+ * When true, deflate generates raw data with no zlib header or trailer, and
+ * will not compute an adler32 check value
+ */
+ final bool raw;
+
+ ZLibEncoder({this.gzip: false,
+ this.level: ZLIB_DFT_LEVEL,
+ this.windowBits: ZLIB_DFT_WINDOW_BITS,
+ this.memLevel: ZLIB_DFT_MEM_LEVEL,
+ this.strategy: ZLIB_STRATEGY_DEFAULT,
+ this.dictionary: null,
+ this.raw: false}) {
+ validateZLibeLevel(level);
+ validateZLibMemLevel(memLevel);
+ validateZLibStrategy(strategy);
+ validateZLibWindowBits(windowBits);
+ }
/**
- * Convert a list of bytes using the options given to the [ZLibEncoder]
+ * Convert a list of bytes using the options given to the ZLibEncoder
* constructor.
*/
List<int> convert(List<int> bytes) {
_BufferSink sink = new _BufferSink();
startChunkedConversion(sink)
- ..add(bytes)
- ..close();
+ ..add(bytes)
+ ..close();
return sink.builder.takeBytes();
}
/**
- * Start a chunked conversion using the options given to the [ZLibEncoder]
+ * Start a chunked conversion using the options given to the ZLibEncoder
Lasse Reichstein Nielsen 2014/02/04 13:11:57 No need to remove '[' and ']'. If you don't want a
vicb 2014/02/04 19:55:14 fixed
* constructor. While it accepts any [ChunkedConversionSink] taking
* [List<int>]'s, the optimal sink to be passed as [sink] is a
* [ByteConversionSink].
@@ -127,21 +326,46 @@ class ZLibEncoder extends Converter<List<int>, List<int>> {
if (sink is! ByteConversionSink) {
sink = new ByteConversionSink.from(sink);
}
- return new _ZLibEncoderSink(sink, gzip, level);
+
+ return new _ZLibEncoderSink(sink, gzip, level, windowBits, memLevel,
+ strategy, dictionary, raw);
}
}
/**
- * The [ZLibDecoder] is the decoder used by [ZLibCodec] and [GZipCodec] to
- * decompress data.
+ * The ZLibDecoder is used by [ZLibCodec] and [GZipCodec] to decompress data.
*/
class ZLibDecoder extends Converter<List<int>, List<int>> {
+ /**
+ * Base two logarithm of the window size (the size of the history buffer). It
+ * should be in the range 8..15. Larger values result in better compression at
+ * the expense of memory usage. The default value is 15
+ */
+ final int windowBits;
/**
- * Create a new [ZLibEncoder] converter.
- */
- const ZLibDecoder();
+ * Initial compression dictionary.
+ *
+ * It should consist of strings (byte sequences) that are likely to be
+ * encountered later in the data to be compressed, with the most commonly used
+ * strings preferably put towards the end of the dictionary. Using a dictionary
+ * is most useful when the data to be compressed is short and can be predicted
+ * with good accuracy; the data can then be compressed better than with the
+ * default empty dictionary.
+ */
+ final List<int> dictionary;
+
+ /**
+ * When true, deflate generates raw data with no zlib header or trailer, and
+ * will not compute an adler32 check value
+ */
+ final bool raw;
+
+ ZLibDecoder({this.windowBits: ZLIB_DFT_WINDOW_BITS, this.dictionary: null,
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Could have const default constructor too.
+ this.raw: false}) {
+ validateZLibWindowBits(windowBits);
+ }
/**
* Convert a list of bytes using the options given to the [ZLibDecoder]
@@ -150,8 +374,8 @@ class ZLibDecoder extends Converter<List<int>, List<int>> {
List<int> convert(List<int> bytes) {
_BufferSink sink = new _BufferSink();
startChunkedConversion(sink)
- ..add(bytes)
- ..close();
+ ..add(bytes)
+ ..close();
return sink.builder.takeBytes();
}
@@ -165,7 +389,7 @@ class ZLibDecoder extends Converter<List<int>, List<int>> {
if (sink is! ByteConversionSink) {
sink = new ByteConversionSink.from(sink);
}
- return new _ZLibDecoderSink(sink);
+ return new _ZLibDecoderSink(sink, windowBits, dictionary, raw);
}
}
@@ -191,14 +415,18 @@ class _BufferSink extends ByteConversionSink {
class _ZLibEncoderSink extends _FilterSink {
- _ZLibEncoderSink(ByteConversionSink sink, bool gzip, int level)
- : super(sink, _Filter.newZLibDeflateFilter(gzip, level));
+ _ZLibEncoderSink(ByteConversionSink sink, bool gzip, int level,
+ int windowBits, int memLevel, int strategy,
+ List<int> dictionary, bool raw)
+ : super(sink, _Filter.newZLibDeflateFilter(gzip, level, windowBits,
+ memLevel, strategy,
+ dictionary, raw));
}
-
class _ZLibDecoderSink extends _FilterSink {
- _ZLibDecoderSink(ByteConversionSink sink)
- : super(sink, _Filter.newZLibInflateFilter());
+ _ZLibDecoderSink(ByteConversionSink sink, int windowBits,
+ List<int> dictionary, bool raw)
+ : super(sink, _Filter.newZLibInflateFilter(windowBits, dictionary, raw));
}
@@ -208,7 +436,7 @@ class _FilterSink extends ByteConversionSink {
bool _closed = false;
bool _empty = true;
- _FilterSink(ByteConversionSink this._sink, _Filter this._filter);
+ _FilterSink(this._sink, this._filter);
void add(List<int> data) {
addSlice(data, 0, data.length, false);
@@ -258,7 +486,6 @@ class _FilterSink extends ByteConversionSink {
}
-
/**
* Private helper-class to handle native filters.
*/
@@ -286,6 +513,46 @@ abstract class _Filter {
*/
void end();
- external static _Filter newZLibDeflateFilter(bool gzip, int level);
- external static _Filter newZLibInflateFilter();
+ external static _Filter newZLibDeflateFilter(bool gzip, int level,
+ int windowBits, int memLevel,
+ int strategy,
+ List<int> dictionary, bool raw);
+
+ external static _Filter newZLibInflateFilter(int windowBits,
+ List<int> dictionary, bool raw);
+}
+
+void validateZLibWindowBits(int windowBits) {
+ if (windowBits is! int ||
Lasse Reichstein Nielsen 2014/02/04 13:11:57 I wouldn't test for int-ness here. The type annota
vicb 2014/02/04 19:55:14 Good to know, fixed
+ windowBits < ZLIB_MIN_WINDOW_BITS ||
+ windowBits > ZLIB_MAX_WINDOW_BITS) {
+ throw new ArgumentError("'windowBits' must be in range "
Lasse Reichstein Nielsen 2014/02/04 13:11:57 You can also use new RangeError.range(windowBit
vicb 2014/02/04 19:55:14 Thanks for the tip
+ "${ZLIB_MIN_WINDOW_BITS}..${ZLIB_MAX_WINDOW_BITS}");
+ }
+}
+
+void validateZLibeLevel(int level) {
+ if (level is! int ||
+ level < ZLIB_MIN_LEVEL ||
+ level > ZLIB_MAX_LEVEL) {
+ throw new ArgumentError("'level' must be in range ${ZLIB_MIN_LEVEL}.."
+ "${ZLIB_MAX_LEVEL}");
+ }
+}
+
+void validateZLibMemLevel(int memLevel) {
+ if (memLevel is! int ||
+ memLevel < ZLIB_MIN_MEM_LEVEL ||
+ memLevel > ZLIB_MAX_MEM_LEVEL) {
+ throw new ArgumentError("'memLevel' must be in range "
+ "${ZLIB_MIN_MEM_LEVEL}..${ZLIB_MAX_MEM_LEVEL}");
+ }
+}
+
+void validateZLibStrategy(int strategy) {
+ var strategies = <int>[ZLIB_STRATEGY_FILTERED, ZLIB_STRATEGY_HUFFMAN_ONLY,
Lasse Reichstein Nielsen 2014/02/04 13:11:57 Maybe make this list const. No need to allocate a
vicb 2014/02/04 19:55:14 done
+ ZLIB_STRATEGY_RLE, ZLIB_STRATEGY_FIXED, ZLIB_STRATEGY_DEFAULT];
+ if (strategies.indexOf(strategy) == -1) {
+ throw new ArgumentError("Unsupported 'strategy'");
+ }
}

Powered by Google App Engine
This is Rietveld 408576698