From 88cef262ec50cf905da8498b7af2d8f167fbf61d Mon Sep 17 00:00:00 2001 From: Przemyslaw Skibinski Date: Thu, 3 Nov 2016 13:25:20 +0100 Subject: documentation moved to doc/ --- README.md | 4 +- doc/lz4_Block_format.md | 127 ++++++++++++++++ doc/lz4_Frame_format.md | 385 ++++++++++++++++++++++++++++++++++++++++++++++++ lib/README.md | 2 +- lib/lz4.h | 2 +- lz4_Block_format.md | 127 ---------------- lz4_Frame_format.md | 385 ------------------------------------------------ 7 files changed, 516 insertions(+), 516 deletions(-) create mode 100644 doc/lz4_Block_format.md create mode 100644 doc/lz4_Frame_format.md delete mode 100644 lz4_Block_format.md delete mode 100644 lz4_Frame_format.md diff --git a/README.md b/README.md index 753f11d..3f9cba6 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,6 @@ A list of known source ports is maintained on the [LZ4 Homepage]. [Open-Source Benchmark program by m^2 (v0.14.3)]: http://encode.ru/threads/1371-Filesystem-benchmark?p=34029&viewfull=1#post34029 [Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia -[lz4_Block_format]: lz4_Block_format.md -[lz4_Frame_format]: lz4_Frame_format.md +[lz4_Block_format]: doc/lz4_Block_format.md +[lz4_Frame_format]: doc/lz4_Frame_format.md [LZ4 Homepage]: http://www.lz4.org diff --git a/doc/lz4_Block_format.md b/doc/lz4_Block_format.md new file mode 100644 index 0000000..0f6a5ba --- /dev/null +++ b/doc/lz4_Block_format.md @@ -0,0 +1,127 @@ +LZ4 Block Format Description +============================ +Last revised: 2015-05-07. +Author : Yann Collet + + +This specification is intended for developers +willing to produce LZ4-compatible compressed data blocks +using any programming language. + +LZ4 is an LZ77-type compressor with a fixed, byte-oriented encoding. +There is no entropy encoder back-end nor framing layer. +The latter is assumed to be handled by other parts of the system (see [LZ4 Frame format]). +This design is assumed to favor simplicity and speed. +It helps later on for optimizations, compactness, and features. + +This document describes only the block format, +not how the compressor nor decompressor actually work. +The correctness of the decompressor should not depend +on implementation details of the compressor, and vice versa. + +[LZ4 Frame format]: lz4_Frame_format.md + + + +Compressed block format +----------------------- +An LZ4 compressed block is composed of sequences. +A sequence is a suite of literals (not-compressed bytes), +followed by a match copy. + +Each sequence starts with a token. +The token is a one byte value, separated into two 4-bits fields. +Therefore each field ranges from 0 to 15. + + +The first field uses the 4 high-bits of the token. +It provides the length of literals to follow. + +If the field value is 0, then there is no literal. +If it is 15, then we need to add some more bytes to indicate the full length. +Each additional byte then represent a value from 0 to 255, +which is added to the previous value to produce a total length. +When the byte value is 255, another byte is output. +There can be any number of bytes following the token. There is no "size limit". +(Side note : this is why a not-compressible input block is expanded by 0.4%). + +Example 1 : A length of 48 will be represented as : + + - 15 : value for the 4-bits High field + - 33 : (=48-15) remaining length to reach 48 + +Example 2 : A length of 280 will be represented as : + + - 15 : value for the 4-bits High field + - 255 : following byte is maxed, since 280-15 >= 255 + - 10 : (=280 - 15 - 255) ) remaining length to reach 280 + +Example 3 : A length of 15 will be represented as : + + - 15 : value for the 4-bits High field + - 0 : (=15-15) yes, the zero must be output + +Following the token and optional length bytes, are the literals themselves. +They are exactly as numerous as previously decoded (length of literals). +It's possible that there are zero literal. + + +Following the literals is the match copy operation. + +It starts by the offset. +This is a 2 bytes value, in little endian format +(the 1st byte is the "low" byte, the 2nd one is the "high" byte). + +The offset represents the position of the match to be copied from. +1 means "current position - 1 byte". +The maximum offset value is 65535, 65536 cannot be coded. +Note that 0 is an invalid value, not used. + +Then we need to extract the match length. +For this, we use the second token field, the low 4-bits. +Value, obviously, ranges from 0 to 15. +However here, 0 means that the copy operation will be minimal. +The minimum length of a match, called minmatch, is 4. +As a consequence, a 0 value means 4 bytes, and a value of 15 means 19+ bytes. +Similar to literal length, on reaching the highest possible value (15), +we output additional bytes, one at a time, with values ranging from 0 to 255. +They are added to total to provide the final match length. +A 255 value means there is another byte to read and add. +There is no limit to the number of optional bytes that can be output this way. +(This points towards a maximum achievable compression ratio of about 250). + +With the offset and the matchlength, +the decoder can now proceed to copy the data from the already decoded buffer. +On decoding the matchlength, we reach the end of the compressed sequence, +and therefore start another one. + + +Parsing restrictions +----------------------- +There are specific parsing rules to respect in order to remain compatible +with assumptions made by the decoder : + +1. The last 5 bytes are always literals +2. The last match must start at least 12 bytes before end of block. + Consequently, a block with less than 13 bytes cannot be compressed. + +These rules are in place to ensure that the decoder +will never read beyond the input buffer, nor write beyond the output buffer. + +Note that the last sequence is also incomplete, +and stops right after literals. + + +Additional notes +----------------------- +There is no assumption nor limits to the way the compressor +searches and selects matches within the source data block. +It could be a fast scan, a multi-probe, a full search using BST, +standard hash chains or MMC, well whatever. + +Advanced parsing strategies can also be implemented, such as lazy match, +or full optimal parsing. + +All these trade-off offer distinctive speed/memory/compression advantages. +Whatever the method used by the compressor, its result will be decodable +by any LZ4 decoder if it follows the format specification described above. diff --git a/doc/lz4_Frame_format.md b/doc/lz4_Frame_format.md new file mode 100644 index 0000000..2ea1a86 --- /dev/null +++ b/doc/lz4_Frame_format.md @@ -0,0 +1,385 @@ +LZ4 Frame Format Description +============================ + +###Notices + +Copyright (c) 2013-2015 Yann Collet + +Permission is granted to copy and distribute this document +for any purpose and without charge, +including translations into other languages +and incorporation into compilations, +provided that the copyright notice and this notice are preserved, +and that any substantive changes or deletions from the original +are clearly marked. +Distribution of this document is unlimited. + +###Version + +1.5.1 (31/03/2015) + + +Introduction +------------ + +The purpose of this document is to define a lossless compressed data format, +that is independent of CPU type, operating system, +file system and character set, suitable for +File compression, Pipe and streaming compression +using the [LZ4 algorithm](http://www.lz4.org). + +The data can be produced or consumed, +even for an arbitrarily long sequentially presented input data stream, +using only an a priori bounded amount of intermediate storage, +and hence can be used in data communications. +The format uses the LZ4 compression method, +and optional [xxHash-32 checksum method](https://github.com/Cyan4973/xxHash), +for detection of data corruption. + +The data format defined by this specification +does not attempt to allow random access to compressed data. + +This specification is intended for use by implementers of software +to compress data into LZ4 format and/or decompress data from LZ4 format. +The text of the specification assumes a basic background in programming +at the level of bits and other primitive data representations. + +Unless otherwise indicated below, +a compliant compressor must produce data sets +that conform to the specifications presented here. +It doesn’t need to support all options though. + +A compliant decompressor must be able to decompress +at least one working set of parameters +that conforms to the specifications presented here. +It may also ignore checksums. +Whenever it does not support a specific parameter within the compressed stream, +it must produce a non-ambiguous error code +and associated error message explaining which parameter is unsupported. + + +General Structure of LZ4 Frame format +------------------------------------- + +| MagicNb | F. Descriptor | Block | (...) | EndMark | C. Checksum | +|:-------:|:-------------:| ----- | ----- | ------- | ----------- | +| 4 bytes | 3-11 bytes | | | 4 bytes | 0-4 bytes | + +__Magic Number__ + +4 Bytes, Little endian format. +Value : 0x184D2204 + +__Frame Descriptor__ + +3 to 11 Bytes, to be detailed in the next part. +Most important part of the spec. + +__Data Blocks__ + +To be detailed later on. +That’s where compressed data is stored. + +__EndMark__ + +The flow of blocks ends when the last data block has a size of “0”. +The size is expressed as a 32-bits value. + +__Content Checksum__ + +Content Checksum verify that the full content has been decoded correctly. +The content checksum is the result +of [xxh32() hash function](https://github.com/Cyan4973/xxHash) +digesting the original (decoded) data as input, and a seed of zero. +Content checksum is only present when its associated flag +is set in the frame descriptor. +Content Checksum validates the result, +that all blocks were fully transmitted in the correct order and without error, +and also that the encoding/decoding process itself generated no distortion. +Its usage is recommended. + +__Frame Concatenation__ + +In some circumstances, it may be preferable to append multiple frames, +for example in order to add new data to an existing compressed file +without re-framing it. + +In such case, each frame has its own set of descriptor flags. +Each frame is considered independent. +The only relation between frames is their sequential order. + +The ability to decode multiple concatenated frames +within a single stream or file +is left outside of this specification. +As an example, the reference lz4 command line utility behavior is +to decode all concatenated frames in their sequential order. + + +Frame Descriptor +---------------- + +| FLG | BD | (Content Size) | HC | +| ------- | ------- |:--------------:| ------- | +| 1 byte | 1 byte | 0 - 8 bytes | 1 byte | + +The descriptor uses a minimum of 3 bytes, +and up to 11 bytes depending on optional parameters. + +__FLG byte__ + +| BitNb | 7-6 | 5 | 4 | 3 | 2 | 1-0 | +| ------- | ------- | ------- | --------- | ------- | --------- | -------- | +|FieldName| Version | B.Indep | B.Checksum| C.Size | C.Checksum|*Reserved*| + + +__BD byte__ + +| BitNb | 7 | 6-5-4 | 3-2-1-0 | +| ------- | -------- | ------------ | -------- | +|FieldName|*Reserved*| Block MaxSize|*Reserved*| + +In the tables, bit 7 is highest bit, while bit 0 is lowest. + +__Version Number__ + +2-bits field, must be set to “01”. +Any other value cannot be decoded by this version of the specification. +Other version numbers will use different flag layouts. + +__Block Independence flag__ + +If this flag is set to “1”, blocks are independent. +If this flag is set to “0”, each block depends on previous ones +(up to LZ4 window size, which is 64 KB). +In such case, it’s necessary to decode all blocks in sequence. + +Block dependency improves compression ratio, especially for small blocks. +On the other hand, it makes direct jumps or multi-threaded decoding impossible. + +__Block checksum flag__ + +If this flag is set, each data block will be followed by a 4-bytes checksum, +calculated by using the xxHash-32 algorithm on the raw (compressed) data block. +The intention is to detect data corruption (storage or transmission errors) +immediately, before decoding. +Block checksum usage is optional. + +__Content Size flag__ + +If this flag is set, the uncompressed size of data included within the frame +will be present as an 8 bytes unsigned little endian value, after the flags. +Content Size usage is optional. + +__Content checksum flag__ + +If this flag is set, a content checksum will be appended after the EndMark. + +Recommended value : “1” (content checksum is present) + +__Block Maximum Size__ + +This information is intended to help the decoder allocate memory. +Size here refers to the original (uncompressed) data size. +Block Maximum Size is one value among the following table : + +| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | +| --- | --- | --- | --- | ----- | ------ | ---- | ---- | +| N/A | N/A | N/A | N/A | 64 KB | 256 KB | 1 MB | 4 MB | + +The decoder may refuse to allocate block sizes above a (system-specific) size. +Unused values may be used in a future revision of the spec. +A decoder conformant to the current version of the spec +is only able to decode blocksizes defined in this spec. + +__Reserved bits__ + +Value of reserved bits **must** be 0 (zero). +Reserved bit might be used in a future version of the specification, +typically enabling new optional features. +If this happens, a decoder respecting the current version of the specification +shall not be able to decode such a frame. + +__Content Size__ + +This is the original (uncompressed) size. +This information is optional, and only present if the associated flag is set. +Content size is provided using unsigned 8 Bytes, for a maximum of 16 HexaBytes. +Format is Little endian. +This value is informational, typically for display or memory allocation. +It can be skipped by a decoder, or used to validate content correctness. + +__Header Checksum__ + +One-byte checksum of combined descriptor fields, including optional ones. +The value is the second byte of xxh32() : ` (xxh32()>>8) & 0xFF ` +using zero as a seed, +and the full Frame Descriptor as an input +(including optional fields when they are present). +A wrong checksum indicates an error in the descriptor. +Header checksum is informational and can be skipped. + + +Data Blocks +----------- + +| Block Size | data | (Block Checksum) | +|:----------:| ------ |:----------------:| +| 4 bytes | | 0 - 4 bytes | + + +__Block Size__ + +This field uses 4-bytes, format is little-endian. + +The highest bit is “1” if data in the block is uncompressed. + +The highest bit is “0” if data in the block is compressed by LZ4. + +All other bits give the size, in bytes, of the following data block +(the size does not include the block checksum if present). + +Block Size shall never be larger than Block Maximum Size. +Such a thing could happen for incompressible source data. +In such case, such a data block shall be passed in uncompressed format. + +__Data__ + +Where the actual data to decode stands. +It might be compressed or not, depending on previous field indications. +Uncompressed size of Data can be any size, up to “block maximum size”. +Note that data block is not necessarily full : +an arbitrary “flush” may happen anytime. Any block can be “partially filled”. + +__Block checksum__ + +Only present if the associated flag is set. +This is a 4-bytes checksum value, in little endian format, +calculated by using the xxHash-32 algorithm on the raw (undecoded) data block, +and a seed of zero. +The intention is to detect data corruption (storage or transmission errors) +before decoding. + +Block checksum is cumulative with Content checksum. + + +Skippable Frames +---------------- + +| Magic Number | Frame Size | User Data | +|:------------:|:----------:| --------- | +| 4 bytes | 4 bytes | | + +Skippable frames allow the integration of user-defined data +into a flow of concatenated frames. +Its design is pretty straightforward, +with the sole objective to allow the decoder to quickly skip +over user-defined data and continue decoding. + +For the purpose of facilitating identification, +it is discouraged to start a flow of concatenated frames with a skippable frame. +If there is a need to start such a flow with some user data +encapsulated into a skippable frame, +it’s recommended to start with a zero-byte LZ4 frame +followed by a skippable frame. +This will make it easier for file type identifiers. + + +__Magic Number__ + +4 Bytes, Little endian format. +Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F. +All 16 values are valid to identify a skippable frame. + +__Frame Size__ + +This is the size, in bytes, of the following User Data +(without including the magic number nor the size field itself). +4 Bytes, Little endian format, unsigned 32-bits. +This means User Data can’t be bigger than (2^32-1) Bytes. + +__User Data__ + +User Data can be anything. Data will just be skipped by the decoder. + + +Legacy frame +------------ + +The Legacy frame format was defined into the initial versions of “LZ4Demo”. +Newer compressors should not use this format anymore, as it is too restrictive. + +Main characteristics of the legacy format : + +- Fixed block size : 8 MB. +- All blocks must be completely filled, except the last one. +- All blocks are always compressed, even when compression is detrimental. +- The last block is detected either because + it is followed by the “EOF” (End of File) mark, + or because it is followed by a known Frame Magic Number. +- No checksum +- Convention is Little endian + +| MagicNb | B.CSize | CData | B.CSize | CData | (...) | EndMark | +| ------- | ------- | ----- | ------- | ----- | ------- | ------- | +| 4 bytes | 4 bytes | CSize | 4 bytes | CSize | x times | EOF | + + +__Magic Number__ + +4 Bytes, Little endian format. +Value : 0x184C2102 + +__Block Compressed Size__ + +This is the size, in bytes, of the following compressed data block. +4 Bytes, Little endian format. + +__Data__ + +Where the actual compressed data stands. +Data is always compressed, even when compression is detrimental. + +__EndMark__ + +End of legacy frame is implicit only. +It must be followed by a standard EOF (End Of File) signal, +wether it is a file or a stream. + +Alternatively, if the frame is followed by a valid Frame Magic Number, +it is considered completed. +It makes legacy frames compatible with frame concatenation. + +Any other value will be interpreted as a block size, +and trigger an error if it does not fit within acceptable range. + + +Version changes +--------------- + +1.5.1 : changed format to MarkDown compatible + +1.5 : removed Dictionary ID from specification + +1.4.1 : changed wording from “stream” to “frame” + +1.4 : added skippable streams, re-added stream checksum + +1.3 : modified header checksum + +1.2 : reduced choice of “block size”, to postpone decision on “dynamic size of BlockSize Field”. + +1.1 : optional fields are now part of the descriptor + +1.0 : changed “block size” specification, adding a compressed/uncompressed flag + +0.9 : reduced scale of “block maximum size” table + +0.8 : removed : high compression flag + +0.7 : removed : stream checksum + +0.6 : settled : stream size uses 8 bytes, endian convention is little endian + +0.5: added copyright notice + +0.4 : changed format to Google Doc compatible OpenDocument diff --git a/lib/README.md b/lib/README.md index f932d42..209cac1 100644 --- a/lib/README.md +++ b/lib/README.md @@ -37,4 +37,4 @@ Other files present in the directory are not source code. There are : - liblz4.pc.in : for pkg-config (make install) - README.md : this file -[official interoperable frame format]: ../lz4_Frame_format.md +[official interoperable frame format]: ../doc/lz4_Frame_format.md diff --git a/lib/lz4.h b/lib/lz4.h index 3ece4dd..0f17c7c 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -44,7 +44,7 @@ extern "C" { * Block compression functions are not-enough to send information, * since it's still necessary to provide metadata (such as compressed size), * and each application can do it in whichever way it wants. - * For interoperability, there is LZ4 frame specification (lz4_Frame_format.md). + * For interoperability, there is LZ4 frame specification (doc/lz4_Frame_format.md). * A library is provided to take care of it, see lz4frame.h. */ diff --git a/lz4_Block_format.md b/lz4_Block_format.md deleted file mode 100644 index 0f6a5ba..0000000 --- a/lz4_Block_format.md +++ /dev/null @@ -1,127 +0,0 @@ -LZ4 Block Format Description -============================ -Last revised: 2015-05-07. -Author : Yann Collet - - -This specification is intended for developers -willing to produce LZ4-compatible compressed data blocks -using any programming language. - -LZ4 is an LZ77-type compressor with a fixed, byte-oriented encoding. -There is no entropy encoder back-end nor framing layer. -The latter is assumed to be handled by other parts of the system (see [LZ4 Frame format]). -This design is assumed to favor simplicity and speed. -It helps later on for optimizations, compactness, and features. - -This document describes only the block format, -not how the compressor nor decompressor actually work. -The correctness of the decompressor should not depend -on implementation details of the compressor, and vice versa. - -[LZ4 Frame format]: lz4_Frame_format.md - - - -Compressed block format ------------------------ -An LZ4 compressed block is composed of sequences. -A sequence is a suite of literals (not-compressed bytes), -followed by a match copy. - -Each sequence starts with a token. -The token is a one byte value, separated into two 4-bits fields. -Therefore each field ranges from 0 to 15. - - -The first field uses the 4 high-bits of the token. -It provides the length of literals to follow. - -If the field value is 0, then there is no literal. -If it is 15, then we need to add some more bytes to indicate the full length. -Each additional byte then represent a value from 0 to 255, -which is added to the previous value to produce a total length. -When the byte value is 255, another byte is output. -There can be any number of bytes following the token. There is no "size limit". -(Side note : this is why a not-compressible input block is expanded by 0.4%). - -Example 1 : A length of 48 will be represented as : - - - 15 : value for the 4-bits High field - - 33 : (=48-15) remaining length to reach 48 - -Example 2 : A length of 280 will be represented as : - - - 15 : value for the 4-bits High field - - 255 : following byte is maxed, since 280-15 >= 255 - - 10 : (=280 - 15 - 255) ) remaining length to reach 280 - -Example 3 : A length of 15 will be represented as : - - - 15 : value for the 4-bits High field - - 0 : (=15-15) yes, the zero must be output - -Following the token and optional length bytes, are the literals themselves. -They are exactly as numerous as previously decoded (length of literals). -It's possible that there are zero literal. - - -Following the literals is the match copy operation. - -It starts by the offset. -This is a 2 bytes value, in little endian format -(the 1st byte is the "low" byte, the 2nd one is the "high" byte). - -The offset represents the position of the match to be copied from. -1 means "current position - 1 byte". -The maximum offset value is 65535, 65536 cannot be coded. -Note that 0 is an invalid value, not used. - -Then we need to extract the match length. -For this, we use the second token field, the low 4-bits. -Value, obviously, ranges from 0 to 15. -However here, 0 means that the copy operation will be minimal. -The minimum length of a match, called minmatch, is 4. -As a consequence, a 0 value means 4 bytes, and a value of 15 means 19+ bytes. -Similar to literal length, on reaching the highest possible value (15), -we output additional bytes, one at a time, with values ranging from 0 to 255. -They are added to total to provide the final match length. -A 255 value means there is another byte to read and add. -There is no limit to the number of optional bytes that can be output this way. -(This points towards a maximum achievable compression ratio of about 250). - -With the offset and the matchlength, -the decoder can now proceed to copy the data from the already decoded buffer. -On decoding the matchlength, we reach the end of the compressed sequence, -and therefore start another one. - - -Parsing restrictions ------------------------ -There are specific parsing rules to respect in order to remain compatible -with assumptions made by the decoder : - -1. The last 5 bytes are always literals -2. The last match must start at least 12 bytes before end of block. - Consequently, a block with less than 13 bytes cannot be compressed. - -These rules are in place to ensure that the decoder -will never read beyond the input buffer, nor write beyond the output buffer. - -Note that the last sequence is also incomplete, -and stops right after literals. - - -Additional notes ------------------------ -There is no assumption nor limits to the way the compressor -searches and selects matches within the source data block. -It could be a fast scan, a multi-probe, a full search using BST, -standard hash chains or MMC, well whatever. - -Advanced parsing strategies can also be implemented, such as lazy match, -or full optimal parsing. - -All these trade-off offer distinctive speed/memory/compression advantages. -Whatever the method used by the compressor, its result will be decodable -by any LZ4 decoder if it follows the format specification described above. diff --git a/lz4_Frame_format.md b/lz4_Frame_format.md deleted file mode 100644 index 2ea1a86..0000000 --- a/lz4_Frame_format.md +++ /dev/null @@ -1,385 +0,0 @@ -LZ4 Frame Format Description -============================ - -###Notices - -Copyright (c) 2013-2015 Yann Collet - -Permission is granted to copy and distribute this document -for any purpose and without charge, -including translations into other languages -and incorporation into compilations, -provided that the copyright notice and this notice are preserved, -and that any substantive changes or deletions from the original -are clearly marked. -Distribution of this document is unlimited. - -###Version - -1.5.1 (31/03/2015) - - -Introduction ------------- - -The purpose of this document is to define a lossless compressed data format, -that is independent of CPU type, operating system, -file system and character set, suitable for -File compression, Pipe and streaming compression -using the [LZ4 algorithm](http://www.lz4.org). - -The data can be produced or consumed, -even for an arbitrarily long sequentially presented input data stream, -using only an a priori bounded amount of intermediate storage, -and hence can be used in data communications. -The format uses the LZ4 compression method, -and optional [xxHash-32 checksum method](https://github.com/Cyan4973/xxHash), -for detection of data corruption. - -The data format defined by this specification -does not attempt to allow random access to compressed data. - -This specification is intended for use by implementers of software -to compress data into LZ4 format and/or decompress data from LZ4 format. -The text of the specification assumes a basic background in programming -at the level of bits and other primitive data representations. - -Unless otherwise indicated below, -a compliant compressor must produce data sets -that conform to the specifications presented here. -It doesn’t need to support all options though. - -A compliant decompressor must be able to decompress -at least one working set of parameters -that conforms to the specifications presented here. -It may also ignore checksums. -Whenever it does not support a specific parameter within the compressed stream, -it must produce a non-ambiguous error code -and associated error message explaining which parameter is unsupported. - - -General Structure of LZ4 Frame format -------------------------------------- - -| MagicNb | F. Descriptor | Block | (...) | EndMark | C. Checksum | -|:-------:|:-------------:| ----- | ----- | ------- | ----------- | -| 4 bytes | 3-11 bytes | | | 4 bytes | 0-4 bytes | - -__Magic Number__ - -4 Bytes, Little endian format. -Value : 0x184D2204 - -__Frame Descriptor__ - -3 to 11 Bytes, to be detailed in the next part. -Most important part of the spec. - -__Data Blocks__ - -To be detailed later on. -That’s where compressed data is stored. - -__EndMark__ - -The flow of blocks ends when the last data block has a size of “0”. -The size is expressed as a 32-bits value. - -__Content Checksum__ - -Content Checksum verify that the full content has been decoded correctly. -The content checksum is the result -of [xxh32() hash function](https://github.com/Cyan4973/xxHash) -digesting the original (decoded) data as input, and a seed of zero. -Content checksum is only present when its associated flag -is set in the frame descriptor. -Content Checksum validates the result, -that all blocks were fully transmitted in the correct order and without error, -and also that the encoding/decoding process itself generated no distortion. -Its usage is recommended. - -__Frame Concatenation__ - -In some circumstances, it may be preferable to append multiple frames, -for example in order to add new data to an existing compressed file -without re-framing it. - -In such case, each frame has its own set of descriptor flags. -Each frame is considered independent. -The only relation between frames is their sequential order. - -The ability to decode multiple concatenated frames -within a single stream or file -is left outside of this specification. -As an example, the reference lz4 command line utility behavior is -to decode all concatenated frames in their sequential order. - - -Frame Descriptor ----------------- - -| FLG | BD | (Content Size) | HC | -| ------- | ------- |:--------------:| ------- | -| 1 byte | 1 byte | 0 - 8 bytes | 1 byte | - -The descriptor uses a minimum of 3 bytes, -and up to 11 bytes depending on optional parameters. - -__FLG byte__ - -| BitNb | 7-6 | 5 | 4 | 3 | 2 | 1-0 | -| ------- | ------- | ------- | --------- | ------- | --------- | -------- | -|FieldName| Version | B.Indep | B.Checksum| C.Size | C.Checksum|*Reserved*| - - -__BD byte__ - -| BitNb | 7 | 6-5-4 | 3-2-1-0 | -| ------- | -------- | ------------ | -------- | -|FieldName|*Reserved*| Block MaxSize|*Reserved*| - -In the tables, bit 7 is highest bit, while bit 0 is lowest. - -__Version Number__ - -2-bits field, must be set to “01”. -Any other value cannot be decoded by this version of the specification. -Other version numbers will use different flag layouts. - -__Block Independence flag__ - -If this flag is set to “1”, blocks are independent. -If this flag is set to “0”, each block depends on previous ones -(up to LZ4 window size, which is 64 KB). -In such case, it’s necessary to decode all blocks in sequence. - -Block dependency improves compression ratio, especially for small blocks. -On the other hand, it makes direct jumps or multi-threaded decoding impossible. - -__Block checksum flag__ - -If this flag is set, each data block will be followed by a 4-bytes checksum, -calculated by using the xxHash-32 algorithm on the raw (compressed) data block. -The intention is to detect data corruption (storage or transmission errors) -immediately, before decoding. -Block checksum usage is optional. - -__Content Size flag__ - -If this flag is set, the uncompressed size of data included within the frame -will be present as an 8 bytes unsigned little endian value, after the flags. -Content Size usage is optional. - -__Content checksum flag__ - -If this flag is set, a content checksum will be appended after the EndMark. - -Recommended value : “1” (content checksum is present) - -__Block Maximum Size__ - -This information is intended to help the decoder allocate memory. -Size here refers to the original (uncompressed) data size. -Block Maximum Size is one value among the following table : - -| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | -| --- | --- | --- | --- | ----- | ------ | ---- | ---- | -| N/A | N/A | N/A | N/A | 64 KB | 256 KB | 1 MB | 4 MB | - -The decoder may refuse to allocate block sizes above a (system-specific) size. -Unused values may be used in a future revision of the spec. -A decoder conformant to the current version of the spec -is only able to decode blocksizes defined in this spec. - -__Reserved bits__ - -Value of reserved bits **must** be 0 (zero). -Reserved bit might be used in a future version of the specification, -typically enabling new optional features. -If this happens, a decoder respecting the current version of the specification -shall not be able to decode such a frame. - -__Content Size__ - -This is the original (uncompressed) size. -This information is optional, and only present if the associated flag is set. -Content size is provided using unsigned 8 Bytes, for a maximum of 16 HexaBytes. -Format is Little endian. -This value is informational, typically for display or memory allocation. -It can be skipped by a decoder, or used to validate content correctness. - -__Header Checksum__ - -One-byte checksum of combined descriptor fields, including optional ones. -The value is the second byte of xxh32() : ` (xxh32()>>8) & 0xFF ` -using zero as a seed, -and the full Frame Descriptor as an input -(including optional fields when they are present). -A wrong checksum indicates an error in the descriptor. -Header checksum is informational and can be skipped. - - -Data Blocks ------------ - -| Block Size | data | (Block Checksum) | -|:----------:| ------ |:----------------:| -| 4 bytes | | 0 - 4 bytes | - - -__Block Size__ - -This field uses 4-bytes, format is little-endian. - -The highest bit is “1” if data in the block is uncompressed. - -The highest bit is “0” if data in the block is compressed by LZ4. - -All other bits give the size, in bytes, of the following data block -(the size does not include the block checksum if present). - -Block Size shall never be larger than Block Maximum Size. -Such a thing could happen for incompressible source data. -In such case, such a data block shall be passed in uncompressed format. - -__Data__ - -Where the actual data to decode stands. -It might be compressed or not, depending on previous field indications. -Uncompressed size of Data can be any size, up to “block maximum size”. -Note that data block is not necessarily full : -an arbitrary “flush” may happen anytime. Any block can be “partially filled”. - -__Block checksum__ - -Only present if the associated flag is set. -This is a 4-bytes checksum value, in little endian format, -calculated by using the xxHash-32 algorithm on the raw (undecoded) data block, -and a seed of zero. -The intention is to detect data corruption (storage or transmission errors) -before decoding. - -Block checksum is cumulative with Content checksum. - - -Skippable Frames ----------------- - -| Magic Number | Frame Size | User Data | -|:------------:|:----------:| --------- | -| 4 bytes | 4 bytes | | - -Skippable frames allow the integration of user-defined data -into a flow of concatenated frames. -Its design is pretty straightforward, -with the sole objective to allow the decoder to quickly skip -over user-defined data and continue decoding. - -For the purpose of facilitating identification, -it is discouraged to start a flow of concatenated frames with a skippable frame. -If there is a need to start such a flow with some user data -encapsulated into a skippable frame, -it’s recommended to start with a zero-byte LZ4 frame -followed by a skippable frame. -This will make it easier for file type identifiers. - - -__Magic Number__ - -4 Bytes, Little endian format. -Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F. -All 16 values are valid to identify a skippable frame. - -__Frame Size__ - -This is the size, in bytes, of the following User Data -(without including the magic number nor the size field itself). -4 Bytes, Little endian format, unsigned 32-bits. -This means User Data can’t be bigger than (2^32-1) Bytes. - -__User Data__ - -User Data can be anything. Data will just be skipped by the decoder. - - -Legacy frame ------------- - -The Legacy frame format was defined into the initial versions of “LZ4Demo”. -Newer compressors should not use this format anymore, as it is too restrictive. - -Main characteristics of the legacy format : - -- Fixed block size : 8 MB. -- All blocks must be completely filled, except the last one. -- All blocks are always compressed, even when compression is detrimental. -- The last block is detected either because - it is followed by the “EOF” (End of File) mark, - or because it is followed by a known Frame Magic Number. -- No checksum -- Convention is Little endian - -| MagicNb | B.CSize | CData | B.CSize | CData | (...) | EndMark | -| ------- | ------- | ----- | ------- | ----- | ------- | ------- | -| 4 bytes | 4 bytes | CSize | 4 bytes | CSize | x times | EOF | - - -__Magic Number__ - -4 Bytes, Little endian format. -Value : 0x184C2102 - -__Block Compressed Size__ - -This is the size, in bytes, of the following compressed data block. -4 Bytes, Little endian format. - -__Data__ - -Where the actual compressed data stands. -Data is always compressed, even when compression is detrimental. - -__EndMark__ - -End of legacy frame is implicit only. -It must be followed by a standard EOF (End Of File) signal, -wether it is a file or a stream. - -Alternatively, if the frame is followed by a valid Frame Magic Number, -it is considered completed. -It makes legacy frames compatible with frame concatenation. - -Any other value will be interpreted as a block size, -and trigger an error if it does not fit within acceptable range. - - -Version changes ---------------- - -1.5.1 : changed format to MarkDown compatible - -1.5 : removed Dictionary ID from specification - -1.4.1 : changed wording from “stream” to “frame” - -1.4 : added skippable streams, re-added stream checksum - -1.3 : modified header checksum - -1.2 : reduced choice of “block size”, to postpone decision on “dynamic size of BlockSize Field”. - -1.1 : optional fields are now part of the descriptor - -1.0 : changed “block size” specification, adding a compressed/uncompressed flag - -0.9 : reduced scale of “block maximum size” table - -0.8 : removed : high compression flag - -0.7 : removed : stream checksum - -0.6 : settled : stream size uses 8 bytes, endian convention is little endian - -0.5: added copyright notice - -0.4 : changed format to Google Doc compatible OpenDocument -- cgit v0.12 From e624978ea2a0ede65a9d43fc4f514addb3d8b12a Mon Sep 17 00:00:00 2001 From: Przemyslaw Skibinski Date: Thu, 3 Nov 2016 15:03:43 +0100 Subject: test programs moved to tests/ --- Makefile | 5 +- programs/Makefile | 198 +-------- programs/datagencli.c | 190 -------- programs/frametest.c | 866 ------------------------------------ programs/fullbench.c | 813 ---------------------------------- programs/fuzzer.c | 1169 ------------------------------------------------- tests/Makefile | 280 ++++++++++++ tests/datagencli.c | 190 ++++++++ tests/frametest.c | 866 ++++++++++++++++++++++++++++++++++++ tests/fullbench.c | 813 ++++++++++++++++++++++++++++++++++ tests/fuzzer.c | 1169 +++++++++++++++++++++++++++++++++++++++++++++++++ 11 files changed, 3324 insertions(+), 3235 deletions(-) delete mode 100644 programs/datagencli.c delete mode 100644 programs/frametest.c delete mode 100644 programs/fullbench.c delete mode 100644 programs/fuzzer.c create mode 100644 tests/Makefile create mode 100644 tests/datagencli.c create mode 100644 tests/frametest.c create mode 100644 tests/fullbench.c create mode 100644 tests/fuzzer.c diff --git a/Makefile b/Makefile index 518ffbd..3df03f7 100644 --- a/Makefile +++ b/Makefile @@ -37,6 +37,7 @@ LIBDIR ?= $(PREFIX)/lib INCLUDEDIR=$(PREFIX)/include PRGDIR = programs LZ4DIR = lib +TESTDIR = tests # Define nul output @@ -64,6 +65,7 @@ lz4: clean: @$(MAKE) -C $(PRGDIR) $@ > $(VOID) + @$(MAKE) -C $(TESTDIR) $@ > $(VOID) @$(MAKE) -C $(LZ4DIR) $@ > $(VOID) @$(MAKE) -C examples $@ > $(VOID) @$(MAKE) -C versionsTest $@ > $(VOID) @@ -88,7 +90,7 @@ travis-install: sudo $(MAKE) install test: - $(MAKE) -C $(PRGDIR) test + $(MAKE) -C $(TESTDIR) test cmake: @cd contrib/cmake_unofficial; cmake CMakeLists.txt; $(MAKE) @@ -108,6 +110,7 @@ staticAnalyze: clean armtest: clean CFLAGS="-O3 -Werror" $(MAKE) -C $(LZ4DIR) all CC=arm-linux-gnueabi-gcc CFLAGS="-O3 -Werror" $(MAKE) -C $(PRGDIR) bins CC=arm-linux-gnueabi-gcc + CFLAGS="-O3 -Werror" $(MAKE) -C $(TESTDIR) bins CC=arm-linux-gnueabi-gcc versionsTest: clean $(MAKE) -C versionsTest diff --git a/programs/Makefile b/programs/Makefile index 6214084..8cb416c 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -25,13 +25,6 @@ # lz4 : Command Line Utility, supporting gzip-like arguments # lz4c : CLU, supporting also legacy lz4demo arguments # lz4c32: Same as lz4c, but forced to compile in 32-bits mode -# fuzzer : Test tool, to check lz4 integrity on target platform -# fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode -# frametest : Test tool, to check lz4frame integrity on target platform -# frametest32: Same as frametest, but forced to compile in 32-bits mode -# fullbench : Precisely measure speed for each LZ4 function variant -# fullbench32: Same as fullbench, but forced to compile in 32-bits mode -# datagen : generates synthetic data samples for tests & benchmarks # ########################################################################## DESTDIR ?= @@ -64,9 +57,9 @@ FUZZER_TIME := -T9mn default: lz4 -m32: lz4c32 fullbench32 fuzzer32 frametest32 +m32: lz4c32 -bins: lz4 lz4c fullbench fuzzer frametest datagen +bins: lz4 lz4c all: bins m32 @@ -79,35 +72,10 @@ lz4c : $(LZ4DIR)/lz4.o $(LZ4DIR)/lz4hc.o $(LZ4DIR)/lz4frame.o $(LZ4DIR)/xxhash. lz4c32: $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4frame.c $(LZ4DIR)/xxhash.c bench.c lz4io.c lz4cli.c $(CC) -m32 $(FLAGS) -DENABLE_LZ4C_LEGACY_OPTIONS $^ -o $@$(EXT) -fullbench : $(LZ4DIR)/lz4.o $(LZ4DIR)/lz4hc.o $(LZ4DIR)/lz4frame.o $(LZ4DIR)/xxhash.o fullbench.c - $(CC) $(FLAGS) $^ -o $@$(EXT) - -fullbench32: $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4frame.c $(LZ4DIR)/xxhash.c fullbench.c - $(CC) -m32 $(FLAGS) $^ -o $@$(EXT) - -fuzzer : $(LZ4DIR)/lz4.o $(LZ4DIR)/lz4hc.o $(LZ4DIR)/xxhash.o fuzzer.c - $(CC) $(FLAGS) $^ -o $@$(EXT) - -fuzzer32: $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/xxhash.c fuzzer.c - $(CC) -m32 $(FLAGS) $^ -o $@$(EXT) - -frametest: $(LZ4DIR)/lz4frame.o $(LZ4DIR)/lz4.o $(LZ4DIR)/lz4hc.o $(LZ4DIR)/xxhash.o frametest.c - $(CC) $(FLAGS) $^ -o $@$(EXT) - -frametest32: $(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/xxhash.c frametest.c - $(CC) -m32 $(FLAGS) $^ -o $@$(EXT) - -datagen : datagen.o datagencli.c - $(CC) $(FLAGS) $^ -o $@$(EXT) - clean: @$(MAKE) -C $(LZ4DIR) $@ > $(VOID) @$(RM) core *.o *.test tmp* \ lz4$(EXT) lz4c$(EXT) lz4c32$(EXT) \ - fullbench$(EXT) fullbench32$(EXT) \ - fuzzer$(EXT) fuzzer32$(EXT) \ - frametest$(EXT) frametest32$(EXT) \ - datagen$(EXT) @echo Cleaning completed @@ -141,166 +109,4 @@ uninstall: @$(RM) $(DESTDIR)$(MANDIR)/unlz4.1 @echo lz4 programs successfully uninstalled -test: test-lz4 test-lz4c test-frametest test-fullbench test-fuzzer test-mem - -test32: test-lz4c32 test-frametest32 test-fullbench32 test-fuzzer32 test-mem32 - -test-all: test test32 - -test-lz4-sparse: lz4 datagen - @echo "\n ---- test sparse file support ----" - ./datagen -g5M -P100 > tmpSrc - ./lz4 -B4D tmpSrc | ./lz4 -dv --sparse > tmpB4 - diff -s tmpSrc tmpB4 - ./lz4 -B5D tmpSrc | ./lz4 -dv --sparse > tmpB5 - diff -s tmpSrc tmpB5 - ./lz4 -B6D tmpSrc | ./lz4 -dv --sparse > tmpB6 - diff -s tmpSrc tmpB6 - ./lz4 -B7D tmpSrc | ./lz4 -dv --sparse > tmpB7 - diff -s tmpSrc tmpB7 - ./lz4 tmpSrc | ./lz4 -dv --no-sparse > tmpNoSparse - diff -s tmpSrc tmpNoSparse - ls -ls tmp* - ./datagen -s1 -g1200007 -P100 | ./lz4 | ./lz4 -dv --sparse > tmpOdd # Odd size file (to generate non-full last block) - ./datagen -s1 -g1200007 -P100 | diff -s - tmpOdd - ls -ls tmpOdd - @$(RM) tmp* - @echo "\n Compatibility with Console :" - echo "Hello World 1 !" | ./lz4 | ./lz4 -d -c - echo "Hello World 2 !" | ./lz4 | ./lz4 -d | cat - echo "Hello World 3 !" | ./lz4 --no-frame-crc | ./lz4 -d -c - @echo "\n Compatibility with Append :" - ./datagen -P100 -g1M > tmp1M - cat tmp1M tmp1M > tmp2M - ./lz4 -B5 -v tmp1M tmpC - ./lz4 -d -v tmpC tmpR - ./lz4 -d -v tmpC >> tmpR - ls -ls tmp* - diff tmp2M tmpR - @$(RM) tmp* - -test-lz4-contentSize: lz4 datagen - @echo "\n ---- test original size support ----" - ./datagen -g15M > tmp - ./lz4 -v tmp | ./lz4 -t - ./lz4 -v --content-size tmp | ./lz4 -d > tmp2 - diff -s tmp tmp2 - # test large size [2-4] GB - @./datagen -g3G -P100 | ./lz4 | ./lz4 --decompress --force --sparse - tmp - @ls -ls tmp - ./lz4 --quiet --content-size tmp | ./lz4 --verbose --decompress --force --sparse - tmp2 - @ls -ls tmp2 - @$(RM) tmp* - -test-lz4-frame-concatenation: lz4 datagen - @echo "\n ---- test frame concatenation ----" - @echo -n > empty.test - @echo hi > nonempty.test - cat nonempty.test empty.test nonempty.test > orig.test - @./lz4 -zq empty.test > empty.lz4.test - @./lz4 -zq nonempty.test > nonempty.lz4.test - cat nonempty.lz4.test empty.lz4.test nonempty.lz4.test > concat.lz4.test - ./lz4 -d concat.lz4.test > result.test - sdiff orig.test result.test - @$(RM) *.test - @echo frame concatenation test completed - -test-lz4-multiple: lz4 datagen - @echo "\n ---- test multiple files ----" - @./datagen -s1 > tmp1 2> $(VOID) - @./datagen -s2 -g100K > tmp2 2> $(VOID) - @./datagen -s3 -g1M > tmp3 2> $(VOID) - ./lz4 -f -m tmp* - ls -ls tmp* - @$(RM) tmp1 tmp2 tmp3 - ./lz4 -df -m *.lz4 - ls -ls tmp* - ./lz4 -f -m tmp1 notHere tmp2; echo $$? - @$(RM) tmp* - -test-lz4-basic: lz4 datagen - @echo "\n ---- test lz4 basic compression/decompression ----" - ./datagen -g0 | ./lz4 -v | ./lz4 -t - ./datagen -g16KB | ./lz4 -9 | ./lz4 -t - ./datagen -g20KB > tmpSrc - ./lz4 < tmpSrc | ./lz4 -d > tmpRes - diff -q tmpSrc tmpRes - ./lz4 --no-frame-crc < tmpSrc | ./lz4 -d > tmpRes - diff -q tmpSrc tmpRes - ./datagen | ./lz4 | ./lz4 -t - ./datagen -g6M -P99 | ./lz4 -9BD | ./lz4 -t - ./datagen -g17M | ./lz4 -9v | ./lz4 -qt - ./datagen -g33M | ./lz4 --no-frame-crc | ./lz4 -t - ./datagen -g256MB | ./lz4 -vqB4D | ./lz4 -t - ./datagen -g6GB | ./lz4 -vqB5D | ./lz4 -qt - ./datagen -g6GB | ./lz4 -vq9BD | ./lz4 -qt - @$(RM) tmp* - -test-lz4: lz4 datagen test-lz4-basic test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation - @echo "\n ---- test pass-through ----" - ./datagen | ./lz4 -tf - -test-lz4c: lz4c datagen - @echo "\n ---- test lz4c version ----" - ./datagen -g256MB | ./lz4c -l -v | ./lz4c -t - -test-interop-32-64: lz4 lz4c32 datagen - @echo "\n ---- test interoperability 32-bits -vs- 64 bits ----" - ./datagen -g16KB | ./lz4c32 -9 | ./lz4 -t - ./datagen -P10 | ./lz4 -9B4 | ./lz4c32 -t - ./datagen | ./lz4c32 | ./lz4 -t - ./datagen -g1M | ./lz4 -3B5 | ./lz4c32 -t - ./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4 -qt - ./datagen -g1G -P90 | ./lz4 | ./lz4c32 -t - ./datagen -g6GB | ./lz4c32 -vq9BD | ./lz4 -qt - -test-lz4c32-basic: lz4c32 datagen - @echo "\n ---- test lz4c32 32-bits version ----" - ./datagen -g16KB | ./lz4c32 -9 | ./lz4c32 -t - ./datagen | ./lz4c32 | ./lz4c32 -t - ./datagen -g256MB | ./lz4c32 -vqB4D | ./lz4c32 -qt - ./datagen -g6GB | ./lz4c32 -vqB5D | ./lz4c32 -qt - -test-lz4c32: test-lz4c32-basic test-interop-32-64 - -test-fullbench: fullbench - ./fullbench --no-prompt $(NB_LOOPS) $(TEST_FILES) - -test-fullbench32: fullbench32 - ./fullbench32 --no-prompt $(NB_LOOPS) $(TEST_FILES) - -test-fuzzer: fuzzer - ./fuzzer $(FUZZER_TIME) - -test-fuzzer32: fuzzer32 - ./fuzzer32 $(FUZZER_TIME) - -test-frametest: frametest - ./frametest $(FUZZER_TIME) - -test-frametest32: frametest32 - ./frametest32 $(FUZZER_TIME) - -test-mem: lz4 datagen fuzzer frametest fullbench - @echo "\n ---- valgrind tests : memory analyzer ----" - valgrind --leak-check=yes --error-exitcode=1 ./datagen -g50M > $(VOID) - ./datagen -g16KB > tmp - valgrind --leak-check=yes --error-exitcode=1 ./lz4 -9 -BD -f tmp $(VOID) - ./datagen -g16KB -s2 > tmp2 - ./datagen -g16KB -s3 > tmp3 - valgrind --leak-check=yes --error-exitcode=1 ./lz4 --force --multiple tmp tmp2 tmp3 - ./datagen -g16MB > tmp - valgrind --leak-check=yes --error-exitcode=1 ./lz4 -9 -B5D -f tmp tmp2 - valgrind --leak-check=yes --error-exitcode=1 ./lz4 -t tmp2 - valgrind --leak-check=yes --error-exitcode=1 ./lz4 -bi1 tmp - valgrind --leak-check=yes --error-exitcode=1 ./fullbench -i1 tmp tmp2 - ./datagen -g256MB > tmp - valgrind --leak-check=yes --error-exitcode=1 ./lz4 -B4D -f -vq tmp $(VOID) - $(RM) tmp* - valgrind --leak-check=yes --error-exitcode=1 ./fuzzer -i64 -t1 - valgrind --leak-check=yes --error-exitcode=1 ./frametest -i256 - -test-mem32: lz4c32 datagen -# unfortunately, valgrind doesn't seem to work with non-native binary... - endif diff --git a/programs/datagencli.c b/programs/datagencli.c deleted file mode 100644 index 2fbcc50..0000000 --- a/programs/datagencli.c +++ /dev/null @@ -1,190 +0,0 @@ -/* - datagencli.c - compressible data command line generator - Copyright (C) Yann Collet 2012-2015 - - GPL v2 License - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License along - with this program; if not, write to the Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 - - Public forum : https://groups.google.com/forum/#!forum/lz4c -*/ - -/************************************** -* Includes -**************************************/ -#include /* fprintf, stderr */ -#include "datagen.h" /* RDG_generate */ -#include "lz4.h" /* LZ4_VERSION_STRING */ - - -/************************************** -* Basic Types -**************************************/ -#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ -# include - typedef uint8_t BYTE; - typedef uint16_t U16; - typedef uint32_t U32; - typedef int32_t S32; - typedef uint64_t U64; -#else - typedef unsigned char BYTE; - typedef unsigned short U16; - typedef unsigned int U32; - typedef signed int S32; - typedef unsigned long long U64; -#endif - - -/************************************** -* Constants -**************************************/ -#define KB *(1 <<10) -#define MB *(1 <<20) -#define GB *(1U<<30) - -#define SIZE_DEFAULT (64 KB) -#define SEED_DEFAULT 0 -#define COMPRESSIBILITY_DEFAULT 50 - - -/************************************** -* Macros -**************************************/ -#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) -#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); } -static unsigned displayLevel = 2; - - -/********************************************************* -* Command line -*********************************************************/ -static int usage(char* programName) -{ - DISPLAY( "Compressible data generator\n"); - DISPLAY( "Usage :\n"); - DISPLAY( " %s [size] [args]\n", programName); - DISPLAY( "\n"); - DISPLAY( "Arguments :\n"); - DISPLAY( " -g# : generate # data (default:%i)\n", SIZE_DEFAULT); - DISPLAY( " -s# : Select seed (default:%i)\n", SEED_DEFAULT); - DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", COMPRESSIBILITY_DEFAULT); - DISPLAY( " -h : display help and exit\n"); - DISPLAY( "Special values :\n"); - DISPLAY( " -P0 : generate incompressible noise\n"); - DISPLAY( " -P100 : generate sparse files\n"); - return 0; -} - - -int main(int argc, char** argv) -{ - int argNb; - double proba = (double)COMPRESSIBILITY_DEFAULT / 100; - double litProba = 0.0; - U64 size = SIZE_DEFAULT; - U32 seed = SEED_DEFAULT; - char* programName; - - /* Check command line */ - programName = argv[0]; - for(argNb=1; argNb='0') && (*argument<='9')) - { - size *= 10; - size += *argument - '0'; - argument++; - } - if (*argument=='K') { size <<= 10; argument++; } - if (*argument=='M') { size <<= 20; argument++; } - if (*argument=='G') { size <<= 30; argument++; } - if (*argument=='B') { argument++; } - break; - case 's': - argument++; - seed=0; - while ((*argument>='0') && (*argument<='9')) - { - seed *= 10; - seed += *argument - '0'; - argument++; - } - break; - case 'P': - argument++; - proba=0.0; - while ((*argument>='0') && (*argument<='9')) - { - proba *= 10; - proba += *argument - '0'; - argument++; - } - if (proba>100.) proba=100.; - proba /= 100.; - break; - case 'L': /* hidden argument : Literal distribution probability */ - argument++; - litProba=0.; - while ((*argument>='0') && (*argument<='9')) - { - litProba *= 10; - litProba += *argument - '0'; - argument++; - } - if (litProba>100.) litProba=100.; - litProba /= 100.; - break; - case 'v': - displayLevel = 4; - argument++; - break; - default: - return usage(programName); - } - } - - } - } - - DISPLAYLEVEL(4, "Data Generator %s \n", LZ4_VERSION_STRING); - DISPLAYLEVEL(3, "Seed = %u \n", seed); - if (proba!=COMPRESSIBILITY_DEFAULT) DISPLAYLEVEL(3, "Compressibility : %i%%\n", (U32)(proba*100)); - - RDG_genOut(size, proba, litProba, seed); - DISPLAYLEVEL(1, "\n"); - - return 0; -} diff --git a/programs/frametest.c b/programs/frametest.c deleted file mode 100644 index 4322ab8..0000000 --- a/programs/frametest.c +++ /dev/null @@ -1,866 +0,0 @@ -/* - frameTest - test tool for lz4frame - Copyright (C) Yann Collet 2014-2016 - - GPL v2 License - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License along - with this program; if not, write to the Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - You can contact the author at : - - LZ4 homepage : http://www.lz4.org - - LZ4 source repository : https://github.com/Cyan4973/lz4 -*/ - -/*-************************************ -* Compiler specific -**************************************/ -#ifdef _MSC_VER /* Visual Studio */ -# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ -# pragma warning(disable : 4146) /* disable: C4146: minus unsigned expression */ -#endif - - -/*-************************************ -* Includes -**************************************/ -#include /* malloc, free */ -#include /* fprintf */ -#include /* strcmp */ -#include /* clock_t, clock(), CLOCKS_PER_SEC */ -#include "lz4frame_static.h" -#include "lz4.h" /* LZ4_VERSION_STRING */ -#define XXH_STATIC_LINKING_ONLY -#include "xxhash.h" /* XXH64 */ - - -/*-************************************ -* Basic Types -**************************************/ -#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ -# include -typedef uint8_t BYTE; -typedef uint16_t U16; -typedef uint32_t U32; -typedef int32_t S32; -typedef uint64_t U64; -#else -typedef unsigned char BYTE; -typedef unsigned short U16; -typedef unsigned int U32; -typedef signed int S32; -typedef unsigned long long U64; -#endif - - -/* unoptimized version; solves endianess & alignment issues */ -static void FUZ_writeLE32 (void* dstVoidPtr, U32 value32) -{ - BYTE* dstPtr = (BYTE*)dstVoidPtr; - dstPtr[0] = (BYTE)value32; - dstPtr[1] = (BYTE)(value32 >> 8); - dstPtr[2] = (BYTE)(value32 >> 16); - dstPtr[3] = (BYTE)(value32 >> 24); -} - - -/*-************************************ -* Constants -**************************************/ -#define LZ4F_MAGIC_SKIPPABLE_START 0x184D2A50U - -#define KB *(1U<<10) -#define MB *(1U<<20) -#define GB *(1U<<30) - -static const U32 nbTestsDefault = 256 KB; -#define COMPRESSIBLE_NOISE_LENGTH (2 MB) -#define FUZ_COMPRESSIBILITY_DEFAULT 50 -static const U32 prime1 = 2654435761U; -static const U32 prime2 = 2246822519U; - - - -/*-************************************ -* Macros -**************************************/ -#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) -#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); } -#define DISPLAYUPDATE(l, ...) if (displayLevel>=l) { \ - if ((FUZ_GetClockSpan(g_clockTime) > refreshRate) || (displayLevel>=4)) \ - { g_clockTime = clock(); DISPLAY(__VA_ARGS__); \ - if (displayLevel>=4) fflush(stdout); } } -static const clock_t refreshRate = CLOCKS_PER_SEC / 6; -static clock_t g_clockTime = 0; - - -/*-*************************************** -* Local Parameters -*****************************************/ -static U32 no_prompt = 0; -static char* programName; -static U32 displayLevel = 2; -static U32 pause = 0; - - -/*-******************************************************* -* Fuzzer functions -*********************************************************/ -static clock_t FUZ_GetClockSpan(clock_t clockStart) -{ - return clock() - clockStart; /* works even if overflow; max span ~ 30 mn */ -} - - -#define FUZ_rotl32(x,r) ((x << r) | (x >> (32 - r))) -unsigned int FUZ_rand(unsigned int* src) -{ - U32 rand32 = *src; - rand32 *= prime1; - rand32 += prime2; - rand32 = FUZ_rotl32(rand32, 13); - *src = rand32; - return rand32 >> 5; -} - - -#define FUZ_RAND15BITS (FUZ_rand(seed) & 0x7FFF) -#define FUZ_RANDLENGTH ( (FUZ_rand(seed) & 3) ? (FUZ_rand(seed) % 15) : (FUZ_rand(seed) % 510) + 15) -static void FUZ_fillCompressibleNoiseBuffer(void* buffer, unsigned bufferSize, double proba, U32* seed) -{ - BYTE* BBuffer = (BYTE*)buffer; - unsigned pos = 0; - U32 P32 = (U32)(32768 * proba); - - /* First Byte */ - BBuffer[pos++] = (BYTE)(FUZ_rand(seed)); - - while (pos < bufferSize) { - /* Select : Literal (noise) or copy (within 64K) */ - if (FUZ_RAND15BITS < P32) { - /* Copy (within 64K) */ - unsigned match, end; - unsigned length = FUZ_RANDLENGTH + 4; - unsigned offset = FUZ_RAND15BITS + 1; - if (offset > pos) offset = pos; - if (pos + length > bufferSize) length = bufferSize - pos; - match = pos - offset; - end = pos + length; - while (pos < end) BBuffer[pos++] = BBuffer[match++]; - } else { - /* Literal (noise) */ - unsigned end; - unsigned length = FUZ_RANDLENGTH; - if (pos + length > bufferSize) length = bufferSize - pos; - end = pos + length; - while (pos < end) BBuffer[pos++] = (BYTE)(FUZ_rand(seed) >> 5); - } - } -} - - -static unsigned FUZ_highbit(U32 v32) -{ - unsigned nbBits = 0; - if (v32==0) return 0; - while (v32) v32 >>= 1, nbBits ++; - return nbBits; -} - - -int basicTests(U32 seed, double compressibility) -{ - int testResult = 0; - void* CNBuffer; - void* compressedBuffer; - void* decodedBuffer; - U32 randState = seed; - size_t cSize, testSize; - LZ4F_preferences_t prefs; - LZ4F_decompressionContext_t dCtx = NULL; - LZ4F_compressionContext_t cctx = NULL; - U64 crcOrig; - - /* Create compressible test buffer */ - memset(&prefs, 0, sizeof(prefs)); - CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); - compressedBuffer = malloc(LZ4F_compressFrameBound(COMPRESSIBLE_NOISE_LENGTH, NULL)); - decodedBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); - FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState); - crcOrig = XXH64(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); - - /* Special case : null-content frame */ - testSize = 0; - DISPLAYLEVEL(3, "LZ4F_compressFrame, compress null content : \n"); - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, NULL), CNBuffer, testSize, NULL); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed null content into a %i bytes frame \n", (int)cSize); - - DISPLAYLEVEL(3, "LZ4F_createDecompressionContext \n"); - { LZ4F_errorCode_t const errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) goto _output_error; } - - DISPLAYLEVEL(3, "LZ4F_getFrameInfo on null-content frame (#157) \n"); - { size_t avail_in = cSize; - LZ4F_frameInfo_t frame_info; - LZ4F_errorCode_t const errorCode = LZ4F_getFrameInfo(dCtx, &frame_info, compressedBuffer, &avail_in); - if (LZ4F_isError(errorCode)) goto _output_error; - } - - DISPLAYLEVEL(3, "LZ4F_freeDecompressionContext \n"); - { LZ4F_errorCode_t const errorCode = LZ4F_freeDecompressionContext(dCtx); - if (LZ4F_isError(errorCode)) goto _output_error; } - - /* Trivial tests : one-step frame */ - testSize = COMPRESSIBLE_NOISE_LENGTH; - DISPLAYLEVEL(3, "LZ4F_compressFrame, using default preferences : \n"); - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, NULL), CNBuffer, testSize, NULL); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "Decompression test : \n"); - { size_t decodedBufferSize = COMPRESSIBLE_NOISE_LENGTH; - size_t compressedBufferSize = cSize; - BYTE* op = (BYTE*)decodedBuffer; - BYTE* const oend = (BYTE*)decodedBuffer + COMPRESSIBLE_NOISE_LENGTH; - BYTE* ip = (BYTE*)compressedBuffer; - BYTE* const iend = (BYTE*)compressedBuffer + cSize; - U64 crcDest; - - LZ4F_errorCode_t errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) goto _output_error; - - DISPLAYLEVEL(3, "Single Block : \n"); - errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &compressedBufferSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); - if (crcDest != crcOrig) goto _output_error; - DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize); - - DISPLAYLEVEL(3, "Reusing decompression context \n"); - { size_t iSize = compressedBufferSize - 4; - const BYTE* cBuff = (const BYTE*) compressedBuffer; - size_t decResult; - DISPLAYLEVEL(3, "Missing last 4 bytes : "); - decResult = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL); - if (LZ4F_isError(decResult)) goto _output_error; - if (!decResult) goto _output_error; /* not finished */ - DISPLAYLEVEL(3, "indeed, request %u bytes \n", (unsigned)decResult); - cBuff += iSize; - iSize = decResult; - decResult = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL); - if (decResult != 0) goto _output_error; /* should finish now */ - crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); - if (crcDest != crcOrig) goto _output_error; - } - - { size_t oSize = 0; - size_t iSize = 0; - LZ4F_frameInfo_t fi; - - DISPLAYLEVEL(3, "Start by feeding 0 bytes, to get next input size : "); - errorCode = LZ4F_decompress(dCtx, NULL, &oSize, ip, &iSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - DISPLAYLEVEL(3, " %u \n", (unsigned)errorCode); - - DISPLAYLEVEL(3, "get FrameInfo on null input : "); - errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); - if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error; - DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode)); - - DISPLAYLEVEL(3, "get FrameInfo on not enough input : "); - iSize = 6; - errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); - if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error; - DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode)); - ip += iSize; - - DISPLAYLEVEL(3, "get FrameInfo on enough input : "); - iSize = 15 - iSize; - errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); - if (LZ4F_isError(errorCode)) goto _output_error; - DISPLAYLEVEL(3, " correctly decoded \n"); - ip += iSize; - } - - DISPLAYLEVEL(3, "Byte after byte : \n"); - while (ip < iend) { - size_t oSize = oend-op; - size_t iSize = 1; - errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += oSize; - ip += iSize; - } - crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); - if (crcDest != crcOrig) goto _output_error; - DISPLAYLEVEL(3, "Regenerated %u/%u bytes \n", (unsigned)(op-(BYTE*)decodedBuffer), COMPRESSIBLE_NOISE_LENGTH); - - errorCode = LZ4F_freeDecompressionContext(dCtx); - if (LZ4F_isError(errorCode)) goto _output_error; - } - - DISPLAYLEVEL(3, "Using 64 KB block : \n"); - prefs.frameInfo.blockSizeID = LZ4F_max64KB; - prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "Using 256 KB block : \n"); - prefs.frameInfo.blockSizeID = LZ4F_max256KB; - prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "Decompression test : \n"); - { size_t decodedBufferSize = COMPRESSIBLE_NOISE_LENGTH; - unsigned maxBits = FUZ_highbit((U32)decodedBufferSize); - BYTE* op = (BYTE*)decodedBuffer; - BYTE* const oend = (BYTE*)decodedBuffer + COMPRESSIBLE_NOISE_LENGTH; - BYTE* ip = (BYTE*)compressedBuffer; - BYTE* const iend = (BYTE*)compressedBuffer + cSize; - U64 crcDest; - - LZ4F_errorCode_t errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) goto _output_error; - - DISPLAYLEVEL(3, "random segment sizes : \n"); - while (ip < iend) { - unsigned nbBits = FUZ_rand(&randState) % maxBits; - size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; - //DISPLAY("%7i : + %6i\n", (int)(ip-(BYTE*)compressedBuffer), (int)iSize); - errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += oSize; - ip += iSize; - } - crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); - if (crcDest != crcOrig) goto _output_error; - DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize); - - errorCode = LZ4F_freeDecompressionContext(dCtx); - if (LZ4F_isError(errorCode)) goto _output_error; - } - - DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "Using 1 MB block : \n"); - prefs.frameInfo.blockSizeID = LZ4F_max1MB; - prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "Using 4 MB block : \n"); - prefs.frameInfo.blockSizeID = LZ4F_max4MB; - prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - DISPLAYLEVEL(3, "without checksum : \n"); - prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; - cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); - if (LZ4F_isError(cSize)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); - - { size_t errorCode; - BYTE* const ostart = (BYTE*)compressedBuffer; - BYTE* op = ostart; - errorCode = LZ4F_createCompressionContext(&cctx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) goto _output_error; - - DISPLAYLEVEL(3, "compress without frameSize : \n"); - memset(&(prefs.frameInfo), 0, sizeof(prefs.frameInfo)); - errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs); - if (LZ4F_isError(errorCode)) goto _output_error; - op += errorCode; - errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += errorCode; - errorCode = LZ4F_compressEnd(cctx, compressedBuffer, testSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)(op-ostart)); - - DISPLAYLEVEL(3, "compress with frameSize : \n"); - prefs.frameInfo.contentSize = testSize; - op = ostart; - errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs); - if (LZ4F_isError(errorCode)) goto _output_error; - op += errorCode; - errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += errorCode; - errorCode = LZ4F_compressEnd(cctx, compressedBuffer, testSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)(op-ostart)); - - DISPLAYLEVEL(3, "compress with wrong frameSize : \n"); - prefs.frameInfo.contentSize = testSize+1; - op = ostart; - errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs); - if (LZ4F_isError(errorCode)) goto _output_error; - op += errorCode; - errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += errorCode; - errorCode = LZ4F_compressEnd(cctx, op, testSize, NULL); - if (LZ4F_isError(errorCode)) { DISPLAYLEVEL(3, "Error correctly detected : %s \n", LZ4F_getErrorName(errorCode)); } - else - goto _output_error; - - errorCode = LZ4F_freeCompressionContext(cctx); - if (LZ4F_isError(errorCode)) goto _output_error; - cctx = NULL; - } - - DISPLAYLEVEL(3, "Skippable frame test : \n"); - { size_t decodedBufferSize = COMPRESSIBLE_NOISE_LENGTH; - unsigned maxBits = FUZ_highbit((U32)decodedBufferSize); - BYTE* op = (BYTE*)decodedBuffer; - BYTE* const oend = (BYTE*)decodedBuffer + COMPRESSIBLE_NOISE_LENGTH; - BYTE* ip = (BYTE*)compressedBuffer; - BYTE* iend = (BYTE*)compressedBuffer + cSize + 8; - - LZ4F_errorCode_t errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) goto _output_error; - - /* generate skippable frame */ - FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START); - FUZ_writeLE32(ip+4, (U32)cSize); - - DISPLAYLEVEL(3, "random segment sizes : \n"); - while (ip < iend) { - unsigned nbBits = FUZ_rand(&randState) % maxBits; - size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; - errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += oSize; - ip += iSize; - } - DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)decodedBufferSize); - - /* generate zero-size skippable frame */ - DISPLAYLEVEL(3, "zero-size skippable frame\n"); - ip = (BYTE*)compressedBuffer; - op = (BYTE*)decodedBuffer; - FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START+1); - FUZ_writeLE32(ip+4, 0); - iend = ip+8; - - while (ip < iend) { - unsigned nbBits = FUZ_rand(&randState) % maxBits; - size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; - errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += oSize; - ip += iSize; - } - DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)(ip - (BYTE*)compressedBuffer - 8)); - - DISPLAYLEVEL(3, "Skippable frame header complete in first call \n"); - ip = (BYTE*)compressedBuffer; - op = (BYTE*)decodedBuffer; - FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START+2); - FUZ_writeLE32(ip+4, 10); - iend = ip+18; - while (ip < iend) { - size_t iSize = 10; - size_t oSize = 10; - if (iSize > (size_t)(iend-ip)) iSize = iend-ip; - errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); - if (LZ4F_isError(errorCode)) goto _output_error; - op += oSize; - ip += iSize; - } - DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)(ip - (BYTE*)compressedBuffer - 8)); - } - - DISPLAY("Basic tests completed \n"); -_end: - free(CNBuffer); - free(compressedBuffer); - free(decodedBuffer); - LZ4F_freeDecompressionContext(dCtx); dCtx = NULL; - LZ4F_freeCompressionContext(cctx); cctx = NULL; - return testResult; - -_output_error: - testResult = 1; - DISPLAY("Error detected ! \n"); - goto _end; -} - - -static void locateBuffDiff(const void* buff1, const void* buff2, size_t size, unsigned nonContiguous) -{ - int p=0; - const BYTE* b1=(const BYTE*)buff1; - const BYTE* b2=(const BYTE*)buff2; - if (nonContiguous) { - DISPLAY("Non-contiguous output test (%i bytes)\n", (int)size); - return; - } - while (b1[p]==b2[p]) p++; - DISPLAY("Error at pos %i/%i : %02X != %02X \n", p, (int)size, b1[p], b2[p]); -} - - -static const U32 srcDataLength = 9 MB; /* needs to be > 2x4MB to test large blocks */ - -int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressibility, U32 duration_s) -{ - unsigned testResult = 0; - unsigned testNb = 0; - void* srcBuffer = NULL; - void* compressedBuffer = NULL; - void* decodedBuffer = NULL; - U32 coreRand = seed; - LZ4F_decompressionContext_t dCtx = NULL; - LZ4F_compressionContext_t cCtx = NULL; - size_t result; - clock_t const startClock = clock(); - clock_t const clockDuration = duration_s * CLOCKS_PER_SEC; - XXH64_state_t xxh64; -# define CHECK(cond, ...) if (cond) { DISPLAY("Error => "); DISPLAY(__VA_ARGS__); \ - DISPLAY(" (seed %u, test nb %u) \n", seed, testNb); goto _output_error; } - - /* Create buffers */ - result = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); - CHECK(LZ4F_isError(result), "Allocation failed (error %i)", (int)result); - result = LZ4F_createCompressionContext(&cCtx, LZ4F_VERSION); - CHECK(LZ4F_isError(result), "Allocation failed (error %i)", (int)result); - srcBuffer = malloc(srcDataLength); - CHECK(srcBuffer==NULL, "srcBuffer Allocation failed"); - compressedBuffer = malloc(LZ4F_compressFrameBound(srcDataLength, NULL)); - CHECK(compressedBuffer==NULL, "compressedBuffer Allocation failed"); - decodedBuffer = calloc(1, srcDataLength); /* calloc avoids decodedBuffer being considered "garbage" by scan-build */ - CHECK(decodedBuffer==NULL, "decodedBuffer Allocation failed"); - FUZ_fillCompressibleNoiseBuffer(srcBuffer, srcDataLength, compressibility, &coreRand); - - /* jump to requested testNb */ - for (testNb =0; (testNb < startTest); testNb++) (void)FUZ_rand(&coreRand); // sync randomizer - - /* main fuzzer test loop */ - for ( ; (testNb < nbTests) || (clockDuration > FUZ_GetClockSpan(startClock)) ; testNb++) { - U32 randState = coreRand ^ prime1; - unsigned BSId = 4 + (FUZ_rand(&randState) & 3); - unsigned BMId = FUZ_rand(&randState) & 1; - unsigned CCflag = FUZ_rand(&randState) & 1; - unsigned autoflush = (FUZ_rand(&randState) & 7) == 2; - LZ4F_preferences_t prefs; - LZ4F_compressOptions_t cOptions; - LZ4F_decompressOptions_t dOptions; - unsigned nbBits = (FUZ_rand(&randState) % (FUZ_highbit(srcDataLength-1) - 1)) + 1; - size_t srcSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; - cOptions.stableSrc = ((FUZ_rand(&randState) & 3) == 1); - - result = LZ4F_compressUpdate(cCtx, op, oSize, ip, iSize, &cOptions); - CHECK(LZ4F_isError(result), "Compression failed (error %i)", (int)result); - op += result; - ip += iSize; - - if (forceFlush) { - result = LZ4F_flush(cCtx, op, oend-op, &cOptions); - CHECK(LZ4F_isError(result), "Compression failed (error %i)", (int)result); - op += result; - } - } - result = LZ4F_compressEnd(cCtx, op, oend-op, &cOptions); - CHECK(LZ4F_isError(result), "Compression completion failed (error %i)", (int)result); - op += result; - cSize = op-(BYTE*)compressedBuffer; - } - - { const BYTE* ip = (const BYTE*)compressedBuffer; - const BYTE* const iend = ip + cSize; - BYTE* op = (BYTE*)decodedBuffer; - BYTE* const oend = op + srcDataLength; - size_t totalOut = 0; - unsigned maxBits = FUZ_highbit((U32)cSize); - unsigned nonContiguousDst = (FUZ_rand(&randState) & 3) == 1; /* 0 : contiguous; 1 : non-contiguous; 2 : dst overwritten */ - nonContiguousDst += FUZ_rand(&randState) & nonContiguousDst; /* 0=>0; 1=>1,2 */ - XXH64_reset(&xxh64, 1); - if (maxBits < 3) maxBits = 3; - while (ip < iend) { - unsigned const nbBitsI = (FUZ_rand(&randState) % (maxBits-1)) + 1; - unsigned const nbBitsO = (FUZ_rand(&randState) % (maxBits)) + 1; - size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; - if (oSize > (size_t)(oend-op)) oSize = oend-op; - dOptions.stableDst = FUZ_rand(&randState) & 1; - if (nonContiguousDst==2) dOptions.stableDst = 0; - result = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, &dOptions); - if (result == (size_t)-LZ4F_ERROR_contentChecksum_invalid) - locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst); - CHECK(LZ4F_isError(result), "Decompression failed (error %i:%s)", (int)result, LZ4F_getErrorName((LZ4F_errorCode_t)result)); - XXH64_update(&xxh64, op, (U32)oSize); - totalOut += oSize; - op += oSize; - ip += iSize; - op += nonContiguousDst; - if (nonContiguousDst==2) op = (BYTE*)decodedBuffer; /* overwritten destination */ - } - CHECK(result != 0, "Frame decompression failed (error %i)", (int)result); - if (totalOut) { /* otherwise, it's a skippable frame */ - U64 const crcDecoded = XXH64_digest(&xxh64); - if (crcDecoded != crcOrig) locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst); - CHECK(crcDecoded != crcOrig, "Decompression corruption"); - } - } - } - - DISPLAYLEVEL(2, "\rAll tests completed \n"); - -_end: - LZ4F_freeDecompressionContext(dCtx); - LZ4F_freeCompressionContext(cCtx); - free(srcBuffer); - free(compressedBuffer); - free(decodedBuffer); - - if (pause) { - DISPLAY("press enter to finish \n"); - (void)getchar(); - } - return testResult; - -_output_error: - testResult = 1; - goto _end; -} - - -int FUZ_usage(void) -{ - DISPLAY( "Usage :\n"); - DISPLAY( " %s [args]\n", programName); - DISPLAY( "\n"); - DISPLAY( "Arguments :\n"); - DISPLAY( " -i# : Nb of tests (default:%u) \n", nbTestsDefault); - DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n"); - DISPLAY( " -s# : Select seed (default:prompt user)\n"); - DISPLAY( " -t# : Select starting test number (default:0)\n"); - DISPLAY( " -p# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); - DISPLAY( " -v : verbose\n"); - DISPLAY( " -h : display help and exit\n"); - return 0; -} - - -int main(int argc, char** argv) -{ - U32 seed=0; - int seedset=0; - int argNb; - int nbTests = nbTestsDefault; - int testNb = 0; - int proba = FUZ_COMPRESSIBILITY_DEFAULT; - int result=0; - U32 duration=0; - - /* Check command line */ - programName = argv[0]; - for(argNb=1; argNb='0') && (*argument<='9')) { - nbTests *= 10; - nbTests += *argument - '0'; - argument++; - } - break; - - case 'T': - argument++; - nbTests = 0; duration = 0; - for (;;) { - switch(*argument) - { - case 'm': duration *= 60; argument++; continue; - case 's': - case 'n': argument++; continue; - case '0': - case '1': - case '2': - case '3': - case '4': - case '5': - case '6': - case '7': - case '8': - case '9': duration *= 10; duration += *argument++ - '0'; continue; - } - break; - } - break; - - case 's': - argument++; - seed=0; - seedset=1; - while ((*argument>='0') && (*argument<='9')) { - seed *= 10; - seed += *argument - '0'; - argument++; - } - break; - case 't': - argument++; - testNb=0; - while ((*argument>='0') && (*argument<='9')) { - testNb *= 10; - testNb += *argument - '0'; - argument++; - } - break; - case 'P': /* compressibility % */ - argument++; - proba=0; - while ((*argument>='0') && (*argument<='9')) { - proba *= 10; - proba += *argument - '0'; - argument++; - } - if (proba<0) proba=0; - if (proba>100) proba=100; - break; - default: - ; - return FUZ_usage(); - } - } - } - } - - /* Get Seed */ - printf("Starting lz4frame tester (%i-bits, %s)\n", (int)(sizeof(size_t)*8), LZ4_VERSION_STRING); - - if (!seedset) { - time_t const t = time(NULL); - U32 const h = XXH32(&t, sizeof(t), 1); - seed = h % 10000; - } - printf("Seed = %u\n", seed); - if (proba!=FUZ_COMPRESSIBILITY_DEFAULT) printf("Compressibility : %i%%\n", proba); - - if (nbTests<=0) nbTests=1; - - if (testNb==0) result = basicTests(seed, ((double)proba) / 100); - if (result) return 1; - return fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100, duration); -} diff --git a/programs/fullbench.c b/programs/fullbench.c deleted file mode 100644 index ab6bba1..0000000 --- a/programs/fullbench.c +++ /dev/null @@ -1,813 +0,0 @@ -/* - bench.c - Demo program to benchmark open-source compression algorithm - Copyright (C) Yann Collet 2012-2015 - - GPL v2 License - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License along - with this program; if not, write to the Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 - - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c -*/ - -/************************************** -* Compiler Options -**************************************/ -/* Disable some Visual warning messages */ -#define _CRT_SECURE_NO_WARNINGS -#define _CRT_SECURE_NO_DEPRECATE /* VS2005 */ - -/* Unix Large Files support (>4GB) */ -#if (defined(__sun__) && (!defined(__LP64__))) // Sun Solaris 32-bits requires specific definitions -# define _LARGEFILE_SOURCE -# define _FILE_OFFSET_BITS 64 -#elif ! defined(__LP64__) // No point defining Large file for 64 bit -# define _LARGEFILE64_SOURCE -#endif - -// S_ISREG & gettimeofday() are not supported by MSVC -#if defined(_MSC_VER) || defined(_WIN32) -# define BMK_LEGACY_TIMER 1 -#endif - - -/************************************** -* Includes -**************************************/ -#include /* malloc, free */ -#include /* fprintf, fopen, ftello64 */ -#include /* stat64 */ -#include /* stat64 */ -#include /* strcmp */ -#include /* clock_t, clock(), CLOCKS_PER_SEC */ - -#include "lz4.h" -#include "lz4hc.h" -#include "lz4frame.h" - -#include "xxhash.h" - - -/************************************** -* Compiler Options -**************************************/ -/* S_ISREG & gettimeofday() are not supported by MSVC */ -#if !defined(S_ISREG) -# define S_ISREG(x) (((x) & S_IFMT) == S_IFREG) -#endif - - -/************************************** -* Basic Types -**************************************/ -#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ -# include - typedef uint8_t BYTE; - typedef uint16_t U16; - typedef uint32_t U32; - typedef int32_t S32; - typedef uint64_t U64; -#else - typedef unsigned char BYTE; - typedef unsigned short U16; - typedef unsigned int U32; - typedef signed int S32; - typedef unsigned long long U64; -#endif - - -/************************************** -* Constants -**************************************/ -#define PROGRAM_DESCRIPTION "LZ4 speed analyzer" -#define AUTHOR "Yann Collet" -#define WELCOME_MESSAGE "*** %s v%s %i-bits, by %s ***\n", PROGRAM_DESCRIPTION, LZ4_VERSION_STRING, (int)(sizeof(void*)*8), AUTHOR - -#define NBLOOPS 6 -#define TIMELOOP (CLOCKS_PER_SEC * 25 / 10) - -#define KB *(1 <<10) -#define MB *(1 <<20) -#define GB *(1U<<30) - -#define KNUTH 2654435761U -#define MAX_MEM (1920 MB) -#define DEFAULT_CHUNKSIZE (4 MB) - -#define ALL_COMPRESSORS 0 -#define ALL_DECOMPRESSORS 0 - - -/************************************** -* Local structures -**************************************/ -struct chunkParameters -{ - U32 id; - char* origBuffer; - char* compressedBuffer; - int origSize; - int compressedSize; -}; - - -/************************************** -* Macros -**************************************/ -#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) -#define PROGRESS(...) g_noPrompt ? 0 : DISPLAY(__VA_ARGS__) - - -/************************************** -* Benchmark Parameters -**************************************/ -static int g_chunkSize = DEFAULT_CHUNKSIZE; -static int g_nbIterations = NBLOOPS; -static int g_pause = 0; -static int g_compressionTest = 1; -static int g_compressionAlgo = ALL_COMPRESSORS; -static int g_decompressionTest = 1; -static int g_decompressionAlgo = ALL_DECOMPRESSORS; -static int g_noPrompt = 0; - -static void BMK_setBlocksize(int bsize) -{ - g_chunkSize = bsize; - DISPLAY("-Using Block Size of %i KB-\n", g_chunkSize>>10); -} - -static void BMK_setNbIterations(int nbLoops) -{ - g_nbIterations = nbLoops; - DISPLAY("- %i iterations -\n", g_nbIterations); -} - -static void BMK_setPause(void) -{ - g_pause = 1; -} - - -/********************************************************* -* Private functions -*********************************************************/ -static clock_t BMK_GetClockSpan( clock_t clockStart ) -{ - return clock() - clockStart; /* works even if overflow; max span ~30 mn */ -} - - -static size_t BMK_findMaxMem(U64 requiredMem) -{ - size_t step = 64 MB; - BYTE* testmem=NULL; - - requiredMem = (((requiredMem >> 26) + 1) << 26); - requiredMem += 2*step; - if (requiredMem > MAX_MEM) requiredMem = MAX_MEM; - - while (!testmem) { - if (requiredMem > step) requiredMem -= step; - else requiredMem >>= 1; - testmem = (BYTE*) malloc ((size_t)requiredMem); - } - free (testmem); - - /* keep some space available */ - if (requiredMem > step) requiredMem -= step; - else requiredMem >>= 1; - - return (size_t)requiredMem; -} - - -static U64 BMK_GetFileSize(const char* infilename) -{ - int r; -#if defined(_MSC_VER) - struct _stat64 statbuf; - r = _stat64(infilename, &statbuf); -#else - struct stat statbuf; - r = stat(infilename, &statbuf); -#endif - if (r || !S_ISREG(statbuf.st_mode)) return 0; /* No good... */ - return (U64)statbuf.st_size; -} - - -/********************************************************* -* Benchmark function -*********************************************************/ -static LZ4_stream_t LZ4_stream; -static void local_LZ4_resetDictT(void) -{ - LZ4_resetStream(&LZ4_stream); -} - -static void local_LZ4_createStream(void) -{ - LZ4_resetStream(&LZ4_stream); -} - -static int local_LZ4_saveDict(const char* in, char* out, int inSize) -{ - (void)in; - return LZ4_saveDict(&LZ4_stream, out, inSize); -} - -static int local_LZ4_compress_limitedOutput(const char* in, char* out, int inSize) -{ - return LZ4_compress_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); -} - -static int local_LZ4_compress_default_large(const char* in, char* out, int inSize) -{ - return LZ4_compress_default(in, out, inSize, LZ4_compressBound(inSize)); -} - -static int local_LZ4_compress_default_small(const char* in, char* out, int inSize) -{ - return LZ4_compress_default(in, out, inSize, LZ4_compressBound(inSize)-1); -} - -static int local_LZ4_compress_fast0(const char* in, char* out, int inSize) -{ - return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 0); -} - -static int local_LZ4_compress_fast1(const char* in, char* out, int inSize) -{ - return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 1); -} - -static int local_LZ4_compress_fast2(const char* in, char* out, int inSize) -{ - return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 2); -} - -static int local_LZ4_compress_fast17(const char* in, char* out, int inSize) -{ - return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 17); -} - -static int local_LZ4_compress_fast_extState0(const char* in, char* out, int inSize) -{ - return LZ4_compress_fast_extState(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize), 0); -} - -static int local_LZ4_compress_fast_continue0(const char* in, char* out, int inSize) -{ - return LZ4_compress_fast_continue(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize), 0); -} - -static int local_LZ4_compress_withState(const char* in, char* out, int inSize) -{ - return LZ4_compress_withState(&LZ4_stream, in, out, inSize); -} - -static int local_LZ4_compress_limitedOutput_withState(const char* in, char* out, int inSize) -{ - return LZ4_compress_limitedOutput_withState(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize)-1); -} - -static int local_LZ4_compress_continue(const char* in, char* out, int inSize) -{ - return LZ4_compress_continue(&LZ4_stream, in, out, inSize); -} - -static int local_LZ4_compress_limitedOutput_continue(const char* in, char* out, int inSize) -{ - return LZ4_compress_limitedOutput_continue(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize)-1); -} - -/* declare hidden function */ -int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize); - -static int local_LZ4_compress_forceDict(const char* in, char* out, int inSize) -{ - return LZ4_compress_forceExtDict(&LZ4_stream, in, out, inSize); -} - - -/* HC compression functions */ -LZ4_streamHC_t LZ4_streamHC; -static void local_LZ4_resetStreamHC(void) -{ - LZ4_resetStreamHC(&LZ4_streamHC, 0); -} - -static int local_LZ4_saveDictHC(const char* in, char* out, int inSize) -{ - (void)in; - return LZ4_saveDictHC(&LZ4_streamHC, out, inSize); -} - -static int local_LZ4_compressHC_withStateHC(const char* in, char* out, int inSize) -{ - return LZ4_compressHC_withStateHC(&LZ4_streamHC, in, out, inSize); -} - -static int local_LZ4_compressHC_limitedOutput_withStateHC(const char* in, char* out, int inSize) -{ - return LZ4_compressHC_limitedOutput_withStateHC(&LZ4_streamHC, in, out, inSize, LZ4_compressBound(inSize)-1); -} - -static int local_LZ4_compressHC_limitedOutput(const char* in, char* out, int inSize) -{ - return LZ4_compressHC_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); -} - -static int local_LZ4_compressHC_continue(const char* in, char* out, int inSize) -{ - return LZ4_compressHC_continue(&LZ4_streamHC, in, out, inSize); -} - -static int local_LZ4_compressHC_limitedOutput_continue(const char* in, char* out, int inSize) -{ - return LZ4_compressHC_limitedOutput_continue(&LZ4_streamHC, in, out, inSize, LZ4_compressBound(inSize)-1); -} - - -/* decompression functions */ -static int local_LZ4_decompress_fast(const char* in, char* out, int inSize, int outSize) -{ - (void)inSize; - LZ4_decompress_fast(in, out, outSize); - return outSize; -} - -static int local_LZ4_decompress_fast_usingDict(const char* in, char* out, int inSize, int outSize) -{ - (void)inSize; - LZ4_decompress_fast_usingDict(in, out, outSize, out - 65536, 65536); - return outSize; -} - -static int local_LZ4_decompress_safe_usingDict(const char* in, char* out, int inSize, int outSize) -{ - (void)inSize; - LZ4_decompress_safe_usingDict(in, out, inSize, outSize, out - 65536, 65536); - return outSize; -} - -extern int LZ4_decompress_safe_forceExtDict(const char* in, char* out, int inSize, int outSize, const char* dict, int dictSize); - -static int local_LZ4_decompress_safe_forceExtDict(const char* in, char* out, int inSize, int outSize) -{ - (void)inSize; - LZ4_decompress_safe_forceExtDict(in, out, inSize, outSize, out - 65536, 65536); - return outSize; -} - -static int local_LZ4_decompress_safe_partial(const char* in, char* out, int inSize, int outSize) -{ - return LZ4_decompress_safe_partial(in, out, inSize, outSize - 5, outSize); -} - - -/* frame functions */ -static int local_LZ4F_compressFrame(const char* in, char* out, int inSize) -{ - return (int)LZ4F_compressFrame(out, 2*inSize + 16, in, inSize, NULL); -} - -static LZ4F_decompressionContext_t g_dCtx; - -static int local_LZ4F_decompress(const char* in, char* out, int inSize, int outSize) -{ - size_t srcSize = inSize; - size_t dstSize = outSize; - size_t result; - result = LZ4F_decompress(g_dCtx, out, &dstSize, in, &srcSize, NULL); - if (result!=0) { DISPLAY("Error decompressing frame : unfinished frame\n"); exit(8); } - if (srcSize != (size_t)inSize) { DISPLAY("Error decompressing frame : read size incorrect\n"); exit(9); } - return (int)dstSize; -} - - -#define NB_COMPRESSION_ALGORITHMS 100 -#define NB_DECOMPRESSION_ALGORITHMS 100 -int fullSpeedBench(const char** fileNamesTable, int nbFiles) -{ - int fileIdx=0; - - /* Init */ - { size_t const errorCode = LZ4F_createDecompressionContext(&g_dCtx, LZ4F_VERSION); - if (LZ4F_isError(errorCode)) { DISPLAY("dctx allocation issue \n"); return 10; } } - - /* Loop for each fileName */ - while (fileIdx inFileSize) benchedSize = (size_t)inFileSize; - if (benchedSize < inFileSize) - DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); - - /* Allocation */ - chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)g_chunkSize)+1) * sizeof(struct chunkParameters)); - orig_buff = (char*) malloc(benchedSize); - nbChunks = (int) ((benchedSize + (g_chunkSize-1)) / g_chunkSize); - maxCompressedChunkSize = LZ4_compressBound(g_chunkSize); - compressedBuffSize = nbChunks * maxCompressedChunkSize; - compressed_buff = (char*)malloc((size_t)compressedBuffSize); - if(!chunkP || !orig_buff || !compressed_buff) { - DISPLAY("\nError: not enough memory!\n"); - fclose(inFile); - free(orig_buff); - free(compressed_buff); - free(chunkP); - return(12); - } - - /* Fill in src buffer */ - DISPLAY("Loading %s... \r", inFileName); - readSize = fread(orig_buff, 1, benchedSize, inFile); - fclose(inFile); - - if (readSize != benchedSize) { - DISPLAY("\nError: problem reading file '%s' !! \n", inFileName); - free(orig_buff); - free(compressed_buff); - free(chunkP); - return 13; - } - - /* Calculating input Checksum */ - crcOriginal = XXH32(orig_buff, benchedSize,0); - - - /* Bench */ - { int loopNb, nb_loops, chunkNb, cAlgNb, dAlgNb; - size_t cSize=0; - double ratio=0.; - - DISPLAY("\r%79s\r", ""); - DISPLAY(" %s : \n", inFileName); - - /* Bench Compression Algorithms */ - for (cAlgNb=0; (cAlgNb <= NB_COMPRESSION_ALGORITHMS) && (g_compressionTest); cAlgNb++) { - const char* compressorName; - int (*compressionFunction)(const char*, char*, int); - void (*initFunction)(void) = NULL; - double bestTime = 100000000.; - - /* filter compressionAlgo only */ - if ((g_compressionAlgo != ALL_COMPRESSORS) && (g_compressionAlgo != cAlgNb)) continue; - - /* Init data chunks */ - { int i; - size_t remaining = benchedSize; - char* in = orig_buff; - char* out = compressed_buff; - nbChunks = (int) (((int)benchedSize + (g_chunkSize-1))/ g_chunkSize); - for (i=0; i g_chunkSize) { chunkP[i].origSize = g_chunkSize; remaining -= g_chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } - chunkP[i].compressedBuffer = out; out += maxCompressedChunkSize; - chunkP[i].compressedSize = 0; - } - } - - switch(cAlgNb) - { - case 0 : DISPLAY("Compression functions : \n"); continue; - case 1 : compressionFunction = local_LZ4_compress_default_large; compressorName = "LZ4_compress_default"; break; - case 2 : compressionFunction = local_LZ4_compress_default_small; compressorName = "LZ4_compress_default(small dst)"; break; - case 3 : compressionFunction = local_LZ4_compress_fast0; compressorName = "LZ4_compress_fast(0)"; break; - case 4 : compressionFunction = local_LZ4_compress_fast1; compressorName = "LZ4_compress_fast(1)"; break; - case 5 : compressionFunction = local_LZ4_compress_fast2; compressorName = "LZ4_compress_fast(2)"; break; - case 6 : compressionFunction = local_LZ4_compress_fast17; compressorName = "LZ4_compress_fast(17)"; break; - case 7 : compressionFunction = local_LZ4_compress_fast_extState0; compressorName = "LZ4_compress_fast_extState(0)"; break; - case 8 : compressionFunction = local_LZ4_compress_fast_continue0; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_fast_continue(0)"; break; - - case 10: compressionFunction = LZ4_compressHC; compressorName = "LZ4_compressHC"; break; - case 11: compressionFunction = local_LZ4_compressHC_limitedOutput; compressorName = "LZ4_compressHC_limitedOutput"; break; - case 12 : compressionFunction = local_LZ4_compressHC_withStateHC; compressorName = "LZ4_compressHC_withStateHC"; break; - case 13: compressionFunction = local_LZ4_compressHC_limitedOutput_withStateHC; compressorName = "LZ4_compressHC_limitedOutput_withStateHC"; break; - case 14: compressionFunction = local_LZ4_compressHC_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_continue"; break; - case 15: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; - case 20: compressionFunction = local_LZ4_compress_forceDict; initFunction = local_LZ4_resetDictT; compressorName = "LZ4_compress_forceDict"; break; - case 30: compressionFunction = local_LZ4F_compressFrame; compressorName = "LZ4F_compressFrame"; - chunkP[0].origSize = (int)benchedSize; nbChunks=1; - break; - case 40: compressionFunction = local_LZ4_saveDict; compressorName = "LZ4_saveDict"; - LZ4_loadDict(&LZ4_stream, chunkP[0].origBuffer, chunkP[0].origSize); - break; - case 41: compressionFunction = local_LZ4_saveDictHC; compressorName = "LZ4_saveDictHC"; - LZ4_loadDictHC(&LZ4_streamHC, chunkP[0].origBuffer, chunkP[0].origSize); - break; - case 60: DISPLAY("Obsolete compression functions : \n"); continue; - case 61: compressionFunction = LZ4_compress; compressorName = "LZ4_compress"; break; - case 62: compressionFunction = local_LZ4_compress_limitedOutput; compressorName = "LZ4_compress_limitedOutput"; break; - case 63: compressionFunction = local_LZ4_compress_withState; compressorName = "LZ4_compress_withState"; break; - case 64: compressionFunction = local_LZ4_compress_limitedOutput_withState; compressorName = "LZ4_compress_limitedOutput_withState"; break; - case 65: compressionFunction = local_LZ4_compress_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_continue"; break; - case 66: compressionFunction = local_LZ4_compress_limitedOutput_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_limitedOutput_continue"; break; - default : - continue; /* unknown ID : just skip */ - } - - for (loopNb = 1; loopNb <= g_nbIterations; loopNb++) { - double averageTime; - clock_t clockTime; - - PROGRESS("%1i- %-28.28s :%9i ->\r", loopNb, compressorName, (int)benchedSize); - { size_t i; for (i=0; i%9i (%5.2f%%),%7.1f MB/s\r", loopNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 1000000); - } - - if (ratio<100.) - DISPLAY("%2i-%-28.28s :%9i ->%9i (%5.2f%%),%7.1f MB/s\n", cAlgNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 1000000); - else - DISPLAY("%2i-%-28.28s :%9i ->%9i (%5.1f%%),%7.1f MB/s\n", cAlgNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 100000); - } - - /* Prepare layout for decompression */ - /* Init data chunks */ - { int i; - size_t remaining = benchedSize; - char* in = orig_buff; - char* out = compressed_buff; - - nbChunks = (int) (((int)benchedSize + (g_chunkSize-1))/ g_chunkSize); - for (i=0; i g_chunkSize) { chunkP[i].origSize = g_chunkSize; remaining -= g_chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } - chunkP[i].compressedBuffer = out; out += maxCompressedChunkSize; - chunkP[i].compressedSize = 0; - } - } - for (chunkNb=0; chunkNb\r", loopNb, dName, (int)benchedSize); - - nb_loops = 0; - clockTime = clock(); - while(clock() == clockTime); - clockTime = clock(); - while(BMK_GetClockSpan(clockTime) < TIMELOOP) { - for (chunkNb=0; chunkNb %7.1f MB/s\r", loopNb, dName, (int)benchedSize, (double)benchedSize / bestTime / 1000000); - - /* CRC Checking */ - crcDecoded = XXH32(orig_buff, (int)benchedSize, 0); - if (crcOriginal!=crcDecoded) { DISPLAY("\n!!! WARNING !!! %14s : Invalid Checksum : %x != %x\n", inFileName, (unsigned)crcOriginal, (unsigned)crcDecoded); exit(1); } - } - - DISPLAY("%2i-%-29.29s :%10i -> %7.1f MB/s\n", dAlgNb, dName, (int)benchedSize, (double)benchedSize / bestTime / 1000000); - } - } - free(orig_buff); - free(compressed_buff); - free(chunkP); - } - - LZ4F_freeDecompressionContext(g_dCtx); - if (g_pause) { printf("press enter...\n"); (void)getchar(); } - - return 0; -} - - -static int usage(const char* exename) -{ - DISPLAY( "Usage :\n"); - DISPLAY( " %s [arg] file1 file2 ... fileX\n", exename); - DISPLAY( "Arguments :\n"); - DISPLAY( " -c : compression tests only\n"); - DISPLAY( " -d : decompression tests only\n"); - DISPLAY( " -H/-h : Help (this text + advanced options)\n"); - return 0; -} - -static int usage_advanced(void) -{ - DISPLAY( "\nAdvanced options :\n"); - DISPLAY( " -c# : test only compression function # [1-%i]\n", NB_COMPRESSION_ALGORITHMS); - DISPLAY( " -d# : test only decompression function # [1-%i]\n", NB_DECOMPRESSION_ALGORITHMS); - DISPLAY( " -i# : iteration loops [1-9](default : %i)\n", NBLOOPS); - DISPLAY( " -B# : Block size [4-7](default : 7)\n"); - return 0; -} - -static int badusage(const char* exename) -{ - DISPLAY("Wrong parameters\n"); - usage(exename); - return 0; -} - -int main(int argc, const char** argv) -{ - int i, - filenamesStart=2; - const char* exename = argv[0]; - const char* input_filename=0; - - // Welcome message - DISPLAY(WELCOME_MESSAGE); - - if (argc<2) { badusage(exename); return 1; } - - for(i=1; i= '0') && (argument[1]<= '9')) { - g_compressionAlgo *= 10; - g_compressionAlgo += argument[1] - '0'; - argument++; - } - break; - - // Select decompression algorithm only - case 'd': - g_compressionTest = 0; - while ((argument[1]>= '0') && (argument[1]<= '9')) { - g_decompressionAlgo *= 10; - g_decompressionAlgo += argument[1] - '0'; - argument++; - } - break; - - // Display help on usage - case 'h' : - case 'H': usage(exename); usage_advanced(); return 0; - - // Modify Block Properties - case 'B': - while (argument[1]!=0) - switch(argument[1]) - { - case '4': - case '5': - case '6': - case '7': - { int B = argument[1] - '0'; - int S = 1 << (8 + 2*B); - BMK_setBlocksize(S); - argument++; - break; - } - case 'D': argument++; break; - default : goto _exit_blockProperties; - } -_exit_blockProperties: - break; - - // Modify Nb Iterations - case 'i': - if ((argument[1] >='0') && (argument[1] <='9')) { - int iters = argument[1] - '0'; - BMK_setNbIterations(iters); - argument++; - } - break; - - // Pause at the end (hidden option) - case 'p': BMK_setPause(); break; - - // Unknown command - default : badusage(exename); return 1; - } - } - continue; - } - - // first provided filename is input - if (!input_filename) { input_filename=argument; filenamesStart=i; continue; } - - } - - // No input filename ==> Error - if(!input_filename) { badusage(exename); return 1; } - - return fullSpeedBench(argv+filenamesStart, argc-filenamesStart); - -} diff --git a/programs/fuzzer.c b/programs/fuzzer.c deleted file mode 100644 index db7ad07..0000000 --- a/programs/fuzzer.c +++ /dev/null @@ -1,1169 +0,0 @@ -/* - fuzzer.c - Fuzzer test tool for LZ4 - Copyright (C) Yann Collet 2012-2016 - - GPL v2 License - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License along - with this program; if not, write to the Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - - You can contact the author at : - - LZ4 homepage : http://www.lz4.org - - LZ4 source repo : https://github.com/Cyan4973/lz4 -*/ - -/*-************************************ -* Compiler options -**************************************/ -#ifdef _MSC_VER /* Visual Studio */ -# define _CRT_SECURE_NO_WARNINGS /* fgets */ -# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ -# pragma warning(disable : 4146) /* disable: C4146: minus unsigned expression */ -# pragma warning(disable : 4310) /* disable: C4310: constant char value > 127 */ -#endif - -/* S_ISREG & gettimeofday() are not supported by MSVC */ -#if defined(_MSC_VER) || defined(_WIN32) -# define FUZ_LEGACY_TIMER 1 -#endif - - -/*-************************************ -* Includes -**************************************/ -#include -#include /* fgets, sscanf */ -#include /* strcmp */ -#include /* clock_t, clock, CLOCKS_PER_SEC */ -#include "lz4.h" /* LZ4_VERSION_STRING */ -#include "lz4hc.h" -#define XXH_STATIC_LINKING_ONLY -#include "xxhash.h" - - -/*-************************************ -* Basic Types -**************************************/ -#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ -# include -typedef uint8_t BYTE; -typedef uint16_t U16; -typedef uint32_t U32; -typedef int32_t S32; -typedef uint64_t U64; -#else -typedef unsigned char BYTE; -typedef unsigned short U16; -typedef unsigned int U32; -typedef signed int S32; -typedef unsigned long long U64; -#endif - - -/*-************************************ -* Constants -**************************************/ -#define NB_ATTEMPTS (1<<16) -#define COMPRESSIBLE_NOISE_LENGTH (1 << 21) -#define FUZ_MAX_BLOCK_SIZE (1 << 17) -#define FUZ_MAX_DICT_SIZE (1 << 15) -#define FUZ_COMPRESSIBILITY_DEFAULT 60 -#define PRIME1 2654435761U -#define PRIME2 2246822519U -#define PRIME3 3266489917U - -#define KB *(1U<<10) -#define MB *(1U<<20) -#define GB *(1U<<30) - - -/*-*************************************** -* Macros -*****************************************/ -#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) -#define DISPLAYLEVEL(l, ...) if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); } -static int g_displayLevel = 2; -static const clock_t g_refreshRate = CLOCKS_PER_SEC * 25 / 100; -static clock_t g_time = 0; - - -/*-******************************************************* -* Fuzzer functions -*********************************************************/ -static clock_t FUZ_GetClockSpan(clock_t clockStart) -{ - return clock() - clockStart; /* works even if overflow; max span ~ 30mn */ -} - -static U32 FUZ_rotl32(U32 u32, U32 nbBits) -{ - return ((u32 << nbBits) | (u32 >> (32 - nbBits))); -} - -static U32 FUZ_rand(U32* src) -{ - U32 rand32 = *src; - rand32 *= PRIME1; - rand32 ^= PRIME2; - rand32 = FUZ_rotl32(rand32, 13); - *src = rand32; - return rand32; -} - - -#define FUZ_RAND15BITS ((FUZ_rand(seed) >> 3) & 32767) -#define FUZ_RANDLENGTH ( ((FUZ_rand(seed) >> 7) & 3) ? (FUZ_rand(seed) % 15) : (FUZ_rand(seed) % 510) + 15) -static void FUZ_fillCompressibleNoiseBuffer(void* buffer, size_t bufferSize, double proba, U32* seed) -{ - BYTE* BBuffer = (BYTE*)buffer; - size_t pos = 0; - U32 P32 = (U32)(32768 * proba); - - /* First Bytes */ - while (pos < 20) - BBuffer[pos++] = (BYTE)(FUZ_rand(seed)); - - while (pos < bufferSize) - { - /* Select : Literal (noise) or copy (within 64K) */ - if (FUZ_RAND15BITS < P32) - { - /* Copy (within 64K) */ - size_t match, d; - size_t length = FUZ_RANDLENGTH + 4; - size_t offset = FUZ_RAND15BITS + 1; - while (offset > pos) offset >>= 1; - d = pos + length; - while (d > bufferSize) d = bufferSize; - match = pos - offset; - while (pos < d) BBuffer[pos++] = BBuffer[match++]; - } - else - { - /* Literal (noise) */ - size_t d; - size_t length = FUZ_RANDLENGTH; - d = pos + length; - if (d > bufferSize) d = bufferSize; - while (pos < d) BBuffer[pos++] = (BYTE)(FUZ_rand(seed) >> 5); - } - } -} - - -#define MAX_NB_BUFF_I134 150 -#define BLOCKSIZE_I134 (32 MB) -/*! FUZ_AddressOverflow() : -* Aggressively pushes memory allocation limits, -* and generates patterns which create address space overflow. -* only possible in 32-bits mode */ -static int FUZ_AddressOverflow(void) -{ - char* buffers[MAX_NB_BUFF_I134+1]; - int i, nbBuff=0; - int highAddress = 0; - - DISPLAY("Overflow tests : "); - - /* Only possible in 32-bits */ - if (sizeof(void*)==8) - { - DISPLAY("64 bits mode : no overflow \n"); - fflush(stdout); - return 0; - } - - buffers[0] = (char*)malloc(BLOCKSIZE_I134); - buffers[1] = (char*)malloc(BLOCKSIZE_I134); - if ((!buffers[0]) || (!buffers[1])) { - DISPLAY("not enough memory for tests \n"); - return 0; - } - - for (nbBuff=2; nbBuff < MAX_NB_BUFF_I134; nbBuff++) { - DISPLAY("%3i \b\b\b\b", nbBuff); - buffers[nbBuff] = (char*)malloc(BLOCKSIZE_I134); - if (buffers[nbBuff]==NULL) goto _endOfTests; - - if (((size_t)buffers[nbBuff] > (size_t)0x80000000) && (!highAddress)) { - DISPLAY("high address detected : "); - fflush(stdout); - highAddress=1; - } - - { size_t const sizeToGenerateOverflow = (size_t)(- ((size_t)buffers[nbBuff-1]) + 512); - int const nbOf255 = (int)((sizeToGenerateOverflow / 255) + 1); - char* const input = buffers[nbBuff-1]; - char* output = buffers[nbBuff]; - int r; - input[0] = (char)0xF0; /* Literal length overflow */ - input[1] = (char)0xFF; - input[2] = (char)0xFF; - input[3] = (char)0xFF; - for(i = 4; i <= nbOf255+4; i++) input[i] = (char)0xff; - r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); - if (r>0) goto _overflowError; - input[0] = (char)0x1F; /* Match length overflow */ - input[1] = (char)0x01; - input[2] = (char)0x01; - input[3] = (char)0x00; - r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); - if (r>0) goto _overflowError; - - output = buffers[nbBuff-2]; /* Reverse in/out pointer order */ - input[0] = (char)0xF0; /* Literal length overflow */ - input[1] = (char)0xFF; - input[2] = (char)0xFF; - input[3] = (char)0xFF; - r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); - if (r>0) goto _overflowError; - input[0] = (char)0x1F; /* Match length overflow */ - input[1] = (char)0x01; - input[2] = (char)0x01; - input[3] = (char)0x00; - r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); - if (r>0) goto _overflowError; - } - } - - nbBuff++; -_endOfTests: - for (i=0 ; i g_refreshRate) | (g_displayLevel>=3)) { - g_time = clock(); - DISPLAY("\r%5u ", testNb); - if (g_displayLevel>=3) fflush(stdout); - } -} - - -/*! FUZ_findDiff() : -* find the first different byte between buff1 and buff2. -* presumes buff1 != buff2. -* presumes a difference exists before end of either buffer. -* Typically invoked after a checksum mismatch. -*/ -static void FUZ_findDiff(const void* buff1, const void* buff2) -{ - const BYTE* const b1 = (const BYTE*)buff1; - const BYTE* const b2 = (const BYTE*)buff2; - size_t i=0; - while (b1[i]==b2[i]) i++; - DISPLAY("Wrong Byte at position %u\n", (unsigned)i); -} - - -static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double compressibility, U32 duration_s) -{ - unsigned long long bytes = 0; - unsigned long long cbytes = 0; - unsigned long long hcbytes = 0; - unsigned long long ccbytes = 0; - void* CNBuffer; - char* compressedBuffer; - char* decodedBuffer; -# define FUZ_max LZ4_COMPRESSBOUND(LEN) - int ret; - unsigned cycleNb; -# define FUZ_CHECKTEST(cond, ...) if (cond) { printf("Test %u : ", testNb); printf(__VA_ARGS__); \ - printf(" (seed %u, cycle %u) \n", seed, cycleNb); goto _output_error; } -# define FUZ_DISPLAYTEST { testNb++; g_displayLevel<3 ? 0 : printf("%2u\b\b", testNb); if (g_displayLevel==4) fflush(stdout); } - void* stateLZ4 = malloc(LZ4_sizeofState()); - void* stateLZ4HC = malloc(LZ4_sizeofStateHC()); - LZ4_stream_t LZ4dict; - LZ4_streamHC_t LZ4dictHC; - U32 crcOrig, crcCheck; - U32 coreRandState = seed; - U32 randState = coreRandState ^ PRIME3; - int result = 0; - clock_t const clockStart = clock(); - clock_t const clockDuration = (clock_t)duration_s * CLOCKS_PER_SEC; - - - /* init */ - memset(&LZ4dict, 0, sizeof(LZ4dict)); - - /* Create compressible test buffer */ - CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); - FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState); - compressedBuffer = (char*)malloc(LZ4_compressBound(FUZ_MAX_BLOCK_SIZE)); - decodedBuffer = (char*)malloc(FUZ_MAX_DICT_SIZE + FUZ_MAX_BLOCK_SIZE); - - /* move to startCycle */ - for (cycleNb = 0; cycleNb < startCycle; cycleNb++) { - (void)FUZ_rand(&coreRandState); - - if (0) { /* some problems can be related to dictionary re-use; in this case, enable this loop */ - int dictSize, blockSize, blockStart; - char* dict; - char* block; - FUZ_displayUpdate(cycleNb); - randState = coreRandState ^ PRIME3; - blockSize = FUZ_rand(&randState) % FUZ_MAX_BLOCK_SIZE; - blockStart = FUZ_rand(&randState) % (COMPRESSIBLE_NOISE_LENGTH - blockSize); - dictSize = FUZ_rand(&randState) % FUZ_MAX_DICT_SIZE; - if (dictSize > blockStart) dictSize = blockStart; - block = ((char*)CNBuffer) + blockStart; - dict = block - dictSize; - LZ4_loadDict(&LZ4dict, dict, dictSize); - LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); - LZ4_loadDict(&LZ4dict, dict, dictSize); - LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); - LZ4_loadDict(&LZ4dict, dict, dictSize); - LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); - } } - - /* Main test loop */ - for (cycleNb = startCycle; (cycleNb < nbCycles) || (FUZ_GetClockSpan(clockStart) < clockDuration) ; cycleNb++) { - U32 testNb = 0; - char* dict; - char* block; - int dictSize, blockSize, blockStart, compressedSize, HCcompressedSize; - int blockContinueCompressedSize; - - FUZ_displayUpdate(cycleNb); - (void)FUZ_rand(&coreRandState); - randState = coreRandState ^ PRIME3; - - /* Select block to test */ - blockSize = (FUZ_rand(&randState) % (FUZ_MAX_BLOCK_SIZE-1)) + 1; - blockStart = FUZ_rand(&randState) % (COMPRESSIBLE_NOISE_LENGTH - blockSize); - dictSize = FUZ_rand(&randState) % FUZ_MAX_DICT_SIZE; - if (dictSize > blockStart) dictSize = blockStart; - block = ((char*)CNBuffer) + blockStart; - dict = block - dictSize; - - /* Compression tests */ - - /* Test compression destSize */ - FUZ_DISPLAYTEST; - { int srcSize = blockSize; - int targetSize = srcSize * ((FUZ_rand(&randState) & 127)+1) >> 7; - char endCheck = FUZ_rand(&randState) & 255; - compressedBuffer[targetSize] = endCheck; - ret = LZ4_compress_destSize(block, compressedBuffer, &srcSize, targetSize); - FUZ_CHECKTEST(ret > targetSize, "LZ4_compress_destSize() result larger than dst buffer !"); - FUZ_CHECKTEST(compressedBuffer[targetSize] != endCheck, "LZ4_compress_destSize() overwrite dst buffer !"); - FUZ_CHECKTEST(srcSize > blockSize, "LZ4_compress_destSize() fed more than src buffer !"); - DISPLAYLEVEL(5, "destSize : %7i/%7i; content%7i/%7i ", ret, targetSize, srcSize, blockSize); - if (targetSize>0) { - FUZ_CHECKTEST((ret==0), "LZ4_compress_destSize() compression failed"); - /* check correctness */ - FUZ_DISPLAYTEST; - - crcOrig = XXH32(block, srcSize, 0); - compressedSize = ret; - endCheck = FUZ_rand(&randState) & 255; - decodedBuffer[srcSize] = endCheck; - ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, srcSize); - FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe() failed on data compressed by LZ4_compress_destSize"); - FUZ_CHECKTEST(ret!=srcSize, "LZ4_decompress_safe() failed : did not fully decompressed data"); - FUZ_CHECKTEST(decodedBuffer[srcSize] != endCheck, "LZ4_decompress_safe() overwrite dst buffer !"); - crcCheck = XXH32(decodedBuffer, srcSize, 0); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe() corrupted decoded data"); - - DISPLAYLEVEL(5, " OK \n"); - } - else - DISPLAYLEVEL(5, " \n"); - } - - /* Test compression HC */ - FUZ_DISPLAYTEST; - ret = LZ4_compressHC(block, compressedBuffer, blockSize); - FUZ_CHECKTEST(ret==0, "LZ4_compressHC() failed"); - HCcompressedSize = ret; - - /* Test compression HC using external state */ - FUZ_DISPLAYTEST; - ret = LZ4_compressHC_withStateHC(stateLZ4HC, block, compressedBuffer, blockSize); - FUZ_CHECKTEST(ret==0, "LZ4_compressHC_withStateHC() failed"); - - /* Test compression using external state */ - FUZ_DISPLAYTEST; - ret = LZ4_compress_withState(stateLZ4, block, compressedBuffer, blockSize); - FUZ_CHECKTEST(ret==0, "LZ4_compress_withState() failed"); - - /* Test compression */ - FUZ_DISPLAYTEST; - ret = LZ4_compress(block, compressedBuffer, blockSize); - FUZ_CHECKTEST(ret==0, "LZ4_compress() failed"); - compressedSize = ret; - - /* Decompression tests */ - - crcOrig = XXH32(block, blockSize, 0); - - /* Test decoding with output size being exactly what's necessary => must work */ - FUZ_DISPLAYTEST; - ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize); - FUZ_CHECKTEST(ret<0, "LZ4_decompress_fast failed despite correct space"); - FUZ_CHECKTEST(ret!=compressedSize, "LZ4_decompress_fast failed : did not fully read compressed data"); - crcCheck = XXH32(decodedBuffer, blockSize, 0); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast corrupted decoded data"); - - /* Test decoding with one byte missing => must fail */ - FUZ_DISPLAYTEST; - decodedBuffer[blockSize-1] = 0; - ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize-1); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too small"); - FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast overrun specified output buffer"); - - /* Test decoding with one byte too much => must fail */ - FUZ_DISPLAYTEST; - ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize+1); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too large"); - - /* Test decoding with output size exactly what's necessary => must work */ - FUZ_DISPLAYTEST; - decodedBuffer[blockSize] = 0; - ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize); - FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe failed despite sufficient space"); - FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe did not regenerate original data"); - FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe overrun specified output buffer size"); - crcCheck = XXH32(decodedBuffer, blockSize, 0); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe corrupted decoded data"); - - // Test decoding with more than enough output size => must work - FUZ_DISPLAYTEST; - decodedBuffer[blockSize] = 0; - decodedBuffer[blockSize+1] = 0; - ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize+1); - FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe failed despite amply sufficient space"); - FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe did not regenerate original data"); - //FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe wrote more than (unknown) target size"); // well, is that an issue ? - FUZ_CHECKTEST(decodedBuffer[blockSize+1], "LZ4_decompress_safe overrun specified output buffer size"); - crcCheck = XXH32(decodedBuffer, blockSize, 0); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe corrupted decoded data"); - - // Test decoding with output size being one byte too short => must fail - FUZ_DISPLAYTEST; - decodedBuffer[blockSize-1] = 0; - ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize-1); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to Output Size being one byte too short"); - FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe overrun specified output buffer size"); - - // Test decoding with output size being 10 bytes too short => must fail - FUZ_DISPLAYTEST; - if (blockSize>10) - { - decodedBuffer[blockSize-10] = 0; - ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize-10); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to Output Size being 10 bytes too short"); - FUZ_CHECKTEST(decodedBuffer[blockSize-10], "LZ4_decompress_safe overrun specified output buffer size"); - } - - // Test decoding with input size being one byte too short => must fail - FUZ_DISPLAYTEST; - ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize-1, blockSize); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to input size being one byte too short (blockSize=%i, ret=%i, compressedSize=%i)", blockSize, ret, compressedSize); - - // Test decoding with input size being one byte too large => must fail - FUZ_DISPLAYTEST; - decodedBuffer[blockSize] = 0; - ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize+1, blockSize); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to input size being too large"); - FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe overrun specified output buffer size"); - - // Test partial decoding with target output size being max/2 => must work - FUZ_DISPLAYTEST; - ret = LZ4_decompress_safe_partial(compressedBuffer, decodedBuffer, compressedSize, blockSize/2, blockSize); - FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe_partial failed despite sufficient space"); - - // Test partial decoding with target output size being just below max => must work - FUZ_DISPLAYTEST; - ret = LZ4_decompress_safe_partial(compressedBuffer, decodedBuffer, compressedSize, blockSize-3, blockSize); - FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe_partial failed despite sufficient space"); - - /* Test Compression with limited output size */ - - /* Test compression with output size being exactly what's necessary (should work) */ - FUZ_DISPLAYTEST; - ret = LZ4_compress_limitedOutput(block, compressedBuffer, blockSize, compressedSize); - FUZ_CHECKTEST(ret==0, "LZ4_compress_limitedOutput() failed despite sufficient space"); - - /* Test compression with output size being exactly what's necessary and external state (should work) */ - FUZ_DISPLAYTEST; - ret = LZ4_compress_limitedOutput_withState(stateLZ4, block, compressedBuffer, blockSize, compressedSize); - FUZ_CHECKTEST(ret==0, "LZ4_compress_limitedOutput_withState() failed despite sufficient space"); - - /* Test HC compression with output size being exactly what's necessary (should work) */ - FUZ_DISPLAYTEST; - ret = LZ4_compressHC_limitedOutput(block, compressedBuffer, blockSize, HCcompressedSize); - FUZ_CHECKTEST(ret==0, "LZ4_compressHC_limitedOutput() failed despite sufficient space"); - - /* Test HC compression with output size being exactly what's necessary (should work) */ - FUZ_DISPLAYTEST; - ret = LZ4_compressHC_limitedOutput_withStateHC(stateLZ4HC, block, compressedBuffer, blockSize, HCcompressedSize); - FUZ_CHECKTEST(ret==0, "LZ4_compressHC_limitedOutput_withStateHC() failed despite sufficient space"); - - /* Test compression with missing bytes into output buffer => must fail */ - FUZ_DISPLAYTEST; - { int missingBytes = (FUZ_rand(&randState) % 0x3F) + 1; - if (missingBytes >= compressedSize) missingBytes = compressedSize-1; - missingBytes += !missingBytes; /* avoid special case missingBytes==0 */ - compressedBuffer[compressedSize-missingBytes] = 0; - ret = LZ4_compress_limitedOutput(block, compressedBuffer, blockSize, compressedSize-missingBytes); - FUZ_CHECKTEST(ret, "LZ4_compress_limitedOutput should have failed (output buffer too small by %i byte)", missingBytes); - FUZ_CHECKTEST(compressedBuffer[compressedSize-missingBytes], "LZ4_compress_limitedOutput overran output buffer ! (%i missingBytes)", missingBytes) - } - - /* Test HC compression with missing bytes into output buffer => must fail */ - FUZ_DISPLAYTEST; - { int missingBytes = (FUZ_rand(&randState) % 0x3F) + 1; - if (missingBytes >= HCcompressedSize) missingBytes = HCcompressedSize-1; - missingBytes += !missingBytes; /* avoid special case missingBytes==0 */ - compressedBuffer[HCcompressedSize-missingBytes] = 0; - ret = LZ4_compressHC_limitedOutput(block, compressedBuffer, blockSize, HCcompressedSize-missingBytes); - FUZ_CHECKTEST(ret, "LZ4_compressHC_limitedOutput should have failed (output buffer too small by %i byte)", missingBytes); - FUZ_CHECKTEST(compressedBuffer[HCcompressedSize-missingBytes], "LZ4_compressHC_limitedOutput overran output buffer ! (%i missingBytes)", missingBytes) - } - - - /*-******************/ - /* Dictionary tests */ - /*-******************/ - - /* Compress using dictionary */ - FUZ_DISPLAYTEST; - { LZ4_stream_t LZ4_stream; - LZ4_resetStream(&LZ4_stream); - LZ4_compress_continue (&LZ4_stream, dict, compressedBuffer, dictSize); /* Just to fill hash tables */ - blockContinueCompressedSize = LZ4_compress_continue (&LZ4_stream, block, compressedBuffer, blockSize); - FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed"); - } - - /* Decompress with dictionary as prefix */ - FUZ_DISPLAYTEST; - memcpy(decodedBuffer, dict, dictSize); - ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer+dictSize, blockSize, decodedBuffer, dictSize); - FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_decompress_fast_withPrefix64k did not read all compressed block input"); - crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0); - if (crcCheck!=crcOrig) { - int i=0; - while (block[i]==decodedBuffer[i]) i++; - printf("Wrong Byte at position %i/%i\n", i, blockSize); - - } - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_withPrefix64k corrupted decoded data (dict %i)", dictSize); - - FUZ_DISPLAYTEST; - ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer+dictSize, blockContinueCompressedSize, blockSize, decodedBuffer, dictSize); - FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data"); - crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); - - /* Compress using External dictionary */ - FUZ_DISPLAYTEST; - dict -= (FUZ_rand(&randState) & 0xF) + 1; /* Separation, so it is an ExtDict */ - if (dict < (char*)CNBuffer) dict = (char*)CNBuffer; - LZ4_loadDict(&LZ4dict, dict, dictSize); - blockContinueCompressedSize = LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); - FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed"); - - FUZ_DISPLAYTEST; - LZ4_loadDict(&LZ4dict, dict, dictSize); - ret = LZ4_compress_limitedOutput_continue(&LZ4dict, block, compressedBuffer, blockSize, blockContinueCompressedSize-1); - FUZ_CHECKTEST(ret>0, "LZ4_compress_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer : %i written, %i buffer", ret, blockContinueCompressedSize); - - FUZ_DISPLAYTEST; - LZ4_loadDict(&LZ4dict, dict, dictSize); - ret = LZ4_compress_limitedOutput_continue(&LZ4dict, block, compressedBuffer, blockSize, blockContinueCompressedSize); - FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_compress_limitedOutput_compressed size is different (%i != %i)", ret, blockContinueCompressedSize); - FUZ_CHECKTEST(ret<=0, "LZ4_compress_limitedOutput_continue should work : enough size available within output buffer"); - - /* Decompress with dictionary as external */ - FUZ_DISPLAYTEST; - decodedBuffer[blockSize] = 0; - ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize, dict, dictSize); - FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_decompress_fast_usingDict did not read all compressed block input"); - FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_fast_usingDict overrun specified output buffer size") - crcCheck = XXH32(decodedBuffer, blockSize, 0); - if (crcCheck!=crcOrig) - FUZ_findDiff(block, decodedBuffer); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_usingDict corrupted decoded data (dict %i)", dictSize); - - FUZ_DISPLAYTEST; - decodedBuffer[blockSize] = 0; - ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize, dict, dictSize); - FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data"); - FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe_usingDict overrun specified output buffer size") - crcCheck = XXH32(decodedBuffer, blockSize, 0); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); - - FUZ_DISPLAYTEST; - decodedBuffer[blockSize-1] = 0; - ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)"); - FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size"); - - FUZ_DISPLAYTEST; - decodedBuffer[blockSize-1] = 0; - ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-1, dict, dictSize); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : not enough output size (-1 byte)"); - FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe_usingDict overrun specified output buffer size"); - - FUZ_DISPLAYTEST; - { U32 const missingBytes = (FUZ_rand(&randState) & 0xF) + 2; - if ((U32)blockSize > missingBytes) { - decodedBuffer[blockSize-missingBytes] = 0; - ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-missingBytes, dict, dictSize); - FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : output buffer too small (-%u byte)", missingBytes); - FUZ_CHECKTEST(decodedBuffer[blockSize-missingBytes], "LZ4_decompress_safe_usingDict overrun specified output buffer size (-%u byte) (blockSize=%i)", missingBytes, blockSize); - } } - - /* Compress HC using External dictionary */ - FUZ_DISPLAYTEST; - dict -= (FUZ_rand(&randState) & 7); /* even bigger separation */ - if (dict < (char*)CNBuffer) dict = (char*)CNBuffer; - LZ4_resetStreamHC (&LZ4dictHC, FUZ_rand(&randState) & 0x7); - LZ4_loadDictHC(&LZ4dictHC, dict, dictSize); - blockContinueCompressedSize = LZ4_compressHC_continue(&LZ4dictHC, block, compressedBuffer, blockSize); - FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compressHC_continue failed"); - - FUZ_DISPLAYTEST; - LZ4_loadDictHC(&LZ4dictHC, dict, dictSize); - ret = LZ4_compressHC_limitedOutput_continue(&LZ4dictHC, block, compressedBuffer, blockSize, blockContinueCompressedSize-1); - FUZ_CHECKTEST(ret>0, "LZ4_compressHC_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer"); - - FUZ_DISPLAYTEST; - LZ4_loadDictHC(&LZ4dictHC, dict, dictSize); - ret = LZ4_compressHC_limitedOutput_continue(&LZ4dictHC, block, compressedBuffer, blockSize, blockContinueCompressedSize); - FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_compress_limitedOutput_compressed size is different (%i != %i)", ret, blockContinueCompressedSize); - FUZ_CHECKTEST(ret<=0, "LZ4_compress_limitedOutput_continue should work : enough size available within output buffer"); - - FUZ_DISPLAYTEST; - decodedBuffer[blockSize] = 0; - ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize, dict, dictSize); - FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data"); - FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe_usingDict overrun specified output buffer size") - crcCheck = XXH32(decodedBuffer, blockSize, 0); - if (crcCheck!=crcOrig) - FUZ_findDiff(block, decodedBuffer); - FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); - - /* ***** End of tests *** */ - /* Fill stats */ - bytes += blockSize; - cbytes += compressedSize; - hcbytes += HCcompressedSize; - ccbytes += blockContinueCompressedSize; - } - - if (nbCycles<=1) nbCycles = cycleNb; /* end by time */ - bytes += !bytes; /* avoid division by 0 */ - printf("\r%7u /%7u - ", cycleNb, nbCycles); - printf("all tests completed successfully \n"); - printf("compression ratio: %0.3f%%\n", (double)cbytes/bytes*100); - printf("HC compression ratio: %0.3f%%\n", (double)hcbytes/bytes*100); - printf("ratio with dict: %0.3f%%\n", (double)ccbytes/bytes*100); - - /* release memory */ - { -_exit: - free(CNBuffer); - free(compressedBuffer); - free(decodedBuffer); - free(stateLZ4); - free(stateLZ4HC); - return result; - -_output_error: - result = 1; - goto _exit; - } -} - - -#define testInputSize (192 KB) -#define testCompressedSize (128 KB) -#define ringBufferSize (8 KB) - -static void FUZ_unitTests(void) -{ - const unsigned testNb = 0; - const unsigned seed = 0; - const unsigned cycleNb= 0; - char testInput[testInputSize]; - char testCompressed[testCompressedSize]; - char testVerify[testInputSize]; - char ringBuffer[ringBufferSize]; - U32 randState = 1; - - /* Init */ - FUZ_fillCompressibleNoiseBuffer(testInput, testInputSize, 0.50, &randState); - - /* 32-bits address space overflow test */ - FUZ_AddressOverflow(); - - /* LZ4 streaming tests */ - { LZ4_stream_t* statePtr; - LZ4_stream_t streamingState; - U64 crcOrig; - U64 crcNew; - int result; - - /* Allocation test */ - statePtr = LZ4_createStream(); - FUZ_CHECKTEST(statePtr==NULL, "LZ4_createStream() allocation failed"); - LZ4_freeStream(statePtr); - - /* simple compression test */ - crcOrig = XXH64(testInput, testCompressedSize, 0); - LZ4_resetStream(&streamingState); - result = LZ4_compress_limitedOutput_continue(&streamingState, testInput, testCompressed, testCompressedSize, testCompressedSize-1); - FUZ_CHECKTEST(result==0, "LZ4_compress_limitedOutput_continue() compression failed"); - - result = LZ4_decompress_safe(testCompressed, testVerify, result, testCompressedSize); - FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe() decompression failed"); - crcNew = XXH64(testVerify, testCompressedSize, 0); - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); - - /* ring buffer test */ - { XXH64_state_t xxhOrig; - XXH64_state_t xxhNew; - LZ4_streamDecode_t decodeState; - const U32 maxMessageSizeLog = 10; - const U32 maxMessageSizeMask = (1< ringBufferSize) rNext = 0; - if (dNext + messageSize > dBufferSize) dNext = 0; - } - } - } - - /* LZ4 HC streaming tests */ - { LZ4_streamHC_t* sp; - LZ4_streamHC_t sHC; - U64 crcOrig; - U64 crcNew; - int result; - - /* Allocation test */ - sp = LZ4_createStreamHC(); - FUZ_CHECKTEST(sp==NULL, "LZ4_createStreamHC() allocation failed"); - LZ4_freeStreamHC(sp); - - /* simple HC compression test */ - crcOrig = XXH64(testInput, testCompressedSize, 0); - LZ4_resetStreamHC(&sHC, 0); - result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput, testCompressed, testCompressedSize, testCompressedSize-1); - FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() compression failed"); - - result = LZ4_decompress_safe(testCompressed, testVerify, result, testCompressedSize); - FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe() decompression failed"); - crcNew = XXH64(testVerify, testCompressedSize, 0); - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); - - /* simple dictionary HC compression test */ - crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0); - LZ4_resetStreamHC(&sHC, 0); - LZ4_loadDictHC(&sHC, testInput, 64 KB); - result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + 64 KB, testCompressed, testCompressedSize, testCompressedSize-1); - FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result); - - result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result, testCompressedSize, testInput, 64 KB); - FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe() simple dictionary decompression test failed"); - crcNew = XXH64(testVerify, testCompressedSize, 0); - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() simple dictionary decompression test : corruption"); - - /* multiple HC compression test with dictionary */ - { int result1, result2; - int segSize = testCompressedSize / 2; - crcOrig = XXH64(testInput + segSize, testCompressedSize, 0); - LZ4_resetStreamHC(&sHC, 0); - LZ4_loadDictHC(&sHC, testInput, segSize); - result1 = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + segSize, testCompressed, segSize, segSize -1); - FUZ_CHECKTEST(result1==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result1); - result2 = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + 2*segSize, testCompressed+result1, segSize, segSize-1); - FUZ_CHECKTEST(result2==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result2); - - result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result1, segSize, testInput, segSize); - FUZ_CHECKTEST(result!=segSize, "LZ4_decompress_safe() dictionary decompression part 1 failed"); - result = LZ4_decompress_safe_usingDict(testCompressed+result1, testVerify+segSize, result2, segSize, testInput, 2*segSize); - FUZ_CHECKTEST(result!=segSize, "LZ4_decompress_safe() dictionary decompression part 2 failed"); - crcNew = XXH64(testVerify, testCompressedSize, 0); - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() dictionary decompression corruption"); - } - - /* remote dictionary HC compression test */ - crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0); - LZ4_resetStreamHC(&sHC, 0); - LZ4_loadDictHC(&sHC, testInput, 32 KB); - result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + 64 KB, testCompressed, testCompressedSize, testCompressedSize-1); - FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() remote dictionary failed : result = %i", result); - - result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result, testCompressedSize, testInput, 32 KB); - FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe_usingDict() decompression failed following remote dictionary HC compression test"); - crcNew = XXH64(testVerify, testCompressedSize, 0); - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe_usingDict() decompression corruption"); - - /* multiple HC compression with ext. dictionary */ - { XXH64_state_t crcOrigState; - XXH64_state_t crcNewState; - const char* dict = testInput + 3; - int dictSize = (FUZ_rand(&randState) & 8191); - char* dst = testVerify; - - size_t segStart = dictSize + 7; - int segSize = (FUZ_rand(&randState) & 8191); - int segNb = 1; - - LZ4_resetStreamHC(&sHC, 0); - LZ4_loadDictHC(&sHC, dict, dictSize); - - XXH64_reset(&crcOrigState, 0); - XXH64_reset(&crcNewState, 0); - - while (segStart + segSize < testInputSize) { - XXH64_update(&crcOrigState, testInput + segStart, segSize); - crcOrig = XXH64_digest(&crcOrigState); - result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + segStart, testCompressed, segSize, LZ4_compressBound(segSize)); - FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result); - - result = LZ4_decompress_safe_usingDict(testCompressed, dst, result, segSize, dict, dictSize); - FUZ_CHECKTEST(result!=segSize, "LZ4_decompress_safe_usingDict() dictionary decompression part %i failed", segNb); - XXH64_update(&crcNewState, dst, segSize); - crcNew = XXH64_digest(&crcNewState); - if (crcOrig!=crcNew) { - size_t c=0; - while (dst[c] == testInput[segStart+c]) c++; - DISPLAY("Bad decompression at %u / %u \n", (U32)c, (U32)segSize); - } - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe_usingDict() part %i corruption", segNb); - - dict = dst; - //dict = testInput + segStart; - dictSize = segSize; - - dst += segSize + 1; - segNb ++; - - segStart += segSize + (FUZ_rand(&randState) & 0xF) + 1; - segSize = (FUZ_rand(&randState) & 8191); - } - } - - /* ring buffer test */ - { XXH64_state_t xxhOrig; - XXH64_state_t xxhNew; - LZ4_streamDecode_t decodeState; - const U32 maxMessageSizeLog = 10; - const U32 maxMessageSizeMask = (1< ringBufferSize) rNext = 0; - if (dNext + messageSize > dBufferSize) dNext = 0; - } - } - - /* small decoder-side ring buffer test */ - { XXH64_state_t xxhOrig; - XXH64_state_t xxhNew; - LZ4_streamDecode_t decodeState; - const U32 maxMessageSizeLog = 12; - const U32 maxMessageSizeMask = (1< dBufferSize) dNext = 0; - - while (totalMessageSize < 9 MB) { - XXH64_update(&xxhOrig, testInput + iNext, messageSize); - crcOrig = XXH64_digest(&xxhOrig); - - result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + iNext, testCompressed, messageSize, testCompressedSize-ringBufferSize); - FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() compression failed"); - - result = LZ4_decompress_safe_continue(&decodeState, testCompressed, testVerify + dNext, result, messageSize); - FUZ_CHECKTEST(result!=(int)messageSize, "64K D.ringBuffer : LZ4_decompress_safe() test failed"); - - XXH64_update(&xxhNew, testVerify + dNext, messageSize); - crcNew = XXH64_digest(&xxhNew); - if (crcOrig != crcNew) - FUZ_findDiff(testInput + iNext, testVerify + dNext); - FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption during small decoder-side ring buffer test"); - - /* prepare next message */ - dNext += messageSize; - totalMessageSize += messageSize; - messageSize = (FUZ_rand(&randState) & maxMessageSizeMask) + 1; - iNext = (FUZ_rand(&randState) & 65535); - if (dNext > dBufferSize) dNext = 0; - } - } - } - - printf("All unit tests completed successfully \n"); - return; -_output_error: - exit(1); -} - - -static int FUZ_usage(char* programName) -{ - DISPLAY( "Usage :\n"); - DISPLAY( " %s [args]\n", programName); - DISPLAY( "\n"); - DISPLAY( "Arguments :\n"); - DISPLAY( " -i# : Nb of tests (default:%i) \n", NB_ATTEMPTS); - DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n"); - DISPLAY( " -s# : Select seed (default:prompt user)\n"); - DISPLAY( " -t# : Select starting test number (default:0)\n"); - DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); - DISPLAY( " -v : verbose\n"); - DISPLAY( " -p : pause at the end\n"); - DISPLAY( " -h : display help and exit\n"); - return 0; -} - - -int main(int argc, char** argv) -{ - U32 seed=0; - int seedset=0; - int argNb; - int nbTests = NB_ATTEMPTS; - int testNb = 0; - int proba = FUZ_COMPRESSIBILITY_DEFAULT; - int pause = 0; - char* programName = argv[0]; - U32 duration = 0; - - /* Check command line */ - for(argNb=1; argNb='0') && (*argument<='9')) { - nbTests *= 10; - nbTests += *argument - '0'; - argument++; - } - break; - - case 'T': - argument++; - nbTests = 0; duration = 0; - for (;;) { - switch(*argument) - { - case 'm': duration *= 60; argument++; continue; - case 's': - case 'n': argument++; continue; - case '0': - case '1': - case '2': - case '3': - case '4': - case '5': - case '6': - case '7': - case '8': - case '9': duration *= 10; duration += *argument++ - '0'; continue; - } - break; - } - break; - - case 's': - argument++; - seed=0; seedset=1; - while ((*argument>='0') && (*argument<='9')) { - seed *= 10; - seed += *argument - '0'; - argument++; - } - break; - - case 't': /* select starting test nb */ - argument++; - testNb=0; - while ((*argument>='0') && (*argument<='9')) { - testNb *= 10; - testNb += *argument - '0'; - argument++; - } - break; - - case 'P': /* change probability */ - argument++; - proba=0; - while ((*argument>='0') && (*argument<='9')) { - proba *= 10; - proba += *argument - '0'; - argument++; - } - if (proba<0) proba=0; - if (proba>100) proba=100; - break; - default: ; - } - } - } - } - - printf("Starting LZ4 fuzzer (%i-bits, v%s)\n", (int)(sizeof(size_t)*8), LZ4_VERSION_STRING); - - if (!seedset) { - time_t const t = time(NULL); - U32 const h = XXH32(&t, sizeof(t), 1); - seed = h % 10000; - } - printf("Seed = %u\n", seed); - - if (proba!=FUZ_COMPRESSIBILITY_DEFAULT) printf("Compressibility : %i%%\n", proba); - - if ((seedset==0) && (testNb==0)) FUZ_unitTests(); - - if (nbTests<=0) nbTests=1; - - { int const result = FUZ_test(seed, nbTests, testNb, ((double)proba) / 100, duration); - if (pause) { - DISPLAY("press enter ... \n"); - (void)getchar(); - } - return result; - } -} diff --git a/tests/Makefile b/tests/Makefile new file mode 100644 index 0000000..7652476 --- /dev/null +++ b/tests/Makefile @@ -0,0 +1,280 @@ + +# ########################################################################## +# LZ4 programs - Makefile +# Copyright (C) Yann Collet 2011-2016 +# +# GPL v2 License +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# You can contact the author at : +# - LZ4 homepage : http://www.lz4.org +# - LZ4 source repository : https://github.com/Cyan4973/lz4 +# ########################################################################## +# fuzzer : Test tool, to check lz4 integrity on target platform +# fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode +# frametest : Test tool, to check lz4frame integrity on target platform +# frametest32: Same as frametest, but forced to compile in 32-bits mode +# fullbench : Precisely measure speed for each LZ4 function variant +# fullbench32: Same as fullbench, but forced to compile in 32-bits mode +# datagen : generates synthetic data samples for tests & benchmarks +# ########################################################################## + +DESTDIR ?= +PREFIX ?= /usr/local +BINDIR := $(PREFIX)/bin +MANDIR := $(PREFIX)/share/man/man1 +LZ4DIR := ../lib +PRGDIR := ../programs + +CFLAGS ?= -O3 # can select custom flags. For example : CFLAGS="-O2 -g" make +CFLAGS += -Wall -Wextra -Wundef -Wcast-qual -Wcast-align -Wshadow -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes +CFLAGS += $(MOREFLAGS) +CPPFLAGS:= -I$(LZ4DIR) -DXXH_NAMESPACE=LZ4_ -DLZ4_DLL_EXPORT=1 +FLAGS := $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) + + +# Define *.exe as extension for Windows systems +ifneq (,$(filter Windows%,$(OS))) +EXT =.exe +VOID = nul +else +EXT = +VOID = /dev/null +endif + + +# Default test parameters +TEST_FILES := COPYING +FUZZER_TIME := -T9mn + + +default: bins + +m32: fullbench32 fuzzer32 frametest32 + +bins: fullbench fuzzer frametest datagen + +all: bins m32 + + +lz4: + $(MAKE) -C $(PRGDIR) $@ + +lz4c: + $(MAKE) -C $(PRGDIR) $@ + +lz4c32: + $(MAKE) -C $(PRGDIR) $@ + +fullbench : $(LZ4DIR)/lz4.o $(LZ4DIR)/lz4hc.o $(LZ4DIR)/lz4frame.o $(LZ4DIR)/xxhash.o fullbench.c + $(CC) $(FLAGS) $^ -o $@$(EXT) + +fullbench32: $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/lz4frame.c $(LZ4DIR)/xxhash.c fullbench.c + $(CC) -m32 $(FLAGS) $^ -o $@$(EXT) + +fuzzer : $(LZ4DIR)/lz4.o $(LZ4DIR)/lz4hc.o $(LZ4DIR)/xxhash.o fuzzer.c + $(CC) $(FLAGS) $^ -o $@$(EXT) + +fuzzer32: $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/xxhash.c fuzzer.c + $(CC) -m32 $(FLAGS) $^ -o $@$(EXT) + +frametest: $(LZ4DIR)/lz4frame.o $(LZ4DIR)/lz4.o $(LZ4DIR)/lz4hc.o $(LZ4DIR)/xxhash.o frametest.c + $(CC) $(FLAGS) $^ -o $@$(EXT) + +frametest32: $(LZ4DIR)/lz4frame.c $(LZ4DIR)/lz4.c $(LZ4DIR)/lz4hc.c $(LZ4DIR)/xxhash.c frametest.c + $(CC) -m32 $(FLAGS) $^ -o $@$(EXT) + +datagen : $(PRGDIR)/datagen.c datagencli.c + $(CC) $(FLAGS) -I$(PRGDIR) $^ -o $@$(EXT) + +clean: + @$(MAKE) -C $(LZ4DIR) $@ > $(VOID) + @$(RM) core *.o *.test tmp* \ + fullbench$(EXT) fullbench32$(EXT) \ + fuzzer$(EXT) fuzzer32$(EXT) \ + frametest$(EXT) frametest32$(EXT) \ + datagen$(EXT) + @echo Cleaning completed + + +#------------------------------------------------------------------------ +#make install is validated only for Linux, OSX, kFreeBSD, Hurd and +#FreeBSD targets +ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU FreeBSD)) + +test: test-lz4 test-lz4c test-frametest test-fullbench test-fuzzer test-mem + +test32: test-lz4c32 test-frametest32 test-fullbench32 test-fuzzer32 test-mem32 + +test-all: test test32 + +test-lz4-sparse: lz4 datagen + @echo "\n ---- test sparse file support ----" + ./datagen -g5M -P100 > tmpSrc + $(PRGDIR)/lz4 -B4D tmpSrc | $(PRGDIR)/lz4 -dv --sparse > tmpB4 + diff -s tmpSrc tmpB4 + $(PRGDIR)/lz4 -B5D tmpSrc | $(PRGDIR)/lz4 -dv --sparse > tmpB5 + diff -s tmpSrc tmpB5 + $(PRGDIR)/lz4 -B6D tmpSrc | $(PRGDIR)/lz4 -dv --sparse > tmpB6 + diff -s tmpSrc tmpB6 + $(PRGDIR)/lz4 -B7D tmpSrc | $(PRGDIR)/lz4 -dv --sparse > tmpB7 + diff -s tmpSrc tmpB7 + $(PRGDIR)/lz4 tmpSrc | $(PRGDIR)/lz4 -dv --no-sparse > tmpNoSparse + diff -s tmpSrc tmpNoSparse + ls -ls tmp* + ./datagen -s1 -g1200007 -P100 | $(PRGDIR)/lz4 | $(PRGDIR)/lz4 -dv --sparse > tmpOdd # Odd size file (to generate non-full last block) + ./datagen -s1 -g1200007 -P100 | diff -s - tmpOdd + ls -ls tmpOdd + @$(RM) tmp* + @echo "\n Compatibility with Console :" + echo "Hello World 1 !" | $(PRGDIR)/lz4 | $(PRGDIR)/lz4 -d -c + echo "Hello World 2 !" | $(PRGDIR)/lz4 | $(PRGDIR)/lz4 -d | cat + echo "Hello World 3 !" | $(PRGDIR)/lz4 --no-frame-crc | $(PRGDIR)/lz4 -d -c + @echo "\n Compatibility with Append :" + ./datagen -P100 -g1M > tmp1M + cat tmp1M tmp1M > tmp2M + $(PRGDIR)/lz4 -B5 -v tmp1M tmpC + $(PRGDIR)/lz4 -d -v tmpC tmpR + $(PRGDIR)/lz4 -d -v tmpC >> tmpR + ls -ls tmp* + diff tmp2M tmpR + @$(RM) tmp* + +test-lz4-contentSize: lz4 datagen + @echo "\n ---- test original size support ----" + ./datagen -g15M > tmp + $(PRGDIR)/lz4 -v tmp | $(PRGDIR)/lz4 -t + $(PRGDIR)/lz4 -v --content-size tmp | $(PRGDIR)/lz4 -d > tmp2 + diff -s tmp tmp2 + # test large size [2-4] GB + @./datagen -g3G -P100 | $(PRGDIR)/lz4 | $(PRGDIR)/lz4 --decompress --force --sparse - tmp + @ls -ls tmp + $(PRGDIR)/lz4 --quiet --content-size tmp | $(PRGDIR)/lz4 --verbose --decompress --force --sparse - tmp2 + @ls -ls tmp2 + @$(RM) tmp* + +test-lz4-frame-concatenation: lz4 datagen + @echo "\n ---- test frame concatenation ----" + @echo -n > empty.test + @echo hi > nonempty.test + cat nonempty.test empty.test nonempty.test > orig.test + @$(PRGDIR)/lz4 -zq empty.test > empty.lz4.test + @$(PRGDIR)/lz4 -zq nonempty.test > nonempty.lz4.test + cat nonempty.lz4.test empty.lz4.test nonempty.lz4.test > concat.lz4.test + $(PRGDIR)/lz4 -d concat.lz4.test > result.test + sdiff orig.test result.test + @$(RM) *.test + @echo frame concatenation test completed + +test-lz4-multiple: lz4 datagen + @echo "\n ---- test multiple files ----" + @./datagen -s1 > tmp1 2> $(VOID) + @./datagen -s2 -g100K > tmp2 2> $(VOID) + @./datagen -s3 -g1M > tmp3 2> $(VOID) + $(PRGDIR)/lz4 -f -m tmp* + ls -ls tmp* + @$(RM) tmp1 tmp2 tmp3 + $(PRGDIR)/lz4 -df -m *.lz4 + ls -ls tmp* + $(PRGDIR)/lz4 -f -m tmp1 notHere tmp2; echo $$? + @$(RM) tmp* + +test-lz4-basic: lz4 datagen + @echo "\n ---- test lz4 basic compression/decompression ----" + ./datagen -g0 | $(PRGDIR)/lz4 -v | $(PRGDIR)/lz4 -t + ./datagen -g16KB | $(PRGDIR)/lz4 -9 | $(PRGDIR)/lz4 -t + ./datagen -g20KB > tmpSrc + $(PRGDIR)/lz4 < tmpSrc | $(PRGDIR)/lz4 -d > tmpRes + diff -q tmpSrc tmpRes + $(PRGDIR)/lz4 --no-frame-crc < tmpSrc | $(PRGDIR)/lz4 -d > tmpRes + diff -q tmpSrc tmpRes + ./datagen | $(PRGDIR)/lz4 | $(PRGDIR)/lz4 -t + ./datagen -g6M -P99 | $(PRGDIR)/lz4 -9BD | $(PRGDIR)/lz4 -t + ./datagen -g17M | $(PRGDIR)/lz4 -9v | $(PRGDIR)/lz4 -qt + ./datagen -g33M | $(PRGDIR)/lz4 --no-frame-crc | $(PRGDIR)/lz4 -t + ./datagen -g256MB | $(PRGDIR)/lz4 -vqB4D | $(PRGDIR)/lz4 -t + ./datagen -g6GB | $(PRGDIR)/lz4 -vqB5D | $(PRGDIR)/lz4 -qt + ./datagen -g6GB | $(PRGDIR)/lz4 -vq9BD | $(PRGDIR)/lz4 -qt + @$(RM) tmp* + +test-lz4: lz4 datagen test-lz4-basic test-lz4-multiple test-lz4-sparse test-lz4-contentSize test-lz4-frame-concatenation + @echo "\n ---- test pass-through ----" + ./datagen | $(PRGDIR)/lz4 -tf + +test-lz4c: lz4c datagen + @echo "\n ---- test lz4c version ----" + ./datagen -g256MB | $(PRGDIR)/lz4c -l -v | $(PRGDIR)/lz4c -t + +test-interop-32-64: lz4 lz4c32 datagen + @echo "\n ---- test interoperability 32-bits -vs- 64 bits ----" + ./datagen -g16KB | $(PRGDIR)/lz4c32 -9 | $(PRGDIR)/lz4 -t + ./datagen -P10 | $(PRGDIR)/lz4 -9B4 | $(PRGDIR)/lz4c32 -t + ./datagen | $(PRGDIR)/lz4c32 | $(PRGDIR)/lz4 -t + ./datagen -g1M | $(PRGDIR)/lz4 -3B5 | $(PRGDIR)/lz4c32 -t + ./datagen -g256MB | $(PRGDIR)/lz4c32 -vqB4D | $(PRGDIR)/lz4 -qt + ./datagen -g1G -P90 | $(PRGDIR)/lz4 | $(PRGDIR)/lz4c32 -t + ./datagen -g6GB | $(PRGDIR)/lz4c32 -vq9BD | $(PRGDIR)/lz4 -qt + +test-lz4c32-basic: lz4c32 datagen + @echo "\n ---- test lz4c32 32-bits version ----" + ./datagen -g16KB | $(PRGDIR)/lz4c32 -9 | $(PRGDIR)/lz4c32 -t + ./datagen | $(PRGDIR)/lz4c32 | $(PRGDIR)/lz4c32 -t + ./datagen -g256MB | $(PRGDIR)/lz4c32 -vqB4D | $(PRGDIR)/lz4c32 -qt + ./datagen -g6GB | $(PRGDIR)/lz4c32 -vqB5D | $(PRGDIR)/lz4c32 -qt + +test-lz4c32: test-lz4c32-basic test-interop-32-64 + +test-fullbench: fullbench + ./fullbench --no-prompt $(NB_LOOPS) $(TEST_FILES) + +test-fullbench32: fullbench32 + ./fullbench32 --no-prompt $(NB_LOOPS) $(TEST_FILES) + +test-fuzzer: fuzzer + ./fuzzer $(FUZZER_TIME) + +test-fuzzer32: fuzzer32 + ./fuzzer32 $(FUZZER_TIME) + +test-frametest: frametest + ./frametest $(FUZZER_TIME) + +test-frametest32: frametest32 + ./frametest32 $(FUZZER_TIME) + +test-mem: lz4 datagen fuzzer frametest fullbench + @echo "\n ---- valgrind tests : memory analyzer ----" + valgrind --leak-check=yes --error-exitcode=1 ./datagen -g50M > $(VOID) + ./datagen -g16KB > tmp + valgrind --leak-check=yes --error-exitcode=1 $(PRGDIR)/lz4 -9 -BD -f tmp $(VOID) + ./datagen -g16KB -s2 > tmp2 + ./datagen -g16KB -s3 > tmp3 + valgrind --leak-check=yes --error-exitcode=1 $(PRGDIR)/lz4 --force --multiple tmp tmp2 tmp3 + ./datagen -g16MB > tmp + valgrind --leak-check=yes --error-exitcode=1 $(PRGDIR)/lz4 -9 -B5D -f tmp tmp2 + valgrind --leak-check=yes --error-exitcode=1 $(PRGDIR)/lz4 -t tmp2 + valgrind --leak-check=yes --error-exitcode=1 $(PRGDIR)/lz4 -bi1 tmp + valgrind --leak-check=yes --error-exitcode=1 ./fullbench -i1 tmp tmp2 + ./datagen -g256MB > tmp + valgrind --leak-check=yes --error-exitcode=1 $(PRGDIR)/lz4 -B4D -f -vq tmp $(VOID) + $(RM) tmp* + valgrind --leak-check=yes --error-exitcode=1 ./fuzzer -i64 -t1 + valgrind --leak-check=yes --error-exitcode=1 ./frametest -i256 + +test-mem32: lz4c32 datagen +# unfortunately, valgrind doesn't seem to work with non-native binary... + +endif diff --git a/tests/datagencli.c b/tests/datagencli.c new file mode 100644 index 0000000..2fbcc50 --- /dev/null +++ b/tests/datagencli.c @@ -0,0 +1,190 @@ +/* + datagencli.c + compressible data command line generator + Copyright (C) Yann Collet 2012-2015 + + GPL v2 License + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + You can contact the author at : + - LZ4 source repository : https://github.com/Cyan4973/lz4 + - Public forum : https://groups.google.com/forum/#!forum/lz4c +*/ + +/************************************** +* Includes +**************************************/ +#include /* fprintf, stderr */ +#include "datagen.h" /* RDG_generate */ +#include "lz4.h" /* LZ4_VERSION_STRING */ + + +/************************************** +* Basic Types +**************************************/ +#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ +# include + typedef uint8_t BYTE; + typedef uint16_t U16; + typedef uint32_t U32; + typedef int32_t S32; + typedef uint64_t U64; +#else + typedef unsigned char BYTE; + typedef unsigned short U16; + typedef unsigned int U32; + typedef signed int S32; + typedef unsigned long long U64; +#endif + + +/************************************** +* Constants +**************************************/ +#define KB *(1 <<10) +#define MB *(1 <<20) +#define GB *(1U<<30) + +#define SIZE_DEFAULT (64 KB) +#define SEED_DEFAULT 0 +#define COMPRESSIBILITY_DEFAULT 50 + + +/************************************** +* Macros +**************************************/ +#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) +#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); } +static unsigned displayLevel = 2; + + +/********************************************************* +* Command line +*********************************************************/ +static int usage(char* programName) +{ + DISPLAY( "Compressible data generator\n"); + DISPLAY( "Usage :\n"); + DISPLAY( " %s [size] [args]\n", programName); + DISPLAY( "\n"); + DISPLAY( "Arguments :\n"); + DISPLAY( " -g# : generate # data (default:%i)\n", SIZE_DEFAULT); + DISPLAY( " -s# : Select seed (default:%i)\n", SEED_DEFAULT); + DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", COMPRESSIBILITY_DEFAULT); + DISPLAY( " -h : display help and exit\n"); + DISPLAY( "Special values :\n"); + DISPLAY( " -P0 : generate incompressible noise\n"); + DISPLAY( " -P100 : generate sparse files\n"); + return 0; +} + + +int main(int argc, char** argv) +{ + int argNb; + double proba = (double)COMPRESSIBILITY_DEFAULT / 100; + double litProba = 0.0; + U64 size = SIZE_DEFAULT; + U32 seed = SEED_DEFAULT; + char* programName; + + /* Check command line */ + programName = argv[0]; + for(argNb=1; argNb='0') && (*argument<='9')) + { + size *= 10; + size += *argument - '0'; + argument++; + } + if (*argument=='K') { size <<= 10; argument++; } + if (*argument=='M') { size <<= 20; argument++; } + if (*argument=='G') { size <<= 30; argument++; } + if (*argument=='B') { argument++; } + break; + case 's': + argument++; + seed=0; + while ((*argument>='0') && (*argument<='9')) + { + seed *= 10; + seed += *argument - '0'; + argument++; + } + break; + case 'P': + argument++; + proba=0.0; + while ((*argument>='0') && (*argument<='9')) + { + proba *= 10; + proba += *argument - '0'; + argument++; + } + if (proba>100.) proba=100.; + proba /= 100.; + break; + case 'L': /* hidden argument : Literal distribution probability */ + argument++; + litProba=0.; + while ((*argument>='0') && (*argument<='9')) + { + litProba *= 10; + litProba += *argument - '0'; + argument++; + } + if (litProba>100.) litProba=100.; + litProba /= 100.; + break; + case 'v': + displayLevel = 4; + argument++; + break; + default: + return usage(programName); + } + } + + } + } + + DISPLAYLEVEL(4, "Data Generator %s \n", LZ4_VERSION_STRING); + DISPLAYLEVEL(3, "Seed = %u \n", seed); + if (proba!=COMPRESSIBILITY_DEFAULT) DISPLAYLEVEL(3, "Compressibility : %i%%\n", (U32)(proba*100)); + + RDG_genOut(size, proba, litProba, seed); + DISPLAYLEVEL(1, "\n"); + + return 0; +} diff --git a/tests/frametest.c b/tests/frametest.c new file mode 100644 index 0000000..4322ab8 --- /dev/null +++ b/tests/frametest.c @@ -0,0 +1,866 @@ +/* + frameTest - test tool for lz4frame + Copyright (C) Yann Collet 2014-2016 + + GPL v2 License + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + You can contact the author at : + - LZ4 homepage : http://www.lz4.org + - LZ4 source repository : https://github.com/Cyan4973/lz4 +*/ + +/*-************************************ +* Compiler specific +**************************************/ +#ifdef _MSC_VER /* Visual Studio */ +# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ +# pragma warning(disable : 4146) /* disable: C4146: minus unsigned expression */ +#endif + + +/*-************************************ +* Includes +**************************************/ +#include /* malloc, free */ +#include /* fprintf */ +#include /* strcmp */ +#include /* clock_t, clock(), CLOCKS_PER_SEC */ +#include "lz4frame_static.h" +#include "lz4.h" /* LZ4_VERSION_STRING */ +#define XXH_STATIC_LINKING_ONLY +#include "xxhash.h" /* XXH64 */ + + +/*-************************************ +* Basic Types +**************************************/ +#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ +# include +typedef uint8_t BYTE; +typedef uint16_t U16; +typedef uint32_t U32; +typedef int32_t S32; +typedef uint64_t U64; +#else +typedef unsigned char BYTE; +typedef unsigned short U16; +typedef unsigned int U32; +typedef signed int S32; +typedef unsigned long long U64; +#endif + + +/* unoptimized version; solves endianess & alignment issues */ +static void FUZ_writeLE32 (void* dstVoidPtr, U32 value32) +{ + BYTE* dstPtr = (BYTE*)dstVoidPtr; + dstPtr[0] = (BYTE)value32; + dstPtr[1] = (BYTE)(value32 >> 8); + dstPtr[2] = (BYTE)(value32 >> 16); + dstPtr[3] = (BYTE)(value32 >> 24); +} + + +/*-************************************ +* Constants +**************************************/ +#define LZ4F_MAGIC_SKIPPABLE_START 0x184D2A50U + +#define KB *(1U<<10) +#define MB *(1U<<20) +#define GB *(1U<<30) + +static const U32 nbTestsDefault = 256 KB; +#define COMPRESSIBLE_NOISE_LENGTH (2 MB) +#define FUZ_COMPRESSIBILITY_DEFAULT 50 +static const U32 prime1 = 2654435761U; +static const U32 prime2 = 2246822519U; + + + +/*-************************************ +* Macros +**************************************/ +#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) +#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); } +#define DISPLAYUPDATE(l, ...) if (displayLevel>=l) { \ + if ((FUZ_GetClockSpan(g_clockTime) > refreshRate) || (displayLevel>=4)) \ + { g_clockTime = clock(); DISPLAY(__VA_ARGS__); \ + if (displayLevel>=4) fflush(stdout); } } +static const clock_t refreshRate = CLOCKS_PER_SEC / 6; +static clock_t g_clockTime = 0; + + +/*-*************************************** +* Local Parameters +*****************************************/ +static U32 no_prompt = 0; +static char* programName; +static U32 displayLevel = 2; +static U32 pause = 0; + + +/*-******************************************************* +* Fuzzer functions +*********************************************************/ +static clock_t FUZ_GetClockSpan(clock_t clockStart) +{ + return clock() - clockStart; /* works even if overflow; max span ~ 30 mn */ +} + + +#define FUZ_rotl32(x,r) ((x << r) | (x >> (32 - r))) +unsigned int FUZ_rand(unsigned int* src) +{ + U32 rand32 = *src; + rand32 *= prime1; + rand32 += prime2; + rand32 = FUZ_rotl32(rand32, 13); + *src = rand32; + return rand32 >> 5; +} + + +#define FUZ_RAND15BITS (FUZ_rand(seed) & 0x7FFF) +#define FUZ_RANDLENGTH ( (FUZ_rand(seed) & 3) ? (FUZ_rand(seed) % 15) : (FUZ_rand(seed) % 510) + 15) +static void FUZ_fillCompressibleNoiseBuffer(void* buffer, unsigned bufferSize, double proba, U32* seed) +{ + BYTE* BBuffer = (BYTE*)buffer; + unsigned pos = 0; + U32 P32 = (U32)(32768 * proba); + + /* First Byte */ + BBuffer[pos++] = (BYTE)(FUZ_rand(seed)); + + while (pos < bufferSize) { + /* Select : Literal (noise) or copy (within 64K) */ + if (FUZ_RAND15BITS < P32) { + /* Copy (within 64K) */ + unsigned match, end; + unsigned length = FUZ_RANDLENGTH + 4; + unsigned offset = FUZ_RAND15BITS + 1; + if (offset > pos) offset = pos; + if (pos + length > bufferSize) length = bufferSize - pos; + match = pos - offset; + end = pos + length; + while (pos < end) BBuffer[pos++] = BBuffer[match++]; + } else { + /* Literal (noise) */ + unsigned end; + unsigned length = FUZ_RANDLENGTH; + if (pos + length > bufferSize) length = bufferSize - pos; + end = pos + length; + while (pos < end) BBuffer[pos++] = (BYTE)(FUZ_rand(seed) >> 5); + } + } +} + + +static unsigned FUZ_highbit(U32 v32) +{ + unsigned nbBits = 0; + if (v32==0) return 0; + while (v32) v32 >>= 1, nbBits ++; + return nbBits; +} + + +int basicTests(U32 seed, double compressibility) +{ + int testResult = 0; + void* CNBuffer; + void* compressedBuffer; + void* decodedBuffer; + U32 randState = seed; + size_t cSize, testSize; + LZ4F_preferences_t prefs; + LZ4F_decompressionContext_t dCtx = NULL; + LZ4F_compressionContext_t cctx = NULL; + U64 crcOrig; + + /* Create compressible test buffer */ + memset(&prefs, 0, sizeof(prefs)); + CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); + compressedBuffer = malloc(LZ4F_compressFrameBound(COMPRESSIBLE_NOISE_LENGTH, NULL)); + decodedBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); + FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState); + crcOrig = XXH64(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); + + /* Special case : null-content frame */ + testSize = 0; + DISPLAYLEVEL(3, "LZ4F_compressFrame, compress null content : \n"); + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, NULL), CNBuffer, testSize, NULL); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed null content into a %i bytes frame \n", (int)cSize); + + DISPLAYLEVEL(3, "LZ4F_createDecompressionContext \n"); + { LZ4F_errorCode_t const errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) goto _output_error; } + + DISPLAYLEVEL(3, "LZ4F_getFrameInfo on null-content frame (#157) \n"); + { size_t avail_in = cSize; + LZ4F_frameInfo_t frame_info; + LZ4F_errorCode_t const errorCode = LZ4F_getFrameInfo(dCtx, &frame_info, compressedBuffer, &avail_in); + if (LZ4F_isError(errorCode)) goto _output_error; + } + + DISPLAYLEVEL(3, "LZ4F_freeDecompressionContext \n"); + { LZ4F_errorCode_t const errorCode = LZ4F_freeDecompressionContext(dCtx); + if (LZ4F_isError(errorCode)) goto _output_error; } + + /* Trivial tests : one-step frame */ + testSize = COMPRESSIBLE_NOISE_LENGTH; + DISPLAYLEVEL(3, "LZ4F_compressFrame, using default preferences : \n"); + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, NULL), CNBuffer, testSize, NULL); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "Decompression test : \n"); + { size_t decodedBufferSize = COMPRESSIBLE_NOISE_LENGTH; + size_t compressedBufferSize = cSize; + BYTE* op = (BYTE*)decodedBuffer; + BYTE* const oend = (BYTE*)decodedBuffer + COMPRESSIBLE_NOISE_LENGTH; + BYTE* ip = (BYTE*)compressedBuffer; + BYTE* const iend = (BYTE*)compressedBuffer + cSize; + U64 crcDest; + + LZ4F_errorCode_t errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) goto _output_error; + + DISPLAYLEVEL(3, "Single Block : \n"); + errorCode = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &compressedBufferSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); + if (crcDest != crcOrig) goto _output_error; + DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize); + + DISPLAYLEVEL(3, "Reusing decompression context \n"); + { size_t iSize = compressedBufferSize - 4; + const BYTE* cBuff = (const BYTE*) compressedBuffer; + size_t decResult; + DISPLAYLEVEL(3, "Missing last 4 bytes : "); + decResult = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL); + if (LZ4F_isError(decResult)) goto _output_error; + if (!decResult) goto _output_error; /* not finished */ + DISPLAYLEVEL(3, "indeed, request %u bytes \n", (unsigned)decResult); + cBuff += iSize; + iSize = decResult; + decResult = LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, cBuff, &iSize, NULL); + if (decResult != 0) goto _output_error; /* should finish now */ + crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); + if (crcDest != crcOrig) goto _output_error; + } + + { size_t oSize = 0; + size_t iSize = 0; + LZ4F_frameInfo_t fi; + + DISPLAYLEVEL(3, "Start by feeding 0 bytes, to get next input size : "); + errorCode = LZ4F_decompress(dCtx, NULL, &oSize, ip, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + DISPLAYLEVEL(3, " %u \n", (unsigned)errorCode); + + DISPLAYLEVEL(3, "get FrameInfo on null input : "); + errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); + if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error; + DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode)); + + DISPLAYLEVEL(3, "get FrameInfo on not enough input : "); + iSize = 6; + errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); + if (errorCode != (size_t)-LZ4F_ERROR_frameHeader_incomplete) goto _output_error; + DISPLAYLEVEL(3, " correctly failed : %s \n", LZ4F_getErrorName(errorCode)); + ip += iSize; + + DISPLAYLEVEL(3, "get FrameInfo on enough input : "); + iSize = 15 - iSize; + errorCode = LZ4F_getFrameInfo(dCtx, &fi, ip, &iSize); + if (LZ4F_isError(errorCode)) goto _output_error; + DISPLAYLEVEL(3, " correctly decoded \n"); + ip += iSize; + } + + DISPLAYLEVEL(3, "Byte after byte : \n"); + while (ip < iend) { + size_t oSize = oend-op; + size_t iSize = 1; + errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += oSize; + ip += iSize; + } + crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); + if (crcDest != crcOrig) goto _output_error; + DISPLAYLEVEL(3, "Regenerated %u/%u bytes \n", (unsigned)(op-(BYTE*)decodedBuffer), COMPRESSIBLE_NOISE_LENGTH); + + errorCode = LZ4F_freeDecompressionContext(dCtx); + if (LZ4F_isError(errorCode)) goto _output_error; + } + + DISPLAYLEVEL(3, "Using 64 KB block : \n"); + prefs.frameInfo.blockSizeID = LZ4F_max64KB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "without checksum : \n"); + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "Using 256 KB block : \n"); + prefs.frameInfo.blockSizeID = LZ4F_max256KB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "Decompression test : \n"); + { size_t decodedBufferSize = COMPRESSIBLE_NOISE_LENGTH; + unsigned maxBits = FUZ_highbit((U32)decodedBufferSize); + BYTE* op = (BYTE*)decodedBuffer; + BYTE* const oend = (BYTE*)decodedBuffer + COMPRESSIBLE_NOISE_LENGTH; + BYTE* ip = (BYTE*)compressedBuffer; + BYTE* const iend = (BYTE*)compressedBuffer + cSize; + U64 crcDest; + + LZ4F_errorCode_t errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) goto _output_error; + + DISPLAYLEVEL(3, "random segment sizes : \n"); + while (ip < iend) { + unsigned nbBits = FUZ_rand(&randState) % maxBits; + size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; + //DISPLAY("%7i : + %6i\n", (int)(ip-(BYTE*)compressedBuffer), (int)iSize); + errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += oSize; + ip += iSize; + } + crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1); + if (crcDest != crcOrig) goto _output_error; + DISPLAYLEVEL(3, "Regenerated %i bytes \n", (int)decodedBufferSize); + + errorCode = LZ4F_freeDecompressionContext(dCtx); + if (LZ4F_isError(errorCode)) goto _output_error; + } + + DISPLAYLEVEL(3, "without checksum : \n"); + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "Using 1 MB block : \n"); + prefs.frameInfo.blockSizeID = LZ4F_max1MB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "without checksum : \n"); + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "Using 4 MB block : \n"); + prefs.frameInfo.blockSizeID = LZ4F_max4MB; + prefs.frameInfo.contentChecksumFlag = LZ4F_contentChecksumEnabled; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + DISPLAYLEVEL(3, "without checksum : \n"); + prefs.frameInfo.contentChecksumFlag = LZ4F_noContentChecksum; + cSize = LZ4F_compressFrame(compressedBuffer, LZ4F_compressFrameBound(testSize, &prefs), CNBuffer, testSize, &prefs); + if (LZ4F_isError(cSize)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)cSize); + + { size_t errorCode; + BYTE* const ostart = (BYTE*)compressedBuffer; + BYTE* op = ostart; + errorCode = LZ4F_createCompressionContext(&cctx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) goto _output_error; + + DISPLAYLEVEL(3, "compress without frameSize : \n"); + memset(&(prefs.frameInfo), 0, sizeof(prefs.frameInfo)); + errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs); + if (LZ4F_isError(errorCode)) goto _output_error; + op += errorCode; + errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += errorCode; + errorCode = LZ4F_compressEnd(cctx, compressedBuffer, testSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)(op-ostart)); + + DISPLAYLEVEL(3, "compress with frameSize : \n"); + prefs.frameInfo.contentSize = testSize; + op = ostart; + errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs); + if (LZ4F_isError(errorCode)) goto _output_error; + op += errorCode; + errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += errorCode; + errorCode = LZ4F_compressEnd(cctx, compressedBuffer, testSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + DISPLAYLEVEL(3, "Compressed %i bytes into a %i bytes frame \n", (int)testSize, (int)(op-ostart)); + + DISPLAYLEVEL(3, "compress with wrong frameSize : \n"); + prefs.frameInfo.contentSize = testSize+1; + op = ostart; + errorCode = LZ4F_compressBegin(cctx, compressedBuffer, testSize, &prefs); + if (LZ4F_isError(errorCode)) goto _output_error; + op += errorCode; + errorCode = LZ4F_compressUpdate(cctx, op, LZ4F_compressBound(testSize, &prefs), CNBuffer, testSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += errorCode; + errorCode = LZ4F_compressEnd(cctx, op, testSize, NULL); + if (LZ4F_isError(errorCode)) { DISPLAYLEVEL(3, "Error correctly detected : %s \n", LZ4F_getErrorName(errorCode)); } + else + goto _output_error; + + errorCode = LZ4F_freeCompressionContext(cctx); + if (LZ4F_isError(errorCode)) goto _output_error; + cctx = NULL; + } + + DISPLAYLEVEL(3, "Skippable frame test : \n"); + { size_t decodedBufferSize = COMPRESSIBLE_NOISE_LENGTH; + unsigned maxBits = FUZ_highbit((U32)decodedBufferSize); + BYTE* op = (BYTE*)decodedBuffer; + BYTE* const oend = (BYTE*)decodedBuffer + COMPRESSIBLE_NOISE_LENGTH; + BYTE* ip = (BYTE*)compressedBuffer; + BYTE* iend = (BYTE*)compressedBuffer + cSize + 8; + + LZ4F_errorCode_t errorCode = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) goto _output_error; + + /* generate skippable frame */ + FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START); + FUZ_writeLE32(ip+4, (U32)cSize); + + DISPLAYLEVEL(3, "random segment sizes : \n"); + while (ip < iend) { + unsigned nbBits = FUZ_rand(&randState) % maxBits; + size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; + errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += oSize; + ip += iSize; + } + DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)decodedBufferSize); + + /* generate zero-size skippable frame */ + DISPLAYLEVEL(3, "zero-size skippable frame\n"); + ip = (BYTE*)compressedBuffer; + op = (BYTE*)decodedBuffer; + FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START+1); + FUZ_writeLE32(ip+4, 0); + iend = ip+8; + + while (ip < iend) { + unsigned nbBits = FUZ_rand(&randState) % maxBits; + size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; + errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += oSize; + ip += iSize; + } + DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)(ip - (BYTE*)compressedBuffer - 8)); + + DISPLAYLEVEL(3, "Skippable frame header complete in first call \n"); + ip = (BYTE*)compressedBuffer; + op = (BYTE*)decodedBuffer; + FUZ_writeLE32(ip, LZ4F_MAGIC_SKIPPABLE_START+2); + FUZ_writeLE32(ip+4, 10); + iend = ip+18; + while (ip < iend) { + size_t iSize = 10; + size_t oSize = 10; + if (iSize > (size_t)(iend-ip)) iSize = iend-ip; + errorCode = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, NULL); + if (LZ4F_isError(errorCode)) goto _output_error; + op += oSize; + ip += iSize; + } + DISPLAYLEVEL(3, "Skipped %i bytes \n", (int)(ip - (BYTE*)compressedBuffer - 8)); + } + + DISPLAY("Basic tests completed \n"); +_end: + free(CNBuffer); + free(compressedBuffer); + free(decodedBuffer); + LZ4F_freeDecompressionContext(dCtx); dCtx = NULL; + LZ4F_freeCompressionContext(cctx); cctx = NULL; + return testResult; + +_output_error: + testResult = 1; + DISPLAY("Error detected ! \n"); + goto _end; +} + + +static void locateBuffDiff(const void* buff1, const void* buff2, size_t size, unsigned nonContiguous) +{ + int p=0; + const BYTE* b1=(const BYTE*)buff1; + const BYTE* b2=(const BYTE*)buff2; + if (nonContiguous) { + DISPLAY("Non-contiguous output test (%i bytes)\n", (int)size); + return; + } + while (b1[p]==b2[p]) p++; + DISPLAY("Error at pos %i/%i : %02X != %02X \n", p, (int)size, b1[p], b2[p]); +} + + +static const U32 srcDataLength = 9 MB; /* needs to be > 2x4MB to test large blocks */ + +int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressibility, U32 duration_s) +{ + unsigned testResult = 0; + unsigned testNb = 0; + void* srcBuffer = NULL; + void* compressedBuffer = NULL; + void* decodedBuffer = NULL; + U32 coreRand = seed; + LZ4F_decompressionContext_t dCtx = NULL; + LZ4F_compressionContext_t cCtx = NULL; + size_t result; + clock_t const startClock = clock(); + clock_t const clockDuration = duration_s * CLOCKS_PER_SEC; + XXH64_state_t xxh64; +# define CHECK(cond, ...) if (cond) { DISPLAY("Error => "); DISPLAY(__VA_ARGS__); \ + DISPLAY(" (seed %u, test nb %u) \n", seed, testNb); goto _output_error; } + + /* Create buffers */ + result = LZ4F_createDecompressionContext(&dCtx, LZ4F_VERSION); + CHECK(LZ4F_isError(result), "Allocation failed (error %i)", (int)result); + result = LZ4F_createCompressionContext(&cCtx, LZ4F_VERSION); + CHECK(LZ4F_isError(result), "Allocation failed (error %i)", (int)result); + srcBuffer = malloc(srcDataLength); + CHECK(srcBuffer==NULL, "srcBuffer Allocation failed"); + compressedBuffer = malloc(LZ4F_compressFrameBound(srcDataLength, NULL)); + CHECK(compressedBuffer==NULL, "compressedBuffer Allocation failed"); + decodedBuffer = calloc(1, srcDataLength); /* calloc avoids decodedBuffer being considered "garbage" by scan-build */ + CHECK(decodedBuffer==NULL, "decodedBuffer Allocation failed"); + FUZ_fillCompressibleNoiseBuffer(srcBuffer, srcDataLength, compressibility, &coreRand); + + /* jump to requested testNb */ + for (testNb =0; (testNb < startTest); testNb++) (void)FUZ_rand(&coreRand); // sync randomizer + + /* main fuzzer test loop */ + for ( ; (testNb < nbTests) || (clockDuration > FUZ_GetClockSpan(startClock)) ; testNb++) { + U32 randState = coreRand ^ prime1; + unsigned BSId = 4 + (FUZ_rand(&randState) & 3); + unsigned BMId = FUZ_rand(&randState) & 1; + unsigned CCflag = FUZ_rand(&randState) & 1; + unsigned autoflush = (FUZ_rand(&randState) & 7) == 2; + LZ4F_preferences_t prefs; + LZ4F_compressOptions_t cOptions; + LZ4F_decompressOptions_t dOptions; + unsigned nbBits = (FUZ_rand(&randState) % (FUZ_highbit(srcDataLength-1) - 1)) + 1; + size_t srcSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; + cOptions.stableSrc = ((FUZ_rand(&randState) & 3) == 1); + + result = LZ4F_compressUpdate(cCtx, op, oSize, ip, iSize, &cOptions); + CHECK(LZ4F_isError(result), "Compression failed (error %i)", (int)result); + op += result; + ip += iSize; + + if (forceFlush) { + result = LZ4F_flush(cCtx, op, oend-op, &cOptions); + CHECK(LZ4F_isError(result), "Compression failed (error %i)", (int)result); + op += result; + } + } + result = LZ4F_compressEnd(cCtx, op, oend-op, &cOptions); + CHECK(LZ4F_isError(result), "Compression completion failed (error %i)", (int)result); + op += result; + cSize = op-(BYTE*)compressedBuffer; + } + + { const BYTE* ip = (const BYTE*)compressedBuffer; + const BYTE* const iend = ip + cSize; + BYTE* op = (BYTE*)decodedBuffer; + BYTE* const oend = op + srcDataLength; + size_t totalOut = 0; + unsigned maxBits = FUZ_highbit((U32)cSize); + unsigned nonContiguousDst = (FUZ_rand(&randState) & 3) == 1; /* 0 : contiguous; 1 : non-contiguous; 2 : dst overwritten */ + nonContiguousDst += FUZ_rand(&randState) & nonContiguousDst; /* 0=>0; 1=>1,2 */ + XXH64_reset(&xxh64, 1); + if (maxBits < 3) maxBits = 3; + while (ip < iend) { + unsigned const nbBitsI = (FUZ_rand(&randState) % (maxBits-1)) + 1; + unsigned const nbBitsO = (FUZ_rand(&randState) % (maxBits)) + 1; + size_t iSize = (FUZ_rand(&randState) & ((1< (size_t)(iend-ip)) iSize = iend-ip; + if (oSize > (size_t)(oend-op)) oSize = oend-op; + dOptions.stableDst = FUZ_rand(&randState) & 1; + if (nonContiguousDst==2) dOptions.stableDst = 0; + result = LZ4F_decompress(dCtx, op, &oSize, ip, &iSize, &dOptions); + if (result == (size_t)-LZ4F_ERROR_contentChecksum_invalid) + locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst); + CHECK(LZ4F_isError(result), "Decompression failed (error %i:%s)", (int)result, LZ4F_getErrorName((LZ4F_errorCode_t)result)); + XXH64_update(&xxh64, op, (U32)oSize); + totalOut += oSize; + op += oSize; + ip += iSize; + op += nonContiguousDst; + if (nonContiguousDst==2) op = (BYTE*)decodedBuffer; /* overwritten destination */ + } + CHECK(result != 0, "Frame decompression failed (error %i)", (int)result); + if (totalOut) { /* otherwise, it's a skippable frame */ + U64 const crcDecoded = XXH64_digest(&xxh64); + if (crcDecoded != crcOrig) locateBuffDiff((BYTE*)srcBuffer+srcStart, decodedBuffer, srcSize, nonContiguousDst); + CHECK(crcDecoded != crcOrig, "Decompression corruption"); + } + } + } + + DISPLAYLEVEL(2, "\rAll tests completed \n"); + +_end: + LZ4F_freeDecompressionContext(dCtx); + LZ4F_freeCompressionContext(cCtx); + free(srcBuffer); + free(compressedBuffer); + free(decodedBuffer); + + if (pause) { + DISPLAY("press enter to finish \n"); + (void)getchar(); + } + return testResult; + +_output_error: + testResult = 1; + goto _end; +} + + +int FUZ_usage(void) +{ + DISPLAY( "Usage :\n"); + DISPLAY( " %s [args]\n", programName); + DISPLAY( "\n"); + DISPLAY( "Arguments :\n"); + DISPLAY( " -i# : Nb of tests (default:%u) \n", nbTestsDefault); + DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n"); + DISPLAY( " -s# : Select seed (default:prompt user)\n"); + DISPLAY( " -t# : Select starting test number (default:0)\n"); + DISPLAY( " -p# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); + DISPLAY( " -v : verbose\n"); + DISPLAY( " -h : display help and exit\n"); + return 0; +} + + +int main(int argc, char** argv) +{ + U32 seed=0; + int seedset=0; + int argNb; + int nbTests = nbTestsDefault; + int testNb = 0; + int proba = FUZ_COMPRESSIBILITY_DEFAULT; + int result=0; + U32 duration=0; + + /* Check command line */ + programName = argv[0]; + for(argNb=1; argNb='0') && (*argument<='9')) { + nbTests *= 10; + nbTests += *argument - '0'; + argument++; + } + break; + + case 'T': + argument++; + nbTests = 0; duration = 0; + for (;;) { + switch(*argument) + { + case 'm': duration *= 60; argument++; continue; + case 's': + case 'n': argument++; continue; + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': duration *= 10; duration += *argument++ - '0'; continue; + } + break; + } + break; + + case 's': + argument++; + seed=0; + seedset=1; + while ((*argument>='0') && (*argument<='9')) { + seed *= 10; + seed += *argument - '0'; + argument++; + } + break; + case 't': + argument++; + testNb=0; + while ((*argument>='0') && (*argument<='9')) { + testNb *= 10; + testNb += *argument - '0'; + argument++; + } + break; + case 'P': /* compressibility % */ + argument++; + proba=0; + while ((*argument>='0') && (*argument<='9')) { + proba *= 10; + proba += *argument - '0'; + argument++; + } + if (proba<0) proba=0; + if (proba>100) proba=100; + break; + default: + ; + return FUZ_usage(); + } + } + } + } + + /* Get Seed */ + printf("Starting lz4frame tester (%i-bits, %s)\n", (int)(sizeof(size_t)*8), LZ4_VERSION_STRING); + + if (!seedset) { + time_t const t = time(NULL); + U32 const h = XXH32(&t, sizeof(t), 1); + seed = h % 10000; + } + printf("Seed = %u\n", seed); + if (proba!=FUZ_COMPRESSIBILITY_DEFAULT) printf("Compressibility : %i%%\n", proba); + + if (nbTests<=0) nbTests=1; + + if (testNb==0) result = basicTests(seed, ((double)proba) / 100); + if (result) return 1; + return fuzzerTests(seed, nbTests, testNb, ((double)proba) / 100, duration); +} diff --git a/tests/fullbench.c b/tests/fullbench.c new file mode 100644 index 0000000..ab6bba1 --- /dev/null +++ b/tests/fullbench.c @@ -0,0 +1,813 @@ +/* + bench.c - Demo program to benchmark open-source compression algorithm + Copyright (C) Yann Collet 2012-2015 + + GPL v2 License + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + You can contact the author at : + - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c +*/ + +/************************************** +* Compiler Options +**************************************/ +/* Disable some Visual warning messages */ +#define _CRT_SECURE_NO_WARNINGS +#define _CRT_SECURE_NO_DEPRECATE /* VS2005 */ + +/* Unix Large Files support (>4GB) */ +#if (defined(__sun__) && (!defined(__LP64__))) // Sun Solaris 32-bits requires specific definitions +# define _LARGEFILE_SOURCE +# define _FILE_OFFSET_BITS 64 +#elif ! defined(__LP64__) // No point defining Large file for 64 bit +# define _LARGEFILE64_SOURCE +#endif + +// S_ISREG & gettimeofday() are not supported by MSVC +#if defined(_MSC_VER) || defined(_WIN32) +# define BMK_LEGACY_TIMER 1 +#endif + + +/************************************** +* Includes +**************************************/ +#include /* malloc, free */ +#include /* fprintf, fopen, ftello64 */ +#include /* stat64 */ +#include /* stat64 */ +#include /* strcmp */ +#include /* clock_t, clock(), CLOCKS_PER_SEC */ + +#include "lz4.h" +#include "lz4hc.h" +#include "lz4frame.h" + +#include "xxhash.h" + + +/************************************** +* Compiler Options +**************************************/ +/* S_ISREG & gettimeofday() are not supported by MSVC */ +#if !defined(S_ISREG) +# define S_ISREG(x) (((x) & S_IFMT) == S_IFREG) +#endif + + +/************************************** +* Basic Types +**************************************/ +#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ +# include + typedef uint8_t BYTE; + typedef uint16_t U16; + typedef uint32_t U32; + typedef int32_t S32; + typedef uint64_t U64; +#else + typedef unsigned char BYTE; + typedef unsigned short U16; + typedef unsigned int U32; + typedef signed int S32; + typedef unsigned long long U64; +#endif + + +/************************************** +* Constants +**************************************/ +#define PROGRAM_DESCRIPTION "LZ4 speed analyzer" +#define AUTHOR "Yann Collet" +#define WELCOME_MESSAGE "*** %s v%s %i-bits, by %s ***\n", PROGRAM_DESCRIPTION, LZ4_VERSION_STRING, (int)(sizeof(void*)*8), AUTHOR + +#define NBLOOPS 6 +#define TIMELOOP (CLOCKS_PER_SEC * 25 / 10) + +#define KB *(1 <<10) +#define MB *(1 <<20) +#define GB *(1U<<30) + +#define KNUTH 2654435761U +#define MAX_MEM (1920 MB) +#define DEFAULT_CHUNKSIZE (4 MB) + +#define ALL_COMPRESSORS 0 +#define ALL_DECOMPRESSORS 0 + + +/************************************** +* Local structures +**************************************/ +struct chunkParameters +{ + U32 id; + char* origBuffer; + char* compressedBuffer; + int origSize; + int compressedSize; +}; + + +/************************************** +* Macros +**************************************/ +#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) +#define PROGRESS(...) g_noPrompt ? 0 : DISPLAY(__VA_ARGS__) + + +/************************************** +* Benchmark Parameters +**************************************/ +static int g_chunkSize = DEFAULT_CHUNKSIZE; +static int g_nbIterations = NBLOOPS; +static int g_pause = 0; +static int g_compressionTest = 1; +static int g_compressionAlgo = ALL_COMPRESSORS; +static int g_decompressionTest = 1; +static int g_decompressionAlgo = ALL_DECOMPRESSORS; +static int g_noPrompt = 0; + +static void BMK_setBlocksize(int bsize) +{ + g_chunkSize = bsize; + DISPLAY("-Using Block Size of %i KB-\n", g_chunkSize>>10); +} + +static void BMK_setNbIterations(int nbLoops) +{ + g_nbIterations = nbLoops; + DISPLAY("- %i iterations -\n", g_nbIterations); +} + +static void BMK_setPause(void) +{ + g_pause = 1; +} + + +/********************************************************* +* Private functions +*********************************************************/ +static clock_t BMK_GetClockSpan( clock_t clockStart ) +{ + return clock() - clockStart; /* works even if overflow; max span ~30 mn */ +} + + +static size_t BMK_findMaxMem(U64 requiredMem) +{ + size_t step = 64 MB; + BYTE* testmem=NULL; + + requiredMem = (((requiredMem >> 26) + 1) << 26); + requiredMem += 2*step; + if (requiredMem > MAX_MEM) requiredMem = MAX_MEM; + + while (!testmem) { + if (requiredMem > step) requiredMem -= step; + else requiredMem >>= 1; + testmem = (BYTE*) malloc ((size_t)requiredMem); + } + free (testmem); + + /* keep some space available */ + if (requiredMem > step) requiredMem -= step; + else requiredMem >>= 1; + + return (size_t)requiredMem; +} + + +static U64 BMK_GetFileSize(const char* infilename) +{ + int r; +#if defined(_MSC_VER) + struct _stat64 statbuf; + r = _stat64(infilename, &statbuf); +#else + struct stat statbuf; + r = stat(infilename, &statbuf); +#endif + if (r || !S_ISREG(statbuf.st_mode)) return 0; /* No good... */ + return (U64)statbuf.st_size; +} + + +/********************************************************* +* Benchmark function +*********************************************************/ +static LZ4_stream_t LZ4_stream; +static void local_LZ4_resetDictT(void) +{ + LZ4_resetStream(&LZ4_stream); +} + +static void local_LZ4_createStream(void) +{ + LZ4_resetStream(&LZ4_stream); +} + +static int local_LZ4_saveDict(const char* in, char* out, int inSize) +{ + (void)in; + return LZ4_saveDict(&LZ4_stream, out, inSize); +} + +static int local_LZ4_compress_limitedOutput(const char* in, char* out, int inSize) +{ + return LZ4_compress_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); +} + +static int local_LZ4_compress_default_large(const char* in, char* out, int inSize) +{ + return LZ4_compress_default(in, out, inSize, LZ4_compressBound(inSize)); +} + +static int local_LZ4_compress_default_small(const char* in, char* out, int inSize) +{ + return LZ4_compress_default(in, out, inSize, LZ4_compressBound(inSize)-1); +} + +static int local_LZ4_compress_fast0(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 0); +} + +static int local_LZ4_compress_fast1(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 1); +} + +static int local_LZ4_compress_fast2(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 2); +} + +static int local_LZ4_compress_fast17(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast(in, out, inSize, LZ4_compressBound(inSize), 17); +} + +static int local_LZ4_compress_fast_extState0(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast_extState(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize), 0); +} + +static int local_LZ4_compress_fast_continue0(const char* in, char* out, int inSize) +{ + return LZ4_compress_fast_continue(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize), 0); +} + +static int local_LZ4_compress_withState(const char* in, char* out, int inSize) +{ + return LZ4_compress_withState(&LZ4_stream, in, out, inSize); +} + +static int local_LZ4_compress_limitedOutput_withState(const char* in, char* out, int inSize) +{ + return LZ4_compress_limitedOutput_withState(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize)-1); +} + +static int local_LZ4_compress_continue(const char* in, char* out, int inSize) +{ + return LZ4_compress_continue(&LZ4_stream, in, out, inSize); +} + +static int local_LZ4_compress_limitedOutput_continue(const char* in, char* out, int inSize) +{ + return LZ4_compress_limitedOutput_continue(&LZ4_stream, in, out, inSize, LZ4_compressBound(inSize)-1); +} + +/* declare hidden function */ +int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize); + +static int local_LZ4_compress_forceDict(const char* in, char* out, int inSize) +{ + return LZ4_compress_forceExtDict(&LZ4_stream, in, out, inSize); +} + + +/* HC compression functions */ +LZ4_streamHC_t LZ4_streamHC; +static void local_LZ4_resetStreamHC(void) +{ + LZ4_resetStreamHC(&LZ4_streamHC, 0); +} + +static int local_LZ4_saveDictHC(const char* in, char* out, int inSize) +{ + (void)in; + return LZ4_saveDictHC(&LZ4_streamHC, out, inSize); +} + +static int local_LZ4_compressHC_withStateHC(const char* in, char* out, int inSize) +{ + return LZ4_compressHC_withStateHC(&LZ4_streamHC, in, out, inSize); +} + +static int local_LZ4_compressHC_limitedOutput_withStateHC(const char* in, char* out, int inSize) +{ + return LZ4_compressHC_limitedOutput_withStateHC(&LZ4_streamHC, in, out, inSize, LZ4_compressBound(inSize)-1); +} + +static int local_LZ4_compressHC_limitedOutput(const char* in, char* out, int inSize) +{ + return LZ4_compressHC_limitedOutput(in, out, inSize, LZ4_compressBound(inSize)-1); +} + +static int local_LZ4_compressHC_continue(const char* in, char* out, int inSize) +{ + return LZ4_compressHC_continue(&LZ4_streamHC, in, out, inSize); +} + +static int local_LZ4_compressHC_limitedOutput_continue(const char* in, char* out, int inSize) +{ + return LZ4_compressHC_limitedOutput_continue(&LZ4_streamHC, in, out, inSize, LZ4_compressBound(inSize)-1); +} + + +/* decompression functions */ +static int local_LZ4_decompress_fast(const char* in, char* out, int inSize, int outSize) +{ + (void)inSize; + LZ4_decompress_fast(in, out, outSize); + return outSize; +} + +static int local_LZ4_decompress_fast_usingDict(const char* in, char* out, int inSize, int outSize) +{ + (void)inSize; + LZ4_decompress_fast_usingDict(in, out, outSize, out - 65536, 65536); + return outSize; +} + +static int local_LZ4_decompress_safe_usingDict(const char* in, char* out, int inSize, int outSize) +{ + (void)inSize; + LZ4_decompress_safe_usingDict(in, out, inSize, outSize, out - 65536, 65536); + return outSize; +} + +extern int LZ4_decompress_safe_forceExtDict(const char* in, char* out, int inSize, int outSize, const char* dict, int dictSize); + +static int local_LZ4_decompress_safe_forceExtDict(const char* in, char* out, int inSize, int outSize) +{ + (void)inSize; + LZ4_decompress_safe_forceExtDict(in, out, inSize, outSize, out - 65536, 65536); + return outSize; +} + +static int local_LZ4_decompress_safe_partial(const char* in, char* out, int inSize, int outSize) +{ + return LZ4_decompress_safe_partial(in, out, inSize, outSize - 5, outSize); +} + + +/* frame functions */ +static int local_LZ4F_compressFrame(const char* in, char* out, int inSize) +{ + return (int)LZ4F_compressFrame(out, 2*inSize + 16, in, inSize, NULL); +} + +static LZ4F_decompressionContext_t g_dCtx; + +static int local_LZ4F_decompress(const char* in, char* out, int inSize, int outSize) +{ + size_t srcSize = inSize; + size_t dstSize = outSize; + size_t result; + result = LZ4F_decompress(g_dCtx, out, &dstSize, in, &srcSize, NULL); + if (result!=0) { DISPLAY("Error decompressing frame : unfinished frame\n"); exit(8); } + if (srcSize != (size_t)inSize) { DISPLAY("Error decompressing frame : read size incorrect\n"); exit(9); } + return (int)dstSize; +} + + +#define NB_COMPRESSION_ALGORITHMS 100 +#define NB_DECOMPRESSION_ALGORITHMS 100 +int fullSpeedBench(const char** fileNamesTable, int nbFiles) +{ + int fileIdx=0; + + /* Init */ + { size_t const errorCode = LZ4F_createDecompressionContext(&g_dCtx, LZ4F_VERSION); + if (LZ4F_isError(errorCode)) { DISPLAY("dctx allocation issue \n"); return 10; } } + + /* Loop for each fileName */ + while (fileIdx inFileSize) benchedSize = (size_t)inFileSize; + if (benchedSize < inFileSize) + DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", inFileName, (int)(benchedSize>>20)); + + /* Allocation */ + chunkP = (struct chunkParameters*) malloc(((benchedSize / (size_t)g_chunkSize)+1) * sizeof(struct chunkParameters)); + orig_buff = (char*) malloc(benchedSize); + nbChunks = (int) ((benchedSize + (g_chunkSize-1)) / g_chunkSize); + maxCompressedChunkSize = LZ4_compressBound(g_chunkSize); + compressedBuffSize = nbChunks * maxCompressedChunkSize; + compressed_buff = (char*)malloc((size_t)compressedBuffSize); + if(!chunkP || !orig_buff || !compressed_buff) { + DISPLAY("\nError: not enough memory!\n"); + fclose(inFile); + free(orig_buff); + free(compressed_buff); + free(chunkP); + return(12); + } + + /* Fill in src buffer */ + DISPLAY("Loading %s... \r", inFileName); + readSize = fread(orig_buff, 1, benchedSize, inFile); + fclose(inFile); + + if (readSize != benchedSize) { + DISPLAY("\nError: problem reading file '%s' !! \n", inFileName); + free(orig_buff); + free(compressed_buff); + free(chunkP); + return 13; + } + + /* Calculating input Checksum */ + crcOriginal = XXH32(orig_buff, benchedSize,0); + + + /* Bench */ + { int loopNb, nb_loops, chunkNb, cAlgNb, dAlgNb; + size_t cSize=0; + double ratio=0.; + + DISPLAY("\r%79s\r", ""); + DISPLAY(" %s : \n", inFileName); + + /* Bench Compression Algorithms */ + for (cAlgNb=0; (cAlgNb <= NB_COMPRESSION_ALGORITHMS) && (g_compressionTest); cAlgNb++) { + const char* compressorName; + int (*compressionFunction)(const char*, char*, int); + void (*initFunction)(void) = NULL; + double bestTime = 100000000.; + + /* filter compressionAlgo only */ + if ((g_compressionAlgo != ALL_COMPRESSORS) && (g_compressionAlgo != cAlgNb)) continue; + + /* Init data chunks */ + { int i; + size_t remaining = benchedSize; + char* in = orig_buff; + char* out = compressed_buff; + nbChunks = (int) (((int)benchedSize + (g_chunkSize-1))/ g_chunkSize); + for (i=0; i g_chunkSize) { chunkP[i].origSize = g_chunkSize; remaining -= g_chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } + chunkP[i].compressedBuffer = out; out += maxCompressedChunkSize; + chunkP[i].compressedSize = 0; + } + } + + switch(cAlgNb) + { + case 0 : DISPLAY("Compression functions : \n"); continue; + case 1 : compressionFunction = local_LZ4_compress_default_large; compressorName = "LZ4_compress_default"; break; + case 2 : compressionFunction = local_LZ4_compress_default_small; compressorName = "LZ4_compress_default(small dst)"; break; + case 3 : compressionFunction = local_LZ4_compress_fast0; compressorName = "LZ4_compress_fast(0)"; break; + case 4 : compressionFunction = local_LZ4_compress_fast1; compressorName = "LZ4_compress_fast(1)"; break; + case 5 : compressionFunction = local_LZ4_compress_fast2; compressorName = "LZ4_compress_fast(2)"; break; + case 6 : compressionFunction = local_LZ4_compress_fast17; compressorName = "LZ4_compress_fast(17)"; break; + case 7 : compressionFunction = local_LZ4_compress_fast_extState0; compressorName = "LZ4_compress_fast_extState(0)"; break; + case 8 : compressionFunction = local_LZ4_compress_fast_continue0; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_fast_continue(0)"; break; + + case 10: compressionFunction = LZ4_compressHC; compressorName = "LZ4_compressHC"; break; + case 11: compressionFunction = local_LZ4_compressHC_limitedOutput; compressorName = "LZ4_compressHC_limitedOutput"; break; + case 12 : compressionFunction = local_LZ4_compressHC_withStateHC; compressorName = "LZ4_compressHC_withStateHC"; break; + case 13: compressionFunction = local_LZ4_compressHC_limitedOutput_withStateHC; compressorName = "LZ4_compressHC_limitedOutput_withStateHC"; break; + case 14: compressionFunction = local_LZ4_compressHC_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_continue"; break; + case 15: compressionFunction = local_LZ4_compressHC_limitedOutput_continue; initFunction = local_LZ4_resetStreamHC; compressorName = "LZ4_compressHC_limitedOutput_continue"; break; + case 20: compressionFunction = local_LZ4_compress_forceDict; initFunction = local_LZ4_resetDictT; compressorName = "LZ4_compress_forceDict"; break; + case 30: compressionFunction = local_LZ4F_compressFrame; compressorName = "LZ4F_compressFrame"; + chunkP[0].origSize = (int)benchedSize; nbChunks=1; + break; + case 40: compressionFunction = local_LZ4_saveDict; compressorName = "LZ4_saveDict"; + LZ4_loadDict(&LZ4_stream, chunkP[0].origBuffer, chunkP[0].origSize); + break; + case 41: compressionFunction = local_LZ4_saveDictHC; compressorName = "LZ4_saveDictHC"; + LZ4_loadDictHC(&LZ4_streamHC, chunkP[0].origBuffer, chunkP[0].origSize); + break; + case 60: DISPLAY("Obsolete compression functions : \n"); continue; + case 61: compressionFunction = LZ4_compress; compressorName = "LZ4_compress"; break; + case 62: compressionFunction = local_LZ4_compress_limitedOutput; compressorName = "LZ4_compress_limitedOutput"; break; + case 63: compressionFunction = local_LZ4_compress_withState; compressorName = "LZ4_compress_withState"; break; + case 64: compressionFunction = local_LZ4_compress_limitedOutput_withState; compressorName = "LZ4_compress_limitedOutput_withState"; break; + case 65: compressionFunction = local_LZ4_compress_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_continue"; break; + case 66: compressionFunction = local_LZ4_compress_limitedOutput_continue; initFunction = local_LZ4_createStream; compressorName = "LZ4_compress_limitedOutput_continue"; break; + default : + continue; /* unknown ID : just skip */ + } + + for (loopNb = 1; loopNb <= g_nbIterations; loopNb++) { + double averageTime; + clock_t clockTime; + + PROGRESS("%1i- %-28.28s :%9i ->\r", loopNb, compressorName, (int)benchedSize); + { size_t i; for (i=0; i%9i (%5.2f%%),%7.1f MB/s\r", loopNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 1000000); + } + + if (ratio<100.) + DISPLAY("%2i-%-28.28s :%9i ->%9i (%5.2f%%),%7.1f MB/s\n", cAlgNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 1000000); + else + DISPLAY("%2i-%-28.28s :%9i ->%9i (%5.1f%%),%7.1f MB/s\n", cAlgNb, compressorName, (int)benchedSize, (int)cSize, ratio, (double)benchedSize / bestTime / 100000); + } + + /* Prepare layout for decompression */ + /* Init data chunks */ + { int i; + size_t remaining = benchedSize; + char* in = orig_buff; + char* out = compressed_buff; + + nbChunks = (int) (((int)benchedSize + (g_chunkSize-1))/ g_chunkSize); + for (i=0; i g_chunkSize) { chunkP[i].origSize = g_chunkSize; remaining -= g_chunkSize; } else { chunkP[i].origSize = (int)remaining; remaining = 0; } + chunkP[i].compressedBuffer = out; out += maxCompressedChunkSize; + chunkP[i].compressedSize = 0; + } + } + for (chunkNb=0; chunkNb\r", loopNb, dName, (int)benchedSize); + + nb_loops = 0; + clockTime = clock(); + while(clock() == clockTime); + clockTime = clock(); + while(BMK_GetClockSpan(clockTime) < TIMELOOP) { + for (chunkNb=0; chunkNb %7.1f MB/s\r", loopNb, dName, (int)benchedSize, (double)benchedSize / bestTime / 1000000); + + /* CRC Checking */ + crcDecoded = XXH32(orig_buff, (int)benchedSize, 0); + if (crcOriginal!=crcDecoded) { DISPLAY("\n!!! WARNING !!! %14s : Invalid Checksum : %x != %x\n", inFileName, (unsigned)crcOriginal, (unsigned)crcDecoded); exit(1); } + } + + DISPLAY("%2i-%-29.29s :%10i -> %7.1f MB/s\n", dAlgNb, dName, (int)benchedSize, (double)benchedSize / bestTime / 1000000); + } + } + free(orig_buff); + free(compressed_buff); + free(chunkP); + } + + LZ4F_freeDecompressionContext(g_dCtx); + if (g_pause) { printf("press enter...\n"); (void)getchar(); } + + return 0; +} + + +static int usage(const char* exename) +{ + DISPLAY( "Usage :\n"); + DISPLAY( " %s [arg] file1 file2 ... fileX\n", exename); + DISPLAY( "Arguments :\n"); + DISPLAY( " -c : compression tests only\n"); + DISPLAY( " -d : decompression tests only\n"); + DISPLAY( " -H/-h : Help (this text + advanced options)\n"); + return 0; +} + +static int usage_advanced(void) +{ + DISPLAY( "\nAdvanced options :\n"); + DISPLAY( " -c# : test only compression function # [1-%i]\n", NB_COMPRESSION_ALGORITHMS); + DISPLAY( " -d# : test only decompression function # [1-%i]\n", NB_DECOMPRESSION_ALGORITHMS); + DISPLAY( " -i# : iteration loops [1-9](default : %i)\n", NBLOOPS); + DISPLAY( " -B# : Block size [4-7](default : 7)\n"); + return 0; +} + +static int badusage(const char* exename) +{ + DISPLAY("Wrong parameters\n"); + usage(exename); + return 0; +} + +int main(int argc, const char** argv) +{ + int i, + filenamesStart=2; + const char* exename = argv[0]; + const char* input_filename=0; + + // Welcome message + DISPLAY(WELCOME_MESSAGE); + + if (argc<2) { badusage(exename); return 1; } + + for(i=1; i= '0') && (argument[1]<= '9')) { + g_compressionAlgo *= 10; + g_compressionAlgo += argument[1] - '0'; + argument++; + } + break; + + // Select decompression algorithm only + case 'd': + g_compressionTest = 0; + while ((argument[1]>= '0') && (argument[1]<= '9')) { + g_decompressionAlgo *= 10; + g_decompressionAlgo += argument[1] - '0'; + argument++; + } + break; + + // Display help on usage + case 'h' : + case 'H': usage(exename); usage_advanced(); return 0; + + // Modify Block Properties + case 'B': + while (argument[1]!=0) + switch(argument[1]) + { + case '4': + case '5': + case '6': + case '7': + { int B = argument[1] - '0'; + int S = 1 << (8 + 2*B); + BMK_setBlocksize(S); + argument++; + break; + } + case 'D': argument++; break; + default : goto _exit_blockProperties; + } +_exit_blockProperties: + break; + + // Modify Nb Iterations + case 'i': + if ((argument[1] >='0') && (argument[1] <='9')) { + int iters = argument[1] - '0'; + BMK_setNbIterations(iters); + argument++; + } + break; + + // Pause at the end (hidden option) + case 'p': BMK_setPause(); break; + + // Unknown command + default : badusage(exename); return 1; + } + } + continue; + } + + // first provided filename is input + if (!input_filename) { input_filename=argument; filenamesStart=i; continue; } + + } + + // No input filename ==> Error + if(!input_filename) { badusage(exename); return 1; } + + return fullSpeedBench(argv+filenamesStart, argc-filenamesStart); + +} diff --git a/tests/fuzzer.c b/tests/fuzzer.c new file mode 100644 index 0000000..db7ad07 --- /dev/null +++ b/tests/fuzzer.c @@ -0,0 +1,1169 @@ +/* + fuzzer.c - Fuzzer test tool for LZ4 + Copyright (C) Yann Collet 2012-2016 + + GPL v2 License + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + + You can contact the author at : + - LZ4 homepage : http://www.lz4.org + - LZ4 source repo : https://github.com/Cyan4973/lz4 +*/ + +/*-************************************ +* Compiler options +**************************************/ +#ifdef _MSC_VER /* Visual Studio */ +# define _CRT_SECURE_NO_WARNINGS /* fgets */ +# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ +# pragma warning(disable : 4146) /* disable: C4146: minus unsigned expression */ +# pragma warning(disable : 4310) /* disable: C4310: constant char value > 127 */ +#endif + +/* S_ISREG & gettimeofday() are not supported by MSVC */ +#if defined(_MSC_VER) || defined(_WIN32) +# define FUZ_LEGACY_TIMER 1 +#endif + + +/*-************************************ +* Includes +**************************************/ +#include +#include /* fgets, sscanf */ +#include /* strcmp */ +#include /* clock_t, clock, CLOCKS_PER_SEC */ +#include "lz4.h" /* LZ4_VERSION_STRING */ +#include "lz4hc.h" +#define XXH_STATIC_LINKING_ONLY +#include "xxhash.h" + + +/*-************************************ +* Basic Types +**************************************/ +#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */ +# include +typedef uint8_t BYTE; +typedef uint16_t U16; +typedef uint32_t U32; +typedef int32_t S32; +typedef uint64_t U64; +#else +typedef unsigned char BYTE; +typedef unsigned short U16; +typedef unsigned int U32; +typedef signed int S32; +typedef unsigned long long U64; +#endif + + +/*-************************************ +* Constants +**************************************/ +#define NB_ATTEMPTS (1<<16) +#define COMPRESSIBLE_NOISE_LENGTH (1 << 21) +#define FUZ_MAX_BLOCK_SIZE (1 << 17) +#define FUZ_MAX_DICT_SIZE (1 << 15) +#define FUZ_COMPRESSIBILITY_DEFAULT 60 +#define PRIME1 2654435761U +#define PRIME2 2246822519U +#define PRIME3 3266489917U + +#define KB *(1U<<10) +#define MB *(1U<<20) +#define GB *(1U<<30) + + +/*-*************************************** +* Macros +*****************************************/ +#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) +#define DISPLAYLEVEL(l, ...) if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); } +static int g_displayLevel = 2; +static const clock_t g_refreshRate = CLOCKS_PER_SEC * 25 / 100; +static clock_t g_time = 0; + + +/*-******************************************************* +* Fuzzer functions +*********************************************************/ +static clock_t FUZ_GetClockSpan(clock_t clockStart) +{ + return clock() - clockStart; /* works even if overflow; max span ~ 30mn */ +} + +static U32 FUZ_rotl32(U32 u32, U32 nbBits) +{ + return ((u32 << nbBits) | (u32 >> (32 - nbBits))); +} + +static U32 FUZ_rand(U32* src) +{ + U32 rand32 = *src; + rand32 *= PRIME1; + rand32 ^= PRIME2; + rand32 = FUZ_rotl32(rand32, 13); + *src = rand32; + return rand32; +} + + +#define FUZ_RAND15BITS ((FUZ_rand(seed) >> 3) & 32767) +#define FUZ_RANDLENGTH ( ((FUZ_rand(seed) >> 7) & 3) ? (FUZ_rand(seed) % 15) : (FUZ_rand(seed) % 510) + 15) +static void FUZ_fillCompressibleNoiseBuffer(void* buffer, size_t bufferSize, double proba, U32* seed) +{ + BYTE* BBuffer = (BYTE*)buffer; + size_t pos = 0; + U32 P32 = (U32)(32768 * proba); + + /* First Bytes */ + while (pos < 20) + BBuffer[pos++] = (BYTE)(FUZ_rand(seed)); + + while (pos < bufferSize) + { + /* Select : Literal (noise) or copy (within 64K) */ + if (FUZ_RAND15BITS < P32) + { + /* Copy (within 64K) */ + size_t match, d; + size_t length = FUZ_RANDLENGTH + 4; + size_t offset = FUZ_RAND15BITS + 1; + while (offset > pos) offset >>= 1; + d = pos + length; + while (d > bufferSize) d = bufferSize; + match = pos - offset; + while (pos < d) BBuffer[pos++] = BBuffer[match++]; + } + else + { + /* Literal (noise) */ + size_t d; + size_t length = FUZ_RANDLENGTH; + d = pos + length; + if (d > bufferSize) d = bufferSize; + while (pos < d) BBuffer[pos++] = (BYTE)(FUZ_rand(seed) >> 5); + } + } +} + + +#define MAX_NB_BUFF_I134 150 +#define BLOCKSIZE_I134 (32 MB) +/*! FUZ_AddressOverflow() : +* Aggressively pushes memory allocation limits, +* and generates patterns which create address space overflow. +* only possible in 32-bits mode */ +static int FUZ_AddressOverflow(void) +{ + char* buffers[MAX_NB_BUFF_I134+1]; + int i, nbBuff=0; + int highAddress = 0; + + DISPLAY("Overflow tests : "); + + /* Only possible in 32-bits */ + if (sizeof(void*)==8) + { + DISPLAY("64 bits mode : no overflow \n"); + fflush(stdout); + return 0; + } + + buffers[0] = (char*)malloc(BLOCKSIZE_I134); + buffers[1] = (char*)malloc(BLOCKSIZE_I134); + if ((!buffers[0]) || (!buffers[1])) { + DISPLAY("not enough memory for tests \n"); + return 0; + } + + for (nbBuff=2; nbBuff < MAX_NB_BUFF_I134; nbBuff++) { + DISPLAY("%3i \b\b\b\b", nbBuff); + buffers[nbBuff] = (char*)malloc(BLOCKSIZE_I134); + if (buffers[nbBuff]==NULL) goto _endOfTests; + + if (((size_t)buffers[nbBuff] > (size_t)0x80000000) && (!highAddress)) { + DISPLAY("high address detected : "); + fflush(stdout); + highAddress=1; + } + + { size_t const sizeToGenerateOverflow = (size_t)(- ((size_t)buffers[nbBuff-1]) + 512); + int const nbOf255 = (int)((sizeToGenerateOverflow / 255) + 1); + char* const input = buffers[nbBuff-1]; + char* output = buffers[nbBuff]; + int r; + input[0] = (char)0xF0; /* Literal length overflow */ + input[1] = (char)0xFF; + input[2] = (char)0xFF; + input[3] = (char)0xFF; + for(i = 4; i <= nbOf255+4; i++) input[i] = (char)0xff; + r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); + if (r>0) goto _overflowError; + input[0] = (char)0x1F; /* Match length overflow */ + input[1] = (char)0x01; + input[2] = (char)0x01; + input[3] = (char)0x00; + r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); + if (r>0) goto _overflowError; + + output = buffers[nbBuff-2]; /* Reverse in/out pointer order */ + input[0] = (char)0xF0; /* Literal length overflow */ + input[1] = (char)0xFF; + input[2] = (char)0xFF; + input[3] = (char)0xFF; + r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); + if (r>0) goto _overflowError; + input[0] = (char)0x1F; /* Match length overflow */ + input[1] = (char)0x01; + input[2] = (char)0x01; + input[3] = (char)0x00; + r = LZ4_decompress_safe(input, output, nbOf255+64, BLOCKSIZE_I134); + if (r>0) goto _overflowError; + } + } + + nbBuff++; +_endOfTests: + for (i=0 ; i g_refreshRate) | (g_displayLevel>=3)) { + g_time = clock(); + DISPLAY("\r%5u ", testNb); + if (g_displayLevel>=3) fflush(stdout); + } +} + + +/*! FUZ_findDiff() : +* find the first different byte between buff1 and buff2. +* presumes buff1 != buff2. +* presumes a difference exists before end of either buffer. +* Typically invoked after a checksum mismatch. +*/ +static void FUZ_findDiff(const void* buff1, const void* buff2) +{ + const BYTE* const b1 = (const BYTE*)buff1; + const BYTE* const b2 = (const BYTE*)buff2; + size_t i=0; + while (b1[i]==b2[i]) i++; + DISPLAY("Wrong Byte at position %u\n", (unsigned)i); +} + + +static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double compressibility, U32 duration_s) +{ + unsigned long long bytes = 0; + unsigned long long cbytes = 0; + unsigned long long hcbytes = 0; + unsigned long long ccbytes = 0; + void* CNBuffer; + char* compressedBuffer; + char* decodedBuffer; +# define FUZ_max LZ4_COMPRESSBOUND(LEN) + int ret; + unsigned cycleNb; +# define FUZ_CHECKTEST(cond, ...) if (cond) { printf("Test %u : ", testNb); printf(__VA_ARGS__); \ + printf(" (seed %u, cycle %u) \n", seed, cycleNb); goto _output_error; } +# define FUZ_DISPLAYTEST { testNb++; g_displayLevel<3 ? 0 : printf("%2u\b\b", testNb); if (g_displayLevel==4) fflush(stdout); } + void* stateLZ4 = malloc(LZ4_sizeofState()); + void* stateLZ4HC = malloc(LZ4_sizeofStateHC()); + LZ4_stream_t LZ4dict; + LZ4_streamHC_t LZ4dictHC; + U32 crcOrig, crcCheck; + U32 coreRandState = seed; + U32 randState = coreRandState ^ PRIME3; + int result = 0; + clock_t const clockStart = clock(); + clock_t const clockDuration = (clock_t)duration_s * CLOCKS_PER_SEC; + + + /* init */ + memset(&LZ4dict, 0, sizeof(LZ4dict)); + + /* Create compressible test buffer */ + CNBuffer = malloc(COMPRESSIBLE_NOISE_LENGTH); + FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, &randState); + compressedBuffer = (char*)malloc(LZ4_compressBound(FUZ_MAX_BLOCK_SIZE)); + decodedBuffer = (char*)malloc(FUZ_MAX_DICT_SIZE + FUZ_MAX_BLOCK_SIZE); + + /* move to startCycle */ + for (cycleNb = 0; cycleNb < startCycle; cycleNb++) { + (void)FUZ_rand(&coreRandState); + + if (0) { /* some problems can be related to dictionary re-use; in this case, enable this loop */ + int dictSize, blockSize, blockStart; + char* dict; + char* block; + FUZ_displayUpdate(cycleNb); + randState = coreRandState ^ PRIME3; + blockSize = FUZ_rand(&randState) % FUZ_MAX_BLOCK_SIZE; + blockStart = FUZ_rand(&randState) % (COMPRESSIBLE_NOISE_LENGTH - blockSize); + dictSize = FUZ_rand(&randState) % FUZ_MAX_DICT_SIZE; + if (dictSize > blockStart) dictSize = blockStart; + block = ((char*)CNBuffer) + blockStart; + dict = block - dictSize; + LZ4_loadDict(&LZ4dict, dict, dictSize); + LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); + LZ4_loadDict(&LZ4dict, dict, dictSize); + LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); + LZ4_loadDict(&LZ4dict, dict, dictSize); + LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); + } } + + /* Main test loop */ + for (cycleNb = startCycle; (cycleNb < nbCycles) || (FUZ_GetClockSpan(clockStart) < clockDuration) ; cycleNb++) { + U32 testNb = 0; + char* dict; + char* block; + int dictSize, blockSize, blockStart, compressedSize, HCcompressedSize; + int blockContinueCompressedSize; + + FUZ_displayUpdate(cycleNb); + (void)FUZ_rand(&coreRandState); + randState = coreRandState ^ PRIME3; + + /* Select block to test */ + blockSize = (FUZ_rand(&randState) % (FUZ_MAX_BLOCK_SIZE-1)) + 1; + blockStart = FUZ_rand(&randState) % (COMPRESSIBLE_NOISE_LENGTH - blockSize); + dictSize = FUZ_rand(&randState) % FUZ_MAX_DICT_SIZE; + if (dictSize > blockStart) dictSize = blockStart; + block = ((char*)CNBuffer) + blockStart; + dict = block - dictSize; + + /* Compression tests */ + + /* Test compression destSize */ + FUZ_DISPLAYTEST; + { int srcSize = blockSize; + int targetSize = srcSize * ((FUZ_rand(&randState) & 127)+1) >> 7; + char endCheck = FUZ_rand(&randState) & 255; + compressedBuffer[targetSize] = endCheck; + ret = LZ4_compress_destSize(block, compressedBuffer, &srcSize, targetSize); + FUZ_CHECKTEST(ret > targetSize, "LZ4_compress_destSize() result larger than dst buffer !"); + FUZ_CHECKTEST(compressedBuffer[targetSize] != endCheck, "LZ4_compress_destSize() overwrite dst buffer !"); + FUZ_CHECKTEST(srcSize > blockSize, "LZ4_compress_destSize() fed more than src buffer !"); + DISPLAYLEVEL(5, "destSize : %7i/%7i; content%7i/%7i ", ret, targetSize, srcSize, blockSize); + if (targetSize>0) { + FUZ_CHECKTEST((ret==0), "LZ4_compress_destSize() compression failed"); + /* check correctness */ + FUZ_DISPLAYTEST; + + crcOrig = XXH32(block, srcSize, 0); + compressedSize = ret; + endCheck = FUZ_rand(&randState) & 255; + decodedBuffer[srcSize] = endCheck; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, srcSize); + FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe() failed on data compressed by LZ4_compress_destSize"); + FUZ_CHECKTEST(ret!=srcSize, "LZ4_decompress_safe() failed : did not fully decompressed data"); + FUZ_CHECKTEST(decodedBuffer[srcSize] != endCheck, "LZ4_decompress_safe() overwrite dst buffer !"); + crcCheck = XXH32(decodedBuffer, srcSize, 0); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe() corrupted decoded data"); + + DISPLAYLEVEL(5, " OK \n"); + } + else + DISPLAYLEVEL(5, " \n"); + } + + /* Test compression HC */ + FUZ_DISPLAYTEST; + ret = LZ4_compressHC(block, compressedBuffer, blockSize); + FUZ_CHECKTEST(ret==0, "LZ4_compressHC() failed"); + HCcompressedSize = ret; + + /* Test compression HC using external state */ + FUZ_DISPLAYTEST; + ret = LZ4_compressHC_withStateHC(stateLZ4HC, block, compressedBuffer, blockSize); + FUZ_CHECKTEST(ret==0, "LZ4_compressHC_withStateHC() failed"); + + /* Test compression using external state */ + FUZ_DISPLAYTEST; + ret = LZ4_compress_withState(stateLZ4, block, compressedBuffer, blockSize); + FUZ_CHECKTEST(ret==0, "LZ4_compress_withState() failed"); + + /* Test compression */ + FUZ_DISPLAYTEST; + ret = LZ4_compress(block, compressedBuffer, blockSize); + FUZ_CHECKTEST(ret==0, "LZ4_compress() failed"); + compressedSize = ret; + + /* Decompression tests */ + + crcOrig = XXH32(block, blockSize, 0); + + /* Test decoding with output size being exactly what's necessary => must work */ + FUZ_DISPLAYTEST; + ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize); + FUZ_CHECKTEST(ret<0, "LZ4_decompress_fast failed despite correct space"); + FUZ_CHECKTEST(ret!=compressedSize, "LZ4_decompress_fast failed : did not fully read compressed data"); + crcCheck = XXH32(decodedBuffer, blockSize, 0); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast corrupted decoded data"); + + /* Test decoding with one byte missing => must fail */ + FUZ_DISPLAYTEST; + decodedBuffer[blockSize-1] = 0; + ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize-1); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too small"); + FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast overrun specified output buffer"); + + /* Test decoding with one byte too much => must fail */ + FUZ_DISPLAYTEST; + ret = LZ4_decompress_fast(compressedBuffer, decodedBuffer, blockSize+1); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast should have failed, due to Output Size being too large"); + + /* Test decoding with output size exactly what's necessary => must work */ + FUZ_DISPLAYTEST; + decodedBuffer[blockSize] = 0; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize); + FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe failed despite sufficient space"); + FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe did not regenerate original data"); + FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe overrun specified output buffer size"); + crcCheck = XXH32(decodedBuffer, blockSize, 0); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe corrupted decoded data"); + + // Test decoding with more than enough output size => must work + FUZ_DISPLAYTEST; + decodedBuffer[blockSize] = 0; + decodedBuffer[blockSize+1] = 0; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize+1); + FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe failed despite amply sufficient space"); + FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe did not regenerate original data"); + //FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe wrote more than (unknown) target size"); // well, is that an issue ? + FUZ_CHECKTEST(decodedBuffer[blockSize+1], "LZ4_decompress_safe overrun specified output buffer size"); + crcCheck = XXH32(decodedBuffer, blockSize, 0); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe corrupted decoded data"); + + // Test decoding with output size being one byte too short => must fail + FUZ_DISPLAYTEST; + decodedBuffer[blockSize-1] = 0; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize-1); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to Output Size being one byte too short"); + FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe overrun specified output buffer size"); + + // Test decoding with output size being 10 bytes too short => must fail + FUZ_DISPLAYTEST; + if (blockSize>10) + { + decodedBuffer[blockSize-10] = 0; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize, blockSize-10); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to Output Size being 10 bytes too short"); + FUZ_CHECKTEST(decodedBuffer[blockSize-10], "LZ4_decompress_safe overrun specified output buffer size"); + } + + // Test decoding with input size being one byte too short => must fail + FUZ_DISPLAYTEST; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize-1, blockSize); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to input size being one byte too short (blockSize=%i, ret=%i, compressedSize=%i)", blockSize, ret, compressedSize); + + // Test decoding with input size being one byte too large => must fail + FUZ_DISPLAYTEST; + decodedBuffer[blockSize] = 0; + ret = LZ4_decompress_safe(compressedBuffer, decodedBuffer, compressedSize+1, blockSize); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe should have failed, due to input size being too large"); + FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe overrun specified output buffer size"); + + // Test partial decoding with target output size being max/2 => must work + FUZ_DISPLAYTEST; + ret = LZ4_decompress_safe_partial(compressedBuffer, decodedBuffer, compressedSize, blockSize/2, blockSize); + FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe_partial failed despite sufficient space"); + + // Test partial decoding with target output size being just below max => must work + FUZ_DISPLAYTEST; + ret = LZ4_decompress_safe_partial(compressedBuffer, decodedBuffer, compressedSize, blockSize-3, blockSize); + FUZ_CHECKTEST(ret<0, "LZ4_decompress_safe_partial failed despite sufficient space"); + + /* Test Compression with limited output size */ + + /* Test compression with output size being exactly what's necessary (should work) */ + FUZ_DISPLAYTEST; + ret = LZ4_compress_limitedOutput(block, compressedBuffer, blockSize, compressedSize); + FUZ_CHECKTEST(ret==0, "LZ4_compress_limitedOutput() failed despite sufficient space"); + + /* Test compression with output size being exactly what's necessary and external state (should work) */ + FUZ_DISPLAYTEST; + ret = LZ4_compress_limitedOutput_withState(stateLZ4, block, compressedBuffer, blockSize, compressedSize); + FUZ_CHECKTEST(ret==0, "LZ4_compress_limitedOutput_withState() failed despite sufficient space"); + + /* Test HC compression with output size being exactly what's necessary (should work) */ + FUZ_DISPLAYTEST; + ret = LZ4_compressHC_limitedOutput(block, compressedBuffer, blockSize, HCcompressedSize); + FUZ_CHECKTEST(ret==0, "LZ4_compressHC_limitedOutput() failed despite sufficient space"); + + /* Test HC compression with output size being exactly what's necessary (should work) */ + FUZ_DISPLAYTEST; + ret = LZ4_compressHC_limitedOutput_withStateHC(stateLZ4HC, block, compressedBuffer, blockSize, HCcompressedSize); + FUZ_CHECKTEST(ret==0, "LZ4_compressHC_limitedOutput_withStateHC() failed despite sufficient space"); + + /* Test compression with missing bytes into output buffer => must fail */ + FUZ_DISPLAYTEST; + { int missingBytes = (FUZ_rand(&randState) % 0x3F) + 1; + if (missingBytes >= compressedSize) missingBytes = compressedSize-1; + missingBytes += !missingBytes; /* avoid special case missingBytes==0 */ + compressedBuffer[compressedSize-missingBytes] = 0; + ret = LZ4_compress_limitedOutput(block, compressedBuffer, blockSize, compressedSize-missingBytes); + FUZ_CHECKTEST(ret, "LZ4_compress_limitedOutput should have failed (output buffer too small by %i byte)", missingBytes); + FUZ_CHECKTEST(compressedBuffer[compressedSize-missingBytes], "LZ4_compress_limitedOutput overran output buffer ! (%i missingBytes)", missingBytes) + } + + /* Test HC compression with missing bytes into output buffer => must fail */ + FUZ_DISPLAYTEST; + { int missingBytes = (FUZ_rand(&randState) % 0x3F) + 1; + if (missingBytes >= HCcompressedSize) missingBytes = HCcompressedSize-1; + missingBytes += !missingBytes; /* avoid special case missingBytes==0 */ + compressedBuffer[HCcompressedSize-missingBytes] = 0; + ret = LZ4_compressHC_limitedOutput(block, compressedBuffer, blockSize, HCcompressedSize-missingBytes); + FUZ_CHECKTEST(ret, "LZ4_compressHC_limitedOutput should have failed (output buffer too small by %i byte)", missingBytes); + FUZ_CHECKTEST(compressedBuffer[HCcompressedSize-missingBytes], "LZ4_compressHC_limitedOutput overran output buffer ! (%i missingBytes)", missingBytes) + } + + + /*-******************/ + /* Dictionary tests */ + /*-******************/ + + /* Compress using dictionary */ + FUZ_DISPLAYTEST; + { LZ4_stream_t LZ4_stream; + LZ4_resetStream(&LZ4_stream); + LZ4_compress_continue (&LZ4_stream, dict, compressedBuffer, dictSize); /* Just to fill hash tables */ + blockContinueCompressedSize = LZ4_compress_continue (&LZ4_stream, block, compressedBuffer, blockSize); + FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed"); + } + + /* Decompress with dictionary as prefix */ + FUZ_DISPLAYTEST; + memcpy(decodedBuffer, dict, dictSize); + ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer+dictSize, blockSize, decodedBuffer, dictSize); + FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_decompress_fast_withPrefix64k did not read all compressed block input"); + crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0); + if (crcCheck!=crcOrig) { + int i=0; + while (block[i]==decodedBuffer[i]) i++; + printf("Wrong Byte at position %i/%i\n", i, blockSize); + + } + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_withPrefix64k corrupted decoded data (dict %i)", dictSize); + + FUZ_DISPLAYTEST; + ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer+dictSize, blockContinueCompressedSize, blockSize, decodedBuffer, dictSize); + FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data"); + crcCheck = XXH32(decodedBuffer+dictSize, blockSize, 0); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); + + /* Compress using External dictionary */ + FUZ_DISPLAYTEST; + dict -= (FUZ_rand(&randState) & 0xF) + 1; /* Separation, so it is an ExtDict */ + if (dict < (char*)CNBuffer) dict = (char*)CNBuffer; + LZ4_loadDict(&LZ4dict, dict, dictSize); + blockContinueCompressedSize = LZ4_compress_continue(&LZ4dict, block, compressedBuffer, blockSize); + FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compress_continue failed"); + + FUZ_DISPLAYTEST; + LZ4_loadDict(&LZ4dict, dict, dictSize); + ret = LZ4_compress_limitedOutput_continue(&LZ4dict, block, compressedBuffer, blockSize, blockContinueCompressedSize-1); + FUZ_CHECKTEST(ret>0, "LZ4_compress_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer : %i written, %i buffer", ret, blockContinueCompressedSize); + + FUZ_DISPLAYTEST; + LZ4_loadDict(&LZ4dict, dict, dictSize); + ret = LZ4_compress_limitedOutput_continue(&LZ4dict, block, compressedBuffer, blockSize, blockContinueCompressedSize); + FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_compress_limitedOutput_compressed size is different (%i != %i)", ret, blockContinueCompressedSize); + FUZ_CHECKTEST(ret<=0, "LZ4_compress_limitedOutput_continue should work : enough size available within output buffer"); + + /* Decompress with dictionary as external */ + FUZ_DISPLAYTEST; + decodedBuffer[blockSize] = 0; + ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize, dict, dictSize); + FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_decompress_fast_usingDict did not read all compressed block input"); + FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_fast_usingDict overrun specified output buffer size") + crcCheck = XXH32(decodedBuffer, blockSize, 0); + if (crcCheck!=crcOrig) + FUZ_findDiff(block, decodedBuffer); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_fast_usingDict corrupted decoded data (dict %i)", dictSize); + + FUZ_DISPLAYTEST; + decodedBuffer[blockSize] = 0; + ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize, dict, dictSize); + FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data"); + FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe_usingDict overrun specified output buffer size") + crcCheck = XXH32(decodedBuffer, blockSize, 0); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); + + FUZ_DISPLAYTEST; + decodedBuffer[blockSize-1] = 0; + ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)"); + FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size"); + + FUZ_DISPLAYTEST; + decodedBuffer[blockSize-1] = 0; + ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-1, dict, dictSize); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : not enough output size (-1 byte)"); + FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe_usingDict overrun specified output buffer size"); + + FUZ_DISPLAYTEST; + { U32 const missingBytes = (FUZ_rand(&randState) & 0xF) + 2; + if ((U32)blockSize > missingBytes) { + decodedBuffer[blockSize-missingBytes] = 0; + ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-missingBytes, dict, dictSize); + FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : output buffer too small (-%u byte)", missingBytes); + FUZ_CHECKTEST(decodedBuffer[blockSize-missingBytes], "LZ4_decompress_safe_usingDict overrun specified output buffer size (-%u byte) (blockSize=%i)", missingBytes, blockSize); + } } + + /* Compress HC using External dictionary */ + FUZ_DISPLAYTEST; + dict -= (FUZ_rand(&randState) & 7); /* even bigger separation */ + if (dict < (char*)CNBuffer) dict = (char*)CNBuffer; + LZ4_resetStreamHC (&LZ4dictHC, FUZ_rand(&randState) & 0x7); + LZ4_loadDictHC(&LZ4dictHC, dict, dictSize); + blockContinueCompressedSize = LZ4_compressHC_continue(&LZ4dictHC, block, compressedBuffer, blockSize); + FUZ_CHECKTEST(blockContinueCompressedSize==0, "LZ4_compressHC_continue failed"); + + FUZ_DISPLAYTEST; + LZ4_loadDictHC(&LZ4dictHC, dict, dictSize); + ret = LZ4_compressHC_limitedOutput_continue(&LZ4dictHC, block, compressedBuffer, blockSize, blockContinueCompressedSize-1); + FUZ_CHECKTEST(ret>0, "LZ4_compressHC_limitedOutput_continue using ExtDict should fail : one missing byte for output buffer"); + + FUZ_DISPLAYTEST; + LZ4_loadDictHC(&LZ4dictHC, dict, dictSize); + ret = LZ4_compressHC_limitedOutput_continue(&LZ4dictHC, block, compressedBuffer, blockSize, blockContinueCompressedSize); + FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_compress_limitedOutput_compressed size is different (%i != %i)", ret, blockContinueCompressedSize); + FUZ_CHECKTEST(ret<=0, "LZ4_compress_limitedOutput_continue should work : enough size available within output buffer"); + + FUZ_DISPLAYTEST; + decodedBuffer[blockSize] = 0; + ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize, dict, dictSize); + FUZ_CHECKTEST(ret!=blockSize, "LZ4_decompress_safe_usingDict did not regenerate original data"); + FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe_usingDict overrun specified output buffer size") + crcCheck = XXH32(decodedBuffer, blockSize, 0); + if (crcCheck!=crcOrig) + FUZ_findDiff(block, decodedBuffer); + FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data"); + + /* ***** End of tests *** */ + /* Fill stats */ + bytes += blockSize; + cbytes += compressedSize; + hcbytes += HCcompressedSize; + ccbytes += blockContinueCompressedSize; + } + + if (nbCycles<=1) nbCycles = cycleNb; /* end by time */ + bytes += !bytes; /* avoid division by 0 */ + printf("\r%7u /%7u - ", cycleNb, nbCycles); + printf("all tests completed successfully \n"); + printf("compression ratio: %0.3f%%\n", (double)cbytes/bytes*100); + printf("HC compression ratio: %0.3f%%\n", (double)hcbytes/bytes*100); + printf("ratio with dict: %0.3f%%\n", (double)ccbytes/bytes*100); + + /* release memory */ + { +_exit: + free(CNBuffer); + free(compressedBuffer); + free(decodedBuffer); + free(stateLZ4); + free(stateLZ4HC); + return result; + +_output_error: + result = 1; + goto _exit; + } +} + + +#define testInputSize (192 KB) +#define testCompressedSize (128 KB) +#define ringBufferSize (8 KB) + +static void FUZ_unitTests(void) +{ + const unsigned testNb = 0; + const unsigned seed = 0; + const unsigned cycleNb= 0; + char testInput[testInputSize]; + char testCompressed[testCompressedSize]; + char testVerify[testInputSize]; + char ringBuffer[ringBufferSize]; + U32 randState = 1; + + /* Init */ + FUZ_fillCompressibleNoiseBuffer(testInput, testInputSize, 0.50, &randState); + + /* 32-bits address space overflow test */ + FUZ_AddressOverflow(); + + /* LZ4 streaming tests */ + { LZ4_stream_t* statePtr; + LZ4_stream_t streamingState; + U64 crcOrig; + U64 crcNew; + int result; + + /* Allocation test */ + statePtr = LZ4_createStream(); + FUZ_CHECKTEST(statePtr==NULL, "LZ4_createStream() allocation failed"); + LZ4_freeStream(statePtr); + + /* simple compression test */ + crcOrig = XXH64(testInput, testCompressedSize, 0); + LZ4_resetStream(&streamingState); + result = LZ4_compress_limitedOutput_continue(&streamingState, testInput, testCompressed, testCompressedSize, testCompressedSize-1); + FUZ_CHECKTEST(result==0, "LZ4_compress_limitedOutput_continue() compression failed"); + + result = LZ4_decompress_safe(testCompressed, testVerify, result, testCompressedSize); + FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe() decompression failed"); + crcNew = XXH64(testVerify, testCompressedSize, 0); + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); + + /* ring buffer test */ + { XXH64_state_t xxhOrig; + XXH64_state_t xxhNew; + LZ4_streamDecode_t decodeState; + const U32 maxMessageSizeLog = 10; + const U32 maxMessageSizeMask = (1< ringBufferSize) rNext = 0; + if (dNext + messageSize > dBufferSize) dNext = 0; + } + } + } + + /* LZ4 HC streaming tests */ + { LZ4_streamHC_t* sp; + LZ4_streamHC_t sHC; + U64 crcOrig; + U64 crcNew; + int result; + + /* Allocation test */ + sp = LZ4_createStreamHC(); + FUZ_CHECKTEST(sp==NULL, "LZ4_createStreamHC() allocation failed"); + LZ4_freeStreamHC(sp); + + /* simple HC compression test */ + crcOrig = XXH64(testInput, testCompressedSize, 0); + LZ4_resetStreamHC(&sHC, 0); + result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput, testCompressed, testCompressedSize, testCompressedSize-1); + FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() compression failed"); + + result = LZ4_decompress_safe(testCompressed, testVerify, result, testCompressedSize); + FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe() decompression failed"); + crcNew = XXH64(testVerify, testCompressedSize, 0); + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption"); + + /* simple dictionary HC compression test */ + crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0); + LZ4_resetStreamHC(&sHC, 0); + LZ4_loadDictHC(&sHC, testInput, 64 KB); + result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + 64 KB, testCompressed, testCompressedSize, testCompressedSize-1); + FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result); + + result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result, testCompressedSize, testInput, 64 KB); + FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe() simple dictionary decompression test failed"); + crcNew = XXH64(testVerify, testCompressedSize, 0); + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() simple dictionary decompression test : corruption"); + + /* multiple HC compression test with dictionary */ + { int result1, result2; + int segSize = testCompressedSize / 2; + crcOrig = XXH64(testInput + segSize, testCompressedSize, 0); + LZ4_resetStreamHC(&sHC, 0); + LZ4_loadDictHC(&sHC, testInput, segSize); + result1 = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + segSize, testCompressed, segSize, segSize -1); + FUZ_CHECKTEST(result1==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result1); + result2 = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + 2*segSize, testCompressed+result1, segSize, segSize-1); + FUZ_CHECKTEST(result2==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result2); + + result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result1, segSize, testInput, segSize); + FUZ_CHECKTEST(result!=segSize, "LZ4_decompress_safe() dictionary decompression part 1 failed"); + result = LZ4_decompress_safe_usingDict(testCompressed+result1, testVerify+segSize, result2, segSize, testInput, 2*segSize); + FUZ_CHECKTEST(result!=segSize, "LZ4_decompress_safe() dictionary decompression part 2 failed"); + crcNew = XXH64(testVerify, testCompressedSize, 0); + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() dictionary decompression corruption"); + } + + /* remote dictionary HC compression test */ + crcOrig = XXH64(testInput + 64 KB, testCompressedSize, 0); + LZ4_resetStreamHC(&sHC, 0); + LZ4_loadDictHC(&sHC, testInput, 32 KB); + result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + 64 KB, testCompressed, testCompressedSize, testCompressedSize-1); + FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() remote dictionary failed : result = %i", result); + + result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result, testCompressedSize, testInput, 32 KB); + FUZ_CHECKTEST(result!=(int)testCompressedSize, "LZ4_decompress_safe_usingDict() decompression failed following remote dictionary HC compression test"); + crcNew = XXH64(testVerify, testCompressedSize, 0); + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe_usingDict() decompression corruption"); + + /* multiple HC compression with ext. dictionary */ + { XXH64_state_t crcOrigState; + XXH64_state_t crcNewState; + const char* dict = testInput + 3; + int dictSize = (FUZ_rand(&randState) & 8191); + char* dst = testVerify; + + size_t segStart = dictSize + 7; + int segSize = (FUZ_rand(&randState) & 8191); + int segNb = 1; + + LZ4_resetStreamHC(&sHC, 0); + LZ4_loadDictHC(&sHC, dict, dictSize); + + XXH64_reset(&crcOrigState, 0); + XXH64_reset(&crcNewState, 0); + + while (segStart + segSize < testInputSize) { + XXH64_update(&crcOrigState, testInput + segStart, segSize); + crcOrig = XXH64_digest(&crcOrigState); + result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + segStart, testCompressed, segSize, LZ4_compressBound(segSize)); + FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result); + + result = LZ4_decompress_safe_usingDict(testCompressed, dst, result, segSize, dict, dictSize); + FUZ_CHECKTEST(result!=segSize, "LZ4_decompress_safe_usingDict() dictionary decompression part %i failed", segNb); + XXH64_update(&crcNewState, dst, segSize); + crcNew = XXH64_digest(&crcNewState); + if (crcOrig!=crcNew) { + size_t c=0; + while (dst[c] == testInput[segStart+c]) c++; + DISPLAY("Bad decompression at %u / %u \n", (U32)c, (U32)segSize); + } + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe_usingDict() part %i corruption", segNb); + + dict = dst; + //dict = testInput + segStart; + dictSize = segSize; + + dst += segSize + 1; + segNb ++; + + segStart += segSize + (FUZ_rand(&randState) & 0xF) + 1; + segSize = (FUZ_rand(&randState) & 8191); + } + } + + /* ring buffer test */ + { XXH64_state_t xxhOrig; + XXH64_state_t xxhNew; + LZ4_streamDecode_t decodeState; + const U32 maxMessageSizeLog = 10; + const U32 maxMessageSizeMask = (1< ringBufferSize) rNext = 0; + if (dNext + messageSize > dBufferSize) dNext = 0; + } + } + + /* small decoder-side ring buffer test */ + { XXH64_state_t xxhOrig; + XXH64_state_t xxhNew; + LZ4_streamDecode_t decodeState; + const U32 maxMessageSizeLog = 12; + const U32 maxMessageSizeMask = (1< dBufferSize) dNext = 0; + + while (totalMessageSize < 9 MB) { + XXH64_update(&xxhOrig, testInput + iNext, messageSize); + crcOrig = XXH64_digest(&xxhOrig); + + result = LZ4_compressHC_limitedOutput_continue(&sHC, testInput + iNext, testCompressed, messageSize, testCompressedSize-ringBufferSize); + FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() compression failed"); + + result = LZ4_decompress_safe_continue(&decodeState, testCompressed, testVerify + dNext, result, messageSize); + FUZ_CHECKTEST(result!=(int)messageSize, "64K D.ringBuffer : LZ4_decompress_safe() test failed"); + + XXH64_update(&xxhNew, testVerify + dNext, messageSize); + crcNew = XXH64_digest(&xxhNew); + if (crcOrig != crcNew) + FUZ_findDiff(testInput + iNext, testVerify + dNext); + FUZ_CHECKTEST(crcOrig!=crcNew, "LZ4_decompress_safe() decompression corruption during small decoder-side ring buffer test"); + + /* prepare next message */ + dNext += messageSize; + totalMessageSize += messageSize; + messageSize = (FUZ_rand(&randState) & maxMessageSizeMask) + 1; + iNext = (FUZ_rand(&randState) & 65535); + if (dNext > dBufferSize) dNext = 0; + } + } + } + + printf("All unit tests completed successfully \n"); + return; +_output_error: + exit(1); +} + + +static int FUZ_usage(char* programName) +{ + DISPLAY( "Usage :\n"); + DISPLAY( " %s [args]\n", programName); + DISPLAY( "\n"); + DISPLAY( "Arguments :\n"); + DISPLAY( " -i# : Nb of tests (default:%i) \n", NB_ATTEMPTS); + DISPLAY( " -T# : Duration of tests, in seconds (default: use Nb of tests) \n"); + DISPLAY( " -s# : Select seed (default:prompt user)\n"); + DISPLAY( " -t# : Select starting test number (default:0)\n"); + DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n", FUZ_COMPRESSIBILITY_DEFAULT); + DISPLAY( " -v : verbose\n"); + DISPLAY( " -p : pause at the end\n"); + DISPLAY( " -h : display help and exit\n"); + return 0; +} + + +int main(int argc, char** argv) +{ + U32 seed=0; + int seedset=0; + int argNb; + int nbTests = NB_ATTEMPTS; + int testNb = 0; + int proba = FUZ_COMPRESSIBILITY_DEFAULT; + int pause = 0; + char* programName = argv[0]; + U32 duration = 0; + + /* Check command line */ + for(argNb=1; argNb='0') && (*argument<='9')) { + nbTests *= 10; + nbTests += *argument - '0'; + argument++; + } + break; + + case 'T': + argument++; + nbTests = 0; duration = 0; + for (;;) { + switch(*argument) + { + case 'm': duration *= 60; argument++; continue; + case 's': + case 'n': argument++; continue; + case '0': + case '1': + case '2': + case '3': + case '4': + case '5': + case '6': + case '7': + case '8': + case '9': duration *= 10; duration += *argument++ - '0'; continue; + } + break; + } + break; + + case 's': + argument++; + seed=0; seedset=1; + while ((*argument>='0') && (*argument<='9')) { + seed *= 10; + seed += *argument - '0'; + argument++; + } + break; + + case 't': /* select starting test nb */ + argument++; + testNb=0; + while ((*argument>='0') && (*argument<='9')) { + testNb *= 10; + testNb += *argument - '0'; + argument++; + } + break; + + case 'P': /* change probability */ + argument++; + proba=0; + while ((*argument>='0') && (*argument<='9')) { + proba *= 10; + proba += *argument - '0'; + argument++; + } + if (proba<0) proba=0; + if (proba>100) proba=100; + break; + default: ; + } + } + } + } + + printf("Starting LZ4 fuzzer (%i-bits, v%s)\n", (int)(sizeof(size_t)*8), LZ4_VERSION_STRING); + + if (!seedset) { + time_t const t = time(NULL); + U32 const h = XXH32(&t, sizeof(t), 1); + seed = h % 10000; + } + printf("Seed = %u\n", seed); + + if (proba!=FUZ_COMPRESSIBILITY_DEFAULT) printf("Compressibility : %i%%\n", proba); + + if ((seedset==0) && (testNb==0)) FUZ_unitTests(); + + if (nbTests<=0) nbTests=1; + + { int const result = FUZ_test(seed, nbTests, testNb, ((double)proba) / 100, duration); + if (pause) { + DISPLAY("press enter ... \n"); + (void)getchar(); + } + return result; + } +} -- cgit v0.12 From 84cedb4632ab87fbb108b4f7ed6e9ec164b6a4d4 Mon Sep 17 00:00:00 2001 From: Przemyslaw Skibinski Date: Thu, 3 Nov 2016 15:12:57 +0100 Subject: updated links to LZ4 repository --- lib/lz4.c | 2 +- lib/lz4.h | 2 +- lib/lz4frame.c | 2 +- lib/lz4frame.h | 2 +- lib/lz4frame_static.h | 2 +- lib/lz4hc.c | 2 +- lib/lz4hc.h | 2 +- programs/bench.c | 2 +- programs/bench.h | 2 +- programs/datagen.c | 2 +- programs/datagen.h | 2 +- programs/lz4cli.c | 2 +- programs/lz4io.c | 2 +- programs/lz4io.h | 2 +- tests/Makefile | 2 +- tests/datagencli.c | 2 +- tests/frametest.c | 2 +- tests/fullbench.c | 2 +- tests/fuzzer.c | 2 +- 19 files changed, 19 insertions(+), 19 deletions(-) diff --git a/lib/lz4.c b/lib/lz4.c index 08128bb..5b7d71f 100644 --- a/lib/lz4.c +++ b/lib/lz4.c @@ -29,7 +29,7 @@ You can contact the author at : - LZ4 homepage : http://www.lz4.org - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 */ diff --git a/lib/lz4.h b/lib/lz4.h index 0f17c7c..fc27c95 100644 --- a/lib/lz4.h +++ b/lib/lz4.h @@ -30,7 +30,7 @@ You can contact the author at : - LZ4 homepage : http://www.lz4.org - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 */ #ifndef LZ4_H_2983827168210 #define LZ4_H_2983827168210 diff --git a/lib/lz4frame.c b/lib/lz4frame.c index 9df2493..6bb76a1 100644 --- a/lib/lz4frame.c +++ b/lib/lz4frame.c @@ -29,7 +29,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - LZ4 homepage : http://www.lz4.org -- LZ4 source repository : https://github.com/Cyan4973/lz4 +- LZ4 source repository : https://github.com/lz4/lz4 */ /* LZ4F is a stand-alone API to create LZ4-compressed Frames diff --git a/lib/lz4frame.h b/lib/lz4frame.h index a70d0e0..79164b8 100644 --- a/lib/lz4frame.h +++ b/lib/lz4frame.h @@ -28,7 +28,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ diff --git a/lib/lz4frame_static.h b/lib/lz4frame_static.h index 0d90975..79e481c 100644 --- a/lib/lz4frame_static.h +++ b/lib/lz4frame_static.h @@ -29,7 +29,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ diff --git a/lib/lz4hc.c b/lib/lz4hc.c index 68afa98..a1d1a55 100644 --- a/lib/lz4hc.c +++ b/lib/lz4hc.c @@ -28,7 +28,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ diff --git a/lib/lz4hc.h b/lib/lz4hc.h index 3995ada..4f676aa 100644 --- a/lib/lz4hc.h +++ b/lib/lz4hc.h @@ -28,7 +28,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ #ifndef LZ4_HC_H_19834876238432 diff --git a/programs/bench.c b/programs/bench.c index 5952f47..bec93f9 100644 --- a/programs/bench.c +++ b/programs/bench.c @@ -20,7 +20,7 @@ You can contact the author at : - LZ4 homepage : http://www.lz4.org - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 */ /*-************************************ diff --git a/programs/bench.h b/programs/bench.h index 1e3df78..e84aaee 100644 --- a/programs/bench.h +++ b/programs/bench.h @@ -17,7 +17,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ #pragma once diff --git a/programs/datagen.c b/programs/datagen.c index 5e4cdd2..d851418 100644 --- a/programs/datagen.c +++ b/programs/datagen.c @@ -19,7 +19,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - Public forum : https://groups.google.com/forum/#!forum/lz4c */ diff --git a/programs/datagen.h b/programs/datagen.h index 2bf3d7f..680afa2 100644 --- a/programs/datagen.h +++ b/programs/datagen.h @@ -19,7 +19,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - Public forum : https://groups.google.com/forum/#!forum/lz4c */ diff --git a/programs/lz4cli.c b/programs/lz4cli.c index 961bea1..7cf035e 100644 --- a/programs/lz4cli.c +++ b/programs/lz4cli.c @@ -19,7 +19,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ /* diff --git a/programs/lz4io.c b/programs/lz4io.c index 492363f..29bf359 100644 --- a/programs/lz4io.c +++ b/programs/lz4io.c @@ -19,7 +19,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ /* diff --git a/programs/lz4io.h b/programs/lz4io.h index 11c8fdb..4f0e02a 100644 --- a/programs/lz4io.h +++ b/programs/lz4io.h @@ -18,7 +18,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ /* diff --git a/tests/Makefile b/tests/Makefile index 7652476..24cb37a 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -21,7 +21,7 @@ # # You can contact the author at : # - LZ4 homepage : http://www.lz4.org -# - LZ4 source repository : https://github.com/Cyan4973/lz4 +# - LZ4 source repository : https://github.com/lz4/lz4 # ########################################################################## # fuzzer : Test tool, to check lz4 integrity on target platform # fuzzer32: Same as fuzzer, but forced to compile in 32-bits mode diff --git a/tests/datagencli.c b/tests/datagencli.c index 2fbcc50..d74d5dd 100644 --- a/tests/datagencli.c +++ b/tests/datagencli.c @@ -20,7 +20,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - Public forum : https://groups.google.com/forum/#!forum/lz4c */ diff --git a/tests/frametest.c b/tests/frametest.c index 4322ab8..ccf4278 100644 --- a/tests/frametest.c +++ b/tests/frametest.c @@ -20,7 +20,7 @@ You can contact the author at : - LZ4 homepage : http://www.lz4.org - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 */ /*-************************************ diff --git a/tests/fullbench.c b/tests/fullbench.c index ab6bba1..67442b2 100644 --- a/tests/fullbench.c +++ b/tests/fullbench.c @@ -19,7 +19,7 @@ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. You can contact the author at : - - LZ4 source repository : https://github.com/Cyan4973/lz4 + - LZ4 source repository : https://github.com/lz4/lz4 - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c */ diff --git a/tests/fuzzer.c b/tests/fuzzer.c index db7ad07..bbf4e14 100644 --- a/tests/fuzzer.c +++ b/tests/fuzzer.c @@ -20,7 +20,7 @@ You can contact the author at : - LZ4 homepage : http://www.lz4.org - - LZ4 source repo : https://github.com/Cyan4973/lz4 + - LZ4 source repo : https://github.com/lz4/lz4 */ /*-************************************ -- cgit v0.12 From 9f15a4c6e6aa1d6aadfc87501576a4bf3d16c771 Mon Sep 17 00:00:00 2001 From: Przemyslaw Skibinski Date: Thu, 3 Nov 2016 15:18:59 +0100 Subject: updated Visual Studio projects --- visual/VS2010/datagen/datagen.vcxproj | 2 +- visual/VS2010/frametest/frametest.vcxproj | 2 +- visual/VS2010/fullbench/fullbench.vcxproj | 2 +- visual/VS2010/fuzzer/fuzzer.vcxproj | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/visual/VS2010/datagen/datagen.vcxproj b/visual/VS2010/datagen/datagen.vcxproj index 74aabba..d64a3c2 100644 --- a/visual/VS2010/datagen/datagen.vcxproj +++ b/visual/VS2010/datagen/datagen.vcxproj @@ -153,7 +153,7 @@ - + diff --git a/visual/VS2010/frametest/frametest.vcxproj b/visual/VS2010/frametest/frametest.vcxproj index b44a3e6..5def90e 100644 --- a/visual/VS2010/frametest/frametest.vcxproj +++ b/visual/VS2010/frametest/frametest.vcxproj @@ -156,7 +156,7 @@ - + diff --git a/visual/VS2010/fullbench/fullbench.vcxproj b/visual/VS2010/fullbench/fullbench.vcxproj index 84b22c6..3d858a3 100644 --- a/visual/VS2010/fullbench/fullbench.vcxproj +++ b/visual/VS2010/fullbench/fullbench.vcxproj @@ -156,7 +156,7 @@ - + diff --git a/visual/VS2010/fuzzer/fuzzer.vcxproj b/visual/VS2010/fuzzer/fuzzer.vcxproj index 9b36c8a..2b2d58d 100644 --- a/visual/VS2010/fuzzer/fuzzer.vcxproj +++ b/visual/VS2010/fuzzer/fuzzer.vcxproj @@ -155,7 +155,7 @@ - + -- cgit v0.12 From 74ba377d7220fce86df74bf83eca504c40b3212e Mon Sep 17 00:00:00 2001 From: Przemyslaw Skibinski Date: Thu, 3 Nov 2016 15:28:50 +0100 Subject: updated .travis.yml --- .travis.yml | 34 +++++++++++++++++----------------- programs/Makefile | 4 ---- visual/VS2010/datagen/datagen.vcxproj | 8 ++++---- 3 files changed, 21 insertions(+), 25 deletions(-) diff --git a/.travis.yml b/.travis.yml index 4966e11..2a0eaab 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,7 +4,7 @@ matrix: include: # OS X Mavericks - os: osx - env: Ubu="OS X Mavericks" PARAMS='-C programs test-lz4 CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion"' COMPILER=clang + env: Ubu="OS X Mavericks" PARAMS='-C tests test-lz4 CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion"' COMPILER=clang # Container-based 12.04 LTS Server Edition 64 bit (doesn't support 32-bit includes) - os: linux sudo: false @@ -23,22 +23,22 @@ matrix: env: Ubu="12.04cont" PARAMS=gpptest COMPILER=cc - os: linux sudo: false - env: Ubu="12.04cont" PARAMS='-C programs test-lz4' COMPILER=cc + env: Ubu="12.04cont" PARAMS='-C tests test-lz4' COMPILER=cc - os: linux sudo: false - env: Ubu="12.04cont" PARAMS='-C programs test-lz4 MOREFLAGS="-std=c99"' COMPILER=cc + env: Ubu="12.04cont" PARAMS='-C tests test-lz4 MOREFLAGS="-std=c99"' COMPILER=cc - os: linux sudo: false - env: Ubu="12.04cont" PARAMS='-C programs test-lz4c' COMPILER=cc + env: Ubu="12.04cont" PARAMS='-C tests test-lz4c' COMPILER=cc - os: linux sudo: false - env: Ubu="12.04cont" PARAMS='-C programs test-fullbench' COMPILER=cc + env: Ubu="12.04cont" PARAMS='-C tests test-fullbench' COMPILER=cc - os: linux sudo: false - env: Ubu="12.04cont" PARAMS='-C programs test-fuzzer' COMPILER=cc + env: Ubu="12.04cont" PARAMS='-C tests test-fuzzer' COMPILER=cc - os: linux sudo: false - env: Ubu="12.04cont" PARAMS='-C programs test-frametest' COMPILER=cc + env: Ubu="12.04cont" PARAMS='-C tests test-frametest' COMPILER=cc # 14.04 LTS Server Edition 64 bit - env: Ubu="14.04" PARAMS=sanitize COMPILER=clang dist: trusty @@ -72,10 +72,10 @@ matrix: - python3 - libc6-dev-i386 - gcc-multilib - - env: Ubu="14.04" PARAMS='-C programs test-lz4' COMPILER=cc + - env: Ubu="14.04" PARAMS='-C tests test-lz4' COMPILER=cc dist: trusty sudo: required - - env: Ubu="14.04" PARAMS='-C programs test-lz4c32' COMPILER=cc + - env: Ubu="14.04" PARAMS='-C tests test-lz4c32' COMPILER=cc dist: trusty sudo: required addons: @@ -83,7 +83,7 @@ matrix: packages: - libc6-dev-i386 - gcc-multilib - - env: Ubu="14.04" PARAMS='-C programs test-lz4 CC=gcc-5' COMPILER=gcc-5 + - env: Ubu="14.04" PARAMS='-C tests test-lz4 CC=gcc-5' COMPILER=gcc-5 dist: trusty sudo: required addons: @@ -92,7 +92,7 @@ matrix: - ubuntu-toolchain-r-test packages: - gcc-5 - - env: Ubu="14.04" PARAMS='-C programs test-lz4c32 CC=gcc-5' COMPILER=gcc-5 + - env: Ubu="14.04" PARAMS='-C tests test-lz4c32 CC=gcc-5' COMPILER=gcc-5 dist: trusty sudo: required addons: @@ -102,7 +102,7 @@ matrix: packages: - gcc-5 - gcc-5-multilib - - env: Ubu="14.04" PARAMS='-C programs test-lz4 CC=gcc-6' COMPILER=gcc-6 + - env: Ubu="14.04" PARAMS='-C tests test-lz4 CC=gcc-6' COMPILER=gcc-6 dist: trusty sudo: required addons: @@ -111,7 +111,7 @@ matrix: - ubuntu-toolchain-r-test packages: - gcc-6 - - env: Ubu="14.04" PARAMS='-C programs test-lz4 CC=clang-3.8' COMPILER=clang-3.8 + - env: Ubu="14.04" PARAMS='-C tests test-lz4 CC=clang-3.8' COMPILER=clang-3.8 dist: trusty sudo: required addons: @@ -121,7 +121,7 @@ matrix: - llvm-toolchain-precise-3.8 packages: - clang-3.8 - - env: Ubu="14.04" PARAMS='-C programs test-fullbench32' COMPILER=cc + - env: Ubu="14.04" PARAMS='-C tests test-fullbench32' COMPILER=cc dist: trusty sudo: required addons: @@ -129,7 +129,7 @@ matrix: packages: - libc6-dev-i386 - gcc-multilib - - env: Ubu="14.04" PARAMS='-C programs test-fuzzer32' COMPILER=cc + - env: Ubu="14.04" PARAMS='-C tests test-fuzzer32' COMPILER=cc dist: trusty sudo: required addons: @@ -137,7 +137,7 @@ matrix: packages: - libc6-dev-i386 - gcc-multilib - - env: Ubu="14.04" PARAMS='-C programs test-frametest32' COMPILER=cc + - env: Ubu="14.04" PARAMS='-C tests test-frametest32' COMPILER=cc dist: trusty sudo: required addons: @@ -145,7 +145,7 @@ matrix: packages: - libc6-dev-i386 - gcc-multilib - - env: Ubu="14.04" PARAMS='-C programs test-mem' COMPILER=cc + - env: Ubu="14.04" PARAMS='-C tests test-mem' COMPILER=cc dist: trusty sudo: required addons: diff --git a/programs/Makefile b/programs/Makefile index 8cb416c..59eb5e7 100644 --- a/programs/Makefile +++ b/programs/Makefile @@ -50,10 +50,6 @@ VOID = /dev/null endif -# Default test parameters -TEST_FILES := COPYING -FUZZER_TIME := -T9mn - default: lz4 diff --git a/visual/VS2010/datagen/datagen.vcxproj b/visual/VS2010/datagen/datagen.vcxproj index d64a3c2..4d7516e 100644 --- a/visual/VS2010/datagen/datagen.vcxproj +++ b/visual/VS2010/datagen/datagen.vcxproj @@ -65,20 +65,20 @@ true - $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); true - $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); true false - $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); false - $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); + $(IncludePath);$(UniversalCRT_IncludePath);$(SolutionDir)..\..\programs;$(SolutionDir)..\..\lib;$(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSDK_IncludePath); true -- cgit v0.12 From f38adde9e0c2ffcac66bf094c3f24e25ec24c8a6 Mon Sep 17 00:00:00 2001 From: Przemyslaw Skibinski Date: Thu, 3 Nov 2016 16:01:50 +0100 Subject: versionsTests moved to tests/ --- tests/COPYING | 339 ++++++++++++++++++++++++++++++++++++++ tests/Makefile | 7 +- tests/README.md | 23 +++ tests/test-lz4-versions.py | 154 +++++++++++++++++ versionsTest/Makefile | 40 ----- versionsTest/test-lz4-versions.py | 147 ----------------- 6 files changed, 522 insertions(+), 188 deletions(-) create mode 100644 tests/COPYING create mode 100644 tests/README.md create mode 100644 tests/test-lz4-versions.py delete mode 100644 versionsTest/Makefile delete mode 100644 versionsTest/test-lz4-versions.py diff --git a/tests/COPYING b/tests/COPYING new file mode 100644 index 0000000..d159169 --- /dev/null +++ b/tests/COPYING @@ -0,0 +1,339 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. diff --git a/tests/Makefile b/tests/Makefile index 24cb37a..824f7f3 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -1,4 +1,3 @@ - # ########################################################################## # LZ4 programs - Makefile # Copyright (C) Yann Collet 2011-2016 @@ -38,6 +37,8 @@ BINDIR := $(PREFIX)/bin MANDIR := $(PREFIX)/share/man/man1 LZ4DIR := ../lib PRGDIR := ../programs +TESTDIR := lz4test +PYTHON?= python3 CFLAGS ?= -O3 # can select custom flags. For example : CFLAGS="-O2 -g" make CFLAGS += -Wall -Wextra -Wundef -Wcast-qual -Wcast-align -Wshadow -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes @@ -107,8 +108,12 @@ clean: fuzzer$(EXT) fuzzer32$(EXT) \ frametest$(EXT) frametest32$(EXT) \ datagen$(EXT) + @rm -fR $(TESTDIR) @echo Cleaning completed +versionsTest: + $(PYTHON) test-lz4-versions.py + #------------------------------------------------------------------------ #make install is validated only for Linux, OSX, kFreeBSD, Hurd and diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..9070317 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,23 @@ +Programs and scripts for automated testing of LZ4 +======================================================= + +This directory contains the following programs and scripts: +- `datagen` : Synthetic and parametrable data generator, for tests +- `frametest` : Test tool that checks lz4frame integrity on target platform +- `fullbench` : Precisely measure speed for each lz4 inner functions +- `fuzzer` : Test tool, to check lz4 integrity on target platform +- `test-lz4-versions.py` : compatibility test between lz4 versions stored on Github + + +#### `test-lz4-versions.py` - script for testing lz4 interoperability between versions + +This script creates `versionsTest` directory to which lz4 repository is cloned. +Then all taged (released) versions of lz4 are compiled. +In the following step interoperability between lz4 versions is checked. + + +#### License + +All files in this directory are licensed under GPL-v2. +See [COPYING](COPYING) for details. +The text of the license is also included at the top of each source file. diff --git a/tests/test-lz4-versions.py b/tests/test-lz4-versions.py new file mode 100644 index 0000000..54b7e13 --- /dev/null +++ b/tests/test-lz4-versions.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python3 +"""Test LZ4 interoperability between versions""" + +# +# Copyright (C) 2011-present, Takayuki Matsuoka +# All rights reserved. +# GPL v2 License +# + +import glob +import subprocess +import filecmp +import os +import shutil +import sys +import hashlib + +repo_url = 'https://github.com/lz4/lz4.git' +tmp_dir_name = 'tests/versionsTest' +make_cmd = 'make' +git_cmd = 'git' +test_dat_src = 'README.md' +test_dat = 'test_dat' +head = 'r999' + +def proc(cmd_args, pipe=True, dummy=False): + if dummy: + return + if pipe: + subproc = subprocess.Popen(cmd_args, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + else: + subproc = subprocess.Popen(cmd_args) + return subproc.communicate() + +def make(args, pipe=True): + return proc([make_cmd] + args, pipe) + +def git(args, pipe=True): + return proc([git_cmd] + args, pipe) + +def get_git_tags(): + stdout, stderr = git(['tag', '-l', 'r[0-9][0-9][0-9]']) + tags = stdout.decode('utf-8').split() + return tags + +# https://stackoverflow.com/a/19711609/2132223 +def sha1_of_file(filepath): + with open(filepath, 'rb') as f: + return hashlib.sha1(f.read()).hexdigest() + +if __name__ == '__main__': + error_code = 0 + base_dir = os.getcwd() + '/..' # /path/to/lz4 + tmp_dir = base_dir + '/' + tmp_dir_name # /path/to/lz4/tests/versionsTest + clone_dir = tmp_dir + '/' + 'lz4' # /path/to/lz4/tests/versionsTest/lz4 + programs_dir = base_dir + '/programs' # /path/to/lz4/programs + os.makedirs(tmp_dir, exist_ok=True) + + # since Travis clones limited depth, we should clone full repository + if not os.path.isdir(clone_dir): + git(['clone', repo_url, clone_dir]) + + shutil.copy2(base_dir + '/' + test_dat_src, tmp_dir + '/' + test_dat) + + # Retrieve all release tags + print('Retrieve all release tags :') + os.chdir(clone_dir) + tags = [head] + get_git_tags() + print(tags); + + # Build all release lz4c and lz4c32 + for tag in tags: + os.chdir(base_dir) + dst_lz4c = '{}/lz4c.{}' .format(tmp_dir, tag) # /path/to/lz4/test/lz4test/lz4c. + dst_lz4c32 = '{}/lz4c32.{}'.format(tmp_dir, tag) # /path/to/lz4/test/lz4test/lz4c32. + if not os.path.isfile(dst_lz4c) or not os.path.isfile(dst_lz4c32) or tag == head: + if tag != head: + r_dir = '{}/{}'.format(tmp_dir, tag) # /path/to/lz4/test/lz4test/ + os.makedirs(r_dir, exist_ok=True) + os.chdir(clone_dir) + git(['--work-tree=' + r_dir, 'checkout', tag, '--', '.'], False) + os.chdir(r_dir + '/programs') # /path/to/lz4/lz4test//programs + make(['clean', 'lz4c', 'lz4c32'], False) + else: + os.chdir(programs_dir) + make(['lz4c', 'lz4c32'], False) + shutil.copy2('lz4c', dst_lz4c) + shutil.copy2('lz4c32', dst_lz4c32) + + # Compress test.dat by all released lz4c and lz4c32 + print('Compress test.dat by all released lz4c and lz4c32') + os.chdir(tmp_dir) + for lz4 in glob.glob("*.lz4"): + os.remove(lz4) + for tag in tags: + proc(['./lz4c.' + tag, '-1fz', test_dat, test_dat + '_1_64_' + tag + '.lz4']) + proc(['./lz4c.' + tag, '-9fz', test_dat, test_dat + '_9_64_' + tag + '.lz4']) + proc(['./lz4c32.' + tag, '-1fz', test_dat, test_dat + '_1_32_' + tag + '.lz4']) + proc(['./lz4c32.' + tag, '-9fz', test_dat, test_dat + '_9_32_' + tag + '.lz4']) + + print('Full list of compressed files') + lz4s = sorted(glob.glob('*.lz4')) + for lz4 in lz4s: + print(lz4 + ' : ' + repr(os.path.getsize(lz4))) + + # Remove duplicated .lz4 files + print('') + print('Duplicated files') + lz4s = sorted(glob.glob('*.lz4')) + for i, lz4 in enumerate(lz4s): + if not os.path.isfile(lz4): + continue + for j in range(i+1, len(lz4s)): + lz4t = lz4s[j] + if not os.path.isfile(lz4t): + continue + if filecmp.cmp(lz4, lz4t): + os.remove(lz4t) + print('{} == {}'.format(lz4, lz4t)) + + print('Enumerate only different compressed files') + lz4s = sorted(glob.glob('*.lz4')) + for lz4 in lz4s: + print(lz4 + ' : ' + repr(os.path.getsize(lz4)) + ', ' + sha1_of_file(lz4)) + + # Decompress remained .lz4 files by all released lz4c and lz4c32 + print('Decompression tests and verifications') + lz4s = sorted(glob.glob('*.lz4')) + for dec in glob.glob("*.dec"): + os.remove(dec) + for lz4 in lz4s: + print(lz4, end=" ") + for tag in tags: + print(tag, end=" ") + proc(['./lz4c.' + tag, '-df', lz4, lz4 + '_d64_' + tag + '.dec']) + proc(['./lz4c32.' + tag, '-df', lz4, lz4 + '_d32_' + tag + '.dec']) + print(' OK') # well, here, decompression has worked; but file is not yet verified + + # Compare all '.dec' files with test_dat + decs = glob.glob('*.dec') + for dec in decs: + if not filecmp.cmp(dec, test_dat): + print('ERR : ' + dec) + error_code = 1 + else: + print('OK : ' + dec) + os.remove(dec) + + if error_code != 0: + print('ERROR') + + sys.exit(error_code) diff --git a/versionsTest/Makefile b/versionsTest/Makefile deleted file mode 100644 index 2ce719a..0000000 --- a/versionsTest/Makefile +++ /dev/null @@ -1,40 +0,0 @@ -# ########################################################################## -# LZ4 tests - Makefile -# Copyright (C) Takayuki Matsuoka - Yann Collet 2011-2015 -# -# GPL v2 License -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -# -# You can contact the author at : -# - LZ4 source repository : https://github.com/Cyan4973/lz4 -# - LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c -# ########################################################################## -# versionstest : Compatibility test between LZ4 versions stored on Github (r116+) -# ########################################################################## - -PYTHON?= python3 -TESTDIR := lz4test - -default: all - -all: versionsTest - -versionsTest: - $(PYTHON) test-lz4-versions.py - -clean: - @rm -fR $(TESTDIR) - @echo Cleaning completed diff --git a/versionsTest/test-lz4-versions.py b/versionsTest/test-lz4-versions.py deleted file mode 100644 index ec2b7b2..0000000 --- a/versionsTest/test-lz4-versions.py +++ /dev/null @@ -1,147 +0,0 @@ -#!/usr/bin/env python3 - -import glob -import subprocess -import filecmp -import os -import shutil -import sys -import hashlib - -repo_url = 'https://github.com/Cyan4973/lz4.git' -tmp_dir_name = 'versionsTest/lz4test' -make_cmd = 'make' -git_cmd = 'git' -test_dat_src = 'README.md' -test_dat = 'test_dat' -head = 'r999' - -def proc(cmd_args, pipe=True, dummy=False): - if dummy: - return - if pipe: - subproc = subprocess.Popen(cmd_args, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - else: - subproc = subprocess.Popen(cmd_args) - return subproc.communicate() - -def make(args, pipe=True): - return proc([make_cmd] + args, pipe) - -def git(args, pipe=True): - return proc([git_cmd] + args, pipe) - -def get_git_tags(): - stdout, stderr = git(['tag', '-l', 'r[0-9][0-9][0-9]']) - tags = stdout.decode('utf-8').split() - return tags - -# https://stackoverflow.com/a/19711609/2132223 -def sha1_of_file(filepath): - with open(filepath, 'rb') as f: - return hashlib.sha1(f.read()).hexdigest() - -if __name__ == '__main__': - error_code = 0 - base_dir = os.getcwd() + '/..' # /path/to/lz4 - tmp_dir = base_dir + '/' + tmp_dir_name # /path/to/lz4/versionsTest/lz4test - clone_dir = tmp_dir + '/' + 'lz4' # /path/to/lz4/versionsTest/lz4test/lz4 - programs_dir = base_dir + '/programs' # /path/to/lz4/programs - os.makedirs(tmp_dir, exist_ok=True) - - # since Travis clones limited depth, we should clone full repository - if not os.path.isdir(clone_dir): - git(['clone', repo_url, clone_dir]) - - shutil.copy2(base_dir + '/' + test_dat_src, tmp_dir + '/' + test_dat) - - # Retrieve all release tags - print('Retrieve all release tags :') - os.chdir(clone_dir) - tags = [head] + get_git_tags() - print(tags); - - # Build all release lz4c and lz4c32 - for tag in tags: - os.chdir(base_dir) - dst_lz4c = '{}/lz4c.{}' .format(tmp_dir, tag) # /path/to/lz4/test/lz4test/lz4c. - dst_lz4c32 = '{}/lz4c32.{}'.format(tmp_dir, tag) # /path/to/lz4/test/lz4test/lz4c32. - if not os.path.isfile(dst_lz4c) or not os.path.isfile(dst_lz4c32) or tag == head: - if tag != head: - r_dir = '{}/{}'.format(tmp_dir, tag) # /path/to/lz4/test/lz4test/ - os.makedirs(r_dir, exist_ok=True) - os.chdir(clone_dir) - git(['--work-tree=' + r_dir, 'checkout', tag, '--', '.'], False) - os.chdir(r_dir + '/programs') # /path/to/lz4/lz4test//programs - make(['clean', 'lz4c', 'lz4c32'], False) - else: - os.chdir(programs_dir) - make(['lz4c', 'lz4c32'], False) - shutil.copy2('lz4c', dst_lz4c) - shutil.copy2('lz4c32', dst_lz4c32) - - # Compress test.dat by all released lz4c and lz4c32 - print('Compress test.dat by all released lz4c and lz4c32') - os.chdir(tmp_dir) - for lz4 in glob.glob("*.lz4"): - os.remove(lz4) - for tag in tags: - proc(['./lz4c.' + tag, '-1fz', test_dat, test_dat + '_1_64_' + tag + '.lz4']) - proc(['./lz4c.' + tag, '-9fz', test_dat, test_dat + '_9_64_' + tag + '.lz4']) - proc(['./lz4c32.' + tag, '-1fz', test_dat, test_dat + '_1_32_' + tag + '.lz4']) - proc(['./lz4c32.' + tag, '-9fz', test_dat, test_dat + '_9_32_' + tag + '.lz4']) - - print('Full list of compressed files') - lz4s = sorted(glob.glob('*.lz4')) - for lz4 in lz4s: - print(lz4 + ' : ' + repr(os.path.getsize(lz4))) - - # Remove duplicated .lz4 files - print('') - print('Duplicated files') - lz4s = sorted(glob.glob('*.lz4')) - for i, lz4 in enumerate(lz4s): - if not os.path.isfile(lz4): - continue - for j in range(i+1, len(lz4s)): - lz4t = lz4s[j] - if not os.path.isfile(lz4t): - continue - if filecmp.cmp(lz4, lz4t): - os.remove(lz4t) - print('{} == {}'.format(lz4, lz4t)) - - print('Enumerate only different compressed files') - lz4s = sorted(glob.glob('*.lz4')) - for lz4 in lz4s: - print(lz4 + ' : ' + repr(os.path.getsize(lz4)) + ', ' + sha1_of_file(lz4)) - - # Decompress remained .lz4 files by all released lz4c and lz4c32 - print('Decompression tests and verifications') - lz4s = sorted(glob.glob('*.lz4')) - for dec in glob.glob("*.dec"): - os.remove(dec) - for lz4 in lz4s: - print(lz4, end=" ") - for tag in tags: - print(tag, end=" ") - proc(['./lz4c.' + tag, '-df', lz4, lz4 + '_d64_' + tag + '.dec']) - proc(['./lz4c32.' + tag, '-df', lz4, lz4 + '_d32_' + tag + '.dec']) - print(' OK') # well, here, decompression has worked; but file is not yet verified - - # Compare all '.dec' files with test_dat - decs = glob.glob('*.dec') - for dec in decs: - if not filecmp.cmp(dec, test_dat): - print('ERR : ' + dec) - error_code = 1 - else: - print('OK : ' + dec) - os.remove(dec) - - if error_code != 0: - print('ERROR') - - sys.exit(error_code) -- cgit v0.12 From 8847faffbf78ad7166826a56fb15902fca0e2603 Mon Sep 17 00:00:00 2001 From: Przemyslaw Skibinski Date: Thu, 3 Nov 2016 16:17:38 +0100 Subject: updated programs/README.md --- Makefile | 3 +-- programs/README.md | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++--- tests/Makefile | 2 +- 3 files changed, 66 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 3df03f7..6997b19 100644 --- a/Makefile +++ b/Makefile @@ -68,7 +68,6 @@ clean: @$(MAKE) -C $(TESTDIR) $@ > $(VOID) @$(MAKE) -C $(LZ4DIR) $@ > $(VOID) @$(MAKE) -C examples $@ > $(VOID) - @$(MAKE) -C versionsTest $@ > $(VOID) @$(RM) lz4$(EXT) @echo Cleaning completed @@ -113,7 +112,7 @@ armtest: clean CFLAGS="-O3 -Werror" $(MAKE) -C $(TESTDIR) bins CC=arm-linux-gnueabi-gcc versionsTest: clean - $(MAKE) -C versionsTest + $(MAKE) -C $(TESTDIR) $@ examples: $(MAKE) -C $(LZ4DIR) diff --git a/programs/README.md b/programs/README.md index a952c2f..7a5d2e2 100644 --- a/programs/README.md +++ b/programs/README.md @@ -1,6 +1,67 @@ -This directory `programs` contains source files -to generate `lz4` command line utility, -and many test tools, such as `fuzzer`, `fullbench` and such. +Command Line Interface for LZ4 library +============================================ + +Command Line Interface (CLI) can be created using the `make` command without any additional parameters. +There are however other Makefile targets that create different variations of CLI: +- `lz4` : default CLI supporting gzip-like arguments +- `lz4c` : Same as `lz4` with additional support for decompression of legacy lz4 versions +- `lz4c32` : Same as `lz4c`, but forced to compile in 32-bits mode + + +#### Aggregation of parameters +CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`. + + + +#### Benchmark in Command Line Interface +CLI includes in-memory compression benchmark module for lz4. +The benchmark is conducted using a given filename. +The file is read into memory. +It makes benchmark more precise as it eliminates I/O overhead. + +The benchmark measures ratio, compressed size, compression and decompression speed. +One can select compression levels starting from `-b` and ending with `-e`. +The `-i` parameter selects a number of iterations used for each of tested levels. + + + +#### Usage of Command Line Interface +The full list of options can be obtained with `-h` or `-H` parameter: +``` +Usage : + lz4 [arg] [input] [output] + +input : a filename + with no FILE, or when FILE is - or stdin, read standard input +Arguments : + -1 : Fast compression (default) + -9 : High compression + -d : decompression (default for .lz4 extension) + -z : force compression + -f : overwrite output without prompting + -h/-H : display help/long help and exit + +Advanced arguments : + -V : display Version number and exit + -v : verbose mode + -q : suppress warnings; specify twice to suppress errors too + -c : force write to standard output, even if it is the console + -t : test compressed file integrity + -m : multiple input files (implies automatic output filenames) + -l : compress using Legacy format (Linux kernel compression) + -B# : Block size [4-7](default : 7) + -BD : Block dependency (improve compression ratio) +--no-frame-crc : disable stream checksum (default:enabled) +--content-size : compressed frame includes original size (default:not present) +--[no-]sparse : sparse mode (default:enabled on file, disabled on stdout) +Benchmark arguments : +Benchmark arguments : + -b# : benchmark file(s), using # compression level (default : 1) + -e# : test all compression levels from -bX to # (default: 1) + -i# : iteration loops [1-9](default : 3), benchmark mode only + ``` + +#### License All files in this directory are licensed under GPL-v2. See [COPYING](COPYING) for details. diff --git a/tests/Makefile b/tests/Makefile index 824f7f3..d7a195a 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -37,7 +37,7 @@ BINDIR := $(PREFIX)/bin MANDIR := $(PREFIX)/share/man/man1 LZ4DIR := ../lib PRGDIR := ../programs -TESTDIR := lz4test +TESTDIR := versionsTest PYTHON?= python3 CFLAGS ?= -O3 # can select custom flags. For example : CFLAGS="-O2 -g" make -- cgit v0.12