diff options
author | Takayuki MATSUOKA <takayuki.matsuoka@gmail.com> | 2015-03-24 22:52:17 (GMT) |
---|---|---|
committer | Takayuki MATSUOKA <takayuki.matsuoka@gmail.com> | 2015-03-24 22:52:17 (GMT) |
commit | 438fee9169d2267f46ddd54f770b5f22a61ebb86 (patch) | |
tree | 48f870c63a986f0a0e830d8ba8ee3463157fd8dd /examples/blockStreaming_doubleBuffer.md | |
parent | a38166bd5261ac8ff93dad8fa87eb6ee3deeea2d (diff) | |
download | lz4-438fee9169d2267f46ddd54f770b5f22a61ebb86.zip lz4-438fee9169d2267f46ddd54f770b5f22a61ebb86.tar.gz lz4-438fee9169d2267f46ddd54f770b5f22a61ebb86.tar.bz2 |
Add document for "Double Buffer" example
Diffstat (limited to 'examples/blockStreaming_doubleBuffer.md')
-rw-r--r-- | examples/blockStreaming_doubleBuffer.md | 99 |
1 files changed, 99 insertions, 0 deletions
diff --git a/examples/blockStreaming_doubleBuffer.md b/examples/blockStreaming_doubleBuffer.md new file mode 100644 index 0000000..cd295df --- /dev/null +++ b/examples/blockStreaming_doubleBuffer.md @@ -0,0 +1,99 @@ +# LZ4 Streaming API Example : Double Buffer + +`blockStreaming_doubleBuffer.c` is LZ4 Straming API example which implements double buffer (de)compression. + +Please note : + + - Firstly, read "LZ4 Streaming API Basics". + - This is relatively advanced application example. + - Output file is not compatible with lz4frame and platform dependent. + + +## What's the point of this example ? + + - Handle huge file in small amount of memory + - Always better compression ratio than Block API + - Uniform block size + + +## How the compression works + +First of all, allocate "Double Buffer" for input and LZ4 compressed data buffer for output. +Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2). + +``` + Double Buffer + + Page#1 Page#2 + +---------+---------+ + | Block#1 | | + +----+----+---------+ + | + v + {Out#1} + + + Prefix Dependency + +---------+ + | | + v | + +---------+----+----+ + | Block#1 | Block#2 | + +---------+----+----+ + | + v + {Out#2} + + + External Dictionary Mode + +---------+ + | | + | v + +----+----+---------+ + | Block#3 | Block#2 | + +----+----+---------+ + | + v + {Out#3} + + + Prefix Dependency + +---------+ + | | + v | + +---------+----+----+ + | Block#3 | Block#4 | + +---------+----+----+ + | + v + {Out#4} +``` + +Next, read first block to double buffer's first page. And compress it by `LZ4_compress_continue()`. +For the first time, LZ4 doesn't know any previous dependencies, +so it just compress the line without dependencies and generates compressed block {Out#1} to LZ4 compressed data buffer. +After that, write {Out#1} to the file. + +Next, read second block to double buffer's second page. And compress it. +In this time, LZ4 can use dependency to Block#1 to improve compression ratio. +This dependency is called "Prefix mode". + +Next, read third block to double buffer's *first* page. And compress it. +Also this time, LZ4 can use dependency to Block#2. +This dependency is called "External Dictonaly mode". + +Continue these procedure to the end of the file. + + +## How the decompression works + +Decompression will do reverse order. + + - Read first compressed block. + - Decompress it to the first page and write that page to the file. + - Read second compressed block. + - Decompress it to the second page and write that page to the file. + - Read third compressed block. + - Decompress it to the *first* page and write that page to the file. + +Continue these procedure to the end of the compressed file. |