| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
| |
Otherwise, the output from decoding LZ4-compressed input could be
platform dependent.
Also add a compile-time check to confirm the existing code's assumptions
that, if <stdint.h> isn't used, then sizeof(int) == 4.
Updates #792
|
|
|
|
|
|
|
|
|
|
| |
PR#756 fixed the data corruption bug, but didn't clear `ip`. PR#760
fixed that off-by-one error, but missed the case where `ip == filledIp`,
which is harder for the fuzzers to find (it took 20 days not 1 day).
Verified this fixed the issue reported by OSS-Fuzz.
Credit to OSS-Fuzz.
|
|
|
|
|
| |
We do want to bump, even if the dictionary is empty, but we **don't** want to
bump if the dictionary is null.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When using an empty dictionary, we bail out of loading or attaching it in
ways that leave the working context in potentially slightly different states.
In particular, in some paths, we will cause the currentOffset to be non-zero,
while in others we would allow it to remain 0.
This difference in behavior is perfectly harmless, but in some situations, it
can produce slight differences in the compressed output. For sanity's sake,
we currently try to maintain a strict correspondence between the behavior of
the dict attachment and the dict loading paths. This patch restores them to
behaving identically.
This shouldn't have any negative side-effects, as far as I can tell. When
writing the dict attachment code, I tried to preserve zeroed currentOffsets
when possible, since they benchmarked as very slightly faster. However, the
case of attaching an empty dictionary is probably rare enought that it's
acceptable to minisculely degrade performance in that corner case.
|
|\
| |
| | |
silence msan warning when offset==0
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
When the match is very long and found quickly, we can do
matchLength * nbCompares iterations through the chain
swapping, which can really slow down compression.
|
| |
| |
| |
| |
| |
| | |
The pattern detection in extDict mode could put `matchIndex`
within the last 3 bytes of the dictionary. This would cause
a read out of bounds.
|
| |
| |
| |
| |
| |
| | |
We should be comparing `matchPtr` not `ip`. This bug just means
that this branch was not taken, so we might miss some of the
forward length.
|
|/
|
|
|
|
|
|
|
|
|
|
| |
It is important to continue to look backwards if the current pattern
reaches `lowPrefixPtr`. If the pattern detection doesn't go all the
way to the beginning of the pattern, or the end of the pattern it
slows down the search instead of speeding it up.
The slow unit in `round_trip_stream_fuzzer` used to take 12 seconds
to run with -O3, now it takes 0.2 seconds.
Credit to OSS-Fuzz
|
|
|
|
| |
Fixes #761.
|
|\
| |
| | |
[lz4frame] Fix unused variable warnings in fuzzing mode
|
| | |
|
|\ \
| |/
|/| |
[LZ4_compress_destSize] Fix off-by-one error in fix
|
| |
| |
| |
| |
| |
| |
| | |
The next match is looking at the current ip, not the next ip,
so it needs to be cleared as well.
Credit to OSS-Fuzz
|
| |
| |
| |
| |
| | |
When `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined we skip
magic and checksum checks. This makes it easier to fuzz decompression.
|
| | |
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
| |
This diff fixes an issue in which we failed to clear the `dictCtx` in HC
compression. The `dictCtx` is not supposed to be used when an `extDict` is
present: matches found in the `dictCtx` do not account for the presence of an
`extDict` segment, and their offsets are therefore miscalculated when one is
present. This can lead to data corruption.
This diff clears the `dictCtx` whenever setting an `extDict`.
This issue was uncovered by @terrelln's fuzzing work.
|
| |
|
| |
|
| |
|
|\
| |
| | |
[ossfuzz] Improve the fuzzers
|
| |
| |
| |
| |
| | |
* Partial decoding could read a few bytes beyond the end of the input
* Partial decoding returned an error with an empty output buffer
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It's now possible to select a custom LZ4_DISTANCE_MAX at compile time,
provided it's <= 65535.
However, in some cases (when compressing in byU16 mode),
the new distance wasn't respected,
as it used to implied that it was necessarily within range.
Added a distance check for this case.
Also : added a new TravisCI test which ensures that
custom LZ4_DISTANCE_MAX compiles correctly
and compresses correctly (relying on `assert()` to find outsized offsets).
|
|/
|
|
| |
value of LZ4_DISTANCE_MAX,
|
| |
|
|
|
|
| |
to reduce risks that future bug reports in `dev` branch report `v1.9.1` as the failing version.
|
| |
|
|
|
|
| |
and that such metadata must be provided / sent / saved by the application.
|
| |
|
|\
| |
| | |
Added documentation and macro to support in-place compression and decompression
|
| | |
|
| |
| |
| |
| | |
for compatibility with in-place decompression scenarios.
|
| |
| |
| |
| |
| | |
to pass worst case scenario.
Now adds margin proportional to input size to counter local expansion.
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
order
ensure correct propagation of LZ4_DISTANCE_MAX
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When I did FAST_DEC_LOOP performance test, I found the
offset check is much more than v1.8.3
You will see the condition check difference via lzbench with dickens test case.
v1.8.3 34959
v.1.9.x 1055885
After investigate the code, we could see the difference.
v.1.8.3 SKIP the condition check if
if condition is true in:
https://github.com/lz4/lz4/blob/v1.8.3/lib/lz4.c#L1463
AND below condition is true
https://github.com/lz4/lz4/blob/v1.8.3/lib/lz4.c#L1478\
The offset check should be invoked.
v1.9.3
The offset check code will be invoked in every loop which lead to downgrade.
So the fix would be move this check to specific condition
to avoid useless condition check.
After this change, the call number is same as v1.8.3
|
|/ |
|
|\
| |
| | |
Add multiframe report to --list command
|
| | |
|
|/ |
|
| |
|
| |
|
|\
| |
| | |
Fix test-amalgamation
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Use the list of prerequisites instead of listing those files manually,
this way they will never fall out of sync.
Also update the amalgamation example to use a single cat command.
Fixes: a7e8d394 ("[amalgamation] add test")
Fixes: b192c86b ("[amalgamation] lz4frame.c")
|