Skip to content

Commit

Permalink
zstd: Fix corrupted output in "best" (#876)
Browse files Browse the repository at this point in the history
* zstd: Fix corrupted output in "best"

Regression from #784 and followup #793

Fixes #875

A 0 offset backreference was possible when "improve" was successful twice in a row in the "skipBeginning" part, only finding 2 (previously unmatches) length 4 matches, but where start offset decreased by 2 in both cases.

This would result in output where the end offset would equal to the next 's', thereby doing a self-reference.

Add a general check in "improve" and just reject these. Will also guard against similar issues in the future.

This also hints at some potentially suboptimal hash indexing - but I will take that improvement separately.

Fuzz test set updated.
  • Loading branch information
klauspost authored Oct 22, 2023
1 parent 68eabfa commit 8eb6542
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 5 deletions.
11 changes: 6 additions & 5 deletions zstd/enc_best.go
Original file line number Diff line number Diff line change
Expand Up @@ -197,12 +197,13 @@ encodeLoop:

// Set m to a match at offset if it looks like that will improve compression.
improve := func(m *match, offset int32, s int32, first uint32, rep int32) {
if s-offset >= e.maxMatchOff || load3232(src, offset) != first {
delta := s - offset
if delta >= e.maxMatchOff || delta <= 0 || load3232(src, offset) != first {
return
}
if debugAsserts {
if offset <= 0 {
panic(offset)
if offset >= s {
panic(fmt.Sprintf("offset: %d - s:%d - rep: %d - cur :%d - max: %d", offset, s, rep, e.cur, e.maxMatchOff))
}
if !bytes.Equal(src[s:s+4], src[offset:offset+4]) {
panic(fmt.Sprintf("first match mismatch: %v != %v, first: %08x", src[s:s+4], src[offset:offset+4], first))
Expand Down Expand Up @@ -343,8 +344,8 @@ encodeLoop:
if best.rep > 0 {
var seq seq
seq.matchLen = uint32(best.length - zstdMinMatch)
if debugAsserts && s <= nextEmit {
panic("s <= nextEmit")
if debugAsserts && s < nextEmit {
panic("s < nextEmit")
}
addLiterals(&seq, best.s)

Expand Down
Binary file modified zstd/testdata/fuzz/encode-corpus-raw.zip
Binary file not shown.

0 comments on commit 8eb6542

Please sign in to comment.