deadly data example

[fufuok]

package main

import (
	"bytes"
	"compress/zlib"
	"encoding/hex"
	"fmt"
	"log"

	"github.com/4kills/go-libdeflate/v2"
	czlib "github.com/4kills/go-zlib"
)

var (
	s = "789c7d90316f83301085f7fc0ac45cac3b1b63cc0685542c55a4264b1784825bd1024686284851fe7b8104c454c" +
		"b8bdff7de9def6e3b6b3cf6db298dedc0b26f02384014858efb2af78ecb1573c224e24e92301e4919d158c0dd7e7" +
		"9a40a5de765f39ed76aca2e6a364caf5a17cacdbe2a7dcd8c6ab5e9d790eafab49d2ce87ac4f308724a5c5c70553" +
		"6bfe1b93f968fa214287570bcfc882c6034a0fc736bfde873f38f99aee666fccfb3ad0f043d4604122917dc1abdc" +
		"c6178deaa81a8416d60afcfba9ae8293e2cf2a5536609753f04194881be106c1de6d2eec7058c9c3180cdfc87691" +
		"f814505305c77796d9e66042e372dd2622ec001c143ff09bec7bea151f90c05f3e916cc22f57cc976f73fc74374"
	comp, _ = hex.DecodeString(s)
)

func main() {
	deWithZlib()       // nice, return error
	deWithLibdeflate() // hang, endless loop
	deWithCgoZlib()    // hang, endless loop
}

func deWithLibdeflate() {
	dc, _ := libdeflate.NewDecompressor()
	_, decompressed, err := dc.Decompress(comp, nil, libdeflate.ModeZlib)
	if err != nil {
		log.Println("deWithLibdeflate", err)
		return
	}
	fmt.Println(decompressed)
	dc.Close()
}

func deWithCgoZlib() {
	dc, _ := czlib.NewReader(nil)
	_, decompressed, err := dc.ReadBuffer(comp, nil)
	if err != nil {
		log.Println("deWithCgoZlib", err)
		return
	}
	fmt.Println(decompressed)
	_ = dc.Close()
}

func deWithZlib() {
	var decompressed bytes.Buffer
	dc, _ := zlib.NewReader(bytes.NewReader(comp))
	_, err := decompressed.ReadFrom(dc)
	if err != nil {
		log.Println("deWithZlib", err) // unexpected EOF
		return
	}
	fmt.Println(decompressed)
	_ = dc.Close()
}

[4kills]

Thank you for your issue. I am very sorry you had to experience this. I have a hunch about what might be the issue. I will look into it tonight.
Can you please provide the exact error message you get from compress/zlib and the Go version used?

[fufuok]

root@DevBeta:~/go/zlib# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal
root@DevBeta:~/go/zlib# go version
go version go1.19.4 linux/amd64
root@DevBeta:~/go/zlib# cat go.mod
module gozlib

go 1.17

require (
        github.com/4kills/go-libdeflate/v2 v2.0.2
        github.com/4kills/go-zlib v1.1.1
)
root@DevBeta:~/go/zlib# go build
root@DevBeta:~/go/zlib# ./gozlib 
2022/12/15 17:15:14 deWithZlib unexpected EOF
////// No response //////

top

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                           
 448656 root      20   0 1298452  33352   2532 R 108.6   0.4   0:55.42 gozlib  

[4kills]

@fufuok What did you provide as deadly data exactly?
I have indentified the issue which is that the program keeps allocating more memory for storing the uncompressed data since the deflate algorithm has not enough output space.

[4kills]

The underlying problem is within the C library (libdeflate by ebiggers) used by this project. The C library does not give a more nuanced error message, so I will probably open an issue over there. Tomorrow I will provide a temporary fix until the issue with the C library is resolved

[fufuok]

Maybe the title is a bit misleading, sorry for the trouble.

What I mean is: an illegal data will cause the program to not work properly.

The above sample data is obtained in our production, resulting in high memory usage. After investigation, it was found that it was caused by go-zlib, so I made this demo for you.

Thanks for your quick response, hope it gets better and better.

[4kills]

I have hot fixed it for now. It should work again. I will also create an issue over at the libdeflate c library to figure out the root cause.
@fufuok Is the input you provided trusted? If this was provided by a user this could potentially be a DOS type attack at your systems.

[ebiggers]

Please just update to the latest version of libdeflate, instead of implementing workarounds for issues in older versions.

Also be aware that DEFLATE streams can legitimately decompress to up to 1032x their original size. Decompressing with an unknown output size with no limit should be avoided when possible.