Binary file seems to be corrupted with method Deflated

[mass10]

Summary

Binary file seems to be corrupted with method zip::CompressionMethod::Deflated.

Code to reproduce

use std::io::{Read, Write};

/// Example of deflate.
fn main() -> Result<(), Box<dyn std::error::Error>> {
	// ARCHIVE
	{
		// START
		let w = std::fs::File::create("notepad.exe.zip")?;
		let mut archiver = zip::ZipWriter::new(w);

		let options = zip::write::FileOptions::default();
		// let options = options.compression_method(zip::CompressionMethod::Stored); // SAFE
		let options = options.compression_method(zip::CompressionMethod::Deflated); // CORRUPT🔥
		archiver.start_file("notepad.exe", options)?;

		// READ
		let mut stream = std::fs::File::open("C:\\Windows\\system32\\notepad.exe")?;
		loop {
			let mut buffer = [0; 1000];
			let bytes_read = stream.read(&mut buffer)?;
			if bytes_read == 0 {
				break;
			}

			// WRITE
			let write_buffer = &buffer[..bytes_read];
			archiver.write(&write_buffer)?;
		}

		// COMPLETE
		archiver.finish().unwrap();
	}

	// UNZIP
	{
		let command = std::process::Command::new("wsl.exe").args(&["unzip", "notepad.exe.zip"]).spawn()?.wait()?;
		println!("Command exited with: {}", command);
	}

	// ORIGINAL FILE SIZE
	{
		let meta = std::fs::metadata("C:\\Windows\\system32\\notepad.exe")?;
		println!("Original NOTEPAD is: {} bytes.", meta.len());
	}

	// EXTRACTED FILE SIZE
	{
		let meta = std::fs::metadata("notepad.exe")?;
		println!("Extracted NOTEPAD is: {} bytes.", meta.len());
	}

	return Ok(());
}

Result is

Original NOTEPAD is: 201216 bytes.
Extracted NOTEPAD is: 200903 bytes.

Thank you!

[mass10]

zip::CompressionMethod::Zstd sounds good.

[mass10]

But I found neither of the following supports ZStandard:

• 7-zip
• explorer.exe
• unzip

[milesj]

Just ran into this also. Have to use Stored for now...

[axnsan12]

You are using archiver.write (an implementation of std::io::Write::write) and ignoring the return value. You should be calling write in a loop checking the count of bytes written, or use archiver.write_all.

https://doc.rust-lang.org/stable/std/io/trait.Write.html#tymethod.write

This function will attempt to write the entire contents of buf, but the entire write might not succeed, or the write may also generate an error. A call to write represents at most one attempt to write to any wrapped object.

[mass10]

Oh, It was fixed!

Thank you!

[milesj]

I'm using write_all and it still fails, here's my impl: https://github.com/moonrepo/starbase/blob/master/crates/archive/src/zip.rs

[mass10]

@milesj I thought I want to use fs::read_file_bytes() instead of fs::read_file() in your code.

[milesj]

When I try that I get this error "stream did not contain valid UTF-8", which is the same as using fs::read_file(file)?.as_bytes().

Both are interesting, since these test cases are very simple.

I also tried this, which failed with the same error.

let mut buffer = Vec::new();
let mut stream = fs::open_file(file)?;
stream.read_to_end(&mut buffer).unwrap();

I feel like something else may be going on.

[mass10]

I tried this.

/// Succeeds🌔
fn diagnose_1(path: &str) {
    let data = std::fs::read(path).unwrap();
    println!("(1) {} is {} bytes.", path, data.len());
}

/// Succeeds🌔
fn diagnose_2(path: &str) {
    use std::io::Read;
    let mut file = std::fs::File::open(path).unwrap();
    let mut buffer = Vec::new();
    file.read_to_end(&mut buffer).unwrap();
    println!("(2) {} is {} bytes.", path, buffer.len());
}

/// Fails🌒
fn diagnose_3(path: &str) {
    let text = std::fs::read_to_string(path).unwrap(); // 🔥stream did not contain valid UTF-8
    let data = text.as_bytes();
    println!("(3) {} is {} bytes.", path, data.len());
}

pub fn main() {
    diagnose_1(r"C:\Windows\system32\notepad.exe"); //🌔
    diagnose_2(r"C:\Windows\system32\notepad.exe"); //🌔
    diagnose_3(r"C:\Windows\system32\notepad.exe"); //🌒
}

Result was

(1) C:\Windows\system32\notepad.exe is 201216 bytes.
(2) C:\Windows\system32\notepad.exe is 201216 bytes.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: InvalidData, message: "stream did not contain valid UTF-8" }', src\test2.rs:18:46

[milesj]

It's weird, because even if I do this it fails:

self.archive
    .start_file(name, options)
    .map_err(|error| ZipError::AddFailure {
        source: file.to_path_buf(),
        error,
    })?;

self.archive
    .write_all(&std::fs::read(file).unwrap())
    .map_err(|error| FsError::Write {
        path: file.to_path_buf(),
        error,
    })?;

I wonder if this abstraction is causing issues once compiled.

Example PR: moonrepo/starbase#33

[mass10]

@milesj What kind of file is broken or fail？

[milesj]

Not sure which file is the problem, but this is the fixture: https://github.com/moonrepo/starbase/tree/master/crates/archive/tests/__fixtures__/archives

They are very simple files.

[milesj]

I figured out the problem, was my mistake. The issue was I was using by_index_raw instead of by_index when unzipping.

I'm using Copilot and I think it generated that 😞

[mass10]

That's good! Everything has become clear.