Compression

● Intermediate4 minBefore this: BytesSpec 14.21

std.compress makes bytes smaller and whole again, in gzip, deflate, zstd, brotli and lz4: the formats web servers, packages and games exchange.

use std.compress

fn main() ! Error {
    const page = "<li>an item of the list</li>\n".repeat(100).bytes()
    print("plain: {page.len()} bytes")
    for format in [Format.Gzip, Format.Zstd, Format.Brotli, Format.Lz4] {
        const packed = compress.encode(page, format)
        const back = try compress.decode(packed, format)
        print("{format.name()}: {packed.len()} bytes, same when decoded: {back == page}")
    }
}

kavach run round-trip.kav

plain: 2900 bytesgzip: 73 bytes, same when decoded: truezstd: 47 bytes, same when decoded: truebr: 37 bytes, same when decoded: truelz4: 67 bytes, same when decoded: true

encode packs Bytes in a Format; decode gives back the same bytes.
It needs no capability: it changes nothing outside the program.
The formats come from Rust libraries, built into the runtime.

Levels

Each format has its own levels: a higher one is smaller and slower. Leave level out for the format's usual one:

`Format`	Levels	Usual level
`Gzip`, `Deflate`	0 (stored, not compressed) to 9	6
`Zstd`	1 to 22	3
`Brotli`	0 to 11	5
`Lz4`	one level: `level` must be `none` or 0	0

A level outside its format's range is a mistake in the program, not in the data, so it breaks encode's contract and panics. compress.level_fits checks one first:

use std.compress

fn main() {
    const text = "hello, hello, hello".bytes()
    print(compress.encode(text, Format.Zstd, level: 19).len())
    print(compress.level_fits(Format.Zstd, 30))
    print(compress.encode(text, Format.Zstd, level: 30).len())
}

kavach run level.kav

23falsepanic: contract failed: require level_fits(format, level)

The format a header names

HTTP names formats in headers such as accept-encoding. compress.from_name reads such a name, in any case, and format.name() writes one:

use std.compress

fn main() {
    for asked in ["br", "GZIP", "x-gzip", "compress"] {
        if let format = compress.from_name(asked) {
            print("{asked} -> {format}, sent back as {format.name()}")
        } else {
            print("{asked} -> not a format we know")
        }
    }
}

kavach run header.kav

br -> Brotli, sent back as brGZIP -> Gzip, sent back as gzipx-gzip -> Gzip, sent back as gzipcompress -> not a format we know

Decoding stops at most

A small input can expand to gigabytes, a "zip bomb". So decode gives at most most bytes, 64 MB unless the call says otherwise, and fails with TooLarge past it instead of filling memory. Data that isn't in the format, or was cut short, fails with Corrupt:

use std.compress

fn unpack(data: Bytes) -> String {
    const body = compress.decode(data, Format.Gzip, most: 1_000) catch e => match e {
        CompressError.TooLarge(most) => return "refused: more than {most} bytes",
        CompressError.Corrupt(why) => return "not gzip: {why}",
    }
    "{body.len()} bytes"
}

fn main() {
    const small = compress.encode("hello".bytes(), Format.Gzip)
    const bomb = compress.encode("0".repeat(1_000_000).bytes(), Format.Gzip)
    print("bomb: {bomb.len()} bytes packed")
    print(unpack(small))
    print(unpack(bomb))
    print(unpack("hello".bytes()))
}

kavach run most.kav

bomb: 1004 bytes packed5 bytesrefused: more than 1000 bytesnot gzip: unexpected end of file

`CompressError`	When
`TooLarge(most: Int)`	The data holds more than `most` bytes
`Corrupt(reason: String)`	The data isn't in `format`, or was cut short

Every function

Function	Gives
`compress.encode(data, format, level: none) -> Bytes`	`data` (`Bytes`) compressed in `format`, at `level`, or the format's usual level when `none`
`compress.decode(data, format, most: 64_000_000) -> Bytes ! CompressError`	The bytes `data` holds, which must be in `format`; at most `most` of them
`compress.level_fits(format, level) -> Bool`	Whether `level` is one of `format`'s levels (below); `none` always is
`compress.from_name(name) -> Format?`	The format a name in an HTTP header stands for: `gzip` (or `x-gzip`), `deflate`, `zstd`, `br` or `lz4`, in any case; `none` for any other
`format.name() -> String`	The format's name in a header: `Format.Brotli.name()` is `br`

spec 14.21 · std.compress tested most, tested levels, tested decision 0103