Bytes

● Intermediate5 minBefore this: StringsSpec 2.13

Bytes holds raw data: a file's contents, an image, a network message. It is stored compactly, one byte per byte.

fn main() {
    const greeting = "hi".bytes()
    const header = Bytes([137, 80, 78, 71])
    print(greeting.len())
    print(header[1])
    print(greeting)
    print(greeting.text() ?? "?")
    print((greeting + header).len())
}

kavach run bytes.kav

280Bytes(6869)hi6

Written	Gives
`s.bytes()`	The text `s` as UTF-8 `Bytes`
`Bytes(list)`	`Bytes` from a `[UInt8]`
`b.len()`	`Int`: how many bytes
`b.is_empty()`	`Bool`: whether there are none
`b[i]`	`UInt8`: byte `i`; panics when `i` is out of range
`b.get(i)`	`UInt8?`: byte `i`, or `none`
`b.slice(start, end)`	`Bytes` `start..end`; positions past the end are clamped
`b.index_of(part)`	`Int?`: where the bytes `part` first appear in `b`, or `none`; an empty `part` is at 0
`b.crc32c()`	The CRC32C checksum (Castagnoli, as iSCSI, ext4 and gRPC use), an `Int` in `0..2^32`; the processor computes it where it can
`b.xxhash64()`	The XXH64 hash (seed 0), a `UInt64`: a fast check that data didn't change by accident, as storage engines use it; not for passwords or signatures
`b.text()`	`String?`: the bytes as UTF-8 text, or `none` when they aren't valid UTF-8
`b.to_list()`	`[UInt8]`
`b.as_list<T>()`	`[T]?`: the bytes read as numbers of type `T`, or `none` when their length isn't a whole number of them
`b.read<T>(at, big_endian: false)`	`T?`: the number of type `T` whose bytes start at `at`, or `none` when they don't all fit
`Bytes.of<T>(n, big_endian: false)`	`Bytes`: the bytes of the number `n` of type `T`
`a + b`	`a` followed by `b`
`a == b`	Whether they hold the same bytes

Checking data

fn main() {
    const data = "hello, world".bytes()
    print(data.slice(0, 5).text() ?? "?")
    print(data.get(100) ?? 0)
    print(data.crc32c())
    print(data.xxhash64())
    const changed = "hello, World".bytes()
    print(data.crc32c() == changed.crc32c())
}

kavach run check.kav

hello0177167670312914696790927413826false

A checksum changes when the data does, even by one letter, so it shows data that was damaged on the way. get gives none past the end, where b[i] would stop the program.

Reading numbers

Data often holds numbers, such as a chunk of a file. b.as_list<T>() reads them all, without going through the bytes one by one. It gives none when the length isn't a whole number of them:

fn main() {
    const data = Bytes([0, 0, 128, 63, 0, 0, 0, 64])
    print(data.as_list<Float32>() ?? [])
    print(data.as_list<Int>() ?? [])
    print(data.slice(0, 3).as_list<Int32>())

    // A list like any other: it can grow, and be summed.
    let xs = data.as_list<Float32>() ?? []
    xs.push(5)
    print(xs.sum(x => x))
}

kavach run as-list.kav

[1.0, 2.0][4611686019492741120]none8.0

T says how to read the bytes, and how many make one number:

T	Bytes each	Reads	From	Gives
`Int8`	1	A whole number, which can be negative	`Bytes([255])`	`[-1]`
`UInt8`	1	A whole number, never negative	`Bytes([255])`	`[255]`
`Int16`	2	A whole number, which can be negative	`Bytes([255, 255])`	`[-1]`
`UInt16`	2	A whole number, never negative	`Bytes([255, 255])`	`[65535]`
`Int32`	4	A whole number, which can be negative	`Bytes([255, 255, 255, 255])`	`[-1]`
`UInt32`	4	A whole number, never negative	`Bytes([255, 255, 255, 255])`	`[4294967295]`
`Int`	8	A whole number, which can be negative	`Bytes([255, 255, 255, 255, 255, 255, 255, 255])`	`[-1]`
`UInt64`	8	A whole number, never negative	`Bytes([255, 255, 255, 255, 255, 255, 255, 255])`	`[18446744073709551615]`
`Float32`	4	A number with a fraction	`Bytes([0, 0, 128, 63])`	`[1.0]`
`Float`	8	A number with a fraction	`Bytes([0, 0, 0, 0, 0, 0, 240, 63])`	`[1.0]`

The bytes are little-endian, smallest part first. Nearly all files and processors use that order, so data reads the same on every machine.
The list is a copy, made in one go, with no work for each number. Changing the list leaves the bytes as they were.
Any other type is refused, and so is leaving the type out:

fn main() {
    print("hi".bytes().as_list<String>())
}

kavach run as-list-other-type.kav

error: `as_list` reads numbers of a fixed size: `Int8`, `Int16`, `Int32`, `Int`, `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Float32` or `Float`

fn main() {
    print("hi".bytes().as_list())
}

kavach run as-list-no-type.kav

error: write the type of the numbers: `b.as_list<Float32>()`

One number at a place

A file format or a network message puts each number at its own place. b.read<T>(at) reads the one whose bytes start at at, and Bytes.of<T>(n) gives a number's bytes, to build a message with +:

fn main() uses Console {
    // A message: its kind in 2 bytes, big-endian, then a 4-byte count.
    const message = Bytes([0, 6, 0, 0, 1, 0, 0, 0])
    print(message.read<UInt16>(0, big_endian: true) ?? 0)
    print(message.read<Int32>(2) ?? 0)
    print(message.read<UInt32>(6))

    // Build one: each number's bytes, joined with +.
    const reply = Bytes.of<UInt16>(7, big_endian: true) + Bytes.of<Int32>(65536)
    print(reply)
    print(reply.read<Int32>(2) ?? 0)
}

kavach run read-of.kav

665536noneBytes(000700000100)65536

The bytes are little-endian unless big_endian: true, the order network protocols and some file formats use.
A read whose bytes don't all fit gives none, as b.get(i) does past the end. So does a negative at.
T is a number of fixed size, as for as_list:

fn main() uses Console {
    const data = Bytes([1, 2, 3, 4])
    print(data.read<String>(0))
}

kavach run read-other-type.kav

error: `read` reads numbers of a fixed size: `Int8`, `Int16`, `Int32`, `Int`, `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Float32` or `Float`

read, tested Bytes.of, tested fixed-size numbers, tested

spec 2.13 · bytes reading numbers, tested only fixed-size numbers, tested

In JSON

JSON has no bytes, so Bytes is written as base64 text, as Protocol Buffers' JSON and OpenAPI write them. Decoding reads the URL-safe alphabet too, with or without padding:

use std.json

type Blob {
    name: String
    data: Bytes
}

fn main() ! Error {
    const blob = Blob(name: "hi", data: Bytes([104, 105]))
    print(json.of(blob))

    const back = try json.decode<Blob>("\{\"name\": \"hi\", \"data\": \"aGk\"}")
    print(back.data)
}

kavach run bytes-base64.kav

{"name":"hi","data":"aGk="}Bytes(6869)

It holds for a bare value as for a field, and in lists, maps and optionals.
Text that isn't base64 is a Mismatch error.
A spawn process passes Bytes this way too.
ask<T> still refuses Bytes: models write base64 badly.

spec 2.12 · Bytes in JSON base64, tested spawn process, tested

Files are read as Bytes with fs.read_bytes, or in pieces with fs.chunks: see Reading files.