GPU: Sums in a loop

There is nothing to write. In a compiled program, a loop of this shape goes on while the GPU works:

use std.fs

gpu fn work(x: Int32) -> Int32 {
    x * 2 + 1
}

fn main() ! Error {
    const numbers = Bytes([
        1, 0, 0, 0, 2, 0, 0, 0,
        3, 0, 0, 0, 4, 0, 0, 0,
    ])
    try fs.write_bytes("numbers.bin", numbers)
    let total = 0
    for chunk in try fs.chunks("numbers.bin", 8) {
        const xs = chunk.as_list<Int32>() ?? []
        total += Int(xs.map(x => work(x)).sum(x => x))
    }
    print(total)
}

24

The loop goes on when all of this holds:

• A statement like total += xs.map(f).sum(g) sets the total, with sum, min or max of a list a GPU map or filter made.
• That statement is the only place the loop mentions total.
• It only works out numbers, from values the loop doesn't change.

A const s = xs.map(f).sum(g) used once, in the next statement, counts as part of it.

Only the time changes

• Each statement still runs once for each round, in order, with its checks, before any code after the loop. So total is right whenever it can be read.
• A few rounds' work may be under way at once, fewer for big chunks.
• Only the moment of a panic shows. If a statement overflows, the code after it in the loop, and later rounds, may have run by then, as for a map that doesn't wait.
• Plain kavach run doesn't use the GPU, so its loops run as written, with the same answers.

Which loops wait

kavach check says which loops still wait for the GPU, and why, with the line to look at. A note isn't an error: it goes to standard error, and the exit code stays 0. kavach check --format json leaves the notes out. These are all the reasons:

gpu fn step(x: Int32) -> Int32 {
    x + 1
}

fn logged(n: Int) -> Int {
    print(n)
    n
}

fn main() ! Error {
    const rounds: [[Int32]] = [[1, 2], [3, 4]]

    // These two don't wait, so they get no note.
    let quiet = 0
    for xs in rounds {
        quiet += Int(xs.map(step).sum(x => x))
    }
    let kept = 0
    for xs in rounds {
        const s = xs.map(step).sum(x => x)
        kept += Int(s)
    }

    let a = 0
    for xs in rounds {
        a += Int(xs.map(step).sum(x => x))
        print(a)
    }
    let b = 0
    for xs in rounds {
        b += logged(Int(xs.map(step).sum(x => x)))
    }
    let c = 0
    for xs in rounds {
        c += Int(xs.map(step).sum(x => x))
        const size = if xs.len() > 0 { xs.len() } else { fail Error("empty") }
        print(size)
    }
    let d = 0
    let k: Int32 = 1
    for xs in rounds {
        d += Int(xs.map(step).sum(x => x + k))
        k = k + 1
    }
    for xs in rounds {
        print(xs.map(step).sum(x => x))
    }
    for xs in rounds {
        print(xs.map(step))
    }
    print("{quiet} {kept} {a} {b} {c} {d}")
}

src/main.kav:26: note: this loop waits for the GPU each round: the loop also reads what it adds up to
src/main.kav:31: note: this loop waits for the GPU each round: the statement does more than work out numbers
src/main.kav:35: note: this loop waits for the GPU each round: the loop can leave in the middle of an expression
src/main.kav:42: note: this loop waits for the GPU each round: the statement reads a value the loop changes
src/main.kav:46: note: this loop waits for the GPU each round: the round uses the sum at once
src/main.kav:49: note: this loop waits for the GPU each round: the CPU reads the result each round
src/main.kav: no errors

What it saves

This loop on a file of 2 GB, on the same machine as CUDA, a C++ program that manages its own overlap. 7 runs in turns against a compiler from before this change, in ms, with the same sum every time and the same as CUDA's:

• This table comes from before a cheaper check in the kernel: a multiply no longer divides to check two numbers of 16 bits, which can't overflow. Over 5 runs in turns, the 200 steps then went from 4,008–4,062 to 2,785–2,991 ms, and the 1 step stayed at 1,217–1,297 against 1,235–1,468. Results and panics are the same.
• The laptop was in use while it measured, so two runs far from the rest are left out, both of the old compiler. CUDA's column is 5 runs, from a comparison in turns the same day.
• A file of 8 GB, too large for the GPU and the system's cache, took 63–65 s in all four cases, Kavach and CUDA: there the disk sets the speed, and neither loop holds it up.

Measured again the next day, 5 runs in turns against the compiler of the day before, the middle run in ms, with the file in the system's cache, so the disk no longer holds either side back:

• as_list no longer copies a big chunk: the chunk is read straight into the list's memory, and as_list takes it when nothing else holds the chunk.
• A kernel divides through unsigned numbers. On this GPU a signed % took about a third longer than an unsigned one. Results and panics are the same.
• CUDA is still about 1.8× ahead on 1 step and 2.6× on 200. Up to about 200 steps the time hardly changes, for either: the kernel runs while the next chunk is read, so the difference is the CPU's work for each chunk.
• Copying a chunk over to the GPU takes as long as CUDA's copy. But the GPU library made a new 64 MB buffer to copy through every few chunks, so Kavach now copies through two buffers of its own, used in turn. Run 6 times in turns, 32 chunks held in memory went from 869 to 764 ms (the uploads alone 410 → 304), and the 1-step stream from about 1,250 to 1,040 ms. Most of what is left is opening the GPU, which Kavach's time includes and CUDA's doesn't.
• Past that, the kernel sets the speed. The same kernel written by hand, at 800 steps, took 7.1 s with today's checks, 5.3 s without them, and 4.3 s with wgpu's own counter in each loop off too, against CUDA's 1.7 s. That counter stops a kernel that never ends. So since decision 0077, a kernel whose every loop is a for over a range that stops before its end goes without it: over 5 runs in turns the same day, 200 steps went from 2,816 to 2,541 and 800 steps from 6,007 to 4,909, against CUDA's 1,062 and 2,233 ms, with the same sums. A while or a ..= range keeps the counter.