Ten weekends of CUDA in Go: 30 functions became 106
gocudrv is my weekend project: CUDA's driver API from Go with no cgo, no toolkit at build time, and the driver library loaded at runtime. When I posted the week-3 version, someone in the comments counted my binding table and pointed out I had implemented about 30 functions out of the roughly 380 in cuda.h.
It's 106 now, and more importantly the parts around the bindings, the parts that made me want a wrapper in the first place, actually work: contexts that survive goroutines, streams and events, embedded PTX, and a static binary that runs on a machine that has never seen the CUDA toolkit.
Still a weekend project. Still learning CUDA in public.
Where this left off
The pitch hasn't changed: most Go CUDA packages need cgo and the
toolkit headers at build time, which breaks cross-compilation and makes
deploy images heavy. gocudrv loads libcuda.so.1 (or nvcuda.dll) at
runtime the way purego loads libc, does the calling-convention work
itself, and puts a Go-shaped API on top. Your build is
CGO_ENABLED=0 go build, from any OS, and the only thing the
target machine needs is the NVIDIA driver it already has.
What exists now
Since week 3: the binding table went from ~30 to 106 driver symbols,
typed device buffers (cuda.Alloc[float32]), primary
contexts, streams, events with hardware timing, module loading from
embedded PTX with the driver's JIT, host callbacks, occupancy queries,
and a device-info tool. Tagged v0.2.0 this week. The four examples in the
repo all run from a fresh clone; the event-pipeline one runs two
overlapped copy/compute batches and reports GPU time from CUDA events
(5.5ms for 2x1M floats on my 4070 Ti, driver 13.1).
The executor, grown up
The thing I teased in the week-3 post is now the load-bearing piece. CUDA's driver tracks the current context as per-OS-thread state. Go moves goroutines between OS threads whenever it likes. Put those two sentences next to each other and you can see the crash coming: a goroutine sets up a context, gets migrated, and its next driver call executes on a thread where the context stack is empty or, worse, belongs to a different device.
gocudrv's answer: every context owns one worker goroutine pinned to an
OS thread with runtime.LockOSThread, and every driver call
for that context is a closure sent down a channel to that worker. Your
goroutines never touch the driver directly, so there is no
thread-affinity footgun to hold wrong.
The price is a channel hop per call. For chatty per-call workloads that matters and I want to measure it honestly against cgo before making claims, but the design goal was correctness first: the race is the kind that works fine on your machine and corrupts a context in production.
Some hardship I discovered along the way
In the week-3 post I mentioned a discrepancy that confused me: my CPU
timer said a 10M-element vector add took ~160µs, CUDA's event API
said ~434µs. I assumed I had an overhead problem somewhere. I did
not. I had a measurement problem: kernel launches are asynchronous, so
time.Since around the launch call measures how long it took
to enqueue the work, not to do it.
Now that events work properly, here is the same 10M vector add on the 4070 Ti today, three consecutive timed runs after a warm-up:
Steady state the gap is roughly 18x. It gets worse: the enqueue cost itself drops from ~172µs on the first timed run to ~16µs once the driver has warmed up, so a naive benchmark is wrong twice in opposite directions. If you are timing GPU work with a host clock and not synchronizing first, you are publishing enqueue latency. The event API is not optional; it is the measurement.
One small binary, no toolkit
The part that still makes me smile: go generate compiles
the .cu to PTX once (that is the only step that needs nvcc, on my dev
machine), the PTX ships inside the binary via go:embed, and
the driver JIT-compiles it for whatever GPU it lands on. The stripped
vector-add demo is a 1.78MB static binary. Copy it to any Linux box with
an NVIDIA driver and it runs. No toolkit, no shared-library hunt, no
container layer full of CUDA packages. Module JIT for that small kernel
costs well under a millisecond warm; cold-start JIT across kernel sizes
is the next post, because the numbers surprised me and I want to measure
them properly before writing them down.
Honest limits
NVIDIA-only, Linux and Windows only, and the surface is 106 functions out of roughly 380, grown strictly by what I actually use and test. The per-call channel-hop overhead is real and unmeasured against cgo, so I make no performance claims about launch-heavy workloads yet. CUDA Graphs and multi-GPU are still on the workbench. And I remain a CUDA noob on a weekend schedule; the project moves slowly on purpose, because I want to understand every function I bind.
What's next
The JIT cold-start measurements, CUDA Graphs, and the FFI overhead comparison people keep asking about. If you have a 50-series card, or any setup weirder than my single 4070 Ti, I would genuinely love failure reports: the repo is github.com/eitamring/gocudrv and the examples run in under a minute from a fresh clone.