The author mentions Rust's portable SIMD library [0]. The only issue with portable SIMD is it's only available on nightly. I used it in my FFT crate, but we had to switch to the fearless_simd crate in order to get a portable SIMD solution that works on stable [1].
My heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don't understand why it would be ported to GPU - huge kudos to managing to surprise me
GPU "cores" are basically what a CPU would call SIMD lanes. So a GPU with 1024 'CUDA cores' might be structured as 16 relatively independent pieces that a CPU might call a core, each with a 64 wide SIMD unit.
When you dig through the CUDA developer docs instead of the promotional materials, you can develop a view of Nvidia GPUs as having 8-128 processing cores, each with 4 hyperthreads, running 32-lane SIMD for almost everything. Where a lane is 32 bits wide.
The promotional material likes to label the individual lanes as “cores” because it sounds more impressive. And, it’s not entirely incorrect.
Even the dev docs use the marketing terminology. The description I gave above needs a bit of piecing together.
There is something very SIMD-coded in GPU programming which is coalesced stores/loads, if a warp (32 threads) handles contiguous memory, it will create ~4 transactions instead of 32.
Welcome to the lucky 10,000! SIMD is actually a pretty integral part of how GPUs are able to work efficiently, it's part of why there's such a strong focus on branchless programming in the field.
It should really be read/advertised as "portabler SIMD". It beats hoping the compiler autovectorizes everything well forever or writing architecture specific code manually again but is going to compromise on average performance vs platform specific SIMD.
> There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width.
What should it choose then? I have a Zen 3 processor, and benchmarking some simd I did recently says 32 byte or 64 byte chunks was fastest. But I'm sure I'd get a different result on a different Zen, and different again on Intel's.
How would the library decide what SIMD width I should use?
It'd need some kind of compile-time hardware-feature-detection, yea? That seems probably feasible since proc macros can do essentially anything they like (worryingly).
Only if the end-user is the one compiling the software, on the same very system they'll be running it on. Which is true of GPU shader kernels, due to how GPU drivers work; but isn't generally true of CPU object code (unless you're on Gentoo.)
What you'd actually want is a matrix of variant implementations burned into the binary, with runtime (or process-boot-time) hardware detection that swaps symbols out to point to the correct variant.
Sure but there's no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That's pretty much impossible at the moment.
Which is why I've never quite understood the appeal of portable SIMD libraries for performance-critical code. If I'm explicitly writing SIMD rather than relying on the auto-vectorizer, it's usually because I want access to the particular capabilities of the target ISA.
For many problems, choosing the right instruction or instruction sequence makes a large difference. Portable SIMD abstractions necessarily expose some common semantic layer, but SIMD ISAs don't actually have equivalent capabilities. Instructions like pshufb, for example, enable algorithmic tricks that don't necessarily have an equally efficient analogue on another architecture.
If maximum performance matters, I generally want intrinsics and architecture-specific implementations; if portability matters more, I'd rather move further up the abstraction stack and use something designed to target multiple architectures, such as ISPC. There are certainly cases where portable SIMD gets close enough to optimal, but I don't think there's a compiler or abstraction that can express every useful SIMD idiom and lower it equally efficiently across fundamentally different ISAs.
What is vectorware's business model? Are you planning to sell support/consulting to companies using your stack? Or are you looking to sell licenses to your tool? Or something else?
The tentative plan is to open source all the compiler and `std` bits with our products built on top (compilers are not good businesses). More about our products coming in the next couple of months!
The post is kind of vague on the IR you're targeting. Can you give some examples of what the SIMD-ized IR looks like, and how it maps to the target PTX?
Each family of operations is a trait parameterized by the operation itself:
pub trait EvaluateReduction<Operation, T>: LaneEvaluator {
/// Reduce one distributed definition to an ordinary uniform scalar.
fn evaluate_reduction(&self, value: LaneValue<Self, role::Distributed, T>) -> T;
}
Call sites name the operation:
let one = evaluator.splat::<Splat, _>(1_u32);
let two = evaluator.splat::<Splat, _>(2_u32);
let three = evaluator.binary::<Add, _>(one, two);
let total = evaluator.reduce::<Sum, u32>(three); // a uniform u32
let running = <Executor as EvaluateScan<Scan<Sum, Exclusive>, u32>>::scan(&evaluator, three);
Operations like Sum, Max, ReduceXor, Inclusive, and Exclusive are all distinct types.
As mentioned in the post, execution shape is typed too. A static shuffle takes its control as a type-level constant, and the shuffle mode constrains which controls are expressible:
// Shift down one lane, keeping our own value where the source is inactive.
let down = <Executor as EvaluateShuffle<Shuffle<Down>, DownOrSelf<1>, u32>>::shuffle(&ev, v);
// Broadcast from lane zero.
let bcast = <Executor as EvaluateShuffle<Shuffle<Broadcast>, WarpLane<0>, u32>>::shuffle(&ev, down);
// Butterfly exchange with the neighbor one bit away.
let bfly = <Executor as EvaluateShuffle<Shuffle<Xor>, Butterfly<1>, u32>>::shuffle(&ev, bcast);
For an example of errors caught, a warp-scoped executor for a device-scoped barrier is a compile error:
<ScopedWarpExecutor<'_, WarpUniform> as EvaluateBarrier<Barrier<Device>>>::barrier(evaluator)
// error[E0277]: the trait bound `Device: NvptxBarrierScope` is not satisfied
// help: the trait `NvptxBarrierScope` is implemented for `Warp`
Strip mining is typed on the amount of work and the lane capacity, and it hands back one chunk at a time along with the predicate saying which lanes live in that chunk:
// Six work items across four active lanes: two chunks, based at 0 and 4.
<Executor as EvaluateStripMine<StripMine, (WorkItems, ActiveLanes<StripMined<4>>), i32>>::
for_each_strip_mined(
&evaluator,
(WorkItems::new(6)?, ActiveLanes::new(4)?),
|index, active| {
// ...
},
);
I'm confused too. How does this fit between these approaches for paraellization:
- CUDA kernels and Tiles (e.g. Cudarc, cuda-oxide, rust-gpu etc) - SIMD on the GPU. (E.g. as in the title...)
- CPU SIMD using avx or SSE instructions (And probably thin wrappers for vectors so you can have sane syntax). Or the maybe-upcoming core simd which should abstract over architecture-specific instructions. Magic floats etc which do 4-16 computations at once, but are a bit clumsy to work with
- Rayon thread pools - arbitrary parallel computations, including SIMD, one per CPU core.
It looks like from the code samples like maybe a cleaner syntax for writing code on the GPU than CUDA kernels? E.g. without mucking with serialization, host and device by abstracting over it? And inspired by core::simd. (Good choice if so, in the interest of standardizing on syntax; I did this for my x86 SIMD vector/quaternion lib as well)
If you have to express your computation using an "array programming DSL" with things like scan and gather anyways - why not opt to use torch/tensorflow/jax or anything else that targets MLIR? An example of writing a relu using an embedded array DSL is really not helping your case either - that's exactly the problem that these other solutions mentioned above are successfully solving for the past ~15y (starting with theano etc). Not sure what this brings to the table - doing that AoT instead of at runtime?
The goal of this work is to run existing unmodified CPU libraries (which may use core::simd) on the GPU. If you are manually writing ML-shaped workloads, it doesn't add any value over writing with tech like torch/tensorflow/jax which are custom built for those use-cases (except maybe familiarity if you are a CPU programmer).
This is really cool! It sounds like y'all have a compiler fork that you are using to make this work. I wanna tinker with this, is your compiler available?
I don't get what's the value of it not being enabled by default what does the toggle get us, really? Maybe I don't understand web design and it makes it harder to read for some, I am dyslexic and never had any issues.
It's just a way for us to add minutia and details that most don't care / need to know about. There are three audiences we try to make the posts accessible for: Rust people who don't know about GPUs, GPU people who don't know about Rust, and non-Rust non-GPU people. The toggle lets knowledgable readers go "wait, what about..." and hopefully the toggle answers it.
The author mentions Rust's portable SIMD library [0]. The only issue with portable SIMD is it's only available on nightly. I used it in my FFT crate, but we had to switch to the fearless_simd crate in order to get a portable SIMD solution that works on stable [1].
[0] https://doc.rust-lang.org/std/simd/index.html
[1] https://github.com/linebender/fearless_simd
Pretty common for Rust to cook things in nightly for a very long time; I wouldn't consider it a bad thing, tbh.
My heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don't understand why it would be ported to GPU - huge kudos to managing to surprise me
GPU "cores" are basically what a CPU would call SIMD lanes. So a GPU with 1024 'CUDA cores' might be structured as 16 relatively independent pieces that a CPU might call a core, each with a 64 wide SIMD unit.
64 what? Bits/bytes/something bigger?
When you dig through the CUDA developer docs instead of the promotional materials, you can develop a view of Nvidia GPUs as having 8-128 processing cores, each with 4 hyperthreads, running 32-lane SIMD for almost everything. Where a lane is 32 bits wide.
The promotional material likes to label the individual lanes as “cores” because it sounds more impressive. And, it’s not entirely incorrect.
Even the dev docs use the marketing terminology. The description I gave above needs a bit of piecing together.
32 wide - only AMD has a 64 wide mode
GPUs work on vectors and matrices very often, that's what they are good at, so it makes a lot of sense that they can operate with SIMD I think!
There is something very SIMD-coded in GPU programming which is coalesced stores/loads, if a warp (32 threads) handles contiguous memory, it will create ~4 transactions instead of 32.
Welcome to the lucky 10,000! SIMD is actually a pretty integral part of how GPUs are able to work efficiently, it's part of why there's such a strong focus on branchless programming in the field.
I love how ever example of portable SIMD isn't portable.
They specifies a constant SIMD width so it's non-portable. Well, not performance portable, but why are we using SIMD again?
It should really be read/advertised as "portabler SIMD". It beats hoping the compiler autovectorizes everything well forever or writing architecture specific code manually again but is going to compromise on average performance vs platform specific SIMD.
Why should it be portable? Honest question.
SIMD seems to me, to be very platform specific. Maybe there are times one SIMD unit is not anothers' SIMD unit?
The create is called portable_simd.
There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width.
But the design and documentation of portable_simd makes the fixed size syntactically easy/the default and the width agnostic code harder.
> There is no reason a portable_simd relu_dot implemention should need to specify the SIMD width.
What should it choose then? I have a Zen 3 processor, and benchmarking some simd I did recently says 32 byte or 64 byte chunks was fastest. But I'm sure I'd get a different result on a different Zen, and different again on Intel's.
How would the library decide what SIMD width I should use?
It'd need some kind of compile-time hardware-feature-detection, yea? That seems probably feasible since proc macros can do essentially anything they like (worryingly).
Only if the end-user is the one compiling the software, on the same very system they'll be running it on. Which is true of GPU shader kernels, due to how GPU drivers work; but isn't generally true of CPU object code (unless you're on Gentoo.)
What you'd actually want is a matrix of variant implementations burned into the binary, with runtime (or process-boot-time) hardware detection that swaps symbols out to point to the correct variant.
Go's implementation is vector size independant https://pkg.go.dev/simd@master
Sure but there's no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That's pretty much impossible at the moment.
Which is why I've never quite understood the appeal of portable SIMD libraries for performance-critical code. If I'm explicitly writing SIMD rather than relying on the auto-vectorizer, it's usually because I want access to the particular capabilities of the target ISA.
For many problems, choosing the right instruction or instruction sequence makes a large difference. Portable SIMD abstractions necessarily expose some common semantic layer, but SIMD ISAs don't actually have equivalent capabilities. Instructions like pshufb, for example, enable algorithmic tricks that don't necessarily have an equally efficient analogue on another architecture.
If maximum performance matters, I generally want intrinsics and architecture-specific implementations; if portability matters more, I'd rather move further up the abstraction stack and use something designed to target multiple architectures, such as ISPC. There are certainly cases where portable SIMD gets close enough to optimal, but I don't think there's a compiler or abstraction that can express every useful SIMD idiom and lower it equally efficiently across fundamentally different ISAs.
I would love to have an open source Rust SIMD library with the scope and maturity that https://github.com/google/highway brings to C++.
This is basically the goal of fearless_simd, but of course achieving the same level of maturity will take time.
Do you have examples of complex algorithms running on the gpu with rust with competative performance? Radix sort might be a good one to start with
Really exciting work and great write up, thanks a lot and all the best to your startup!
`core` instead of `std` is great too!
This will become useful in one of my sideproject where I use bitmaps to speed up pathfinding, exited to try it out!
Author here, AMA.
What is vectorware's business model? Are you planning to sell support/consulting to companies using your stack? Or are you looking to sell licenses to your tool? Or something else?
The tentative plan is to open source all the compiler and `std` bits with our products built on top (compilers are not good businesses). More about our products coming in the next couple of months!
Looking forward to reading more about it. Good luck on the launch :)
The post is kind of vague on the IR you're targeting. Can you give some examples of what the SIMD-ized IR looks like, and how it maps to the target PTX?
Didn't want to go into crazy detail in the post.
Each family of operations is a trait parameterized by the operation itself:
Call sites name the operation: Operations like Sum, Max, ReduceXor, Inclusive, and Exclusive are all distinct types.As mentioned in the post, execution shape is typed too. A static shuffle takes its control as a type-level constant, and the shuffle mode constrains which controls are expressible:
For an example of errors caught, a warp-scoped executor for a device-scoped barrier is a compile error: Strip mining is typed on the amount of work and the lane capacity, and it hands back one chunk at a time along with the predicate saying which lanes live in that chunk: Hopefully that gives the flavor of it.I'm confused too. How does this fit between these approaches for paraellization:
It looks like from the code samples like maybe a cleaner syntax for writing code on the GPU than CUDA kernels? E.g. without mucking with serialization, host and device by abstracting over it? And inspired by core::simd. (Good choice if so, in the interest of standardizing on syntax; I did this for my x86 SIMD vector/quaternion lib as well)If you have to express your computation using an "array programming DSL" with things like scan and gather anyways - why not opt to use torch/tensorflow/jax or anything else that targets MLIR? An example of writing a relu using an embedded array DSL is really not helping your case either - that's exactly the problem that these other solutions mentioned above are successfully solving for the past ~15y (starting with theano etc). Not sure what this brings to the table - doing that AoT instead of at runtime?
The goal of this work is to run existing unmodified CPU libraries (which may use core::simd) on the GPU. If you are manually writing ML-shaped workloads, it doesn't add any value over writing with tech like torch/tensorflow/jax which are custom built for those use-cases (except maybe familiarity if you are a CPU programmer).
This is really cool! It sounds like y'all have a compiler fork that you are using to make this work. I wanna tinker with this, is your compiler available?
It is not currently available but we intend to make it available after we launch our products.
Given the massive demand for GPUs for LLMs, what sorts of work do you expect to economically benefit from utilizing GPUs more?
Part of our thesis is that decent GPUs are in every shipping device and most software doesn't use them and should.
I guess you're looking at consumer hardware then since servers have exactly what you pay for.
Can you say more about the application space you're targeting?
Hm is the intent to one day replace the CPU?
The goal is to use similar abstractions and code across both the CPU and GPU where it makes sense.
Any thoughts about SIMD-related crates?
How was your day?
Congrats to the Rust-GPU folks! Nice to see the good work flowing.
Love the pendantic mode setting on the website
I don't get what's the value of it not being enabled by default what does the toggle get us, really? Maybe I don't understand web design and it makes it harder to read for some, I am dyslexic and never had any issues.
It's just a way for us to add minutia and details that most don't care / need to know about. There are three audiences we try to make the posts accessible for: Rust people who don't know about GPUs, GPU people who don't know about Rust, and non-Rust non-GPU people. The toggle lets knowledgable readers go "wait, what about..." and hopefully the toggle answers it.
Does making it the default hurt anyone though? I don't think that 1 subscript adds anything to it..
Just distracting / ugly to my eyes.
Good job!
Hey - this is probably off-topic/meta, but what is going on with the comments here? Is it bots?
No idea, but it seems HN needs POW challenges.
Could also just be trolls attracted by the Rust topic.
I bet some kid is bored out of his mind and wrote a bot.