Flide is extremely fast when one input carries many questions, as the shared input is computed only once. Even with a single question, it can still be up to 3x faster, depending on the model size and input length.
Good point. In that case, the current setting may not help much. But I think we can extend the framework to put the question first, so all the states share one question. We'd decide dynamically whether to place state or question first.
You really don't need an entire new inference engine for this. I modified vllm to work just fine for my decision model experiments.
It properly uses the single shared context and only recomputes the questions, or if the questions are longer, I let the questions go first and then the context that changes often can go after, and then the answer menu tokens... I got roughly the same speed increase with the right workloads that you're saying though.
Nice, similar point! I agree it works fine for full attention. But for Gated DeltaNet on Qwen3.5, vLLM only supports caching at block boundaries. So we developed a new one. Also, we optimize the CUDA graph usage and can keep answers bit-identical at any load.
I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!
Not yet, unfortunately. We plan to support it in the next few versions. But since GLiDE uses bidirectional attention, it can't be accelerated as much as others.
I looked at their paper and the biggest gains come when you have many questions from one context, which is the way these models are supposed to be used. It's around 1.5-3x faster than vLLM with single questions. That's the worst case where you can't reuse anything.
The way I understand it most of their speedup comes from caching intermediate results. They freeze the state after processing context then branch each question off that point. It means they can process questions in parallel after the context stage is finished, and don't waste time doing the same work N times.
Thanks for the clarification! That's our main idea. I think a shared state with many independent questions is the nature of decision tasks, so we push it to the extreme. This is a decision-native design.
So, if your problem is: tag these 10 000 emails (work, spam, personal, English, French...) it's fast? (Feed large context of mails (input) and question context).
If it's answer if this one email is spam - not so fast?
Currently, the speedup mainly depends on the number of questions per input. The more questions for each input, the lower the average time per question, as the input is computed only once by our engine.
For tagging one email is spam or not, our engine can still be up to 3x faster, but the speedup is not as significant as with multiple questions per state.
So max benefits would come from picking an email and asking "is this spam, is this in english, does this contain grammar errors, does it contain a signature, is it a commercial pitch"?
Thanks for the questions! I think tancop's summary has already captured the essence, and I can add some details.
> Under what specific circumstances is it actually 9.5x faster?
The 9.5x is Qwen3.5-0.8B on QuALITY, one long article that has 5.8k tokens and 18 questions on average. The more questions share a context, the bigger the gain.
> Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?
Yes, with a state that has only a single question, it ranges from a near tie with 27B on a long input to more than 3x with a small model on a short input. In this case nothing is shared, so the gain only comes from lower per-request overhead, since Flide runs the whole forward pass as one prerecorded CUDA graph.
> How did you achieve the biggest speed up?
The big speedup comes from cases where one state carries many questions, where we only compute the shared context once. We achieve this by using a mask that keeps the questions from seeing each other, and each question carries the correct position index just as if it came right after the state, so we can get the answers in one pass. The second is the prerecorded CUDA graph.
> Are there any tradeoffs? "While keeping exactness" implies not.
There is no accuracy tradeoff, since each question gets the identical computation as if asked alone. What remains is floating-point rounding that shifts slightly with batch size in every engine, even for a question asked alone, and our results are still the closest to the FP32 results among all the engines.
> Are there any specifics about how one must structure their context to take full advantage of it?
To get the most out of it, you can put everything shared in the state and ask all the questions in one request. And for continuous agentic decisions, use --session-cache-gb to get a further speedup.
Yes, Flide is self-hosted and serves the model on your own GPU. 3050 card is totally OK for the 0.8B model, and there's plenty of room for long inputs. We've only tested Flide on Linux, so on Windows you may need WSL2.
Is there a lower limit of hardware for "good enough" performance? I plan to integrate Flide into an existing product that is a desktop app for lesser-powered laptops that needs a local model.
Thanks! If the laptop has a 6GB NVIDIA GPU, Flide can serve the 0.8B model on it.
Otherwise, llama.cpp or LM Studio is a better choice for CPU-only devices or smaller GPUs.
Amazing, love it
Do you have numbers for running something like Laya locally with/without Flide?
Flide is extremely fast when one input carries many questions, as the shared input is computed only once. Even with a single question, it can still be up to 3x faster, depending on the model size and input length.
Unfortunately, the reverse situation, where the same question is asked about many inputs, is much more useful in practice.
Good point. In that case, the current setting may not help much. But I think we can extend the framework to put the question first, so all the states share one question. We'd decide dynamically whether to place state or question first.
Really depends on the use case. Look at the voice assistant demo from Jev. One input, many questions.
You really don't need an entire new inference engine for this. I modified vllm to work just fine for my decision model experiments.
It properly uses the single shared context and only recomputes the questions, or if the questions are longer, I let the questions go first and then the context that changes often can go after, and then the answer menu tokens... I got roughly the same speed increase with the right workloads that you're saying though.
Nice, similar point! I agree it works fine for full attention. But for Gated DeltaNet on Qwen3.5, vLLM only supports caching at block boundaries. So we developed a new one. Also, we optimize the CUDA graph usage and can keep answers bit-identical at any load.
Cool!
Thanks! Feel free to try!
I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!
Thanks! Welcome to give it a try.
You can check the Decision Index leaderboard at https://huggingface.co/spaces/multimodalart/jev-decision-ind... for reference, and we already support many of the models (see our README).
Or you can try our own model, OneJev (https://github.com/OmniJev/OneJev):
We're actively improving it.Is GLiDE 9B+ supported? It's a Qwen fine tune so it's not clear.
Not yet, unfortunately. We plan to support it in the next few versions. But since GLiDE uses bidirectional attention, it can't be accelerated as much as others.
Hi HN! I'm one of the Flide team.
We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.
Here is the source code: https://github.com/OmniJev/Flide
Look forward to hearing your feedback and ideas!
Under what specific circumstances is it actually 9.5x faster?
Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?
How did you achieve the biggest speed up?
Are there any tradeoffs? “While keeping exactness” implies not.
Are there any specifics about how one must structure their context to take full advantage of it?
I looked at their paper and the biggest gains come when you have many questions from one context, which is the way these models are supposed to be used. It's around 1.5-3x faster than vLLM with single questions. That's the worst case where you can't reuse anything.
The way I understand it most of their speedup comes from caching intermediate results. They freeze the state after processing context then branch each question off that point. It means they can process questions in parallel after the context stage is finished, and don't waste time doing the same work N times.
Thanks for the clarification! That's our main idea. I think a shared state with many independent questions is the nature of decision tasks, so we push it to the extreme. This is a decision-native design.
So, if your problem is: tag these 10 000 emails (work, spam, personal, English, French...) it's fast? (Feed large context of mails (input) and question context).
If it's answer if this one email is spam - not so fast?
Yes, but not quite. Let me clarify.
Currently, the speedup mainly depends on the number of questions per input. The more questions for each input, the lower the average time per question, as the input is computed only once by our engine.
For tagging one email is spam or not, our engine can still be up to 3x faster, but the speedup is not as significant as with multiple questions per state.
So max benefits would come from picking an email and asking "is this spam, is this in english, does this contain grammar errors, does it contain a signature, is it a commercial pitch"?
Exactly!
Thanks for the questions! I think tancop's summary has already captured the essence, and I can add some details.
> Under what specific circumstances is it actually 9.5x faster?
The 9.5x is Qwen3.5-0.8B on QuALITY, one long article that has 5.8k tokens and 18 questions on average. The more questions share a context, the bigger the gain.
> Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?
Yes, with a state that has only a single question, it ranges from a near tie with 27B on a long input to more than 3x with a small model on a short input. In this case nothing is shared, so the gain only comes from lower per-request overhead, since Flide runs the whole forward pass as one prerecorded CUDA graph.
> How did you achieve the biggest speed up?
The big speedup comes from cases where one state carries many questions, where we only compute the shared context once. We achieve this by using a mask that keeps the questions from seeing each other, and each question carries the correct position index just as if it came right after the state, so we can get the answers in one pass. The second is the prerecorded CUDA graph.
> Are there any tradeoffs? "While keeping exactness" implies not.
There is no accuracy tradeoff, since each question gets the identical computation as if asked alone. What remains is floating-point rounding that shifts slightly with batch size in every engine, even for a question asked alone, and our results are still the closest to the FP32 results among all the engines.
> Are there any specifics about how one must structure their context to take full advantage of it?
To get the most out of it, you can put everything shared in the state and ask all the questions in one request. And for continuous agentic decisions, use --session-cache-gb to get a further speedup.
do i need to host the model to do this ? if yes what model can i host in 3050 card
Yes, Flide is self-hosted and serves the model on your own GPU. 3050 card is totally OK for the 0.8B model, and there's plenty of room for long inputs. We've only tested Flide on Linux, so on Windows you may need WSL2.
Is there a lower limit of hardware for "good enough" performance? I plan to integrate Flide into an existing product that is a desktop app for lesser-powered laptops that needs a local model.
Thanks! If the laptop has a 6GB NVIDIA GPU, Flide can serve the 0.8B model on it. Otherwise, llama.cpp or LM Studio is a better choice for CPU-only devices or smaller GPUs.