In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation. I gave the agents access to the compiler's profiler and also Intel's VTune, which has fantastic output. In a couple of hours the LLM generated SSE and AVX implementations of the compression and decompression algorithms that almost doubled performance with a single core. Then I asked it to create a CUDA implementation using NVIDIA's NSIGHT profiler as a guide and it also started doing some good work.
Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot
I have used Opus 5 and some Fable 5 to finally get realtime transcoding of 4K 10-bit HEVC (to 1080p or smaller SDR AVC) working on a Raspberry Pi 4. It was very good at writing optimized NEON kernels. the Argon HEVC hardware decoder outputs SAND30 which is a tiled format that is annoying to work with and not really supported by anything else, the big performance issue has been with converting and scaling it, but as it turned out a lot of it was really with just moving memory around, so by fusing multiple steps into a single kernel it became fast enough. Experimenting with writing the NEON kernels for the different combinations would have taken forever by hand.
I do wonder if auto-research would have reached something similar, it did take a significant amount of steering from me to get it to the point where it was working realtime.
I did something similar recently with Google's C# protobuf library. I had spotted I was getting CPU bound rather than memory bandwidth bound when doing streaming of uint32 buffers in dotnet gRPC.
I then asked claude to compare the C#/.NET implementation in the library with the C++ version, and it quickly identified that the C# library was missing a couple of fairly cheap optimisations that were present in the C++ version.
If I can help get a PR merged, then it'll be by far the biggest impact of any work I've ever done.
I also compared the Rust version, it had this specific optimisation. The far more popular Tokio/Prost library did not.
Given appropriate guardrails, LLMs are impossibly fast at iterating to find root causes and specific performance bottlenecks.
You presented another thing LLMs excel at: integrating something from a project that is not present in another one. I think they work so well at this because both the starting and ending points have an already existing structure, so the LLM can guide itself effectively. In your case it's even more egregious because we are talking about the same exact algorithm/functionality implemented in two different, but rather similar, programming languages.
Could you have manually profiled and compared the execution paths? Sure. Could you have translated the C++ optimizations to C#? Sure. But in such an obvious case, the LLM managed itself.
I tried kernel autoreasearch using DeepSeek-V4-Flash as well. It spent about 1-2 hours to complete the FlashAttention optimization job (https://github.com/fengwang/FA5090/tree/main/v7) and cost me only $0.2. I believe we are ready to offload a lot of this kind well-defined constrained optimization problems to AI Agent autoresearch.
> I tried kernel autoreasearch using DeepSeek-V4-Flash as well. It spent about 1-2 hours to complete the FlashAttention optimization job
Doing the same, re-implementing a lot of LLM/diffusion models in Rust+CUDA for my own usage, usually the initial implementation takes 1-2 days (of 100% autonomous work) then I put an agent to optimize the implementation which tends to get close to SOTA performance within another day or two.
As long as you can point the agent at "This is the correct baseline, make sure any optimizations still pass this", seemingly you can leave them and they come back after N hours with a faster program that just works.
I've had a lot of success decompiling old video game ROMs in exactly this way. Like you say - give it a way of verifying correctness - put it in a loop - and they are quite surprising.
Even the cheap LLMs are great in doing the awful crud work in the beginning: finding offsets, firmware update file structures, brute forcing checksums, etc.
It still produces a lot of crap in the later steps (understanding the implementation itself) but I'm happy doing this stuff myself then.
> It still produces a lot of crap in the later steps (understanding the implementation itself)
I've had success here by adding a phase called "grounding" that attempts to verify its "understanding" by creating tests that modify the running executable to ensure its made the right inference.
Is this variable really MARIO_X? Change it and see if Mario moves. Etc.
As an example in Donkey Kong - the system had trouble deciding if an array controlled barrels or fireballs. There was conflicting evidence.
After many trips through the loop - it realized it does BOTH, depending on which level you're on.
Oh wow, this is almost exactly what I’ve been doing with Zelda LTTP. I have it in rust now, but just finished the “first pass” you reference. Mine is still not really readable, second step is the modernizing the actual code. I’ve really struggled with needing to handhold it though, I’ll see if I can plagiarize from you!
Very cool, me too! I've been working on Final Fantasy Legend (Game Boy and WonderSwan Color) and King's Bounty (PC - DOS). It's great for reversing. Really interesting to see the guts of the games, including bugs.
The most interesting thing I've found so far is the anti-tampering mechanisms.
In Time Pilot - there are three routines that are called constantly from inside the main loop. Each routine computes the checksum of the other routine's code to see if it's been modified. If so it jumps into random junk data.
There are other less exotic routines that make sure the copyright string hasn't been modified, etc.
Had a similar experience with my Rust implementation for JSONLogic expression evaluation engine. As it has a full test suite with 1000s of cases and a benchmarking script, I was able to give some basic hints to try different optimization techniques and the end result was impressive. Reached from 1.6s to 200ms for a full benchmarking test.
https://github.com/GoPlasmatic/datalogic-rs
First 3 versions were hand written and maintained for 3yrs, and now 4th version came out in less than a month's time with impressive performance.
> Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot
This!
People keep measuring how good AI is by one-shotting a problem, but I e found that a back and fourth via the Superpowers SKILL.md is how the model builds these constraints… then when it has all the holes, it gets to work
6 months ago I would have said the ralph loop plugin.
with opus 5, I'm finding that asking itself to write such a loop skill that does benchmark - profile - verify - research - improve, keep logs for each iteration, etc. (just like autoresearch), and then using the skill, works very nicely.
I've had pretty good luck with the following process for performance optimization loops:
- Have an agent generate unit tests until it gets to 100% path (not just statement) coverage, with every numerical test asserting checks against golden values to prevent regressions
- Let it rip on a performance improvement loop, for the widest E2E representative test case you have. Have it generate flamegraphs along the way so you can check in and steer it as necessary.
- Optionally allow for 1 ULP changes in output values so that it doesn't kill itself getting bit-exact results.
- Have it flag correctness errors as it goes, since your code probably isn't bug free.
This is also how I've done language ports from python to rust, and having the ironclad test coverage protects you from drifting.
One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones.
The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds.
The takeaway from this is that these approaches will always solve for specificity, but it's a much harder task to steer the model into making general solutions. So if you're an inference provider for some specific model shape, fantastic, go for it. If you are a maintainer of a open-source library, this is not useful.
This is one of the dilemmas that I am trying to wrap my head around. I love optimizing software pipelines, which often boils down to figuring out the operational constraints that the compiler and the generic libraries can’t assume. Then I exploit these to squeeze out performance. But in a world I can start from scratch and code a domain specific solution from line zero in a matter of hours/days, I do not need general libraries as much as I used to. On one hand the code won’t be as well tested as a good general library. On the other hand, it also won’t have a plethora of library bugs that are there because the code is generic and opaque. One counter argument is that things are never static and you can’t have specific code for too long. A counter to that is that you can then change the code to be specific to the new reality at very low cost. This is the mental loop I ride constantly. Disclaimer: My circumstances are definitely not general, I am not writing code that is truly large scale.
Many open-source communities are wrestling with similar questions. In many ways the generic frameworks end up being reference implementations for the specific solutions to copy and hill-climb towards, and it's as you say unclear what the point is to maintain generic things beyond that.
What I feel is lacking with the solutions that have for example agent-generated and tuned GPU kernels is that the use-cases for them are unclear. If you are a researcher on second-order optimizers, you probably want to be able to handle variable input shapes to experiment, you also might want something readable to understand intermediate steps and perhaps build on that. If you are a neolab running massive training runs for 80% of your VC funding, you need to know that every line in your training code is bitwise identical to the theory/reference because a divergent run from some LLM-generated numerical bug will set you back or bankrupt you, so you can't just plop in a random kernel even if it promises good performance.
So who is the agentically-looped end result for? Except for Openai and Anthropic of course who sold the tools.
this is true. in one of the later problems (cholesky decomposition), the organizer ran the submissions on a tiny training run to validate... and also provided code for same for our reference. most of the top solutions hit 4/8 or so. not very numerically stable.
i found out that as i learnt more domain wise, i was (obviously) able to steer better. doing a re-write can also remove lots of slop and context rot (and subsequently make it easier for both human and LLM to make solution more numerically stable, less reward hackish)
Training material seems to be especially rich re GPU kernels and SIMD.
I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?
Well, GPU kernels are co-designed really hard. A lot of it is, async tile pipelines + spam my MMA primtives.
Obviously it's still hard, but the point is that, by construction (cause like, NVIDIA literally releases primitives like this, and/or people like TK build slightly higher-level primitives over the base hardware primitives), if you learn the complicated language, you can get really good results, and on some level you "know" you're right by construction even before you go to the actual empirical tests (since you're operating over a higher-level "language", and not arbitrary byte accesses).
Honestly a lot of interfaces and frameworks you could argue are like that, so it's not really a point for GPU kernels relative to other things. But maybe a hint as to what I personally think is important in the AI era - finding the right cuts, the right high leverage abstractions, as otherwise AI is going to produce spaghetti nonsense.
Anecdotally, I saw Opus 5 come up with a complicated loop unrolling technique when I asked it to implement a simple biquad in SIMD, missing a simpler solution. Maybe it was a downgraded session, who knows. That SIMD instruction set, the one for the ESP32-P4, is not very popular and all the documentation it has is a couple of blog posts. So I'm pretty sure it has at most seen some code for a predecessor during its training. However, the LLM was able to derive a full listing of the operations and their arguments from gcc to get us started, and that's why I was able to come up with my own implementation. Along the way, it also came up with insights about possible gotchas. Then, when implementing algorithms, it has been able to reason things out and get things working, despite the ISA not being extremely well known.
Off-topic, but imagine us collectively being okay with (or powerless to do anything about) this sentiment about any other software service provided like two years ago.
Because pre-LLMs humans partially "autogenerated" kernels through hyperparameter search and in some sense eating the code complexity in return for performance, and thus built tools for the same automatic verifiability that is useful for LLMs.
In some other tasks, we never built the same level of automatic verifiability since the level of automation in creation being much lower meant it's not giving you as much of a marginal benefit. We prefer code readability and simplicity and such in say, web services, because, say, the database IO time is going to dominate. Here getting an LLM to write a cromulent C# web service is more difficult since it's not easy to automatically verify whether code is cromulent or not. So if you put up LLMs to it, you end up with slop (which works).
OTOH, in kernel design, you give it access to every perf counter, every observable possible and have it optimise all of them. And all are verifiable/hill-climbable and you generally don't give a crap if the code is readable or reusable.
It's been fascinating doing a custom variant for GFQL, the first OSS embeddable Cypher property graph query engine for CPU+GPU -
- accelerated launch of our new backends like polars, including a new lazy mode & planner, which are fundamentally new paths
- while we initially aimed for top GPU benchmark scores, we now also maintain top CPU scores too!
Long-term, more interesting to me is this opens rethinking what it means to be a query engine. Right now we are making it the fastest in general, especially on workloads from our own use, major industry benchmarks, and our users. At the same time, similar to jit and multistage computing, we're looking at new ahead-of-time optimization techniques users can do that are more interesting than plugging in custom indexes. Essentially, if our agents can do fast specializations, there should be safe hooks that we can expose to our user's agents too!
Isn't cholesky - used to substitute householder at a point - faster but less stable in some cases? I'm just recalling from memory since I had done a small project on qr decomposition with householder for an exam this year. I mean, if it is faster than the standard torch operation probably there are good reasons for which it is not the default standard torch operation. Might as well be wrong, I'm not sure
Mirrors my experience: LLMs are really good at optimizing, better than most humans. But also, they tend to not reach absolute peak performance where people made an effort to optimize something.
Since most problems see fairly little optimization, that's still a big win most of the time.
This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?
1. labs have lots of inference capacity
2. they will have domain experts working on this so their efficiency is gonna be exponentially more (can direct LLM better, save money, reach same results faster)
You can't exceed roofline performance on hardware. There is an performance cap you can hit. This recursive self improvement stuff lets you be closer to the pareto frontier, but the idea that it is leading to some exponential growth is a total pipe dream.
Does the edge matter? I know you added significant as your hedge, but once you have feedback, your gain is largely irrelevant. Gain buys you bandwidth, so we are constructing systems run by the most powerful corporations where they are now optimizing for latency, as Archer says, do you want to flash crash civilization? This is how you do it.
this is the first time ive heard of beam search. i would have reached for a genetic algorithm of some sort, although it seems like some stochastic versions of beam search exist to avoid local minima. i wonder if there are any good frameworks for building these that agents can construct and use.
I think the question is how do you keep track of the ideas that the agent is pursuing - like I was working on this for implementing a fft, and doing the optimizations, but I held its hand and was like - hey let's go back and retry this older thing you discounted because of a 3 % slowdown.
Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.
Just reporting work as done that’s not even close mainly. Then if you manually screenshot both and ask if they are the same it’s like “yeah looks great boss”. The visual understanding of the models is just leagues worse than their text/language understanding.
Or guessing colors rather than sampling from the image or pulling from figma is another stupid thing they do constantly.
People are always going to hate auto-research and "loop engineering". Because it's got 2 properties:
1) it's the only way to get something out of models (or people for that matter) that they don't know yet.
2) it's harder to do with an LLM than without. Not easier.
3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a robot leg on the ground is just to crash it into the ground.
you specify the goal. if the goal is achieved, it's achieved.
the code the LLM writes will be read and maintained and developed further by LLMs. so it doesn't really matter what it produces as long as all the tests are green and it achieves exactly what you want it to achieve.
In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation. I gave the agents access to the compiler's profiler and also Intel's VTune, which has fantastic output. In a couple of hours the LLM generated SSE and AVX implementations of the compression and decompression algorithms that almost doubled performance with a single core. Then I asked it to create a CUDA implementation using NVIDIA's NSIGHT profiler as a guide and it also started doing some good work.
Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot
I have used Opus 5 and some Fable 5 to finally get realtime transcoding of 4K 10-bit HEVC (to 1080p or smaller SDR AVC) working on a Raspberry Pi 4. It was very good at writing optimized NEON kernels. the Argon HEVC hardware decoder outputs SAND30 which is a tiled format that is annoying to work with and not really supported by anything else, the big performance issue has been with converting and scaling it, but as it turned out a lot of it was really with just moving memory around, so by fusing multiple steps into a single kernel it became fast enough. Experimenting with writing the NEON kernels for the different combinations would have taken forever by hand.
I do wonder if auto-research would have reached something similar, it did take a significant amount of steering from me to get it to the point where it was working realtime.
For anyone interested the ffmpeg is at https://github.com/poizan42/jellyfin-rpi-ffmpeg and a shim for using it with stock jellyfin at https://github.com/poizan42/jellyfin-rpi-ffmpeg-shim
I did something similar recently with Google's C# protobuf library. I had spotted I was getting CPU bound rather than memory bandwidth bound when doing streaming of uint32 buffers in dotnet gRPC.
I then asked claude to compare the C#/.NET implementation in the library with the C++ version, and it quickly identified that the C# library was missing a couple of fairly cheap optimisations that were present in the C++ version.
If I can help get a PR merged, then it'll be by far the biggest impact of any work I've ever done.
I also compared the Rust version, it had this specific optimisation. The far more popular Tokio/Prost library did not.
Given appropriate guardrails, LLMs are impossibly fast at iterating to find root causes and specific performance bottlenecks.
You presented another thing LLMs excel at: integrating something from a project that is not present in another one. I think they work so well at this because both the starting and ending points have an already existing structure, so the LLM can guide itself effectively. In your case it's even more egregious because we are talking about the same exact algorithm/functionality implemented in two different, but rather similar, programming languages.
Could you have manually profiled and compared the execution paths? Sure. Could you have translated the C++ optimizations to C#? Sure. But in such an obvious case, the LLM managed itself.
I tried kernel autoreasearch using DeepSeek-V4-Flash as well. It spent about 1-2 hours to complete the FlashAttention optimization job (https://github.com/fengwang/FA5090/tree/main/v7) and cost me only $0.2. I believe we are ready to offload a lot of this kind well-defined constrained optimization problems to AI Agent autoresearch.
> I tried kernel autoreasearch using DeepSeek-V4-Flash as well. It spent about 1-2 hours to complete the FlashAttention optimization job
Doing the same, re-implementing a lot of LLM/diffusion models in Rust+CUDA for my own usage, usually the initial implementation takes 1-2 days (of 100% autonomous work) then I put an agent to optimize the implementation which tends to get close to SOTA performance within another day or two.
As long as you can point the agent at "This is the correct baseline, make sure any optimizations still pass this", seemingly you can leave them and they come back after N hours with a faster program that just works.
I've had a lot of success decompiling old video game ROMs in exactly this way. Like you say - give it a way of verifying correctness - put it in a loop - and they are quite surprising.
https://github.com/qarl/arcade-js
Same. I love reverse engineering embedded stuff.
Even the cheap LLMs are great in doing the awful crud work in the beginning: finding offsets, firmware update file structures, brute forcing checksums, etc.
It still produces a lot of crap in the later steps (understanding the implementation itself) but I'm happy doing this stuff myself then.
> It still produces a lot of crap in the later steps (understanding the implementation itself)
I've had success here by adding a phase called "grounding" that attempts to verify its "understanding" by creating tests that modify the running executable to ensure its made the right inference.
Is this variable really MARIO_X? Change it and see if Mario moves. Etc.
As an example in Donkey Kong - the system had trouble deciding if an array controlled barrels or fireballs. There was conflicting evidence.
After many trips through the loop - it realized it does BOTH, depending on which level you're on.
So the "understanding" grows with each iteration.
Oh wow, this is almost exactly what I’ve been doing with Zelda LTTP. I have it in rust now, but just finished the “first pass” you reference. Mine is still not really readable, second step is the modernizing the actual code. I’ve really struggled with needing to handhold it though, I’ll see if I can plagiarize from you!
I'm working on getting the "handholding" down to zero. Frogger is almost done and I haven't had to intervene once.
Very cool, me too! I've been working on Final Fantasy Legend (Game Boy and WonderSwan Color) and King's Bounty (PC - DOS). It's great for reversing. Really interesting to see the guts of the games, including bugs.
The most interesting thing I've found so far is the anti-tampering mechanisms.
In Time Pilot - there are three routines that are called constantly from inside the main loop. Each routine computes the checksum of the other routine's code to see if it's been modified. If so it jumps into random junk data.
There are other less exotic routines that make sure the copyright string hasn't been modified, etc.
https://github.com/qarl/arcade-js/blob/main/games/timeplt/id...
Fascinating.
Had a similar experience with my Rust implementation for JSONLogic expression evaluation engine. As it has a full test suite with 1000s of cases and a benchmarking script, I was able to give some basic hints to try different optimization techniques and the end result was impressive. Reached from 1.6s to 200ms for a full benchmarking test. https://github.com/GoPlasmatic/datalogic-rs First 3 versions were hand written and maintained for 3yrs, and now 4th version came out in less than a month's time with impressive performance.
> Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot
This!
People keep measuring how good AI is by one-shotting a problem, but I e found that a back and fourth via the Superpowers SKILL.md is how the model builds these constraints… then when it has all the holes, it gets to work
This is exactly how I use it. I mean not on abandoned repos, but in a benchmark - profile - verify - research - improve loop.
What’s the best way to loop it?
6 months ago I would have said the ralph loop plugin.
with opus 5, I'm finding that asking itself to write such a loop skill that does benchmark - profile - verify - research - improve, keep logs for each iteration, etc. (just like autoresearch), and then using the skill, works very nicely.
I'm hoping to release a native Wine for MacOS w/ Win32 FEX support in the next few days.
Watching claude and codex play winquake and age of empires, and debug support for Firefox 52 has been wild.
Is Mojo programming language useful in the age of AI?
This is the way. Checkout the technique mentioned in the alphaevolve paper and see if it works well for your setting.
Thank you for the reference, I hope to be smart enough to try it out!
It's a straightforward "prompting" + single evolutionary algorithm technique, The paper looks like well, a paper, but the actual thing is simple.
I've had pretty good luck with the following process for performance optimization loops:
- Have an agent generate unit tests until it gets to 100% path (not just statement) coverage, with every numerical test asserting checks against golden values to prevent regressions
- Let it rip on a performance improvement loop, for the widest E2E representative test case you have. Have it generate flamegraphs along the way so you can check in and steer it as necessary.
- Optionally allow for 1 ULP changes in output values so that it doesn't kill itself getting bit-exact results.
- Have it flag correctness errors as it goes, since your code probably isn't bug free.
This is also how I've done language ports from python to rust, and having the ironclad test coverage protects you from drifting.
One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones.
The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds.
The takeaway from this is that these approaches will always solve for specificity, but it's a much harder task to steer the model into making general solutions. So if you're an inference provider for some specific model shape, fantastic, go for it. If you are a maintainer of a open-source library, this is not useful.
This is one of the dilemmas that I am trying to wrap my head around. I love optimizing software pipelines, which often boils down to figuring out the operational constraints that the compiler and the generic libraries can’t assume. Then I exploit these to squeeze out performance. But in a world I can start from scratch and code a domain specific solution from line zero in a matter of hours/days, I do not need general libraries as much as I used to. On one hand the code won’t be as well tested as a good general library. On the other hand, it also won’t have a plethora of library bugs that are there because the code is generic and opaque. One counter argument is that things are never static and you can’t have specific code for too long. A counter to that is that you can then change the code to be specific to the new reality at very low cost. This is the mental loop I ride constantly. Disclaimer: My circumstances are definitely not general, I am not writing code that is truly large scale.
Many open-source communities are wrestling with similar questions. In many ways the generic frameworks end up being reference implementations for the specific solutions to copy and hill-climb towards, and it's as you say unclear what the point is to maintain generic things beyond that.
What I feel is lacking with the solutions that have for example agent-generated and tuned GPU kernels is that the use-cases for them are unclear. If you are a researcher on second-order optimizers, you probably want to be able to handle variable input shapes to experiment, you also might want something readable to understand intermediate steps and perhaps build on that. If you are a neolab running massive training runs for 80% of your VC funding, you need to know that every line in your training code is bitwise identical to the theory/reference because a divergent run from some LLM-generated numerical bug will set you back or bankrupt you, so you can't just plop in a random kernel even if it promises good performance.
So who is the agentically-looped end result for? Except for Openai and Anthropic of course who sold the tools.
Overfitting to the input is part of the meta in this type of challenges.
The goal is not to create good, general or maintainable code. The only goal is to produce the fastest code.
Nonetheless gives appropriate context to the speed up for folks like myself who were just skimming.
fair argument
Test coverage is important. You need to test a variety of inputs.
Also "will always" is way overconfident. A year ago nothing close to this existed at all. Next year it will fill yet a different role.
this is true. in one of the later problems (cholesky decomposition), the organizer ran the submissions on a tiny training run to validate... and also provided code for same for our reference. most of the top solutions hit 4/8 or so. not very numerically stable.
i found out that as i learnt more domain wise, i was (obviously) able to steer better. doing a re-write can also remove lots of slop and context rot (and subsequently make it easier for both human and LLM to make solution more numerically stable, less reward hackish)
Meta commentary but it felt fresh to read a long wall of text that didn't seem to be AI generated. Thanks.
How is this meta commentary when this or its negative version is present on literally every post.
First: Whataboutism, and second: Those are meta comments as well, they just don't declare themselves as one.
author here!
welcome! check out my featured section
Training material seems to be especially rich re GPU kernels and SIMD.
I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?
Well, GPU kernels are co-designed really hard. A lot of it is, async tile pipelines + spam my MMA primtives.
Obviously it's still hard, but the point is that, by construction (cause like, NVIDIA literally releases primitives like this, and/or people like TK build slightly higher-level primitives over the base hardware primitives), if you learn the complicated language, you can get really good results, and on some level you "know" you're right by construction even before you go to the actual empirical tests (since you're operating over a higher-level "language", and not arbitrary byte accesses).
Honestly a lot of interfaces and frameworks you could argue are like that, so it's not really a point for GPU kernels relative to other things. But maybe a hint as to what I personally think is important in the AI era - finding the right cuts, the right high leverage abstractions, as otherwise AI is going to produce spaghetti nonsense.
Anecdotally, I saw Opus 5 come up with a complicated loop unrolling technique when I asked it to implement a simple biquad in SIMD, missing a simpler solution. Maybe it was a downgraded session, who knows. That SIMD instruction set, the one for the ESP32-P4, is not very popular and all the documentation it has is a couple of blog posts. So I'm pretty sure it has at most seen some code for a predecessor during its training. However, the LLM was able to derive a full listing of the operations and their arguments from gcc to get us started, and that's why I was able to come up with my own implementation. Along the way, it also came up with insights about possible gotchas. Then, when implementing algorithms, it has been able to reason things out and get things working, despite the ISA not being extremely well known.
> Maybe it was a downgraded session, who knows.
Off-topic, but imagine us collectively being okay with (or powerless to do anything about) this sentiment about any other software service provided like two years ago.
How the times have changed…
we live in exciting and scary times...
Were other services so historically resource-constrained?
They are easily verifiable and hill-climbable.
Because pre-LLMs humans partially "autogenerated" kernels through hyperparameter search and in some sense eating the code complexity in return for performance, and thus built tools for the same automatic verifiability that is useful for LLMs.
In some other tasks, we never built the same level of automatic verifiability since the level of automation in creation being much lower meant it's not giving you as much of a marginal benefit. We prefer code readability and simplicity and such in say, web services, because, say, the database IO time is going to dominate. Here getting an LLM to write a cromulent C# web service is more difficult since it's not easy to automatically verify whether code is cromulent or not. So if you put up LLMs to it, you end up with slop (which works).
OTOH, in kernel design, you give it access to every perf counter, every observable possible and have it optimise all of them. And all are verifiable/hill-climbable and you generally don't give a crap if the code is readable or reusable.
It's been fascinating doing a custom variant for GFQL, the first OSS embeddable Cypher property graph query engine for CPU+GPU -
- accelerated launch of our new backends like polars, including a new lazy mode & planner, which are fundamentally new paths
- while we initially aimed for top GPU benchmark scores, we now also maintain top CPU scores too!
Long-term, more interesting to me is this opens rethinking what it means to be a query engine. Right now we are making it the fastest in general, especially on workloads from our own use, major industry benchmarks, and our users. At the same time, similar to jit and multistage computing, we're looking at new ahead-of-time optimization techniques users can do that are more interesting than plugging in custom indexes. Essentially, if our agents can do fast specializations, there should be safe hooks that we can expose to our user's agents too!
Isn't cholesky - used to substitute householder at a point - faster but less stable in some cases? I'm just recalling from memory since I had done a small project on qr decomposition with householder for an exam this year. I mean, if it is faster than the standard torch operation probably there are good reasons for which it is not the default standard torch operation. Might as well be wrong, I'm not sure
For once Cholesky is less general than QR: Cholesky only makes sense for positive matrices, while QR works for any matrix (including non square).
Also QR is a primitive for operations like finding eigenvalues, and I don't think Cholesky can be used there.
Mirrors my experience: LLMs are really good at optimizing, better than most humans. But also, they tend to not reach absolute peak performance where people made an effort to optimize something.
Since most problems see fairly little optimization, that's still a big win most of the time.
some of these submissions seem to be omitting the actual rules. the #1 on edinh has a line that says "bypass ban check"
submission #2 by gau nernst is most numerically stable
Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.
This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?
hello author here.
yes, it gives labs edge and leads to self-recursive improvement loops.
also i was myself able to finish 7th in a later competition with 2-3 other approaches which are variants of the method discussed in this blog.
in general, having a harness as thin as possible with some problem specific instructions while controlling for context rot is the key.
point i am trying to make is there are a lot of optimisation surface areas possible.
you may notice Kimi, GLM have also started telling how their model is able to optimise it's own inference pipeline
https://www.kimi.com/blog/kimi-k3
I don't know. That just sounds like throwing money at a problem until it goes away. I'm not convinced that is the correct path forward.
1. labs have lots of inference capacity 2. they will have domain experts working on this so their efficiency is gonna be exponentially more (can direct LLM better, save money, reach same results faster)
You can't exceed roofline performance on hardware. There is an performance cap you can hit. This recursive self improvement stuff lets you be closer to the pareto frontier, but the idea that it is leading to some exponential growth is a total pipe dream.
fair enough
You say this like that isn’t how the vast majority of problems are solved…
Does the edge matter? I know you added significant as your hedge, but once you have feedback, your gain is largely irrelevant. Gain buys you bandwidth, so we are constructing systems run by the most powerful corporations where they are now optimizing for latency, as Archer says, do you want to flash crash civilization? This is how you do it.
That’s not what a moat is =]
Which I believe was the word intended.
This is really cool - I really like the beam search idea,
this is the first time ive heard of beam search. i would have reached for a genetic algorithm of some sort, although it seems like some stochastic versions of beam search exist to avoid local minima. i wonder if there are any good frameworks for building these that agents can construct and use.
I think the question is how do you keep track of the ideas that the agent is pursuing - like I was working on this for implementing a fft, and doing the optimizations, but I held its hand and was like - hey let's go back and retry this older thing you discounted because of a 3 % slowdown.
beam search works when you have enumerable branches or some predefined dimension
Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.
I’ve tried it on simple UI tasks. Give it screenshot to work towards (or figma MCP), let it get screenshots from chrome to check its work.
The existing models are surprisingly bad at it.
It's really difficult to understand what your definition of surprisingly bad is. What was it continuously having problems with?
Just reporting work as done that’s not even close mainly. Then if you manually screenshot both and ask if they are the same it’s like “yeah looks great boss”. The visual understanding of the models is just leagues worse than their text/language understanding.
Or guessing colors rather than sampling from the image or pulling from figma is another stupid thing they do constantly.
I imagine Gemini would do better on these types of tasks. Curious to try it out.
People are always going to hate auto-research and "loop engineering". Because it's got 2 properties:
1) it's the only way to get something out of models (or people for that matter) that they don't know yet.
2) it's harder to do with an LLM than without. Not easier.
3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a robot leg on the ground is just to crash it into the ground.
What do you mean by 2?
LLMs will take shortcuts and cheat in ways that a human would realize are counterproductive.
you specify the goal. if the goal is achieved, it's achieved.
the code the LLM writes will be read and maintained and developed further by LLMs. so it doesn't really matter what it produces as long as all the tests are green and it achieves exactly what you want it to achieve.
depends on the implementation.
the claude loop/goal just decides it doesnt feel like doing it anymore and ends the loop or goal
in an other thread I've commented, but I use a benchmark - profile - verify - research - improve loop.
I agree with you but perhaps I'm misread you, how else would you do autoresearch and loop engineering without LLMs?
LLMs take shortcuts if allowed.
Humans do it ignorantly.
The LLMs will improve while average human IQ in the west dips closer and closer to the 80s on the global scale.
I've seen this several times now. Disturbing.
IMO, LLMs will be a dead end to anything close to AGI because of this and hallucinations.
We're missing something in the mix, which I suspect is some kind of advanced JEPA model.
Achieving a 232x speedup on a kernel via automated tooling is an incredible engineering feat. Fascinating read on optimization.