The motivation is simply to have the runners share a local docker build cache rather than pushing and pulling cache from registry which is a ridiculous best practice.
We’ve been using runs on for maybe a year now. It’s a managed service for self hosted runners. Pretty great if you want to get off expensive GH actions boxes and do self hosting but don’t have the time to manage it yourself.
I'm sure my scale is many magnitudes smaller but as a solo developer/founder I ran into this same issue recently as I've been creating more PRs than normal and also increasing my GH Actions usage for things like LLM review and just more unit/e2e/etc tests.
Last month I ran up $45 of overage spending (yes, I know, I know, this is tiny but for me it's not) and decided to make use of my homelab that was effectively twiddling it's thumbs most of time.
I settled on small VMs that I delete and restore from a snapshot after every run and it's worked out great so far. They only take up a tiny bit of ram when they are first started up and only take more from the system when they pick up a job. Even though each worker can use 6GB they rarely do and I can run 6-8 of these on my box safely (even if all use 6GB).
I used a custom image containing only what I needed and that's worked really well for me so far and it's easy to add more to it if needed in the future.
After this change I went from spending $2-4/day to <$0.25 (I left a few very small tasks on GH Actions).
It's not always the right call and for paid workers I'd reach for WarpBuild or similar most of the time (WarpBuild still runs my deploys/iOS builds) but for all the PR-check actions, those work just fine on my local server.
With Claude it was super easy to get set up. It's not a very complicated process at all but I was always unsure of the ROI for the time to set it up and tune it. After the overage bill I figured I'd try it out and it worked out even better than I had hoped for (including having a number of the jobs run faster since my workers were bigger/better/faster/stronger than the default GH worker, no surprise there). I even considered running 1-2 VMs on my MBP as well so it could pick up some of the load.
All that said, if your GH Actions bill is going up (which it probably is), I can highly recommend looking into running your own workers for at least part of the load.
AI slop is strong with this one... I saw "per-runner swap" and I was wondering how it's implemented, given that Linux did not support swap partitioning last time I've checked
Turns out it's just a lie. The script creates a swapfile per runner, but then merges them all into one big happy system pool, where every swapfile is shared between every runner.
I had a different plan in mind, keep everything in cheap s3, save a declarative script on a public repository that deploys forgejo and harbor containerized on the local ephemeral Origin/cache and pull heavy sources and code from s3.
Some context for why this hit us harder than most: we run a ~20-agent dev harness — coding agents that work in parallel, open PRs, and push commits more or less around the clock. Great for shipping. Rough on your CI meter. A team of humans pushes a few dozen times a day; 20 tireless agents push a lot more, and every push fires the full test matrix. Our GitHub Actions bill climbed accordingly — nobody made a bad decision, the machines just never stop typing.
So we did the obvious thing: moved CI to a single self-hosted box. Fixed monthly cost instead of a per-minute meter, and the builds actually got faster because the box keeps a warm Docker cache and doesn't cold-start every job.
It sounds like a five-minute job — spin up a VM, run config.sh, done. It mostly is. But three things bit us that nobody warns you about, and they're the reason we bothered to package this up:
1. OOM. A memory-constrained box running a couple of heavy jobs at once just gets its builds OOM-killed, silently, and you spend an afternoon blaming flaky tests. Fix: give the box swap — specifically one swapfile per runner — and set vm.swappiness=10 so it only leans on swap under real pressure. Swap isn't the hot path; it's the airbag.
2. "No space left on device" — but the disk looks half empty. The culprit was /tmp silt: dead job-workspace directories that never get cleaned up, slowly filling the disk until a big build tips over. Fix: give each runner its own /scratch (so they don't fight over one /tmp), and add a systemd-tmpfiles rule that ages /tmp + scratch every 6h. Set-and-forget.
3. The test database. Our suite needs Postgres. One shared instance every job writes into means parallel runs clobber each other's data and burn through connections; the other extreme, a fresh Postgres container per job, is heavier and slower than it needs to be. What worked: one small native Postgres on the box (128MB shared_buffers — deliberately tiny), shared by all runners, with each run isolated in its own schema — create it on the way in, point search_path at it via DATABASE_URL, drop it in an always-run cleanup step. Small-and-shared beats N heavy instances on a constrained box, and per-run schemas make concurrent runs safe without paying for a container spin-up on every job.
The operating rule we landed on for an 8 GB box: provision 3 runners, but cap concurrency at 2. The third is a warm spare and gives you headroom; running all three heavy jobs at once is how you meet the OOM killer again. Bigger box, bump both numbers — it's just policy.
We wrote it up as a few idempotent scripts + a README so it's reproducible instead of tribal knowledge:
- provision-box.sh — swap / per-runner scratch / reaper
- install-runners.sh — download, register, systemd-install N runners
- provision-postgres.sh — one small shared Postgres, with a per-run-schema example workflow
- check-runners.sh — how many are actually online (a dropped runner silently slow-walks every PR)
Everything's parameterized through a config.env, and the README has the exact reference box config (a small Hetzner VM, Ubuntu 24.04, Docker) with copy-paste setup steps.
Fair warning / the tradeoffs: self-hosted runners are great for a private repo you control. Do not point them at a public repo that runs untrusted-PR workflows — that's remote code execution on your box by design. One box is also a single point of failure, and you own patching it now. For our case (private repo, trusted team + agents, predictable cost) it's been a clear win.
ah. it writes and i edit it. me and Stanley Duckenmiller. it's a way better writer than i am. so is my cousin who is a golf writer for 30 years. so you're correct. i am excellent at editing though :)
Your parent comment reads like AI slop, so you may not be as good at editing as you think. Worth thinking about the impression you’re conveying about yourself.
I use this script for provisioning runners (with org-level single use access token).
https://github.com/vbem/multi-runners
The motivation is simply to have the runners share a local docker build cache rather than pushing and pulling cache from registry which is a ridiculous best practice.
How is that not a native thing with Actions? GitLab had multi runners since early on.
This looks awesome and just what I need, with the difficulty of ARC in my corp environment
We’ve been using runs on for maybe a year now. It’s a managed service for self hosted runners. Pretty great if you want to get off expensive GH actions boxes and do self hosting but don’t have the time to manage it yourself.
https://runs-on.com/
Shame this is AWS specific and not just a Docker image
I'm sure my scale is many magnitudes smaller but as a solo developer/founder I ran into this same issue recently as I've been creating more PRs than normal and also increasing my GH Actions usage for things like LLM review and just more unit/e2e/etc tests.
Last month I ran up $45 of overage spending (yes, I know, I know, this is tiny but for me it's not) and decided to make use of my homelab that was effectively twiddling it's thumbs most of time.
I settled on small VMs that I delete and restore from a snapshot after every run and it's worked out great so far. They only take up a tiny bit of ram when they are first started up and only take more from the system when they pick up a job. Even though each worker can use 6GB they rarely do and I can run 6-8 of these on my box safely (even if all use 6GB).
I used a custom image containing only what I needed and that's worked really well for me so far and it's easy to add more to it if needed in the future.
After this change I went from spending $2-4/day to <$0.25 (I left a few very small tasks on GH Actions).
It's not always the right call and for paid workers I'd reach for WarpBuild or similar most of the time (WarpBuild still runs my deploys/iOS builds) but for all the PR-check actions, those work just fine on my local server.
With Claude it was super easy to get set up. It's not a very complicated process at all but I was always unsure of the ROI for the time to set it up and tune it. After the overage bill I figured I'd try it out and it worked out even better than I had hoped for (including having a number of the jobs run faster since my workers were bigger/better/faster/stronger than the default GH worker, no surprise there). I even considered running 1-2 VMs on my MBP as well so it could pick up some of the load.
All that said, if your GH Actions bill is going up (which it probably is), I can highly recommend looking into running your own workers for at least part of the load.
thanks! mine hit $145. i'll have to check out your solution too.
AI slop is strong with this one... I saw "per-runner swap" and I was wondering how it's implemented, given that Linux did not support swap partitioning last time I've checked
Turns out it's just a lie. The script creates a swapfile per runner, but then merges them all into one big happy system pool, where every swapfile is shared between every runner.
I had a different plan in mind, keep everything in cheap s3, save a declarative script on a public repository that deploys forgejo and harbor containerized on the local ephemeral Origin/cache and pull heavy sources and code from s3.
Some context for why this hit us harder than most: we run a ~20-agent dev harness — coding agents that work in parallel, open PRs, and push commits more or less around the clock. Great for shipping. Rough on your CI meter. A team of humans pushes a few dozen times a day; 20 tireless agents push a lot more, and every push fires the full test matrix. Our GitHub Actions bill climbed accordingly — nobody made a bad decision, the machines just never stop typing.
So we did the obvious thing: moved CI to a single self-hosted box. Fixed monthly cost instead of a per-minute meter, and the builds actually got faster because the box keeps a warm Docker cache and doesn't cold-start every job.
It sounds like a five-minute job — spin up a VM, run config.sh, done. It mostly is. But three things bit us that nobody warns you about, and they're the reason we bothered to package this up:
1. OOM. A memory-constrained box running a couple of heavy jobs at once just gets its builds OOM-killed, silently, and you spend an afternoon blaming flaky tests. Fix: give the box swap — specifically one swapfile per runner — and set vm.swappiness=10 so it only leans on swap under real pressure. Swap isn't the hot path; it's the airbag.
2. "No space left on device" — but the disk looks half empty. The culprit was /tmp silt: dead job-workspace directories that never get cleaned up, slowly filling the disk until a big build tips over. Fix: give each runner its own /scratch (so they don't fight over one /tmp), and add a systemd-tmpfiles rule that ages /tmp + scratch every 6h. Set-and-forget.
3. The test database. Our suite needs Postgres. One shared instance every job writes into means parallel runs clobber each other's data and burn through connections; the other extreme, a fresh Postgres container per job, is heavier and slower than it needs to be. What worked: one small native Postgres on the box (128MB shared_buffers — deliberately tiny), shared by all runners, with each run isolated in its own schema — create it on the way in, point search_path at it via DATABASE_URL, drop it in an always-run cleanup step. Small-and-shared beats N heavy instances on a constrained box, and per-run schemas make concurrent runs safe without paying for a container spin-up on every job.
The operating rule we landed on for an 8 GB box: provision 3 runners, but cap concurrency at 2. The third is a warm spare and gives you headroom; running all three heavy jobs at once is how you meet the OOM killer again. Bigger box, bump both numbers — it's just policy.
We wrote it up as a few idempotent scripts + a README so it's reproducible instead of tribal knowledge:
https://github.com/senoff/self-hosted-ci-runner
- provision-box.sh — swap / per-runner scratch / reaper - install-runners.sh — download, register, systemd-install N runners - provision-postgres.sh — one small shared Postgres, with a per-run-schema example workflow - check-runners.sh — how many are actually online (a dropped runner silently slow-walks every PR)
Everything's parameterized through a config.env, and the README has the exact reference box config (a small Hetzner VM, Ubuntu 24.04, Docker) with copy-paste setup steps.
Fair warning / the tradeoffs: self-hosted runners are great for a private repo you control. Do not point them at a public repo that runs untrusted-PR workflows — that's remote code execution on your box by design. One box is also a single point of failure, and you own patching it now. For our case (private repo, trusted team + agents, predictable cost) it's been a clear win.
Does anyone write anything anymore themselves? This is insulting, even for an AI coding enthusiast like myself.
because you want to code and stand up ci runners yourself or because i just shared what i did for anyone to adopt including the code?
not sure what's insulting. I just shared why and how I did this, but hey.
I think it's about the comment you wrote, or rather that AI (appears to have) written entirely.
ah. it writes and i edit it. me and Stanley Duckenmiller. it's a way better writer than i am. so is my cousin who is a golf writer for 30 years. so you're correct. i am excellent at editing though :)
Your parent comment reads like AI slop, so you may not be as good at editing as you think. Worth thinking about the impression you’re conveying about yourself.
Yeah I completely gave up after a paragraph of overly wordy AI garbage.
There's no way just writing it out would have been worse
I’d suggest adding emdashes to your filter list.