Thanks! I tried to make the sandbox-eval stdin interface as simple as possible so you could easily plug it anywhere. The executable docs run an actual full shell in a VM, since they're so fast and cheap to spin up- I hope the example made it clear how to use this feature!
Built in replicated database and filesystem is pretty rad. Fly leaves this as an exercise for the user.
You should license the backend to enterprises that want a cheaper/lighter weight option to kubernetes for the surge of agentically coded internal apps.
This looks awesome! I really love the clean and minimalist website design too.
I've been excited about using (lib)krun as a drop in replacement for crun/runc for container engines, to make them run qemu microvms. I'm exploring what it could look like if every process of an OS ran like that. I envy you, would love to work on something like this!
Hey this is amazing! I see the sqlite db is replicated but how does it handle multiple instances as writers? Curious if the underlying sqlite engine is syzy or if there's other restrictions to be aware of
Thanks! Yes, the write-replication part is syzy, which hooks into the standard unmodified sqlite engine that reads and writes normal sqlite db files locally - syzy just keeps multiple instances in sync by capturing writes and applying them elsewhere.
It handles multiple writers by capturing changes as logical operations, then resolving conflicts in a globally-deterministic order based on their contents based on a CRDT model.
There are some limitations (see https://github.com/wjordan/syzy/blob/main/sqlite/docs/LIMITA...), mostly around primary keys (they have to be non-NULL) and UNIQUE constraints (there's some subtle differences in behavior between nullable and NOT NULL uniqueness). But in general, the goal is for all of SQLite, DML and DDL, to just work normally!
A replicated SQLite is very cool! One question I would have is why expose that as the storage instead of the platform providing a managed storage API? I guess I might be concerned about replication lag and complexity for consumers with the replicated SQLite.
Is it to have more sympathy for LLMs? More 1:1 with a local dev environment?
The idea is for you to be able to publish a stateful app 'everywhere' that moves with your workload without micro-managing regions or resources, like a traditional CDN does for static content. The built-in replicated SQLite is what makes this possible and convenient.
The database is eventually consistent by design. For applications that can tolerate eventual consistency across regions, this gives fast local queries everywhere, and services that are resilient to network partitions. A database sitting alone in Virginia gives you slow requests waiting on queries from distant regions, and global downtime if that one load-bearing region goes unavailable for any reason.
The platform does also support persistent volumes (https://kedge.dev/docs/volumes), which is the managed storage API. I've framed it as an escape hatch for those that still need to run a more traditional service with stable members in a fixed region.
On the HN frontpage is Superlogical (I have no affiliation) right now, and I used the shell provided by the Kedge News demo to `ssh superlogical.jobs` from my browser on a phone on an instance started seconds earlier. :)
I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
> I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
Persistent volumes use local NVMe that continuously syncs to object storage, so a similar architecture to Fly Sprites. It automatically recovers from host failures (with on-demand lazy restore), and you can manually migrate stable members to another region if needed.
I've positioned persistent volumes as an escape hatch for when you need an existing database app running in a VM, or when performance requirements outgrow the built-in replicated SQLite.
Gotcha, interesting. I would love it to death if there were a way to run some CLI, maybe on the system, that gave you information on the current replication state of the disk!
Pretty much! It's not dynamic memory hotplug or resizing, just dynamic billing. Usage is based on the actual RSS used by the app so you can think of the VM's defined size as just a limit, for both CPU and RAM. This means you don't have to spend time profiling and right-sizing instances to optimize costs, just make your application fast and efficient and don't worry about it!
Two questions about the fork boundary. The snapshot is taken once when the process calls listen, so every later instance is a clone of the same frozen memory image, and anything seeded during init is identical across every clone in every region. That is exactly what the Linux vmgenid driver exists for, and it is why Lambda SnapStart had to ship explicit reseeding guidance. I could not find entropy, RNG or reseed anywhere in the runtime, performance or security docs, so is there a post-restore reseed hook, and does it fire before the first accept?
The other one is syzy. CRDT.md says eviction is operator-driven, that PRUNING.md is a design-stage contract and not yet code, and that a replica lagging beyond the hold window may transiently observe the departing duplicate, while the product headline offers scale to zero when idle, which mints offline replicas continuously. Who calls syzy_evict in the managed product?
These are both later-stage detail questions for when this project is further hardened for production workloads, but here's where each of these currently stands:
- Yes, there is a post-restore kernel reseed hook that happens after fork, and just before the trapped listen is continued. It doesn't use the vmgenid driver, it just calls the RNDRESEEDCRNG ioctl, so it would not be compatible with applications that depend on vmgenid notifications to reseed userspace PRNGs.
- In Syzy, the full pruning implementation is deferred for now; at the managed product's current small scale, it can simply eat the tiny overhead costs of un-collected garbage, while the user is only billed for the logical storage space consumed.
The warm-pool tree (kernel → base runtime → app) with CoW memory pages is a nice design — that's the part that makes 3ms believable rather than marketing.
Question on the shared SQLite: with multi-writer CRDT replication and eventual consistency, how do you handle operations that need global agreement — unique usernames at signup, or "did my vote count exactly once"? The HN-clone demo has both auth and voting, so I'm curious whether the HTML app model resolves those at the CRDT layer or accepts last-write-wins and tolerates the edge cases.
https://kedge.dev/docs/sandboxes
Caught my eye with this one. I'll be taking a closer look as this is one of those "I'd rather let someone else do it." type infrastructure bits :)
Thanks! I tried to make the sandbox-eval stdin interface as simple as possible so you could easily plug it anywhere. The executable docs run an actual full shell in a VM, since they're so fast and cheap to spin up- I hope the example made it clear how to use this feature!
This is also extremely interesting to me.
An instant, ephemeral cloud sandbox is pretty handy.
Built in replicated database and filesystem is pretty rad. Fly leaves this as an exercise for the user.
You should license the backend to enterprises that want a cheaper/lighter weight option to kubernetes for the surge of agentically coded internal apps.
This looks awesome! I really love the clean and minimalist website design too.
I've been excited about using (lib)krun as a drop in replacement for crun/runc for container engines, to make them run qemu microvms. I'm exploring what it could look like if every process of an OS ran like that. I envy you, would love to work on something like this!
Hey this is amazing! I see the sqlite db is replicated but how does it handle multiple instances as writers? Curious if the underlying sqlite engine is syzy or if there's other restrictions to be aware of
Thanks! Yes, the write-replication part is syzy, which hooks into the standard unmodified sqlite engine that reads and writes normal sqlite db files locally - syzy just keeps multiple instances in sync by capturing writes and applying them elsewhere.
It handles multiple writers by capturing changes as logical operations, then resolving conflicts in a globally-deterministic order based on their contents based on a CRDT model.
There are some limitations (see https://github.com/wjordan/syzy/blob/main/sqlite/docs/LIMITA...), mostly around primary keys (they have to be non-NULL) and UNIQUE constraints (there's some subtle differences in behavior between nullable and NOT NULL uniqueness). But in general, the goal is for all of SQLite, DML and DDL, to just work normally!
this is super cool, bookmarking to give it a whirl
> As a demo, I made a Hacker News clone with story submission, votes, comments and auth in about 60 lines of Markdown, plus CSS
although why would you use it for evil
> although why would you use it for evil
I couldn't help it! Ben Johnson recently wrote a Litestream post touching on this topic [1]:
> multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain
I offer my sincerest apologies if this database release may have inadvertently opened the gates of Hell.
[1] https://fly.io/blog/litestream-writable-vfs/
A replicated SQLite is very cool! One question I would have is why expose that as the storage instead of the platform providing a managed storage API? I guess I might be concerned about replication lag and complexity for consumers with the replicated SQLite.
Is it to have more sympathy for LLMs? More 1:1 with a local dev environment?
The idea is for you to be able to publish a stateful app 'everywhere' that moves with your workload without micro-managing regions or resources, like a traditional CDN does for static content. The built-in replicated SQLite is what makes this possible and convenient.
The database is eventually consistent by design. For applications that can tolerate eventual consistency across regions, this gives fast local queries everywhere, and services that are resilient to network partitions. A database sitting alone in Virginia gives you slow requests waiting on queries from distant regions, and global downtime if that one load-bearing region goes unavailable for any reason.
The platform does also support persistent volumes (https://kedge.dev/docs/volumes), which is the managed storage API. I've framed it as an escape hatch for those that still need to run a more traditional service with stable members in a fixed region.
This is phenomenal, great work!
On the HN frontpage is Superlogical (I have no affiliation) right now, and I used the shell provided by the Kedge News demo to `ssh superlogical.jobs` from my browser on a phone on an instance started seconds earlier. :)
I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
Thanks for the kind words!
> I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?
Persistent volumes use local NVMe that continuously syncs to object storage, so a similar architecture to Fly Sprites. It automatically recovers from host failures (with on-demand lazy restore), and you can manually migrate stable members to another region if needed.
I've positioned persistent volumes as an escape hatch for when you need an existing database app running in a VM, or when performance requirements outgrow the built-in replicated SQLite.
Gotcha, interesting. I would love it to death if there were a way to run some CLI, maybe on the system, that gave you information on the current replication state of the disk!
> Billed per-second on actual resources consumed, … memory tracks active resident pages
Wait, are you saying it has dynamic vertical scaling of RAM?
Pretty much! It's not dynamic memory hotplug or resizing, just dynamic billing. Usage is based on the actual RSS used by the app so you can think of the VM's defined size as just a limit, for both CPU and RAM. This means you don't have to spend time profiling and right-sizing instances to optimize costs, just make your application fast and efficient and don't worry about it!
Two questions about the fork boundary. The snapshot is taken once when the process calls listen, so every later instance is a clone of the same frozen memory image, and anything seeded during init is identical across every clone in every region. That is exactly what the Linux vmgenid driver exists for, and it is why Lambda SnapStart had to ship explicit reseeding guidance. I could not find entropy, RNG or reseed anywhere in the runtime, performance or security docs, so is there a post-restore reseed hook, and does it fire before the first accept?
The other one is syzy. CRDT.md says eviction is operator-driven, that PRUNING.md is a design-stage contract and not yet code, and that a replica lagging beyond the hold window may transiently observe the departing duplicate, while the product headline offers scale to zero when idle, which mints offline replicas continuously. Who calls syzy_evict in the managed product?
These are both later-stage detail questions for when this project is further hardened for production workloads, but here's where each of these currently stands:
- Yes, there is a post-restore kernel reseed hook that happens after fork, and just before the trapped listen is continued. It doesn't use the vmgenid driver, it just calls the RNDRESEEDCRNG ioctl, so it would not be compatible with applications that depend on vmgenid notifications to reseed userspace PRNGs.
- In Syzy, the full pruning implementation is deferred for now; at the managed product's current small scale, it can simply eat the tiny overhead costs of un-collected garbage, while the user is only billed for the logical storage space consumed.
LLM comments are not allowed.
certainly sounded like one of
The warm-pool tree (kernel → base runtime → app) with CoW memory pages is a nice design — that's the part that makes 3ms believable rather than marketing.
Question on the shared SQLite: with multi-writer CRDT replication and eventual consistency, how do you handle operations that need global agreement — unique usernames at signup, or "did my vote count exactly once"? The HN-clone demo has both auth and voting, so I'm curious whether the HTML app model resolves those at the CRDT layer or accepts last-write-wins and tolerates the edge cases.
Fun fact: we also built our cloud platform based on Fly.io’s ideas, but we supercharged it with a custom Rust filesystem.
This platform powers full-stack applications for business users, including Playcode.ai.
For more information about our cloud platform, visit: https://playcode.io/cloud