Firstly, can there be an easier way to stop a microVM mid execution in this single executable bottlefire format and then rerun that and it would start mid execution. (something akin to how criu does it?)
Secondly, and although this isn't related to your product but rather about microvm and that is if something like microvm could be run in normal cloud infrastructure?
I saw this article (https://blog.alexellis.io/how-to-run-firecracker-without-kvm...) and It mentioned PVM and I was thinking if this can be applied here because then I feel like if I can run a container as a microvm on big cloud and then stop it mid execution and rerun it elsewhere, then I have essentially got an incentive to use spot instances which are (I think) 80% more cheaper and I am just curious about it from a devops side of things.
> Firstly, can there be an easier way to stop a microVM mid execution in this single executable bottlefire format and then rerun that and it would start mid execution. (something akin to how criu does it?)
Not yet - Firecracker supports snapshotting so this should be doable though!
> if something like microvm could be run in normal cloud infrastructure?
Some cloud providers - like GCP and DigitalOcean - do support nested virtualization, and they work pretty well with Firecracker. Using VM migration to run stable workloads on spot instances sounds very interesting :)
> Some cloud providers - like GCP and DigitalOcean - do support nested virtualization, and they work pretty well with Firecracker. Using VM migration to run stable workloads on spot instances sounds very interesting :)
not necessarily. you can build custom kernel with pvm[1] and do it on aws.
Yes I also came to know across pvm. I feel like doing it on top of aws instances can bring a really nice way of migrating from spot isntances and paying less bills.
What are your thoughts on the other hand in using criu with docker and then deploying it on aws spot instances, is it possible
> What are your thoughts on the other hand in using criu with docker and then deploying it on aws spot instances, is it possible
I don't see why not really
but the last few years, spot has been reclaimed way too often and the price discount is not as good as it used to be (era 2016-2017) so I prefer to use saving plan now. although quite a big portion of our fleet still use spot.
hm that was an interesting take, I had seen this youtube video by codedamn [1] on how spot instances are really cheap and had always wondered why people weren't using this, well now I understand that the incentives have changed. Thanks for telling me, I didn't knew it or maybe the creator of that video had created it quite recently (10 months from now isn't that much of a time unless things have changed)
Have things changed quite a lot in 10 months or was it the author maybe overhyping the usecase I suppose.
I am really wondering but is there any software stack that can work with multi cloud approach the best way. I feel like typescript is really great for such purposes for the most part, I hope that this doesn't get counted as too off topic. I am not a dev ops guy but I just like being frugal and checking different options etc. and I am just wondering what is the best "just works" cloud 2025 without being too much expensive like vercel or netlify.
I genuinely hope that you can please consider the 1st question regarding snapshotting and its doability in bottlefire's / bake's roadmap.
Regarding the 2nd question, I feel like something can definitely be crafted that can enable running stable workloads on gcp/digitalocean spot instances and maybe what bake can do is really make the automation aspect of spot instances / VM migration easier...
Please don't get me wrong, this project looks really cool but I would actually like a first hand response as to (preferably) why this/bake project was created and when/why should someone use this..
I also have many more questions and I feel like haivng a community place can be really helpful here.
Although my open source purist heart wishes for you to use matrix, Its also understandable if you use discord. Do note that there are bridges so you could technically have both matrix and discord and bridge them.
Been trying to work out how to put all of this together locally but didn't know bake existed, looks awesome.
Any chance of looking into 9p for rootfs?
I've been wanting to grab images, extract locally and then launch the image in a lightweight VM whilst keeping the "guest" fs visible as a normal directory. (Complex but doable, I've had a PoC for a while but have been wanting to take it further)
You don't need it, it's essentially a hosted version of their tool bake[0], which is actually pretty slick - without any setup you can go from a system with just KVM to running any random docker container. I can think of a few use cases for this, especially anything where you want to use containers as the delivery mechanism
I hope you have read the article. They literally say as the other commentor points, that they essentially use bake. You could rather host something like this pretty trivially I suppose.
Interesting - I somehow didn't realize that KVM didn't require root access.
Also, I wonder if this could be adapted to use Apple's Hypervisor.framework. That one also doesn't require root and ought to be able to spin up and down very quickly.
Crun can be/is used by podman/docker too so you could think that its essentially stripping docker down to its core of running oci containers and just taking the most lightweight and (fastest?) approach and embedding it directly into the executable.
I am sure that I might get somethings wrong, I usually do, so if I have an error, please do let me know as I don't wish to spread misinformation
This project on the other hand seems to be using bake https://github.com/losfair/bake which seems to take something like a docker container and I think essentially push it into something like .iso using squashfs
The idea here is that they seem to embed firecracker itself into a single binary to actually just make it an executable that can run that microvm as it is.
And I think that bottlefire just seems to do something similar but they do it on their server side and seem to just provide you a binary to make things easy for you...
I am not sure with microvm's architecture but it seems that there are some additional features that you can get since they are VM, maybe you can get the ability of https://criu.org/Main_Page out of the box but this doesn't seem to be done right now but In my opinion is really very doable.
Another idea might be more security. I am going to refer to an article [1] which talks about flatpak but it has this line and I quote `containers are not sandboxed`, so I suppose that this can help blur the lines b/w containers by converting containers into microvm's and then leveraging vm's technology for sandboxing or some other mechanism rather easily as compared to docker (I see it use landlock which was new to me!)
I have 2 questions which I hope can be answered
Firstly, can there be an easier way to stop a microVM mid execution in this single executable bottlefire format and then rerun that and it would start mid execution. (something akin to how criu does it?)
Secondly, and although this isn't related to your product but rather about microvm and that is if something like microvm could be run in normal cloud infrastructure?
I saw this article (https://blog.alexellis.io/how-to-run-firecracker-without-kvm...) and It mentioned PVM and I was thinking if this can be applied here because then I feel like if I can run a container as a microvm on big cloud and then stop it mid execution and rerun it elsewhere, then I have essentially got an incentive to use spot instances which are (I think) 80% more cheaper and I am just curious about it from a devops side of things.
> Firstly, can there be an easier way to stop a microVM mid execution in this single executable bottlefire format and then rerun that and it would start mid execution. (something akin to how criu does it?)
Not yet - Firecracker supports snapshotting so this should be doable though!
> if something like microvm could be run in normal cloud infrastructure?
Some cloud providers - like GCP and DigitalOcean - do support nested virtualization, and they work pretty well with Firecracker. Using VM migration to run stable workloads on spot instances sounds very interesting :)
> Some cloud providers - like GCP and DigitalOcean - do support nested virtualization, and they work pretty well with Firecracker. Using VM migration to run stable workloads on spot instances sounds very interesting :)
not necessarily. you can build custom kernel with pvm[1] and do it on aws.
[1]: https://lwn.net/Articles/963718/
Yes I also came to know across pvm. I feel like doing it on top of aws instances can bring a really nice way of migrating from spot isntances and paying less bills.
What are your thoughts on the other hand in using criu with docker and then deploying it on aws spot instances, is it possible
> What are your thoughts on the other hand in using criu with docker and then deploying it on aws spot instances, is it possible
I don't see why not really
but the last few years, spot has been reclaimed way too often and the price discount is not as good as it used to be (era 2016-2017) so I prefer to use saving plan now. although quite a big portion of our fleet still use spot.
hm that was an interesting take, I had seen this youtube video by codedamn [1] on how spot instances are really cheap and had always wondered why people weren't using this, well now I understand that the incentives have changed. Thanks for telling me, I didn't knew it or maybe the creator of that video had created it quite recently (10 months from now isn't that much of a time unless things have changed)
Video [1] : https://www.youtube.com/watch?v=6hM4ZRIuD5g
Have things changed quite a lot in 10 months or was it the author maybe overhyping the usecase I suppose.
I am really wondering but is there any software stack that can work with multi cloud approach the best way. I feel like typescript is really great for such purposes for the most part, I hope that this doesn't get counted as too off topic. I am not a dev ops guy but I just like being frugal and checking different options etc. and I am just wondering what is the best "just works" cloud 2025 without being too much expensive like vercel or netlify.
Have a nice day!
Wow, I didn't know about PVM! So this should finally allow nesting of container/VM sandboxes?
yes you can now
I genuinely hope that you can please consider the 1st question regarding snapshotting and its doability in bottlefire's / bake's roadmap.
Regarding the 2nd question, I feel like something can definitely be crafted that can enable running stable workloads on gcp/digitalocean spot instances and maybe what bake can do is really make the automation aspect of spot instances / VM migration easier...
Please don't get me wrong, this project looks really cool but I would actually like a first hand response as to (preferably) why this/bake project was created and when/why should someone use this..
I also have many more questions and I feel like haivng a community place can be really helpful here.
Although my open source purist heart wishes for you to use matrix, Its also understandable if you use discord. Do note that there are bridges so you could technically have both matrix and discord and bridge them.
Best wishes for your project.
Been trying to work out how to put all of this together locally but didn't know bake existed, looks awesome.
Any chance of looking into 9p for rootfs? I've been wanting to grab images, extract locally and then launch the image in a lightweight VM whilst keeping the "guest" fs visible as a normal directory. (Complex but doable, I've had a PoC for a while but have been wanting to take it further)
Why do you need a "service" for this? It looks like something you could run very well locally.
You don't need it, it's essentially a hosted version of their tool bake[0], which is actually pretty slick - without any setup you can go from a system with just KVM to running any random docker container. I can think of a few use cases for this, especially anything where you want to use containers as the delivery mechanism
[0]: https://github.com/losfair/bake
I hope you have read the article. They literally say as the other commentor points, that they essentially use bake. You could rather host something like this pretty trivially I suppose.
Interesting - I somehow didn't realize that KVM didn't require root access.
Also, I wonder if this could be adapted to use Apple's Hypervisor.framework. That one also doesn't require root and ought to be able to spin up and down very quickly.
How does it differ from docker2exe [1] or dockerc [2]?
[1] https://github.com/rzane/docker2exe?tab=readme-ov-file#embed...
[2] https://github.com/NilsIrl/dockerc
I don't know about the [1] but I think that [2] actually embed's crun https://github.com/containers/crun in the executable itself.
Crun can be/is used by podman/docker too so you could think that its essentially stripping docker down to its core of running oci containers and just taking the most lightweight and (fastest?) approach and embedding it directly into the executable.
I am sure that I might get somethings wrong, I usually do, so if I have an error, please do let me know as I don't wish to spread misinformation
This project on the other hand seems to be using bake https://github.com/losfair/bake which seems to take something like a docker container and I think essentially push it into something like .iso using squashfs The idea here is that they seem to embed firecracker itself into a single binary to actually just make it an executable that can run that microvm as it is.
And I think that bottlefire just seems to do something similar but they do it on their server side and seem to just provide you a binary to make things easy for you...
I am not sure with microvm's architecture but it seems that there are some additional features that you can get since they are VM, maybe you can get the ability of https://criu.org/Main_Page out of the box but this doesn't seem to be done right now but In my opinion is really very doable.
Another idea might be more security. I am going to refer to an article [1] which talks about flatpak but it has this line and I quote `containers are not sandboxed`, so I suppose that this can help blur the lines b/w containers by converting containers into microvm's and then leveraging vm's technology for sandboxing or some other mechanism rather easily as compared to docker (I see it use landlock which was new to me!)
article [1]: https://hanako.codeberg.page/
How is the performance of the mounted host directories? It’s some file server over vsock?
Does Firecracker support snapshotting the GPU state of a MicroVM? Can you suspend a pyToch session?
I'd be interested more generally in what is supported in terms of GPU. Is there such an overview somewhere?
Looks awesome :)
We're doing something similar at E2B, we should chat!
I am very unfamiliar with this. I took a look, but couldn't find any information. How is networking configured?
Seems to use a SOCKS5 proxy over tun2socks[1] to then use the host's networking setup https://github.com/losfair/bake/blob/ddf1edaa9daf340bd167794... so I think the short version to your question is "proxy server"
1: https://github.com/xjasonlyu/tun2socks#readme
Now we need a gui for firecracker like portainer !
Would proxmox work?
Would love to have the ability to spin up firecracker micro vms in proxmox as an alternative to qemu