For context, since a lot of people on HN haven't worked on games - this is not intended to compete with Git for general software development. This is a competitor with Perforce for game development.
Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files that game developers need to collaborate on. For example, one artist might need to obtain an exclusive lock on some art assets while editing them, because there is no sane way to merge two artists' async edits.
The SOTA in this area is Perforce (https://www.perforce.com/products/helix-core), a proprietary system. From what my gamedev friends tell me, when Perforce works it's great, but it hits enough snags that you need a tools engineer to manage it and occasionally fix issues manually. Git LFS is an alternative, but my gamedev friends all prefer Perforce especially when working on team projects beyond like 3-4 people.
Something else that git isn't good at: permissions. In gamedev, you might have proprietary work that you want to restrict to certain users. In P4, you can add restrictions to certain directories for only those who have signed the required NDAs. That's not something that you can do in git: it's all or nothing. Maybe you can set something up with submodules, but that's going to upend your repository if you hadn't planned for it.
I once worked in a git repository that required those kinds of restrictions.
This was within a bank and the code in question was related to enabling Apple Pay from within the banking application. The consequences of that information and code leaking or being seen by anyone who had not signed the NDA were very serious (don't remember the details but it made the lawyers were extremely stressed about it).
Needing to figure out a way to protect those parts of the codebase it was decided in the end that the "easiest" way of doing this was to split the repository in half, with the actual artifact building taking place from the half that had the NDA code. The rest of the application (basically the whole application) was then used as a dependency by it.
Still didn't quite solve the issue, but access to that repository was heavily controlled.
As others have said, it's Apple and they do not take kindly to other people leaking their technology/announcements ahead of time.
See also: the time that ATI's CEO told his employees that their chips would be powering Apple's to-be-announced hardware a few days before the announcement. Steve Jobs responded by pulling all of ATI's hardware from its demo units at the announcement, not mentioning ATI at all, cancelling a joint demonstration of the Radeon card that was going to be in the system, and never partnering with ATI again.
Except of course shipping ATI hardware for years afterwards, then also using nvidia, then dropping nvidia and only using ATI/AMD until transitioning to Apple Silicon.
Because it's Apple. They are huge, have scary lawyers, write scary contracts, and want to "delight the user" with features only when they announce them. They hate leaks, and demand separate teams for basically any/all development.
It seems this wasn’t about the code itself, it was about Apple Pay not being announced yet. So only people under NDA would be allowed to even know what they are working on.
> security by obscurity thing... What am I missing here?
You are looking at the problem from the wrong direction.
If you build a honeypot, to trap hackers, does it behove you to explain what the bait is, and how the trap works?
Know your customer, fraud detection heuristics, finger prints, behavioral triggers are all areas where banks, and financial institutions need to keep the sauce secret. Telling the other party "how" you catch them just gives them the steps of what not to do.
AFAIK the issue with using submodules is you still need the rights to pull the other source repo. However, you can use submodules or LFS to pull a specific build artifact from a build artifact repo or source instead of the source repo, which provides a neat way to manage the dep without fattening the main repo and allows the source repo to be kept separate and high security. I'd certainly do this before changing RCS/VCS solutions. That said, reverse engineering has become relatively trivial in the AI age so the practical utility of providing built rather than source elements is dropping.
I ended up writing my own layer over git for permissions for a specific client a long time ago. It has a huge amount of useful features - sadly, I never took the idea further.
That depends on you distributing working keys for any components you want to restrict access to, and managing those keys for all users, revoking them when access permissions change, etc. It's a lot more complex, more work, and harder to manage than centralized RBAC or similar.
Agreed. I use and love git crypt, but it doesn't get enough use. I think because it's easy to screw up gpg keys. Most of my uses (for one to three devs) have become symmetric keys shared out-of-band instead of using gpg keys because we've had lots of onboarding pain even from people who are quite competent. There are just a lot of sharp edges in gpg that you don't know when you don't know.
Not really, precisely because it’s decentralized. You can’t audit whether a user accessed one of the hidden files, or really even who can access it once you accept the reality of the risk that some team will put a key on S3 or a shared drive or whatever.
It’s fine for things that you want devs to be able to see without the Git host being able to see them, it’s less good at RBAC because there’s no real “identity” component at read-time.
That's by design of git, you can't forget that git is first developed for a bazaar model of information flow, especially with a big decentrailized project like the Linux Kernel, not the silo and isolated corporate NDA and closed model you described. Git prefers open information and discourages information closures and segregation of information by placing restrictions exactly like this.
Git enthusiast would often tell you to do this separately with a submodule, and set permission on the version control forge software level (which means Gitea/Github private RBAC access to certain repos for cloning), sure, but that is also painful as hell.
But my point is that all of this is exactly by design from Linus Torvalds's need for Linux Kernel to replace BitKeeper. Git simply isn't the tool for everything, it was developed for a software project with liberalism in mind, but corporate stuff is monoculture and prefers proprietary, shut-in model, and the eat your own dog food mindset, and no wonder it is so painful to deal with.
Git submodules aren’t convenient either. For the silo and corporate development use case, just use multiple repositories and make your build tool aware of multiple repositories. It is slightly less painful than submodules.
I feel like submodules could be a lot easier to work with if the git command made it easy to update all submodules in one go based on branch head for the submodule.
One thing I don't like about Git LFS is that there is no way to delete very old history. It's the 'git spirit' to not allow deleting history, but in the context of LFS it sounds horrible. Especially if you use Github.
If there is an asset that is updated very frequently in the early stage of development, you'll be charged for all the storage for the rest of the repo's life. That happens a lot in gamedev: most assets go back and forth early on but once it's done no one will touch them ever.
There's also a new player called diversion (diversion.dev) which I think may be a YC startup? Anyway it takes a different approach of being more like Google drive but bringing in VCS behavior making it more indie and designer friendly.
At my previous game-dev-company job we ended up splitting things up into:
1. Code - Git
2. WIP art, shared assets (logos, marketing materials, etc) - Google Drive (because things are often changing, getting passed around, etc)
3. Finished assets (PSD files you're done with, or you think you're done with) - SVN (because we wanted a log of who contributed to what, wanted artists to be able to pick up where someone else left off; having a log of who made changes to a given PSD)
4. Assets rendered out to PNG to include in the app bundle/publish to the static file servers - Git (because those files never changed after being published so the git history wasn't polluted with unneeded files)
I've also used LFS, which is... a fine workaround, but still not great. Users who don't have it configured can still commit binary blobs; users who don't have it configured will clone files incorrectly; if the LFS server is slow, unavailable, unreliable, then the system starts to behave oddly; you need a Git server that supports it.
It was a huge hassle to manage; having a system like this would have been a godsend at that company, and if I still worked there I would be spending all day importing our codebase and assets into it to see how well it works.
what happens if WIP stuff has rapid shifts or changes? art direction changes on the product level, etc? or even something as simple as an asset designer quits or get sick for a while
SVN makes sense cuz it's done and dusted, but I could see the Drive gettin real messy real fast if things change a lot
I wonder how useful this could be as a generalized version control for regular user systems, as a way to rollback, or scrub through history. Presumably if this is designed to work at Epic and Big Game studio scale, it should work at home computer scale
This presumption has destroyed far, far more companies and projects than the opposite assumption (that something built for small will scale to big, then doesn't).
P4 is also really well integrated into IDEs and UE Editor so that I don't need to think about it as much as I need, compared to Git. Locking assets, releasing them, merging into streams etc., is overall pretty streamlined. When it works, it's great, but when it doesn't work though, it's pretty hard to diagnose issues.
A significant part of my job, unfortunately, is helping people fix their workspaces when Perforce (p4) goes bad, or creating guardrails and wrappers to stop Perforce doing bad things.
In fairness, p4 predates most of the VCSes we consider "modern", so I empathize with a lot of the underlying architecture decisions. However, it has and continues to utterly fail at improving at a reasonable pace.
For example:
- p4 tracks file metadata of client workspaces on the server (sync'ed locally, opened for edit, file revision, etc) and uses this as the basis to avoid doing unneeded work. If this becomes desync'ed, a reconcile or force sync must be used. A reconcile can take hours, potentially days; it tries do detect file moves by default, so likely at least O(c^n) for some c>1. I have never personally seen a default reconcile operation _complete_ over any modestly large game code base, and in practice, people accumulate a litany of workarounds and scripts to fix this for themselves.
- Scripting p4 is a nightmare. Documentation is poor, schemas do not exist, and all the language-specific libraries are just thin wrappers over its C++ API.
- By default, p4 "helps" you with text files by "correcting" line endings on sync or even converting between encodings. This works until you have a mixed-OS environment, and discover a part of the pipechain that _must_ have a certain style. There are various levers to pull to make this better, but I've yet to find something fool proof.
- By default, p4 keeps flies read-only, only unlocking them when explicitly marked as being edited. This means, to avoid having to do this manually, every tool you use needs to be p4-aware. Or, you can turn this off, and choose to contend reconcile instead. (See above)
- Branching a modest game project, with, say, Unreal source code, can take hours. And this is the quick version where you ask the server to simply create new metadata, with no file transfer to a client.
- p4 is licensed by the user-account. Every user entity in p4 not intended exclusively for performing backups and maintenance operations counts toward this, including users required to integrate with other services. Plus, often times, these integration users must have admin access to be useful. The security posture is horrific.
It can be SOTA and still be garbage if there's nothing better (and there's nothing better, sadly). This is extremely exciting for anyone who's had to manage revision control for game devs.
Google still uses Piper, which started as a Perforce clone, though many people use it through a frontend like `fig` (try pronouncing ‘Piper HG’) or `jj`.
That sucks, git is so absolutely horrible. It's crazy to me that nobody has made anything better yet. Although I could start that myself and yet have not.
SVN was more straightforward to use, but that straightforwardness lost a lot in terms of fidelity.
The fact that it was easy to clone a subdirectory was nice; the fact that branches were just subdirectories also was not nice. The fact that tags were mutable since they were also just subdirectories... the fact that every operation you ever did required going to the server (commit, log, checkout, everything) made it a pain if you were on a slow link.
I can't count the number of times I was inspecting SVN history and had to just 'svn log > /tmp/svn.log' so I would have the whole log locally rather than having to hit the server each time I wanted to refine a grep.
Git is fit for purpose. That purpose is to host a monorepo, with out a lot of 3rd party dependancies, distributed, patch based.
Thats not how everyone else works.
We're all using package managers to help with massive amounts of 3rd party dependancies (why are you version pinning in any place other than your repo, why arent you pulling updates through your repo and reviewing them)
We're reliant on tools like artifactory to make sure those depedancys dont disappear or are not corrupted.
We use yet other tools to manage our binary files (this tool would fix that).
Github, gittea, gitlab, bitbucket... have all added piles of tooling around git, that are grafted on around its short comings.
> It's crazy to me that nobody has made anything better yet.
> Although I could start that myself and yet have not.
Because unless your a Google or a Linus, no one is going to look twice at your tool for something that is this important. Im not even sure that epic games has the good will, or trust to launch this.
I am going to give them the benefit of the doubt and take a long hard look at it, but my optimism is tempered. But unless it offers a LOT more than git, the extra overhead (lacking IDE support, deployment changes and all the other tooling in GIT's orbit) it isnt going to be a worth while change.
This tool is not for pure source code. It's for videogames. Videogame-specific VCS have been lacking much more than Git has, since the start. As others have said, the biggest problem is undiffable binary files.
Jonathan Blow found it convenient to represent all assets in a large number of text files, to enable merging. For instance he'd have one text file per entity on a map. The game and editor could read either this or the compiled binary version.
Jonathan Blow works with extremely small team sizes relative to the big studios. When you only have a couple people working on a project you don’t need all of the same coordination features.
He and Casey Muratory make a lot of cool instructive content, but their condescending attitude towards the industry always made me thing "Huh, must be really nice working alone and making all the decisions yourself."
I respect his work, but I had to unfollow him on Twitter because he was so condescending to everything and everyone except his loyal fan base.
He’s in a category of influencers who post constantly about gripes and grievances and smug superiority. Some people like that content but I can’t stand it.
I really like hearing about indie development and small teams, but you don’t have to present everything as condescending superiority over the industry. That’s not the part I find interesting.
I think there is an element of audience capture that sets up a self reinforcing feedback loop that drives out the normies and ends up rather cult like.
Is it not also possible that there are an overwhelming number of problems with the big AAA type studios in games right now? I feel right now we're in sort of tale of two cities, because AAA games have turned into barely functioning uninspired parasitically monetized crap, while smaller scale development is in an absolute golden age. And it's likely that LLMs will only add fuel onto this turd burning fire.
That is what makes it cult like and not a full on cult, there is a lot of truth to what he says. The problems is when the conclusions are extrapolated out to absurdity - it’s hard for me to listen to it. I didn’t take sufficient notes to give a proper recount here and it’s a bit too much work for me to go through it again.
I’ll chip in here and say theres ime a world of difference between the amount of condescension and acerbic noise produced by Blow versus Casey. Casey comes of as grumpy but fundamentally pretty respectful in the stuff I’ve seen him in.
I have only seen Casey's writing in the whole "why is the terminal so slow" debacle, but he was a massive jerk in that. He was right! But still a jerk.
And then they usually back their words by doing things like "you claim that outputting text to a terminal emulator is a PhD level problem, so here I did it in a weekend".
Huge teams are more often than not the sign of bloat and inefficiences.
To be fair, game developers have been rendering text on the GPU for over two decades. I've done it in college a decade ago with bmfont [1] (nowadays the engine does rasterization during import). Whoever thought was making a case with "outputting text to a terminal emulator is a PhD level problem" was really out of their depth and was making a case for unnecessary inefficiencies. Kudos to Casey for proving a point.
I remember when that terminal situation first happened, and my main takeaway wasn't "wow, this Casey guy is a genius" but rather reinforcement of my pre-existing belief that Microsoft is full of incompetent and lazy people. Anyone who has ever dabbled in engine or low-level game dev has implemented basic GPU text rendering at some point.
Yeah, like when Blow claimed he could replace PowerPoint in a weekend and ended up implementing a presentation software that had about 2% of what PowerPoint offers.
Now there's an argument to be made that many don't need the remaining ones but to claim that you 'replaced PowerPoint' for anyone but yourself is ridiculous.
Casey was right, though. The windows terminal was (is, it's still there even if you use the new Terminal) atrocious. The performance is so bad, due to going through all the layers it does, which Casey exposed. And it's not even packed with features, pressing up on a new console doesn't bring you a command from history, which Linux terminals and 3rd party Windows ones have been doing for decades, even Powershell does that. The support for colors was also bad, the very limited options for font configuration, and it renders fonts as if it was Win2k... Thankfully, the Windows Terminal solves most of those, and includes tabs and other useful features. Too late for me as I already jumped ship to Linux.
> pressing up on a new console doesn't bring you a command from history, which Linux terminals and 3rd party Windows ones have been doing for decades, even Powershell does that.
I'm wondering if you're confusing Windows terminal with cmd.exe?
Windows terminal is not the shell. It's a terminal emulator. You run a shell inside of the terminal, for example you can run... Powershell.
He's also the sort of person who I suspect works in a very idiosyncratic way, which is great for him and his mind but probably not everyone else. (This is not a criticism.)
That's fine for database-like meta-data (e.g. game entity properties), but not for images, videos or audio files. Just writing those as hex dumps into text files doesn't make them any easier to merge.
You really can't merge binary data, such as textures, meshes, audio, etc. It doesn't matter if you base64 encode the data and stuff it in a text file: it's a jumble of data (assuming this is the implication of what Blow did).
Git LFS has file locking, and no VCS can provide you with the tools for diffing binary assets. I don't see any meaningful difference between Perforce, Diversion or Lore and git + LFS + file locking. Unless there's a meaningful performance impact for large projects (I only work on small / medium projects), the capabilities are the same. However, I get excellent git support for code in any editor, as opposed to Diversion or Lore which have none.
Git LFS for example does not support file chunking. So a single byte change on a large (100s of gigs) file means downloading the whole file again. Lore does chunking of binary files which means faster downloads and better de-dupping on the backend.
Also, Lore seems to support checking out only the assets you actually need (on-demand hydration and sparse checkouts), meaning that a level designer can check out just the level that they're working on without having to manually configure a git sparse-checkout (and then not being able to see any of the non-checked-out files).
If this supports dynamic hydration of files, either as they're accessed (like Dropbox with offline files) or by somehow knowing which files need which other files (building a dependency graph) then it could be a massive win both for speed and efficiency of downloads but also for conserving disk space on developer machines.
And since it has API bindings, it's possible that's something that could be built into IDE plugins, so that your editor (Godot, Unity, etc) can know which assets need which other assets and automatically trigger hydration, including when you e.g. try to use a new model/texture/etc in a scene that hasn't used it yet.
I haven't made games for a long time so I can't speak for my experience, only my friends. From what I understand (1) Perforce has decent integrations with the game engine editors my friends work with, so editor support is no factor for them and (2) it has better delta support for the file types they work with - I believe Git LFS mostly uses a generic xdelta diff which is kind of mediocre at everything versus Perforce can understand different file types and be extended to support custom types.
I guess you never worked with anything but git? The devil is in the details, and those details generally suck more in git (or generally: distributed version control systems) than in traditional centralized VCS's.
Also git-lfs is a crutch that breaks more often than it works :/
(I agree though that for small game projects, git is mostly 'good enough', even without lfs).
There's also the tooling. Game teams have artists and designers where baroque command line incantations are headwinds to their workflow pace.
For the longest time Git tools were really poor. In recent years there's a few ok ones, like Git Fork, though I wouldn't know if those tools scale to the level of a AAA team size repo and not fall over.
Git LFS breaks so often that it can't be seen as a serious professional tool tbh. I've had nothing but trouble with it. If it wants to be serious it needs to be built into Git, not added as some after thought.
> Git LFS breaks so often that it can't be seen as a serious professional tool tbh. I've had nothing but trouble with it. If it wants to be serious it needs to be built into Git, not added as some after thought.
At Fortinet we migrated our SVN repositories to git and ran into a ton of issues; developers over the past ten years had done tons of little mistakes that added up, like accidentally checking an entire Windows virtual machine into the repo. In SVN they deleted it and no one ended up caring, but in Git of course it became part of the repo history.
I did a huge amount of work for the migration, 99% of which was analyzing each repo to find out what files/file extensions were overly large, and then either:
1. Filtering them out of the git history completely during import
2. Converting them to LFS objects after the import
The LFS process was certainly better than the other alternatives, which were 'check everything into the git history' or 'remove all the un-diffable binary files and hope that they weren't needed for anything', but it was still not ideal.
Every developer (out of thousands, across multiple countries, timezones, and native languages) had to set their system up properly; if you missed a command, or if you reinstalled your OS and forgot to set up one of the aliases or hooks, then you would end up checking binary blobs into git rather than LFS, or checking out LFS idents rather than the actual files they needed.
We also had the issue of developers fetching code over SSH but LFS files over HTTPS, which would be fine except that we wanted to prevent access to HTTPS from most subnets, so while the developers could use SSH to clone or pull using their 2FA token their client would then make an HTTP request that wouldn't work unless they were on the version control VPN, which.... blah blah blah.
So yeah, it worked better than the alternative, but it did not work _well_ a lot of the time.
I mean, Git LFS 'supports branches' in that the LFS content identifiers are checked into git as files and Git operated normally; LFS is just a way to replace those content identifiers with the actual content, and then vice-versa when you commit.
I think branching is the one thing that didn't get more complicated with LFS.
> lore stage covers adds, edits, and deletes — you use the same command for all three. Stage a deleted file and Lore records the deletion for the next commit. Moves and renames are tracked too, through a dedicated subcommand: lore stage move <from> <to> records the rename so the file keeps its identity and history across the move instead of registering as a delete plus an add.
Oof. So this isn't compatible with any tools that move or rename files. I can't see how this will be acceptable for real-world use.
Just today as I pushed some changes to Github, I was thinking how user-unfriendly Git's UI is:
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 10 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 290 bytes | 290.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
I know all of these things communicate something to the die-hard Git user, but for most people (even most people using Git, I bet) this is just complete gobbledegook. What the hell is "delta compression"? Why do I care how many threads it's using? What is an 'object' and what does it mean when it's 'local'? What does 'pack-reused' mean?
From the documentation, it looks like Lore does a bit better in this regard:
Pushing 1 fragment(s)
Pushed 1 fragment(s), 124.00 bytes
Pushing a3f8c2d1... to branch main
Pushed revision 1 -> a3f8c2d1... to branch main
Objects are your files. Underlying git is a content-addressable filesystem.
The objects are referenced by trees. A tree is just a directory.
The trees are then referenced by commits and/or tags into a DAG with named pointers into various parts of it (which are your branch and tag references):
Because it would be terribly in-efficient to have a bunch of loose objects, git periodically groups them together into packs. To save space, the objects are compressed against one another (delta compression) within the packs.
When pushing or pulling, the git transfer protocol basically enumerates what objects each side has so that it only needs to transfer the difference. On top of that, it delta compresses the objects on each side that aren't already grouped into packs against each other to save space.
Now explain this to an artist with very little programming experience beyond what they picked up from their coworkers.
I don't mean to be too glib, but some programmers have this decrepit idea that anyone working with computers should understand programming to be able to fully utilize them.
I worked in gamedev, and many of my colleagues were brilliant, but your comment would read as complete nonsense to many of them. That is the problem git has in the gamedev space. You're trying to manage teams of programmers, designers, sound engineers, gameplay specialists, producers...even c-levels. The parent comment to yours is right, to many many people who work on things that Lore could be useful for would find git to be gobblygook.
I think we can all agree that information should be behind a -v CLA. It's probably just something no one has thought of doing. I've learned over the decades to just ignore it.
It's not really feasible to maintain private forks of every piece of software you use (especially something like git that needs regular security patches)
> It's probably just something no one has thought of doing.
One might reasonably think that about a number of git's rough edges, and one might be surprised at the reality.
Some years ago, the annoyance of git's inconsistent terminology drove me to look into consolidating "cache", "index", and "staging area" in git's help text and documentation. What I found was that others had (of course) thought of it before, but when they tried to do it, it was rejected by git's gatekeepers.
Every place I worked at has a git introduction where all new employees learn about how git works internally. Takes 1h, and all junior devs stops memories random commands and actually start to understand. I highly recommend to you to poking around in the .git directory.
The git support for new employees drops basically to zero.
gritzko@spot ~/beagle $ be get
19:07 get ?#0ac49e6a
16:58 post ?0ac49e6a#POST-018 put:/post: banner on stdout
19:07 new beagle/test/be-post-put-banner.sh
19:07 upd dog/INDEX.md
...more stuff...
19:07 del test/post/01-bare-msg/01.put.err.txt
19:07 del test/post/01-bare-msg/02.post.err.txt
19:07 get abc?4222dfab
Basically, sometimes there commands take a long time. It's nice to have feedback that something is happening even if it's just effectively a blinking light.
Those are just the sounds that animal makes. Live with the animal long enough, you learn how the sounds correspond to its internal states, even if you don’t really know what they mean.
I’d be a bit worried if git didn’t heave that particular contented sigh when I ask it to push
I'd rather see some gobbledegook than extended pauses or idealized (read: fake) information. Those are specific tasks it is doing when you run that command, there's a simplicity to it.
Not saying Lore's approach is bad, but sometimes "worse is better".
I feel like, everyone near Git has decided, "Well, all abstractions leak - so we might as well stand in the rain like Andy Dufresne when he escaped from Shawshank Prison!"
He makes a good tool? Honestly I don’t get the git hate on HN. I’ve been using it for years with no issue. I just read the first 3 parts of the git book and never looked back. I even setup a git server at home with the basic tools.
This is definitely a bit snarky but you read any of the documentation at all, and therefore know more than a large percentage of git users about what it’s actually doing. Most people seem to treat git like some sort of mysterious orb which if you speak the right incantations will perform magic for you.
Linus really has very little to do with git's development. He has stated that himself multiple times, and it's the factual truth. "This is what happens when a kernel developer creates tools..." is funny but not factual.
You don't care about any of this information, but that's fine; unless something is going wrong, you can ignore any information that isn't interesting to you.
Having this output is useful for when it does break and you need to copy-paste your terminal output to someone who does understand it to explain it to you or explain how to fix it, but you're correct that 90% of this is effectively debug output that is almost never useful or relevant.
In most cases I would say they should remove any output that isn't necessary, but given that some git operations can be extremely long-running it's beneficial to have some kind of output so the user knows what's going on.
Case in point, this is the output I get when I try to clone the Linux kernel:
Generally not useful information most of the time, but if I didn't have it I would be staring at a blank terminal for an hour wondering what was happening.
Also, I assume you're not but in case anyone is interested in the answers to these questions:
> What the hell is "delta compression"?
The 'delta' is the difference between one thing and another - usually one version of a file and another. Git does some fancy thinking to figure out which files are which other files but with changes, so that it can store just the changes from one version to the next.
For example, a 100 KB file where we only changed 500 bytes ten times would be 1000 KB, but because Git can store the deltas from one to the next it can be 100 KB (the original) plus ten 500-byte changes, for a total of about 105 KB.
> Why do I care how many threads it's using?
Because it directly affects how fast the process works; using 16 threads is 16x faster than using 1 thread (on average). Git automatically detects how many CPU threads are available and uses as many as it can, but if it's being very slow you might look and see 'oh, right, this VM only has two CPUs'.
> What is an 'object' and what does it mean when it's 'local'?
Uh, this one is deliberately vague I guess. An object is a thing that Git keeps track of. Usually this will refer to a blob, which is 'a bunch of bytes that make up a file', or a 'tree', which is a list of files and other trees - basically a directory structure, or a commit's information, but anything that Git keeps track of is an object.
Local just means that you already have a copy on your system. in the 'remote:' line you see output from the other end (where you're pushing to), so that's the server saying that it's using the files it already has.
> What does 'pack-reused' mean?
To be efficient, Git can take all the 'objects' and smush them into one big packfile (rather than having to keep track of hundreds or thousands of separate files). Since Git keeps track of files based on their contents, two identical files are just stored as one copy referenced twice, so it's possible that the file that you're pushing already exists in a pack file and can just be reused rather than having to push another copy.
This is a very promising announcement for Unreal game development specifically. For any other purpose I wouldn't care as much.
Perforce definitely needs a challenger. It is not the incumbent because it is particularily simple to use or administer. Git is actually way simpler when it comes to branching operations for example.
The reasons why p4 is often preferred in gamedev have already been mentioned in other comments: large project support, permissions, file locking and so on.
Another key reason p4 is the king for Unreal dev is just how well it's supported inside the engine. Not perfect, but it's the best supported VCS because it's what Epic uses. Even the Git plugin is painfully unfinished, because Epic does not internally use it. So with Lore I expect them to give it first class support. I'd recommend Git a lot more if the support in Unreal was better.
(background; I've been in gamedev for almost two decades now, 2-200 person companies, every kind of engine and version control system. I prefer git where I can use it: for Unreal that means small projects and/or tech savvy team members. Pick the tool that is right for the job and the team.)
Turns out it is not really new but only open sourced it now. From the FQA.
>Lore, formerly called Unreal Revision Control, is the built-in version control system for UEFN (Unreal Editor for Fortnite), where creators have been using it to version their islands. It is also seeing progressive adoption by internal Epic teams, and is being implemented as the backing store for UEFN’s cook pipeline, where it replaces traditional intermediary storage layers—eliminating redundant file transfers and significantly reducing the time between publishing changes and those changes being playable.
Surprised it is in Rust and not Epic C++ or Verse. I wonder why.
I suspect the use of Rust rather than C++ might have something to do with the fact that Simon Peyton Jones and Lennart Augustsson (both of Haskell fame) both work at Epic and there would have been a strong internal push to do this in a language with some functional programming features. Rust rather than Verse because that would probably not be the right tool for the job (even if Simon works on it). Rust rather than Haskell probably because of performance -- DARCS never caught on partly for performance reasons.
C++'s stdlib collection algorithms were basically all mutate-in-place instead of return-a-new-value until C++20 introduced ranges, and I still don't know anyone who actually uses those. By contrast, people actually use the `container.iter().map(...)` etc. in Rust.
C++ also lacks much of the fancy pattern matching features which are frequently associated with functional programming. Such features may not be fundamental to functional programming, but they would fall under "some functional programming features", in the sense that they're common in the languages of that ecosystem.
I think the things that make Rust safer than C++ make it look more like a functional programming language than C++. The main point is that there are some influential people at Epic (SPJ has often spoken favourably about Rust) who would favour it over C++.
The premise is that Git-LFS sucks, so we need to build a new data versioning system (in Rust, from scratch). While I mostly agree with this premise, but there are already lots of existing (mature) data versioning systems with the same tricks under the hood:
I guess with AI, anyone can vibe code a content-addressed, chunk-level deduped, versioning system in Rust these days...
But jokes aside, Lore seems really cool! What's interesting is the realization that different domains/industries have similar problems, but they don't seem to be cross-polinating. In this case AI and Gaming both need a storage system that can version control large binary files at scale. I think there's lots of opportunities to share ideas here, but perhaps the lack of idea sharing (currently) creates opportunity!
I don't think the needs are exactly the same. I believe in AI the big binary files are normally written once, while in gamedev, they are constantly updated.
That already warrants different storage architectures.
Yeah, I'm still concerned about crystalline entities suddenly showing up. Have they ever fixed it? I don't see anything in the issue tracker, probably because no one was left last time to report it...
"Lore" is appropriate. Epic games is a very unethical company that steals from people. Myself in paticular. I bought Rocket League the game for linux from Psyonix. Epic bought Psyonix and immediately removed the game clients for linux and mac os. I can no longer play. They stole from me and many others. It'd be one thing if they just shut down the game entirely, but stealing it from only some people while keeping it going for others is worse than just killing games.
"Full-surface API" is a feature nobody here has mentioned. Is that a dig at how git intentionally has no linkable library? I saw this earlier https://news.ycombinator.com/item?id=48470604
We had to use Perforce (Helix Core Cloud) at my last game studio, and it is the de facto industry standard that most of your creative staff is already familiar with. The programmers don't love it, but they don't rule the roost in games. It's also the safe, verified default for working with Unreal Engine 5.
It does show its years though. We were one of the first users of the Perforce cloud offering, as we were small and didn't want to self-host ourselves, but it was a bit of a rickety experience. You had to register an Azure account in order to be able to access the service, and you had to ask support to modify things like triggers. Coming from the world of GitHub and other SaaS products, you could tell it was an attempt to retrofit an older model into a new skin.
The Git LFS path has some unofficial support as well, but you are on your own when things go poorly. Epic doesn't provide much help there.
Competition in this space is welcome, especially if they're planning to make it fully officially supported by the Engine.
There was and still is a very nice competitor in this space called PlasticSCM. They were bought by Unity a few years ago. Unity has not been a good steward. They should have done what Epic is doing and open sourced it. But instead they chose to give it P&L responsibility. Curious what it's contributing to their financials.
> Open-source the desktop client so the community can build on its full graphical experience, not just download it. An early desktop client already exists as a binary download, but it isn't open source yet — it depends on some proprietary components, including Epic's internal design system. We're working to make all of it available in the open so that the client can ship as source alongside the rest of Lore. Lore is an open project, so it is important that the desktop client — which will be one of the main ways many people will interact with Lore — is also fully open so that the community is free to review, extend, and shape it.
Thanks. I still think it's a bit weird to say "fully" open source while your flagship client is currently closed. I realize they're referring to the VCS itself, but—well, if they just dropped the "fully" for the time being, I wouldn't bat an eye.
to be fair, that's just the desktop client. You can use or build on top of the CLI
they do say they will open source it, but who knows:
"It isn’t open source yet—it currently depends on some proprietary components, including Epic’s internal design system—but we’re committed to open-sourcing it in the future"
I don't see a workflow for locking assets while they are under modification. This is kind of important for assets? Since we don't really have great merge workflows for meshes/animations/sounds/etc.
I also don't see any sort of GUI client? So the whole art team is going to have to get up close and personal with the CLI
Git-ish CLI is great. The GUI is more important though. Non-programmers don't want to dabble with CLI. One reason why Perforce is the defacto standard IMO. The GUI covers 99% of daily used operations and is easy to use.
This is a very long document that says nothing about chunking at first skim. If chunking is actually wrong, then just explain why, here. Wasting space is not actually a problem if it’s optimized for other purposes instead.
When it comes to large assets, wasting large chunks of space is a problem. If your chunks are 64 kib average (from the Lore document), but changes only average 1 kib (which could be a high estimate), then you will still run out of space 64 times faster and need to read 64 times more data off of the disk for certain operations.
It also makes diffing hard, as well as diff viewing.
The five command part isn't really possible but you can use custom diffs for merges, git diff, etc. pretty easily. There are projects like diffsitter ( https://github.com/afnanenayet/diffsitter ) for doing more intelligent diffs like this for supported languages.
They’ve been dabbling in this space within Unreal Engine for a few years. Perforce is the de facto standard in AAA studios from my experience, curious to see what’s going to happen to them.
The link to Architectural Decision Records is empty, but they're present in the repo to look at[0]. Curiously the decision with the most deciders is the implementation of JavaScript bindings[1].
This looks very cool! I maintain a FreeCAD workbench for 3d model version control[0], and it currently uses git as the VCS, simply because that's what I was already using. Thinking long-term, I see it eventually morphing into a broader PDM (Product Data Management) system, perhaps even PLM (Product Lifecycle Management). Lore has a lot of the requirements already built-in, like centralized locking (better than SVN), and it's better suited for for binary files. I implemented the git backend as a protocol/port so it'd be pretty easy to swap it out. I'll be watching Lore closely.
The idea sounds good, even if Epic's recent track record of tools is not inspiring. But the commit messages etc. are very clearly products of vibe-coding. And version control is not the situation where "works 97% of the time" is a good-enough bar.
About the recent track record, is there some technical problem or just drama with Unreal Engine that I'm not aware of? (I already have the same opinion about using AI-coded VCS.)
> Git’s content-addressed revision graph is excellent, but it treats binary files as second-class citizens—large files require bolted-on LFS rather than first-class chunked storage, sparse checkouts have sharp edges in offline use, and there is no native multi-tenant isolation.
I'm trying to figure out what Lore can accomplish that git+LFS can't. I've read about big binaries chunking, native interface and permission, is there anything else? Weren't those problems already solvable in the git+LFS ecosystem?
If you've used git+LFS for any extended period of times, you'd know how often it breaks, especially when used with forges like GitHub. Both GitHub and Git treat LFS as an after-thought and second class citizen.
The fact that it's even referred to as git+LFS instead of just git... If I needed to work with large files frequently, I wouldn't want such a basic feature bolted on. Not a criticism of git, just can see why Epic doesn't use git.
Yeah, they should have posted it to LoreHub! I just checked for the availability of lorehub.com, and you can buy it for only ~$13,000 and start a competing business to GitHub.
Well, that's easy for a Rust binary. But they don't put any effort into having a great UX for Linux devs with Unreal Engine, for example. It barely works on Linux and is almost impossible to run under Wayland.
this looks cool for game development, because using Git for projects in Unity and Unreal Engine definitely has it's issues. I'm personally not a fan of Git LFS, especially since GitHub charges you to use it (which makes sense, binaries and assets are big, code is small, relatively speaking).
Their docs seem entirely LLM written. It seems especially obvious in the FAQ. While I'm not against using LLMs for writing assistance, they've left a lot of the unnecessary language and typical stylistic choices in there, which erodes my trust in the project a bit. Perhaps it's a very good game-oriented version control system, but the lack of human attention on the docs makes me wonder how much they care
I reluctantly agree. I was interested to read the system design doc[1] but it's so many pages long and so full of redundant statements and needless details[2] that I gave up a couple sections in. The numerous "it's not X, it's Y" constructions give the game away. If they can't be bothered to read these docs themselves, why should I?
Anyway it's probably fine software and I am genuinely going to give it a shot for a usecase I have involving large image files. But the LLM-generated docs don't inspire confidence.
[2] They literally have a section header "10.1 Revision state as a 320-byte fragment". The byte size isn't even relevant in the code as an implementation detail, much less belongs in a design doc. No one read this doc before publishing it.
Yeah, I agree about the docs. I started on the system design page and my head started to swim after about 5 minutes. So exhausting to read!
On the flip side, I expect the project itself will be workable - well, assuming they're actually using it themselves! UE is a big pile of Stuff on its own, and Fortnite must have god knows how much additional crap in there, so if this is (or will be) their replacement for Perforce internally, then it'll be getting a good deal of testing. (If they're just chucking it over the wall, though... well, sheesh, you first...)
(Perforce is the standard thing for games, and pretty well it works too, and hopefully this will deliver it a well-needed kick. It was sold to private equity about 10 years ago, and it feels like they've been coasting ever since. (Perhaps users in other sectors are happier though?))
Yeah, I hope I'm wrong, but it feels a little bit like "planned project no longer has internal support, let's see if we can make it open-source to garner goodwill and recoup some of our investment." Which isn't the worst thing that can happen.
Lore itself is not an example of a program that meaningfully benefits from any of the key features of Lore.
Lore is meant for situations where your repository is going to contain gigabytes of binary files, such as art assets for games. Git is technically great at everything but that, and even the external solutions for that situation still kind of suck.
Git is currently industry standard tool for software development, which means that almost every IDE and code editor has good and mature GIT integration. It will take a while for Lore to achieve similar level of support. Considering the intended usecase seems likely that they focus the effort on integrating into game engines and game development specific needs instead of making plugin for every last code editor. Not only code editor but every other software related tool interacts with version control - code review tools, CI, package managers and code forges. That's a lot more work than just creating a version control system. Why waste effort on areas that already have good alternatives.
Code forges lead to the next reason -> they are making it open source (see the faq for why). If they want community collaboration it makes sense to prefer industry standard tooling. Organization internal consistency doesn't matter to external collaborators. Various Google projects with all their custom tooling are good example of what happens if this gets ignored. Every time I have to deal with depot_tools or repo it's a pain.
Third reason not to self host is that it's not quite production ready (see "Is lore production-ready?" in the FAQ). Can't be self hosting from day 0 and once you have a working setup you need better argument for switching just because you can. If lore was attempting to make a statement that it's a superior method of version control for regular source code it would be a different story.
The incredible laggyness of that website does not inspire confidence. Much of the text selection is also broken, and chrome consumes nearly a full core trying to render.. something?
Its remarkable that anyone thought this website was fit for release, and it gives off strong slop vibes
I also have absolutely zero trust in a product like version control being provided by a for-profit company. It seems like a terrible idea to tie your software stack to Epic Games of all people, given their track record
This is primarily going to be targeted at Unreal Engine users, for whom the source control tooling they use is the smallest of concerns when it comes to being tied to Epic Games.
I'm in the market for a better VCS designed for gamedev-specific concerns, but reading the system design doc, it's LLM-generated. Not exactly confidence inspiring.
Git does great with text files, but game development contains a lot of binary assets (textures, videos, 3d models) and correspondingly huge repos. git-lfs tries to patch around that, but that makes a complex tool that creatives struggle to understand even more complex. Perforce is a pretty popular solution, and was used by Epic in the past
Have to think they're doing this out of a real need, given they were already using Perforce and must've considered Git too. It's also not like 2010s when version control was a hyped thing.
> It’s optimized for projects—including games and entertainment—that combine code with large binary assets, and caters for the needs of developers and artists alike.
When you have a game that weighs in at 100GB, only a tiny fraction of that is built from code. The rest of it is binary assets that most VCSs struggle with. What makes this worth considering is the fact that they designed it to handle binary assets.
Well it seems to be intended for repos with large blobs like video games so it kind of makes sense that their own pure source code is still managed with git.
Epic's Unreal Engine development is made in Perforce, and then it was mirrored to git/Github. Or that's how it was a couple years ago. That's kind of expected, VCS is one thing, a forge is another thing.
Interesting to note that this does not seem like a DVCS in the traditional sense because it depends on coordinating with a central server where all repositories will be hosted. I can't tell if servers can pull/push from eachother.
So it's more like git in practice than git in principle, I guess.
I am not sure it is much of an issue to implement a centralised system to solve domain-specific problems that are caused by or are in the context of inevitable centralisation.
I can see a bunch of media companies liking this, maybe even small design shops. Quite a lot of CMS and change control work for media is not much more than asset management; version control is often a bit of an afterthought.
Web interfaces to this might end up interesting for non-profits and charities with significant media outreach.
Citizen Sleeper, the game that is going free tomorrow on the Epic store is really good. Space Cyberpunk themed RPG/Survival Sim.
It is interesting that people are so cynical about Epic giving out free games. I get that people love Steam, but competition in the storefront market is not bad.
> It is interesting that people are so cynical about Epic giving out free games.
It's a mischaracterization to call the games free, if they require you to install unrelated third-party software you'd rather not install (and which at least in the past has been known to snoop your data without consent). In that sense, you may see it as backlash around characterizing the games as free in the first place, when they obviously are not.
I definitely have. But they're all the games that they gave out in the early days, the more recent games haven't really appealed to me. And it didn't convince me to spend any money, I still haven't spent a single cent on anything Epic Games.
I've been getting them for years now and have enjoyed many of them (though not a large percentage), and would love to use the Epic store more, if only they would implement some version of reviews/scores. Without that, I feel that I can't trust them as a marketplace.
i've paid for brotato (Its a pretty good game, it was worth my money) and played city skylines (which i haven't paid for due to its excessive amount of dlcs)
This is just going to become another way to lock developers into UE. Then they will start charging for licenses, same as Unity did for its versioning feature. It might be open source but that doesn’t stop the commercial use of it being charged for.
For context, since a lot of people on HN haven't worked on games - this is not intended to compete with Git for general software development. This is a competitor with Perforce for game development.
Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files that game developers need to collaborate on. For example, one artist might need to obtain an exclusive lock on some art assets while editing them, because there is no sane way to merge two artists' async edits.
The SOTA in this area is Perforce (https://www.perforce.com/products/helix-core), a proprietary system. From what my gamedev friends tell me, when Perforce works it's great, but it hits enough snags that you need a tools engineer to manage it and occasionally fix issues manually. Git LFS is an alternative, but my gamedev friends all prefer Perforce especially when working on team projects beyond like 3-4 people.
Something else that git isn't good at: permissions. In gamedev, you might have proprietary work that you want to restrict to certain users. In P4, you can add restrictions to certain directories for only those who have signed the required NDAs. That's not something that you can do in git: it's all or nothing. Maybe you can set something up with submodules, but that's going to upend your repository if you hadn't planned for it.
I once worked in a git repository that required those kinds of restrictions.
This was within a bank and the code in question was related to enabling Apple Pay from within the banking application. The consequences of that information and code leaking or being seen by anyone who had not signed the NDA were very serious (don't remember the details but it made the lawyers were extremely stressed about it).
Needing to figure out a way to protect those parts of the codebase it was decided in the end that the "easiest" way of doing this was to split the repository in half, with the actual artifact building taking place from the half that had the NDA code. The rest of the application (basically the whole application) was then used as a dependency by it.
Still didn't quite solve the issue, but access to that repository was heavily controlled.
Strikes me as bizarre that payment code would be sensitive, unless it's a security by obscurity thing (which would also be concerning).
Keys, secrets, etc. yes. But code? What am I missing here?
As others have said, it's Apple and they do not take kindly to other people leaking their technology/announcements ahead of time.
See also: the time that ATI's CEO told his employees that their chips would be powering Apple's to-be-announced hardware a few days before the announcement. Steve Jobs responded by pulling all of ATI's hardware from its demo units at the announcement, not mentioning ATI at all, cancelling a joint demonstration of the Radeon card that was going to be in the system, and never partnering with ATI again.
https://web.archive.org/web/20001216031800/https://www.zdnet...
Sounds like a bit of a dick...
One word: "Courage"
> and never partnering with ATI again.
Except of course shipping ATI hardware for years afterwards, then also using nvidia, then dropping nvidia and only using ATI/AMD until transitioning to Apple Silicon.
Because it's Apple. They are huge, have scary lawyers, write scary contracts, and want to "delight the user" with features only when they announce them. They hate leaks, and demand separate teams for basically any/all development.
It seems this wasn’t about the code itself, it was about Apple Pay not being announced yet. So only people under NDA would be allowed to even know what they are working on.
> security by obscurity thing... What am I missing here?
You are looking at the problem from the wrong direction.
If you build a honeypot, to trap hackers, does it behove you to explain what the bait is, and how the trap works?
Know your customer, fraud detection heuristics, finger prints, behavioral triggers are all areas where banks, and financial institutions need to keep the sauce secret. Telling the other party "how" you catch them just gives them the steps of what not to do.
It's kinda like that, there could be a proprietary fraud detection heuristic in there that you don't want to get out.
Maybe that’s some scoring to decide if you should be able to pay or not with some method.
Can confirm split repos is an excellent solution for protecting IP.
The way I usually solve this is by using git submodules.
Oh man, I've been laughing at this for 37 minutes straight now.
AFAIK the issue with using submodules is you still need the rights to pull the other source repo. However, you can use submodules or LFS to pull a specific build artifact from a build artifact repo or source instead of the source repo, which provides a neat way to manage the dep without fattening the main repo and allows the source repo to be kept separate and high security. I'd certainly do this before changing RCS/VCS solutions. That said, reverse engineering has become relatively trivial in the AI age so the practical utility of providing built rather than source elements is dropping.
I ended up writing my own layer over git for permissions for a specific client a long time ago. It has a huge amount of useful features - sadly, I never took the idea further.
Doesn't git crypt solve this? You can have encrypted blobs in a repo that will be auto decrypted if you have a working key.
That depends on you distributing working keys for any components you want to restrict access to, and managing those keys for all users, revoking them when access permissions change, etc. It's a lot more complex, more work, and harder to manage than centralized RBAC or similar.
People don't use git crypt nearly enough unfortunately.
Agreed. I use and love git crypt, but it doesn't get enough use. I think because it's easy to screw up gpg keys. Most of my uses (for one to three devs) have become symmetric keys shared out-of-band instead of using gpg keys because we've had lots of onboarding pain even from people who are quite competent. There are just a lot of sharp edges in gpg that you don't know when you don't know.
Not really, precisely because it’s decentralized. You can’t audit whether a user accessed one of the hidden files, or really even who can access it once you accept the reality of the risk that some team will put a key on S3 or a shared drive or whatever.
It’s fine for things that you want devs to be able to see without the Git host being able to see them, it’s less good at RBAC because there’s no real “identity” component at read-time.
You can use Mozilla SOPS instead with IAM roles and KMS instead of gpg. They also shifted to AGE over gpg.
Git submodules + SSH keys is another (somewhat "homebrew") solution to this.
> That's not something that you can do in git: it's all or nothing.
That is partially incorrect; you can restrict writes via hooks but not reads; you'd need a workaround like submodules
Does `--no-verify` override the restriction via hooks, or are there some kind of server-side hooks that can be used?
That's by design of git, you can't forget that git is first developed for a bazaar model of information flow, especially with a big decentrailized project like the Linux Kernel, not the silo and isolated corporate NDA and closed model you described. Git prefers open information and discourages information closures and segregation of information by placing restrictions exactly like this.
Git enthusiast would often tell you to do this separately with a submodule, and set permission on the version control forge software level (which means Gitea/Github private RBAC access to certain repos for cloning), sure, but that is also painful as hell.
But my point is that all of this is exactly by design from Linus Torvalds's need for Linux Kernel to replace BitKeeper. Git simply isn't the tool for everything, it was developed for a software project with liberalism in mind, but corporate stuff is monoculture and prefers proprietary, shut-in model, and the eat your own dog food mindset, and no wonder it is so painful to deal with.
Git submodules aren’t convenient either. For the silo and corporate development use case, just use multiple repositories and make your build tool aware of multiple repositories. It is slightly less painful than submodules.
I feel like submodules could be a lot easier to work with if the git command made it easy to update all submodules in one go based on branch head for the submodule.
I think everyone knows that this is a consequence of git's design. Nobody's disputing that.
Unfortunately there are many people who think git is a panacea and is suitable for all version control tasks of anything.
One thing I don't like about Git LFS is that there is no way to delete very old history. It's the 'git spirit' to not allow deleting history, but in the context of LFS it sounds horrible. Especially if you use Github.
If there is an asset that is updated very frequently in the early stage of development, you'll be charged for all the storage for the rest of the repo's life. That happens a lot in gamedev: most assets go back and forth early on but once it's done no one will touch them ever.
Its important to understand that in Game Dev a ’git clone’, aka ’p4 sync’, can be a terabyte of stuff.
Git is bad at such volumes of binary assets, textures, models, sounds, etc.
Git LFS is a major PITA, and if you use GitHub is even worse since there are quotas and rate limits that are charged separately.
One of those ideas that sounds clever in theory but in reality doesn’t work very well
Also... it's kind of weird taking a decentralized system and recentralizing it.
I am building a small asset heavy game. Ran into a similar problem. Built a storage cost efficient tool for exactly this [1].
[1] https://github.com/debarshibasak/assets
There's also a new player called diversion (diversion.dev) which I think may be a YC startup? Anyway it takes a different approach of being more like Google drive but bringing in VCS behavior making it more indie and designer friendly.
At my previous game-dev-company job we ended up splitting things up into:
1. Code - Git
2. WIP art, shared assets (logos, marketing materials, etc) - Google Drive (because things are often changing, getting passed around, etc)
3. Finished assets (PSD files you're done with, or you think you're done with) - SVN (because we wanted a log of who contributed to what, wanted artists to be able to pick up where someone else left off; having a log of who made changes to a given PSD)
4. Assets rendered out to PNG to include in the app bundle/publish to the static file servers - Git (because those files never changed after being published so the git history wasn't polluted with unneeded files)
I've also used LFS, which is... a fine workaround, but still not great. Users who don't have it configured can still commit binary blobs; users who don't have it configured will clone files incorrectly; if the LFS server is slow, unavailable, unreliable, then the system starts to behave oddly; you need a Git server that supports it.
It was a huge hassle to manage; having a system like this would have been a godsend at that company, and if I still worked there I would be spending all day importing our codebase and assets into it to see how well it works.
what happens if WIP stuff has rapid shifts or changes? art direction changes on the product level, etc? or even something as simple as an asset designer quits or get sick for a while
SVN makes sense cuz it's done and dusted, but I could see the Drive gettin real messy real fast if things change a lot
I wonder how useful this could be as a generalized version control for regular user systems, as a way to rollback, or scrub through history. Presumably if this is designed to work at Epic and Big Game studio scale, it should work at home computer scale
This presumption has destroyed far, far more companies and projects than the opposite assumption (that something built for small will scale to big, then doesn't).
Can you name five?
Can confirm, we have a team dedicated to the care and feeding of p4.
P4 is more "industry standard" than "state of the art"... But it does handle large files and partial checkouts without feeling bolted on.
Has nothing to do with Perforce being the Oracle of VCS because it’s baked into the big 3? Riiiight.
It's not even baked into Google anymore.
> Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files
Git-annex ?
P4 is also really well integrated into IDEs and UE Editor so that I don't need to think about it as much as I need, compared to Git. Locking assets, releasing them, merging into streams etc., is overall pretty streamlined. When it works, it's great, but when it doesn't work though, it's pretty hard to diagnose issues.
A significant part of my job, unfortunately, is helping people fix their workspaces when Perforce (p4) goes bad, or creating guardrails and wrappers to stop Perforce doing bad things.
In fairness, p4 predates most of the VCSes we consider "modern", so I empathize with a lot of the underlying architecture decisions. However, it has and continues to utterly fail at improving at a reasonable pace.
For example:
> p4 predates most of the VCSes we consider "modern"
p4 also significantly predates VCSes we consider obsolete. p4 is almost a decade older than SVN.
I’ll add some more
- The P4 cpp api was apparently designed before any modern Cpp std lib was available. And is at best archaic, and stringly to use.
- P4 encoding support is pain in the ass to configure. And ensist on adding or removing bom to files.
Git is certainly not great with binary assets, but calling perforce SOTA ... ouch.
If perforce is the best there is out there for large binary asset management, then there is a blue ocean worth of potential improvement for git.
Perforce is a piece of crap, a relic of the 20th century that must die in a fiery inferno.
It can be SOTA and still be garbage if there's nothing better (and there's nothing better, sadly). This is extremely exciting for anyone who's had to manage revision control for game devs.
I’ve spent more than a decade working in games and unfortunately perforce is the best out there for a variety of reasons. None of them are good.
15 years ago, both Google and Microsoft were on perforce. (The latter through a fork with a different name.)
Google still uses Piper, which started as a Perforce clone, though many people use it through a frontend like `fig` (try pronouncing ‘Piper HG’) or `jj`.
That sucks, git is so absolutely horrible. It's crazy to me that nobody has made anything better yet. Although I could start that myself and yet have not.
With respect, were you around to use any of its predecessors?
I was. I thought, and still think, that svn was much more pleasant to use than git. Alas, I am in the minority.
We are legion ;)
SVN was actually quite decent for game development, definitely more robust and (non-technical-) user-friendly than git+lfs.
(and SVN isn't really compatible with the work-from-home era unfortunately, you really needed a big server on a gigabit LAN)
SVN was more straightforward to use, but that straightforwardness lost a lot in terms of fidelity.
The fact that it was easy to clone a subdirectory was nice; the fact that branches were just subdirectories also was not nice. The fact that tags were mutable since they were also just subdirectories... the fact that every operation you ever did required going to the server (commit, log, checkout, everything) made it a pain if you were on a slow link.
I can't count the number of times I was inspecting SVN history and had to just 'svn log > /tmp/svn.log' so I would have the whole log locally rather than having to hit the server each time I wanted to refine a grep.
git-svn did wonders when we had the code base on subversion
There are some projects:
* https://github.com/jj-vcs/jj
* https://nest.pijul.com/pijul/pijul
turns out version control is hard
We did, mercurial just didn't win.
Git is fit for purpose. That purpose is to host a monorepo, with out a lot of 3rd party dependancies, distributed, patch based.
Thats not how everyone else works.
We're all using package managers to help with massive amounts of 3rd party dependancies (why are you version pinning in any place other than your repo, why arent you pulling updates through your repo and reviewing them)
We're reliant on tools like artifactory to make sure those depedancys dont disappear or are not corrupted.
We use yet other tools to manage our binary files (this tool would fix that).
Github, gittea, gitlab, bitbucket... have all added piles of tooling around git, that are grafted on around its short comings.
> It's crazy to me that nobody has made anything better yet.
Because our entire industry has fallen into the rut of "more tools", of stacking turtles (https://en.wikipedia.org/wiki/Turtles_all_the_way_down ) rather than fixing the real issues that hold us back.
> Although I could start that myself and yet have not.
Because unless your a Google or a Linus, no one is going to look twice at your tool for something that is this important. Im not even sure that epic games has the good will, or trust to launch this.
I am going to give them the benefit of the doubt and take a long hard look at it, but my optimism is tempered. But unless it offers a LOT more than git, the extra overhead (lacking IDE support, deployment changes and all the other tooling in GIT's orbit) it isnt going to be a worth while change.
This tool is not for pure source code. It's for videogames. Videogame-specific VCS have been lacking much more than Git has, since the start. As others have said, the biggest problem is undiffable binary files.
Jonathan Blow found it convenient to represent all assets in a large number of text files, to enable merging. For instance he'd have one text file per entity on a map. The game and editor could read either this or the compiled binary version.
Jonathan Blow works with extremely small team sizes relative to the big studios. When you only have a couple people working on a project you don’t need all of the same coordination features.
He and Casey Muratory make a lot of cool instructive content, but their condescending attitude towards the industry always made me thing "Huh, must be really nice working alone and making all the decisions yourself."
I respect his work, but I had to unfollow him on Twitter because he was so condescending to everything and everyone except his loyal fan base.
He’s in a category of influencers who post constantly about gripes and grievances and smug superiority. Some people like that content but I can’t stand it.
I really like hearing about indie development and small teams, but you don’t have to present everything as condescending superiority over the industry. That’s not the part I find interesting.
I think there is an element of audience capture that sets up a self reinforcing feedback loop that drives out the normies and ends up rather cult like.
Is it not also possible that there are an overwhelming number of problems with the big AAA type studios in games right now? I feel right now we're in sort of tale of two cities, because AAA games have turned into barely functioning uninspired parasitically monetized crap, while smaller scale development is in an absolute golden age. And it's likely that LLMs will only add fuel onto this turd burning fire.
That is what makes it cult like and not a full on cult, there is a lot of truth to what he says. The problems is when the conclusions are extrapolated out to absurdity - it’s hard for me to listen to it. I didn’t take sufficient notes to give a proper recount here and it’s a bit too much work for me to go through it again.
I’ll chip in here and say theres ime a world of difference between the amount of condescension and acerbic noise produced by Blow versus Casey. Casey comes of as grumpy but fundamentally pretty respectful in the stuff I’ve seen him in.
I have only seen Casey's writing in the whole "why is the terminal so slow" debacle, but he was a massive jerk in that. He was right! But still a jerk.
IMO this is just garden variety effects of being a programming influencer. It’s a weird position to be in.
I think being influential just does that to people, with high regularity.
And then they usually back their words by doing things like "you claim that outputting text to a terminal emulator is a PhD level problem, so here I did it in a weekend".
Huge teams are more often than not the sign of bloat and inefficiences.
To be fair, game developers have been rendering text on the GPU for over two decades. I've done it in college a decade ago with bmfont [1] (nowadays the engine does rasterization during import). Whoever thought was making a case with "outputting text to a terminal emulator is a PhD level problem" was really out of their depth and was making a case for unnecessary inefficiencies. Kudos to Casey for proving a point.
1. https://www.angelcode.com/products/bmfont/
I remember when that terminal situation first happened, and my main takeaway wasn't "wow, this Casey guy is a genius" but rather reinforcement of my pre-existing belief that Microsoft is full of incompetent and lazy people. Anyone who has ever dabbled in engine or low-level game dev has implemented basic GPU text rendering at some point.
Yeah, like when Blow claimed he could replace PowerPoint in a weekend and ended up implementing a presentation software that had about 2% of what PowerPoint offers.
Now there's an argument to be made that many don't need the remaining ones but to claim that you 'replaced PowerPoint' for anyone but yourself is ridiculous.
They're good at demos, I give them that.
There's a classic saying along those lines, "everyone is only using 5% of Word. The tricky part is that everyone is using a different 5%"
Casey was right, though. The windows terminal was (is, it's still there even if you use the new Terminal) atrocious. The performance is so bad, due to going through all the layers it does, which Casey exposed. And it's not even packed with features, pressing up on a new console doesn't bring you a command from history, which Linux terminals and 3rd party Windows ones have been doing for decades, even Powershell does that. The support for colors was also bad, the very limited options for font configuration, and it renders fonts as if it was Win2k... Thankfully, the Windows Terminal solves most of those, and includes tabs and other useful features. Too late for me as I already jumped ship to Linux.
> pressing up on a new console doesn't bring you a command from history, which Linux terminals and 3rd party Windows ones have been doing for decades, even Powershell does that.
I'm wondering if you're confusing Windows terminal with cmd.exe?
Windows terminal is not the shell. It's a terminal emulator. You run a shell inside of the terminal, for example you can run... Powershell.
Command history is a feature of the shell.
Isn't that kind of the point though? Doing more with less?
He's also the sort of person who I suspect works in a very idiosyncratic way, which is great for him and his mind but probably not everyone else. (This is not a criticism.)
That's fine for database-like meta-data (e.g. game entity properties), but not for images, videos or audio files. Just writing those as hex dumps into text files doesn't make them any easier to merge.
He uses SVN and specifically has stated that Git isn't suitable for the work he does due to big binaries in source control.
You really can't merge binary data, such as textures, meshes, audio, etc. It doesn't matter if you base64 encode the data and stuff it in a text file: it's a jumble of data (assuming this is the implication of what Blow did).
How do you merge changes to a texture, mesh, audio file, etc?
Git LFS has file locking, and no VCS can provide you with the tools for diffing binary assets. I don't see any meaningful difference between Perforce, Diversion or Lore and git + LFS + file locking. Unless there's a meaningful performance impact for large projects (I only work on small / medium projects), the capabilities are the same. However, I get excellent git support for code in any editor, as opposed to Diversion or Lore which have none.
Git LFS for example does not support file chunking. So a single byte change on a large (100s of gigs) file means downloading the whole file again. Lore does chunking of binary files which means faster downloads and better de-dupping on the backend.
Permissions is another thing. Git permissions are done one a per repo basis.
https://epicgames.github.io/lore/explanation/system-design/#...
Also, Lore seems to support checking out only the assets you actually need (on-demand hydration and sparse checkouts), meaning that a level designer can check out just the level that they're working on without having to manually configure a git sparse-checkout (and then not being able to see any of the non-checked-out files).
If this supports dynamic hydration of files, either as they're accessed (like Dropbox with offline files) or by somehow knowing which files need which other files (building a dependency graph) then it could be a massive win both for speed and efficiency of downloads but also for conserving disk space on developer machines.
And since it has API bindings, it's possible that's something that could be built into IDE plugins, so that your editor (Godot, Unity, etc) can know which assets need which other assets and automatically trigger hydration, including when you e.g. try to use a new model/texture/etc in a scene that hasn't used it yet.
I haven't made games for a long time so I can't speak for my experience, only my friends. From what I understand (1) Perforce has decent integrations with the game engine editors my friends work with, so editor support is no factor for them and (2) it has better delta support for the file types they work with - I believe Git LFS mostly uses a generic xdelta diff which is kind of mediocre at everything versus Perforce can understand different file types and be extended to support custom types.
I guess you never worked with anything but git? The devil is in the details, and those details generally suck more in git (or generally: distributed version control systems) than in traditional centralized VCS's.
Also git-lfs is a crutch that breaks more often than it works :/
(I agree though that for small game projects, git is mostly 'good enough', even without lfs).
There's also the tooling. Game teams have artists and designers where baroque command line incantations are headwinds to their workflow pace.
For the longest time Git tools were really poor. In recent years there's a few ok ones, like Git Fork, though I wouldn't know if those tools scale to the level of a AAA team size repo and not fall over.
Git LFS breaks so often that it can't be seen as a serious professional tool tbh. I've had nothing but trouble with it. If it wants to be serious it needs to be built into Git, not added as some after thought.
> Git LFS breaks so often that it can't be seen as a serious professional tool tbh. I've had nothing but trouble with it. If it wants to be serious it needs to be built into Git, not added as some after thought.
At Fortinet we migrated our SVN repositories to git and ran into a ton of issues; developers over the past ten years had done tons of little mistakes that added up, like accidentally checking an entire Windows virtual machine into the repo. In SVN they deleted it and no one ended up caring, but in Git of course it became part of the repo history.
I did a huge amount of work for the migration, 99% of which was analyzing each repo to find out what files/file extensions were overly large, and then either:
1. Filtering them out of the git history completely during import
2. Converting them to LFS objects after the import
The LFS process was certainly better than the other alternatives, which were 'check everything into the git history' or 'remove all the un-diffable binary files and hope that they weren't needed for anything', but it was still not ideal.
Every developer (out of thousands, across multiple countries, timezones, and native languages) had to set their system up properly; if you missed a command, or if you reinstalled your OS and forgot to set up one of the aliases or hooks, then you would end up checking binary blobs into git rather than LFS, or checking out LFS idents rather than the actual files they needed.
We also had the issue of developers fetching code over SSH but LFS files over HTTPS, which would be fine except that we wanted to prevent access to HTTPS from most subnets, so while the developers could use SSH to clone or pull using their 2FA token their client would then make an HTTP request that wouldn't work unless they were on the version control VPN, which.... blah blah blah.
So yeah, it worked better than the alternative, but it did not work _well_ a lot of the time.
don't think it supports branches
it's also tough when you have 1TB of data, over 1mm files and you might want to lock hundreds files in one go
I mean, Git LFS 'supports branches' in that the LFS content identifiers are checked into git as files and Git operated normally; LFS is just a way to replace those content identifiers with the actual content, and then vice-versa when you commit.
I think branching is the one thing that didn't get more complicated with LFS.
> lore stage covers adds, edits, and deletes — you use the same command for all three. Stage a deleted file and Lore records the deletion for the next commit. Moves and renames are tracked too, through a dedicated subcommand: lore stage move <from> <to> records the rename so the file keeps its identity and history across the move instead of registering as a delete plus an add.
Oof. So this isn't compatible with any tools that move or rename files. I can't see how this will be acceptable for real-world use.
Just today as I pushed some changes to Github, I was thinking how user-unfriendly Git's UI is:
I know all of these things communicate something to the die-hard Git user, but for most people (even most people using Git, I bet) this is just complete gobbledegook. What the hell is "delta compression"? Why do I care how many threads it's using? What is an 'object' and what does it mean when it's 'local'? What does 'pack-reused' mean?From the documentation, it looks like Lore does a bit better in this regard:
Objects are your files. Underlying git is a content-addressable filesystem.
The objects are referenced by trees. A tree is just a directory.
The trees are then referenced by commits and/or tags into a DAG with named pointers into various parts of it (which are your branch and tag references):
https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
Because it would be terribly in-efficient to have a bunch of loose objects, git periodically groups them together into packs. To save space, the objects are compressed against one another (delta compression) within the packs.
https://git-scm.com/docs/git-pack-objects
https://github.com/git/git/blob/master/Documentation/technic...
When pushing or pulling, the git transfer protocol basically enumerates what objects each side has so that it only needs to transfer the difference. On top of that, it delta compresses the objects on each side that aren't already grouped into packs against each other to save space.
https://github.com/git/git/blob/master/Documentation/technic...
Because git is an open-source project written by nerds, it shows you all of this information. Feel free to ignore it!
But if you really want to know, it's all documented both in the git book and git documentation directory, both linked above.
(Caveat: I'm working from memory and surely got some detail at least slightly wrong.)
Now explain this to an artist with very little programming experience beyond what they picked up from their coworkers.
I don't mean to be too glib, but some programmers have this decrepit idea that anyone working with computers should understand programming to be able to fully utilize them.
I worked in gamedev, and many of my colleagues were brilliant, but your comment would read as complete nonsense to many of them. That is the problem git has in the gamedev space. You're trying to manage teams of programmers, designers, sound engineers, gameplay specialists, producers...even c-levels. The parent comment to yours is right, to many many people who work on things that Lore could be useful for would find git to be gobblygook.
I think we can all agree that information should be behind a -v CLA. It's probably just something no one has thought of doing. I've learned over the decades to just ignore it.
> It's probably just something no one has thought of doing
There are 1000 things that's true of about git. At a certain point that becomes a problem in and of itself.
Luckily it’s an open source project so you could go in and clean up the ux.
Probably unacceptable as many would have built tooling around the specific default behaviors.
If tooling is relying on the output format of porcelain commands, it's wrong. The output of various subcommands have already changed more than once.
You can have your own version. You don’t have to push all improvements upstream.
It's not really feasible to maintain private forks of every piece of software you use (especially something like git that needs regular security patches)
> It's probably just something no one has thought of doing.
One might reasonably think that about a number of git's rough edges, and one might be surprised at the reality.
Some years ago, the annoyance of git's inconsistent terminology drove me to look into consolidating "cache", "index", and "staging area" in git's help text and documentation. What I found was that others had (of course) thought of it before, but when they tried to do it, it was rejected by git's gatekeepers.
That makes me sad. This termainology is really one of the major flaws of git, along with the checkout/reset thing
[delayed]
I understand there's friction in contributing to git, but it's absolutely possible to contribute documentation improvements.
https://github.com/search?q=repo%3Agit%2Fgit+staging+area&ty...
https://github.com/git/git/commit/be8ef33c1deabf34a82e4af98b...
Do you have a link to your proposal on the git mailing list?
Every place I worked at has a git introduction where all new employees learn about how git works internally. Takes 1h, and all junior devs stops memories random commands and actually start to understand. I highly recommend to you to poking around in the .git directory.
The git support for new employees drops basically to zero.
Every Beagle command:
The lights are blinking, so everything must be working!
Basically, sometimes there commands take a long time. It's nice to have feedback that something is happening even if it's just effectively a blinking light.
I actually like this underlying logs. Could have a concise / project level summary though.
Those are just the sounds that animal makes. Live with the animal long enough, you learn how the sounds correspond to its internal states, even if you don’t really know what they mean.
I’d be a bit worried if git didn’t heave that particular contented sigh when I ask it to push
I'd rather see some gobbledegook than extended pauses or idealized (read: fake) information. Those are specific tasks it is doing when you run that command, there's a simplicity to it.
Not saying Lore's approach is bad, but sometimes "worse is better".
Git as a data structure is clever, but Git as a CLI is atrocious.
I feel like, everyone near Git has decided, "Well, all abstractions leak - so we might as well stand in the rain like Andy Dufresne when he escaped from Shawshank Prison!"
Obligatory: https://git-man-page-generator.lokaltog.net/
Yes, the famous debate between plumbing and porcelain
Still the porcelain is more like cold stainless steel
This is what happens when a kernel developer creates tools that need some kind of UX (I say this both as a shitty UX developer and Linus fan)
He makes a good tool? Honestly I don’t get the git hate on HN. I’ve been using it for years with no issue. I just read the first 3 parts of the git book and never looked back. I even setup a git server at home with the basic tools.
This is definitely a bit snarky but you read any of the documentation at all, and therefore know more than a large percentage of git users about what it’s actually doing. Most people seem to treat git like some sort of mysterious orb which if you speak the right incantations will perform magic for you.
Reading the docs is kind of a superpower for devs. With AI I think most technology is going to be treated like praying to the machine spirit.
Linus really has very little to do with git's development. He has stated that himself multiple times, and it's the factual truth. "This is what happens when a kernel developer creates tools..." is funny but not factual.
You don't care about any of this information, but that's fine; unless something is going wrong, you can ignore any information that isn't interesting to you.
Having this output is useful for when it does break and you need to copy-paste your terminal output to someone who does understand it to explain it to you or explain how to fix it, but you're correct that 90% of this is effectively debug output that is almost never useful or relevant.
In most cases I would say they should remove any output that isn't necessary, but given that some git operations can be extremely long-running it's beneficial to have some kind of output so the user knows what's going on.
Case in point, this is the output I get when I try to clone the Linux kernel:
Generally not useful information most of the time, but if I didn't have it I would be staring at a blank terminal for an hour wondering what was happening.Also, I assume you're not but in case anyone is interested in the answers to these questions:
> What the hell is "delta compression"?
The 'delta' is the difference between one thing and another - usually one version of a file and another. Git does some fancy thinking to figure out which files are which other files but with changes, so that it can store just the changes from one version to the next.
For example, a 100 KB file where we only changed 500 bytes ten times would be 1000 KB, but because Git can store the deltas from one to the next it can be 100 KB (the original) plus ten 500-byte changes, for a total of about 105 KB.
> Why do I care how many threads it's using?
Because it directly affects how fast the process works; using 16 threads is 16x faster than using 1 thread (on average). Git automatically detects how many CPU threads are available and uses as many as it can, but if it's being very slow you might look and see 'oh, right, this VM only has two CPUs'.
> What is an 'object' and what does it mean when it's 'local'?
Uh, this one is deliberately vague I guess. An object is a thing that Git keeps track of. Usually this will refer to a blob, which is 'a bunch of bytes that make up a file', or a 'tree', which is a list of files and other trees - basically a directory structure, or a commit's information, but anything that Git keeps track of is an object.
Local just means that you already have a copy on your system. in the 'remote:' line you see output from the other end (where you're pushing to), so that's the server saying that it's using the files it already has.
> What does 'pack-reused' mean?
To be efficient, Git can take all the 'objects' and smush them into one big packfile (rather than having to keep track of hundreds or thousands of separate files). Since Git keeps track of files based on their contents, two identical files are just stored as one copy referenced twice, so it's possible that the file that you're pushing already exists in a pack file and can just be reused rather than having to push another copy.
I use GitHub desktop app that pushes to my local Gitlab. It’s a nice and simple GUI, it might be what you’re looking for.
This is a very promising announcement for Unreal game development specifically. For any other purpose I wouldn't care as much.
Perforce definitely needs a challenger. It is not the incumbent because it is particularily simple to use or administer. Git is actually way simpler when it comes to branching operations for example.
The reasons why p4 is often preferred in gamedev have already been mentioned in other comments: large project support, permissions, file locking and so on. Another key reason p4 is the king for Unreal dev is just how well it's supported inside the engine. Not perfect, but it's the best supported VCS because it's what Epic uses. Even the Git plugin is painfully unfinished, because Epic does not internally use it. So with Lore I expect them to give it first class support. I'd recommend Git a lot more if the support in Unreal was better.
(background; I've been in gamedev for almost two decades now, 2-200 person companies, every kind of engine and version control system. I prefer git where I can use it: for Unreal that means small projects and/or tech savvy team members. Pick the tool that is right for the job and the team.)
Turns out it is not really new but only open sourced it now. From the FQA.
>Lore, formerly called Unreal Revision Control, is the built-in version control system for UEFN (Unreal Editor for Fortnite), where creators have been using it to version their islands. It is also seeing progressive adoption by internal Epic teams, and is being implemented as the backing store for UEFN’s cook pipeline, where it replaces traditional intermediary storage layers—eliminating redundant file transfers and significantly reducing the time between publishing changes and those changes being playable.
Surprised it is in Rust and not Epic C++ or Verse. I wonder why.
I suspect the use of Rust rather than C++ might have something to do with the fact that Simon Peyton Jones and Lennart Augustsson (both of Haskell fame) both work at Epic and there would have been a strong internal push to do this in a language with some functional programming features. Rust rather than Verse because that would probably not be the right tool for the job (even if Simon works on it). Rust rather than Haskell probably because of performance -- DARCS never caught on partly for performance reasons.
I don't see how Rust is more functional programming oriented than C++
C++'s stdlib collection algorithms were basically all mutate-in-place instead of return-a-new-value until C++20 introduced ranges, and I still don't know anyone who actually uses those. By contrast, people actually use the `container.iter().map(...)` etc. in Rust.
C++ also lacks much of the fancy pattern matching features which are frequently associated with functional programming. Such features may not be fundamental to functional programming, but they would fall under "some functional programming features", in the sense that they're common in the languages of that ecosystem.
I think the things that make Rust safer than C++ make it look more like a functional programming language than C++. The main point is that there are some influential people at Epic (SPJ has often spoken favourably about Rust) who would favour it over C++.
The premise is that Git-LFS sucks, so we need to build a new data versioning system (in Rust, from scratch). While I mostly agree with this premise, but there are already lots of existing (mature) data versioning systems with the same tricks under the hood:
- Pachyderm (Go): https://github.com/pachyderm/pachyderm
- XetHub (acquired by HuggingFace): https://huggingface.co/blog/xethub-joins-hf
- LakeFS (Go): https://github.com/treeverse/lakeFS
- Oxen (Rust): https://github.com/Oxen-AI/Oxen
I guess with AI, anyone can vibe code a content-addressed, chunk-level deduped, versioning system in Rust these days...
But jokes aside, Lore seems really cool! What's interesting is the realization that different domains/industries have similar problems, but they don't seem to be cross-polinating. In this case AI and Gaming both need a storage system that can version control large binary files at scale. I think there's lots of opportunities to share ideas here, but perhaps the lack of idea sharing (currently) creates opportunity!
I don't think the needs are exactly the same. I believe in AI the big binary files are normally written once, while in gamedev, they are constantly updated.
That already warrants different storage architectures.
I'd trust this project more if it was named Data.
Yeah, I'm still concerned about crystalline entities suddenly showing up. Have they ever fixed it? I don't see anything in the issue tracker, probably because no one was left last time to report it...
If it were Data instead of Lore, it would reject any commit messages that used contractions such as "I'm" and "can't" ;-)
Dunno, this seems fully functional?
Oh brother.
Don't be too hard on Lore.
All Data is Lore. I mean lore is a superset of data. I mean data is lore with a special attribute.
I'm not just picking nits here. And this is not cynicism.
so there you go.
I'm not the parent, but I suppose it was a joke reference to Star Trek where Data (an android character) discovers he has a brother named Lore [0].
[0] https://en.wikipedia.org/wiki/List_of_minor_Star_Trek:_The_N...
Oh, so that's why JetBrains data thing is called DataLore. TIL
What is this, the sequel to the evil Spock with a beard?
an evil brother
No, just different. It's not like he had a goatee.
The likely source for Lore - https://memory-alpha.fandom.com/wiki/Lore
"Lore" is appropriate. Epic games is a very unethical company that steals from people. Myself in paticular. I bought Rocket League the game for linux from Psyonix. Epic bought Psyonix and immediately removed the game clients for linux and mac os. I can no longer play. They stole from me and many others. It'd be one thing if they just shut down the game entirely, but stealing it from only some people while keeping it going for others is worse than just killing games.
I haven't played it since the buyout either, but I'm told it works perfectly with Proton.
Yes, running the windows version of Rocket League might be possible. But that's not what I bought.
> They stole from me and many others.
It's more likely that they took advantage of some things in the license that you weren't aware of, since stealing would be illegal.
"Full-surface API" is a feature nobody here has mentioned. Is that a dig at how git intentionally has no linkable library? I saw this earlier https://news.ycombinator.com/item?id=48470604
We had to use Perforce (Helix Core Cloud) at my last game studio, and it is the de facto industry standard that most of your creative staff is already familiar with. The programmers don't love it, but they don't rule the roost in games. It's also the safe, verified default for working with Unreal Engine 5.
It does show its years though. We were one of the first users of the Perforce cloud offering, as we were small and didn't want to self-host ourselves, but it was a bit of a rickety experience. You had to register an Azure account in order to be able to access the service, and you had to ask support to modify things like triggers. Coming from the world of GitHub and other SaaS products, you could tell it was an attempt to retrofit an older model into a new skin.
The Git LFS path has some unofficial support as well, but you are on your own when things go poorly. Epic doesn't provide much help there.
Competition in this space is welcome, especially if they're planning to make it fully officially supported by the Engine.
I wrote about why merging files isn't as common in the world of game dev for folks coming from the world of text: https://www.kuril.in/blog/why-game-devs-dont-merge-files/
Does this support using S3 as the backing store?
That would be very powerful for various use cases.
There was and still is a very nice competitor in this space called PlasticSCM. They were bought by Unity a few years ago. Unity has not been a good steward. They should have done what Epic is doing and open sourced it. But instead they chose to give it P&L responsibility. Curious what it's contributing to their financials.
>fully open source >look inside >Lore Desktop Client is available as binaries only, download the installer for your platform here:
https://epicgames.github.io/lore/roadmap/#desktop-client
> Open-source the desktop client so the community can build on its full graphical experience, not just download it. An early desktop client already exists as a binary download, but it isn't open source yet — it depends on some proprietary components, including Epic's internal design system. We're working to make all of it available in the open so that the client can ship as source alongside the rest of Lore. Lore is an open project, so it is important that the desktop client — which will be one of the main ways many people will interact with Lore — is also fully open so that the community is free to review, extend, and shape it.
Thanks. I still think it's a bit weird to say "fully" open source while your flagship client is currently closed. I realize they're referring to the VCS itself, but—well, if they just dropped the "fully" for the time being, I wouldn't bat an eye.
It’s pretty obvious the entire surrounding text of this project (and presumably the code itself) is vibed. A lot of that is probably aspirational.
I was excited about this project, but you convinced me to look into it and yeah, I'm seeing emdashes and overly verbose/elaborate commit messages :(
What a shame
to be fair, that's just the desktop client. You can use or build on top of the CLI
they do say they will open source it, but who knows:
"It isn’t open source yet—it currently depends on some proprietary components, including Epic’s internal design system—but we’re committed to open-sourcing it in the future"
> to be fair, that's just the desktop client.
I expect fully to mean fully, though.
There’s a “Repositories” link at the top of the page that links to https://lore.org/#repositories, WhyHow links to various GitHub repos, including https://github.com/EpicGames/lore, which claims to have code for the CLI. I see no reason to suspect that claim is incorrect. The code likely lives in https://github.com/EpicGames/lore/tree/main/lore-client)
I don't see a workflow for locking assets while they are under modification. This is kind of important for assets? Since we don't really have great merge workflows for meshes/animations/sounds/etc.
I also don't see any sort of GUI client? So the whole art team is going to have to get up close and personal with the CLI
Too bad it does not support fully local/detached (without server). It would be nice to be able to run it similar to a local repo without remote
repo: https://github.com/EpicGames/lore
Looks very git-ish. But probably better equipped for large binary files.
Git-ish CLI is great. The GUI is more important though. Non-programmers don't want to dabble with CLI. One reason why Perforce is the defacto standard IMO. The GUI covers 99% of daily used operations and is easy to use.
As someone who has thought a lot about VCS design [1] [2], the chunking approach is the wrong one and will still waste space.
[1]: https://gavinhoward.com/uploads/designs/yore.md
[2]: My WIP VCS has been named Yore for at least two years; I did not copy Lore's name.
This is a very long document that says nothing about chunking at first skim. If chunking is actually wrong, then just explain why, here. Wasting space is not actually a problem if it’s optimized for other purposes instead.
When it comes to large assets, wasting large chunks of space is a problem. If your chunks are 64 kib average (from the Lore document), but changes only average 1 kib (which could be a high estimate), then you will still run out of space 64 times faster and need to read 64 times more data off of the disk for certain operations.
It also makes diffing hard, as well as diff viewing.
What do you do instead of chunking your snapshots? Storing diffs is usually the other approach.
The best answer I have is for you to read the "History Model" section of that design doc through the "Implementing CRDTs" subsection.
I’ve always wanted a git with five commands, and maybe with AST based diffing
The five command part isn't really possible but you can use custom diffs for merges, git diff, etc. pretty easily. There are projects like diffsitter ( https://github.com/afnanenayet/diffsitter ) for doing more intelligent diffs like this for supported languages.
EDIT: and then an example for the merge stuff I couldn't find while typing before: https://mergiraf.org/ and HN discussion a few years ago: https://news.ycombinator.com/item?id=42093756
Five is enough. Beagle uses five HTTP verbs: GET, POST, PUT, DELETE, PATCH. And it is syntax-aware.
https://replicated.wiki/blog/uris.html
clone, pull, push, branch, merge, add, commit are the ones I use, but that's 7
I had wanted the same thing for a long time and jj + difftastic has satisfied me.
I like everything I've read on this site so for, for it is also something I've been wanting.
If the roadmap's "Web client and code review tools" could replace gerrit for me, this would be a easy switch.
Moreover, it looks like they designed both the mutable store and immutable store to be able to easily store their state directly on an s3 like system.
There are a number of features that would greatly speed up CI/CD system operations I belive.
https://epicgames.github.io/lore/explanation/system-design/#...
Helpful page that gets closer to the details
I wonder how many websites ChatGPT did by now...
They’ve been dabbling in this space within Unreal Engine for a few years. Perforce is the de facto standard in AAA studios from my experience, curious to see what’s going to happen to them.
Fantastic. Every time I mix git and games I end up almost having an aneurysm.
I must say lore is an awesome name for a version control system. Much better than git in that regards.
The link to Architectural Decision Records is empty, but they're present in the repo to look at[0]. Curiously the decision with the most deciders is the implementation of JavaScript bindings[1].
[0] https://github.com/EpicGames/lore/tree/main/docs/developing/...
[1] https://github.com/EpicGames/lore/blob/main/docs/developing/...
This looks very cool! I maintain a FreeCAD workbench for 3d model version control[0], and it currently uses git as the VCS, simply because that's what I was already using. Thinking long-term, I see it eventually morphing into a broader PDM (Product Data Management) system, perhaps even PLM (Product Lifecycle Management). Lore has a lot of the requirements already built-in, like centralized locking (better than SVN), and it's better suited for for binary files. I implemented the git backend as a protocol/port so it'd be pretty easy to swap it out. I'll be watching Lore closely.
[0] https://github.com/eblanshey/HistoryWorkbench
It’s great to finally see a possible alternative to Perforce.
The idea sounds good, even if Epic's recent track record of tools is not inspiring. But the commit messages etc. are very clearly products of vibe-coding. And version control is not the situation where "works 97% of the time" is a good-enough bar.
Passing for now.
About the recent track record, is there some technical problem or just drama with Unreal Engine that I'm not aware of? (I already have the same opinion about using AI-coded VCS.)
> Git’s content-addressed revision graph is excellent, but it treats binary files as second-class citizens—large files require bolted-on LFS rather than first-class chunked storage, sparse checkouts have sharp edges in offline use, and there is no native multi-tenant isolation.
I'm trying to figure out what Lore can accomplish that git+LFS can't. I've read about big binaries chunking, native interface and permission, is there anything else? Weren't those problems already solvable in the git+LFS ecosystem?
If you've used git+LFS for any extended period of times, you'd know how often it breaks, especially when used with forges like GitHub. Both GitHub and Git treat LFS as an after-thought and second class citizen.
Can you tell me where did it break? Or what feature you wished it had? I’m just curious, still trying to form an opinion on this.
The fact that it's even referred to as git+LFS instead of just git... If I needed to work with large files frequently, I wouldn't want such a basic feature bolted on. Not a criticism of git, just can see why Epic doesn't use git.
I've used git+LFS for unity projects without much issue. If I was working with more than 2 people, I would definitely reach for something centralized.
I came here hoping Epic Games somehow had launched a reliable alternative to GitHub, but saw their code is hosted on GitHub
Yeah, they should have posted it to LoreHub! I just checked for the availability of lorehub.com, and you can buy it for only ~$13,000 and start a competing business to GitHub.
Guess this is announced as part of epics state of unreal and if so this is already off to an amazing start even if this is all there is!
As long as Epic Games is anti Linux I will never use any Epic Games product.
Count on it.
They are so anti linux that they ship a linux version of Lore.
Well, that's easy for a Rust binary. But they don't put any effort into having a great UX for Linux devs with Unreal Engine, for example. It barely works on Linux and is almost impossible to run under Wayland.
They also buy studios and then kill Linux support when it was already working just fine.
From the horses mouth itself:
https://x.com/timsweeneyepic/status/964284402741149698?lang=...
He straight up makes fun of Linux because he hates Valve and GabeN.
https://x.com/TimSweeneyEpic/status/2054680825677910439?s=20
Game engine, programming language, VCS…will Epic launch a brand new OS?
Kind of funny that it's on GitHub no hate.
Missed opportunity for Lorehub.
does anyone have a proper comparison of binary control systems like lore, xet, etc? i'd love to see how it handles mixed case workloads.
Nice, this seems sort of like Git-but-for-giant-monorepos. That has been a gap in the opensource VCS market
Would lore be good for a tech company monorepo?
What sort of scale are we talking about when mentioning tech company's monorepo though?
Isnt that what SVN is good for?
Not immutable. Not good with binary files.
Interesting. Was‘nt aware of that immutable part. I use it with big bin files for year. Maybe i was lucky
this looks cool for game development, because using Git for projects in Unity and Unreal Engine definitely has it's issues. I'm personally not a fan of Git LFS, especially since GitHub charges you to use it (which makes sense, binaries and assets are big, code is small, relatively speaking).
Their docs seem entirely LLM written. It seems especially obvious in the FAQ. While I'm not against using LLMs for writing assistance, they've left a lot of the unnecessary language and typical stylistic choices in there, which erodes my trust in the project a bit. Perhaps it's a very good game-oriented version control system, but the lack of human attention on the docs makes me wonder how much they care
I reluctantly agree. I was interested to read the system design doc[1] but it's so many pages long and so full of redundant statements and needless details[2] that I gave up a couple sections in. The numerous "it's not X, it's Y" constructions give the game away. If they can't be bothered to read these docs themselves, why should I?
Anyway it's probably fine software and I am genuinely going to give it a shot for a usecase I have involving large image files. But the LLM-generated docs don't inspire confidence.
[1] https://epicgames.github.io/lore/explanation/system-design/
[2] They literally have a section header "10.1 Revision state as a 320-byte fragment". The byte size isn't even relevant in the code as an implementation detail, much less belongs in a design doc. No one read this doc before publishing it.
Yeah, I agree about the docs. I started on the system design page and my head started to swim after about 5 minutes. So exhausting to read!
On the flip side, I expect the project itself will be workable - well, assuming they're actually using it themselves! UE is a big pile of Stuff on its own, and Fortnite must have god knows how much additional crap in there, so if this is (or will be) their replacement for Perforce internally, then it'll be getting a good deal of testing. (If they're just chucking it over the wall, though... well, sheesh, you first...)
(Perforce is the standard thing for games, and pretty well it works too, and hopefully this will deliver it a well-needed kick. It was sold to private equity about 10 years ago, and it feels like they've been coasting ever since. (Perhaps users in other sectors are happier though?))
Yeah, I hope I'm wrong, but it feels a little bit like "planned project no longer has internal support, let's see if we can make it open-source to garner goodwill and recoup some of our investment." Which isn't the worst thing that can happen.
Hosted onn GitHub. Heh.
Lore itself is not an example of a program that meaningfully benefits from any of the key features of Lore.
Lore is meant for situations where your repository is going to contain gigabytes of binary files, such as art assets for games. Git is technically great at everything but that, and even the external solutions for that situation still kind of suck.
Is Lore worse at managing text files, though? If not, it might make sense to adopt it fully in an organization so you can just use one tool.
Git is currently industry standard tool for software development, which means that almost every IDE and code editor has good and mature GIT integration. It will take a while for Lore to achieve similar level of support. Considering the intended usecase seems likely that they focus the effort on integrating into game engines and game development specific needs instead of making plugin for every last code editor. Not only code editor but every other software related tool interacts with version control - code review tools, CI, package managers and code forges. That's a lot more work than just creating a version control system. Why waste effort on areas that already have good alternatives.
Code forges lead to the next reason -> they are making it open source (see the faq for why). If they want community collaboration it makes sense to prefer industry standard tooling. Organization internal consistency doesn't matter to external collaborators. Various Google projects with all their custom tooling are good example of what happens if this gets ignored. Every time I have to deal with depot_tools or repo it's a pain.
Third reason not to self host is that it's not quite production ready (see "Is lore production-ready?" in the FAQ). Can't be self hosting from day 0 and once you have a working setup you need better argument for switching just because you can. If lore was attempting to make a statement that it's a superior method of version control for regular source code it would be a different story.
Interested in this as perforce is pretty terrible a lot of the time
It’s like anything you do has to talk to the server
Even something as simple as diffing a file will just hang if there are server issues
The incredible laggyness of that website does not inspire confidence. Much of the text selection is also broken, and chrome consumes nearly a full core trying to render.. something?
Its remarkable that anyone thought this website was fit for release, and it gives off strong slop vibes
I also have absolutely zero trust in a product like version control being provided by a for-profit company. It seems like a terrible idea to tie your software stack to Epic Games of all people, given their track record
This is primarily going to be targeted at Unreal Engine users, for whom the source control tooling they use is the smallest of concerns when it comes to being tied to Epic Games.
This looks really good. I have been using git to store some PDF (tens of GBs) and git is really not well suited for this. No GFS is not a solution.
I'm in the market for a better VCS designed for gamedev-specific concerns, but reading the system design doc, it's LLM-generated. Not exactly confidence inspiring.
What makes lore better or worth considering... when svn and git never failed me...
Git does great with text files, but game development contains a lot of binary assets (textures, videos, 3d models) and correspondingly huge repos. git-lfs tries to patch around that, but that makes a complex tool that creatives struggle to understand even more complex. Perforce is a pretty popular solution, and was used by Epic in the past
Have to think they're doing this out of a real need, given they were already using Perforce and must've considered Git too. It's also not like 2010s when version control was a hyped thing.
Is there no git trick to turn off version control on non-text files but still store them? How does Lore handle them better?
You still want version control, and locking so that two artists don't concurrently edit the same asset.
> It’s optimized for projects—including games and entertainment—that combine code with large binary assets, and caters for the needs of developers and artists alike.
When you have a game that weighs in at 100GB, only a tiny fraction of that is built from code. The rest of it is binary assets that most VCSs struggle with. What makes this worth considering is the fact that they designed it to handle binary assets.
Apparently, how it handles binaries when developing games.
Why not just use Alienbrain?
never trust epic
Why not?
Ahah, the second and third links on the page are to GitHub
Well it seems to be intended for repos with large blobs like video games so it kind of makes sense that their own pure source code is still managed with git.
It makes total sense. It’s just kind of ironic.
Epic's Unreal Engine development is made in Perforce, and then it was mirrored to git/Github. Or that's how it was a couple years ago. That's kind of expected, VCS is one thing, a forge is another thing.
Interesting to note that this does not seem like a DVCS in the traditional sense because it depends on coordinating with a central server where all repositories will be hosted. I can't tell if servers can pull/push from eachother.
So it's more like git in practice than git in principle, I guess.
I am not sure it is much of an issue to implement a centralised system to solve domain-specific problems that are caused by or are in the context of inevitable centralisation.
I can see a bunch of media companies liking this, maybe even small design shops. Quite a lot of CMS and change control work for media is not much more than asset management; version control is often a bit of an afterthought.
Web interfaces to this might end up interesting for non-profits and charities with significant media outreach.
It is not a DVCS.
> 3.2 Explicit non-goals¶
> Peer-to-peer decentralization. Lore is centralized by design. Two clients communicate through the remote, not directly.
What a waste of a phenomenal domain name.
My first thought exactly :(
I can't imagine it was cheap to acquire it, so any company with VC/big Corp money would've taken it any ways.
How long before Epic starts giving away other software and suing git to support lore?
I can’t remember the last time anyone actually played the game they got for free on epic’s store
Citizen Sleeper, the game that is going free tomorrow on the Epic store is really good. Space Cyberpunk themed RPG/Survival Sim.
It is interesting that people are so cynical about Epic giving out free games. I get that people love Steam, but competition in the storefront market is not bad.
> It is interesting that people are so cynical about Epic giving out free games.
It's a mischaracterization to call the games free, if they require you to install unrelated third-party software you'd rather not install (and which at least in the past has been known to snoop your data without consent). In that sense, you may see it as backlash around characterizing the games as free in the first place, when they obviously are not.
Does it support Linux?
Kerbal Space Program, Civilization 6, Hogwarts Legacy. All games nobody ever played…
I definitely have. But they're all the games that they gave out in the early days, the more recent games haven't really appealed to me. And it didn't convince me to spend any money, I still haven't spent a single cent on anything Epic Games.
I've been getting them for years now and have enjoyed many of them (though not a large percentage), and would love to use the Epic store more, if only they would implement some version of reviews/scores. Without that, I feel that I can't trust them as a marketplace.
I played hogwarts legacy for about an hour before realising it was boring, does that count?
I play Brotato fairly often. I am not sure I have played any of the other free games I’ve gotten though.
i've paid for brotato (Its a pretty good game, it was worth my money) and played city skylines (which i haven't paid for due to its excessive amount of dlcs)
They gave away Death Stranding awhile ago, and I hear that one is good, which I don't dispute given its Kojima origins.
But Kojima games require me to be in a very specific mood and I have been trying to work my way up to it.
This is just going to become another way to lock developers into UE. Then they will start charging for licenses, same as Unity did for its versioning feature. It might be open source but that doesn’t stop the commercial use of it being charged for.
It has an MIT License
It doesn’t matter
https://en.wikipedia.org/wiki/Open-core_model
We're also working on an open source large asset versioning tool called "oxen" - https://github.com/Oxen-AI/Oxen
Would love any feedback on it or contributions if people are interested :)