This article is full of mistakes and misleading claims:
1) It's claiming SHA1 insecurity is theoretical, while SHAttered from 2017 was specifically a pratical proof of concept. The only reason Git wasn't affected, is because they didn't bother bruteforcing a git-blob prefix.
2) It's claiming collision attacks don't matter, only second-preimage attacks do. This is incorrect, collision attacks are enough for code-smuggling problems, when two repositories are on the same git commit (verified by the full commit hash), yet contain different code in their git checkout.
3) The Linus quote "The real security is in distribution" is arguing that "git's content-addressed system should not be used to address content". It's arguing that, in case of curl|sh, you shouldn't use a sha256sum-gate to pin the content to something you've reviewed, you should instead ensure curl is fetching from an https server.
1) I link to the SHAttered paper, as well as Shambles. Git projects were not affected because it is an inefficient attack vector. I say it's impractical to exploit, which I think everyone agrees with.
2) I specifically argue that even if both attacks were practical and cheap, it's still not the problem we should be focusing on.
3) Have you read this email (that I linked to)? It is almost the same general message (20 years ago) that this blog post is. It literally goes though a theoretical object replacement attack and how dumb this scenario is and so SHA-1 is fine.
> 1) I link to the SHAttered paper, as well as Shambles. Git projects were not affected because it is an inefficient attack vector. I say it's impractical to exploit, which I think everyone agrees with.
It seems unlikely it will stay that way forever. Typically attacks get more efficient over time as researchers find improvements, not to mention computers getting better.
In 2015 it was estimated to cost $100,000, now the estimate is down to $10,000. Where will it be in 2035?
The problem of a SH1 collision happening by coincidence is vanishingly low and theoretical.
Nothing else matters.
Git hashes are not supposed to be a security mechanism. If your basis for trusting that you have the right checkout is the git hash, in a situation where you have legitimate concern about untrusted parties manipulating remote repositories, then you're simply wrong.
When I check out code from a git repository in a pipeline using a git hash, I expect the code to be exactly what has been reviewed by me under that hash.
Everything else would just be a crazy invitation to make supply chain attacks uncircumventable.
And so if you don't trust the server that is hosted on or the security of the transport mechanism like TLS/SSL, such that the content may be manipulated by adversaries, you think that git hashes are good enough?
Well, what about someone who is fetching the commit from that server for the first time and has nothing to compare the hash against?
Oh, that would never be a problem for widely disseminated, popular, open source project, so it doesn't matter.
Dealing with potentially-hostile hosts is quite common, actually. See for example how most Linux mirrors work, or Subresource Integrity with HTML.
Turns out securing a service to transfer a single hash is a lot easier than securing a service to transfer gigabytes of data.
Even if I don't fully trust Github, it is still incredibly convenient to be able to upload my code there and then send someone an email telling them to fetch commit `123abc` from some repo link. As long as my email isn't compromised, that should be secure.
Because you're not killling two birds; you're not killing the security bird with a better content hash.
A SHA-256 sum, though very good, only assures you with great confidence that you're looking at the same thing you looked at before, or that someone else is looking at elsewhere.
It is not a digital signature, and we don't want digital signatures to serve the role of content hashes.
Speaking of signatures, we have support for them in Git; you can use gpg to sign commits, and set it up to be done automatically.
Nobody is going to fake your commit such that the fake has the same SH-1 hash and your GPG signature.
The worry there is that the key holder (whether the legitimate one, or a malicious party who got a hold of the key) somehow does this: creates a new commit, signed with their key, which somehow has the same SH-1 as an existing signed commit. The git hash includes the GPG signature, so there is a significant layer of difficulty there which is likely harder than faking an unsigned SHA-256 commit.
Please educate yourself what a merkle tree is. It's a well understood building block of various security systems, including certificate transparency (which explicitly uses sha256).
You refer to PGP signed Git objects, but you also argue:
> Git hashes are not supposed to be a security mechanism
I understand that if we sign a commit with the help of some arbitrarily strong hash, it doesn't protect the parent commit(s). The integrity of the SHA-1 hash references to the parent commits is not in question, but the authenticity of those commits themselves.
That doesn't make a difference: with sha1 a malicious change in content will still result in the same content hash, so the signature will still be valid, and the commit hash will still be the same.
Only if the GPG signing process stupidly relies on the SHA-1 hash. I.e. if it takes an unsigned commit and signs only its SHA-1 hash and then creates a new commit with GPG headers. If that's how it works, that is massively stupid and can be fixed without forcing SHA-256 as a git hash. Just have the signing calculate its own digest for its own purposes.
That digest can be the SHA-256; since the infrastructure is there for it, signing should use SHA-256 regardless of what hash is used by the repository for identifying and linking content.
One of my favorite fun facts about Fossil SCM (another source control by the devs of sqlite) is that they patched their use of SHA1 6 days after the shattered attack was published:
"Both Fossil and Git started out using only SHA1 hashes. But when the SHAttered attack against SHA1 was published on 2017-02-23, the need to migrate to a stronger hash algorithm was recognized. Fossil added the ability to use SHA3-256 as an alternative on 2017-03-01 (six days after the SHAttered attack was first published). SHA3-256 is now the default for all new repositories and check-ins in Fossil, though older check-ins that occurred prior to SHAttered can still use their original SHA1 hash. Hence, no repositories had to be rebuilt and no hyperlinks were broken."
To me it's so interesting watching in realtime Git is still battling with this decision and for Fossil it was just another week of development.
That whole page is fun to read. Another fun fact somewhere else in the docs is that Fossil uses a grow-only set to store commits. They came up with this scheme some years before it was formalized by CRDTs!
I mean, there are two things here. One is how difficult it is to have a different hashing mechanism. Brian and other heroes in the Git core group have done amazing work to make this _technically_ possible on a repo level. To test some of my theories, I trivially implemented MD5 and an insanely dumb and easily breakable hash backend. It's not _hard_ to change the mechanism now. It's about the community.
Fossil isn't difficult to change not because it's technically harder for Git but because Git has a community and ecosystem that Fossil does not. The cost is not in the individual project for Git, the cost is because there is _so much_ in Git and this bifurcates everything.
Also, interestingly, Git today does _not_ use a straight SHA1 because of these attacks. It uses `sha1dc`, a slower collision detecting variant that specifically checks for this vector of attacks. So currently, Git's SHA-1 variant is not susceptible to the SHAttered/Shambles attacks.
To me it's more of a reality of creating a tool with a huge active community and a community of contributors and creating a tool with a small team and small community.
It's easier to make world breaking changes when the world is really small. If git could magically just get everything and everyone to cut over and use git 3.0 in a magic instant, it wouldn't be having this problem.
> but the point is the SHA-1, as far as Git is concerned, isn't even a security feature. It's purely a consistency check. The security parts are elsewhere, so a lot of people assume that since Git uses SHA-1 and SHA-1 is used for cryptographically secure stuff, they think that, Okay, it's a huge security feature. It has nothing at all to do with security, it's just the best hash you can get. ... [1]
Yeah, in 2005, SHA-1 was just about the best you can do given the constraints of the time (without picking something much more esoteric, much slower, etc.). Using SHA-256 at that time would have been noticeably slower on the computers of the time, and made repo metadata take up a lot more space.
It was a mistake to assume a fixed algorithm in the repository format and client-server protocol. I remember being surprised when I learned about that choice, being familiar with cryptographic protocols and formats where the hash algorithm is usually a parameter that can vary for each concrete hash.
> It was a mistake to assume a fixed algorithm in the repository format and client-server protocol.
See also perhaps Wireguard, which touts itself as not having "cryptographic agility" because they wanted to avoid all (perceived) problems and complications of IPsec. But now that PQC is (allegedly) approaching there's no easy to update things because (AIUI) there's no negotiation possible in the protocol; you're basically standing up a 'Wireguard 2.0' that runs separately than the original.
I don't understand why Git is not making the SHA-1 and SHA-256 modes far more compatible with each other.
SHA1-hashed objects should be able to refer to SHA-256-hashed objects, although this seems somewhat pointless.
But SHA-256-hashed objects should also be able to refer to SHA1-hashed objects, with a major caveat: if those objects themselves are part of a collision pair, then there is a genuine problem. But this is avoidable! Suppose that Linux decided to migrate to SHA-256. The upstream project could choose a pair of dates, say January 1 2027 and March 1 2027. Up to the first date, maintainers would be welcome to submit hashes of objects that are not yet in the repo but that they think they might submit later on, and, on that date, the upstream tree would finalize the list of these objects and reference it in the repo (with a new mechanism for this purpose). Effective the second date, the repo would start publishing SHA-256 commits and would never again accept a SHA1-hashed object that was not in the repo at the cutoff date or referenced as part of the Jan 1 block.
And now it would be impossible to get a new SHA1 collision in to the repo.
The only new git features needed would be:
a) actual compatibility so that a SHA-256-hashed object could reference a SHA1-hashed object
b) a new object type that's a list of allowed SHA1 hashes (or probably a tree of them) that is itself hashed with SHA-256 and a mechanism to link to one of these from a commit
c) a policy mechanism to set a repo to only allow SHA1-hashed-objects that a reachable from a preconfigured SHA-256-hashed commit
I skimmed the video, and I didn't quite catch that. Near the end of the video, however, she did mention that interop is in the works[0].
In any case, even if Git 3.0 were completely incompatible, it would suck, but it's not the end of the world. You just treat it as if you were migrating from one SCM system to another. CVS -> SVN -> Perforce -> Git -> Git 3.0 -> [...] been-there-done-that. This is something that both open-source and commercial projects have had to deal with over the years.
Or maybe it would be a repeat of Python 2.x -> 3.x. ¯\_(ツ)_/¯ With AI assistance, hopefully porting the tooling over may go a lot quicker and smoother.
Thanks. I went back and re-watched that part, and her point's that "[a] tree's cryptographic strength is equal to the weakest hash algorithm anywhere in the tree," and that's fair. I also understand why Git 3.0 may not want to give user the choice, though I would still rather it be given.
> "[a] tree's cryptographic strength is equal to the weakest hash algorithm anywhere in the tree"
This is simply wrong IMO, for two reasons:
1. The attack on SHA-1 is a collision attack. Once you have frozen a hash, you cannot attack it with existing cryptanalysis. If there were a preimage attack it would be a different story.
2. Even if there were preimage attacks, one could freeze a mapping from SHA1 hash to SHA-256 hash.
In fact, #2 seems like en excellent design. Objects could reference such a mapping, and a repo could disallow conflicting mappings (the mappings would only be accepted if the mapped objects are reachable from the mapping and the mapping is correct).
They do give the user the choice, but the default is changing. My point is not necessarily to rip out the SHA-256 option, but simply to not make it the default. Because then people will create repos in that format that do not understand the ramifications, where the opposite should be true.
My point is not that it's the end of the world (or the end of Git), but that it will be painful and unclear and confusing to lots of people. That would be fine if it made a huge difference in trust or protection, but it's the wrong way to do that.
Hm but the date is stored inside of the commit. The only way we can know that a commit's date is authentic... is through its hash. If I can forge commits with any SHA1 hash at will, I can make a repository whose head commit has the same SHA1 as the one in torvalds:
/linux but where any commit was replaced by a malicious commit with the same SHA1 and a fake date. You have no way to detect that my repo is inauthentic other than through a deep history comparison. The whole idea behind a merkle tree is that just checking the hash of the top is sufficient to know the identity of the whole tree.
I don't know what the solution is, but I'm inclined to believe that any repo with a single SHA1 commit is as weak as a repo with all SHA1 commits.
The date that a repo receives a commit is known to that repo. And a repo can stop accepting new SHA1 objects. And a SHA256 object could have a flag that says that no SHA1 objects may ever reference it.
The design of Git, as a Merkle tree, is meant to allow for use cases like this:
* I host a mirror of the Linux git repo.
* You download Linux from my mirror.
* You check out a commit, say fd179f8a05be3ccae366b9b96e176b51fbe54aab, which you know is a genuine commit through some out-of-band mechanism (mailing list, GitHub web interface, a line in a Nix file, whatever).
* You check whether the repository I gave you is legitimate or not by re-computing the hash of the commit which I claimed was fd179f8a05be3ccae366b9b96e176b51fbe54aab. If it comes out to be fd179f8a05be3ccae366b9b96e176b51fbe54aab, you know it's legitimate. If it doesn't, you know it's fake.
This is a completely normal use of Git. People download from mirrors all the time. People rely on commit hashes to identify a specific source tree. People trust that if whatever the mirror gave them hashes to the right value, it's genuine. That way, you don't have to trust the mirror.
If I can forge my own commits to have any hash I want, this whole model breaks down. I can replace some old commit in the repo with my own forged commit with the same hash, and when you download a copy of the Linux repo from my mirror, you'll receive a repo with malicious content, but it'll hash to the same fd179f8a05be3ccae366b9b96e176b51fbe54aab hash as a genuine repo would. This breaks the security model of Git.
> You check out a commit, say fd179f8a05be3ccae366b9b96e176b51fbe54aab, which you know is a genuine commit through some out-of-band mechanism
That's a 160 bit hash, which is SHA-1, which has the security properties of SHA-1.
Suppose you check out a commit with a given SHA-256 hash. That commit object represent the root of a tree where all the edges are hashes (and types, etc). I'm suggesting one of two designs:
a) (Simpler but weaker) If Linus has published that commit, then he is confident that he hasn't pulled in any too-new SHA-1 hashes and that there are no collisions present in what he thinks the tree is. So, by induction on the traversal depth, there is only one actual object identified by each edge, and those objects contain the hashes of their child edges, so those hashes are all correct.
This breaks if there is a malicious collision already in the tree.
b) (Stronger but higher overhead and more complex) There would be an object or objects, discoverable from the root by following only SHA-256 edges, that encode a duplicate-free mapping from SHA-1 hash to SHA-256 hash. The client finds and parses that and then, as it traverses the tree, each time it reads a SHA-1 hash, it computes the SHA-1 and SHA-256 hash of the referenced object, verifies that the pair is in the mapping and also verifies that the SHA-1 hash matches what the edge requires.
I think that (b) is genuinely cryptographically secure in the sense that, if you can construct a commit that has the same SHA-256 hash as an official upstream commit but different contents, then there is necessarily a SHA-256 collision.
For A), I don't understand what the point is? I never mentioned what Linus is confident about, I talked about what you can verify when you pull from my mirror. I could replace a commit from 2010 with a malicious one
For B), I would think this could work, but it's a completely different solution from what you proposed and what I responded to.
Tools like git-filter-repo[1] support rewriting commit hashes in commit messages. git-filter-repo actually does it by default; see `--preserve-commit-hashes` in the manual[2].
... do not migrate old repos? I'm not sure why people would do that. Or, if they do, why would they replace the current repo name instead of creating a different one and keeping the old one closed to make the references work.
I don't think this is going to be a problem at all.
Once this starts being actual pain, we will each vibe the replacement index creator (git already supports replacement objects), for back-forth conversion, populated on pack and object indexing.
For massive perf and mem use damage. But oh well. And then we will wait for official version
There are plans to keep sha1s around in a database, but as far as I know, no way to transmit those, so they seem specific to individual forges. They can be recomputed, sure, but again, any signatures break and it's possible that in the case of an actual replacement, the recomputation is now wrong and not easily comparable. So what is the point?
This is not about forges, it is about the repo I have in a folder on my computer. The sha1 hashes shouldn't go away. Yes the forges also need to support this.
The difference with IPv6 adoption is that the internet relies heavily on network effects: so long as some hosts only have an IPv4 address, you need an IPv4 address for full connectivity, but then if everyone has an IPv4 address anyway, there is no immediate need to migrate to IPv6.
(Yes us Hacker News users have plenty of use cases for IPv6, like self-hosting and peer-to-peer networking and so on; we are not the average user.)
This effect doesn't exist for the Git migration. Each repo can be updated independently; it doesn't affect users of other repositories, and most likely, the majority of devs will work on some SHA-1 repos and some SHA-256 repos with no issue.
If anything, I would compare it with the Python 2 to Python 3 migration, which was also painful, but succeeded eventually (despite being much less necessary in the first place).
Which is considered a Really Bad Idea because tags aren't immutable, so there's absolutely zero guarantee that it'll point to the same commit a few months from now.
The GitHub Actions ecosystem found out the hard way, through some rather high-profile compromises. They hotfixed it by adding "immutable tags" to their platform, and are now working on adding a lockfile to... easily reference a commit hash.
Is it? There are many benefits in principle to IPv6, but if my ISP continues to assign me a single dynamic IP, those benefits are entirely moot for me.
Plenty of ISPs do it right (examples: my last 3). That the parent poster's apparently doesn't isn't an argument against the technology their ISP fails to deploy correctly.
Reality is that lots of ISPs do this right. And have for a very long time.
... if you use ISPs that don't deploy the technology correctly. Faced with multiple providers of a technology, are we to judge its benefits by the worst ones?
Right, but just because you don't happen to have IPv6 right now, how does that remove the benefits for others to have IPv6? That's like saying having a faster CPU wouldn't mean faster performance, because I don't have that CPU yet.
There's a massive push right now from top down to have secure software supply chains. Google SBOM and SigStore. It's not an organic need but if you have government customers you don't have many options.
I thought this would be a snark but it's an extremely well put together argument against the "Hashmageddon".
If you're replacing the weakness of SHA-1 just by going to another algorithm, you better be prepared to go to the next one when sha256 collisions happen, and it doesn't sound like git's design would be easy to modify for this type of crypto agility.
I do like their proposal for using signatures to establish trust and allow swapping sha256 for whatever comes next.
Technically, git's design (thanks to very smart people trying to solve this problem like brian and others) is _very_ easy to modify to different hashing algorithms now. A lot of amazing work has gone into this in recent years.
However, it's not a git problem. It's an ecosystem problem. It's that every git repo has to choose one and they're entirely incompatible with each other. That is the cost and the difficulty.
I think there is a question though when that will happen and if it will be in our lifetime. SHA-1 started showing weakness in 2005 (collision in 2^69 instead of expected 2^80. This was later brought down to 2^61 in 2011), the same year git was invented. Nobody has found a similar weakness in SHA-256 as of yet. SHA-256 is still at its design strength of 2^128
It took 20 years to go from vulnerability in sha-1 to having to replace it out of caution. There is no such vuln in sha-256 yet. It could easily be 25 years before we find one, and another 25 years before we have to do something about it. Perhaps longer. Will git still be used 50 years from now?
> All it takes is just one collision to consider it broken right?
No, its considered broken before that stage. i.e. when someone discovers an attack that would allow someone to create a collision faster than they should while still being impractical.
> With the kind of compute power available nowadays and AI models I wouldn't be surprised we see it much sooner.
Computer power doesn't super matter, what matters is algorithmic breakthroughs. So far i dont think there are any examples of major breakthroughs of that type via AI, although perhaps i am just misinformed. Its still early in the AI revolution, it might still happen, but as it stands i don't think there is any reason to worry about that.
It does though, even if it shouldn't be blindly taken as gospel. Arguments help, but some folks have a better brand of apple box to stand on, and "I live and breathe git" helps quite a bit ;)
So they’ve been talking about this for many years, planning, and finally announce when they’re going to switch the default.
So this is the right time to post that everything they’re doing is wrong? Did you engage in all the discussions about it and how best to handle it? Whether SHA-256 was the best solution?
I don’t see anywhere that it talks about alternate proposals or why they might have been better. Why the particular suggestions here were rejected.
This seems like a bunch of Monday morning quarterbacking.
I do mention this in like the first paragraph. I don't feel great about it, but I've listened to these issues for years now during contributor summits and Git Merge talks and while it's always seemed problematic, I thought they would come up with a good solution. This last Git Merge confirmed that it's close to the switch and not in any way solved or improved. I don't want to just go with it for groupthink reasons. I never thought it was a good idea and I have said that, but we have a last chance to rethink this, so I'm curious if I'm alone or in the silent majority.
Your argument is persuasive and well illustrated. I think the problem is the intro paragraphs come off as too certain of catastrophe which, when juxtaposed with your claim that "smarter people than me have been working on this", makes it sound like you don't actually believe they're smarter than you. The rest of your essay feels fair and not judgmental.
I do believe they're smarter than me, but sometimes very smart groups talk themselves into ultimately impractical solutions because they're all smart. Sometimes you need a dumb guy to come in and say "are you sure this is right?"
I believe you are sincere. But "is about to be a huge, costly, global train wreck" lacks the nuance of "are you sure this is right?" and will rub some people the wrong way.
Edit: I'm not suggesting you should have written it any differently. I think you made the right choice to be a bit provocative because it grabs the attention that's needed.
I do believe this. But that doesn't mean I can't be convinced otherwise by a good argument. The point of this post is to see if anyone has a great counterargument to change my mind.
It's a hitchhiker guide to the galaxy reference, where sure, something is technically available but not clearly published and there are hoops even for those who know what they're looking for.
(No clue if it's applicable here, I'm not aware of this case, but I believe that's the reference if it helps :)
Edit : exact quote, as Arthur's house is about to be demolished for a highway bypass:
"But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.
I believe it's a reference to the hitchhikers guide to the galaxy – where the plans to remove the protagonists building to build a bypass road was hidden in this way.
nofunsir is a stoichastic parrot, matching to a bit in Hitchhikers Guide to the Galaxy, wherin the protagonist should have known to protest a plan to demolish his home where plans where clearly documented in a hard to find place that they could not have known about. It is not a good pattern match, because git has been discussing this in public on documented mailing lists for years.
Since 1999, we've had Google to help people find things. True, a mailing list isn't an Instagram reel delivered directly to your face with audio and blasted out on Fox News, but if one was interested in the development of git, an LLM or a Google search would readily tell you about the existence of those mailing lists.
A number of years ago when I heard about this, I was pretty angry and made a private fork of git immediately in which I tried to scrub away the SHA-256 bullshit. But that's basically just paddling upstream with a spoon for a oar.
The stewards of Git are going to do whatever they want, and there is nothing you can do about it if you don't have the clout to create a fork that takes the lead.
No amount of discussion will do anything because they've already decided that their view of the situation is correct. Git hashes are not just content identification but a digital certificate mechanism, and their collision resistance is a grave issue that must be fixed, the end.
You will be browbeaten in any discussion; it's not worth the energy in a world replete with issues.
>it will be an incomprehensibly expensive and ultimately valueless and avoidable global nightmare.
thought "costly" in the title and "incomprehensibly expensive" in the subheader meant this piece would discuss how much less performant sha-256 is on modern machines, but didn't see anything. isn't there hardware acceleration? how much worse is it?
Actually, I think sha-256 is possibly faster than the sha1dc variant that Git currently uses.
I just sent a patch series to the list that enables sha1dc to be accelerated on modern CPU architectures to close to normal SHA1 speeds, but since it was ported from a Rust project by an agent, it will never be applied.
Yes every repo is either one or the other but you fix that by rehashing the entire repo. Everyone can do this independently. It's entirely possible to maintain to identical repos in SHA1 and SHA256 mode but for the most part I suspect once updated people will simply pull down the new repo and use git 3.0 as a required version.
As migrations go, it's reading as simple to me. You'll just have to backpoint the commit signatures. I must assume there's a backwards compatible reference for them in git 3, right?
Or drop them and reference the old structure in a dire pinch.
Re-hash the entire repo as in rewriting all history? Hooo boy will that be a mess, I deal with things which reverence commits by hash in repos all the damn time. There are thousands of them in every Yocto project!
Do you have any external references to any commits that matter, for example in your communication platforms (emails, Slack) or your bug tracker? Or, worse yet, in places where they aren't just text format references, but used for things like CI/CD caching decisions or security scans?
Once you rehash the entire repo, every single one of those external references will be broken. Because no, there's no support for looking up old hash -> new hash or the reverse.
"The migration to the new format is simple; Just re-write everything in the new format, but also keep the old format around forever too since data is lost in the new format!"
I positively don't care about the collision issue.
If you need to certify the authenticity of some code, and you've decided that a Git hash of any kind is going to be your certificate, you have a problem between keyboard and chair which is not fixable by stronger hashes in Git.
Can someone more cyber-pilled than me explain what the actual risk with Git hashes being susceptible to collision attacks is? Obviously accidental collisions are problematic, but to my understanding the probability of that is still approximately zero.
Best I can tell, all a forced collision would do is let someone who already has control of a repo modify the history in a far from plausibly deniable way. Which in practical terms, they already could do simply by replacing the whole thing, because who's out here using git hashes as a security tool? Every pinning I've ever seen has been to tags (which can be modified at will), or hashes of the actual payload (which doesn't need to be the same as what git uses).
Oh, agreed this sounds like a terrible migration path and shouldn't really be needed in the first place.
What I'm missing in the article is whether any Git server accepts replacing a SHA-1 identified object it already has. If it doesn't, then the distribution trust discussed holds, and keeping SHA-1 seems fine. Adding additional signatures seems fine for those who need transitive trust.
Ugh, I didn't know that SHA-1 submodules wouldn't be supported in SHA-256 repos. That changes the transition from painless to a major dumpster fire. Having to maintain converted forks, and use different hashes from upstream is going to be a mess.
I agree, but you're not going to sell that argument to a herd which has decided that git hashes are digital certificates which must be replaced with SHA-256, or the sky will fall.
I don't agree that SHA-1 is much longer feasible for git.
But I also don't think that switching to SHA-256 must be painful. A git2->git3 converted repo could just store all the past hashes, so existing links don't break.
schacon: Really like the "Independent Tree Hash Headers" idea.
How difficult would this be to get this functionality into git?
Would it cause any breaking changes with older versions?
Have you discussed this with any git devs to see if they are open to adding it?
Actually, this entire blog post came out of a short chat at Git Merge a few weeks ago with Jeff King. I argued more or less this and he didn't _entirely_ disagree, though he has good counterarguments on the list over the last few years, so I don't really know how he thinks about it ultimately.
I would write this to the mailing list, but I thought a conversation that includes people outside that list is more interesting to me. Ultimately I'm not sure if I'm dumb about this or the whistle blower that's willing to actually say "maybe this isn't the right call"
It was theoretical - the point was that maybe some paper is published or some new tech or issue comes up. Now we have to do this again. If we separate the concerns, then we don't have to deal with both as though they're one problem. We can deal with one thing for content addressing and another for trust and security.
The post's argument that hash collisions are irrelevant in practice is not convincing at all. Basically they amount to:
1. Collisions aren't as bad as preimage attacks
2. Even if you made a file-with-malicious-hash, how would you get people to pull it?
3. Other attacks are a bigger problem (social engineering)
(2) is laughable in a world with github. It's common for unknown people to submit pull requests to code bases, and for those changes to be reviewed and merged. For example, as part of reviewing pull requests, I have `git fetch`'d proposed changes to my local machine to check behavior on some additional test cases. "If you fetch it you're fucked" is unacceptable as a security boundary.
(1) and (3) are just tu-quoque arguments about other attacks being worse. The relevant question isn't how bad other attacks are, it's how bad this attack is.
The fundamental problem with collisions is that software often assumes they can't happen (or is not tested against them). Thus collisions can trigger bugs, or otherwise cause surprising behavior. For example, webkit figured the colliding PDFs demonstrating a sha1 collision would be excellent for unit tests, so they merged the PDFs into their SVN repo... which completely fucked it [1]. I don't know the exact internals of git so I can't comment on how you would get surprising things to happen, but "oops the file you merged was different than the file you reviewed" and "oops the repository got corrupted" seem entirely plausible.
(2 counter) is impractical because all nodes of git will not replace objects if it thinks it already has it. So any attack has to assume this is the first time the node fetched, which is difficult before trust is established, which is difficult. This is part of the argument Linus originally outlined for this vector, which is that it only works for _very recent_ objects.
(1/3 counter) is not what I argued. I argued from the worst-case position that collision and preimages were theoretically cheap and fast. Even in that case, I feel my arguments hold.
The main issue here is that you assume you can replace an existing object with a replaced one, which you cannot. Not only that, but in all known cases, the sha1dc variant of SHA1 that Git uses will even _tell_ you that someone tried to do this, which singles out the source quickly.
It doesn't reject, but it will not replace. Same for a fetch/pull. That is another issue with this attack vector (that Linus also mentions) - it has to be the _first_ time that a node has seen this object. It makes the attack even more difficult than it already is (in like 4 different major ways)
I'm going to completely ignore the first part of this post because I'm not interested in arguing about how severe the issues with SHA-1 are. I think it's accepted that there are flaws.
So given that, I'm more interested in the arguments for why migrating to SHA-256 is problematic.
The biggest issue I see, after skimming over it, is the submodule breakage for new projects trying to link to old projects. This seems solvable frankly, but is the only serious issue I see. Everything else will be worked out as software is updated IMO.
A lot of replies here seem to be asserting that this "isn't that hard" without addressing the thing that makes it most hard: submodule compatibility and the breadth of tooling
They're the best way we have to reference other repositories from one repository. All other solutions don't have the benefit of being built in to git and having support built in to all git forges.
Ecosystems like Yocto are built around having meta layers as submodules. And, despite the usability flaws of submodules, it works really well.
I also use submodules to include dependencies into C++ projects a lot. It works fine.
git subtree and git subrepo are compatible with all git forges and don't require normal developers to install the extensions. Only the person/bot doing the occasional sync to the external repo has to install the extension. I prefer git subrepo for most (but not all) use cases.
Note that subtree and subrepo have the same SHA-1/SHA-256 incompatibility issue that submodules do, so this will be just as much of a trainwreck for them as well.
What's the advantage to using git subtree or git subrepo instead of git submodules? I've never heard of this, what's the difference between them? If it's an extension, how do people without the extensions end up downloading the code from the other repos?
How does it work with MRs, can I submit an MR which consists of changing the referenced SHA (and have it not show up as changes to every file in the referenced repo)?
They work by copying one repo inside another and providing tools to copy/sync it back out again. It's not a link, it's a copy. It's almost the same as copying the files into your repo and git add'ing them, but there are accounting and tools to pull changes from the subrepo back to the external repo.
The trade-offs are relatively obvious. It'd be a poor option for Yocto, but is a better option for most corporate repos.
Oh, I didn't want to vendor another repo into mine, I just want to store a reference to it. I'll keep using submodules then, as they're easier to work with than tools like gclient and repo.
I really don't get the hate. They're not hard to work with. Just a bit shitty UX but if you're using Git you're used to that already.
I won't defend submodules, but I also don't accept this as a response because it's irrelevant. They are used and it will be an unbearable pain when they break.
Someone started this FUD a long time ago and it has worked. Instead of using an elegant mechanism, project have built inelegant wrappers on top of git like go.mod which are actual mistakes.
It is the same people who say Git is unseasonably hard to learn. It isn't. Git and its command line is ugly and hard to remember. Understanding what those commands do at a commit and a branch level isn't really difficult. Even the internal object model is only moderately difficult.
I say this as a person who strongly dislikes many many aspects of Unix and Linux due to bad design and terrible UX. Git has a better design than any Unix program you get.
Submodules work okay. It is just Git LFS but for Git repos. Get over it.
The first reason the author lists for why this will be bad is only an "issue" on Git hosts that don't allow repo creation on push (which is brain dead of GitHub). Any other host, you push your new repo, and it will see the hashing algorithm, and receive the contents accordingly.
Submodules is a legitimate argument against this, though I don't know how widely this feature is actually used, and similar to the arguments in favor of switching the default branch from master to main, this is simply a setting which can be changed.
I do like the idea of commits having both hashes, and am surprised that idea has not been explored further.
Generally though, I think the author's strongest argument is simply that the change isn't strictly "needed", and all the other issues presented aren't the strongest arguments against change.
The problem is existing repositories have SHA-1 commit hashes, and if you also end up changing those...
There's lots of tools that refer to git commit IDs. Some of those tools may even hardcode a commit ID to be 40 hex digits long. The fact that these tools are external also means that "oh, just rewrite the commit messages or code to refer to the new IDs" isn't feasible. The only way to not break the world is to let people refer to existing commits with their SHA-1 hashes in perpetuity, and it doesn't sound like git is set up to allow this in any way, which means that existing repositories have to stay SHA-1 in perpetuity and that will cause fun down the line if you start having to make SHA-1 and SHA-256 repositories.
Changing from master to main is a one-off change. It might require changing your scripts once to refer to 'origin/main' instead of 'origin/master', but other than that, there is essentially nothing more that needs to be done, there is no risk to historical artifacts that needs to be mitigated.
Thanks for writing this. I'd only been loosely following it and I hadn't realised how bad this is going to be. I have repos with tens of submodules and it's going to be a nightmare if any of them switch to sha256 in place. Not to mention I won't be able to use any new projects unless I rebuild my repo and all the submodules therein.
I thought the master to main thing was bad enough but this is going to suck. And just like the master rename it achieves basically nothing.
What is it about these projects that attracts people who just want to change things for the sake of it? Real engineering means coming up with a solution for backwards compatibility. This is just irresponsible and, frankly, a fuck you to everyone who will be affected by this.
« people who just want to change things for the sake of it » — Not for the sake of it, but to “make the world a better place.” The intention is noble (well, mostly, at least let's assume it is). Of course, there is disregard for history and her deplorables (or not so deplorables), comes with being “progressive”. Which is why I like Windows better (not 11 though, will probably have to go back to Linux at some point).
« What is it about these projects » — Maybe that they're “at the forefront.”
The alternative to making sha256 the default is to leave sha1 the default. Nobody changes to sha256. sha1 is broken in 10 years. Suddenly everyone has to switch all at once on the same day because it is a critical security issue, but github never implemented sha256 because they didn't have to. This would be a major problem.
This is very very easy to fix if you run into it.
1. Adopt git 3.0 if you can with sha256.
2. If you can't use sha256, set the config to put things back to sha1. Wherever you need to do this you probably already set dozens of ENV vars or settings, just add a new one.
Or write a 15 page analysis about how the above is so hard people will probably just find it catastrophic to even think about.
> sha1 is broken in 10 years. Suddenly everyone has to switch all at once on the same day because it is a critical security issue
If you read the OP article, the entire point he's making is that this would never happen, because a hash algorithm being "broken" doesn't matter in practice, because true supply chain security has nothing to do with file hashes.
It's easy for _one person_ to fix. It's not easy for the entire git ecosystem as a whole. GitHub, large internal corporate git repos, CI/CD systems, projects with submodules, etc. The second half the article explains all of this.
It’s already broken, but even though it’s broken it’s hard to generate git collisions because of the repo metadata. It’s easy to generate (for instance) standalone PDFs with identical hashes, but doing this with git in a useful way is much harder.
That said, it’s still a good idea to migrate to a more robust hashing algorithm. Defense in depth, etc. Just because it’s a difficult migration doesn’t mean it shouldn’t be done.
You can certainly do this, as I said, this is Google's backup plan. But defaults matter. People will start running this and getting repos that are uselessly incompatible with other repos, tools, libraries and server instances. Having it as an option is one thing. Making it a default will cause a lot of pain for people who don't want to care about this.
No, my argument is that the change should not happen at all and nobody wants it and it gains the community very, very little but the default change is forcing it on everyone and most will be _entirely_ unaware - now having to solve problems that are difficult to understand. Defaults also matter when they are the wrong defaults.
Who is "you" in the context of a distributed version control system? I think this is not just the plural you, but the unbounded you -- it's all people who not just interact with your project now, but who you hope may interact with it in the future. The question is what the cost is of committing a near-infinite population to this migration, not the cost of doing a single `brew update` on your personal machine, no?
For the record, Y2K was not fud. It was very real, in a long list of datetime problems that are to come. Further datetime problems are coming at scheduled dates.
It was definitely FUD. There was a real problem (date counters would roll over), but the impacts of it were so ridiculously overstated that it eclipsed any sane discussion of the issue. We had people at the time predicting that planes would literally fall out of the sky when Y2k hit, which was never a realistic possibility.
That's the problem with deniers. When responsible persons take preemptive action to prevent tragedy, like with Y2K, the diners say it was FUD. When people don't take action, like with climate change, they say it wasn't important considering it's not them who's dead, totally discounting those who have suffered or died as a consequence. In summary, the deniers are so incompetent that they can't be trusted to correctly maintain a car, let alone civilization, considering they would never even the replace the necessary parts at the right schedules in their car.
They address this very theoretical. In short: Yes. Which makes sense if you don't treat the hash as a form of security against malice, especially in the case of attacks that are already impractical, which is the entire thrust of the article.
As linked by another commenter in this thread, Linus worked out years ago that even if someone inserted a malicious object into the kernel repo, it would at best be a nuisance and not a major concern.
This article is full of mistakes and misleading claims:
1) It's claiming SHA1 insecurity is theoretical, while SHAttered from 2017 was specifically a pratical proof of concept. The only reason Git wasn't affected, is because they didn't bother bruteforcing a git-blob prefix.
2) It's claiming collision attacks don't matter, only second-preimage attacks do. This is incorrect, collision attacks are enough for code-smuggling problems, when two repositories are on the same git commit (verified by the full commit hash), yet contain different code in their git checkout.
3) The Linus quote "The real security is in distribution" is arguing that "git's content-addressed system should not be used to address content". It's arguing that, in case of curl|sh, you shouldn't use a sha256sum-gate to pin the content to something you've reviewed, you should instead ensure curl is fetching from an https server.
1) I link to the SHAttered paper, as well as Shambles. Git projects were not affected because it is an inefficient attack vector. I say it's impractical to exploit, which I think everyone agrees with.
2) I specifically argue that even if both attacks were practical and cheap, it's still not the problem we should be focusing on.
3) Have you read this email (that I linked to)? It is almost the same general message (20 years ago) that this blog post is. It literally goes though a theoretical object replacement attack and how dumb this scenario is and so SHA-1 is fine.
https://lore.kernel.org/git/Pine.LNX.4.58.0504291221250.1890...
Basing your cryptographic advice on a 20 year old opinion-piece from somebody with no background in cryptography is not the flex you think it is.
> 1) I link to the SHAttered paper, as well as Shambles. Git projects were not affected because it is an inefficient attack vector. I say it's impractical to exploit, which I think everyone agrees with.
It seems unlikely it will stay that way forever. Typically attacks get more efficient over time as researchers find improvements, not to mention computers getting better.
In 2015 it was estimated to cost $100,000, now the estimate is down to $10,000. Where will it be in 2035?
I specifically argue that it doesn't matter if it's $1 and base my argument and solution around that. So it's irrelevant where it is in 2035.
You still mentioned the (1) part, which is what i object to. I agree its not fatal to your argument.
Even if it costs zero to create. How do You force people to pull from Your repo?
Hack into the system holding a trusted repository, and swap in your variant with same signature?
The problem of a SH1 collision happening by coincidence is vanishingly low and theoretical.
Nothing else matters.
Git hashes are not supposed to be a security mechanism. If your basis for trusting that you have the right checkout is the git hash, in a situation where you have legitimate concern about untrusted parties manipulating remote repositories, then you're simply wrong.
Sorry, no.
When I check out code from a git repository in a pipeline using a git hash, I expect the code to be exactly what has been reviewed by me under that hash.
Everything else would just be a crazy invitation to make supply chain attacks uncircumventable.
And so if you don't trust the server that is hosted on or the security of the transport mechanism like TLS/SSL, such that the content may be manipulated by adversaries, you think that git hashes are good enough?
Well, what about someone who is fetching the commit from that server for the first time and has nothing to compare the hash against?
Oh, that would never be a problem for widely disseminated, popular, open source project, so it doesn't matter.
Dealing with potentially-hostile hosts is quite common, actually. See for example how most Linux mirrors work, or Subresource Integrity with HTML.
Turns out securing a service to transfer a single hash is a lot easier than securing a service to transfer gigabytes of data.
Even if I don't fully trust Github, it is still incredibly convenient to be able to upload my code there and then send someone an email telling them to fetch commit `123abc` from some repo link. As long as my email isn't compromised, that should be secure.
> Git hashes are not supposed to be a security mechanism
Probably a naive question, but why not kill two birds with one stone if it can be done for a reasonable cost?
Because you're not killling two birds; you're not killing the security bird with a better content hash.
A SHA-256 sum, though very good, only assures you with great confidence that you're looking at the same thing you looked at before, or that someone else is looking at elsewhere.
It is not a digital signature, and we don't want digital signatures to serve the role of content hashes.
Speaking of signatures, we have support for them in Git; you can use gpg to sign commits, and set it up to be done automatically.
Nobody is going to fake your commit such that the fake has the same SH-1 hash and your GPG signature.
The worry there is that the key holder (whether the legitimate one, or a malicious party who got a hold of the key) somehow does this: creates a new commit, signed with their key, which somehow has the same SH-1 as an existing signed commit. The git hash includes the GPG signature, so there is a significant layer of difficulty there which is likely harder than faking an unsigned SHA-256 commit.
Please educate yourself what a merkle tree is. It's a well understood building block of various security systems, including certificate transparency (which explicitly uses sha256).
You refer to PGP signed Git objects, but you also argue:
> Git hashes are not supposed to be a security mechanism
Guess what the Git PGP signature is signing.
This is exactly right. A signature is only worth as much as the hash that it’s signing. And all the usual signature algorithms are signing a hash.
The GPG signature is not signing the git hash, if that's what you mean.
The GPG signature signs some kind of hash calculated over the commit, minus the GPG header, which is thereby added.
The git hash is then calculated over the whole thing. The git hash is on the outside, and not part of the signing.
> The GPG signature is not signing the git hash, if that's what you mean.
It kind of is - it’s signing the hash of the tree object, which is the actual thing that you’d attack with a hash collision
I understand that if we sign a commit with the help of some arbitrarily strong hash, it doesn't protect the parent commit(s). The integrity of the SHA-1 hash references to the parent commits is not in question, but the authenticity of those commits themselves.
No, not the abstract tree formed by a series of commits.
The actual git ‘tree’ object, which is the thing a commit actually points to, referenced by a hash in the commit. That is signed by the GPG signature.
That doesn't make a difference: with sha1 a malicious change in content will still result in the same content hash, so the signature will still be valid, and the commit hash will still be the same.
Only if the GPG signing process stupidly relies on the SHA-1 hash. I.e. if it takes an unsigned commit and signs only its SHA-1 hash and then creates a new commit with GPG headers. If that's how it works, that is massively stupid and can be fixed without forcing SHA-256 as a git hash. Just have the signing calculate its own digest for its own purposes.
That digest can be the SHA-256; since the infrastructure is there for it, signing should use SHA-256 regardless of what hash is used by the repository for identifying and linking content.
> Git hashes are not supposed to be a security mechanism.
Commit signing indicates otherwise.
One of my favorite fun facts about Fossil SCM (another source control by the devs of sqlite) is that they patched their use of SHA1 6 days after the shattered attack was published:
"Both Fossil and Git started out using only SHA1 hashes. But when the SHAttered attack against SHA1 was published on 2017-02-23, the need to migrate to a stronger hash algorithm was recognized. Fossil added the ability to use SHA3-256 as an alternative on 2017-03-01 (six days after the SHAttered attack was first published). SHA3-256 is now the default for all new repositories and check-ins in Fossil, though older check-ins that occurred prior to SHAttered can still use their original SHA1 hash. Hence, no repositories had to be rebuilt and no hyperlinks were broken."
https://fossil-scm.org/home/doc/trunk/www/hundredandone.md
To me it's so interesting watching in realtime Git is still battling with this decision and for Fossil it was just another week of development.
That whole page is fun to read. Another fun fact somewhere else in the docs is that Fossil uses a grow-only set to store commits. They came up with this scheme some years before it was formalized by CRDTs!
That's impressive. I suppose they had a more flexible architecture to make that change so fast.
Is there any writeup on why it was easy for them and not for git?
My guess is that it's less about the architecture and more about the blast radius and the number of users
Hmm probably nothing architecture wise. It's probably just the fact that fossil is developed by way fewer devs.
From the skim I read of this article it seems both projects arrived at the same solution: support both but make SHA-256 the default.
I mean, there are two things here. One is how difficult it is to have a different hashing mechanism. Brian and other heroes in the Git core group have done amazing work to make this _technically_ possible on a repo level. To test some of my theories, I trivially implemented MD5 and an insanely dumb and easily breakable hash backend. It's not _hard_ to change the mechanism now. It's about the community.
Fossil isn't difficult to change not because it's technically harder for Git but because Git has a community and ecosystem that Fossil does not. The cost is not in the individual project for Git, the cost is because there is _so much_ in Git and this bifurcates everything.
Also, interestingly, Git today does _not_ use a straight SHA1 because of these attacks. It uses `sha1dc`, a slower collision detecting variant that specifically checks for this vector of attacks. So currently, Git's SHA-1 variant is not susceptible to the SHAttered/Shambles attacks.
Agreed! This isn't a tech dig at all.
To me it's more of a reality of creating a tool with a huge active community and a community of contributors and creating a tool with a small team and small community.
It's easier to make world breaking changes when the world is really small. If git could magically just get everything and everyone to cut over and use git 3.0 in a magic instant, it wouldn't be having this problem.
Here's a stoichiometric bird for you:
:%s/git 3\.0/python 3.0/g
Or IPv6 or windows 11.
:x
Neovim's lazyvim plugin sucks because it takes over H C and L.
Linus Torvalds in 2007:
> but the point is the SHA-1, as far as Git is concerned, isn't even a security feature. It's purely a consistency check. The security parts are elsewhere, so a lot of people assume that since Git uses SHA-1 and SHA-1 is used for cryptographically secure stuff, they think that, Okay, it's a huge security feature. It has nothing at all to do with security, it's just the best hash you can get. ... [1]
[1] https://www.youtube.com/watch?v=4XpnKHJAok8&t=56m20s
So Torvalds used SHA-1 purely because he needed a hash function with no other property than identifying content.
Exactly. But that's why I think SHA was a mistake. He should have gone with something like murmur to avoid all this frothing at the mouth.
Linus, in 2005, couldn’t have gone for murmur, from 2008.
Was there “something like murmur” in 2005 that’s cryptographically better than SHA1?
Yeah, in 2005, SHA-1 was just about the best you can do given the constraints of the time (without picking something much more esoteric, much slower, etc.). Using SHA-256 at that time would have been noticeably slower on the computers of the time, and made repo metadata take up a lot more space.
It was a mistake to assume a fixed algorithm in the repository format and client-server protocol. I remember being surprised when I learned about that choice, being familiar with cryptographic protocols and formats where the hash algorithm is usually a parameter that can vary for each concrete hash.
> It was a mistake to assume a fixed algorithm in the repository format and client-server protocol.
See also perhaps Wireguard, which touts itself as not having "cryptographic agility" because they wanted to avoid all (perceived) problems and complications of IPsec. But now that PQC is (allegedly) approaching there's no easy to update things because (AIUI) there's no negotiation possible in the protocol; you're basically standing up a 'Wireguard 2.0' that runs separately than the original.
If that was true, i doubt git would have switched to using the slower version of sha-1 that detects attacks.
Frothers gonna froth, though.
I don't understand why Git is not making the SHA-1 and SHA-256 modes far more compatible with each other.
SHA1-hashed objects should be able to refer to SHA-256-hashed objects, although this seems somewhat pointless.
But SHA-256-hashed objects should also be able to refer to SHA1-hashed objects, with a major caveat: if those objects themselves are part of a collision pair, then there is a genuine problem. But this is avoidable! Suppose that Linux decided to migrate to SHA-256. The upstream project could choose a pair of dates, say January 1 2027 and March 1 2027. Up to the first date, maintainers would be welcome to submit hashes of objects that are not yet in the repo but that they think they might submit later on, and, on that date, the upstream tree would finalize the list of these objects and reference it in the repo (with a new mechanism for this purpose). Effective the second date, the repo would start publishing SHA-256 commits and would never again accept a SHA1-hashed object that was not in the repo at the cutoff date or referenced as part of the Jan 1 block.
And now it would be impossible to get a new SHA1 collision in to the repo.
The only new git features needed would be:
a) actual compatibility so that a SHA-256-hashed object could reference a SHA1-hashed object
b) a new object type that's a list of allowed SHA1 hashes (or probably a tree of them) that is itself hashed with SHA-256 and a mechanism to link to one of these from a commit
c) a policy mechanism to set a repo to only allow SHA1-hashed-objects that a reachable from a preconfigured SHA-256-hashed commit
Emily's talk does a pretty good job of summarizing the issues with intermixing the hashes: https://youtu.be/eJJp0RE7cd4
I skimmed the video, and I didn't quite catch that. Near the end of the video, however, she did mention that interop is in the works[0].
In any case, even if Git 3.0 were completely incompatible, it would suck, but it's not the end of the world. You just treat it as if you were migrating from one SCM system to another. CVS -> SVN -> Perforce -> Git -> Git 3.0 -> [...] been-there-done-that. This is something that both open-source and commercial projects have had to deal with over the years.
Or maybe it would be a repeat of Python 2.x -> 3.x. ¯\_(ツ)_/¯ With AI assistance, hopefully porting the tooling over may go a lot quicker and smoother.
[0] https://www.youtube.com/watch?v=eJJp0RE7cd4&t=1134s
Her entire section 2 (starting at 6:12) is basically about why git does not and will never allow mixing of SHA1 and SHA256.
The interop discussed is using copybara as a copy tool to move data from SHA1 based repos to SHA256 based repos and vice versa.
Thanks. I went back and re-watched that part, and her point's that "[a] tree's cryptographic strength is equal to the weakest hash algorithm anywhere in the tree," and that's fair. I also understand why Git 3.0 may not want to give user the choice, though I would still rather it be given.
> "[a] tree's cryptographic strength is equal to the weakest hash algorithm anywhere in the tree"
This is simply wrong IMO, for two reasons:
1. The attack on SHA-1 is a collision attack. Once you have frozen a hash, you cannot attack it with existing cryptanalysis. If there were a preimage attack it would be a different story.
2. Even if there were preimage attacks, one could freeze a mapping from SHA1 hash to SHA-256 hash.
In fact, #2 seems like en excellent design. Objects could reference such a mapping, and a repo could disallow conflicting mappings (the mappings would only be accepted if the mapped objects are reachable from the mapping and the mapping is correct).
They do give the user the choice, but the default is changing. My point is not necessarily to rip out the SHA-256 option, but simply to not make it the default. Because then people will create repos in that format that do not understand the ramifications, where the opposite should be true.
My point is not that it's the end of the world (or the end of Git), but that it will be painful and unclear and confusing to lots of people. That would be fine if it made a huge difference in trust or protection, but it's the wrong way to do that.
Hm but the date is stored inside of the commit. The only way we can know that a commit's date is authentic... is through its hash. If I can forge commits with any SHA1 hash at will, I can make a repository whose head commit has the same SHA1 as the one in torvalds: /linux but where any commit was replaced by a malicious commit with the same SHA1 and a fake date. You have no way to detect that my repo is inauthentic other than through a deep history comparison. The whole idea behind a merkle tree is that just checking the hash of the top is sufficient to know the identity of the whole tree.
I don't know what the solution is, but I'm inclined to believe that any repo with a single SHA1 commit is as weak as a repo with all SHA1 commits.
The date that a repo receives a commit is known to that repo. And a repo can stop accepting new SHA1 objects. And a SHA256 object could have a flag that says that no SHA1 objects may ever reference it.
The design of Git, as a Merkle tree, is meant to allow for use cases like this:
* I host a mirror of the Linux git repo.
* You download Linux from my mirror.
* You check out a commit, say fd179f8a05be3ccae366b9b96e176b51fbe54aab, which you know is a genuine commit through some out-of-band mechanism (mailing list, GitHub web interface, a line in a Nix file, whatever).
* You check whether the repository I gave you is legitimate or not by re-computing the hash of the commit which I claimed was fd179f8a05be3ccae366b9b96e176b51fbe54aab. If it comes out to be fd179f8a05be3ccae366b9b96e176b51fbe54aab, you know it's legitimate. If it doesn't, you know it's fake.
This is a completely normal use of Git. People download from mirrors all the time. People rely on commit hashes to identify a specific source tree. People trust that if whatever the mirror gave them hashes to the right value, it's genuine. That way, you don't have to trust the mirror.
If I can forge my own commits to have any hash I want, this whole model breaks down. I can replace some old commit in the repo with my own forged commit with the same hash, and when you download a copy of the Linux repo from my mirror, you'll receive a repo with malicious content, but it'll hash to the same fd179f8a05be3ccae366b9b96e176b51fbe54aab hash as a genuine repo would. This breaks the security model of Git.
> You check out a commit, say fd179f8a05be3ccae366b9b96e176b51fbe54aab, which you know is a genuine commit through some out-of-band mechanism
That's a 160 bit hash, which is SHA-1, which has the security properties of SHA-1.
Suppose you check out a commit with a given SHA-256 hash. That commit object represent the root of a tree where all the edges are hashes (and types, etc). I'm suggesting one of two designs:
a) (Simpler but weaker) If Linus has published that commit, then he is confident that he hasn't pulled in any too-new SHA-1 hashes and that there are no collisions present in what he thinks the tree is. So, by induction on the traversal depth, there is only one actual object identified by each edge, and those objects contain the hashes of their child edges, so those hashes are all correct.
This breaks if there is a malicious collision already in the tree.
b) (Stronger but higher overhead and more complex) There would be an object or objects, discoverable from the root by following only SHA-256 edges, that encode a duplicate-free mapping from SHA-1 hash to SHA-256 hash. The client finds and parses that and then, as it traverses the tree, each time it reads a SHA-1 hash, it computes the SHA-1 and SHA-256 hash of the referenced object, verifies that the pair is in the mapping and also verifies that the SHA-1 hash matches what the edge requires.
I think that (b) is genuinely cryptographically secure in the sense that, if you can construct a commit that has the same SHA-256 hash as an official upstream commit but different contents, then there is necessarily a SHA-256 collision.
For A), I don't understand what the point is? I never mentioned what Linus is confident about, I talked about what you can verify when you pull from my mirror. I could replace a commit from 2010 with a malicious one
For B), I would think this could work, but it's a completely different solution from what you proposed and what I responded to.
> I could replace a commit from 2010 with a malicious one
How? Remember, there are (currently, anyway) no known SHA-1 preimage attacks.
This means, if you migrate your repo, every single commit message that contains text like: "please see commit <sha1>" will now be broken.
This will be a train wreck. I hope they don't release before adding compatibility modes to keep the existing sha1's around in the database.
Tools like git-filter-repo[1] support rewriting commit hashes in commit messages. git-filter-repo actually does it by default; see `--preserve-commit-hashes` in the manual[2].
[1]: https://github.com/newren/git-filter-repo
[2]: https://htmlpreview.github.io/?https://github.com/newren/git...
I need git-filter-repo to rewrite entire documentation and also resurrect and rehire earlier employees to repeat their GPG signatures.
Sure, but that's not going to rewrite Slack messages, emails, GitHub links, docs
Came to say something exactly like this... The old sha1 handle needs to be still available in the same way an HTTP 301 redirect would work.
... do not migrate old repos? I'm not sure why people would do that. Or, if they do, why would they replace the current repo name instead of creating a different one and keeping the old one closed to make the references work.
I don't think this is going to be a problem at all.
Hmm. If this is a security issue that matters, would you not be expected to migrate?
Not talking about archived code, but active projects that pre-date git 3.0
Once this starts being actual pain, we will each vibe the replacement index creator (git already supports replacement objects), for back-forth conversion, populated on pack and object indexing.
For massive perf and mem use damage. But oh well. And then we will wait for official version
There are plans to keep sha1s around in a database, but as far as I know, no way to transmit those, so they seem specific to individual forges. They can be recomputed, sure, but again, any signatures break and it's possible that in the case of an actual replacement, the recomputation is now wrong and not easily comparable. So what is the point?
This is not about forges, it is about the repo I have in a folder on my computer. The sha1 hashes shouldn't go away. Yes the forges also need to support this.
Since SHA-1 is already broken (just expensive in terms of GPU-time), then the text "please see commit <sha1>" is also already broken.
You can't attack an existing normal commit.
But also collisions there aren't a big deal. People will cite short hashes when referring to things and that's not "broken".
If it's an existing commit and you're already converting the repo, you can just convert the commit messages as well.
That is essentially only a second preimage problem, which is basically impossible.
From what I'd read, SHA256 in git is showing every sign of being another IPv6. In particular:
- It's implemented in a non-backwards-compatible way
- The benefits over the older model are a bit nebulous
- There's a large amount of tooling that needs to catch up, and little sign that there is movement there
The difference with IPv6 adoption is that the internet relies heavily on network effects: so long as some hosts only have an IPv4 address, you need an IPv4 address for full connectivity, but then if everyone has an IPv4 address anyway, there is no immediate need to migrate to IPv6.
(Yes us Hacker News users have plenty of use cases for IPv6, like self-hosting and peer-to-peer networking and so on; we are not the average user.)
This effect doesn't exist for the Git migration. Each repo can be updated independently; it doesn't affect users of other repositories, and most likely, the majority of devs will work on some SHA-1 repos and some SHA-256 repos with no issue.
If anything, I would compare it with the Python 2 to Python 3 migration, which was also painful, but succeeded eventually (despite being much less necessary in the first place).
changing repos to the new IDs would break any existing links to content on the pre-migration repos.
Unless you link using tags.
Which is considered a Really Bad Idea because tags aren't immutable, so there's absolutely zero guarantee that it'll point to the same commit a few months from now.
The GitHub Actions ecosystem found out the hard way, through some rather high-profile compromises. They hotfixed it by adding "immutable tags" to their platform, and are now working on adding a lockfile to... easily reference a commit hash.
> - The benefits over the older model are a bit nebulous
This is far from the case with IPv6!
Is it? There are many benefits in principle to IPv6, but if my ISP continues to assign me a single dynamic IP, those benefits are entirely moot for me.
If your ISP is assigning you a single IPv6 address, they are doing IPv6 wrong. You should be getting your own /56.
Every time you use the word "should" you're fighting reality.
Plenty of ISPs do it right (examples: my last 3). That the parent poster's apparently doesn't isn't an argument against the technology their ISP fails to deploy correctly.
Reality is that lots of ISPs do this right. And have for a very long time.
Even a /64, while wrong, would be a significant improvement over IPv4.
Your ISP assigns you a single dynamic /128? Which ISP is that?
My ISP (Bell Aliant) doesn't support IPv6 at all. Rogers though, for example, assigns you a /56 block but it's dynamic.
Is the argument that there are no benefits to IPv6 for anyone/society because your specific ISP messes it up?
Of course not. The argument is that the benefits are nebulous.
... if you use ISPs that don't deploy the technology correctly. Faced with multiple providers of a technology, are we to judge its benefits by the worst ones?
Right, but just because you don't happen to have IPv6 right now, how does that remove the benefits for others to have IPv6? That's like saying having a faster CPU wouldn't mean faster performance, because I don't have that CPU yet.
There's a massive push right now from top down to have secure software supply chains. Google SBOM and SigStore. It's not an organic need but if you have government customers you don't have many options.
Ironically rewriting git history is a perfect opportunity for a supply chain attack.
I thought this would be a snark but it's an extremely well put together argument against the "Hashmageddon".
If you're replacing the weakness of SHA-1 just by going to another algorithm, you better be prepared to go to the next one when sha256 collisions happen, and it doesn't sound like git's design would be easy to modify for this type of crypto agility.
I do like their proposal for using signatures to establish trust and allow swapping sha256 for whatever comes next.
Technically, git's design (thanks to very smart people trying to solve this problem like brian and others) is _very_ easy to modify to different hashing algorithms now. A lot of amazing work has gone into this in recent years.
However, it's not a git problem. It's an ecosystem problem. It's that every git repo has to choose one and they're entirely incompatible with each other. That is the cost and the difficulty.
I think there is a question though when that will happen and if it will be in our lifetime. SHA-1 started showing weakness in 2005 (collision in 2^69 instead of expected 2^80. This was later brought down to 2^61 in 2011), the same year git was invented. Nobody has found a similar weakness in SHA-256 as of yet. SHA-256 is still at its design strength of 2^128
It took 20 years to go from vulnerability in sha-1 to having to replace it out of caution. There is no such vuln in sha-256 yet. It could easily be 25 years before we find one, and another 25 years before we have to do something about it. Perhaps longer. Will git still be used 50 years from now?
With the kind of compute power available nowadays and AI models I wouldn't be surprised we see it much sooner.
All it takes is just one collision to consider it broken right?
But hey maybe the attempt to fix it makes git controversial enough it falls out of favor, and nobody uses it anymore in 2 years, problem solved? sure.
> All it takes is just one collision to consider it broken right?
No, its considered broken before that stage. i.e. when someone discovers an attack that would allow someone to create a collision faster than they should while still being impractical.
> With the kind of compute power available nowadays and AI models I wouldn't be surprised we see it much sooner.
Computer power doesn't super matter, what matters is algorithmic breakthroughs. So far i dont think there are any examples of major breakthroughs of that type via AI, although perhaps i am just misinformed. Its still early in the AI revolution, it might still happen, but as it stands i don't think there is any reason to worry about that.
Perhaps not clear enough, but Scott was a cofounder of GitHub, so he knows a thing or two about git in the real world =)
Being a founder of GitHub doesn't make my opinion more interesting. I hope the argument stands no matter who wrote it. :)
It does though, even if it shouldn't be blindly taken as gospel. Arguments help, but some folks have a better brand of apple box to stand on, and "I live and breathe git" helps quite a bit ;)
So they’ve been talking about this for many years, planning, and finally announce when they’re going to switch the default.
So this is the right time to post that everything they’re doing is wrong? Did you engage in all the discussions about it and how best to handle it? Whether SHA-256 was the best solution?
I don’t see anywhere that it talks about alternate proposals or why they might have been better. Why the particular suggestions here were rejected.
This seems like a bunch of Monday morning quarterbacking.
I do mention this in like the first paragraph. I don't feel great about it, but I've listened to these issues for years now during contributor summits and Git Merge talks and while it's always seemed problematic, I thought they would come up with a good solution. This last Git Merge confirmed that it's close to the switch and not in any way solved or improved. I don't want to just go with it for groupthink reasons. I never thought it was a good idea and I have said that, but we have a last chance to rethink this, so I'm curious if I'm alone or in the silent majority.
Your argument is persuasive and well illustrated. I think the problem is the intro paragraphs come off as too certain of catastrophe which, when juxtaposed with your claim that "smarter people than me have been working on this", makes it sound like you don't actually believe they're smarter than you. The rest of your essay feels fair and not judgmental.
I do believe they're smarter than me, but sometimes very smart groups talk themselves into ultimately impractical solutions because they're all smart. Sometimes you need a dumb guy to come in and say "are you sure this is right?"
I believe you are sincere. But "is about to be a huge, costly, global train wreck" lacks the nuance of "are you sure this is right?" and will rub some people the wrong way.
Edit: I'm not suggesting you should have written it any differently. I think you made the right choice to be a bit provocative because it grabs the attention that's needed.
I do believe this. But that doesn't mean I can't be convinced otherwise by a good argument. The point of this post is to see if anyone has a great counterargument to change my mind.
The plans have been on display in a cellar. Beware of the leopard.
This isn't really applicable. The plans have been talked about for a while very publicly.
I don’t understand what this is supposed to mean.
It's a hitchhiker guide to the galaxy reference, where sure, something is technically available but not clearly published and there are hoops even for those who know what they're looking for.
(No clue if it's applicable here, I'm not aware of this case, but I believe that's the reference if it helps :)
Edit : exact quote, as Arthur's house is about to be demolished for a highway bypass:
"But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.
It's not at all applicable.
I believe it's a reference to the hitchhikers guide to the galaxy – where the plans to remove the protagonists building to build a bypass road was hidden in this way.
nofunsir is a stoichastic parrot, matching to a bit in Hitchhikers Guide to the Galaxy, wherin the protagonist should have known to protest a plan to demolish his home where plans where clearly documented in a hard to find place that they could not have known about. It is not a good pattern match, because git has been discussing this in public on documented mailing lists for years.
No I'm not. Real thumbs here. If anything this paragraph sounds like LLM slop
Run it through pangram if you want, I wrote it with my human brain. What I did not do, however, was parrot an inapplicable quote.
In what way has git's discussion of their move been hidden away in a metaphorical basement?
I'm sure the cellar was also documented, Arthur eventually found it, did he not?
Mailing lists are basements in 2026
Since 1999, we've had Google to help people find things. True, a mailing list isn't an Instagram reel delivered directly to your face with audio and blasted out on Fox News, but if one was interested in the development of git, an LLM or a Google search would readily tell you about the existence of those mailing lists.
A number of years ago when I heard about this, I was pretty angry and made a private fork of git immediately in which I tried to scrub away the SHA-256 bullshit. But that's basically just paddling upstream with a spoon for a oar.
The stewards of Git are going to do whatever they want, and there is nothing you can do about it if you don't have the clout to create a fork that takes the lead.
No amount of discussion will do anything because they've already decided that their view of the situation is correct. Git hashes are not just content identification but a digital certificate mechanism, and their collision resistance is a grave issue that must be fixed, the end.
You will be browbeaten in any discussion; it's not worth the energy in a world replete with issues.
>it will be an incomprehensibly expensive and ultimately valueless and avoidable global nightmare.
thought "costly" in the title and "incomprehensibly expensive" in the subheader meant this piece would discuss how much less performant sha-256 is on modern machines, but didn't see anything. isn't there hardware acceleration? how much worse is it?
Actually, I think sha-256 is possibly faster than the sha1dc variant that Git currently uses.
I just sent a patch series to the list that enables sha1dc to be accelerated on modern CPU architectures to close to normal SHA1 speeds, but since it was ported from a Rust project by an agent, it will never be applied.
https://lore.kernel.org/git/20260929112544.86511-1-scott@git...
He means costly in terms of human effort and wasted time.
Yes every repo is either one or the other but you fix that by rehashing the entire repo. Everyone can do this independently. It's entirely possible to maintain to identical repos in SHA1 and SHA256 mode but for the most part I suspect once updated people will simply pull down the new repo and use git 3.0 as a required version.
As migrations go, it's reading as simple to me. You'll just have to backpoint the commit signatures. I must assume there's a backwards compatible reference for them in git 3, right?
Or drop them and reference the old structure in a dire pinch.
Re-hash the entire repo as in rewriting all history? Hooo boy will that be a mess, I deal with things which reverence commits by hash in repos all the damn time. There are thousands of them in every Yocto project!
Yes, this would cause big issues for Nix based build systems or any others that reference commits by hash.
Do you have any external references to any commits that matter, for example in your communication platforms (emails, Slack) or your bug tracker? Or, worse yet, in places where they aren't just text format references, but used for things like CI/CD caching decisions or security scans?
Once you rehash the entire repo, every single one of those external references will be broken. Because no, there's no support for looking up old hash -> new hash or the reverse.
"The migration to the new format is simple; Just re-write everything in the new format, but also keep the old format around forever too since data is lost in the new format!"
I positively don't care about the collision issue.
If you need to certify the authenticity of some code, and you've decided that a Git hash of any kind is going to be your certificate, you have a problem between keyboard and chair which is not fixable by stronger hashes in Git.
I don't want instability and churn in tooling.
Can someone more cyber-pilled than me explain what the actual risk with Git hashes being susceptible to collision attacks is? Obviously accidental collisions are problematic, but to my understanding the probability of that is still approximately zero.
Best I can tell, all a forced collision would do is let someone who already has control of a repo modify the history in a far from plausibly deniable way. Which in practical terms, they already could do simply by replacing the whole thing, because who's out here using git hashes as a security tool? Every pinning I've ever seen has been to tags (which can be modified at will), or hashes of the actual payload (which doesn't need to be the same as what git uses).
I see commit hashes used all the time, like in yocto recipes for example
Oh, agreed this sounds like a terrible migration path and shouldn't really be needed in the first place.
What I'm missing in the article is whether any Git server accepts replacing a SHA-1 identified object it already has. If it doesn't, then the distribution trust discussed holds, and keeping SHA-1 seems fine. Adding additional signatures seems fine for those who need transitive trust.
Ugh, I didn't know that SHA-1 submodules wouldn't be supported in SHA-256 repos. That changes the transition from painless to a major dumpster fire. Having to maintain converted forks, and use different hashes from upstream is going to be a mess.
git submodules has always been a major dumpster fire. You’re honestly better avoiding regardless of SHA-256 incompatibilities
Make git init use SHA-256 if git is invoked as git3, SHA1 if invoked as git2.
Plain git init could fail with a diagnostic: informing to use one of the two aliases or an option.
What people don't want is making git repos SHA-256 by accident and finding out later that they made repos not compatible with older git.
just make SHA1 the default in all cases unless the user specifies otherwise
I agree, but you're not going to sell that argument to a herd which has decided that git hashes are digital certificates which must be replaced with SHA-256, or the sky will fall.
I don't agree that SHA-1 is much longer feasible for git.
But I also don't think that switching to SHA-256 must be painful. A git2->git3 converted repo could just store all the past hashes, so existing links don't break.
schacon: Really like the "Independent Tree Hash Headers" idea. How difficult would this be to get this functionality into git? Would it cause any breaking changes with older versions? Have you discussed this with any git devs to see if they are open to adding it?
Actually, this entire blog post came out of a short chat at Git Merge a few weeks ago with Jeff King. I argued more or less this and he didn't _entirely_ disagree, though he has good counterarguments on the list over the last few years, so I don't really know how he thinks about it ultimately.
I would write this to the mailing list, but I thought a conversation that includes people outside that list is more interesting to me. Ultimately I'm not sure if I'm dumb about this or the whistle blower that's willing to actually say "maybe this isn't the right call"
Also, functionally, this is incredibly easy to add to Git.
> We can go through years of this SHA-1 to SHA-256 migration and then quantum computers break 256 and we're back in the same stupid boat again.
SHA-256 is considered quantum safe by the NIST and is left out of PQC migration guidance entirely.
It was theoretical - the point was that maybe some paper is published or some new tech or issue comes up. Now we have to do this again. If we separate the concerns, then we don't have to deal with both as though they're one problem. We can deal with one thing for content addressing and another for trust and security.
Until it isn't.
Things like this have a tendency to to be revised as time passes.
For now. Turns out the pigeon hole principle still holds.
1x10^77 is a lot of pigeon holes.
Yep. So was 1.46 x 10^48
The post's argument that hash collisions are irrelevant in practice is not convincing at all. Basically they amount to:
1. Collisions aren't as bad as preimage attacks
2. Even if you made a file-with-malicious-hash, how would you get people to pull it?
3. Other attacks are a bigger problem (social engineering)
(2) is laughable in a world with github. It's common for unknown people to submit pull requests to code bases, and for those changes to be reviewed and merged. For example, as part of reviewing pull requests, I have `git fetch`'d proposed changes to my local machine to check behavior on some additional test cases. "If you fetch it you're fucked" is unacceptable as a security boundary.
(1) and (3) are just tu-quoque arguments about other attacks being worse. The relevant question isn't how bad other attacks are, it's how bad this attack is.
The fundamental problem with collisions is that software often assumes they can't happen (or is not tested against them). Thus collisions can trigger bugs, or otherwise cause surprising behavior. For example, webkit figured the colliding PDFs demonstrating a sha1 collision would be excellent for unit tests, so they merged the PDFs into their SVN repo... which completely fucked it [1]. I don't know the exact internals of git so I can't comment on how you would get surprising things to happen, but "oops the file you merged was different than the file you reviewed" and "oops the repository got corrupted" seem entirely plausible.
[1]: https://www.reddit.com/r/programming/comments/5vyhy2/webkit_...
(2 counter) is impractical because all nodes of git will not replace objects if it thinks it already has it. So any attack has to assume this is the first time the node fetched, which is difficult before trust is established, which is difficult. This is part of the argument Linus originally outlined for this vector, which is that it only works for _very recent_ objects.
(1/3 counter) is not what I argued. I argued from the worst-case position that collision and preimages were theoretically cheap and fast. Even in that case, I feel my arguments hold.
The main issue here is that you assume you can replace an existing object with a replaced one, which you cannot. Not only that, but in all known cases, the sha1dc variant of SHA1 that Git uses will even _tell_ you that someone tried to do this, which singles out the source quickly.
couldn't github reject a push that contains an existing hash in the repo?
It doesn't reject, but it will not replace. Same for a fetch/pull. That is another issue with this attack vector (that Linus also mentions) - it has to be the _first_ time that a node has seen this object. It makes the attack even more difficult than it already is (in like 4 different major ways)
My prediction: 20 years from now everyone will still use SHA-1 git. That will be simply easier.
Do you want to bet on that prediction
I'm going to completely ignore the first part of this post because I'm not interested in arguing about how severe the issues with SHA-1 are. I think it's accepted that there are flaws.
So given that, I'm more interested in the arguments for why migrating to SHA-256 is problematic.
The biggest issue I see, after skimming over it, is the submodule breakage for new projects trying to link to old projects. This seems solvable frankly, but is the only serious issue I see. Everything else will be worked out as software is updated IMO.
I run a small git/jj forge and for us it's already painful dealing with this. Can't imagine how GitHub is going to handle it.
The xz backdoor is basically his point in practice — that was a maintainer-trust compromise, not a hash collision.
A lot of replies here seem to be asserting that this "isn't that hard" without addressing the thing that makes it most hard: submodule compatibility and the breadth of tooling
Submodules are a mistake.
They're the best way we have to reference other repositories from one repository. All other solutions don't have the benefit of being built in to git and having support built in to all git forges.
Ecosystems like Yocto are built around having meta layers as submodules. And, despite the usability flaws of submodules, it works really well.
I also use submodules to include dependencies into C++ projects a lot. It works fine.
git subtree and git subrepo are compatible with all git forges and don't require normal developers to install the extensions. Only the person/bot doing the occasional sync to the external repo has to install the extension. I prefer git subrepo for most (but not all) use cases.
Note that subtree and subrepo have the same SHA-1/SHA-256 incompatibility issue that submodules do, so this will be just as much of a trainwreck for them as well.
What's the advantage to using git subtree or git subrepo instead of git submodules? I've never heard of this, what's the difference between them? If it's an extension, how do people without the extensions end up downloading the code from the other repos?
How does it work with MRs, can I submit an MR which consists of changing the referenced SHA (and have it not show up as changes to every file in the referenced repo)?
They work by copying one repo inside another and providing tools to copy/sync it back out again. It's not a link, it's a copy. It's almost the same as copying the files into your repo and git add'ing them, but there are accounting and tools to pull changes from the subrepo back to the external repo.
The trade-offs are relatively obvious. It'd be a poor option for Yocto, but is a better option for most corporate repos.
Oh, I didn't want to vendor another repo into mine, I just want to store a reference to it. I'll keep using submodules then, as they're easier to work with than tools like gclient and repo.
I really don't get the hate. They're not hard to work with. Just a bit shitty UX but if you're using Git you're used to that already.
I won't defend submodules, but I also don't accept this as a response because it's irrelevant. They are used and it will be an unbearable pain when they break.
> Submodules are a mistake.
Someone started this FUD a long time ago and it has worked. Instead of using an elegant mechanism, project have built inelegant wrappers on top of git like go.mod which are actual mistakes.
It is the same people who say Git is unseasonably hard to learn. It isn't. Git and its command line is ugly and hard to remember. Understanding what those commands do at a commit and a branch level isn't really difficult. Even the internal object model is only moderately difficult.
I say this as a person who strongly dislikes many many aspects of Unix and Linux due to bad design and terrible UX. Git has a better design than any Unix program you get.
Submodules work okay. It is just Git LFS but for Git repos. Get over it.
I’m not the author but I agree with their opinion.
Compare the UX of go mod with git submodules. One is easy and the other is about as fun as having teeth extracted.
git’s UX has never been its strong point. But submodules takes that pain to a whole new level.
I find them extremely useful. It gives me a monorepo experience in repos that otherwise aren't/can't exist as one monorepo for various reasons.
The first reason the author lists for why this will be bad is only an "issue" on Git hosts that don't allow repo creation on push (which is brain dead of GitHub). Any other host, you push your new repo, and it will see the hashing algorithm, and receive the contents accordingly.
Submodules is a legitimate argument against this, though I don't know how widely this feature is actually used, and similar to the arguments in favor of switching the default branch from master to main, this is simply a setting which can be changed.
I do like the idea of commits having both hashes, and am surprised that idea has not been explored further.
Generally though, I think the author's strongest argument is simply that the change isn't strictly "needed", and all the other issues presented aren't the strongest arguments against change.
I suspect everyone renaming their branch from master to main caused more unnecessary breakages and toil than this ever will.
The problem is existing repositories have SHA-1 commit hashes, and if you also end up changing those...
There's lots of tools that refer to git commit IDs. Some of those tools may even hardcode a commit ID to be 40 hex digits long. The fact that these tools are external also means that "oh, just rewrite the commit messages or code to refer to the new IDs" isn't feasible. The only way to not break the world is to let people refer to existing commits with their SHA-1 hashes in perpetuity, and it doesn't sound like git is set up to allow this in any way, which means that existing repositories have to stay SHA-1 in perpetuity and that will cause fun down the line if you start having to make SHA-1 and SHA-256 repositories.
Changing from master to main is a one-off change. It might require changing your scripts once to refer to 'origin/main' instead of 'origin/master', but other than that, there is essentially nothing more that needs to be done, there is no risk to historical artifacts that needs to be mitigated.
they could just have taken the sha1 of the sha256.
Thanks for writing this. I'd only been loosely following it and I hadn't realised how bad this is going to be. I have repos with tens of submodules and it's going to be a nightmare if any of them switch to sha256 in place. Not to mention I won't be able to use any new projects unless I rebuild my repo and all the submodules therein.
I thought the master to main thing was bad enough but this is going to suck. And just like the master rename it achieves basically nothing.
What is it about these projects that attracts people who just want to change things for the sake of it? Real engineering means coming up with a solution for backwards compatibility. This is just irresponsible and, frankly, a fuck you to everyone who will be affected by this.
« people who just want to change things for the sake of it » — Not for the sake of it, but to “make the world a better place.” The intention is noble (well, mostly, at least let's assume it is). Of course, there is disregard for history and her deplorables (or not so deplorables), comes with being “progressive”. Which is why I like Windows better (not 11 though, will probably have to go back to Linux at some point).
« What is it about these projects » — Maybe that they're “at the forefront.”
This seems like Y2K fud.
The alternative to making sha256 the default is to leave sha1 the default. Nobody changes to sha256. sha1 is broken in 10 years. Suddenly everyone has to switch all at once on the same day because it is a critical security issue, but github never implemented sha256 because they didn't have to. This would be a major problem.
This is very very easy to fix if you run into it.
1. Adopt git 3.0 if you can with sha256.
2. If you can't use sha256, set the config to put things back to sha1. Wherever you need to do this you probably already set dozens of ENV vars or settings, just add a new one.
Or write a 15 page analysis about how the above is so hard people will probably just find it catastrophic to even think about.
> sha1 is broken in 10 years. Suddenly everyone has to switch all at once on the same day because it is a critical security issue
If you read the OP article, the entire point he's making is that this would never happen, because a hash algorithm being "broken" doesn't matter in practice, because true supply chain security has nothing to do with file hashes.
It's easy for _one person_ to fix. It's not easy for the entire git ecosystem as a whole. GitHub, large internal corporate git repos, CI/CD systems, projects with submodules, etc. The second half the article explains all of this.
It’s already broken, but even though it’s broken it’s hard to generate git collisions because of the repo metadata. It’s easy to generate (for instance) standalone PDFs with identical hashes, but doing this with git in a useful way is much harder.
That said, it’s still a good idea to migrate to a more robust hashing algorithm. Defense in depth, etc. Just because it’s a difficult migration doesn’t mean it shouldn’t be done.
You can certainly do this, as I said, this is Google's backup plan. But defaults matter. People will start running this and getting repos that are uselessly incompatible with other repos, tools, libraries and server instances. Having it as an option is one thing. Making it a default will cause a lot of pain for people who don't want to care about this.
"defaults matter" is an argument for this change, not against it.
No, my argument is that the change should not happen at all and nobody wants it and it gains the community very, very little but the default change is forcing it on everyone and most will be _entirely_ unaware - now having to solve problems that are difficult to understand. Defaults also matter when they are the wrong defaults.
This is not difficult to understand. It's very easy to understand.
> Adopt git 3.0 if you can with sha256.
Who is "you" in the context of a distributed version control system? I think this is not just the plural you, but the unbounded you -- it's all people who not just interact with your project now, but who you hope may interact with it in the future. The question is what the cost is of committing a near-infinite population to this migration, not the cost of doing a single `brew update` on your personal machine, no?
Just clone the repo again. Jesus Christ, you act like the simplest thing in the world is some kind of insurmountable challenge.
For the record, Y2K was not fud. It was very real, in a long list of datetime problems that are to come. Further datetime problems are coming at scheduled dates.
It was definitely FUD. There was a real problem (date counters would roll over), but the impacts of it were so ridiculously overstated that it eclipsed any sane discussion of the issue. We had people at the time predicting that planes would literally fall out of the sky when Y2k hit, which was never a realistic possibility.
> which was never a realistic possibility.
Because a lot of work was done to prepare and fix potential issues.
That's the problem with deniers. When responsible persons take preemptive action to prevent tragedy, like with Y2K, the diners say it was FUD. When people don't take action, like with climate change, they say it wasn't important considering it's not them who's dead, totally discounting those who have suffered or died as a consequence. In summary, the deniers are so incompetent that they can't be trusted to correctly maintain a car, let alone civilization, considering they would never even the replace the necessary parts at the right schedules in their car.
…
What does OpenAI have anything to do with this?
Would the author feel the same if git had used MD5 instead of SHA-1?
I do actually literally write in this that if it was MD5 it also would not be a problem.
Fair cop.
They address this very theoretical. In short: Yes. Which makes sense if you don't treat the hash as a form of security against malice, especially in the case of attacks that are already impractical, which is the entire thrust of the article.
The hash isn't the security, the distribution is.
https://lore.kernel.org/git/Pine.LNX.4.58.0504291221250.1890...
As linked by another commenter in this thread, Linus worked out years ago that even if someone inserted a malicious object into the kernel repo, it would at best be a nuisance and not a major concern.