> In my opinion, every commerical software development team using Go should be using custom domains for namespacing their internal libraries and packages.
I’d remove “go” from the above, i.e. I think same applies to other stacks.
Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.
I mean yeah, but other languages don't make it quite so easy to make this mistake. Once your Go project gets to a size where it makes sense for different files to live in different folders (which, given Go's folder-based module system, often happens at just a couple hundred lines of code), the most straightforward way which the tooling nudges you towards means putting GitHub URLs (or URLs to whatever git host website you happen to use) in your source code.
I'm confused by the article, and this comment, treating these URLs as difficult to replace.
Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.
Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.
I've gone through such a migration. Huge Go code base distributed across a lot of git repositories had to be moved to a different git host.
It was horrible. A team of people spent weeks. Different projects depended on different versions of the same internal libraries, we had to create a branch for each depended-on commit and make a version of that commit with the new URLs. The expressed goal was to end up with a system that's "exactly the same" as the old, just on a new host. Changing which version of libraries projects depends on would introduce unnecessary risk.
And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
What about anyone else using your code? You can find and replace your own code, but do you have access to all of the code relying on your go library? Is it all yours? Customers? Other developers?
Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.
Even easier than that, you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from; you can point to a folder on disk or to another forge, and if you want total control you can indirect everything to your own artifact cache via GOPROXY (which can be something as simple as a static folder of source code).
The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.
This is great and I think for any company/individual who is going to ensure their domain is registered and maintained this makes a lot of sense. It would be catastrophic, but there is no reason Github couldn't disappear or otherwise change some policies that require moving away from it. The way Go works makes committing these package names tied to Github so much more weighty than simply the place you pull from.
The one thing that I was worried about was returning 301 in the example Nginx config. If you ever wanted to change the url that clients are redirected to then any browsers that visited the old url config would be forced to go to the old config's redirect url. For `go ...` and `curl` it wouldn't matter, but Chrome/Firefox will cache that 301 permanently and break the intended redirect. Not sure if this is really an issue in practice though.
> That is, if you move your git hosting to GitLab then you have to change your code!
You can also just use "replace github.com/example/example => gitlab.com/example/example" in your go.mod file and everything will keep working. That seems like a very pre-mature optimization for something that doesn't really matter.
What about everyone else using your code? You are going to ask them all to update their code as well? Many of them probably won't realize, and will get stuck on an old version that never gets updated, even to fix bugs and security issues.
cmd+shift+r in goland, push to a branch, open a PR, ask devops to make the old location archived and prevent further writes. After a couple months remove access to the old one and see what breaks.
You know, normal development task, small amount of story points..
You're now arguing that it's easy, which is a completely different argument. The comment I responded to argued that it's unnecessary ("just do the rewrite in go.mod, leave source files unchanged").
Yep that sounds pretty bad. Personally, we'd just eat not being able to build old versions, we archive previous builds, and in the case of a hotfix or whatever we'd use the go.mod rewrite trick. Not sure why you needed the commit/branch setup, did you change the history?
Have libfoo with a bunch of different versions. Authservice depends on libfoo 1.2.0. APIservice depends on libfoo 1.3.0.
Now you want to move from github.com to git.example.org. You change libfoo, authservice and apiservice to use git.example.org, that part is just simple tedious work. But the v1.2.0 and v1.3.0 tags of libfoo are old commits from before the move, so they still reference github.com! Now you need to branch off of the v1.2.0 and v1.3.0 tags of libfoo and do the same change there.
So we have only 3 repositories with only 2 dependencies and we already have to do the search/replace 5 times.
Imagine now that apiservice depends on authservice v2.3.7 just to include some type definitions. Authservice is currently on version 2.4.0 but the types haven't changed so apiservice hasn't upgraded its dependency. Now you need to branch off of authservice v2.3.7 too and do the job there. Oh and authservice v2.3.7 depends on libfoo v1.2.5, so now you need to make a branch off of libfoo v1.2.5 with the search/replace.
3 repositories with a straightforward dependency relationship, 7 search/replace jobs.
Seems like an error to use a URL. This is perfect for a URN or some other form of URI.
It could be a URN that used the DNS as a back end (though that’s a lot like a URL) so better would be something more abstract with multiple possible resolvers and a signature.
GitHub is almost forever. Your custom domain disappears when you stop paying the bills, which if you’re an open source developer has a higher likelihood than GitHub disappearing.
One day, we’re all going back to vendoring dependencies.
I think its a good question. The short answer is: because tooling doesn't default to this or make it easy.
The answer to the question of why THAT is the case - is not so easy to answer. The main benefit I can see with using lock files instead of vendoring is that it saves a lot of storage and diff history from entering your repository. So clones are much faster, backups smaller etc.
I think Go used to work this way (automated vendoring) but it’s the only language I can think of that ever did in terms of standard tooling. It would be instructive to learn why that changed.
Because it sucks. But maybe it should suck? It would make us think twice before adding dependencies.
I use dotnet and I never liked seeing dlls and binary files in my diffs. I would argue if we are adding vendor code to our projects, we should demand the FULL source code instead of dlls. Maybe it is already possible with things like x unit. I have never given it much thought... But then that vendoree code has to come from somewhere as well, right? I mean there is something to be said about provenance or something here?
Sorry if this feels like a stream of consciousness because it is ↔
Vendoring dependencies doesn’t necessarily mean you have to include binary objects in your repo. They could be content-addressable artifacts in your org’s private blob store.
Updates mean you have to copy over all the code into your repo, which creates a large diff, and hope you aren't overwriting any local changes someone might have made.
It bloats your repo, both with the actual code, and the large diffs when you update it.
You have to manually track new versions, without something to tell you if new versions are available, or if your version has known security vulnerabilities.
If the dependency has it's own dependencies, you have to vendor those too recursively. And if multiple dependencies have the same transitive dependency, it is up to you to deduplicate them, and make sure you have a version compatible with all dependents.
Thanks for answering my question! So yes, if I intend to use go and study the packages I'd have to rely on beforehand, if they're all hosted on GitHub should make me steer away from Golang. Or at least plan to cache the packages I need locally so CI always have something to work with when building.
Alternatively, could we ourselves build an automatic mirror so there's a redundant supply provider that doesn't depend on a maintainer's choice of git forge
Isn't this going to introduce the dependency of domain management? I understand the positive side of it, but put some infra level manamgment layer to individual Open Source devs, just a thought. But yeah, positives vs negatives weigh and pick.
I have not worked with Go, so I thought I'd ask: why wouldn't simple find and replace be able to fix this? And why wouldn't a coding agent be able to do this for you quickly?
And yes I'm aware of the irony of hosting this on GitHub... still figuring out a good workflow for maintaining our OSS on Codeberg and GitHub in parallel fed from the internal forge. The dependency on our own domain is real but we favor it.
Even better reason: you can later point this domain at an artifact registry. This not only gives you reliability and flexibility, it also secures your software supply chain. You don't need an SBOM or anything fancy to get started, just pull all your artifacts into a central source and improve it over time. Install an artifact registry anywhere you can run a container, use dumb static shared credentials, and start with "proxy mode". Later on you can pin or restrict versions, verify checksums, implement SSO, etc. This is going to become table stakes in the new security landscape.
> I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.
People can delete projects from GitHub. Businesses whose priorities might change may not keep an old project set to public up on GitHub because they won't want people to continue contacting them for support, or they don't want to be responsible for updating security vulnerabilities for projects they are abandoning so they'd rather just pull it offline, etc.
Whether the library you're importing is hosted on GitHub or not, never assume it will be there tomorrow. *Always vendor your dependencies.*
> In my opinion, every commerical software development team using Go should be using custom domains for namespacing their internal libraries and packages.
I’d remove “go” from the above, i.e. I think same applies to other stacks.
Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.
I mean yeah, but other languages don't make it quite so easy to make this mistake. Once your Go project gets to a size where it makes sense for different files to live in different folders (which, given Go's folder-based module system, often happens at just a couple hundred lines of code), the most straightforward way which the tooling nudges you towards means putting GitHub URLs (or URLs to whatever git host website you happen to use) in your source code.
I'm confused by the article, and this comment, treating these URLs as difficult to replace.
Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.
Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.
I've gone through such a migration. Huge Go code base distributed across a lot of git repositories had to be moved to a different git host.
It was horrible. A team of people spent weeks. Different projects depended on different versions of the same internal libraries, we had to create a branch for each depended-on commit and make a version of that commit with the new URLs. The expressed goal was to end up with a system that's "exactly the same" as the old, just on a new host. Changing which version of libraries projects depends on would introduce unnecessary risk.
And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
What about anyone else using your code? You can find and replace your own code, but do you have access to all of the code relying on your go library? Is it all yours? Customers? Other developers?
Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.
Even easier than that, you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from; you can point to a folder on disk or to another forge, and if you want total control you can indirect everything to your own artifact cache via GOPROXY (which can be something as simple as a static folder of source code).
The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.
And now all your source files contain URLs to abandoned infrastructure. Is that what you'd want long term?
It's harder to replace in history.
This is great and I think for any company/individual who is going to ensure their domain is registered and maintained this makes a lot of sense. It would be catastrophic, but there is no reason Github couldn't disappear or otherwise change some policies that require moving away from it. The way Go works makes committing these package names tied to Github so much more weighty than simply the place you pull from.
The one thing that I was worried about was returning 301 in the example Nginx config. If you ever wanted to change the url that clients are redirected to then any browsers that visited the old url config would be forced to go to the old config's redirect url. For `go ...` and `curl` it wouldn't matter, but Chrome/Firefox will cache that 301 permanently and break the intended redirect. Not sure if this is really an issue in practice though.
> That is, if you move your git hosting to GitLab then you have to change your code!
You can also just use "replace github.com/example/example => gitlab.com/example/example" in your go.mod file and everything will keep working. That seems like a very pre-mature optimization for something that doesn't really matter.
What about everyone else using your code? You are going to ask them all to update their code as well? Many of them probably won't realize, and will get stuck on an old version that never gets updated, even to fix bugs and security issues.
By this logic, those folks should be pointing to their own hosted mirrors of the project, no?
And you'll just leave references to abandoned infrastructure in your code forever?
cmd+shift+r in goland, push to a branch, open a PR, ask devops to make the old location archived and prevent further writes. After a couple months remove access to the old one and see what breaks.
You know, normal development task, small amount of story points..
You're now arguing that it's easy, which is a completely different argument. The comment I responded to argued that it's unnecessary ("just do the rewrite in go.mod, leave source files unchanged").
For my thoughts about why it's not easy, see this comment: https://news.ycombinator.com/item?id=49870363
Yep that sounds pretty bad. Personally, we'd just eat not being able to build old versions, we archive previous builds, and in the case of a hotfix or whatever we'd use the go.mod rewrite trick. Not sure why you needed the commit/branch setup, did you change the history?
Have libfoo with a bunch of different versions. Authservice depends on libfoo 1.2.0. APIservice depends on libfoo 1.3.0.
Now you want to move from github.com to git.example.org. You change libfoo, authservice and apiservice to use git.example.org, that part is just simple tedious work. But the v1.2.0 and v1.3.0 tags of libfoo are old commits from before the move, so they still reference github.com! Now you need to branch off of the v1.2.0 and v1.3.0 tags of libfoo and do the same change there.
So we have only 3 repositories with only 2 dependencies and we already have to do the search/replace 5 times.
Imagine now that apiservice depends on authservice v2.3.7 just to include some type definitions. Authservice is currently on version 2.4.0 but the types haven't changed so apiservice hasn't upgraded its dependency. Now you need to branch off of authservice v2.3.7 too and do the job there. Oh and authservice v2.3.7 depends on libfoo v1.2.5, so now you need to make a branch off of libfoo v1.2.5 with the search/replace.
3 repositories with a straightforward dependency relationship, 7 search/replace jobs.
The numbers get terrifying as you scale this up.
Seems like an error to use a URL. This is perfect for a URN or some other form of URI.
It could be a URN that used the DNS as a back end (though that’s a lot like a URL) so better would be something more abstract with multiple possible resolvers and a signature.
GitHub is almost forever. Your custom domain disappears when you stop paying the bills, which if you’re an open source developer has a higher likelihood than GitHub disappearing.
One day, we’re all going back to vendoring dependencies.
Why did we stop vendoring dependencies in the first place?
I think its a good question. The short answer is: because tooling doesn't default to this or make it easy.
The answer to the question of why THAT is the case - is not so easy to answer. The main benefit I can see with using lock files instead of vendoring is that it saves a lot of storage and diff history from entering your repository. So clones are much faster, backups smaller etc.
I think Go used to work this way (automated vendoring) but it’s the only language I can think of that ever did in terms of standard tooling. It would be instructive to learn why that changed.
Because it sucks. But maybe it should suck? It would make us think twice before adding dependencies.
I use dotnet and I never liked seeing dlls and binary files in my diffs. I would argue if we are adding vendor code to our projects, we should demand the FULL source code instead of dlls. Maybe it is already possible with things like x unit. I have never given it much thought... But then that vendoree code has to come from somewhere as well, right? I mean there is something to be said about provenance or something here?
Sorry if this feels like a stream of consciousness because it is ↔
Vendoring dependencies doesn’t necessarily mean you have to include binary objects in your repo. They could be content-addressable artifacts in your org’s private blob store.
Because its not as convenient
Can you elaborate?
Updates mean you have to copy over all the code into your repo, which creates a large diff, and hope you aren't overwriting any local changes someone might have made.
It bloats your repo, both with the actual code, and the large diffs when you update it.
You have to manually track new versions, without something to tell you if new versions are available, or if your version has known security vulnerabilities.
If the dependency has it's own dependencies, you have to vendor those too recursively. And if multiple dependencies have the same transitive dependency, it is up to you to deduplicate them, and make sure you have a version compatible with all dependents.
Etc.
We didn't!
In the case of a typical software enterprise having your domain gone means that you probably don't care about the code anymore anyway.
Thanks for answering my question! So yes, if I intend to use go and study the packages I'd have to rely on beforehand, if they're all hosted on GitHub should make me steer away from Golang. Or at least plan to cache the packages I need locally so CI always have something to work with when building.
Alternatively, could we ourselves build an automatic mirror so there's a redundant supply provider that doesn't depend on a maintainer's choice of git forge
https://news.ycombinator.com/item?id=49434625
That's a beautiful solution. I love it! The alternative work went with was to just build everything ourselves.
Isn't this going to introduce the dependency of domain management? I understand the positive side of it, but put some infra level manamgment layer to individual Open Source devs, just a thought. But yeah, positives vs negatives weigh and pick.
I have not worked with Go, so I thought I'd ask: why wouldn't simple find and replace be able to fix this? And why wouldn't a coding agent be able to do this for you quickly?
See: https://news.ycombinator.com/item?id=49870363
There are also a few Hugo templates for vanity import paths.
Here's mine: https://github.com/foundata/hugo-theme-govanity (e.g. used at https://golang.foundata.com/ )
And yes I'm aware of the irony of hosting this on GitHub... still figuring out a good workflow for maintaining our OSS on Codeberg and GitHub in parallel fed from the internal forge. The dependency on our own domain is real but we favor it.
You shouldn't be using fully qualified domains at all. Use relative paths instead, like sibling-hosted git submodules.
braid has been a treat for us to manage external dependencies. Much cleaner than submodules.
Even better reason: you can later point this domain at an artifact registry. This not only gives you reliability and flexibility, it also secures your software supply chain. You don't need an SBOM or anything fancy to get started, just pull all your artifacts into a central source and improve it over time. Install an artifact registry anywhere you can run a container, use dumb static shared credentials, and start with "proxy mode". Later on you can pin or restrict versions, verify checksums, implement SSO, etc. This is going to become table stakes in the new security landscape.
You don't need an SBOM or anything fancy to get started
-- Curious how this avoids SBOM need?
Not really convinced. I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.
> I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.
People can delete projects from GitHub. Businesses whose priorities might change may not keep an old project set to public up on GitHub because they won't want people to continue contacting them for support, or they don't want to be responsible for updating security vulnerabilities for projects they are abandoning so they'd rather just pull it offline, etc.
Whether the library you're importing is hosted on GitHub or not, never assume it will be there tomorrow. *Always vendor your dependencies.*
I believe they'd also be cached by proxy.golang.org? But companies should probably run their own proxy.