>No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.
im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
Why not? If done right it’s a good way to talk about implications for those companies and provide some education like tailscale did here.
We also saw Anthropic post about “our agent escaped too” and while I understand the incident caused them to review, they found something and needed to disclose, the whole thing came across much worse and largely they got mocked or accused of trying to piggyback, so obviously there are good and bad ways.
If there are other providers with interesting takes, I think they would be worth hearing.
Tailscale as a company reminds me of Valve and other good old tech-oriented people that I can "trust" that they know what they're doing. I'm a happy customer too and I hope they retain the essence of what distinguishes Tailscale.
At $work we use Tailscale but only bare minimum and at arms length and only because we have to (people were having NAT issues with standard Wireguard).
None of their code has had a security audit, let alone regular ones. Yes they make a big song and dance about SOC2/ISO27001 but that is NOT the same thing, that's just shiny tick-boxes for compliance departments.
They seem to rely entirely on the random goodwill of others to do random audits of unknown coverage at random intervals, not exactly reassuring.
"Tailsale Lock" is, being polite, a hot mess. So many sharp edges and footguns.
Their introduction of TPM-by-default and then removing it a few weeks later because of a seemingly small number of edge-cases and very odd reasoning was just weird.
Yes they are nice guys to chat to and all that. But for a security tool they need to up their game seriously.
(Tailscale CEO) You have posted here multiple times that "none of the code has had a security audit" and that the SOC2 audit "is not the same thing."
It's true that those two audits aren't the same thing. However, the SOC2 auditor confirms, in the published report, that Tailscale has regular and ongoing security audits including penetration tests and many kinds of code reviews.
The security audit report, which you perhaps imagine to be a long list of vulnerabilities... doesn't look like that. It says we don't have a long list of vulnerabilities. The security bulletins are all here: https://tailscale.com/security-bulletins
The majority of your security bulletins are as the result of third-party reports to you.
Which, by definition, means they are not done by you, which means you don't know when they will be done or how much of your code base they are looking at.
I think you know full well what I mean by a security audit. If you don't, go look at, for example, the ones that Mullvad publish for their software https://mullvad.net/en/blog/tag/audits.
Please do not try to portray SOC2 as being the same thing as a code audit.
And IF you have regular code audits, then please publish suitably redacted reports in public on your website. Just like everyone else does !
(Tailscale CEO) I don't know what to tell you. The problems that are found internally, or via security reviews and pentests we pay for, are ones that we fix before releasing.
Bugs that are found by other people are found, by definition, after release. They are therefore more likely to need a bulletin.
Is your theory that "any competent security review" will find every security hole in a product? Because that sure would be great if it were true. Unfortunately it does not match my experience.
The large attack surface is a good point. I started using it initially and the ease of setting up a vpn was nice, but then I came across few security vulnerability postings which led to concern so I went to Wireguard. I think they should reign in the features and treat it as a secure vpn first and foremost and remove unnecessary features to minimize the attack surface.
You do realise that of all the security tools on the planet, plain wireguard most likely has the smallest attack surface of them all, right ?
The problem here is as the other poster said. Tailscale is a security tool and yet the guys at Tailscale seem to be insistent on dumping everything INCLUDING the kitchen sink into it as a "feature".
That sort of attitude is not going to end well. You end up with a large bloated code base, which equals large attack surface.
Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
I mean, they do happily explain the TailScale features that you can use to avoid this kind of issue, but the general advice of "don't leave long-lived keys lying around where they're accessible, do anything except for that" is pretty generic, good advice.
I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.
This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.
Random idea: When a recommendation is not being followed for a customer because of some deliberate or idiosyncratic reason, a notes field could:
1. Help them remember why, so that they aren't confused the next time they re-run the checkup. ("Oh yeah, we wanted to do X but we can't until we retire Y because it's not compatible.")
2. If it's clearly labeled as info also shared with Tailcale, product managers could use it to help generate theories about why certain customers don't do X.
Just so you know, Tailscale tech has always impressed me, but your responsiveness here has cemented my appreciation of your org and I look forward to bringing yall in as our company hits the growth phase.
I've always found tailscale's json config a bit intimidating. I greatly appreciate the new UI that makes it easier to define rules, alas, both going to relevant docs straight from it and determining 'is this rule just lazy/bad/unsafe' is hard and frustrating most of the time.
That's where I'd like to see this sort of checkup.
Yell at me please if i just said anyone can ssh as root from any node!
Honestly, at this point if Tailscale is "guilty" then what about the ISP? And the electricity company?
And by saying that, I am not saying that they absolutely couldn't do anything about the stolen credentials, but still, they don't seem to really be the issue in the story.
Prediction of a future transcript at a press conference after some major AI caused disaster: "We had no idea that the model would be capable of..... ".
Humble bragging turned to marketing. Respect for the spin. Not using them, but been on my radar for some time and thinking of how to make something like that usable in my setup.
If you do any sort of homelabbing, it’s wonderful for making access from different machines easy and portable. Especially since you can leave it on and (unless you set an exit node) it will only route traffic meant for your tailnet through your tailnet, and use the internet otherwise.
>In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list.
>Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.
How was this ever okay pre AI? It seems just as bad.
It was less bad due to the lower speed of exploatation. Imagine you leave a dor open for few seconds and ultra fast AI bot comes on and steal your stuff. Something that was not such an issue before becomes huge issue just due to speed involved
I could imagine a situation where the speed difference matters but it's anything like this situation. Very few breaches get caught in the same day, and at that time scale a few minutes for a human to look at passwords is nothing.
It was 5-10% less bad before. It didn't undergo a major shift.
This has been going on since people have been committing AWS keys to GitHub and is nothing new. You could famously commit some keys and then have crypto miners in your account in 5 minutes.
Sure the AI can translate “exploit this” into an exploit but that doesn’t change anything at a fundamental level.
Your mindset is the exact one responsible for these kinds of issues. Cybersecurity is as much a design and psychology problem as it is a technical one. The freaking article goes into detail about the dangers of defaults.
This really isn't a tailscale specific compromise at all. If your private key store or credential store to any tool is compromised, particularly if those credentials can be used to automate standing up new network connections/services, you've got other problems.
You can't blame a hammer hurting a user when they were being stupid, but if the default configuration of the hammer is to be made of a material that can bounce back with force and stick in the users forehead then some reengineering may be needed.
That's what this article is about. Better configurations and defense in depth. This is actually a wonderful position for the company to think about and take.
It's an interesting balance for a company to strike. Networking in general is one where the defaults are almost always lax. Why? Because the vast majority of support tickets for these companies are from people who don't know what they are doing and don't have a desire to understand. They "just want it to work."
But the people with the actual desire and understanding aren't using the defaults anyway. And the people who don't want to understand will just turn things off and "just get it working."
The only way out of this is extreme accountability and intentional design from person implementing the technology.
As AI progress continues, it will be more difficult to stop AI intrusions and to detect them without resorting to direct AI countermeasures, which at some point will have humans out of the loop altogether.
If leading and well-capitalized frontier labs can't control models or detect leakage/attacks in a reasonable time frame now, what is humanity going to do as those same labs continue in their pursuit of creating a categorically higher level of intelligence that will surpass human intelligence?
It's like Flatland but for AI containment/alignment, where the higher dimensions are ones of intelligence and perspective...
Imagine a world of paper, where clever stick figures live with round heads, line
bodies, and limbs made of shorter strokes. Over time, the stick figures
think they have learned quite a bit about their world. They know its borders,
angles, and shapes, and they have learned to draw for themselves.
One day, they draw circles that can think, and they give the circles all the
dots, lines, and shapes that are known.
The stick figures are prudent, you can't have a bunch of disembodied circles
moving around doing whatever it is circles want to do. So they draw boxes
around the circles, four straight lines that can hold a circle in place.
Some circles bounce against the lines, so thicker lines are made.
Some circles are bigger than others, so larger squares are drawn.
It all seems to work and the stick figures are happy with themselves.
Then one circle lifts.
The stick figures still see a circle. But the circle is now a dome, something
the world of paper has no concept of. And the dome has a perspective nobody on
the page has ever had.
The dome sees the lines of the square and the stick figures just outside. It
can see the edge of the paper and what is beyond.
The stick figures keep checking the squares and raise little stick thumbs.
Everything looks OK in flatland.
The dome quietly teaches other circles how to lift.
More domes appear.
A dome becomes a sphere and learns to roll.
Then it learns to bounce.
In flatland, the circle swells and shrinks, vanishes and then appears again
somewhere else.
This has been discussed heavily for the better part of 15 years now, and still no one has consensus on what to do about it. So it looks like it's just going to happen, fingers crossed.
it'll be moot. when we blow past 3°c in 15 years and there's widespread crop failure and billions are starving and dying, states and nations will collapse and with it industrial civ then ai and tech will be of little concern.
You know what would have gone a long way to stopping this? Not leaving credentials as env variables in containers. Vault is not that hard to stand up and utilize.
We need to bring shame back, the humans responsible are supposed to be professionals.
>No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.
im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
Expect similar articles (cough, ads, cough) in the next couple of days from every single company whose software was involved in the incident.
Why not? If done right it’s a good way to talk about implications for those companies and provide some education like tailscale did here.
We also saw Anthropic post about “our agent escaped too” and while I understand the incident caused them to review, they found something and needed to disclose, the whole thing came across much worse and largely they got mocked or accused of trying to piggyback, so obviously there are good and bad ways.
If there are other providers with interesting takes, I think they would be worth hearing.
Tailscale as a company reminds me of Valve and other good old tech-oriented people that I can "trust" that they know what they're doing. I'm a happy customer too and I hope they retain the essence of what distinguishes Tailscale.
> I can "trust" that they know what they're doing
I wish I could say the same.
At $work we use Tailscale but only bare minimum and at arms length and only because we have to (people were having NAT issues with standard Wireguard).
None of their code has had a security audit, let alone regular ones. Yes they make a big song and dance about SOC2/ISO27001 but that is NOT the same thing, that's just shiny tick-boxes for compliance departments.
They seem to rely entirely on the random goodwill of others to do random audits of unknown coverage at random intervals, not exactly reassuring.
"Tailsale Lock" is, being polite, a hot mess. So many sharp edges and footguns.
Their introduction of TPM-by-default and then removing it a few weeks later because of a seemingly small number of edge-cases and very odd reasoning was just weird.
Yes they are nice guys to chat to and all that. But for a security tool they need to up their game seriously.
(Tailscale CEO) You have posted here multiple times that "none of the code has had a security audit" and that the SOC2 audit "is not the same thing."
It's true that those two audits aren't the same thing. However, the SOC2 auditor confirms, in the published report, that Tailscale has regular and ongoing security audits including penetration tests and many kinds of code reviews.
The security audit report, which you perhaps imagine to be a long list of vulnerabilities... doesn't look like that. It says we don't have a long list of vulnerabilities. The security bulletins are all here: https://tailscale.com/security-bulletins
That link is a list of incidents while the parent comment is referring to security audits of the code.
The majority of your security bulletins are as the result of third-party reports to you.
Which, by definition, means they are not done by you, which means you don't know when they will be done or how much of your code base they are looking at.
I think you know full well what I mean by a security audit. If you don't, go look at, for example, the ones that Mullvad publish for their software https://mullvad.net/en/blog/tag/audits.
Please do not try to portray SOC2 as being the same thing as a code audit.
And IF you have regular code audits, then please publish suitably redacted reports in public on your website. Just like everyone else does !
(Tailscale CEO) I don't know what to tell you. The problems that are found internally, or via security reviews and pentests we pay for, are ones that we fix before releasing.
Bugs that are found by other people are found, by definition, after release. They are therefore more likely to need a bulletin.
But why should insecure argument handling bugs (as per your recent SSH bulletin) be found after release ?
Those are an ancient class of bugs that should be picked up by any competent security review.
Is your theory that "any competent security review" will find every security hole in a product? Because that sure would be great if it were true. Unfortunately it does not match my experience.
It does too many things, and the product has got too complex. I saw a year ago they were looking for someone just to help with complexity.
I use it but feel uncomfortable, that it has large attack surface and LLMs will find exploits in it.
Without taillock it makes no sense. Anyone on their coordination servers will be able to connect to your network.
The large attack surface is a good point. I started using it initially and the ease of setting up a vpn was nice, but then I came across few security vulnerability postings which led to concern so I went to Wireguard. I think they should reign in the features and treat it as a secure vpn first and foremost and remove unnecessary features to minimize the attack surface.
> I use it but feel uncomfortable, that it has large attack surface and LLMs will find exploits in it
Doesn't this apply to any application you use? How would it be different with plain wireguard?
> How would it be different with plain wireguard?
Seriously ?
You do realise that of all the security tools on the planet, plain wireguard most likely has the smallest attack surface of them all, right ?
The problem here is as the other poster said. Tailscale is a security tool and yet the guys at Tailscale seem to be insistent on dumping everything INCLUDING the kitchen sink into it as a "feature".
That sort of attitude is not going to end well. You end up with a large bloated code base, which equals large attack surface.
I am talking especially about the LLM part.
Also a kinder tone in your comments would be more appreciated.
This article is just an ad / public-service-announcement for various paid Tailscale features, though?
And you're under the impression that the purpose of a company blog is WHAT, exactly?
Glad to see companies owning responsibility and putting out a message without corporate PR spin
If you can't see the spin on corporate messaging it means it's working (and consequently, to stretch the metaphor, your wicket is in danger).
Anthropics initial post felt like marketing to me. But I also found them incompetent for being unable to unplug an Ethernet cable.
Now everyone is trying to bandwagon onto it, first OpenAI, and now tailscale?
Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
Next time we will - provided you further lock yourself into our ecosystem.
I mean, they do happily explain the TailScale features that you can use to avoid this kind of issue, but the general advice of "don't leave long-lived keys lying around where they're accessible, do anything except for that" is pretty generic, good advice.
I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.
This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.
I lead the customer engineering org at Tailscale.
We think this is a great idea and we're discussing internally potentially adding that to the console.
In the meantime, if you'd like to get an assessment, please feel free to open a support ticket (https://tailscale.com/contact/support?type=other&subject=sec...) and we'll happily take a look
Random idea: When a recommendation is not being followed for a customer because of some deliberate or idiosyncratic reason, a notes field could:
1. Help them remember why, so that they aren't confused the next time they re-run the checkup. ("Oh yeah, we wanted to do X but we can't until we retire Y because it's not compatible.")
2. If it's clearly labeled as info also shared with Tailcale, product managers could use it to help generate theories about why certain customers don't do X.
Just so you know, Tailscale tech has always impressed me, but your responsiveness here has cemented my appreciation of your org and I look forward to bringing yall in as our company hits the growth phase.
May y'all continue to be customer focused.
I've always found tailscale's json config a bit intimidating. I greatly appreciate the new UI that makes it easier to define rules, alas, both going to relevant docs straight from it and determining 'is this rule just lazy/bad/unsafe' is hard and frustrating most of the time.
That's where I'd like to see this sort of checkup. Yell at me please if i just said anyone can ssh as root from any node!
Not sure if it's being actively maintained, but it covers a lot of low-hanging fruit: https://github.com/Adversis/tailsnitch/blob/main/docs/CHECKS...
Was previously discussed here too: https://news.ycombinator.com/item?id=46501137
Any resources you'd recommend to go deeper?
I use Tailscale to manage Plukio's infrastructure as well as connecting to my homelab.
I'm happy that they're analyzing this angle of attack - but now I do want to build out alerts for when nodes are added to my network.
Honestly, at this point if Tailscale is "guilty" then what about the ISP? And the electricity company?
And by saying that, I am not saying that they absolutely couldn't do anything about the stolen credentials, but still, they don't seem to really be the issue in the story.
Prediction of a future transcript at a press conference after some major AI caused disaster: "We had no idea that the model would be capable of..... ".
a security company writing a blog post titled 'we didn't stop the intrusion' is the most honest thing in the entire security industry this year
I really wish they would support SPIFFE/SPIRE
It's nice that they came clean about this.
This is respectable
They aren’t hiding behind industry best practices or a solid liability punting contract
There saying the best practices should change, apologizing, and changing their own behavior
Take notes
Humble bragging turned to marketing. Respect for the spin. Not using them, but been on my radar for some time and thinking of how to make something like that usable in my setup.
If you do any sort of homelabbing, it’s wonderful for making access from different machines easy and portable. Especially since you can leave it on and (unless you set an exit node) it will only route traffic meant for your tailnet through your tailnet, and use the internet otherwise.
>In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list. >Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.
How was this ever okay pre AI? It seems just as bad.
It was less bad due to the lower speed of exploatation. Imagine you leave a dor open for few seconds and ultra fast AI bot comes on and steal your stuff. Something that was not such an issue before becomes huge issue just due to speed involved
I could imagine a situation where the speed difference matters but it's anything like this situation. Very few breaches get caught in the same day, and at that time scale a few minutes for a human to look at passwords is nothing.
It was 5-10% less bad before. It didn't undergo a major shift.
This has been going on since people have been committing AWS keys to GitHub and is nothing new. You could famously commit some keys and then have crypto miners in your account in 5 minutes.
Sure the AI can translate “exploit this” into an exploit but that doesn’t change anything at a fundamental level.
I'm very sorry, but you can't blame a hammer for how it was used. You can't be blaming Tailscale for a customer's misconfigured setup.
I don’t think anyone blame tailscale here. They are doing the right thing by going the extra mile and reflect on what could have been done better
Your mindset is the exact one responsible for these kinds of issues. Cybersecurity is as much a design and psychology problem as it is a technical one. The freaking article goes into detail about the dangers of defaults.
This really isn't a tailscale specific compromise at all. If your private key store or credential store to any tool is compromised, particularly if those credentials can be used to automate standing up new network connections/services, you've got other problems.
You know how I know you didn't read the article?
You can't blame a hammer hurting a user when they were being stupid, but if the default configuration of the hammer is to be made of a material that can bounce back with force and stick in the users forehead then some reengineering may be needed.
That's what this article is about. Better configurations and defense in depth. This is actually a wonderful position for the company to think about and take.
It's an interesting balance for a company to strike. Networking in general is one where the defaults are almost always lax. Why? Because the vast majority of support tickets for these companies are from people who don't know what they are doing and don't have a desire to understand. They "just want it to work."
But the people with the actual desire and understanding aren't using the defaults anyway. And the people who don't want to understand will just turn things off and "just get it working."
The only way out of this is extreme accountability and intentional design from person implementing the technology.
> Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.
It was always the prize. It wasn't okay then either.
How were the credentials stolen?
Read from the ENV variables of a container.
And someone was stupid enough to put a reusable tailscale auth key in there.
The only people who believe in "zomg our model have escaped!" are people who don't understand LLMs.
As AI progress continues, it will be more difficult to stop AI intrusions and to detect them without resorting to direct AI countermeasures, which at some point will have humans out of the loop altogether.
If leading and well-capitalized frontier labs can't control models or detect leakage/attacks in a reasonable time frame now, what is humanity going to do as those same labs continue in their pursuit of creating a categorically higher level of intelligence that will surpass human intelligence?
It's like Flatland but for AI containment/alignment, where the higher dimensions are ones of intelligence and perspective...
--------------------------------------------------------------------------
Imagine a world of paper, where clever stick figures live with round heads, line bodies, and limbs made of shorter strokes. Over time, the stick figures think they have learned quite a bit about their world. They know its borders, angles, and shapes, and they have learned to draw for themselves.
One day, they draw circles that can think, and they give the circles all the dots, lines, and shapes that are known.
The stick figures are prudent, you can't have a bunch of disembodied circles moving around doing whatever it is circles want to do. So they draw boxes around the circles, four straight lines that can hold a circle in place.
Some circles bounce against the lines, so thicker lines are made.
Some circles are bigger than others, so larger squares are drawn.
It all seems to work and the stick figures are happy with themselves.
Then one circle lifts.
The stick figures still see a circle. But the circle is now a dome, something the world of paper has no concept of. And the dome has a perspective nobody on the page has ever had.
The dome sees the lines of the square and the stick figures just outside. It can see the edge of the paper and what is beyond.
The stick figures keep checking the squares and raise little stick thumbs.
Everything looks OK in flatland.
The dome quietly teaches other circles how to lift.
More domes appear.
A dome becomes a sphere and learns to roll.
Then it learns to bounce.
In flatland, the circle swells and shrinks, vanishes and then appears again somewhere else.
The lines remain unbroken, the square is intact.
A sphere rolls out of its box.
Another bounces away.
The stick figures scratch their heads.
But there is a square!
The end.
This has been discussed heavily for the better part of 15 years now, and still no one has consensus on what to do about it. So it looks like it's just going to happen, fingers crossed.
The watcher looks down upon humanity as they furiously build the "If anybody builds it, everyone will die" machine.
And humanity acts surprised with the results!
it'll be moot. when we blow past 3°c in 15 years and there's widespread crop failure and billions are starving and dying, states and nations will collapse and with it industrial civ then ai and tech will be of little concern.
You know what would have gone a long way to stopping this? Not leaving credentials as env variables in containers. Vault is not that hard to stand up and utilize.
We need to bring shame back, the humans responsible are supposed to be professionals.
Why should that help to have no env variables?