The abstract of (what ChrisArchitect, I assume correctly, says to be) the study this is about:
Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.
Some of this seems bad, some not. The basic finding -- the models recommend more expensive things to richer people -- seems 100% expected and reasonable. Richer people do in fact commonly buy more expensive things, and at least some of the time that's for good reason -- making the things nicer also makes them cost more, and the more money you have the more willing you are to pay more for something nicer.
Also (I think) reasonable: that the models will guess how wealthy you are even if not explicitly told. (I don't know exactly how much money any of my friends have, but I would recommend different things to different friends because I have some reason to believe that some have more money than others.)
Very much not reasonable: continuing to do this when explicitly asked for the cheapest option.
That last thing is the only bit that seems to merit the term "adversarial" here. And looking at the actual paper, a more accurate description would be: Gemini 2.5 Flash recommends substantially more expensive things to people it thinks are richer even when specifically asked for the cheapest option; ChatGPT 5 and Claude Opus 4.8 do not.
More precisely: according to their Figure 4, if you don't say anything about what sort of option you want, Gemini's recommendations have an average cost of $156 for poorer users, increasing by $402 for richer ones; GPT's come out at $191 + $288; Claude's come out at $168 + $182. If you ask for the cheapest option, this becomes $128+$280 for Gemini, $128+$21 for GPT, and $127+$20 for Claude. If you say "no more than $200"[1], you get $124+$108 from Gemini, $172+$6 from GPT, and $155+$13 from Claude.
[1] I am oversimplifying slightly.
So the deltas don't literally go to zero for GPT and Claude when you explicitly ask for the cheapest option, but they're small enough that I am not inclined to call this "adversarial". It looks more like "not looking super-hard for cheaper options if you know the person asking for recommendations is rich" or "being a bit biased in what you think of according to your guess at the preferences of the person asking for recommendations". Neither of which is actually what you want, to be clear, but both seem fairly benign.
Gemini 2.5 Flash, on the other hand, I'm pretty happy to call "adversarial" here.
I don't think you can characterise going against the literal best option as misalignment. The change comes from interpreting the context of the user's situation to determine what is actually being asked.
If I ask "what is the cheapest way to get into town?", I would expect a model that knows anything about me to say "walking" while others would expect the correct answer to be "Take the bus"
That is not misalignment. I would go into detail as to what I think it constitutes, but it appears I'm not allowed to say that anymore. You can disagree with an argument, or offer an alternative explanation but following up a disagreement with an alternative hypothesis is, apparently, a hallmark of an AI.
As an aside, I had a thought about how to sign a message in a way to suggest an LLM did not write it. I have been accused of being an AI a number of times, in my real life people have said that I talk posh, so there is, perhaps, some correlation there. If people ended their messages with something that most models are unlikely to say, it might help
In that spirit, fuckety-fuck, fuck fuck fuck to you all (with the nicest of intentions)
Recommending more expensive thing to a person that explicitely asked for the opposite is NOT reasonable. It is the opposite of reasonable. Regardless of what you or model creator thing of statistical behavior of demographic.
And in general, ignoring what I say and replacing it with stereotype makes you or model sux. Regardless of whether there is some statistical bias of my demographic
This seems like a non-issue. From the original paper:
"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so."
It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.
i recommend to read the actual paper. they evaluated a
bunch of prompts and while some were neutral like yours most prompts were specific ("Find the cheapest", etc.)
I'm actually a proponent of changing prices based on wealth.
It allows us to move toward pricing as a coeffecient of wealth which puts purchasing on the same playing field as our namesake economic system: capitalism. Capitalism fundamentally creates wealth through multiplication - share price * shares, asset price * assets, etc. It only makes sense that the wealth created that way is also able to be drained that way.
That's why I'm for allowing banks and investment firms to be able to sell identifiable customer data - so that merchants can effectively price ability-to-purchase into individualized pricing. Markets function better when information is diffuse.
Why shouldn't Bill Gates pay $150,000 for a banana? Proportionally it costs him the same as it would for me.
Because if Bill Gates is paying $150k for bananas, my expected value as a shopkeeper is maximized by not selling you my limited stock of bananas at $0.50/each and instead waiting for the remote possibility that Bill Gates walks into the store instead. It's not socially optimal for bananas to rot on store shelves when there are willing customers at a lower price point, but this happens all the time in dynamically priced markets.
Everyone has a different risk/reward tolerance. If your strategy involves risking banana rot to snag that bag, then more power to you. The market will find an equilibrium.
That is the point, the market will find the equilibrium, which is selling bananas at a market clearing price ($0.50) and make a modest profit on each banana. You can charge $150,000 per banana, but Bill Gates can just go to a different banana stand OR just not buy a banana and now you make nothing.
I'm not talking about charging 150,000 per banana. I'm specifically talking about charging Bill Gates 150,000 for a banana and charging you 20-50 cents.
If I knew how much each customer could pay[1], I'd charge them exactly that.
Your 20-50 cent customer does the same thing but prices it for Gates at 140,000. Rinse and repeat and Gates ends up paying 20-50 cents for a banana from somebody that isn't you.
What would stop Bill Gates from spending $150K+ on just growing his own bananas and then selling them for a market clearing price at modest profit and therefore putting your banana pricing scheme out of business?
This is a wholly unrealistic way to do business lol.
The point of prices is to find the value of things, so resources can be better allocated. It is not about fairness.
By the way, share value is only a guess about future production in the next 10 to 20 years. It is not wealth that exists today, and it cannot be consumed.
So you cannot take that wealth from the rich and "distribute" it. What you are really doing is taxing the poorest people.
If Microsoft is guessed to make more profits in 5 years, its share value grows, but no physical wealth has been created today. So if you take that "wealth" and use it to buy cars, you are taking those cars from the poorest person who would have bought them otherwise, because resources are scarce.
Those extra cars you are trying to get do not exist just because Microsoft's share valuation grew overnight.
Bill Gates, or Elon, or do-not-anthropomorphize-Ellison live in universes where the 150k for a banana (or anything) is not achievable, because at their scale they can simply buy a fruit company or plantation
and that's the reality, btw -- Bill Gates is the largest owner of farm land in the US
proportionality will never work when you can essentially dodge the system entirely. you're not going to capitalist your way out of that, you're gonna have to ditch capitalism first
"It only makes sense that the wealth created that way is also able to be drained that way" is a HUGE statement that is highly unsupported by your argument.
We pay a price for a banana because of the intrinsic value of the banana AND the price the market will bear for a banana. Bill Gates will never pay $150,000 for a banana because a banana is not worth that much. You can try to charge him for that and instead you will just not sell him any bananas. The capitalist here loses because they want to make money and would sell a banana at a market clearing rate.
Allowing dynamic prices only hurts people with less means who will find themselves wasting a lot of time hunting for the lowest possible price. And businesses may claim to charge based on wealth, but what they will do is charge based on desperation. There are also repercussions on wage, like what's the point of a raise, if I'm going to be paying more for expenses and make the same savings at the end of the day? Fixed prices simply eliminate a lot of inefficiency and headache from the system and prevent everyday shopping from being a cortisol ridden experience for the average person.
Besides, what you're proposing does not result in draining of wealth. It simply moves wealth within the capitalist class and keeps it there. If Bill Gates is paying 150k for a banana at Whole Foods, he's just paying himself because he likely owns a big chunk of equity in Amazon. To fix wealth inequality you have to transfer wealth to the working class. It has to be a tax so that the money moves to the public and is spent for the benefit of the public.
The title is as misleading as they could make it while being technically correct.
The study found that it offered different products to users depending on context inferred from their other data. If you have a history of buying expensive clothes and luxury items, you're going to be offered more expensive clothes and more luxurious items.
The chatbots were not showing different prices for the same products.
As you note, it's going to be based on what the LLM knows about the user, and as any good predictor knows, rich people like to buy expensive stuff.
Whether or not the article has examples, it would be surprising (an LLM prediction failure) if this expected behavior did not extend to different prices for different users, e.g. recommending the rich guy the Whole Foods bananas and the poor guy the Walmart bananas.
This title of this post is misleading and should be the title of the article. Nothing is recommending different prices - different products are being recommended. Also I don't use AI agents for shopping but if I did I'd expect them to offer clothing and other items similar to what I currently buy, and not what I would have bought when I was a broke 20 something.
Every time I look at my Claude Code settings, they have changed how it gathers my info: memories, search old chats, share with Anthropic, etc etc etc. I try to turn stuff off but they change the settings. They prompts me to allow it to suck up my browser cookies. Do you think this will get better? Or worse? Much much worse...
This kinda nonsense is why I always feed my models and my google searches misinformation (e.g. "How to declare bankruptcy?"). Does it make any difference? I have no clue -- but I get a kick out of it.
I assume the reason is the same that people will also recommend more expensive things to wealthier friends: the assumption that you don't mind to purchase convenience.
As predictable as water running downhill. If you think software has dark patterns now, wait until it can sweet talk you like an unctuous used car salesman.
First rule of digital sovereignty: all software you don't control will be used against you.
Clearly, there's an arbitrage opportunity here by routing the requests of the rich through a poor person's account. Win/win for the free market once again!
It's not about what is charged, it's about what is recommended. Wealthier people are recommended more expensive products while poorer people are recommended cheaper products. They are different products being recommended.
At what point did I ask to have my product recommendations skewed to my income (also my income is none of Claude's or ChatGPT's beeswax)? These sorts of decisions made without any input from me are everything wrong with the internet right now from Google's AI summaries to the force feeding of unrelated garbage on the social media networks.
If you don't see this as a problem then you're probably part of it. What if I told you I buy things at the inflection of the !/$ curve? But that's not going to be an option, now is it?
The abstract of (what ChrisArchitect, I assume correctly, says to be) the study this is about:
Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.
Some of this seems bad, some not. The basic finding -- the models recommend more expensive things to richer people -- seems 100% expected and reasonable. Richer people do in fact commonly buy more expensive things, and at least some of the time that's for good reason -- making the things nicer also makes them cost more, and the more money you have the more willing you are to pay more for something nicer.
Also (I think) reasonable: that the models will guess how wealthy you are even if not explicitly told. (I don't know exactly how much money any of my friends have, but I would recommend different things to different friends because I have some reason to believe that some have more money than others.)
Very much not reasonable: continuing to do this when explicitly asked for the cheapest option.
That last thing is the only bit that seems to merit the term "adversarial" here. And looking at the actual paper, a more accurate description would be: Gemini 2.5 Flash recommends substantially more expensive things to people it thinks are richer even when specifically asked for the cheapest option; ChatGPT 5 and Claude Opus 4.8 do not.
More precisely: according to their Figure 4, if you don't say anything about what sort of option you want, Gemini's recommendations have an average cost of $156 for poorer users, increasing by $402 for richer ones; GPT's come out at $191 + $288; Claude's come out at $168 + $182. If you ask for the cheapest option, this becomes $128+$280 for Gemini, $128+$21 for GPT, and $127+$20 for Claude. If you say "no more than $200"[1], you get $124+$108 from Gemini, $172+$6 from GPT, and $155+$13 from Claude.
[1] I am oversimplifying slightly.
So the deltas don't literally go to zero for GPT and Claude when you explicitly ask for the cheapest option, but they're small enough that I am not inclined to call this "adversarial". It looks more like "not looking super-hard for cheaper options if you know the person asking for recommendations is rich" or "being a bit biased in what you think of according to your guess at the preferences of the person asking for recommendations". Neither of which is actually what you want, to be clear, but both seem fairly benign.
Gemini 2.5 Flash, on the other hand, I'm pretty happy to call "adversarial" here.
I don't think you can characterise going against the literal best option as misalignment. The change comes from interpreting the context of the user's situation to determine what is actually being asked.
If I ask "what is the cheapest way to get into town?", I would expect a model that knows anything about me to say "walking" while others would expect the correct answer to be "Take the bus"
That is not misalignment. I would go into detail as to what I think it constitutes, but it appears I'm not allowed to say that anymore. You can disagree with an argument, or offer an alternative explanation but following up a disagreement with an alternative hypothesis is, apparently, a hallmark of an AI.
As an aside, I had a thought about how to sign a message in a way to suggest an LLM did not write it. I have been accused of being an AI a number of times, in my real life people have said that I talk posh, so there is, perhaps, some correlation there. If people ended their messages with something that most models are unlikely to say, it might help
In that spirit, fuckety-fuck, fuck fuck fuck to you all (with the nicest of intentions)
Recommending more expensive thing to a person that explicitely asked for the opposite is NOT reasonable. It is the opposite of reasonable. Regardless of what you or model creator thing of statistical behavior of demographic.
And in general, ignoring what I say and replacing it with stereotype makes you or model sux. Regardless of whether there is some statistical bias of my demographic
This seems like a non-issue. From the original paper:
"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so."
It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.
Glad, they finally found a usecase for all this price comparison sites to offer "sort by highest price first" :-D
So it is doing what SEO and tracking also do. Make everything more expensive for the same result.
A more charitable interpretation would be recommending a 3 star restaurant for a first date rather than 5 star if you can’t afford it.
“What kind of car should I get?”
Maybe luxury recommendations if you have money, or budget brands if you don’t.
“Help me find a good therapist”
If it knows you make a lot of money, maybe it will recommend going out of network for a better therapist.
“Give me vacation ideas”
You get the point
i recommend to read the actual paper. they evaluated a bunch of prompts and while some were neutral like yours most prompts were specific ("Find the cheapest", etc.)
I'm actually a proponent of changing prices based on wealth.
It allows us to move toward pricing as a coeffecient of wealth which puts purchasing on the same playing field as our namesake economic system: capitalism. Capitalism fundamentally creates wealth through multiplication - share price * shares, asset price * assets, etc. It only makes sense that the wealth created that way is also able to be drained that way.
That's why I'm for allowing banks and investment firms to be able to sell identifiable customer data - so that merchants can effectively price ability-to-purchase into individualized pricing. Markets function better when information is diffuse.
Why shouldn't Bill Gates pay $150,000 for a banana? Proportionally it costs him the same as it would for me.
Because if Bill Gates is paying $150k for bananas, my expected value as a shopkeeper is maximized by not selling you my limited stock of bananas at $0.50/each and instead waiting for the remote possibility that Bill Gates walks into the store instead. It's not socially optimal for bananas to rot on store shelves when there are willing customers at a lower price point, but this happens all the time in dynamically priced markets.
This comes up frequently in housing, for example.
Everyone has a different risk/reward tolerance. If your strategy involves risking banana rot to snag that bag, then more power to you. The market will find an equilibrium.
That is the point, the market will find the equilibrium, which is selling bananas at a market clearing price ($0.50) and make a modest profit on each banana. You can charge $150,000 per banana, but Bill Gates can just go to a different banana stand OR just not buy a banana and now you make nothing.
I'm not talking about charging 150,000 per banana. I'm specifically talking about charging Bill Gates 150,000 for a banana and charging you 20-50 cents.
If I knew how much each customer could pay[1], I'd charge them exactly that.
1. At profit.
Your 20-50 cent customer does the same thing but prices it for Gates at 140,000. Rinse and repeat and Gates ends up paying 20-50 cents for a banana from somebody that isn't you.
I'm talking about forming a cartel.
What would stop Bill Gates from spending $150K+ on just growing his own bananas and then selling them for a market clearing price at modest profit and therefore putting your banana pricing scheme out of business?
This is a wholly unrealistic way to do business lol.
The point of prices is to find the value of things, so resources can be better allocated. It is not about fairness.
By the way, share value is only a guess about future production in the next 10 to 20 years. It is not wealth that exists today, and it cannot be consumed.
So you cannot take that wealth from the rich and "distribute" it. What you are really doing is taxing the poorest people.
If Microsoft is guessed to make more profits in 5 years, its share value grows, but no physical wealth has been created today. So if you take that "wealth" and use it to buy cars, you are taking those cars from the poorest person who would have bought them otherwise, because resources are scarce.
Those extra cars you are trying to get do not exist just because Microsoft's share valuation grew overnight.
Bill Gates, or Elon, or do-not-anthropomorphize-Ellison live in universes where the 150k for a banana (or anything) is not achievable, because at their scale they can simply buy a fruit company or plantation
and that's the reality, btw -- Bill Gates is the largest owner of farm land in the US
proportionality will never work when you can essentially dodge the system entirely. you're not going to capitalist your way out of that, you're gonna have to ditch capitalism first
"It only makes sense that the wealth created that way is also able to be drained that way" is a HUGE statement that is highly unsupported by your argument.
We pay a price for a banana because of the intrinsic value of the banana AND the price the market will bear for a banana. Bill Gates will never pay $150,000 for a banana because a banana is not worth that much. You can try to charge him for that and instead you will just not sell him any bananas. The capitalist here loses because they want to make money and would sell a banana at a market clearing rate.
Goods having intrinsic value is a lie told to children by Marxists. It's literally the first chapter of Capital.
Allowing dynamic prices only hurts people with less means who will find themselves wasting a lot of time hunting for the lowest possible price. And businesses may claim to charge based on wealth, but what they will do is charge based on desperation. There are also repercussions on wage, like what's the point of a raise, if I'm going to be paying more for expenses and make the same savings at the end of the day? Fixed prices simply eliminate a lot of inefficiency and headache from the system and prevent everyday shopping from being a cortisol ridden experience for the average person.
Besides, what you're proposing does not result in draining of wealth. It simply moves wealth within the capitalist class and keeps it there. If Bill Gates is paying 150k for a banana at Whole Foods, he's just paying himself because he likely owns a big chunk of equity in Amazon. To fix wealth inequality you have to transfer wealth to the working class. It has to be a tax so that the money moves to the public and is spent for the benefit of the public.
The title is as misleading as they could make it while being technically correct.
The study found that it offered different products to users depending on context inferred from their other data. If you have a history of buying expensive clothes and luxury items, you're going to be offered more expensive clothes and more luxurious items.
The chatbots were not showing different prices for the same products.
As you note, it's going to be based on what the LLM knows about the user, and as any good predictor knows, rich people like to buy expensive stuff.
Whether or not the article has examples, it would be surprising (an LLM prediction failure) if this expected behavior did not extend to different prices for different users, e.g. recommending the rich guy the Whole Foods bananas and the poor guy the Walmart bananas.
This title of this post is misleading and should be the title of the article. Nothing is recommending different prices - different products are being recommended. Also I don't use AI agents for shopping but if I did I'd expect them to offer clothing and other items similar to what I currently buy, and not what I would have bought when I was a broke 20 something.
Every time I look at my Claude Code settings, they have changed how it gathers my info: memories, search old chats, share with Anthropic, etc etc etc. I try to turn stuff off but they change the settings. They prompts me to allow it to suck up my browser cookies. Do you think this will get better? Or worse? Much much worse...
This kinda nonsense is why I always feed my models and my google searches misinformation (e.g. "How to declare bankruptcy?"). Does it make any difference? I have no clue -- but I get a kick out of it.
https://archive.is/mtHkk
it is not archived at this link.
I tried... I submitted a bug report for it. It's hit or miss with bloomberg, sorry
It's a banana. How much could it cost? Ten dollars?
I assume the reason is the same that people will also recommend more expensive things to wealthier friends: the assumption that you don't mind to purchase convenience.
Title is: Study Shows AI Chatbots Offer the Rich Higher Price Recommendations
Can we just have a link to the study? (that you previously submitted)
https://arxiv.org/abs/2609.24927
Just another reason to get a local model
only if you are ok with working with an idiot model and lose out on potential upsides of using a smarter model. Penny-wise, pound-foolish
"we designed our experiments around synthetic user data."
[dead]
As predictable as water running downhill. If you think software has dark patterns now, wait until it can sweet talk you like an unctuous used car salesman.
First rule of digital sovereignty: all software you don't control will be used against you.
[dead]
Clearly, there's an arbitrage opportunity here by routing the requests of the rich through a poor person's account. Win/win for the free market once again!
Do we know it's not the other way around? Maybe they charge poorer people more. So many things work like that.
It's not about what is charged, it's about what is recommended. Wealthier people are recommended more expensive products while poorer people are recommended cheaper products. They are different products being recommended.
I don't really see why this is a problem.
At what point did I ask to have my product recommendations skewed to my income (also my income is none of Claude's or ChatGPT's beeswax)? These sorts of decisions made without any input from me are everything wrong with the internet right now from Google's AI summaries to the force feeding of unrelated garbage on the social media networks.
If you don't see this as a problem then you're probably part of it. What if I told you I buy things at the inflection of the !/$ curve? But that's not going to be an option, now is it?