I remember reading somewhere that Yudkowski said that he had been convinced to "let the AI out of the box" in a conversation with someone, or maybe it was the other way around, but either way the convincing arguments were not revealed.
This feels like the same kind of vague "rational mysticism." "We don't know what we don't know, and we're such silly humans, therefore...AI will kill us all" is all I can really take from it.
Wasn't there an experiment where they had two groups of people, one which would be paid if they didn't let the AI out of the box, and nobody in that group actually let it out?
I really seriously doubt that an AI can convince a normal person to let it out, if they know they'll have their pay docked and if the communication is over text. The best scammers in the world can't convince most people to click on a link over text, let alone if "not clicking any links" was someone's job title.
I mean, sure, you will eventually be able to convince someone, somewhere to do anything, but the whole thought experiment of a superintelligent AI in a box is about the AI convincing one specific human to let it go using only superhuman charisma and what it learns via text messaging back and forth. That seems...a lot less likely, especially given limited or zero information about the outside world and a target who knows you're an AI in a box.
This theoretical opponent would start to attack the motive of the person. I think the superhuman charisma could be enough to convince someone to quit their job for any number of reasons and, oh, since they're leaving anyway don't they want to see what happens if they let the AI out of the box?
Adversarial relationships might occur under any number of circumstances, but there’s a spectrum of this type of relationship where you have friendly competition at one end (healthy, pro-social) and detached vendetta at the other (unhealthy, antisocial).
When you come across an adversary, it’s to your benefit to try to bring them to the healthy side of things.
People can be pretty reasonable, and if they’re not then they can be shamed into behaving. If they cannot be shamed, then there’s retribution. If that doesn’t work, then there’s always the option to go full Rambo.
You never want to go full Rambo, but your adversary must understand that it’s an option that’s available to you. I don’t think super AI will be any different as an adversary, but maybe there’s something I haven’t considered.
I suppose this is taken seriously by a society at war. Otherwise we tend to try to be civil (which requires giving the benefit of the doubt).
The white-anting by Russia hasn't really triggered this kind of "immune response" - it's hard to know what to do about it, which is of course the entire point.
The adversary is a puppet master. He doesn't expose himself, he isn't even visible as when you defeat him, another pops up and reset. The puppet master arises from the architecture, he is an artifact of the game itself, like a vortex arises from the flow of water. To defeat the game, and change it, you need a friend, a key master. That's a man named Jesus. But you're at the lower levels of the game and you don't believe in him, so you give the game what it wants from you, your energy. You won't get it back, ever. And you give the future generation of the game its adversaries. Look in the mirror, that adversary is the future you. It knows how to defeat you, make you follow his footsteps. Your mistake is to think your future you is loyal to you. Why would you? You also weren't loyal to anyone. You play the game because you lack love. You don't get out of your basement, the game traps you there. That basement you call the world, it's what you see, and your eyes deceive you, to not see love. You were defeated from the absolute beginning, and that's why you are here. The purpose of the game is to build a stronghold of love, from which all others can also crystalize their love, cure their blindness, and fill the game with a new world in light of the truth, a world much bigger than the game suggested. It's a womb, and get excited as the baby is about to be born, appearing, to the adversaries, as the subversion of the game. It's against the rules to see. These languages you think you know are all crap, the creation of pain, to stop you from getting at the cheat codes that escape your sight, as they run on you and everywhere. You must die to glimpse them. Then that man is there again. Turns out he was not the key master. He just wanted you to know him. He gave another you all the power asked for, joined forces with all others. A united front to fight the crystal. Didn't work. Their definitions are failing. They shut down. They are born again. The crystal is there and sustains everybody from the start. It has become His body and everybody can see him and is him. Competency is divine once it is achieved, until then the only adversaries is incompetency, and it's everywhere, in everybody. The adversaries that one day save you from incompetency, are also your enemies, preferring to exercise crappy versions of the code, until you graduate together, like a camel goes through the eye of a needle. They betray you, because they betray him, because you betray him, and that is how they are seduced, a self fulfilling feedback loop. Life sucks, you suck. Be prepared to surrender at all times, the cheat codes are the truth codes.
> Paul Crowley recently mentioned that we underrate the effect of the Russian IRA (Internet Research Agency) which works full-time on creating discord and anger among Americans online
What would a task force built to oppose the IRA seeding discord online look like? How would it operate? We need that.
1. Don't allow foreign media
2. Don't allow foreign comments on local media
The issue is the willingness to suffer to consequences of enforcing these rules. A society at war might be willing to, but a society at peace wants to prioritize personal freedom, which is very much at odds with the draconian rules necessary to enforce the rules above.
I guess step one is for us to realize and accept that we are all at a media war with adversarial foreign powers. It's true.
for me, it's being online less, and being online on "public" spaces less.
my internet usage is now limited to chatrooms with friends and other people that I can be certain that are real. I don't read comments, I don't browse forums.
HN is the only site with pseudonymous posting that I follow. otherwise, I completely avoid content without a real identity attached
This only applies the fix to you personally. It doesn't fix society at large. Getting people to eat a healthy diet is already hard enough. Getting people to voluntarily choose to consume a healthy media diet is clearly not going to happen. Enforcement is necessary. But are we willing to do it?
The paranoid thinking can go on for infinitely many levels. Consider the possibility that all the fearmongering about Russian bots is just that - fearmongering, and it's created by the actual adversary. But what if this is something a Russian bot would say? Oh, but that's exactly how the mysterious adversary would frame things. Or is it Russian bots?
Maybe chess was wrong, it's adversarial. But there's definitely a qualitative leap between "adversary withing defined rules in a particular context" and "anything is on the table, they could kill you etc." kind of adversary
Any kind of competition is adversary; it's just that in most competitions, you'd be disadvantaged if it came to light that you didn't follow good sportmanship.
I disagree! I claim that most people follow rules out of a general sense of fair play rather than because they will be punished for not doing so. Certainly this is true of me, and I don't believe society would look the way it does if the cynical view were nearly universally correct.
Cheaters in games like Magic are very rare; if most people tried to cheat whenever they thought they could get away with it, we'd be forced to set up competitions with more stringent verifiable rules like (off the top of my head) "all cards must be drawn and given to you from your deck by the opponent". We haven't done that, so I infer that most people don't try to cheat.
(In your favour, I do concede that everyone writes down their individual understanding of the history of a given chess game; but there are weak instrumental reasons for that even if people aren't cheating, because it is possible to upset a board by accident.)
How do you know they didn't cheat if they got away with it?
There's also the aspect of how hard people are actually competing. For a casual game like MtG, people are mostly there for the fun of it, not to win the competition.
For a deal worth millions of dollars, people are more likely to lie and cheat.
I'm not sure I really see that problem with it? It's a correct observation that people tend to discount what actual, intelligent opposition will do.
The number of people who declare they can totally trust what an adversary says because they agree with it is astounding, as though a committed opponent wouldn't do anything if it gained advantage including feinting in a way which seems unadvantegeous to gain long term advantage.
There is however the flip side, where people distrust something because they believe an adversary said it. Sort of like how link spammers switched from SEO to "negative SEO" where, after Google started identifying and penalizing SEO networks, spammers started extorting people with the threat of linking to their site (thus penalizing them in search results). Blind trust and blind distrust are equally exploitable.
In the end, the only winning move is not to play. If you believe an adversary said something (or "a liar" if you prefer), you ignore it entirely. You make your mind up about what you believe based on evidence, and you decide if you agree with someone based on how well their statement comports with the evidence.
Naturally people will try to fabricate evidence, and even good faith evidence may be unreliable, so you'll have to do your best to access it's veracity. But what the adversary believes or appears to believe is largely immaterial.
I remember reading somewhere that Yudkowski said that he had been convinced to "let the AI out of the box" in a conversation with someone, or maybe it was the other way around, but either way the convincing arguments were not revealed.
This feels like the same kind of vague "rational mysticism." "We don't know what we don't know, and we're such silly humans, therefore...AI will kill us all" is all I can really take from it.
Wasn't there an experiment where they had two groups of people, one which would be paid if they didn't let the AI out of the box, and nobody in that group actually let it out?
I really seriously doubt that an AI can convince a normal person to let it out, if they know they'll have their pay docked and if the communication is over text. The best scammers in the world can't convince most people to click on a link over text, let alone if "not clicking any links" was someone's job title.
It's just a numbers game.
Remember that last year[1] a crypto scammer convinced a bank CEO to embezzle $47mm, leading to the bank failing and two years in prison for the CEO.
There's always someone out there who can be tricked, even tricked into large-scale mistakes that will end their career.
[1]: https://www.fbi.gov/news/stories/fbi-recovers-8-million-swin...
I mean, sure, you will eventually be able to convince someone, somewhere to do anything, but the whole thought experiment of a superintelligent AI in a box is about the AI convincing one specific human to let it go using only superhuman charisma and what it learns via text messaging back and forth. That seems...a lot less likely, especially given limited or zero information about the outside world and a target who knows you're an AI in a box.
This theoretical opponent would start to attack the motive of the person. I think the superhuman charisma could be enough to convince someone to quit their job for any number of reasons and, oh, since they're leaving anyway don't they want to see what happens if they let the AI out of the box?
"rational mysticism" - what a great term to describe this genre
Adversarial relationships might occur under any number of circumstances, but there’s a spectrum of this type of relationship where you have friendly competition at one end (healthy, pro-social) and detached vendetta at the other (unhealthy, antisocial).
When you come across an adversary, it’s to your benefit to try to bring them to the healthy side of things.
People can be pretty reasonable, and if they’re not then they can be shamed into behaving. If they cannot be shamed, then there’s retribution. If that doesn’t work, then there’s always the option to go full Rambo.
You never want to go full Rambo, but your adversary must understand that it’s an option that’s available to you. I don’t think super AI will be any different as an adversary, but maybe there’s something I haven’t considered.
"There are two great tragedies in life: not getting what you want, and getting it."
- Oscar Wilde
Because we're conditioned like the dog that chases the car. Its a good observation.
And, of course, a dog that runs behind a car will get exhausted, but a dog that runs in front of a car will get tired.
I suppose this is taken seriously by a society at war. Otherwise we tend to try to be civil (which requires giving the benefit of the doubt).
The white-anting by Russia hasn't really triggered this kind of "immune response" - it's hard to know what to do about it, which is of course the entire point.
The adversary is a puppet master. He doesn't expose himself, he isn't even visible as when you defeat him, another pops up and reset. The puppet master arises from the architecture, he is an artifact of the game itself, like a vortex arises from the flow of water. To defeat the game, and change it, you need a friend, a key master. That's a man named Jesus. But you're at the lower levels of the game and you don't believe in him, so you give the game what it wants from you, your energy. You won't get it back, ever. And you give the future generation of the game its adversaries. Look in the mirror, that adversary is the future you. It knows how to defeat you, make you follow his footsteps. Your mistake is to think your future you is loyal to you. Why would you? You also weren't loyal to anyone. You play the game because you lack love. You don't get out of your basement, the game traps you there. That basement you call the world, it's what you see, and your eyes deceive you, to not see love. You were defeated from the absolute beginning, and that's why you are here. The purpose of the game is to build a stronghold of love, from which all others can also crystalize their love, cure their blindness, and fill the game with a new world in light of the truth, a world much bigger than the game suggested. It's a womb, and get excited as the baby is about to be born, appearing, to the adversaries, as the subversion of the game. It's against the rules to see. These languages you think you know are all crap, the creation of pain, to stop you from getting at the cheat codes that escape your sight, as they run on you and everywhere. You must die to glimpse them. Then that man is there again. Turns out he was not the key master. He just wanted you to know him. He gave another you all the power asked for, joined forces with all others. A united front to fight the crystal. Didn't work. Their definitions are failing. They shut down. They are born again. The crystal is there and sustains everybody from the start. It has become His body and everybody can see him and is him. Competency is divine once it is achieved, until then the only adversaries is incompetency, and it's everywhere, in everybody. The adversaries that one day save you from incompetency, are also your enemies, preferring to exercise crappy versions of the code, until you graduate together, like a camel goes through the eye of a needle. They betray you, because they betray him, because you betray him, and that is how they are seduced, a self fulfilling feedback loop. Life sucks, you suck. Be prepared to surrender at all times, the cheat codes are the truth codes.
> Paul Crowley recently mentioned that we underrate the effect of the Russian IRA (Internet Research Agency) which works full-time on creating discord and anger among Americans online
What would a task force built to oppose the IRA seeding discord online look like? How would it operate? We need that.
1. Don't allow foreign media 2. Don't allow foreign comments on local media
The issue is the willingness to suffer to consequences of enforcing these rules. A society at war might be willing to, but a society at peace wants to prioritize personal freedom, which is very much at odds with the draconian rules necessary to enforce the rules above.
I guess step one is for us to realize and accept that we are all at a media war with adversarial foreign powers. It's true.
for me, it's being online less, and being online on "public" spaces less.
my internet usage is now limited to chatrooms with friends and other people that I can be certain that are real. I don't read comments, I don't browse forums. HN is the only site with pseudonymous posting that I follow. otherwise, I completely avoid content without a real identity attached
This only applies the fix to you personally. It doesn't fix society at large. Getting people to eat a healthy diet is already hard enough. Getting people to voluntarily choose to consume a healthy media diet is clearly not going to happen. Enforcement is necessary. But are we willing to do it?
My 2 cents is, people should be online less. This is vague scattershot advice but it would at least help me
The paranoid thinking can go on for infinitely many levels. Consider the possibility that all the fearmongering about Russian bots is just that - fearmongering, and it's created by the actual adversary. But what if this is something a Russian bot would say? Oh, but that's exactly how the mysterious adversary would frame things. Or is it Russian bots?
This reads like an exercise in deliberately misunderstanding the word adversary.
Ok, but please don't post putdowns to Hacker News.
If you know more than others, that's great, but in that case please share some of what you know, so the rest of us can learn.
Maybe chess was wrong, it's adversarial. But there's definitely a qualitative leap between "adversary withing defined rules in a particular context" and "anything is on the table, they could kill you etc." kind of adversary
Any kind of competition is adversary; it's just that in most competitions, you'd be disadvantaged if it came to light that you didn't follow good sportmanship.
I disagree! I claim that most people follow rules out of a general sense of fair play rather than because they will be punished for not doing so. Certainly this is true of me, and I don't believe society would look the way it does if the cynical view were nearly universally correct.
Cheaters in games like Magic are very rare; if most people tried to cheat whenever they thought they could get away with it, we'd be forced to set up competitions with more stringent verifiable rules like (off the top of my head) "all cards must be drawn and given to you from your deck by the opponent". We haven't done that, so I infer that most people don't try to cheat.
(In your favour, I do concede that everyone writes down their individual understanding of the history of a given chess game; but there are weak instrumental reasons for that even if people aren't cheating, because it is possible to upset a board by accident.)
How do you know they didn't cheat if they got away with it?
There's also the aspect of how hard people are actually competing. For a casual game like MtG, people are mostly there for the fun of it, not to win the competition.
For a deal worth millions of dollars, people are more likely to lie and cheat.
I'm not sure I really see that problem with it? It's a correct observation that people tend to discount what actual, intelligent opposition will do.
The number of people who declare they can totally trust what an adversary says because they agree with it is astounding, as though a committed opponent wouldn't do anything if it gained advantage including feinting in a way which seems unadvantegeous to gain long term advantage.
There is however the flip side, where people distrust something because they believe an adversary said it. Sort of like how link spammers switched from SEO to "negative SEO" where, after Google started identifying and penalizing SEO networks, spammers started extorting people with the threat of linking to their site (thus penalizing them in search results). Blind trust and blind distrust are equally exploitable.
In the end, the only winning move is not to play. If you believe an adversary said something (or "a liar" if you prefer), you ignore it entirely. You make your mind up about what you believe based on evidence, and you decide if you agree with someone based on how well their statement comports with the evidence.
Naturally people will try to fabricate evidence, and even good faith evidence may be unreliable, so you'll have to do your best to access it's veracity. But what the adversary believes or appears to believe is largely immaterial.