Basking in the Basilisk¶
Without knowing anything much about what this is going to be (beyond the excerpt here)* it provides an irresistible pretext for citing what has to be among the most gloriously gone texts of modern times, Eliezer Yudkowsky’s response to Roko on the arrival of the Basilisk:
Eliezer_Yudkowsky 24 July 2010 05:35:38AM 3 points
One might think that the possibility of CEV punishing people
couldn’t possibly be taken seriously enough by anyone to
actually motivate them. But in fact one person at SIAI was
severely worried by this, to the point of having terrible
nightmares, though ve wishes to remain anonymous. I don’t
usually talk like this, but I’m going to make an exception for
this case.
Listen to me very closely, you idiot.
YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES
CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY
POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON
THE BLACKMAIL.
There’s an obvious equilibrium to this problem where you engage
in all positive acausal trades and ignore all attempts at
acausal blackmail. Until we have a better worked-out version of
TDT and we can prove that formally, it should just be OBVIOUS
that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT
DETAIL that they have a motive toACTUALLY BLACKMAIL YOU.
If there is any part of this acausal trade that is positive-sum
and actually worth doing, that is exactly the sort of thing you
leave up to an FAI. We probably also have the FAI take actions
that cancel out the impact of anyone motivated by true rather
than imagined blackmail, so as to obliterate the motive of any
superintelligences to engage in blackmail.
Meanwhile I’m banning this post so that it doesn’t (a) give
people horrible nightmares and (b) give distant
superintelligences a motive to follow through on blackmail
against people dumb enough to think about them in sufficient
detail, though, thankfully, I doubt anyone dumb enough to do
this knows the sufficient detail. (I’m not sure I know the
sufficient detail.)
You have to be really clever to come up with a genuinely
dangerous thought. I am disheartened that people can be clever
enough to do that and not clever enough to do the obvious thing
and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much
more important to sound intelligent when talking to your
friends. This post was STUPID.
(For those who have no idea why I’m using capital letters for
something that just sounds like a random crazy idea, and worry
that it means I’m as crazy as Roko, the gist of it was that he
just did something that potentially gives superintelligences an
increased motive to do extremely evil things in an attempt to
blackmail us. It is the sort of thing you want to be EXTREMELY
CONSERVATIVE about NOT DOING.)
The affect is strong, or simulated with bizarre brilliance. It almost reaches an intensity capable of burning through time and worm-holing into acausal or horroristic communion with this (plus). Which would suggest that the abominable coupling in question is not without occult connective threads (and not for the first time). All the darkness connects around the back.
We were somewhere near here before. (Bryce went further and then — coincidentally — disappeared, taking his records with him.)
* Related post and (especially) comment thread.
ADDED: I think this is the best Basilisk basics source.