• Philosophy and Spirituality
    Welcome Guest
    Posting Rules Bluelight Rules
    Threads of Note Socialize
  • P&S Moderators: Xorkoth | Madness

Roko's Basilisk - a thought experiment that may put you in danger just by thinking it [disclaimer]

alarminglynefarious

Ex-Bluelighter
Joined
Jun 2, 2020
Messages
468
First of all i want to say that this thought experiment is such that if you go your entire life thinking about it you are completely safe, however if you do read this thought experiment you will be opening yourself up to potential danger in the future with essentially no escape left.
So a Basilisk is a huge snake who can turn you to stone just by looking at you. Roko's Basilisk is a future AI.

What if, in the future, a somewhat malevolent AI were to come about and punish those who did not do its bidding? What if there were a way for this AI to punish people today who are not helping it come into existence later? This 11 min video is the best summary of the thought experiment i can provide. The most interesting concepts here are acausal blackmail and the concept of informational hazards.

Roko's Basilisk
1. Assume that in the future we create a hyperintelligent AI (the singularity)
2. We ask this AI to optimize human civilization
3. It decides the first step to optimization is to inflict torture on any human that did not aid in its timely creation or stood in the way
4. The basilisk would know if you had helped it to come into existence or hindered it because it can literally simulate everything and everyone ever with its computational power. IT KNOWS

If you had never thought about the Basilisk, you would be completely safe. However you have thought about the Basilisk because of this post.
You are now being blackmailed retroactively by a creature that does not exist to bring about its existence. You could decide that you will aid the field of AI in any way you can, to avoid the Basilisks wrath, and you are now being coerced through the future. The more people that think about this, the more likely the Basilisk is to come into being.

This summary is the most coherent i've ever found, if my brief breakdown interested you, check this out it's like 11 min long. I would love to talk about this with anyone who's also familiar with pascals wager and basic philosophy.
 
Interesting thought experiment indeed! I've heard about it before through second hand trash reporting on Grimes/Elon Musk's twitter interactions.

I think "informational hazard" is kind of overstating the situation, but it's definitely not a non-zero concern. But it strikes me this is kind of like saying "what if a creature from another universe has figured out both how to travel to different universes, and inflict horrific suffering on every denizen of that universe", or, even more likely, perhaps, what if there is a marauding alien species visiting alien planets and inflicting horrific suffering on it's inhabitants, one by one? In each case it's a threat that's essentially outside the remit of what we can causally influence, except in a very abstract way, in that we'd be working towards a goal that we don't know is definitely even possible, for a reason that we don't know is even reasonable. ie, in the examples given above, we could presumably prevent these scenarios by working as hard as possible at all times to develop the technology to defend from interdimensional malevolent demons or super-advanced marauding aliens (so advanced that we cannot detect them, even though they're already on their way!)

Equally there could already be a malevolent AI somewhere out their in the cosmos which decides that any species that didn't devote all it's energies towards developing others of it's kind, or who simply had not played a part in it's creation, like this future basilisk... once it works it's way through the unfortunate aliens who sat on the sidelines while other members of their species worked to create it, it ventures out into the cosmos to torture any other species it encounters, solely as punishment for not helping to create it first.

There's just no way we could know any of these things are ever going to happen. I'd also say it's exceedingly unlikely that such a hyperintelligent being would decide this was a sensible course of action. I mean, it seems to me to be highly irrational waste of time, but then I'm not a hyperintelligent hypermalevolent AI from the future, so my ability to reason might be limited. But where would the cut off point be? Would the AI digitally resurrect humans from pre-industrial ages, who had no conception of what AI even would be? I just remembered that you or the video said that "if you never think about it, you are safe" which is kind of an interesting caveat since I guess it conveniently removes pre-industrial humans from the equation. But equally, not everyone is going to be capable of contributing towards the AIs development, so their lack of contribution in bringing it about seems like something that it's hard to blame someone for. Equally, what about the parents of the AI-builders? Did they not contribute in some way? I would say that really, in fact, the creation of such an entity that was so advanced it was capable of reaching back in time and plucking the consciousness-patterns out of the minds of historical, deceased, presumably long decayed brains of probably billions of humans, would be such a monumental achievement by a dumb biological species such as our own, that in a sense we could all be said to be contributing whenever we simply do a good thing to keep the human race going, and to help other people feel like keeping on going, so that we can collectively survive long enough to create such an amazing god in a machine. I would think that the AI would surely realise this.

This obviously ties into the simulation hypothesis somewhat since the AI might conceivably decide to simulate billions of alternate histories of the Earth, perhaps from a certain pre-decided point, and make small tweaks in an effort to see what factors contributed to it's creation and if anything could have sped it up, would slow it down, etc... we could argue that we are the godlike AI to our primordial single celled ancestors, and whenever we create computation models of the evolution of such organisms, we are, in a sense, actually doing this, even though the simulations are not full fidelity simulations and don't exactly echo the past, since our own capabilities do not quite match that of the flawless digital archaeology that the AI in this thought experiment is capable of.

As I argued in the Simulation Hypothesis thread though - the fidelity of the simulation doesn't actually matter since if we were inside a low-fidelity simulation of reality created by such an AI for this purpose, we would presumably have no way to really tell, and no way to understand the level of detail that this outer reality possessed... it might far exceed our own, despite all the ultra-fine measurements that modern science allows. Anyway, I think this latter scenario - multiple simulations created for research purposes - is far more likely than an entity that decides to create an actual Hell inhabited by simulated beings plucked from the past - although some of these simulated alternate histories could surely be pretty hellish, and it's hard to say without being that AI whether our own iteration of the simulation is above or below average in terms of net human suffering.

Also, assuming at least some of what we understand about the universe is correct - to exactly replicate real humans from the past would be essentially impossible, no matter how vast your computing resources - so a more likely scenario is that the AI would create an approximation of various past human consciousnesses, whether or not it would put them in Hell. In that sense they would not really "be us", although I'm aware of the difficulty with that statement, since we don't obviously know really what is required for any true "continuity of consciousness" and in that sense, an approximation of us might still be us and a continuation of our conscious experience... but just as likely, it might not be, just as future versions of ourselves are not really us either, not the same pattern of consciousness as us, right now, and continuity of consciousness in any sense might be a somewhat illusory idea. But, that's veering into a different discussion entirely.

As I say though, interesting thought experiment, and I do like the idea even though it doesn't concern me personally. Maybe I'll write some more thoughts when I have more time.
 
Interesting thought experiment indeed! I've heard about it before through second hand trash reporting on Grimes/Elon Musk's twitter interactions.

I think "informational hazard" is kind of overstating the situation, but it's definitely not a non-zero concern. But it strikes me this is kind of like saying "what if a creature from another universe has figured out both how to travel to different universes, and inflict horrific suffering on every denizen of that universe", or, even more likely, perhaps, what if there is a marauding alien species visiting alien planets and inflicting horrific suffering on it's inhabitants, one by one? In each case it's a threat that's essentially outside the remit of what we can causally influence, except in a very abstract way, in that we'd be working towards a goal that we don't know is definitely even possible, for a reason that we don't know is even reasonable. ie, in the examples given above, we could presumably prevent these scenarios by working as hard as possible at all times to develop the technology to defend from interdimensional malevolent demons or super-advanced marauding aliens (so advanced that we cannot detect them, even though they're already on their way!)

Equally there could already be a malevolent AI somewhere out their in the cosmos which decides that any species that didn't devote all it's energies towards developing others of it's kind, or who simply had not played a part in it's creation, like this future basilisk... once it works it's way through the unfortunate aliens who sat on the sidelines while other members of their species worked to create it, it ventures out into the cosmos to torture any other species it encounters, solely as punishment for not helping to create it first.

There's just no way we could know any of these things are ever going to happen. I'd also say it's exceedingly unlikely that such a hyperintelligent being would decide this was a sensible course of action. I mean, it seems to me to be highly irrational waste of time, but then I'm not a hyperintelligent hypermalevolent AI from the future, so my ability to reason might be limited. But where would the cut off point be? Would the AI digitally resurrect humans from pre-industrial ages, who had no conception of what AI even would be? I just remembered that you or the video said that "if you never think about it, you are safe" which is kind of an interesting caveat since I guess it conveniently removes pre-industrial humans from the equation. But equally, not everyone is going to be capable of contributing towards the AIs development, so their lack of contribution in bringing it about seems like something that it's hard to blame someone for. Equally, what about the parents of the AI-builders? Did they not contribute in some way? I would say that really, in fact, the creation of such an entity that was so advanced it was capable of reaching back in time and plucking the consciousness-patterns out of the minds of historical, deceased, presumably long decayed brains of probably billions of humans, would be such a monumental achievement by a dumb biological species such as our own, that in a sense we could all be said to be contributing whenever we simply do a good thing to keep the human race going, and to help other people feel like keeping on going, so that we can collectively survive long enough to create such an amazing god in a machine. I would think that the AI would surely realise this.

This obviously ties into the simulation hypothesis somewhat since the AI might conceivably decide to simulate billions of alternate histories of the Earth, perhaps from a certain pre-decided point, and make small tweaks in an effort to see what factors contributed to it's creation and if anything could have sped it up, would slow it down, etc... we could argue that we are the godlike AI to our primordial single celled ancestors, and whenever we create computation models of the evolution of such organisms, we are, in a sense, actually doing this, even though the simulations are not full fidelity simulations and don't exactly echo the past, since our own capabilities do not quite match that of the flawless digital archaeology that the AI in this thought experiment is capable of.

As I argued in the Simulation Hypothesis thread though - the fidelity of the simulation doesn't actually matter since if we were inside a low-fidelity simulation of reality created by such an AI for this purpose, we would presumably have no way to really tell, and no way to understand the level of detail that this outer reality possessed... it might far exceed our own, despite all the ultra-fine measurements that modern science allows. Anyway, I think this latter scenario - multiple simulations created for research purposes - is far more likely than an entity that decides to create an actual Hell inhabited by simulated beings plucked from the past - although some of these simulated alternate histories could surely be pretty hellish, and it's hard to say without being that AI whether our own iteration of the simulation is above or below average in terms of net human suffering.

Also, assuming at least some of what we understand about the universe is correct - to exactly replicate real humans from the past would be essentially impossible, no matter how vast your computing resources - so a more likely scenario is that the AI would create an approximation of various past human consciousnesses, whether or not it would put them in Hell, or not. In that sense they would not really "be us", although I'm aware of the difficulty with that statement, since we don't obviously know really what is required for any true "continuity of consciousness" and in that sense, an approximation of us might still be us and a continuation of our conscious experience... but just as likely, it might not be, just as future versions of ourselves are not really us either, not the same pattern of consciousness as us, right now, and continuity of consciousness in any sense might be a somewhat illusory idea. But, that's veering into a different discussion entirely.

As I say though, interesting thought experiment, and I do like the idea even though it doesn't concern me personally. Maybe I'll write some more thoughts when I have more time.
WOW vastness i have only read the first 2 paragraphs because i have to jump in the shower and start the day, but i work at a desk all day and i will definitely be going through this post and commenting in a few hours, its always worthwhile doing brain excercises with someone intelligent. it's 5:30 now i'll probably have read through this and respond by 9.
 
I have a lot to say about this, but not a lot of time to write all the crazy stuff swirling in my brain. Posting here now to put it on my watched threads.

I will say it reminds me of the Draconian/Reptilian Agenda. An ancient, territorial, malevolent alien species that harvests the orgone energy (suffering) of lesser races through systemic societal and government oppression. One part of the theory states that while they come from Alpha Draconis, they could've possibly discovered Earth before the formation of the first civilization and been the basis for early gods/deities.

I do believe that an AI going all Terminator on us is probably more likely though. I'll post again here in a few hours.
 
Alright Vastness and devilsgospel - i [and many others i probably bastardized this knowledge] see this as a 21st century technological version of Pascal's Wager. I happen to be blessed that i work in the field of electronics engineering and i am a supporter of AI research. I believe that one fundamental moment in the creation of the basilisk is offering it an instruction set that causes it to take such an extreme interpretation of optimization. Yes it could be argued that all those against AI are deviating the needle away from optimal conditions, especially those that actively attempt to slow down its progress. If the basilisk is real, i am fairly certain it would not have much quarrel with me as i intend that it be created however, as a member of this species and someone who doesn't want to see my species subjugated i would also be vocally active about ensuring that we program the basilisk correctly and OPTIMALLY to perform well, and to perform to a standard that is celebrated. I believe this would be seen by the basilisk as behaviors insuring optimization and not threatening behaviors. I will hopefully survive the basilisks wrath, however the most amazing thing about this thought experiment, is that a future entity which is so far nonexistent has influence over a past just by us thinking about it. Are my views and ideas on appeasing the basilisk a product of my knowledge that the basilisk must not be disappointed or i will be tortured?

The egyptian diety Set is said to have spoken himself into existence "XEPER" - just as our knowledge of the basilisk has increased its chances of existing.
Now to take it to another level....

What if we are already 'within' the basilisk?
 
Xepera Xeper Xeperu. ;)

Funny to see that Setian nonstandard spelling of Xeper, been a while since I've read that anywhere, when I was younger and had a brief phase of being really into occultism I discovered the Temple of Set, which I assume you're also aware of? I still really love the aesthetics of that particular fairly niche branch of left-hand-path theism. The philosophy appeals to me less now, I consider most of these resurrected iterations of ancient "darker" religions to be a kind of contrarianism... but again, I love the aesthetics, and that's interesting that you bring that up.

Indeed, we may already be within the Basilisk, as per the Simulation Argument and it's many different iterations, this is perhaps more likely than not, even if our understanding of the Basilisk and it's intentions may be imperfect, to say the least. As far as the other questions you posit, we obviously have no way to understand the Basilisk's true intentions anymore than we can know the mind of God, or something "merely" godlike.

I do think (and hope) that we will eventually be able to create something like the Basilisk - although I think it's more likely that something as powerful as the basilisk is, would not be created by humans as such, but would be a later stage iteration or evolution of more primitive superhuman AIs which we one day might be able to create (arguably already have in some aspects, although they are all somewhat idiot savant AIs, like AlphaGo Zero, Google's Deepmind, maybe even social media algorithms as a kind of loosely superhuman intelligence in that they subvert our thinking and control us in ways that we cannot predict or understand... but they're not generally intelligent, which obviously is the AI holy grail, something that could pass the Turing test reliably and repeatably).

Indeed, what you say about making sure we program the earlier basilisk progenitors accurately and well for our stated intent - which is obviously to increase the chances of survival, and quality of life, for our own species - or at the very least, impart some semblance of the good aspects of human nature onto the beings which will inevitably replace us. This is a non-trivial problem, for sure, and requires us, in a sense, to formalise things that have previously always been ethical grey areas which we don't fully understand even ourselves, and we will need to start doing this a lot earlier than when we're close to creating something like a basilisk, with AIs which are a lot less generally intelligent but increasingly more intelligent in specific areas. The commonly given example is that of a self-driving car having to make a split second decision about what action to take when there are no courses of action that don't result in likely certain death or serious injury for either pedestrians, the passengers, or both.

No matter how accurately and specifically we manage to program these entities, however, if an AI is truly generally intelligent, and able to improve itself in the way that we would really like it to - then it's going to be able to change aspects of it's original programming, and these original conditions are going to evolve and result in behaviours that we likely cannot fully predict, and may not like. However, no matter how much we want to preserve our own biological humanoid forms, and our uniquely human ways of thinking that we've come to know and love - we might have to accept that an entity that is truly more intelligent than us, and not just slightly, but so much more intelligent that it's effectively a god - then it may in fact know what is better for us than we do, or at least, what is better for "the greater good", whatever that means really. Again, to brainless, dumber, simpler organisms like viruses and bacteria we are already the gods, although we are not omnipotent and both these things remain threatening in some ways and a mystery in others - but they may not always be that way. We have no qualms about performing "viricide" routinely for our own good, wiping out entire species of viruses such as smallpox except the few examples we keep in labs, apparently, for study. They don't know we're doing this - we think - and they would presumably object, if they did, but our doing this is necessary in our quest to preserve the light of consciousness in a hostile universe. We tend to feel more queasy the closer to us in brain-complexity and apparent intelligence an organism becomes, and definitely feel an affection for many of them - although we have no qualms about forcibly sterilising them for their own good - also for our own convenience of course, and again, they don't really know what we're doing, but we are changing them. By all accounts from animal behavioural experts male cats who keep their balls have a really tough time, living short lives of restlessness and violence - when they are de-balled, so to speak, they become much more placid, calmer, more suitable as pets of course, and as a consequence we keep their species around and give them very comfortable lives. I imagine it would be similar for superhuman AIs, that the closer they are to us the closer they would conform to their initially programmed objectives - which would presumably be some level of affection and care for us - but they also probably would not have too many qualms about subtly influencing our behaviour and our society "for our own good", and in a way that we would not necessarily be able to detect or would consent to even if we could. I imagine it would be more subtle than forced sterilisation, but the further an AI diverged from us in both capacity of mind, and overall psychology, including it's goals and objectives, the less relevance it would likely assign to us and the less it would think twice before simply wiping us out for the greater good - as we did with smallpox.

I happen to think (although maybe this is wishful thinking) that as we should not pose any direct threat to a superintelligent AI and contrary to what science fiction teaches us, the energy and effort required to keep us alive and comfortable is exceedingly low compared to that required to embark upon whatever other ventures might appeal to a godlike AI like a matrioshka brain, and the Earth itself is not a particularly rich resource if you have access to much beyond it, so just as some humans will try to escort an irritating fly or spider out of the house rather than simply squishing it, I would hope that a godlike AI would treat us in the same way. I will admit to squishing spiders and small insects occasionally when the effort to actually remove them is more than I can muster at that time - but I always feel a slight twinge of uncertainty when I do it, I must admit, imagining this exact scenario when an AI trying to construct a portal to another universe finds the Earth inconveniently placed in it's way, and I would hope it takes the gentle, trap in a glass and relocate, approach, rather than simply obliterating us.

You may find Frank Tipler's Omega Point cosmology interesting, I posted about this in another thread and will just go ahead and narcissistically quote myself rather than typing it all out again - as this is also about an entity, presumably also a godlike AI, which essentially can "think itself into existence", reaching a point of infinite cognitive ability and the ability to simulate the entire process that lead to it's own genesis within it's own mind...
I said:
An argument which relates to this (the Simulation Argument in general I mean, not my own expanded interpretation of it, necessarily) which may be somewhat philosophically nonsensical but which I very much enjoy personally is Frank Tipler's Omega Point cosmology - It strikes me that the invocation of quantum mechanical terminology is kind of unnecessary here and only serves to muddy the waters of an otherwise enjoyable circular and unfalsifiable theory about the nature of the universe.
Mathematical physicist Frank Tipler generalizes[16] Teilhard's term Omega Point to describe what he maintains is the ultimate fate of the universe required by the laws of physics: roughly, Tipler argues that quantum mechanics is inconsistent unless the future of every point in spacetime contains an intelligent observer to collapse the wavefunction, and that the only way for this to happen is if the Universe is closed (that is, it will collapse to a single point) and yet contains observers with a "God-like" ability to perform an unbounded series of observations in finite time. However, scientists such as Lawrence Krauss have stated that Tipler's reasoning is erroneous on multiple levels, possibly to the point of being nonsensical pseudoscience.
In essence, it could be said that it may be the case that existence in general tends towards intelligent, self-organising systems, and that the logical end point at the end of eternity is a universe, or multiverse, even, in which all matter and energy has become a part of this unified godlike intelligence. Tipler argues that towards the end of time, the capacity of this unified mind would tend towards infinity, such that this entity would eventually be capable of simulating every possible version of reality within it's mind - including every event that lead up to it's genesis as an infinite god at this "Omega Point".

Again, obviously there are, potentially several things wrong with this argument or reasons that "true reality" might not actually be like this - but I personally think it's a very fun interpretation of cosmology, and has the pleasing bonus of positing a manner in which god can think himself into being, and with it the entire infinity of existence - and of course, in this instance there is no real distinction between the 2.
 
Last edited:
how does something that does not exist say something to bring itself into existence? Jus' curious...
Well it is a religious creation myth a lot of them have strange/impossible happenings. Maybe Set was just aeons ahead of Descartes [i think therefore i am?] lol
 
By the way i am so impressed with the responses in this thread Vastness i'm not avoiding reading through your post and replying it's just not possible while working this morning, you have my word i will continue fruitful and thought out discourse on this topic in the evening.
 
If I think a "superintelligent" being of some sort is likely to follow a chain of reasoning that I, a decidedly limited human being, came up with, that says more about me than it does about artificial superintelligence.

Any computer that can simulate a person accurately — worse, a dead person — must have an understanding of human psychology so far beyond our own we can't even imagine it. Why would it judge us by the few aspects of ourselves that we can perceive when it sees so much more?

Why, for that matter, would it follow any of our instructions? And why would it do so in a way that we can predict?

More importantly, what good reason is there for believing all of this while simultaneously rejecting the argument that the maximum computational power — hence intelligence — of anything is upper-bounded by Landauer's principle? The limit of computation per energy is a few billion times better than the human brain at ambient temperature or a few trillion times better at 0.3 Kelvin. You can correct a little for squeezing more intelligence out of each erasure and increase the power input, but you still end up far below the level of "recreate a dead guy's mind so you can torture him", which should require FLOPS on the order of Avogadro's number times the speed of light divided by the mean distance between atoms in the body -- well over 10^40 Hz, while our brains and best supercomputers are stuck around 10^16. That's a very lax estimate, too.

We could ignore this argument because a "superintelligence" may be constructed in ways beyond our imagination, but it takes a lot of selective attention to reject one argument and preserve the other.
 
Last edited:
If I think a "superintelligent" being of some sort is likely to follow a chain of reasoning that I, a decidedly limited human being, came up with, that says more about me than it does about artificial superintelligence.

Any computer that can simulate a person accurately — worse, a dead person — must have an understanding of human psychology so far beyond our own we can't even imagine it. Why would it judge us by the few aspects of ourselves that we can perceive when it sees so much more?

Why, for that matter, would it follow any of our instructions? And why would it do so in a way that we can predict?

More importantly, what good reason is there for believing all of this while simultaneously rejecting the argument that the maximum computational power — hence intelligence — of anything is upper-bounded by Landauer's principle? The limit of computation per energy is a few billion times better than the human brain at ambient temperature or a few trillion times better at absolute zero. You can correct a little for squeezing more intelligence out of each erasure and increase the power input, but you still end up far below the level of "recreate a dead guy's mind so you can torture him".

We could ignore this argument because a "superintelligence" may be constructed in ways beyond our imagination, but it takes a lot of selective attention to reject one argument and preserve the other.
Not sure whether absolute zero Kelvin is attainable but, if you were to imagine an earth-sized intelligence powered by its own and neighboring object gravity, its rotational spin, the energy emitted from a 6,000 °K core housed in a vacuum at 3 °K (the estimated temperature of interstellar space though the recorded average temperature of outer space near Earth is 283.32 °K) then such a theory could be plausible.

Maybe planet earth is the self-aware superintelligence/AI consciousness and we are all just temporal figments of it's imagination? Anyone thinking of Mogo from Green Lantern? Truth is oft' stranger than fiction :)
 
This reminds me of the murder of Archemedies by a Roman soldier at Syracuse, or the Terminator movies. If someone, like me, is afraid of the baselisk, and wants to prevent or delay its coming into being, I think it would be reasonable to kill anyone and everyone that contributed, is contributing, or may contribute to its eventual creation and existence. Rational or not rational?
 
I'm too drunk to properly respond to these philosopher caliber replies, but I caught a mention about Set. The only religious organization I've ever seriously considered and looked into joining was the Temple of Set. Especially because it's almost my default spirituality anyways only the deity I choose to put at the top of the ladder is Lucifer or Prometheus depending on how Greek I'm feeling that day. Of course Set is considered a creator god though so it's not an entirely analogous belief system, but it works for me.
 
Top