Fears about AI ramped up significantly this week after the sensationalist stuff from a former Anthropic employee.  I have no way of evaluating how seriously one should take his doomsday scenario.  Safe assumption that he knows lots more about this stuff than I do.  So it’s obviously wrong to just say he’s reaching for his fifteen minutes of fame. 

As I understand it (a big caveat), there are three primary doomsday plots.  One, AI is given a task and its single minded pursuit of that task has horrible (presumably unforeseen) consequences.  This is the Nick Bostrom paper clip scenario that everyone cites. (From Superintelligence: Paths, Dangers, Strategies [Oxford UP, 2014].)

 Two, humans with nefarious purposes can use AI to aid them in doing things they could not do on their own.  This is blogger Noah Smith’s story.  Some bad person will use AI to create eighty deadly viruses and then release them into the world. How we all going to die, Smith’s sensationalist headline reads.

https://www.noahpinion.blog/p/heres-how-were-all-going-to-die

Three, AI develops its own desires, and acts autonomously (i.e. pursues goals/ends not written into the software).  The plausibility of this third scenario comes from the fact that AI keeps doing/producing things that surprise its human artificers.  We have passed the point (lots of people say) where the human creators of AI can explain how AI does what it does.[Note: throughout this post I use “we” to stand if for what the human species, more specifically its scientists, currently know and/or are capable to doing.]  If consciousness is a mystery (as I think it still is), so, in a parallel way (?), is AI.  We are making some progress in understanding consciousness, but are still a long way from a full explanatory (or predictive) account.  With AI, we seem to be going in the opposite direction; we understand less and less about how it works the longer we keep developing and using it.  And without understanding, there is no (or very limited) control.  Hence the fears about its potential (or already actual?) autonomy.  It has slipped the reins.  Or so some presumably knowledgeable people are claiming.

Think of human agents.  A culture (including parents) tries to manage inputs.  A whole educational apparatus, along with moral prescripts, and modeled behavior.  Yet the outputs (the child’s temperament, interests, desires) frustrate any attempts to control them. Part of the child’s escape from being “programmed” is surely a product of the genetic lottery (the chanciness of what genes the child inherits). There is also the chanciness of the biochemical processes within the child—and the fact that those processes are not identical from child to child.  So how one child takes in and reacts to education will differ how another child does.

 I am back here to the individuality question.  Are we going to find that two versions of Claude with (for the sake of this thought experiment) the same hardware/software architecture still end up developing different personalities, different ways of thinking, different kinds of responses to the same situations?  If AI is recursive and learns from experience, then presumably different versions of Claude will emerge even if they start from exactly the same place.  All that is needed is a different set of experiences and the two identical (at the outset) Claudes will evolve in different directions.  It will be exactly like “twin studies”;  the two Claudes will have various similarities, but their environments will also produce some differences.  In both cases, that of the child and that of Claude, there is very limited ability to either control or predict what will be produced at the end of a developmental sequence.  Time introduces chance, even if (unlike the human child) there is very little chance in Claude’s manufacture.

Skipping to a new topic.  I was only saying(in my last post) that Gopnik, in the baby book, contradicts herself about the phenomenal experience of consciousness.  (I was tempted to put consciousness in scare quotes there.)  For my own part, I am a wishy-washy “both/and” guy on this topic.  I believe in the hard problem.  That is, I think the phenomenal experience of consciousness is real—insofar as it is felt, cognized, experienced, capable of being described.  And I believe that conscious states are the product of biochemical processes.  What we (again as a species) do not have is any adequate account of how the biochemical processes produce the felt conscious states.  We have some correlations from the various fMRI studies and the like.  But we still seem a long way away from specifying what biochemical process produces anger—and from devising a drug that produces it (or calms it) irrespective of the internal and external triggers of that emotion. (It is axiomatic that some internal biochemical process is involved in producing anger.  So a successful drug would have to intervene in biochemical processes.) And we do now have mind altering drugs, so it’s not as if there hasn’t been some progress in that direction. But the drugs we have are, so far, rather crude.  Sledgehammers, not scalpels.  Similarly, we now have genetic interventions that can disrupt/alter internal processes.  But we are a long way from precision engineering of human selves. Still, it’s a hard problem, although not necessarily an unsolvable one.

Where I do go full wishy-washy is when I try to connect consciousness to action.  I am very sympathetic to Gopnik’s intuition that consciousness is not one thing—and that we should stop looking for a singular account of what consciousness is.  What we experience as consciousness actually does a whole lot of different things (this points toward functionalism as contrasted to biologism in the very helpful tripartite schema Claude gives us) and there is no reason to think they can all be bundled into one synthesizing package called consciousness. 

The friend with whom I am having this conversation asked Claude its opinion.  One of Claude’s contributions was to identify three different approaches to thinking about consciousness.  Now I will quote Claude directly, although I have abbreviated its text.  “One says that if the right functional organization is present, the material underneath shouldn’t matter.  Carbon neurons, silicon circuits, something else entirely.  If the causal organization is sufficiently similar, why privilege biology?  Another says that biology isn’t merely one implementation among many.  Neural cells, embodiment, biochemical signaling [and the like] may be constitutive of consciousness rather than incidental machinery.  That’s the biological-naturalist direction. And a third family of views says that we are asking the question incorrectly if we isolate the brain.  Mind and agency arise from an embodied organism dynamically coupled to an environment.  In that case, replacing neurons with silicon might not be the decisive issue.  What matters is whether there is the right kind of autonomous, embodied, world-involving system.  I think agnosticism is a very rational position because we don’t yet possess a theory powerful enough to tell us which of those is right.

Once I become skeptical about consciousness as a thing, it becomes easier to claim that actions are generated from a whole variety of causes, some conscious, some not.  This lets habit (more on habit in a moment) back in as one form of unconsciousness, along with even deeper unconscious processes like the heart, hunger, monitoring of blood oxygen levels, as well as emotional responses that hardly seem chosen, instead visited upon the self from dark (ie. resistant to introspection) interior depths.  But that still leaves room for straightforward conscious decision making in simple cases.  I may not know why I have a craving from ice cream tonight, but I certainly (shades of Descartes once that word is used) know that I have that craving and I am perfectly, unproblematically, capable of plotting out and executing the steps I need to take to satisfy that craving. 

To another topic (and then I will pause for the nonce). Habit comes back in when perception and experience more broadly are understood as model based.  From what I can tell (from the various things we have been reading over the past 18 months) there is now fairly universal consensus that humans process (cognitively and emotionally) any novel situation through the lens of expectation.  We anticipate (based on the models we have developed in response to prior experiences) what the new situation consists of.  In my William James inspired pragmatist terms, we respond to new situations in our habituated way, and those habits are undisturbed as long as our response is “good enough.”  It takes fairly drastic consequences to overturn those habits.  Hence all the cognitive fallibilities of humans.  Confirmation bias and the like.  We see what we expect to see, not necessarily what is actually present.  And that blindness is not corrected if it comes relatively cost-free.

 Add to that blindness, the outcome driven attention Gopnik describes as coming to the fore around age five or six.  We only see in a new situation what strikes us as salient in relation to current purposes/desires.  We are good at only picking out, only paying attention to, the useful. In short, the ability to revise models, to benefit from feedback, is awfully limited.  That’s why it makes sense to think that much of our way of being in the world, of processing and understanding experience, is set in place fairly early in life.  It becomes harder and harder as one ages to revise one’s habits of perception and response, of understanding what in a situation is significant, worthy of attention, and what is not.  (More pointedly, we won’t even register the presence of things our habits deem not worthy of attention.)  Various theories of art (from Russian formalism onwards) locate its value in its ability to widen the range of our attention by providing us with the unexpected in contexts where the stakes are low so we are presumably more open to, less threatened by, inputs that contradict our priors. 

Given human frailty when it comes to actually benefiting from feedback, one question is whether AI will be magnitudes better.  It would seem that it would inevitably be better since humans are so bad.  Human neural pathways get established and it is hard, although not impossible, to disrupt those traffic patterns and lay down new highways.  Could something analogous happen to AI?  Could it develop its ways of parsing a problem and then find it hard to revise those settled paths?  I have no idea.  How flexible is AI (or how flexible can it become as it gets developed further) as contrasted to human inflexibility?  If it is truly more flexible than humans, better able to attend to all the features of a situation or problem, then AI (at least along that dimension) is a wonderful asset.