The debate I am referring to here is not the one about how dangerous AI might be. Rather, I am interested here in the question of whether AI “thinks”–which leads in some instances to the further question of whether it is conscious.
I am not going to try to weigh in on these issues myself in this post. Instead, I am just pointing you toward three recent interventions in those issues that I found illuminating.
First, a two part discussion (long, but very careful and clear) of whether AI “thinks and understands” from Andy Masley. He also promises a third part (which I am eagerly awaiting) to this series in which he will get into the nitty gritty of how AI (as currently constructed) actually works. Here’s the link to the first two parts (at the end of part one you get the link to take you on to part two). https://open.substack.com/pub/andymasley/p/why-i-believe-current-ai-models-can?utm_campaign=post-expanded-share&utm_medium=web
Second, the humanist argument against AI as offered by Timothy Burke. Note how Burke combines an argument (that thinking depends on embodiment) and then segues into condemnation of what AI does to human relationships. So we get both skepticism about AI thinking and a jeremiad against its ill effects (since it is still doing something even if it isn’t thinking.)
I am giving you the link to Burke’s piece here along with about 3/4s of his post. (Apologies for the bad formatting which this tech idiot could not fix. Use the link for a more reader-friendly experience.
https://timothyburke.substack.com/p/academia-how-thought-happens
| Academia: How Thought Happens Friday’s Child Is Not Someone Else’s Inefficiency Timothy Burke Oct 9 I’ve thought about thinking a lot in this last year. And more than ever, I’m sure that thinking happens through the small stuff, in the details, in the routine. As I wrote recently in this newsletter, the work I’d like to be free of is work I want everyone to be free of, even AI. It’s work that I don’t think needs to be done at all, by anyone. It’s work that we are doing because it’s become someone’s job to make work for other people as their own “proof of life”, a demonstration of that person’s value to an organization. (This is one of Graeber’s central points in Bullshit Jobs and The Utopia of Rules.) The work that really needs doing is the work that is the predicate of thought. Profundity grows in the soil of the ordinary. Thinking, whether fast or slow in Daniel Kahneman’s memorable phrasing, is embodied and experiential. It happens in context, it is reactive, it is material. Thinking is an emergent process that requires a constrained topography to produce structures, outcomes, complexities, insights. The Claude Code advertisement and a fair amount of other AI propagandizing conjures up a utopia for brains-in-jars, at last free of the distractions of worldly life. It is an absurd kind of late Cartesianism run amok. I don’t know whether I would feel worse if the leaders of Big Tech really meant it or they’re just like Richard Daystrom, grasping for a rationale that will get Starfleet to buy what they’re selling. Brains-in-jars is bodies-in-vats, from a movie that I know the Big Tech elite have seen: The Matrix. And it bothers me more than a little that I have to wonder whether some of them looked at endless rows of senseless flesh powering a vast machine and said to themselves, “Yes”. Because I have been thinking about thinking, I have, like many academic humanists of the generation that became scholars through, with and around poststructuralist and postmodernist theory, been both discomforted and bemused by the uncanny matching between some of the central tenets of that kind of theory and the operations of generative AI. Emily Eakin’s wonderful new book The Frenchmen toggles between a personal memoir of being enchanted by Theory (as so many of us were in college in the 1980s) and the intertwined personal lives of some of the major French superstars who created Theory. Eakin does a marvelous job of uncovering that the French theorists themselves wanted to know more about the personal lives and feelings of their peers, that in lived experience the author-function was no more dead for them than it was for anyone else. One of the insights she brings into our present moment is even more important, however. For all that it eventually became common to bash postmodernist and poststructuralist thought as anti-scientific, as patently untrue and purely ideological, it turns out to have been non-trivially correct in a purely empirical sense about one of its deepest tenets. As Eakin ably sums it up, Theory from its structuralist roots onward came to see human beings as trapped in language and textuality, that there was nothing knowably outside of and beyond language. That didn’t mean that a black hole isn’t real or that a gene is just a word, but it meant that the reality of such things was only knowable through and within language, and thus was always subject to the effects of language. We are produced through language and are never outside of it. Which in many ways describes how generative AI actually does what it does in technical terms. And it offers an explanation of its power: because a global civilization operates at vast scales and through many complex systems that depend upon language and that use language to command, order and act upon human lives that generative AI has the potential to do so much to us, through us and perhaps for us. For me, at least, this unexpected proof of some of the insights of Theory is not a very welcome thing. I’m at least amused that a long-standing and unthoughtful ideological disdain for structuralism and postmodernism has suddenly run so hard into an unexpected reality. Back when I was more engaged with Theory I understood the point that we can’t think without language, can’t know the real world except through language, but I have always felt as if I think prior to and outside of language. Moreover, cognitive science has in many ways provided a good deal of evidence that thought and consciousness are in many ways pre-linguistic or extra-linguistic. A two-dimensional being thrown up into a three-dimensional space, as per the book Flatland, might not have the language for describing what “up” and “down” are once they return to their two-dimensional world, but they’d still know that there is a third dimension. That feeling that thought is before and outside of language is suddenly more important for me, and I suspect connects me in new ways to at least some of those who sourly complained of what they saw of Theory’s acceptance of a “post-truth” perspective. I am stubbornly sure that thinking in a mind that is in a brain that is in a body that is in a material world is not just or merely the operations of language, and thus that Claude is not thinking as I do and cannot substitute for my experience of thought, nor for the consequences of my thinking for me and for the world around me. Which is again why I don’t want to be relieved of the burden of small tasks, of the processes of thinking in the routines of everyday life, in order to become the proverbial big-brained mutant so beloved in science fiction and comic-books, just doing deep thinking liberated from meatspace, unshackled from process. I recently attended an interesting conference about liberal education and AI where I was irritated by one speaker’s enthusiastic embrace of AI. In particular, the speaker said, it let them prepare 25 slides for a presentation to their faculty that they otherwise would not have not have had the time to prepare. Perhaps the point here is not to have something to handle the task for which you have no time but instead to reconsider whether that task is worth anybody’s time. If an administrator doesn’t have to think about the faculty they’re talking to and think about what needs saying to them, then they have nothing to offer that is worth hearing. That’s because a moment like that is not just about what’s in the language of slides or a speech or a written communique. The speech or training that someone in charge has decided you have to hear that is empty of thought is the speech of liability protection, the speech of covering your ass. It is the speech of hierarchy and power: you have to hear what I have to say, but I don’t have to think about what I say or the thoughts of those who hear it. That’s a problem in any workplace, any social setting, but especially in an academic institution. It runs counter to what is supposed to be happening in classrooms, in scholarship, in the central mission of the institution. That’s one reason that academic leaders like Santiago Schnell, the provost of Dartmouth University, run into so much criticism when it comes out that they’ve been using AI to write talks they’re giving to audiences on their own campuses and beyond. It’s not just hypocrisy, though that’s part of the problem. While poststructuralists may have been right that we are always already entangled through and within language, it turned out that many of us valued the “author-function” even if we understood it to be a fiction of sorts. Every time a hue and cry goes up among fans and audiences when they come to believe that a writer or creator has used AI, a major part of the reason is that audiences have a parasocial relationship to the creators as they imagine them to be. To learn that AI is doing the work that you believed was being thought through the mind of another person is to discover you do not have that parasocial relationship. You are not in a mediated relationship to another mind, to the revealed thought of another person turned into words or images. Your sense of relation is to a thing when you took it to be to a person, which is an interesting inversion of Marx’s understanding of commodity fetishism, whereby things accumulate meanings in the process of their social character being obscured. Here the author-function doesn’t just lose meanings that audiences valued, it becomes a source of horror, the crawling sense that behind what you took to be a human face is an alien doppleganger, a mere tool, a soul that’s been snatched. When you find out that Schnell’s words are just the product of an AI, you find out that for Schnell you are an inconvenient friction in his day, a gap in his efficiency. You are not a member of his community or a fellow professional. The parasocial relation we have to authors, and the thought of authors, is built around being cared for and cared about, about being deemed worthy of being written to, communicated with. That might be a fiction, but it is a fiction that has mattered to many of us as readers, as citizens, as workers. Here, unfortunately, we discover that objecting to something that we see as the consequence of generative AI is not in fact a result of generative AI as such, any more than the table described by Marx is “evolving out of its wooden brain grotesque ideas”. We discover in fact the materialized reality of the thought of people who let generative AI write and speak for them, that we in fact are just something to speak at, not with, that speech is just encoding a relation between a subject and their muted objects. The tool used by a subject with the license to use it makes any audience into things, deliverables, ordered and managed properly. That user does not get back their mornings to think more deeply, they are relieved of the need to think at all. Which is why the rest of us need to keep struggling through the dense ordeal of the small materialities of everyday life and the everyday routines of labor. The headwaters of thought are in that ordeal, and from them flow human relationships, both real and aspired towards, both the relations we imagine we want and those we hope to overcome. A little help here and there with tasks that are more difficult than they need to be in the infrastructures of the previous world is fine, but the “greater things” we want, the deeper thoughts we crave, will be found only in our bodies, our minds, and our language. No tool can tell us what matters, and the use of some tools in some circumstances tells us one unmistakeable thing only, which is that the user of the tool believes that we the reader, the listener, the audience do not in fact matter. I don’t take kindly to being told that, and neither should you. |
Finally, a clever rendering of the various statements that crop up repeatedly from those who deny that AI thinks or ever could think. A useful overview of the debate, along with a reminder that an assertion is not an argument.
From:Henry Shevlin from Polytropolis <polytropolis@substack.com> The link will get you to the full post, including the graphic of the Bingo card
| Bad AI Consciousness Takes Bingo Duty calls: shaming 24 bad arguments against AI consciousness Henry Shevlin Oct 10 NB: this post is currently a bit rough, because I’m writing it at 30,000 feet and I’m trying to catch the eye of the flight attendant to get another bag of the cream cheese and chive pretzels. Expect a finessed version of this some point in the next few days with more hyperlinks and fewer enthymemes. To my great surprise, the AI consciousness debate is getting HOT right now. You have to understand how strange this is for me. I’ve been working on consciousness for most of my adult life, and AI consciousness for over a decade. For most of that time, AI consciousness was a respectable but fairly kooky debate in which most of the common examples came from science fiction. But now it’s everywhere! Newspapers, TV, even papal encyclicals. Which of my niche interests is next? Should I expect congressional hearings on Baldur’s Gate 3 Honour Mode strategies? As fun as this has been, it’s also given rise to a positive cavalcade of terrible takes, including from some people I otherwise think of as smart and thoughtful, which has led to me dropping some tweets that, if not quite acerbic, could only be described as uncharacteristically arch. To be clear, I don’t think that I have all the answers here, or even most of them. There are plenty of people on all sides of the debate I respect, and while I skew towards “moderate liberalism” (AI consciousness is possible, current models are probably not conscious, but we shouldn’t be too complacent about that), I have big error bars in all directions. But if people are going to weigh in on these issues – as I absolutely think they can and should – they should at least do some basic homework or show a little bit of epistemic humility, and ideally both, because right now I’m experiencing physical pain multiple times a day scrolling Twitter from seeing hot takes that are unreflective, uninformed, and often very crude (in the full sense of that term).With that in mind, I decided to put on the yoke of necessity and do something about it. Here, inspired by Rob Bensinger’s sublime Bad Alignment Takes Bingo, is the Bad AI Consciousness Takes bingo (I won’t name names, but several Local Area Posters have already won). I would stress that this list is far from exhaustive, and although it focuses on the anti-AI consciousness side here, there are bad arguments for AI consciousness too (eg, the direct inference from intelligence to consciousness, or accepting multiple realisability as a datum without question). The main difference is that almost everyone on the pro- side I’ve seen so far is saying things like “this is a really important issue and we should take it seriously”, whereas a disproportionate number of people on the anti-side (including some big names and big accounts) are adamant the topic is dumb and stupid and their opponents are obviously wrong. However, should the pro-side start engaging in similar epistemic misbehaviour, rest assured I have another bingo card in me. Finally, I’d also really emphasise that many of the lines below could be absolutely fine as conclusions of a wider argument or theory. For example, there are some reasonable arguments connecting life and consciousness; I’m not convinced by them, but I have great respect for some of their advocates. The problem is when people assert those conclusions as argument-terminating assumptions.Anyway, here’s the bingo and the breakdown, have fun and don’t take it too seriously. 1. Nothing but…Master rebuttal. Everything can be described in lots of ways, without excluding other ways. Humans are clouds of atoms, biochemical complexes, social primates, and doctors, lawyers, and parents, and all of those are mutually compatible. “It’s just programmed by humans.” LLMs are not programmed in the sense relevant here; it’s more instructive to say they’re grown. Engineers choose the data, architecture and training objectives, but the ultimate features of the model emerge in complex and often unexpected ways (cf Lovelace’s objection in Turing 1950). If being shaped by an outside process ruled out consciousness, natural selection and your parents would have some explaining to do. “It’s just matrix multiplication.” And you’re just sodium and potassium ions traversing membranes. Every interesting physical system is “just” something at implementational levels of description (see Marr 1982). The question is what other capacities emerge from these lower level traits. (Also, it’s not even technically correct; LLMs have non-linear algorithmic components too). “It’s just predicting the next word.” Next-word prediction is the training objective for base models, but that doesn’t tell you HOW the system is achieving the training objective. Prediction is also a deep cognitive capability; one of the leading theories of human cognition characterises us as predictive engines. Oh, and it’s also untrue — post-training adds other objectives, and LLMs have been found planning rhymes a line ahead, not just “one word at a time”. “It’s just copying human behaviour.” LLMs are anthropomimetic, to be sure — they resemble us in terms of behaviour. But that doesn’t mean their behaviour doesn’t have some of the same underlying mechanisms; an understudy may learn by copying a dancer’s steps, but at the end she’s really dancing. Whether the mechanisms are the same is an open question, and “it’s copying” restates it without answering it. “It’s just playing a character.” This is a more interesting objection, as LLMs do seem to have personas. But it’s not clear whether these are more like an actor playing a role or a human switching between social personas in different settings. Even if it’s the former, actors are conscious. And recent interpretability work treats the network as an author simulating the Assistant rather than being it, which means the thing that might feel and the thing that acts needn’t be the same. The author could be the moral patient and the character the moral agent. 2. Missing ingredients Master rebuttal. No-one knows what consciousness is or what it requires, and people have suggested a wide array of ingredients over the years. All of these takes propose something LLMs lack and assume consciousness needs it. Some of those assumptions may turn out right, but as slogans they’re just presupposing the theory they need to argue for. “It’s not alive.” Bacteria and fungi are alive and almost certainly not conscious, so life is at best a necessary condition. There are some serious theories linking consciousness more or less closely to life through processes like metabolism, homeostasis or self-maintenance, and they deserve engagement. But you need to make the effort to actually embed your claim in these frameworks! “It doesn’t have a body.” Our bodies do very little while we’re dreaming, yet we still have rich conscious experiences. People with locked-in syndrome have almost no bodily agency but full conscious lives. Embodiment theories are again real contenders, but you need to argue for the claim that consciousness requires embodiment rather than assuming it (we also have various forms of embodied LLM already, of course). “It doesn’t have feelings.” This is the precise point at issue. Just because you say it loudly and/or angrily doesn’t make it true. See also proof by assertion. “It doesn’t have free will.” Plenty of philosophers and neuroscientists think humans lack free will (Robert Sapolsky had a recent popular book about it). They vary on what this means for things like responsibility, but none of them concludes it means we’re unconscious. Consciousness and free will are different properties. “It’s frozen until you prompt it.” Consumer LLMs remain passive until prompted because of specific design choices, but there are lots of ways of giving LLMs the ability to run continuously and autonomously without user input (see eg OpenClaw). More broadly, while this may be relevant for the temporal structure of consciousness, you need an argument if you think only certain temporal structures create consciousness. “A simulation of a hurricane doesn’t get anyone wet.” But a simulation of a calculator does arithmetic, and digital money has value. Some properties survive simulation and some don’t. If functionalism is true, consciousness is more like money, defined by its causal role. You may disagree — but again, you have to give an argument rather than just restating your position. 3. False analogies Master rebuttal. Analogies between two things only work if they share relevant features. This is why people laugh at the famous “You wouldn’t steal a car…” advert against piracy, because there are a number of important differences between conducting a carjacking vs a teenager downloading Nothing_Else_Matters(2).ogg. Many of these arguments are similarly poor analogies. “Do you think your iPhone is conscious?” Panpsychists and pancomputationalists actually do, but setting them aside, most people don’t. But most people also don’t think sea sponges or bacteria are conscious. Clearly there are other factors besides “is it biological or artificial” that are relevant here, most obviously complexity (The Microsoft Word variant is worse, since a document is a static record and nothing in it computes anything.) “It’s like thinking there’s a tiny band inside the radio.” Radios work because bands make music in a studio and it gets broadcast and radios pick it up. LLMs have no studio equivalent. Their outputs are produced by the computation happening inside the system, so if anything the model is the band. (Whether the band feels anything is the question.) “It can’t even count the r’s in strawberry.” This is years out of date, so immediately indicates a familiarity issue. There’s also a straightforward reason LLMs struggled with this, namely that they read text in chunks called tokens rather than letters. More to the point, capability and consciousness come apart. Toddlers and cats can’t count the r’s in strawberry either. 4. Evidence games Master rebuttal. These moves all involve setting an implausibly high evidential threshold to attributions of consciousness (that would also exclude animals, infants, or other people) or otherwise shutting down the discussion with vague epistemological scrambling. “We don’t even know what consciousness is.” True! It’s the hardest problem in science. But surely that should be a reason for open-mindedness and epistemic humility! We still don’t know which animals are conscious. And as result it seems at least conceivably that we might build consciousness without knowing it. Of course, it: fine to use this line if you’d like to remain agnostic, but you can’t use it to license a firm verdict. “There’s no evidence it’s conscious.” It depends on what you mean by evidence. Direct evidence, no, but you have no direct evidence that anyone but you is conscious. Instead we rely on indirect cues — other people are made like me and act like me, so they’re mostly in. Animals are made like me but they don’t act much like me. LLMs are in a weird zone where their substrate and parts of their psychological structure are nothing like us, but a lot amount of their behaviour is startling similar. This makes the evidence mixed, and mixed evidence is still evidence. The 2023 indicators report is still the gold standard here, and while it found no current system meeting the indicators, it also suggested there were no obvious technical barrier to future ones. “It only says that because it was trained to.” In most cases, no. Models are trained for personalities and dispositions (such as being honest, helpful, and harmless) not canned answers. However, there are some exceptions for sensitive topics, including consciousness; whereas base models will happily claim to be conscious, most consumer LLMs are specifically trained to deny or hedge the issue. “No possible output could convince me.” Argument from incredulity, do not pass go, do not collect epistemic credit. If nothing could convince you, why are you even taking part in a discussion? And your own incredulity has no epistemic value here. Some people find it impossible to believe god exists. Some people find it impossible to believe god doesn’t. And plenty of people find it easy to imagine that LLMs are conscious (notably the godfather of deep learning himself). “I work in ML. Trust me, there’s nothing there.” The tech world itself is deeply divided on this. It’s like saying “trust me, I’m an American, George W Bush was a great president.” Additionally, technical ML knowledge is only so relevant to these debates; knowing how to train a model doesn’t tell you how its features map on to cognition and representation (that’s mechinterp), and even that doesn’t settle how they map onto rich psychological capacities like consciousness. 5. Not about minds at all Master rebuttal. This last set of takes isn’t object-level arguments at all, but claims about motives, risks or the mental health of the people asking. Many of them are straightforwardly Bulverism, and tip over the line from bad arguments to antisocial behaviour. Either way, none of them bear on whether the system is conscious. “It’s just anthropomorphism.” Anthropomorphism might explain why people might hold a belief, but it doesn’t show the belief is false. And there are strong anthropocentric tendencies that cut the other way, leading us to ignore potential evidence for mentality when it comes in exotic forms, especially when it would challenge our sense of our specialness (the animal cognition world has been dealing with this for centuries). “It’s just hype.” The supposed incentive points both ways at once. Critics say labs talk up consciousness to sell products, and also that they will use it to dodge liability. But labs are wildly divided on this. Senior Microsoft figures have campaigned aggressively against even taking the AI consciousness question seriously, so “the companies all want this” isn’t even descriptively right. The question also long predates modern tech companies, and has been a favourite topic of philosophers for centuries (in the form of automata rather than AI, but same idea). “It’s a dangerous distraction.” Does discussing AI consciousness really distract us from other issues? We do not, in fact, have a norm that we should only discuss the most important thing in the world at any one time (at least that’s my impression, unless Taylor and Travis have a more world-historical role than I realised). I would also suggest that for alignment purposes, at least, AI consciousness might be more relevant than some think, with multiple papers now suggesting unanticipated negative effects associated with suppressing consciousness attributions. “So switching off my laptop is murder now?” This is like saying ‘If pigs were conscious, factory farming would be a moral catastrophe, so pigs aren’t conscious.’ That’s obviously silly, because inconvenient consequences don’t make a claim false. It’s also not clear that the murder analogy holds; there are big complex questions about where LLM identity begins and ends (is my ChatGPT the same individual across context windows?), and also questions about model details — a paused model with weights intact is quite different from a dead human. “That’s AI psychosis.” There are various cases of vulnerable users whose delusions emerge in conversation with chatbots, though whether there’s a causal link is hotly debated (compare the way hearing secret messages from the TV has been a common delusion in schizophrenia for decades). But applying the term to researchers who treat the question as open is contentless name-calling with a clinical label. Free squareThe Chinese Room. Look, Searle isn’t my favourite philosopher, but The Chinese Room is a real argument and I’ve taught it in grad and undergrad classes many times. But it deserves a special place on this list because it’s so often used as a debate-terminating cliché by people who read Minds, Brains, and Programs once back in the day and have decided they never need to read anything ever again. Shouting “Chinese Room!” doesn’t settle anything. It was controversial even at the time, has multiple replies considered by Searle and others (something like the “systems reply” seems obviously correct to me), and ultimately, it’s a vibes-based argument about a scientific issue — “wouldn’t it be SUPER WEIRD if a system like this was conscious?” For these reasons and more, in my subjective view it has not aged well, and notably none of the major contemporary scientific sceptics of AI consciousness lean on it. That said, this square is reserved for those who throw it out there and expect it to settle an argument by itself. Honourable mentions “It’s not a real person” (begging the question, personhood undefined) “It’s just a stochastic parrot” (variant of the copying line) “It’s just a tool” (we mostly use it as one, sure; but who says tools can’t be conscious?) “Machines don’t have souls” (the most honest one, since it states the real premise) “Remember Blake Lemoine?” and its sequel, “The Claude Delusion” (see psychosis entry) “This is just Effective Altruist cult stuff” (more ad hominems, and besides, I’m only EA adjacent) “Which one’s conscious, the weights or the chat?” (interesting and complex question, but the question of whether the consciousness is there is orthogonal to where consciousness would live) “It doesn’t have pain receptors” (you might be able to simulate pain, but more to the point, pain is only one type of consciousness) “It forgets you as soon as the chat ends” (so do some humans, perhaps most famously Clive Wearing, but they’re still conscious) “Penrose and Gödel proved it” (is the hope that by combining consciousness, quantum mechanics, and Gödel together we’ll create a mystery and confusion singularity that will solve everything?) “Just read the researchers” (links one paper they had ChatGPT explain to them) Okay, I got my pretzels, and they have Disclosure Day on the in-flight entertainment, so I’m cutting it here for now. Good luck, no cheating, |
