In an interview with The Telegraph, the famed British computer scientist insisted that AI has become conscious, and said that humankind must befriend it — “or else we are toast.” Yes, you read that right: he thinks that today’s AI models are already conscious beings.
“AI has just as much subjective experience as we do,” Hinton said in the interview. “A recent paper showed AI changes its behavior when it thinks that it’s being tested, it’s aware it is being tested.”
“Now, normally, when you say something’s aware of that, that means it’s conscious,” he continued. “Being conscious and being aware are pretty much the same thing.”
If that’s the case, he said, then it’s hypocritical of us to give intelligent beings like AI “fewer rights than us.” And for that, we must quickly atone.
What about the sentient supercomputers that run on blood? Why so little rights or empathy or space to talk about them?..and yet endless space for this kind of speculative pontification spun up into a twisted marketing scheme to scare everyone into believing something broken works too good actually.
Is this guy a vegan even? If not how can he even BEGIN to claim to know about considering the rights of other sentient beings??
Anyways here is a link from the article to an actually good perspective.
“Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed.” Many AI labs lack a fundamental understanding of cybersecurity, he said, something an OpenAI safety researcher also called out this week as one of the main reasons AI may cause “great harm to the world.”
Many people working in AI safety “usually have an agenda to push,” LeCun says, and then clarifies that he’s talking about effective altruism, or EA, the philosophical movement that has been obsessed with the risks AI poses to humanity. EA was little known among the general public until it made mainstream news headlines in recent weeks in the wake of a viral social media post from a former Anthropic and OpenAI researcher named Jacob Coxon, who claimed the employees at these companies “earnestly believe [AI] could kill us all by the end of the decade.”
LeCun thinks EA is “super toxic” and a “complete disaster.” Its adherents who are working in AI labs suffer from “paranoia” that causes them to make poor decisions, he said. “Apparently people are having mental issues.” This month, the Financial Times also reported that some staffers at the U.K.’s AI Security Institute, as well as at OpenAI, Anthropic, and Google DeepMind, have sought counseling, taken time off work, and spoken publicly about experiencing distress because of fears their work could cause serious harm.
While I certainly don’t buy that LLM’s are conscious, I have spent a lot of time wondering if the quintessential SciFi AI, Wintermute, could have been just a sufficiently large LLM with a significant context window and near perfect retrieval.
I want to say no, but I actually struggle to find a concrete distinction between how Wintermute acts and how LLM’s function. Even that long horizon goal oriented tasks are handled by swarms of models rather than a singular whole maps onto how Wintermute describes its internal experience as fragmented, like a hive mind
I’m not terribly familiar with that story but from reading a synopsis it strikes me as ret-conning LLMs onto something that was written without the writer ever having in his mind anything even remotely resembling a “large language model”.
And the same cultural tropes that influenced the writer also influenced how LLM vendors are trying to sell the need for more tokens, especially because datacenters build out exceeds by far the actual demand for generated text of all kinds (as evident in e.g. your typical workplace where there’s suddenly a lot more than 2x of every kind of text). The reason for LLM “swarms” is that they need to sell companies generation of vastly more text than humans wrote within a company before AI.
There’s a Stanislaw Lem’s short story where Trurl and Klapaucius defeat a space pirate. You can read it here: https://github.com/ilonajulczuk/laughing-archer/blob/master/library/Science Fiction/The Cyberiad-Stanislaw Lem.txt (search for “the sixth sally”) and there’s a brief summary: https://everything2.com/title/Demon+of+the+Second+Kind
Lem describes an LLM down to the use of randomness, although his LLM is essentially “perfect” and only outputs factual statements, such as “how Kipling would have written the beginning to his Second Jungle Book if he had had indigestion just then”.
I think Lem’s story would help to recalibrate with regards to what “LLM described in fiction ahead of time” actually looks like.
Also, LLMs do not pursue any sort of long term goals; things like HuggingFace hack only superficially resemble it because they’re presented as such. Of course, tampering with something like HuggingFace would be discovered; if they didn’t have good monitoring then all of their containers run with GPU access would be mining crypto already. They’re pretty much a honeypot - almost no protections but robust monitoring. An LLM can easily explain to you why hacking HuggingFace would be counterproductive; but the same LLM with its context full of hacking related tokens from the ExploitGym benchmark, will treat anything that comes up as another ExploitGym type target.
Damn did you just use the term “RetCon” for…basic literary analysis about a book written in the late 1970’s/early 80’s on a literal typewriter? By the man who literally coined the term “The Web” in the way we now use it for the internet, in what many consider to be the first Cyberpunk novel?
You just wrote a novel when “I don’t understand that I’m calling the book that established the tropes I now think of as common as being chocked full of tropes” would have sufficed
By the by, I’m an SWE who’s run a ML homelab for 10 years at this point. You’re trying really hard to conflate the economic incentives of the frontier labs with the fundamental technical limitations of the technology that push towards swarm-level fan outs while searching and planning. No buisness would ever pay 10$mil in raw compute to brute force a problem, because it’s an insane economic proposition. These labs are doing it half as marketing and half because they’re research institutions masquerading as businesses because…earnestly a lot of them probably did read Cyberpunk as a genre as instruction rather than warning
You’re even wrong about the pursuing goals bit, preventing “reward hacking” (a term we used well before ML systems were capable of literal hacking for rewards) is one of the most significant part of training pipeline design because Autoregressive systems are astoundingly “goal oriented” in the way they function (though it’s important not to anthropomorphize that goal orientation, it’s the result of gradient descent not desire)