Truth is, we have no idea when something is or is not conscious.
I am conscious. I was not at conception. When did my consciousness emerge? No idea.
I am conscious. My earliest ancestors likely were not. Which of my ancestors was the first to be conscious? No one knows.
Truth is probably that consciousness is not binary, you can have varying degrees of it. Also, it is incredibly difficult, if not impossible to properly define consciousness.
So how can we say that AI isn’t conscious and never will be? We can’t, at least not scientifically.
To be clear, I don’t believe LLMs are conscious either. But I would be extremely careful with the claim that AI categorically cannot be. Why would that be the case?
And if there is the chance that something might be even a little bit conscious, wouldn’t it be unethical to turture it? I think it is.
Intelligence is similarly hard to define as consciousness.
Turing considered machines intelligent if their communication becomes indistinguishable from that of a human. Most people seemed to agree with that idea, until LLMs started actually beating Turing tests. Since then, the goal post has been moving pretty much constantly. We come up with a new way of measuring intelligence and a year later LLMs beat it. If we take the standard way of measuring intelligence (Mensa IQ test), LLMs have surpassed most humans (source: https://trackingai.org/home).