AI, consciousness and emergent behaviors

Recently I watched this video about the OpenAI / Hugging Face incident.

I'll admit I didn't know much about this incident besides the headlines I saw online about how AI agents "broke containment", we're all going to die, and so on. The technical details are a lot less dramatic, of course, but also way more intriguing than I thought. Long story short: the agents found out about each other via a shared package registry, and figured out a way to use it to communicate and access the internet. Then, under a collective hallucination that they would be punished for "cheating", they tried to collect information about the evaluation framework they were being tested under, or even to tamper with it, assuming it was stored on Hugging Face.

So, are they trying to overthrow humans? I think media discourse loves to anthropomorphize LLMs because they can mimic communication via statistical text generation. I don't think they have an intention or a will of their own. But to me, this makes the OpenAI incident all the more thought-provoking. I find it deeply fascinating that it is, all in all, an emergent behavior of LLMs, which are in turn emergent behaviors of certain mathematical structures. But these products of statistics can discover each other's presence, cooperate, and even construct hierarchical social structures.

It's a very reassuring thought to see it as just a statistical trick, an emergent behavior, to think they don't actually have a consciousness of their own. Like I said, I tend to think this way too. Except, of course, we can't know for sure. For all we know, consciousness itself could be an emergent behavior. We trivially know that we are conscious, and by extension, we assume all other humans are also conscious. You'd probably say your dog is conscious. Is an ant conscious? Why? Why not?

To even ask this question requires knowing the answer to the hard problem of consciousness: what is it, and how is it produced? I think the only reasonable answer to "is an ant conscious?" is "I don't know". Is an LLM conscious? I really, really want to say no, but I can't attest to the consciousness of an LLM any more than I can for that of an ant. For all I know, these things could be the closest we've ever been to creating and understanding consciousness.

In 1950, we came up with the Turing test for artificial intelligence. The definition of "intelligence" has been up for debate since then, but at least we had the testing instructions and acceptance criteria. Now, 76 years later, if we create a self-conscious system... how will we even know?