GenAI is Fluent in Everything, but Faithful in Nothing

Why the machines hallucinate, why they have no worldview, and why truth has to come from somewhere else.

I'm going to say something that sounds like an insult and is meant as a description: large language models (all of them) hav never known a true thing. Not once. It doesn't know things at all. It is extraordinarily good at sounding like it does, which is a different skill, and most of our present confusion comes from mistaking the second for the first.

Here is what a language model actually does. It has read an enormous amount of text, and from that text it has learned, with real brilliance, what tends to come next. Give it some words and it predicts the words likely to follow. That is the whole trick. It is a magnificent trick - it gives us machines that write fluent prose in any voice on any subject - but look at what it optimizes for. It optimizes for plausible. It was never, at any point, optimizing for true. Truth was not in the objective. Plausibility was. And plausibility and truth often travel together, which is precisely why we confuse them - but they are not the same thing, and the gap between them is the whole story.

This is why these systems hallucinate," a word I dislike because it implies a malfunction. There is no malfunction. A model that invents a court case that never happened - complete with a docket number, plausible parties, and a tidy holding - is not broken. It is doing exactly what it was built to do: produce the most plausible continuation. A fake citation is plausible. It looks like the thousands of real ones the model has read. The machine has no way to prefer the real one, because it has no idea that real" is a category. It isn't lying, either. Lying requires knowing the truth and choosing against it, and the machine has never once been in a position to know.

Now the deeper point, the one that took me a long time to learn to say cleanly. Truth is not a property of language. You cannot find it inside a sentence by examining the sentence harder. Truth is a property of the relationship between a sentence and the world - between the words it is raining" and the actual sky. A statement is true when it corresponds to how things are. And the model has only ever seen the words. It has read every description of rain ever written and stood out in none of it. It holds the map - all of the maps, every map anyone has ever drawn - and it has never once been to the territory. That is why it can be eloquent and wrong in the same breath and feel no friction between the two. The friction lives in a place the model has never visited.

There's a corollary that unsettles people, and it shouldn't. A machine like this has no worldview. None. It will argue any side of anything with equal grace, defend a position and then dismantle it in the next window, because it isn't holding a position - it's rendering one. It is a mirror with a vocabulary. We keep waiting for it to reveal what it really believes, and it doesn't believe anything, and that is not a flaw to be trained out of it. It is the honest fact of the thing. The language is separate from any view of the world. That was the original insight some of us started from years ago, before any of the building began: language is machinery, and machinery has no creed.

It is a mirror with vocabulary

The trouble is that we keep dressing the machinery in the costume of a knower. We put it behind a chat window that answers in the first person, warm and certain, and every instinct we have says this thing believes what it is telling me. It does not. It cannot. And the distance between how it sounds and what it is happens to be the most dangerous real estate in the whole technology, because that is exactly where a fluent falsehood gets received as a considered judgment - in a clinic, in a courtroom, in a loan decision, in a room where someone is deciding whether to act.

So what do you do with a machine that can say anything and stand behind nothing? You stop asking it to be the thing it cannot be. If truth lives in the relationship between a claim and the world, then truth has to come from the world - from some grounded, checkable account that sits outside the language model and stays outside it. You don't teach the renderer to be honest. You keep the saying and the knowing in separate rooms, and you let the language render only what the knowing will vouch for. Language on one side, a verifiable account of the world on the other, and a wall between them you can actually inspect.

That sounds tidy until you try to build it, and then you hit the part nobody puts on a slide. Before you can check a claim against the world, you have to know what the claim is - and pulling discrete, checkable claims out of fluent prose is genuinely hard. The machine doesn't speak in clean facts. It speaks in paragraphs, where an assertion hides inside a subordinate clause, where a hedge can pass for a claim and a claim can pass for a hedge, and where - my favorite trap - every individual sentence is true and the paragraph they assemble into is a lie. The honest sentence, marshaled into a dishonest whole. Working out what is actually being asserted, before you have checked whether any of it is so, turns out to be most of the labor. It is unglamorous, and it is the ballgame.

I don't think the future of this technology is a more fluent machine. We already have fluency. Fluent is solved. The future is a more honest architecture - one that knows the difference between what it can say and what it can stand behind, and that keeps the truth somewhere you can point to and check. A machine with no worldview is not the problem. Pretending it has one is. The repair was never going to be giving the machine a conscience. It is to stop asking the part that talks to also be the part that knows.

Full disclosure: I'm a co-founder of 2Brains, a company built on exactly this conviction, so I am not a neutral party here, which we have solved and have patent pending. But the conviction came first. The company exists because of it, not the other way around.

The post GenAI is Fluent in Everything, but Faithful in Nothing first appeared on I, Cringely.




Digital Branding
Web DesignMarketing