Robert Polding's Home

I keep thinking about a fairly minor legal case from a couple of years back. Minor in the sense that it involved about eight hundred Canadian dollars and one grieving customer, not exactly a landmark. But it gets at something every organisation bolting AI onto its website is going to run into eventually, which is… when the bot says something that isn’t true, whose problem is that?

The case is Moffatt v. Air Canada. Ars Technica covered it well at the time, and their write-up is worth reading in full: Air Canada must honor refund policy invented by airline’s chatbot.

Here’s what happened. Jake Moffatt needed to fly for his grandmother’s funeral. He asked the airline’s website chatbot about bereavement fares, and the bot told him he could book at full price and then claim the discounted rate back afterwards, as long as he applied within 90 days. So that’s what he did. Screenshotted the conversation, booked, flew, and then put in his refund request.

Except the policy the bot described didn’t exist. Air Canada’s actual bereavement rules, which lived on a different page of the same website, said you have to sort the discount out before you travel, not after. So they turned him down. He showed them the screenshot. They turned him down again. He spent a couple of months going back and forth before finally taking it to the Civil Resolution Tribunal, which sided with him and ordered the airline to pay up.

The money isn’t the interesting bit. The interesting bit is how Air Canada tried to defend itself.

They argued that the chatbot was, and I’m quoting the reporting here, “a separate legal entity” responsible for its own actions. Which is a genuinely remarkable thing for a company to say out loud in a courtroom. Don’t blame us, blame the bot. The bot did it.

The tribunal didn’t buy it, and the reasoning was about as straightforward as you’d hope. The chatbot is part of Air Canada’s website. Air Canada is responsible for what’s on its website. It doesn’t matter whether the words came from a normal page or a chat window. And there’s one line in the decision I love: the tribunal noted that Air Canada never bothered to explain why a customer should be expected to go and cross-reference the chatbot’s answer against some other page to figure out which version of the policy was the real one.

That, to me, is the whole thing. If you publish two contradictory answers and then act surprised when the customer believes the wrong one, you don’t get to call that their mistake.

It would be easy to read this as a funny airline story and leave it there. I don’t think it is one, or not only one. We’re plugging language models into the front counters of banks, hospitals, universities, government offices, more or less as fast as we can, and these systems are spectacularly good at producing text that sounds right. Sounding right and being right are not the same thing, and the whole case sits in the gap between them.

I’d also quietly object to the usual way this gets described, which is that the chatbot “hallucinated” the policy. That word makes it sound like a rare malfunction, the system briefly losing its mind before returning to normal. But making up a plausible-sounding bereavement policy is not a malfunction. It’s the machine doing exactly what it does, which is assemble convincing text without any real stake in whether the text is true. When it lands on something correct, we call it an answer. When it doesn’t and someone’s out of pocket, we call it a hallucination. Same process both times.

What I find reassuring, honestly, is that the legal principle the tribunal reached for isn’t new or exotic. Companies are responsible for what they tell the public, and it doesn’t matter whether a human said it or a machine did. There’s decades of case law about this going back to ordinary computer systems. All that’s new is applying it to a chatbot, and confirming that wrapping your software in the language of autonomy and “it’s basically its own agent” doesn’t get you off the hook.

Which is really the lesson, and it’s not a complicated one. You can’t outsource the blame to a tool. If you put an AI system in front of people, whatever it says is something you said. That’s not some unfair burden to grumble about, it’s just the cost of admission. If the convenience is worth having, then so is the work of making sure the thing tells the truth.

The organisations that get this right are going to be the ones that treat these systems as instruments to be aimed and checked and supervised, not as trusted colleagues and definitely not as convenient people to blame when it goes wrong. Air Canada worked that out the expensive way, over an eight hundred dollar refund. Cheaper, I’d suggest, to just take the point.

---

Original reporting: Ars Technica, “Air Canada must honor refund policy invented by airline’s chatbot”.