# Klarna is hiring humans again: AI use cases need operational evidence, not vendor blogs

> Put RAG, agents and recommender systems in one table, and the easiest use cases to control share one trait: either a person signs off on the result, or there is a metric that can be measured every day.

Original: https://fdetimes.net/en/analysis/klarna-ai-use-cases-operational-evidence/

In February 2024, Klarna announced that its AI assistant, built on OpenAI, had handled 2.3 million conversations in its first month, or two-thirds of the company's customer service chats. Fifteen months later, the same company's CEO said that investing in the quality of human support was the way forward.

The two statements do not contradict each other. They show a pattern that anyone deploying AI in a business runs into sooner or later: a demo is measured by the share of work automated, while a system that runs for the long term is measured by who fixes things when the machine gets them wrong.

If you are trying to move into forward deployed engineering, this is the most valuable lesson available. Clients will arrive with a list of use cases taken from vendor blogs. Your job is to say which ones have operational evidence behind them, and what each one needs so it does not fall apart when real users arrive.

## Vendor lists are long, but not every item comes with numbers

Read enough use-case roundups and it is easy to conclude that AI is already present in every industry. Glean groups RAG by industry: customer service uses RAG to pull product information and customer history and personalise replies, while in legal work the system searches case law and statutes to help lawyers research and draft.

Codica lists agents in finance that can detect anomalies and signs of fraud. All of this is plausible, but it describes capability, not operational evidence.

A vendor blog post tells you what kind of problem can be tackled. It does not tell you the rate of wrong answers, whether users trust the system, or who is accountable when something goes wrong.

So when you read about a use case, ask five questions: where does the data come from, does the answer point to its source, who reviews the result before it reaches the end user, when must the system stop and hand over to a person, and if nobody reviews it, what is measured instead.

Answer those five questions for each of the examples below and they fall into a few clear groups.

## RAG works best when someone reads the output

Two use cases described by theblue.ai deserve a close look, because they share a structure that is easy to control. The first is internal Q&A: the system answers within seconds and names the documents it used to answer.

The second is repetitive compliance questionnaires or tender documents: RAG drafts answers from policies and technical documentation, then an expert reviews and approves them.

What the two have in common is that the machine does not have the final say. An answer with citations can be checked by the reader. A draft reviewed by an expert means a wrong sentence is caught before it goes out, and case-law research follows the same pattern: the lawyer still does the research and the drafting.

Consider a hypothetical: a B2B software company receives about 40 security questionnaires a quarter, each with several hundred questions, most of them repeated across questionnaires. If RAG can draft most of the answers and the expert only has to fix the hard ones, the time saved is real, and the risk stays in the hands of someone who knows the work.

That is exactly the shape of a use case that gets its budget approved.

**Key point:** The easiest AI use cases to control are those where a person reads the result, or where there is a metric that can be measured every day.

## Klarna shows where customer service agents are strong and where they break

According to figures the company published itself, customer satisfaction with the AI assistant was on a par with human agents and repeat enquiries fell by 25%, although these are the company's own numbers rather than those of an independent auditor.

In May 2025 the Klarna CEO changed tone: according to Entrepreneur, the company was rehiring customer service staff and promising that customers who wanted to speak to a real person would always be able to. The lesson is not that AI failed. It can be read as a shift from a design that treats automation as the goal to one that treats the handoff to a human as a mandatory feature.

Picture a customer asking about a refund, but the refund details sit in the payments system and the complaint history sits in the ticketing system, so the bot sees only part of the picture and answers confidently, yet wrongly.

That is why, for an FDE, the first question on a customer service project is which systems the agent can read, not which model to choose. The second is what signal the agent relies on to know it lacks enough information and must hand over to a person.

## Product recommendations are the AI with the clearest metrics

On recommender systems, Algolia cites figures suggesting recommendations account for as much as 31% of e-commerce revenue, and repeats the claim that 80% of what is watched on Netflix comes from its recommendation system. Both are second-hand figures in a post by a vendor that sells search and recommendation tools, so treat them as indicative only.

What is more instructive is how this problem works in practice. Nobody reviews each recommendation, but a bad one usually just makes the user scroll past, and the outcome is measured through behaviour: whether users click, buy or keep watching. The system can run fully automatically because of this feedback loop, not because the model is smarter.

Put the three groups in one table and a pattern emerges that no use-case roundup states outright.

| Use case | Industry | Data needed | Who signs off | Evidence available |
|---|---|---|---|---|
| Internal Q&A with citations | All industries | Internal documents | The person asking checks the source | Vendor description |
| Drafting compliance questionnaires, tenders | B2B | Policies, technical documentation | Expert review | Vendor description |
| Case law and statute research | Legal | Case law, legislation | Lawyer | Vendor description |
| Customer service agent | Fintech, retail | Products, customer history, multiple systems | Agent, handing off to a human when needed | Self-reported by Klarna, then adjusted in 2025 |
| Anomaly and fraud detection | Finance | Transactions | Must be designed case by case | Still only a list of capabilities |
| Product recommendations | E-commerce, streaming | User behaviour | Automatic, measured by behaviour | Second-hand figures, measurable |

The pattern is this: the higher the cost of being wrong, the more a person needs to sign off, and where nobody signs off, there must be a metric measured every day. A use case with neither, such as an agent answering every complaint on its own with no path to a human, is exactly where a project is likely to have to reverse course.

## An FDE portfolio should show the handoff to a human, not a chatbot

If you are preparing to move into an FDE role, do not build your portfolio around an "ask me anything" chatbot.

Build a narrow RAG system on real data, such as public legal texts, a university's regulations or the product documentation of an open-source project, require every answer to cite its source, then publish the citation accuracy rate you measured yourself.

On a CV, a line such as "designed a handoff-to-human flow for low-confidence answers, tracked weekly handoff rate" carries far more weight than "built a chatbot using GPT".

When reading job descriptions, look for mentions of evaluation, human-in-the-loop or integration across multiple source systems, because they signal a company that has moved past the demo stage and is dealing with real operations.

And in your first working session with a client, ask before you propose anything: who signs off on the answer today, and how will they check the machine? The answers to those two questions all but decide what the first use case should be.

The Klarna case does not prove that customer service agents are a bad idea. It shows that even when AI is already handling two-thirds of chats, customers still need a person on the other side when they want one.

**Try this week:**

- Build a small RAG over 20-30 public legal documents, require every answer to name its source document, then score how many answers cite correctly
- Take an agent use case you have seen, map out three situations in which the agent must hand over to a person, along with the data needed to recognise each one
- Read 5 job descriptions for FDE or AI engineer roles and mark those that mention evaluation, human-in-the-loop or escalation

## Sources

- [Retrieval augmented generation use cases: Transforming data into insights (Glean)](https://www.glean.com/blog/retrieval-augmented-generation-use-cases)

- [Retrieval Augmented Generation (RAG) – 5 Use Cases (theblue.ai)](https://theblue.ai/blog/rag-news/)

- [Klarna AI assistant handles two-thirds of customer service chats in its first month](https://www.klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/)

- [Klarna Is Hiring Customer Service Agents After AI Couldn't Cut It on Calls, According to the Company's CEO](https://www.entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396)

- [What Role does AI Play in Recommendation Systems and Engines? (Algolia)](https://www.algolia.com/blog/ai/what-role-does-ai-play-in-recommendation-systems-and-engines/)

- [A Brief Guide on AI Agents: Benefits and Use Cases (Codica)](https://www.codica.com/blog/brief-guide-on-ai-agents/)

- [Top 15 Use Cases Of AI Agents In Business (Ampcome)](https://www.ampcome.com/post/15-use-cases-of-ai-agents-in-business)