Early AI exploration

Finding a useful problem for a new technology

This was one of the company’s first AI projects. Large language models, the technology behind tools such as ChatGPT, were new. There were few established examples of using them in business software, and the organisation did not yet have clear rules for designing or testing AI features.

We began with Case Management, where support agents spend a lot of time reading customer histories and trying to understand what happened before they can act.

The initial question was broad: where could this technology genuinely help?

That distinction mattered. It was easy to imagine impressive demos. It was much harder to find a use that saved time in the real product and remained useful when the output was imperfect.

1. Choosing problems AI could realistically help with

I worked with product, engineering and research to compare possible uses against two things: how much they could help support agents and whether we could build them safely with the models available at the time.

Three ideas came forward:

1. Case summaries: Help agents understand long and complicated cases faster.
2. Similar cases: Surface past cases that might contain useful context or a previous solution.
3. Proposed solutions: Tell the agent how to resolve the case.

We moved forward with the first two. Proposed solutions sounded more ambitious, but the early models were not reliable enough for that level of responsibility. They could only produce a single response, with no conversation to clarify missing context or correct a weak answer.

Deferring it was an important product decision. We did not need to use the most impressive capability. We needed to find the point where AI could help without asking users to trust it too much.

2. Testing real output, not the best demo

AI can produce an excellent result once and fail on the next case. We needed to understand its normal behaviour, not select the most convincing examples for a presentation.

I interviewed support experts to learn which details they needed before acting on a case. Together, we created ideal summaries for a set of real, anonymised case histories. These became the reference for evaluating generated output.

We built an internal testing space where prompts, the instructions sent to the AI, could be changed and run against the same cases repeatedly. We stored each prompt, its result and what we expected to improve next.

I also created a consistent set of review questions. Did the summary include the events an expert considered essential? Was the current situation clear? Could anything mislead the agent? This meant we were not judging every result from scratch.

This made failures visible. We could see whether a summary omitted a key event, gave too much weight to an old message or sounded clear while misrepresenting what happened.

3. Changing the summary when one paragraph was not enough

The early summaries worked for simple cases but struggled with long histories. A single paragraph could sound fluent while leaving out an important event or merging several moments into the wrong conclusion.

Instead of trying to solve everything through a better prompt, I changed the interface.

The new summary had two layers. A short overview explained the current situation. Underneath it, a chronological timeline summarised each important activity separately. Agents could understand the case quickly, then inspect the sequence when they needed more confidence.

This reduced the pressure on one generated paragraph to be complete and perfectly accurate.

4. Helping agents judge whether a similar case was useful

The system could calculate how mathematically similar two cases were and turn that score into a percentage. A number such as 87% looked precise, but it did not explain whether the old case was relevant to the problem in front of the agent.

I removed the percentages.

The interface ranked the similar cases and showed the title with a short summary of the customer’s original message. That gave agents information they could understand and let them decide whether opening the case was worthwhile.

The system did the searching. The person made the judgment.

5. Making the wait and cost feel intentional

Generating a summary took time and every request cost money. Leadership was concerned about calling the model automatically whenever someone opened a case.

We added a button so the agent could request a summary when it was useful. While the AI worked, a skeleton placeholder showed that content was loading. Once generated, the summary was stored and reused until new activity changed the case.

This avoided repeated calls, reduced cost and gave the user a clear reason for the wait. Other AI features could reuse the same approach.

What changed

• Around 70–75% less time spent understanding a case: Agents could see what had happened and what still required attention much faster.
• Around 40–50% fewer customer complaints: Complaints about incorrect or irrelevant support responses fell significantly after the new experience was introduced.
• Focused on the AI uses that showed real value: We moved forward with case summaries and similar cases, and stopped work on proposed solutions when the models were not reliable enough. This avoided spending more time on an idea that was not ready.
• A way to test AI output that others could reuse: The review method gave other teams a practical way to compare generated results with what experts expected.
• Interaction patterns that others could reuse: Other teams adopted the two-part summary, button-triggered generation and stored results that only refreshed when the source information changed.

What I learned

The hardest part was not finding something the model could generate. It was deciding where that output was useful, how much responsibility it should have and how we would know when it failed.

Real cases and expert reference answers made the conversation more honest. They shifted reviews away from whether an example looked impressive and towards whether the feature helped across different situations.

The project also changed my role. Designing the screens was only one part. I worked with engineers on what the models could do, with support experts on what a good answer contained and with product leaders on cost. I then had to bring those decisions together in the product.

Trust did not come from telling users the AI was intelligent. It came from giving them enough structure and context to judge the result themselves.