SHARP: an AI analyst that shows its confidence

The Value Quantification team at Mastercard had a bottleneck that nobody loved but everybody had made peace with: pulling the right benchmarks for a request took an analyst about 2.5 hours. Not because the data was hard to find, but because it was scattered — different sources, different formats, different ways of framing the same number.

I built SHARP, an LLM-based tool on Microsoft Copilot, to automate that retrieval. The mechanics of the retrieval pipeline came together faster than I expected. The harder problem showed up later: once SHARP could produce an answer in 15 to 20 minutes instead of 2.5 hours, how was anyone supposed to know when to trust it?

Speed wasn't the hard part

Retrieval is a well-understood problem. Point the model at the right sources, structure the prompt, validate the output format. SHARP got from a request to a formatted answer quickly, and the 2.5-hour-to-15-minute improvement was obvious and easy to demonstrate.

What wasn't obvious was what stakeholders actually needed from the tool. They didn't just want a fast answer — they wanted to know how much weight to put on it before they used it in a client conversation.

Designing the confidence-scoring rubric

So I built a scoring rubric that rated each SHARP output on how much a stakeholder should trust it, based on things like source agreement and data recency. That rubric took longer to get right than the retrieval pipeline did. A fast wrong answer is worse than a slow right one, and a fast answer with no signal about its own reliability just moves the guesswork downstream to whoever has to act on it.

That's the part of building with LLMs that I think gets underrated: the interesting product work isn't making the model answer faster, it's making the model honest about what it doesn't know.