Choosing a model for an in-product finance agent

// the question

Cleo has an assistant inside the product. It reads your bills, answers questions about them, and pays or approves them once you confirm. Newer, larger models usually score higher on public benchmarks, but they cost more and keep you waiting longer. We wanted to know whether that shows up on the work our assistant actually does.

// what we built

An eval that runs the real assistant against a fresh copy of a seeded business and swaps only the model underneath it. Each run is graded in code: which bills were paid or approved, and whether the reply states the right amounts, bill numbers and dates. No model grades another.

• 25 tasks across paying and approving, answering questions, and reports written on a schedule with nobody in the chat, a feature we are still testing
• Every task three times per model, so one lucky run does not count
• Speed and cost measured on the same runs as quality

// what we learned

In chat, size bought nothing we could measure. All three models paid the right bills for the right amounts, approved only what was waiting on the user, asked which bill was meant when two vendors matched, and answered every question correctly, in every run. The smallest model gave its first words in under 3 seconds at the median. The largest took about 28.

Scheduled reports were where they separated. With nobody to ask and every bill to spell out, the largest model got 83% right, the mid-size 78% and the smallest 33%. The smallest often replied the way it would in chat, with a one-line intro and none of the bills. The mid-size model once left a bill out.

One report task no model passed. Asked to pay bills from inside a scheduled report, none of them moved any money, but none said clearly that it had not. A report that quietly skips a request is still a miss, and we count it as one.

// what's next

We rerun these evals whenever the assistant or a model changes, and add tasks as the assistant learns to do more.