Key takeaways
- Measure cost per task alongside the monthly bill
- Send simple work to smaller models
- Cache what repeats
The monthly invoice from a model provider tells you very little. It goes up, and nobody knows whether that is because usage grew, a prompt got longer or one customer is running a script against your chatbot. Start by making cost visible at the level where you can act on it.
Measure
- Track cost per completed task and per user. "Rs 1.80 per invoice processed" is something you can manage; a monthly total is not.
- Set budgets and alerts per environment and per customer, so a runaway test job in staging does not wait for month end to be noticed.
Reduce
- Send simple requests to smaller, faster models. Sorting messages does not need the most capable model you can buy.
- Use prompt caching for long instructions and documents that are sent again and again.
- Trim retrieved context to what the answer needs. Ten passages where three would do is paying for seven you did not use.
- Batch background work, such as overnight summaries, where nobody is waiting for a reply.
Review
- Each month, look at the ten most expensive tasks and ask why each one costs what it does.
- Re-test cheaper models when new versions arrive. Last year's expensive choice may now have a cheaper equal.
- Switch off features nobody uses. They still cost money every time they run.
Have a question about this guide? Write to us or ask Colony, our AI guide.