← Field notes

Guide

Choose a model with a small evaluation

Compare the same text workload across models before giving one a production budget.

Build a small set

Collect a few representative prompts and define what a useful answer looks like. Include one ambiguous request, one long request that fits the gateway limits and one request with an output format to follow. Use examples that are appropriate to share with a model provider.

Keep the test comparable

Keep the messages and output-token cap identical across models. Run the same test for each candidate and record format compliance, errors and output length. Use a separate key for evaluation so you can review usage afterward.

Make the tradeoff explicit

Start with the catalog’s workload suggestions. Evaluate the cheapest candidate that meets your acceptance criteria, and keep a second model in your test set for the cases it misses. Compare actual charges after the run, including cached input when reported.

Compare models ↗