Traditional scaling focuses heavily on training: larger models, more data and more optimization steps. Inference-time compute shifts part of the budget to the moment when the model is solving a task.

Chrome GPU editorial
FIG. 01 — Sometimes the model needs more than one glamorous attempt.

What Counts as Test-Time Compute?

  • Generating several candidate answers
  • Searching through multiple reasoning paths
  • Using a verifier or reward model to rank outputs
  • Iterative refinement
  • Tool use and external retrieval

One Model, Different Budgets

The same model can behave differently depending on how much computation it receives at inference. A low-cost setting might produce one immediate answer. A higher-cost setting might sample several candidates, inspect intermediate results and select the best response.

Inference strategyCompute costPotential benefit
Single passLowFast response
Multiple samplesMediumBetter candidate diversity
Search + verifierHighStronger task performance
Tool-augmented loopVariableExternal information and actions
Model intelligence may be partly a property of the system around the model, not only the weights inside it.

The Systems Question

Extra inference compute increases latency and serving cost. It therefore becomes a scheduling and systems problem: which requests deserve more computation, how should candidates be batched, and when is a verifier worth the additional expense?

Experiment Idea

For a small reasoning benchmark, I will compare a single greedy answer, multiple sampled answers with majority voting, and a small search procedure. The objective is not only to measure accuracy but also improvement per unit of added compute.

candidates = [ model.generate(prompt, temperature=0.8) for _ in range(num_samples) ] best = verifier.rank(prompt, candidates)[0]

Metrics

  • Accuracy
  • Latency
  • Total generated tokens
  • GPU time
  • Improvement per additional candidate