On Turn-Level Best-of-N Sampling for Scaling LLM Agents

Özetle: In this work, we evaluate a simple inference-time scaling approach based on textit{turn-level best-of-N sampling} (method) for large language model (LLM) agents on complex, long-horizon tasks.

Özet

In this work, we evaluate a simple inference-time scaling approach based on textit{turn-level best-of-N sampling} (method) for large language model (LLM) agents on complex, long-horizon tasks. We instantiate this using a textit{generator-selector} framework with two LLMs; at each turn, a emph{generator} model generates $N$ candidate actions and a emph{selector} model selects one of them, which is then executed in the environment. This paper is closely related to the work of cite{zainullina2025guided} who used critic models for guided search at inference time. Different from their work, we evaluate if off-the-shelf, open-weight LLMs can be effective at comparing different candidate actions directly. Experiments on long-horizon terminal tasks show that method can lead to significant performance improvements, both in terms of accuracy (5–10%) and consistency across repeated attempts. Analysis of agent trajectories reveal a shift in agent behaviour towards desirable patterns such as reduction in number of attempts to solve the task, improved task instruction following, debugging and testing.

Orijinal Özet (İngilizce)

In this work, we evaluate a simple inference-time scaling approach based on textit{turn-level best-of-N sampling} (method) for large language model (LLM) agents on complex, long-horizon tasks. We instantiate this using a textit{generator-selector} framework with two LLMs; at each turn, a emph{generator} model generates $N$ candidate actions and a emph{selector} model selects one of them, which is then executed in the environment. This paper is closely related to the work of cite{zainullina2025guided} who used critic models for guided search at inference time. Different from their work, we evaluate if off-the-shelf, open-weight LLMs can be effective at comparing different candidate actions directly. Experiments on long-horizon terminal tasks show that method can lead to significant performance improvements, both in terms of accuracy (5–10%) and consistency across repeated attempts. Analysis of agent trajectories reveal a shift in agent behaviour towards desirable patterns such as reduction in number of attempts to solve the task, improved task instruction following, debugging and testing.

Kaynak: arXiv:2611.06868 · PDF