Evaluating Frontier AI Agents with Shadow Evaluations

Practical Summary

Shadow evaluation is a methodology designed to assess the capabilities of frontier AI agents in conducting open-ended research. Unlike standard benchmarks that rely on verifiable, narrow tasks, shadow evaluations involve tasking agents with replicating or extending unpublished research papers. This approach tests an agent's ability to manage long-horizon goals, allocate resources, and exercise judgment in environments where success is not immediately clear. By comparing agent-generated outputs against the original research conducted by human experts, researchers can identify critical bottlenecks in agent performance, such as a lack of creativity, poor backtracking, and failure to adhere to complex, multi-step instructions. This method provides a more realistic assessment of progress toward recursive self-improvement than narrow benchmarks.

Methodology and Setup

To conduct a shadow evaluation, researchers partner with authors of unpublished research to draft the main research questions. The methodology, as detailed at https://www.normaltech.ai/p/ai-agents-cant-yet-do-open-ended, involves providing these questions to frontier AI agents. The evaluation requires a research scaffold—an executive architecture that manages the agent's planning, exploration, and review processes. Researchers should allocate sufficient API credits and compute resources to allow the agent to navigate the research task. The evaluation team must include expert reviewers who are capable of assessing the quality of the agent's research output. The team should ideally include collaborators with diverse priors to mitigate potential bias in the interpretation of results.

Documented Workflow

The workflow begins by providing the agent with the main research questions from an unpublished paper. The agent is then given a set timeframe to conduct research. Throughout this period, the agent is expected to monitor its own progress, manage its API budget, and utilize self-review tools. The process involves the agent proposing research directions, testing hypotheses, and potentially backtracking if initial approaches prove unpromising. Finally, the agent produces a paper which is then reviewed by the original human authors of the study. This review process is central to the evaluation, as it highlights discrepancies between human research judgment and agent-driven decision-making, providing a qualitative assessment of the agent's performance.

Limitations and Challenges

Shadow evaluations face several inherent limitations. The sample size is typically small due to the intensive nature of the review process, which requires significant researcher flexibility in design and interpretation. There is also the risk of researcher bias, as the team's prior beliefs regarding recursive self-improvement may influence the study's execution. Furthermore, expert reviewers are aware that the papers are AI-generated, which may introduce subjective preferences for human-led approaches. Additionally, the performance observed is a combination of the underlying foundation model and the specific research scaffold used, making it difficult to isolate the capabilities of the model itself from the executive architecture.

Interpreting Agent Performance

When analyzing the output of a shadow evaluation, researchers should focus on specific behavioral markers. Evidence suggests that current frontier agents often struggle with epistemic lock-in, where they double down on unpromising directions despite negative feedback. Agents may also fail to effectively utilize their allocated resources, ending experiments with significant budget and time remaining. A key indicator of performance is how the agent responds to its own self-reviews; if an agent identifies issues but fails to creatively address them, it suggests a lack of genuine research judgment. These evaluations are intended to surface bottlenecks that are not currently visible in narrow, verifiable benchmarks.

Choosing When to Use Shadow Evaluations

Shadow evaluations are most appropriate for researchers and organizations aiming to understand the limitations of AI agents in complex, non-verifiable domains. They should be used when the goal is to assess progress toward recursive self-improvement rather than simple task efficiency. By identifying where agents fail to backtrack, follow instructions, or maintain long-term research goals, teams can better prioritize scaffold improvements and targeted training. This method is particularly valuable for those investigating whether current AI progress is bottlenecked by agent capabilities or if there are deeper, structural limitations to fully automated research that require further investigation.

Executive Architecture Considerations

It is scientifically important to distinguish between the foundation model and the research scaffold. Many of the failures observed—such as maintaining long-horizon goals, budgeting time and API usage, deciding when to transition from exploration to exploitation, invoking critics and reviewers, keeping a research notebook, and determining when work is good enough—are properties of the overall executive architecture. If a future scaffold substantially improves these executive functions while the underlying model changes little, researchers must attribute the improvement correctly. Consequently, publishing the operational prompts and orchestration logic is essential for reproducibility and for distinguishing improvements in frontier models from improvements in agent design.

Implications for AI Progress

The results of shadow evaluations suggest that conducting open-ended research remains challenging for frontier AI agents. If these findings hold up, they imply that while faster progress is possible on narrow, verifiable tasks like improving efficiency, it may not lead to broad recursive self-improvement or explosive progress. Researchers must measure how quickly current limitations, such as the lack of creativity and judgment, can be overcome through targeted training and scaffold improvements. Furthermore, if the world contains many remaining bottlenecks that are hard to overcome, Amdahl’s law suggests that even a hundredfold speedup in the parts amenable to AI would only lead to a small speedup in the overall pace of progress.