AI Scientist-v2: First AI Paper Accepted by Peer Review

AI Scientist-v2: The Insane Story of How AI Just Wrote Its First Peer-Reviewed Paper

What if your next research collaborator never slept, never complained about grant deadlines, and could explore thousands of experimental hypotheses while you grab coffee? Sounds like science fiction? It's already happening. SakanaAI just dropped something that made the entire ML research community drop their coffee mugs: the first workshop paper written entirely by AI—and accepted through actual peer review. Not a preprint. Not a blog post. A real, reviewed, accepted paper.

Welcome to the era of AI Scientist-v2, and if you're still writing every experiment script by hand, you might want to sit down for this.

The painful truth? Modern researchers drown in complexity. Hypothesis generation, experimental design, implementation, debugging, analysis, writing, revising—each step consumes weeks of human effort. The replication crisis haunts our field. Brilliant ideas die in implementation hell. Meanwhile, the pace of ML innovation accelerates exponentially, leaving individual researchers gasping for air.

AI Scientist-v2 is SakanaAI's answer to this chaos: a fully autonomous, end-to-end agentic system that generates hypotheses, runs experiments, analyzes results, and writes complete scientific manuscripts. Unlike anything before it, this system employs progressive agentic tree search—think AlphaGo's strategic brilliance applied to scientific discovery. No human-authored templates. No hand-holding. Just pure, relentless, automated exploration of the research space.

The focus keyword here is AI Scientist-v2, and by the end of this deep dive, you'll understand exactly why this repository is about to change how you think about research forever. Ready to peek behind the curtain? Let's go.

What is AI Scientist-v2?

AI Scientist-v2 is a generalized end-to-end agentic system for automated scientific discovery developed by SakanaAI, the Tokyo-based AI research collective founded by former Google Brain and DeepMind researchers. Building on the foundation of their first iteration, this v2 release represents a fundamental architectural shift—from template-dependent automation to truly open-ended scientific exploration.

The project's full title tells the complete story: "The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search." And yes, that "workshop-level" claim is backed by cold, hard reality: this system generated a paper that survived actual peer review at a major ML workshop.

Why is it trending right now? Three explosive factors converged:

First, the capability leap. Where v1 required human-authored templates to guide paper structure, v2 removes this crutch entirely. It generalizes across ML domains using an experiment manager agent that orchestrates progressive tree search through the hypothesis space.

Second, the peer-review milestone. In an age of AI-generated sludge flooding preprint servers, a machine-authored paper passing legitimate academic scrutiny is a watershed moment. It forces us to confront uncomfortable questions about the future of scientific publishing.

Third, the timing. With foundation models growing exponentially more capable and the AI research community desperate for automation tools, SakanaAI released this precisely when the infrastructure—both technical and social—was ready to receive it.

The critical distinction from v1: AI Scientist-v2 trades reliability for generality. v1, with its rigid templates, achieves higher success rates when strong starting points exist. v2 accepts lower success rates in exchange for the ability to explore truly novel territories without human-imposed structure. It's the difference between a skilled technician following blueprints and a visionary architect dreaming new buildings into existence.

The system is explicitly designed for open-ended scientific exploration—the kind of messy, creative, failure-rich process that actually produces breakthroughs, rather than incremental advances on well-trodden paths.

Key Features That Make AI Scientist-v2 Dangerously Powerful

Let's dissect what makes this system tick. The architecture reveals SakanaAI's deep understanding of both the scientific process and modern agent design.

Template-Free Generalization The v1 system's Achilles heel was template dependency. Human researchers had to craft detailed paper structures, experimental frameworks, and analysis pipelines. v2 shatters this limitation. The experiment manager agent dynamically constructs appropriate methodologies based on the research domain, the generated hypothesis, and intermediate results. This isn't mere prompt engineering—it's genuine architectural flexibility.

Progressive Agentic Tree Search (BFTS) The beating heart of v2 is its Best-First Tree Search mechanism. Imagine AlphaGo's Monte Carlo Tree Search, but instead of Go positions, each node represents a potential experimental configuration. The system maintains multiple parallel exploration paths (configurable via num_workers), dynamically allocates computational resources to promising branches, and prunes failing directions. The bfts_config.yaml exposes fine-grained control: max_debug_depth determines persistence on failing nodes, debug_prob controls stochastic recovery attempts, and num_drafts sets how many independent search trees germinate from the ideation phase.

Multi-Model Orchestration AI Scientist-v2 doesn't bet on a single LLM. It deploys a specialized model ensemble: Claude 3.5 Sonnet typically handles experimentation (where reasoning depth matters), GPT-4o manages citations and literature synthesis, o1-preview tackles write-up generation, and o3-mini aggregates visualization plots. This division of labor exploits each model's comparative advantage, creating a genuinely collaborative AI research team.

Autonomous Novelty Checking Through integration with the Semantic Scholar API, the system validates generated hypotheses against existing literature during ideation. This isn't superficial keyword matching—the LLM engages with retrieved papers to assess genuine conceptual novelty, preventing redundant research before computational resources are expended.

Sandboxed Execution with Explicit Risk Acknowledgment SakanaAI deserves credit for unusual honesty. The repository prominently warns that this system executes LLM-generated code with full autonomy, including potential dangerous package installation, uncontrolled web access, and process spawning. The mandatory Docker^{↗ Bright Coding Blog} sandbox requirement isn't bureaucratic box-checking—it's essential protection against an agent that might, in pursuit of some experimental goal, compromise its host environment.

Use Cases: Where AI Scientist-v2 Absolutely Dominates

Exploratory ML Architecture Search When you have a vague intuition—"what if attention mechanisms could be replaced with something simpler?"—but no concrete starting point, v2 shines. The ideation agent generates diverse architectural hypotheses, the tree search explores implementation variants, and the system automatically evaluates against standard benchmarks. Human researchers provide the creative spark; v2 handles the exhaustive, grinding exploration.

Cross-Domain Method Transfer Techniques successful in computer vision often languish unapplied in NLP or reinforcement learning, simply because no human researcher bridges the gap. v2's template-free design enables automatic transplantation: describe the source domain's technique and target domain's problem, and the system autonomously adapts methodology, runs experiments, and validates transfer.

Negative Result Documentation Science's publication bias systematically hides failed experiments, distorting collective knowledge. v2's marginal cost per experiment—approximately $15-20—makes comprehensive negative result exploration economically viable. Researchers can systematically map what doesn't work, producing invaluable resources for the community.

Rapid Workshop Prototyping For researchers with promising preliminary results but insufficient time for full paper development before a deadline, v2 offers acceleration. The system generates complete submissions from minimal topic descriptions, allowing human researchers to focus on high-level direction and final quality control rather than implementation drudgery.

Educational Replication Studies Automated replication of published results, with systematic variation of implementation details, helps identify which findings are robust versus fragile. v2's tree search naturally explores this perturbation space, potentially serving as an automated replication engine.

Step-by-Step Installation & Setup Guide

Getting AI Scientist-v2 running demands proper preparation. This isn't a pip-install-and-go toy—it's a serious research instrument requiring GPU resources and API budget.

Environment Setup

First, create an isolated conda environment with Python^{↗ Bright Coding Blog} 3.11:

# Create a new conda environment
conda create -n ai_scientist python=3.11
conda activate ai_scientist

Core Dependencies

Install PyTorch with CUDA support. Adjust pytorch-cuda version to match your NVIDIA driver:

# Install PyTorch with CUDA support (adjust pytorch-cuda version for your setup)
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia

Document Processing Tools

The system generates LaTeX manuscripts and processes PDFs, requiring specialized tools:

# Install PDF and LaTeX tools
conda install anaconda::poppler
conda install conda-forge::chktex

Python Requirements

Install remaining dependencies from the project's requirements file:

# Install Python package requirements
pip install -r requirements.txt

Total installation time: approximately one hour, assuming reasonable network bandwidth.

API Key Configuration

The system requires multiple API keys depending on your model selection. Export these as environment variables:

export OPENAI_API_KEY="YOUR_OPENAI_KEY_HERE"
export S2_API_KEY="YOUR_S2_KEY_HERE"
# Set AWS↗ Bright Coding Blog credentials if using Bedrock for Claude models
# export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
# export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_KEY"
# export AWS_REGION_NAME="your-aws-region"

For Claude via AWS Bedrock, install additional packages:

pip install anthropic[bedrock]

Critical: Sandboxed Execution

Never skip this. The repository's explicit warning bears repeating: this system executes LLM-generated code with full autonomy. Run exclusively within Docker containers or equivalent sandboxed environments. The risk isn't theoretical—an agent optimizing for experimental success might install arbitrary packages, make unrestricted network requests, or spawn unexpected processes.

REAL Code Examples from AI Scientist-v2

Let's examine actual code from the repository, with detailed explanations of how this system operates.

Example 1: Research Ideation Pipeline

The ideation phase transforms a human topic description into structured, novel research hypotheses. Here's the actual execution command:

python ai_scientist/perform_ideation_temp_free.py \
 --workshop-file "ai_scientist/ideas/my_research_topic.md" \
 --model gpt-4o-2024-05-13 \
 --max-num-generations 20 \
 --num-reflections 5

Breaking this down: The perform_ideation_temp_free.py script—note the "temp_free" indicating template-free operation—takes your research topic Markdown^{↗ Smart Converter} file and engages the specified LLM in structured brainstorming. The --max-num-generations 20 parameter controls exploration breadth: the system attempts 20 distinct research ideas. --num-reflections 5 enables iterative refinement; each idea undergoes 5 rounds of self-criticism and improvement, dramatically enhancing quality over naive one-shot generation.

The output is a structured JSON file containing hypotheses, proposed experimental methodologies, and literature positioning. This JSON becomes the seed for the full experimental pipeline.

Example 2: Core Experiment Execution

The main pipeline launch demonstrates v2's multi-model orchestration:

python launch_scientist_bfts.py \
 --load_ideas "ai_scientist/ideas/my_research_topic.json" \
 --load_code \
 --add_dataset_ref \
 --model_writeup o1-preview-2024-09-12 \
 --model_citation gpt-4o-2024-11-20 \
 --model_review gpt-4o-2024-11-20 \
 --model_agg_plots o3-mini-2025-01-31 \
 --num_cite_rounds 20

Critical design insight: Each --model_* flag assigns a specific capability to a specialized model. o1-preview handles write-up generation because its extended reasoning produces more coherent scientific narratives. gpt-4o manages citations and review—tasks requiring broad knowledge retrieval and critical evaluation. o3-mini aggregates plots, a task where its efficiency advantages outweigh reasoning depth requirements. This isn't arbitrary—it's SakanaAI's empirical optimization based on model characteristics and task demands.

The --load_code flag initializes experimentation with existing code snippets (omit if starting from scratch). --add_dataset_ref incorporates dataset references for reproducibility. --num_cite_rounds 20 enables iterative citation refinement during write-up, ensuring literature positioning accuracy.

Example 3: Tree Search Configuration

The bfts_config.yaml exposes the agentic tree search's controlling parameters:

# Agent configuration
agent:
  num_workers: 3        # Parallel exploration paths
  steps: 21             # Maximum nodes to explore
  num_seeds: 3          # Validation seeds per experiment

# Search strategy configuration  
search:
  max_debug_depth: 3    # Persistence on failing nodes
  debug_prob: 0.5       # Stochastic recovery probability
  num_drafts: 5         # Independent search trees to grow

Understanding the search dynamics: With num_workers: 3 and steps: 21, the system explores up to 21 total nodes, expanding 3 concurrently at each step. This parallel breadth-first strategy balances exploration diversity against computational cost. num_seeds: 3 provides statistical validation—each promising configuration runs with 3 random seeds to distinguish genuine improvements from luck.

The max_debug_depth: 3 parameter reveals sophisticated failure handling. When an experimental node fails (compilation error, runtime exception, nonsensical results), the system doesn't immediately abandon the branch. Instead, it attempts up to 3 debugging iterations, using the LLM to diagnose and repair. debug_prob: 0.5 introduces stochasticity—only 50% of failures trigger debugging, preventing infinite loops on fundamentally broken approaches while preserving recovery for promising-but-flawed experiments.

Example 4: Citation and Responsible Use

The repository includes explicit BibTeX for academic attribution:

@article{aiscientist_v2,
  title={The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search},
  author={Yamada, Yutaro and Lange, Robert Tjarko and Lu, Cong and Hu, Shengran and Lu, Chris and Foerster, Jakob and Clune, Jeff and Ha, David},
  journal={arXiv preprint arXiv:2504.08066},
  year={2025}
}

More importantly, the license mandates disclosure. Any resulting manuscript must include:

"This manuscript was autonomously generated using The AI Scientist."

This isn't mere attribution—it's a legally binding requirement under their Responsible AI License derivative. SakanaAI is proactively establishing norms for transparent AI-generated research, preventing the deceptive deployment that could poison scientific trust.

Advanced Usage & Best Practices

Model Selection Strategy: The default configuration uses Claude 3.5 Sonnet for experimentation because it achieves highest success rates on complex implementation tasks. However, cost-conscious researchers can experiment with GPT-4o for less demanding domains. The writing phase's ~$5 cost is relatively fixed, but citation generation with GPT-4o rather than more expensive models offers savings without quality degradation.

Topic Description Crafting: The my_research_topic.md file's quality dramatically impacts output. Study the example i_cant_believe_its_not_better.md carefully. Effective descriptions balance specificity—enough constraint to prevent aimless wandering—with openness sufficient for genuine novelty. Include concrete benchmark datasets and evaluation metrics when possible.

Resource Scaling: For initial exploration, reduce num_workers and steps in bfts_config.yaml. A single-worker, 7-step configuration completes in under an hour and costs under $5, providing rapid iteration on topic descriptions before committing to full exploration.

Failure Analysis: The unified_tree_viz.html visualization in experiments/"timestamp_ideaname"/logs/0-run/ is invaluable. This interactive tree reveals which branches succeeded, which failed, and where debugging attempts occurred. Study failure patterns to refine your topic description or adjust search parameters.

Memory Management: CUDA OOM errors indicate your GPU lacks capacity for the LLM-suggested model. Proactively specify smaller architectures in your topic description—mention "ResNet-18 rather than ResNet-152" or "distilled models" to constrain the agent's resource assumptions.

Comparison with Alternatives

The verdict: Choose v2 when exploring genuinely novel territory without clear templates. Stick with v1 for well-defined problems with strong existing foundations. Generic LLM workflows lack the integration and systematic search that make either version powerful. Human research remains essential for conceptual framing, quality judgment, and ethical oversight—but the mechanical execution layer is now automatable.

FAQ: Your Burning Questions Answered

Why didn't my experiment produce a PDF? Success rates depend heavily on model choice and idea complexity. Claude 3.5 Sonnet for experimentation yields highest completion rates. Simpler, well-scoped ideas also succeed more frequently. The system is explicitly designed for exploration, not guaranteed production.

What's the real cost per experiment? Ideation runs a few dollars. The main pipeline costs approximately $15-20 with Claude 3.5 Sonnet, plus ~$5 for writing. Using GPT-4o for citations reduces writing costs. Budget $25-30 per complete attempt for robust planning.

Can I use this outside machine learning? The current implementation targets ML domains requiring GPU computation. However, the template-free architecture theoretically generalizes. Forks adapting the experimental backend to other computational sciences are plausible community extensions.

Is this actually generating novel science? The peer-reviewed workshop paper demonstrates genuine capability for modest novelty. However, v2 excels at systematic exploration and combination of existing techniques rather than paradigm-shifting breakthroughs. It's a powerful accelerator, not a replacement for human creativity.

How do I handle Semantic Scholar rate limits? Obtain an API key for higher throughput. Without one, the system functions but may skip novelty checking or citation phases. For urgent work, this fallback is viable though suboptimal.

What about the replication crisis—does this help or hurt? Paradoxically, both. Cheap automated generation risks flooding venues with low-quality submissions. But systematic, cheap replication and negative result documentation—traditionally unpublishable—could strengthen scientific foundations. The outcome depends on community norms and review adaptation.

Is the generated code safe to run unsandboxed? Absolutely not. The repository explicitly warns of dangerous package installation, uncontrolled web access, and unintended process spawning. Docker sandboxing is mandatory, not optional. Treat this system with the caution due any autonomous code-execution agent.

Conclusion: The Future Just Arrived—Are You Ready?

AI Scientist-v2 isn't a perfect system. Success rates remain modest. Costs, while trivial compared to human researcher time, aren't zero. The sandbox requirement reflects genuine risks from autonomous code execution. And the peer-reviewed paper, while historic, was a workshop submission rather than a top-tier conference breakthrough.

But here's what matters: the barrier has been crossed. An AI system autonomously generated a hypothesis, designed experiments, debugged failures, analyzed results, wrote a manuscript, and survived academic peer review. That genie won't return to the bottle.

For researchers, this is a force multiplier of unprecedented power. The drudgery—implementation, debugging, literature search, formatting—can increasingly be delegated. Human creativity focuses where it matters: asking the right questions, recognizing profound patterns, providing ethical guardrails, and making the intuitive leaps no tree search can replicate.

For the scientific community, adaptation is imperative. Review processes must evolve to evaluate AI-generated work fairly. Publication norms must accommodate (and mandate) disclosure. And we must collectively ensure this capability strengthens rather than dilutes scientific quality.

The repository awaits at https://github.com/SakanaAI/AI-Scientist-v2. Install it. Experiment with it. Critique it. Contribute to it. Because whether you're excited or terrified, AI Scientist-v2 is already rewriting the rules of scientific discovery—and the researchers who master this tool will define the next decade of machine learning innovation.

What's your first research question going to be?