AgentGym-RL: The Secret Weapon Beating GPT-4o with 7B Parameters

What if I told you that a 7-billion parameter open-source model just crushed GPT-4o at its own game? Not in some cherry-picked benchmark. Not in a narrow, single-turn math problem. We're talking about real-world, multi-turn interactive tasks—web navigation, scientific research, embodied reasoning, and deep search. The kind of tasks that make current LLM agents stumble, hallucinate, and loop endlessly.

Here's the brutal truth most AI researchers won't admit: single-turn reinforcement learning has hit a wall. You can fine-tune your model on human demonstrations all day long, but as Richard Sutton famously emphasized, true intelligence emerges from exploration and interaction with the environment—not imitation. The problem? Multi-turn RL is a nightmare. The search space explodes exponentially. Training signals become noisy and unstable. Agents forget what they learned three turns ago or get trapped in infinite loops of pointless exploration.

Enter AgentGym-RL, the framework that's rewriting the rules of LLM agent training. Developed by researchers from Fudan University and ByteDance, this isn't just another RL wrapper—it's a complete ecosystem for training agents that think, plan, and act across 27 diverse tasks spanning five real-world scenarios. And the results are insane: their ScalingInter-7B model doesn't just match GPT-4o—it surpasses it on WebArena benchmarks, while performing on par with DeepSeek-R1-0528 and Gemini-2.5-Pro.

Ready to discover how they pulled off this impossible feat? Let's dive deep into the architecture, the algorithms, and the exact code you need to replicate these results.

What is AgentGym-RL?

AgentGym-RL is an open-source framework designed specifically for training LLM agents through multi-turn reinforcement learning. Born from the research paper "AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning" by Zhiheng Xi et al., this project represents a fundamental shift in how we approach autonomous AI systems.

The core insight is deceptively simple yet profound: most existing RL methods for LLMs are stuck in single-turn paradigms. Math problems? Single answer. Code generation? Single output. But the real world doesn't work that way. Booking a flight requires multiple searches, comparisons, and form submissions. Conducting scientific research demands iterative hypothesis testing and experimental refinement. These long-horizon decision-making tasks require agents that can maintain coherence, learn from feedback, and adapt strategies across dozens of interaction turns.

AgentGym-RL addresses this gap through three architectural pillars:

Environment Diversity: Unlike prior work that tests on toy environments, AgentGym-RL integrates five distinct real-world scenarios—web navigation, deep search, digital games, embodied tasks, and scientific experiments. This isn't just about breadth; it's about generalization. An agent trained across these diverse environments develops robust, transferable reasoning capabilities.
Algorithmic Flexibility: The framework supports mainstream RL algorithms including PPO, GRPO, RLOO, and REINFORCE++, alongside complementary methods like SFT, DPO, and AgentEvol. This isn't a monolithic system—it's a modular toolkit that adapts to your research needs.
Training Stability: Through their novel ScalingInter-RL method, the team solved the notorious exploration-exploitation trade-off that plagues multi-turn RL. The result? Stable, sustained behavioral improvement instead of the catastrophic collapse that typically ends multi-turn training runs.

The project has already gained significant traction in the research community, earning Oral presentation at ICLR 2026—one of the most prestigious venues in machine learning. With datasets available on Hugging Face and a growing ecosystem of environments, AgentGym-RL is rapidly becoming the de facto standard for serious LLM agent research.

Key Features That Set AgentGym-RL Apart

Modular System Design

AgentGym-RL employs a clean, decoupled architecture organized into three specialized modules:

Environment Module: Provides diverse scenarios through a standardized server-client architecture with unified HTTP protocols and parallel request handling. This means you can swap environments without rewriting your training pipeline. The parallel request system is crucial—multi-turn interactions are inherently slow, and sequential processing would make training prohibitively expensive.
Agent Module: Encapsulates the complete reasoning and decision-making process during multi-turn interactions. Beyond simple action selection, it supports long-horizon planning and self-reflection mechanisms. This is where the magic happens—agents don't just react^{↗ Bright Coding Blog}; they think about their thinking.
Training Module: Implements the full RL pipeline with optimizations for multi-turn scenarios. Built on top of Verl but significantly extended to handle the unique challenges of agent training.

Comprehensive Environment Suite

The framework ships with production-grade environments that mirror real-world complexity:

Environment	Domain	What Agents Do
WebArena	Web Navigation	Navigate e-commerce, forums, CMS, and development platforms to complete complex tasks
Search-R1	Deep Search	Perform multi-turn retrieval and reasoning with search engines
TextCraft	Digital Games	Complete crafting tasks through natural language planning
BabyAI	Embodied Tasks	Execute grid-world navigation with text instructions
SciWorld	Scientific Tasks	Conduct experiments through iterative hypothesis-test cycles

ScalingInter-RL: The Breakthrough Algorithm

This is where AgentGym-RL transcends conventional approaches. ScalingInter-RL uses a progressive horizon-scaling strategy that fundamentally reimagines how agents learn:

Phase 1: Start with shorter interaction horizons. Agents quickly master simple tasks, building confidence and stable policy foundations.
Phase 2: Gradually extend the horizon. As competence grows, so does complexity. Agents learn to plan further ahead without the destabilizing noise of premature long-horizon exploration.

The genius? It mirrors human learning. You don't throw calculus at a student on day one. You build from arithmetic, through algebra, to the deep stuff. ScalingInter-RL applies this curriculum learning principle to reinforcement learning, and the results speak for themselves: higher final performance with more stable training dynamics.

Extended Verl Infrastructure

The team made substantial modifications to Verl to support agent training:

RolloutHandler: Manages complex attention masks, loss masks, position IDs, and sequence IDs across multi-turn trajectories. Handles historical messages, environment status, and reward computation automatically.
EnvClient: Provides clean abstractions for environment interaction—observation(), available_actions(), step(), reset()—with parallel environment initialization for efficiency.
Corrected Advantage Computation: Fixed GAE and REINFORCE++ implementations to handle both single-turn and multi-turn scenarios correctly.

Real-World Use Cases Where AgentGym-RL Dominates

1. Autonomous Web Navigation at Scale

Imagine deploying an agent that can independently navigate complex web applications—not just simple form filling, but multi-step workflows across authentication systems, search interfaces, and transactional platforms. WebArena's four domains (shopping, forums, CMS, development) represent $4.9 trillion in annual e-commerce volume alone. Traditional agents fail here because they can't handle dynamic content, session management, or error recovery across turns. AgentGym-RL-trained agents match or exceed GPT-4o on these tasks, with the 7B ScalingInter model achieving best-in-category performance on shopping and CMS tasks.

2. Iterative Scientific Research Assistance

SciWorld enables agents to conduct authentic scientific experiments through text-driven interactions. This isn't multiple-choice chemistry—it's iterative hypothesis formulation, experimental design, observation, and refinement. For pharmaceutical research, materials science, and climate modeling, this capability could accelerate discovery cycles by orders of magnitude. The multi-turn nature is essential: a failed experiment isn't an endpoint; it's feedback for the next iteration.

3. Intelligent Information Retrieval Systems

The Search-R1 integration creates agents that don't just search—they investigate. Multi-turn retrieval with reasoning means following leads, refining queries based on intermediate findings, and synthesizing information across sources. Compare this to current RAG systems that choke on complex, multi-faceted questions. AgentGym-RL agents plan their search strategy, execute it across turns, and adapt when initial approaches fail.

4. Embodied AI for Robotics and Simulation

BabyAI's grid-world tasks translate directly to robotics control, autonomous navigation, and industrial automation. The text-instruction interface means these capabilities extend to natural language command of physical systems. The multi-turn aspect captures real-world execution: observe, act, observe the result, correct. This closed-loop interaction is fundamentally different from open-loop planning that dominates current robotics.

Step-by-Step Installation & Setup Guide

Getting started with AgentGym-RL requires careful environment configuration. Follow these exact steps to replicate the paper's results.

Prerequisites

CUDA 12.4
Python^{↗ Bright Coding Blog} 3.10
PyTorch 2.4
Linux system (tested on x86_64)

Complete Installation

# Create isolated conda environment
echo "Preparing environment for agentgym-rl..."
conda create -n agentgym-rl python==3.10 -y
conda activate agentgym-rl

# Install PyTorch with CUDA 12.4 support
pip3 install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124

# Install Flash Attention for efficient attention computation
FLASH_ATTENTION_URL="https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.3/flash_attn-2.7.3+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl"
FLASH_ATTENTION_NAME="flash_attn-2.7.3+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl"
wget -q $FLASH_ATTENTION_URL -O $FLASH_ATTENTION_NAME
pip3 install $FLASH_ATTENTION_NAME
rm -f $FLASH_ATTENTION_NAME  # Clean up downloaded wheel

# Install AgentGym-RL training framework
cd AgentGym-RL
pip3 install -e .

# Install AgentGym environment infrastructure
echo "Preparing environment for agentenv..."
cd AgentGym/agentenv
pip3 install -e .
pip3 install transformers==4.51.3  # Pin transformer version for compatibility

Critical notes:

Flash Attention installation is mandatory for memory-efficient training of long trajectories
The transformers==4.51.3 pin prevents compatibility issues with newer versions
Both AgentGym-RL and AgentGym repositories must be installed in editable mode (-e)

Data Preparation

Download the pre-built RL dataset from Hugging Face:

# Dataset: AgentGym/AgentGym-RL-Data-ID
# Available at: https://huggingface.co/datasets/AgentGym/AgentGym-RL-Data-ID

This dataset contains curated interaction trajectories across all five environment types, formatted for immediate training use.

Environment Server Launch

Before training, you must launch the appropriate environment server. Refer to AgentGym's documentation for environment-specific setup.

REAL Code Examples from the Repository

Let's examine production-ready code from the AgentGym-RL repository, with detailed explanations of how each component enables multi-turn RL training.

Example 1: Launching AgentGym-RL Training with Fixed Interaction Horizon

This script trains an agent with a fixed maximum of 15 interaction turns, establishing baseline performance before progressive scaling:

# Standard AgentGym-RL training launch
# Located at: examples/train/AgentGym-RL/webarena_train.sh

bash webarena_train.sh \
  # Core training configuration
  algorithm.rounds_ctrl.type=fixed \           # Fixed horizon: no progressive scaling
  algorithm.rounds_ctrl.rounds=15 \            # Hard limit: 15 turns maximum per episode
  data.max_prompt_length=2048 \                # Task description prompt length cap
  data.max_response_length=8192 \              # Total trajectory tokens (excluding prompt)
  actor_rollout_ref.agentgym.task_name=webarena \  # Target environment
  actor_rollout_ref.agentgym.env_addr=http://localhost:8000 \  # Environment server URL
  actor_rollout_ref.rollout.max_tokens=512 \   # Single-turn response limit
  actor_rollout_ref.rollout.rollout_log_dir=./rollouts  # Trajectory storage for analysis

What's happening here? The fixed round controller creates a stable training environment with predictable episode lengths. This is crucial for baseline comparison—it eliminates horizon variation as a confounding factor. The max_response_length parameter (8192 tokens) must accommodate the entire multi-turn trajectory, not just a single response. This is dramatically larger than typical RLHF setups and reflects the fundamental difference between single-turn and multi-turn training.

Example 2: ScalingInter-RL Progressive Training Launch

This is where the breakthrough happens. The ScalingInter-RL configuration progressively extends the interaction horizon:

# ScalingInter-RL training with progressive horizon expansion
# Located at: examples/train/ScalingInter-RL/webarena_train.sh

bash webarena_train.sh \
  algorithm.rounds_ctrl.type=scaling_inter_stepwise \  # Progressive scaling mode
  algorithm.rounds_ctrl.steps_scaling_inter=100 \       # Increase horizon every 100 training steps
  algorithm.rounds_ctrl.rounds=[10,20,30] \             # Horizon progression: 10 → 20 → 30 turns
  # ... (other parameters identical to fixed mode)

The curriculum learning mechanism explained:

Training Phase	Steps	Horizon	Agent Behavior
Early	0-99	10 turns	Masters simple, short-horizon tasks; builds stable policy foundation
Middle	100-199	20 turns	Extends planning; explores medium-complexity strategies
Late	200+	30 turns	Executes complex long-horizon reasoning; maximizes task completion

The StepRoundsScheduler (invoked by scaling_inter_stepwise) implements this progression automatically. Compare this to naive approaches: starting with 30-turn episodes causes policy collapse due to excessive exploration noise, while fixed 10-turn training hits a performance ceiling on complex tasks. ScalingInter-RL gets the best of both worlds.

Example 3: Evaluation Pipeline

After training, evaluate your agent with the standardized evaluation script:

# Evaluation launch
# Located at: examples/eval/webarena_eval.sh

bash webarena_eval.sh \
  # Uses generation.yaml config for inference parameters
  # Automatically loads trained checkpoint
  # Connects to running environment server
  # Outputs structured results for benchmark comparison

The evaluation script uses AgentGym-RL/verl/agent_trainer/config/generation.yaml for generation hyperparameters, ensuring training-evaluation consistency. This separation of training and generation configs is deliberate—it allows independent optimization of exploration (training) versus exploitation (evaluation) behaviors.

Example 4: Core Configuration Schema

The ppo_trainer.yaml configuration reveals the full parameter space:

# Key configuration parameters from ppo_trainer.yaml
# Full path: AgentGym-RL/verl/agent_trainer/config/ppo_trainer.yaml

algorithm:
  rounds_ctrl:
    type: fixed  # or scaling_inter_stepwise
    rounds: 15   # base/maximum turns
    steps_scaling_inter: 100  # scaling frequency (ScalingInter-RL only)

data:
  max_prompt_length: 2048     # static task description length
  max_response_length: 8192   # dynamic interaction trajectory length

actor_rollout_ref:
  agentgym:
    task_name: webarena       # environment selector
    env_addr: "http://localhost:8000"  # server endpoint
  rollout:
    max_tokens: 512           # per-turn generation limit
    rollout_log_dir: "./rollouts"      # trajectory logging for debugging

Critical insight: The max_response_length (8192) versus max_tokens (512) distinction is architecturally fundamental. max_tokens controls single-turn verbosity—preventing runaway generation. max_response_length controls total episode memory—ensuring the model can attend to complete interaction histories. This dual-limit design prevents both per-turn bloat and episode truncation.

Advanced Usage & Best Practices

Hyperparameter Tuning for Your Environment

The steps_scaling_inter parameter controls curriculum pacing. For simpler environments (BabyAI), try 50 steps. For complex web navigation, 100-200 steps prevents premature horizon expansion. Monitor the rollout_log_dir trajectories—visual inspection of failure modes beats aggregate metrics for debugging.

Parallel Environment Scaling

The EnvClient supports parallel initialization. For maximum throughput:

Launch multiple environment server instances on different ports
Distribute rollouts across them using env_addr arrays
Monitor GPU memory—long trajectories consume significant VRAM for attention computation

Custom Environment Integration

AgentGym-RL's modular design supports custom environments. Implement the standardized HTTP protocol from AgentGym's 2nd-development tutorial, and your environment plugs into the full training pipeline automatically.

Debugging Multi-Turn Failures

Use the visualized interactive interface (documented here) to replay complete trajectories. Common failure patterns to watch for:

Loop detection: Agent repeating the same action
Premature termination: Agent ending episodes before task completion
Context truncation: Critical history falling outside attention window

Comparison with Alternatives

Feature	AgentGym-RL	Standard RLHF (PPO/DPO)	Single-Turn Agent Frameworks	Commercial APIs (GPT-4o)
Multi-turn RL training	✅ Native	❌ Limited	❌ N/A	❌ Black-box
Real-world environments	5 domains	Toy tasks only	1-2 domains	Unknown
Open-source	✅ Full	✅ Partial	Varies	❌ Proprietary
7B model performance	Matches GPT-4o	Far below	Below GPT-4o	N/A (is GPT-4o)
Training stability	ScalingInter-RL	Unstable for agents	N/A	N/A
Custom environment support	✅ Modular	Difficult	Limited	❌ Impossible
Reproducible research	✅ Full pipeline	Partial	Partial	❌ No

Why AgentGym-RL wins: It's the only open-source framework that combines multi-turn RL algorithms, diverse real-world environments, and stable training through progressive horizon scaling. Commercial APIs offer no training transparency. Standard RLHF tools collapse on multi-turn tasks. Single-turn frameworks fundamentally cannot address long-horizon decision making.

FAQ

Q: Can I use AgentGym-RL with my own custom LLM architecture? A: Yes. The framework is model-agnostic at the architecture level. The paper demonstrates results with Qwen2.5 models, but any transformer-based LLM compatible with Hugging Face transformers should work. You'll need to ensure your model fits within GPU memory constraints for the target max_response_length.

Q: How much GPU memory do I need for training? A: For the 7B models with full trajectory lengths (8192 tokens), expect 40GB+ VRAM per training instance. The Flash Attention dependency is mandatory precisely for memory efficiency—without it, training becomes impractical. Multi-GPU setups with DeepSpeed ZeRO-3 are recommended for production training.

Q: What's the difference between AgentGym and AgentGym-RL? A: AgentGym provides the environment infrastructure and behavior cloning baselines. AgentGym-RL adds the complete reinforcement learning training pipeline, including the modified Verl infrastructure, ScalingInter-RL algorithm, and RL-specific optimizations. You need both installed.

Q: Can I use this for commercial applications? A: Check the specific license in the GitHub repository. The underlying environments (WebArena, SciWorld, etc.) have their own licenses that may affect commercial deployment. The training framework itself is research-oriented.

Q: How does ScalingInter-RL compare to standard curriculum learning? A: Traditional curriculum learning manually designs task difficulty progressions. ScalingInter-RL automates this through horizon scaling within the same task distribution—no manual task selection needed. The agent naturally encounters simpler sub-tasks with shorter horizons and complex ones with longer horizons.

Q: What if my environment server crashes during training? A: The EnvClient includes automatic reconnection logic. For production training, implement environment server health checks and maintain redundant instances. The rollout_log_dir enables training resumption from checkpoints without losing trajectory data.

Q: How long does training take to match GPT-4o performance? A: Per the paper, ScalingInter-7B reaches GPT-4o parity on WebArena after full curriculum progression. Exact wall-clock time depends on GPU count and environment latency. With 8x A100 GPUs, expect several days for complete training. The fixed-horizon baseline trains faster but achieves lower final performance.

Conclusion

AgentGym-RL represents a paradigm shift in how we train autonomous LLM agents. By tackling the fundamental challenges of multi-turn reinforcement learning—environment diversity, training stability, and exploration-exploitation balance—this framework achieves what seemed impossible just months ago: open-source 7B models matching and exceeding top-tier proprietary systems on real-world interactive tasks.

The ScalingInter-RL algorithm is the secret sauce, applying curriculum learning principles to horizon scaling in a way that no prior work has attempted. The results are unambiguous: stable training, superior final performance, and genuine generalization across 27 diverse tasks.

For researchers, this is an unprecedented opportunity to study emergent agent behaviors in controlled, reproducible settings. For practitioners, it's a roadmap to building autonomous systems that can navigate the complexity of real-world digital environments.

The era of experience-driven agent intelligence is here. Don't get left behind with imitation learning and single-turn benchmarks.

👉 Star the repository, clone the code, and start training agents that actually do things: https://github.com/WooooDyy/AgentGym-RL

The future belongs to agents that learn by doing. AgentGym-RL is how you build them.

P.S. — The ICLR 2026 Oral presentation means this research is already influencing the field's direction. Early adoption now puts you ahead of the curve when these methods become standard practice.