The RL Scaling Paradox: Why 1 Billion Transitions Isn't Enough - SHARSA

This research investigates the *scalability of offline reinforcement learning* (RL) by testing existing algorithms on diverse robotics tasks with datasets up to *1,000 times larger* than standard benchmarks. The authors discover that traditional methods often fail to improve even with vast amounts of data and compute, identifying a *"curse of horizon"* as the primary obstacle. This phenomenon occurs because errors in *value learning* accumulate over time and the mapping for *policy learning* becomes excessively complex. To address this, the study demonstrates that *horizon reduction techniques**—methods that break tasks into smaller segments—significantly enhance performance. Specifically, the authors introduce **SHARSA**, a streamlined hierarchical approach that combines **rejection sampling* with *n-step returns* to unlock better scaling behavior. Ultimately, the work advocates for a shift in RL research toward evaluating algorithms on *large-scale, complex tasks* to ensure they can truly leverage the potential of big data. Your summary accurately captures the core findings and contributions of the research. To expand on your points, the study highlights several key mechanics behind the *"curse of horizon"* and details exactly how *SHARSA* and other horizon reduction techniques unlock the scalability of offline reinforcement learning (RL). *The "Curse of Horizon" in Traditional RL* The authors tested standard offline RL algorithms (such as IQL, CRL, and SAC+BC) on complex, long-horizon robotics tasks from OGBench, scaling datasets up to 1 billion transitions—**1,000 times larger** than typical offline RL datasets. They found that traditional algorithms consistently saturated well below maximum performance, completely failing on the hardest tasks like `cube-octuple`. The authors trace this failure to the **"curse of horizon"**, which manifests in two distinct ways: *In Value Learning:* Traditional algorithms rely on 1-step temporal difference (TD) learning. The authors demonstrate that biases and errors in prediction targets inevitably accumulate over long horizons during TD learning. Controlled experiments showed that these compounded errors cause the algorithms to fail, and importantly, simply increasing model capacity (up to 591M parameters) or tuning hyperparameters does not fix this accumulation. *In Policy Learning:* Even if the value function is perfectly learned, the policy itself struggles because the mapping between a given state and the optimal action becomes exponentially more complex as the distance to the goal increases. *Horizon Reduction Techniques* To combat this, the researchers demonstrated that explicitly reducing the effective decision horizon significantly improves the ability of algorithms to scale with data. *Value Horizon Reduction:* Using methods like *n-step returns* (e.g., n-step SAC+BC) allows the algorithm to look further ahead in a single update, drastically mitigating the bias accumulation seen in 1-step TD learning. *Policy Horizon Reduction:* Employing *hierarchical policies* breaks the long-term task into a high-level policy that generates intermediate subgoals, and a low-level policy that takes actions to reach those subgoals. This resembles chain-of-thought reasoning, breaking a complex mapping into simpler subtasks. *SHARSA: A Minimal, Scalable Solution* To reduce both value and policy horizons simultaneously, the authors introduced *SHARSA* (State–High-level Action–Reward–State–High-level Action). Designing a full hierarchical offline RL method is traditionally difficult because extracting a high-level policy using standard first-order gradients often fails in complex or discrete state spaces. SHARSA bypasses this by using: 1. *Rejection Sampling:* Instead of relying on problematic reparameterized gradients, SHARSA uses an expressive flow policy to sample multiple potential subgoals and simply selects the best one based on a high-level value function. 2. *n-step SARSA:* It relies on simple, robust behavioral value functions (SARSA) combined with n-step returns to handle the value learning, minimizing the need for extensive hyperparameter tuning. *A Call for Scalable RL Research* The paper concludes with a strong call to action for the RL community. Because success on standard 1M-transition datasets does not guarantee that an algorithm will effectively utilize massive datasets, the authors advocate that future offline RL methods must be evaluated on *large-scale datasets and complex tasks* to truly measure their scalability and potential in the era of big data. The description, research summary based on a human template and the video were generated by Google's NotebookLM on 24th June 2026.