The Automation of AI Research FARS Architecture and Empirical

FARS: A Large-Scale Fully Automated AI Research System 1️⃣ FARS (Fully Automated Research System) is a large-scale, end-to-end autonomous research system designed to independently execute the complete scientific workflow without human intervention, specifically operating within the AI-for-AI research domain. 2️⃣ The system’s architecture organizes the research process into four distinct, sequential stages: Ideation, Planning, Experiment, and Writing. 3️⃣ All stages are coordinated via an artifact-centered shared workspace that serves as persistent project memory. This structure preserves all intermediate artifacts—including research proposals, experiment plans, code, logs, and results—ensuring the entire research trajectory is fully auditable. 4️⃣ In its first public live deployment, FARS demonstrated unprecedented scale by producing 166 complete research papers across 67 fine-grained AI/ML topics. The deployment ran for 417 hours and consumed 21.6 billion model tokens. 5️⃣ To rigorously evaluate these outputs, the researchers conducted a large-scale human review study consisting of 282 structured reviews from 88 screened volunteer experts covering 140 of the generated papers. This process utilized ICLR-style rubrics alongside a custom AI Integrity Audit. 6️⃣ The human reviews revealed that while FARS can produce review-worthy artifacts, only 11.4% of the reviewed papers reached the formal acceptance threshold (an average score of 6 out of 10), indicating that the tail of genuinely strong research outputs remains relatively thin. 7️⃣ Despite this thin tail, FARS significantly outperforms prior automated research systems. When evaluated with the Stanford Agentic Reviewer, FARS achieved the highest mean rating (5.00) compared to all baseline systems (which scored 4.13 or lower) and was the only system to generate papers that met the automated acceptance threshold. 8️⃣ Reviewers consistently rewarded the system for its strong foundational execution, frequently citing methodological soundness, solid experimental design, clear motivation, and candid failure analysis as the generated papers' primary strengths. 9️⃣ The most persistent and critical weaknesses identified by reviewers related to experimental sufficiency and evidence quality. Even among the highest-rated papers, reviewers frequently penalized FARS for narrow evaluations, limited domain coverage, missing ablations, and poor generalization. 🔟 The specialized AI Integrity Audit exposed that lower-rated papers suffered from code-verifiable integrity failure modes, such as experimental-design pathologies, internal inconsistencies, hallucinated citations, and fabricated results. This proves that standard text-based evaluation is insufficient for AI-generated research, requiring deep, artifact-grounded human audits.