QuantUniversity AI Spring School: Poor Measurement in AI

AI performance is often measured using benchmarks that fail to capture real-world effectiveness. Common pitfalls—like confirmation bias, task contamination, and Goodhart’s law—can lead to misleading assessments, raising concerns about the reliability of today’s AI evaluation practices. In this insightful session, Patrick Hall will break down the risks of poor AI measurement and explore cutting-edge solutions, including NIST ARIA, H2O.ai HCAT, and stakeholder-informed evaluations that ensure AI systems are fair, transparent, and robust. 🔎 What you'll learn: ✅ The hidden risks in AI benchmarking and evaluation ✅ How AI measurement failures impact decision-making ✅ Promising new frameworks and methodologies shaping the future of AI assessment