Mesurer l’immesurable : Comment évaluer les systèmes à base d’IA générative ?

Presentation by: Erin Pacquetet (SCIAM) 📕 Summary: Generative AI is revolutionizing application development, opening up a variety of uses: assistants, content generation, augmented search, and facilitating complex tasks. But a major challenge remains: accurately evaluating products based on models that are both creative and unpredictable. This session explores this paradox: leveraging LLM while controlling the evaluation of its results. We will see how to adjust criteria and methods to assess technical accuracy, consistency, and business relevance. The program includes: limitations of traditional metrics, automated evaluation via "LLM-as-a-judge" (and its biases), the importance of human evaluation, and continuous monitoring to detect deviations and side effects. We will analyze the case of a RAG chatbot, where linguistic creativity and the requirement for truthfulness clash. The evaluation, balancing factuality and fluency, controls accuracy without controlling the question asked. This real-world case study will serve as our guide to implementing a comprehensive and reproducible evaluation pipeline. This session provides benchmarks and tools for methodically evaluating generative systems and leveraging them as a strategic asset in AI. Recorded in April 2026 in Paris, Palais des Congrès, Porte Maillot. 🔥 To stay up-to-date with Devoxx France news, follow us on: LinkedIn:   / devoxx-france   Bluesky: https://bsky.app/profile/devoxx.fr Visit our website: https://www.devoxx.fr/