From Pixels To Agents: Optimizing On-Device Performance of ... - Savas Ozkan, Samsung Electronics

Join us at the premier vendor-neutral open source conference, where developers and technologists come together to collaborate, share knowledge, and explore the latest innovations and advancements in open source technology. Learn more at https://events.linuxfoundation.org/ From Pixels To Agents: Optimizing On-Device Performance of Confidential Computing in AI Evolution - Savas Ozkan, Samsung Research UK, Samsung Electronics Unlike server-side confidential AI, on-device confidential AI must balance strong protection of sensitive personal data with efficient operation under limited computational resources. In this talk, we explore the impacts of CC on on-device AI performance for various AI models and tools by identifying some root-causes. First, we recognise that CC overheads vary across AI models during critical operations such as data read/write, model loading and inference phases, supported by detailed experiments. Second, we investigate multiple designs for AI agent tools in CC, especially by considering different AI memory modules, that present distinct overheads compared to traditional AI models. To enable systematic evaluation, we develop a modular software framework integrated with the open-source ISLET CC project. This framework supports configurable benchmarking of AI agent tools, and will be publicly released to foster the reproducibility and collaboration within the CC community. Lastly, since these performance drops can negatively impact the user experience, we propose a set of techniques that minimise the overhead related with model loading while ensuring robust privacy protection.