Big Techday 26: Securing confidential AI workloads on untrusted GPUs - TNG

Securing confidential AI workloads on untrusted GPUs with Intel TDX and NVIDIA Confidential Compute For years, processing confidential data with AI required on-premise infrastructure. TNG has followed this path, but the reality is that the hardware costs and operational complexity of maintaining private clusters are often impractical for rapid scaling. Since using untrusted cloud GPU providers bears the risk of loosing confidentiality, it is necessary to investigate how to build trust through hardware-enforced security while retaining the flexibility of cloud resources. In this talk, Dr. Benjamin Merkel, Dr. Marcel Lippmann, and Dr. Fabian Roll demonstrate how Intel Trust Domain Extensions (TDX) and NVIDIA Confidential Compute enable cryptographically verifiable execution environments on untrusted hosts. Leveraging remote attestation, they maintain workload integrity for sensitive operations like LLM inference. They share their learnings about these technologies, trust boundaries, and operational challenges. About the speakers: Dr. Benjamin Merkel: Dr. Benjamin Merkel is a Principal Consultant at TNG Technology Consulting. He is responsible for the smooth operation of AI services on TNG's own GPU cluster. His work focuses on the development of modern language models and the optimization of their performance. Before joining TNG, he earned his doctorate in physics with experiments in quantum optics. Dr. Marcel Lippmann: Dr. Marcel Lippmann is a Senior Consultant at TNG Technology Consulting GmbH, specializing in software architecture as well as Cloud and DevOps infrastructures. Since 2017, he has been developing technical solutions for industries such as automotive, e-commerce, and IoT – ranging from CI/CD pipelines and microservices to enterprise architectures. His current focus is on the further development and operation of a Kubernetes-based platform for Large Language Models on TNG's own GPU cluster. Additionally, as Data Protection Officer, he advises on compliance questions and regulatory requirements in data protection. Prior to joining TNG, he received his doctorate in computer science with research focuses on ontology languages, temporalised description logics, and database theory. Dr. Fabian Roll: Dr. Fabian Roll holds a PhD in mathematics and works as a Software Consultant at TNG Technology Consulting. During his doctoral studies at TU Munich, he researched geometric complexes in topological data analysis, with the goal of advancing leading methods for detecting nonlinear structures in datasets. Since 2024, he has been working at TNG in the area of AI infrastructure, building and operating a Kubernetes-based platform for AI models on TNG's own GPU cluster. Most recently, he has been focusing on Confidential Computing solutions for protecting sensitive AI workloads on untrusted infrastructure.