End-to-End GenAI Observability: Infrastructure, Agents, and Applications

Join this L300 session to discover how to build comprehensive observability across your entire GenAI stack: from GPU utilization and infrastructure health to agent decision flows, tool invocations, and application performance. CloudWatch OTLP for Metrics Leverage OpenTelemetry Protocol (OTLP) for standardized metrics collection Configure CloudWatch to receive OTLP metrics from distributed GenAI workloads Enable vendor-agnostic observability across your infrastructure Infrastructure Observability Monitor inference workloads on Amazon EKS using CloudWatch Container Insights Collect GPU metrics (utilization, memory, power draw, temperature) with Nvidia DCGM exporter Visualize infrastructure and inference performance using Amazon Managed Prometheus and Grafana Leverage community-driven Grafana dashboards for zero-configuration monitoring Inference Performance Metrics Track VLLM (open-source LLM serving tool) performance metrics Monitor time-to-first-token and end-to-end request latency Analyze token consumption and generation patterns Agent Observability Beyond Agent Core Instrument agents deployed on EKS (outside Bedrock Agent Core runtime) using OpenTelemetry Enable auto-instrumentation without code changes using AWS Distro for OpenTelemetry Configure telemetry collection through environment variables in Kubernetes deployments Use CloudWatch GenAI Agent Core observability capabilities for agents on any platform Session and Trace Management View complete traces with timeline and trajectory visualizations Track token counts (input/output) for each model invocation Analyze agent reasoning, tool calls, and decision flows Correlate traces across multiple agent interactions using session IDs Access integrated logs, metrics, and traces in CloudWatch Key Takeaways Zero infrastructure overhead for observability setup OpenTelemetry-based approach works with agents on EKS, ECS, EC2, or any platform CloudWatch GenAI observability features extend beyond Bedrock Agent Core runtime For more events like this see our Cloud Operations Enablement series: https://aws-experience.com/amer/smb/e...