The Speed of Metrics, the Fidelity of Traces: Architecting Post-Collection Aggregation - Zack Owens

Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Mumbai, India (18-19 June, 2026), Yokohama, Japan (29-30 July, 2026), and Shanghai, China (8-9 September, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io The Speed of Metrics, the Fidelity of Traces: Architecting Post-Collection Aggregation - Zack Owens, New Relic As organizations adopt observability practices, they face a scalability paradox: systems now generate petabytes of traces and logs, but querying this raw telemetry over long time horizons becomes prohibitively slow and expensive due to the data volume. The standard solution of pre-aggregating high-cardinality telemetry into metrics at collection time through features in the OpenTelemetry collector works well for known patterns but fails when engineers need to ask new questions about historical data. This creates an uncomfortable choice for engineers and operators: fast dashboards with pre-aggregated metrics, or high-fidelity traces and logs that become unusable beyond short time windows. This talk presents a post-collection aggregation approach that enables fast queries over long time periods of detailed telemetry without changes to collector-side configuration. This session explores techniques for incremental view materialization that work with timeseries data. Attendees will leave with concrete architectural patterns which are applicable to open source databases like ClickHouse or OpenSearch to answer novel questions without sacrificing query speed or data fidelity.