Traditional Kafka offset lag metrics don't actually measure how old your data is — they just tell you how far behind a consumer is, according to a technical report published on InfoQ. For teams running Apache Hudi pipelines at scale, confusing these two measurements leads to violations of data freshness service-level agreements, even when offset lag looks healthy. The report details how Twilio's data engineering team built a time-based lag monitoring system that measures the true age of data in their lake by reading Kafka checkpoints directly from Hudi commit files stored in S3.

At Twilio, the data lake processes over five trillion records monthly as of Q4 2025 across self-hosted Kafka clusters, peaking at 12.9 million messages per second on Cyber Monday 2025. The company's data lake underpins analytics, reporting, and machine learning across product lines including messaging, email, and voice. Standard consumer lag metrics like records-lag-max and even Hudi's kafkaDelayCount appeared normal, yet downstream analytics teams kept reporting stale data that was sometimes hours old. The issue wasn't Kafka throughput; it was a visibility gap created because Hudi Delta Streamer manages its own checkpoints, stored alongside table data in S3 and separate from Kafka's consumer group offset tracking, making standard lag monitoring tools like Burrow unable to determine whether Hudi had actually committed data to the lake.

The solution calculates a time-in-queue metric by reading the Kafka checkpoint from the latest Hudi commit file in S3, seeking to that offset in the Kafka topic, and measuring the timestamp difference between that message and the current time. The metrics reporter runs every fifteen minutes in production, fetching the latest Hudi commit from the active timeline and walking through commits in reverse chronological order to find the most recent one containing a deltastreamer.checkpoint.key. The algorithm then seeks to the checkpoint offset in each Kafka partition, reads the first message Hudi hasn't yet committed to the lake, and computes lag as the current timestamp minus that message's timestamp, capped at seven days for inactive pipelines. For multi-partition topics, the algorithm takes the record with the earliest timestamp across all partitions rather than the average or latest, representing the oldest data still waiting and the worst-case lag that matters for service-level agreement enforcement.

The algorithm had to handle several edge cases that emerged only in production. During migration periods when both legacy pipelines and new Kafka-sourced pipelines wrote to the same Hudi table simultaneously, the most recent commit frequently came from the legacy pipeline, which embedded no checkpoint key because it sourced data from S3 rather than Kafka. This caused the original algorithm to find no checkpoint metadata and default lag to the seven-day cap, producing false spikes. The fix was to stop assuming the latest commit is the right one and instead walk back through the timeline up to MAX_COMMIT_DEPTH commits — defaulting to 100 — until finding the most recent commit that contains a deltastreamer.checkpoint.key. The report also describes handling clock skew across producers, which can make lag appear artificially low or even negative when producer system clocks drift ahead, by flooring lag at zero and alerting separately on sustained negative values.

Each pipeline now defines its own service-level agreement threshold via a slaInMinutes field in its onboarding configuration, defaulting to sixty minutes for streaming pipelines and 1,440 minutes for batch. Breach is reported as a ratio between 0.0 and 1.0, where 1.0 represents a full breach, giving teams a leading signal that allows warning alerts at 0.7 and critical alerts at 1.0 rather than a single all-or-nothing threshold. The report notes that no changes were required to any existing pipeline because the reporter is purely an observation layer, requiring neither new instrumentation nor producer changes since the offset Hudi already commits to S3 and the timestamps already on Kafka messages are enough to compute true data freshness. Twilio's roadmap includes evaluating this approach for Iceberg pipelines as they migrate from Hudi and pairing the metric with anomaly detection libraries like Prophet or Luminaire rather than relying solely on fixed thresholds. Organizations treating observability as an afterthought will find themselves debugging invisible problems long after customers notice them, while those who instrument for the questions that matter — not just the metrics that are easy — build systems that reveal their own failure modes before users do.