Netflix has redesigned the streaming pipeline powering Service Topology, its real-time map of service dependencies, after some instances experienced traffic surges up to 100 times typical levels while simultaneously handling heavy input-output enrichment tasks. The company described the overhaul in a recent post on InfoQ, explaining how it separated intermediary resolution from enrichment and persistence into three distinct stages, replaced gRPC with server-sent events for high-volume internal transfers, and now propagates backpressure to Kafka instead of discarding records. The updated architecture addresses production-scale challenges that emerged as Netflix's dependency-mapping system processed network-flow data from eBPF network flows, inter-process communication metrics, and distributed traces.
The previous design concentrated workload around popular destinations, creating severe imbalances when intermediary resolution brought relevant flows together. Netflix reports that certain instances handled up to 100 times the normal traffic volume while also executing I/O-intensive enrichment operations. The redesigned pipeline now operates in three phases: the first consumes multi-region Kafka streams, filters invalid records, batches data into five-minute windows, and creates initial aggregators; the second resolves intermediaries into direct application-to-application edges and redistributes results; the final stage enriches nodes with health, ownership, and metadata before writing them to the graph database. The IPC pipeline doesn't require the extra redistribution step because its metrics already describe application-level calls and arrive partitioned by application from the start, allowing single-stage aggregation.
The system uses Apache Pekko Streams to manage backpressure, sending demand signals upstream through processing stages until the Kafka consumer pauses when graph storage can't keep pace. According to the post, this approach leaves records in Kafka until capacity returns, resulting in delayed freshness under load rather than dropped data or an incomplete map—an outcome Netflix considers preferable to batch-generated maps that may already be stale during an incident. The company replaced gRPC between pipeline stages with server-sent events after finding that serialization, connection-pool management, and streaming-response memory pressure became expensive at its volume, describing SSE as lighter-weight and compatible with reactive backpressure. Netflix's processing fleet expands and contracts with demand, using consistent hashing based on the current list of healthy instances from the service registry to determine which instance owns each aggregator, automatically moving only affected aggregators when instances join or leave.
The separation of stages allows Netflix to redistribute work that previously concentrated on hot instances, addressing the bottleneck that emerged when resolution logic and enrichment competed for resources on the same machines. Raw flow records capture network hops through load balancers, NAT gateways, API gateways, and proxies rather than logical application dependencies, requiring processing to convert them into meaningful service-to-service edges. By splitting resolution from enrichment, the system can spread the computational load more evenly, preventing individual instances from becoming overwhelmed when popular destinations generate disproportionate traffic. The pipeline's backpressure mechanism ensures that when downstream components slow down—such as when graph writes lag—the system doesn't lose data by dropping records but instead delays ingestion, preserving completeness at the cost of temporary staleness. Netflix teams use Service Topology for incident investigation, blast-radius analysis, dependency understanding, and production change management, making data completeness critical even when freshness suffers briefly.
The post also describes historical reconstruction capabilities: rather than retaining complete graph snapshots or replaying event logs, Service Topology maintains time-windowed aggregator snapshots and property-level mutation history, enabling engineers to reconstruct topology at a specified moment and examine dependency changes around incidents. The architecture's ability to pause Kafka consumption under pressure rather than shed load means the system can recover without permanent data loss, a design choice that prioritizes eventual consistency over real-time responsiveness when capacity constraints emerge. The shift from gRPC to server-sent events for internal stage-to-stage communication reduces overhead while maintaining compatibility with the reactive backpressure model, keeping the gRPC API intact for Service Topology clients while optimizing internal data movement. The business case for maintaining topology freshness—even with occasional delays—reflects the operational reality that during incidents, engineers need accurate dependency maps more than they need maps updated within seconds, making completeness the higher priority.