A (In)Cast of Thousands: Scaling Datacenter TCP to Kiloservers and Gigabits
Carnegie Mellon University Parallel Data Lab Technical Report CMU-PDL-09-101, Feb 2009.Vijay Vasudevan, Amar Phanishayee, Hiral Shah, Elie Krevat, David G. Andersen,
Gregory R. Ganger, Garth A. Gibson
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
This paper presents a practical solution to the problem of high-fan-in, high-bandwidth synchronized TCP workloads in datacenter Ethernets—the Incast problem. In these networks, receivers often experience a drastic reduction in throughput when simultaneously requesting data from many servers using TCP. Inbound data overfills small switch buffers, leading to TCP timeouts lasting hundreds of milliseconds. For many datacenter workloads that have a synchronization requirement (e.g., filesystem reads and parallel dataintensive queries), incast can reduce throughput by up to 90%.
Our solution for incast uses high-resolution timers in TCP to allow for microsecond-granularity timeouts. We show that this
technique is effective in avoiding incast using simulation and real-world experiments. Last, we show that eliminating the minimum
retransmission timeout bound is safe for all environments, including the wide-area.
KEYWORDS:Cluster-based storage systems, TCP, performance measurement and analysis
FULL TR: pdf