Need To Move 1.2 Exabytes Across the World Every Day? Just Effingo

An anonymous reader shares a report: Google has revealed technical details of its in-house data transfer tool, called Effingo, and bragged that it uses the project to move an average of 1.2 exabytes every day. As explained in a paper [PDF] and video to be presented on Thursday at the SIGCOMM 2024 conference in Sydney, bandwidth constraints and the stubbornly steady speed of light mean that not even Google is immune to the need to replicate data so it is located close to where it is processed or served. Indeed, the paper describes managed data transfer as "an unsung hero of large-scale, globally-distributed systems" because it "reduces the network latency from across-globe hundreds to in-continent dozens of milliseconds." The paper also points out that data transfer tools are not hard to find, and asks why a management layer like Effingo is needed. The answer is that the tools Google could find either optimized for transfer time or handled point-to-point data streams -- and weren't up to the job of handling the 1.2 exabytes Effingo moves on an average day, at 14 terabytes per second. To shift all those bits, Effingo "balances infrastructure efficiency and users' needs" and recognizes that "some users and some transfers are more important than the others: eg, disaster recovery for a serving database, compared to migrating data from a cluster with maintenance scheduled a week from now."

Read more of this story at Slashdot.