Introducing Social Hash Partitioner, a scalable distributed hypergraph partitioner

Facebook serves billions of people each day. To support this scale, we distribute our workloads at many different levels. Our users' traffic is routed to one of several worldwide data-centers to improve scalability, fault tolerance and latency. As a single host has limited storage and compute resources, our storage systems shard data items over multiple hosts and our batch jobs execute over clusters of thousands of workers, to scale and speed-up the computation. At the heart of these systems, is a fundamental decision on how to assign a set of elements (requests, data items, or jobs) to one of several groups (data centers, hosts, or workers).