Bandana: Using Non-volatile Memory for Storing Deep Learning Models

Typical large-scale recommender systems use deep learning models that are stored on a large amount of DRAM. These models often rely on embeddings, which consume most of the required memory. We present Bandana, a storage system that reduces the DRAM footprint of embeddings, by using Nonvolatile Memory (NVM) as the primary storage medium, with a small amount of DRAM as cache. The main challenge in storing embeddings on NVM is its limited read bandwidth compared to DRAM. Bandana uses two primary techniques to address this limitation: first, it stores embedding vectors that are likely to be read together in the same physical location, using hypergraph partitioning, and second, it decides the number of embedding vectors to cache in DRAM by simulating dozens of small caches. These techniques allow Bandana to increase the effective read bandwidth of NVM by 2-3 and thereby significantly reduce the total cost of ownership. An increasing number of web-scale applications are relying on deep learning models, including online search (Clark, 2015), online ads (Zhou et al., 2018) and content recommendation systems (Covington et al., 2016). Therefore, application providers are dedicating ever more compute and storage resources to training, storing and accessing deep learning models. For example, at Facebook, thousands of servers are dedicated to storing deep learning models to recommend relevant posts or content to users. The deep learning features are often represented by vectors called embeddings, which encode the meaning of each feature, such that similar vectors are closer in the embedding Euclidean space. To compute the most relevant posts to serve each user, Facebook uses two types of embeddings: user and post embeddings.