Graph-based machine learning: Part 2 – Insight Data

In my previous post, we discussed the foundation of community detection using modularity optimization. One major constraint however, is that your graph needs to fit in memory. This quickly turns problematic as your number of nodes surpass billions, and the number of edges becomes trillions. Thankfully we can leverage distributed computation systems in order to solve this limitation. To do this we first need to define the state of a node so that it contains all the information needed during computation; this will serve as a basic structure to pass around between the machines of our distributed cluster.