announcing-ray.html?utm_content=buffer01978&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

This post announces Ray, a framework for efficiently running Python code on clusters and large multi-core machines. Like remote functions, actor methods return object IDs (that is, futures) that can be passed into other tasks and whose values can be retrieved with ray.get. The time required for deserialization is particularly important because one of the most common patterns in machine learning is to aggregate a large number of values (for example, neural net weights, rollouts, or other values) in a single process, so the deserialization step could happen hundreds of times in a row. To minimize the time required to deserialize objects in shared memory, we use the Apache Arrow data layout.