The modern data stack is many things to many people. It's BI plus AI! To get a better visibility into just what the modern data stack is, how it's evolving, and why it all matters, we look to Fivetran's "Multi-Cloud Modern Data Stack: Fireside Chat with Industry Trailblazers" for some insight. For Ali Ghodsi, the CEO and co-founder of Databricks, things are pretty clear: The modern data stack is the open lakehouse architecture, which combines elements of a data warehouse and a data lakes to provide high-quality data in support of BI and AI workloads. "What's going to happen over the next five years," Ghodsi says during the fireside chat, "is "companies like Fivetran and Databricks and many others that are to come [are] going to re-envision how things are done in this amazing new infrastructure that we have. And it's going to be much, much simpler.
Towards the end of each year, I receive a slew of predictions, from data/analytics industry executives and luminaries, focused on the year ahead. This year, those predictions filled a 49-page-long document. While I couldn't include all of them, I've rounded up many of this year's prognostications, from over 30 companies, in this post. The roster includes numerous well-known data/analytics players, including Cloudera, Databricks, Micro Focus, Qlik, SAS, and Snowflake, to name a few. Thoughts from execs at Andreessen Horowitz, the Deloitte AI Institute and O'Reilly are in the mix as well, as are those from executives at smaller but still important industry players.
Successful data strategies are built on a foundation of meticulous data management, creating enterprise architectures that "democratize" data access and usage, yielding measurable results from machine learning platforms. The reality, according to an examination of the emerging "AI organization," is that few data-driven organizations are able to deliver on their data strategy. A survey commissioned by Databricks and conducted by MIT Technology Review Insights found that a mere 13 percent of those polled actually achieve measurable business results. MIT Technology Review Insights said it polled 351 CDOs, chief analytics officers as well as CIOs, CTOs and senior technology executives. It also interviewed several other senior technology leaders.
As a result, all major cloud providers are either offering or promising to offer Kubernetes options that run on-premises and in multiple clouds. While Kubernetes is making the cloud more open, cloud providers are trying to become "stickier" with more vertical integration. From database-as-a-service (DBaaS) to AI/ML services, the cloud providers are offering options that make it easier and faster to code. Organizations should not take a "one size fits all" approach to the cloud. For applications and environments that can scale quickly, Kubernetes may be the right option. For stable applications, leveraging DBaaS and built-in AI/ML could be the perfect solution. For infrastructure services, SaaS offerings may be the optimal approach. The number of options will increase, so create basic business guidelines for your teams.
Pati says the data market has aligned with the vision laid out years ago by Dataworks founder, Amar Arsikere. A Google engineer who built some of the first products on BigTable, Arsikere, who is Infoworks CTO and chief product officer, envisioned a solution that could eliminate much of the technical complexity involved with moving, transforming, and managing data.