In the not too a distant future, the business world will be split into two camps – companies which have an agile analytics capability and those companies that get eaten by the first group. Even today, it is already clear that the data and analytics capability of a company will be a critically determining factor in its success over the competition.
Today, most companies typically implement a very limited number of data analytics use cases and deploy them production each year. If they are able to implement ten ( 10 ) new end to end use cases in a single year, most people would say they are doing well, But the state of the art for an organization is not just ten, but tens, with an “s”, of use cases per year. However, if you want to have a true digital transformation to occur in your organization, you will need to be able to implement hundreds or thousands of new analytics use cases in a year. That may sound like a high number, but the ability to conceive a data analytic use case, quickly implement that use case, then decide if you can use that information to guide your decision making is what separates the Googles and Amazons from the rest of the pack.
Some of the use cases will have lots of data, and some not so much. The ability to process a large amount of data is now table stakes. The true differentiator will be in the agility to implement large number of data-driven use cases, and implement them very quickly.
It is hard to imagine, but there was a time, when Google was using traditional data and analytics stack, traditional ETL and relational data warehouses for several of its core internal analytics, just like most companies are today. And much like most companies today, adding new use cases could take months. One of the jobs I had at Google was to migrate and automate its internal systems from a traditional analytics stack, to what became known as a “big data” stack. The goal was to be able to execute new use cases several orders of magnitude faster than before. We wanted to be able to create new data analytics use cases almost as fast as we could think of them. So we set out on a journey to automate the creation and operations of data analytics pipelines. This gave the company the agility to hyper-analyze all kinds of data about its operations.
Today every company is looking to emulate data companies such as Google, Amazon and Facebook in their own domains. Data and analytics has become a strategic company initiative. C-level executives and the boards of directors across all kinds of different industry segments are looking to gain the agility to transform their businesses. Retailers don’t just sell goods, they are now also data companies as they figure out what products are most popular and what associated products to promote to you. Oil & gas exploration companies are now data companies as they analyze real-time drilling data to optimize well depth and maximize the yield they get from an oil field. Shipping logistics companies are now data companies as they figure out strategies to minimize the number of stops and maximize truck utilization and gas mileage .
The real question you should be considering is: How do you and your organization become an agile data analytics powerhouse?
This is the fundamental problem we at Infoworks have set out to solve in a completely holistic way that has not yet been addressed by the industry… until now. At Infoworks, we believe the solution to enabling any company to become an agile data-driven company has three main components:
I will be writing more about the importance of agility in analytics and automation of data engineering and data operations in the months and years to come. If you take away anything from my first blog, is that the revolution in analytics that is happening now isn’t just about the velocity of data, it is about the velocity and agility with which you can execute new data use cases and projects.