This Eckerson Group report recommends 10 vital steps to attain success in DataOps.READ MORE
In 1955 the first Fortune 500 list appeared with General Motors, Jersey Standard, U.S. Steel, General Electric, Esmark, Chrysler, Armour, Gulf Oil, Mobil and DuPont all enjoying their debut in the top 10. Of all the companies that appeared on the Fortune 500 in 1955, only 53 remain on the list today.
The number one disruptive force leading to the fall-out according to analysts at Credit Suisse? Automation.
None of this is lost on the Fortune 500’s oldest company, the Bank of New York Mellon, founded by Alexander Hamilton as the Bank of New York all the way back in 1784. One of its subsidiaries, Eagle Investment Systems, helps worldwide financial institutions like Charles Schwab, CIGNA, JP Morgan Chase & Co., US Bancorp, and many others grow their assets through an innovative suite of data management, investment accounting, and performance measurement solutions. Eagle Investment Systems is a data pioneer that knew it needed to digitally transform itself even further to compete with today’s younger, well-funded data innovators that launched with a lot of the hard-to-find engineering talent that other companies like Eagle have to chase after.
“We have a ton of data,” Eagle Digital Transformation Engineering Leader Chris Stirrat said during his presentation at Strata New York last September. But according to Stirrat, that also meant they had a “big data problem.” He explained their current product line had existed for 25 years. It was time for a tune-up. “We were sitting on all this data for our customers and it’s locked away in an Oracle database.” Sure, they could run SQL queries to get at the data, but they wanted to use the data more creatively, allowing for ad hoc queries and data blending that could unearth valuable new insights. They also wanted to make it easier for end users to create customized and dashboards without having to ask IT to deliver requested changes that would take months to complete. According to Stirrat, “That really lends itself much more to a data lake with many different types of database technologies.”
Eagle Investment Systems decided they needed to digitally transform their operation by building a highly resilient and scalable big data platform that powers everything. Their caveats: fast time to completion and no downtime during the switchover from the existing system to the new one. In order to build out the new system without disrupting their existing business, Eagle started a brand new team completely from scratch. However, the team was relatively small, and Eagle wanted to automate their data engineering and data ops processes, so they decided to use commercial software wherever possible to minimize internal development.
Eagle had a few key criteria they routinely use when selecting commercial vendors beyond the obvious point that they had to solve a need:
Eagle Investment Systems also set a philosophy to “fail fast”. For example, they evaluated various big data vendors and quickly did some due diligence, settling on Hortonworks as their Hadoop provider. They could have spent months doing the analysis to choose a vendor, but with this fail-fast philosophy, they were prepared to change suppliers if it turned out they made a mistake. This approach was very different from the way the rest of the bank made decisions, but it was also critical in order to get the project done quickly. It was acceptable to make a mistake, Eagle reasoned, as long as you could quickly recognize it and quickly make the needed change.
After they stood up their Hadoop infrastructure, they had to get the data into their lake. “We’ve got this new data lake and great functionality with parallel calculations that scale out,” Chris said, “But we had to get data into this thing.” That’s where Infoworks came in. Through Infoworks, Eagle was able to automate the flow of data into their system, replacing time and labor-intensive processes and hand-coding with automated processes. With Infoworks, Eagle could:
“(Now) everything we do is automated,” said Chris. Plus: “We were able to fully operationalize the data pipelines we created. You can schedule them… you can run them on-demand.”
But this is only the beginning. Now that Eagle Investment Systems is able to efficiently ingest all their existing data into the data lake, their plan is to use Infoworks to automate the building of data pipelines that can pull data from all kinds of new data sources. More data means more kinds of analysis for their big data platform to perform even more informed decision making while delivering more robust recommendations for customers.
Want to learn more about how Infoworks empowers digital transformation through end to end big data automation? Check out this brief introductory video on the capabilities and solutions that Infoworks provides.
And be sure to check out the summarized video presentation (12 minutes) Chris gave at the Strata Data Conference here.