The Rise of the Citizen Data Engineer

Data Engineering

Written by Dominic Sartorio - March 24, 2021 | Category: Data Engineering

The Rise of the Citizen Data Engineer

Business groups of all shapes and sizes have digitally transformed and are implementing a wide variety of data science and data analytics projects.  Whether it’s the marketing group, sales operations group, customer service, finance, supply chain, product or any other function, these groups’ hunger for well-governed, high-quality and up-to-date data, ready-made for their analysis needs, has never been greater.  Up to now, many organizations have tried to meet this need with centralized data engineering teams, but such teams can struggle to keep up with the increasing demand from all of these various functions with the timeliness that they are expecting.

Enter the Citizen Data Engineer.  Usually embedded within business groups outside of IT, they are often the most hands-on data-savvy analyst within their respective analyst teams.  While typically not data engineers by training, many have picked up SQL and scripting skills along the way, and are able to stand up their own simple data pipelines to address their groups’ specific needs.  Additionally, citizen data engineers act at the service of their own group, laser-focusing on getting the data they need and getting it now.  As such, there is little patience for heavy enterprise data platforms that take days or weeks to implement the data pipelines they need, and in the absence of a fit-for-purpose tool for their needs, they often go their own way, hand-scripting their own pipelines or cobbling together various point tools to get the job done.  Agility and self-service are critical.

While not a brand new role (Dice, the career website, picked up on the increasing frequency of this role over a year ago: https://insights.dice.com/2020/01/10/citizen-data-engineer-year-2020/), the citizen data engineer is still emerging in many organizations.  Most Chief Data Officers and other enterprise-wide data leaders certainly don’t want to slow down the business and are striving to empower this role, but are struggling to do so with centrally managed tools that don’t meet the citizen data engineers’ need for “just in time” pipeline creation and subsequent change management.  This has resulted in a “wild west” situation, in which each individual citizen data engineer chooses their own tools that meet their own needs, and adopts their own standards for data quality, data recency, data privacy and other “Data SLAs” for their respective teams.  

The Cloud further empowers the citizen data engineer, and has further exacerbated this “wild west” problem.  As more organizations stand up data lakes and data warehouses in the Cloud (e.g. Databricks, Snowflake, Amazon Redshift, Azure Synapse, Google Bigquery etc), it is very easy for the citizen data engineer to get going with cloud-hosted tools without waiting on IT or a service provider to provision data integration software for them.  Indeed, as the pandemic forced more organizations to work remotely and move their data analytics to the cloud, this past year has seen a proliferation of citizen data engineers standing up data pipelines in the cloud.  Meeting this demand, there has also been a proliferation of point tools that promise agility and self-service of data pipelines in the cloud – but then fail to ensure consistency in data quality and governance across teams, as well as fail to keep up as each team’s various data demands grow.  

As a result, many organizations struggle to strike a balance.  How to empower citizen data engineers to be highly agile and self-serve, while still maintaining enterprise-wide standards for data governance and and scalability?  

Infoworks can help!   As the world’s only single platform for enterprise data operations and orchestration, Infoworks provides an end-to-end solution for onboarding, preparing and operationalizing data for all data analytics use cases, and includes capabilities for data governance as well as monitoring for performance and other SLAs.   But it is not just intended for central data engineering teams standing up enterprise data warehouses and data lakes; Infoworks is also intended to help citizen data engineers all over the organization be successful.   As a cloud-hosted solution with a step-by-step user experience for data onboarding and pipeline creation, citizen data engineers can get started quickly, while also having the confidence that the product can scale as their data volumes and complexity grows over time.   Moreover, Infoworks enables citizen data engineer self-service and high productivity with a highly automated approach – just connect and crawl source schemas and auto-generate onboarding and preparation jobs from the resulting metadata – without writing custom scripts that must then be maintained.  Meanwhile, the central teams can use Infoworks to establish baseline governance, such as onboarding curated datasets to enterprise data lakes for all citizen data engineers to consume, as well as dashboards to monitor usage, performance and other SLAs.   

As such, Infoworks brings the best of both worlds:  Central data engineering teams are empowered to manage enterprise data lakes with consistent governance standards, while citizen data engineers are empowered to get going quickly on their own.

If you’re interested in learning more, then why wait, try Infoworks yourself!  Visit our Test Drive at https://www.infoworks.io/try/

 

 

About this Author
Dominic Sartorio
Dominic Sartorio is Senior Vice President, Product of Infoworks. Prior to Infoworks, he held senior leadership positions in product and product management at a number of enterprise software companies including Informatica and Potegrity. At Informatica, Dominic served in various senior product management roles, and as VP of Product Operations was responsible for coordinating product planning, portfolio management, and go-to-market across all of Informatica’s product lines. As SVP, Products and Development at Protegrity, Dominic was responsible for product strategy and management, R&D, global customer support, and technology partnerships.