So far, we've considered ingestion, transformation, and the rest, as if you have only one such process going on. But we all know that's not true.
In reality, coordinating how data streams into a lake and through its transformations and motions involves many moving parts. You have numerous external systems, each with their own load requirements. Let's take a look at the way Infoworks lets you orchestrate the dance.
Your fully operational unified data platform might well involve dozens of ingestion sources, pipelines, targets, cubes, queries, exports, and more. 1 DataFoundry's workflow design allows you to initiate, pause, resume, and restart these processes, in parallel or in sequential dependence.
The dependencies for each task are under your granular control. Tasks may depend on mere completion, or success, or failure, for either all or at least one parent.
Further, you can implement decision trees within workflows, which can launch pipelines, and even launch other workflows. You have the logical flow control you might expect from a visual coding environment, but at a level of automation and abstraction which expands the capabilities of a broader range of technical staffing. And, where appropriate, you can run external scripts to integrate with external workflow systems.
As workflows run, you have continual insight into their completion state. You can pause a workflow, and even restart from a failure point without having the restart the entire workflow. And, to help with integrating DataFoundry with your broader ecosystem, you can launch even workflows externally via REST API.
So, what you have learned? 1 Ingestion and transformation can involve numerous external systems, each with their own requirements. 2 DataFoundry workflows let you coordinate and manage the impacts these processes have on your broader data ecosystem. 3 Tasks may run sequentially or in parallel, and you have granular control over how prior task completions initiate subsequent tasks. 4 Workflows can branch and launch new workflows. 5 Workflows can be externally launched via REST API. 6 And, Workflows let you monitor runs, pause them, and restart from failure points.
If you're looking for greater control over your full data ecosystem, look further into Infoworks.