From Shape to Execution

Moving from the Resource DSL and Data DSL to the Execution DSL

In the previous post I discussed the domain of the Resource and Data DSLs, this involved determining the shape of the compute fabric and then determining how to partition data to feed this fabric with data.  We touched on the realm of the Routing DSL by discussing replicating compute fabrics.

In this post, we will move to the realm of the Execution DSL.  Once we have a compute fabric materialized and data to feed into it, we must concern ourselves with orchestration between the different compute components, this is the job of the Execution DSL.

It can be imagined that we would have portions of the fabric that execute sequentially and other parts that execute in parallel.   In previous posts, I had proposed using a system of Channels created in Go to handle the orchestration.  A flow could be represented as a series of channels and would ensure that data is processed in an orderly manner.

Several patterns can be defined using Go channels.  Fan In, Fan Out, Cancellation, etc.  The different choices would give many options for optimization and would provide a great deal of freedom and expressiveness.

See : https://blog.golang.org/pipelines

This is similar to Flow based programming, and this system of channels might be a way to implement flows through the fabric of our compute engine.

Given that the fact that most channel patterns have straightforward implementations, it should be possible to define a graphical DSL or more practically a textual DSL that would enable the generation of the code defining the channels.  Off the shelf tools such as Sirius  https://www.eclipse.org/sirius/ and Xtext  https://www.eclipse.org/Xtext/ would allow the creation of such DSLs, complete with IDE's for editing and validation of the DSL.

The Externalized DSL would provide benefits. First, the flow could be externalized into files that could be checked into version control and then supplied to the engine at runtime. This could facilitate the evolution of a flow design.  Additionally, externalized flows could even enable the optimization of a flow.  The editing of the DSL could be automated to provide an auto tuning feature.  This tuning could be based on feedback from system monitoring; sort of a built in machine learning of sorts.

The Execution DSL defines the shape of the "track" that cars drive on.