DSL for Execution of Batch operations (Part 4)

In previous posts, I had talked about the possibility of creating a DSL to express a data flow orchestrated by a set of Go channels.

Goals of this DSL:

1. Direct the creation of pods that contained 2 types of containers (orchestrators and workers)
2. Direct the creation of an execution flow graph expressed by Go Channels. 

How would this fit into an Operator system?

1. An operator would manage high-level CRDs representing the Pods and the flow patterns.  
2. The CRD based view of the structure would be transformed into the actual DSL via templating.
3. Controllers would detect changes in the CRDs and deploy new pods with new structures and flows based.

This tool looks like it could be part of the process:

https://github.com/k14s/ytt/blob/master/docs/README.md

Note:  it was suggested that validation could be applied to CRD structures in JSON.  This would still apply.  This base data could be supplied to something like a JTT templating engine.

This "layered" approach to DSL creation would help to standardize parts of the DSL language.  Templating could assemble the base structure of the DSL and then overlays could fine-tune the DSL before handing to a code generator.

 

https://developer.ibm.com/videos/introducing-the-yaml-templating-tool-ytt/

As for how to generate the code that expresses the set of channels and Goroutines expressed by the DSL structure there are a few Open Source projects out there that shed some light:

https://github.com/kamildrazkiewicz/go-flow