DSL for Execution of Batch operations (Part 5)

Part 5:  How to limit resource consumption across Calculation Sets.

For the CalcSet_Pod based implementation of CalculationSet you will have a flow that uses the containers in a pod to execute the flow, but the container list in a Pod is an immutable structure.  You cannot change containers without restarting the Pod.  In the case where you have complex graph, spinning up a container for each node in the graph would not be practical (wasted resources).  Instead, it is suggested that containers be reused based upon the current state of the graph.  This does change the one function per container semantics and will complicate container creation, yet, as of this time, there does not appear to be a way to dynamically load and unload containers in a pod.  (Alibaba had mentioned something close to this that might be worth a look though)

For the CalcSet_List based implementation of the CalculationSet you will have a flow that is implemented across service calls to Deployments implemented with CalcWorkers_Pod's as the workers.  Again, if the flow is large you will not want to spin up all the pods at once.

Both scenarios can be orchestrated with Go Channels and the common DSL.

Workers are represented by Circles.  Note: all the circles do not need to be instantiated at the same time.  For instance, when transitioning from 1 to 2, in addition to doing the work in 2, a create signal could be sent to create 3 on the fly.   In a container implemented version, the options are less flexible as you cannot create on the fly, but it is envisioned that re-purpose events (similar to actors switch behavior) could be handled during the execution flow.

Note: Gluing method calls together in a similar way is supported in Gloo (See Function-level routing allows integration of legacy applications, microservices and serverless) : https://github.com/solo-io/gloo#using-gloo)  

This design above does similar function level orchestration, just within the confines of a strictly defined environment with respect to file systems and data locality. You can count on a function having local access to shared data on the file-system in the case of Container based workers (via shared volume mounts)  and for Pod based workers, you can count on collocation of the Data Pond.

Another facet of the framework is to allow time windowed batch processing across a set of K8's resource without worrying about issues syncing data structures across the distributed network

Another facet of the framework is to allow time windowed batch processing across a set of K8's resources without worrying about issues syncing data structures across the distributed network.

As per:  https://dzone.com/articles/atomicboolean-lock-idiom

in "Java Concurrency In Practice," Brian Goetz describes how objects can be safely shared across threads. He lists four options:

The object is thread-confined, meaning updates to the object are performed by only one owning thread

The object is shared read-only, meaning this object only needs one-time publication

The object is inherently thread-safe, meaning it performs synchronization internally

The object is guarded by some lock

Note: I am extending this principle above to the use case of distributed resources. 

Instead of using the other options that are clearly more code intensive, let the system itself guarantee that ALL data is based on the same snapshot.  This includes all derived data.  This means that inside a pod containers can cooperate and share intermediate results without versioning. 

This is the power of programming Kubernetes, by handling data locality and shared mutability at the system level we have greatly simplified the code base.  It reminds me of AOP, but for systems.  Handle the large concerns at the system level.  This simplifies downstream code.