Abstraction based on Shape

Something like "Form Follows Function"

As we turn our focus to processing data streams with higher order resource graphs, it could be beneficial to structure the data feed to mirror the engine.  I had blogged about Synchronization Sets in earlier posts.  It seems that the Resource DSL and Data DSL could have a binding in this area.

Note: the shape may not be a one to one match, and it would only roughly apply to highest level of grouping.  For instance A could consume a complex data structure represented by Data A, the point is, there is a mapping between the resources and the data for each part of the resource Graph.  Note, here also could be Data components that are shared between Synchronization Sets.  The Data graph is independent and its own entity, but it is aware of the resource graph.   

Each part of the Resource graph can change its not just scaling at the highest level.  Inner parts can be adjusted.  This means the mapping between data and components must be at the inner level.

This is important as when scaling resources, the data must also change.  If you have sharded a data set into 50 segments and you change your resource configuration to be 500 then your data needs to reflect that.  

There is separation of concerns, but both the Data DSL and the Resource DSL need to communicate. The k8's infrastructure would provide the mechanism for this communication.  Resources and Data would be represented as CRDs in the system.  Updates to one CRD would result in updates to the other.   

This is why we use the term DSL to describe the components.  They are configurations, but they can respond to changes.  This is not a static system where data is positioned once and is frozen in place.  Data adapts and positions itself according to the changing structure of the resources.  In this way, it should be possible to have a higher degree of control over data locality.