In this first diagram we see the concept of the Compute Wheel. It consists of compute resource positioned on nodes that each have their own local storage. I have suggested MemSQL database in the past for it's ability to sync via Kafka, but it could be any storage mechanism that can accept a data feed. SQL databases do offer time sliced querying of data - this would be needed.
Zooming out, we can start to see the pattern of self similarity emerge. Complex systems are comprised of simpler components. As I have stated in previous posts, the work is being done at the lowest level. The higher levels are constructs that control the shape of the solution, possibly via AI feedback between the layers.
In earlier posts I had mentioned that the ability to select the right target is what really makes a difference. Could this be a viable target for development effort? It could be the pattern that allows the implementation of the "Windowed Mandelbrot" that I described earlier, a name created based on the shape of the overall blueprint that expresses the concept that only parts of the graph are in existence at any given point in time. This may be the way to realize the full potential of containers and orchestration.
Note data locality with respect to the initial base data is at the node level. As data is aggregated and transformed by higher level constructs re-localization is performed. The arrows in the diagrams above represent data transfer.
The question here is what types of patterns can be abstracted from the diagrams above? The process of building this framework should not be seen as creating a static model. It should be in finding and exploiting patterns for processing data. Free form application of the basic patterns will result in more flexible systems.
No comments:
Post a Comment