An arbitrarily complex flow of operations can be realized as the combined output of a series of related sub calculations. Its form is that of a graph of nodes with child leaves (not recursive). Nodes could be at the pod level and containers (leaves) could share a workspace (the pod)
Data can be shared from container to container via the pod. The containers in a pod share local IPC address space. DaemonSet provisioned database existing at the physical node level.
This has layered caching built in. Data can be pushed as close to compute as possible flowing from the source to the bottom most leaves of the node structure; common points along a path would allow for caching at virtually any level.
Note the graph changes as the idea progresses through the graph. Container images used for A B C sections not necessarily needed for the D E F operations and so on. There is possibility of crop rotation like feeding of input data if you can change the image on a container at runtime (you can https://github.com/openkruise/kruise/blob/master/docs/tutorial/sidecarset.md). This prevents uber containers... and minimizes total footprint as resources from A B C are reclaimed for D E F
No comments:
Post a Comment