Evolving Skynetes to have infrastructure layer

My previous posts on Skynetes were an attempt to describe a possible set of CRD's that would act to sync up statefull services in k8's. These services are based on idea that in memory objects are different across services and are dependent on snapshot data. It was assumed snapshots would be taken of data and that groups of services would be bound to a snapshot - with sets of services contributing to a global set of answers for that particular snapshot. In looking at the this problem deeper it appears that certain infrastructure related CRDS could help define the base environment to build such a system. At the base of this idea is breaking up a large compute problem via a shared nothing approach. For this to work each node in the compute mesh will need to satisfy some requirements. Each Node: 1. have a local database (MemSQL) that can ingest data from Kafka stream 2. have a kafka ingest component that will listen on sharded topics 3. have a snapshoting component that will snapshot local data 4. support versioned services that all contribute to global answer for a specific snapshot More to come on this but its looking like development should probably start not on the service versioning but on the infrastructure creation layer. Once this layer is built then service can be added to nodes as listed in step 4. Other operators out there are doing similar things at the service layer: (creating custom CRDs that abstract services) https://github.com/kontena/akrobateo might https://github.com/rancher/k3s/blob/master/pkg/servicelb/controller.go What would be needed here is a set of abstractions that sit over k8's nodes and maintain the desired state of each node via a set of controllers. Note on portability : Different cloud providers have different API for provisioning nodes. It should be possible to abstracSt out node definitions to allow deployment on the various cloud providers. I know that openshift is doing similar things with their code. Idea would be to start with a single cloud then expand out. A good starting point would be https://github.com/paulczar/gThe cp-cloud-compute-operator Some aspects of stateful sets / daemoansets may come into play here. The local database, kafka listener and napshotting component are good candidates for deamonsets. Using stateful set might allow control of local database without having to know about the node itself. Maybe this translates into stateful sets with pods containing these containers: local database in memory but backed up to NFS for durability kafka injest snapshot another pod would contain the services and istio could handle versioning the services sending requests for snapshot1 related requests to the correct service via layer 7 routing. MUltiple service versions can exist on the same node. Being able to decouple the infrastructure pod from service pod seems like a good idea. We do want local access a particular set of sharded data. So these pods need to be on the same node. Need some way of saying shard 1 is on node x and shard 2 is on node y As for sharding across nodes this seems to apply: https://docs.mongodb.com/kubernetes-operator/master/tutorial/deploy-sharded-cluster/. Could be a good idea to look into that operator to see how they are doing that.