AWS, K8's and KBL's

 

I have been blogging about Kubernetes as the possible primordial soup that can house KBL's - Kubernetes based life-forms.  KBLs are formed using kubernetes constructs and are designed to function in larger ecosystems; via a pattern of self-similarity, a system built with KBLs is designed to evolve.  

Now, of course, these are not actual life-forms, but the name is important as it puts you in the right frame of mind.  The KBL is an attempt to create a system that is designed to evolve.  It can replicate and adjust it's implementation.  

The KBL is specialized pattern in Kubernetes with a focus on data locality.  This focus on data localization forces the design to have to take into account physical node locations. This however is not strictly part of the Kubernetes domain.  Kubernetes, by default, is designed to abstract away the location of any particular resource.  We however, for performance reasons, are compelled to find a way to add data locality back into the equation.  

This is where AWS comes into the picture.  Note this also might have easily been GCP, but for now, we will focus on AWS.   AWS is the underlying technology that provisions the nodes; being able to control AWS node creation programmatically is needed to enable the system to grow or shrink itself.  This is not horizontal scaling, this is altering the shape of graph in response to feedback obtained from the environment, perhaps a collection of KBLs.

In the past few weeks, I have been diving into using Boto3, CloudFormation, and now the Amazon Cloud Development Kit (CDK) to control resource creation.  I am starting to see where bootstrap information and feedback from the KBLs could be used to create and destroy new cloud resources.  It appears there should be a way to integrate with AWS to allow our system of cooperating KBLs to grow and shrink as needed.   

In a previous post I had suggested an imaginary construct, the "windowed mandelbrot", and proposed that in order for a design based on self similarity to be cost effective only parts of the graph would be active at any given point.  AWS provides the API to make this happen.  

One interesting way of passing deployment instructions into CDK is using a Queue.  Importing boto3  classes into your application code running inside Kubernetes would allow sending messages to an AWS based queue and scheduled lambda processes could monitor that queue and run deployment operations based on data passed into the Queue.  .

The CDK exposes an addNodeGroup method.

As a result of this call, the nodeGroup would be provisioned. In previous posts, I had suggested a raid like distribution of data among nodes in a group to prevent data loss in the event of node failure.  This may be leading to each KBL being in it's own NodeGroup?

The blue/green parts below are in AWS and the yellow parts are in Kubernetes.

https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/using-sqs-with-go-sdk.html

https://github.com/aws-samples/aws-cdk-examples/tree/master/python/api-sqs-lambda

https://youtu.be/PdR3SlkwjLM