I have been exploring Kubernetes Native solutions for processing streaming data. The idea has been that custom CRD's (DSLs) could be used to simplify application logic. It seems to me that the being able create code that sits at the same level as the core kubernetes objects is a powerful concept. It opens up the door to providing cross cutting concerns to a suite of applications.
It is important to note that the CRDs act to delegate to implementations. They form the blueprint for the solution but are not themselves necessarily the implementation code. For instance, it was envisioned that a portion of the Data DSL would delegate to other operators that are designed to deploy databases.
I think there could be advantages of a K8's native streaming solution that doesn't force an application to conform to specific APIs, but there is wealth of knowledge out there that is contained in the existing streaming frameworks.
One of the concepts of first principles thinking as I understand it is to let yourself rethink existing processes. I have a feeling that there may be a solution that doesn't force specific APIs but without a deeper dive, I cannot be sure. It seems the existing solutions integrate with K8's as many others apps do - as an afterthought or a deployment target. What I want to do is look at it from a Kubernetes first perspective and see what materializes. I am not trying to hit every use case only the use case that ensures correctness and data consistency. Instead of having to conform to a framework API maybe its possible to let Kubernetes handle the data consistency?
https://www.oreilly.com/radar/the-world-beyond-batch-streaming-102/
https://learning.oreilly.com/library/view/stream-processing-with/9781491974285/
No comments:
Post a Comment