HPC in Kubernetes - getting closer to implementation spec?

The design of the proposed data DSL that has been discussed in previous posts can probably be improved by looking at how HPC is being done in Kubernetes right now.  The Lustre file system is being used to power low-latency stateless services.  I think this can be incorporated into the design of the Data DSL.   The one part that the data dsl did define, the usage of ephemeral storage, could still stay.  It could be that Lustre is used as an intermediate source for data - the final destination of the data being truly local and next to the compute (on the same machine).

Below I have included some research on using Lustre.  This would not be the final destination as there is still latency in network hop using Lustre...but Lustre could be used as a "to the curb" data location where we get close but not all the way to the destination.  Lustre could simplify this level.  

Databases can be installed in the Lustre file system.  These databases could serve as a staging area for the data that is pushed directly to the instance.

Operators are the enabling technology for this application of HPC in kubernetes,

 This video has some insights on creating operators to assist with HPC.

Nice integration with the cluster autoscaler and creating new aws instances.

https://youtu.be/LAz_TG2Qtik

The article also mentions using Multus to increase network performance.

https://github.com/k8snetworkplumbingwg/multus-cni

It seems maybe the Data DSL has another option.  

https://aws.amazon.com/fsx/lustre/

It appears you can an RDBMS on FSx 

https://aws.amazon.com/blogs/storage/simplify-your-microsoft-sql-server-high-availability-deployments-using-amazon-fsx-for-windows-file-server/

https://www.eksworkshop.com/beginner/190_fsx_lustre/

It may be possible to further optimize data locality by installing RDBMS on ephemeral storage?  This would take out the network hop.  

In a system that cycles compute from instance to instance that we have described earlier in this blog, each member of a compute cycle would be on a separate machine.  In the event that a machine failed it would only mean the frequency of the data push out of the compute cycle would be reduced by 1 for the time that was needed to restore the instance.   Even using ephemeral data store, the system would still have fault tolerance built in.

https://youtu.be/ttCAqwnN_Qo

Lustre is also available on GCP

https://codelabs.developers.google.com/codelabs/lustre-on-gcp#0