After a few weekends with Dart/Flutter, I spent the majority of this weekend looking a Julia. After noticing that Julia was advertised to be very fast and designed from the group up for data science, I finally decided to jump in.
After a bit of research, it is starting to look like the features of Julia are uniquely positioned to fit in well as the implementation language in the Execution DSL.
Two language features in particular make it well suited for the Execution DSL
Julia support distributed computing with built in language primitives.
Julia supports meta programming
See:
https://docs.julialang.org/en/latest/stdlib/Distributed/#Cluster-Manager-Interface-1
It seems that the Resources DSL could provide the specification that determines what the graph of compute resources looks like (the shape of the compute solution)
Using the information in the Resources DSL, it would seem that properly constructed Julia code could use meta programming to adapt to this structure and create the corresponding execution resources inside this graph.
Update 3/23:
Flow based programming could connect pieces of the puzzle together. Just for the internals of solving the actual problems within a node of a flow based system, you must be able to allocate resources and then have your compute solution take advantage of the allocated resources. This is where Julia could come into play. Instead of trying to tack orchestration between nodes on top of FBP, keep FBP simple, input and output.
Within a box in a series of connected boxes, a finer level orchestration is taking place. It seems Julia "machines" mapping to containers deployed in Kubernetes could be one implementation of an orchestration mechanism "inside the box"
See: https://docs.julialang.org/en/v1/stdlib/Distributed/index.html
So instead of each container being a box in the FBP world and having to orchestrate communication via FBP mechanisms, each box in FBP would correspond to a "stage" in processing flow. (Potentially many containers) And in each stage orchestration between containers happens via Julia distributed programming techniques.
So you have a phase that needs certain resources, this could possibly be provisioned by Volcano. Inside a phase communication is using Julia primitives. Zooming out one layer it looks like a FBP problem flowing from stage to stage. Inside each stage you have an orchestration langues similar to Go with Channels etc but geared more towards scientific calculation workloads.
It may be the fact that Go and Channels are what orchestrates the "macro" level. Orchestrating parallel and sequential phases, but at the micro level you have a language that offers both the capability for scientific workloads and distributed computing via language primitives.
Update 4/5/2020
Over the week I have been continuing to research Julia and have found that the GRPC support is not quite there yet. This is not optimal, but in continuing to look around, I found two frameworks that could help to expose Julia Services
First - there is Diana for for creating a GraphQL API:
https://github.com/codeneomatrix/Diana.jl
https://codeneomatrix.github.io/Diana.jl/v0.2/
and second, there is Genie for REST and Websockets
https://github.com/GenieFramework/Genie.jl
https://genieframework.com/
Support for GRPC seems to still be in development:
https://github.com/JuliaIO/ProtoBuf.jl
No comments:
Post a Comment