Sunday, January 12, 2020

Optimizations in Time based Windowed APIs

Being able to share intermediate results across a series of calculations has the potential to optimize the process.  Having isolated environments is an advantage when it comes to synchronization, but if the answer to parts of the equation are already known, it makes sense to reuse them.  The question becomes how to determine what data is re-usable? (maybe a tree of bloom filters?)   In functional programming there is the idea of referential transparency - a function can be replaced by a value without altering the result of the program.  If the assembly of the answer was directed by a structure that either made function calls or used cached values - then all the calculations would not need to be run every time.

In essence, this would be the transfer of data from between instances of a Pluggable Universes.  Where in the K8's mesh of objects we have created could this occur?

The system that has been described in earlier posts has emphasized isolation and using immutable data as a way to keep consistency across a set of distributed processes.  This does not mean we can't reuse partial results that have base data that has not changed from snapshot to snapshot.  

I think the key to finding the answer to this optimization would be to follow the principles of referential transparency in the core framework code that invokes each partial calculation.  There would need to be a mechanism that was able to decide whether or not to perform a calculation or sub in a cached result.

Where would this data be kept?  In the proposed system, it could be kept at the node level in MemSQL  Remember each set of calculations was fed by a local MemSQL database located on the same node.  So this same database could act to serve both live data and cached results.




No comments: