Tuesday, November 03, 2020

A view set for each time slice

"View sets" are windows into the node level storage.   When storing a time series in normal database tables you use append only technique that allows you to query for state at a specific instance in time.  This leads to as slightly different query strategy vs a standard relational structure.  In the former version you are querying to find out what the state was at a particular point in time.  You use the same marker in all data tables in the local instance to make it possible to do these queries across many tables.


Essentially what you are doing is creating views into an ever changing data set.  When a valid marker point is hit (there will be rules to determine what is valid marker point) then its possible to update existing views to "roll" to this new time slice.

Say you have 5 time slices.  For each base table in the system you will have 5 views that will show the state of the data at the marker defining the timeslice.  The process of keeping the views updated is by using ALTER VIEW during rollover time and updating the marker information in the query.  The result of this is less complicated SQL for the applications - they can point at a series of views and can assume all the underlying data is correctly sliced.  

We see the same recycling pattern here that we see with resources.   Views are re-used and the number of "view sets" ends up equaling the number of time slices.




MemSQL supports both create view and alter view DDL statements.

https://docs.singlestore.com/v7.1/reference/sql-reference/data-definition-language-ddl/create-view/

https://docs.singlestore.com/v7.1/reference/sql-reference/data-definition-language-ddl/alter-view/

The previously mentioned data DSL and operator will handle the creation of the view sets and handle updating of the views in concert with the Resource and Routing DSLs.  All the DSLs are working to cycle compute nodes in this data ingest and process cycle.

No comments: