Reveloo is configured by a set of SQL and YAML files in a Github repository. Each model is a unique directory; within the directories, SQL files define the data to be exposed, and YAML files annotate the data, describing what data means and how models connect to each other.
Models are materialized in the
rloo_dev. Redshift schemas.
There are three types of models:
idthat defines an external key, and properties that are either unchanging or change by-day.
If you make a mistake in your YAML, the build process will raise an error with feedback.
The code in your Github repo defines the models that are built. Code on the
master branch builds the main model in the
rloo. schema, and code on the
dev branch builds the dev model in the
rloo_dev. schema, viewable in the web application when “Dev Mode” is toggled. You can rebuild each model in the web app by selecting the menu option to “Build” and then checking logs to ensure success. If a model build fails, the service’s data is not updated. Models should be tested in the dev environment before pushing to master.
Admins can trigger manual or scheduled builds by visiting the build page.
Models can reference each other by joining data in the
rloo. schema. If a
rloo. join is detected, an implicit dependency is created whereby the dependent model will be built after the model on which it depends.
Models are defined with strict SQL, which means data teams can easily copy/paste and test queries an subqueries. However, teams can use a special syntax to enable dynamic injection of dates, so models can be generated for a specific date range. Any code between
rloo:end tags will be dynamically replaced with dates specified by the build command; e.g.,
/*rloo:start*/CURRENT_DATE/*rloo:start*/ may become
/*rloo:start*/'2019-01-01 00:00:00'/*rloo:start*/. This enables query caching and dynamic period generation, and allows rapid testing of models by using short default date ranges.
Building a model incrementally performs a delete-and-replace of daily and/or event tables for the date range supplied. Base dimension tables are always fully replaced, regardless of whether the build is incremental.
Non-incremental builds completely replace all tables for the date range supplied. Incremental builds are recommended for automated builds, but schema changes to daily or event models require a full rebuild.