Cubes – pluggable model explained
-
Upload
stefan-urbanek -
Category
Technology
-
view
3.593 -
download
1
description
Transcript of Cubes – pluggable model explained
![Page 1: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/1.jpg)
Pluggable ModelCubes Analytical Workspace Redesign
data brewery
February 2014Stefan Urbanek – @Stiivi
![Page 2: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/2.jpg)
Original CubesCubes before 1.0
![Page 3: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/3.jpg)
Model
■ single JSON or a model bundle
■ contains all model objects
■ full description required
![Page 4: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/4.jpg)
backends
model browser✂
server
http
workspace
formatters
modules
one file or one directory bundle
one per serving:[workspace] backend=sql url=postgresql://localhost/database
![Page 5: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/5.jpg)
Browser
SQL Snowflake Browser
Aggregation Browser
SQL Denormalized Browser MongoDB Browser Some HTTP Data
Service Browser
?
multiple backends available
![Page 6: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/6.jpg)
Backend
■ implemented as python module with an entry point create_workspace()
■ provides Workspace and Browser workspace represents data storage
■ only one Workspace per serving only one kind of storage per serving
![Page 7: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/7.jpg)
Requirements
![Page 8: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/8.jpg)
Model
■ composed of multiple parts
■ external model definition provided from external source, such as analytical service
■ shared dimension descriptions only one dimension description is necessary per composed model
![Page 9: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/9.jpg)
Backend
■ heterogenous storage multiple data stores, different types of data stores
■ different schemas in same store
■ multiple environments dev, test, production, ...
![Page 10: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/10.jpg)
Redesign
![Page 11: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/11.jpg)
Backend
■ “backend” are multiple objects:
!
■ better plug-in system instead of Python module
■ more flexible composition
|Browser
"Store
#Provider
![Page 12: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/12.jpg)
Backend Objects
■ Browser – performs aggregated browsing
■ Store – maintains database connection
■ Model Provider – provides model
Note: not every kind has to be implemented
![Page 13: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/13.jpg)
Logical Physical
physical data store(database or API)
|Browser
"Store
#Provider
∑aggregate
connectcreate model
model
cubes
dimensions
model
backend objects
![Page 14: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/14.jpg)
Browser
![Page 15: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/15.jpg)
Browser
■ depends on the logical model
■ implements aggregation aggregate(), values(), …
■ gets data from associated store
![Page 16: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/16.jpg)
Logical Physical
physical data store(database or API)
|Browser
"Store
∑aggregate
model
browser
![Page 17: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/17.jpg)
Browser Methods
■ features()
■ aggregate()
■ members()
■ facts()
■ fact()
![Page 18: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/18.jpg)
Store
![Page 19: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/19.jpg)
Store
■ provides database or API connection
■ might provide a model
■ slicer tool actions physical mapping validation, model from schema generation, schema from model generation, schema conversions and optimization, ...
*former backend’s “Workspace” object
*
![Page 20: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/20.jpg)
Logical Physical
physical data store(database or API)
|Browser
"Store
connect
store
![Page 21: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/21.jpg)
Store Methods
■ validate(cube) – does logical map to physical?
■ create(object) – create physical structure
Store is not required to implement any methods at this time. Future:
![Page 22: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/22.jpg)
Model Provider
![Page 23: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/23.jpg)
Model Provider
■ creates model from external source
■ might suggest store to be used
![Page 24: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/24.jpg)
Logical
!Provider
create model
model
cubes
dimensions
model
model provider
![Page 25: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/25.jpg)
Provider Methods
■ dimension_metadata(name,temps,locale)
■ cube_metadata(name,locale)
or
■ dimension(name,temps,locale)
■ cube(name,locale)
![Page 26: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/26.jpg)
example backends
SQL Backend Mongo Backend Google Analytics Backend
|Snowflake Browser
"SQL Store
|Mongo Browser
"Mongo Store
|GA Browser
"GA Store
#GA Model Provider
![Page 27: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/27.jpg)
from cubes import | AggregateBrowser, " Store !class " SQLStore(" Store): | default_browser_name = “sql_snowflake” ! def __init__(self, # **options): # initialize the store here ! def validate_cube(self, cube): return True # if valid !!class | SQLSnowflakeBrowser(| Browser): def __init__(self, model, locale): # initialize the browser ! def features(self): # return list of browser features def aggregate(self, cell, ...): # return aggregation of the cell
from slicer.ini
![Page 28: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/28.jpg)
New Workspace
■ global object at library level
■ provides appropriate browser
■ contains run-time configuration
■ might have state persistence
*former backend Workspace is now Store
*
![Page 29: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/29.jpg)
Future Workspace
■ caching
■ cube composition
■ … ?
![Page 30: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/30.jpg)
Workspace Example
![Page 31: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/31.jpg)
heterogenous environment
Workspace
Cubes
Model Providers
Stores
sales churn eventsactivations
Static Model Provider
API Model Provider
BI Data(Postgres)
BI Data 2(Mongo)
Events(API)
![Page 32: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/32.jpg)
Workspace
Cubes
Model Providers
Stores
sales churn eventsactivations
Static Model Provider
BI Data(Postgres)
BI Data 2(Mongo)
crm sales events
[workspace] models_path: /var/lib/cubes/models ![models] crm: crm.cubesmodel sales: sales.cubesmodel events: events.cubesmodel ![datastore_bidata] type: sql url: postgresql://localhost/crm ![datastore_bidata2] type: mongo host: localhost collection: events
![Page 33: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/33.jpg)
Conclusion
![Page 34: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/34.jpg)
Conclusion
■ heterogenous pluggable environment
■ externally provided models
■ easier backend implementation
![Page 35: Cubes – pluggable model explained](https://reader033.fdocuments.in/reader033/viewer/2022042606/54963f26b47959604d8b4f57/html5/thumbnails/35.jpg)
Cubes Home
cubes.databrewery.org
github
github.com/Stiivi/cubes
Development Documentation
cubes.databrewery.org/dev/doc/for github master HEAD