Biml for Beginners: Speed up your SSIS development (SQLSaturday Iceland)
Biml Academy 2 - Lesson 5: Importing source metadata into Biml
-
Upload
cathrine-wilhelmsen -
Category
Data & Analytics
-
view
1.824 -
download
1
Transcript of Biml Academy 2 - Lesson 5: Importing source metadata into Biml
![Page 1: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/1.jpg)
Biml Academy 2
Importing, Staging, and Loading Data with Biml
Andy Leonard · Scott Currie · Cathrine Wilhelmsen
![Page 2: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/2.jpg)
Biml Academy 2
Lesson 5: Importing source metadata into Biml (Cathrine)
Lesson 6: Staging data with Biml (Andy)
Lesson 7: Metadata-driven staging pattern with Biml (Andy)
Lesson 8: BimlFlex for Data Vault data warehouses (Scott)
Lesson 9: Biml Q&A with several BimlHeroes!
![Page 3: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/3.jpg)
Cathrine Wilhelmsen
@cathrinew
cathrinew.net
Data Warehouse ArchitectBusiness Intelligence Developer
![Page 4: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/4.jpg)
Lesson 5:
Importing source metadata into Biml
![Page 5: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/5.jpg)
Methods for importing metadata
ImportTableNodes
ImportDB
GetDatabaseSchema
![Page 6: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/6.jpg)
Methods for importing metadata
ImportTableNodes
Import tables and views in one schema
Filter tables using LIKE % wildcard syntax
![Page 7: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/7.jpg)
Methods for importing metadata
ImportDB
Import all schemas, tables and views
Filter using LIKE % wildcard syntax
![Page 8: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/8.jpg)
Methods for importing metadata
GetDatabaseSchema
Import all schemas, tables and views
Filter using collections
![Page 9: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/9.jpg)
Why ImportDB and GetDatabaseSchema?
ImportDB / ImportTableNodes
• SELECT * on each imported table
• SELECT to get defaults, indexes etc.
• Using SchemaOnly mode
• Doesn't actually run the query, but returns schema for the result set
GetDatabaseSchema
• Figures out target database type (SQL Server, Oracle, Teradata, DB2, PostgreSQL, Ingres, OpenEdge etc.)
• Queries the relevant information schema for that database type
A few queries for the entire importMultiple queries per table
![Page 10: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/10.jpg)
But why keep both if one is faster?
Support for legacy scripts :)
They can return different results – where both are valid/correct
Different data type mapping defaults in different systems(Bulk Copy, SSIS and Excel are all different)
Biml supports the two most commonly used approaches
![Page 11: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/11.jpg)
Methods and parameters
ImportTableNodes(schema, tableFilter)
ImportTableNodes(schema, tableFilter, importOptions)
ImportDB()
ImportDB(schemaFilter, tableFilter)
ImportDB(schemaFilter, tableFilter, importOptions)
GetDatabaseSchema()
GetDatabaseSchema(importOptions)
GetDatabaseSchema(includedSchemas, includedTables, importOptions)
![Page 12: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/12.jpg)
Methods and parameters: string value
ImportTableNodes(schema, tableFilter)
ImportTableNodes(schema, tableFilter, importOptions)
ImportDB()
ImportDB(schemaFilter, tableFilter)
ImportDB(schemaFilter, tableFilter, importOptions)
GetDatabaseSchema()
GetDatabaseSchema(importOptions)
GetDatabaseSchema(includedSchemas, includedTables, importOptions)
![Page 13: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/13.jpg)
Methods and parameters: string filter
ImportTableNodes(schema, tableFilter)
ImportTableNodes(schema, tableFilter, importOptions)
ImportDB()
ImportDB(schemaFilter, tableFilter)
ImportDB(schemaFilter, tableFilter, importOptions)
GetDatabaseSchema()
GetDatabaseSchema(importOptions)
GetDatabaseSchema(includedSchemas, includedTables, importOptions)
![Page 14: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/14.jpg)
Methods and parameters: collections
ImportTableNodes(schema, tableFilter)
ImportTableNodes(schema, tableFilter, importOptions)
ImportDB()
ImportDB(schemaFilter, tableFilter)
ImportDB(schemaFilter, tableFilter, importOptions)
GetDatabaseSchema()
GetDatabaseSchema(importOptions)
GetDatabaseSchema(includedSchemas, includedTables, importOptions)
![Page 15: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/15.jpg)
Methods and parameters: ImportOptions
ImportTableNodes(schema, tableFilter)
ImportTableNodes(schema, tableFilter, importOptions)
ImportDB()
ImportDB(schemaFilter, tableFilter)
ImportDB(schemaFilter, tableFilter, importOptions)
GetDatabaseSchema()
GetDatabaseSchema(importOptions)
GetDatabaseSchema(includedSchemas, includedTables, importOptions)
![Page 16: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/16.jpg)
ImportOptions specify objects to exclude
Don't exclude anything
Views
Identity Specifications
Primary Keys
Foreign Keys
Unique Keys
Indexes
Column Defaults
Check Constraints
ImportOptions.None
ImportOptions.ExcludeViews
ImportOptions.ExcludeIdentity
ImportOptions.ExcludePrimaryKey
ImportOptions.ExcludeForeignKey
ImportOptions.ExcludeUniqueKey
ImportOptions.ExcludeIndex
ImportOptions.ExcludeColumnDefault
ImportOptions.ExcludeCheckConstraint
![Page 17: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/17.jpg)
Demo time!
![Page 18: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/18.jpg)
Additional methods for importing metadata
GenerateSchemaNode
Returns AstSchemaNode
GenerateSchemaNodes
Returns IEnumerable<AstSchemaNode>
![Page 19: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/19.jpg)
Additional methods for importing metadata
GenerateTableNode
Returns AstTableNode
GenerateTableNodes
Returns IEnumerable<AstTableNode>
![Page 20: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/20.jpg)
Demo time!
![Page 21: Biml Academy 2 - Lesson 5: Importing source metadata into Biml](https://reader031.fdocuments.in/reader031/viewer/2022021813/587542811a28abb8208b5491/html5/thumbnails/21.jpg)
Biml Academy 2
Lesson 5: Importing source metadata into Biml (Cathrine)
Lesson 6: Staging data with Biml (Andy)
Lesson 7: Metadata-driven staging pattern with Biml (Andy)
Lesson 8: BimlFlex for delivering Data Vault data warehouses (Scott)
Lesson 9: Biml Q&A with several BimlHeroes!