Solving Problems in ETL using SSIS Allan Mitchell SQL Server MVP .
Solving Problems in ETL using SSIS
description
Transcript of Solving Problems in ETL using SSIS
![Page 1: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/1.jpg)
Solving Problems in ETL using SSIS
Allan MitchellSQL Server MVPwww.Konesans.com
![Page 2: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/2.jpg)
Why am I here
• Worked with DTS since the beginning• Worked onsite with SSIS before RTM• Co-Author on Wrox Professional SSIS Book• Run
−www.SQLDTS.com and −www.SQLIS.com
• I am not Microsoft.
![Page 3: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/3.jpg)
Agenda/Content Slide
• What is ETL and why would I use it?• Microsoft ETL through the Years
−Data Transformation Services−SQL Server Integration Services−What makes Microsoft ETL a good choice
• Demos
![Page 4: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/4.jpg)
What is ETL?
• Extraction, Transformation and Load• Moving data from A B• Transform, conform data
• Most often used in Data Warehouse loads• Also used for
−Data Synchronisation−Data Replication
![Page 5: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/5.jpg)
Traditional ETL Architecture
In this traditional scenario, the integration process simply conforms data and loads the database serverThe database performs aggregations, sorting and other operations……but has to contend with competing demands for resources from user queriesThis solution does not scale to very large volumes of data and multiple, complex aggregations
Traditional warehouse loading
ELT
![Page 6: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/6.jpg)
Rethinking ETL Architecture
Leveraging SQL Server Integration Services
![Page 7: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/7.jpg)
Is it important?
• Getting ETL right is very Important• 70% of a BI project is about the ETL• Why?
−No matter how good the cube guy, the reporter. −Bad data and they are dead in the water
![Page 8: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/8.jpg)
Microsoft ETL through the years
• SQL Server 7– Introduced Data Transformation Services– Drag and Drop paradigm– Easy to use– Freely redistributable– Quick– Forgiving
![Page 9: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/9.jpg)
Microsoft ETL through the years
• SQL Server 2000– Enhancements to DTS– Mainly engine based changes
– Some things however were painful.
![Page 10: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/10.jpg)
Microsoft ETL through the years
• SQL Server 2005– Microsoft listened to years of input– Complete rewrite– Separates control flow and data flow logic– Containers
• Looping• Transactions boundaries• Restart Points
– Easy to make environment agnostic
![Page 11: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/11.jpg)
Microsoft ETL through the years
• SQL Server 2008– Enhances SSIS from SQL Server 2005– Engine multi threading– Persisted lookups– Better feedback from the engine
![Page 12: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/12.jpg)
Demos
![Page 13: Solving Problems in ETL using SSIS](https://reader035.fdocuments.in/reader035/viewer/2022062315/56814d9b550346895dbaf468/html5/thumbnails/13.jpg)
Contact• eMail– [email protected]
• Live Meetings– www.SQLServerFAQ.com
• Websites– www.SQLIS.com– www.SQLDTS.com