SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero...
-
Upload
truonghanh -
Category
Documents
-
view
223 -
download
1
Transcript of SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero...
![Page 1: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/1.jpg)
SQL Azure and SqlBulkCopy
Presented by Herve Roggero
Managing Partner, Blue Syntax Consulting
SQL Azure MVP
Email: [email protected]
Twitter: @hroggero
LinkedIn: SQL Azure and SQL Server Security
LinkedIn: Azure Florida Association
www.bluesyntax.net
![Page 2: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/2.jpg)
Agenda Available tools to load data
INSERT commands
SqlDataAdapter
Analyzing BCP
Analyzing SqlBulkCopy
Performance test
![Page 3: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/3.jpg)
Available tools/techniques INSERT
INSERT (2)
INSERT BULK
SqlDataAdapter
Migration Wizard
SSIS
BCP / INSERT BULK
SqlBulkCopy
![Page 4: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/4.jpg)
What makes up load time? Connection Time
Send Query
Compile Query
Execute Query
Send Data
![Page 5: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/5.jpg)
INSERT Classic way to insert records
Logged operation
Usually single record at a time
All packets are encrypted => too verbose
VERY SLOW...
INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES ('Helmet', 25.50)
INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES ('Wheel', 30.00);
![Page 6: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/6.jpg)
INSERT (2) New INSERT command in SQL Server 2008
Table Value Constructor
Not as verbose
Operation is still logged
Faster than regular INSERT, but still SLOW…
INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES
('Helmet', 25.50),
('Wheel', 30.00);
![Page 7: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/7.jpg)
INSERT (summary)
INSERT … INTO…
INSERT … INTO…
INSERT … INTO… SQL Azure
SQL Azure
SQL Azure
![Page 8: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/8.jpg)
BULK INSERT NOT SUPPORTED in SQL AZURE…
INSERT BULK … INTO…
SQL Server
Raw packets
Raw packets
Raw packets
TERMINATION
File
![Page 9: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/9.jpg)
INSERT BULK Supported… but requires feeding “raw” TDS packets
=> Must be used by a Tool
Very efficient and lean
Sends 1 INSERT statement
Everything that follows are raw data packets
NO FURTHER COMPILATION/QUERY EXECUTION
SQL Azure waits until Session sends “TERMINATION”
Operation not logged
FAST…
![Page 10: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/10.jpg)
INSERT BULK (cont.)
INSERT BULK … INTO…
SQL Azure
Raw packets
Raw packets
Raw packets
TERMINATION
![Page 11: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/11.jpg)
BCP Uses INSERT BULK
Performance of BCP depends on INSERT BULK
However…
BCP can also extract data
FAST
![Page 12: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/12.jpg)
Tools Tools that use the BCP program:
SSIS: Bulk Insert Task
SQL Azure Migration Wizard
http://sqlazuremw.codeplex.com/
Tools that use the SQL Bulk Copy API
Enzo Data Copy
![Page 13: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/13.jpg)
SqlDataAdapter Based on XML Datagrams (differences)
Very memory intensive
Can be very slow depending on approach
Provides a way to work in Batches
Sends INSERT operations
SLOW…
![Page 14: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/14.jpg)
SqlBulkCopy Uses INSERT BULK operation
Programmatic access, options similar to BCP
Can send data in Batch
FAST
![Page 15: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/15.jpg)
Techniques Summary Tool/Technique Operation Logged Speed
INSERT INSERT Yes SLOW
BULK INSERT BULK INSERT No n/a
INSERT BULK INSERT BULK No FAST
BCP INSERT BULK No FAST
SSIS/ Migration Wizard
INSERT BULK No FAST
Data Adapter INSERT Yes SLOW
SqlBulkCopy INSERT BULK No FAST
Enzo Data Copy INSERT BULK No FAST
![Page 16: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/16.jpg)
About the performance test Load 40,000 records (on-premise -> SQL Azure)
Uses multithreading (TPL) – 8 threads
Comparing
Data Adapter
Stored Procedure
SqlBulkCopy API
INSERTs
![Page 17: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/17.jpg)
Performance Chart X: Batch Size; Y: Load Time {40K records; DOP = 8}
Batch Size
Lo
ad
Tim
e (
sec)
2000 3000 4000 1000 500 100
10
20
30
Data Adapter
Stored Procedure
SqlBulkCopy
INSERT 2 [limited to batch of 1000]
![Page 18: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/18.jpg)
Comparing Tools
40,000 Records (insecs)
2,000,000 Records(in mins)
SSIS 4.2 2.17
SQL Bulk Copy 2.18 4.3
Enzo Data Copy 25 1.41
0
5
10
15
20
25
30
SSIS
SQL Bulk Copy
Enzo Data Copy
![Page 19: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/19.jpg)
Demos Sample Code
Uses parallel processing on all methods
Compares:
Data Adapter
Inline INSERTs
Stored Procedure INSERTs
SqlBulkCopy
SSIS
Enzo Data Copy (www.bluesyntax.net)
![Page 20: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/20.jpg)
Considerations Resilience to Throttling
Adapting to throttling conditions
Changing workload depending on errors
Connection Retries
Retrying for connection loss
Causes include throttling, load balancing, network…
http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-connection-management.aspx
![Page 21: SQL Azure and SqlBulkCopy - Blue Syntax · SQL Azure and SqlBulkCopy Presented by Herve Roggero Managing Partner, Blue Syntax Consulting SQL Azure MVP Email: hroggero@bluesyntax.net](https://reader031.fdocuments.in/reader031/viewer/2022022507/5acbcff07f8b9a6a678b9672/html5/thumbnails/21.jpg)
Thank you
Presented by Herve Roggero
Managing Partner, Blue Syntax Consulting
SQL Azure MVP
Email: [email protected]
Twitter: @hroggero
LinkedIn: SQL Azure and SQL Server Security
LinkedIn: Azure Florida Association
Blog: www.geekswithblogs.net/hroggero
www.bluesyntax.net