Post on 08-Apr-2018
8/7/2019 ADO dot Net Advanced PAtterns
1/26
8/7/2019 ADO dot Net Advanced PAtterns
2/26
About Me
Independent Software Developer
20 years developing
Microsoft .NET MVP
ASPInsider
INETA Speaker
Various publications & conferences
Blogs: thedatafarm.com/blog, blog.ziffdavis.com/devlife
Founder and leader ofVermont .NET
INETAAdvisory Board Member
8/7/2019 ADO dot Net Advanced PAtterns
3/26
Agenda
ADO.NET 2.0 DataSet performance
Pushing huge data back to SQL Server
Caching huge amounts of data Querying huge amounts of data in memory
8/7/2019 ADO dot Net Advanced PAtterns
4/26
ADO.NET 2.0 Features for this talk
Batch Updating with DataAdapter
SqlBulkCopy Class
SqlDependency Class
8/7/2019 ADO dot Net Advanced PAtterns
5/26
ADO.NET 2.0 Engine Improvements
DataSet Scalability across the board
Can handle lots of rows
Indexing Engine completely re-written Incremental indexing updates
much more
Low overhead for using Data
Views
Serialization: Faster, more compact
Supports Binary & Schema-less serialization
8/7/2019 ADO dot Net Advanced PAtterns
6/26
DEMO
Indexing Engine Performance
8/7/2019 ADO dot Net Advanced PAtterns
7/26
Moving Huge Amounts of Data
Getting data is easy and fast!
Uploading is a different story
1.1 Update Options DataAdapter.Update one row at a time
Controlled stored procedure update one row at a
time
8/7/2019 ADO dot Net Advanced PAtterns
8/26
Updating in 2.0
DataAdapterBatch Update
Pushes groups of rows up to SQL Server
SQL Server still updates one at a time Find sweet spot for UpdateBatchSize value
Bulk Copy
Streamed Insert using SQL ServerBulk Copy
*slide slightly modified from original printed version
8/7/2019 ADO dot Net Advanced PAtterns
9/26
DEMO
Batch Update & Bulk Copy Class
8/7/2019 ADO dot Net Advanced PAtterns
10/26
8/7/2019 ADO dot Net Advanced PAtterns
11/26
DEMO
Using BCL & DML to process huge
amounts of data
8/7/2019 ADO dot Net Advanced PAtterns
12/26
DataSet Caching
DataSet improvements enable storing lots
of data into cache
SqlDependency enables automatednotification of changes to .NET
Combine SqlDependency and Caching for
serious resource reduction
8/7/2019 ADO dot Net Advanced PAtterns
13/26
DEMO
SqlDependency & DataSet Cache
Part 1
8/7/2019 ADO dot Net Advanced PAtterns
14/26
Big DataSet cache begs granularity
SqlDependency is all or nothing
Create separate dependencies for logical
sections of your cache
8/7/2019 ADO dot Net Advanced PAtterns
15/26
DEMO
SqlDependency & DataSet Cache
Part 2: Achieve granularity
8/7/2019 ADO dot Net Advanced PAtterns
16/26
Divide & Conquer your huge dataset
In-Memory Querying is coming!
LiNQ
VB9
/C#3
What about today?
ADO.NET 2.0 has everything you need to
build a simple query processor
Great example of QP from ADO.NET Team
Technical Lead Pablo Castro (for demo only)
8/7/2019 ADO dot Net Advanced PAtterns
17/26
High Level view of Query Processor
Familiar Query Syntax eg SELECT, WHERE, JOIN
Query is parsed to Algebra Tree Some of the Algebraic Operators
Scan: enumeration over the logical tree Projection: identifying what columns to return
Filter, Join, Sort
Just a data structure
Iterator is created for each operator Does the actual work of processing the data and extracting what
is defined by operator Projection Iterator returns all the rows but only containing the
desired fields
Filter iterator processes all the rows and returns only rows matchingfilter rules
8/7/2019 ADO dot Net Advanced PAtterns
18/26
Roll your own Query Processor
cached
data
ProjectIterator
cached
data
Join
iterator
Narrowed
down to
specific
columns
Flattened tomatch criteria
Enhanced
with related
data
8/7/2019 ADO dot Net Advanced PAtterns
19/26
Our Custom QP Iterators
Scan Enumerates over dataset rows (eg leaf-nodes of tree)
Select (Project)
DataSet Expressions can pull out columns Filter
DataTable.Select for on-the-fly indexing
Consider DataView for a permanent index and its raw.NET 2.0 power
DataTable.Select can leverage this index Sort
Bundle this with the filter leverage DataTable.Select or DataView and indexing
8/7/2019 ADO dot Net Advanced PAtterns
20/26
Use DataViews new power for JOIN
JOIN will be your weakest link in a query
operation
Iterates through joined table for every row in
primary table
Try a DataView instead
DataViews indexing will speed this up
8/7/2019 ADO dot Net Advanced PAtterns
21/26
Beware Operator Constraints
DataView/DT.Select uses == and !=
need to be creative for implementing things
like >, IN orBETWEEN
Cant leverage indexing
Roll your own?
8/7/2019 ADO dot Net Advanced PAtterns
22/26
DEMO
A look at a sample Query
Processor written with
ADO.NET 2.0
8/7/2019 ADO dot Net Advanced PAtterns
23/26
Summary
ADO.NET 2.0s performance andfunctional improvements put new power inour hands
You can work with huge amounts of dataefficiently
Functionality like BatchUpdate gives you
easy access to some of this power With a little more code, you can do
wonders!
8/7/2019 ADO dot Net Advanced PAtterns
24/26
Resources
PDC05 Session DAT408, Pablo Castro,
ADO.NET Team Technical Lead
ADO.NET Team Blog:blogs.msdn.com/dataaccess
MSDN Data Access Dev Center
msdn.microsoft.com/data
Whats New in ADO.NET 2.0, Julia Lerman,
MSDNMagazine April 2005
8/7/2019 ADO dot Net Advanced PAtterns
25/26
8/7/2019 ADO dot Net Advanced PAtterns
26/26
Contact Info
Julie Lerman
jlerman@thedatafarm.comwww.thedatafarm.com
Blogs
www.thedatafarm.com/blogblogs.ziffdavis.com/devlife