A Performance Comparison of Microsoft ® SQL Server ® 2008 Transparent Data Encryption ® and...
-
date post
21-Dec-2015 -
Category
Documents
-
view
220 -
download
0
Transcript of A Performance Comparison of Microsoft ® SQL Server ® 2008 Transparent Data Encryption ® and...
A Performance Comparison of Microsoft® SQL Server®
2008 Transparent Data Encryption® and NetLib®
Encryptionizer®
Phil Colbert
CSCI 693 Fall 2008
Presentation Overview
Introduction Data Set Related Work Experimental Procedures Experimental Results Conclusions Future Work
SQL Server, Microsoft, Transparent Data Encryption, Virtual PC, Transact-SQL, T-SQL, and Windows XP are registered trademarks of the Microsoft Corporation, Inc. NetLib and Encryptionizer are registered trademarks of Communication Horizons LLC. VMware is a registered trademark of VMware, Inc.
Introduction Data and information security is a critical element
of our modern world Relational database management systems
(RDBMS) key component Microsoft SQL Server 2008 Enterprise introduces
Transparent Data Encryption (TDE), or whole database encryption
NetLib Encryptionizer has been available for nearly a decade, and works on all versions of Microsoft SQL Server since version 6
Benchmark performance comparisonSources: The Canadian Press [1], C. Boss [2], D. Migoya [3], S. Hsueh [4], NetLib [5]
Introduction: Acronyms
TDE: Transparent Data Encryption NLE: NetLib Encryptionizer AES: Advanced Encryption Standard DES: Data Encryption Standard TSQL: Microsoft Transact-SQL MSSQL: Microsoft SQL Server 2008
Enterprise BigNW: Expanded Northwind database
Introduction: Encryption
128-bit AES AES replaced DES in 2002 for the United
States government as the cryptographic security standard for all sensitive data
TDE and NLE both support AES
Source: FIPS [6]
Data Set
Microsoft Northwind database Expanded by a factor of 100 by BigNW
TSQL script authored by Scott Mauvais BigNW TSQL modified to work with
MSSQL
Sources: Microsoft [7], S. Mauvais [11]
Related Work
No significant benchmarks comparing these two products
Microsoft estimates 3-5% reduction in overall performance for low processor and low input/output systems using TDE
Microsoft estimates up to 28% reduction in overal performance for high load systems using TDE
Industry standard TPC benchmark cost-prohibitive Virtual environments viable benchmark platform
Sources: S. Hsueh [4], Transaction Processing Performance Council [9], M.L. Catalan et al [8]
Experimental Procedures: Overview
Development and implementation of a proprietary benchmark methodology based on ANSI SQL Standard Scalable and Portable (AS3AP) benchmark guidelines
Batch file, table-based, and TSQL script-based solution Experimental results saved in real-time to test results
database and tables Three MSSQL instances:
– Base: unencrypted baseline
– TDE: TDE encrypted
– NLE: NLE encrypted
Utilized MSSQL cache clearing methods
Source: C. Turbyfill et al [10]
Experimental Procedures: Platform and Software
Intel® Core™2 Duo 2.13 GHz Processor with 3.25 GB RAM running Microsoft Windows XP Professional operating system with Service Pack 3
Microsoft Virtual PC used version 6.0.156.0 with 1024 MB RAM running Microsoft Windows XP Professional operating system with Service Pack 3
VMware Workstation used version 6.5.0 build-118166 with 1024 MB RAM running Microsoft Windows XP Professional operating system with Service Pack 3
Microsoft SQL Server 2008 Enterprise code name "Katmai" (CTP) version 10.0.1075.23
NetLib Encryptionizer version was 2007.101.20.8.3.4a Hardware virtualization enabled All software packages and operating systems were 32-bit
Experimental Procedures: TSQL Benchmark
Fig. 1 Benchmark flow diagram
Experimental Procedures:Benchmarks
Ten benchmarks in five categories:1. Data retrieval, 2. Insertion, 3.Deletion, 4. Update, 5. Backup
Category Benchmark Description
Retrieve Select orders ordered on or after 1/1/1996 using index seek
Retrieve Select orders shipped to Germany using index scan
Retrieve Select all orders joined with customers table
Retrieve Select all invoices from Invoices view
Retrieve Execute stored procedure Employee Sales by Country inserted into temp table
Add Select all orders and insert into temporary table
Add Insert new orders
Update Update orders information
Delete Delete orders
Backup Add a backup device and backup the entire database
Fig. 2 Benchmark descriptions by category
Experimental Procedures: Unbalanced
Benchmark executions with no attempt to account for indeterminate system anomalies
Executions by benchmark by instance Record script execution times using TSQL timing
syntax Performed on both virtual environments, and dual-
processor non-virtual environment Derived baseline number of iterations per script to
execute within a 5-20 second time interval
Experimental Procedures:Balanced
Sequential execution methodology of each script across each instance
A measurable unit is the iterative execution of a single script, on a single database instance
Each measurable execution unit repeated 1000 times for dual processor environment, 100 times for each virtual environment
Reduce data outliers
Benchmark Iterations
1 12
2 19
3 32
4 2
5 7
6 12
7 143
8 10
9 100
10 3
Fig. 3 Derived iterations
Experimental Results: BalancedVirtual PC
Mean Execution Time (ms) ± % At 95%Confidence Level
Benchmark Base TDE NLE Base TDE NLE
1 7963 9524 8523 2.8 1.1 0.9
2 6531 8118 6685 1.6 1.2 1.6
3 5202 6974 6757 0.8 3.1 3.5
4 6838 7881 7679 0.4 1.0 2.7
5 6092 8796 7894 3.4 2.5 1.0
6 6314 7684 7163 1.5 0.5 4.0
7 7414 10151 8741 3.6 5.1 3.4
8 3532 4698 4298 1.0 0.8 2.2
9 8691 11953 11501 8.9 6.4 9.3
10 8135 7968 8624 3.0 2.7 2.9
Fig. 4 Balanced Virtual PC benchmark results, with an average sample mean decrease in execution speed for NLE compared to TDE of 6.9% (± 2.8% at the 95% confidence level).
Experimental Results: BalancedVirtual PC
Fig. 5 Balanced Virtual PC mean execution time (ms) at 95% confidence level
0 2000 4000 6000 8000 10000 12000 14000
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BM 1
BM 2
BM 3
BM 4
BM 5
BM 6
BM 7
BM 8
BM 9
BM 10
Experimental Results: BalancedVMware
Mean Execution Time (ms) ± % At 95%Confidence Level
Benchmark Base TDE NLE Base TDE NLE
1 6301 8719 8138 1.9 1.1 1.3
2 4886 7083 6313 1.0 0.5 0.7
3 5398 6435 6630 1.3 0.9 0.8
4 6112 6600 6882 1.0 1.0 1.1
5 4999 8079 7227 0.6 0.4 0.4
6 4904 6896 6373 1.6 1.3 1.3
7 7820 8039 7002 4.7 5.5 3.1
8 10565 13606 12576 1.0 0.9 0.8
9 16283 15785 15493 0.4 1.5 0.9
10 5751 5694 6246 1.0 0.9 0.9
Fig. 6 Balanced VMware benchmark results, with an average sample mean decrease in execution speed for NLE compared to TDE of 4.1% (± 1.3% at the 95% confidence level).
Experimental Results: BalancedVMware
0 5000 10000 15000 20000
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BaseTDENLE
BM 1
BM 2
BM 3
BM 4
BM 5
BM 6
BM 7
BM 8
BM 9
BM 10
Fig. 7 Balanced VMware mean execution time (ms) at 95% confidence level
Experimental Results: BalancedNon-virtual dual-processor
Mean Execution Time (ms) ± % At 95%Confidence Level
Benchmark Base TDE NLE Base TDE NLE
1 7066 9324 7014 0.4 0.2 0.3
2 6763 11173 7172 0.6 0.3 0.5
3 6580 9000 7715 0.2 0.2 0.3
4 6618 7318 6749 0.1 0.1 0.1
5 7093 8283 7107 0.2 0.2 0.2
6 7108 9317 7280 0.4 0.2 0.3
7 3958 9653 6580 1.1 0.7 1.7
8 15377 15581 15568 0.1 0.2 0.1
9 4651 10684 7345 3.4 0.3 1.5
10 8783 8564 9419 0.1 0.1 0.1
Fig. 8 Balanced dual-processor benchmark results, with an average sample mean decrease in execution speed for NLE compared to TDE of 17.2% (± 0.4% at the 95% confidence level).
Experimental Results: BalancedNon-virtual dual-processor
0 5 10 15 20
BaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLibBaseTDE
NetLib
BM 1
BM 2
BM 3
BM 4
BM 5
BM 6
BM 7
BM 8
BM 9
BM 10
Time (seconds)
Fig. 9 Balanced dual-processor mean execution time (ms) at 95% confidence level
Conclusions
Experimental results indicate that– Both TDE and NLE increase mean execution
time– NLE is 17.2% (± 0.4%1) faster than TDE on a
non-virtual dual-processor system, 4.1% (± 1.3%) faster on VMware, and 6.9% (± 2.8%) faster on Virtual PC
– TDE mean execution time increased by 25.2% (± 0.4%) over the baseline instance on a non-virtual dual-processor system
1 All ± % at the 95% confidence level
Future Work Benchmark on a more diverse pool of computer hardware and software
– Processor count and configurations– Total memory– 64-bit and 32-bit operating systems– 64-bit and 32-bit MSSQL and NLE
Increased database size Increased benchmark query complexity Simulated load Industry standard TPC benchmarks Stronger encryption bit strength (256-bit) CPU ticks and IO busy ticks Increased iterations Completely isolated computer system with minimal software to
prevent indeterminate anomalies
References
[1] The Canadian Press, "National Bank reports theft of laptop with mortgage loan database," September 23, 2008.
[2] C. Boss, "Reynoldsburg student information stolen," The Columbus Dispatch, August 28, 2008.
[3] D. Migoya, "Stolen state database puts 1.4 million at ID-theft risk," Denver Post, November 2, 2006.
[4] S. Hsueh, "Database Encryption in SQL Server 2008 Enterprise Edition," SQL Server Technical Article, Feb. 2008. Retrieved from the World Wide Web Sep. 24, 2008, http://msdn.microsoft.com/en-us/library/cc278098.aspx.
[5] NetLib, "Performance Benchmarks – Whole Database Encryption," Communications Horizon, LLC., Retrieved from the World Wide Web Sep. 24, 2008, http://www.netlib.com/files/performance_benchmarks_wholedb.pdf.
[6] National Institute of Standards and Technology (NIST), "Announcing the Advanced Encryption Standard (AES)," Federal Information Processing Standards Publication (FIPS) 197, United States of America Federal Government, November 26, 2001. Retrieved from the World Wide Web Oct. 4, 2008, http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf.
[7] Microsoft Corporation, Inc., "Access 2000 Tutorial: Northwind Traders Sample Database," June 22, 1999. Retrieved from the World Wide Web Sep. 24, 2008, http://www.microsoft.com/downloads/details.aspx?familyid=C6661372-8DBE-422B-8676-C632D66C529C&displaylang=en..
[8] M.L. Catalan, R. Ludena, A. Dennis, H. Umeno, "VM-Based Benchmark and Analysis System for Testing Online Transaction Processing," Second International Conference on Innovative Computing, Information and Control, ICICIC 2007, 2008, p. 4427665.
[9] Transaction Processing Performance Council, "TPC Benchmark E, Standard Specification, Version 1.6.0," Technical White Paper, 2008.
[10] C. Turbyfill, C. Orji, D. Bitton, "AS3AP – A Comparative Relational Database Benchmark," IEEE Computer Society International Conference, pp. 560-64, Feb. 1989.
[11] S. Mauvais, "Big Northwind Sample Code," May 23, 2007. Retrieved from the World Wide Web Sep. 24, 2008. http://www.mauvais.com/Download/ZD-BigNW.htm.
Thank You. Questions?
Benchmark Schema
Fig. 10 Benchmark schema
Benchmark Installation and Execution
Step 1: Create instances (Base, TDE, NLE) Step 2: Install NetLib Step 3: Configure NetLib with NLE instance Step 4: Create Northwind database within each instance Step 5: Create BigNW database from Northwind for each each
instance Step 6: Configure TDE for BigNW within TDE instance Step 7: Verify BigNW database in NLE instance is encrypted Step 8: Verify BIgNW database in TDE instance is encrypted Step 9: Verify benchmarks execution Step 10: Create TestResults database with each instance Step 11: Execute benchmarks Step 12: Evaluate benchmark data
TSQL Driver: Page 1-- CPU and timing variablesDECLARE @CPUStart int, @CPUEnd intDECLARE @StartTime datetime, @EndTime datetime-- Loop variablesDECLARE @InnerLoopCount int SET @InnerLoopCount = 1DECLARE @OuterLoopCount int SET @OuterLoopCount = 1DECLARE @InnerLoops intDECLARE @OuterLoops int-- Cursor variablesDECLARE @ScriptID int, @Benchmark int, @Description nchar(300), @Script nchar(300), @Iterate int, @Repeat int, @DisableCache bit,
@UseTable nchar(50)
-- Clean buffersCHECKPOINTDBCC DROPCLEANBUFFERS WITH NO_INFOMSGSDBCC FREESYSTEMCACHE ('ALL') WITH NO_INFOMSGS
-- Disable interactive output during processingSET NOCOUNT ON
-- Retrieve scriptsDECLARE curScripts CURSOR
FORSELECT [ScriptID], [Benchmark], [Description], [Script], [Iterate], [Repeat], [DisableCaching], [UseTable] FROM [TestResults].[dbo].[Scripts] ORDER BY [ScriptID]
OPEN curScripts
TSQL Driver: Page 2-- Loop over and execute scriptsFETCH NEXT FROM curScripts INTO @ScriptID, @Benchmark, @Description, @Script, @Iterate, @Repeat, @DisableCache, @UseTableWHILE (@@FETCH_STATUS = 0)BEGIN
-- Setup looping variablesSET @OuterLoops = @Repeat IF @OuterLoops <= 0Begin
SET @OuterLoops = 1End
SET @InnerLoops = @IterateIF @InnerLoops <= 0Begin
SET @InnerLoops = 1End
SET @OuterLoopCount = 1
PRINT 'Processing ' + Convert(varchar,@ScriptID) + ': ' + Convert(varchar(300),@Description)PRINT ' Instance = ' + Convert(varchar,'$(WhichInstance)')PRINT ' Benchmark = ' + Convert(varchar,@Benchmark)PRINT ' Iterations = ' + Convert(varchar,@Iterate)PRINT ' Repeat = ' + Convert(varchar,@Repeat)PRINT ' DisableCache = ' + Convert(varchar,@DisableCache)PRINT ''
TSQL Driver: Page 3-- Start outer loop block
While @OuterLoopCount <= @OuterLoops
BEGIN
-- Start performance block
SET @InnerLoopCount = 1
SET @CPUStart = @@CPU_BUSY
SET @StartTime = GETDATE()
WHILE @InnerLoopCount <= @InnerLoops
BEGIN
-- Begin dynamic execution
-- Select database to use for running scripts against and
-- execute script
EXEC ('USE ' + @UseTable + ';' + @Script)
-- Caching, if enabled
If (@DisableCache = 1)
BEGIN
CHECKPOINT
DBCC DROPCLEANBUFFERS WITH NO_INFOMSGS
DBCC FREESYSTEMCACHE ('ALL') WITH NO_INFOMSGS
END
-- End dynamic execution
SET @InnerLoopCount = @InnerLoopCount + 1
END
TSQL Driver: Page 4Set @EndTime = GETDATE()
SET @CPUEnd = @@CPU_BUSY
-- End performance block
-- Insert results into test table
INSERT INTO [TestResults].[dbo].[Results] ([Benchmark], [WhichInstance], [Cipher],[KeySize],[CPUTicks],[RunTime],[Query],[Iterate], [Repeat], [DisableCaching]) Values (@Benchmark,'$(WhichInstance)','AES','128',@CPUEnd-@CPUStart,DATEDIFF(ms,@StartTime,@EndTime),@ScriptID,@Iterate,@Repeat,@DisableCache)
-- Debug line
-- PRINT 'O: ' + Convert(varchar,@OuterLoopCount) + ', I: ' + Convert(varchar,@InnerLoopCount) + ', Diff: ' + Convert(varchar,DATEDIFF(ms,@StartTime,@EndTime))
--PRINT Convert(varchar,DATEDIFF(ms,@StartTime,@EndTime))
SET @OuterLoopCount = @OuterLoopCount + 1
END
FETCH NEXT FROM curScripts INTO @ScriptID, @Benchmark, @Description, @Script, @Iterate, @Repeat, @DisableCache, @UseTable
END
DEALLOCATE curScripts
SET NOCOUNT OFF
Benchmark Scripts: Page 1 Benchmark 1: Select orders ordered on or after 1/1/1996
– Select * INTO #Temp FROM [dbo].[Orders] WHERE [OrderDate] >= '1/1/1996'; IF OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;
Benchmark 2: Select orders shipped to Germany– Select * INTO #Temp FROM [dbo].[Orders] WHERE [ShipCountry] = 'Germany'; IF
OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;
Benchmark 3: Select all orders joined with customers– Select [dbo].[Orders].[OrderID], [dbo].[Customers].[ContactName] INTO #Temp FROM
[dbo].[Orders] LEFT JOIN [dbo].[Customers] ON [dbo].[Orders].[CustomerID] = [dbo].[Customers].[CustomerID]; IF OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;
Benchmark 4: Select invoices from Invoices view– Select * INTO #Temp FROM [dbo].[Invoices]; IF OBJECT_ID('TempDB..#Temp') IS NOT
NULL DROP TABLE #Temp;
Benchmark 5: Execute stored procedure Employee Sales by Country inserted into temp table
– EXECUTE [dbo].[Employee Sales by Country2] '1/1/1997','12/31/1997‘;
Benchmark Scripts: Page 2 Benchmark 6: Select ORDERS table into temporary table
– SELECT * INTO #Temp FROM [dbo].[Orders] IF OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;
Benchmark 7: Insert into Orders table– INSERT INTO [dbo].[Orders] ([CustomerID], [EmployeeID], [OrderDate]) Values
('VINET',5,'1/1/2005');
Benchmark 8: Updates Orders table– UPDATE [dbo].[Orders] SET [dbo].[Orders].[EmployeeID]=6 WHERE [dbo].[Orders].
[CustomerID]='VINET' AND [dbo].[Orders].[OrderDate]='1/1/2005‘;
Benchmark 9: INSERT and DELETE records from Orders table– INSERT INTO [dbo].[Orders] ([CustomerID], [EmployeeID], [OrderDate]) Values
('VINET',5,'1/1/2005'); DELETE FROM [dbo].[Orders] WHERE [dbo].[Orders].[CustomerID]='VINET' AND [dbo].[Orders].[OrderDate]='1/1/2005‘;
Benchmark 10: Add a backup device and backup the complete database– IF EXISTS (SELECT * FROM master.dbo.sysdevices WHERE NAME = 'BigNWBackupTest'
AND Status = 16) EXEC('sp_dropdevice BigNWBackupTest'); EXECUTE sp_addumpdevice 'disk', 'BigNWBackupTest','BigNWBackupTest.bak'; BACKUP DATABASE BigNW TO BigNWBackupTest WITH INIT;
Sample Graph:Raw Data Comparison
Benchmark 1 Run Chart - Raw Data Comparison
4000
6000
8000
10000
12000
14000
Tim
e (m
s)
Base TDE NetLib
Sample Graph:Delta Comparison
Benchmark 1 Run Chart - Delta Comparison
Base - TDE Base - TDE TDE - NetLib
Sample Chart:Scatter
Benchmark 1 Scatter Run Chart - Raw Data Comparison
4000
6000
8000
10000
12000
14000
Tim
e (m
s)
Base TDE NetLib