BENCHMARKING BENCHMARKING DATABASESDATABASES

BENCHMARKING BENCHMARKING DATABASESDATABASES

By Samy KabanguBy Samy Kabangu

SupervisorSupervisor : Mr. John: Mr. John EbdenEbden

Computer Science DepartmentRhodes University

BACKGROUNDBACKGROUNDBACKGROUNDBACKGROUND

DATABASE WARS!DATABASE WARS!

There is a high competition in the Database computing industry.

Spectacular claims are made by different vendors in favour of their products in terms of performance.

Here is one of the claims published Here is one of the claims published in a white paper by Microsoft.in a white paper by Microsoft.

““ Microsoft SQL Server has steadily gained ground on Microsoft SQL Server has steadily gained ground on other database systems and now surpasses the other database systems and now surpasses the competition in terms of performance, scalability, competition in terms of performance, scalability, security, developer productivity, business intelligence security, developer productivity, business intelligence (BI), and compatibility with the 2007 Microsoft Office (BI), and compatibility with the 2007 Microsoft Office System. It achieves this at a considerably lower cost System. It achieves this at a considerably lower cost than does Oracle Database 11g.”than does Oracle Database 11g.”

Microsoft, 2009Microsoft, 2009

The question is: “should the public, database The question is: “should the public, database professionals, database users and academics professionals, database users and academics

believe in such claims ?”believe in such claims ?”

Industry Standard BenchmarksIndustry Standard Benchmarks

Benchmarking a database is the process of Benchmarking a database is the process of performing well defined tests on that performing well defined tests on that particular database in order to evaluate its particular database in order to evaluate its performance.performance.

Industry standard database benchmarks Industry standard database benchmarks provide a platform for benchmarking various provide a platform for benchmarking various database products.database products.

The obtained results are used to compare The obtained results are used to compare and rank database products.and rank database products.

Some of the Issues with the Industry Some of the Issues with the Industry Standard BenchmarksStandard Benchmarks

Though they provide means of knowing the performance of Though they provide means of knowing the performance of database products, their adoption might be a little difficult.database products, their adoption might be a little difficult.

They are performed on specific hardware platform and They are performed on specific hardware platform and operating system.operating system.

They simulate real world workload that might not reflect the They simulate real world workload that might not reflect the actual workload of a specific organisation database system.actual workload of a specific organisation database system.

Some vendors use techniques such as caching the data and Some vendors use techniques such as caching the data and SQL statement into memory(RAM) in order to avoid disk SQL statement into memory(RAM) in order to avoid disk access so as to improve performanceaccess so as to improve performance

Vendors cite the benchmarks tests results that favour their Vendors cite the benchmarks tests results that favour their product. product.

STATEMENT OF THE STATEMENT OF THE PROBLEMPROBLEMSTATEMENT OF THE STATEMENT OF THE PROBLEMPROBLEM

Our objective is to come up with a best way of Our objective is to come up with a best way of testing a database management system on testing a database management system on several different hardware and software several different hardware and software platforms. The variables are: platforms. The variables are:

64 bit + quad processor @ 2.66GHz each + 2.96G 64 bit + quad processor @ 2.66GHz each + 2.96G RAM with 32 bit and then 64 bit software, RAM with 32 bit and then 64 bit software,

32 bit P4 @ 2.8GHz 1 GB RAM with 32 bit 32 bit P4 @ 2.8GHz 1 GB RAM with 32 bit software,software,

An investigation into to the most critical An investigation into to the most critical benchmarking techniques and factors in tuning benchmarking techniques and factors in tuning database management systems will be the main database management systems will be the main focus of this project. focus of this project.

We will firstly provide ourselves with a sound understanding of the factors that influence the performance of a database management system.

Which will enable us in the investigation of some of the well known industry standard database benchmarks such as the TPC.

Run some TPC benchmark tests on SQL Server 2008.

INTENDED APPROACHINTENDED APPROACHINTENDED APPROACHINTENDED APPROACH

EXPECTED RESULTSEXPECTED RESULTSEXPECTED RESULTSEXPECTED RESULTS

Implementation of sound practice techniques Implementation of sound practice techniques in tuning a database in order to obtain an in tuning a database in order to obtain an optimum performance from it.optimum performance from it.

To be able to answer the question:”Why To be able to answer the question:”Why using the TPC Benchmarks is better than using the TPC Benchmarks is better than using other industry standard benchmarks?”.using other industry standard benchmarks?”.

Observe the performance of Microsoft SQL Observe the performance of Microsoft SQL server 2008 on a core 2 Quad processor if it server 2008 on a core 2 Quad processor if it will match the one published by the TPC-H will match the one published by the TPC-H run on mainframe type hardware?run on mainframe type hardware?

EXPECTED RESULTSEXPECTED RESULTSEXPECTED RESULTSEXPECTED RESULTS

Can new software (Microsoft SQL Can new software (Microsoft SQL Server 2008 – 32 and 64 bit versions) Server 2008 – 32 and 64 bit versions) being run on slow hardware match the being run on slow hardware match the performance of old software (Microsoft performance of old software (Microsoft SQL Server 2005) run on fast SQL Server 2005) run on fast hardware?hardware?

The primary rigging database methods The primary rigging database methods used by vendors to increase the used by vendors to increase the performance of their products.performance of their products.

Can SQL Server “beat” Oracle (as per Can SQL Server “beat” Oracle (as per white paper mentioned above).white paper mentioned above).

FUTURE EXTENSIONFUTURE EXTENSION

This project can be extended to the design of database benchmarks.

QUESTIONS?QUESTIONS?