Software Quality Assurance Automated Testing Techniques

Michael BiwerSoftware Quality AssuranceAutomated Testing TechniquesApologyI did not communicate to Dr. Yang my Abstract and References by Monday 8:00AM.This is because I completely forgot and dropped the ball on the simplest part of this course.This is why there had been a late communication saying that I was to present today.Thankfully, Dr. Yang and me worked out a deal.

AgendaSoftware Quality Assurance: Automated TestingOverviewTesting MethodologyOverview of types of testsAutomated Testing SoftwareVersus Manual Testing, Pros, Cons, And Basic FunctionalityTypes of Automated Testing FrameworkModular, Data-Driven and Keyword-Driven Frameworks.DefinitionThe management and performance of test activities, to include the development and execution of test scripts so as to verify test requirements, using an automated test tool[2].HistoryBegan to evolve with the rest of software development as pressure was put on developers to:Reduce time, cut costs and increase quality.This began to create more incremental builds:Shown with the increase of Rapid Application Development methods such as Agile.Posed the difficulties of faster changes and additions to project requirements.Higher customer appreciation of quality control. (ISO 9000)Therefore, Software Quality Assurance: Automation Testing began to evolve and third party companies began to create automation suites.Developer TestingVS.Quality Assurance TestingDeveloper Testing:Ensures that the all of the projects requirements have been met through testing the processes.Usually, done through testing the code from an internal standpoint.Overall, looks to ensure that the process work correctly.Quality Assurance Testing:Ensures that all of the project requirements have been met through testing the product.Usually, done through testing the product from an external standpoint. Most instances mimics users view of the product.Overall, looks to improve the product as a whole.

Manual Testing Vs. Automated TestingManual:Labor intensive, testing take many man-hours to complete.Does not require software knowledge or skills from the testers.Repetitive tests still require many hours to complete and may be narrowed to fit a time schedule.Automated:Capital intensive.Large upfront cost for the software. Scalability might require hardware upgrades.Will not receive any benefits until the 2 3 project, because the creation of the test framework.Repetitive testing shows rates of returns around the 10th time it is run.Requires more specialist knowledge bases for the automation engineer.

Automated TestingManual TestingTest Case Creation1Need for test createdNeed for test created2Business Analyst creates test requirements, test plan and test cases which reflects the need in automated testing tool software.Business Analyst creates test requirements, test plan and test cases which reflects the need in a document.3Automation Engineer creates test script framework which reflect the test plan and test cases requirements.Manual Tester carries out the test plan, recording results of tests.4Automation Testers create driver scripts to run the test scripts generated by the automation Engineer.Results of tests and expected results are compared and bugs are recorded for the next build.5Tests are run automatically and results are analyzed. Business Analyst logs bugs for next build/run of test.N/AAll Runs of the Test Case Afterwards6The Business Analyst reviews test, makes adjustments if needed and re-run the test in the automation software.The Business Analyst reviews test, makes adjustments.7N/AManual Tester carries out the test plan, recording results of tests.Why?Proper defect tracking.Aim to enhance the overall quality of the product, not to just ensure it works correctly.Different eyes on the testing ensures that project requirements were communicated efficiently between customer and development team.Because not all developers are good testers, it is good to have a specialist test the product to ensure consistent quality.Ensures that the customer has input, in order to make sure that they will be happy with the end result.

Pros24/7 Testing.Repetitive testing is easily accomplished and cost effective in long running projects or in maintenance.In depth reporting tools.Consistent testing.Fewer, more skilled workers.ConsCostHeavy upfront cost with the purchase of the software and the time devoted to the creation of a automation framework.In addition, you will not see a return on investment right away.Must pay your employees more.You will require an automation engineer which has software programming knowledge.You will have to train your automation testing team in the automation software tool you choose.

Overview of Testing TypesTwo overall types of testing:White-Box Testing:Testing that is done from an internal viewpoint of the application.

Black-Box Testing:Testing that is done from an external viewpoint of the application with only the working knowledge of how the application should work.InputOutputInputProcessOutputOverview of Testing Methods (cont..)Unit TestingTests that target specific code to ensure that it can operate separately from the system given the correct inputs.Integration TestingTesting that aims to test the applications code as a whole to ensure that the components interact correctly.System TestingTesting that makes sure that the application meets the requirements set down for the software and that it doesnt interfere with any other platforms/applications.Acceptance TestingTesting where either the user or developer/QA team is given a build of the program in order to verify the requirements have been met.Overview of Testing Methods(cont)Smoke Testing:Build Verification Testing, Ensures that no serious problem has occurred between build of the application.Regression TestingTesting normally performed after a major code overhaul has been made to make sure that there are no persisting defects and that no functionality has been lost. Functional TestingTesting that only cover a particular flow or process within the program.Non-Functional TestingTesting that confirms the overall quality of the product in reference to the user.Performance TestingTesting that checks the scalability, reliability and resource usage of certain aspect of the program.Usability TestingTesting of the Graphical User Interface (GUI) works correctly in the given layout.When to Automate a TestIt makes the most sense to automate tests for products that are going to have continuous development or maintenance.This ensures that the test cases can be reused, increasing the return on investment of the automation software.Such as for the purposes of regression or smoke tests.Side Note:Developers often use automated testing methods for unit or integration tests. (Junit Tests.)Basic Automation Software FeaturesRecord/PlaybackProcess:Click the play button in the automation suite.The Automation Software tracks your mouse movements, clicks and keyboard actions.Click the stop button when the test completes.The Software then compiles the movements into a test cases for you to insert into a test plan.You are then able to perform those exact actions multiple times.Sometimes features of the automation suite include auto-generation of record/playback tests into test scripts.

Basic Automation Software FeaturesRecord/Playback (cont)Benefits:Simple, does not require a software background to utilize.Quick, there is not much set up to create this type of test. Can be used for repetitive testing on a basic level.DrawbacksThe types of tests that you can performs are limited to external options.If you have to change the test data, you must redo the entire test.Therefore, there can be a lot of maintenance.

Example of Record/Playbackhttp://youtu.be/facNRr8SHpk?t=45s Basic Automation Software FeaturesObject RecognitionAllows you to have an external look at the objects that are currently being displayed in the application being tested.Aka. GUI.You are able to store these objects in an Object Repository in order to easily reference them or you are able to Call them directly within your test scripts.They are formed in a hierarchy of container objects.Automation software with this ability gives you libraries which allow you to perform functions and retrieve/change properties of the objects.Basic Automation Software FeaturesObject Recognition(cont)Benefits:Allows you to set up more in-depth specific tests within test scripts.Drawbacks:Requires a test script framework to create test on these objects.Creates large initial overhead.Problems occur when custom objects/properties are created by the developers which the automation suite cannot recognize.This would require you to create your own Object Script Extender to recognize the object/property.

Example of HP QuickTest ProObject Recognition

Basic Automation Software FeaturesTest Script Execution FrameworkAllows you to create libraries in order to test any portion of your application.CodeGUIAPIsOnly limited to the amount of coding you want to do.These scripts are accessed through drivers and can be used across many different test cases.Commonly these test scripts are created using VBScript, Java, Perl, or CGI.The automation tools have pre-loaded libraries to give you extra capabilities with testing objects.Simple test script example.Uses a direct calling of objects to interact with them.

Basic Automation Software FeaturesTest Cases ReportsAble to view the passes and the fails of your automation tests in a report format.Since this is based on your test scripts, you only get as much detailed information as you allow the test case record to collect from the tests.HP QTP:ExampleReport

ComponentsTest PlanA set of test cases which verify similar requirements of the application.Test CaseA test to verify requirements of the application.Driver Scripts:The overall scripts that call your test scripts that make up your test cases.In essence, will reflect your test plan.Test Scripts:Code which allows for the tests actions to be carried out.Test Data:The data that you will be testing in your application.

Meta-Language DevelopmentMeta-Language is a abstract language used in running scripts. It has been simplified for the average user with no knowledge of programing to use.This allows testers that only have the business knowledge of the system to utilize testing tools.Allows for manual testers to migrate onto the new system easily.This is essential for the testers to take ownership of the test cases.It is used at the highest visible level for the users to see, the Driver Script.In essence however, the Meta-Language, will be a list of parameters that will be parsed/translated and used in the test scripts.

HP QTP Example:

Input parameters that the automation engineer has set for this Driver ScriptWhere the automation tester inputs the test parameters.Types:Modular FrameworkCreating specific test scripts in a library which are used within the Driver Scripts to perform the tests.IE: In the windows calculator, having a script that would verify that 5 + 5 = 10.Benefits:Quick and easy to put together.Drawbacks:Difficult to maintain.Less reusable code.Data is within the test scripts.Can easily have multiple copies of the same code.Because the test scripts have all of the control, this means that the ownership of the test cases lie in the automation engineer.

Types:Modular Framework (cou)Driver ScriptTest Library 1Test Library 2Function_1Function_2Function_3Function_4Function_5Function_6Example of framework setup:Example of Psuedocode for a Modular Framework Driver Script:

MainRead Parameters

Window_1 FunctionsFunction_1 CallFunction_2 CallFunction_3 CallWindow_2 FunctionsFunction_4 CallFunction_5 CallFunction_6 Call

End MainAction_TypeName_Of_ScriptRun_ScriptVerify_Calc_ScriptExample of Driver Script Parameters:Types:Data-Driven FrameworkThis framework looks to fix the problem of data being within the test script. The input is commonly moved to the driver script or to an external database or spreadsheet which is fed into the test scripts.IE. In the windows calculator, you have a spreadsheet with numbers which are run through a script that adds them together.Benefits:Gives ownership of the test cases to the testers.Maintenance is more manageable.Drawbacks:Test case information can be split into different areas confusing large operations.Scalability is a real problem.Types: Data-Driven Framework (cou)Driver FileScript_1Script_2Script_3Data_File_1Data_File_2Data_File_3Comparison of Expected and Actual OutputsExample of framework structure:Pseudocode for Data-Driven Driver Script:

MainRead in ParametersPerform required Script with given parametersScript_1 Call Script_2 CallScript_3 CallEnd Main

Script_NameData_FileNumbersExpected_ResultAdd_Calc_ScriptValue_Add_To_10.xmlA1-A2B1Sub_Calc_ScriptValue_Sub_To_0.xmlA1-A2B1Mul_Calc_ScriptValue_Mul_To_49.xmlA1-A2B1Example of the Data-Driven Driver Script:Types:Keyword-Driven FrameworksThis framework looks to provide maximum ownership to the testers and maximum flexibility.Does this by providing a object by object look at the test cases and providing actions that can be taken out on those objects.Benefits:Provides all ownership to the testers.Maximum flexibility and highly scalable.Maintenance is on the testers and is standardized.Drawbacks:Large overhead time for creating the framework.A driver file must be created for each window of the application.An iterative approach which can increase test case execution times.

Types: Keyword-Driven Framework (cou)

Pseudocode for Keyword-Driven Driver Script:

MainRead in ParametersParse parameters into correct Object+Function+Data groupsFor Each Object in windowSelected ActionObject_1 Action CallsObject_2 Action CallsObject_3 Action CallsEnd MainIterationAction_TypeButton_1Edit_1CheckBox_11ClickClickClick2InsertHello3ClickHighlight4Key_CNTRL_X5clickClick6Key_CNTRL_V7PropertyText=HelloChecked=TrueTypes: Hybrid FrameworkCombines the previous frameworks together in various levels to provide maximum testing capabilities.Normally involves connecting to a test database for various inputs.

Example of Hybrid Framework Psuedocode:

MainRead ParametersGroup ParametersLink to databaseSelect a specific windowFor each object in a specific windowPerform the needed scriptsEnd MainWhat we discussedSoftware Quality Assurance: Automated TestingOverviewTesting MethodologyOverview of types of testsAutomated Testing SoftwareVersus Manual Testing, Pros, Cons, And FunctionalityTypes of Automated Testing Framework

ReferencesChaos manifesto 2012: The year of the executive sponsor. (2012). Retrieved from http://versionone.com/assets/img/files/CHAOSManifesto2012.pdfElfriede, D., Rashka, J., & Paul, J. (2008).Automated software testing: Introduction, management and performance. (13 ed.). Castleton, New York: Hamilton. Retrieved from http://books.google.com/books?id=kl2H0G6EFf0C&lpg=PR5&ots=wN_PrWrBx1&dq=automation software test qa&lr&pg=PA43Mosley, D., & Posey, B. (2002).Just enough software test automation. Upper Saddle Hill, new Jersey: Prentice Hall PTR. Retrieved from http://books.google.com/books?id=PEBvfWESIt4C&lpg=PR15&ots=xGxZPGvp2P&dq=software test automation &lr&pg=PA3Laukkanen, P. (2006).Data-driven and keyword-driven test automation frameworks. (Master's thesis)Retrieved from http://eliga.fi/Thesis-Pekka-Laukkanen.pdfSoftware testing. (n.d.). Retrieved from http://en.wikipedia.org/wiki/Software_testingAllott, S. (1999, 10).Automate your tests - you wont regress it!. Seventeenth annual pacific northwest software quality conference, Oregon Convention Center Portland, Oregon. Retrieved from http://www.snapshots.pnsqc.org/2011-theme/pnsqc-04-06-2011/proceedings/pnsqc99.pdfKumar, S. (Designer). (2007, April 9). File:SampleCode.JPG [Print Photo]. Retrieved from http://en.wikipedia.org/wiki/File:SampleCode.JPGKiranpaul , K. (n.d.).Quick test professional tutorials - qtp fundamentals, qtp features, vb scripting examples in qtp, qtp framework and vb script.. Retrieved from http://qtp4free.blogspot.com/2010/05/object-spy-in-qtp-92.htmlJain, A. (Photographer). (2013, July 5). Good Looking and Enhanced Results Viewer [Web Graphic]. Retrieved from http://2.bp.blogspot.com/_iTz5d0BSy6g/TPY5-7SOIGI/AAAAAAAAAPM/wxlEl9BHMqM/s1600/QTP_11_RV_SS.pngWalter , T. (Photographer). (2012, July 6). Add Parameters to specify the Ranorex Automation [Web Graphic]. Retrieved from http://www.ranorex.com/blog/wp-content/uploads/2012/07/Add-Parameters-To-Test.pngTutor. (2012, Aug 6).Software testing life cycle (stlc) complete tutorial. Retrieved from http://www.sdlc.ws/software-testing-life-cycle-stlc-complete-tutorial/Function_1 + Object_1 + Data Function_1 + Object_2 + Data Function_2 + Object_2 + Data

Driver Script

Test LibrariesFunction_1Function_2