QaOps - NYC Selenium Users Meetup - 03/15/12

Christopher Hoey

■ Senior IT Manager - Wireless Generation

■ Started in QA

■ Switched over to Operations

■ Selenium abuser since 2006

■ Got to see the Selenium vs Webdriver Cage Fight live

Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012

http://www.youtube.com/watch?v=Vlz-WmcrBL8


Founded 12/14/06 Closed 07/16/09

Thanks!

Ashley WilsonLindsey BaronSauce LabsConductor

NYC Metro Selenium Users

Selenium Developers

Jmeter Developers

Show of hands

Numbers collected and updated before sharing slides

About 35 attended and participated in informal poll


Show of hands - primary job title

Quality Assurance ?12



Developer ? 10



Operations ? 5


Show of hands - experience with Selenium

Have not used it?5



Tried Selenium IDE?

8Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012


Mastered Selenium IDE?



Tried Selenium RC/Webdriver?



Mastered Selenium RC/Webdriver?



Selenium tied to CI build system?



Running a Selenium Grid?


Show of hands - use of Selenium

Functional Testing New Features?



Regression Testing?



Load/Stress Testing? really?



Benchmarking Performance?



Automating Data Entry?



Monitoring for Operations?



Something Strange?

2 Filling out forms to renew trial licenses when expired

Running build jobs for a build tool that does not have a cli


DisclaimersNot a talk about best practices

Forcing Selenium to do things not originally intended

A lot of talk about Selenium IDE QAOps term kind of meant as a joke but might have meaning? Times have changed, tools improved and storms have passed


Dev + Ops = DevOps --> QA + Ops = QAOps


About Wireless Generation

● Create tools that make teacher's jobs easier

● Primarily focused around assessment and reporting apps

● Mix of mature code/platforms and new development projects

● Experiencing rapid growth year over year


Faith in monitors pretty low Manual configuration management Serious bugs in third party tools Large increases in user traffic Lots of new projects to juggle while keeping existing work online Sysadmins manually checking site =(

Reuse anything that helps!

Going MacGyver in early days of Ops

Selenium to the rescue!


● Use Selenium IDE so anyone in Operations with Firefox can use

● Suites are happy path smoke tests that are not brittle ● Focus on touching all moving parts of infrastructure

● All suites can run with any username

● All suites can run on any environment

● Sysadmins running Selenium tests -- still done manually =/

Suites of Smoke Tests for Operations


Suites for Operations

DEMO TIME

Removing brittleness from Selenium IDE tests against a site with difficult

locators


Suites for Operations - DEMO TIME

All we really want to know is if we threw any errors while loading pages that touch on all parts of our platform. Two simple regex searches for every page gets us the desired minimal coverage

Yes that looks way too simple but it did the job very well.

These tests started finding issues before our customers did.

Confidence increased dramatically!


Suites for Operations - DEMO TIME

https://zzz.yyy.net/wgen/tpri/LoadTpriClasseSummary.do?ppkDefaultVersionId=7&ppkPeriodId=20&ppkClassId=550987013&ppkCompetencyDisplayId=1&ppkInstId=33645685&ppkSchoolGroupId...

We know the starting URL pattern.

The rest of the URL is really dynamic and the page locators that happen to be available are not Selenium friendly. We could use XPath but that has proven to be really brittle for us. Solution - Store the html source and use storeEval with a regex to find the correct link every time


Automation - judges.py

■ Headless Selenium servers

■ Suites of tests for each production webapp using smoke tests

■ Run through suites serially in a never ending loop

■ Any failure in suites marks the status as failed for that server

■ Look at each servers total result time running the

suite based on thresholds

■ Running helps see what server is currently being tested

■ Results projected on wall in Ops area


We have a fundamental problem

Sysadmins Customers


Solution - Any browser any platform

Bromine

Grid


Loadtesting Selenium vs JMeter

● Lots of start up overhead

● ~30 concurrent sessions before strange things happen

● Lots of requirements / coordination headless is a hack

● Loads quickly

● ~300 concurrent sessions before strange things happen

● Java and a jar, headless is easy


Loadtesting with JMeter

Queries to find interesting users

Scripts to replace variables for Selenium suites

Playback Selenium scripts through JMeter proxy

Selenium Scripts IN

JMeter Scripts OUT


Loadtesting with JMeter

DEMO TIME

Setting up jmeter as a proxy recorder for Selenium

playback


Loadtesting with JMeter - DEMO TIME

Start JMeter and create a new basic WorkBench



Add an HTTP Proxy Server: Edit -> Add -> Non-Test Elements -> HTTP Proxy Server



Set your port, any record filters of what to include or exclude and click start to fire up the proxy



Set Firefox network settings to use localhost and the port you configured JMeter proxy to use



Playback your tests using Selenium and watch as JMeter records the hits



Add a thread group by right clicking Test Plan -> Add -> Threads -> Thread Group



Name your thread group, pick how many threads you want, ramp up time and how many loops



Run your tests and set servers ablaze. Run -> Start


Finding hard bugs

Uh oh! Just throwing more memory at the problem will not solve this!


Finding hard bugs

Did we see this in the environments below Production? Not very noticeable unless looking hard for it.


Finding hard bugs

Ramping up JMeter threads of a specific user in an endless loop reproduces issue in every environment


Dashboard Setup?

Use Selenium to login and navigate to your various dashboards for TV's and sharing with teams.

QAOps?


Getting most of the focus these days. Certainly best of all worlds coming together. Takes a while to be a good practitioner based on tech debt | servers | timeboxes | complexity | culture

A lot of quick wins can be made here.

Selenium IDE makes it trivial for Ops to start running and developing tests

Questions?

We are hiring!

http://www.wirelessgeneration.comDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012

http://www.wirelessgeneration.com

Picture Credits


http://www.russpainter.com/shop/images/clip.jpg

http://g-ecx.images-amazon.com/images/G/01/stores/sport-goods/1209-huntsman.jpg

http://uploads.neatorama.com/wp-content/uploads/2010/06/macgyver.jpg

http://www.agilegeoscience.com/storage/post-images/python-logo-small.png?__SQUARESPACE_CACHEVERSION=1316752792052

http://www.packagingsuppliesbymail.com/images/C/Duct%20Tape.jpg

Picture Credits


http://upload.wikimedia.org/wikipedia/en/0/07/Jakarta_jmeter_logo.jpg

http://cdn.babble.com/momcrunch/files/2012/02/SauceLabs_Logo_Vert_150px.gif

http://www.pptbackground.net/plog-content/images/powerpoint/education-backgrounds/raising-hands-together-powerpoint-backgrounds.jpg

http://gigaom2.files.wordpress.com/2011/01/firefox-ie.png

http://seleniumhq.org/images/selenium-ide-logo.png

Picture Credits


http://www.portlandmercury.com/imager/dr-strangelove/b/original/3791589/80b9/mwbw-drstrangelove.jpg

http://stylreesta.files.wordpress.com/2011/07/caution_big.png

http://3.bp.blogspot.com/_t5Xqp0AYeg4/S74FdH-SO-I/AAAAAAAAAOU/OlMrI9m8VIE/s320/smileyface1.jpg

http://en.wikipedia.org/wiki/File:Devops.png

http://blogs.babycenter.com/wp-content/uploads/2008/02/crying-face.jpg

Picture Credits


http://seleniumhq.org/images/selenium-grid-logo.png

http://www.studiosysadmins.com/site_media/attachments.gallery/2009/05/12/ServerRoomFireWeb.jpg

http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Tux.png/220px-Tux.png

http://blogs.splunk.com/wp-content/uploads/2010/02/Picture-65.png

http://www.wisedatasecurity.com/imagenes/ie-8.jpg

QaOps - NYC Selenium Users Meetup - 03/15/12

Technology

Transcript of QaOps - NYC Selenium Users Meetup - 03/15/12