JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation

© 2012 IBM Corporation

Testing with Bytecode Instrumentation (BCI) JavaOne 2012 - CON3961

Paul Thwaite – Java 8 Test LeadWednesday 3 October 2012

2 © 2012 IBM Corporation

Important Disclaimers

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.

ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.

IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.

IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS


Introduction to the speaker – Paul Thwaite

■ 11 years experience developing and deploying Java SDKs

■ Recent work focus:– Quality Assurance lead

• Leading cross-continent teams• System Test on IBM SDKs

– Test lead for Java 8• Focus on testing in OpenJDK• OpenJDK contributor

■ My contact information:– [email protected]– linkedin.com/profile/view?id=26327969


What you will learn from this talk

■ Why me and what I do■ What is BCI?■ Typical problems encountered during testing■ Two demos showing how BCI can help with testing■ Introduction to Java agents■ How BCI works■ Limitations of BCI■ Overview of the different BCI approaches


Why me and what I do

■ Test IBM's Java SDK– I break Java

■ Focus on system testing in QA– Multi-threaded load testing– Third-party applications

■ System test the Java SDK means writing Java■ Can the VM handle any application or Java code?


Bytecode Instrumentation


What is BCI?

■ Bytecode Instrumentation (BCI) is modifying the bytes of a class at runtime■ Classes can be modified as the JVM loads them or at any point afterwards■ Application and SDK classes can be changed (with few restrictions)■ Abstraction frameworks exist to make BCI easier

Code

CodeClass

BCI

JVM

Code

Code

Class

Snippets

Disk Runtime


ASM

■ ASM is an all purpose Java bytecode manipulation framework■ It can be used to modify existing classes or dynamically generate classes, directly in binary

form■ Many products such as AspectJ and JRuby use ASM■ The examples in this presentation use ASM to abstract out the complexity of transforming

classes


Problems encountered in testing


Typical problems encountered during testing

■ Test exception handling

■ Test for memory exhaustion

■ Test secure paths through code

■ Test specific code paths in very large systems such as WebSphere Application Server


Problem 1: Testing all exception paths

try {

Socket aSocket = new Socket("www.example.com", 1234);

// more code...

} catch (UnknownHostException e) {

//Handle correctly

} catch (IOException e) {

//Handle correctly

} catch (SecurityException e) {

//Handle correctly

}


java.net.Socket



try {


// more code...


//Handle correctly


//Handle correctly


//Handle correctly

}

■ The test running normally



try {


// more code...


//Handle correctly


//Handle correctly


//Handle correctly

}

■ How do you force the exception?– Pull the network cable out of the test machine– Update /etc/hosts to force the exception

■ Not practical– Manual, test-specific, other tests may run on the same machine



try {


// more code...


//Handle correctly


//Handle correctly


//Handle correctly

}

■ How do you force the exception?– Intercept the connection– IOException is a “catch all” and very hard to create– You would need to understand the implementation in order to test properly

■ Not practical



try {


// more code...


//Handle correctly


//Handle correctly


//Handle correctly

}

■ How do you force the exception?– A SecurityManager would need to be used– Complicated to setup– Not practical especially if testing a complex system– Likely would need two versions of the test


Demo using BCI to help solvethese problems


Problem 1: Demo of UnknownHostException

■ What does the test do?– Uses java.net.Socket to create a connection to three hosts– Prints to stdout whether the connection was successful or not– BCI is used to trigger the UnknownHostException on www.oracle.com

■ Test 1 – the test running normally

java -cp testcase tests.NetworkTest

■ Test 2 – the test running with BCI

java -Xbootclasspath/a:lib/asm-all-4.0.jar:lib/networkTrans.jar -javaagent:lib/networkTrans.jar=www.oracle.com -cp testcase tests.NetworkTest

Pass in the host on which to force the UnknownHostException

http://www.oracle.com/


Problem 1: Demo of UnknownHostException

■ What does the test do?– Uses java.net.Socket to create a connection to three hosts– Prints to stdout whether the connection was successful or not– BCI is used to trigger the UnknownHostException on www.oracle.com



Connected to host: www.example.comConnected to host: www.oracle.comConnected to host: www.ibm.com

Connected to host: www.example.comCouldn't find host: www.oracle.com UnknownHostException thrownConnected to host: www.ibm.com


Problem 1: Breakdown of the Java command

java -Xbootclasspath/a:lib/asm-all-4.0.jar:lib/networkTrans.jar -javaagent:lib/networkTrans.jar=www.oracle.com -cp testcase tests.NetworkTest

Append the ASMlibrary to thebootclasspath

Append a Javaagent library tothe bootclasspath

Enable the Java agent Pass in options tothe agent

Run the test as normal - the test does not change

Note: The bootclasspath is used because the Socket class is being modified.


Problem 2: Testing Resource Exhaustion

■ A large system, such as WebSphere Application Server, should be able to handle an OutOfMemoryError

■ An OutOfMemoryError can occur at any time– It can also happen in different places within the system

■ Hard and complicated to test without side effects– Changing the application server's environment is not a good idea– Using a reduced heap will not guarantee all cases are covered


Problem 2: Demo of OutOfMemoryError – Java command

■ What does the test do?– Fills up the heap with Intergers until an OutOfMemoryError exception is thrown– It will take a very long time for the OutOfMemoryError exception to be thrown– BCI is used to set the trigger to just three Integers without having to change the test


java -cp testcase tests.AllocateTest


java -javaagent:lib/resourceTrans.jar=3 -cp testcase:lib/asm-all-4.0.jar tests.AllocateTest

Option to trigger OOME at just 3 Integers


Problem 2: Demo of OutOfMemoryError – stdout

■ What does the test do?– Fills up the heap with Intergers until an OutOfMemoryError exception is thrown– It will take a very long time for the OutOfMemoryError exception to be thrown– BCI is used to set the trigger to just three Integers without having to change the test



Adding new value 0Adding new value 1Adding new value 2......Adding new value 450...

Adding new value 0Adding new value 1Adding new value 2Adding new value 3Exception in thread "main" java.lang.OutOfMemoryError at transformer.ResourceTransformer.resourceHelper(ResourceTransformer.java:79) at tests.AllocateTest.increment(AllocateTest.java) at tests.AllocateTest.main(AllocateTest.java:12)


Problem 2: Breakdown of the Java command

java -javaagent:lib/resourceTrans.jar=3 -cp testcase:lib/asm-all-4.0.jar tests.AllocateTest

Enable the Java agent Pass in options tothe agent

Run the test as normal - the test does not change

Add the ASM library to the classpath


Recap – why BCI?

■ Some parts of Java applications are hard to test■ BCI provides a solution to this problem■ With BCI, you can:

– Replace your method bodies at runtime– Replace your entire class– All without having to change your code

■ BCI is performed by inserting new bytecodes into your application at runtime■ Using the UnknownHostException example, there are two ways to perform BCI

– Change the testcase to use a mock version of java.net.Socket which calls a different class i.e. MySocketClass

– Modify the java.net.Socket class■ Whichever you choose, you will be playing with bytecodes


Recap – why BCI?

■ Some parts of Java applications are hard to test■ BCI provides a solution to this problem■ With BCI, you can:

– Replace your method bodies at runtime– Replace your entire class– All without having to change your code

■ BCI is performed by inserting new bytecodes into your application at runtime■ Using the UnknownHostException example, there are two ways to perform BCI

– Change the testcase to use a mock version of java.net.Socket which calls a different class i.e. MySocketClass

– Modify the java.net.Socket class■ Whichever you choose, you will be playing with bytecodes■ But don't fear – there are tools available which can help you!


Java agents


Java agent overview

■ A Java agent provides instrumentation capabilities to a Java application■ Java agents provide the means to perform bytecode transformation■ An agent must have a premain method

■ It is packaged inside a JAR file■ The JAR file manifest defines the agent class■ The agent has 'no modelling restrictions' i.e. it can do anything any other java code can do

– Start threads, listen on the network etc

■ The Instrumention object is used to perform class transformations

import java.lang.instrument.Instrumentation;public class MyJavaAgent { public static void premain(String agentArgument, Instrumentation instrumentation) {} } Manifest-Version: 1.0

Premain-Class: MyJavaAgent


How BCI works


Using BCI to modify java.net.Socket

Socket

Socket(InetAddress addr)

Socket(String host,int port)

visitMethod(info) isTargetMethod?

visitCode()insert new code

ASM



Socket



ASM





Socket



ASM



if(hostname.equals(host)) throw newUnknownHostException();


if (hostname.equals(host)) throw newUnknownHostException();

mv.visitVarInsn(Opcodes.ALOAD, 1);

mv.visitMethodInsn(INVOKESPECIAL, "java/net/UnknownHostException", "<init>", "()V");

mv.visitCode();

ASM logic


if (hostname.equals(host)) throw newUnknownHostException();

0: aload_1 1: ldc #2; // String host 3: invokevirtual #3; //Method java/lang/String.equals:(Ljava/lang/Object;)Z 6: ifeq 17 9: new #4; //class java/net/UnknownHostException 12: dup 13: invokespecial #5; //Method java/net/UnknownHostException."<init>":()V 16: athrow 17: return

Bytecodes


Limitations of BCI at runtime

■ Restrictions when retransforming classes at runtime:– Fields cannot be added or removed– Methods cannot be added or removed– Method signatures cannot be modified

■ Effectively only method bodies can be modified■ Most classes can be modified (including most SDK classes)


BCI Options


Replace a class using a pre-compiled version of your new class

■ Pre-compile a new version of a class■ When the class to be loaded next is the one you want to replace■ You can return the bytes for your new class and discard the old one■ No knowledge of bytecode is required■ Original source must be available and unchanged■ Need to match the correct compiler version


Modify bytecodes with a BCI framework

■ Use a framework– Provides an abstraction layer– Easier to learn (provides symbolic constants, helper APIs)– Adds an additional dependency

■ ASM – Widely used– Uder active development– Friendly license

■ ASM gives you a good symbolic framework for writing bytecodes■ Knowledge of the JVM architecture is required


Use javap to create bytecodes

■ Write some java code that does what you want■ Compile your class to byte-code■ Use a de-compiler to examine the class file■ JVM ships with javap■ verbose option (-v) decompiles method bytecodes

void setupConnections();

Code:

0: ldc #2 // String www.example.com

2: sipush 1234

5: invokestatic #3 // Method MySocketFactory.newSocket:(Ljava/lang/String;I)Ljava/net/Socket;

8: astore_1

9: return


ASMifier – Create ASM logic

■ ASMifier– Part of standard ASM distribution– Turn a Java method into byte code logic for insertion into an ASM class – An Eclipse plugin is available

mv.visitLdcInsn("www.example.com");

mv.visitIntInsn(SIPUSH, 1234);

mv.visitMethodInsn(INVOKESTATIC, "transformer/SocketHelper$MySocketFactory", "newSocket", "(Ljava/lang/String;I)Ljava/net/Socket;");

mv.visitVarInsn(ASTORE, 1);

mv.visitInsn(RETURN);


What can go wrong? (Dealing with class verify problems)

■ If the JVM is unhappy with your modified class, it may throw a java.lang.VerifyError

■ 'stack shape inconsistent'/'stack underflow'– Attempting to operate on the wrong type of variable/stack is empty– Often due to failure to push/pop all the correct variables

■ 'Inconsistent stackmap frames'– Stackmaps are method meta-data used by the JVM– A map of all the variables on the stack at the target of any branch/goto instruction– BCI cause targets to move, invalidating the stack maps– ASM will re-calculate them for you

■ 'stack overflow'– Maximum stack size specified in method meta data– Any operations added via BCI may increase required stack– ASM will calculate that for you too


What can go wrong? (Runtime Problems)

■ No compile time checking

■ Access control– Calling a private method is valid byte-code, fails at runtime

■ Calling a non-existent method– Incorrect argument types– Incorrect argument ordering– Typos


Conclusion

■ BCI is very powerful■ BCI is used in many products■ JVM knowledge is required but...■ There are many tools available to make it easier■ BCI allows testing of otherwise impossible or tricky situations

■ Slides will be available on Slideshare after the conference■ Examples are available on request

– [email protected]


References

■ Get Products and Technologies:– IBM Java Runtimes and SDKs:

• https://www.ibm.com/developerworks/java/jdk/– IBM Monitoring and Diagnostic Tools for Java:

• https://www.ibm.com/developerworks/java/jdk/tools/

■ Learn:– IBM Java InfoCenter:

• http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp

■ Discuss:– IBM Java Runtimes and SDKs Forum:

• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0

https://www.ibm.com/developerworks/java/jdk/

https://www.ibm.com/developerworks/java/jdk/tools/

http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0


References

■ Eclipse Bytecode Outline plugin (allows you to use asmifier in Eclipse)– http://andrei.gmxhome.de/bytecode/index.html

■ ASM– http://asm.ow2.org/index.html

■ Java VM Spec– http://docs.oracle.com/javase/specs/jvms/se7/html/index.html

■ java.lang.instrument package documentation– http://docs.oracle.com/javase/7/docs/api/index.html?java/lang/instrument/package-summary.html


Copyright and Trademarks

© IBM Corporation 2012. All Rights Reserved.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., and registered in many jurisdictions worldwide.

Other product and service names might be trademarks of IBM or other companies.

A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademark information” page at URL: www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/legal/copytrade.shtml

JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation

Technology

Transcript of JavaOne 2012 CON 3961 Innovative Testing Techniques Using Bytecode Instrumentation