Axis2 client memory leak

Outline

• Issue & diagnosis• Postmortem from technique point– Axis2 1.4/1.4.1/1.5.6 client– Implement & design issue

Issue

CServer exhaust 8G memory -> memory leak

diagnosis

jmap –histo pid

num #instances #bytes class name---------------------------------------------- 1: 1001744 246906600 [C //char array 2: 2855952 137085696 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment 3: 282338 115692192 [I // int array 4: 677362 102508648 [Ljava.util.HashMap$Entry; 5: 192117 99164392 [B // byte array 6: 2856122 91732120 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry; 7: 2855952 91390464 edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync 8: 1337498 53499920 java.lang.String 9: 433634 43130120 [Ljava.lang.Object; 10: 597908 38266112 java.util.HashMap

1) The main change in this release is using web service client (axis2 1.4.1)

From the histogram, the memory leak is caused by Axis2.

2) char[] / edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap

3) Google ‘axis2 1.4.1 concurrentHashMap memory leak‘

diagnosis

• We got a 2.39G memory heap dump– Very fast, less than 30 seconds (because 8G

memory in production)• Problem : too big to open in dev box– Need enough physical memory– (May) need 64 bit OS– Try Jhat/Jprofile/YJP/Jmat eclipse plugin

(windows/2G)

jmap –dump:file=cdump pid

Open by Jmat standalone application

diagnosis

AxisConfiguration retain 540.9m heap

diagnosis

Hashtable allEndPoints holds more than 2k axis2 endpoint instances

diagnosis

All of the endpoint instances are about FavoriteService

1) Confirm in the code, only FavoriteService related did not call cleanup

Axis2 1.4 memory leak

• Reproduce

for( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request); }

Axis2 1.4 memory leak

• Informal Model (OO design)

AxisConfiguration

Map allServices = new Hashtable();Map allEndpoints = new Hashtable();

Axis2 1.4 memory leak

• Cause : Stub Initiation, add into AxisConfiguration

//Init, AddallServices.put(serviceName, axisService);..allEndpoints.put(serviceName + "." + endpointName, axisService);

Axis2 1.4 memory leak

• Cause : Stub Finalize

//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}

public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName());}

//AsixConigurationpublic synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }

Axis2 1.4 memory leak

• Cause : Client Finalize

//ServiceClientprotected void finalize() throws Throwable { super.finalize(); cleanup();}

public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); …}

//AsixConigurationpublic AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices();

while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName());

… } …}

Axis2 1.4 memory leak

• Cause

• Memory Leak : allEndpoints

//Init, AddallServices.put(serviceName, axisService);..allEndpoints.put(serviceName + "." + endpointName, axisService);

//Cleanup(Finalize), RemoveallServices.put(serviceName, axisService);

Axis2 1.4.1 fix

• Fix the bug AXIS2-3870

//AsixConiguration.removeServiceGroup

//removes the endpoints to this service String serviceName = axisService.getName(); String key = null; for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){ key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key); }

Axis2 1.4.1 memory leak

• Reproduce

for( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request); stub.cleanup(); }

Programmer: 1.4 has memory leak issue, so call cleanup

Axis2 1.4.1 memory leak

• Cause : Stub Finalize (no change)

//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}

public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName());}

//AsixConigurationpublic synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }

Axis2 1.4.1 memory leak

• Cause: Client Finalize (no change)

//ServiceClientprotected void finalize() throws Throwable { super.finalize(); cleanup();}

public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); …}

Axis2 1.4.1 memory leak

• Cause: Client Finalize (no change)

//AsixConigurationpublic AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices();

while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName());

… for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){ key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key); } …

}

…}

Servce are already removed from ServiceGroup in Stub cleanupThe while loop would never enter

Axis2 1.4.1 memory leak

• Cause– Stub cleanup and ServiceClient cleanup have

dependency– Can not call in below order

• Two are two memory leak bugs in 1.4, 1.4.1 only fix 1 bug

Stub.cleanup ServiceClient.cleanup

Axis2 1.5(.6) fix

• Fix the bug AXIS2-4007 AXIS2-4163

//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}

public void cleanup() throws AxisFault { // _service.getAxisConfiguration().removeService(_service.getName()); _serviceClient.cleanup();}

Implement issue

• Forget to cleanup– Container object: add only, no remove– Resource object: apply only, no return/close

• Cleanup dependency– One object cleanup depend on other objects– Two object cleanup/two method has order

dependency

Design Issue

• AxisConfiguration is a global shared object– Usually only 1 instance even in client side.

• Purpose for put service/endpoint map in this global object AxisConfiguration ?

Design Issue

• Message Dispatch in server side // RequestURIBasedServiceDispatcher public AxisService findService(MessageContext messageContext) throws AxisFault {

AxisConfiguration registry = configurationContext.getAxisConfiguration();

AxisService axisService = registry.getService(values[0]);

// If the axisService is not null we get the binding that the request came to add // add it as a property to the messageContext if (axisService != null) { Map endpoints = axisService.getEndpoints(); if (endpoints != null) { if (endpoints.size() == 1) { messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(axisService.getEndpointName())); } else { String endpointName = values[0].substring(values[0].indexOf(".") + 1); messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(endpointName)); } } }

return axisService; }

Design Issue

• Service is ‘singleton’ in server side // AxisConfiguration public AxisService getService(String name) throws AxisFault { AxisService axisService = (AxisService) allServices.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } else { axisService = (AxisService) allEndpoints.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } } return null; }

Design Issue

• Client side– (Usually) New instance every method invocation– No need for message routing• Why still register in Axi2Configuration ?

axisConfig = configContext.getAxisConfiguration(); if (axisService == null) { axisService = createAnonymousService(); } this.axisService = axisService; if (axisConfig.getService(axisService.getName()) == null) { axisService.setClientSide(true); axisConfig.addService(axisService); } else { … }

…