Glassfish and its Memory Leaks
Glassfish is an widely used application server with full Java EE 6 support. One of the website's statements is:
- 'The best open source application server in the industry today.'
Well, decide for yourself, if that slogan is true.
Disclaimer
Note that bugs get fixed time by time and this page is not updated frequently.
More bugs filed
#Update-20140801
The matter here is the memory leak in glassfish on ear undeployment
If you are developing applications with glassfish you know what I'm talking about.
Have a look to the loaded classes after server start (having the ear deployed) and five successive deployments:
You can clearly see the increasing count of loaded classes and how deployments are slowly eat your perm gen space.
Preview of the result
Tracking down the memory leaks
Once the glassfish is fixed, it looks like this:
Tracking down the memory leaks
The plural is correct, it's more than one issue to find and eliminate. But where do you start? First you need some tools. Here is what I used:
- jhat and jmap, provided with the JDK
- VisualVM
jhat und jmap
jhat and jmap are command line tools where jhat record a heap snapshot into a file
jhat -J-Xmx1024m data.leak
and jmap starts a web server so you can browse the heap snapshot.
jmap -dump:format=b,file=data.leak
VisualVM
VisualVM is a gui based tool which also allows to take heap snapshots and explore these snapshots.
What to search?
You quickly find that there is always one instance of an EarClassLoader for each deployed ear. So the scenario is
- start the server with the no deployed ear.
- deploy the ear
- undeploy the ear
- force garbage collection
- search hard references to the dead EarClassLoader instance.
In VisualVM you will use the context menu Show Nearest GC Root. But the this will only find the first path. Since there are hard references also paths via weak references will show up. So you have to deal with them too.
There is a Dead Cow
You found a hard reference? Remove it somehow:
- The best way would be to acquire all sources, but it's a hurdle.
- If you can't find the sources, translating the class back into readable code might work too. Jad might still work.
- A quicker way is to write add code to a ContextListener.contextDestroyed() method and use introspection to poke some values to null.
I mainly used the last method, because it's the fastest way.
Once you've found a hard reference where none should be, set it to null.
Or if weak references are showing up, set them to null and mark it with a comment.
Once you've managed it that everything is freed, you comment out the code for the weak references.
The Dead Cows in Glassfish
Here is what I found
- http://java.net/jira/browse/GLASSFISH-17468 (already fixed)
private void fixGlassfishBug17468() {
try {
// find the context config and flush the error handler.
final Object digester =
getMember(null, Class.forName("org.apache.catalina.startup.ContextConfig"), "contextDigester");
if (digester != null) {
setMember(digester, "errorHandler", null);
}
} catch (Exception ex) {
LOG.error(ex);
}
}
- http://java.net/jira/browse/WSIT-1655
private void fixGlassfishBugWSIT1655(final Set<Object> myUrlSet, final ClassLoader myClassLoader) {
try {
// clean BaseAuthConfigFactory
final Map<?, ?> provider2IdsMap =
(Map<?, ?>) getMember(null, Class.forName("com.sun.jaspic.config.factory.BaseAuthConfigFactory"),
"provider2IdsMap");
for (final Object acp : provider2IdsMap.keySet()) {
try {
final Map<?, ?> serverConfigMap = (Map<?, ?>) getMember(acp, "serverConfigMap");
for (Iterator<?> i = serverConfigMap.entrySet().iterator(); i.hasNext();) {
final Entry<?, ?> e = (Entry<?, ?>) i.next();
try {
final ClassLoader cl =
(ClassLoader) getMember(e.getValue(),
"callbackHandler.handler.handlerContext.this$0.seiModel.classLoader");
if (cl != null && refersToMyEar(cl, myClassLoader, myUrlSet)) {
i.remove();
continue;
}
} catch (Exception ex2) {
ex2.toString();
}
try {
final Class<?>[] classes =
(Class<?>[]) getMember(e.getValue(),
"callbackHandler.handler.handlerContext.this$0.seiModel.jaxbContext.classes");
if (classes != null)
for (Class<?> c : classes) {
if (refersToMyEar(c.getClassLoader(), myClassLoader, myUrlSet)) {
i.remove();
break;
}
}
} catch (Exception ex2) {
}
}
} catch (Exception ex) {
}
}
} catch (Throwable ex) {
LOG.error(ex, ex);
}
}
- http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7199818 (fixed in Java 8)
private void fixJavaBug7199818(final Set<Object> myUrlSet, final ClassLoader myClassLoader) {
try {
final Class<?> configuration = Class.forName("javax.security.auth.login.Configuration");
ClassLoader cl = (ClassLoader) getMember(null, configuration, "contextClassLoader");
if (refersToMyEar(cl, myClassLoader, myUrlSet)) {
do {
cl = cl.getParent();
} while (refersToMyEar(cl, myClassLoader, myUrlSet));
setMember(null, configuration, "contextClassLoader", cl);
}
} catch (Exception ex) {
LOG.error(ex, ex);
}
}
- http://java.net/jira/browse/GLASSFISH-19096
// no code for the ContextListener - see ticket
- more bugs to file... hang on.
Update-20140801
Here are some more bugs which need a fix:
- https://java.net/jira/browse/GLASSFISH-21152
The ActionReporter keeps references to the deployment actions which keep a reference to the deployed application... bye bye memory.
You might try my ugly fix using SoftReferences. It' working for me, maybe also for you?
Here is the source ActionReporter.java and here is a precompiled class /download/ActionReporter.class to patch the kernel.jar.
- https://java.net/jira/browse/GLASSFISH-21112
Woot! Now only the system class path is used to locate the ORB. No way to work in an OSGI environment?
Well, let's build our own singleton:
private final static String arg[] = {};
private static ORB THEORB;
public synchronized static ORB getORB() {
if (THEORB == null) {
try {
THEORB = ORB.init();
} catch (RuntimeException ex) {
// fix for Java 8
try {
Method initJava8 = ORB.class.getDeclaredMethod("init", arg.getClass(), Properties.class);
THEORB = (ORB) initJava8.invoke(null, null, null);
} catch (Exception ex2) {
throw new RuntimeException(ex);
}
}
}
return THEORB;
}
}
and then replace all 'ORB.init()' calls with 'getORB()' in
* TxIORInterceptor
* PropagationContextHelper
* CoordinatorHelper
* ...
maybe it's easier to just patch the class org.omg.CORBA.ORB in the rt.jar? Source is provided and here is the modified method:
...
private final static String NOARGS[] = {};
...
public static synchronized ORB init() {
if (singleton == null) {
String className = getSystemProperty(ORBSingletonClassKey);
if (className == null)
className = getPropertyFromFile(ORBSingletonClassKey);
if ((className == null) ||
(className.equals("com.sun.corba.se.impl.orb.ORBSingleton"))) {
singleton = new com.sun.corba.se.impl.orb.ORBSingleton();
} else {
try {
singleton = create_impl_with_systemclassloader(className);
} catch (RuntimeException re) {
singleton = init(NOARGS, null);
}
}
}
return singleton;
}
the helper methods getMember() and setMember()
/**
* Helper function to set a member of the given object.
*
* @param reference
* the object to search for member
* @param memberName
* the name of the member
* @param value
* the value to set
* @throws NoSuchFieldException
* @throws IllegalAccessException
*/
public void setMember(final Object reference, String memberName, Object value) throws NoSuchFieldException,
IllegalAccessException {
setMember(reference, reference.getClass(), memberName, value);
}
private void setMember(Object reference, Class<? extends Object> clazz, String memberName, Object value)
throws NoSuchFieldException, SecurityException, IllegalArgumentException, IllegalAccessException {
try {
final Field field = clazz.getDeclaredField(memberName);
field.setAccessible(true);
field.set(reference, value);
} catch (NoSuchFieldException nsfe) {
if (clazz.getSuperclass() == null)
throw nsfe;
setMember(reference, clazz.getSuperclass(), memberName, value);
}
}
/**
* Helper function to get a member of the given object.
*
* @param reference
* the object to search for member
* @param memberName
* the name of the member
* @return the object
* @throws NoSuchFieldException
* @throws IllegalAccessException
*/
public static Object getMember(Object reference, String memberName) {
try {
for (StringTokenizer st = new StringTokenizer(memberName, "."); st.hasMoreTokens();) {
reference = getMember(reference, reference.getClass(), st.nextToken());
}
return reference;
} catch (Exception ex) {
}
return null;
}
private static Object getMember(Object reference, Class<?> clazz, String memberName) throws NoSuchFieldException,
IllegalAccessException {
try {
final Field memberField = clazz.getDeclaredField(memberName);
memberField.setAccessible(true);
final Object object = memberField.get(reference);
return object;
} catch (NoSuchFieldException nsfe) {
if (clazz.getSuperclass() == null)
throw nsfe;
return getMember(reference, clazz.getSuperclass(), memberName);
}
}
Your job is now
Visit all open bugs and vote for them. This helps to get them fixed!
rev: 1.17