OutOfMemoryError email alert

When you profiling an application or looking for some memory leak taking heapdump is always essential. There are multiple ways how to do it manually. For instance using directly jmap command or some monitoring tool like VisualVM, JConsole, JMC etc. However when bad things happen in production it’s useful to be alerted and have a heapdump stored. For a such scenario JDK offers several useful command line options:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath
-XX:OnOutOfMemoryError

which could be used in a following manner:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdumps -XX:OnOutOfMemoryError=/opt/app/send_alert_email.sh

I think above configuration is self explaining. When OutOfMemoryError occurs, JDK will automatically save heapdump of the application into /tmp/heapdumps folder. Note that, if the heapdumps directory doesn’t exist it won’t be created and the JVM stores dump into a new file called heapdumps instead. Because the file gets overridden every time the application throws java.lang.OutOfMemoryError you should create the target directory manually otherwise the dump from previous application startup would be last forever. Moreover, shell script send_alert_email.sh is called, that actually sends an email notification.

Below is a notification script I normally use for this purpose. It sends an empty email with subject 'Java App - OutOfMemory alert!' to email recipient@domain.com using pre-configured SMTP server with authentication.

#!/bin/sh
mailx -v -s 'Java App - OutOfMemory alert!' -S smtp=10.0.0.1:25 -S from=app@domain.com -S smtp-auth=login -S smtp-auth-user="smtp-user-name" -S smtp-auth-password="smtp-user-password" recipient@domain.com < /dev/null
echo "Alert email sent"

Script must be executable by an user the java application is running under. The above configuration is easy to test using some memory-leaking application. You can use this simple one:

public class OutOfMemoryTest {
    public static void main(String[] args) {
        StringBuilder sb = new StringBuilder();
        while (true) {
            sb.append(new String("test"));
        }
    }
}

Save the class into OutOfMemoryTest.java file, create /tmp/heapdumps folder, compile class and run built test application:

$ javac OutOfMemoryTest.java

$ java -Xmx5m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdumps -XX:OnOutOfMemoryError=./send_alert_email.sh -cp . OutOfMemoryTest
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/heapdumps/java_pid25781.hprof ...
Heap dump file created [3101545 bytes in 0.023 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="./send_alert_email.sh"
#   Executing /bin/sh -c "./send_alert_email.sh"...
Alert email sent
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2367)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
	at java.lang.StringBuilder.append(StringBuilder.java:132)
	at OutOfMemoryTest.main(OutOfMemoryTest.java:6)