JMX monitoring for JBoss AS 7 with Munin

Continuing the last story how to setup the basic monitoring cases for JBoss AS 7 with Munin, I took the next step to enhance the monitoring in Munin by reading JMX values from the server. If you have setup and configured JBoss AS 7 correctly, you can access JMX with the jconsole application; but JBoss AS 7 comes with a different JMX transport, so as the first step, make sure you start $JBOSS_HOME/bin/jconsole.sh.

Choose remote process and enter service:jmx:remoting-jmx://HOST:PORT, where HOST is the hostname or IP of your server, port defaults to 9999. As username and password you need to enter the values you have configured in mgmt-users.properties.

With this setup working, I had a look on the JMX plugin in Munin, and discovered it needs to be enhanced to support the different protocol JBoss has introduced for JMX. Currently it only supports the MI/JNDI based access. Before going into details what I changed, and how you can get the changes, here is the final setup:

In /etc/munin/plugin-conf.d/munin-node we will add a new chapter for JMX:


[jmx_*]
env.ip HOST
env.port PORT
env.username USER
env.password PASS

Make sure to replace the constants with your values. Additionally you may need to install an additional package for the java support in munin (on Fedora / Centos you need to yum install munin-java-plugins).

In /etc/munin/plugins/ the jmx_ executable needs to be linked to a munin probe.


ln -s /usr/share/munin/plugins/jmx_ jmx_Threads

Now you can test it with munin-run jmx_Threads -d and will see that you can’t connect to the JXM server in JBoss AS 7. As said before the munin plugin can not yet handle the jmx-remote protocol.

Looking add the code in the JMX plugin, it uses the given IP and port to build the URL to connect to. We need to override this with our own url, so we need to have one new parameter, which replaces the ip and port: env.url takes the full URL to the JMX port. And because the jmx-remoting protocol needs additionally libraries – this is why there is an extra startup script for jconsole in the JBoss distribution – we need a second parameter env.classpath to add JARs to the classpath.


[jmx_*]
env.url service:jmx:remoting-jmx://HOST:PORT
env.port PORT
env.username USER
env.password PASS
env.classpath jboss-logging.jar:jboss-marshalling.jar:jboss-marshalling-river.jar:jboss-remoting.jar:jboss-remoting-jmx.jar:xnio-api.jar:xnio-nio.jar

I’ve forked the munin codebase to https://github.com/kifj/munin, in plugins/javalib and plugins/node.d.java you’ll find the changed files.

As a result the jmx_ script and the munin-jmx-plugins.jar are handling the additional parameters. You can build it yourself or download the 2 files from here: jmx_ and munin-jmx-plugins.jar. These need to replace the original files in /usr/share/munin/plugins/jmx_ and /usr/share/java/munin-jmx-plugins.jar. The libraries from env.classpath need to copied or symlinked to /usr/share/java too.

Tagged : , , , ,

6 Responses to “JMX monitoring for JBoss AS 7 with Munin”

  1. Dennis Moul Says:
    March 2, 2013 at 12:54 am

    Thanks for this, it’s a huge help. I had a nice monitoring setup working for JBoss 6, which got entirely broken with the changes in JBoss 7. Your updated jmx_ plugin has been great for restoring monitoring with the new version.

    A couple tips for novices:
    This plugin has a “multigraph” mode, where it grabs a whole bunch of metrics in a single invocation instead of you having to symlink many different times (once per metric) in the traditional munin way. The multigraph approach is way more efficient (one java invocation/startup overhead instead of dozens) and also relieves you of having to figure out what all metrics are available. To use this mode, just symlink /etc/munin/plugins/jmx_MultigraphAll to /usr/share/munin/plugins/jmx_.

    Joe mentions that you have to symlink some libraries into /usr/share/java; where those libraries are located is not intuitively obvious in JBoss7. Using find/locate commands I found them all, these are the links that seemed to work for me:

    cd /usr/share/java
    ln -s /opt/jboss/modules/org/jboss/logging/main/jboss-logging-3.1.0.GA.jar jboss-logging.jar
    ln -s /opt/jboss/modules/org/jboss/marshalling/main/jboss-marshalling-1.3.11.GA.jar jboss-marshalling.jar
    ln -s /opt/jboss/modules/org/jboss/marshalling/river/main/jboss-marshalling-river-1.3.11.GA.jar jboss-marshalling-river.jar
    ln -s /opt/jboss/modules/org/jboss/remoting3/main/jboss-remoting-3.2.3.GA.jar jboss-remoting.jar
    ln -s /opt/jboss/modules/org/jboss/remoting3/remoting-jmx/main/remoting-jmx-1.0.2.Final.jar jboss-remoting-jmx.jar
    ln -s /opt/jboss/modules/org/jboss/xnio/main/xnio-api-3.0.3.GA.jar xnio-api.jar
    ln -s /opt/jboss/modules/org/jboss/xnio/nio/main/xnio-nio-3.0.3.GA.jar xnio-nio.jar

    • Thanks for providing the locations of the jar files.

      For the problem with the stopped JBoss server, you are right – this needs to be fixed at the Java code.
      Running munin-run jmx_Threads yields:


      Failed to prepare values for class class org.munin.plugin.jmx.MemoryAllocatedNonHeap: Operation failed with status WAITING
      java.lang.RuntimeException: Operation failed with status WAITING
      at org.jboss.remotingjmx.RemotingConnector.connect(RemotingConnector.java:154)
      at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
      at org.munin.plugin.jmx.BasicMBeanConnection.get(BasicMBeanConnection.java:19)

      This needs to be caught and handled.

  2. Dennis Moul Says:
    March 2, 2013 at 12:59 am

    Another tip: in multigraph mode I got three graphs, not showing much detail. Took me a while to find the rest of the detailed graphs – they are nested UNDERNEATH the three parent JMX graphs. So just click on the “Allocated Memory” graph and you get a new page with more than a dozen additional graphs; likewise the other two top-level JMX graphs have some additional detail when clicked upon.

  3. Dennis Moul Says:
    March 2, 2013 at 1:20 am

    I noticed today that when JBoss is not running, the jmx_* graphs disappear from the munin node’s graph page altogether… this can be problematic, since then I can’t see the behavior leading up to a crash/fail until JBoss is restarted (or I go digging manually for the graphs in the filesystem).

    Most munin plugins that I’ve worked with just have blank sections in their graphs for the time periods in which their target is not available. I’m wondering what would be involved in getting the jmx_ plugin to “fail gracefully” in this way, so that the graphs are all still there but have blank sections when JBoss is down.

    I took a look at the perl code of jmx_ itself to see if there was something I could fix, but it seems mostly a wrapper around the Java code in munin-jmx-plugins.jar, and digging into that is a bit outside my skill set.

    Any thoughts on improving the handling of JBoss-down situations? Doesn’t happen a lot in production, but is pretty common in dev/test environments where you may be monitoring still for performance/QA reasons.

  4. I’ve fixed several issues with the plugin when the JMX connection could not be established – the code is up to date on github and you can download an updated jar file here: http://johannes-beck.name/binary/munin-jmx-plugins.jar

    • Thanks for this article.

      With your new munin-jmx-plugins.jar we still get the error you mentioned above on a running Jboss:

      Failed to prepare values for class class org.munin.plugin.jmx.MemoryAllocatedNonHeap: Operation failed with status WAITING
      java.lang.RuntimeException: Operation failed with status WAITING

Leave a Reply to joe