JMX Heap monitoring issues
yuvarajan1989k opened this issue · 15 comments
Hi,
I received the below error.
Nov 24 06:41:00 *** collectd[31721]: cjni_init_native: Cannot find the API class "org.collectd.api.Collectd". Please set the correct class path using 'JVMArg "-Djava.class.path=..."'.
Nov 24 06:41:00 ** collectd[31721]: java plugin: cjni_create_jvm: cjni_init_native failed.
Nov 24 06:41:00 ** collectd[31721]: java plugin: cjni_thread_attach: cjni_create_jvm failed.
Nov 24 06:41:00 ** collectd[31721]: java plugin: Configuration block for `GenericJMX' found, but no such configuration callback has been registered. Please make sure, the `LoadPlugin' lines precede the `Plugin' blocks.
JMX is configured and listening in the port 1097.
ServiceURL "service:jmx:rmi:///jndi/rmi://localhost:1097/jmxrmi"
Java version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
@yuvarajan1989k - Can you provide your Collectd configuration and confirm the LoadPlugin lines precede the Plugin blocks?
Thanks for your help.
@yuvarajan1989k, what version of the Stackdriver monitoring agent package are you running? On what OS?
Can you please also list the contents of the /opt/stackdriver/collectd/share/collectd/java directory on your system?
Hi,
OS Version:- Ubuntu 18.04
stackdriver-agent: 5.5.2-1001.bionic
ls -tlr /opt/stackdriver/collectd/share/collectd/java
-rw-r--r-- 1 root root 16759 Nov 14 11:08 generic-jmx.jar
-rw-r--r-- 1 root root 13737 Nov 14 11:08 collectd-api.jar
Interesting. The files are right there, and added to the classpath, but not visible to the JVM.
Is the agent running at root?
Are you seeing any errors in the logs prior to "Cannot find the API class"? E.g., "class file has the wrong version" or "bad class file"?
Stackdriver running as root
root 19400 1 0 Nov26 ? 00:00:30 /opt/stackdriver/collectd/sbin/stackdriver-collectd -C /etc/stackdriver/collectd.conf -P /var/run/stackdriver-agent.pid
FYI:-
Nov 26 23:37:31 ** collectd[19393]: plugin_load: plugin "java" successfully loaded.
Nov 26 23:37:31 ** collectd[19399]: plugin_load: plugin "java" successfully loaded.
Nov 26 23:37:31 ** collectd[19400]: cjni_init_native: Cannot find the API class "org.collectd.api.Collectd". Please set the correct class path using 'JVMArg "-Djava.class.path=..."'.
Nov 26 23:37:31 ** collectd[19400]: java plugin: cjni_create_jvm: cjni_init_native failed.
Nov 26 23:37:31 ** collectd[19400]: java plugin: cjni_thread_attach: cjni_create_jvm failed.
Nov 26 23:37:31 ** collectd[19400]: java plugin: Configuration block for `GenericJMX' found, but no such configuration callback has been registered. Please make sure, the `LoadPlugin' lines precede the `Plugin' blocks.
JAVA VERSION:-
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
@sophieyfang, you were just (successfully) testing JVM monitoring on Bionic, right? Can you please compare your environment with the above? Were you also using JRE 1.8? I think you may have tested with the full JDK, though it shouldn't affect the ability to load the plugin classes. Any other discrepancies?
I just tested it again on ubuntu1804 with jdk8 and jdk11. Previously, i was only testing it on jdk11 as it's latest one supported on ubuntu1804. jdk 11 has no issues and i can see the jvm metrics flow in. However, jdk8 does give the below error:
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8991]: plugin_load: plugin "target_set" successfully loaded.
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8991]: plugin_load: plugin "target_replace" successfully loaded.
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 stackdriver-agent[8951]: ...done.
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 systemd[1]: Started LSB: start and stop Stackdriver Agent.
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8992]: cjni_init_native: Cannot find the API class "org.collectd.api.Collectd". Please set the correct class path using 'JVMArg "-Djava.class.path=..."
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8992]: java plugin: cjni_create_jvm: cjni_init_native failed.
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8992]: java plugin: cjni_thread_attach: cjni_create_jvm failed.
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8992]: java plugin: Configuration block for `GenericJMX' found, but no such configuration callback has been registered. Please make sure, the `LoadPlug
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8992]: Initialization complete, entering read-loop.
Nov 27 04:17:43 sophieyfang-ubuntu1804-jvm8 collectd[8992]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
@sophieyfang is it possible to run this with java 1.8 ?
@yuvarajan1989k, thanks for the report. We should support all JDK versions available from the default repos on a given distro (presumably including JDK 8 on Bionic). As you can see, we've reproduced your issue, and will work on fixing it.
@yuvarajan1989k In the meantime, you could try the following workaround:
- install JDK 11, but don't make it the default.
- add the location of that JDK's
libjvm.soasJAVA_LIB_DIRto/etc/default/stackdriver-agent, e.g.:JAVA_LIB_DIR="/usr/lib/jvm/java-11-openjdk-amd64/jre/lib/amd64/server" - restart the agent
Please let us know if it works for you.
Yes. Above solution has helped me to solve the issues. thank you.
This is actually fixed by Stackdriver/collectd#183 (and Stackdriver/collectd#184 for the older agents). The former will be part of the 6.0.3 release.
The fix has been released in version 6.0.3 of the Stackdriver monitoring agent.