Nullpointer exception in Priam.Cassandra not starting up.
Closed this issue · 5 comments
Good morning,
I am struggling to get Priam working with Cassandra and after hacking at it for two past two weeks, am finally reaching out to you all with hopes that you might be able to help.
I have the following installed:
- Java 7 (openJDK)
- Cassandra 2.0.17
- Priam 2.x (says 2.8.0-SNAPSHOT)
I followed the Wiki to a tee but keep seeing the following with no other relevant information in any log files:
cassand+ 18663 18621 0 03:26 ? 00:00:00 grep -q Error: Exception thrown by the agent : java.lang.NullPointerException
We then attached a java debugger (jdb) to get to the source of the error and this what we see:
Cassandra running in the debugger, and discovered it's stuck in
com.netflix.priam.cassandra.extensions.PriamStartupAgent.setPriamProperties (PriamStartupAgent.java:75)
It tries to load configuration with the following code, and then if it fails, it sleeps for 5 seconds and tries again.
token = DataFetcher.fetchData("http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_token");
seeds = DataFetcher.fetchData("http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_seeds");
isReplace = Boolean.parseBoolean(DataFetcher.fetchData("http://127.0.0.1:8080/Priam/REST/v1/cassconfig/is_replace_token"));
replacedIp = DataFetcher.fetchData("http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_replaced_ip");
extraEnvParams = DataFetcher.fetchData("http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_extra_env_params");
Obviously, it wasn't able to load the config via the REST API running in Tomcat. Tomcat had been shutdown, so I restarted it. The curl command (using the first "get_token" URL above) then got stuck waiting. Checked catalina.out log file from tomcat, and discovered this line repeated over and over again:
INFO 04:44:10,453 Fail wrting cassandra.yml file. Retry again!
INFO 04:44:10,453 Fail wrting cassandra.yml file. Retry again!
INFO 04:44:10,453 Fail wrting cassandra.yml file. Retry again!
We've been running tomcat as root, and also set up a link from "cassnadra.yml" to "cassandra.yaml":
$ ip-xx-xx-xx-xx:~$ ls -l /etc/cassandra/
total 92
-rw-r--r-- 1 root root 10837 May 8 03:09 cassandra-env.sh
-rw-r--r-- 1 root root 1199 Sep 18 2015 cassandra-rackdc.properties
-rw-r--r-- 1 root root 1358 Sep 18 2015 cassandra-topology.properties
-rw-r--r-- 1 root root 916 Sep 18 2015 cassandra-topology.yaml
-rw-r--r-- 1 root root 33360 Sep 18 2015 cassandra.yaml
lrwxrwxrwx 1 root root 14 May 9 14:41 cassandra.yml -> cassandra.yaml
-rw-r--r-- 1 root root 2374 Sep 18 2015 commitlog_archiving.properties
-rw-r--r-- 1 root root 1850 Sep 18 2015 log4j-server.properties
-rw-r--r-- 1 root root 1228 Sep 18 2015 log4j-tools.properties
drwxr-xr-x 2 root root 4096 May 8 02:28 triggers
Turns out Priam was looking for the yaml file in /mnt/cassandra/conf for some reason, instead of /etc/cassandra.yaml. Rather than trying to track down where in the world it got that value, we simply created a symbolic link with that path and pointed it to /etc/cassandra.yaml.
main[1] stop at com.netflix.priam.utils.TuneCassandra:50
Deferring breakpoint com.netflix.priam.utils.TuneCassandra:50.
It will be set after the class is loaded.
main[1] cont
Set deferred breakpoint com.netflix.priam.utils.TuneCassandra:50
Breakpoint hit: "thread=localhost-startStop-1", com.netflix.priam.utils.TuneCassandra.execute(), line=50 bci=6
50 tuner.writeAllProperties(config.getYamlLocation(), null, config.getSeedProviderName());
localhost-startStop-1[1] print config.getYamlLocation()
config.getYamlLocation() = "/mnt/cassandra/conf/cassandra.yaml"
localhost-startStop-1[1] print config.getSeedProviderName()
config.getSeedProviderName() = "com.netflix.priam.cassandra.extensions.NFSeedProvider"
localhost-startStop-1[1] cont
That prevented the error message, but still not able to use the REST API.
ubuntu@ip-xx-xx-xx-xx:~$ curl http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_token
<title>Apache Tomcat/8.0.33 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style>HTTP Status 404 - /Priam/REST/v1/cassconfig/get_token
type Status report
message /Priam/REST/v1/cassconfig/get_token
description The requested resource is not available.
Apache Tomcat/8.0.33
There is no way that the install instructions for Priam are correct, as I have shown by all the issues that it is running into. Why is Netflix open sourcing software without correct install instructions?
Is anyone able to help with this?
To answer some of your questions:
- "Turns out Priam was looking for the yaml file in /mnt/cassandra/conf for some reason, instead of /etc/cassandra.yaml. Rather than trying to track down where in the world it got that value,..." --- Priam fetches it from properties (see PriamConfiguration.getYamlLocation()).
- /Priam/REST/v1/cassconfig/get_token 404 -- this could be a myriad of reasaons. 404 usually means that Tomcat is rejecting your request, possibly due to incorrect web.xml of your Tomcat instance. Is any http request working to Priam?
@calicartel any other issues, if not, I will close this ticket.