/cluster API response status 400

[mars]

For a Kong 0.6.1 cluster running with two members in a Private Space, at some point the /cluster admin API starts returning HTTP 400.

• Other admin & proxy APIs continue to function correctly.
• Restarting the app heroku restart does not restore functionality.

The following is logged. Unsure if correlated, but I've rarely seen this mutex error until now.

app[web.2]: 2016/03/01 02:37:16 [error] 98#0: [lua] cluster.lua:40: Cassandra error: Could not create lock for prepare request: Error locking mutex: timeout, context: ngx.timer
app[web.1]: 2016/03/01 02:37:16 [error] 98#0: [lua] cluster.lua:40: Cassandra error: Could not create lock for prepare request: Error locking mutex: timeout, context: ngx.timer

May be related to #6

[mars]

That same cluster appears healthy when checked via kong CLI:

~ $ kong cluster members -c $KONG_CONF
[INFO] Using configuration: /app/config/kong.yml
dyno-64714c8b-3f9f-417c-8007-d3c998070a9b_10.0.142.56:7946   10.0.142.56:7946   alive  
dyno-967fef17-e4b4-4de5-b253-3575960fe701_10.0.150.62:7946   10.0.150.62:7946   alive  
dyno-da5dab93-30c2-4a11-ab22-7f1c7ee165c2_10.0.143.137:7946  10.0.143.137:7946  alive

~ $ kong cluster reachability -c $KONG_CONF
[INFO] Using configuration: /app/config/kong.yml
Total members: 3, live members: 3
Starting reachability test...
Successfully contacted all live nodes

That shows 3 nodes: 2 are web dynos; 1 is the CLI session in an interactive dyno

[mars]

Also worth noting: a single-node Kong 0.6.1 responds status 200 as expected.

[mars]

Solved the problem: I was sending an empty JSON object in the body of my /cluster API requests, which caused the status 400 response. Sending an empty body results in successful responses.