oracle/nosql-examples

Facing service unavailable error while implementing load balanced architecture

Akshaykale84 opened this issue · 6 comments

Akshaykale84 commented

KV Version : 22.2.13
Node driver Version : 5.3.4

I am using Oracle NoSQL Database for one of the project. I have configured the proxy server and the communication between DB and Application is established.

 

I created 4 instances of proxy to enable load balanced architecture and also configured the HAproxy to distribute load to the four proxy servers.

 

When I am trying to make NoSQL client with nodeJS driver it is throwing an error message stating "Service Unavailable"(HTTP_RESPONSE_CODE 503).

 

Kindly help in this regard.

 

And also suggest if there is any other way to implement load balanced architecture in Oracle NoSQL Database (on-prem).

dario-vega commented

I will review it tomorrow and provide you instructions to setup HA-proxy
@Akshaykale84 Is it a secure or non-secure NoSQL cluster?

Akshaykale84 commented

I will review it tomorrow and provide you instructions to setup HA-proxy
@Akshaykale84 Is it a secure or non-secure NoSQL cluster?

Sure.

It's non-secured NoSQL cluster

dario-vega commented

Node.js application connection

        return new NoSQLClient({
            serviceType: ServiceType.KVSTORE,
            endpoint: process.env.NOSQL_ENDPOINT + ":" + process.env.NOSQL_PORT
	});

Deploy your cluster and start the HTTP proxy in the nodes.

Note: The Oracle NoSQL Database Proxy can run in one or multiple dedicated hosts. It can be hosted inside the nodes of the NoSQL Cluster.

e.g node1-nosql, node2-nosql, node3-nosql in my case, I am using the port 8080.

java -jar $KVHOME/lib/httpproxy.jar -helperHosts $KVHOST:5000 -storeName $KVSTORE -httpPort 8080 -verbose true

Test using curl for all nodes hosting a HTTP proxy node1-nosql, node2-nosql, node3-nosql

curl -v http://node1-nosql:8080

Test using your Oracle NoSQL SDK for Node.js example for all nodes hosting a HTTP proxy node1-nosql, node2-nosql, node3-nosql

export NOSQL_ENDPOINT=node1-nosql
export NOSQL_PORT=8080 
node test.js

Configuring the Load Balancer

You can use a load balancer as frontend which has a backend set of multiple NoSQL proxies on different hosts. In this case, you can setup HA proxy.

Install haproxy - in an external node to run the Load Balancer
e.g nosql-lb

sudo yum install haproxy

Add the following lines at the end of the file /etc/haproxy/haproxy.cfg

in my case, node1-nosql, node2-nosql, node3-nosql and using the port 8080 for all nodes including the load balancer

# Configure HAProxy to listen on port 80
frontend http_front
   bind *:8080
   stats uri /haproxy?stats
   default_backend http_back

# Configure HAProxy to route requests to Oracle NoSQL Database Proxy nodes on port 8080
backend http_back
   balance roundrobin
   server node1-nosql 10.0.0.56:8080 check
   server node2-nosql 10.0.0.167:8080 check
   server node3-nosql 10.0.0.143:8080 check

Restart HA proxy and validate the status

sudo systemctl stop haproxy.service
sudo systemctl start haproxy.service
sudo systemctl status haproxy.service
$ sudo systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-02-24 10:25:52 GMT; 2min 3s ago
  Process: 2205764 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $CFGDIR -c -q $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2205766 (haproxy)
    Tasks: 2 (limit: 100047)
   Memory: 4.5M
   CGroup: /system.slice/haproxy.service
           ├─2205766 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid
           └─2205769 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid

Test using curl

curl -v http://nosql-lb:8080

Test using your Oracle NoSQL SDK for Node.js example

export NOSQL_ENDPOINT=nosql-lb
export NOSQL_PORT=8080 
node test.js

Troubleshooting

The following error happens if:

  • the load balancer cannot contact any of the IP addresses in backend http_back - you can use fqdn instead of IP addresses
  • all the HTTP proxy are down
NoSQLServiceError: [SERVICE_ERROR] Unsuccessful HTTP response.  Status code: 503. Status message: Service Unavailable
    at HttpClient._handleResponse (/home/opc/demo-graphql-nosql/blogloader/node_modules/oracle-nosqldb/lib/http_client.js:93:33)
    at IncomingMessage.<anonymous> (/home/opc/demo-graphql-nosql/blogloader/node_modules/oracle-nosqldb/lib/http_client.js:150:22)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Let me know if it works, Otherwise, please provide the output for the command documented here

  • 👍1
Akshaykale84 commented

Thanks @dario-vega for the detailed explanation.

I tried the same on my environment.

It's working till creating the client of DB.

return new NoSQLClient({ serviceType: ServiceType.KVSTORE, endpoint: process.env.NOSQL_ENDPOINT + ":" + process.env.NOSQL_PORT 	});

But when I try to put some data in the table using the client facing the following error:

client.put('tableName', {id:10, name:'akshay'}).then(res => console.log('inserted'));

Error Msg:

'[REQUEST_TIMEOUT] Operation timed out after 5000 ms and 5 retries; Caused by: [SERVI CE_ERROR] Unsuccessful HTTP response. Status code: 503. Status message: Service Unavailable'

The same code is working when I change the endpoint from load balancer host to proxy host.

Please assist here..

dario-vega commented

Provide the HA proxy version used.

haproxy -v

Here is an extract of my configuration file

# Configure HAProxy to route requests to Oracle NoSQL Database Proxy nodes on port 8080
backend http_back
   balance roundrobin
   server node1-nosql 10.0.0.56:8080 check
   server node2-nosql 10.0.0.167:8080 check
   server node3-nosql 10.0.0.143:8080 check

Please provide the output for the following commands (use your IPs and ports) for each line with the server directive.

ping 10.0.0.56
ping 10.0.0.167
ping 10.0.0.143
curl -v http://10.0.0.56:8080
curl -v http://10.0.0.167:8080
curl -v http://10.0.0.143:8080

You can also use DNS entries.

# Configure HAProxy to route requests to Oracle NoSQL Database Proxy nodes on port 8080
backend http_back
   balance roundrobin
   server node1-nosql node1-nosql:8080 check
   server node2-nosql node2-nosql:8080 check
   server node3-nosql node3-nosql:8080 check

FYI, my test code is creating a table and inserting data, not just making a connection

I am using

proxyVersion=5.9.1
kvclientVersion=22.2.13
Akshaykale84 commented

I restarted the haproxy and proxy instances and now it's working properly.

Thanks @dario-vega for your help..

  • 👍1