zevenet/zlb

Problem with HTTP farms 5.10.1

Closed this issue · 13 comments

Wan6698 commented

Hello!
After update Zevenet 5.0 or 5.9 to version 5.10.1 Http farms not working correctly
If in backend certificate install trust, load balancing work well, but if I install self-signed certificate in backend node, cert not change from Zevenet and I have error 503.
In new version ZPROXY module is in charge of this, but it not work correctly.
In version 4 and 5, all good.

abdessamad-zevenet commented

Hi,

Yes, zproxy is in charge of the http farms. latest 5.10.1 use the most recent Openssl implementation, Make sure that you are generating certificate correctly and that you have updated zproxy to the latest version in the repositories.

Also, enable debug level, for so, change LogLevel parameter in the farm configuration file (usr/local/zevenet/config/_proxy.cfg ) to value 8 and test with a simple CURL request with the option "-v" to the farm vip and from the loadbalancer itself to the backend server.

regards!

Wan6698 commented

Thanks for answer

Logs after enable

Can you help me?

Apr 1 13:24:23 Zevenet zapi.cgi[26318]: (INFO) webgui :: STATUS: 200 REQUEST: PUT /zapi/v4.0/zapi.cgi/farms/DMZCAS443/actions
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) fd: 21:-1 Client closed connection
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) fd: 23:-1 Client closed connection
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) fd: 21:-1 Client closed connection
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) SSL: TLSv1.3, Not REUSED, Ciphers: TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD#012
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) (7f9f03fff700) could not get SNI host name to (null)
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443,443) 1 [443] GET / HTTP/1.1 [172.16.6.253 (21) -> 172.16.20.83 (-1)]
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443,443,0) Backend closed connection
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443,443,0) (7f9f03fff700) BackEnd 172.16.20.83:443 dead (killed) in farm: 'DMZCAS443', service: '443'
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443,443,0) (7f9f03fff700) e503 no backend GET / HTTP/1.1 from 172.16.6.253
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) fd: 21:-1 Client closed connection
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) SSL: TLSv1.3, REUSED, Ciphers: TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD#012
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) (7f9f03fff700) could not get SNI host name to (null)
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443,443) (7f9f03fff700) e503 no backend GET /robots.txt HTTP/1.1 from 172.16.6.253
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) fd: 21:-1 Client closed connection
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) SSL: TLSv1.3, Not REUSED, Ciphers: TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD#012
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443) (7f9f126fc700) could not get SNI host name to (null)
Apr 1 13:24:30 Zevenet zproxy[26369]: (DMZCAS443,443) (7f9f126fc700) e503 no backend GET /favicon.ico HTTP/1.1 from 172.16.6.253
Apr 1 13:24:33 Zevenet zproxy[26369]: BackEnd 172.16.20.83:443 resurrect in farm: 'DMZCAS443', service: '443'
Apr 1 13:24:33 Zevenet zproxy[26369]: (DMZCAS443) fd: 21:-1 Client closed connection
Apr 1 13:24:33 Zevenet zproxy[26369]: (DMZCAS443) fd: 23:-1 Client closed connection
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) SSL: TLSv1.3, Not REUSED, Ciphers: TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD#012
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) (7f9f03fff700) could not get SNI host name to (null)
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443,443) 2 [443] GET /owa HTTP/1.1 [172.16.6.253 (21) -> 172.16.20.83 (-1)]
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443,443,0) Backend closed connection
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443,443,0) (7f9f03fff700) BackEnd 172.16.20.83:443 dead (killed) in farm: 'DMZCAS443', service: '443'
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443,443,0) (7f9f03fff700) e503 no backend GET /owa HTTP/1.1 from 172.16.6.253
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) fd: 21:-1 Client closed connection
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) SSL: TLSv1.3, REUSED, Ciphers: TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD#012
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) (7f9f037fe700) could not get SNI host name to (null)
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443,443) (7f9f037fe700) e503 no backend GET /robots.txt HTTP/1.1 from 172.16.6.253
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) fd: 21:-1 Client closed connection
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) SSL: TLSv1.3, Not REUSED, Ciphers: TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD#012
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443) (7f9f03fff700) could not get SNI host name to (null)
Apr 1 13:24:34 Zevenet zproxy[26369]: (DMZCAS443,443) (7f9f03fff700) e503 no backend GET /favicon.ico HTTP/1.1 from 172.16.6.253
Apr 1 13:24:42 Zevenet zapi.cgi[26387]: (INFO) webgui :: STATUS: 200 REQUEST: GET /zapi/v4.0/zapi.cgi/system/logs
Apr 1 13:24:43 Zevenet zproxy[26369]: BackEnd 172.16.20.83:443 resurrect in farm: 'DMZCAS443', service: '443'
Apr 1 13:24:48 Zevenet zapi.cgi[26422]: (INFO) webgui :: STATUS: 200 REQUEST: GET /zapi/v4.0/zapi.cgi/system/logs/syslog/lines/50
Apr 1 13:25:01 Zevenet CRON[26459]: (root) CMD (/usr/local/zevenet/bin/zenrrd &>/dev/null)
Apr 1 13:25:16 Zevenet zapi.cgi[26539]: (INFO) webgui :: STATUS: 200 REQUEST: GET /zapi/v4.0/zapi.cgi/system/logs

abdessamad-zevenet commented

Hi,
it seems that the the backend server is not accepting the connection and closing it immediately.

Please, share your farm configuration file (/usr/local/zevenet/config/DMZCAS443_proxy.cfg), to check if its correctly configured.

Also, execute from the load balancer an HTTP request using CURL to the backend server:
$ curl -v -k https://172.16.20.83:443

Wan6698 commented

CFG file

######################################################################
##GLOBAL OPTIONS
User "root"
Group "root"
Name DMZCAS443

allow PUT and DELETE also (by default only GET, POST and HEAD)?:

#ExtendedHTTP 0

Logging: (goes to syslog by default)

0 no logging

1 normal

2 extended

3 Apache-style (common log format)

#LogFacility local5
LogLevel 8

check timeouts:

Timeout 45
ConnTO 20
Alive 10
Client 30
ThreadModel dynamic
Control "/tmp/DMZCAS443_proxy.socket"
DHParams "/usr/local/zevenet/app/zproxy/etc/dh2048.pem"
ECDHCurve "prime256v1"

#HTTP(S) LISTENERS
ListenHTTPS
Err414 "/usr/local/zevenet/config/DMZCAS443_Err414.html"
Err500 "/usr/local/zevenet/config/DMZCAS443_Err500.html"
Err501 "/usr/local/zevenet/config/DMZCAS443_Err501.html"
Err503 "/usr/local/zevenet/config/DMZCAS443_Err503.html"
Address 192.168.130.87
Port 443
xHTTP 4
RewriteLocation 1

    Cert "/usr/local/zevenet/config/fullcert.pem"
    Ciphers "ALL"
    Disable SSLv3
    SSLHonorCipherOrder 1
    #ZWACL-INI

    Service "443"
            ##True##HTTPS-backend##
            #DynScale 1
            #BackendCookie "ZENSESSIONID" "domainname.com" "/" 0
            #HeadRequire "Host: "
            #Url ""
            #Redirect ""
            StrictTransportSecurity 21600000
            #Session
                    #Type nothing
                    #TTL 120
                    #ID "sessionname"
            #End
            #BackEnd

            BackEnd
                    HTTPS
                    Address 172.16.20.83
                    Port 443
                    TimeOut 1
                    Priority 1
            End
            #End
    End
    #ZWACL-END


    #Service "DMZCAS443"
            ##False##HTTPS-backend##
            #DynScale 1
            #BackendCookie "ZENSESSIONID" "domainname.com" "/" 0
            #HeadRequire "Host: "
            #Url ""
            #Redirect ""
            #StrictTransportSecurity 21600000
            #Session
                    #Type nothing
                    #TTL 120
                    #ID "sessionname"
            #End
            #BackEnd

            #End
    #End

End

And

curl -v -k https://172.16.20.83:443

  • Expire in 0 ms for 6 (transfer 0x555b5c3acf50)
  • Trying 172.16.20.83...
  • TCP_NODELAY set
  • Expire in 200 ms for 4 (transfer 0x555b5c3acf50)
  • Connected to 172.16.20.83 (172.16.20.83) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 172.16.20.83:443
  • Closing connection 0
    curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 172.16.20.83:443
abdessamad-zevenet commented

With curl you can verify that the issue is in the backend server side, since it's not allowing ssl connections, make sure sure that the backend server has ssl enabled with a valid certificate.

Notes on the configuration file after fixing the backend server:

  1. remove the "Priority 1", since it's not used with only one backend.
  2. you should consider increasing the TimeOut, depending on your network and your application response time.
  3. comment out the line ECDHCurve "prime256v1", the ecdh curve now it select the proper one automatically.
Wan6698 commented

Certificate on this server is self-signed and I cannot change it now(
I'm going to change certificate for clients who connect to Server by Zevenet
Versions 5.0 and 5.9 max accept 120-130 established connections..(
But I need 300-500 connections by Zevenet.

adanmarin-zevenet commented

Are you still experiencing the issue?

Wan6698 commented
emiliocampos-zevenet commented
Wan6698 commented
emiliocampos-zevenet commented
edhedh commented

Hi,

I'm a newbie to Zevenet community edition.

I have two HTTPS backends: backend1.acme.com and backend2.acme.com
I want my clients to connect to Zevenet frontend.acme.com via HTTPS (https://frontend.acme.com)
Upon receiving https://frontend.acme.com requests I want Zevenet to load balance the requests, using sticky sessions, to https://backend1.acme.com and https://backend2.acme.com in eg round robin mode.

I'm confused about which certificates I need on the Zevenet end and what their contents (common name, SAN names if SAN is supported ?) should be:

  • To allow clients to connect via https to https://frontend.acme.com
  • To allow Zevenet to connect via https to the two https backends.
    Assuming I have no intermediate certificates and all certificates, both the frontend and the backend certificates, are signed by the same root CA.

To configure the backends in Zevenet you have the "Https backends" option.
If my backends, as is the case in my example, are https backends, I guess I need to enable that option ?
If enabled, how can Zevenet build a trusted connection to the backends ?
Do I need a particular certificate on the Zevenet end ? The certificate of the root ca that signed the certificates of the backend servers ? How do I add that root ca certificate to Zevenet and how do I tell Zevenet to use that for connecting to the backends ?

Can you configure the Zevenet Community Edition as SSL passthrough and if yes how ?
Does Zevenet Community Edition support SAN certificates ?

Please advise.
Kr
EDH

emiliocampos-zevenet commented

please don't mix issues as it is not related to the subject.

In case you want help configuring farms please refer to the official documentation or ask in the mail distribution list.

Thanks in advance.