elastic/ansible-elasticsearch

ES_JAVA_OPTS overrided by sysconfig

realkcn opened this issue · 6 comments

realkcn commented

Describe the feature:

Elasticsearch version 7.5.0

Role version: (If using master please specify github sha) 7.5.0

JVM version (java -version): openjdk 1.8

OS version (uname -a if on a Unix-like system): Centos 7.7

Description of the problem including expected versus actual behaviour:
es_proxy_host and es_proxy_port defined isn't effective.

Playbook:
Please specify the full playbook used to reproduce this issue.

- hosts: es-master
  roles:
    - role: elastic.elasticsearch
  vars:
    es_version: 7.5.0
    es_add_repository: false
    es_heap_size: "1g"
    es_config:
      node.name: "es-master-{{ inventory_hostname_short }}"
      cluster.name: "es-cluster"
      cluster.initial_master_nodes: "{{ es_master_host }}:9300"
      discovery.seed_hosts: "{{ es_master_host }}:9300"
      http.port: 9200
      node.data: false
      node.master: true
      bootstrap.memory_lock: false
    es_plugins:
     - plugin: ingest-attachment
    es_proxy_host: 10.1.4.2
    es_proxy_port: 3128

- hosts: es-data
  roles:
    - role: elastic.elasticsearch
  vars:
    es_add_repository: false
    es_data_dirs:
      - "/opt/data/elasticsearch"
    es_config:
      node.name: "es-data-{{ inventory_hostname_short }}"
      cluster.name: "es-cluster"
      cluster.initial_master_nodes: "{{ es_master_host }}:9300"
      discovery.seed_hosts: "{{ es_master_host }}:9300"
      http.port: 9200
      node.data: true
      node.master: false
      bootstrap.memory_lock: false
    es_plugins:
      - plugin: ingest-attachment
    es_proxy_host: 10.1.4.2
    es_proxy_port: 3128

Provide logs from Ansible:
2019-12-18 14:37:24,247 p=ansible u=3466 | failed: [k8s-master2] (item={u'plugin': u'ingest-attachment'}) => {"ansible_loop_var": "item", "attempts": 1, "changed": false, "cmd": ["/usr/share/elasticsearch/bin/elasticsearch-plugin", "install", "ingest-attachment", "--batch", "--silent"], "delta": "0:02:09.928312", "end": "2019-12-18 14:37:24.153155", "item": {"plugin": "ingest-attachment"}, "msg": "non-zero return code", "rc": 1, "start": "2019-12-18 14:35:14.224843", "stderr": "Exception in thread "main" java.net.ConnectException: Connection timed out\n\tat java.base/sun.nio.ch.Net.connect0(Native Method)\n\tat java.base/sun.nio.ch.Net.connect(Net.java:493)\n\tat java.base/sun.nio.ch.Net.connect(Net.java:482)\n\tat java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)\n\tat java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339)\n\tat java.base/java.net.Socket.connect(Socket.java:603)\n\tat java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:290)\n\tat java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)\n\tat java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)\n\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)\n\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)\n\tat java.base/sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:265)\n\tat java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)\n\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:177)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)\n\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)\n\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:163)\n\tat java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142)\n\tat org.elasticsearch.plugins.InstallPluginCommand.urlExists(InstallPluginCommand.java:353)\n\tat org.elasticsearch.plugins.InstallPluginCommand.getElasticUrl(InstallPluginCommand.java:315)\n\tat org.elasticsearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:255)\n\tat org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:230)\n\tat org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:217)\n\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)\n\tat org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)\n\tat org.elasticsearch.cli.Command.main(Command.java:90)\n\tat org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)", "stderr_lines": ["Exception in thread "main" java.net.ConnectException: Connection timed out", "\tat java.base/sun.nio.ch.Net.connect0(Native Method)", "\tat java.base/sun.nio.ch.Net.connect(Net.java:493)", "\tat java.base/sun.nio.ch.Net.connect(Net.java:482)", "\tat java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)", "\tat java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339)", "\tat java.base/java.net.Socket.connect(Socket.java:603)", "\tat java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:290)", "\tat java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)", "\tat java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)", "\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)", "\tat java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)", "\tat java.base/sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:265)", "\tat java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)", "\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:177)", "\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)", "\tat java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)", "\tat java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:163)", "\tat java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142)", "\tat org.elasticsearch.plugins.InstallPluginCommand.urlExists(InstallPluginCommand.java:353)", "\tat org.elasticsearch.plugins.InstallPluginCommand.getElasticUrl(InstallPluginCommand.java:315)", "\tat org.elasticsearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:255)", "\tat org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:230)", "\tat org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:217)", "\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)", "\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)", "\tat org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)", "\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)", "\tat org.elasticsearch.cli.Command.main(Command.java:90)", "\tat org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)"], "stdout": "", "stdout_lines": []}

ES Logs if relevant:

realkcn commented

Trace into the elasticsearch-plugin script, I found it caused by sysconfig of elasticsearch. The templates/elasticsearch.j2 has one line:
ES_JAVA_OPTS=
Every time call the playbook, /etc/sysconfig/elasticsearch is regenerated and elasticsearch-cli use the file and overided environment variable ES_JAVA_OPTS which include proxy setting("-Dhttps.proxyHost ...").So the script can't get the ES_JAVA_OPTS correctly always.
Comment the ES_JAVA_OPTS line in templates/elasticsearch.j2 then it work.

jmlrt commented

Hi @realkcn,
I created #648 which should fix it, however, I don't have a proper way to test it (a VM without direct Internet Access and proxy).
Could you test a new deployment while applying the changes in my PR and confirm me if it's solving your issue?

realkcn commented

The issue solved.
BTW: proxy_host and proxy_port setting on es_plugins section has the same problem.The example on document can't work too.
es_plugins:
- plugin: ingest-attachment
proxy_host: 10.1.4.2
proxy_port: 3128

jmlrt commented

BTW: proxy_host and proxy_port setting on es_plugins section has the same problem.The example on document can't work too.

You mean specifying dedicated proxy per plugin like that?

es_plugins:
- plugin: ingest-attachment
  proxy_host: 10.1.4.2
  proxy_port: 3128

Yeah I removed https://github.com/elastic/ansible-elasticsearch/pull/648/files#diff-3f419eb05dd0502a4e53b27340029a92L78 because ES_JAVA-OPTS in tasks/elasticsearch-plugins.yml because if it's overrided by ES_JAVA_OPTS in /etc/sysconfig/elasticsearch for global proxy host, I don't see how it could work for plugin specific proxy.

I'm not sure we have real use cases where people needs to specify different proxies for plugin install than the default proxy used for global Elasticsearch.

Are you in that use-case?

realkcn commented

BTW: proxy_host and proxy_port setting on es_plugins section has the same problem.The example on document can't work too.

You mean specifying dedicated proxy per plugin like that?

es_plugins:
- plugin: ingest-attachment
  proxy_host: 10.1.4.2
  proxy_port: 3128

Yeah I removed https://github.com/elastic/ansible-elasticsearch/pull/648/files#diff-3f419eb05dd0502a4e53b27340029a92L78 because ES_JAVA-OPTS in tasks/elasticsearch-plugins.yml because if it's overrided by ES_JAVA_OPTS in /etc/sysconfig/elasticsearch for global proxy host, I don't see how it could work for plugin specific proxy.

I'm not sure we have real use cases where people needs to specify different proxies for plugin install than the default proxy used for global Elasticsearch.

Are you in that use-case?

I think es_proxy_host is good enough for deployment.The specify proxy setting for plugin is useless.

jmlrt commented

OK I removed also the instructions to use specific proxies for plugins.