h2oai/driverlessai-recipes

driverless h2oai python pipeline tries to connect to H2O server at port 12345

cyrilthank opened this issue · 9 comments

cyrilthank commented

Hi

I am currently using the h2oai trial for scoping the features.

I am finding that after completing the training process and downloading the scoring pipeline, when i execute the run_example.py it fails while connecting to localhost:12345

Question: does the h2oai server need to be up and running while the run_example.py is executed

Is there a way to workaround this and have run_example.py execute (in isolation) without connecting to the h2o server?

Why is the run_example.py trying to connect to h2o server

goldentom42 commented

Hi @cyrilthank,

Thank you for trying our free trial of DriverlessAI.
Could you please tell us or give us a link to the run_example.py script you are trying to execute?

Best regards.

cyrilthank commented

Entire error message is as below

  • pm=pip
    ++ pwd
  • current_dir=/h2o/scoring-pipeline
    +++ dirname ./run_example.sh
    ++ cd .
    ++ pwd
  • script_dir=/h2o/scoring-pipeline
  • cd /h2o/scoring-pipeline
  • source ./common-functions.sh
  • main
  • parse_args_and_exec
  • '[' '' '!=' '' ']'
  • [[ pip == \p\i\p ]]
  • run_using_pip
  • '[' '!' -d env ']'
  • source env/bin/activate
    ++ '[' env/bin/activate = ./run_example.sh ']'
    ++ deactivate nondestructive
    ++ unset -f pydoc
    ++ '[' -z '' ']'
    ++ '[' -z '' ']'
    ++ '[' -n /bin/bash ']'
    ++ hash -r
    ++ '[' -z '' ']'
    ++ unset VIRTUAL_ENV
    ++ '[' '!' nondestructive = nondestructive ']'
    ++ VIRTUAL_ENV=/h2o/scoring-pipeline/env
    ++ export VIRTUAL_ENV
    ++ _OLD_VIRTUAL_PATH=/home/cyril/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:.
    ++ PATH=/h2o/scoring-pipeline/env/bin:/home/cyril/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:.
    ++ export PATH
    ++ '[' -z '' ']'
    ++ '[' -z '' ']'
    ++ _OLD_VIRTUAL_PS1=
    ++ '[' x '!=' x ']'
    +++ basename /h2o/scoring-pipeline/env
    ++ PS1='(env) '
    ++ export PS1
    ++ alias pydoc
    ++ true
    ++ '[' -n /bin/bash ']'
    ++ hash -r
  • python example.py
    2019-08-29 04:50:07,534 C: D: M: 15605 DEBUG : Could not locate cudart (None) or other error: #_python example.pyget_cuda_versions_subprocess: undefined symbol: cudaRuntimeGetVersion
    2019-08-29 04:50:07,654 C: D: M: 15605 DEBUG : Could not locate cudnn (None) or other error #_python example.pyget_cuda_versions_subprocess: undefined symbol: cudnnGetVersion
    2019-08-29 04:50:21,894 C:▇ D:80.1GB M:1.9GB 15462 INFO : Starting H2O server for recipes
    Traceback (most recent call last):
    File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/h2o/h2o.py", line 265, in init
    "connected.", "not found."))
    File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/h2o/backend/connection.py", line 323, in open
    conn._cluster = conn._test_connection(retries, messages=_msgs)
    File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/h2o/backend/connection.py", line 603, in _test_connection
    % (self._base_url, max_retries, "\n".join(errors)))
    h2o.exceptions.H2OConnectionError: Could not establish link to the H2O cloud http://localhost:50331 after 5 retries
    [50:22.18] H2OConnectionError: Unexpected HTTP error: HTTPConnectionPool(host='localhost', port=50331): Max retries exceeded with url: /3/Cloud (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1f3494b5f8>: Failed to establish a new connection: [Errno 111] Connection refused',))
    [50:22.39] H2OConnectionError: Unexpected HTTP error: HTTPConnectionPool(host='localhost', port=50331): Max retries exceeded with url: /3/Cloud (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1f3494bd30>: Failed to establish a new connection: [Errno 111] Connection refused',))
    [50:22.59] H2OConnectionError: Unexpected HTTP error: HTTPConnectionPool(host='localhost', port=50331): Max retries exceeded with url: /3/Cloud (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1f3494b6d8>: Failed to establish a new connection: [Errno 111] Connection refused',))
    [50:22.79] H2OConnectionError: Unexpected HTTP error: HTTPConnectionPool(host='localhost', port=50331): Max retries exceeded with url: /3/Cloud (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1f34963438>: Failed to establish a new connection: [Errno 111] Connection refused',))
    [50:23.00] H2OConnectionError: Unexpected HTTP error: HTTPConnectionPool(host='localhost', port=50331): Max retries exceeded with url: /3/Cloud (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1f34963b00>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "example.py", line 7, in
from scoring_h2oai_experiment_49205d0e_c403_11e9_9fbc_0800273a584e import Scorer
File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/scoring_h2oai_experiment_49205d0e_c403_11e9_9fbc_0800273a584e/init.py", line 1, in
from .scorer import Scorer
File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/scoring_h2oai_experiment_49205d0e_c403_11e9_9fbc_0800273a584e/scorer.py", line 18, in
load_all_custom_recipes()
File "h2oaicore/recipe_server_support.py", line 43, in h2oaicore.recipe_server_support.load_all_custom_recipes
File "h2oaicore/recipe_server_support.py", line 23, in h2oaicore.recipe_server_support.start_servers_for_recipes
File "h2oaicore/recipe_server_support.py", line 117, in h2oaicore.recipe_server_support.start_h2o_recipes
File "h2oaicore/recipe_server_support.py", line 118, in h2oaicore.recipe_server_support.start_h2o_recipes
File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/h2o/h2o.py", line 277, in init
bind_to_localhost=bind_to_localhost)
File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/h2o/backend/server.py", line 138, in start
bind_to_localhost=bind_to_localhost, log_dir=log_dir, log_level=log_level)
File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/h2o/backend/server.py", line 271, in _launch_server
java = self._find_java()
File "/h2o/scoring-pipeline/env/lib/python3.6/site-packages/h2o/backend/server.py", line 410, in _find_java
raise H2OStartupError("Cannot find Java. Please install the latest JRE from\n"
h2o.exceptions.H2OStartupError: Cannot find Java. Please install the latest JRE from
http://www.oracle.com/technetwork/java/javase/downloads/index.html
cyril@h2oclient:/h2o/scoring-pipeline$

cyrilthank commented 
==============calling code

cyril@h2oclient:/h2o/scoring-pipeline$ cat run.sh
#/h2o/scoring-pipeline/h2oai.txt
export DRIVERLESS_AI_LICENSE_FILE="/h2o/scoring-pipeline/h2oai.txt"
run_example.sh 
cyril@h2oclient:/h2o/scoring-pipeline$ 
=============================
cyrilthank commented 
==============run_example.sh

#!/usr/bin/env bash

set -o pipefail
set -ex

            deactivate
        fi
    fi

    source env/bin/activate
    python example.py
    deactivate
}

run_using_conda(){
    env_name="$(head -n 1 environment.yml | cut -d ' ' -f 2)"
    create_conda_env ${env_name}
    source activate ${env_name}
    python example.py
    source deactivate
}

cd "${script_dir}"
source "./common-functions.sh"

main $@
cd "${current_dir}"
cyrilthank commented 
==========example.py

# Copyright 2019 H2O.ai; Proprietary License;  -*- encoding: utf-8 -*-
print(train_munged)  # for model fitting (use entire frame, no cross-validation)
print(valid_munged)  # for model validation (parameter tuning)
print(test_munged)   # for final pipeline testing (one time)

#
# To retrieve the original feature column names, use the Scorer.get_column_names() method:
# This method retrieves the input column names
#

print('---------- Retrieve column names ----------')
print(scorer.get_column_names())

#
# To retrieve the transformed column names, use the Scorer.get_transformed_column_names() method:
# This method retrieves the transformed column names
#

print('---------- Retrieve transformed column names ----------')
print(scorer.get_transformed_column_names())

cyril@h2oclient:/h2o/scoring-pipeline$
cyril@h2oclient:/h2o/scoring-pipeline$ vi run_example.sh
print(train_munged)  # for model fitting (use entire frame, no cross-validation)
print(valid_munged)  # for model validation (parameter tuning)
print(test_munged)   # for final pipeline testing (one time)

#
# To retrieve the original feature column names, use the Scorer.get_column_names() method:
# This method retrieves the input column names 
#

print('---------- Retrieve column names ----------')
print(scorer.get_column_names())

#
# To retrieve the transformed column names, use the Scorer.get_transformed_column_names() method:
# This method retrieves the transformed column names
#

print('---------- Retrieve transformed column names ----------')
print(scorer.get_transformed_column_names())
cyrilthank commented

apologies for the bold letters caused by 'dumb copy & paste'
files.zip

goldentom42 commented

I suspect java is not installed on the server and therefore it cannot start H2O open source clusters:

raise H2OStartupError("Cannot find Java. Please install the latest JRE from\n")

Could you try installing and check if that solves the problem ?

cyrilthank commented

Thanks @goldentom42 you are right

However after installing i get the below

---------- Score Row ----------
Stalling, checking threads for check_period=28 stalled_period=2 [75.5, 0.0, 0.0, 0.0, 3.3, 0.0, 0.4, 0.0, 0.5, 0.6, 0.4] [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] : proctitle: |python example.py2/3 (2 1) (True) [0, 0, 0, 3, 0, 0.4, 0, 0.5, 0.6, 0.4] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Stalling, checking threads for check_period=28 stalled_period=2 [75.5, 0.0, 0.0, 0.0, 3.3, 0.0, 0.4, 0.0, 0.5, 0.6, 0.4] [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] : proctitle: |python example.py2/3 (2 1) (True) [0, 0, 0, 3, 0, 0.4, 0, 0.5, 0.6, 0.4] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Stalling, checking threads for check_period=30 stalled_period=3 [75.5, 0.0, 0.0, 0.0, 3.3, 0.0, 0.4, 0.0, 0.5, 0.6, 0.4, 0.5, 0.3] [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] : proctitle: |python example.py2/3 (2 1) (True) [0, 3, 0, 0.4, 0, 0.5, 0.6, 0.4, 0.5, 0.3] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Stalling, checking threads for check_period=30 stalled_period=3 [75.5, 0.0, 0.0, 0.0, 3.3, 0.0, 0.4, 0.0, 0.5, 0.6, 0.4, 0.5, 0.3] [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] : proctitle: |python example.py2/3 (2 1) (True) [0, 3, 0, 0.4, 0, 0.5, 0.6, 0.4, 0.5, 0.3] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Stalling for too long > 440.02 and CPU: [75.5, 0.0, 0.0, 0.0, 3.3, 0.0, 0.4, 0.0, 0.5, 0.6, 0.4, 0.5, 0.3, 0.3, 1.4, 0.8, 1.8, 0.6, 0.0, 1.2] % and GPU: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] % for n_gpus=0 : proctitle: |python example.py2/3 (2 1) (True) [0.6, 0.4, 0.5, 0.3, 0.3, 1, 0.8, 1, 0.6, 0] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Stalling for too long > 440.02 and CPU: [75.5, 0.0, 0.0, 0.0, 3.3, 0.0, 0.4, 0.0, 0.5, 0.6, 0.4, 0.5, 0.3, 0.3, 1.4, 0.8, 1.8, 0.6, 0.0, 1.2] % and GPU: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] % for n_gpus=0 : proctitle: |python example.py2/3 (2 1) (True) [0.6, 0.4, 0.5, 0.3, 0.3, 1, 0.8, 1, 0.6, 0] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Traceback (most recent call last):
  File "example.py", line 33, in <module>
    'At 2.7 MM, my bid is .70/sf higher than his unitsunder construction.',  # text
  File "/h2o/scoring-pipeline/scoring_h2oai_experiment_b90fbaca_c984_11e9_8285_0800273a584e/scorer.py", line 198, in score
    fast_approx=fast_approx)
  File "/h2o/scoring-pipeline/scoring_h2oai_experiment_b90fbaca_c984_11e9_8285_0800273a584e/scorer.py", line 98, in _predict
    preds = predict(input_frame, output_margin=output_margin, pred_contribs=pred_contribs, fast_approx=fast_approx)
  File "/h2o/scoring-pipeline/scoring_h2oai_experiment_b90fbaca_c984_11e9_8285_0800273a584e/scorer.py", line 69, in _predict
    fast_approx=fast_approx, **kwargs)
  File "h2oaicore/models.py", line 7473, in h2oaicore.models.StackedEnsemble.predict_base
  File "h2oaicore/models.py", line 7502, in h2oaicore.models.StackedEnsemble._predict_with_transform
  File "h2oaicore/models.py", line 7048, in h2oaicore.models.StackedBaseModels.transform
  File "h2oaicore/systemutils.py", line 2046, in h2oaicore.systemutils.mppool.submit_tryget
  File "h2oaicore/systemutils.py", line 1675, in h2oaicore.systemutils.mppool.try_get_internal
RuntimeError: Stalling for too long > 440.02 and CPU: [75.5, 0.0, 0.0, 0.0, 3.3, 0.0, 0.4, 0.0, 0.5, 0.6, 0.4, 0.5, 0.3, 0.3, 1.4, 0.8, 1.8, 0.6, 0.0, 1.2] % and GPU: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] % for n_gpus=0 : proctitle: |python example.py2/3 (2 1) (True) [0.6, 0.4, 0.5, 0.3, 0.3, 1, 0.8, 1, 0.6, 0] [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
cyril@h2oclient:/h2o/scoring-pipeline$

can you please help

cyrilthank commented

can you please help with this error which i get after i run (post java install)