Unable to install airflow_package via chef in kitchen environment
Idun0 opened this issue · 5 comments
Idun0 commented
Ive been trying to bring up a clean kitchen instance for a couple days now, i was getting a setuptools issue until i was able to get past that with attribute assigning the pip_version to 18.0. But i can not seem to get past this error:
`* python_package[apache-airflow] action install[2018-10-18T15:19:20+00:00] INFO: Processing python_package[apache-airflow] action install (airflow::packages line 67)
================================================================================
Error executing action `install` on resource 'python_package[apache-airflow]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
STDOUT: Collecting apache-airflow==1.10.0
Downloading https://files.pythonhosted.org/packages/da/2a/6e9efcd40193850e2f636c7306eede2ff5607aa9f81ff9f7a151d9b13ff8/apache-airflow-1.10.0.tar.gz (4.3MB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 393, in <module>
do_setup()
File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 258, in do_setup
verify_gpl_dependency()
File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 49, in verify_gpl_dependency
raise RuntimeError("By default one of Airflow's dependencies installs a GPL "
RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE
----------------------------------------
STDERR: The directory '/home/vagrant/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
You are using pip version 6.1.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jtxQR7/apache-airflow
---- End output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
Ran ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] returned 1
Cookbook Trace:
---------------
/tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:201:in `tap'
/tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:201:in `language_command_shell_out!'
/tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:216:in `block in language_command_mixin'
/tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:282:in `pip_command'
/tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:296:in `pip_install'
/tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:209:in `install_package'
Resource Declaration:
---------------------
# In /tmp/kitchen/cache/cookbooks/airflow/recipes/packages.rb
67: python_package node['airflow']['airflow_package'] do
68: version node['airflow']['version']
69: end
70:
Compiled Resource:
------------------
# Declared in /tmp/kitchen/cache/cookbooks/airflow/recipes/packages.rb:67:in `from_file'
python_package("apache-airflow") do
package_name "apache-airflow"
action [:install]
retries 0
retry_delay 2
default_guard_interpreter :default
declared_type :python_package
cookbook_name "airflow"
recipe_name "packages"
parent_python python_runtime[2]
timeout 900
end
Platform:
---------
x86_64-linux
[2018-10-18T15:19:25+00:00] INFO: Running queued delayed notifications before re-raising exception
Running handlers:
[2018-10-18T15:19:25+00:00] ERROR: Running exception handlers
Running handlers complete
[2018-10-18T15:19:25+00:00] ERROR: Exception handlers complete
Chef Client failed. 38 resources updated in 02 minutes 31 seconds
[2018-10-18T15:19:25+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
[2018-10-18T15:19:25+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-10-18T15:19:25+00:00] ERROR: python_package[apache-airflow] (airflow::packages line 67) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
STDOUT: Collecting apache-airflow==1.10.0
Downloading https://files.pythonhosted.org/packages/da/2a/6e9efcd40193850e2f636c7306eede2ff5607aa9f81ff9f7a151d9b13ff8/apache-airflow-1.10.0.tar.gz (4.3MB)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 393, in <module>
do_setup()
File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 258, in do_setup
verify_gpl_dependency()
File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 49, in verify_gpl_dependency
raise RuntimeError("By default one of Airflow's dependencies installs a GPL "
RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE
----------------------------------------
STDERR: The directory '/home/vagrant/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
You are using pip version 6.1.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jtxQR7/apache-airflow
---- End output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
Ran ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] returned 1
[2018-10-18T15:19:25+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)`
cloneluke commented
I have not had time to test this cookbook with airflow 1.10 yet.
I do know that is a known issue with airflow 1.10
https://issues.apache.org/jira/browse/AIRFLOW-3117
also discussed at length here: apache-airflow.slack.com
I will try to get a fix for this, it should be pretty simple, just add
magic_shell_environment 'SLUGIFY_USES_TEXT_UNIDECODE' do value 'yes' end
Idun0 commented
@cloneluke Thank you very much, I really appreciate it!!!
cloneluke commented
add
magic_shell_environment 'SLUGIFY_USES_TEXT_UNIDECODE' do value 'yes' end
as a resource before you call airflow cookbook, or manually set the env var to test it before you run the recipe/cookbook
let me know if that works
Idun0 commented
This worked thank you