system-sphinx POD stays in CrashLoopBackoff

[dlydiard]

Overall 3scale was able to startup with externa postgres databases. The Routes were created and i was able to login to the admin portal. However, the system-sphinx POD stays in CrashLoopBackoff, with the following logs:

declare -x ZYNC_PORT_8080_TCP_PROTO="tcp"
declare -x ZYNC_SERVICE_HOST="172.21.39.147"
declare -x ZYNC_SERVICE_PORT="8080"
declare -x ZYNC_SERVICE_PORT_8080_TCP="8080"
declare -x container="oci"
declare -x npm_config_nodedir="/remote-source/nodejs/app"
NOTE: Gem::Specification#has_rdoc= is deprecated with no replacement. It will be removed on or after 2018-12-01.
Gem::Specification#has_rdoc= called from /opt/deps/rubygems/github.com/3scale/prawn/prawn-external-gitcommit-88aead0d97e230d08e9f51a18711d01317965bfe/app/prawn-core.gemspec:36.
fatal: not a git repository (or any of the parent directories): .git
rake aborted!
SignalException: SIGTERM
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.131.2/lib/aws-sdk-core/json.rb:9:in `require_relative'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.131.2/lib/aws-sdk-core/json.rb:9:in `<top (required)>'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.131.2/lib/aws-sdk-core.rb:69:in `require_relative'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.131.2/lib/aws-sdk-core.rb:69:in `<top (required)>'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-ses-1.47.0/lib/aws-sdk-ses.rb:11:in `require'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-ses-1.47.0/lib/aws-sdk-ses.rb:11:in `<top (required)>'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-rails-3.6.2/lib/aws/rails/mailer.rb:3:in `require'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-rails-3.6.2/lib/aws/rails/mailer.rb:3:in `<top (required)>'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-rails-3.6.2/lib/aws-sdk-rails.rb:3:in `require_relative'
/opt/system/vendor/bundle/ruby/2.6.0/gems/aws-sdk-rails-3.6.2/lib/aws-sdk-rails.rb:3:in `<top (required)>'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/runtime.rb:60:in `require'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/runtime.rb:55:in `each'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/runtime.rb:55:in `block in require'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/runtime.rb:44:in `each'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/runtime.rb:44:in `require'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler.rb:174:in `require'
/opt/system/config/application.rb:14:in `<top (required)>'
/opt/system/Rakefile:1:in `require'
/opt/system/Rakefile:1:in `<top (required)>'
/opt/system/vendor/bundle/ruby/2.6.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/cli/exec.rb:63:in `load'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/cli/exec.rb:63:in `kernel_load'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/cli/exec.rb:28:in `run'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/cli.rb:481:in `exec'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/cli.rb:31:in `dispatch'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/cli.rb:25:in `start'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/exe/bundle:49:in `block in <top (required)>'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/lib/bundler/friendly_errors.rb:128:in `with_friendly_errors'
/opt/rh/rh-ruby26/root/usr/local/share/gems/gems/bundler-2.2.25/exe/bundle:37:in `<top (required)>'
/opt/rh/rh-ruby26/root/usr/local/bin/bundle:23:in `load'
/opt/rh/rh-ruby26/root/usr/local/bin/bundle:23:in `<main>'
(See full trace by running task with --trace)

The POD events say that it is failing the liveness probe.

Any ideas?

[eguzki]

Sphinx has been replaced by searchd recently #818

Which version of 3scale are you running?

For specifically sphinx related issues, better to ask in https://github.com/3scale/porta

[dlydiard]

We are on 3scale-operator.v0.10.2-0.1679333916.p

This eventually started working overnight, after around 230 restarts:

fatal: not a git repository (or any of the parent directories): .git
Removing :environment prerequisite from db:create
Generating configuration to db/sphinx/production.conf
[Fri May  5 15:34:29.725 2023] [61] using config file 'db/sphinx/production.conf'...
[Fri May  5 15:34:29.727 2023] [61] listening on all interfaces, port=9306
listening on all interfaces, port=9306
Sphinx 2.2.11-id64-release (95ae9a6)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
precaching index 'account_core'
precaching index 'topic_core'
precaching index 'backend_api_core'
precaching index 'cms_page_core'
precaching index 'email_configuration_core'
precaching index 'metric_core'
precaching index 'plan_core'
precaching index 'proxy_rule_core'
precaching index 'service_core'
[Fri May  5 15:34:29.765 2023] [61] accepting connections
accepting connections