Unable to spin up Fleet-server

Question

Unable to spin up Fleet-server

dikshachauhan-qasource opened this issue 2 years ago · 24 comments

dikshachauhan-qasource commented 2 years ago

While using make commands and setting up the environment, we are getting errors when spinning up fleet-server. Below is the error details:

TASK [Start stack without any tag to spin up the stack using Fleet's bootstrapping code] **********************************************
fatal: [3.141.85.36]: FAILED! => {"changed": true, "cmd": "TAGS='non-existing-tag' /home/admin/e2e-testing/.ci/scripts/functional-test.sh", "delta": "0:00:35.436131", "end": "2022-11-23 10:02:09.082902", "msg": "non-zero return code", "rc": 2, "start": "2022-11-23 10:01:33.646771", "stderr": "+ BASE_DIR=/home/admin/e2e-testing\n+ SUITE=fleet\n+ REPORT_PREFIX=junit\n+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin\n+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin\n+ BASE_DIR=/home/admin/e2e-testing\n+ /home/admin/e2e-testing/.ci/scripts/install-test-dependencies.sh fleet\n+ BASE_DIR=/home/admin/e2e-testing\n+ SUITE=fleet\n+ '[' -f /home/admin/e2e-testing/.ci/scripts/install-fleet-test-dependencies.sh ']'\n+ echo 'Not installing test dependencies for fleet'\n++ echo junit\n++ sed -r 's/[ @~]+//g'\n+ REPORT_PREFIX=junit\n++ date +%Y-%m-%d-%H:%M:%S\n+ SEED=2022-11-23-10:01:33\n+ REPORT=/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33\n+ echo 'REPORT=\"/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33\"'\n+ echo 'Removing previous test files in the case the workspace is reused'\n+ rm -f '/home/admin/e2e-testing/outputs/TEST-*.*'\n+ TAGS=non-existent-tag\n+ FORMAT=pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.xml\n+ make --no-print-directory -C /home/admin/e2e-testing/e2e/_suites/fleet functional-test\ngo: downloading github.com/Jeffail/gabs/v2 v2.6.0\ngo: downloading github.com/cenkalti/backoff/v4 v4.1.2\ngo: downloading github.com/cucumber/godog v0.12.4\ngo: downloading github.com/google/uuid v1.3.0\ngo: downloading github.com/pkg/errors v0.9.1\ngo: downloading github.com/sirupsen/logrus v1.8.1\ngo: downloading go.elastic.co/apm v1.13.0\ngo: downloading github.com/docker/go-connections v0.4.0\ngo: downloading github.com/spf13/pflag v1.0.5\ngo: downloading github.com/docker/cli v20.10.11+incompatible\ngo: downloading github.com/docker/docker v20.10.12+incompatible\ngo: downloading github.com/mitchellh/go-homedir v1.1.0\ngo: downloading github.com/testcontainers/testcontainers-go v0.13.0\ngo: downloading github.com/elastic/go-elasticsearch/v8 v8.0.0-20210317102009-a9d74cec0186\ngo: downloading go.elastic.co/apm/module/apmelasticsearch v1.10.0\ngo: downloading github.com/shirou/gopsutil/v3 v3.21.10\ngo: downloading golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e\ngo: downloading github.com/cucumber/messages-go/v16 v16.0.1\ngo: downloading go.elastic.co/apm/module/apmhttp v1.10.0\ngo: downloading github.com/gobuffalo/packr/v2 v2.8.1\ngo: downloading github.com/joho/godotenv v1.4.0\ngo: downloading gopkg.in/yaml.v2 v2.4.0\ngo: downloading github.com/elastic/go-sysinfo v1.1.1\ngo: downloading go.elastic.co/fastjson v1.1.0\ngo: downloading github.com/docker/go-units v0.4.0\ngo: downloading github.com/opencontainers/image-spec v1.0.2\ngo: downloading github.com/containerd/containerd v1.5.9\ngo: downloading github.com/docker/distribution v2.7.1+incompatible\ngo: downloading github.com/opencontainers/go-digest v1.0.0\ngo: downloading github.com/magiconair/properties v1.8.5\ngo: downloading github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6\ngo: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b\ngo: downloading github.com/cucumber/gherkin-go/v19 v19.0.3\ngo: downloading github.com/hashicorp/go-memdb v1.3.0\ngo: downloading github.com/gofrs/uuid v4.0.0+incompatible\ngo: downloading github.com/tklauser/go-sysconf v0.3.9\ngo: downloading github.com/gobuffalo/packd v1.0.0\ngo: downloading github.com/markbates/oncer v1.0.0\ngo: downloading github.com/markbates/safe v1.0.1\ngo: downloading github.com/armon/go-radix v1.0.0\ngo: downloading google.golang.org/grpc v1.43.0\ngo: downloading golang.org/x/net v0.0.0-20220107192237-5cfca573fb4d\ngo: downloading github.com/morikuni/aec v1.0.0\ngo: downloading github.com/hashicorp/go-immutable-radix v1.3.1\ngo: downloading github.com/tklauser/numcpus v0.3.0\ngo: downloading github.com/karrick/godirwalk v1.15.8\ngo: downloading github.com/markbates/errx v1.1.0\ngo: downloading github.com/gobuffalo/logger v1.0.3\ngo: downloading howett.net/plist v0.0.0-20201203080718-1454fab16a06\ngo: downloading github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901\ngo: downloading github.com/prometheus/procfs v0.7.3\ngo: downloading github.com/gogo/protobuf v1.3.2\ngo: downloading github.com/opencontainers/runc v1.0.2\ngo: downloading github.com/moby/sys/mount v0.2.0\ngo: downloading github.com/hashicorp/golang-lru v0.5.4\ngo: downloading golang.org/x/crypto v0.0.0-20211117183948-ae814b36b871\ngo: downloading google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368\ngo: downloading github.com/moby/sys/mountinfo v0.5.0\ngo: downloading github.com/golang/protobuf v1.5.2\ngo: downloading google.golang.org/protobuf v1.27.1\ngo: downloading golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b\n# github.com/elastic/e2e-testing/internal/kibana\n../../../internal/kibana/policies.go:207:10: no new variables on left side of :=\n../../../internal/kibana/policies.go:213:26: no new variables on left side of :=\n../../../internal/kibana/policies.go:214:18: no new variables on left side of :=\n../../../internal/kibana/policies.go:277:42: syntax error: unexpected context, expecting comma or )\n../../../internal/kibana/policies.go:280:3: syntax error: unexpected ) after top level declaration\n../../../internal/kibana/policies.go:307:2: syntax error: non-declaration statement outside function body\nmake: *** [../../commons-test.mk:65: functional-test] Error 2", "stderr_lines": ["+ BASE_DIR=/home/admin/e2e-testing", "+ SUITE=fleet", "+ REPORT_PREFIX=junit", "+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin", "+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin", "+ BASE_DIR=/home/admin/e2e-testing", "+ /home/admin/e2e-testing/.ci/scripts/install-test-dependencies.sh fleet", "+ BASE_DIR=/home/admin/e2e-testing", "+ SUITE=fleet", "+ '[' -f /home/admin/e2e-testing/.ci/scripts/install-fleet-test-dependencies.sh ']'", "+ echo 'Not installing test dependencies for fleet'", "++ echo junit", "++ sed -r 's/[ @~]+//g'", "+ REPORT_PREFIX=junit", "++ date +%Y-%m-%d-%H:%M:%S", "+ SEED=2022-11-23-10:01:33", "+ REPORT=/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33", "+ echo 'REPORT=\"/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33\"'", "+ echo 'Removing previous test files in the case the workspace is reused'", "+ rm -f '/home/admin/e2e-testing/outputs/TEST-*.*'", "+ TAGS=non-existent-tag", "+ FORMAT=pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.xml", "+ make --no-print-directory -C /home/admin/e2e-testing/e2e/_suites/fleet functional-test", "go: downloading github.com/Jeffail/gabs/v2 v2.6.0", "go: downloading github.com/cenkalti/backoff/v4 v4.1.2", "go: downloading github.com/cucumber/godog v0.12.4", "go: downloading github.com/google/uuid v1.3.0", "go: downloading github.com/pkg/errors v0.9.1", "go: downloading github.com/sirupsen/logrus v1.8.1", "go: downloading go.elastic.co/apm v1.13.0", "go: downloading github.com/docker/go-connections v0.4.0", "go: downloading github.com/spf13/pflag v1.0.5", "go: downloading github.com/docker/cli v20.10.11+incompatible", "go: downloading github.com/docker/docker v20.10.12+incompatible", "go: downloading github.com/mitchellh/go-homedir v1.1.0", "go: downloading github.com/testcontainers/testcontainers-go v0.13.0", "go: downloading github.com/elastic/go-elasticsearch/v8 v8.0.0-20210317102009-a9d74cec0186", "go: downloading go.elastic.co/apm/module/apmelasticsearch v1.10.0", "go: downloading github.com/shirou/gopsutil/v3 v3.21.10", "go: downloading golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e", "go: downloading github.com/cucumber/messages-go/v16 v16.0.1", "go: downloading go.elastic.co/apm/module/apmhttp v1.10.0", "go: downloading github.com/gobuffalo/packr/v2 v2.8.1", "go: downloading github.com/joho/godotenv v1.4.0", "go: downloading gopkg.in/yaml.v2 v2.4.0", "go: downloading github.com/elastic/go-sysinfo v1.1.1", "go: downloading go.elastic.co/fastjson v1.1.0", "go: downloading github.com/docker/go-units v0.4.0", "go: downloading github.com/opencontainers/image-spec v1.0.2", "go: downloading github.com/containerd/containerd v1.5.9", "go: downloading github.com/docker/distribution v2.7.1+incompatible", "go: downloading github.com/opencontainers/go-digest v1.0.0", "go: downloading github.com/magiconair/properties v1.8.5", "go: downloading github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6", "go: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b", "go: downloading github.com/cucumber/gherkin-go/v19 v19.0.3", "go: downloading github.com/hashicorp/go-memdb v1.3.0", "go: downloading github.com/gofrs/uuid v4.0.0+incompatible", "go: downloading github.com/tklauser/go-sysconf v0.3.9", "go: downloading github.com/gobuffalo/packd v1.0.0", "go: downloading github.com/markbates/oncer v1.0.0", "go: downloading github.com/markbates/safe v1.0.1", "go: downloading github.com/armon/go-radix v1.0.0", "go: downloading google.golang.org/grpc v1.43.0", "go: downloading golang.org/x/net v0.0.0-20220107192237-5cfca573fb4d", "go: downloading github.com/morikuni/aec v1.0.0", "go: downloading github.com/hashicorp/go-immutable-radix v1.3.1", "go: downloading github.com/tklauser/numcpus v0.3.0", "go: downloading github.com/karrick/godirwalk v1.15.8", "go: downloading github.com/markbates/errx v1.1.0", "go: downloading github.com/gobuffalo/logger v1.0.3", "go: downloading howett.net/plist v0.0.0-20201203080718-1454fab16a06", "go: downloading github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901", "go: downloading github.com/prometheus/procfs v0.7.3", "go: downloading github.com/gogo/protobuf v1.3.2", "go: downloading github.com/opencontainers/runc v1.0.2", "go: downloading github.com/moby/sys/mount v0.2.0", "go: downloading github.com/hashicorp/golang-lru v0.5.4", "go: downloading golang.org/x/crypto v0.0.0-20211117183948-ae814b36b871", "go: downloading google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368", "go: downloading github.com/moby/sys/mountinfo v0.5.0", "go: downloading github.com/golang/protobuf v1.5.2", "go: downloading google.golang.org/protobuf v1.27.1", "go: downloading golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b", "# github.com/elastic/e2e-testing/internal/kibana", "../../../internal/kibana/policies.go:207:10: no new variables on left side of :=", "../../../internal/kibana/policies.go:213:26: no new variables on left side of :=", "../../../internal/kibana/policies.go:214:18: no new variables on left side of :=", "../../../internal/kibana/policies.go:277:42: syntax error: unexpected context, expecting comma or )", "../../../internal/kibana/policies.go:280:3: syntax error: unexpected ) after top level declaration", "../../../internal/kibana/policies.go:307:2: syntax error: non-declaration statement outside function body", "make: *** [../../commons-test.mk:65: functional-test] Error 2"], "stdout": "Not installing test dependencies for fleet\nRemoving previous test files in the case the workspace is reused\nOP_LOG_LEVEL=INFO \\\nOP_LOG_INCLUDE_TIMESTAMP=TRUE \\\nTIMEOUT_FACTOR=3 \\\nSTACK_VERSION= \\\nDEVELOPER_MODE=false \\\nELASTIC_APM_SERVICE_NAME=\"E2E Tests\" \\\nELASTIC_APM_CENTRAL_CONFIG=\"false\" \\\nELASTIC_APM_GLOBAL_LABELS= \\\nELASTIC_APM_SERVICE_VERSION=\"\" \\\nELASTIC_APM_ENVIRONMENT=\"local\" \\\nELASTIC_APM_SECRET_TOKEN=\"\" \\\nELASTIC_APM_SERVER_URL=\"\" \\\nBRANCH_NAME=\"\" \\\nTRACEPARENT=\"\" \\\ngo test -timeout 90m -v --godog.format=\"pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.xml\"  --godog.tags=\"non-existent-tag && ~skip\"\nFAIL\tgithub.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]", "stdout_lines": ["Not installing test dependencies for fleet", "Removing previous test files in the case the workspace is reused", "OP_LOG_LEVEL=INFO \\", "OP_LOG_INCLUDE_TIMESTAMP=TRUE \\", "TIMEOUT_FACTOR=3 \\", "STACK_VERSION= \\", "DEVELOPER_MODE=false \\", "ELASTIC_APM_SERVICE_NAME=\"E2E Tests\" \\", "ELASTIC_APM_CENTRAL_CONFIG=\"false\" \\", "ELASTIC_APM_GLOBAL_LABELS= \\", "ELASTIC_APM_SERVICE_VERSION=\"\" \\", "ELASTIC_APM_ENVIRONMENT=\"local\" \\", "ELASTIC_APM_SECRET_TOKEN=\"\" \\", "ELASTIC_APM_SERVER_URL=\"\" \\", "BRANCH_NAME=\"\" \\", "TRACEPARENT=\"\" \\", "go test -timeout 90m -v --godog.format=\"pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-39e5cd94-2022-11-23-10:01:33.xml\"  --godog.tags=\"non-existent-tag && ~skip\"", "FAIL\tgithub.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]"]}

PLAY RECAP ****************************************************************************************************************************
3.141.85.36                : ok=13   changed=7    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0

make: *** [Makefile:204: start-elastic-stack] Error 2
make: Leaving directory '/home/admin/e2e-testing/.ci

Can anybody take a look on above, as we are unable to perform testing around scripts.

Answer 1 · 2022-11-23T12:24:56.000Z

cc @pazone

Answer 2 · 2022-11-30T08:02:25.000Z

@pazone This seems as if it is negatively impacting the ability of @dikshachauhan-qasource 's team to perform Fleet testing. What's the status on this issue?

Answer 3 · 2022-11-30T12:03:51.000Z

@dikshachauhan-qasource could you please attach the full build log?

Answer 4 · 2022-11-30T14:00:32.000Z

Hi @pazone

Please find the CI job logs for reference highlighting the same issue.

https://beats-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/e2e-tests/pipelines/e2e-testing-mbp/pipelines/PR-3252/runs/1/steps/383/log/?start=0

Thanks!

Answer 5 · 2022-12-01T11:55:48.000Z

Hi @pazone ,

As a workaround, fleet server issue was resolved once ansible version is reset. However, if its get once down, and we try to restart it, the above reported error continues to occur.

I have validate issue on two different VMs and thus again blocking us to continue on regression part.

Logs captured are:

TASK [Start stack without any tag to spin up the stack using Fleet's bootstrapping code] **********************************************
fatal: [13.58.183.81]: FAILED! => {"changed": true, "cmd": "TAGS='non-existing-tag' /home/admin/e2e-testing/.ci/scripts/functional-test.sh", "delta": "0:00:00.655275", "end": "2022-12-01 11:50:18.132193", "msg": "non-zero return code", "rc": 2, "start": "2022-12-01 11:50:17.476918", "stderr": "+ BASE_DIR=/home/admin/e2e-testing\n+ SUITE=fleet\n+ REPORT_PREFIX=junit\n+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin\n+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin\n+ BASE_DIR=/home/admin/e2e-testing\n+ /home/admin/e2e-testing/.ci/scripts/install-test-dependencies.sh fleet\n+ BASE_DIR=/home/admin/e2e-testing\n+ SUITE=fleet\n+ '[' -f /home/admin/e2e-testing/.ci/scripts/install-fleet-test-dependencies.sh ']'\n+ echo 'Not installing test dependencies for fleet'\n++ echo junit\n++ sed -r 's/[ @~]+//g'\n+ REPORT_PREFIX=junit\n++ date +%Y-%m-%d-%H:%M:%S\n+ SEED=2022-12-01-11:50:17\n+ REPORT=/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17\n+ echo 'REPORT=\"/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17\"'\n+ echo 'Removing previous test files in the case the workspace is reused'\n+ rm -f '/home/admin/e2e-testing/outputs/TEST-*.*'\n+ TAGS=non-existent-tag\n+ FORMAT=pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.xml\n+ make --no-print-directory -C /home/admin/e2e-testing/e2e/_suites/fleet functional-test\n# github.com/elastic/e2e-testing/e2e/_suites/fleet [github.com/elastic/e2e-testing/e2e/_suites/fleet.test]\n./integrations.go:175:10: syntax error: unexpected newline, expecting comma or }\n./integrations.go:180:5: syntax error: unexpected newline, expecting comma or }\nmake: *** [../../commons-test.mk:65: functional-test] Error 2", "stderr_lines": ["+ BASE_DIR=/home/admin/e2e-testing", "+ SUITE=fleet", "+ REPORT_PREFIX=junit", "+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin", "+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin", "+ BASE_DIR=/home/admin/e2e-testing", "+ /home/admin/e2e-testing/.ci/scripts/install-test-dependencies.sh fleet", "+ BASE_DIR=/home/admin/e2e-testing", "+ SUITE=fleet", "+ '[' -f /home/admin/e2e-testing/.ci/scripts/install-fleet-test-dependencies.sh ']'", "+ echo 'Not installing test dependencies for fleet'", "++ echo junit", "++ sed -r 's/[ @~]+//g'", "+ REPORT_PREFIX=junit", "++ date +%Y-%m-%d-%H:%M:%S", "+ SEED=2022-12-01-11:50:17", "+ REPORT=/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17", "+ echo 'REPORT=\"/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17\"'", "+ echo 'Removing previous test files in the case the workspace is reused'", "+ rm -f '/home/admin/e2e-testing/outputs/TEST-*.*'", "+ TAGS=non-existent-tag", "+ FORMAT=pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.xml", "+ make --no-print-directory -C /home/admin/e2e-testing/e2e/_suites/fleet functional-test", "# github.com/elastic/e2e-testing/e2e/_suites/fleet [github.com/elastic/e2e-testing/e2e/_suites/fleet.test]", "./integrations.go:180:5: syntax error: unexpected newline, expecting comma or }", "./integrations.go:180:5: syntax error: unexpected newline, expecting comma or }", "make: *** [../../commons-test.mk:65: functional-test] Error 2"], "stdout": "Not installing test dependencies for fleet\nRemoving previous test files in the case the workspace is reused\nOP_LOG_LEVEL=INFO \\\nOP_LOG_INCLUDE_TIMESTAMP=TRUE \\\nTIMEOUT_FACTOR=3 \\\nSTACK_VERSION= \\\nDEVELOPER_MODE=false \\\nELASTIC_APM_SERVICE_NAME=\"E2E Tests\" \\\nELASTIC_APM_CENTRAL_CONFIG=\"false\" \\\nELASTIC_APM_GLOBAL_LABELS= \\\nELASTIC_APM_SERVICE_VERSION=\"\" \\\nELASTIC_APM_ENVIRONMENT=\"local\" \\\nELASTIC_APM_SECRET_TOKEN=\"\" \\\nELASTIC_APM_SERVER_URL=\"\" \\\nBRANCH_NAME=\"\" \\\nTRACEPARENT=\"\" \\\ngo test -timeout 90m -v --godog.format=\"pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.xml\"  --godog.tags=\"non-existent-tag && ~skip\"\nFAIL\tgithub.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]", "stdout_lines": ["Not installing test dependencies for fleet", "Removing previous test files in the case the workspace is reused", "OP_LOG_LEVEL=INFO \\", "OP_LOG_INCLUDE_TIMESTAMP=TRUE \\", "TIMEOUT_FACTOR=3 \\", "STACK_VERSION= \\", "DEVELOPER_MODE=false \\", "ELASTIC_APM_SERVICE_NAME=\"E2E Tests\" \\", "ELASTIC_APM_CENTRAL_CONFIG=\"false\" \\", "ELASTIC_APM_GLOBAL_LABELS= \\", "ELASTIC_APM_SERVICE_VERSION=\"\" \\", "ELASTIC_APM_ENVIRONMENT=\"local\" \\", "ELASTIC_APM_SECRET_TOKEN=\"\" \\", "ELASTIC_APM_SERVER_URL=\"\" \\", "BRANCH_NAME=\"\" \\", "TRACEPARENT=\"\" \\", "go test -timeout 90m -v --godog.format=\"pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-3d1ac05b-2022-12-01-11:50:17.xml\"  --godog.tags=\"non-existent-tag && ~skip\"", "FAIL\tgithub.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]"]}

PLAY RECAP ****************************************************************************************************************************
13.58.183.81               : ok=13   changed=6    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0

make: *** [Makefile:204: start-elastic-stack] Error 2
make: Leaving directory '/home/admin/e2e-testing/.ci'

Please let me know if more info is required.

Thanks

Answer 6 · 2022-12-01T13:30:18.000Z

It is failing in the test fleet_debian_10_arm64_system_integration / Adding memory system/metrics Integration to a Policy – System Integration
https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/PR-3252/1/tests

Answer 7 · 2022-12-01T13:33:56.000Z

@dikshachauhan-qasource Is this a part of the testing you're doing locally? If so, can you give me the sequence of commands that you're running which produce this error?

Answer 8 · 2022-12-01T13:35:37.000Z

the test fails on Debian 10 ARM64, it is a really specific failure, I will wait to try to retry to confirm the error is consistent first of all.

Answer 9 · 2022-12-01T13:41:28.000Z

We have cucumber files at Artifacts https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/PR-3252/1/artifacts in our case, we are looking for fleet_debian_10_arm64_system_integration

https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/PR-3252/1/artifact/outputs/18.119.158.69/TEST-fleet_debian_10_arm64_system_integration-a7393d5f-2022-11-30-05%3A59%3A13.json.html#collapseScenariocucumber_html_reporter0_5

@memory
Scenario Outline:Adding memory system/metrics Integration to a Policy
2 1 1
Given an agent is deployed to Fleet with "tar" installer 256026m 1s 960ms
And the agent is listed in Fleet as "online" 2158565m 34s 942ms
When the policy is updated to have "system/metrics" set to "memory" 2910479m 56s 13ms [+ Show Error](https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/PR-3252/1/artifact/outputs/18.119.158.69/TEST-fleet_debian_10_arm64_system_integration-a7393d5f-2022-11-30-05%3A59%3A13.json.html#error0_52)
package system not found in policy 3c680d10-7075-11ed-b09e-551e43e67b8d
Then "system/metrics" with "memory" metrics are present in the datastreams 0s

this pre-condition sounds weird to me And the agent is listed in Fleet as "online" 2158565m 34s 942ms

Answer 10 · 2022-12-01T13:44:01.000Z

Now I see panic: runtime error: invalid memory address or nil pointer dereference. Can it be the reason?

Answer 11 · 2022-12-01T13:44:51.000Z

Now I see panic: runtime error: invalid memory address or nil pointer dereference. Can it be the reason?

It could be, I remember that one from some time ago I reported it in some place

Answer 12 · 2022-12-01T13:56:16.000Z

[2022-11-30T06:24:23.083Z] Nov 30 06:07:28 ip-172-31-8-249 systemd[1]: Started Elastic Agent is a unified agent to observe, monitor and protect your system..
[2022-11-30T06:24:23.083Z] Nov 30 06:07:29 ip-172-31-8-249 elastic-agent[15774]: 2022-11-30T06:07:29.103Z        INFO        application/application.go:67        Detecting execution mode
[2022-11-30T06:24:23.083Z] Nov 30 06:07:29 ip-172-31-8-249 elastic-agent[15774]: 2022-11-30T06:07:29.104Z        INFO        application/application.go:76        Agent is managed locally
[2022-11-30T06:24:23.083Z] Nov 30 06:07:29 ip-172-31-8-249 elastic-agent[15774]: 2022-11-30T06:07:29.105Z        INFO        capabilities/capabilities.go:59        capabilities file not found in /opt/Elastic/Agent/capabilities.yml
[2022-11-30T06:24:23.083Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: 2022-11-30T06:07:30.412Z        INFO        [api]        api/server.go:62        Starting stats endpoint
[2022-11-30T06:24:23.083Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: 2022-11-30T06:07:30.412Z        INFO        application/local_mode.go:168        Agent is starting
[2022-11-30T06:24:23.083Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: 2022-11-30T06:07:30.412Z        INFO        application/local_mode.go:178        Agent is stopped
[2022-11-30T06:24:23.083Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: 2022-11-30T06:07:30.413Z        INFO        [api]        api/server.go:64        Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
[2022-11-30T06:24:23.083Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: panic: runtime error: invalid memory address or nil pointer dereference
[2022-11-30T06:24:23.083Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xaaaab490f78c]
[2022-11-30T06:24:23.084Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: goroutine 84 [running]:
[2022-11-30T06:24:23.084Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: google.golang.org/grpc.(*Server).Serve(0x4000192700?, {0xaaaab5e3a168?, 0x4000717260?})
[2022-11-30T06:24:23.084Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]:         /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:740 +0x3c
[2022-11-30T06:24:23.084Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/core/server.(*Server).Start.func1()
[2022-11-30T06:24:23.084Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]:         /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/server/server.go:159 +0x34
[2022-11-30T06:24:23.084Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]: created by github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/core/server.(*Server).Start
[2022-11-30T06:24:23.084Z] Nov 30 06:07:30 ip-172-31-8-249 elastic-agent[15774]:         /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/core/server/server.go:158 +0x364

Answer 13 · 2022-12-01T13:59:54.000Z

I think this is the code related
https://github.com/elastic/elastic-agent/blob/8.5/pkg/core/server/server.go#L158-L166

Answer 14 · 2022-12-01T14:11:42.000Z

Hi @cachedout

Below are the commands that I have executed to run the tests locally.

make -C .ci create-stack
make -C .ci start-elastic-stack
make -C .ci list-platforms
export NODE_LABEL=debian_11_amd64
make -C .ci create-node
cd .ci
sudo vi .e2e-tests.yaml  : [To enable the integrations tag]
cd ..
make -C .ci run-tests DEVELOPER_MODE=true TAGS="integrations"
make -C .ci show-stack
make -C .ci run-tests DEVELOPER_MODE=true TAGS="integrations"
make -C .ci start-elastic-stack
make -C .ci show-stack
make -C .ci start-elastic-stack

Answer 15 · 2022-12-01T14:39:00.000Z

make -C .ci list-platforms
export NODE_LABEL=debian_11_amd64

The test is failing in ARM64 not in AMD64

Answer 16 · 2022-12-01T15:01:35.000Z

BTW the retry has passed https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/PR-3252/2/pipeline

Answer 17 · 2022-12-01T15:10:28.000Z

Notified the elastic agent team. The found error looks dangerous

Answer 18 · 2022-12-02T11:05:20.000Z

The reason has been found. Occasional test failures are going to be fixed here: elastic/elastic-agent#1863

Answer 19 · 2022-12-08T09:20:22.000Z

Hi @cachedout

cc @pazone

We have revalidated the Fleet server CI after the fix of above issue under #33937
on the 7.17 and main branch as well.

However, Issue is still reproducible and not fixed.

Logs recorded:


TASK [Start stack without any tag to spin up the stack using Fleet's bootstrapping code] *********************************************
fatal: [3.137.217.15]: FAILED! => {"changed": true, "cmd": "TAGS='non-existing-tag' /home/admin/e2e-testing/.ci/scripts/functional-test.sh", "delta": "0:00:00.861350", "end": "2022-12-08 09:12:39.723171", "msg": "non-zero return code", "rc": 2, "start": "2022-12-08 09:12:38.861821", "stderr": "+ BASE_DIR=/home/admin/e2e-testing\n+ SUITE=fleet\n+ REPORT_PREFIX=junit\n+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin\n+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin\n+ BASE_DIR=/home/admin/e2e-testing\n+ /home/admin/e2e-testing/.ci/scripts/install-test-dependencies.sh fleet\n+ BASE_DIR=/home/admin/e2e-testing\n+ SUITE=fleet\n+ '[' -f /home/admin/e2e-testing/.ci/scripts/install-fleet-test-dependencies.sh ']'\n+ echo 'Not installing test dependencies for fleet'\n++ echo junit\n++ sed -r 's/[ @~]+//g'\n+ REPORT_PREFIX=junit\n++ date +%Y-%m-%d-%H:%M:%S\n+ SEED=2022-12-08-09:12:38\n+ REPORT=/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38\n+ echo 'REPORT=\"/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38\"'\n+ echo 'Removing previous test files in the case the workspace is reused'\n+ rm -f '/home/admin/e2e-testing/outputs/TEST-*.*'\n+ TAGS=non-existent-tag\n+ FORMAT=pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.xml\n+ make --no-print-directory -C /home/admin/e2e-testing/e2e/_suites/fleet functional-test\n# github.com/elastic/e2e-testing/e2e/_suites/fleet [github.com/elastic/e2e-testing/e2e/_suites/fleet.test]\n./integrations.go:180:5: syntax error: unexpected newline, expecting comma or }\nmake: *** [../../commons-test.mk:65: functional-test] Error 2", "stderr_lines": ["+ BASE_DIR=/home/admin/e2e-testing", "+ SUITE=fleet", "+ REPORT_PREFIX=junit", "+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin", "+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin", "+ BASE_DIR=/home/admin/e2e-testing", "+ /home/admin/e2e-testing/.ci/scripts/install-test-dependencies.sh fleet", "+ BASE_DIR=/home/admin/e2e-testing", "+ SUITE=fleet", "+ '[' -f /home/admin/e2e-testing/.ci/scripts/install-fleet-test-dependencies.sh ']'", "+ echo 'Not installing test dependencies for fleet'", "++ echo junit", "++ sed -r 's/[ @~]+//g'", "+ REPORT_PREFIX=junit", "++ date +%Y-%m-%d-%H:%M:%S", "+ SEED=2022-12-08-09:12:38", "+ REPORT=/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38", "+ echo 'REPORT=\"/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38\"'", "+ echo 'Removing previous test files in the case the workspace is reused'", "+ rm -f '/home/admin/e2e-testing/outputs/TEST-*.*'", "+ TAGS=non-existent-tag", "+ FORMAT=pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.xml", "+ make --no-print-directory -C /home/admin/e2e-testing/e2e/_suites/fleet functional-test", "# github.com/elastic/e2e-testing/e2e/_suites/fleet [github.com/elastic/e2e-testing/e2e/_suites/fleet.test]", "./integrations.go:180:5: syntax error: unexpected newline, expecting comma or }", "make: *** [../../commons-test.mk:65: functional-test] Error 2"], "stdout": "Not installing test dependencies for fleet\nRemoving previous test files in the case the workspace is reused\nOP_LOG_LEVEL=INFO \\\nOP_LOG_INCLUDE_TIMESTAMP=TRUE \\\nTIMEOUT_FACTOR=3 \\\nSTACK_VERSION= \\\nDEVELOPER_MODE=false \\\nELASTIC_APM_SERVICE_NAME=\"E2E Tests\" \\\nELASTIC_APM_CENTRAL_CONFIG=\"false\" \\\nELASTIC_APM_GLOBAL_LABELS= \\\nELASTIC_APM_SERVICE_VERSION=\"\" \\\nELASTIC_APM_ENVIRONMENT=\"local\" \\\nELASTIC_APM_SECRET_TOKEN=\"\" \\\nELASTIC_APM_SERVER_URL=\"\" \\\nBRANCH_NAME=\"\" \\\nTRACEPARENT=\"\" \\\ngo test -timeout 90m -v --godog.format=\"pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.xml\"  --godog.tags=\"non-existent-tag && ~skip\"\nFAIL\tgithub.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]", "stdout_lines": ["Not installing test dependencies for fleet", "Removing previous test files in the case the workspace is reused", "OP_LOG_LEVEL=INFO \\", "OP_LOG_INCLUDE_TIMESTAMP=TRUE \\", "TIMEOUT_FACTOR=3 \\", "STACK_VERSION= \\", "DEVELOPER_MODE=false \\", "ELASTIC_APM_SERVICE_NAME=\"E2E Tests\" \\", "ELASTIC_APM_CENTRAL_CONFIG=\"false\" \\", "ELASTIC_APM_GLOBAL_LABELS= \\", "ELASTIC_APM_SERVICE_VERSION=\"\" \\", "ELASTIC_APM_ENVIRONMENT=\"local\" \\", "ELASTIC_APM_SECRET_TOKEN=\"\" \\", "ELASTIC_APM_SERVER_URL=\"\" \\", "BRANCH_NAME=\"\" \\", "TRACEPARENT=\"\" \\", "go test -timeout 90m -v --godog.format=\"pretty,cucumber:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.json,junit:/home/admin/e2e-testing/outputs/TEST-junit-7b2afe35-2022-12-08-09:12:38.xml\"  --godog.tags=\"non-existent-tag && ~skip\"", "FAIL\tgithub.com/elastic/e2e-testing/e2e/_suites/fleet [build failed]"]}

PLAY RECAP ***************************************************************************************************************************
3.137.217.15               : ok=13   changed=6    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0

make: *** [Makefile:204: start-elastic-stack] Error 2
make: Leaving directory '/home/ubuntu/e2e-testing/.ci'

So, we are reopening it.

Please let me know if more info is required.

Answer 20 · 2022-12-12T15:12:43.000Z

Looks like the container could not start up properly. Now the snapshot version is being updated here.
I think it would be great to attach containers logs to the build. Or send the logs to kibana.

Answer 21 · 2022-12-15T19:38:15.000Z

The root cause is that the functional test does not wait for Kibana to be available. Because of that, you have to make the make -C .ci start-elastic-stack twice, the second time Kibana is ready. I am working on the fix.

Answer 22 · 2022-12-20T11:53:10.000Z

There is another issue if the docker pull takes more than 5 min the token is not valid

Answer 23 · 2022-12-30T09:20:01.000Z

I just merged #3285 with a bunch of fixes, now the stack provision is more reliable.
Some of the improvements are:

Node provision and configuration are more reliable.
Now we wait for Kibana to be up and running before launching the Fleet server
The main env vars related to the test are shown in the logs
The logs verbosity was reduced
Some logs messages are moved to the proper log level
Docker, Kubernetes, and Elastic Agent logs are grabbed after the execution of the test suit, then are uploaded to artifacts in the CI
The artifact now have a name associated with the test that generated those artifacts
Some Ansible updates
Fix teardown resources
Allow overwriting the Elastic Stack configuration with environment variables:
- ELASTICSEARCH_URL: URL to Elasticsearch deployment
- KIBANA_URL: URL to Kibana deployment
- FLEET_URL: URL to Fleet server deployment
- ELASTICSEARCH_PASSWORD: Elastic user password
- KIBANA_PASSWORD: Elastic user password

Answer 24 · 2022-12-30T13:08:37.000Z

Hi @kuisathaverat

Thanks for fixing this out.

We have revalidated this issue and found it not reproducible.

We are able to spin up kibana along with running fleet server under Fleet test suite.

Screenshot for reference:

Tested on Git branch origin/8.6

However, Fleet server agent logs are currently not available under Agent logs, I am working upon this to validate and will share feedback if reproduced consistently.

Thanks!!