Failed to start X-Ray Daemon

Question

Failed to start X-Ray Daemon

Closed this issue 6 years ago · 13 comments

Since moving to v2.1.2 the X-Ray Daemon will no longer start on our Ubuntu servers.

Files pulled with:
wget https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-2.x.deb

And installed with:
sudo dpkg -i aws-xray-daemon-2.x.deb

However, upon running:
sudo service xray status

I see the following errors:

● xray.service - AWS X-Ray Daemon
Loaded: loaded (/lib/systemd/system/xray.service; disabled; vendor preset: enabled)
Active: inactive (dead)

May 17 20:46:10 ip-10-2-112-10 systemd[1]: xray.service: Unit entered failed state.
May 17 20:46:10 ip-10-2-112-10 systemd[1]: xray.service: Failed with result 'exit-code'.
May 17 20:46:10 ip-10-2-112-10 systemd[1]: xray.service: Service hold-off time over, scheduling restart.
May 17 20:46:10 ip-10-2-112-10 systemd[1]: Stopped AWS X-Ray Daemon.
May 17 20:46:10 ip-10-2-112-10 systemd[1]: xray.service: Start request repeated too quickly.
May 17 20:46:10 ip-10-2-112-10 systemd[1]: Failed to start AWS X-Ray Daemon.
May 17 20:46:18 ip-10-2-112-10 systemd[1]: [/lib/systemd/system/xray.service:18] Unknown lvalue 'LogsDirectory' in section 'Service'
May 17 20:46:18 ip-10-2-112-10 systemd[1]: [/lib/systemd/system/xray.service:19] Unknown lvalue 'LogsDirectoryMode' in section 'Service'
May 17 20:46:18 ip-10-2-112-10 systemd[1]: [/lib/systemd/system/xray.service:20] Unknown lvalue 'ConfigurationDirectory' in section 'Service'
May 17 20:46:18 ip-10-2-112-10 systemd[1]: [/lib/systemd/system/xray.service:21] Unknown lvalue 'ConfigurationDirectoryMode' in section 'Service'

I've tried to remove those lines from the /lib/systemd/system/xray.service file but it still will not start due to a group issue:

● xray.service - AWS X-Ray Daemon
Loaded: loaded (/lib/systemd/system/xray.service; disabled; vendor preset: enabled)
Active: inactive (dead)

May 17 21:05:51 ip-10-2-112-10 systemd[1]: xray.service: Service hold-off time over, scheduling restart.
May 17 21:05:51 ip-10-2-112-10 systemd[1]: Stopped AWS X-Ray Daemon.
May 17 21:05:51 ip-10-2-112-10 systemd[1]: Started AWS X-Ray Daemon.
May 17 21:05:51 ip-10-2-112-10 systemd[1]: xray.service: Main process exited, code=exited, status=216/GROUP
May 17 21:05:51 ip-10-2-112-10 systemd[1]: xray.service: Unit entered failed state.
May 17 21:05:51 ip-10-2-112-10 systemd[1]: xray.service: Failed with result 'exit-code'.
May 17 21:05:52 ip-10-2-112-10 systemd[1]: xray.service: Service hold-off time over, scheduling restart.
May 17 21:05:52 ip-10-2-112-10 systemd[1]: Stopped AWS X-Ray Daemon.
May 17 21:05:52 ip-10-2-112-10 systemd[1]: xray.service: Start request repeated too quickly.
May 17 21:05:52 ip-10-2-112-10 systemd[1]: Failed to start AWS X-Ray Daemon.

Answer 1 · 2018-05-17T20:15:25.000Z

Hi @smuffff ,
Thank you for letting us know the issue. We are investigating it.
Could you please let us know your Ubuntu OS version ?

Thanks,
Yogi

Answer 2 · 2018-05-17T21:34:49.000Z

Sorry for any convenience. We will post an update as soon as we can. In the mean time you can pin v2.1.1 https://s3.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-2.1.1.deb to get unblocked.

Answer 3 · 2018-05-18T01:05:39.000Z

Hi @smuffff ,
We tested daemon releases V2.1.2 and V2.1.1 on Ubuntu 16.04 Amazon EC2 instances.

We executed the following script as user data when launching the EC2 instances:
For V2.1.2 daemon version instance :

#!/bin/bash

wget https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-2.x.deb
sudo dpkg -i aws-xray-daemon-2.x.deb

For V2.1.1 daemon version instance:

#!/bin/bash

wget https://s3.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-2.1.1.deb
sudo dpkg -i aws-xray-daemon-2.1.1.deb

We could successfully launch AWS X-Ray daemon in both versions and could not reproduce the issue you are facing.

Can you please provide us:

The version of Ubuntu OS you are using and whether the instance is on-premise or an Amazon EC2 instance?
If the X-Ray daemon installation/start command is part of server launching script and if it is possible to share. Feel free to redact any sensitive information.

Thanks,
Yogi

Answer 4 · 2018-05-18T08:59:04.000Z

Hi Yogi

We are using EC2 Ubuntu 16.04, however these have had their OS hardened which could be causing the issue I guess - not sure where I'd look to find more info on this.
We run this code as part of the instances User Data script:

wget https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-2.x.deb
dpkg -i aws-xray-daemon-2.x.deb

This works if we pin it to 2.1.1

Thanks

Adam

Answer 5 · 2018-05-18T22:56:57.000Z

Hi @smuffff ,

Thank you for the response.
We want to deep dive into the issue and following questions will help us:

Is it possible to let us know the hardened Ubuntu OS version/name and place to get it, in order to reproduce this issue? (is the AMI available in the Amazon Marketplace)
Do you see daemon logs located at /var/log/xray/xray.log, if yes, can you provide us?
For V2.1.1, can you let us know the group under which the xray user is?
Command : groups xray
For V2.1.2 can you remove Groups from /lib/systemd/system/xray.service file and try deploying X-Ray daemon?

Best,
Yogi

Answer 6 · 2018-05-20T03:16:55.000Z

Hi @smuffff ,

In addition to @yogiraj07's request, could you also kindly provide the output of the following command: cat /etc/login.defs | grep USERGROUPS_ENAB

I suspect the issue is caused by the lines [1] and [2]

[1] explicitly requested the service to be run under group xray, which usually should've been created by [2]. However the default behavior (creating a new group with the same name as the user) might've been disabled in your hardening process.

- Liu

Answer 7 · 2019-02-27T06:03:24.000Z

@yogiraj07 I have configured x-ray daemon on aws EC2 as mentioned here https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ec2.html. I could see the xray service running on the ec2 instance. Below are the x-ray logs,
**$ /usr/bin/xray
2019-02-27T05:57:33Z [Info] Initializing AWS X-Ray daemon 2.1.3
2019-02-27T05:57:33Z [Error] listen udp 127.0.0.1:2000: bind: address already in use

$ cat /var/log/xray/xray.log
2019-01-30T15:21:10Z [Info] Initializing AWS X-Ray daemon 2.1.3
2019-01-30T15:21:10Z [Info] Using buffer memory limit of 160 MB
2019-01-30T15:21:10Z [Info] 2560 segment buffers allocated
2019-01-30T15:21:10Z [Info] Using region: ap-south-1**

But still i don't see any traces in the x-ray console. EC2 instance has xray:* permission. Am i missing any configuration here.

Appreciate your valuable help on this!!!

Answer 8 · 2019-02-27T20:02:25.000Z

Looking at the timestamps, I see only entries for Jan 30th and Feb 27th.
Is this expected? I'm wondering if the Daemon is actually starting up at all.

I believe it is not. yogiraj07 can correct me but if the address is already bound it will fail to start.

Can you identify the process running on udp port 2000? It may be an old process of the daemon stuck in the background.

lsof -i udp:2000

Answer 9 · 2019-02-27T20:14:25.000Z

@sandygvs and @awssandra , taking this discussion on issue 23. Yes the daemon will fail to start if address is already bound.

Answer 10 · 2022-06-12T13:13:19.000Z

Hi @yogiraj07 / @awssandra / @smuffff ,

For X-Ray, is the following logs means that AWS X-Ray Daemon is running well in my EC2 server?

To test if X-Ray Daemon working fine, how should I do some simple test in my EC2 server?

Logs:

2022-06-12T08:46:40Z [Info] Initializing AWS X-Ray daemon 3.3.3
2022-06-12T08:46:40Z [Info] Using buffer memory limit of 19 MB
2022-06-12T08:46:40Z [Info] 304 segment buffers allocated
2022-06-12T08:46:40Z [Info] Using region: ap-southeast-1
2022-06-12T08:46:40Z [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.ap-southeast-1.amazonaws.com
2022-06-12T08:46:40Z [Info] Starting proxy http server on 127.0.0.1:2000

Screenshot:

Thanks in advance!

Answer 11 · 2022-06-12T19:13:17.000Z

Hi @jerrychong25 - yes that means the daemon is running correctly. To test it, you can spin up one of our simple sample apps, e.g. https://github.com/aws-samples/aws-xray-sdk-node-sample to make sure the segments it creates are sent to X-Ray.

Answer 12 · 2022-06-22T07:00:09.000Z

Hi @jerrychong25 - yes that means the daemon is running correctly. To test it, you can spin up one of our simple sample apps, e.g. https://github.com/aws-samples/aws-xray-sdk-node-sample to make sure the segments it creates are sent to X-Ray.

Hi @willarmiros ,

Noted with thanks!

Answer 13 · 2024-12-20T05:59:12.000Z

failing to start the daemon on RHEL8 although it's installed and enabled.

sh-5.1$ sudo systemctl start xray.service
Job for xray.service failed because the control process exited with error code.
sh-5.1$ systemctl status xray.service
● xray.service - AWS X-Ray Daemon
   Loaded: loaded (/etc/systemd/system/xray.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2024-12-20 13:56:34 +08; 3s ago
  Process: 2294 ExecStart=/usr/bin/xray -f /var/log/xray/xray.log (code=exited, status=216/GROUP)
 Main PID: 2294 (code=exited, status=216/GROUP)