Setup 1: Using Ceph Ansible

Current Setup

Spin up 9 instances with the appropriate rules using AWS Console

AMI: Ubuntu Server 18.04 LTS (HVM), SSD Volume Type
Instance Type: t2.medium
Instance Details: For learning purpose, set number of instances to 9
Add Storage: For learning sake, add more volumes and set the sizes to 50GB (Sandbox limit)
Security Group: SSH, ICMP, TCP 8080, TCP 9283 (For learning sake, I allowed All Traffic so I dont have to worry about the ports to open)

Name the instances

Admin: admin
Managers: mgr
Monitors: mon
OSDs: osd1, osd2, osd3
MDSs: mds
Grafana Server: gfs
Client: client

Admin configuration

We need ssh-keys to be able to communicate with the servers.

We can generate one or simply use what we generated from the AWS console.

Console Keys

AWS generated a key pair and gave us the private key but stored the public key on the servers, specifically in the location ~/.ssh/authorized_keys

We will use the authorized_keys as the public key and the private key as what we have on our local machine.

Run the commands below

cp ~/.ssh/authorized_keys ~/.ssh/id_rsa.pub

Display the content of the private on your local machine

cat keyPair.pem

Copy what you see from beginning to the end.. eg. below

Head onto the admin

nano ~/.ssh/id_rsa

Past the content there. Save the file and exit

Change the file permissions

chmod 400 ~/.ssh/id_rsa

The next thing is to add all hosts to the admin host file

sudo nano /etc/hosts

172.31.23.65 mgr
172.31.24.49 mon
172.31.21.226 osd1
172.31.23.115 osd2
172.31.18.39 osd3
172.31.23.73 gfs
172.31.20.221 mds
172.31.18.1 client

Get the right ip address for each server

Try to connect to servers

ssh mgr
exit
ssh mon
exit
ssh osd1
exit
ssh osd2
exit
ssh osd3
exit
ssh gfs
exit
ssh mds
exit
ssh client
exit

You are seeing ubuntu because ubuntu is my username on all the servers.

Ceph Configuration setup on the admin

Let's first install ansible. Ansible would help us automate the installation and configuration of ceph on all our nodes.

Add the package repository

sudo add-apt-repository ppa:ansible/ansible

Install ansible now:

sudo apt update
sudo apt install ansible -y

Install another dependency package

sudo apt install python-netaddr -y

Adding configuration files

Clone the ceph ansible repo

cd ~
git clone https://github.com/ceph/ceph-ansible.git
cd ceph-ansible
git checkout stable-5.0

Copy and paste the content of the following files. Read all YML files

nano group_vars/all.yml
nano group_vars/clients.yml
nano group_vars/mdss.yml
nano group_vars/mgrs.yml
nano group_vars/mons.yml
nano group_vars/osds.yml

For the osds.yml, take not of the devices added:

#osds.yml
devices:
  - /dev/xvda
  - /dev/xvdb
  - /dev/xvdc
  - /dev/xvdd
  - /dev/xvde
#  - /dev/sdb
#  - /dev/sdc
#  - /dev/sdd
#  - /dev/sde

For the all.yml file, take note of the ff:

#all.yml
monitor_interface: eth0
cluster_network: 172.31.0.0/16
radosgw_interface: eth0

Get the lists of hosts

nano hosts

Copy the content below into it

#Ceph admin user for ssh and sudo
[all:vars]
ansible_ssh_user=ubuntu
ansible_become=true
ansible_become_method=sudo
ansible_become_user=root

[mgrs]
mgr

[mons]
mon

[mdss]
mds

[rgws]
mon

[osds]
osd1
osd2
osd3

[grafana-server]
gfs

[clients]
client

Let's ping all the hosts using ansible:

ansible all -m ping -i hosts

Output:

client | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
mon | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
osd2 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
mds | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
gfs | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
mgr | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
osd3 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
osd1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Get the site.yml file

cp site.yml.sample site.yml

Run the playbook

ansible-playbook -i hosts site.yml

Dashboard

ssh mon

Verify the dashboard url

sudo ceph mgr services

Output

{
    "dashboard": "https://mgr:8080/",
    "prometheus": "http://mgr:9283/"
}

When all is done, and works successfully with no errors, you can locate the dashboard running on the manager...

http://mgr:8080

Replace mgr in the url with the public ip address of the monitor. eg. http://34.56.78.345:8080

admin
p@ssw0rd

Orchestrator

Enabling the orchestrator will allow have other functionalities on the dashboard.

ssh mon
sudo ceph mgr module enable test_orchestrator
sudo ceph orch set backend test_orchestrator
sudo ceph orch status

Kalkulus1/ceph-playbook