Try-Ceph : What is it

Try-Ceph requires bare mininum work to get your FIRST CEPH CLUSTER running for your testing purpose and learning purpose. Anyone who wants to have a quick look at CEPH , this is the fastes way to get to their first Ceph cluster.

Try-Ceph features includes

Quickest way to get your first test Ceph cluster up and running.
Automated Ceph installation and configuration within the box.
Installs and configure ceph-dash : A Ceph graphical dashboard for monitoring.
Installs and configure Object Storage which is accessible via S3 or Swift API
User can define number of Monitor and OSD nodes for the Ceph cluster.
It uses Vagrant --> Ansible --> Ceph --> Ceph-dash

Requirements

Host machine OS - Mac OSX or any LINUX distribution ( tested on CentOS , Ubuntu )
Virtuliaztion platform - Oracle VirtualBox ( tested ) or VMWare workstation
Disk Space on host machine - >5GB ( 1 mon & 1 OSD node with 2 OSD's )
Host configuration management - Ansible

How to get started with Try-Ceph

Prerequiste : Make sure your host machine should comply with the requirements mentioned above. Before "vagrant up" operation sometimes ' git config --global url."https://".insteadOf git:// ' is needed to solve Ceph installation packages.

STEP - 1 : $ git clone https://github.com/ksingh7/Try-Ceph.git

$ git clone https://github.com/ksingh7/Try-Ceph.git
Cloning into 'Try-Ceph'...
remote: Counting objects: 127, done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 127 (delta 19), reused 105 (delta 11)
Receiving objects: 100% (127/127), 29.23 KiB | 0 bytes/s, done.
Resolving deltas: 100% (19/19), done.
Checking connectivity... done.
$

STEP - 1.1 : Install Ansible on the host machine , see the below instructions for your OS

Ubuntu : https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-ubuntu-14-04 CentOS : http://www.tecmint.com/install-and-configure-ansible-automation-tool-in-linux/

STEP - 2 : $ cd Try-Ceph

NOTE : By default vagrant will provision 1 MON and 3 OSD node , if you need more MON / OSD nodes , please change the NMONS and NOSDS parameteraunder  Vagrantfile.

STEP - 3 : $ vagrant up

# vagrant up
Bringing machine 'mon0' up with 'virtualbox' provider...
Bringing machine 'osd0' up with 'virtualbox' provider...
Bringing machine 'osd1' up with 'virtualbox' provider...
Bringing machine 'osd2' up with 'virtualbox' provider...
==> mon0: Importing base box 'ceph/centos7'...
==> mon0: Matching MAC address for NAT networking...
==> mon0: Checking if box 'ceph/centos7' is up to date...
==> mon0: Setting the name of the VM: Try-Ceph_mon0_1448744827979_6033
==> mon0: Clearing any previously set network interfaces...
==> mon0: Preparing network interfaces based on configuration...
    mon0: Adapter 1: nat
    mon0: Adapter 2: hostonly
==> mon0: Forwarding ports...
    mon0: 22 => 2222 (adapter 1)
==> mon0: Running 'pre-boot' VM customizations...
==> mon0: Booting VM...
==> mon0: Waiting for machine to boot. This may take a few minutes...
    mon0: SSH address: 127.0.0.1:2222
    mon0: SSH username: vagrant
    mon0: SSH auth method: private key
    mon0: Warning: Connection timeout. Retrying...
    mon0: Warning: Remote connection disconnect. Retrying...
==> mon0: Machine booted and ready!
==> mon0: Checking for guest additions in VM...
==> mon0: Setting hostname...
==> mon0: Configuring and enabling network interfaces...
==> mon0: Mounting shared folders...
    mon0: /vagrant => /root/git/Try-Ceph
==> osd0: Importing base box 'ceph/centos7'...
==> osd0: Matching MAC address for NAT networking...
==> osd0: Checking if box 'ceph/centos7' is up to date...
==> osd0: Setting the name of the VM: Try-Ceph_osd0_1448744879956_64498
==> osd0: Fixed port collision for 22 => 2222. Now on port 2200.
==> osd0: Clearing any previously set network interfaces...
==> osd0: Preparing network interfaces based on configuration...
    osd0: Adapter 1: nat
    osd0: Adapter 2: hostonly
    osd0: Adapter 3: hostonly
==> osd0: Forwarding ports...
    osd0: 22 => 2200 (adapter 1)
==> osd0: Running 'pre-boot' VM customizations...
==> osd0: Booting VM...
==> osd0: Waiting for machine to boot. This may take a few minutes...
    osd0: SSH address: 127.0.0.1:2200
    osd0: SSH username: vagrant
    osd0: SSH auth method: private key
    osd0: Warning: Connection timeout. Retrying...
==> osd0: Machine booted and ready!
==> osd0: Checking for guest additions in VM...
==> osd0: Setting hostname...
==> osd0: Configuring and enabling network interfaces...
==> osd0: Mounting shared folders...
    osd0: /vagrant => /root/git/Try-Ceph
==> osd1: Importing base box 'ceph/centos7'...
==> osd1: Matching MAC address for NAT networking...
==> osd1: Checking if box 'ceph/centos7' is up to date...
==> osd1: Setting the name of the VM: Try-Ceph_osd1_1448744941270_1671
==> osd1: Fixed port collision for 22 => 2222. Now on port 2201.
==> osd1: Clearing any previously set network interfaces...
==> osd1: Preparing network interfaces based on configuration...
    osd1: Adapter 1: nat
    osd1: Adapter 2: hostonly
    osd1: Adapter 3: hostonly
==> osd1: Forwarding ports...
    osd1: 22 => 2201 (adapter 1)
==> osd1: Running 'pre-boot' VM customizations...
==> osd1: Booting VM...
==> osd1: Waiting for machine to boot. This may take a few minutes...
    osd1: SSH address: 127.0.0.1:2201
    osd1: SSH username: vagrant
    osd1: SSH auth method: private key
    osd1: Warning: Connection timeout. Retrying...
==> osd1: Machine booted and ready!
==> osd1: Checking for guest additions in VM...
==> osd1: Setting hostname...
==> osd1: Configuring and enabling network interfaces...
==> osd1: Mounting shared folders...
    osd1: /vagrant => /root/git/Try-Ceph
==> osd2: Importing base box 'ceph/centos7'...
==> osd2: Matching MAC address for NAT networking...
==> osd2: Checking if box 'ceph/centos7' is up to date...
==> osd2: Setting the name of the VM: Try-Ceph_osd2_1448745001796_20960
==> osd2: Fixed port collision for 22 => 2222. Now on port 2202.
==> osd2: Clearing any previously set network interfaces...
==> osd2: Preparing network interfaces based on configuration...
    osd2: Adapter 1: nat
    osd2: Adapter 2: hostonly
    osd2: Adapter 3: hostonly
==> osd2: Forwarding ports...
    osd2: 22 => 2202 (adapter 1)
==> osd2: Running 'pre-boot' VM customizations...
==> osd2: Booting VM...
==> osd2: Waiting for machine to boot. This may take a few minutes...
    osd2: SSH address: 127.0.0.1:2202
    osd2: SSH username: vagrant
    osd2: SSH auth method: private key
    osd2: Warning: Connection timeout. Retrying...
==> osd2: Machine booted and ready!
==> osd2: Checking for guest additions in VM...
==> osd2: Setting hostname...
==> osd2: Configuring and enabling network interfaces...
==> osd2: Mounting shared folders...
    osd2: /vagrant => /root/git/Try-Ceph
==> osd2: Running provisioner: ansible...
PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/root/.vagrant.d/insecure_private_key --user=vagrant --connection=ssh --limit='all' --inventory-file=/root/git/Try-Ceph/.vagrant/provisioners/ansible/inventory --extra-vars={"fsid":"10c95f01-2dd2-4863-affa-60c4eafcd8d2","monitor_secret":"AQBNTxZRWId7JxAA/Ac4ToR7ZfNdOGDSToGHpA=="} site.yml

PLAY [mons;osds] **************************************************************

GATHERING FACTS ***************************************************************
ok: [osd2]
ok: [osd1]
ok: [mon0]
ok: [osd0]

TASK: [common | Copying Ceph key] *********************************************
changed: [mon0]
changed: [osd0]
changed: [osd2]
changed: [osd1]

TASK: [common | Importing Ceph GPG  key] **************************************
changed: [osd2]
changed: [osd1]
changed: [mon0]
changed: [osd0]

TASK: [common | Copying EPEL key] *********************************************
changed: [mon0]
changed: [osd0]
changed: [osd1]
changed: [osd2]

TASK: [common | Importing EPEL GPG  key] **************************************
changed: [mon0]
changed: [osd0]
changed: [osd2]
changed: [osd1]

TASK: [common | Adding EPEL repo] *********************************************
changed: [mon0]
changed: [osd1]
changed: [osd2]
changed: [osd0]

TASK: [common | Adding ceph to YUM repo] **************************************
changed: [osd0]
changed: [mon0]
changed: [osd2]
changed: [osd1]

TASK: [common | Disable firewalld] ********************************************
changed: [mon0]
changed: [osd2]
changed: [osd1]
changed: [osd0]

TASK: [common | Stop firewalld] ***********************************************
changed: [mon0]
changed: [osd1]
changed: [osd0]
changed: [osd2]

TASK: [common | Creating directories required by ceph [/var/lib/ceph]] ********
changed: [mon0]
changed: [osd1]
changed: [osd0]
changed: [osd2]

TASK: [common | Creating directories required by ceph [/var/lib/ceph/{{ item }}]] ***
changed: [mon0] => (item=tmp)
changed: [osd0] => (item=tmp)
changed: [osd2] => (item=tmp)
changed: [osd1] => (item=tmp)
changed: [mon0] => (item=mon)
changed: [osd2] => (item=mon)
changed: [osd1] => (item=mon)
changed: [osd0] => (item=mon)
changed: [mon0] => (item=bootstrap-osd)
changed: [osd1] => (item=bootstrap-osd)
changed: [osd2] => (item=bootstrap-osd)
changed: [osd0] => (item=bootstrap-osd)

TASK: [common | Creating directories required by ceph [/var/log/ceph]] ********
changed: [mon0]
changed: [osd0]
changed: [osd1]
changed: [osd2]

TASK: [common | Creating directories required by ceph [/etc/ceph]] ************
changed: [mon0]
changed: [osd0]
changed: [osd2]
changed: [osd1]

TASK: [common | Installing Ceph packages] *************************************
changed: [osd2]
changed: [osd0]
changed: [osd1]
changed: [mon0]

TASK: [common | Generating Ceph configuration file] ***************************
changed: [osd2]
changed: [osd1]
changed: [osd0]
changed: [mon0]

PLAY [mons] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [mon0]

TASK: [mons | Creating monitor initial keyring] *******************************
changed: [mon0]

TASK: [mons | Setting initial monitor key permissions] ************************
ok: [mon0]

TASK: [mons | Creating monitor directory] *************************************
changed: [mon0]

TASK: [mons | Creating administrator keyring , Generating a client.admin user and adding the user to the keyring] ***
changed: [mon0]

TASK: [mons | Adding the client.admin key to the ceph.mon.keyring.] ***********
changed: [mon0]

TASK: [mons | Generating a monitor map using the hostname,host IP address and the FSID. Save it as /tmp/monmap] ***
changed: [mon0]

TASK: [mons | Populating the monitor daemon with the monitor map and keyring.] ***
changed: [mon0]

TASK: [mons | Start and add that the monitor service to the init sequence] ****
changed: [mon0]

TASK: [mons | Adding the new monitor to the cluster] **************************
changed: [mon0]

TASK: [mons | Copy keys to the ansible server under fetch dirctory] ***********
changed: [mon0] => (item=/var/lib/ceph/bootstrap-osd/ceph.keyring)
changed: [mon0] => (item=/var/lib/ceph/mon/ceph-mon0/keyring)
ok: [mon0] => (item=/etc/ceph/ceph.bootstrap-osd.keyring)
ok: [mon0] => (item=/etc/ceph/ceph.mon.keyring)
changed: [mon0] => (item=/etc/ceph/ceph.conf)
changed: [mon0] => (item=/etc/ceph/ceph.client.admin.keyring)

TASK: [mons | Cloning ceph-dash from github] **********************************
changed: [mon0]

TASK: [mons | Upgrading jinja2 for ceph-dash] *********************************
changed: [mon0]

TASK: [mons | Update ceph-dash.py] ********************************************
changed: [mon0]

TASK: [mons | Executing ceph-dash] ********************************************
<job 137746701184.5009> finished on mon0

PLAY [osds] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [osd2]
ok: [osd1]
ok: [osd0]

TASK: [osds | Creating directories required by ceph] **************************
ok: [osd1] => (item=tmp)
ok: [osd0] => (item=tmp)
ok: [osd2] => (item=tmp)
ok: [osd1] => (item=mon)
ok: [osd0] => (item=mon)
ok: [osd2] => (item=mon)
ok: [osd1] => (item=mds)
ok: [osd2] => (item=mds)
ok: [osd0] => (item=mds)
ok: [osd1] => (item=bootstrap-osd)
ok: [osd0] => (item=bootstrap-osd)
ok: [osd2] => (item=bootstrap-osd)
ok: [osd1] => (item=bootstrap-mds)
ok: [osd0] => (item=bootstrap-mds)
ok: [osd2] => (item=bootstrap-mds)

TASK: [osds | Gathering keys from MON node to OSD node [ STEP-1 ]] ************
changed: [osd1]
changed: [osd0]
changed: [osd2]

TASK: [osds | Copying OSD bootstrap key to OSD node] **************************
changed: [osd0]
changed: [osd2]
changed: [osd1]

TASK: [osds | Zapping OSD] ****************************************************
changed: [osd0] => (item=/dev/sdb)
changed: [osd1] => (item=/dev/sdb)
changed: [osd2] => (item=/dev/sdb)
changed: [osd1] => (item=/dev/sdc)
changed: [osd0] => (item=/dev/sdc)
changed: [osd2] => (item=/dev/sdc)

TASK: [osds | Prepare OSD disk(s)] ********************************************
changed: [osd0] => (item=/dev/sdb)
changed: [osd1] => (item=/dev/sdb)
changed: [osd2] => (item=/dev/sdb)
changed: [osd0] => (item=/dev/sdc)
changed: [osd2] => (item=/dev/sdc)
changed: [osd1] => (item=/dev/sdc)

TASK: [osds | Activating  OSD] ************************************************
changed: [osd0] => (item=/dev/sdb)
changed: [osd0] => (item=/dev/sdc)
changed: [osd2] => (item=/dev/sdb)
changed: [osd1] => (item=/dev/sdb)
changed: [osd2] => (item=/dev/sdc)
changed: [osd1] => (item=/dev/sdc)

TASK: [osds | Start and add that the OSD service to the init sequence] ********
changed: [osd1]
changed: [osd2]
changed: [osd0]

PLAY [mons] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [mon0]

TASK: [rgw | Install Dnsmasq] *************************************************
ok: [mon0]

TASK: [rgw | Set configuration file] ******************************************
changed: [mon0]

TASK: [rgw | Set resolve.conf file] *******************************************
changed: [mon0]

TASK: [rgw | Make sure Dnsmasq is running] ************************************
changed: [mon0]

TASK: [rgw | Install RGW] *****************************************************
changed: [mon0]

TASK: [rgw | Creating RGW keyring] ********************************************
changed: [mon0]

TASK: [rgw | Setting RGW keyring persmissions] ********************************
ok: [mon0]

TASK: [rgw | Generate key for RGW instance] ***********************************
changed: [mon0]

TASK: [rgw | Add capabilities to RGW key] *************************************
changed: [mon0]

TASK: [rgw | Add RGW key to cluster] ******************************************
changed: [mon0]

TASK: [rgw | Updating hosts file] *********************************************
changed: [mon0]

TASK: [rgw | Start RGW] *******************************************************
changed: [mon0]

TASK: [rgw | Creating Rados Gateway user for S3 Access] ***********************
changed: [mon0]

TASK: [rgw | Creating Rados Gateway Swift user] *******************************
changed: [mon0]

TASK: [rgw | Adding Secret for Radow Gateway swift user] **********************
changed: [mon0]

TASK: [rgw | Installing s3cmd] ************************************************
changed: [mon0]

TASK: [rgw | adding s3cmd configuration file] *********************************
changed: [mon0]

NOTIFIED: [rgw | restart dnsmasq] *********************************************
changed: [mon0]

PLAY RECAP ********************************************************************
mon0                       : ok=50   changed=42   unreachable=0    failed=0
osd0                       : ok=23   changed=20   unreachable=0    failed=0
osd1                       : ok=23   changed=20   unreachable=0    failed=0
osd2                       : ok=23   changed=20   unreachable=0    failed=0

#

STEP - 4 : Check Ceph cluster status

$ vagrant ssh mon0 -c "sudo ceph -s"
    cluster 10c95f01-2dd2-4863-affa-60c4eafcd8d2
     health HEALTH_OK
     monmap e1: 1 mons at {mon0=192.168.101.10:6789/0}, election epoch 2, quorum 0 mon0
     osdmap e17: 6 osds: 6 up, 6 in
      pgmap v25: 192 pgs, 3 pools, 0 bytes data, 0 objects
            207 MB used, 65126 MB / 65333 MB avail
                 192 active+clean
Connection to 127.0.0.1 closed.
$

STEP - 5 : To check status of your cluster using Ceph dashboard. Point your host browser to http://192.168.101.10:5000

STEP - 6 : To access object storage via S3

Node mon0 is preconfigured as Ceph Rados Gateway which can be accessed via s3cmd tool

    # vagrant ssh mon0 
    $ sudo su -
    # s3cmd ls                          # List S3 bucket
    # s3cmd mb s3://first-bucket        # Make S3 bucket
    # s3cmd ls                          # List S3 bucket

STEP - 7 : To access object storage via SWIFT

Install swift cli tools to a node that have access to mon0 machine ( 192.168.101.10 )
Finally access object storage via Swift Call

# swift -A http://192.168.101.10/auth/1.0 -U demo:swift -K 'SwiftSecretKey' list

Destro Try-Ceph setup

STEP - 1 : Check status of your vagrant instances $ vagrant status

STEP - 2 : Halt vagrant instances $ vagrant halt mon0 osd0 osd1 osd2

STEP - 3 : Destroy vagrant instances $ vagrant destroy -f mon0 osd0 osd1 osd2

Troubleshooting

If you encounter any problem during any step of ansible run , you can manually start ansible provisioning at the step it got stoppoed. Using the below command

# ansible-playbook site.yml -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory --private-key=~/.vagrant.d/insecure_private_key -u vagrant --start-at-task="Installing Ceph packages"

Video Demo

Click Here For Video Demo