Ceph - howto, rbd, lvm, cluster

Install ceph

wget -q -O- https://raw.github.com/ceph/ceph/master/keys/release.asc | apt-key add -
echo deb http://ceph.com/debian/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
apt-get update && apt-get install ceph

Video to ceph intro

https://www.youtube.com/watch?v=UXcZ2bnnGZg
http://www.youtube.com/watch?v=BBOBHMvKfyc&feature=g-high

Rebooting node stops everything / Set number of replicas across all nodes

Make sure that the min replica count is set to nodes-1.

ceph osd pool set <poolname> min_size 1

Then the remaing node[s] will start up with just 1 node if everything else is down.

Keep in mind this can potentially make stuff ugly as there are no replicas now.

More info here: http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10481

Add disks (OSD) or entire nodes

Prepare the disk as usual (partition or entire disk) - format with filesystem of choosing. Add to fstab and mount. Add to /etc/ceph/ceph.conf and replicate the new conf to the other nodes.

Start the disk, I'm assuming we've added osd.12 on ceph1 here.

## Auth stuff to make sure that the OSD is accepted into the cluser:
mkdir /srv/ceph/osd12
ceph-osd -i 12 --mkfs --mkkey
ceph auth add osd.12 osd 'allow *' mon 'allow rwx' -i /etc/ceph/keyring.osd.12

## Create disk and start it
ceph osd create osd.12
/etc/init.d/ceph start osd.12

## Add it to the cluster and allow replicated based on CRUSH map.
ceph osd crush set 12 osd.12 1.0 pool=default rack=unknownrack host=ceph1

In the line above, if you exchange the pool/rack/host you can place your disk/node where you want.

If you add a new host entry, it will be the same as adding a new node (with the disk).

Check that is in the right place with:

ceph osd tree

More info here:

• http://ceph.com/docs/master/rados/operations/pools/
• http://ceph.com/docs/master/rados/operations/add-or-rm-osds/

Delete pools/OSD

Make sure you have the right disk, run

ceph osd tree

to get an overview.

Delete a OSD

ceph osd crush remove osd.12

Add Monitor node/service

Install ceph, add keys, ceph.conf, host files and prepare a storage for containing the maps.

Then add the monitor into the system (To keep quorum, keep either 1 or 3+ - not 2). Examples are adding monitor mon.21 with ip 192.168.0.68

cd /tmp; mkdir add_monitor; cd add_monitor
ceph auth get mon. -o key
> exported keyring for mon.

ceph mon getmap -o map
> got latest monmap

ceph-mon -i 21 --mkfs --monmap map --keyring key 
> ceph-mon: created monfs at /srv/ceph/mon21 for mon.21

ceph mon add 21 192.168.0.68
> port defaulted to 6789added mon.21 at 192.168.0.68:6789/0

/etc/init.d/ceph start mon.21

Add the info to the ceph.conf file:

[mon]
...
[mon.21]
    host = ceph2-mon
    mon addr = 192.168.0.68:6789
...

Replicating from OSD-based to replication across hosts in a ceph cluster

More info here: http://jcftang.github.com/2012/09/06/going-from-replicating-across-osds-to-replicating-across-hosts-in-a-ceph-cluster/

Replication - see current level pr. OSD

ceph osd dump

CRUSH maps

Redistributing, [de]assembling and finetuning; more info here:

http://hpc.admin-magazine.com/Articles/RADOS-and-Ceph-Part-2

KVM - add disk

Pr host:

    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol="rbd" name="test/disk2-qemu-5g:rbd_cache=1">
        <host name='192.168.0.67' port='6789'/>
        <host name='192.168.0.68' port='6789'/>
      </source>
      <auth username='admin' type='ceph'>
        <secret type='ceph' uuid='7a91dc24-b072-43c4-98fb-4b2415322b0f'/>
      </auth>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

Pr pool:

<pool type='rbd'>
   <name>rbd</name>
   <uuid>f959641f-f518-4505-9e85-17d994e2a399</uuid>
   <source>
     <host name='192.168.0.67' port='6789'/>
     <host name='192.168.0.68' port='6789'/>
     <host name='192.168.0.69' port='6789'/>
     <name>test</name>
     <auth username='admin' type='ceph'>
       <secret type='ceph' uuid='7a91dc24-b072-43c4-98fb-4b2415322b0f'/>
     </auth>
   </source>
</pool>

KVM - add secret/auth for use with ceph

Create a secret.xml file:

<secret ephemeral='no' private='no'>
   <uuid>7a91dc24-b072-43c4-98fb-4b2415322b0f</uuid>
   <usage type='ceph'>
     <name>admin</name>
   </usage>
</secret>

Use it:

virsh secret-define secret.xml
virsh secret-set-value 7a91dc24-b072-43c4-98fb-4b2415322b0f AQDAD8JQOLS9IxAAbox00eOmlM1h5ZLGPxHGHw==

The last key is the key from your /etc/ceph/keyring.admin

cat /etc/ceph/keyring.admin 
[client.admin]
        key = AQDAD8JQOLS9IxAAbox00eOmlM1h5ZLGPxHGHw==

Online resizing of KVM images (rbd)

Resize the desired block-image (here going from 30GB -> 40GB)

qemu-img resize -f rbd rbd:sata/disk3 40G
> Image resized.

Find the attached target device:

virsh domblklist rbd-test
> Target     Source
> ------------------------------------------------
> vdb        sata/disk2-qemu-5g:rbd_cache=1
> vdc        sata/disk3:rbd_cache=1
> hdc        -

Then use virsh to tell the guest that the disk has a new size:

virsh blockresize --domain rbd-test --path "vdc" --size 40G
> Block device 'vdc' is resized

Check raw rbd info

rbd --pool sata info disk3
> rbd image 'disk3':
>        size 40960 MB in 10240 objects
>        order 22 (4096 KB objects)
>        block_name_prefix: rb.0.13fb.23353e97
>        parent:  (pool -1)