Installing CEPH Cluster

What is CEPH? 

Ceph stores data as objects within logical storage pools. Using the CRUSH algorithm, Ceph calculates which placement group should contain the object, and further calculates which Ceph OSD Daemon should store the placement group. The CRUSH algorithm enables the Ceph Storage Cluster to scale, re-balance, and recover dynamically. 

Requirements  

System used for our CEPH example

• CentOS 7.8.2003
• EPEL CentOS Repository
• NTP Service - Time Synchronization
• Create ceph user account to manage cluster
• Configure firewall
• Disable SELinux
• Create ssh-keygen and copy SSH ID to OSD nodes
• Install CentOS 7.8.2003 on all nodes, including monitor node (We are using CentOS minimal CD/DVD ISO)
• Install your favorite editor 
• Create sudo access for ceph user profile
• Enable SSH logins at startup for all nodes, including mon node
• Create a separate drive partitions without LVM
• Plan CEPH cluster with Puppet by collecting commands and information for all configuration files

Repositories

Repository configuration for our setup. 

# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

# CentOS-Ceph-Luminous.repo
#
# Please see http://wiki.centos.org/SpecialInterestGroup/Storage for more
# information

[centos-ceph-luminous]
name=CentOS-$releasever - Ceph Luminous
baseurl=http://mirror.centos.org/$contentdir/$releasever/storage/$basearch/ceph-luminous/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage

[centos-ceph-luminous-test]
name=CentOS-$releasever - Ceph Luminous Testing
baseurl=http://buildlogs.centos.org/centos/$releasever/storage/$basearch/ceph-luminous/
gpgcheck=0
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage

[centos-ceph-luminous-source]
name=CentOS-$releasever - Ceph Luminous Source
baseurl=http://vault.centos.org/$contentdir/$releasever/storage/Source/ceph-luminous/
gpgcheck=0
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
# CentOS-CR.repo
#
# The Continuous Release ( CR )  repository contains rpms that are due in the next
# release for a specific CentOS Version ( eg. next release in CentOS-7 ); these rpms
# are far less tested, with no integration checking or update path testing having
# taken place. They are still built from the upstream sources, but might not map
# to an exact upstream distro release.
#
# These packages are made available soon after they are built, for people willing
# to test their environments, provide feedback on content for the next release, and
# for people looking for early-access to next release content.
#
# The CR repo is shipped in a disabled state by default; its important that users
# understand the implications of turning this on.
#
# NOTE: We do not use a mirrorlist for the CR repos, to ensure content is available
#       to everyone as soon as possible, and not need to wait for the external
#       mirror network to seed first. However, many local mirrors will carry CR repos
#       and if desired you can use one of these local mirrors by editing the baseurl
#       line in the repo config below.
#

[cr]
name=CentOS-$releasever - cr
baseurl=http://mirror.centos.org/centos/$releasever/cr/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0
# CentOS-Debug.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#

# All debug packages from all the various CentOS-7 releases
# are merged into a single repo, split by BaseArch
#
# Note: packages in the debuginfo repo are currently not signed
#

[base-debuginfo]
name=CentOS-7 - Debuginfo
baseurl=http://debuginfo.centos.org/7/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Debug-7
enabled=0
#
#CentOS-fasttrack.repo

[fasttrack]
name=CentOS-7 - fasttrack
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=fasttrack&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/fasttrack/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

# CentOS-Media.repo
#
#  This repo can be used with mounted DVD media, verify the mount point for
#  CentOS-7.  You can use this repo and yum to install items directly off the
#  DVD ISO that we release.
#
# To use this repo, put in your DVD and use it with the other repos too:
#  yum --enablerepo=c7-media [command]
#
# or for ONLY the media repo, do this:
#
#  yum --disablerepo=\* --enablerepo=c7-media [command]

[c7-media]
name=CentOS-$releasever - Media
baseurl=file:///media/CentOS/
        file:///media/cdrom/
        file:///media/cdrecorder/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

# CentOS-OpenStack-queens.repo
#
# Please see http://wiki.centos.org/SpecialInterestGroup/Cloud for more
# information

[centos-openstack-queens]
name=CentOS-7 - OpenStack queens
baseurl=http://mirror.centos.org/$contentdir/$releasever/cloud/$basearch/openstack-queens/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
exclude=sip,PyQt4

[centos-openstack-queens-test]
name=CentOS-7 - OpenStack queens Testing
baseurl=https://buildlogs.centos.org/centos/7/cloud/$basearch/openstack-queens/
gpgcheck=0
enabled=0
exclude=sip,PyQt4

[centos-openstack-queens-debuginfo]
name=CentOS-7 - OpenStack queens - Debug
baseurl=http://debuginfo.centos.org/centos/7/cloud/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
exclude=sip,PyQt4

[centos-openstack-queens-source]
name=CentOS-7 - OpenStack queens - Source
baseurl=http://vault.centos.org/centos/7/cloud/Source/openstack-queens/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
exclude=sip,PyQt4

[rdo-trunk-queens-tested]
name=OpenStack queens Trunk Tested
baseurl=https://trunk.rdoproject.org/centos7-queens/current-passed-ci/
gpgcheck=0
enabled=0
# CentOS-QEMU-EV.repo
#
# Please see http://wiki.centos.org/SpecialInterestGroup/Virtualization for more
# information

[centos-qemu-ev]
name=CentOS-$releasever - QEMU EV
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=virt-kvm-common
#baseurl=http://mirror.centos.org/$contentdir/$releasever/virt/$basearch/kvm-common/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization

[centos-qemu-ev-test]
name=CentOS-$releasever - QEMU EV Testing
baseurl=http://buildlogs.centos.org/centos/$releasever/virt/$basearch/kvm-common/
gpgcheck=0
enabled=0

# CentOS-Sources.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base-source]
name=CentOS-$releasever - Base Sources
baseurl=http://vault.centos.org/centos/$releasever/os/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates-source]
name=CentOS-$releasever - Updates Sources
baseurl=http://vault.centos.org/centos/$releasever/updates/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras-source]
name=CentOS-$releasever - Extras Sources
baseurl=http://vault.centos.org/centos/$releasever/extras/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus-source]
name=CentOS-$releasever - Plus Sources
baseurl=http://vault.centos.org/centos/$releasever/centosplus/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
# CentOS-Storage.repo
#
# Please see http://wiki.centos.org/SpecialInterestGroup/Storage for more
# information

[centos-storage-debuginfo]
name=CentOS-$releasever - Storage SIG - debuginfo
baseurl=http://debuginfo.centos.org/$contentdir/$releasever/storage/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage

# CentOS Vault contains rpms from older releases in the CentOS-7
# tree.

# C7.0.1406
[C7.0.1406-base]
name=CentOS-7.0.1406 - Base
baseurl=http://vault.centos.org/7.0.1406/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-updates]
name=CentOS-7.0.1406 - Updates
baseurl=http://vault.centos.org/7.0.1406/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-extras]
name=CentOS-7.0.1406 - Extras
baseurl=http://vault.centos.org/7.0.1406/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-centosplus]
name=CentOS-7.0.1406 - CentOSPlus
baseurl=http://vault.centos.org/7.0.1406/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-fasttrack]
name=CentOS-7.0.1406 - Fasttrack
baseurl=http://vault.centos.org/7.0.1406/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.1.1503
[C7.1.1503-base]
name=CentOS-7.1.1503 - Base
baseurl=http://vault.centos.org/7.1.1503/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-updates]
name=CentOS-7.1.1503 - Updates
baseurl=http://vault.centos.org/7.1.1503/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-extras]
name=CentOS-7.1.1503 - Extras
baseurl=http://vault.centos.org/7.1.1503/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-centosplus]
name=CentOS-7.1.1503 - CentOSPlus
baseurl=http://vault.centos.org/7.1.1503/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-fasttrack]
name=CentOS-7.1.1503 - Fasttrack
baseurl=http://vault.centos.org/7.1.1503/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.2.1511
[C7.2.1511-base]
name=CentOS-7.2.1511 - Base
baseurl=http://vault.centos.org/7.2.1511/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-updates]
name=CentOS-7.2.1511 - Updates
baseurl=http://vault.centos.org/7.2.1511/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-extras]
name=CentOS-7.2.1511 - Extras
baseurl=http://vault.centos.org/7.2.1511/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-centosplus]
name=CentOS-7.2.1511 - CentOSPlus
baseurl=http://vault.centos.org/7.2.1511/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-fasttrack]
name=CentOS-7.2.1511 - Fasttrack
baseurl=http://vault.centos.org/7.2.1511/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.3.1611
[C7.3.1611-base]
name=CentOS-7.3.1611 - Base
baseurl=http://vault.centos.org/7.3.1611/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-updates]
name=CentOS-7.3.1611 - Updates
baseurl=http://vault.centos.org/7.3.1611/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-extras]
name=CentOS-7.3.1611 - Extras
baseurl=http://vault.centos.org/7.3.1611/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-centosplus]
name=CentOS-7.3.1611 - CentOSPlus
baseurl=http://vault.centos.org/7.3.1611/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-fasttrack]
name=CentOS-7.3.1611 - Fasttrack
baseurl=http://vault.centos.org/7.3.1611/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.4.1708
[C7.4.1708-base]
name=CentOS-7.4.1708 - Base
baseurl=http://vault.centos.org/7.4.1708/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-updates]
name=CentOS-7.4.1708 - Updates
baseurl=http://vault.centos.org/7.4.1708/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-extras]
name=CentOS-7.4.1708 - Extras
baseurl=http://vault.centos.org/7.4.1708/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-centosplus]
name=CentOS-7.4.1708 - CentOSPlus
baseurl=http://vault.centos.org/7.4.1708/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-fasttrack]
name=CentOS-7.4.1708 - Fasttrack
baseurl=http://vault.centos.org/7.4.1708/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.5.1804
[C7.5.1804-base]
name=CentOS-7.5.1804 - Base
baseurl=http://vault.centos.org/7.5.1804/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.5.1804-updates]
name=CentOS-7.5.1804 - Updates
baseurl=http://vault.centos.org/7.5.1804/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.5.1804-extras]
name=CentOS-7.5.1804 - Extras
baseurl=http://vault.centos.org/7.5.1804/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.5.1804-centosplus]
name=CentOS-7.5.1804 - CentOSPlus
baseurl=http://vault.centos.org/7.5.1804/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.5.1804-fasttrack]
name=CentOS-7.5.1804 - Fasttrack
baseurl=http://vault.centos.org/7.5.1804/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.6.1810
[C7.6.1810-base]
name=CentOS-7.6.1810 - Base
baseurl=http://vault.centos.org/7.6.1810/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.6.1810-updates]
name=CentOS-7.6.1810 - Updates
baseurl=http://vault.centos.org/7.6.1810/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.6.1810-extras]
name=CentOS-7.6.1810 - Extras
baseurl=http://vault.centos.org/7.6.1810/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.6.1810-centosplus]
name=CentOS-7.6.1810 - CentOSPlus
baseurl=http://vault.centos.org/7.6.1810/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.6.1810-fasttrack]
name=CentOS-7.6.1810 - Fasttrack
baseurl=http://vault.centos.org/7.6.1810/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.7.1908
[C7.7.1908-base]
name=CentOS-7.7.1908 - Base
baseurl=http://vault.centos.org/7.7.1908/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.7.1908-updates]
name=CentOS-7.7.1908 - Updates
baseurl=http://vault.centos.org/7.7.1908/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.7.1908-extras]
name=CentOS-7.7.1908 - Extras
baseurl=http://vault.centos.org/7.7.1908/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.7.1908-centosplus]
name=CentOS-7.7.1908 - CentOSPlus
baseurl=http://vault.centos.org/7.7.1908/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.7.1908-fasttrack]
name=CentOS-7.7.1908 - Fasttrack
baseurl=http://vault.centos.org/7.7.1908/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0
[centos-kernel]
name=CentOS LTS Kernels for $basearch
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=kernel&infra=$infra
#baseurl=http://mirror.centos.org/altarch/7/kernel/$basearch/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centos-kernel-experimental]
name=CentOS Experimental Kernels for $basearch
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=experimental&infra=$infra
#baseurl=http://mirror.centos.org/altarch/7/experimental/$basearch/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[epel-testing]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-testing-debuginfo]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-debug-epel7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-testing-source]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-source-epel7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
 

Configuring Chrony

We will configure chrony to retrieve updates from the central node ceph-mon NTP service IP/Domain name, then update the OSD nodes from ceph-mon so all nodes are synchronized to the main monitor node of CEPH monitor node. 

Install CentOS 

Installing CentOS 7.8.2003

1. Configure hostname
2. Configure static address or MAP mapping through DHCP (ifcfg-enp0s3 or nmtui)
3. Configure extra users for sudo operations - won't be used to manage CEPH
4. Create user CEPH
5. Configure /etc/hosts:

• ceph-mon
• ceph-osd1
• ceph-osd2
• ceph-osd3
  
  

   6. Disabled SELinux

   7. Install EPEL from Fedora Project website

   8. Upgrade system to current patches and fixes

   9.Create passwordless to each machine

Create Second Partition 

Since we added a secondary drive we have our OSD disks on each CEPH node as /dev/sdb and the main operating system on /dev/sda.  We will now configure our second drive /dev/sdb as /dev/sdb1 for each CEPH node.

 Output:

Now that we have extra space for our OSD drives we can now configure the secondary drive for each of the OSD nodes (ceph-osd1,2,3).

Install and Configure OpenSSH Server 

Make sure that openssh server has been installed and started. 

 Start and enable the sshd daemon service

 Configuring Hostname and IP

In our setup the configuration for hostname and IP will be the following 

1. ceph-mon - 192.168.2.70
2. ceph-osd1 - 192.168.2.71
3. ceph-osd2 - 192.168.2.72
4. ceph-osd3 - 192.168.2.73

Hostname Configuration 

Configure hostname during the installation or by typing below commands.

On the main node which will hold our monitor node called ceph-mon, rename it to ceph-mon if the name hasn't been configured yet:

The same for all OSD{1..3} nodes by typing. 

Install an Editor 

Configure IP Address 

Configure the IP address manually or by using nmtui. We will use nmtui since it's simple and easy. 

Verify Connectivity 

Ping each node to each node and make sure you edit the /etc/hosts file to reflect node information. Please read further for more information on howto. 

Create ceph User 

Create a user called ceph on every ceph node. 

 Now create a password for the user: ceph

Create Sudo Configuration 

 Create a file, add the below properties into the file which allows the ceph user account to run root commands without a password.

 Add below lines to the ceph file we created.

ceph ALL = (root) NOPASSWD:ALL

Now we need to change the appropriate properties to the ceph file we just created.

Generate SSH on Monitor CEPH Node 

Generate the SSH keys on the ceph-mon node. 

Configure HOSTS file 

Go to each ceph node including the master ceph-mon node and add the below information to reflect all OSD nodes and MON node 

Add below information 

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.2.70 ceph-mon node0
192.168.2.71 ceph-osd1 node1
192.168.2.72 ceph-osd2 node2
192.168.2.73 ceph-osd3 node3

The reason we added the node{0..3} is for easy manageability. 

Firewall Configuration 

If you have a firewall installed, we configure it to allow ceph communications. 

Copy All SSH Keys to Nodes

From the ceph-mon node we will run a script to deploy to all OSD nodes 

We also want to add the names ceph-osd{1..3}, so type command below. 

The following line will prompt for (y)es and the password for each node which is 3 nodes in all. You might also want to add a key to the monitor node, which it would be to it's self.

Example of SSH-COPY-ID 

Disabling SELinux 

Let's disable selinux.

Change line "SELINUX=" to show like below example. 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

Disable SELinux 

Install Editor on All Nodes

Installation of EPEL Repository 

 Install EPEL on each node of the ceph cluster nodes.

Easier and faster way to install using the below command line from the ceph-mon node.

OR 

This command installs EPEL on all nodes of the CEPH cluster 

Upgrade ALL Nodes

Command to upgrade (fixes/updates) all nodes including the monitor master node for CEPH. 

 Installing Chrony

Starting chronyd daemon on all machines at the same time.

Configure the hosts file on all nodes, including the ceph-mon node.

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.2.70  ceph-mon node0
192.168.2.71  ceph-osd1 node1
192.168.2.72  ceph-osd2 node2
192.168.2.73  ceph-osd3 node3 

Install ceph Monitor Node Packages 

 Login into the ceph-mon to install required packages.

Install CEPH Monitor Node with ceph-deploy 

Login to the ceph-mon node with ceph. 

 Create ceph-deploy Directory

Change into ceph-deploy directory.

Create Initial Monitor 

Edit the ceph.conf 

Add the below lines to the end of ceph.conf 

osd pool default size = 2
mon_pg_warn_max_per_osd = 0

 Install ceph in ceph Monitor Node

 Add initial monitor and gather keys

Install ceph to OSD Node with ceph-deploy 

Activate ceph-osd node data disks. Login as ceph account and change into the ceph-deploy directory.

Activating the OSD disks 

After the initialization has completed with the cephfs system creation we need to check the health of the ceph cluster by typing below commands 

CEPH Status & Health of Cluster 

Checking cluster health after creating all the OSD nodes 

Print Ceph OSD Map 

Please stay tuned to the next article explaining how to integrate with OpenStack and how to deploy with Puppet.

Article created by AJA