使用的是fc12操作系統(tǒng)，內(nèi)核是2.6.33，先升級到2.6.36。起初在fc11上升級，make modules時遇到如下錯誤 INSTALL /lib/firmware/bnx2/bnx2-rv2p-06-5.0.0.j3.fw
DEPMOD 2.6.36
/bin/sh: line 1: 15321 Segmentation fault /sbin/depmod -ae -F System.map 2.6.36
最后找了個fc12的系統(tǒng)，升級。
1. 下載ceph-0.29.1.tar.gz，解壓
cd ceph.0.29.1/
./autogen.sh
./configure
checking for NSS... yes
configure: using nss for cryptography
checking for FCGX_Init in -lfcgi... no
checking for fuse_main in -lfuse... no
configure: error: no FUSE found (use --without-fuse to disable)
=====> 安裝fuse,fuse-devel
checking for fuse_main in -lfuse... yes
checking for malloc in -ltcmalloc... no
configure: error: in `/home/ceph-0.29.1':
configure: error: no tcmalloc found (use --without-tcmalloc to disable)
====> yum install google-perftools-devel
hecking for FCGX_Init in -lfcgi... no
checking for fuse_main in -lfuse... yes
checking for malloc in -ltcmalloc... yes
checking jni.h usability... no
checking jni.h presence... no
checking for jni.h... no
checking for LIBEDIT... no
configure: error: in `/home/ceph-0.29.1':
configure: error: No usable version of libedit found.
====>FC12的LIBEDIT是2.11，yum install libedit-devel
checking for jni.h... no
checking for LIBEDIT... yes
checking atomic_ops.h usability... no
checking atomic_ops.h presence... no
checking for atomic_ops.h... no
configure: error: in `/home/ceph-0.29.1':
configure: error: no libatomic-ops found (use --without-libatomic-ops to disable)
====>yum install libatomic_ops-devel
末了，
#make && make install ；這個過程有點慢
#cp ./src/sample.* /usr/local/etc/ceph/ ##拷貝ceph的配置文件，后面要修改
#mv /usr/local/etc/ceph/sample.ceph.conf /usr/local/etc/ceph/ceph.conf
#mv /usr/local/etc/ceph/sample.fetch_config /usr/local/etc/ceph/fetch_config
#修改/usr/local/etc/ceph/ceph.conf和/usr/local/etc/ceph/fetch_config文件，按照網(wǎng)上的要求，可以適當(dāng)?shù)囟啻蜷_log以了解ceph內(nèi)部實現(xiàn)。
#cp ./src/init-ceph /etc/init.d/ceph
#mkdir /var/log/ceph; ##存放log，現(xiàn)在ceph自己還不自動建這個目錄
#mkdir /data ##存儲相應(yīng)信息，下面配置會用到
2. 打算用2臺MDS，2臺OSD，1臺AS+MON，所以上述的機(jī)器復(fù)制4臺。
在5臺機(jī)器上modporbe ceph
3. 兩臺OSD，加虛擬硬盤，格成btrfs格式
[root@ceph-osd59 ~]# mkfs.btrfs /dev/sdb
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 2.10GB
Btrfs Btrfs v0.19
4. 然后在AS+MON上，創(chuàng)建文件系統(tǒng)，并啟動
[root@ceph-as-mon70 ~]# mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
temp dir is /tmp/mkcephfs.ujjZwfgtYH
preparing monmap in /tmp/mkcephfs.ujjZwfgtYH/monmap
/usr/local/bin/monmaptool --create --clobber --add alpha 10.10.37.70:6789 --print /tmp/mkcephfs.ujjZwfgtYH/monmap
/usr/local/bin/monmaptool: monmap file /tmp/mkcephfs.ujjZwfgtYH/monmap
/usr/local/bin/monmaptool: generated fsid f2ca9063-bee3-a227-76be-bd7bdb38b666
epoch 1
fsid f2ca9063-bee3-a227-76be-bd7bdb38b666
last_changed 2011-06-27 17:47:01.317472
created 2011-06-27 17:47:01.317472
0: 10.10.37.70:6789/0 mon.alpha
/usr/local/bin/monmaptool: writing epoch 1 to /tmp/mkcephfs.ujjZwfgtYH/monmap (1 monitors)
=== osd.0 ===
pushing conf and monmap to ceph-osd58:/tmp/mkfs.ceph.2961
umount: /data/osd.0: not mounted
umount: /dev/sdb: not mounted
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 2.10GB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
failed to read /dev/fd0
failed to read /dev/sr0
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
** WARNING: 'filestore btrfs snap' is enabled (for safe transactions,
rollback), but btrfs does not support the SNAP_CREATE_V2 ioctl
(added in Linux 2.6.37). Expect slow btrfs sync/commit
performance.
2011-06-27 17:47:03.336988 7f4fb110c720 created object store /data/osd.0 journal /data/osd.0/journal for osd0 fsid f2ca9063-bee3-a227-76be-bd7bdb38b666
creating private key for osd.0 keyring /tmp/mkfs.ceph.2961/keyring.osd.0
creating /tmp/mkfs.ceph.2961/keyring.osd.0
collecting osd.0 key
=== osd.1 ===
pushing conf and monmap to ceph-osd59:/tmp/mkfs.ceph.2961
umount: /data/osd.1: not mounted
umount: /dev/sdb: not mounted
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdb
nodesize 4096 leafsize 4096 sectorsize 4096 size 2.10GB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
failed to read /dev/fd0
failed to read /dev/sr0
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
** WARNING: 'filestore btrfs snap' is enabled (for safe transactions,
rollback), but btrfs does not support the SNAP_CREATE_V2 ioctl
(added in Linux 2.6.37). Expect slow btrfs sync/commit
performance.
2011-06-27 17:47:05.706919 7fa46eb33720 created object store /data/osd.1 journal /data/osd.1/journal for osd1 fsid f2ca9063-bee3-a227-76be-bd7bdb38b666
creating private key for osd.1 keyring /tmp/mkfs.ceph.2961/keyring.osd.1
creating /tmp/mkfs.ceph.2961/keyring.osd.1
collecting osd.1 key
=== mds.alpha ===
pushing conf and monmap to cephmds56:/tmp/mkfs.ceph.2961
creating private key for mds.alpha keyring /data/keyring.mds.alpha
creating /data/keyring.mds.alpha
collecting mds.alpha key
=== mds.beta ===
pushing conf and monmap to cephmds57:/tmp/mkfs.ceph.2961
creating private key for mds.beta keyring /data/keyring.mds.beta
creating /data/keyring.mds.beta
collecting mds.beta key
Building generic osdmap
highest numbered osd in /tmp/mkcephfs.ujjZwfgtYH/conf is osd.1
num osd = 2
/usr/local/bin/osdmaptool: osdmap file '/tmp/mkcephfs.ujjZwfgtYH/osdmap'
/usr/local/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.ujjZwfgtYH/osdmap
Generating admin key at /tmp/mkcephfs.ujjZwfgtYH/keyring.admin
creating /tmp/mkcephfs.ujjZwfgtYH/keyring.admin
Building initial monitor keyring
added entity mds.alpha auth auth(auid = 18446744073709551615 key=AQCYUQhOKEfJHBAA7dBdwvgmvpYmdChwmG0j2w== with 0 caps)
added entity mds.beta auth auth(auid = 18446744073709551615 key=AQCcUQhOiJzeLhAAktfC2QEvanlhHqOYcluUeA== with 0 caps)
added entity osd.0 auth auth(auid = 18446744073709551615 key=AQCXUQhO6P3zFhAA8eseLUW9RSRWkm0ar+tvpQ== with 0 caps)
added entity osd.1 auth auth(auid = 18446744073709551615 key=AQCZUQhOyPddLRAA61EgalinvHHe9tOU1Cif+Q== with 0 caps)
=== mon.alpha ===
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
/usr/local/bin/cmon: created monfs at /data/mon.alpha for mon.alpha
placing client.admin keyring in /etc/ceph/keyring
[root@ceph-as-mon70 ~]# /etc/init.d/ceph -a start
=== mon.alpha ===
Starting Ceph mon.alpha on ceph-as-mon70...
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
starting mon.alpha rank 0 at 10.10.37.70:6789/0 mon_data /data/mon.alpha fsid f2ca9063-bee3-a227-76be-bd7bdb38b666
=== mds.alpha ===
Starting Ceph mds.alpha on cephmds56...
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
starting mds.alpha at 0.0.0.0:6800/1873
=== mds.beta ===
Starting Ceph mds.beta on cephmds57...
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
starting mds.beta at 0.0.0.0:6800/1799
=== osd.0 ===
Mounting Btrfs on ceph-osd58:/data/osd.0
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
Scanning for Btrfs filesystems
failed to read /dev/fd0
failed to read /dev/sr0
Starting Ceph osd.0 on ceph-osd58...
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
starting osd0 at 0.0.0.0:6800/2190 osd_data /data/osd.0 /data/osd.0/journal
=== osd.1 ===
Mounting Btrfs on ceph-osd59:/data/osd.1
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
Scanning for Btrfs filesystems
failed to read /dev/fd0
failed to read /dev/sr0
Starting Ceph osd.1 on ceph-osd59...
** WARNING: Ceph is still under development. Any feedback can be directed **
** at or http://ceph.newdream.net/. **
starting osd1 at 0.0.0.0:6800/2191 osd_data /data/osd.1 /data/osd.1/journal
[root@ceph-as-mon70 ~]#
5. 在AS+MON機(jī)器上mount ceph文件系統(tǒng)
mkdir /ceph
mount.ceph 10.10.37.70:/ /ceph
【出現(xiàn)問題】
其中在mount的時候，出現(xiàn)error
> # dmesg -c
> libceph: no secret set (for auth_x protocol)
> libceph: error -22 on auth protocol 2 init
后來加上
[root@ceph-as-mon70 ceph-0.29.1]# cat /etc/ceph/keyring
[client.admin]
key = AQCVDghOOMZONRAANdh11UcePt5qxB+NMYDM3g==
auid = 18446744073709551615
[root@ceph-as-mon70 ceph-0.29.1]# mount.ceph 10.10.37.70:6789:/ /ceph -v -o name=admin,secret=AQCVDghOOMZONRAANdh11UcePt5qxB+NMYDM3g==
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
parsing options: name=admin,secret=AQCVDghOOMZONRAANdh11UcePt5qxB+NMYDM3g==
error adding secret to kernel client.admin : No such devicemount error 5 = Input/output error
也還是出錯，
最后通過關(guān)掉SELINUX解決，修改/etc/selinux/config
SELINUX=disabled
#SELINUX=enforcing
重啟后[root@cephmds56 ~]# sestatus
SELinux status: disabled
