GlusterFS学习手记02-cluster/unify的学习,及负载均衡的实现

本文作者:深夜的蚊子
本文链接:http://wenzizone.cn/?p=9
版权所有。转载时请以链接形式注明作者和原始出处及本声明,谢谢

上 次的学习已经对glusterfs这个系统有了初步的认识,这次继续,上次只是简单的实现了一个类似NFS的配置,但这个并不是glusterfs出现的 主要目的,而且也不是我们应用glusterfs的主要目的,我们要是实现的是分布式存储,另外一个最简单的想法就是,如何能把多台server目前空余 的硬盘空间利用起来。这其实就是glusterfs最主要的功能之一,多存储空间的聚合。

实现目的:3存储节点,1clinet,3存储节点使用round-robin的方式进行存储
试验环境:vmware6.0 linux2个,分别server:192.168.211.128 client:192.168.211.129

软件安装这里不在概述
先来看看server端的配置
首先在home目录下建立4个用于共享出来的文件夹,其中一个是用于namespace的

mkdir -p /home/{dir1,dir2,dir3,dir4}
chmod 1777 /home/dir[1-4]

然后看下3个glusterfs-server的配置文件
cat /etc/glusterfs/server1.vol

### Export volume “brick” with the contents of “/home/export” directory.
volume brick
type storage/posix # POSIX FS translator
option directory /home/dir1 # Export this directory
end-volume

### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server
option bind-address 192.168.211.128 # Default is to listen on all interfaces
option listen-port 6996 # Default is 6996
subvolumes brick
option auth.addr.brick.allow * # Allow access to “brick” volume
end-volume

cat /etc/glusterfs/server2.vol

### Export volume “brick” with the contents of “/home/export” directory.
volume brick
type storage/posix # POSIX FS translator
option directory /home/dir2 # Export this directory
end-volume

### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server
option bind-address 192.168.211.128 # Default is to listen on all interfaces
option listen-port 6997 # Default is 6996
subvolumes brick
option auth.addr.brick.allow * # Allow access to “brick” volume
end-volume

cat /etc/glusterfs/server3.vol

### Export volume “brick” with the contents of “/home/export” directory.
volume brick
type storage/posix # POSIX FS translator
option directory /home/dir3 # Export this directory
end-volume

### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server
option bind-address 192.168.211.128 # Default is to listen on all interfaces
option listen-port 6998 # Default is 6996
subvolumes brick
option auth.addr.brick.allow * # Allow access to “brick” volume
end-volume

cat /etc/glusterfs/server4.vol

### Export volume “brick” with the contents of “/home/export” directory.
volume brick
type storage/posix # POSIX FS translator
option directory /home/dir4 # Export this directory
end-volume

### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server
option bind-address 192.168.211.128 # Default is to listen on all interfaces
option listen-port 6999 # Default is 6996
subvolumes brick
option auth.addr.brick.allow * # Allow access to “brick” volume
end-volume

接下来看下client的配置
cat /etc/glusterfs/client.vol

### Add client feature and attach to remote subvolume
volume client1
type protocol/client
option transport-type tcp/client
option remote-host 192.168.211.128 # IP address of the remote brick
option remote-port 6996 # default server port is 6996
option remote-subvolume brick # name of the remote volume
end-volume

volume client2
type protocol/client
option transport-type tcp/client
option remote-host 192.168.211.128
option remote-port 6997
option remote-subvolume brick
end-volume

volume client3
type protocol/client
option transport-type tcp/client
option remote-host 192.168.211.128
option remote-port 6998
option remote-subvolume brick
end-volume

volume namespacenode
type protocol/client
option transport-type tcp
option remote-host 192.168.211.128
option remote-port 6999
option remote-subvolume brick
end-volume

volume bricks
type cluster/unify
subvolumes client1 client2 client3
option scheduler rr
option namespace namespcenode
end-volume

### Add writeback feature
volume writeback
type performance/write-behind
option block-size 1MB
option cache-size 2MB
option flush-behind off
subvolumes bricks
end-volume

### Add readahead feature
volume readahead
type performance/read-ahead
option page-size 1MB # unit in bytes
option page-count 2 # cache per file = (page-count x page-size)
subvolumes writeback
end-volume

这样所有准备工作就做完了,接下来启动服务
server端

glusterfsd -f /etc/glusterfs/server1.vol
glusterfsd -f /etc/glusterfs/server2.vol
glusterfsd -f /etc/glusterfs/server3.vol
glusterfsd -f /etc/glusterfs/server4.vol
如果启动没有报错,可以执行
ps fax|grep gluseterfs进行查看
1762 tty6 Ss+ 0:00 /sbin/mingetty tty6
1858 ? Ssl 0:00 glusterfsd -f ./server1.vol
1861 ? Ssl 0:00 glusterfsd -f ./server2.vol
1864 ? Ssl 0:00 glusterfsd -f ./server3.vol
1867 ? Ssl 0:00 glusterfsd -f ./server4.vol
当然还可以通过端口进行查看
netstat -ln
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.168.211.128:6996 0.0.0.0:* LISTEN
tcp 0 0 192.168.211.128:6997 0.0.0.0:* LISTEN
tcp 0 0 192.168.211.128:6998 0.0.0.0:* LISTEN
tcp 0 0 192.168.211.128:6999 0.0.0.0:* LISTEN

到此,服务器端启动完毕
然后启动client端

modprobe fuse 先挂着fuse模块
glusterfs -l /tmp/glusterfs.log -f /etc/glusterfs/client.vol /mnt
执行完毕后可以使用df -h进行查看是否mount成功,如果成功结果如下
[root@contos5-1-4 glusterfs]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.1G 2.5G 4.3G 37% /
/dev/sda1 190M 11M 170M 7% /boot
tmpfs 125M 0 125M 0% /dev/shm
glusterfs 22G 8.7G 12G 43% /mnt
如果没有如上显示,则说明挂载没有成功,可以查看tmp下建立的日志进行排错

接下来就准备进行测试工作了
测试目的很简单,就是对round-robin进行测试,主要使用touch命令

在client端的/mnt目录下
touch {1,2,3,4,5,6,7,8,9}
然后执行ls查看结果如下
[root@contos5-1-4 mnt]# ls
1 2 3 4 5 6 7 8 9
现在在到server端去查看
cd到/home目录
执行ls *进行查看,结果如下
[root@contos5-1-1 home]# ls *
dir1:
1 4 7

dir2:
2 5 8

dir3:
3 6 9

dir4:
1 2 3 4 5 6 7 8 9

由上面看到,9个新的文件是依次创建到了dir1 dir2 dir3中,dir4就是我们配置的namespace,用于交换空间

到此,我的试验就算完成了,而且试验目的也达成了
但到此我却发现了几个问题
1,交换空间也就是namespace需要设置多大,是应该每个存储空间之和还是和一个空间大小一样就行
2,如果其中一个节点down机,上面的数据如何处理

对于以上几个问题,我会再之后的学习中进行研究,并实现glusterfs的其他功能,对于其中的参数下次一并讨论吧

GlusterFS学习手记01-初步了解及简单的配置应用

本文作者:深夜的蚊子
本文链接:http://wenzizone.cn/?p=8
版权所有。转载时请以链接形式注明作者和原始出处及本声明,谢谢

对于分布式存储已经被越来越多的公司所应用,对于这方面的学习自然不能忽略,之前就也看过相关的内容,像红帽的GFS,google的googlefs,开源的现在也越来越多,像Fastdfs,ZFS,Lustre,Hadoop,GlusterFS。

大概的也都接触了一下,Fastdfs号称小巧,速度快占用资源小,但据我了解他是提供api来进行操作的,而目前我还是希望可以像NFS一样,直接mount来使用,并且提供高的可用性,和文件的容错功能,一路扫下来,瞄准了glusterfs。

Glusterfs是一个具有可以扩展到几个PB数量级的集群文件系统。它可以把多个不同类型的存储块通过Infiniband RDMA或者TCP/IP汇聚成一个大的并行网络文件系统。

http://www.gluster.org可以了解到更多并可以下载到最新的安装包,glusterfs使用了fuse,fuse可以到http://fuse.sourceforge.net/进行下载

首先是fuse的安装

./configure –enable-dependency-tracking –enable-kernel-module –enable-lib –enable-util
make
make install

如果没有什么错误,就是安装完毕了,接下来是glusterfs的安装

tar zxvf glusterfs-2.0.0rc1.tar.gz
./configure
make
make install

如果没有报错就是安装完毕了
然后执行ldconfig
执行完毕后执行
glusterfs –version 可以看到

glusterfs 2.0.0rc1 built on Feb 10 2009 11:39:40
Repository revision: glusterfs–mainline–3.0–patch-844
Copyright (c) 2006, 2007, 2008 Z RESEARCH Inc. <http://www.zresearch.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

说明glusterfs已经正确安装完毕了

下面来做个简单的配置,类似NFS的配置

环境:vmware6.0 模拟两个linux,ip分别Server:192.168.211.128 client:192.168.211.129,软件都已经安装完毕

server端的配置

先建立需要share的文件夹
make -p /home/dir1
chmod 1777 /home/dir1
然后建立配置文件
vi /etc/glusterfs/server.vol

### file: server-volume.vol.sample

#####################################
### GlusterFS Server Volume File ##
#####################################

#### CONFIG FILE RULES:
### “#” is comment character.
### – Config file is case sensitive
### – Options within a volume block can be in any order.
### – Spaces or tabs are used as delimitter within a line.
### – Multiple values to options will be : delimitted.
### – Each option should end within a line.
### – Missing or commented fields will assume default values.
### – Blank/commented lines are allowed.
### – Sub-volumes should already be defined above before referring.

### Export volume “brick” with the contents of “/home/export” directory.
volume brick
type storage/posix # POSIX FS translator
option directory /home/dir1 # Export this directory
end-volume

### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp/server
option bind-address 192.168.211.128 # Default is to listen on all interfaces
option listen-port 6996 # Default is 6996
subvolumes brick
option auth.addr.brick.allow * # Allow access to “brick” volume
end-volume

client端的配置

modprobe -i fuse
vi /etc/glusterfs/client.vol

### file: client-volume.vol.sample

#####################################
### GlusterFS Client Volume File ##
#####################################

#### CONFIG FILE RULES:
### “#” is comment character.
### – Config file is case sensitive
### – Options within a volume block can be in any order.
### – Spaces or tabs are used as delimitter within a line.
### – Each option should end within a line.
### – Missing or commented fields will assume default values.
### – Blank/commented lines are allowed.
### – Sub-volumes should already be defined above before referring.

### Add client feature and attach to remote subvolume
volume client0
type protocol/client
option transport-type tcp/client
option remote-host 192.168.211.128 # IP address of the remote brick
option remote-port 6996 # default server port is 6996
option remote-subvolume brick # name of the remote volume
end-volume

这样两台机器就都配置完毕了

首先在server上启动服务

gulsterfsd -f /etc/glusterfs/server.vol

[root@contos5-1-1 etc]# ps ax|grep -v grep |grep gluster
2028 ? Ssl 0:00 glusterfsd -f ./server.vol

看到这个证明服务器端就启动了

接下来是client端的启动

glusterfs -l /tmp/glustfs.log -f /etc/glusterfs/client.vol /mnt/

[root@contos5-1-4 glusterfs]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.1G 2.5G 4.3G 37% /
/dev/sda1 190M 11M 170M 7% /boot
tmpfs 125M 0 125M 0% /dev/shm
glusterfs 7.1G 2.9G 3.9G 43% /mnt

看到这个了,就说明client端mount正常,如果没有请查看/tmp/glusterfs.log文件

到此,一个简单的1存储1客户端的配置就完成了,这篇文章仅仅是个开始,glusterfs还有很多的功能,我也会在以后的时间进行深入的研究,希望有相同兴趣的朋友一起讨论