GPFS安装手册

目录

GPFS 基本安装过程 ........................................................................................................................ 1 GPFS performance and Infiniband: ................................................................................................... 5 adding extra subnets（可选）: ........................................................................................................ 7 与LUSTRE文件系统共存环境搭建： ............................................................................................. 8 Gpfs 基本卸载过程 ......................................................................................................................... 9

GPFS 基本安装过程

同步所有服务器时间，系统时间 1. 准备并安装rpm包，升级包

2. 准备依赖包，cp -rf /root/GPFS/6 /var/cache/yum/x86_64/

3. yum -y install rsh ; yum -y install libc.so.6 ; yum -y install ksh ; yum -y install libstdc++.so.5;

yum -y install compat-libstdc++-33-3.2.3-69.el6.x86_64 4. cd /root/GPFS;rpm -ivh *.rpm ; rpm -Uvh update/*.rpm 5. Linux客户端ssh无密码登录

ssh-keygen -t rsa

会创建/root/.ssh/id-ras.pub id-ras 公钥：/root/.ssh/id-ras.pub 私钥：/root/.ssh/id-ras

把公钥复制到authorized_keys ,并复制/root/.ssh到所有机器上的/root/目录.

If your GPFS cluster is configured to use SSH/SCP, it is suggested that you increase the value of MaxStartups in sshd_config to at least 1024. 6. 设置环境变量

vi /root/.bash_profile

PATH=$PATH:/usr/lpp/mmfs/bin 启用环境变量

source /root/.bash_profile 7. 编译安装

cd /usr/lpp/mmfs/src make Autoconfig make World make InstallImages

cd /usr/lpp/mmfs/src; make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig; make World;make InstallImages

When using GPFS 3.2.1 and earlier there is not an rpm option to the GPFS GPL build process. In this case to distribute the kernel module binaries you can copy the files manually. You can see alist of these files when the make InstallImages process completes. These files are mmfslinux, mmfs26, lxtrace, dumpconv, tracedevcd, and they need to be copied to the /usr/lpp/mmfs/bin directory on the other nodes. 命令如下：

mkdir /tmp/gpfs_cmds/

cp -f /usr/lpp/mmfs/src/bin/mmfslinux /tmp/gpfs_cmds/mmfslinux cp -f /usr/lpp/mmfs/src/bin/mmfs26 /tmp/gpfs_cmds/mmfs26 cp -f /usr/lpp/mmfs/src/bin/tracedev /tmp/gpfs_cmds/tracedev cp -f /usr/lpp/mmfs/src/bin/lxtrace.sial /tmp/gpfs_cmds/lxtrace.sial

cp -f /usr/lpp/mmfs/src/bin/mmdumpfilocks.sial /tmp/gpfs_cmds/mmdumpfilocks.sial cp -f /usr/lpp/mmfs/src/bin/lxtrace /tmp/gpfs_cmds/lxtrace cp -f /usr/lpp/mmfs/src/bin/dumpconv /tmp/gpfs_cmds/dumpconv cd /tmp

tar -cf gpfs_cmds.tar ./gpfs_cmds

on other nodes copy cmds to /usr/lpp/mmfs/bin

8. 创建cluster

mmcrcluster -N hosta:quorum,hostb:quorum -p hosta -s hostb -r /usr/bin/ssh -R /usr/bin/scp

9. mmchlicense server --accept -N hosta,hostb 10. (可选)修改GPFS文件，支持device-mapper设备

cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices 修改文件使之能够输出这种格式\，mpath1为多路径磁盘 for i in `multipath -ll | grep CERESDAT | awk '{print $1}'` do

echo mapper/$i dmm done

11. NSD配置文件(dataAndMetadata、dataOnly、metadataOnly、descOnly)

DiskUsage

If a value is not specified, the disk usage remains unchanged: dataAndMetadata

Indicates that the disk contains both data and metadata. This is the default. dataOnly

Indicates that the disk contains data and does not contain metadata. metadataOnly

Indicates that the disk contains metadata and does not contain data. descOnly

Indicates that the disk contains no data and no file metadata. Such a disk is used solely to

keep a copy of the file system descriptor, and can be used as a third failure group in certain disaster recovery configurations. For more information, see General Parallel File

System: Advanced Administration and search on Synchronous mirroring utilizing GPFS replication.

格式：DiskName:ServerList::DiskUsage:FailureGroup:DesiredName

ServerList中可指定主备nsd server,分散磁盘顺序以实现负载均衡 [root@node01 ~]# cat diskdef

/dev/mapper/meta01:io01:io02:metadataOnly:failgroup01: /dev/mapper/meta02:io03:io04:metadataOnly:failgroup02: 12. [root@hosta ~]# vi diskdef

/dev/sdc:hosta:hostb::01: /dev/sdd:hostb:hosta::02: /dev/sde:hosta:hostb::01: /dev/sdf:hostb:hosta::02: /dev/sdg:hosta:hostb::01: /dev/sdh:hostb:hosta::02: /dev/sdi:hosta:hostb::01: /dev/sdj:hostb:hosta::02:

13. 创建NSD

mmcrnsd -F diskdef -v no （可选）创建仲裁盘:

mmchconfig tiebreakerDisks=\14. 启动GPFS,并查看启动状态

mmstartup -a;mmgetstate -a 15. 创建文件系统

mmcrfs vol_data -F diskdef -B 64K -T /vol_data -v no 16. 挂载文件系统

mmmount vol_data -a

mmchdisk vol_data stop -d gpfs12nsd mmlsdisk vol_data

disk driver sector failure holds holds storage name type size group metadata data status availability pool ------------ -------- ------ ------- -------- ----- ------------- ------------ ------------

gpfs6nsd nsd 512 1 Yes Yes ready up system gpfs7nsd nsd 512 2 Yes Yes ready up system gpfs8nsd nsd 512 1 Yes Yes ready up system gpfs9nsd nsd 512 2 Yes Yes ready up system gpfs10nsd nsd 512 1 Yes Yes ready up system gpfs11nsd nsd 512 2 Yes Yes ready up system gpfs12nsd nsd 512 1 Yes Yes ready down system gpfs13nsd nsd 512 2 Yes Yes ready up system vi add_diskdef

17. (可选)变相替换磁盘过程（首先数据得有冗余，加粗字体为命令）

/dev/sdj:hosta:hostb::01: mmcrnsd -F add_diskdef mmlsnsd

File system Disk name NSD servers --------------------------------------------------------------------------- vol_data gpfs6nsd hosta,hostb vol_data gpfs7nsd hostb,hosta vol_data gpfs8nsd hosta,hostb vol_data gpfs9nsd hostb,hosta vol_data gpfs10nsd hosta,hostb vol_data gpfs11nsd hostb,hosta vol_data gpfs12nsd hosta,hostb vol_data gpfs13nsd hostb,hosta (free disk) gpfs14nsd hosta,hostb mmadddisk vol_data -F add_diskdef -r

The following disks of vol_data will be formatted on node hosta: gpfs14nsd: size 586061784 KB Extending Allocation Map

Checking Allocation Map for storage pool 'system' 12 % complete on Fri Jan 4 07:11:45 2013 100 % complete on Fri Jan 4 07:12:23 2013 Completed adding disks to file system vol_data.

mmadddisk: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. Restriping vol_data ...

Scanning file system metadata, phase 1 ... 1 % complete on Fri Jan 4 07:12:37 2013 100 % complete on Fri Jan 4 07:12:50 2013 Scan completed successfully.

Scanning file system metadata, phase 2 ... 4 % complete on Fri Jan 4 07:12:53 2013 100 % complete on Fri Jan 4 07:13:54 2013 Scan completed successfully.

Scanning file system metadata, phase 3 ... Scan completed successfully.

Scanning file system metadata, phase 4 ... Scan completed successfully. Scanning user file metadata ...

23.60 % complete on Fri Jan 4 07:14:16 2013 ( 487171 inodes 1702 MB) 100.00 % complete on Fri Jan 4 07:14:46 2013 Scan completed successfully. Done

18. 增加GPFS客户端节点

在安装好GPFS rpm情况下（安装方式同上面所提），运行命令：mmaddnode -N client01

GPFS performance and Infiniband:

linux磁盘预读访问优化：

for i in `ls /dev/mapper/ | grep -v control | grep -v Vol`;do blockdev --setra 1024 /dev/mapper/$i;done

for i in `ls /dev/mapper/ | grep -v control | grep -v Vol`;do blockdev --getra /dev/mapper/$i;done

mmchconfig prefetchThreads=96 mmchconfig worker1Threads=144 maxMBpS=3200 pagepool=4096M verbsPorts verbsRdma [enable|disable]

[root@node009 ~]# ibstatus

Infiniband device 'mlx4_0' port 1 status:

default gid: fe80:0000:0000:0000:0030:4800:4413:0001 base lid: 0x4b sm lid: 0x35 state: 4: ACTIVE phys state: 5: LinkUp

rate: 20 Gb/sec (4X DDR)

Infiniband device 'mlx4_0' port 2 status:

default gid: fe80:0000:0000:0000:0030:4800:4413:0002 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X)

mmchconfig verbsPorts=\ mmchconfig versRdma=enable

Once these two parameters are set, if a node has these ports available they are used once the GPFS daemon has started. GPFS decides to use the IB RDMA connection instead of TCP/IP before looking at the subnets parameter. If an IB RDMA connection is available on a node it is used. If there are no IB ports available on a node then the subnets processing rules take effect. For enhanced performance when multiple verbPorts are defined on a node GPFS multiplexes the traffic over all of the available ports.

本文来源：https://www.bwwdw.com/article/tk1w.html