zhangguanzhang's Blog

记录LVM和raid的练习

字数统计: 7.1k阅读时长: 40 min
2017/04/20

LVM和raid的创建和调整

LVM就不用讲了,动态调整大小…
总结来说就是:
磁盘创建成PV物理卷
PV物理卷组成VG卷组
然后在卷组上创建LV

lvm

用分区模拟硬盘

  • 创建分区给LVM实例使用(没有多个硬盘,没办法,只有用一个盘分成多个分区来模仿实际生活里的多块硬盘了)
  • 用的云主机,俩块硬盘,用另一块没使用的数据盘做例子,分成四个分区
  • 使用fdisk来分区,n是增加分区表,p是打印分区表,创建三个逻辑分区然后一个扩展分区
[root@guan ~]# fdisk -l
Disk /dev/vda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000736d7
Device Boot Start End Blocks Id System /dev/vda1 * 1 2611 20970496 83 Linux
Disk /dev/vdb: 75.2 GB, 75161927680 bytes 16 heads, 63 sectors/track, 145635 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x32af4d52
Device Boot Start End Blocks Id System
[root@guan ~]# fdisk /dev/vdb WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-145635, default 1): Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-145635, default 145635): +18G Command (m for help): n Command action e extended p primary partition (1-4) p rtition number (1-4): 2 First cylinder (37451-145635, default 37451): Using default value 37451 Last cylinder, +cylinders or +size{K,M,G} (37451-145635, default 145635): +18G Command (m for help): p Disk /dev/vdb: 75.2 GB, 75161927680 bytes 16 heads, 63 sectors/track, 145635 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x32af4d52 Device Boot Start End Blocks Id System /dev/vdb1 1 37450 18874768+ 83 Linux /dev/vdb2 37451 74900 18874800 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (74901-145635, default 74901): Using default value 74901 Last cylinder, +cylinders or +size{K,M,G} (74901-145635, default 145635): +18G Command (m for help): p Disk /dev/vdb: 75.2 GB, 75161927680 bytes 16 heads, 63 sectors/track, 145635 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x32af4d52 Device Boot Start End Blocks Id System /dev/vdb1 1 37450 18874768+ 83 Linux /dev/vdb2 37451 74900 18874800 83 Linux /dev/vdb3 74901 112350 18874800 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) e Selected partition 4 First cylinder (112351-145635, default 112351): Using default value 112351 Last cylinder, +cylinders or +size{K,M,G} (112351-145635, default 145635): Using default value 145635 Command (m for help): n First cylinder (112351-145635, default 112351): Using default value 112351 Last cylinder, +cylinders or +size{K,M,G} (112351-145635, default 145635): +15G Command (m for help): p Disk /dev/vdb: 75.2 GB, 75161927680 bytes 16 heads, 63 sectors/track, 145635 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x32af4d52 Device Boot Start End Blocks Id System /dev/vdb1 1 37450 18874768+ 83 Linux /dev/vdb2 37451 74900 18874800 83 Linux /dev/vdb3 74901 112350 18874800 83 Linux /dev/vdb4 112351 145635 16775640 5 Extended /dev/vdb5 112351 143559 15729304+ 83 Linux Command (m for help): t Partition number (1-5): 1 Hex code (type L to list codes): 8e #修改分区编号 Changed system type of partition 1 to 8e (Linux LVM) Command (m for help): t Partition number (1-5): 2 Hex code (type L to list codes): 8e #修改分区编号 Changed system type of partition 2 to 8e (Linux LVM) Command (m for help): t Partition number (1-5): 3 Hex code (type L to list codes): 8e #修改分区编号 Changed system type of partition 3 to 8e (Linux LVM) Command (m for help): t Partition number (1-5): 5 Hex code (type L to list codes): 8e #修改分区编号 Changed system type of partition 5 to 8e (Linux LVM) Command (m for help): p Disk /dev/vdb: 75.2 GB, 75161927680 bytes 16 heads, 63 sectors/track, 145635 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x32af4d52 Device Boot Start End Blocks Id System /dev/vdb1 1 37450 18874768+ 8e Linux LVM /dev/vdb2 37451 74900 18874800 8e Linux LVM /dev/vdb3 74901 112350 18874800 8e Linux LVM /dev/vdb4 112351 145635 16775640 5 Extended /dev/vdb5 112351 143559 15729304+ 8e Linux LVM Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@guan ~]# fdisk -l Disk /dev/vda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000736d7 Device Boot Start End Blocks Id System /dev/vda1 * 1 2611 20970496 83 Linux Disk /dev/vdb: 75.2 GB, 75161927680 bytes 16 heads, 63 sectors/track, 145635 cylinders Units = cylinders of 1008 * 512 = 516096 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x32af4d52 Device Boot Start End Blocks Id System /dev/vdb1 1 37450 18874768+ 8e Linux LVM /dev/vdb2 37451 74900 18874800 8e Linux LVM /dev/vdb3 74901 112350 18874800 8e Linux LVM /dev/vdb4 112351 145635 16775640 5 Extended /dev/vdb5 112351 143559 15729304+ 8e Linux LVM [root@guan ~]# parted GNU Parted 2.1 Using /dev/vda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print list #查看下分区 Model: Virtio Block Device (virtblk) Disk /dev/vda: 21.5GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags 1 1049kB 21.5GB 21.5GB primary ext3 boot Model: Virtio Block Device (virtblk) Disk /dev/vdb: 75.2GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags 1 32.3kB 19.3GB 19.3GB primary ext3 lvm 2 19.3GB 38.7GB 19.3GB primary lvm 3 38.7GB 58.0GB 19.3GB primary lvm 4 58.0GB 75.2GB 17.2GB extended 5 58.0GB 74.1GB 16.1GB logical lvm (parted) q

2,创建PV物理卷并查看

[root@guan ~]# pvcreate /dev/vdb{1,2,3,5}
  Physical volume "/dev/vdb1" successfully created
  Physical volume "/dev/vdb2" successfully created
  Physical volume "/dev/vdb3" successfully created
  Physical volume "/dev/vdb5" successfully created
[root@guan ~]# pvdisplay 
  "/dev/vdb1" is a new physical volume of "18.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/vdb1
  VG Name               
  PV Size               18.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               iBGa4M-ek2A-ZMjT-fogT-V8Y3-Rpk5-XJJIRx
   
  "/dev/vdb2" is a new physical volume of "18.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/vdb2
  VG Name               
  PV Size               18.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               dx1VH5-NHTv-XlnV-cJdJ-7a2a-cMcg-GjKdMJ
   
  "/dev/vdb3" is a new physical volume of "18.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/vdb3
  VG Name               
  PV Size               18.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               kY3170-RecU-l4Ag-Ct96-i3qn-cHk4-hsXfTQ
   
  "/dev/vdb5" is a new physical volume of "15.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/vdb5
  VG Name               
  PV Size               15.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               iXsXMU-xtF1-HU5H-1Shu-qk0a-Gfhz-CufnJf

3,创建卷组VG并查看

[root@guan ~]# vgcreate test_vg /dev/vdb{1,2,3,5}
  Volume group "test_vg" successfully created
[root@guan ~]# vgdisplay 
  --- Volume group ---
  VG Name               test_vg
  System ID             
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               68.98 GiB
  PE Size               4.00 MiB
  Total PE              17660
  Alloc PE / Size       0 / 0   
  Free  PE / Size       17660 / 68.98 GiB
  VG UUID               uApKmt-Bnpz-yYKo-lK4B-u6RU-tbQf-HXpctt

4,创建LV逻辑卷并查看

[root@guan ~]# lvcreate -n test_233 -L 40G test_vg
  Logical volume "test_233" created
[root@guan ~]# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/test_vg/test_233
  LV Name                test_233
  VG Name                test_vg
  LV UUID                dPfsbR-P3xZ-fFWW-G5CG-Cbvg-QwW2-eLgzfK
  LV Write Access        read/write
  LV Creation host, time guan, 2017-05-15 16:27:22 +0800
  LV Status              available
  # open                 0
  LV Size                40.00 GiB
  Current LE             10240
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

5,格式化并且挂载

[root@guan ~]# mkfs.ext3 /dev/test_vg/test_233 
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
2621440 inodes, 10485760 blocks
524288 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
320 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@guan ~]# mkdir /mnt/test
[root@guan ~]# mount /dev/test_vg/test_233 /mnt/test
[root@guan ~]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/vda1                      20G  8.3G   11G  45% /
/dev/mapper/test_vg-test_233   40G  177M   38G   1% /mnt/test

6,给LVM扩容

[root@guan ~]# lvextend -L +10G /dev/test_vg/test_233 
  Extending logical volume test_233 to 50.00 GiB
  Logical volume test_233 successfully resized
[root@guan ~]# resize2fs /dev/test_vg/test_233 
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/test_vg/test_233 is mounted on /mnt/test; on-line resizing required
old desc_blocks = 3, new_desc_blocks = 4
Performing an on-line resize of /dev/test_vg/test_233 to 13107200 (4k) blocks.
The filesystem on /dev/test_vg/test_233 is now 13107200 blocks long.

[root@guan ~]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/vda1                      20G  8.3G   11G  45% /
/dev/mapper/test_vg-test_233   50G  180M   47G   1% /mnt/test

7,给卷组VG扩容


没办法,分区全部分完了...大致就是分一个区然后用pvcreate来转成物理卷
然后用vgextend vgname /dev/partname加进卷组然后用vgdisplay查看下容量是否扩大,然后如果是卷组不够用下想给逻辑卷扩容的话最后参照第六步用lvextend扩容下逻辑卷
8,删除LVM
[root@guan ~]# umount /mnt/test
[root@guan ~]# lvremove /dev/test_vg/test_233 
Do you really want to remove active logical volume test_233? [y/n]: y
  Logical volume "test_233" successfully removed
[root@guan ~]# vgremove test_vg 
  Volume group "test_vg" successfully removed
[root@guan ~]# pvremove /dev/vdb{1,2,3,5}
  Labels on physical volume "/dev/vdb1" successfully wiped
  Labels on physical volume "/dev/vdb2" successfully wiped
  Labels on physical volume "/dev/vdb3" successfully wiped
  Labels on physical volume "/dev/vdb5" successfully wiped
[root@guan ~]# 
  • !!!生产环境里调整文件系统大小前一定要做好数据备份

二:raid部分

1,主机只有一块闲置硬盘,先分区来给raid用


[root@guan ~]# fdisk /dev/vdb 

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-145635, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-145635, default 145635): +18G

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (37451-145635, default 37451): 
Using default value 37451
Last cylinder, +cylinders or +size{K,M,G} (37451-145635, default 145635): +18G

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): p

Disk /dev/vdb: 75.2 GB, 75161927680 bytes
16 heads, 63 sectors/track, 145635 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x32af4d52

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1               1       37450    18874768+  fd  Linux raid autodetect
/dev/vdb2           37451       74900    18874800   fd  Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

raid0

-C  –create   创建阵列;

-a  –auto    同意创建设备,如不加此参数时必须先使用mknod 命令来创建一个RAID设备,不过推荐使用-a yes参数一次性创建;

-l   –level   阵列模式,支持的阵列模式有 linear, raid0, raid1, raid4, raid5, raid6, raid10, multipath, faulty, container;

-n     –raid-devices    阵列中活动磁盘的数目,该数目加上备用磁盘的数目应该等于阵列中总的磁盘数目;


[root@guan ~]# mdadm -C /dev/md0 -l 0 -n1 /dev/vdb{1,2}
mdadm: '1' is an unusual number of drives for an array, so it is probably
     a mistake.  If you really mean it you will need to specify --force before
     setting the number of drives.
[root@guan ~]# mdadm -C /dev/md0 -l 0 -n 2 /dev/vdb1 /dev/vdb2
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@guan ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue May 16 09:14:05 2017
     Raid Level : raid0
     Array Size : 37748736 (36.00 GiB 38.65 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Tue May 16 09:14:05 2017
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 512K

           Name : guan:0  (local to host guan)
           UUID : c5875719:bcd15fa1:26644829:7adb57db
         Events : 0

    Number   Major   Minor   RaidDevice State
       0     252       17        0      active sync   /dev/vdb1
       1     252       18        1      active sync   /dev/vdb2

说明:Raid Level : 阵列级别; 

   Array Size : 阵列容量大小;

   Raid Devices : RAID成员的个数;

   Total Devices : RAID中下属成员的总计个数,因为还有冗余硬盘或分区,也就是spare,为了RAID的正常运珩,随时可以推上去加入RAID的;

   State : clean, degraded, recovering 状态,包括三个状态,clean 表示正常,degraded 表示有问题,recovering 表示正在恢复或构建;

   Active Devices : 被激活的RAID成员个数;

   Working Devices : 正常的工作的RAID成员个数;

   Failed Devices : 出问题的RAID成员;

   Spare Devices : 备用RAID成员个数,当一个RAID的成员出问题时,用其它硬盘或分区来顶替时,RAID要进行构建,在没构建完成时,这个成员也会被认为是spare设备;

   UUID : RAID的UUID值,在系统中是唯一的;

格式化磁盘阵列并挂载


[root@guan ~]# mkfs.ext3 /dev/md0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=256 blocks
2359296 inodes, 9437184 blocks
471859 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
288 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
  32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
  4096000, 7962624

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@guan ~]# mkdir /raid0
[root@guan ~]# mount /dev/md0 /raid0

将磁盘阵列设备设置为开机自动挂载项,centos系统开机会读取磁盘阵列文件的配置文件/etc/mdadm.conf以启动rad设备


[root@guan ~]# echo DEVICE /dev/vdb{1,2}>/etc/mdadm.conf
[root@guan ~]# mdadm –Ds >> /etc/mdadm.conf
[root@guan ~]# cat /etc/mdadm.conf
DEVICE /dev/vdb1 /dev/vdb2
ARRAY /dev/md0 metadata=1.2 name=guan:0 UUID=c5875719:bcd15fa1:26644829:7adb57db
[root@guan ~]# echo '/dev/md0               /raid0             ext3     defaults       0 0'>>/etc/fstab

测试下速度


[root@guan ~]# time dd if=/dev/zero of=md bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.8183 s, 81.8 MB/s

real  0m12.820s
user  0m0.004s
sys 0m3.233s
[root@guan raid0]# time dd if=/dev/zero of=testfile bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.7144 s, 89.5 MB/s

real  0m11.716s
user  0m0.004s
sys 0m2.741s

构建raid5


[root@guan ~]# fdisk /dev/vdb

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (74901-145635, default 74901): 
Using default value 74901
Last cylinder, +cylinders or +size{K,M,G} (74901-145635, default 145635): +18G

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
e
Selected partition 4
First cylinder (112351-145635, default 112351): 
Using default value 112351
Last cylinder, +cylinders or +size{K,M,G} (112351-145635, default 145635): 
Using default value 145635

Command (m for help): n
First cylinder (112351-145635, default 112351): 
Using default value 112351
Last cylinder, +cylinders or +size{K,M,G} (112351-145635, default 145635): 
Using default value 145635

Command (m for help): t
Partition number (1-5): 3
Hex code (type L to list codes): fd
Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-5): 5
Hex code (type L to list codes): fd
Changed system type of partition 5 to fd (Linux raid autodetect)

Command (m for help): p

Disk /dev/vdb: 75.2 GB, 75161927680 bytes
16 heads, 63 sectors/track, 145635 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x32af4d52

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1               1       37450    18874768+  fd  Linux raid autodetect
/dev/vdb2           37451       74900    18874800   fd  Linux raid autodetect
/dev/vdb3           74901      112350    18874800   fd  Linux raid autodetect
/dev/vdb4          112351      145635    16775640    5  Extended
/dev/vdb5          112351      145635    16775608+  fd  Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@guan ~]# mdadm -C /dev/md0 -l 5 -n 3 -x 1  /dev/vdb{1,2,3,5}
mdadm: largest drive (/dev/vdb1) exceeds size (16766976K) by more than 1%
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

查看状态


[root@guan ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue May 16 11:28:40 2017
     Raid Level : raid5
     Array Size : 33533952 (31.98 GiB 34.34 GB)
  Used Dev Size : 16766976 (15.99 GiB 17.17 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue May 16 11:30:57 2017
          State : clean, degraded, recovering 
 Active Devices : 2
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 512K

 Rebuild Status : 22% complete

           Name : guan:0  (local to host guan)
           UUID : 343a8167:2d1ac2dd:75294605:74886ed1
         Events : 4

    Number   Major   Minor   RaidDevice State
       0     252       17        0      active sync   /dev/vdb1
       1     252       18        1      active sync   /dev/vdb2
       4     252       19        2      spare rebuilding   /dev/vdb3

       3     252       21        -      spare   /dev/vdb5

还在构建,构建完成后状态如下


[root@guan ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue May 16 11:28:40 2017
     Raid Level : raid5
     Array Size : 33533952 (31.98 GiB 34.34 GB)
  Used Dev Size : 16766976 (15.99 GiB 17.17 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue May 16 11:39:28 2017
          State : clean 
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : guan:0  (local to host guan)
           UUID : 343a8167:2d1ac2dd:75294605:74886ed1
         Events : 18

    Number   Major   Minor   RaidDevice State
       0     252       17        0      active sync   /dev/vdb1
       1     252       18        1      active sync   /dev/vdb2
       4     252       19        2      active sync   /dev/vdb3

       3     252       21        -      spare   /dev/vdb5

格式化并挂载


[root@guan ~]# mkfs.ext3 /dev/md0
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=256 blocks
2097152 inodes, 8383488 blocks
419174 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
256 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
  32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
  4096000, 7962624

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@guan ~]# mkdir /mnt/raid5   
[root@guan ~]# mount /dev/md0 /mnt/raid5/

模拟硬盘损坏


[root@guan ~]# touch /mnt/raid5/test1
[root@guan ~]# touch /mnt/raid5/test2
[root@guan ~]# mdadm /dev/md0 -f /dev/vdb1
mdadm: set /dev/vdb1 faulty in /dev/md0
[root@guan ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue May 16 11:28:40 2017
     Raid Level : raid5
     Array Size : 33533952 (31.98 GiB 34.34 GB)
  Used Dev Size : 16766976 (15.99 GiB 17.17 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue May 16 11:48:37 2017
          State : clean, degraded, recovering 
 Active Devices : 2
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

 Rebuild Status : 1% complete

           Name : guan:0  (local to host guan)
           UUID : 343a8167:2d1ac2dd:75294605:74886ed1
         Events : 20

    Number   Major   Minor   RaidDevice State
       3     252       21        0      spare rebuilding   /dev/vdb5
       1     252       18        1      active sync   /dev/vdb2
       4     252       19        2      active sync   /dev/vdb3

       0     252       17        -      faulty   /dev/vdb1

可以看到热备盘vdb5顶替了手动模拟损坏的损坏盘vdb1后raid5在进行数据检验与恢复
构建过程好慢….
中途创建了文件也可以,在数据恢复的过程中raid5设备的性能会明显下降


[root@guan ~]# mdadm /dev/md0 -r /dev/vdb1
mdadm: hot removed /dev/vdb1 from /dev/md0
[root@guan ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue May 16 11:28:40 2017
     Raid Level : raid5
     Array Size : 33533952 (31.98 GiB 34.34 GB)
  Used Dev Size : 16766976 (15.99 GiB 17.17 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Tue May 16 12:03:56 2017
          State : clean 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : guan:0  (local to host guan)
           UUID : 343a8167:2d1ac2dd:75294605:74886ed1
         Events : 90

    Number   Major   Minor   RaidDevice State
       3     252       21        0      active sync   /dev/vdb5
       1     252       18        1      active sync   /dev/vdb2
       4     252       19        2      active sync   /dev/vdb3

可以看到损坏盘vdb1已经移除了
新加热备磁盘,注意新增加的硬盘需要与原硬盘大小一致
如果是实际生产中添加新的硬盘,同样需要对新硬盘进行创建分区的操作,这里我们为了方便,将刚才模拟损坏的硬盘再次新加到raid5 中


[root@guan ~]# mdadm /dev/md0 -a /dev/vdb1
mdadm: added /dev/vdb1
[root@guan ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue May 16 11:28:40 2017
     Raid Level : raid5
     Array Size : 33533952 (31.98 GiB 34.34 GB)
  Used Dev Size : 16766976 (15.99 GiB 17.17 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue May 16 12:06:29 2017
          State : clean 
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : guan:0  (local to host guan)
           UUID : 343a8167:2d1ac2dd:75294605:74886ed1
         Events : 91

    Number   Major   Minor   RaidDevice State
       3     252       21        0      active sync   /dev/vdb5
       1     252       18        1      active sync   /dev/vdb2
       4     252       19        2      active sync   /dev/vdb3

       5     252       17        -      spare   /dev/vdb1

可以看到vdb1变成热备盘
如果要向raid里扩容,默认添加进来的盘都是热备盘所以说要扩容d话需要转换为活动盘
这里我没有其余分区了,我就把vdb1当作加进来的盘下转成活动盘吧
注意:-G是增长模式 -n 4 表示使用四块工作磁盘


[root@guan ~]# mdadm -G /dev/md0 -n4
mdadm: Need to backup 3072K of critical section..
[root@guan ~]# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Tue May 16 11:28:40 2017
     Raid Level : raid5
     Array Size : 33533952 (31.98 GiB 34.34 GB)
  Used Dev Size : 16766976 (15.99 GiB 17.17 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue May 16 13:45:24 2017
          State : clean, reshaping 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

 Reshape Status : 0% complete
  Delta Devices : 1, (3->4)

           Name : guan:0  (local to host guan)
           UUID : 343a8167:2d1ac2dd:75294605:74886ed1
         Events : 107

    Number   Major   Minor   RaidDevice State
       3     252       21        0      active sync   /dev/vdb5
       1     252       18        1      active sync   /dev/vdb2
       4     252       19        2      active sync   /dev/vdb3
       5     252       17        3      active sync   /dev/vdb1

可以看到正在构建
将磁盘阵列设备设置为开机自动挂载的话参照上面的部分
RAID设备的删除


[root@guan test]# umount /mnt/raid5/
[root@guan test]# mdadm -S /dev/md0
mdadm: stopped /dev/md0

扩lvm容量的一些实操记录

改硬盘容量没有识别

kvm 上修改是能识别的,但是 vmware上 没有识别,需要重启。实际上可以不重启识别,下面实际硬盘 sda 被修改成 210G

1
2
3
4
5
6
7
8
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 200G 0 disk
└─sda1 8:1 0 195G 0 part /boot
├─centos-root 253:0 0 50G 0 lvm /
├─centos-swap 253:1 0 200M 0 lvm
└─centos-data 253:2 0 144.8G 0 lvm /data
sr0 11:0 1 4.2G 0 rom

查找硬盘属于的host,下面是 host0

1
2
$ ls -l /sys/block/sda
lrwxrwxrwx 1 root root 0 Oct 29 04:07 /sys/block/sda -> ../devices/pci0000:00/0000:00:10.0/host0/target0:0:0/0:0:0:0/block/sda

如果新加的没识别到:

1
echo "- - -" > /sys/class/scsi_host/hostX/scan

下面这个能重新扫描,从而识别到容量

1
echo 1 > /sys/class/block/sda/device/rescan

查看到生效

1
2
3
4
5
6
7
8
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 210G 0 disk
└─sda1 8:1 0 195G 0 part /boot
├─centos-root 253:0 0 50G 0 lvm /
├─centos-swap 253:1 0 200M 0 lvm
└─centos-data 253:2 0 144.8G 0 lvm /data
sr0 11:0 1 4.2G 0 rom

听群友说 sg3_utils 也可以,装了它后 rescan-scsi-bus.sh,不确定这个能识别容量不。

参考:

parted无交互配置lvm

1
2
3
4
5
parted /dev/vdb mklabel gpt;
parted /dev/vdb mkpart primary xfs 1 100%;
parted /dev/vdb set 1 lvm on;
pvcreate /dev/vdb1;
vgextend centos_centos7 /dev/vdb1;

如果是新创建 lv 挂载

1
2
3
4
lvcreate -n data -l +100%FREE centos_centos7;
mkfs.xfs /dev/mapper/centos_centos7-data;
mkdir /data;
mount /dev/mapper/centos_centos7-data /data;

追加到已有lv

1
lvextend -l +100% /dev/mapper/centos_centos7-data

有需要的话,清除parted给硬盘打的label

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@lvm-test2 ~]# gdisk /dev/vdb            
GPT fdisk (gdisk) version 0.8.10

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): x

Expert command (? for help): z
About to wipe out GPT on /dev/vdb. Proceed? (Y/N): y
yGPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Blank out MBR? (Y/N): y

扩容实战

可以接着前面的虚拟化上增大硬盘的容量后操作。但是要注意:扩容只能扩容一个硬盘上的最后一个分区表,推荐使用 parted 命令扩,且推荐使用最新版的 parted。无论这个分区是不是 lvm,都得先扩大分区的大小。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 464K 0 rom
sr1 11:1 1 1024M 0 rom
vda 252:0 0 60G 0 disk
├─vda1 252:1 0 2M 0 part
├─vda2 252:2 0 200M 0 part /boot
└─vda3 252:3 0 39.8G 0 part
└─centos7-root 253:0 0 39.8G 0 lvm /
vdb 252:16 0 100G 0 disk
$ parted /dev/vda
GNU Parted 3.1
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 64.4GB <----------------------------注意这个容量
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 1049kB 3146kB 2097kB primary
2 3146kB 213MB 210MB primary xfs boot
3 213MB 42.9GB 42.7GB primary lvm

vda 原先40G,现在把硬盘容量扩容到60G了,需要把剩下的容量用 parted 加到最后的分区 vda3 上,理论上也只能扩容最后一个分区。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(parted) resizepart 3                                                     
End? [42.9GB]? 64G # <--------- 写前面的容量,稳妥起见保留一点
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 64.4GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 1049kB 3146kB 2097kB primary
2 3146kB 213MB 210MB primary xfs boot
3 213MB 64.0GB 63.8GB primary lvm

(parted) quit
Information: You may need to update /etc/fstab.

扩容 pv

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ pvdisplay                                               
--- Physical volume ---
PV Name /dev/vda3
VG Name centos7
PV Size 39.80 GiB / not usable 0
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 10189
Free PE 0
Allocated PE 10189
PV UUID VVXMV7-zVhi-NSVz-frrY-YKFg-BoJH-ir33p3
$ pvresize /dev/vda3
Physical volume "/dev/vda3" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
$ pvdisplay
--- Physical volume ---
PV Name /dev/vda3
VG Name centos7
PV Size <59.41 GiB / not usable <3.16 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 15207
Free PE 5018
Allocated PE 10189
PV UUID VVXMV7-zVhi-NSVz-frrY-YKFg-BoJH-ir33p3

扩容根分区所在的 lv

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ lvdisplay 
--- Logical volume ---
LV Path /dev/centos7/root
LV Name root
VG Name centos7
LV UUID npjBgn-XPHe-dNS2-bJGd-kYp2-RayO-dWttAc
LV Write Access read/write
LV Creation host, time localhost, 2018-06-30 14:35:21 +0800
LV Status available
# open 1
LV Size 39.80 GiB
Current LE 10189
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
$ lvextend -l +100%FREE /dev/centos7/root
Size of logical volume centos7/root changed from 39.80 GiB (10189 extents) to 59.40 GiB (15207 extents).
Logical volume centos7/root successfully resized.

这里根分区文件系统是 xfs,所以可以使用 xfs_growfs 扩容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ xfs_growfs /dev/centos7/root 
meta-data=/dev/mapper/centos7-root isize=512 agcount=4, agsize=2608384 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=10433536, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=5094, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 10433536 to 15571968
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 464K 0 rom
sr1 11:1 1 1024M 0 rom
vda 252:0 0 60G 0 disk
├─vda1 252:1 0 2M 0 part
├─vda2 252:2 0 200M 0 part /boot
└─vda3 252:3 0 59.4G 0 part
└─centos7-root 253:0 0 59.4G 0 lvm /
vdb 252:16 0 100G 0 disk
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos7-root 60G 1.1G 59G 2% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 9.0M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/vda2 197M 120M 77M 61% /boot
tmpfs 781M 0 781M 0% /run/user/0

centos 6.5 ext4 扩容根,删掉分区信息(不会删除数据或文件系统,当然这样智能扩最后一个分区表)重写

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
[root@asfd ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 464K 0 rom
sr1 11:1 1 1024M 0 rom
vda 252:0 0 100G 0 disk
├─vda1 252:1 0 200M 0 part /boot
└─vda2 252:2 0 39.8G 0 part
└─vg_centos6-LogVol00 (dm-0) 253:0 0 39.8G 0 lvm /
[root@asfd ~]# parted /dev/vda
GNU Parted 2.1
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 211MB 210MB primary ext4 boot
2 211MB 42.9GB 42.7GB primary lvm

(parted) rm 2
Warning: WARNING: the kernel failed to re-read the partition table on /dev/vda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 211MB 210MB primary ext4 boot

(parted) mkpart primary 211MB 98GB
Warning: WARNING: the kernel failed to re-read the partition table on /dev/vda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 1049kB 211MB 210MB primary ext4 boot
2 211MB 98.0GB 97.8GB primary

(parted) quit
Information: You may need to update /etc/fstab.
[root@asfd ~]# e2fsck -f /dev/vda
e2fsck 1.41.12 (17-May-2010)
/dev/vda is in use.
e2fsck: Cannot continue, aborting.
[root@asfd ~]# resize2fs /dev/vda2
resize2fs 1.41.12 (17-May-2010)
resize2fs: Device or resource busy while trying to open /dev/vda2
Couldn't find valid filesystem superblock.

[root@asfd ~]# reboot

[root@asfd ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 464K 0 rom
sr1 11:1 1 1024M 0 rom
vda 252:0 0 100G 0 disk
├─vda1 252:1 0 200M 0 part /boot
└─vda2 252:2 0 91.1G 0 part
└─vg_centos6-LogVol00 (dm-0) 253:0 0 39.8G 0 lvm /
[root@asfd ~]# pvdisplay
--- Physical volume ---
PV Name /dev/vda2
VG Name vg_centos6
PV Size 39.80 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 10189
Free PE 0
Allocated PE 10189
PV UUID CkzQuX-GXe4-umx7-f70N-bcPK-4l02-YEzK06

[root@asfd ~]# pvresize /dev/vda2
Physical volume "/dev/vda2" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
[root@asfd ~]# pvdisplay
--- Physical volume ---
PV Name /dev/vda2
VG Name vg_centos6
PV Size 91.07 GiB / not usable 2.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 23314
Free PE 13125
Allocated PE 10189
PV UUID CkzQuX-GXe4-umx7-f70N-bcPK-4l02-YEzK06

[root@asfd ~]# lvdisplay
--- Logical volume ---
LV Path /dev/vg_centos6/LogVol00
LV Name LogVol00
VG Name vg_centos6
LV UUID aUnWwy-jYdb-sQrT-X7cY-eGzP-eQz3-L41Wx2
LV Write Access read/write
LV Creation host, time centos6.8, 2018-06-30 18:01:01 +0800
LV Status available
# open 1
LV Size 39.80 GiB
Current LE 10189
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

[root@asfd ~]# lvextend -l +100%FREE /dev/vg_centos6/LogVol00
Size of logical volume vg_centos6/LogVol00 changed from 39.80 GiB (10189 extents) to 91.07 GiB (23314 extents).
Logical volume LogVol00 successfully resized.
[root@asfd ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 464K 0 rom
sr1 11:1 1 1024M 0 rom
vda 252:0 0 100G 0 disk
├─vda1 252:1 0 200M 0 part /boot
└─vda2 252:2 0 91.1G 0 part
└─vg_centos6-LogVol00 (dm-0) 253:0 0 91.1G 0 lvm /

参考:

CATALOG
  1. 1. lvm
    1. 1.1. 用分区模拟硬盘
    2. 1.2. 2,创建PV物理卷并查看
    3. 1.3. 3,创建卷组VG并查看
    4. 1.4. 4,创建LV逻辑卷并查看
    5. 1.5. 5,格式化并且挂载
    6. 1.6. 6,给LVM扩容
    7. 1.7. 7,给卷组VG扩容
  2. 2. 二:raid部分
    1. 2.1. 1,主机只有一块闲置硬盘,先分区来给raid用
      1. 2.1.1. raid0
      2. 2.1.2. 构建raid5
    2. 2.2. 模拟硬盘损坏
  3. 3. 扩lvm容量的一些实操记录
    1. 3.1. 改硬盘容量没有识别
    2. 3.2. parted无交互配置lvm
      1. 3.2.1. 如果是新创建 lv 挂载
      2. 3.2.2. 追加到已有lv
    3. 3.3. 扩容实战