1. 服务器中的阵列卡,按照使用工具的不同,可分为Broadcom LSI(工具:storcli), DELL(工具:megacli,虽然DELL服务器使用LSI的阵列卡,但是由于不支持storcli命令,所以使用megacli),HP(PMC芯片卡,工具:hpssacli/ssacli),Microchip Adaptec(工具:arcconf)。
2. 阵列卡上创建的逻辑卷读写缓存不同的设置会影响到raid性能,所以梳理各种阵列卡的读/写缓存打开/关闭的方法以供设置参考。
工具1: storcli
a. 确认软件包是否安装
[root@localhost ~]#rpm -qa | grep storcli
storcli-1.21.06-1.noarch
备注:若检索结果为空,需要安装对应的软件安装包。安装命令如下:
#rpm -ivh storcli-xx-xx-1.noarch.rpm
b. 查看逻辑卷读/写缓存状态
备注:请关注标红部分示例为:NRWTD, 参数解释如下:
NR: No Read Ahead(关闭预读,即读缓存关闭)
R:Read Ahead(开启预读,即开启读缓存)
WB: Write Back(开启回写,即开启写缓存)
AWB: Always Write Back(强制开启回写,强制开启写缓存)
WT: Write Through(直写模式,关闭回写/写缓存)
[root@localhost ~]# /opt/MegaRAID/storcli/storcli64 /c0/v1 show
Controller = 0
Status = Success
Description = None
Virtual Drives :
==============
--------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
--------------------------------------------------------------
1/1 RAID10 Optl RW No NRWTD - ON 2.180 TB
--------------------------------------------------------------
Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked|
Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency
c. 读缓存打开/关闭
语法如下:
storcli /cx/vx set rdcache=RA|NoRA
参数说明:
cx: x为数字,从0开始,代表控制器编号
vx: x为数字,从0开始,代表逻辑卷编号
RA: Read Ahead
NoRA: No Read Ahead
示例:
[root@localhost ~]# /opt/MegaRAID/storcli/storcli64 /c0/v1 set rdcache=ra
Controller = 0
Status = Success
Description = None
Detailed Status :
===============
---------------------------------------
VD Property Value Status ErrMsg ErrCd
---------------------------------------
1 rdCache RA Success - 0
---------------------------------------
[root@localhost ~]# /opt/MegaRAID/storcli/storcli64 /c0/v1 set rdcache=nora
Controller = 0
Status = Success
Description = None
Detailed Status :
===============
---------------------------------------
VD Property Value Status ErrMsg ErrCd
---------------------------------------
1 rdCache NoRA Success - 0
---------------------------------------
d. 写缓存打开/关闭
语法如下:
storcli /cx/vx set wrcache=WT|WB|AWB
参数说明:
cx: x为数字,从0开始,代表控制器编号
vx: x为数字,从0开始,代表逻辑卷编号
WT: Write Through
WB: Write Back
AWB: Always Write Back
示例:
[root@localhost ~]# /opt/MegaRAID/storcli/storcli64 /c0/v1 set wrcache=wt
Controller = 0
Status = Success
Description = None
Detailed Status :
===============
---------------------------------------
VD Property Value Status ErrMsg ErrCd
---------------------------------------
1 wrCache WT Success - 0
---------------------------------------
[root@localhost ~]# /opt/MegaRAID/storcli/storcli64 /c0/v1 set wrcache=wb
Controller = 0
Status = Success
Description = None
Detailed Status :
===============
---------------------------------------
VD Property Value Status ErrMsg ErrCd
---------------------------------------
1 wrCache WB Success - 0
---------------------------------------
[root@localhost ~]# /opt/MegaRAID/storcli/storcli64 /c0/v1 set wrcache=awb
Controller = 0
Status = Success
Description = None
Detailed Status :
===============
---------------------------------------
VD Property Value Status ErrMsg ErrCd
---------------------------------------
1 wrCache AWB Success - 0
---------------------------------------
工具2:megacli
a. 确认软件包是否安装
[root@localhost ~]# rpm -qa | grep -i MegaCli
MegaCli-8.07.10-1.noarch
备注:若检索结果为空,需要安装对应的软件安装包。安装命令如下:
#rpm -ivh MegaCli-xx-xx-1.noarch.rpm
b. 查看逻辑卷读/写缓存状态
备注:
请关注标红部分示例为: WriteBack, ReadAheadNone, Direct, Write Cache OK if Bad BBU, 参数解释如下:
NORA: No Read Ahead(关闭预读,即读缓存关闭)
RA:Read Ahead(开启预读,即开启读缓存)
WB: Write Back(开启回写,即开启写缓存)
ForcedWB: Forced Write Back(强制开启回写,强制开启写缓存)
WT: Write Through(直写模式,关闭回写/写缓存)
示例:
[root@localhost ~]# /opt/MegaRAID/MegaCli/MegaCli64 -ldinfo -l1 -a0
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
Size : 2.180 TB
Sector Size : 512
Is VD emulated : Yes
Mirror Data : 2.180 TB
State : Optimal
Strip Size : 64 KB
Number Of Drives per span:2
Span Depth : 5
Default Cache Policy: WriteBack, ReadAheadNone, Direct, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, Write Cache OK if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Ongoing Progresses:
Check Consistency : Completed 26%, Taken 493 min.
Encryption Type : None
Bad Blocks Exist: No
PI type: No PI
Is VD Cached: No
Exit Code: 0x00
c. 读/写缓存打开/关闭
语法如下:
MegaCli -LDSetProp {-Name LdNamestring} | -RW|RO|Blocked|RemoveBlocked | WT|WB|ForcedWB [-Immediate] |RA|NORA|ADRA |
DsblPI
| Cached|Direct | -EnDskCache|DisDskCache | CachedBadBBU|NoCachedBadBBU
-Lx|-L0,1,2|-Lall -aN|-a0,1,2|-aALL
参数说明:
aN: N为数字,从0开始,代表控制器编号
Lx: x为数字,从0开始,代表逻辑卷编号
NORA: No Read Ahead(关闭预读,即读缓存关闭)
RA:Read Ahead(开启预读,即开启读缓存)
WB: Write Back(开启回写,即开启写缓存)
ForcedWB: Forced Write Back(强制开启回写,强制开启写缓存)
WT: Write Through(直写模式,关闭回写/写缓存)
读缓存修改示例:
[root@localhost ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LdSetProp RA -L1 -a0
Set Read Policy to ReadAhead on Adapter 0, VD 1 (target id: 1) success
Exit Code: 0x00
[root@localhost ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LdSetProp NORA -L1 -a0
Set Read Policy to NoReadAhead on Adapter 0, VD 1 (target id: 1) success
Exit Code: 0x00
读缓存修改示例:
[root@localhost ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LdSetProp WB -L1 -a0
Set Write Policy to WriteBack on Adapter 0, VD 1 (target id: 1) success
Exit Code: 0x00
[root@localhost ~]#/opt/MegaRAID/MegaCli/MegaCli64 -LdSetProp WT -L1 -a0
Set Write Policy to WriteThrough on Adapter 0, VD 1 (target id: 1) success
Exit Code: 0x00
[root@localhost ~]#/opt/MegaRAID/MegaCli/MegaCli64 -LdSetProp ForcedWB -L1 -a0
Set Write Policy to Forced WriteBack on Adapter 0, VD 1 (target id: 1) success
Exit Code: 0x00
工具3:hpssacli/ssacli
说明:
1. hpssacli工具为旧工具,目前大部分线上服务器都安装有对应的安装包,ssacli为新工具,在旧工具无法正常执行的情况下,建议使用新工
具,语法完全一致。
2. ssdsmartpath(固态硬盘智能路径)和caching(写缓存)只能打开一个,在一个打开的状态下,另外一个的值必定是关闭的,例如:
caching:Enable的情况下,ssdsmartpath的值必定为disabled.
a. 确认安装包是否安装
[root@localhost ~]# rpm -qa | grep hpssacli
hpssacli-2.20-11.0.x86_64
备注:若检索结果为空,需要安装对应的软件安装包。安装命令如下:
#rpm -ivh hpssacli-xx-xx.x86_64.rpm
#rpm -ivh ssacli-xx-xx.x86_64.rpm
b. 确认硬盘控制器所在slot插槽
说明:如下所示, 阵列卡型号为Smart Array P840ar, 插槽位置为Slot 0, 务必查询信息,插槽位置后面的命令要用。
[root@localhost ~]# hpssacli ctrl all show
Smart Array P840ar in Slot 0 (Embedded) (sn: PVYKH0BRH710CO)
c. 确认逻辑卷写缓存(caching)状态
参数说明:
ctrl: Controller 简写
slot=0: 硬盘控制器所在PCIe插槽位置
ld 2: ld(logical drive 逻辑卷),后面的数字为逻辑卷编号,顺序从1开始
show: 信息展示
说明:请关注示例中caching的值,示例中为Disabled(即关闭)
[root@localhost ~]# hpssacli ctrl slot=0 ld 2 show
Smart Array P840ar in Slot 0 (Embedded)
array B
Logical Drive: 2
Size: 2.2 TB
Fault Tolerance: 1+0
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 1280 KB
Status: OK
MultiDomain Status: OK
Caching: Disabled
Unique Identifier: 600508B1001C0D8EB99EB2A7EE5060E4
Disk Name: /dev/sdb
Mount Points: /opt/uredis 2.2 TB Partition Number 1
Logical Drive Label: 06172624PVYKH0BRH710COB70A
Mirror Group 1:
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, Solid State SATA, 480.1 GB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, Solid State SATA, 480.1 GB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, Solid State SATA, 480.1 GB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, Solid State SATA, 480.1 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, Solid State SATA, 480.1 GB, OK)
Mirror Group 2:
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, Solid State SATA, 480.1 GB, OK)
physicaldrive 2I:2:1 (port 2I:box 2:bay 1, Solid State SATA, 480.1 GB, OK)
physicaldrive 2I:2:2 (port 2I:box 2:bay 2, Solid State SATA, 480.1 GB, OK)
physicaldrive 2I:2:3 (port 2I:box 2:bay 3, Solid State SATA, 480.1 GB, OK)
physicaldrive 2I:2:4 (port 2I:box 2:bay 4, Solid State SATA, 480.1 GB, OK)
Drive Type: Data
LD Acceleration Method: HP SSD Smart Path
d. 确认逻辑卷ssdsmartpath状态
参数说明:
ctrl: Controller 简写
slot=0: 硬盘控制器所在PCIe插槽位置
array b: ld(array, 磁盘阵列,也即逻辑卷),后面的字母为逻辑卷编号,顺序从a开始
show: 信息展示
说明:请关注示例中ssdsmartpath的值,示例中为Enable(即关闭)
[root@localhost ~]# hpssacli ctrl slot=0 array b show
Smart Array P840ar in Slot 0 (Embedded)
Array: B
Interface Type: Solid State SATA
Unused Space: 0 MB (0.0%)
Used Space: 4.4 TB (100.0%)
Status: OK
MultiDomain Status: OK
Array Type: Data
HP SSD Smart Path: enable
e. 关闭固态硬件智能路径并打开写缓存示例
[root@localhost ~]# hpssacli ctrl slot=0 array b modify ssdsmartpath=disable
[root@localhost ~]# hpssacli ctrl slot=0 ld 2 modify caching=enable
f. 关闭写缓存并打开固态硬盘智能路径
[root@localhost ~]# hpssacli ctrl slot=0 ld 2 modify caching=disable
[root@localhost ~]# hpssacli ctrl slot=0 array b modify ssdsmartpath=enable
工具4: arcconf
a. 确认软件包是否安装
[root@localhost ~]# rpm -qa | grep Arcconf
Arcconf-2.04-22665.x86_64
备注:若检索结果为空,需要安装对应的软件安装包。安装命令如下:
#rpm -ivh Arcconf-xx-xx.x86_64.rpm
b. 确认硬盘控制器信息
说明:控制器编号从1开始,所以示例中控制器ID为1:
[root@localhost ~]# /usr/Arcconf/arcconf list
Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller ID : Status, Slot, Mode, Name, SerialNumber, WWN
----------------------------------------------------------------------
Controller 1: : Optimal, Slot 2147483647, RAID (Expose RAW), , FB488000, 560DA833FB488000
c. 查看逻辑卷读/写缓存状态
备注:
1. 请关注标红部分示例为:参数说明如下:
Read-cache setting : Enabled(读缓存打开)
Write-cache setting : Enabled(写缓存打开)
2. 示例命令行参数说明(/usr/Arcconf/arcconf getconfig 1 ld 1)
getconfig: prints controller information(打印控制器相关信息)
第一个1:控制器编号
ld:logical drive(逻辑卷,即磁盘阵列)
第二个1:逻辑卷编号,从0开始,示例是第二个逻辑卷,所以为1.
[root@localhost ~]# /usr/Arcconf/arcconf getconfig 1 ld 1
Controllers found: 1
----------------------------------------------------------------------
Logical device information
----------------------------------------------------------------------
Logical Device number 1
Logical Device name : data
Block Size of member drives : 512 Bytes
RAID level : 10
Unique Identifier : 6CDE0BB1
Status of Logical Device : Optimal
Additional details : Quick initialized
Size : 1014 MB
Parity space : 1024 MB
Stripe-unit size : 256 KB
Interface Type : Serial ATA
Device Type : HDD
Read-cache setting : Enabled
Read-cache status : On
Write-cache setting : Enabled
Write-cache status : On
Partitioned : No
Protected by Hot-Spare : No
Bootable : No
Failed stripes : No
Power settings : Disabled
--------------------------------------------------------
Logical Device segment information
--------------------------------------------------------
Group 0, Segment 0 : Present (3815447MB, SATA, HDD, Enclosure:0, Slot:2) K4KZMMKL
Group 0, Segment 1 : Present (3815447MB, SATA, HDD, Enclosure:0, Slot:3) K7G2L3SL
Group 1, Segment 0 : Present (3815447MB, SATA, HDD, Enclosure:0, Slot:4) K4KYTSLL
Group 1, Segment 1 : Present (3815447MB, SATA, HDD, Enclosure:0, Slot:5) K7G183TL
d. 读/写缓存设置语法
语法如下:
Usage: SETCACHE <Controller#> LOGICALDRIVE <LogicalDrive#> <logical mode> [noprompt] [nologs]
Usage: SETCACHE <Controller#> DEVICE <Channel# ID#> <physical mode> [nologs]
Usage: SETCACHE <Controller#> DEVICEALL <policy> [noprompt] [nologs]
Example: SETCACHE 1 LOGICALDRIVE 0 ron
Example: SETCACHE 1 DEVICE 0 0 wt
Example: SETCACHE 1 DEVICEALL drivespecific
===================================================================================
Changes a device's cache mode.
LOGICALDRIVE parameters
LogicalDrive# : Number of the logical device whose cache will be altered
Logical Modes : ron - read cache enabled
roff - read cache disabled
wt - write through
wb - write back
wbb - write back with battery/ZMM
con - cache enabled
coff - cache disabled
DEVICE parameters
Channel# ID# : Channel and ID of the device whose cache will be altered
Physical Modes : wt - write through
wb - write back
DEVICEALL parameters
Policy : disable - write back for all physical drives
enable - write through for all physical drives
drivespecific - user can set for an individual physical drive
部分参数说明:
rron: read cache enabled(打开读缓存)
roff: read cache disabled(关闭读缓存)
wt: write through(只写模式,即关闭写缓存)
wb:write back(回写模式,即打开写缓存)
con: cache enabled(打开逻辑卷缓存(打开读/写缓存前提是逻辑卷缓存已打开,在逻辑卷关闭的情况下,无法打开读/写缓存))
coff: cache disabled(关闭逻辑卷缓存)
noprompt:no prompt(没有提示,按照默认y执行,部分命令需要和用户交互,使用noprompt参数可按照默认yes答复执行)
e. 读缓存打开/关闭示例
[root@localhost ~]# /usr/Arcconf/arcconf setcache 1 logicaldrive 1 roff
Controllers found: 1
Command completed successfully.
[root@localhost ~]# /usr/Arcconf/arcconf setcache 1 logicaldrive 1 ron
Controllers found: 1
Command completed successfully.
f. 写缓存打开/关闭示例
[root@localhost ~]# /usr/Arcconf/arcconf setcache 1 logicaldrive 1 wt
Controllers found: 1
Command completed successfully.
You have new mail in /var/spool/mail/root
[root@localhost ~]# /usr/Arcconf/arcconf setcache 1 logicaldrive 1 wb
Controllers found: 1
WARNING: Power failure without battery/ZMM support will lead to data loss.
Do you wish to continue?
Press y, then ENTER to continue or press ENTER to abort: y
Command completed successfully.
[root@localhost ~]# /usr/Arcconf/arcconf setcache 1 logicaldrive 1 wbb
Controllers found: 1
Command completed successfully