oracle 11gr2 linux安装 mutipath 链路变更问题解决_billy

http://blog.sina.com.cn/u/1863394114

首页博文目录关于我

个人资料

微博

加好友发纸条

写留言加关注

博客等级：
博客积分：

博客访问：
关注人气：
获赠金笔：0支
赠出金笔：0支
荣誉徽章：

正文字体大小：大中小

oracle 11gr2 linux安装 mutipath 链路变更问题解决

(2012-03-01 11:11:21)

标签：

数据库

分类： oracle

在对一套oracle 11gr2版本进行安装三个节点，但三个节点找到的multipath盘对应的dm路径都不一致并且重启后对应的dm的路径还变化：

[root@db3 ~]# multipath -l|grep dm
mpath2 (360014380125d8a5200003000008a0000) dm-1 HP,HSV400
| |- 3:0:1:1 sdm 8:192 active undef running
mpath1 (360014380125d8a5200003000008e0000) dm-0 HP,HSV400
mpath9 (360014380125d8a520000300000aa0000) dm-8 HP,HSV400
mpath8 (360014380125d8a520000300000a20000) dm-6 HP,HSV400
mpath7 (360014380125d8a520000300000a60000) dm-7 HP,HSV400
mpath6 (360014380125d8a5200003000009a0000) dm-4 HP,HSV400
mpath5 (360014380125d8a5200003000009e0000) dm-3 HP,HSV400
mpath11 (360014380125d8a520000300000b20000) dm-10 HP,HSV400
mpath4 (360014380125d8a520000300000960000) dm-5 HP,HSV400
mpath10 (360014380125d8a520000300000ae0000) dm-9 HP,HSV400
mpath3 (360014380125d8a520000300000920000) dm-2 HP,HSV400

[root@db1 ~]# multipath -l|grep dm
mpath2 (360014380125d8a520000300000920000) dm-2 HP,HSV400
mpath1 (360014380125d8a5200003000008e0000) dm-0 HP,HSV400
mpath9 (360014380125d8a520000300000ae0000) dm-9 HP,HSV400
mpath8 (360014380125d8a520000300000aa0000) dm-8 HP,HSV400
mpath7 (360014380125d8a520000300000a60000) dm-6 HP,HSV400
mpath6 (360014380125d8a520000300000a20000) dm-7 HP,HSV400
mpath5 (360014380125d8a5200003000009e0000) dm-4 HP,HSV400
mpath11 (360014380125d8a5200003000008a0000) dm-1 HP,HSV400
| |- 3:0:1:1 sdm 8:192 active undef running
mpath4 (360014380125d8a5200003000009a0000) dm-5 HP,HSV400
mpath10 (360014380125d8a520000300000b20000) dm-10 HP,HSV400
mpath3 (360014380125d8a520000300000960000) dm-3 HP,HSV400

[root@db2 oraInventory]# multipath -l|grep dm
mpath2 (360014380125d8a520000300000920000) dm-2 HP,HSV400
mpath1 (360014380125d8a5200003000008e0000) dm-1 HP,HSV400
mpath9 (360014380125d8a520000300000ae0000) dm-9 HP,HSV400
mpath8 (360014380125d8a520000300000aa0000) dm-8 HP,HSV400
mpath7 (360014380125d8a520000300000a60000) dm-7 HP,HSV400
mpath6 (360014380125d8a520000300000a20000) dm-6 HP,HSV400
mpath5 (360014380125d8a5200003000009e0000) dm-5 HP,HSV400
mpath11 (360014380125d8a5200003000008a0000) dm-0 HP,HSV400
| |- 3:0:1:1 sdm 8:192 active undef running
mpath4 (360014380125d8a5200003000009a0000) dm-4 HP,HSV400
mpath10 (360014380125d8a520000300000b20000) dm-10 HP,HSV400
mpath3 (360014380125d8a520000300000960000) dm-3 HP,HSV400

通过更改/etc/multipath/bindings和/etc/multipath.conf文件将三台机器对应的mpath路径对应一致，但是dm路径还是不一致,只能试用此时的/dev/mapper/multipath盘作为ASM 共享disk。

fdisk /dev/mapper/multipath*进行划分分区，安装grid infrastructure在进行创建asm卷组时，使用/dev/mapper/multipath报“不是可用的asm盘”,由于此时没有安装oracleasm、asmlib、asm-support无法判断问题到底是出在哪里，由于安装的版本是red hat linux 6.2，oracle官网已经对此版本没有对asmlib包进行支持，只能重新安装操作系统，将操作系统版本降低到linux 5.7。

安装完成操作系统并且将oracleasm三个包配置完成，通过oracleasm命令进行添加multipath盘：

oracleasm createdisk DISK1 /dev/mapper/mpath*p1

另外两台机器进行加载，加载成功并且能找到添加的盘符。但是在/dev/oracleasm/disks/ 目录中所创建的ASM DISK硬盘三台机器所对应的major和minor号不一致，此时只能试着进行grid安装。在进行asm 卷组创建时，添加asm DISK成功，但是在检查安装环境时，报DISK不是共享磁盘错误:

Device Checks for ASM - This is a pre-check to verify if the specified devices meet the requirements for configuration through the Oracle Universal Storage Manager Configuration Assistant. Error:
-
"/dev/oracleasm/disks/DISK1" is not shared - Cause: Cause Of Problem Not Available - Action: User Action Not Available
-
"/dev/oracleasm/disks/DISK2" is not shared - Cause: Cause Of Problem Not Available - Action: User Action Not Available
-
"/dev/oracleasm/disks/DISK3" is not shared - Cause: Cause Of Problem Not Available - Action: User Action Not Available
-
Could not find the storage - Cause: Cause Of Problem Not Available - Action: User Action Not Available
-
Could not find the storage - Cause: Cause Of Problem Not Available - Action: User Action Not Available

Operation Failed on Nodes: [db3, db2, db1]
Verification result of failed node: db3

Details:
-
PRVF-5150 : Path /dev/oracleasm/disks/DISK5 is not a valid path on all nodes - Cause: - Action:
-
PRVF-5150 : Path /dev/oracleasm/disks/DISK4 is not a valid path on all nodes - Cause: - Action:
-
PRVF-5149 : WARNING: Storage "/dev/oracleasm/disks/DISK3" is not shared on all nodes - Cause: - Action:
-
PRVF-5149 : WARNING: Storage "/dev/oracleasm/disks/DISK2" is not shared on all nodes - Cause: - Action:
-
PRVF-5149 : WARNING: Storage "/dev/oracleasm/disks/DISK1" is not shared on all nodes - Cause: - Action:
忽略此错误，继续安装，当进行第一个节点运行root.sh脚本时，创建data asm卷组成功，但是进行第二、三节点运行root.sh，报以下无法挂载DATA卷组错误：

Mounting Disk Group DATA failed with the following message:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
ORA-15080: synchronous I/O operation to a disk failed

Configuration of ASM ... failed
see asmca logs at /oracle/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /oracle/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6763.
/oracle/app/11.2.0/grid/perl/bin/perl -I/oracle/app/11.2.0/grid/perlb -I/oracle/app/11.2.0/grid/crs/install /oracle/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

通过查询oracle support文档ID 1384504.1中说明中，由于操作系统启动中ASM进程先扫描SD开头的磁盘，来代替multipath（DM）磁盘，所以导致三台机器盘符不一致。

A 'cat /proc/partitions', show the asmlib devices are linked to the 'sd' devices instead the 'multipath (dm)' devices. These asmlib devices used to be linked to the multipath devices before the problem:
解决方法就是通过更改asm扫描盘符设备顺序，先扫描multipath 和dm盘，屏蔽sd盘。

The asmlib file (/etc/sysconfig/oracleasm) to do with mapping asmlib devices to multipath devices, seems is not configure for multipathing:

ORACLEASM_ENABLED=true

ORACLEASM_UID=oracle

ORACLEASM_GID=oinstall

ORACLEASM_SCANBOOT=true

ORACLEASM_SCANORDER=""

ORACLEASM_SCANEXCLUDE=""

After ORACLEASM_SCANORDER and ORACLEASM_EXCLUDE were updated in /etc/sysconfig/oracleasm, ASM was able to display the asmlib disks with v$asm_disk.header_status=MEMBER. Thereafter, ASM was able to mount the diskgroup.

ORACLEASM_SCANORDER="mpath dm"

ORACLEASM_SCANEXCLUDE="sd"

但是通过/proc/partiton文件查看到，三台机器sd的major和minor是不变的，dm一直在变化：

[root@db2 oraInventory]# more /proc/partitions
major minor #blocks name

   8        0 291991552 sda
   8        1     104391 sda1
   8        2   41945715 sda2
   8        3   41945715 sda3
   8        4          1 sda4
   8        5   33463363 sda5
   8        6   10482381 sda6
   8        7 164047716 sda7
   8       16 2147483648 sdb
   8       17 2147480811 sdb1
   8       32 2147483648 sdc
   8       33 2147480811 sdc1
   8       48 2147483648 sdd
   8       49 2147480811 sdd1
   8       64 2147483648 sde
   8       65 2147480811 sde1
   8       80 2147483648 sdf
   8       81 2147480811 sdf1
   8       96 2147483648 sdg
   8       97 2147480811 sdg1
   8      112 2147483648 sdh
   8      113 2147480811 sdh1
   8      128 2147483648 sdi
   8      129 2147480811 sdi1
   8      144 2147483648 sdj
   8      145 2147480811 sdj1
   8      160 2147483648 sdk
   8      161 2147480811 sdk1
   8      176 2147483648 sdl
   8      177 2147480811 sdl1
   8      192 2147483648 sdm
   8      193 2147480811 sdm1
   8      208 2147483648 sdn
   8      209 2147480811 sdn1
   8      224 2147483648 sdo
   8      225 2147480811 sdo1
   8      240 2147483648 sdp
   8      241 2147480811 sdp1

将/etc/sysconfig/oracleasm内容更改为：

ORACLEASM_ENABLED=true
ORACLEASM_UID=oracle
ORACLEASM_GID=oinstall
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="mpath sd"
ORACLEASM_SCANEXCLUDE=""

重启三台机器后,/dev/oracleasm/disks/的major和minor一致

[root@db1 disks]# ls -lrt
total 0
brw-rw---- 1 grid asmadmin 8, 81 Feb 29 22:49 DISK7
brw-rw---- 1 grid asmadmin 8, 97 Feb 29 22:49 DISK6
brw-rw---- 1 grid asmadmin 8, 113 Feb 29 22:49 DISK9
brw-rw---- 1 grid asmadmin 8, 129 Feb 29 22:49 DISK8
brw-rw---- 1 grid asmadmin 8, 145 Feb 29 22:49 DISK10
brw-rw---- 1 grid asmadmin 8, 161 Feb 29 22:49 DISK11
brw-rw---- 1 grid asmadmin 8, 33 Feb 29 22:54 DISK2
brw-rw---- 1 grid asmadmin 8, 49 Feb 29 22:57 DISK4
brw-rw---- 1 grid asmadmin 8, 65 Feb 29 22:58 DISK5
brw-rw---- 1 grid asmadmin 8, 17 Feb 29 22:58 DISK3
brw-rw---- 1 grid asmadmin 8, 177 Feb 29 22:58 DISK1

再进行安装一切顺利，以下是三节点rac资源：

[grid@db1 ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    db1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    db1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    db1
ora.asm        ora.asm.type   ONLINE    ONLINE    db1
ora.cvu        ora.cvu.type   ONLINE    ONLINE    db1
ora....SM1.asm application    ONLINE    ONLINE    db1
ora....B1.lsnr application    ONLINE    ONLINE    db1
ora.db1.gsd    application    OFFLINE   OFFLINE
ora.db1.ons    application    ONLINE    ONLINE    db1
ora.db1.vip    ora....t1.type ONLINE    ONLINE    db1
ora....SM2.asm application    ONLINE    ONLINE    db2
ora....B2.lsnr application    ONLINE    ONLINE    db2
ora.db2.gsd    application    OFFLINE   OFFLINE
ora.db2.ons    application    ONLINE    ONLINE    db2
ora.db2.vip    ora....t1.type ONLINE    ONLINE    db2
ora....SM3.asm application    ONLINE    ONLINE    db3
ora....B3.lsnr application    ONLINE    ONLINE    db3
ora.db3.gsd    application    OFFLINE   OFFLINE
ora.db3.ons    application    ONLINE    ONLINE    db3
ora.db3.vip    ora....t1.type ONLINE    ONLINE    db3
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora.mdsoss.db ora....se.type ONLINE    ONLINE    db1
ora....network ora....rk.type ONLINE    ONLINE    db1
ora.oc4j       ora.oc4j.type ONLINE    ONLINE    db1
ora.ons        ora.ons.type   ONLINE    ONLINE    db1
ora.scan1.vip ora....ip.type ONLINE    ONLINE    db1

阅读┊ 收藏 ┊ 喜欢 ▼ ┊打印┊举报/Report

前一篇：跨平台零中断数据库迁移

后一篇：AIX环境10g rac+dataguard 更改时区步骤

新浪BLOG意见反馈留言板　欢迎批评指正