首页 > 代码库 > rac存储替换
rac存储替换
2017-06-24
实验步骤：
一、将所有节点关机，生成与原共享盘大小相同的几块盘即可，挂到所有的节点上，磁盘为固定大小，状态为共享
二、将所有节点开机，编辑文件/etc/udev/rules.d/60-raw.rules，设置共享磁盘的权限，内容如下所示
ACTION=="add", KERNEL=="sdb", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdd", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sde", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="sdf", RUN+="/bin/raw /dev/raw/raw5 %N"
ACTION=="add", KERNEL=="sdg", RUN+="/bin/raw /dev/raw/raw6 %N"
ACTION=="add", KERNEL=="sdh", RUN+="/bin/raw /dev/raw/raw7 %N"
ACTION=="add", KERNEL=="sdi", RUN+="/bin/raw /dev/raw/raw8 %N"
ACTION=="add", KERNEL=="sdj", RUN+="/bin/raw /dev/raw/raw9 %N"
ACTION=="add", KERNEL=="sdk", RUN+="/bin/raw /dev/raw/raw10 %N"
KERNEL=="raw1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw6", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw7", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw8", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw9", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="raw10", OWNER="grid", GROUP="asmadmin", MODE="0660"
所有节点加载文件，使编辑生效
[root@rac1 rules.d]# start_udev
Starting udev: [ OK ]
检查结果
[root@rac2 rules.d]# ls -l /dev/raw*
total 0
crw-rw---- 1 grid asmadmin 162, 1 Apr 6 15:18 raw1
crw-rw---- 1 grid asmadmin 162, 10 Apr 6 15:18 raw10
crw-rw---- 1 grid asmadmin 162, 2 Apr 6 15:18 raw2
crw-rw---- 1 grid asmadmin 162, 3 Apr 6 15:18 raw3
crw-rw---- 1 grid asmadmin 162, 4 Apr 6 15:18 raw4
crw-rw---- 1 grid asmadmin 162, 5 Apr 6 15:18 raw5
crw-rw---- 1 grid asmadmin 162, 6 Apr 6 15:18 raw6
crw-rw---- 1 grid asmadmin 162, 7 Apr 6 15:18 raw7
crw-rw---- 1 grid asmadmin 162, 8 Apr 6 15:18 raw8
crw-rw---- 1 grid asmadmin 162, 9 Apr 6 15:18 raw9
crw-rw---- 1 root disk 162, 0 Apr 6 15:18 rawctl
三、在任一节点使用grid用户运行asmca，使用图形界面将裸盘分组
分组策略：
数据盘使用DATA_NEW，冗余模式为external，一块盘大小为15G
归档盘使用ARCD_NEW,冗余模式为external，一块盘大小为4G
OCR、VOTE使用相同的磁盘组 OCR_NEW，冗余模式为normal，三块盘，每块大小1G
下图为将磁盘分组后的结果
四、替换OCR与VOTE盘
检查OCR的备份，使用grid用户的环境变量，以root身份查询
[root@rac1 rules.d]# su - grid
[grid@rac1 ~]$ su
Password:
[root@rac2 grid]# ocrconfig -showbackup
rac2 2017/04/05 13:56:48 /u01/11.2.0/grid/cdata/rac-cluster/backup00.ocr
rac2 2017/04/03 21:08:52 /u01/11.2.0/grid/cdata/rac-cluster/backup01.ocr
rac1 2017/04/01 19:01:38 /u01/11.2.0/grid/cdata/rac-cluster/backup02.ocr
rac2 2017/04/05 13:56:48 /u01/11.2.0/grid/cdata/rac-cluster/day.ocr
rac1 2017/03/31 13:16:38 /u01/11.2.0/grid/cdata/rac-cluster/week.ocr
检查当前ocr设置信息
[root@rac2 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3472
Available space (kbytes) : 258648
ID : 343097207
Device/File Name : +OCR
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
当前使用的OCR为旧存储，现将其换成新的ORC_NEW
添加新的OCR_NEW，在添加新的磁盘的同时，OCR信息写入到新的磁盘中
[root@rac1 grid]# ocrconfig -add +OCR_NEW
检查添加结果
[root@rac1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3460
Available space (kbytes) : 258660
ID : 343097207
Device/File Name : +OCR
Device/File integrity check succeeded
Device/File Name : +OCR_NEW
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
删除原有的OCR磁盘组，并检查
[root@rac1 grid]# ocrconfig -delete +OCR
[root@rac1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3460
Available space (kbytes) : 258660
ID : 343097207
Device/File Name : +OCR_NEW
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
替换VOTE盘
检查VOTE信息
[grid@rac1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 2a722d08a6a34fe2bfbd8c8e15f9a3a4 (/dev/raw/raw1) [OCR]
2. ONLINE 79a3bd29995a4fd3bf50eeae98b33e95 (/dev/raw/raw2) [OCR]
3. ONLINE aa005c3d91254fb0bf49789a07bcbbe9 (/dev/raw/raw3) [OCR]
Located 3 voting disk(s).
替换VOTE
[grid@rac1 ~]$ crsctl replace votedisk +OCR_NEW
Successful addition of voting disk 3bcdec446c2a4f3ebf35a222d982f233.
Successful addition of voting disk 39c04a0f01d54fabbfa2bb0e956cb9c8.
Successful addition of voting disk 01ae7539f3644fbebf039a38f8bf28df.
Successful deletion of voting disk 2a722d08a6a34fe2bfbd8c8e15f9a3a4.
Successful deletion of voting disk 79a3bd29995a4fd3bf50eeae98b33e95.
Successful deletion of voting disk aa005c3d91254fb0bf49789a07bcbbe9.
Successfully replaced voting disk group with +OCR_NEW.
CRS-4266: Voting file(s) successfully replaced
五、数据盘移动规划，本次实验使用rman进行数据库的移动
1、全备数据库
RMAN> run{
allocate channel c1 device type disk;
backup database format ‘/backup/full_%d_%T_%s_%p.bak‘;
sql ‘alter system archive log current‘;
sql ‘alter system archive log current‘;
sql ‘alter system archive log current‘;
backup archivelog all format ‘/backup/arch_%d_%T_%s_%p.bak‘;
backup current controlfile format ‘/backup/ctl_%U‘;
release channel c1;
}
2、将spfile导出到pfile，修改参数
SQL> create pfile=‘/home/oracle/init.ora‘ from spfile;
将asm实例的spfile也导出到新ocr盘中，原理与数据库实例spfile相同，此处过程略
File created.
[oracle@rac1 ~]$ vim init.ora
修改其中控制文件位置
*.control_files=‘+DATA_NEW/rac/controlfile/ctl‘
根据提供的路径建立相应的目录
ASMCMD> pwd
+data_new/rac
ASMCMD> mkdir controlfile
ASMCMD> ls
PARAMETERFILE/
controlfile/
关闭实例，将参数从pfile导入到spfile
SQL> create spfile from pfile=‘/home/oracle/init.ora‘;
3、将本地新生成的spfile cp到新的共享磁盘组位置，修改本地init文件使其指向新的spfile位置
ASMCMD> pwd
+data_new/rac
ASMCMD> ls
controlfile/
ASMCMD> cp /u01/app/oracle/product/11.2.0/db_1/dbs/spfilerac1.ora spfilerac.ora
copying /u01/app/oracle/product/11.2.0/db_1/dbs/spfilerac1.ora -> +data_new/rac/spfilerac.ora
注意：在转移结束后，将本地存储的spfile删除，否则在数据库实例开启时，会优先使用本地spfile
修改init文件使其指向新参数文件位置
[oracle@rac1 dbs]$ pwd
/u01/app/oracle/product/11.2.0/db_1/dbs
[oracle@rac1 dbs]$ vim initrac1.ora
[oracle@rac1 dbs]$ cat initrac1.ora
SPFILE=‘+DATA_NEW/RAC/spfilerac.ora‘
4、启动数据库实例，并检查使用的参数文件
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 1152450560 bytes
Fixed Size 2252584 bytes
Variable Size 1006633176 bytes
Database Buffers 134217728 bytes
Redo Buffers 9347072 bytes
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA_NEW/rac/spfilerac.ora
5、恢复控制文件
RMAN> restore controlfile from ‘/backup/ctl_0ts113s2_1_1‘;
6、将数据库启动到mount阶段，检查数据文件物理结构
RMAN> alter database mount;
database mounted
released channel: ORA_DISK_1
RMAN> report schema;
RMAN-06139: WARNING: control file is not current for REPORT SCHEMA
Report of database schema for database with db_unique_name RAC
List of Permanent Datafiles
===========================
File Size(MB) Tablespace RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1 780 SYSTEM *** +DATADG/rac/datafile/system.272.938517081
2 640 SYSAUX *** +DATADG/rac/datafile/sysaux.277.938517083
3 25 UNDOTBS1 *** +DATADG/rac/datafile/undotbs1.282.940501517
4 5 USERS *** +DATADG/rac/datafile/users.281.938517083
5 346 EXAMPLE *** +DATADG/rac/datafile/example.283.938517211
6 25 UNDOTBS2 *** +DATADG/rac/datafile/undotbs2.276.938517477
7 10 FANGFANG *** +DATADG/rac/datafile/fangfang.dbg
List of Temporary Files
=======================
File Size(MB) Tablespace Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1 36 TEMP 32767 +DATADG/rac/tempfile/temp.284.938517201
此时的控制文件文件记录的数据文件物理位置为旧存储，运行还原脚本，将还原路径指定到新的数据存储位置
注意：此处要求与原数据文件后缀号区分，且不同
7、运行还原数据库脚本，将数据文件还原到新的存储位置
run{
allocate channel c1 device type disk;
set newname for datafile 1 to ‘+DATA_NEW/rac/datafile/system.dbf‘;
set newname for datafile 2 to ‘+DATA_NEW/rac/datafile/sysaux.dbf‘;
set newname for datafile 3 to ‘+DATA_NEW/rac/datafile/undotbs1.dbf‘;
set newname for datafile 4 to ‘+DATA_NEW/rac/datafile/users.dbf‘;
set newname for datafile 5 to ‘+DATA_NEW/rac/datafile/undotbs2.dbf‘;
restore database;
switch datafile all;
release channel c1;
}
注意：在运行脚本前，确定目录的存在
恢复数据库
recover database
注意：在恢复完数据库后，打开数据库检查各个文件是否使用的新存储中文件
使用oracle用户将新数据库的信息，及新数据库实例文件的位置信息注册到srv控制器中
（1）删除数据库（仅删除注册信息）
srvctl remove database -d rac
-d为数据库唯一名
（2）增加数据库
srvctl add database -d rac -o /u01/app/oracle/product/11.2.0/db_1 -p +DATA_NEW/rac/spfilerac.ora
-d为数据库唯一名 -o为oracle家目录 -p为参数文件路径
（3）为所有节点添加数据库实例
srvctl add instance -d rac -i rac1 -n rac1
srvctl add instance -d rac -i rac2 -n rac2
-d为数据库唯一名 -i 为数据库实例名 -n为集群节点名
（4）使用srv管理器启动数据库（将数据库设置为自启动）
srvctl start database -d rac
8、将原redo数据切入归档，在新存储位置重建redo log
查看日志组状态
set linesize 150;
set pagesize 50;
column TYPE format a8;
column MEMBER format a60;
select lf.GROUP#,l.THREAD#,lf.TYPE,l.BYTES/1024/1024 MB,lf.MEMBER from v$logfile lf,v$log l where lf.GROUP#=l.GROUP#;
重建在线日志文件
SQL> alter database add logfile thread 1 group 5 ‘+DATA_NEW/rac/onlinelog/group_5_redo‘ size 50m;
Database altered.
SQL> alter database add logfile thread 1 group 6 ‘+DATA_NEW/rac/onlinelog/group_6_redo‘ size 50m;
Database altered.
SQL> alter database add logfile thread 2 group 7 ‘+DATA_NEW/rac/onlinelog/group_7_redo‘ size 50m;
Database altered.
SQL> alter database add logfile thread 2 group 8 ‘+DATA_NEW/rac/onlinelog/group_8_redo‘ size 50m;
Database altered.
查看当前日志状态
set linesize 150;
set pagesize 50;
column TYPE format a8;
column MEMBER format a60;
select lf.GROUP#,l.THREAD#,lf.TYPE,l.BYTES/1024/1024 MB,lf.MEMBER from v$logfile lf,v$log l where lf.GROUP#=l.GROUP#;
切换日志页将一节点至第五组，二节点切换至第七组，再次检查
SQL> alter system archive log current;
System altered.
SQL> /
System altered.
删除位于原存储的1、2、3、4组，在删除之前先清空redolog中的数据
SQL> alter database clear logfile group 1;
Database altered.
SQL> alter database clear logfile group 2;
Database altered.
SQL> alter database clear logfile group 3;
Database altered.
SQL> alter database clear logfile group 4;
Database altered.
SQL> alter database drop logfile group 1;
Database altered.
SQL> alter database drop logfile group 2;
Database altered.
SQL> alter database drop logfile group 3;
Database altered.
SQL> alter database drop logfile group 4;
Database altered.
再次查看
9、在新存储重新建立临时表空间
SQL> CREATE TEMPORARY TABLESPACE temp01 TEMPFILE ‘+DATA_NEW/rac/tempfile/temp001.dbf‘ SIZE 200m AUTOEXTEND ON NEXT 32m MAXSIZE 2048m EXTENT MANAGEMENT LOCAL;
Tablespace created.
将新的临时表空间设置为默认
SQL> ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp01;
Database altered.
删除原存储的临时表空间，如果长时间删除失败，重新登陆sqlplus再次删除即可
SQL> DROP TABLESPACE temp INCLUDING CONTENTS AND DATAFILES;
Tablespace dropped.
六、安全删除物理存储
1、删除原磁盘组
在grid用户下使用sqlplus / as sysasm登陆asm实例
在其他节点卸载旧磁盘组
SQL> alter diskgroup DATADG dismount;
Diskgroup altered.
SQL> alter diskgroup ARCHDG dismount;
Diskgroup altered.
SQL> alter diskgroup OCR dismount;
Diskgroup altered.
2、返回节点一，删除磁盘组
SQL> drop diskgroup DATADG including contents;
Diskgroup dropped.
SQL> drop diskgroup ARCHDG including contents;
Diskgroup dropped.
SQL> drop diskgroup OCR including contents;
Diskgroup dropped.
3、关闭节点，从所有节点移除物理盘（生产环境直接将旧存储拔下即可）
删除上述五块共享旧盘
4、开启节点，使用root用户检查当前操作系统识别的磁盘
[root@rac1 ~]# fdisk -l
Disk /dev/sda: 85.9 GB, 85899280384 bytes
255 heads, 63 sectors/track, 10443 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00013815
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 26 791 6144000 82 Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3 791 10444 77535232 83 Linux
Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdc: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdd: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sde: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdf: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
仔细观察各盘的大小与名称的对应关系
编辑磁盘规则文件，按照上述查找结果生成正确的对应信息
1、如果直接使用的裸盘，执行start_udev加载并刷新对应信息即可
2、如果使用的asm磁盘，编辑/etc/udev/rules.d/99-oracle-asmdevices.rules文件，将其名称对应关系手工编辑为操作系统识别的对应关系
rac存储替换