UDEV SCSI Rules Configuration for ASM in Oracle Linux 5 and 6

For Oracle Automatic Storage Manager (ASM) to use disks, it needs to be able to identify the devices consistently and for them to have the correct ownership and permissions. In Linux you can use ASMLib to manage these tasks, but it is seen as an additional layer of complexity and has never really gained any popularity. Instead, many people use the Linux device manager "udev" to perform these tasks. This article presents a brief overview of setting up udev rules with respect to disks for use with ASM in Oracle 11g. The examples are all done using Oracle Linux 5 and 6, so they will be consistent with RHEL and CentOS 5 and 6.

• Background
• Identify the Disks (/sbin/scsi_id)
• Make SCSI Devices Trusted
• Create UDEV Rules File
• Load Updated Block Device Partitions (/sbin/partprobe)
• Test Rules (udevtest)
• Restart UDEV Service
• Check Ownership and Permissions

Background

Essentially, what udev does is apply rules defined in files in the "/etc/udev/rules.d" directory to the device nodes listed in the "/dev" directory. The rules can be defined in a variety of ways, but what we need to do is identify the device and say what we want udev to do with it.

In this case I know all my disk devices are named "/dev/sd?1", where the "?

" represents a letter from a-d, so I can identify the devices of interest using the following rule parameters.

KERNEL=="sd?1", BUS=="scsi"

I want to tie each specific device to an alias, so it is always identified the same way, regardless of the device name Linux assigns it. So I need to be able to test each device that matches the previous pattern to see if it is the disk I am interested in. Each disk has a unique SCSI ID, so I can place a test into the rule, telling it how to perform the test, and the result it should return for a positive match. The following rule parameters explain how to test the device and what result constitutes a match in Oracle Linux 5.

PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_"

The scsi_id command works a little differently in Oracle Linux 6, so for that the following test works better.

PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_"

Once we have identified the specific device of interest, we need to indicate what actions should be performed on it. The following parameters specify an alias, the ownership and the permissions for the device.

NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"

So the whole rule for each disk will look something like this in Oracle Linux 5.

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"

Or this in Oracle Linux 6.

KERNEL=="sd?
1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"

This means that the device pointing to the partition "sd*1" on the disk with the SCSI ID of "SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_" will always be called "/dev/asm-disk1", regardless of the letter "?" Linux assigns when the device is discovered. In addition, the device will have the correct ownership and permissions for ASM.

There are a number of wildcards and matching patterns that can be used if you don‘t want to write device-specific rules.

Now we know roughly what we are trying to achieve, we will look at each step necessary for setting up the disks for ASM to use.

Identify the Disks (/sbin/scsi_id)

We are going to write device-specific rules, so we need to be able to identify each device consistently, irrespective of the order in which Linux discovers it. To do this we are going to use the SCSI ID for each disk (not the partition), which we get using the scsi_id command. The "-s" option makes the paths relative to the "/sys" directory. For Oracle Linux 5, use the following command.

# /sbin/scsi_id -g -u -s /block/sdb
SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_
# /sbin/scsi_id -g -u -s /block/sdc
SATA_VBOX_HARDDISK_VB46dec7e0-192e8000_
# /sbin/scsi_id -g -u -s /block/sdd
SATA_VBOX_HARDDISK_VBce8c63bb-ac67a172_
# /sbin/scsi_id -g -u -s /block/sde
SATA_VBOX_HARDDISK_VB7437a3b7-95b199cd_
# 

The "-s" is not available in Oracle Linux 6, so you must use the following syntax.

# /sbin/scsi_id -g -u -d /dev/sdb
SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_
# /sbin/scsi_id -g -u -d /dev/sdc
SATA_VBOX_HARDDISK_VB46dec7e0-192e8000_
# /sbin/scsi_id -g -u -d /dev/sdd
SATA_VBOX_HARDDISK_VBce8c63bb-ac67a172_
# /sbin/scsi_id -g -u -d /dev/sde
SATA_VBOX_HARDDISK_VB7437a3b7-95b199cd_
# 

Make SCSI Devices Trusted

Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn‘t already exist.

options=-g

Create UDEV Rules File

Create the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file.

# vi /etc/udev/rules.d/99-oracle-asmdevices.rules

The file should contain the following lines for Oracle Linux 5. The PROGRAM parameter must match the command you used to retrieve the SCSI ID, and the RESULT parameter must match the value returned from your disks.

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?
1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VB46dec7e0-192e8000_", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VBce8c63bb-ac67a172_", NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="SATA_VBOX_HARDDISK_VB7437a3b7-95b199cd_", NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"

The equivalent for Oracle Linux 6 is shown below.

KERNEL=="sd?
1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?
1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="SATA_VBOX_HARDDISK_VB46dec7e0-192e8000_", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="SATA_VBOX_HARDDISK_VBce8c63bb-ac67a172_", NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="SATA_VBOX_HARDDISK_VB7437a3b7-95b199cd_", NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"

Load Updated Block Device Partitions (/sbin/partprobe)

Load updated block device partition tables.

# /sbin/partprobe /dev/sdb1
# /sbin/partprobe /dev/sdc1
# /sbin/partprobe /dev/sdd1
# /sbin/partprobe /dev/sde1

Test Rules (udevtest)

Test the rules are working as expected.

# #OL5
# udevtest /block/sdb/sdb1
# udevtest /block/sdc/sdc1
# udevtest /block/sdd/sdd1
# udevtest /block/sde/sde1

# #OL6
# udevadm test /block/sdb/sdb1
# udevadm test /block/sdc/sdc1
# udevadm test /block/sdd/sdd1
# udevadm test /block/sde/sde1

The output from the first disk should look something like this.

# udevtest /block/sdb/sdb1
main: looking at device ‘/block/sdb/sdb1‘ from subsystem ‘block‘
udev_rules_get_name: add symlink ‘disk/by-id/scsi-SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3-part1‘
udev_rules_get_name: add symlink ‘disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part1‘
run_program: ‘/lib/udev/vol_id --export /dev/.tmp-8-17‘
run_program: ‘/lib/udev/vol_id‘ returned with status 4
run_program: ‘/sbin/scsi_id -g -u -s /block/sdb/sdb1‘
run_program: ‘/sbin/scsi_id‘ (stdout) ‘SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3_‘
run_program: ‘/sbin/scsi_id‘ returned with status 0
udev_rules_get_name: rule applied, ‘sdb1‘ becomes ‘asm-disk1‘
udev_device_event: device ‘/block/sdb/sdb1‘ already in database, validate currently present symlinks
udev_node_add: creating device node ‘/dev/asm-disk1‘, major = ‘8‘, minor = ‘17‘, mode = ‘0660‘, uid = ‘1100‘, gid = ‘1200‘
udev_node_add: creating symlink ‘/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3-part1‘ to ‘../../asm-disk1‘
udev_node_add: creating symlink ‘/dev/disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part1‘ to ‘../../asm-disk1‘
main: run: ‘socket:/org/kernel/dm/multipath_event‘
main: run: ‘socket:/org/kernel/udev/monitor‘
main: run: ‘/lib/udev/udev_run_devd‘
main: run: ‘socket:/org/freedesktop/hal/udev_event‘
main: run: ‘/sbin/pam_console_apply /dev/asm-disk1 /dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBd306dbe0-df3367e3-part1 /dev/disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part1‘
#

Restart UDEV Service

Restart the UDEV service.

# #OL5
# /sbin/udevcontrol reload_rules

# #OL6
# udevadm control --reload-rules

# #OL5 and OL6
# /sbin/start_udev

Check Ownership and Permissions

Check the disks are now available with the "asm-disk*" alias and the correct ownership and permissions.

# cd /dev
# ls -al asm-disk*
brw-rw---- 1 oracle dba 8, 17 Apr  8 22:47 asm-disk1
brw-rw---- 1 oracle dba 8, 33 Apr  8 22:47 asm-disk2
brw-rw---- 1 oracle dba 8, 49 Apr  8 22:47 asm-disk3
brw-rw---- 1 oracle dba 8, 65 Apr  8 22:47 asm-disk4
#

So the ASM_DISKSTRING initialization parameter in the ASM instance can be set to ‘/dev/asm-disk*‘ to identify the ASM disks.