Replacing a Flash Disk Due to Flash Disk Failure

Each Exadata Storage Server is equipped withfour F20 PCIe cards. Each card has four flash disks (FDOMs) for a total of 16 flash disks. The four F20 PCIe cards are present in PCI slot numbers 1, 2, 4, and 5. The F20 PCIe cards are not hot-pluggable such that Exadata Storage Server must be powered down before replacing the flash disks or cards.

To identify a failed flash disk, use the following command:

CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS=critical DETAIL

         name:                   [9:0:2:0]
         diskType:               FlashDisk
         id:                     508002000092e70FMOD2
         luns:                   1_2
         makeModel:              "MARVELL SD88SA02"
         physicalFirmware:       D20R
         physicalInsertTime:     2009-10-27T13:11:16-07:00
         physicalInterface:      sas
         physicalSerial:         508002000092e70FMOD2
         physicalSize:           22.8880615234375G
         slotNumber:             "PCI Slot: 1; FDOM: 2"
         status:                 critical

The slotNumber attribute shows the PCI slot and the FDOM number.

If an flash disk is detected to have failed, then an alert is generated indicating that the flash disk, as well as the LUN on it, has failed. The alert message includes the PCI slot number of the flash card, and the exact FDOM number. These numbers uniquely identify the field replaceable unit (FRU). If you have configured the system for alert notification, then the alert will be sent by e-mail message to the designated address.

A flash disk outage can cause reduction in performance and dataredundancy. The failed disk should be replaced with a new flash disk at the earliest opportunity. If the flash disk is used for flash cache, then the effective cache size for the cell is reduced. If the flash disk is used for grid disks, then the Oracle ASM disks associated with these grid disks are automatically dropped with the FORCE option from the Oracle ASM disk group and an Oracle ASM rebalance will ensue to restore the data redundancy.

The following procedure describes how to replace an FDOM due to disk failure

1. Inactivate all grid disks on the cell.

2. Shut down the cell.

3. Replace the failed flash disk based on the PCI number and FDOM number.

4. Power up the cell. The cell services are started automatically.

5. Bring all grid disks online using the following command:

   CellCLI> ALTER GRIDDISK ALL ACTIVE
   

6. Verify that all grid disks have been successfully put online using the following command:

   CellCLI> LIST GRIDDISK ATTRIBUTES asmmodestatus
   

   Wait until asmmodestatus shows ONLINE for all grid disks.

The new flash disk is automatically used by the system. If the flash disk is used for flash cache, then the effective cache size will increase. If the flash disk is used for grid disks, then the grid disks will be re-created on the new flash disk. If those grid disks were part of an Oracle ASM disk group, then they will be added back to the disk group and the data will be rebalanced on them based on the disk group redundancy and ASM_POWER_LIMIT parameter.

Oracle ASM rebalance occurs when dropping or adding a disk. To check the status of the rebalance, do the following:

• The rebalance operation may have been successfully run. Check the Oracle ASM alert logs to confirm.

• The rebalance operation may be currently running. Check the GV$ASM_OPERATION view to determine if the rebalance operation is still running.

• The rebalance operation may have failed. Check the V$ASM_OPERATION.ERROR view to determine if the rebalance operation failed.

• Rebalance operations from multiple disk groups can be done on different Oracle ASM instances in the same cluster if the physical disk being replaced contains ASM disks from multiple disk groups. One Oracle ASM instance can run one rebalance operation at a time. If all Oracle ASM instances are busy, then rebalance operations are queued.

Replacing a Flash Disk Due to Flash Disk Problems

Exadata Storage Server is equipped with four F20 PCIe cards. Each card has four flash disks (FDOMs) for a total of 16 flash disks. The four F20 PCIe cards are present on PCI slot numbers 1, 2, 4, and 5. The F20 PCIe cards are not hot-pluggable such that Exadata Storage Server must be powered down before replacing the flash disks or cards.

You may need to replace a flash disk because the disk is in predictive failure status or poor performance status.

To identify a predictive failure flash disk, use the following command:

CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS=‘predictive          failure‘ DETAIL

         name: [9:0:2:0]
         diskType: FlashDisk
         id: 508002000092e70FMOD2
         luns: 1_2
         makeModel: "MARVELL SD88SA02"
         physicalFirmware: D20R
         physicalInsertTime: 2009-10-27T13:11:16-07:00
         physicalInterface: sas
         physicalSerial: 508002000092e70FMOD2
         physicalSize: 22.8880615234375G
         slotNumber: "PCI Slot: 1; FDOM: 2"
         status: predictive failure

To identify a poor performance flash disk, use the following command:

CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS= ‘poor performance‘ DETAIL

         name: [9:0:2:0]
         diskType: FlashDisk
         id: 508002000092e70FMOD2
         luns: 1_2
         makeModel: "MARVELL SD88SA02"
         physicalFirmware: D20R
         physicalInsertTime: 2009-10-27T13:11:16-07:00
         physicalInterface: sas
         physicalSerial: 508002000092e70FMOD2
         physicalSize: 22.8880615234375G
         slotNumber: "PCI Slot: 1; FDOM: 2"
         status: poor performance

An alert is generated when a flash disk is in predictive failure or poor performance status. The alert includes specific instructions for replacing the flash disk. If you have configured the system for alert notifications, then the alerts are sent by e-mail message to the designated address.

Flash disk predictive failure status indicates that the flash disk will fail soon, and should be replaced at the earliest opportunity. If the flash disk is used for flash cache, then it will continue to be used as flash cache. If the flash disk is used for grid disks, then the Oracle ASM disks associated with these grid disks are automatically dropped and Oracle ASM rebalance will relocate the data from the predictively failed disk to other disks. Wait until the Oracle ASM disks have been successfully dropped by querying the V$ASM_DISK_STAT view before proceeding with the flash disk replacement. If the normal drop did not complete before the flash disk fails, then the Oracle ASM disks will be automatically dropped with FORCE option from the Oracle ASM disk group.

If the DROP command did not complete before the flash disk fails, then refer to "Replacing a Flash Disk Due to Flash Disk Failure".

Flash disk poor performance status indicates that the flash disk demonstrates extremely poor performance, and should be replaced at the earliest opportunity. If the flash disk is used for flash cache, then flash cache will be dropped from this disk thus reducing the effective flash cache size for Exadata Storage Server. If the flash disk is used for grid disks, then the Oracle ASM disks associated with the grid disks on this flash disk are automatically dropped with FORCE option if possible. If DROP...FORCE cannot succeed due to offline partners, then the grid disks will be automatically dropped normally and Oracle ASM rebalance will relocate the data from the poor performance disk to other disks.

The following procedure describes how to replace a flash disk due to disk problems:

1. Shut down the cell.

2. Replace the failed flash disk based on the PCI number and FDOM number.

3. Power up the cell. The cell services are started automatically.

4. Bring all grid disks are online using the following command:

   CellCLI> ALTER GRIDDISK ALL ACTIVE
   

5. Verify that all grid disks have been successfully put online using the following command:

   CellCLI> LIST GRIDDISK ATTRIBUTES asmmodestatus
   

   Wait until asmmodestatus shows ONLINE for all grid disks.

The new flash disk is automatically used by the system. If the flash disk is used for flash cache, then the effective cache size will increase. If the flash disk is used for grid disks, then the grid disks will be re-created on the new flash disk. If those gird disks were part of an Oracle ASM disk group, then they will be added back to the disk group and the data will be rebalanced on them based on the disk group redundancy and ASM_POWER_LIMIT parameter.

Oracle ASM rebalance occurs when dropping or adding a disk. To check the status of the rebalance, do the following:

• The rebalance operation may have been successfully run. Check the Oracle ASM alert logs to confirm.

• The rebalance operation may be currently running. Check the GV$ASM_OPERATION view to determine if the rebalance operation is still running.

• The rebalance operation may have failed. Check the V$ASM_OPERATION.ERROR view to determine if the rebalance operation failed.

• Rebalance operations from multiple disk groups can be done on different Oracle ASM instances in the same cluster if the physical disk being replaced contains Oracle ASM disks from multiple disk groups. One Oracle ASM instance can run one rebalance operation at a time. If all Oracle ASM instances are busy, then rebalance operations are queued.

Removing Flash Disk Due to Bad Performance

A single bad flash disk can degrade the performance of other good flash disks. It is better to remove the bad flash disk from the system than let it remain. To identify a bad flash disk, use the CALIBRATE command, and look for very low throughput and IOPS for each flash disk.

If a flash disk exhibits extremely poor performance, then it will be marked as poor performance. The flash cache on that flash disk will be automatically disabled, and the grid disks on that flash disk will be automatically dropped from the Oracle ASM disk group.

The following procedure describes how to remove a flash drive once the bad flash disk has been identified:

1. If the flash disk is used for flash cache, then disable flash cache that is part of the flash disk using the following commands:

   CellCLI > DROP FLASHCACHE
   CellCLI > CREATE FLASHCACHE CELLDISK=‘fd1,fd2,fd3,fd4, ...‘ 
   

   Note:

   Do not include the bad flash disk when creating the new flash cache.

2. If the flash disk is used for grid disks, then use the following command to direct Oracle ASM to stop using the bad disk at once:

   SQL> ALTER DISKGROUP diskgroup_name DROP DISK asm_disk_name FORCE 
   

   It is possible that the DROP command with FORCE option could fail due to offline partners. Either restore the Oracle ASM data redundancy by correcting other cell or disk failures and retry DROP...FORCE, or use the following command to direct Oracle ASM to rebalance the data out of the bad disk:

   SQL> ALTER DISKGROUP diskgroup_name DROP DISK asm_disk_name  NOFORCE
   

3. Wait until the Oracle ASM disks associated with the grid disks on this bad flash disk have been successfully dropped. Oracle Exadata Storage Server Software will automatically send an alert when it is safe to replace the flash disk.

4. Shut down the cell.

   See Also:

   "Shutting Down Exadata Storage Server"

5. Remove the bad flash disk and replace it with a new flash disk.

6. Power up the cell. The cell services are started automatically.

7. Add the new flash disk to flash cache using the following commands:

   CellCLI> DROP FLASHCACHE
   CellCLI> CREATE FLASHCACHE ALL
   

8. Bring all grid disks online using the following command:

   CellCLI> ALTER GRIDDISK ALL ACTIVE
   

9. Verify that all grid disks have been successfully put online using the following command:

   CellCLI> LIST GRIDDISK ATTRIBUTES asmmodestatus
   

   Wait until asmmodestatus shows ONLINE for all grid disks.

The flash disks are added as follows:

• If the flash disk is used for grid disks, then the grid disks will be re-created on the new flash disk.

• If these grid disks were part of an Oracle ASM disk group and DROP...FORCE was used in Step 2, then they will be added back to the disk group and the data will be rebalanced on based on disk group redundancy and the ASM_POWER_LIMIT parameter.

• If DROP...NOFORCE was used in Step 2, then you must manually add the grid disks back to the Oracle ASM disk group.

Oracle ASM rebalance occurs when dropping or adding a disk. To check the status of the rebalance, do the following:

• The rebalance operation may have been successfully run. Check the Oracle ASM alert logs to confirm.

• The rebalance operation may be currently running. Check the GV$ASM_OPERATION view to determine if the rebalance operation is still running.

• The rebalance operation may have failed. Check the V$ASM_OPERATION.ERROR view to determine if the rebalance operation failed.

• Rebalance operations from multiple disk groups can be done on different Oracle ASM instances in the same cluster if the physical disk being replaced contains Oracle ASM disks from multiple disk groups. One Oracle ASM instance can run one rebalance operation at a time. If all Oracle ASM instances are busy, then rebalance operations are queued.

Maintaining the Physical Disks of Exadata Storage Servers

The first two disks of Exadata Storage Server are system disks. Oracle Exadata Storage Server Software system software resides on a portion of each of the system disks. These portions on both system disks are referred to as the system area. The nonsystem area of the system disks, referred to as data partitions, is used for normal data storage.All other disks in the cell are called data disks.

You can monitor a physical disk by checking its attributes with theCellCLI  LIST PHYSICALDISK command. For example, a physical disk status equal to critical, or predictive failure is probably having problems and needs to be replaced. The disk firmware maintains the error counters, and marks a drive with Predictive Failure when internal thresholds are exceeded. The drive, not the cell software, determines if it needs replacement.

When disk I/O errors occur,Oracle ASM performs bad extent repair for read errors due to media errors. The disks will stay online, and no alerts are sent. When Oracle ASM gets a read error on a physically-addressed metadata block, it does not have mirroring for the blocks, and takes the disk offline. Oracle ASM force drops the disk.

This section contains the following topics:

• Replacing a Physical Disk Due to Disk Failure

• Replacing a Physical Disk Due to Disk Problems

• Removing a Physical Disk Due to Bad Performance

• Moving All Drives from One Exadata Storage Server to Another Exadata Storage Server

• Repurposing a Physical Disk

• Removing and Replacing the Same Physical Disk

Replacing a Physical Disk Due to Disk Failure

A physical disk outage can cause a reduction in performance and data redundancy. Therefore, the disk should be replaced with a new disk as soon as possible. When the disk fails, the Oracle ASM disks associated with the grid disks on the physical disk are automatically dropped with FORCE option, and an Oracle ASM rebalance follows to restore the data redundancy.

An Exadata alert is generated when a disk fails. The alert includes specificinstructions for replacing the disk. If you have configured the system for alert notifications, then the alert is sent by e-mail to the designated address.

After the physical disk is replaced, the grid disks and cell disks that existed on the previous disk in that slot is re-created on the new physical disk. If those grid disks were part of an Oracle ASM group, then they will be added back to the disk group and the data will be rebalanced on them, based on the disk group redundancy and ASM_POWER_LIMITparameter.

Oracle ASM rebalance occurs when dropping or adding a disk. To check the status of the rebalance, do the following:

• The rebalance operation may have been successfully run. Check the Oracle ASM alert logs to confirm.

• The rebalance operation may be currently running. Check the GV$ASM_OPERATION view to determine if the rebalance operation is still running.

• The rebalance operation may have failed. Check the V$ASM_OPERATION.ERROR view to determine if the rebalance operation failed.

• Rebalance operations from multiple disk groups can be done on different Oracle ASM instances in the same cluster if the physical disk being replaced contains ASM disks from multiple disk groups. One Oracle ASM instance can run one rebalance operation at a time. If all Oracle ASM instances are busy, then rebalance operations will be queued.

The following procedure describes how to replace a disk due to disk failure:

1. Determine the failed disk using the following command:

   CellCLI> LIST PHYSICALDISK WHERE diskType=HardDisk AND status=critical DETAIL
   

   The following is an example of the output from the command. The slot number shows the location of the disk, and the status shows that the disk has failed.

   CellCLI> LIST PHYSICALDISK WHERE diskType=HardDisk AND status=critical DETAIL
   
            name:                   28:5
            deviceId:               21
            diskType:               HardDisk
            enclosureDeviceId:      28
            errMediaCount:          0
            errOtherCount:          0
            foreignState:           false
            luns:                   0_5
            makeModel:              "SEAGATE ST360057SSUN600G"
            physicalFirmware:       0705
            physicalInterface:      sas
            physicalSerial:         A01BC2
            physicalSize:           558.9109999993816G
            slotNumber:         5
           status:             critical
   

2. Replace the physical disk on Exadata Storage Server and wait for three minutes. The physical disk is hot-pluggable, and can be replaced when the power is on.

3. Confirm the disk is online.

   When you replace a physical disk, the disk must be acknowledged by the RAID controller before you can use it. This does not take long. Use the LIST PHYSICALDISKcommand similar to the following to ensure the status is NORMAL.

   CellCLI> LIST PHYSICALDISK WHERE name=28:5 ATTRIBUTES status
   

4. Verify the firmware is correct using the ALTER CELL VALIDATE CONFIGURATION command.

In rare cases, the automatic firmware update may not work, and the LUN will not be rebuilt. This can be confirmed by checking the ms-odl.trc file.

Removing a Physical Disk Due to Bad Performance

A single bad physical disk can degrade the performance of other good disks. It is better to remove the bad disk from the system than let it remain. To identify a bad physical disk, use the CALIBRATE command, and look for very low throughput and IOPS for each physical disk.

The following procedure describes how to remove a physical disk once the bad disk has been identified:

1. Illuminate the physical drive service LED to identify the drive to be replaced using the following command:

   cellcli -e ‘alter physicaldisk disk_name serviceled on‘
   

   In the preceding command, disk_name is the name of the physical disk to be replaced, such as 20:2.

2. Find all the grid disks on the bad disk. Use the following command to direct Oracle ASM to stop using the bad disk immediately:

   ALTER DISKGROUP diskgroup_name DROP DISK asm_disk_name
   

3. Wait until the Oracle ASM disks associated with the grid disks on the bad disk have been successfully dropped by querying the V$ASM_DISK_STAT view.

4. Remove the badly-performing disk. When you remove the disk, you get an alert.

5. When a new disk is available, install the new disk in the system. The cell disks and grid disks are automatically created on the new physical disk.

   Note:

   When you replace a physical disk, the disk must be acknowledged by the RAID controller before you can use it. This does not take long, but you should use the LIST PHYSICALDISK command to ensure the status is NORMAL.

Replacing a Power Distribution Unit

A power distribution unit (PDU) can be replaced while Oracle Exadata Rack is online. The second PDU in the rack maintains the power to all components in the rackexcept for the KVM. The KVM is a non-critical item that is powered from the PDU-B side of the rack. PDU-A is on the left, and PDU-B is on the right when viewing the rack from the rear.