Mounting Diskgroup Fails With ORA-15063 and V$ASM_DISK Shows PROVISIONED

Mounting Diskgroup Fails With ORA-15063 and V$ASM_DISK Shows PROVISIONED (Doc ID 1487443.1)

APPLIES TO:

Oracle Database - Enterprise Edition - Version 10.2.0.1 to 11.2.0.4 [Release 10.2 to 11.2]
Information in this document applies to any platform.

SYMPTOMS

 1. ASM is restarted and some ASM diskgroups will not be mounted with the following similar error.

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"

2. v$asm_disk view returns all ASM device information but v$asm_disk.header_status colume shows the status as "PROVISIONED".

SQL> select group_number gn,disk_number dn, mount_status, header_status,mode_status,state, total_mb, free_mb, label, path from v$asm_disk order by group_number, disk_number;

  GN   DN MOUNT_STATUS HEADER_STATU MODE_STATUS  STATE         TOTAL_MB    FREE_MB        LABEL          PATH                                   
---- ---- ----------------------------------------------------------------------------------------------------------------------------------------
   0    0 CLOSED              PROVISIONED        ONLINE            NORMAL                  0                0       CRS             ORCL:CRS
   0    1 CLOSED              PROVISIONED        ONLINE            NORMAL                  0                0       DATA           ORCL:DATA

3. kfed output for the affected device shows unexpected value on some spare columns.

Example #1>  0xaa55 on kfdhdb.acdb.ub2spare field ( 2 bytes at 510th location ).

$ kfed read /dev/oracleasm/disks/DATA

kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
..
kfdhdb.acdb.ents:                   0 ; 0x1dc: 0x0000
kfdhdb.acdb.ub2spare:     43605 ; 0x1de: 0xaa55                          <<<<<<<<------------ be="" br="" should="">                                                             

Example #2>

kfdhdb.ub4spare[39]:         3465798583 ; 0x198: 0xce93e3b7        <<<<<<<<------------ be="" br="" should="">kfdhdb.ub4spare[40]:                  0 ; 0x19c: 0x00000000

CAUSE

Case #1] 0xaa55 on little-endian server like Linux  or 0x55aa on big-endain server like Sun Sparc indicates boot signature ( or magic number ) on MBR (Master Boot Record )  Partition. 

Case #2] Local backup software ( like Symantec image backup ) touches ASM disk header on  column kfdhdb.ub4spare[39] from kfed output.

This issue could happen outside ASM when some tools on OS ( or human ) put partition information on the affected device

SOLUTION

 This situation can happen while the affected diskgroup is mounted and maintenance window needs to be scheduled to repair the device header.

1. Shutdown ASM instance

2. Take a 10M dd backup just in case.

   $ dd if=/dev/oracleasm/disks/DATA of=/tmp/DATA.dd bs=1M count=10

3. Repair the device header in the following way.  Repeat the following example procedure for the all affected devices.

   Check AU ( Allocation Unit ) size for the disk in the following way for aus option in kfed command .

   $  kfed read /dev/oracleasm/disks/DATA | grep ausize
   kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
                                   ^^^^^^^

   If ASM version is 11.1.0.7 or above,

$ kfed repair /dev/oracleasm/disks/DATA aus=1048576

   If ASM version is less than 11.1.0.7,

$ kfed read /dev/oracleasm/disks/DATA text=./DATA.kfed
$ kfed write /dev/oracleasm/disks/DATA CHKSUM=YES aus=1048576 text=./DATA.kfed  

  

4. Startup ASM and mounte the affected diskgroup.

5. The procedure above only works with the symtom (0xaa55 on 510th location on device header ) described above. And there are many other possibilities causing PROVISIONED status in which the procedure above may not work.

    And note that if the procedure above does not resolve the issue, the corruption must have been spreaded to other blocks beyond disk header.  Contact Oracle Support for further investigation.