ASM Mirroring

Redundancy Level

The type of a ASM disk group is based on three redundancy levels:

• Normal

ASM provides two-way mirroring by default. A loss of one ASM disk is tolerated. Use can optionally choose three-way or unprotected mirroring for a file in a NORMAL redundancy disk group. A file specified with HIGH redundancy (three-way mirroring) in a NORMAL redundancy disk group provides additional protection form a bad disk sector, not protection from a disk failure.

• High

ASM provides triple mirroring by default. A loss of two ASM disks in different failure groups is tolerated.

• External

ASM does not provide mirroring redundancy and relies on the storage system to provide RAID functionality. Any write error causes a forced dismount of the disk group. All disks must be located to successfully mount the disk group.

Failure Group

When ASM allocates an extent for a mirrored file it allocate a primary copy and a mirror copy. ASM chooses the disk on which to store the mirror copy in a different failure group from the primary copy.

A failure group is a subset of the disks in a disk group, which could fail at the same time because they share hardware. The failure of common hardware must be tolerated. The simultaneous failure of all disks in a failure group does not result data loss because all mirrored copies of the disks are in different failure groups. A NORMAL redundancy disk group must contain at least two failure groups. A HIGH redundancy disk group must contain at least three failure group. There are always failure groups even if they are not explicitly created. If you do not specify a failure group for a disk, Oracle automatically creates a new failure group containing just that disk, except for disk group containing disks on Oracle Exadata cells.

Disk Failure

When there is a failure of one or more disks, the disks are first taken offline and then automatically dropped. In this case the disk group remains mounted and serviceable. In addition because of mirroring all of the disk group data remain accessible. After the disk drop operation, ASM performs a re-balance to restore full redundancy for the data on the failed disks.

Recovery from Read or Write I/O Errors

When a read error happens it triggers Oracle ASM instance to attempt bad block remapping. ASM then reads a good copy of the extent and copies it to the disk that has the read error. If the write to the same location succeeds then the underlining allocation unit is deemed healthy. If the write fails, ASM attempts to write the extent to a new allocate unit on the same disk. If this write succeeds, the original allocation unit is marked as unusable. If the write fails the disk is taken offline.

One unique benefit on Oracle ASM based mirroring is that the database instance is aware of the mirroring. For many types of logical corruptions such as a bad checksum or incorrect System Change Number (SCN), the database instance proceeds through the mirror side looking for valid content and proceeds without errors.

When a write error happens, the database instance sends ASM instance a disk offline message. If database can successfully complete a write to at least one extent copy and receive acknowledgment of the offline disk from ASM , the write is consider successful. If the write to all mirror side fails, database takes the appropriate actions  in response to a write error such as taking the tablespace offline.

==========================================================================================

Q) When a normal redundant ASM diskgroup runs out of space, is there still access to the data?  What operations are permitted or forbidden ?

An1:
Using this DATA diskgroup as an example:

CREATE DISKGROUP DATA NORMAL REDUNDANCY
FAILGROUP array1 DISK '/devices/diska1','/devices/diska2','/devices/diska3','/devices/diska4'
FAILGROUP array2 DISK '/devices/diskb1','/devices/diskb2','/devices/diskb3','/devices/diskb4';

In a normal redundancy each diskgroup should have at least 2 failure groups. Each failure group is a mirror of the other group.

With reference to the above example,

If disk '/devices/diskb2' is corrupted or failed this means that all the data inside that disk has to be written to the other disks '/devices/diskb1','/devices/diskb3','/devices/diskb4'.

If these 3 disks do not have enough room for the data in the corrupted disk '/devices/diskb2', you will not have mirroring of some files in that disk.

So, some files may end up with reduced redundancy (EXTERNAL redundancy, since there are only primary extents)

However, the diskgroup will remain mounted and used because it has all the files.  Even though some of them will be mirrored and others will not be mirrored.

If there is a condition where there will be loss of data in which the primary extent and secondary extent are not available, ASM will dismount the disk group immediately to protect other data.

With a Normal redundant diskgroup, you should add another new disk to the failure group as soon as possible and it will rebalance/mirror the files' extents automatically.

======================================================================

How to Calculate Usable_FILE_MB / REQUIRED_MIRROR_FREE_MB (Doc ID 1459611.1)

USABLE_FREE_MB

==============

USABLE_FILE_MB indicates the amount of free space, adjusted for mirroring, that is available for new files to restore redundancy after a disk failure. USABLE_FILE_MB is computed by subtracting REQUIRED_MIRROR_FREE_MB from the total free space in the disk group and then adjusting the value for mirroring. For example, in a normal redundancy disk group where by default the mirrored files use disk space equal to twice their size, if 4 GB of actual usable file space remains, then USABLE_FILE_MB equals roughly 2 GB. You can then add a file that is up to 2 GB.

USABLE_FREE_MB = (FREE_MB - REQUIRED_MIRROR_FREE_MB)/2 for normal redundancy
   or
USABLE_FREE_MB =FREE_MB - REQUIRED_MIRROR_FREE_MB)/3 for high redundancy.

-------------------------------------------------------------------------------------------------------------------
In regards to calculating the usable_file_mb -- the following is how this value is calculated:

Eg:  normal redundancy

GROUP_NUMBER

NAME

SECTOR_SIZE

BLOCK_SIZE

ALLOCATION_UNIT_SIZE

STATE

TYPE

TOTAL_MB

FREE_MB

HOT_USED_MB

COLD_USED_MB

REQUIRED_MIRROR_FREE_MB

USABLE_FILE_MB

1

DATADG

512

4096

1048576

MOUNTED

NORMAL

20362

19568

0

794

0

9784

So based on this:

(FREE_MB - REQUIRED_MIRROR_FREE_MB)/2 = USABLE_FILE_MB

(19568 - 0)/2 = 9784

REQUIRED_MIRROR_FREE_MB

===================

REQUIRED_MIRROR_FREE_MB indicates the amount of space that must be available in a disk group to restore full redundancy after the worst failure that can be tolerated by the disk group without adding additional storage. This requirement ensures that there are sufficient failure groups to restore redundancy. Also, this worst failure refers to a permanent failure where the disks must be dropped, not the case where the disks go offline and then back online.

The amount of space displayed in this column takes the effects of mirroring into account. The value is computed as follows:

Normal redundancy disk group with more than two failure groups

The value is the total raw space for all of the disks in the largest failure group. The largest failure group is the one with the largest total raw capacity. For example, if each disk is in its own failure group, then the value would be the size of the largest capacity disk.

High redundancy disk group with more than three failure groups

The value is the total raw space for all of the disks in the two largest failure groups.

Eg: normal redundancy

REQUIRED_MIRROR_FREE_MB is the size of 1 disk

DATA1 : REQUIRED_MIRROR_FREE_MB :  610352

The size of one of the disks in the DATA1 diskgroup  - DISKA1 is 610352.

=========================================================================

Fine grained striping vs. Coarse

We all aware that Oracle Automatic storage management offers striping of disks but propreitory of its own method.

To stripe data, Oracle ASM separates files into stripes and spreads data evenly across all of the disks in a disk group. It has two methods of striping, Fine & Coarse.

Fine Striping:- Fine striping writes 128 KB data to each ASM Disk in the diskgroup in a round robin fashion, 128 KB goes to the first disk, then the next 128 KB, goes to the next disk, etc. According to manual, The fine-grained stripe size always equals 128 KB in any configuration; this provides lower I/O latency for small I/O operations.” Small I/O operations sure sounds like a good candidate for redo logs, control files etc.

Coarse Striping:-With coarse grained striping ASM writes data to each disk in the same round robin fashion, but writes chunks in the size of the ASM instance’s allocation unit (AU) size, default is 1MB.

Further, starting from 11gR2, Online redo log files are no more in Fine striping contradictory to 10g, as Oracle silently changed the template to Coarse striping.

A note has been written on the same:- Redo Log Striping In 11.2 ASM, is Coarse Or Fine? [ID 1269158.1]

Mirroring as you aware the copy of same extents to multiple disks
Further along with mirroring, Oracle Intelligently place the extents in two modes,

COLD, MIRRORCOLD – use the inner most tracks (closest to spindle) on the disk drive.
HOT, MIRRORHOT – Use the outermost tracks which have greater speed and higher bandwidth.

http://oracle-info.com/category/12c-database/

=======================================================================