Voting Disk and OCR in 11gR2: Some changes

Having just delivered an Oracle Database 11gR2 RAC Admin course, I’d like to point out some remarkable changes in the way we handle now the important Clusterware components Voting Disk and Oracle Cluster Registry (OCR): Amazingly, we can now store the two inside of an Automatic Storage Management (ASM) Disk Group, which was not possible in 10g.

The OCR is striped and mirrored (if we have a redundancy other than external), similar as ordinary Database Files are. So we can now leverage the mirroring capabilities of ASM to mirror the OCR also, without having to use multiple RAW devices for that purpose only. The Voting Disk (or Voting File, as it is now also referred to) is not striped but put as a whole on ASM Disks – if we use a redundancy of normal on the Diskgroup, 3 Voting Files are placed, each on one ASM Disk into a different failgroup. Therefore, you need to have at least 3 failgroups for that diskgroup, like on this picture:

ASM Diskgroup that contains Voting Files and OCR

This is a concern, if our ASM Diskgroups consist of only 2 ASM Disks respectively only 2 failgroups like with Extended RAC! Therefore, the new quorum failgroup clause was introduced:

create diskgroup data normal redundancy
 failgroup fg1 disk 'ORCL:ASMDISK1'
 failgroup fg2 disk 'ORCL:ASMDISK2'
 quorum failgroup fg3 disk 'ORCL:ASMDISK3'
 attribute 'compatible.asm' = '11.2.0.0.0';

The failgroup fg3 above needs only one small Disk (300 MB should be on the safe side here, since the Voting File is only about 280 MB in size) to keep one Mirror of the Voting File. fg1 and fg2 will contain each one Voting File and all the other stripes of the Database Area as well, but fg3 will only get that one Voting File.

[root@uhesse1 ~]#  /u01/app/11.2.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   511de6e64e354f9bbf4be318fc928c28 (ORCL:ASMDISK1) [DATA]
 2. ONLINE   2f1973ed4be84f50bffc2475949b428f (ORCL:ASMDISK2) [DATA]
 3. ONLINE   5ed44fb7e79c4f79bfaf09b402ba70df (ORCL:ASMDISK3) [DATA]

Another important change regarding the Voting File is that it is no longer supported to take a manual backup of it with dd. Instead, the Voting File gets backed up automatically into the OCR. As a New Feature, you can now do a manual backup of the OCR any time you like, without having to wait until that is done automatically – which is also still done:

[root@uhesse1 ~]# /u01/app/11.2.0/grid/bin/ocrconfig -showbackup

uhesse1     2010/10/06 09:37:30     /u01/app/11.2.0/grid/cdata/cluhesse/backup00.ocr
uhesse1     2010/10/06 05:37:29     /u01/app/11.2.0/grid/cdata/cluhesse/backup01.ocr
uhesse1     2010/10/06 01:37:27     /u01/app/11.2.0/grid/cdata/cluhesse/backup02.ocr
uhesse1     2010/10/05 01:37:21     /u01/app/11.2.0/grid/cdata/cluhesse/day.ocr
uhesse1     2010/10/04 13:37:19     /u01/app/11.2.0/grid/cdata/cluhesse/week.ocr

Above are the automatic backups of the OCR as in earlier versions. Now the manual backup:

[root@uhesse1 ~]# /u01/app/11.2.0/grid/bin/ocrconfig -manualbackup
uhesse1     2010/10/06 13:07:03     /u01/app/11.2.0/grid/cdata/cluhesse/backup_20101006_130703.ocr

I got a manual backup on the default location on my master node. We can define another backup location for the automatic backups as well as for the manual backups – preferrable on a Shared Device that is accessible by all the nodes (which is not the case with /home/oracle, unfortunately :-) ):

[root@uhesse1 ~]# /u01/app/11.2.0/grid/bin/ocrconfig -backuploc /home/oracle
[root@uhesse1 ~]# /u01/app/11.2.0/grid/bin/ocrconfig -manualbackup
uhesse1     2010/10/06 13:10:50     /home/oracle/backup_20101006_131050.ocr
uhesse1     2010/10/06 13:07:03     /u01/app/11.2.0/grid/cdata/cluhesse/backup_20101006_130703.ocr

[root@uhesse1 ~]# /u01/app/11.2.0/grid/bin/ocrconfig -showbackup
uhesse1     2010/10/06 09:37:30     /u01/app/11.2.0/grid/cdata/cluhesse/backup00.ocr
uhesse1     2010/10/06 05:37:29     /u01/app/11.2.0/grid/cdata/cluhesse/backup01.ocr
uhesse1     2010/10/06 01:37:27     /u01/app/11.2.0/grid/cdata/cluhesse/backup02.ocr
uhesse1     2010/10/05 01:37:21     /u01/app/11.2.0/grid/cdata/cluhesse/day.ocr
uhesse1     2010/10/04 13:37:19     /u01/app/11.2.0/grid/cdata/cluhesse/week.ocr
uhesse1     2010/10/06 13:10:50     /home/oracle/backup_20101006_131050.ocr
uhesse1     2010/10/06 13:07:03     /u01/app/11.2.0/grid/cdata/cluhesse/backup_20101006_130703.ocr

Conclusion: The way to handle Voting Disk and OCR has changed significantly – they can be kept inside of an ASM Diskgroup especially.

About these ads

, ,

  1. #1 by jason arneil on October 6, 2010 - 17:17

    Hi Uwe,

    Any sign of being able to place the voting files onto multiple diskgroups if using external redundancy? And does it make any sense using the quorum feature with external redundancy?

    regards,

    jason.

  2. #2 by Uwe Hesse on October 7, 2010 - 07:49

    Hi Jason,
    there seems to be no supported way to have multiple Voting Files with external redundancy. Even the “crsctl add css votedisk” command is disabled if Voting Files are stored on ASM initially. You get this error message then: “CRS-4671: This command is not supported for ASM diskgroups.”

    Regarding your second question: I don’t see why we should do that, even though it should be possible technically.

  3. #3 by Srinivasan Krishnan on October 7, 2010 - 10:08

    For the quorum disk feature , should I have to add extra disk ? currently I have only one voting disk ?

  4. #4 by Uwe Hesse on October 7, 2010 - 10:18

    The Quorum Failgroup clause was introduced for setups with Extended RAC and/or for setups with Diskgroups that have only 2 Disks (resp. only 2 Failure Groups) but want to use normal redundancy. If you have currently one Voting Disk and you setup a Diskgroup with redundancy external – you will stay with one Voting Disk that is placed on your Diskgroup (as a whole on one Disk) together with all the other files (Datafiles, Controlfiles, Logfiles) that make up your Database Area. If you choose external redundancy, your storage (RAID) should provide redundancy, though.
    Short answer: No, not necessarily :-)

  5. #5 by Manuel Fuenzalida on March 18, 2011 - 04:35

    Hi uwe:

    I read your blog, very good articles, but this one interest me because i’m installing now an extended Rac 11G R2, two storage, but at the same physical site. I need to know, when the installer ask me for storage options for OCR and voting file, and i tell the installer to use asm with normal redundancy, if i want to have the contents of each voting file mirrored in other file group, made of disk of the second storage…..how can i do that? and how many disks i need to do that ? Because in the installer of grid infrastructure, don’t let me indicate thats options….

    thank you :)

    regards
    Manuel

  6. #6 by Uwe Hesse on March 18, 2011 - 09:02

    Hi Manuel,
    I suppose you are aware that it is not recommended to put 2 of 3 voting files on one site of an Extended RAC, because if this site crashes, the cluster is unavailable. You should instead put one voting file on each site and one on a third node. Look here for a whitepaper that describes that:

    http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

    If you insist to put 2 voting files on one site, you could control this with the mentioned quorum failgroup clause above in the article. Pick one disk on the desired site for that quorum failgroup. But again: From a HA perspective, it is a bad idea to put 2 of 3 voting files on one site of an Extended RAC

  7. #7 by Manuel on March 18, 2011 - 14:04

    Hi uwe:

    Thanks for your reply, i understand that is a bad idea to put 2 of 3 voting files on one site of an Extended RAC, but i dont know how do it….Grid infrastructure installer dont let me configure in the moment i install, the failgroups for disk containing OCR y Voting Files….so that way, asm create the failgroup of each disk in the same disk….i need 1 voting file in one disk from one storage…other voting file in one disk from another storage….and the third voting file, i coul make it a you say, with an nfs or something like that…but for now, i can’t do that way because installer don’t let me specify failgroups…..

    Thak you again.
    Regards
    Manuel

  8. #8 by DanyC on April 12, 2011 - 11:46

    Hi Uwe,

    In case i have 2 nodes accesing 2 storages and the ocr & voting files stored in

    ORCL:CRS_ST1_DISK1
    ORCL:CRS_ST1_DISK2
    ORCL:CRS_ST2_DISK1
    ORCL:CRS_ST2_DISK2
    ORCL:CRS_ST2_DISK3

    is that normal? I’m very confused as one of our dbs said the number should be odd but i didn’t find any notes where Oracle recommend that!?!
    Why i should not have 3 fg on both storages?

    Looking forward to your reply.

    Many thanks,
    Dani

  9. #9 by Uwe Hesse on April 12, 2011 - 20:24

    Dani,
    the number of Voting Files for 11gR2 is 1, 3 or 5, depending on the redundancy (external, normal, high) of the ASM Diskgroup, the Voting Files are kept in. The OCR has 1 file on that Diskgroup, which can be mirrored to another Diskgroup, than you have 2 files. These numbers do not increase even if you create more than 3 Failgroups on your Diskgroups.

  10. #10 by DanyC on April 13, 2011 - 13:48

    Thanks Uwe for your answer.
    Looking on MOS i found a note 877134.1 which says

    “An odd number of voting disks is required for proper clusterware configuration. A node must be able to access strictly more than half of the voting disks at any time. So, in order to tolerate a failure of n voting disks, you must have at least 2n+1 configured. (n=1 means 3 voting disks). Refer to Note 428681.1 for assistance with adding voting disks.”

    Thanks a lot,
    Dani

  11. #11 by Manuel Fuenzalida on May 9, 2011 - 22:56

    Hi Uwe, i’m trying to add a third voting file, but on an nfs file system, i have a problem….other 2 voting files are stored on asm, it’s posible to do this ?

    Regards, Manuel

  12. #12 by Uwe Hesse on May 10, 2011 - 12:30

    Hi Manuel,
    yes it is. That is described in the whitepaper I linked to already in a previous answer to your questions:

    http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

  13. #13 by Manuel on May 10, 2011 - 15:28

    Hi Uwe, yesterday, I tried to add the third voting file following the instructions in the whitepaper, but, ann error ocurred, this error, tells me that i can´t add the third voting file on an nfs file system, because i have the other 2 voting files on asm storage.

    Have you tried this ?

    Best regards, Manuel

  14. #14 by Uwe Hesse on May 11, 2011 - 16:13

    Hi Manuel,
    although I didn’t do that myself, I trust Roland Knapp & Markus Michalewicz (the authors of the whitepaper) that it can be done as described :-)

  15. #15 by Dinesh on May 25, 2011 - 12:32

    I have a partition in shared disk /dev/sdc1.

    While installing Oracle grid and when it prompted to enter the Diskgroup for OCR and Voting disk I gave a name DGDATA and choose only one (external redundancy) disk “/dev/sdc1″ (it does not ask for one for OCR and one for Voting as it used to do in 10g clusterware installation).

    The Oracle grid installation has gone through successfully. when I restart the system I find CRS not starting.

    The log says: “Error PROC:26: Error while accessing the physical storage ASM……..” ORA-01034: oracle not available Could not init OCR, code:26….Linud permission denied.

    For your information….ASM instance is up and I find the diskgroup mounted.

    Can anyone help on this.

    Regards
    Dinesh

  16. #16 by Manuel on May 25, 2011 - 15:44

    Dinesh, before you do the installation, you have to configure asm disks, with “oracleasm” utility, before that, you have to install this utility, with an rpm package in linux, then execute “oracleasm create disk”….

    Best Regards
    Manuel

  17. #17 by Uwe Hesse on May 26, 2011 - 08:08

    Dinesh, apart from Manuel’s valid hint: If you have such a grieve technical problem, your first contact should be Oracle Support. Second good chance (without MOS Account) would be OTN Discussion Forum: http://forums.oracle.com/forums/forum.jspa?forumID=62
    I do this Blog in my spare time and may not answer at all or late – the forum has 100’s of members who will answer much faster.

  18. #18 by Dinesh on May 29, 2011 - 17:42

    Thanks Manuel and Hesse.

    Manuel: I have applied rpms, I have tried to use ORACLEASM command to create a disk (basically labeling the disk). I had issues executing root.sh on the 2nd node while installing grid. Next time I directly specified the /dev/sdc1 for OCR and Voting disk during grid installation. And the whole grid installation works fine. The problems is only when you restart the system?!!!

    Thanks Hesse for the link, I did go through it. Will post my problem there.

  19. #19 by Ora600Tom on November 18, 2011 - 15:14

    Nice post, thank you.

    You said,

    “The failgroup fg3 above needs only one small Disk (300 MB should be on the safe side here, since the Voting File is only about 280 MB in size) to keep one Mirror of the Voting File. fg1 and fg2 will contain each one Voting File and all the other stripes of the Database Area as well, but fg3 will only get that one Voting File.”

    Is this correct? Voting disk alone make 280MB or Voting disk and OCR together makes up 280MB?

    Thanks
    Thomas Saviour

  20. #20 by Uwe Hesse on November 18, 2011 - 19:05

    Thank you for the comment. I think it is indeed true (although a couple of hundred MB may be considered neglectable anyway, these days):

    http://download.oracle.com/docs/cd/E11882_01/install.112/e22489/storage.htm#CWLIN288

  21. #21 by Ora600Tom on November 19, 2011 - 05:33

    Thank you. That means, OCR will be striped only on fg1 and fg2. My initial impression was both OCR and Voting disk will be in all 3 disks.

    But the same document also says

    “If you are upgrading Oracle Clusterware, and your existing cluster uses 100 MB OCR and 20 MB voting disk partitions, then you must extend these partitions to at least 300 MB. Oracle recommends that you do not use partitions, but instead place OCR and voting disks in disk groups marked as QUORUM disk groups.”

    That means QUORUM can hold OCR in certain circumstances?

    Interestingly as per a blog from Riyaj, one node is writing 512 bytes on a specific offset for each node while heart beat checking. Then I do not understand what makes the voting disk size to 280M. In the past Voting disk size was very small.

    http://orainternals.wordpress.com/2010/10/29/whats-in-a-voting-disk/

    Many Thanks
    Thomas

  22. #22 by Uwe Hesse on November 19, 2011 - 10:42

    Thank you for raising this interesting discussion! I really do appreciate that :-)
    Right now, I have no RAC at hand to check, but I doubt that quorum failgroups contain OCR stripes. They got invented to be used on the third (middle) site of an Extended RAC, where no crsd process is running to use the OCR. Will research that as soon as I find the time & hardware for it – which may take a little because I am very busy with course delivery and assisting Oracle Certification to craft an Exadata Exam.

  23. #23 by Momo on November 22, 2011 - 16:48

    Hi guys, thank you for this interesting blog.
    I just want some clarification about the quorum failgroup. i see this syntax :

    SQL> CREATE DISKGROUP TEST NORMAL REDUNDANCY
    FAILGROUP fg1 DISK ‘
    FAILGROUP fg2 DISK ‘

    QUORUM FAILGROUP fg3 DISK ”
    ATTRIBUTE ‘compatible.asm’ = ‘11.2.0.0.0’;

    if i need to place my third voting in a NFS file, how do i create this file, will be working if the file is created by a simple touch?

    Thank you

  24. #24 by Momo on November 22, 2011 - 16:50

    The correct syntaxe

    CREATE DISKGROUP TEST NORMAL REDUNDANCY
    FAILGROUP fg1 DISK ‘
    FAILGROUP fg2 DISK ‘

    QUORUM FAILGROUP
    fg3 DISK ”
    ATTRIBUTE ‘compatible.asm’ = ‘11.2.0.0.0’;

  25. #25 by Momo on November 22, 2011 - 16:51

    CREATE DISKGROUP TEST NORMAL REDUNDANCY
    FAILGROUP fg1 DISK ‘disk in SAN1′
    FAILGROUP fg2 DISK ‘ disk in SAN2′
    QUORUM FAILGROUP fg3 DISK ‘another disk or file’
    ATTRIBUTE ‘compatible.asm’ = ‘11.2.0.0.0’;

  26. #26 by momo on November 23, 2011 - 14:24

    Finally i found the solution (in this document link http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

    Just create the file with the dd command as described.

    Thank you for this useful blog

  27. #27 by Uwe Hesse on November 23, 2011 - 18:28

    Thank YOU for sharing this information here :-)

  28. #28 by Jay on December 5, 2011 - 04:17

    We chose external redundancy for OCR/VD in a new 11.2.0.3 RAC grid infrastructure. When adding OCR copy to a second ASM Diskgroup “ocrconfig -add ” , Should NewDg asm and database compatible attricute to be 11.2.0.0 ?.

    Is there any procedure to create second VD into NewDG with external redundancy?. If not, how we maintain redundant copy of VD with external redundancy?

    Thanks
    Jay

  29. #29 by Sabine on January 10, 2012 - 19:19

    Hi Uwe,
    could you explain what is the Universal File ID shown up in the query “crsctl query css votedisk”?
    We are using Rac one node on iSCSI storage and I couldnt identify this Universal File Id – it is not scsi-wwid nor isci-id.
    we noticed also, that after bringing an offline voting disk online again, this Universal File Id has changed.
    Best Regards
    Sabine

  30. #30 by Uwe Hesse on January 12, 2012 - 12:12

    Sabine,
    thank you for stopping by! I am sorry, but I can’t tell you the deeper meaning of the Universal File Id, though. Will keep an eye on that – maybe I will stumble about it in the future.

  31. #31 by Js on February 5, 2012 - 11:50

    Hi Uwe, nice post

    I have couple of doubt regarding OCR/VOT disks on ASM, gone through with some documentation but still its not clear to me.

    I am confused with the starting order of Clusterware and ASM, before 11gr2, oracle used to start clusterware -> ASM -> DB .. so on. But now when we OCR/VOT are on ASM, how does oracle manage this, which component get started first.

    I would appreciate if you can shed some light.

    Regards,

  32. #32 by Uwe Hesse on February 7, 2012 - 13:25

    Thank you, JS, for the question: The secret lies in the OLR (Oracle Local Registry) that points to the voting files, placed (unstriped) on single disks in an ASM diskgroup. Look here for a lot more details regarding the cluster startup sequence:

    11gR2 Clusterware and Grid Home – What You Need to Know [ID 1053147.1]
    Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]

  33. #33 by Roger on February 27, 2012 - 23:12

    So when I install 11gr2 grid infrastructure I put 3 raw disks into 1 vote diskgroup and that completes the voting disk setup?

  34. #34 by Uwe Hesse on February 28, 2012 - 11:04

    Roger, you don’t have to dedicate a diskgroup just to contain the voting disk. These are some keypoints about voting disk handling in 11gR2:

    1 Voting disks will be kept in an ASM diskgroup
    2 Redundancy of the Voting Disk depends on the Redundancy of that diskgroup
    3 Voting Disks are mirrored across Failuregroups

    A common example would be the Diskgroup DATA with redundancy normal that has at least 3 Failuregroups and contains also the Database Area of a RAC database. In this case 3 Failuregoups would each store one voting disk unstriped on one disk.

  35. #35 by Andrew on March 1, 2012 - 18:36

    Two questions for you if I may?

    1. How many of the three voting disks need to stay online for the cluster to function normally?
    2. Have you ever seen this error when attempting to add a voting disk:

    -bash-3.2# /code/grid/11.2.0.2/bin/crsctl add css votedisk /voteocr3
    Now formatting voting disk: /voteocr3/rac/.
    Failed to initialize voting file /voteocr3/rac/.
    Change to configuration failed, but was successfully rolled back.
    CRS-4000: Command Add failed, or completed with errors.

    Thank you for your article and the information you provide.

  36. #36 by Uwe Hesse on March 3, 2012 - 12:13

    Andrew, you’re welcome :-)

    To your questions:
    1.) Of three voting disks, two need to be accessible for the cssd processes in order to keep the cluster alive
    2.) I did not encounter that yet

  37. #37 by Roger on March 7, 2012 - 23:02

    What I have is one voting disk group and that disk group is made of only one raw disk. Can I simply add 2 more raw disks to make it right and if so can I do this while the database is up.

  38. #38 by Uwe Hesse on March 8, 2012 - 17:24

    Roger,
    when your voting disk is outside of ASM (probably you upgraded from 10g), you could just move it online into an ASM diskgroup with crsctl replace votedisk +data e.g.
    You could also create a new ASM diskgroup (with normal redundancy e.g.) for it.

  39. #39 by Roger on March 9, 2012 - 18:22

    This is a new install of 11gr2 and the voting disk group was created with one raw disk. My question is can I add 2 more raw disk to the voting group and be good to go?

    Also if I can simply add the raw disks is there any special syntax for
    adding raw disks to voting group?

  40. #40 by Uwe Hesse on March 9, 2012 - 18:28

    Roger,
    you cannot change the redundancy of an existing ASM diskgroup – it is determined at creation. SO adding more devices to the existing diskgroup won’t get your voting disk mirrored. But you could create a new diskgroup with at least 3 disks and redundancy normal. Then move the voting disk from the existing into the new diskgroup. This is an online operation that will not harm the availability of your database.

  41. #41 by Roger on March 9, 2012 - 19:14

    My question is based on the assumption that voting disk group requires a minimum of 3 raw disk to function properly and that there is one voting disk group per database.

  42. #42 by Roger on March 17, 2012 - 03:13

    I am installing 11gr2 grid infrastructure and at the end of the install when running the last root.sh script I got this output – I setup the data disk group during the installation and thought I could setup the FRA and OCR_VOTE disk groups after the installation completed however it seemed to take the DATA disk group and use 1 of the disk for the voting disk. I am not sure what it did with the rest of the DATA disk group.

    ASM created and started successfully.

    Disk Group DATA created successfully.

    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user ‘root’, privgrp ‘root’..
    Operation successful.
    CRS-4256: Updating the profile
    Successful addition of voting disk 9c2aea6c433f4fdebf3388c6524b8a66.
    Successfully replaced voting disk group with +DATA.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 9c2aea6c433f4fdebf3388c6524b8a66 (/dev/mapper/asm01p1) [DATA]
    Located 1 voting disk(s).
    CRS-2672: Attempting to start ‘ora.asm’ on ‘lpqmd544′
    CRS-2676: Start of ‘ora.asm’ on ‘lpqmd544′ succeeded
    CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘lpqmd544′
    CRS-2676: Start of ‘ora.DATA.dg’ on ‘lpqmd544′ succeeded

  43. #43 by Uwe Hesse on March 17, 2012 - 11:36

    Roger, you observe standard behavior here. Because of the external redundancy of the diskgroup, you get only one voting disk.

    1) You don’t have to create a dedicated diskgroup only for the Voting Disk and OCR. It is OK to have the Database Area in the same diskgroup
    2) You can create a dedicated diskgroup for that purpose and move Voting Disk and OCR onto it online afterwards

  44. #44 by Roger on March 17, 2012 - 19:18

    Do you have the steps to make a new OCR_VOTE group and copy the existing contents to it and then drop he old group and have the old raw disk available as candidates? I have 3 servers and I need to have identical setups on each one. I want normal redundancy on the voting DG and then my DATA and FRA are externan redundancy

  45. #45 by Roger on March 17, 2012 - 22:19

    This is from V$asm_disk – it shows that one of my 120g drives is a voting under the disk group DATA and I would like to change that so the voting group is it’s own.and return the 120g raw disk to the DATA Disk Group

    V NAME TOTAL_MB FREE_MB PATH
    – ———- ———- ———- ——————–
    Y DATA_0000 122879 122755 /dev/mapper/asm01p1
    N DATA_0001 122879 122784 /dev/mapper/asm02p1
    N DATA_0002 122879 122787 /dev/mapper/asm03p1
    N DATA_0003 122879 122784 /dev/mapper/asm04p1

    By the way Thanks for all your replies – also so people know that when you install Grid Infrastructure and get to the part where you create a disk group it only wants you to create the OCR_VOTE disk group there and then after the installation you can add the other disk groups using asmca.

  46. #46 by Uwe Hesse on March 19, 2012 - 11:12

    Hi Roger, the output above does not mean that the voting disk is the only file on that ASM disk. It will get Datafile stripes etc. in the same way as the other disks also. So we don’t waste 120G for just one voting disk here.

    If you want to use a dedicated diskgroup for voting disk and OCR, this can be done online with these commands:
    1) crsctl replace votedisk +OCR_VOTE
    this will add 3 voting disks to the new diskgroup if it has a redundancy normal (make sure you have at least 3 disks in that diskgroup)
    2) ocrconfig -add +OCR_VOTE
    will add ONE OCR into the diskgroup that is protected according to the redundancy of that diskgroup
    3) ocrconfig -delete +DATA
    will remove the stripes of the OCR from the DATA diskgroup

    You may now drop the DATA diskgroup or use it for the Database you will create subsequently.
    Make sure to take a backup of the OCR before doing the 3 steps above.

  47. #47 by Bertrand on September 14, 2012 - 11:42

    Hi Uwe,

    I have a question regarding 11GR2 extended cluster and the “real” needs of a third site for the third voting disk.

    As the voting disks are stored into ASM that way (http://docs.oracle.com/cd/E11882_01/rac.112/e16794/votocr.htm#CHDFJHHJ) :


    By default, Oracle ASM puts each voting disk in its own failure group within the disk group. A failure group is a subset of the disks in a disk group.

    If voting disks are stored on Oracle ASM with normal or high redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.

    Example :

    3 voting disks into an ASM diskgroup composed of 4 failgroups :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 8a1954c3473a4f50bfb3e2ce01468d34 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F52d0s0) [OCRVOTE2]
    2. ONLINE 2aa67328d87f4fc9bf900b46cc95a673 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F58d0s0) [OCRVOTE2]
    3. ONLINE 9af8c976efa94f83bf175a0a920f7da9 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Cd0s0) [OCRVOTE2]
    Located 3 voting disk(s).

    As far the failgroup :

    DG FAILGROUP PATH
    —————————— —————————— ————————————————–
    OCRVOTE2 FG1 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F52d0s0
    OCRVOTE2 FG2 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F58d0s0
    OCRVOTE2 FG3 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Cd0s0
    OCRVOTE2 FG4 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Ed0s0

    Now if we simulate the lost of one voting disk (the one into FG2 for exemple) :

    SQL> alter diskgroup OCRVOTE2 offline disks in failgroup fg2;

    Diskgroup altered.

    Let’s see if it has been reconstructed into FG4 :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 8a1954c3473a4f50bfb3e2ce01468d34 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F52d0s0) [OCRVOTE2]
    2. ONLINE 9af8c976efa94f83bf175a0a920f7da9 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Cd0s0) [OCRVOTE2]
    3. ONLINE 5d61b8a9596b4f5bbf96855faa384ee7 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Ed0s0) [OCRVOTE2]
    Located 3 voting disk(s).

    Yes that’s the case !

    So, if we use 2 sites with normal redundancy and 2 failure group into each site (total of 2 failure group), then is the third site still needed ?

    (AS a site will not be a single point of failure anymore as the remaining site will be able to host 2 voting disks in case one site crash).

    Thanks for your advice,
    Bertrand

  48. #48 by Bertrand on September 14, 2012 - 14:25

    Uwe,

    Correction : I mean “total of 4 failure groups”

    Thx
    Bertrand

  49. #49 by Uwe Hesse on September 14, 2012 - 15:13

    Bertrand, when you have an extended RAC with 2 Voting Disks on one site A and 1 on site B – how is the cluster supposed to survive a disaster that destroys the whole site A (including 2 Voting Disks)? 1 out of 3 is simply 1 too few, sorry for saying the obvious :-)

  50. #50 by Bertrand on September 14, 2012 - 16:28

    Hi Uwe,

    Thanks for the reply.

    No problem ;-).

    My example describes Site B with 2 failgroups and site A with 2 failgroups as well.

    Then on site B one failgroup is “empty” before the crash (as you said 2 voting disks are on site A).

    One voting disk could then been “automatically” recreated on site B and then site B will get 2 voting disks, right ?

    Thx
    Bertrand

  51. #51 by Uwe Hesse on September 14, 2012 - 16:37

    No, I don’t think so. No matter how many failgroups are present, 1 accessible Voting Disk out of 3 is not sufficient to keep the cluster alive, in my opinion. Which is why a third site is recommended for Extended RAC to hold one Voting Disk.

  52. #52 by Bertrand on September 14, 2012 - 17:46

    I agree, but according to the oracle documentation :


    If voting disks are stored on Oracle ASM with normal or high redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.

    Then, you think that in my example the Site B will be evicted from the cluster before a second voting disk could be recovered into the “empty” failgroup located in Site B.

    Is that correct ?

    Thx
    Bertrand

  53. #53 by Uwe Hesse on September 14, 2012 - 18:30

    Yes, I think that is correct – and the documentation is only talking about the damage of one failure group (only one Voting Disk affected) whereas in case of a disaster at site A, 2 failure groups (each with one Voting Disk on) are affected at the same time.

  54. #54 by Bertrand on September 14, 2012 - 20:17

    Well, i think you are right.
    As you always suggest :-) and as i usually do : i will test it and i will let you know.

    Thx for your time
    Bertrand

  55. #55 by Uwe Hesse on September 17, 2012 - 11:02

    Bertrand, you’re welcome :-) Appreciate if you could share the outcome of your test also.

  56. #56 by Bertrand on September 19, 2012 - 10:20

    Uwe,

    The outcome is the following :

    configuration :

    2 sites (SITEA and SITEB) and 4 failgroups for the OCRVOT2 diskgroup :

    DG FAILGROUP PATH
    —————————— —————————— —————————————-
    OCRVOT2 FG1 ORCL:VOTE1SITEA
    OCRVOT2 FG2 ORCL:VOTE2SITEA
    OCRVOT2 FG3 ORCL:VOTE1SITEB
    OCRVOT2 FG4 ORCL:VOTE2SITEB

    current voting disk configuration :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE a8fcb4667eb54ffdbfe45645a1344c4a (ORCL:VOTE1SITEA) [OCRVOT2]
    2. ONLINE 5f12bca463804f04bfe90eefd03f99d4 (ORCL:VOTE2SITEA) [OCRVOT2]
    3. ONLINE 2456cb13571a4f96bf76d30f366fabfb (ORCL:VOTE1SITEB) [OCRVOT2]
    Located 3 voting disk(s).

    —————————– First test :

    Offline one voting disk :

    SQL> alter diskgroup OCRVOT2 offline disks in failgroup fg1;

    Diskgroup altered.

    => new voting disk configuration :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 5f12bca463804f04bfe90eefd03f99d4 (ORCL:VOTE2SITEA) [OCRVOT2]
    2. ONLINE 2456cb13571a4f96bf76d30f366fabfb (ORCL:VOTE1SITEB) [OCRVOT2]
    3. ONLINE 05c73a222df44f07bfea4f42eaf2f03c (ORCL:VOTE2SITEB) [OCRVOT2]

    Conclusion :
    it works as expected, one voting disk has been re-created into one available failgroup.

    —————————– Second test :

    I put the offlined disk back online

    SQL> alter diskgroup OCRVOT2 online disks in failgroup fg1;

    Diskgroup altered

    Now i shoot the SITEB (that hosts the majority of the voting disks) and the result is :

    [cssd(4205)]CRS-1606:The number of voting files available, 1, is less than the minimum number of voting files required, 2, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log

    on both machines.

    => Both machines have been evicted from the cluster.

    —————————– Conclusions :

    – If one voting disk is lost : it will be re-created into one available diskgroup.

    – If the majority of voting disks are lost at the same time : the voting disks are not re-created into the available failgroup(s) and both nodes are evicted from the cluster.

    – you were right ;-)

  57. #57 by Uwe Hesse on September 19, 2012 - 10:47

    Thank you, Bertrand, for getting back to this site with the result of your test – I really do appreciate it!

  58. #58 by Juris Trošins on September 19, 2012 - 11:25

    Please guys do not fall into the nfs trap described in the document. Or at least pay attention to the “Known Issues” section. I’ve encountered the situations with Extended RAC with 3d voting disk on ASM couple of years ago, when the whole cluster was unusable when NFS got inaccessible. Filed SR to Oracle which resulted in the fix development, which I’m going to put to test in a while. The overall recommendation was to use iscsi as a path for 3d voting disk. It behaves just like scsi without locking resources like NFS.

  59. #59 by Uwe Hesse on September 19, 2012 - 11:35

    Thank you for sharing that information. You refer to this issue that Roland & Markus mention in the whitepaper: “If the NFS device location is not accessible [...] Starting Oracle Clusterware again with “crsctl start crs” will hang, because some of the old clusterware processes will hang on I/O to the NFS voting file.” I suppose? It would be very kind if you could share the result of your test of the fix later.

  60. #60 by Juris Trošins on September 19, 2012 - 11:42

    Yes, Uwe. Almost. In my case there was a hang even in the middle of something. Which resulted of course in hanging crsctl operations as well. You can check patch 10109915, which was developed in response of my case. My customer went to iSCSI and arranging the test is kinda complicated now. I’m going to give it a try in a local vm env, but still had no time. If you’ll have a time to test – I will much appreciated the reply :)

  61. #61 by Bertrand on September 28, 2012 - 07:45

    Hello Guys,

    I’ll have to launch soon a fresh install of rac with a third voting disk based on nfs.
    Then i’ll test this patch and let you know.

  62. #62 by Bertrand on October 9, 2012 - 13:36

    Hello Guys,

    I will not be able to test it soon as due to ACFS issue on RH 6.3 we had to install 14494305 (Implies CRS is now 11.2.0.3.3) but 10109915 is not available on 11.2.0.3.3.

    Bertrand

  63. #63 by Bertrand on October 23, 2012 - 09:25

    Hello Guys,

    I won’t test it because we are moving to ISCSI.

    Bertrand

  64. #64 by Uwe Hesse on October 23, 2012 - 16:55

    Bertrand, thanks for keeping us informed :)

  65. #65 by Jarmo on December 10, 2012 - 14:09

    Hi all
    Does anyone have experiences or found documentation of the case below?

    You have voting disks with normal redundancy on SAN. Then the connection to one of the voting files is lost for a moment. Now, is crs able to recover voting file automaticly, when connection comes working again?

  66. #66 by Ramesh on June 3, 2013 - 20:38

    Nice One.

  67. #67 by Anju Garg on September 20, 2013 - 16:39

    Here is a related link:

    http://oracleinaction.com/voting-disk/

  68. #68 by Jay Vardhan on September 20, 2013 - 16:45

    Hello Sir,

    Thanks for nice explanation of all the changes fr OCR and VOTE disk in 11gR2.
    Sir, since from 11gR2, vote disk is also backup with OCR automatically, I would request if you please brief the steps to use this backup in case of failure of vote disk.

  69. #69 by Aman.... on September 22, 2013 - 12:25

    @Jay Vardhan,

    Check the Oracle documentation. The steps are listed there.

    http://docs.oracle.com/cd/E11882_01/rac.112/e16794/votocr.htm#BGBCHCBI

    Aman….

  70. #70 by Aou on October 7, 2013 - 17:07

    Will 11gR2 voting Disk got backup as well if it is allocated in itsown +VOTDG but not included in +OCRDB??

  71. #71 by Leandro on October 16, 2013 - 03:28

    Hello all,

    I´m trying to move voting disk from High redundancy to Normal(DG 3 Failgroup gb each disk all)

    The command crsctl replace votedisk +GRID01 is hunging up and does not return nothing and any log is uptaded (grid alerts etc)

    The diskgroup was create successfully and ocrcheck as well the new Redundancy. I have a SR at Oracle and i will apreciate if you have already faced this situation.

    Steps:

    Name
    ——–
    === ODM Data Collection ===

    * Create Diskgroup: GRID01 (Normal Redundancy) */

    /* Devices */
    /dev/rdsk/c0d53s0
    /dev/rdsk/c0d54s0
    /dev/rdsk/c0d55s0

    * CREATE */
    CREATE DISKGROUP GRID01 NORMAL REDUNDANCY
    FAILGROUP SITE_A DISK
    ‘/dev/rdsk/c0d53s0′ NAME VOTE_MTC1_VD05_OCR_01
    FAILGROUP SITE_B DISK
    ,’/dev/rdsk/c0d54s0′ NAME VOTE_MTC1_VD05_OCR_04
    FAILGROUP SITE_C DISK
    ,’/dev/rdsk/c0d55s0′ NAME VOTE_MTC1_VD05_OCR_05
    ‘compatible.asm’ = ‘11.2’,
    REBALANCE POWER 10;

    Query CSS Before */

    grid@su-br-pro-dbtc01 ~ [+ASM1] >crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 3834a46ee4b84f49bfbde0adb188340e (/dev/rdsk/c0d9s0) [OCR]
    2. ONLINE 5b313f4e76c94f75bf76c2cbce65c1a7 (/dev/rdsk/c0d8s0) [OCR]
    3. ONLINE 58de0204606d4f2cbf698a8c17a4cd14 (/dev/rdsk/c0d6s0) [OCR]
    4. ONLINE 093aa04606264f10bf5ad9513e582444 (/dev/rdsk/c0d3s0) [OCR]
    5. ONLINE 8b33492ba3b04f1abf54d3a37643b616 (/dev/rdsk/c0d2s0) [OCR]

    — As root
    /* -show backup */
    ocrconfig -showbackup

    /* backup manual */
    ocrconfig -manualbackup

    su-br-pro-dbtc01 2013/10/14 20:15:55 /soft/appdbs/grid/11.2.0/cdata/su-br-pro-tcbr/backup_20131014_201555.ocr

    /*- create OCR redundancy as ROOT*/

    ocrconfig –add +GRID01

    —ocrcheck

    grid@su-br-pro-dbtc01 /dev/rdsk [+ASM1]>ocrcheck
    Status of Oracle Cluster Registry is as follows :
    Version : 3
    Total space (kbytes) : 262120
    Used space (kbytes) : 3732
    Available space (kbytes) : 258388
    ID : 703883108
    Device/File Name : +OCR
    Device/File integrity check succeeded
    Device/File Name : +GRID01
    Device/File integrity check succeeded

    Device/File not configured

    Device/File not configured

    Device/File not configured

    Cluster registry integrity check succeeded

    Logical corruption check bypassed due to non-privileged user

    grid@su-br-pro-dbtc01 /dev/rdsk [+ASM1]>

    /* – Move Voting as GRID */ — (Hanging up)

    crsctl replace votedisk +GRID01

    .

    FileName
    —————-
    VOTING

    FileComment
    ———————-

    Thanks

    Leandro (Brazil)

  72. #72 by Uwe Hesse on October 19, 2013 - 09:21

    Leandro, I hope you have sorted out that issue meanwhile? I didn’t encounter such a problem during voting disk replacement yet.

  73. #73 by Moriah on November 13, 2013 - 13:38

    Its very good Article. please share more updates like thats. Thank you. oracleconnections

  74. #74 by leoxu on February 8, 2014 - 08:01

    hi:
    i read oracle clusterware document that it indicates the following:
    1、Neither should you add a voting disk to a cluster file system in
    addition to the voting disks stored in an Oracle ASM disk group.
    Oracle does not support having voting disks in Oracle ASM and
    directly on a cluster file system for the same cluster at the same
    time.
    2、You cannot combine voting disks
    stored in Oracle ASM and voting disks not stored in Oracle ASM in the same cluster.

    please help me explain it,tks.

  75. #75 by Uwe Hesse on February 10, 2014 - 17:53

    leoxu, in my opinion the docs are quite clear here: If you have voting disks in an ASM diskgroup, that diskgroup determines the number of voting disks implicitly via the diskgroup redundancy. If you try a command like crsctl add css votedisk you get CRS-4671: This command is not supported for ASM diskgroups

  1. October 2010 Blogroll Report | AskDba.org Weblog
  2. Database Migration to ASM with short downtime « The Oracle Instructor
  3. Merry Christmas & A Happy New Year 2012! « The Oracle Instructor
  4. Merry Christmas & A Happy New Year 2012! | Oracle Administrators Blog - by Aman Sood
  5. Drop an ASM Disk that contains a Voting Disk? « The Oracle Instructor
  6. Top 10 postings in 2012 « The Oracle Instructor
  7. Purpose of the Voting Disk for #Oracle RAC | The Oracle Instructor

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,805 other followers

%d bloggers like this: