Pickleball spielen

1. 

   #1 von jason arneil am Oktober 6, 2010 - 17:17

   Hi Uwe,

   Any sign of being able to place the voting files onto multiple diskgroups if using external redundancy? And does it make any sense using the quorum feature with external redundancy?

   regards,

   jason.
2. 

   #2 von Uwe Hesse am Oktober 7, 2010 - 07:49

   Hi Jason,
   there seems to be no supported way to have multiple Voting Files with external redundancy. Even the „crsctl add css votedisk“ command is disabled if Voting Files are stored on ASM initially. You get this error message then: „CRS-4671: This command is not supported for ASM diskgroups.“

   Regarding your second question: I don’t see why we should do that, even though it should be possible technically.
3. 

   #3 von Srinivasan Krishnan am Oktober 7, 2010 - 10:08

   For the quorum disk feature , should I have to add extra disk ? currently I have only one voting disk ?
4. 

   #4 von Uwe Hesse am Oktober 7, 2010 - 10:18

   The Quorum Failgroup clause was introduced for setups with Extended RAC and/or for setups with Diskgroups that have only 2 Disks (resp. only 2 Failure Groups) but want to use normal redundancy. If you have currently one Voting Disk and you setup a Diskgroup with redundancy external – you will stay with one Voting Disk that is placed on your Diskgroup (as a whole on one Disk) together with all the other files (Datafiles, Controlfiles, Logfiles) that make up your Database Area. If you choose external redundancy, your storage (RAID) should provide redundancy, though.
   Short answer: No, not necessarily 🙂
5. 

   #5 von Manuel Fuenzalida am März 18, 2011 - 04:35

   Hi uwe:

   I read your blog, very good articles, but this one interest me because i’m installing now an extended Rac 11G R2, two storage, but at the same physical site. I need to know, when the installer ask me for storage options for OCR and voting file, and i tell the installer to use asm with normal redundancy, if i want to have the contents of each voting file mirrored in other file group, made of disk of the second storage…..how can i do that? and how many disks i need to do that ? Because in the installer of grid infrastructure, don’t let me indicate thats options….

   thank you 🙂

   regards
   Manuel
6. 

   #6 von Uwe Hesse am März 18, 2011 - 09:02

   Hi Manuel,
   I suppose you are aware that it is not recommended to put 2 of 3 voting files on one site of an Extended RAC, because if this site crashes, the cluster is unavailable. You should instead put one voting file on each site and one on a third node. Look here for a whitepaper that describes that:

   Klicke, um auf grid-infra-thirdvoteonnfs-131158.pdf zuzugreifen

   If you insist to put 2 voting files on one site, you could control this with the mentioned quorum failgroup clause above in the article. Pick one disk on the desired site for that quorum failgroup. But again: From a HA perspective, it is a bad idea to put 2 of 3 voting files on one site of an Extended RAC
7. 

   #7 von Manuel am März 18, 2011 - 14:04

   Hi uwe:

   Thanks for your reply, i understand that is a bad idea to put 2 of 3 voting files on one site of an Extended RAC, but i dont know how do it….Grid infrastructure installer dont let me configure in the moment i install, the failgroups for disk containing OCR y Voting Files….so that way, asm create the failgroup of each disk in the same disk….i need 1 voting file in one disk from one storage…other voting file in one disk from another storage….and the third voting file, i coul make it a you say, with an nfs or something like that…but for now, i can’t do that way because installer don’t let me specify failgroups…..

   Thak you again.
   Regards
   Manuel
8. 

   #8 von DanyC am April 12, 2011 - 11:46

   Hi Uwe,

   In case i have 2 nodes accesing 2 storages and the ocr & voting files stored in

   ORCL:CRS_ST1_DISK1
   ORCL:CRS_ST1_DISK2
   ORCL:CRS_ST2_DISK1
   ORCL:CRS_ST2_DISK2
   ORCL:CRS_ST2_DISK3

   is that normal? I’m very confused as one of our dbs said the number should be odd but i didn’t find any notes where Oracle recommend that!?!
   Why i should not have 3 fg on both storages?

   Looking forward to your reply.

   Many thanks,
   Dani
9. 

   #9 von Uwe Hesse am April 12, 2011 - 20:24

   Dani,
   the number of Voting Files for 11gR2 is 1, 3 or 5, depending on the redundancy (external, normal, high) of the ASM Diskgroup, the Voting Files are kept in. The OCR has 1 file on that Diskgroup, which can be mirrored to another Diskgroup, than you have 2 files. These numbers do not increase even if you create more than 3 Failgroups on your Diskgroups.
10. 

    #10 von DanyC am April 13, 2011 - 13:48

    Thanks Uwe for your answer.
    Looking on MOS i found a note 877134.1 which says

    „An odd number of voting disks is required for proper clusterware configuration. A node must be able to access strictly more than half of the voting disks at any time. So, in order to tolerate a failure of n voting disks, you must have at least 2n+1 configured. (n=1 means 3 voting disks). Refer to Note 428681.1 for assistance with adding voting disks.“

    Thanks a lot,
    Dani
11. 

    #11 von Manuel Fuenzalida am Mai 9, 2011 - 22:56

    Hi Uwe, i’m trying to add a third voting file, but on an nfs file system, i have a problem….other 2 voting files are stored on asm, it’s posible to do this ?

    Regards, Manuel
12. 

    #12 von Uwe Hesse am Mai 10, 2011 - 12:30

    Hi Manuel,
    yes it is. That is described in the whitepaper I linked to already in a previous answer to your questions:

    Klicke, um auf grid-infra-thirdvoteonnfs-131158.pdf zuzugreifen
13. 

    #13 von Manuel am Mai 10, 2011 - 15:28

    Hi Uwe, yesterday, I tried to add the third voting file following the instructions in the whitepaper, but, ann error ocurred, this error, tells me that i can´t add the third voting file on an nfs file system, because i have the other 2 voting files on asm storage.

    Have you tried this ?

    Best regards, Manuel
14. 

    #14 von Uwe Hesse am Mai 11, 2011 - 16:13

    Hi Manuel,
    although I didn’t do that myself, I trust Roland Knapp & Markus Michalewicz (the authors of the whitepaper) that it can be done as described 🙂
15. 

    #15 von Dinesh am Mai 25, 2011 - 12:32

    I have a partition in shared disk /dev/sdc1.

    While installing Oracle grid and when it prompted to enter the Diskgroup for OCR and Voting disk I gave a name DGDATA and choose only one (external redundancy) disk „/dev/sdc1“ (it does not ask for one for OCR and one for Voting as it used to do in 10g clusterware installation).

    The Oracle grid installation has gone through successfully. when I restart the system I find CRS not starting.

    The log says: „Error PROC:26: Error while accessing the physical storage ASM……..“ ORA-01034: oracle not available Could not init OCR, code:26….Linud permission denied.

    For your information….ASM instance is up and I find the diskgroup mounted.

    Can anyone help on this.

    Regards
    Dinesh
16. 

    #16 von Manuel am Mai 25, 2011 - 15:44

    Dinesh, before you do the installation, you have to configure asm disks, with „oracleasm“ utility, before that, you have to install this utility, with an rpm package in linux, then execute „oracleasm create disk“….

    Best Regards
    Manuel
17. 

    #17 von Uwe Hesse am Mai 26, 2011 - 08:08

    Dinesh, apart from Manuel’s valid hint: If you have such a grieve technical problem, your first contact should be Oracle Support. Second good chance (without MOS Account) would be OTN Discussion Forum: http://forums.oracle.com/forums/forum.jspa?forumID=62
    I do this Blog in my spare time and may not answer at all or late – the forum has 100’s of members who will answer much faster.
18. 

    #18 von Dinesh am Mai 29, 2011 - 17:42

    Thanks Manuel and Hesse.

    Manuel: I have applied rpms, I have tried to use ORACLEASM command to create a disk (basically labeling the disk). I had issues executing root.sh on the 2nd node while installing grid. Next time I directly specified the /dev/sdc1 for OCR and Voting disk during grid installation. And the whole grid installation works fine. The problems is only when you restart the system?!!!

    Thanks Hesse for the link, I did go through it. Will post my problem there.
19. 

    #19 von Ora600Tom am November 18, 2011 - 15:14

    Nice post, thank you.

    You said,

    „The failgroup fg3 above needs only one small Disk (300 MB should be on the safe side here, since the Voting File is only about 280 MB in size) to keep one Mirror of the Voting File. fg1 and fg2 will contain each one Voting File and all the other stripes of the Database Area as well, but fg3 will only get that one Voting File.“

    Is this correct? Voting disk alone make 280MB or Voting disk and OCR together makes up 280MB?

    Thanks
    Thomas Saviour
20. 

    #20 von Uwe Hesse am November 18, 2011 - 19:05

    Thank you for the comment. I think it is indeed true (although a couple of hundred MB may be considered neglectable anyway, these days):
    http://download.oracle.com/docs/cd/E11882_01/install.112/e22489/storage.htm#CWLIN288
21. 

    #21 von Ora600Tom am November 19, 2011 - 05:33

    Thank you. That means, OCR will be striped only on fg1 and fg2. My initial impression was both OCR and Voting disk will be in all 3 disks.

    But the same document also says

    „If you are upgrading Oracle Clusterware, and your existing cluster uses 100 MB OCR and 20 MB voting disk partitions, then you must extend these partitions to at least 300 MB. Oracle recommends that you do not use partitions, but instead place OCR and voting disks in disk groups marked as QUORUM disk groups.“

    That means QUORUM can hold OCR in certain circumstances?

    Interestingly as per a blog from Riyaj, one node is writing 512 bytes on a specific offset for each node while heart beat checking. Then I do not understand what makes the voting disk size to 280M. In the past Voting disk size was very small.

    What’s in a voting disk?

    Many Thanks
    Thomas
22. 

    #22 von Uwe Hesse am November 19, 2011 - 10:42

    Thank you for raising this interesting discussion! I really do appreciate that 🙂
    Right now, I have no RAC at hand to check, but I doubt that quorum failgroups contain OCR stripes. They got invented to be used on the third (middle) site of an Extended RAC, where no crsd process is running to use the OCR. Will research that as soon as I find the time & hardware for it – which may take a little because I am very busy with course delivery and assisting Oracle Certification to craft an Exadata Exam.
23. 

    #23 von Momo am November 22, 2011 - 16:48

    Hi guys, thank you for this interesting blog.
    I just want some clarification about the quorum failgroup. i see this syntax :

    SQL> CREATE DISKGROUP TEST NORMAL REDUNDANCY
    FAILGROUP fg1 DISK ‘‚
    FAILGROUP fg2 DISK ‚‚
    QUORUM FAILGROUP fg3 DISK “
    ATTRIBUTE ‚compatible.asm‘ = ‚11.2.0.0.0‘;

    if i need to place my third voting in a NFS file, how do i create this file, will be working if the file is created by a simple touch?

    Thank you
24. 

    #24 von Momo am November 22, 2011 - 16:50

    The correct syntaxe

    CREATE DISKGROUP TEST NORMAL REDUNDANCY
    FAILGROUP fg1 DISK ‘‚
    FAILGROUP fg2 DISK ‚‚
    QUORUM FAILGROUP
    fg3 DISK “
    ATTRIBUTE ‚compatible.asm‘ = ‚11.2.0.0.0‘;
25. 

    #25 von Momo am November 22, 2011 - 16:51

    CREATE DISKGROUP TEST NORMAL REDUNDANCY
    FAILGROUP fg1 DISK ‘disk in SAN1′
    FAILGROUP fg2 DISK ‚ disk in SAN2‘
    QUORUM FAILGROUP fg3 DISK ‚another disk or file‘
    ATTRIBUTE ‚compatible.asm‘ = ‚11.2.0.0.0‘;
26. 

    #26 von momo am November 23, 2011 - 14:24

    Finally i found the solution (in this document link http://www.oracle.com/technetwork/database/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

    Just create the file with the dd command as described.

    Thank you for this useful blog
27. 

    #27 von Uwe Hesse am November 23, 2011 - 18:28

    Thank YOU for sharing this information here 🙂
28. 

    #28 von Jay am Dezember 5, 2011 - 04:17

    We chose external redundancy for OCR/VD in a new 11.2.0.3 RAC grid infrastructure. When adding OCR copy to a second ASM Diskgroup „ocrconfig -add “ , Should NewDg asm and database compatible attricute to be 11.2.0.0 ?.

    Is there any procedure to create second VD into NewDG with external redundancy?. If not, how we maintain redundant copy of VD with external redundancy?

    Thanks
    Jay
29. 

    #29 von Sabine am Januar 10, 2012 - 19:19

    Hi Uwe,
    could you explain what is the Universal File ID shown up in the query „crsctl query css votedisk“?
    We are using Rac one node on iSCSI storage and I couldnt identify this Universal File Id – it is not scsi-wwid nor isci-id.
    we noticed also, that after bringing an offline voting disk online again, this Universal File Id has changed.
    Best Regards
    Sabine
30. 

    #30 von Uwe Hesse am Januar 12, 2012 - 12:12

    Sabine,
    thank you for stopping by! I am sorry, but I can’t tell you the deeper meaning of the Universal File Id, though. Will keep an eye on that – maybe I will stumble about it in the future.
31. 

    #31 von Js am Februar 5, 2012 - 11:50

    Hi Uwe, nice post

    I have couple of doubt regarding OCR/VOT disks on ASM, gone through with some documentation but still its not clear to me.

    I am confused with the starting order of Clusterware and ASM, before 11gr2, oracle used to start clusterware -> ASM -> DB .. so on. But now when we OCR/VOT are on ASM, how does oracle manage this, which component get started first.

    I would appreciate if you can shed some light.

    Regards,
32. 

    #32 von Uwe Hesse am Februar 7, 2012 - 13:25

    Thank you, JS, for the question: The secret lies in the OLR (Oracle Local Registry) that points to the voting files, placed (unstriped) on single disks in an ASM diskgroup. Look here for a lot more details regarding the cluster startup sequence:

    11gR2 Clusterware and Grid Home – What You Need to Know [ID 1053147.1]
    Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]
33. 

    #33 von Roger am Februar 27, 2012 - 23:12

    So when I install 11gr2 grid infrastructure I put 3 raw disks into 1 vote diskgroup and that completes the voting disk setup?
34. 

    #34 von Uwe Hesse am Februar 28, 2012 - 11:04

    Roger, you don’t have to dedicate a diskgroup just to contain the voting disk. These are some keypoints about voting disk handling in 11gR2:

    1 Voting disks will be kept in an ASM diskgroup
    2 Redundancy of the Voting Disk depends on the Redundancy of that diskgroup
    3 Voting Disks are mirrored across Failuregroups

    A common example would be the Diskgroup DATA with redundancy normal that has at least 3 Failuregroups and contains also the Database Area of a RAC database. In this case 3 Failuregoups would each store one voting disk unstriped on one disk.
35. 

    #35 von Andrew am März 1, 2012 - 18:36

    Two questions for you if I may?

    1. How many of the three voting disks need to stay online for the cluster to function normally?
    2. Have you ever seen this error when attempting to add a voting disk:

    -bash-3.2# /code/grid/11.2.0.2/bin/crsctl add css votedisk /voteocr3
    Now formatting voting disk: /voteocr3/rac/.
    Failed to initialize voting file /voteocr3/rac/.
    Change to configuration failed, but was successfully rolled back.
    CRS-4000: Command Add failed, or completed with errors.

    Thank you for your article and the information you provide.
36. 

    #36 von Uwe Hesse am März 3, 2012 - 12:13

    Andrew, you’re welcome 🙂

    To your questions:
    1.) Of three voting disks, two need to be accessible for the cssd processes in order to keep the cluster alive
    2.) I did not encounter that yet
37. 

    #37 von Roger am März 7, 2012 - 23:02

    What I have is one voting disk group and that disk group is made of only one raw disk. Can I simply add 2 more raw disks to make it right and if so can I do this while the database is up.
38. 

    #38 von Uwe Hesse am März 8, 2012 - 17:24

    Roger,
    when your voting disk is outside of ASM (probably you upgraded from 10g), you could just move it online into an ASM diskgroup with crsctl replace votedisk +data e.g.
    You could also create a new ASM diskgroup (with normal redundancy e.g.) for it.
39. 

    #39 von Roger am März 9, 2012 - 18:22

    This is a new install of 11gr2 and the voting disk group was created with one raw disk. My question is can I add 2 more raw disk to the voting group and be good to go?

    Also if I can simply add the raw disks is there any special syntax for
    adding raw disks to voting group?
40. 

    #40 von Uwe Hesse am März 9, 2012 - 18:28

    Roger,
    you cannot change the redundancy of an existing ASM diskgroup – it is determined at creation. SO adding more devices to the existing diskgroup won’t get your voting disk mirrored. But you could create a new diskgroup with at least 3 disks and redundancy normal. Then move the voting disk from the existing into the new diskgroup. This is an online operation that will not harm the availability of your database.
41. 

    #41 von Roger am März 9, 2012 - 19:14

    My question is based on the assumption that voting disk group requires a minimum of 3 raw disk to function properly and that there is one voting disk group per database.
42. 

    #42 von Roger am März 17, 2012 - 03:13

    I am installing 11gr2 grid infrastructure and at the end of the install when running the last root.sh script I got this output – I setup the data disk group during the installation and thought I could setup the FRA and OCR_VOTE disk groups after the installation completed however it seemed to take the DATA disk group and use 1 of the disk for the voting disk. I am not sure what it did with the rest of the DATA disk group.

    ASM created and started successfully.

    Disk Group DATA created successfully.

    clscfg: -install mode specified
    Successfully accumulated necessary OCR keys.
    Creating OCR keys for user ‚root‘, privgrp ‚root‘..
    Operation successful.
    CRS-4256: Updating the profile
    Successful addition of voting disk 9c2aea6c433f4fdebf3388c6524b8a66.
    Successfully replaced voting disk group with +DATA.
    CRS-4256: Updating the profile
    CRS-4266: Voting file(s) successfully replaced
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 9c2aea6c433f4fdebf3388c6524b8a66 (/dev/mapper/asm01p1) [DATA]
    Located 1 voting disk(s).
    CRS-2672: Attempting to start ‚ora.asm‘ on ‚lpqmd544‘
    CRS-2676: Start of ‚ora.asm‘ on ‚lpqmd544‘ succeeded
    CRS-2672: Attempting to start ‚ora.DATA.dg‘ on ‚lpqmd544‘
    CRS-2676: Start of ‚ora.DATA.dg‘ on ‚lpqmd544‘ succeeded
43. 

    #43 von Uwe Hesse am März 17, 2012 - 11:36

    Roger, you observe standard behavior here. Because of the external redundancy of the diskgroup, you get only one voting disk.

    1) You don’t have to create a dedicated diskgroup only for the Voting Disk and OCR. It is OK to have the Database Area in the same diskgroup
    2) You can create a dedicated diskgroup for that purpose and move Voting Disk and OCR onto it online afterwards
44. 

    #44 von Roger am März 17, 2012 - 19:18

    Do you have the steps to make a new OCR_VOTE group and copy the existing contents to it and then drop he old group and have the old raw disk available as candidates? I have 3 servers and I need to have identical setups on each one. I want normal redundancy on the voting DG and then my DATA and FRA are externan redundancy
45. 

    #45 von Roger am März 17, 2012 - 22:19

    This is from V$asm_disk – it shows that one of my 120g drives is a voting under the disk group DATA and I would like to change that so the voting group is it’s own.and return the 120g raw disk to the DATA Disk Group

    V NAME TOTAL_MB FREE_MB PATH
    – ———- ———- ———- ——————–
    Y DATA_0000 122879 122755 /dev/mapper/asm01p1
    N DATA_0001 122879 122784 /dev/mapper/asm02p1
    N DATA_0002 122879 122787 /dev/mapper/asm03p1
    N DATA_0003 122879 122784 /dev/mapper/asm04p1

    By the way Thanks for all your replies – also so people know that when you install Grid Infrastructure and get to the part where you create a disk group it only wants you to create the OCR_VOTE disk group there and then after the installation you can add the other disk groups using asmca.
46. 

    #46 von Uwe Hesse am März 19, 2012 - 11:12

    Hi Roger, the output above does not mean that the voting disk is the only file on that ASM disk. It will get Datafile stripes etc. in the same way as the other disks also. So we don’t waste 120G for just one voting disk here.

    If you want to use a dedicated diskgroup for voting disk and OCR, this can be done online with these commands:
    1) crsctl replace votedisk +OCR_VOTE
    this will add 3 voting disks to the new diskgroup if it has a redundancy normal (make sure you have at least 3 disks in that diskgroup)
    2) ocrconfig -add +OCR_VOTE
    will add ONE OCR into the diskgroup that is protected according to the redundancy of that diskgroup
    3) ocrconfig -delete +DATA
    will remove the stripes of the OCR from the DATA diskgroup

    You may now drop the DATA diskgroup or use it for the Database you will create subsequently.
    Make sure to take a backup of the OCR before doing the 3 steps above.
47. 

    #47 von Bertrand am September 14, 2012 - 11:42

    Hi Uwe,

    I have a question regarding 11GR2 extended cluster and the „real“ needs of a third site for the third voting disk.

    As the voting disks are stored into ASM that way (http://docs.oracle.com/cd/E11882_01/rac.112/e16794/votocr.htm#CHDFJHHJ) :

    “
    By default, Oracle ASM puts each voting disk in its own failure group within the disk group. A failure group is a subset of the disks in a disk group.

    If voting disks are stored on Oracle ASM with normal or high redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.
    “

    Example :

    3 voting disks into an ASM diskgroup composed of 4 failgroups :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 8a1954c3473a4f50bfb3e2ce01468d34 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F52d0s0) [OCRVOTE2]
    2. ONLINE 2aa67328d87f4fc9bf900b46cc95a673 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F58d0s0) [OCRVOTE2]
    3. ONLINE 9af8c976efa94f83bf175a0a920f7da9 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Cd0s0) [OCRVOTE2]
    Located 3 voting disk(s).

    As far the failgroup :

    DG FAILGROUP PATH
    —————————— —————————— ————————————————–
    OCRVOTE2 FG1 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F52d0s0
    OCRVOTE2 FG2 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F58d0s0
    OCRVOTE2 FG3 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Cd0s0
    OCRVOTE2 FG4 /dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Ed0s0

    Now if we simulate the lost of one voting disk (the one into FG2 for exemple) :

    SQL> alter diskgroup OCRVOTE2 offline disks in failgroup fg2;

    Diskgroup altered.

    Let’s see if it has been reconstructed into FG4 :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 8a1954c3473a4f50bfb3e2ce01468d34 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F52d0s0) [OCRVOTE2]
    2. ONLINE 9af8c976efa94f83bf175a0a920f7da9 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Cd0s0) [OCRVOTE2]
    3. ONLINE 5d61b8a9596b4f5bbf96855faa384ee7 (/dev/rdsk/c6t60060E80056DEB0000006DEB00000F5Ed0s0) [OCRVOTE2]
    Located 3 voting disk(s).

    Yes that’s the case !

    So, if we use 2 sites with normal redundancy and 2 failure group into each site (total of 2 failure group), then is the third site still needed ?

    (AS a site will not be a single point of failure anymore as the remaining site will be able to host 2 voting disks in case one site crash).

    Thanks for your advice,
    Bertrand
48. 

    #48 von Bertrand am September 14, 2012 - 14:25

    Uwe,

    Correction : I mean „total of 4 failure groups“

    Thx
    Bertrand
49. 

    #49 von Uwe Hesse am September 14, 2012 - 15:13

    Bertrand, when you have an extended RAC with 2 Voting Disks on one site A and 1 on site B – how is the cluster supposed to survive a disaster that destroys the whole site A (including 2 Voting Disks)? 1 out of 3 is simply 1 too few, sorry for saying the obvious 🙂
50. 

    #50 von Bertrand am September 14, 2012 - 16:28

    Hi Uwe,

    Thanks for the reply.

    No problem ;-).

    My example describes Site B with 2 failgroups and site A with 2 failgroups as well.

    Then on site B one failgroup is „empty“ before the crash (as you said 2 voting disks are on site A).

    One voting disk could then been „automatically“ recreated on site B and then site B will get 2 voting disks, right ?

    Thx
    Bertrand
51. 

    #51 von Uwe Hesse am September 14, 2012 - 16:37

    No, I don’t think so. No matter how many failgroups are present, 1 accessible Voting Disk out of 3 is not sufficient to keep the cluster alive, in my opinion. Which is why a third site is recommended for Extended RAC to hold one Voting Disk.
52. 

    #52 von Bertrand am September 14, 2012 - 17:46

    I agree, but according to the oracle documentation :

    “
    If voting disks are stored on Oracle ASM with normal or high redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.
    “

    Then, you think that in my example the Site B will be evicted from the cluster before a second voting disk could be recovered into the „empty“ failgroup located in Site B.

    Is that correct ?

    Thx
    Bertrand
53. 

    #53 von Uwe Hesse am September 14, 2012 - 18:30

    Yes, I think that is correct – and the documentation is only talking about the damage of one failure group (only one Voting Disk affected) whereas in case of a disaster at site A, 2 failure groups (each with one Voting Disk on) are affected at the same time.
54. 

    #54 von Bertrand am September 14, 2012 - 20:17

    Well, i think you are right.
    As you always suggest 🙂 and as i usually do : i will test it and i will let you know.

    Thx for your time
    Bertrand
55. 

    #55 von Uwe Hesse am September 17, 2012 - 11:02

    Bertrand, you’re welcome 🙂 Appreciate if you could share the outcome of your test also.
56. 

    #56 von Bertrand am September 19, 2012 - 10:20

    Uwe,

    The outcome is the following :

    configuration :

    2 sites (SITEA and SITEB) and 4 failgroups for the OCRVOT2 diskgroup :

    DG FAILGROUP PATH
    —————————— —————————— —————————————-
    OCRVOT2 FG1 ORCL:VOTE1SITEA
    OCRVOT2 FG2 ORCL:VOTE2SITEA
    OCRVOT2 FG3 ORCL:VOTE1SITEB
    OCRVOT2 FG4 ORCL:VOTE2SITEB

    current voting disk configuration :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE a8fcb4667eb54ffdbfe45645a1344c4a (ORCL:VOTE1SITEA) [OCRVOT2]
    2. ONLINE 5f12bca463804f04bfe90eefd03f99d4 (ORCL:VOTE2SITEA) [OCRVOT2]
    3. ONLINE 2456cb13571a4f96bf76d30f366fabfb (ORCL:VOTE1SITEB) [OCRVOT2]
    Located 3 voting disk(s).

    —————————– First test :

    Offline one voting disk :

    SQL> alter diskgroup OCRVOT2 offline disks in failgroup fg1;

    Diskgroup altered.

    => new voting disk configuration :

    crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 5f12bca463804f04bfe90eefd03f99d4 (ORCL:VOTE2SITEA) [OCRVOT2]
    2. ONLINE 2456cb13571a4f96bf76d30f366fabfb (ORCL:VOTE1SITEB) [OCRVOT2]
    3. ONLINE 05c73a222df44f07bfea4f42eaf2f03c (ORCL:VOTE2SITEB) [OCRVOT2]

    Conclusion :
    it works as expected, one voting disk has been re-created into one available failgroup.

    —————————– Second test :

    I put the offlined disk back online

    SQL> alter diskgroup OCRVOT2 online disks in failgroup fg1;

    Diskgroup altered

    Now i shoot the SITEB (that hosts the majority of the voting disks) and the result is :

    [cssd(4205)]CRS-1606:The number of voting files available, 1, is less than the minimum number of voting files required, 2, resulting in CSSD termination to ensure data integrity; details at (:CSSNM00018:) in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log

    on both machines.

    => Both machines have been evicted from the cluster.

    —————————– Conclusions :

    – If one voting disk is lost : it will be re-created into one available diskgroup.

    – If the majority of voting disks are lost at the same time : the voting disks are not re-created into the available failgroup(s) and both nodes are evicted from the cluster.

    – you were right 😉
57. 

    #57 von Uwe Hesse am September 19, 2012 - 10:47

    Thank you, Bertrand, for getting back to this site with the result of your test – I really do appreciate it!
58. 

    #58 von Juris Trošins am September 19, 2012 - 11:25

    Please guys do not fall into the nfs trap described in the document. Or at least pay attention to the „Known Issues“ section. I’ve encountered the situations with Extended RAC with 3d voting disk on ASM couple of years ago, when the whole cluster was unusable when NFS got inaccessible. Filed SR to Oracle which resulted in the fix development, which I’m going to put to test in a while. The overall recommendation was to use iscsi as a path for 3d voting disk. It behaves just like scsi without locking resources like NFS.
59. 

    #59 von Uwe Hesse am September 19, 2012 - 11:35

    Thank you for sharing that information. You refer to this issue that Roland & Markus mention in the whitepaper: „If the NFS device location is not accessible […] Starting Oracle Clusterware again with “crsctl start crs” will hang, because some of the old clusterware processes will hang on I/O to the NFS voting file.“ I suppose? It would be very kind if you could share the result of your test of the fix later.
60. 

    #60 von Juris Trošins am September 19, 2012 - 11:42

    Yes, Uwe. Almost. In my case there was a hang even in the middle of something. Which resulted of course in hanging crsctl operations as well. You can check patch 10109915, which was developed in response of my case. My customer went to iSCSI and arranging the test is kinda complicated now. I’m going to give it a try in a local vm env, but still had no time. If you’ll have a time to test – I will much appreciated the reply 🙂
61. 

    #61 von Bertrand am September 28, 2012 - 07:45

    Hello Guys,

    I’ll have to launch soon a fresh install of rac with a third voting disk based on nfs.
    Then i’ll test this patch and let you know.
62. 

    #62 von Bertrand am Oktober 9, 2012 - 13:36

    Hello Guys,

    I will not be able to test it soon as due to ACFS issue on RH 6.3 we had to install 14494305 (Implies CRS is now 11.2.0.3.3) but 10109915 is not available on 11.2.0.3.3.

    Bertrand
63. 

    #63 von Bertrand am Oktober 23, 2012 - 09:25

    Hello Guys,

    I won’t test it because we are moving to ISCSI.

    Bertrand
64. 

    #64 von Uwe Hesse am Oktober 23, 2012 - 16:55

    Bertrand, thanks for keeping us informed 🙂
65. 

    #65 von Jarmo am Dezember 10, 2012 - 14:09

    Hi all
    Does anyone have experiences or found documentation of the case below?

    You have voting disks with normal redundancy on SAN. Then the connection to one of the voting files is lost for a moment. Now, is crs able to recover voting file automaticly, when connection comes working again?
66. 

    #66 von Ramesh am Juni 3, 2013 - 20:38

    Nice One.
67. 

    #67 von Anju Garg am September 20, 2013 - 16:39

    Here is a related link:
    http://oracleinaction.com/voting-disk/
68. 

    #68 von Jay Vardhan am September 20, 2013 - 16:45

    Hello Sir,

    Thanks for nice explanation of all the changes fr OCR and VOTE disk in 11gR2.
    Sir, since from 11gR2, vote disk is also backup with OCR automatically, I would request if you please brief the steps to use this backup in case of failure of vote disk.
69. 

    #69 von Aman.... am September 22, 2013 - 12:25

    @Jay Vardhan,

    Check the Oracle documentation. The steps are listed there.
    http://docs.oracle.com/cd/E11882_01/rac.112/e16794/votocr.htm#BGBCHCBI

    Aman….
70. 

    #70 von Aou am Oktober 7, 2013 - 17:07

    Will 11gR2 voting Disk got backup as well if it is allocated in itsown +VOTDG but not included in +OCRDB??
71. 

    #71 von Leandro am Oktober 16, 2013 - 03:28

    Hello all,

    I´m trying to move voting disk from High redundancy to Normal(DG 3 Failgroup gb each disk all)

    The command crsctl replace votedisk +GRID01 is hunging up and does not return nothing and any log is uptaded (grid alerts etc)

    The diskgroup was create successfully and ocrcheck as well the new Redundancy. I have a SR at Oracle and i will apreciate if you have already faced this situation.

    Steps:

    Name
    ——–
    === ODM Data Collection ===

    * Create Diskgroup: GRID01 (Normal Redundancy) */

    /* Devices */
    /dev/rdsk/c0d53s0
    /dev/rdsk/c0d54s0
    /dev/rdsk/c0d55s0

    * CREATE */
    CREATE DISKGROUP GRID01 NORMAL REDUNDANCY
    FAILGROUP SITE_A DISK
    ‚/dev/rdsk/c0d53s0′ NAME VOTE_MTC1_VD05_OCR_01
    FAILGROUP SITE_B DISK
    ,’/dev/rdsk/c0d54s0′ NAME VOTE_MTC1_VD05_OCR_04
    FAILGROUP SITE_C DISK
    ,’/dev/rdsk/c0d55s0‘ NAME VOTE_MTC1_VD05_OCR_05
    ‚compatible.asm‘ = ‚11.2‘,
    REBALANCE POWER 10;

    Query CSS Before */

    grid@su-br-pro-dbtc01 ~ [+ASM1] >crsctl query css votedisk
    ## STATE File Universal Id File Name Disk group
    — —– —————– ——— ———
    1. ONLINE 3834a46ee4b84f49bfbde0adb188340e (/dev/rdsk/c0d9s0) [OCR]
    2. ONLINE 5b313f4e76c94f75bf76c2cbce65c1a7 (/dev/rdsk/c0d8s0) [OCR]
    3. ONLINE 58de0204606d4f2cbf698a8c17a4cd14 (/dev/rdsk/c0d6s0) [OCR]
    4. ONLINE 093aa04606264f10bf5ad9513e582444 (/dev/rdsk/c0d3s0) [OCR]
    5. ONLINE 8b33492ba3b04f1abf54d3a37643b616 (/dev/rdsk/c0d2s0) [OCR]

    — As root
    /* -show backup */
    ocrconfig -showbackup

    /* backup manual */
    ocrconfig -manualbackup

    su-br-pro-dbtc01 2013/10/14 20:15:55 /soft/appdbs/grid/11.2.0/cdata/su-br-pro-tcbr/backup_20131014_201555.ocr

    /*- create OCR redundancy as ROOT*/

    ocrconfig –add +GRID01

    —ocrcheck

    grid@su-br-pro-dbtc01 /dev/rdsk [+ASM1]>ocrcheck
    Status of Oracle Cluster Registry is as follows :
    Version : 3
    Total space (kbytes) : 262120
    Used space (kbytes) : 3732
    Available space (kbytes) : 258388
    ID : 703883108
    Device/File Name : +OCR
    Device/File integrity check succeeded
    Device/File Name : +GRID01
    Device/File integrity check succeeded

    Device/File not configured

    Device/File not configured

    Device/File not configured

    Cluster registry integrity check succeeded

    Logical corruption check bypassed due to non-privileged user

    grid@su-br-pro-dbtc01 /dev/rdsk [+ASM1]>

    /* – Move Voting as GRID */ — (Hanging up)

    crsctl replace votedisk +GRID01

    .

    FileName
    —————-
    VOTING

    FileComment
    ———————-

    Thanks

    Leandro (Brazil)
72. 

    #72 von Uwe Hesse am Oktober 19, 2013 - 09:21

    Leandro, I hope you have sorted out that issue meanwhile? I didn’t encounter such a problem during voting disk replacement yet.
73. 

    #73 von Moriah am November 13, 2013 - 13:38

    Its very good Article. please share more updates like thats. Thank you. oracleconnections
74. 

    #74 von leoxu am Februar 8, 2014 - 08:01

    hi:
    i read oracle clusterware document that it indicates the following:
    1、Neither should you add a voting disk to a cluster file system in
    addition to the voting disks stored in an Oracle ASM disk group.
    Oracle does not support having voting disks in Oracle ASM and
    directly on a cluster file system for the same cluster at the same
    time.
    2、You cannot combine voting disks
    stored in Oracle ASM and voting disks not stored in Oracle ASM in the same cluster.

    please help me explain it,tks.
75. 

    #75 von Uwe Hesse am Februar 10, 2014 - 17:53

    leoxu, in my opinion the docs are quite clear here: If you have voting disks in an ASM diskgroup, that diskgroup determines the number of voting disks implicitly via the diskgroup redundancy. If you try a command like crsctl add css votedisk you get CRS-4671: This command is not supported for ASM diskgroups
76. 

    #76 von udayjampani am November 13, 2014 - 20:43

    Hi Uwe,

    my question is straight and simple.

    In Normal redundancy with a requirement of minimum 3 fail groups , why ocr is is written in only 2(mirrrored once)?? , where as voting disk is leveraging entire 3 fail groups (3 voting disks) or
    In High redundancy with a requirement of minimum 5 fail groups , why ocr is using only 3(mirrrored twice)??, where as voting disk is leveraging the luxury of entire 5fail groups.(5 voting disks)
77. 

    #77 von Uwe Hesse am November 15, 2014 - 14:55

    Good question! Apparently, the voting disk is more important so we safeguard it more. Especially, there is a need to be able to build a quorum (majority of voting disks) if one is corrupted. Also, the voting disk is not striped the way the OCR is. Could be that we actually use high protection for the OCR even in normal redundancy diskgroups like we do for the database controlfile, though. Must check that somewhen.
78. 

    #78 von Cristian am Juli 18, 2015 - 22:05

    Hi Uwe, I saw all tutorials creates the DATA diskgroup and then move the OCR and vote files to a new diskgroup.

    Questions:

    In a fresh install of Oracle 12c Grid Infrastructure (without repository DB) when we get to the „Create ASM Disk Group“ screen:

    Why don’t we create there the diskgroup OCR_VOTE with normal redundancy using 3 LUNS?

    In this scenario will we achieve the goal of having 3 Voting files and 2 OCR required?

    And after finish Grid Infrastructure installation, we then create the DATA and FRA diskgroups.

    Is there any reason this procedure does not work?

    Thanks in advance

    Cristian
79. 

    #79 von Uwe Hesse am Juli 21, 2015 - 10:50

    Cristian, although I don’t know what tutorial you are talking about, you are correct: The first diskgroup that you create with the installer upon GI installation can be OCR_VOTE with normal redundancy. You may use it for storing only voting disk and OCR. OCR will not be mirrored to another diskgroup automatically, though. Afterwards, you may create diskgroups DATA and FRA for your RAC databases.
80. 

    #80 von Mubarak Ginawi am September 3, 2015 - 09:34

    Thanx Uwe, before read this page i had hold many question in my head but after read this all of them got solved regarding extended RAC.

    we are about to Configuring Oracle Database 11g with Oracle RAC on Extended Clusters using NetApp storage, this days and after finish the configuration, i will share the knowledge we’ll got it as soon as possible.

    and my thanx also to Bertrand for your test.

    but i have a question, if the site A and site C (that contain the third vote disk as NFS) failed at the same time.. the site B will be down.

    is there any way to make it Online and access the database. (I know it’s impossible but i want to make another Fourth mirror for vote disk at any location )
81. 

    #81 von Yasir am März 17, 2016 - 11:36

    create diskgroup data normal redundancy
    failgroup fg1 disk ‚ORCL:ASMDISK1‘
    failgroup fg2 disk ‚ORCL:ASMDISK2‘

    quorum failgroup fg3 disk ‚ORCL:ASMDISK3‘
    attribute ‚compatible.asm‘ = ‚11.2.0.0.0‘;

    In your example, if any of the disk fails, how will oracle maintain the voting disk ?

1. October 2010 Blogroll Report | AskDba.org Weblog
2. Database Migration to ASM with short downtime « The Oracle Instructor
3. Merry Christmas & A Happy New Year 2012! « The Oracle Instructor
4. Merry Christmas & A Happy New Year 2012! | Oracle Administrators Blog - by Aman Sood
5. Drop an ASM Disk that contains a Voting Disk? « The Oracle Instructor
6. Top 10 postings in 2012 « The Oracle Instructor
7. Purpose of the Voting Disk for #Oracle RAC | The Oracle Instructor

Diese Seite verwendet Akismet, um Spam zu reduzieren. Erfahre, wie deine Kommentardaten verarbeitet werden..