An Exadata Database Machine X2-2 Full Rack comes with 8 Database Nodes and 14 Storage Servers (Cells). The Cell Administration can be done on the command line as user celladmin with CellCLI, as shown in previous postings already, but it would be an annoying task to do all commands 14 times. Therefore, dcli was introduced to enable us to control multiple Cells with a single command. It comes with a nice help switch:
[celladmin@cell1 ~]$ dcli -h Distributed Shell for Oracle Storage This script executes commands on multiple cells in parallel threads. The cells are referenced by their domain name or ip address. Local files can be copied to cells and executed on cells. This tool does not support interactive sessions with host applications. Use of this tool assumes ssh is running on local host and cells. The -k option should be used initially to perform key exchange with cells. User may be prompted to acknowledge cell authenticity, and may be prompted for the remote user password. This -k step is serialized to prevent overlayed prompts. After -k option is used once, then subsequent commands to the same cells do not require -k and will not require passwords for that user from the host. Command output (stdout and stderr) is collected and displayed after the copy and command execution has finished on all cells. Options allow this command output to be abbreviated. Return values: 0 -- file or command was copied and executed successfully on all cells 1 -- one or more cells could not be reached or remote execution returned non-zero status. 2 -- An error prevented any command execution Examples: dcli -g mycells -k dcli -c stsd2s2,stsd2s3 vmstat dcli -g mycells cellcli -e alter iormplan active dcli -g mycells -x reConfig.scl usage: dcli [options] [command] options: --version show program's version number and exit -c CELLS comma-separated list of cells -d DESTFILE destination directory or file -f FILE file to be copied -g GROUPFILE file containing list of cells -h, --help show help message and exit -k push ssh key to cell's authorized_keys file -l USERID user to login as on remote cells (default: celladmin) -n abbreviate non-error output -r REGEXP abbreviate output lines matching a regular expression -s SSHOPTIONS string of options passed through to ssh --scp=SCPOPTIONS string of options passed through to scp if different from sshoptions --serial serialize execution over the cells -t list target cells --unkey drop keys from target cells' authorized_keys file -v print extra messages to stdout --vmstat=VMSTATOPS vmstat command options -x EXECFILE file to be copied and executed
We will see some of the most useful (in my view) commands in the following. First, we need to setup user equivalence for the celladmin user to all cells. If the directory .ssh does not exist yet, we need to create it first on all the cells. On my demo machine, I have only 2 cells:
[celladmin@cell1 ~]$ mkdir ~/.ssh [celladmin@cell1 ~]$ chmod 700 ~/.ssh [celladmin@cell2 ~]$ mkdir ~/.ssh [celladmin@cell2 ~]$ chmod 700 ~/.ssh
Very useful is the usage of the -g switch that points to a text file containing all Cells resp. a group of Cells (IP aliases or IP Adresses):
[celladmin@cell1 ~]$ cat cells.txt cell1 cell2
The establishment of user equivalence is made easy by the -k switch:
[celladmin@cell1 ~]$ dcli -k -g cells.txt Error: Neither RSA nor DSA keys have been generated for current user. Run 'ssh-keygen -t dsa' to generate an ssh key pair.
Well, I just do what I’ve been told:
[celladmin@cell1 ~]$ ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/celladmin/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/celladmin/.ssh/id_dsa. Your public key has been saved in /home/celladmin/.ssh/id_dsa.pub. The key fingerprint is: e6:25:1f:2f:22:a9:5c:ec:e4:98:64:67:91:60:ce:9d celladmin@cell1.example.com
Second try:
[celladmin@cell1 ~]$ dcli -k -g cells.txt The authenticity of host 'cell1 (127.0.0.1)' can't be established. RSA key fingerprint is 99:86:a5:3f:f1:98:75:53:e8:92:fc:7d:fd:4d:aa:45. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'cell1' (RSA) to the list of known hosts. celladmin@cell1's password: The authenticity of host 'cell2 (192.168.56.103)' can't be established. RSA key fingerprint is 99:86:a5:3f:f1:98:75:53:e8:92:fc:7d:fd:4d:aa:45. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'cell2,192.168.56.103' (RSA) to the list of known hosts. celladmin@cell2's password: cell1: ssh key added cell2: ssh key added
Now just testing the user equivalence:
[celladmin@cell1 ~]$ ssh cell1 date Tue Mar 22 03:21:48 PDT 2011 [celladmin@cell1 ~]$ ssh cell2 date Tue Mar 22 03:21:53 PDT 2011
Works! Now wasn’t that easy? Our first Multi-Cell-Command will be in the most simple form:
[celladmin@cell1 ~]$ dcli -g cells.txt "cellcli -e list cell" cell1: cell1 online cell2: cell2 online
The next command will validate the configuration of all the Cells. Notice the „Success“ message (Return Code 0) each one returns:
[celladmin@cell1 ~]$ dcli -g cells.txt "cellcli -e alter cell validate configuration" cell1: Cell cell1 successfully altered cell2: Cell cell2 successfully altered
We can suppress those normal messages because we may only want to see if something is not normal. That does the -n switch:
[celladmin@cell1 ~]$ dcli -g cells.txt -n "cellcli -e alter cell validate configuration" OK: ['cell1', 'cell2']
The -x switch will copy a text file to all the cells and execute it. The suffix need to be .scl (small L) to be accepted as a CellCLI Batch file. Otherwise it is assumed that it contains OS commands. For some odd reason, we need to make it OS executable in both cases:
[celladmin@cell1 ~]$ cat mycommands.scl list cell list flashcache [celladmin@cell1 ~]$ dcli -g cells.txt -x mycommands.scl Error: Exec file does not have owner execute permissions [celladmin@cell1 ~]$ chmod 770 *.scl [celladmin@cell1 ~]$ dcli -g cells.txt -x mycommands.scl cell1: cell1 online cell1: cell1_FLASHCACHE normal cell2: cell2 online cell2: cell2_FLASHCACHE normal
The -r switch suppresses output lines with regular expressions. Suppose I want to suppress the griddisks that belong to diskgroup reco:
[celladmin@cell1 ~]$ dcli -g cells.txt -r "reco" "cellcli -e list griddisk" reco: ['cell1', 'cell2'] cell1: data_CD_disk01_cell1 active cell1: data_CD_disk02_cell1 active cell1: data_CD_disk03_cell1 active cell1: data_CD_disk04_cell1 active cell1: data_CD_disk05_cell1 active cell1: data_CD_disk06_cell1 active cell1: data_CD_disk07_cell1 active cell1: data_CD_disk08_cell1 active cell1: data_CD_disk09_cell1 active cell1: data_CD_disk10_cell1 active cell1: data_CD_disk11_cell1 active cell1: data_CD_disk12_cell1 active cell2: data_CD_disk01_cell2 active cell2: data_CD_disk02_cell2 active cell2: data_CD_disk03_cell2 active cell2: data_CD_disk04_cell2 active cell2: data_CD_disk05_cell2 active cell2: data_CD_disk06_cell2 active cell2: data_CD_disk07_cell2 active cell2: data_CD_disk08_cell2 active cell2: data_CD_disk09_cell2 active cell2: data_CD_disk10_cell2 active cell2: data_CD_disk11_cell2 active cell2: data_CD_disk12_cell2 active
The –vmstat switch lets us vmstat all the cells with one command:
[celladmin@cell1 ~]$ dcli -g cells.txt --vmstat="-a 3 2" procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ 04:11:30: r b swpd free inact active si so bi bo in cs us sy id wa st cell1: 0 0 0 430620 326880 698244 0 0 73 50 1062 811 1 14 83 1 0 cell2: 0 0 0 437336 326988 691956 0 0 65 48 1056 680 1 14 84 1 0 Minimum: 0 0 0 430620 326880 691956 0 0 65 48 1056 680 1 14 83 1 0 Maximum: 0 0 0 437336 326988 698244 0 0 73 50 1062 811 1 14 84 1 0 Average: 0 0 0 433978 326934 695100 0 0 69 49 1059 745 1 14 83 1 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ 04:11:34: r b swpd free inact active si so bi bo in cs us sy id wa st cell1: 0 0 0 430620 326900 698252 0 0 0 19 1028 10452 0 16 83 0 0 cell2: 0 0 0 437336 327008 691972 0 0 0 19 1027 11320 0 7 93 0 0 Minimum: 0 0 0 430620 326900 691972 0 0 0 19 1027 10452 0 7 83 0 0 Maximum: 0 0 0 437336 327008 698252 0 0 0 19 1028 11320 0 16 93 0 0 Average: 0 0 0 433978 326954 695112 0 0 0 19 1027 10886 0 11 88 0 0
Conclusion: With dcli, we have a powerful utility to run commands on multiple Cells without much effort.
#1 von Surachart Opun am März 22, 2011 - 12:58
Good for Distributed Shell.
curious… Idea use „ssh“ no password -)
It’s secure, isn’t it?
#2 von Peter am März 22, 2011 - 13:42
Hi,
Question – Isnt the X2-8 a set config with just two ‚big‘ servers with 8 cpu sockets each? And the x2-2 config is ‚configurable‘ (1/4 – 1/2 & full rack), but with smaller servers?
/P
#3 von Uwe Hesse am März 22, 2011 - 13:49
Peter,
you are right. Will correct that in the posting, although the content regarding the Cells is not affected 🙂
#4 von Peter am März 22, 2011 - 15:44
Uwe,
Its an excellent post, I meant to write that earlier 😉
regds
/P
#5 von Uwe Hesse am März 23, 2011 - 08:48
@Surachart: Even without specifying a passphrase, I need to be able to login as celladmin on one cell before I can make use of user equivalence. It’s rather unlikely that all the cells have different passwords for the celladmin user even on a production system. On my demo system, security is a minor concern anyway 🙂
@Peter: Thank you for the nice feedback!
#6 von Radoslav Golian am April 16, 2011 - 20:35
@Surachart Opun @Uwe
Authentication is done by those keys generated by ssh-keygen. It’s asymmetric (public-private keys are used).. Key is saved to cells by command: dcli -k -g cells.txt
see this output:
cell1: ssh key added
cell2: ssh key added
-k push ssh key to cell’s authorized_keys file
when you’re connecting cell looks to the authorized_keys file and tries to find your key..
So it’s very secure (more than symmetric password authentication), but you should protect your private keys..
good practice is to set keystore password
-storepass mystorepass
or key pair password:
-keypass mykeypass
#7 von Radoslav Golian am April 16, 2011 - 20:46
i made a mistake: there’s no switch -storepass or -keypass in ssh-keygen (they are in keytool)..
#8 von Madness am April 17, 2011 - 12:20
Really, what is it the innovation of dcli ?
The remote execution of commands in different machines without password (using public keys) under the SSH protocol is old.
In this case Oracle has invented the wheel.
#9 von SPA2 am Juli 31, 2012 - 15:15
Hi Uwe,
I have a question concerning the storage servers.
We have a quarter rack with 3 storage servers.
How can I check if we have unallocated disk space via command line if possible?
many thanks
regards
SPA2
#10 von Uwe Hesse am August 2, 2012 - 09:14
You go on one DB server, set the Grid Infrastructure env and type
asmcmd lsdg
that show your diskgroups space
asmcmd lsdsk –candidate
that shows whether griddisks are not yet part of diskgroups
#11 von umair am November 11, 2014 - 20:00
Hi…Can we download and use dcli on non exadata RAC environments.
#12 von Uwe Hesse am November 15, 2014 - 14:28
Technically that should be possible. Only that I have not seen a place to download it yet.
#13 von sshdba am Februar 1, 2015 - 17:28
Reblogged this on Easy Oracle DBA and commented:
Uwe’s Brillaint article on the very powerful DCLI utility for Exadata
#14 von sshdba am Februar 1, 2015 - 17:29
Brilliant article as always Uwe