Sep 25, 2018 9:50:03 AM by Anju Garg
Cluster administrators are expected to perform management operations on the cluster which may be hosting various applications and databases. Prior to Oracle clusterware 12c, the effects of cluster management operations could be experienced only after changes had been made to the system by executing the crsctl / srvctl commands. When managing complex and dynamic environments, it is critical that the administrator understands the impact of such operations, for example, a server outage, on various cluster resources prior to actual occurrence of the proposed event. A mistake or lack of appropriate knowledge on part of the administrator could result in undesirable outcomes. Oracle Clusterware 12c introduces an extremely valuable feature “what-if command evaluation” which is designed to evaluate the impact of a command execution against the cluster by simulating the command without making any changes to the system. The output displays the impact of a particular command when it will be actually executed without actually executing the command. As a result, chances of mistake are minimized and administrators can maintain the cluster smoothly. And of course, since the command has not actually been executed, the actual impact doesn’t actually occur.
Whereas cluster administrators are responsible for overseeing various applications hosted in the cluster, database administrates are concerned about the impact of cluster management operation(s) on the databases they are responsible for. Also, the commands to perform management tasks differ for each group of administrators. Whereas cluster administrators perform management of oracle clusterware components and resources using crsctl utility, database administrators carry out similar monitoring and management activities for Oracle-related resources such as database instances and services using srvctl.
Accordingly, there are two levels of the What-If Command Evaluation:
Different command options are available for the srvctl and crsctl which can be used to evaluate nearly all management scenarios prior to actually executing them. Moreover, impact of a component failure can also be predicted by using special commands (e.g. srvctl predict) or command options (e.g. the crsctl command option –fail).
It is worth mentioning here that What-If Command Evaluation does not provide transactional security i.e. the evaluation is performed on the state of the cluster as of the time of start of evaluation. The evaluation result considers this state only. The actual result will be same as that projected by What-If evaluation only if the cluster state remains same between the start of the evaluation and the actual execution of the command. If the cluster state changes, the result of the actual execution may be different from its projection.
In this article I will discuss about
In order to demonstrate examples of What-If Command Evaluation I have setup a cluster as under:
Using the crsctl eval command, cluster administrators can perform what-if command evaluation on
What-if command evaluation can be performed by cluster administrators on user-defined application resources to test the effect of starting, stopping, relocating, modifying, or adding resources. Administrators can also examine the effect of a potential resource failure.
The eval commands available for application resources are:
$ crsctl eval { start | stop | relocate | modify | add | fail } resource
In my current setup, I have created a server pool called apache_sp which currently has both the leaf nodes i.e. host04 and host05 assigned to it.
[grid@host01 bin]$ crsctl status serverpool apache_sp
NAME=apache_sp
ACTIVE_SERVERS=host04 host05
A user-defined application resource apache_res is configured to run on servers in the server pool apache_sp.
[root@host01 ~]# crsctl stat res apache_res -f |grep SERVER_POOLS
SERVER_POOLS=apache_sp
Currently, resource apache_res is executing on host04
[root@host01 bin]# crsctl stat res apache_res
NAME=apache_res
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on host04
If resource apache_res is stopped, it will not execute on any server.
[root@host01 bin]# crsctl eval stop resource apache_res
Stage Group 1:
----------------------------------------------------------------------------
Stage Number Required Action
----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
--------------------------------------------------------------------------
If apache_res is relocated, it will be moved to the only other server in the server pool apache_sp i.e. host05
[root@host01 bin]# crsctl eval relocate resource apache_res
Stage Group 1:
-----------------------------------------------------------------------------
Stage Number Required Action
-----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
Resource 'apache_res' (1/1) will be in state
[ONLINE|INTERMEDIATE] on server [host05]
-----------------------------------------------------------------------------
If resource apache_res is modified so that it is configured to run a hub node, it will execute on host03 which is a hub node.
[root@host01 bin]# crsctl eval modify res apache_res -attr
"SERVER_CATEGORY=ora.hub.category, SERVER_POOLS='*'" -f
Stage Group 1:
-----------------------------------------------------------------------------
Stage Number Required Action
-----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
Resource 'apache_res' (1/1) will be in state
[ONLINE|INTERMEDIATE] on server [host03]
-----------------------------------------------------------------------------
Cluster Administrators can employ the crsctl eval command on Oracle clusterware resources - Servers, server pools and policies to perform what-if command evaluation that tests the effect of:
The eval commands available for Oracle clusterware resources are:
$ crsctl eval { add | delete | modify } serverpool ....
$ crsctl eval { add | relocate | delete } server ...
$ crsctl eval activate policy ...
Since importance of leafpool (10) is higher than that of apache_sp (0), both the servers from apache_sp will be moved to leafpool causing resource apache_res to stop.
[root@host01 ~]# crsctl stat serverpool apache_sp -f | grep IMP
IMPORTANCE=0
[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, MIN_SIZE=1, MAX_SIZE=2" -f
Stage Group 1:
-----------------------------------------------------------------------------
Stage Number Required Action
-----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
Y Server 'host04' will be moved from pools
[apache_sp Free] to pools [leafpool]
Y Server 'host05' will be moved from pools
[apache_sp Free] to pools [leafpool]
-----------------------------------------------------------------------------
In stage-I,
In stage - II,
[root@host01 bin]# crsctl eval delete server host04
Stage Group 1:
-----------------------------------------------------------------------------
Stage Number Required Action
-----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
Y Server 'host04' will be removed from pools
[apache_sp Free]
2 Y Resource 'apache_res' (1/1) will be in state
[ONLINE|INTERMEDIATE] on server [host05]
-----------------------------------------------------------------------------
Let us add day_policy and night_policy policies to the cluster
[root@host01 bin]# crsctl add policy day_policy -attr "DESCRIPTION='The Day policy'"
[root@host01 bin]# crsctl add policy night_policy -attr "DESCRIPTION='The
Night policy'"
Set the SERVER_POOL_NAMES policy set attribute to add server pool apache_sp to the scope of the server pools that are controlled by the policy set.
[root@host01 bin]# crsctl modify policyset -attr "SERVER_POOL_NAMES='Free
apache_sp'" -ksp
Set attributes for the server pool apache_sp in policies day_policy and night_policy so as to have two servers in server pool apache_sp during day time and one server during night time
[root@host01 bin]# crsctl modify serverpool apache_sp –attr
"MAX_SIZE=2,MIN_SIZE=1" -policy day_policy
[root@host01 bin]# crsctl modify serverpool apache_sp -attr
"MAX_SIZE=1,MIN_SIZE=1" -policy night_policy
Now, if we evaluate the impact of activating policies day_policy and night_policy, it can be seen that , as per policies,
[root@host01 bin]# crsctl eval activate policy day_policy -admin -l serverpools
NAME = Free
ACTIVE_SERVERS = host03 host04 host05
NAME = Generic
ACTIVE_SERVERS = host01 host02
NAME = apache_sp
ACTIVE_SERVERS = host04 host05
NAME = ora.amdb
ACTIVE_SERVERS = host01 host02
[root@host01 bin]# crsctl eval activate policy night_policy -admin -l serverpools -f
NAME = Free
ACTIVE_SERVERS = host03 host04 host05
NAME = Generic
ACTIVE_SERVERS = host01 host02
NAME = apache_sp
ACTIVE_SERVERS = host05
NAME = ora.amdb
ACTIVE_SERVERS = host01 host02
The crsctl eval commands for cluster administrators may contain additional parameters to govern the format of the command output. These parameters can be used in conjunction with the -admin option and enable cluster administrators to control the amount of information returned by the commands. The -admin option modifies the output so that the administrator is provided with a summary of the status of clusterware resources and /or server pool assignments resulting from the proposed action. Note that status of application resources is not displayed.
The following additional formatting parameters are available in conjunction with the -admin option:
-l <level> specifies the output display level.
-l serverpools displays server pool information.
-l resources displays resources running on servers in the server pools
-l all displays server pool and resource information
-x shows differences only.
-a shows all resources
$ crsctl eval ... serverpool ... [-admin [-l <level>] [-x] [-a]]
$ crsctl eval ... server ... [-admin [-l <level>] [-x] [-a]]
$ crsctl eval activate policy ... [-admin [-l <level>] [-x] [-a]]
As we saw earlier, if we find out impact of adding a server pool called leafpool with IMPORTANCE = 10, MIN_SIZE=1 and MAX_SIZE=2, without specifying any additional parameters, the output predicts the resulting status of resource apache_res and both the servers (host04 / host05) .
[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10,
MIN_SIZE=1, MAX_SIZE=2" -f
Stage Group 1:
----------------------------------------------------------------------------
Stage Number Required Action
----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
Y Server 'host04' will be moved from pools[Free
apache_sp] to pools [leafpool]
Y Server 'host05' will be moved from pools[Free
apache_sp] to pools [leafpool]
----------------------------------------------------------------------------
If we set output display level to server pools only with –a parameter, it shows the resultant status of all the server pools only. Note that servers host04 and host05 will be moved from server pool apache_sp to leafpool.
[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10,
MIN_SIZE=1, MAX_SIZE=2" -f -admin -l serverpools -a
NAME = Free
ACTIVE_SERVERS = host03
NAME = Generic
ACTIVE_SERVERS = host01 host02
NAME = apache_sp
ACTIVE_SERVERS =
NAME = leafpool
ACTIVE_SERVERS = host04 host05
NAME = ora.amdb
ACTIVE_SERVERS = host01 host02
If we replace the parameter -a with -x, status of only the affected server pools both before and after adding the server pool leafpool is displayed. Note that Generic and ora.amdb pools are not displayed since they are not affected by addition of leafpool.
[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10,
MIN_SIZE=1, MAX_SIZE=2" -f -admin -l serverpools -x
NAME = Free
Old Active Servers = host03 host04 host05
New Active Servers = host03
NAME = apache_sp
Old Active Servers = host04 host05
New Active Servers =
NAME = leafpool
Old Active Servers =
New Active Servers = host04 host05
If we set output display level to resources, the resultant status of all the databases and database services will be displayed. Since database amdb does not have any service configured, database services are not displayed here. Note that status of application resources (apache_res) is not displayed.
[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, MIN_SIZE=1, MAX_SIZE=2" -f -admin -l resources
-----------------------------------------------------------------------------
Name Target State Server Effect
-----------------------------------------------------------------------------
Cluster Resources
-----------------------------------------------------------------------------
ora.amdb.db
1 ONLINE ONLINE host01
2 ONLINE ONLINE host02
ora.mgmtdb
1 ONLINE OFFLINE
-----------------------------------------------------------------------------
As we saw earlier, if we evaluate the impact of deleting the server host04 from cluster without any additional options, resulting effects on both the application resource apache_res and server pool apache_sp are displayed.
[root@host01 bin]# crsctl eval delete server host04
Stage Group 1:
----------------------------------------------------------------------------
Stage Number Required Action
-----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
Y Server 'host04' will be removed from pools
[Free apache_sp]
2 Y Resource 'apache_res' (1/1) will be in state
[ONLINE|INTERMEDIATE] on server [host05]
-----------------------------------------------------------------------------
If we specify additional options -admin -l serverpools and -x, it displays the status of only the affected server pools both before and after deleting the server host04. Note that server host04 has been removed from both Free and apache_sp server pools.
[root@host01 bin]# crsctl eval delete server host04 -admin -l serverpools -x
NAME = Free
Old Active Servers = host03 host04 host05
New Active Servers = host03 host05
NAME = apache_sp
Old Active Servers = host04 host05
New Active Servers = host05
As we saw earlier, if we evaluate the impact of activating night_policy without any additional parameters, impact on both the application resource apache_res and the server pools apache_sp is displayed.
[root@host01 bin]# crsctl eval activate policy night_policy -f
Stage Group 1:
----------------------------------------------------------------------------
Stage Number Required Action
----------------------------------------------------------------------------
1 Y Resource 'apache_res' (1/1) will be in state
[OFFLINE]
Y Server 'host04' will be removed from pools
[apache_sp]
2 Y Resource 'apache_res' (1/1) will be in state
[ONLINE] on server [host05]
----------------------------------------------------------------------------
If we want to see the status of all the server pools only, we can add -admin -l serverpools option
[root@host01 bin]# crsctl eval activate policy night_policy -f -admin -l serverpools
NAME = Free
ACTIVE_SERVERS = host03 host04 host05
NAME = Generic
ACTIVE_SERVERS = host01 host02
NAME = apache_sp
ACTIVE_SERVERS = host05
NAME = ora.amdb
ACTIVE_SERVERS = host01 host02
If we want to see the status of the affected server pools (apache_sp) only , we can add -x option
[root@host01 bin]# crsctl eval activate policy night_policy -f -admin -l serverpools -x
NAME = apache_sp
Old Active Servers = host04 host05
New Active Servers = host05
What-if command evaluation using the crsctl eval command is recommended for use by cluster administrators in conjunction with user-defined application resources only. For resources with the ora. name prefix, one should use the srvctl predict command or the srvctl ... -eval command. If we try to evaluate the effect of stopping resource with ora. name prefix, for example ora.host01.vip, using crsctl, we will receive error ORA-4995.
[root@host01 bin]# crsctl eval stop res ora.host01.vip
CRS-4995: The command 'stop resource' is invalid in crsctl. Use srvctl for this command.
In the next and final articles in this series, I will discuss about :
Oracle 12c Release 1 Clusterware Administration and Deployment Guide
Tags: Oracle
Written by Anju Garg
Anju Garg is an Oracle Ace with over 14 years of experience in IT Industry in various roles. Since 2010, she has been involved in teaching and has trained more than a hundred DBAs from across the world in various core DBA technologies like RAC, Data guard, Performance Tuning, SQL statement tuning, Database Administration etc.
She is a regular speaker at Sangam and OTNYathra. She writes articles about Oracle and is one of the reviewers of the following book published by Pearson Oracle Problem-Solving and Troubleshooting Handbook
She is certified for :
She is passionate about learning and has keen interest in RAC and Performance Tuning. She shares her knowledge via her technical blog at http://oracleinaction.com/
We use cookies to improve your experience with our site. By continuing to use this site, you consent to our use of cookies. Learn more.