Oracle Clusterware 12c: What-if command evaluation - Part I

Cluster administrators are expected to perform management operations on the cluster which may be hosting various applications and databases. Prior to Oracle clusterware 12c, the effects of cluster management operations could be experienced only after changes had been made to the system by executing the crsctl / srvctl commands. When managing complex and dynamic environments, it is critical that the administrator understands the impact of such operations, for example, a server outage, on various cluster resources prior to actual occurrence of the proposed event. A mistake or lack of appropriate knowledge on part of the administrator could result in undesirable outcomes. Oracle Clusterware 12c introduces an extremely valuable feature “what-if command evaluation” which is designed to evaluate the impact of a command execution against the cluster by simulating the command without making any changes to the system. The output displays the impact of a particular command when it will be actually executed without actually executing the command. As a result, chances of mistake are minimized and administrators can maintain the cluster smoothly. And of course, since the command has not actually been executed, the actual impact doesn’t actually occur.

Whereas cluster administrators are responsible for overseeing various applications hosted in the cluster, database administrates are concerned about the impact of cluster management operation(s) on the databases they are responsible for. Also, the commands to perform management tasks differ for each group of administrators. Whereas cluster administrators perform management of oracle clusterware components and resources using crsctl utility, database administrators carry out similar monitoring and management activities for Oracle-related resources such as database instances and services using srvctl.

Accordingly, there are two levels of the What-If Command Evaluation:

• A view for cluster administrators provided by crsctl
• A view for database administrators provided by srvctl

Different command options are available for the srvctl and crsctl which can be used to evaluate nearly all management scenarios prior to actually executing them. Moreover, impact of a component failure can also be predicted by using special commands (e.g. srvctl predict) or command options (e.g. the crsctl command option –fail).

It is worth mentioning here that What-If Command Evaluation does not provide transactional security i.e. the evaluation is performed on the state of the cluster as of the time of start of evaluation. The evaluation result considers this state only. The actual result will be same as that projected by What-If evaluation only if the cluster state remains same between the start of the evaluation and the actual execution of the command. If the cluster state changes, the result of the actual execution may be different from its projection.

In this article I will discuss about

• Cluster administrator’s view of the What-If Command Evaluation
  • What-If Command Evaluation on Application Resources with crsctl
  • What-If Command Evaluation on Oracle Clusterware Resources with crsctl
    • Formatting the Output for What-If Command Evaluation on Oracle Clusterware Resources
  • What-if command evaluation using crsctl eval command: Consideration

• Database administrator’s view of the What-If Command Evaluation
  • What-If Command Evaluation using srvctl command with -eval option
  • Evaluating Failure Consequences with srvctl predict

In order to demonstrate examples of What-If Command Evaluation I have setup a cluster as under:

Current Setup:

• Name of the cluster: Cluster01
• Oracle Clusterware active version : 12.1.0.2.0
• Type of cluster: Flex cluster
• Hub nodes : host01, host02, host03
• Leaf Nodes: host04, host05
• Name of Administrator managed RAC database: amdb
• Instances of RAC database: amdb1@host01, amdb2@host02
• Server pools:
  • Free (ACTIVE_SERVERS=host03 host04 host05)
    • apache_sp : (ACTIVE_SERVERS=host04 host05)

• Generic (ACTIVE_SERVERS=host01 host02)
  • ora.amdb (ACTIVE_SERVERS=host01 host02)

• A user defined application resource configured on servers in the server pool apache_sp : apache_res

Cluster administrator’s view of the What-If Command Evaluation

Using the crsctl eval command, cluster administrators can perform what-if command evaluation on

• User-defined application resources
• Oracle clusterware resources i.e. Servers, server pools and policies

What-If Command Evaluation on Application Resources with crsctl

What-if command evaluation can be performed by cluster administrators on user-defined application resources to test the effect of starting, stopping, relocating, modifying, or adding resources. Administrators can also examine the effect of a potential resource failure.

The eval commands available for application resources are:

$ crsctl eval { start | stop | relocate | modify | add | fail } resource

• crsctl eval start resource: Predicts the effects of starting a resource without making changes to the system.
• crsctl eval stop resource: Predicts the effects of stopping a resource without making changes to the system.
• crsctl eval relocate resource: Simulates relocating a resource without making changes to the system.
• crsctl eval modify resource: Predicts the effects of modifying a resource without making changes to the system.
• crsctl eval add resource: Predicts the effects of adding a resource without making changes to the system.
• crsctl eval fail resource: Predicts the consequences of a failure of a resource.

In my current setup, I have created a server pool called apache_sp which currently has both the leaf nodes i.e. host04 and host05 assigned to it.

[grid@host01 bin]$ crsctl status serverpool apache_sp
NAME=apache_sp
ACTIVE_SERVERS=host04 host05

A user-defined application resource apache_res is configured to run on servers in the server pool apache_sp.

[root@host01 ~]# crsctl stat res apache_res -f |grep SERVER_POOLS
SERVER_POOLS=apache_sp

Currently, resource apache_res is executing on host04

[root@host01 bin]# crsctl stat res apache_res
NAME=apache_res
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on host04

Example: Evaluate effect of stopping resource apache_res

If resource apache_res is stopped, it will not execute on any server.

[root@host01 bin]# crsctl eval stop resource apache_res
Stage Group 1:
----------------------------------------------------------------------------
Stage Number   Required       Action
----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]
--------------------------------------------------------------------------

Example: Evaluate effect of relocating resource apache_res

If apache_res is relocated, it will be moved to the only other server in the server pool apache_sp i.e. host05

[root@host01 bin]# crsctl eval relocate resource apache_res
Stage Group 1:
-----------------------------------------------------------------------------
Stage Number   Required       Action
-----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]
                               Resource 'apache_res' (1/1) will be in state
                               [ONLINE|INTERMEDIATE] on server [host05]
-----------------------------------------------------------------------------

Example: Evaluate effect of modifying resource apache_res

If resource apache_res is modified so that it is configured to run a hub node, it will execute on host03 which is a hub node.

[root@host01 bin]# crsctl eval modify res apache_res -attr 
"SERVER_CATEGORY=ora.hub.category, SERVER_POOLS='*'" -f

Stage Group 1:
-----------------------------------------------------------------------------
Stage Number   Required       Action
-----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]
                               Resource 'apache_res' (1/1) will be in state
                               [ONLINE|INTERMEDIATE] on server [host03]
-----------------------------------------------------------------------------

What-If Command Evaluation on Oracle Clusterware Resources with crsctl

Cluster Administrators can employ the crsctl eval command on Oracle clusterware resources - Servers, server pools and policies to perform what-if command evaluation that tests the effect of:

• Adding, deleting and modifying server pools
• Adding a server to the cluster
• Removing a server from the cluster
• Relocating a server from one server pool to another
• Enabling a specific management policy

The eval commands available for Oracle clusterware resources are:

• Server pools

$ crsctl eval { add | delete | modify } serverpool ....

• crsctl eval add serverpool: Predicts the effects of adding a server pool without making changes to the system.
• crsctl eval modify serverpool: Predicts the effects of modifying a server pool without making changes to the system.
• crsctl eval delete serverpool: Simulates the deletion of a server pool without making changes to the system.

• Servers

$ crsctl eval { add | relocate | delete } server ...

• crsctl eval add server: Simulates the addition of a server without making changes to the system.
• crsctl eval delete server: Predicts the effects of deleting a server without making changes to the system.
• crsctl eval relocate server: Predicts the effects of relocating a server to a different server pool without making changes to the system.

• Policies

$ crsctl eval activate policy ...

• crsctl eval activate policy: Predicts the effects of activating a specific policy without making changes to the system.

Example: Evaluate impact of adding a server pool called leafpool with MIN_SIZE = 1, MAX_SIZE = 2 and IMPORTANCE = 10

Since importance of leafpool (10) is higher than that of apache_sp (0), both the servers from apache_sp will be moved to leafpool causing resource apache_res to stop.

[root@host01 ~]# crsctl stat serverpool apache_sp -f | grep IMP

IMPORTANCE=0

[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, MIN_SIZE=1, MAX_SIZE=2" -f
Stage Group 1:
-----------------------------------------------------------------------------
Stage Number   Required       Action
-----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]

                   Y           Server 'host04' will be moved from pools
                               [apache_sp Free] to pools [leafpool]

                   Y           Server 'host05' will be moved from pools
                               [apache_sp Free] to pools [leafpool]
-----------------------------------------------------------------------------

Example: Evaluate impact of deleting the server host04

In stage-I,

• Resource apache_res which is currently executing on host04, will stop
• Server host04 will be removed from server pool apache_sp

In stage - II,

• Resource apache_res will start executing on server host05, the only other server in the server pool apache_sp

[root@host01 bin]# crsctl eval delete server host04
Stage Group 1:
-----------------------------------------------------------------------------
Stage Number   Required       Action
-----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]

                   Y           Server 'host04' will be removed from pools
                               [apache_sp Free]

     2             Y           Resource 'apache_res' (1/1) will be in state
                               [ONLINE|INTERMEDIATE] on server [host05]
-----------------------------------------------------------------------------

Example: Evaluate impact of activating a policy

Let us add day_policy and night_policy policies to the cluster

[root@host01 bin]# crsctl add policy day_policy -attr "DESCRIPTION='The Day policy'"
[root@host01 bin]# crsctl add policy night_policy -attr "DESCRIPTION='The
Night policy'"

Set the SERVER_POOL_NAMES policy set attribute to add server pool apache_sp to the scope of the server pools that are controlled by the policy set.

[root@host01 bin]# crsctl modify policyset -attr "SERVER_POOL_NAMES='Free
apache_sp'" -ksp

Set attributes for the server pool apache_sp in policies day_policy and night_policy so as to have two servers in server pool apache_sp during day time and one server during night time

[root@host01 bin]# crsctl modify serverpool apache_sp –attr 
"MAX_SIZE=2,MIN_SIZE=1" -policy day_policy

[root@host01 bin]# crsctl modify serverpool apache_sp -attr
"MAX_SIZE=1,MIN_SIZE=1" -policy night_policy

Now, if we evaluate the impact of activating policies day_policy and night_policy, it can be seen that , as per policies,

• when day_policy is activated, apache_sp will have 2 servers assigned to it.
• when night_policy is activated, apache_sp will have 1 server assigned to it.

[root@host01 bin]# crsctl eval activate policy day_policy -admin -l serverpools
NAME = Free
ACTIVE_SERVERS = host03 host04 host05
NAME = Generic
ACTIVE_SERVERS = host01 host02
NAME = apache_sp
ACTIVE_SERVERS = host04 host05
NAME = ora.amdb
ACTIVE_SERVERS = host01 host02

[root@host01 bin]# crsctl eval activate policy night_policy -admin -l serverpools -f
NAME = Free
ACTIVE_SERVERS = host03 host04 host05
NAME = Generic
ACTIVE_SERVERS = host01 host02
NAME = apache_sp
ACTIVE_SERVERS = host05
NAME = ora.amdb
ACTIVE_SERVERS = host01 host02

Formatting the Output of What-If Command Evaluation

The crsctl eval commands for cluster administrators may contain additional parameters to govern the format of the command output. These parameters can be used in conjunction with the -admin option and enable cluster administrators to control the amount of information returned by the commands. The -admin option modifies the output so that the administrator is provided with a summary of the status of clusterware resources and /or server pool assignments resulting from the proposed action. Note that status of application resources is not displayed.

The following additional formatting parameters are available in conjunction with the -admin option:

-l <level> specifies the output display level.

-l serverpools displays server pool information.

-l resources displays resources running on servers in the server pools

-l all displays server pool and resource information

-x shows differences only.

-a shows all resources

Command syntax:

$ crsctl eval ... serverpool ... [-admin [-l <level>] [-x] [-a]]

$ crsctl eval ... server ... [-admin [-l <level>] [-x] [-a]]

$ crsctl eval activate policy ... [-admin [-l <level>] [-x] [-a]]

Example: Display status of all server pools after adding server pool leafpool

As we saw earlier, if we find out impact of adding a server pool called leafpool with IMPORTANCE = 10, MIN_SIZE=1 and MAX_SIZE=2, without specifying any additional parameters, the output predicts the resulting status of resource apache_res and both the servers (host04 / host05) .

[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10,
MIN_SIZE=1, MAX_SIZE=2" -f
Stage Group 1:
----------------------------------------------------------------------------
Stage Number   Required      Action
----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]

                   Y           Server 'host04' will be moved from pools[Free
                               apache_sp] to pools [leafpool]

                   Y           Server 'host05' will be moved from pools[Free
                               apache_sp] to pools [leafpool]
----------------------------------------------------------------------------

If we set output display level to server pools only with –a parameter, it shows the resultant status of all the server pools only. Note that servers host04 and host05 will be moved from server pool apache_sp to leafpool.

[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, 
MIN_SIZE=1, MAX_SIZE=2" -f -admin -l serverpools -a
NAME = Free
ACTIVE_SERVERS = host03

NAME = Generic
ACTIVE_SERVERS = host01 host02

NAME = apache_sp
ACTIVE_SERVERS =

NAME = leafpool
ACTIVE_SERVERS = host04 host05

NAME = ora.amdb
ACTIVE_SERVERS = host01 host02

Example: Display status of server pools affected by adding server pool leafpool

If we replace the parameter -a with -x, status of only the affected server pools both before and after adding the server pool leafpool is displayed. Note that Generic and ora.amdb pools are not displayed since they are not affected by addition of leafpool.

[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10,
MIN_SIZE=1, MAX_SIZE=2" -f -admin -l serverpools -x

NAME = Free
Old Active Servers = host03 host04 host05
New Active Servers = host03

NAME = apache_sp
Old Active Servers = host04 host05
New Active Servers =

NAME = leafpool
Old Active Servers =
New Active Servers = host04 host05

Example: Display status of all non-application resources running on server pools after adding server pool leafpool

If we set output display level to resources, the resultant status of all the databases and database services will be displayed. Since database amdb does not have any service configured, database services are not displayed here. Note that status of application resources (apache_res) is not displayed.

[root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, MIN_SIZE=1, MAX_SIZE=2" -f -admin -l resources
-----------------------------------------------------------------------------
Name           Target State       Server                   Effect              
-----------------------------------------------------------------------------
Cluster Resources
-----------------------------------------------------------------------------
ora.amdb.db
     1       ONLINE ONLINE       host01
     2       ONLINE ONLINE       host02
ora.mgmtdb
     1       ONLINE OFFLINE
-----------------------------------------------------------------------------

Example: Display status of server pools affected by removal of server host04 from cluster

As we saw earlier, if we evaluate the impact of deleting the server host04 from cluster without any additional options, resulting effects on both the application resource apache_res and server pool apache_sp are displayed.

[root@host01 bin]# crsctl eval delete server host04
Stage Group 1:
----------------------------------------------------------------------------
Stage Number   Required       Action
-----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]

                   Y           Server 'host04' will be removed from pools                
                               [Free apache_sp]

     2             Y           Resource 'apache_res' (1/1) will be in state
                               [ONLINE|INTERMEDIATE] on server [host05]
-----------------------------------------------------------------------------

If we specify additional options -admin -l serverpools and -x, it displays the status of only the affected server pools both before and after deleting the server host04. Note that server host04 has been removed from both Free and apache_sp server pools.

[root@host01 bin]# crsctl eval delete server host04 -admin -l serverpools -x

NAME = Free
Old Active Servers = host03 host04 host05
New Active Servers = host03 host05

NAME = apache_sp
Old Active Servers = host04 host05
New Active Servers = host05

Example: Display status of server pools on activating the policy night_policy

As we saw earlier, if we evaluate the impact of activating night_policy without any additional parameters, impact on both the application resource apache_res and the server pools apache_sp is displayed.

[root@host01 bin]# crsctl eval activate policy night_policy -f
Stage Group 1:
----------------------------------------------------------------------------
Stage Number   Required       Action
----------------------------------------------------------------------------
     1             Y           Resource 'apache_res' (1/1) will be in state
                               [OFFLINE]

                   Y           Server 'host04' will be removed from pools
                               [apache_sp]

     2             Y           Resource 'apache_res' (1/1) will be in state
                               [ONLINE] on server [host05]
----------------------------------------------------------------------------

If we want to see the status of all the server pools only, we can add -admin -l serverpools option

[root@host01 bin]# crsctl eval activate policy night_policy -f -admin -l serverpools

NAME = Free
ACTIVE_SERVERS = host03 host04 host05

NAME = Generic
ACTIVE_SERVERS = host01 host02

NAME = apache_sp
ACTIVE_SERVERS = host05

NAME = ora.amdb
ACTIVE_SERVERS = host01 host02

Example: Display status of server pools affected by activation of the policy night_policy

If we want to see the status of the affected server pools (apache_sp) only , we can add -x option

[root@host01 bin]# crsctl eval activate policy night_policy -f -admin -l serverpools -x

NAME = apache_sp

Old Active Servers = host04 host05
New Active Servers = host05

What-if command evaluation using crsctl eval command: Consideration

What-if command evaluation using the crsctl eval command is recommended for use by cluster administrators in conjunction with user-defined application resources only. For resources with the ora. name prefix, one should use the srvctl predict command or the srvctl ... -eval command. If we try to evaluate the effect of stopping resource with ora. name prefix, for example ora.host01.vip, using crsctl, we will receive error ORA-4995.

[root@host01 bin]# crsctl eval stop res ora.host01.vip

CRS-4995: The command 'stop resource' is invalid in crsctl. Use srvctl for this command.

Summary:

• Prior to Oracle clusterware 12c, the effects of cluster management operations could be experienced only after changes had been made to the system by executing the crsctl / srvctl commands. A mistake or lack of appropriate knowledge on part of the administrator could result in undesirable outcomes.
• Oracle Clusterware 12c introduces an extremely valuable feature “what-if command evaluation” which is designed to evaluate the impact of a command execution against the cluster by simulating the command without making any changes to the system.
• There are two levels of the What-If Command Evaluation:
  • A view for cluster administrators provided by crsctl
  • A view for database administrators provided by srvctl
  • Cluster administrators can use crsctl to
    • Perform What-If Command Evaluation
      • On Application Resources
      • On Oracle Clusterware Resources
      • Format the Output for What-If Command Evaluation on Oracle Clusterware Resources
      • What-If Command Evaluation does not provide transactional security

Next Steps

In the next and final articles in this series, I will discuss about :

• Database administrator’s view of the What-If Command Evaluation
  • What-If Command Evaluation using srvctl command with -eval option
  • Evaluating Failure Consequences with srvctl predict

References:

Oracle 12c Release 1 Clusterware Administration and Deployment Guide