Oracle Clusterware 12c: What-if command evaluation - Part I

    Sep 25, 2018 10:50:03 AM by Anju Garg

    Cluster administrators are expected to perform management operations on the cluster which may be hosting various applications and databases. Prior to Oracle clusterware 12c, the effects of cluster management operations could be experienced only after changes had been made to the system by executing the crsctl / srvctl commands. When managing complex and dynamic environments, it is critical that the administrator understands the impact of such operations, for example, a server outage, on various cluster resources prior to actual occurrence of the proposed event. A mistake or lack of appropriate knowledge on part of the administrator could result in undesirable outcomes. Oracle Clusterware 12c introduces an extremely valuable feature “what-if command evaluation” which is designed to evaluate the impact of a command execution against the cluster by simulating the command without making any changes to the system. The output displays the impact of a particular command when it will be actually executed without actually executing the command. As a result, chances of mistake are minimized and administrators can maintain the cluster smoothly. And of course, since the command has not actually been executed, the actual impact doesn’t actually occur.

    Whereas cluster administrators are responsible for overseeing various applications hosted in the cluster, database administrates are concerned about the impact of cluster management operation(s) on the databases they are responsible for. Also, the commands to perform management tasks differ for each group of administrators. Whereas cluster administrators perform management of oracle clusterware components and resources using crsctl utility, database administrators carry out similar monitoring and management activities for Oracle-related resources such as database instances and services using srvctl.

    Accordingly, there are two levels of the What-If Command Evaluation:

    • A view for cluster administrators provided by crsctl
    • A view for database administrators provided by srvctl

    Different command options are available for the srvctl and crsctl which can be used to evaluate nearly all management scenarios prior to actually executing them. Moreover, impact of a component failure can also be predicted by using special commands (e.g. srvctl predict) or command options (e.g. the crsctl command option –fail).

    It is worth mentioning here that What-If Command Evaluation does not provide transactional security i.e. the evaluation is performed on the state of the cluster as of the time of start of evaluation. The evaluation result considers this state only. The actual result will be same as that projected by What-If evaluation only if the cluster state remains same between the start of the evaluation and the actual execution of the command. If the cluster state changes, the result of the actual execution may be different from its projection.

    In this article I will discuss about

    • Cluster administrator’s view of the What-If Command Evaluation
      • What-If Command Evaluation on Application Resources with crsctl
      • What-If Command Evaluation on Oracle Clusterware Resources with crsctl
        • Formatting the Output for What-If Command Evaluation on Oracle Clusterware Resources
      • What-if command evaluation using crsctl eval command: Consideration
    • Database administrator’s view of the What-If Command Evaluation
      • What-If Command Evaluation using srvctl command with -eval option
      • Evaluating Failure Consequences with srvctl predict

    In order to demonstrate examples of What-If Command Evaluation I have setup a cluster as under:

    Current Setup:

    • Name of the cluster: Cluster01
    • Oracle Clusterware active version : 12.1.0.2.0
    • Type of cluster: Flex cluster
    • Hub nodes : host01, host02, host03
    • Leaf Nodes: host04, host05
    • Name of Administrator managed RAC database: amdb
    • Instances of RAC database: amdb1@host01, amdb2@host02
    • Server pools:
      • Free (ACTIVE_SERVERS=host03 host04 host05)
        • apache_sp : (ACTIVE_SERVERS=host04 host05)
    • Generic (ACTIVE_SERVERS=host01 host02)
      • ora.amdb (ACTIVE_SERVERS=host01 host02)
    • A user defined application resource configured on servers in the server pool apache_sp : apache_res

    Cluster administrator’s view of the What-If Command Evaluation

    Using the crsctl eval command, cluster administrators can perform what-if command evaluation on

    • User-defined application resources
    • Oracle clusterware resources i.e. Servers, server pools and policies

    What-If Command Evaluation on Application Resources with crsctl

    What-if command evaluation can be performed by cluster administrators on user-defined application resources to test the effect of starting, stopping, relocating, modifying, or adding resources. Administrators can also examine the effect of a potential resource failure.

    The eval commands available for application resources are:

    $ crsctl eval { start | stop | relocate | modify | add | fail } resource

    • crsctl eval start resource: Predicts the effects of starting a resource without making changes to the system.
    • crsctl eval stop resource: Predicts the effects of stopping a resource without making changes to the system.
    • crsctl eval relocate resource: Simulates relocating a resource without making changes to the system.
    • crsctl eval modify resource: Predicts the effects of modifying a resource without making changes to the system.
    • crsctl eval add resource: Predicts the effects of adding a resource without making changes to the system.
    • crsctl eval fail resource: Predicts the consequences of a failure of a resource.

    In my current setup, I have created a server pool called apache_sp which currently has both the leaf nodes i.e. host04 and host05 assigned to it.

    [grid@host01 bin]$ crsctl status serverpool apache_sp
    NAME=apache_sp
    ACTIVE_SERVERS=host04 host05

    A user-defined application resource apache_res is configured to run on servers in the server pool apache_sp.

    [root@host01 ~]# crsctl stat res apache_res -f |grep SERVER_POOLS
    SERVER_POOLS=apache_sp

    Currently, resource apache_res is executing on host04

    [root@host01 bin]# crsctl stat res apache_res
    NAME=apache_res
    TYPE=cluster_resource
    TARGET=ONLINE
    STATE=ONLINE on host04

    Example: Evaluate effect of stopping resource apache_res

    If resource apache_res is stopped, it will not execute on any server.

    [root@host01 bin]# crsctl eval stop resource apache_res
    Stage Group 1:
    ----------------------------------------------------------------------------
    Stage Number   Required       Action
    ----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]
    --------------------------------------------------------------------------

    Example: Evaluate effect of relocating resource apache_res

    If apache_res is relocated, it will be moved to the only other server in the server pool apache_sp i.e. host05

    [root@host01 bin]# crsctl eval relocate resource apache_res
    Stage Group 1:
    -----------------------------------------------------------------------------
    Stage Number   Required       Action
    -----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]
                                   Resource 'apache_res' (1/1) will be in state
                                   [ONLINE|INTERMEDIATE] on server [host05]
    -----------------------------------------------------------------------------

    Example: Evaluate effect of modifying resource apache_res

    If resource apache_res is modified so that it is configured to run a hub node, it will execute on host03 which is a hub node.

    [root@host01 bin]# crsctl eval modify res apache_res -attr 
    "SERVER_CATEGORY=ora.hub.category, SERVER_POOLS='*'" -f

    Stage Group 1:
    -----------------------------------------------------------------------------
    Stage Number   Required       Action
    -----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]
                                   Resource 'apache_res' (1/1) will be in state
                                   [ONLINE|INTERMEDIATE] on server [host03]
    -----------------------------------------------------------------------------

    What-If Command Evaluation on Oracle Clusterware Resources with crsctl

    Cluster Administrators can employ the crsctl eval command on Oracle clusterware resources - Servers, server pools and policies to perform what-if command evaluation that tests the effect of:

    • Adding, deleting and modifying server pools
    • Adding a server to the cluster
    • Removing a server from the cluster
    • Relocating a server from one server pool to another
    • Enabling a specific management policy

    The eval commands available for Oracle clusterware resources are:

    • Server pools

    $ crsctl eval { add | delete | modify } serverpool ....

    • crsctl eval add serverpool: Predicts the effects of adding a server pool without making changes to the system.
    • crsctl eval modify serverpool: Predicts the effects of modifying a server pool without making changes to the system.
    • crsctl eval delete serverpool: Simulates the deletion of a server pool without making changes to the system.
    • Servers

    $ crsctl eval { add | relocate | delete } server ...

    • crsctl eval add server: Simulates the addition of a server without making changes to the system.
    • crsctl eval delete server: Predicts the effects of deleting a server without making changes to the system.
    • crsctl eval relocate server: Predicts the effects of relocating a server to a different server pool without making changes to the system.
    • Policies

    $ crsctl eval activate policy ...

    • crsctl eval activate policy: Predicts the effects of activating a specific policy without making changes to the system.

     

    Example: Evaluate impact of adding a server pool called leafpool with MIN_SIZE = 1, MAX_SIZE = 2 and IMPORTANCE = 10

    Since importance of leafpool (10) is higher than that of apache_sp (0), both the servers from apache_sp will be moved to leafpool causing resource apache_res to stop.

    [root@host01 ~]# crsctl stat serverpool apache_sp -f | grep IMP

    IMPORTANCE=0

    [root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, MIN_SIZE=1, MAX_SIZE=2" -f
    Stage Group 1:
    -----------------------------------------------------------------------------
    Stage Number   Required       Action
    -----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]

                       Y           Server 'host04' will be moved from pools
                                   [apache_sp Free] to pools [leafpool]

                       Y           Server 'host05' will be moved from pools
                                   [apache_sp Free] to pools [leafpool]
    -----------------------------------------------------------------------------

    Example: Evaluate impact of deleting the server host04

    In stage-I,

    • Resource apache_res which is currently executing on host04, will stop
    • Server host04 will be removed from server pool apache_sp

    In stage - II,

    • Resource apache_res will start executing on server host05, the only other server in the server pool apache_sp

    [root@host01 bin]# crsctl eval delete server host04
    Stage Group 1:
    -----------------------------------------------------------------------------
    Stage Number   Required       Action
    -----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]

                       Y           Server 'host04' will be removed from pools
                                   [apache_sp Free]

         2             Y           Resource 'apache_res' (1/1) will be in state
                                   [ONLINE|INTERMEDIATE] on server [host05]
    -----------------------------------------------------------------------------

    Example: Evaluate impact of activating a policy

    Let us add day_policy and night_policy policies to the cluster

    [root@host01 bin]# crsctl add policy day_policy -attr "DESCRIPTION='The Day policy'"
    [root@host01 bin]# crsctl add policy night_policy -attr "DESCRIPTION='The
    Night policy'"

    Set the SERVER_POOL_NAMES policy set attribute to add server pool apache_sp to the scope of the server pools that are controlled by the policy set.

    [root@host01 bin]# crsctl modify policyset -attr "SERVER_POOL_NAMES='Free
    apache_sp'" -ksp

    Set attributes for the server pool apache_sp in policies day_policy and night_policy so as to have two servers in server pool apache_sp during day time and one server during night time

    [root@host01 bin]# crsctl modify serverpool apache_sp –attr 
    "MAX_SIZE=2,MIN_SIZE=1" -policy day_policy

    [root@host01 bin]# crsctl modify serverpool apache_sp -attr
    "MAX_SIZE=1,MIN_SIZE=1" -policy night_policy

    Now, if we evaluate the impact of activating policies day_policy and night_policy, it can be seen that , as per policies,

    • when day_policy is activated, apache_sp will have 2 servers assigned to it.
    • when night_policy is activated, apache_sp will have 1 server assigned to it.

    [root@host01 bin]# crsctl eval activate policy day_policy -admin -l serverpools
    NAME = Free
    ACTIVE_SERVERS = host03 host04 host05
    NAME = Generic
    ACTIVE_SERVERS = host01 host02
    NAME = apache_sp
    ACTIVE_SERVERS = host04 host05
    NAME = ora.amdb
    ACTIVE_SERVERS = host01 host02

    [root@host01 bin]# crsctl eval activate policy night_policy -admin -l serverpools -f
    NAME = Free
    ACTIVE_SERVERS = host03 host04 host05
    NAME = Generic
    ACTIVE_SERVERS = host01 host02
    NAME = apache_sp
    ACTIVE_SERVERS = host05
    NAME = ora.amdb
    ACTIVE_SERVERS = host01 host02

    Formatting the Output of What-If Command Evaluation

    The crsctl eval commands for cluster administrators may contain additional parameters to govern the format of the command output. These parameters can be used in conjunction with the -admin option and enable cluster administrators to control the amount of information returned by the commands. The -admin option modifies the output so that the administrator is provided with a summary of the status of clusterware resources and /or server pool assignments resulting from the proposed action. Note that status of application resources is not displayed.

    The following additional formatting parameters are available in conjunction with the -admin option:

    -l <level> specifies the output display level.

    -l serverpools displays server pool information.

    -l resources displays resources running on servers in the server pools

    -l all displays server pool and resource information

    -x shows differences only.

    -a shows all resources

    Command syntax:

    $ crsctl eval ... serverpool ... [-admin [-l <level>] [-x] [-a]]

    $ crsctl eval ... server ... [-admin [-l <level>] [-x] [-a]]

    $ crsctl eval activate policy ... [-admin [-l <level>] [-x] [-a]]

    Example: Display status of all server pools after adding server pool leafpool

    As we saw earlier, if we find out impact of adding a server pool called leafpool with IMPORTANCE = 10, MIN_SIZE=1 and MAX_SIZE=2, without specifying any additional parameters, the output predicts the resulting status of resource apache_res and both the servers (host04 / host05) .

    [root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10,
    MIN_SIZE=1, MAX_SIZE=2" -f
    Stage Group 1:
    ----------------------------------------------------------------------------
    Stage Number   Required      Action
    ----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]

                       Y           Server 'host04' will be moved from pools[Free
                                   apache_sp] to pools [leafpool]

                       Y           Server 'host05' will be moved from pools[Free
                                   apache_sp] to pools [leafpool]
    ----------------------------------------------------------------------------

    If we set output display level to server pools only with –a parameter, it shows the resultant status of all the server pools only. Note that servers host04 and host05 will be moved from server pool apache_sp to leafpool.

    [root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, 
    MIN_SIZE=1, MAX_SIZE=2" -f -admin -l serverpools -a
    NAME = Free
    ACTIVE_SERVERS = host03

    NAME = Generic
    ACTIVE_SERVERS = host01 host02

    NAME = apache_sp
    ACTIVE_SERVERS =

    NAME = leafpool
    ACTIVE_SERVERS = host04 host05

    NAME = ora.amdb
    ACTIVE_SERVERS = host01 host02

    Example: Display status of server pools affected by adding server pool leafpool

    If we replace the parameter -a with -x, status of only the affected server pools both before and after adding the server pool leafpool is displayed. Note that Generic and ora.amdb pools are not displayed since they are not affected by addition of leafpool.

    [root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10,
    MIN_SIZE=1, MAX_SIZE=2" -f -admin -l serverpools -x

    NAME = Free
    Old Active Servers = host03 host04 host05
    New Active Servers = host03

    NAME = apache_sp
    Old Active Servers = host04 host05
    New Active Servers =

    NAME = leafpool
    Old Active Servers =
    New Active Servers = host04 host05

    Example: Display status of all non-application resources running on server pools after adding server pool leafpool

    If we set output display level to resources, the resultant status of all the databases and database services will be displayed. Since database amdb does not have any service configured, database services are not displayed here. Note that status of application resources (apache_res) is not displayed.

    [root@host01 bin]# crsctl eval add serverpool leafpool -attr "IMPORTANCE=10, MIN_SIZE=1, MAX_SIZE=2" -f -admin -l resources
    -----------------------------------------------------------------------------
    Name           Target State       Server                   Effect              
    -----------------------------------------------------------------------------
    Cluster Resources
    -----------------------------------------------------------------------------
    ora.amdb.db
         1       ONLINE ONLINE       host01
         2       ONLINE ONLINE       host02
    ora.mgmtdb
         1       ONLINE OFFLINE
    -----------------------------------------------------------------------------

    Example: Display status of server pools affected by removal of server host04 from cluster

    As we saw earlier, if we evaluate the impact of deleting the server host04 from cluster without any additional options, resulting effects on both the application resource apache_res and server pool apache_sp are displayed.

    [root@host01 bin]# crsctl eval delete server host04
    Stage Group 1:
    ----------------------------------------------------------------------------
    Stage Number   Required       Action
    -----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]

                       Y           Server 'host04' will be removed from pools                
                                   [Free apache_sp]

         2             Y           Resource 'apache_res' (1/1) will be in state
                                   [ONLINE|INTERMEDIATE] on server [host05]
    -----------------------------------------------------------------------------

    If we specify additional options -admin -l serverpools and -x, it displays the status of only the affected server pools both before and after deleting the server host04. Note that server host04 has been removed from both Free and apache_sp server pools.

    [root@host01 bin]# crsctl eval delete server host04 -admin -l serverpools -x

    NAME = Free
    Old Active Servers = host03 host04 host05
    New Active Servers = host03 host05

    NAME = apache_sp
    Old Active Servers = host04 host05
    New Active Servers = host05

    Example: Display status of server pools on activating the policy night_policy

    As we saw earlier, if we evaluate the impact of activating night_policy without any additional parameters, impact on both the application resource apache_res and the server pools apache_sp is displayed.

    [root@host01 bin]# crsctl eval activate policy night_policy -f
    Stage Group 1:
    ----------------------------------------------------------------------------
    Stage Number   Required       Action
    ----------------------------------------------------------------------------
         1             Y           Resource 'apache_res' (1/1) will be in state
                                   [OFFLINE]

                       Y           Server 'host04' will be removed from pools
                                   [apache_sp]

         2             Y           Resource 'apache_res' (1/1) will be in state
                                   [ONLINE] on server [host05]
    ----------------------------------------------------------------------------

    If we want to see the status of all the server pools only, we can add -admin -l serverpools option

    [root@host01 bin]# crsctl eval activate policy night_policy -f -admin -l serverpools

    NAME = Free
    ACTIVE_SERVERS = host03 host04 host05

    NAME = Generic
    ACTIVE_SERVERS = host01 host02

    NAME = apache_sp
    ACTIVE_SERVERS = host05

    NAME = ora.amdb
    ACTIVE_SERVERS = host01 host02

    Example: Display status of server pools affected by activation of the policy night_policy

    If we want to see the status of the affected server pools (apache_sp) only , we can add -x option

    [root@host01 bin]# crsctl eval activate policy night_policy -f -admin -l serverpools -x

    NAME = apache_sp

    Old Active Servers = host04 host05
    New Active Servers = host05

    What-if command evaluation using crsctl eval command: Consideration

    What-if command evaluation using the crsctl eval command is recommended for use by cluster administrators in conjunction with user-defined application resources only. For resources with the ora. name prefix, one should use the srvctl predict command or the srvctl ... -eval command. If we try to evaluate the effect of stopping resource with ora. name prefix, for example ora.host01.vip, using crsctl, we will receive error ORA-4995.

    [root@host01 bin]# crsctl eval stop res ora.host01.vip

    CRS-4995: The command 'stop resource' is invalid in crsctl. Use srvctl for this command.

    Summary:

    • Prior to Oracle clusterware 12c, the effects of cluster management operations could be experienced only after changes had been made to the system by executing the crsctl / srvctl commands. A mistake or lack of appropriate knowledge on part of the administrator could result in undesirable outcomes.
    • Oracle Clusterware 12c introduces an extremely valuable feature “what-if command evaluation” which is designed to evaluate the impact of a command execution against the cluster by simulating the command without making any changes to the system.
    • There are two levels of the What-If Command Evaluation:
      • A view for cluster administrators provided by crsctl
      • A view for database administrators provided by srvctl
      • Cluster administrators can use crsctl to
        • Perform What-If Command Evaluation
          • On Application Resources
          • On Oracle Clusterware Resources
          • Format the Output for What-If Command Evaluation on Oracle Clusterware Resources
          • What-If Command Evaluation does not provide transactional security

    Next Steps

    In the next and final articles in this series, I will discuss about :

    • Database administrator’s view of the What-If Command Evaluation
      • What-If Command Evaluation using srvctl command with -eval option
      • Evaluating Failure Consequences with srvctl predict

    References:

    Oracle 12c Release 1 Clusterware Administration and Deployment Guide

    Tags: Oracle

    Anju Garg

    Written by Anju Garg

    Anju Garg is an Oracle Ace with over 14 years of experience in IT Industry in various roles. Since 2010, she has been involved in teaching and has trained more than a hundred DBAs from across the world in various core DBA technologies like RAC, Data guard, Performance Tuning, SQL statement tuning, Database Administration etc. She is a regular speaker at Sangam and OTNYathra. She writes articles about Oracle and is one of the reviewers of the following book published by Pearson Oracle Problem-Solving and Troubleshooting Handbook She is certified for : Oracle 9i Database Administration OCP Oracle 11g Database Administration OCP Oracle 11g Performance Tuning OCE Oracle 11g R2 RAC OCE Oracle 11g SQL Tuning OCE Oracle 12c Database Administration OCP Oracle Real Application Clusters 12c Certified Implementation Specialist She is passionate about learning and has keen interest in RAC and Performance Tuning. She shares her knowledge via her technical blog at http://oracleinaction.com/