Troubleshooting GSM-GDS 12c

Troubleshooting is key part of any system when it is not functioning as expected, like wise when managing and working with GSM/GDS there are various commands available and various techniques in order to evaluate the issues, This article explains what are the techniques available and can be used in order to troubleshoot to catch the valuable information.

Troubleshooting Methods – GSM/GDS

Troubleshooting can be done based on the nature of issue, based on that we have to explore the things, In GSM/GDS case the troubleshooting mainly can be done from the GDSCTL utility and also few other piece of source, we will discuss all the available sources to diagnosis the problem.

Head start – Alert log

Consider database part and we have the alert log and also for the listener we have the log file, like wise for GSM we have the alert log which writes the information of all the activities performed at GSM level like which services are connected and what is the status, monitoring status and much. This file named as alert_gsm.log under the diag destination of the GSM. This information also can be gathered from the GSM configuration like below.

GDSCTL>status gsm
Alias sagsm
Version 12.1.0.2.0
Start Date 28-JUL-2016 05:24:01
Trace Level support
Listener Log File /u01/app/oracle/diag/gsm/ORA-C2/sagsm/alert/log.xml
Listener Trace File /u01/app/oracle/diag/gsm/ORA-C2/sagsm/trace/ora_8238_140045476419008.trc
Endpoint summary (ADDRESS=(HOST=ORA-C2.localdomain)(PORT=1555)(PROTOCOL=tcp))
GSMOCI Version 0.1.11
Mastership Y
Connected to GDS catalog Y
Process Id 8240
Number of reconnections 2
Pending tasks. Total 0
Tasks in process. Total 0
Regional Mastership TRUE
Total messages published 7388
Time Zone +12:00
Orphaned Buddy Regions:
None
GDS region apac

GDSCTL>
[oracle@ORA-C2 trace]$ pwd
/u01/app/oracle/diag/gsm/ORA-C2/sagsm/trace
[oracle@ORA-C2 trace]$ ls -ltr al*
-rw-r—–. 1 oracle oinstall 1647026 Aug 1 03:07 alert_gsm.log
[oracle@ORA-C2 trace]$

In the above output we can see the information of GSM and also the listener log and trace files which show the base directory of all the GSM related diag files. Apart from the alert log we can also find the trace files if we enable any tracing or various other associated trace files.

31-AUG-2016 02:48:28 * (connect_data=(service_name=_Monitor)(operation=trace_level)(arg0=SUPPORT)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.90)(PORT=10748)) * establish * _Monitor * 0
Executing trace_level command
31-AUG-2016 02:48:29 * service_update * sagsm * 0
31-AUG-2016 02:48:47 * service_update * psfin%1 * 0
31-AUG-2016 02:48:49 * service_update * psfin%11 * 0
31-AUG-2016 02:48:59 * service_update * GCAT * 0
31-AUG-2016 02:49:14 * service_update * sagsm * 0
31-AUG-2016 02:49:17 * service_update * psfin%1 * 0
Executing status command
31-AUG-2016 02:49:19 * (connect_data=(service_name=_Monitor)(operation=status)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.90)(PORT=10756)) * establish * _Monitor * 0
31-AUG-2016 02:49:19 * service_update * sagsm * 0
31-AUG-2016 02:49:19 * (connect_data=(service_name=_Monitor)(operation=ocistat)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.90)(PORT=10757)) * establish * _Monitor * 0
Executing ocistat command
31-AUG-2016 02:49:19 * (connect_data=(service_name=_Monitor)(operation=opxystat)) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.90)(PORT=10758)) * establish * _Monitor * 0
31-AUG-2016 02:49:19 * service_update * sagsm * 0
Executing opxystat command
31-AUG-2016 02:49:19 * service_update * sagsm * 0
31-AUG-2016 02:49:19 * service_update * psfin%11 * 0
31-AUG-2016 02:49:41 * service_update * psfin%1 * 0
31-AUG-2016 02:49:44 * service_update * NCAT * 0
31-AUG-2016 02:49:47 * service_update * psfin%1 * 0
31-AUG-2016 02:49:49 * service_update * psfin%11 * 0
31-AUG-2016 02:50:14 * service_update * sagsm * 0
31-AUG-2016 02:50:17 * service_update * psfin%1 * 0
31-AUG-2016 02:50:19 * service_update * psfin%11 * 0
31-AUG-2016 02:50:32 * service_update * psfin%1 * 0
31-AUG-2016 02:50:41 * service_update * NCAT * 0
[oracle@ORA-C2 trace]$

Reviewing the Configuration of the GSM Components

Before heading to start troubleshooting any issue, I feel it is very mandatory to check the configuration first, Once the configuration part is clear then we can consider reviewing around the things. If in case there is an issue with configuration and we may end up spending hours of hours in troubleshooting where there is actual issue is with configuration, So there are various commands available to check the configuration of the components. When managing Global Data Services there can be more than one GSM, hence to check if any particular GSM is running or not (or) to check the other details related to GSM then we can check by querying like below.  

Status of GSM

[oracle@ORA-C2 ~]$ gdsctl gsmadm/oracle@ncat
GDSCTL: Version 12.1.0.2.0 – Production on Thu Sep 01 01:04:21 NZST 2016

Copyright (c) 2011, 2014, Oracle. All rights reserved.

Welcome to GDSCTL, type "help" for information.

GDSCTL>set gsm -gsm sagsm
GDSCTL>status gsm
Alias sagsm
Version 12.1.0.2.0
Start Date 28-JUL-2016 05:24:01
Trace Level off
Listener Log File /u01/app/oracle/diag/gsm/ORA-C2/sagsm/alert/log.xml
Listener Trace File /u01/app/oracle/diag/gsm/ORA-C2/sagsm/trace/ora_8238_140045476419008.trc
Endpoint summary (ADDRESS=(HOST=ORA-C2.localdomain)(PORT=1555)(PROTOCOL=tcp))
GSMOCI Version 0.1.11
Mastership Y
Connected to GDS catalog Y
Process Id 8240
Number of reconnections 2
Pending tasks. Total 0
Tasks in process. Total 0
Regional Mastership TRUE
Total messages published 6904
Time Zone +12:00
Orphaned Buddy Regions:
None
GDS region apac

Services

There can be few issues where the global services are not running as expected or the services are stopped working, in that case to check which service and where it is running the below command can be used which lists all the information related to the services of the configuration.

GDSCTL>status service
Service "app_engine.psfin.oradbcloud" has 1 instance(s). Affinity: ANYWHERE
Instance "psfin%1", name: "ORC1", db: "CANADA", region: "westcan", status: ready.
Service "nvision_report.psfin.oradbcloud" has 1 instance(s). Affinity: ANYWHERE
Instance "psfin%11", name: "ORC1", db: "INDIA", region: "apac", status: ready.

Databases

To check what are the databases are part of the configuration, to know the database role or under which region the specific database falls under, to know all these details we can query below command which helps to understand the architecture of the databases.

GDSCTL>status database
Database: "canada" Registered: Y State: Ok ONS: N. Role: PRIMARY Instances: 1 Region: westcan
Service: "app_engine" Globally started: Y Started: Y
Scan: N Enabled: Y Preferred: Y
Service: "nvision_report" Globally started: Y Started: N
Scan: N Enabled: Y Preferred: Y
Registered instances:
psfin%1
Database: "india" Registered: Y State: Ok ONS: N. Role: PH_STNDBY Instances: 1 Region: apac
Service: "nvision_report" Globally started: Y Started: Y
Scan: N Enabled: Y Preferred: Y
Registered instances:
psfin%11

Configuration of GSM

To review the entire configuration of GSM then there is single command which lists entire configuration like region names, pool, GSMs, Databases and services which are part of the configuration. This command will not describe in depth of each component instead we can say high level of configuration.  

GDSCTL>config
Catalog connection is established

Regions
————————
Name Buddy
—- —–
regionora
apac
westcan

GSMs
————————
sagsm

GDS pools
————————
dbpoolora
psfin

Databases
————————
india
canada

Services
————————
app_engine
nvision_report

GDSCTL pending requests
————————
Command Object Status
——- —— ——
SYNC DATABAE india N
SYNC DATABAE canada N

Global properties
————————
Name: oradbcloud
Master GSM: —-

Configuration of GSM

Above we have seen just the status of the GSM and also the overall review of the configuration, if we want to check all the GSMs configuration then the below command helps to view the end points and region information of each GSM.

GDSCTL>config gsm
Name Region ENDPOINT 
—- —— ——– 
sagsm apac (ADDRESS=(HOST=ORA-C2.localdomain)(PORT=1555)(PROTOCOL=tcp))

Configuration of Region

To know how many regions are configured and part of the configuration then we can use the config region command which displays information of only the regions.

GDSCTL>config region
Name Buddy
—- —–
regionora
apac
westcan

Know about GDS Pools

To check the all GDS pools configured in GDS configuration we can use “config gdspool” , it shows the pool names and also broker information if it is configured. In below example we have PSFIN GDS Pool and the related Data Guard is configured with Broker and hence it shows the status accordingly.

GDSCTL>config gdspool
Name Broker
—- ——
psfin Yes
dbpoolora No

Databases configuration

This command output is same as “database” but database command output shows which databases are running with which services and so on, this command lists output including the pool information and also region names.

GDSCTL>config database
Name Pool Status Region
—- —- —— ——
india psfin Ok apac
canada psfin Ok westcan

Configuration of Services

This command output is similar like config database, this command also provides the nature of service and also includes the network names and also it shows whether the services are running or not.

GDSCTL>config service
Name Network name Pool Started Preferred all
—- ——- —- —- ——- ——— —
nvision_report nvision_report.psfin.oradbcloupsfin Yes Yes
d
app_engine app_engine.psfin.oradbcloud psfin Yes No

GDSCTL>

Tracing at GSM level

So far we have performed basic level of troubleshooting and also 360 degrees review of the configuration, let’s consider we are unable to fix by the above troubleshooting and if we are looking for more details then certainly tracing helps a lot. To enable tracing is very simple, we need to select which GSM we want to enable tracing and also the level of supporting and the tracing can be disabled at any time.

GDSCTL>set trace_level -gsm sagsm SUPPORT
trace_level is set to SUPPORT
GDSCTL>
GDSCTL>status gsm
Alias sagsm
Version 12.1.0.2.0
Start Date 28-JUL-2016 05:24:01
Trace Level support
Listener Log File /u01/app/oracle/diag/gsm/ORA-C2/sagsm/alert/log.xml
Listener Trace File /u01/app/oracle/diag/gsm/ORA-C2/sagsm/trace/ora_8238_140045476419008.trc
Endpoint summary (ADDRESS=(HOST=ORA-C2.localdomain)(PORT=1555)(PROTOCOL=tcp))
GSMOCI Version 0.1.11
Mastership Y
Connected to GDS catalog Y
Process Id 8240
Number of reconnections 2
Pending tasks. Total 0
Tasks in process. Total 0
Regional Mastership TRUE
Total messages published 7324
Time Zone +12:00
Orphaned Buddy Regions:
None
GDS region apac

GDSCTL>exit
[oracle@ORA-C2 ~]$ cd /u01/app/oracle/diag/gsm/ORA-C2/sagsm/trace
[oracle@ORA-C2 trace]$ 
[oracle@ORA-C2 trace]$ ls -ltr
total 2416
-rw-r—–. 1 oracle oinstall 13112 Sep 1 02:49 ora_8240_140103001331456.trm
-rw-r—–. 1 oracle oinstall 66989 Sep 1 02:49 ora_8240_140103001331456.trc
-rw-r—–. 1 oracle oinstall 1532 Sep 1 02:49 ora_8264_139754960172480.trm
-rw-r—–. 1 oracle oinstall 6250 Sep 1 02:49 ora_8264_139754960172480.trc
-rw-r—–. 1 oracle oinstall 1508 Sep 1 02:49 ora_8240_140103013770688.trm
-rw-r—–. 1 oracle oinstall 5330 Sep 1 02:49 ora_8240_140103013770688.trc
-rw-r—–. 1 oracle oinstall 5107 Sep 1 02:49 ora_8258_140328191546816.trm
-rw-r—–. 1 oracle oinstall 23796 Sep 1 02:49 ora_8258_140328191546816.trc
-rw-r—–. 1 oracle oinstall 1640305 Sep 1 02:49 alert_gsm.log
-rw-r—–. 1 oracle oinstall 62167 Sep 1 02:49 ora_8250_140488163562944.trm
-rw-r—–. 1 oracle oinstall 296391 Sep 1 02:49 ora_8250_140488163562944.trc
-rw-r—–. 1 oracle oinstall 15590 Sep 1 02:49 ora_8256_140045087200704.trm
-rw-r—–. 1 oracle oinstall 67657 Sep 1 02:49 ora_8256_140045087200704.trc
-rw-r—–. 1 oracle oinstall 43720 Sep 1 02:49 ora_8238_140045476419008.trm
-rw-r—–. 1 oracle oinstall 181610 Sep 1 02:49 ora_8238_140045476419008.trc
[oracle@ORA-C2 trace]$ date
Thu Sep 1 02:49:58 NZST 2016
[oracle@ORA-C2 trace]$

Disabling Tracing

GDSCTL>set trace_level -gsm sagsm off

trace_level is set to OFF

GDSCTL>

Conclusion

We’ve seen how to perform overall review of configuration which falls under troubleshooting and also what are the sources we have to review in case of any issues with GSM.