Monday, July 28, 2014

ORA-16693: requirements not met for enabling fast-start failover

1.Fast star failover mode is not enabled, even the logxptmode of both the standby and primary database ins async as per the protection mode requirement of maxprotection mode.
DGMGRL>  enable fast_start failover
Error: ORA-16693: requirements not met for enabling fast-start failover
Failed.
DGMGRL> show database orcl_us logxptmode
  LogXptMode = 'ASYNC'
DGMGRL> show database orcl_in logxptmode
  LogXptMode = 'ASYNC'
DGMGRL> show database 'orcl_us'

Database - orcl_us

  Role:              PHYSICAL STANDBY
  Intended State:    APPLY-ON
  Transport Lag:     0 seconds (computed 1 second ago)
  Apply Lag:         0 seconds (computed 1 second ago)
  Apply Rate:        83.00 KByte/s
  Real Time Query:   ON
  Instance(s):
    orcl2

Database Status:
SUCCESS

2.The dataguard logs message shows the flashback mode is not enabled in target standby
version check on database orcl_us detected stale metadata,
      requesting update from primary database
Creating process RSM0
07/28/2014 12:42:29
ORA-16693, FSFO requires the target standby to be in flashback mode

3.Enabled the flashback mode in standby database
DGMGRL> connect sys/sys123@orcl_us
Connected as SYSDBA.

DGMGRL> edit database 'orcl_us' set state='apply-off';
Succeeded.
DGMGRL>  sql "alter database flashback on";
Succeeded.
DGMGRL> enable fast_start failover
Enabled.
DGMGRL> edit database 'orcl_us' set state='apply-on';
Succeeded.
DGMGRL> show configuration

Configuration - broker1

  Protection Mode: MaxPerformance
  Databases:
  orcl_in - Primary database
    Warning: ORA-16819: fast-start failover observer not started

    orcl_us - (*) Physical standby database
      Warning: ORA-16819: fast-start failover observer not started

Fast-Start Failover: ENABLED

Configuration Status:
WARNING

4. Fast start failover is enabled, with a warning message of observer process is not started.

Sunday, July 20, 2014

ORA-16647: could not start more than one observer

1. Error while starting the fsfo observer in new node.
[W000 07/20 22:41:58.95] Failed to start the Observer.
Error: ORA-16647: could not start more than one observer

2.Stop the observer and start the observer in the new node
DGMGRL> show fast_start failover

Fast-Start Failover: ENABLED

  Threshold:          30 seconds
  Target:             orcl_uk
  Observer:           ol6-112-rac1.localdomain
  Lag Limit:          20 seconds
  Shutdown Primary:   TRUE
  Auto-reinstate:     TRUE
  Observer Reconnect: (none)
  Observer Override:  FALSE

Configurable Failover Conditions
  Health Conditions:
    Corrupted Controlfile          YES
    Corrupted Dictionary           YES
    Inaccessible Logfile            NO
    Stuck Archiver                  NO
    Datafile Offline               YES

  Oracle Error Conditions:
    (none)

DGMGRL> stop observer
Done.
DGMGRL> show fast_start failover

Fast-Start Failover: ENABLED

  Threshold:          30 seconds
  Target:             orcl_uk
  Observer:           (none)
  Lag Limit:          20 seconds
  Shutdown Primary:   TRUE
  Auto-reinstate:     TRUE
  Observer Reconnect: (none)
  Observer Override:  FALSE

Configurable Failover Conditions
  Health Conditions:
    Corrupted Controlfile          YES
    Corrupted Dictionary           YES
    Inaccessible Logfile            NO
    Stuck Archiver                  NO
    Datafile Offline               YES

  Oracle Error Conditions:
    (none)

3.Start the observer process
]$ dgmgrl -logfile '/u03/app/oracle/diag/rdbms/orcl_uk/orcl2/trace/observer.log'  'sys/sys123@orcl_uk' 'start observer' &
[1] 12870
[oracle@ol6-112-rac2 trace]$ DGMGRL for Linux: Version 12.1.0.1.0 - 64bit Production

Copyright (c) 2000, 2012, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDBA.

ORA-16693: requirements not met for enabling fast-start failover

1.Error while enabling the fast start failover
 dgmgrl -echo 'sys/sys123@orcl_uk'
DGMGRL for Linux: Version 12.1.0.1.0 - 64bit Production

Copyright (c) 2000, 2012, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDBA.
DGMGRL> show configuration
show configuration

Configuration - broker1

  Protection Mode: MaxPerformance
  Databases:
  orcl_in - Primary database
    orcl_uk - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> enable fast_start failover
enable fast_start failover
Error: ORA-16693: requirements not met for enabling fast-start failover


2.Checked the dataguard logfile for any more detail on the error message
/u03/app/oracle/diag/rdbms/orcl_in/orcl1/trace
[oracle@ol6-112-rac1 trace]$ tail -100f drcorcl1.log
.............
ENABLE FAST_START FAILOVER
MaxPerformance FSFO requires LogXptMode=ASYNC for primary and target standby databases, and no Far Sync is allowed between primary andtarget standby databases.
..................

3.Checked and changed the logxptmode from the sync to async, enabled the fsfo
dgmgrl sys/sys123@orcl_in
DGMGRL for Linux: Version 12.1.0.1.0 - 64bit Production

Copyright (c) 2000, 2012, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected as SYSDBA.
DGMGRL> show database 'orcl_in' logxptmode
  LogXptMode = 'sync'
DGMGRL> show database 'orcl_uk' logxptmode
  LogXptMode = 'sync'
DGMGRL> edit database 'orcl_uk' set property logxptmode='async';
Property "logxptmode" updated
DGMGRL> edit database 'orcl_in' set property logxptmode='async';
Property "logxptmode" updated
DGMGRL> enable fast_start failover
Enabled.


Monday, July 14, 2014

DGMGRL Switchover and ORA-12514: TNS:listener does not currently know of service requested in connect descriptor


1.dgmgrl switchover fails with "ORA-12514: TNS:listener does not currently know of service requested in connect descriptor"
DGMGRL> switchover to 'orcl_uk';
Performing switchover NOW, please wait...
Operation requires shutdown of instance "orcl2" on database "orcl_in"
Shutting down instance "orcl2"...
Database closed.
Database dismounted.
ORACLE instance shut down.
New primary database "orcl_uk" is opening...
Operation requires shutdown of instance "orcl1" on database "orcl_in"
Shutting down instance "orcl1"...
ORACLE instance shut down.
Operation requires startup of instance "orcl1" on database "orcl_in"
Starting instance "orcl1"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:
        start up and mount instance "orcl1" of database "orcl_in"

DGMGRL>


2.Alter log file of the new primary database recorded with the below error message "Heartbeat failed to connect to standby 'orcl'. Error is 1034."
orcl_uk aler log:
===============
Error Message
=============
Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ol6-112-rac1-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=ol6-112-rac2-vip)(PORT=1521))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=orcl_in_DGB)(CID=(PROGRAM=oracle)(HOST=ol6-112-rac2.localdomain)(USER=oracle))))

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.3.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
  Time: 12-JUL-2014 20:57:45
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ol6-112-rac1-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=ol6-112-rac2-vip)(PORT=1521))(LOAD_BALANCE=ON)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=orcl_in_DGB)(CID=(PROGRAM=oracle)(HOST=ol6-112-rac2.localdomain)(USER=oracle))))

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.3.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
  Time: 12-JUL-2014 20:57:45
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
Sat Jul 12 20:57:47 2014
Error 1034 received logging on to the standby
PING[ARC2]: Heartbeat failed to connect to standby 'orcl'. Error is 1034.

3.Check the old primary database listener status and identified the service name for the old primary database is not in proper format (db_unique_name)_DGMGRL. It should be ORCL_IN_DGMGRL instead of ORCL_INK_DGMGRL
istener status for orcl
[oracle@ol6-112-rac1 admin]$ lsnrctl status listener

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 12-JUL-2014 21:54:35

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     listener
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                11-JUL-2014 22:54:28
Uptime                    0 days 23 hr. 0 min. 21 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid/app/11.2.0.3/grid_1/network/admin/listener.ora
Listener Log File         /u01/app/11.2.0/grid_base/diag/tnslsnr/ol6-112-rac1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.111)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.113)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "ORCL_IN" has 2 instance(s).
  Instance "orcl1", status UNKNOWN, has 1 handler(s) for this service...
  Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "ORCL_INK_DGMGRL" has 1 instance(s).
  Instance "orcl1", status UNKNOWN, has 1 handler(s) for this service...
Service "ORCL_IN_DGB" has 1 instance(s).
  Instance "orcl1", status READY, has 1 handler(s) for this service...


4.Change the service_name in the listener.ora file and reload the listener and check the status of listener again
[grid@ol6-112-rac1 admin]$ lsnrctl status listener

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 12-JUL-2014 22:07:24

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     listener
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                11-JUL-2014 22:54:28
Uptime                    0 days 23 hr. 13 min. 5 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid/app/11.2.0.3/grid_1/network/admin/listener.ora
Listener Log File         /u01/app/11.2.0/grid_base/diag/tnslsnr/ol6-112-rac1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
Services Summary...
Service "ORCL_IN" has 1 instance(s).
  Instance "orcl1", status UNKNOWN, has 1 handler(s) for this service...
Service "ORCL_IN_DGMGRL" has 1 instance(s).
  Instance "orcl1", status UNKNOWN, has 1 handler(s) for this 

5.Switchback the database to original primary and switchover to standby once again to check the switchover it without any issues
[oracle@ol6-112-rac1 admin]$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/sys123@crman1
Connected.
DGMGRL>
DGMGRL>
DGMGRL>
DGMGRL> switchover to "orcl_uk";
Performing switchover NOW, please wait...
Operation requires shutdown of instance "orcl2" on database "orcl_in"
Shutting down instance "orcl2"...
Database closed.
Database dismounted.
ORACLE instance shut down.
New primary database "orcl_uk" is opening...
Operation requires shutdown of instance "orcl1" on database "orcl_in"
Shutting down instance "orcl1"...
ORACLE instance shut down.
Operation requires startup of instance "orcl1" on database "orcl_in"
Starting instance "orcl1"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "orcl_uk"

Sunday, July 13, 2014

ORA-00245: control file backup failed; target is likely on a local file system

1.RMAN error message "ORA-00245: control file backup failed; target is likely on a local file system"
[oracle@ol6-112-rac1 admin]$ rman catalog rman/rman123@crman  target sys/sys123@orcl1

Recovery Manager: Release 11.2.0.3.0 - Production on Sun Jul 13 14:38:22 2014

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1374849049, not open)
connected to recovery catalog database

RMAN> resync catalog from db_unique_name orcl_in;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync from db_unique_name command on default channel at 07/13/2014 14:39:09
ORA-00245: control file backup failed; target is likely on a local file system

RMAN> show snapshot controlfile name;

RMAN configuration parameters for database with db_unique_name ORCL_IN are:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/orcl/oracle/product/11.2.0.3/db_1/dbs/snapcf_orcl1.f'; # default


2.Configuring snapshot controlfile name to ASM storage
RMAN> configure snapshot controlfile name to '+DATA/orcl/snapcf_orcl.f';

new RMAN configuration parameters:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DATA/orcl/snapcf_orcl.f';
new RMAN configuration parameters are successfully stored

RMAN> show snapshot controlfile name ;

RMAN configuration parameters for database with db_unique_name ORCL_IN are:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+DATA/orcl/snapcf_orcl.f';

RMAN> resync catalog from db_unique_name 'orcl_in';

Tuesday, July 1, 2014

ORA-16737 :the redo transport service for standby database

1.Error with log shipping to remote standby destination
DGMGRL> show database verbose 'dbrac2_new' 'statusreport';
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT
       new12cdb_rac2      ERROR ORA-16737: the redo transport service for standby database "dbrac1" has an error
Alert log of primary database:-
================================
Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ol6-112-rac1)(PORT=1524))(CONNECT_DATA=(SERVER=DEDICATED)(Service_NAME=DBRAC1_DGMGRL)(CID=(PROGRAM=oracle)(HOST=ol6-112-rac2.localdomain)(USER=oracle))))

  VERSION INFORMATION:
        TNS for Linux: Version 12.1.0.1.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.1.0 - Production
  Time: 02-JUL-2014 11:56:56
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
Wed Jul 02 11:56:56 2014


2. Check the static listener configuration in the standby side.
LISTEN_FOR_NEW12CDB=
(DESCRIPTION_LIST =
        (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.111)(PORT = 1524))
        )
)
SID_LIST_LISTEN_FOR_NEW12CDB=
(SID_LIST=
        (SID_DESC=
        (global_dbname=DBRAC1_DGB)
        (ORACLE_HOME=/u03/app/oracle/product/12.1.0/dbhome_1)
        (SID_NAME=new12cdb)
        )
 )
 

3.There is a mismatch with global_dbname entry in the listener.ora of the standby database with that of the service name it is trying to ship log to the standby database. changing the global_dbname to dbrac1_dgmgrl in the listener.ora file in the standby side
[oracle@ol6-112-rac1 admin]$ lsnrctl reload LISTEN_FOR_NEW12CDB\
> ;

LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 02-JUL-2014 12:04:40

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.111)(PORT=1524)                                                                                       ))
The command completed successfully
[oracle@ol6-112-rac1 admin]$ cat listener.ora
LISTEN_FOR_NEW12CDB=
(DESCRIPTION_LIST =
        (DESCRIPTION =
        (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.111)(PORT = 1524))
        )
)
SID_LIST_LISTEN_FOR_NEW12CDB=
(SID_LIST=
        (SID_DESC=
        (global_dbname=DBRAC1_DGMGRL)
        (ORACLE_HOME=/u03/app/oracle/product/12.1.0/dbhome_1)
        (SID_NAME=new12cdb)
        )
 )

[oracle@ol6-112-rac1 admin]$ lsnrctl status LISTEN_FOR_NEW12CDB

LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 02-JUL-2014 12:05:00

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.111)(PORT=1524)))
STATUS of the LISTENER
------------------------
Alias                     LISTEN_FOR_NEW12CDB
Version                   TNSLSNR for Linux: Version 12.1.0.1.0 - Production
Start Date                01-JUL-2014 12:51:31
Uptime                    0 days 23 hr. 13 min. 28 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u03/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u03/app/oracle/diag/tnslsnr/ol6-112-rac1/listen_for_new12cdb/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.111)(PORT=1524)))
Services Summary...
Service "DBRAC1_DGMGRL" has 1 instance(s).
  Instance "new12cdb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@ol6-112-rac1 admin]$

4.Check the statusreport of the primary database for the errors
DGMGRL> show database verbose 'dbrac2_new' 'statusreport';
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT
DGMGRL> show database dbrac1

Database - dbrac1

  Role:              PHYSICAL STANDBY
  Intended State:    APPLY-ON
  Transport Lag:     0 seconds (computed 1 second ago)
  Apply Lag:         0 seconds (computed 1 second ago)
  Apply Rate:        146.00 KByte/s
  Real Time Query:   ON
  Instance(s):
    new12cdb

Database Status:
SUCCESS


Monday, June 30, 2014

ORA-16627: operation disallowed since no standby databases would remain to support protection mode

1.Change the dataguard configuration protection mode fails with ora-16627
DGMGRL> edit configuration
> set protection mode as
> maxavailability;
Error: ORA-16627: operation disallowed since no standby databases would remain to support protection mode

2.check the standby database logxptmode
DGMGRL> show database dbrac2 'LogXptMode';
  LogXptMode = 'ASYNC'

3.Change the log transport mode to sync from async
DGMGRL> edit database dbrac2
> set property 'LogXptMode'='sync';
Property "LogXptMode" updated

4.Change the protection mode from maxperformance to maxavailability
DGMGRL> edit configuration
> set protection mode as maxavailability;
Succeeded.
DGMGRL> show configuration

Configuration - dgmgrl1

  Protection Mode: MaxAvailability
  Databases:
  dbrac1 - Primary database
    dbrac2 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS