How to Deal with Cutover Phase in R12.2 (ADOP)
Best Backup Practices while using Online Patching(adop) in R12.2
Rollback Patch even after cutover phase in Oracle Apps R12.2 using Oracle Database Flashback
If you are applying patch in Oracle Apps R12.2 using ADOP and your development team comes and request for the patch rollback. You will simply going to run adop phase=abort and your patch will be rollback and development team will be happy for having such a nice and wonderful DBA. But this will be easy only if you haven’t run the cutover phase yet. Suppose you are in middle of cutover phase or you have already executed the cutover phase then you will be left high and dry in such a situation to rollback the patch and will be very angry or upset on your developers. If you have taken application as well as database backup before applying the patch then only you will be able to rollback the patch by restoring to the latest backup.
One of the best feature of Oracle EBS R12.2 is ADOP tool. And great thing about ADOP is the ability to rollback the patch and changes created by the patch, but this roll back will be available till you run the cutover phase. Once the cutover is executed, the changes being made by apply phase are committed, and cannot be rolled back. This constraint brings requirement of taking backup each time patch being applied. Taking backup each time before applying the patch will require lot of time whether it is hot backup or cold backup.
The problem here is not actually the method of the backup. It is the backup operation itself that creates the cost. In other words, taking a backup before applying every online patch in EBS 12.2 is not so applicable, as it costs so much that it complicates the patching operation and it increases the backup maintenance operations; therefore, it even reduces the benefits of using online patching.
So here we are with different approach of backup and guaranteed roll back to the state of system before applying the patch. We will take a use of Oracle Database Flashback feature in this method and restarting the application services from the relevant file system edition depending on whether the decision to roll back the system is done before the file systems were switched.
Discuss a scenario in which we actually need to roll back the system after the cutover phase has been run. After the finalize phase you execute adop phase=cutover, but cutover fails. Suppose you try to fix the problem, causing cutover to fail (as Oracle suggests so) but could not fix it. The only option left is go back to a state before you cutover phase and roll back the changes done in the database and application tiers. To do so we need to have a backup that can restored and used for recovering the database just before running the cutover phase. However, taking an online backup in the cutover phase extends the time needed for patching,and therefore it is not applicable for every patch. If patch size is small (suppose 10KB), to guarantee a roll back in case of failure we need to backup our EBS environment which large is size (generally in GB). So backing up the environment before cutover would increase the patching window time even to apply small size patch. And also would cost us side effects such as heavy I/O created by backup jobs. So taking full backup before applying a small patch is not a great idea and similarly restoring the full backup will be time consuming process, and it will also not a great idea to used in case of patching failure.
So, Let’s go back where we have started and illustrate this scenario again, but this time we will use the flashback option and a couple of application tier operations to restore your EBS 12.2 system to the state it was in before the online patching cutover phase was run
Backing Up:
In this section we will learn how to backup the EBS before cutover.
Step 1:Enable Flashback
Enable flashback and configure it with four settings accordingly, before starting the patching cycle, as follows:
alter system set db_recovery_file_dest='/some_path/' scope=BOTH SID='*';
db_recovery_file_dest is set to a directory where the flashback logs are placed. It can set to an ASM disk group, as well.
alter system set db_recovert_file_dest_size=100G scope=BOTH SID='*';
db_recover_file_dest_size is where flashback is placed; the size of the db_recovery_file_dest should be set according to our needs. It actually depends on how long we want to store the flashback logs. Also in addition to the flashback logs if we place our rman backups and online redo logs in the database recovery file destination, then we also need to consider them for setting the db_recovery_file_dest_size.
alter system set db_flashback_retention_target=120;
db_flashback_retention_target is set to a value which actually represents the upper limit in minutes on how far back in time the database may be flashed back.
alter databse flashback on;
By executing “alter database flashback on”, flashback in the Database gets activated.
Step 2: Create a Flashback Restore Point
The next thing to do is to create a flashback restore point, this restore point may be created just before starting the patching cycle or preferably just before the cutover phase.
NOTE:
1. Stop the web services.
2. Stop the concurrent managers or put the schedule concurrent requests on hold.
3. Stop database-bound third-party applications or cut off their database connections.
4. Force a log switch.
5. Create a guaranteed restore point.
6. Force another log switch.
Guaranteed restore point assures that the database will maintain enough flashback logs to flash back the database to the restore point and by shutting down the web services and putting the critical concurrent requests on hold, you can ensure that you won’t lose any important transactional data. Also, if there are any other database applications that reach the EBS database via database links or any other non-EBS products using the EBS database (actually there should not be), then they should be stopped or their EBS-related activities should prevented by their application owners.
It is advisable to create the restore point just before the cutover phase, because as per the action plan, there are some requirements to stop the application services, which will increase the downtime for the patching operation. So, it is better to create the restore point as late as possible.
After reviewing and ensuring the recommendations are in place, you continue with creating the patching cycle and completing phases one by one until you reach the cutover phase and creating the restore point as follows. To start patching, you create the patching cycle using the adop phase=prepare command and complete the phases one by one as shown here, until the cutover phase:
adop phase=fs_clone
adop phase=prepare
adop phase=apply
adop phase=finalize
Just before the cutover phase, you create the restore point as follows:
alter system switch logfile;
CREATE RESTORE POINT BEFORE_PATCH GUARANTEE FLASHBACK DATABASE;
alter system switch logfile;
Once the restore point is created, you have a database backup that was taken exactly at the time you wanted and can be restored if needed. At this point, you have a backup; thus, you can continue with the critical cutover phase.
adop phase=cutover
Recovering
Now suppose you face an unsolvable error during the cutover phase and decide to roll your EBS 12.2 environment back to the state before the patching cycle was started.
Step 1: Restore Database
This time, rather than a costly standard database and application file system restore, you can use flashback to restore the database to the restore point named BEFORE_PATCH, which was created just before the cutover phase. To accomplish this, you follow this action plan:
- Shut down the database.
SQL>shutdown immediate
2. Start the database in mount mode.SQL>startup mount
3. Restore the database to the restore point BEFORE_PATCH using the flashback feature.
SQL>flashback database to restore point BEFORE_CUTOVER;
4. Start the database in read-only mode.SQL>alter database open read only
5. Check the database with simple queries and ensure that there are no abnormalities.
6. Shut down the database and open it using the resetlogs option.
SQL>shutdown immediate;
SQL>startup mount;
SQL>alter database open resetlogs;
By doing the previous actions, you roll back your database to a state before the cutover phase. After rolling back your database using flashback, you should disable the flashback in the database and leave it to be disabled until the next patching operation. By doing so, you eliminate the risk of running out of space in FRA, which may be caused by retaining the flashback logs. Note that running out of space will cause the EBS database to hang. The following action plan can be used to accomplish that:
1. Disable flashback.SQL>alter database flashback off;
2. Drop the restore point.SQL>drop restore point BEFORE_CUTOVER;
3. Set recovery file destination.SQL>alter system set db_recovery_file_dest=’’;
Step 2: Restore Application File System
So far, so good. We have successfully roll back to restore point in EBS 12.2 database using the flashback option in case of a failure that may be encountered during the cutover phase. But what about the application tier file systems? To fully roll back an EBS 12.2 system in the case of a cutover failure, you need to roll back the file systems of the application nodes, too.
As patching activities change the database objects in the database, they change the file objects in the application file system. Until the cutover phase, all the file object changes are done in the patch file system; likewise, all the database objects changes are done in the patch edition. So, any time before the cutover phase, you can use adop phase=abort and roll back those changes. However, once you start the cutover phase, abort will no longer work, and that’s why you need to do some special actions to roll back the changes done in the cutover phase.
For the database tier, it is a little complicated. An operation like rolling back the changes that occurred before the cutover employs using the flashback option, restore points, and so on. However, it is pretty simple for the application tier.
Suppose you execute the cutover phase during a patching activity and encounter some errors. So, you decide to roll back your EBS system to a state before the cutover phase. To roll back the changes for the database tier, suppose you use the flashback restore point that you created earlier.
Just after the database is restored, you continue with rolling back the application tier.
To do that, you need to know whether the cutover failed before the file systems were switched. You can identify this using the cutover logs ($NE_BASE/EBSapps/log/adop/<current_session_id>/cutover_<timestamp>/ for your session ID).
If the cutover failed before the file systems were switched, then you are lucky, as you just need to shut down all the application services and then start all the application services using the run edition environment like in your standard startup procedure.
If the log file indicates that cutover failed after the file systems were switched, then you have to follow a different action plan. That is, if the file systems were switched, then you need to switch back the file systems in order to roll back your application tier to its state before the cutover phase. What you need to do is as follows:
1. Source the environment on the current run file system.
2. Shut down all the application services using the standard procedure (adstpall.sh).
3. Switch back the file system using txkADOPCutOverPhaseCtrlScript.pl.
Usage :
perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl \
-action=ctxupdate \
-contextfile=<full path to new run context file> \
-patchcontextfile=<full path to new patch file system context file> \
-outdir=<full path to out directory>4. Start up all services from the new run file system using the standard procedure.(Source run edition environment and use the adstrtal.sh script to start all the application services.)
Once you flash back the database and restart the application services from the file system that was your run file system before the last cutover phase, your EBS 12.2 system is rolled back to the state before the cutover phase. At this point, you can decide whether to continue or abort the patching cycle. If you fix the issues causing cutover to fail, you may decide to continue the patching. In that case, you can just execute the
adop phase=cutovercommand. However, if you decide not to continue, you can useadop phase=abortand abort the patching cycle.
References:
- Practical Oracle E-Business Suite An Implementation and Management Guide by Syed Zaheer and Erman Arslan
