ADOP : Online Patching Oracle EBS R12.2

Before Oracle Application R12.2 we need to shutdown application tier to apply patch on production environment and downtime depends on multiple factors, scheduled lengthy downtime window were major problem from more than last two decades in single instance environment in R12.1 and earlier versions of Oracle Apps. Oracle R12.2 finally comes with solution to this by introducing ADOP (Application DBA Online Patching) tool where users can be still connected to system while patch being applied to the system. And very small downtime independents of all factors unlike in R12.1 and older version. Users gets disconnected only during cutover phase. Here in this blog we will discuss about adop.

There are five important stages in ADOP to apply patch on the system. Listed as per below:

1. Prepare File System
2. Apply Patch
3. Finalize
4. Cutover
5. Cleanup

In Brief:

adop phase=prepare : A copy is made from the running file system which resides in both file system as well as database.

adop phase=apply patches=<patch_number> &

adop phase=finalize

Patches are applied to the copied file system while users continue to access the running system.

adop phase=cutover : The copied file system becomes the new running system

adop phase=cleanup : The original running system (now obsolete) is recycled for use in future patching cycles.

Benefits of ADOP:

1. Patching downtime time can now be measured in terms of minutes rather than hours, only being limited by the time it takes to bounce the Application processes in the Middle Tier.
2. All patches and maintenance packs are applied online, requiring only a short cutover period.
3. Maintenance windows can now be better planned
4. Only a minimal patching disruption time is required

File System in R12.2:

Oracle Application R12.2 comes with three different types of file system.

1. Run File System: Systems where users are connected and online. The Run Edition includes a complete application-tier file system along with all objects and data visible in the default edition of the database. Assume fs1 as current run file system in below example.
2. Patch File System: Systems where patches are applied. Users don’t have access to this file system. Assume fs2 as current patch file system.
3. Non-Editioned File System (fs_ne): File system which remains consistent during all the patching cycle and contain details regarding patches applied on systems.

All three file systems forms Oracle Application R12.2 and connects to Oracle Database as described in below diagram.

fs1 and fs2 are interchanged after cutover phase, as shown in below picture.

Connecting to the Run Edition and Patch Edition :

Sourcing the APPS environment in R12.2 , is totally different than earlier version of Oracle Apps. The below picture shows how to source environment in Apps R12.2 and connect to run file edition.

End users can only connects to run file systems but developers may also want to connect the patch file system. Hence we have also covered connecting to patch file edition, as shown in below picture.

Warning: It is only safe to connect to the patch edition while an Online Patching session is in progress. Specifically, the Patch Edition is active after the “adop phase=prepare” operation, and persists until the cutover or abort operation is run.

The application-tier Patch Edition environment is configured to connect to the database patch edition by default. If a database patch edition is not active, then attempting to connect to the database from the application-tier patch edition environment will fail.

As shown in above picture the highlighted red boxes indicates that we can also check the current file edition system in which we are working.

Edition-Based Redefinition (EBR):

Traditionally, applications such as Oracle E-Business Suite have had to be unavailable while their associated database objects are patched or upgraded. The Edition-Based Redefinition (EBR) feature of the Oracle 11gR2 Database allows this downtime to be avoided.

Every database has at least one edition, and can support multiple editions. A new edition is created as a child of an existing edition. The changes are made in the child edition, while the parent edition remains available for use. Changes to data (such as when a patch is applied) are made by writing only to new columns or new tables not seen by the old edition.

EBR allows an application’s database objects to be changed without interrupting the application’s availability. For example, an Oracle E-Business Suite patch can be applied while users are logged in and active. This is accomplished by making the changes in the privacy of a new edition.

Key objects used by EBR include:

1. Editioning views expose a different projection of a changed table to each edition, allowing each to see only its own columns.
2. Cross-edition triggers propagate data changes made by the old edition into the columns of the new edition, which subsequently replaces the old edition

More number of active database editions also causes the problem as it increases the inheritance hence more time required to fetch the objects from older editions and cause the slower database performance.

For more information on the Edition-Based Redefinition feature introduced on Oracle Database 11gR2 see the following:

Edition-Based Redefinition of the Oracle Database Advanced Application Developer’s Guide 11g Release 2 (11.2) — E41502

Prepare

Before applying a patch, you must start an Online Patching Cycle. This is done using the adop “prepare” command. Connect to the primary application-tier node of your target system and source the run edition environment. Then execute the prepare command.

adop phase=prepare

The adop utility may first execute the cleanup phase from the previous cycle if needed, and will then proceed to prepare the patch edition for a new Online Patching Cycle. To prepare the patch edition, adop will:

1. Validate that the system is ready to start a new patching cycle.
2. Create a new database patch edition
3. Synchronize the file system patch edition with the run edition
4. Configure the patch edition for use by the patching tools

File system synchronization may be done by applying the delta (changes) from the previous patching cycle, or by re-creating the entire patch edition file system as a fresh copy of the run edition (called “fs_clone”). When complete, check the exiting status code (success is ‘0’):

If there were any problems with the prepare phase, check Troubleshooting and resolve the problem. Then run the prepare command again.

After a successful prepare phase, the database and file system patch edition will contain a copy of the run edition code and seed data. You can now apply patches to the patch edition.

Apply

Once the Patch Edition is prepared, you can apply any number of patches to the patch edition. Changes to the patch edition are isolated from the run edition, which is still available for use.

Before applying an patch, you must first download the patch bundle from the web user interface (support.oracle.com). The downloads will be in the form of ZIP files. Place the ZIP files in the “$PATCH_TOP” directory on the application-tier installation of your target application system, and then unzip all ZIP files.

Patches are applied to the patch edition using the “adop phase=apply” command. The command accepts a “patches=…” parameter where you can specify a single patch or a comma-separated list of patches.

adop phase=apply patches=16605855 … adop phase=apply patches=15111111,15222222 …

Note that the adop command will apply patches to the patch edition no matter what edition your current environment is set to.

If the adop apply commands fail, check Troubleshooting and correct the problem, then run the adop apply command again, adding the “restart=yes” option.

adop phase=apply patches=16605855 restart=yes

In some cases, an error when applying a patch can be corrected by applying a replacement patch. You can abandon an existing failed patch and apply the replacement patch by running the apply command with the “abandon=yes” parameter.

adop phase=apply patches=16699999 abandon=yes

Finalize

The finalize phase is used by the Online Patching tool to perform any final actions needed to make the system ready for the fastest possible cutover. The finalize command is run as follows:

adop phase=finalize

If the finalize command returns an error, the system is not ready for cutover. In this case, check Troubleshooting correct the problem and run the finalize command again.

After successful completion of the finalize phase, the system is ready for cutover, but you do not need to execute the cutover right away; you can delay executing cutover until a convenient or predetermined time in the future. You may also apply additional patches if needed, but you will need to run the finalize phase again after doing so.

Cutover

The cutover phase will configure the patch edition to become the new run edition, and restart the application on this new run edition.

adop phase=cutover

After successful completion of the cutover phase, the application will be up and running on the new edition, ready for use. Since the run/patch designation of the dual file systems are swapped during cutover, you must re-source the run edition environment directly after cutover.

Important:

Remember to re-source the run edition environment directly after cutover.

If cutover fails there is no option to redirect back except restoration.

You can also check-out my below blog post for best practices while patching with ADOP

Best Backup Practices while using Online Patching(adop) in R12.2

Cleanup

The cleanup phase will remove unnecessary code and data from old editions that are no longer needed by the application. Cleanup should be run after cutover, at any time before the next prepare phase. It is best to run cleanup immediately after cutover so that there is no delay when preparing the next online patching cycle. There are three levels of cleanup available:

• quick — the minimal cleanup required before starting the next patching cycle.
• standard — (default) removes obsolete code and seed data from old editions.
• full — removes all obsolete code, data, and table columns.

Standard cleanup is the default, and is recommended to avoid a buildup of obsolete code objects in old editions.

adop phase=cleanup

Use quick cleanup when you need to start the next patching cycle as soon as possible.

adop phase=cleanup cleanup_mode=quick

Use full cleanup to remove obsolete table columns. Full cleanup is required after aborting an online patching cycle.

adop phase=cleanup cleanup_mode=full

Special Patching Action

There are some more useful patching action in adop, which are also listed here.

fs_clone

The “fs_clone” command recreates the patch edition file system by making a full copy of the run edition file system. This command is required when the patch edition file system cannot be automatically synchronized with the run edition file system by adop. Run the fs_clone command if you have done any of the following actions:

• Applied middle-tier technology patches to the run edition
• Changed run edition files manually (outside of online patching)
• Aborted an online patching cycle

The fs_clone command is executed as follows:

adop phase=fs_clone

Abort

If an online patching cycle has failed for some reason that cannot be corrected, you can abort the online patching cycle and return to normal operation. The adop “abort” command drops the database patch edition and removes or abandons any changes made during the the online patching cycle. If the online patching cycle has completed the cutover phase, then you can no longer abort the patching cycle.

When aborting an online patching cycle, you must also run the cleanup and fs_clone commands to fully eliminate changes from the patching cycle. The abort command is run as follows:

adop phase=abort adop phase=cleanup adop phase=fs_clone

Tools and Scripts for Edition-based Development:

There are a number of SQL*Plus scripts that can provide useful information about the state of your editioned development environment. All ADZD* scripts are found under $AD_TOP/sql.

• ADZDDBCC — database compliance checker, shows violations of the database object development standards documented in the Oracle E-Business Suite Developer’s Guide, Part No. E22961. Warning: this script takes a long time to run.
• ADZDSHOWED — Show database editions and current edition.
• ADZDSHOWLOG — Show full diagnostic log for online patching infrastructure.
• ADZDSHOWLOGEVT — Show only event and error messages from online patching diagnostic log (a useful summary, without the detailed statement text).
• ADZDSHOWLOGERR — Show only error messages from online patching diagnostic log.
• ADZDSHOWEV TABLE_SYNONYM_NAME — Show editioning view column mapping for table.
• ADZDSHOWTAB TABLE_SYNONYM_NAME — Show table information and related objects.
• ADZDSHOWMV MVIEW_NAME — Show materialized view information and related objects.
• ADZDSHOWTS — Show important tablespace status. Ensure that you have enough SYSTEM tablespace.
• ADZDCMPED — Compare Patch Edition with Run Edition. Warning: this script may take a long time to run.
• ADZDSHOWDDLS — Show stored DDL summary by phase.
• ADZDALLDDLS — Show stored DDL statement text.
• ADZDDDLERROR — Show stored DDL execution errors and messages.
• adutlrcmp — Recompile all objects, with before/after status report. Warning: this script may take a long time to run.

The following scripts are for experts:

• ADZDSHOWOBJS — Show Object Summary per edition. Counts of actual and stub (inherited) editioned object per edition.
• ADZDSHOWAOBJS — Show Actual Objects in the current edition. These are the editioned objects that have been changed by the patch.
• ADZDSHOWIOBJS — Show Inherited Objects in the current edition. These are the editioned objects that remain untouched in the patch edition. This script is used to confirm that the adop actualize_all command has worked properly.
• ADZDSHOWCOBJS — Show Covered Object Summary per edition. Count of objects in old editions that have a replacement in the run edition. This script is used to confirm that the adop cleanup command has worked properly.
• ADZDSHOWCOBJX — Show Covered Object List. List of objects in old editions that have a replacement in the run edition.
• ADZDSHOWSM — Show Seed Manager status.
• ADZDSHOWTM — Show Table Manager status.
• ADZDSHOWAD — AD (online patching) database object status
• ADZDSHOWSES — Show sessions connected to the database (by edition).
• ADZDSHOWDEP OBJECT_NAME — Show objects that OBJECT_NAME depends on.
• ADZDSHOWDEPTREE OBJECT_NAME — Show full dependency tree of objects that OBJECT_NAME depends on.

ADOP Parameters

Below table shows all the useful commands used in ADOP tool. You can always use adop -help to get online help at any point of time. It is also nice practice to use adop -status command before running of every phase of adop.

References:

1. Doc ID 1590494.1 : Patching & Maintenance Advisor: E-Business Suite (EBS) R12.2
2. Doc ID 1355068.1 : Oracle E-Business Suite 12.2 Patching Technology Components Guide
3. Doc ID 1577661.1 : Developing and Deploying Customizations in Oracle E-Business Suite Release 12.2
4. https://abhinavkotnala.wordpress.com/category/oracle-e-business-suite-r12-2/
5. Oracle E-Business Suite Maintenance Guide Release 12.2 Part No. E22954–21

More reference for troubleshooting

1. https://k21academy.com/oracle-apps-dba-r12/adop-prepare-phase-issue-fs_clone-failed-weblogic-management-scripting-scriptexception/

Originally posted at

https://pareshzawar.wixsite.com/tech/post/2017/04/27/adop-online-patching-oracle-ebs-r122