This article covers Exadata Patching for on-premises Exadata systems (non-Cloud) or what is also known as software maintenance.

Oracle Platinum Services

Oracle will patch Exadata systems if you are unable to do it yourself. The platinum services team has done a great job tailoring the service to meet the ever-expanding capabilities of Exadata and resulting needs of customers. However, there are cases where customers need more flexibility than Platinum Services can provide, so some customers do Exadata patching themselves or contract for Oracle or 3rd party services to provide that flexibility. Be sure to review the Platinum Services Data Sheet (here), review the Scope of Services (linked from the Data Sheet) and talk to your Oracle Sales Representative to see if Platinum Services is right for you.

Advantages of Exadata in the Cloud

Exadata in the Cloud includes Cloud@Customer, OCI (Oracle Cloud Infrastructure), and Exadata multi-cloud (@AWS, @Azure, and @GCP). While this article covers on-premises Exadata systems, it’s important to note that Exadata in the Cloud offers some distinct advantages when it comes to patching. Some of the complexity outlined in this article is addressed in the Cloud by…

• Standardization
• Oracle Cloud Operations (division of labor)
• Automated Tools

As you will see in this article, patching Exadata includes a great deal of flexibility in how it’s implemented. Oracle Cloud addresses this by first standardizing on the most common approaches which are then automated via the Cloud Control Plane software. This standardized framework is then evolved to include additional options needed by customers. Oracle Cloud includes a division of labor and responsibilities, assigning some of those responsibilities to Oracle Cloud Operations. All hardware maintenance is the responsibility of Oracle Cloud Operations, as well as the lower layers of the Exadata System Software stack (including the Hypervisor). The upper layers (Virtual Machine, GI, and Database software) are the responsibility of the customer, but automated tools are provided to simplify these tasks.

Frequency of Patching

Oracle allows customers to patch Exadata systems according to the frequency that best meets their business needs. In general, any of the following

• Yearly
• Quarterly
• Monthly

Oracle recommends patching Exadata systems at least yearly, while the majority of Oracle customers patch Exadata systems on a quarterly basis. Heightened security concerns (due to the rise of Artificial Intelligence) are pushing more customers toward monthly patching, especially for applying security patches. Few customers patch Exadata systems on a monthly basis due to challenges with reboots and business impact (even with rolling patching). See the “Live Update” section of this article for more information.

Patch Bundles

Oracle provides patch bundles for all layers of the Exadata infrastructure, from O/S and drivers through database software. These bundles are separated into 5 areas as follows:

• Exadata Database Servers (Bare Metal and Virtual Machines)
• Exadata Storage Servers
• Exadata network switches (InfiniBand or RoCE)
• Oracle Grid Infrastructure
• Oracle Database software (binaries & data patch)

The first three on the list above involving patching Exadata components and including drivers, O/S, and Exadata specific software. The included drivers are only those drivers necessary for Exadata systems. Exadata runs a distribution of Oracle Linux with a specialized kernel and minimal set of Linux packages needed for running Oracle Database. Compatibility of Exadata Release with GI and Database software are documented in MOS note KB153930.

Quarterly Full Stack Download Patch (QFSDP)

Oracle combines all patches into a single patch bundle known as the Quarterly Full Stack Download Patch (QFSDP) for the most common combination of versions. The advantage of QFSDP is it simplifies the task of evaluating software version dependencies, since all of those dependencies are built into the QFSDP. However, there might be additional combinations of software versions that ARE compatible beyond what is included in QFSDP. You get a bit more flexibility by not using QFSDP, but at the expense of a bit more work evaluating all of the combinations and permutations of software versions (outlined in MOS note KB153930).

Exadata Software Versions & Releases

All Exadata systems should be patched according to an established schedule as outlined above. It is important to stay current on Exadata software for the most stable and secure system possible, regardless of the hardware age. Exadata System Software includes the following levels:

• Major Version
• Release
• Maintenance Release

Changes in Major Version involve changing the Oracle Linux version, which will sometimes involve more extensive changes. For example, the reserved UID/GID (User ID and Group ID) ranges changed in Oracle Linux 8, which sometimes forced changes to customer systems. Likewise, more extensive changes are expected when moving to Oracle Linux 9 (in Exadata System Software 26.2). One change that can affect customers is the minimum length of SSH keys to deliver greater security in Linux 9.

Recommended Releases

Oracle documents recommended releases in KB153930, and it is not always the latest release. Customers should use the RECOMMENDED release unless there is a feature or hardware compatibility requirement that dictates a later release than what is documented as recommended. Oracle performs extensive certification testing of each software release and hardware generation to ensure quality of the release. The recommended release will change as experience is gained in the field on real-world customer systems.

Current vs. Target Release

The amount of work required during the patching process is at least partially dependent on the current vs. target release. Waiting too long between patching events will often increase the amount of effort involved. Systems need to be at the minimum release level to upgrade directly to certain target releases. If the patching involves a change in Major Version (Linux version change), additional work is often required.

Exadata Database Server: Rolling vs. Non-Rolling Patching

The “state of the art” method for Exadata maintenance is to use rolling (or RAC rolling) patching, where each Database Server gets shutdown individually. RAC (Real Application Clusters) keeps the database running and accessible while nodes are being rebooted. Transparent Application Continuity (TAC) allows connections to move between nodes so users are not impacted. However, some applications don’t support RAC rolling updates.

Be aware that rolling patching of Database Servers will reduce the aggregate CPU, Memory, and I/O bandwidth when each Database Server goes down. Connections should be moved over to a node that’s running if the application is properly configured to use TAC. However, the aggregate capacity will be reduced and won’t be available to run workloads.

Exadata Storage: Rolling vs. Non-Rolling Patching

As with Database Servers, the “state of the art” for Exadata Storage Servers is to perform rolling updates. However, Exadata storage needs to be configured in High Redundancy to use rolling updates. Be aware that rolling patching of Exadata Storage Servers will reduce the aggregate XRMEM and Flash Cache available when each Storage Server is brought down. Exadata software (as of 26.1) pre-populates Flash Cache on partner Storage Servers to avoid a Flash Cache miss during patching. However, the aggregate Flash Cache is still being reduced during the patching process.

Exadata Switches – Always Rolling Patching

The internal switch fabric of Exadata (InfiniBand for older Exadata models, and RoCE in newer Exadata models) is designed to always be patched in a rolling fashion. The network fabric is fully redundant, with multiple network connections for each Database Server, multiple network connections for each Storage Server, and redundant switches as well.

Exadata Documentation

Each generation of the Exadata Documentation is cumulative, so the latest version includes instructions that apply to earlier versions.

MOS Articles

My Oracle Support (MOS) includes Knowledge Base articles that are critical to Exadata patching. Those notes are updated at least monthly, so be sure to review the latest version prior to any Exadata patching. The key KB articles are these:

• KB153930 – Exadata Database Machine and Exadata Storage Server Supported Versions
• KB467937 – Exadata System Software Certification

KB153930 detailed which versions of the various software stack layers are compatible with each other, while KB467937 details which Exadata Software versions have been certified with which Exadata Hardware generations.

Patch Readme Files

Each set of Exadata patches include readme files. These contain details of what is being applied in the update/patch, but may also contain other important information. It’s highly important to review any of the readme files within each download.

Exadata Configuration Check Utility (exachk)

Your Exadata system should generally be in a “healthy” state before you start patching. All Exadata systems should be configured to run the Exadata Configuration Checker (exachk) on a regular basis. The exachk tool will check the health of the system. Oracle periodically updates exachk to include new checks when issues are discovered, so exachk itself should be updated periodically to ensure your system has the latest checks included.

Local Customizations

You should avoid customizing your Exadata system as much as possible. That includes adding software that isn’t explicitly necessary for operating the system. Some 3rd party software or local customizations can stall the upgrade/patching process. If you can’t avoid installing 3rd party software, it’s best to install software in the form of RPM packages because software installed as packages are tracked by Linux in a standard manner. It is possible to customize the system in such a manner that will cause patching to fail. Customers have “root” (super user) authority on Exadata, and that level of access can be used to make changes that cause failures. When Oracle development becomes aware of these issues, checks are added to the exachk utility to detect those problems prior to patching. However, it is not possible to check every possible local customization that will impact patching.

Handling 3rd Party Packages

Oracle introduced features in the 25.1 release that provide much better handling of 3rd party packages. The Exadata software installer (patchmgr) is able to check dependencies among software packages to determine if 3rd party packages need to be upgraded in conjunction with a given Exadata software update. The traditional solution was to de-install the 3rd party software prior to the Exadata software update, then re-install the 3rd party software when Exadata patching is complete. That process is rather inconvenient and it’s undesirable in some cases (especially with security software).

Exadata Live Update

Starting with Exadata System Software 24.1, Oracle provides the Live Update feature which is designed to avoid reboot (keeping the system “live”) when critical updates and especially security updates (CVEs) are applied. Live Update is designed to be deployed as follows:

• Quarterly Full Patching
• Monthly Live Updates for CVEs

The most common Exadata patching interval used by customers is quarterly full patching (QFSDP or not). The vast majority of customers say that more frequent patching would be a burden and isn’t necessarily feasible. The amount of work would be a burden, and coordination with the business and application teams would make that difficult. Oracle designed the Live Update feature to allow customers to apply critical fixes and security updates (CVEs) without a system reboot. For more information about Exadata Live Update, see here.

Exadata Hardware Lifetimes

It’s important to realize that Exadata software should be updated on a regular basis throughout its lifecycle. Oracle anticipates that customers will refresh Exadata hardware after approximately 7 years of use or even more frequently in some cases. MOS Knowledge Base document KB467937 documents which Exadata System Software Releases have been certified on which Exadata Hardware Generations. The matrix in KB467937 shows the earliest certified release on a hardware generation, and will show the final certified release when that occurs. Certification is the process of testing Exadata System Software on a given Exadata Hardware Generation. Oracle runs approximately 25 million tests during each certification run, ensuring customers will not encounter issues. Oracle runs certification tests on hardware until Last Ship Date plus 7 years (LSD+7) or longer as outlined in KB467937.

Data Guard Considerations

Oracle Data Guard is an important component in minimizing downtime in high availability environments. However, it’s important to understand that REDO apply in Data Guard doesn’t serve to warm the Flash Cache. Changed blocks will be pulled into Flash Cache, but read activity is not propagated from primary to standby.

Standby-First Patching (Database patches)

Standby-First Patching is a concept that applies to the Database layer, and has been around since Oracle11g. For patches (RU or Interim) that are flagged as standby-first installable in the patch readme, the binaries (but not the data patch) can be patched on the standby side first.

• Check pre-requisites
• Patch standby binaries
• Switch roles
• Patch primary binaries
• Run Datapatch

The Standby-First Patching process needs to be integrated with the Exadata System Software patching process. For more information on patching databases that use Data Guard, please refer to the manual here.

Virtualized Exadata

Exadata machines can run bare-metal or virtualized. For virtualized systems, the hypervisor (on the physical host) will be patched, and the virtual machine(s) on the host will be patched separately. Virtual machines on a physical server can be members in RAC clusters residing on different physical server. Physical servers (and hypervisors) are not members of clusters. If using Exascale storage, Virtual Machines can move between physical servers and they can be moved without a restart using Live Migration.

Exadata Patching Process

The process of patching Exadata can vary according to the criteria discussed above. However, the general process is essentially as follows:

• Preparation
• Setup Driving System
• Run pre-checks
• Backup
• Apply patches
• Post-Update Verification

Let’s dive into each of these steps in detail.

1) Preparation

You can begin by looking at the basic contours of the patching process. Preparation includes all of the pre-work required to determine what needs to be done. This is the planning, so some of these will lead to steps that are executed during the patching process.

• Exadata Hardware Generation(s)
• Current Exadata System Software Release
• Target Exadata System Software Release & prerequisites
• 3rd Party Software Installed
• Current System Health
• Check for Customizations
• High Availability Requirements & Options for HA

Exadata Hardware Generation(s) involved will dictate which Exadata System Software releases can be applied on that hardware. Watch for older hardware that is approaching End of Certification and needs to be refreshed. Every Exadata System Software release will allow direct upgrade from certain other releases with a minimum Maintenance Release for each. If you’re following the minimum upgrade interval (upgrading at least yearly), you should already be meeting the minimums. However, if you are delaying your patching for a longer interval, it might be necessary to apply a Maintenance Release before applying the target Exadata System Software Release.

2) Setup Driving System

Exadata System Software patching can be done from a separate system or from one of the Exadata Database Servers. The advantage of using a separate system is that particular system doesn’t need to be upgraded as part of the process. If you’re running the upgrade from an Exadata Database Server, you’ll need to switch the driving server at some point in the process. In other words, you cannot use a driving system to update itself.

3) Run Pre-Checks

The overall plan built during the preparation phase will involve some steps that are executed during pre-check. For example, the preparation/planning process might identify certain 3rd party software is installed on the system, which will mean that software will be either uninstalled or RPM dependencies will have to be addressed during the upgrade. In other words, running of pre-checks happens during the execution phase of the patching process, whereas preparation is the planning phase.

4) Backup

A full system backup and database backup need to be completed during the execution phase of patching. This provides a fallback in the event patching fails.

5) Apply Patches

Calling these “patches” is somewhat of a misnomer, since Major Versions, Releases, and Maintenance Releases might be involved in the process. It really depends on the current vs. target release involved and what boundaries of Major Versions and Releases are being crossed.

6) Post-Update Verification

The final step of any system patching or upgrade should always involve verification. Databases need to be running and accessible, applications need connectivity. It’s often important to TEST the application to ensure everything is working properly before declaring an end of the patching process.

Debugging

Technicians executing the patching need to be adept at debugging the process. Fast response to problems during patching will mitigate downtime and business impact. Debugging starts with examining any log files produced by the driving process. For example, the Exadata patchmgr tool generates logs that are helpful in debugging the process.