Sizing Asymmetric RAC Clusters

Oracle Real Application Clusters (RAC) allows workloads to be routed to cluster nodes in a fully load-balanced manner, or certain portions of the workload can be routed to specific nodes. The first case results in a symmetric workload with all nodes receiving approximately the same load. The second case can result in an asymmetric workload, with some nodes receiving a different amount of load. Asymmetric clusters can result in higher that expected costs for Exadata Cloud customers because Dynamic CPU Scaling is built for symmetry with all nodes using the same OCPU/ECPU setting. In this article, we will explore how to detect this condition and provide the proper sizing.

Under Sizing CPU for Cloud

Oracle Cloud database services (ExaDB-D, ExaDB-C@C, etc.) are designed for modern workloads with symmetrical distribution of work across instances or nodes of a RAC cluster. Asymmetric RAC clusters or workloads can result in under-sizing, resulting in higher than expected OCPU/ECPU charges in Oracle Cloud. RAC systems in the Cloud (whether BaseDB or Exadata) are configured for symmetric workloads, so the OCPU/ECPU setting has to match the node or instance with the highest CPU consumption. Let’s look at the distinctions between asymmetric clusters and asymmetric workloads.

Asymmetric Clusters

The nodes of a physical or virtual cluster can be symmetric or asymmetric. The number of processor cores, SMT threads, or relative “horsepower” of cluster nodes can be the same (symmetric) or can be different (asymmetric). Even in cases where the number of cores or threads are the same, the processors in the nodes can be different hardware generations, which can have different performance characteristics. This generally applies to on-premises systems because Oracle doesn’t support RAC on 3rd party Clouds as of this writing.

One example of an Asymmetric Cluster is a customer who deployed powerful multi-chip servers (multiple processor chips with many cores/threads per chip) to run EBS batch processing, while a set of less powerful servers were used for on-line OLTP user sessions. RAC Services can be used to route workloads to specific servers, or workloads can connect to a specific host and SID (Oracle System Identifier) rather than using Services. While this can be done to enable use of different sized servers, it can also help in the case of a low performance cluster interconnect in non-Exadata systems (spreading work to reduce data sharing).

Asymmetric Workloads

Physical server configurations can be entirely symmetric, but running a workload that is asymmetric. Rather than looking at server configuration, it’s more effective to interrogate CPU consumption metrics generated by Oracle and surfaced as “DB CPU Seconds per Second” metrics for each RAC instance. It’s not necessary to even determine how the application connects to the database because the resource consumption metrics tell what needs to be known.

It’s important to understand that RAC clusters will never be perfectly symmetrical. Oracle provides load balancing through several mechanisms, but there will always be some variance in how work is distributed across the cluster. What we are looking for is the DEGREE of asymmetry, answering the question of whether the workload is highly asymmetrical or not.

Percentage of Workload by Instance

The degree of asymmetry is determined by calculating the percentage of workload for each instance, which can be viewed as a pie chart or bar chart. Workload distribution is not expected to be perfectly equal, so there will always be some degree of asymmetry in the workload distribution across instances. Workload distribution across instances will change continuously. The following diagram shows a 4-node cluster running a RAC Database with 4 instances.

Note that Oracle RAC allows up to 100 instances in a RAC cluster, but the majority of deployments have much fewer than the maximum.

Changing RAC Instance Count During Migration

Migrating an existing RAC database often involves changing the number of RAC nodes (servers) or Database Instances. For example, an existing RAC database might run across 8 Database Servers (nodes) in an on-premises environment, but might be moved to an Exadata Cloud system with only 2 Database Servers. The number of processor cores has increased dramatically over the past several hardware generations, so decreasing the number of RAC nodes required is a natural response to the hardware evolution. Depending on how applications manage their connections to Oracle, it can be complex to make this change.

Sizing by Aggregate Resource Consumption

If sizing for OCPU/ECPU of a RAC cluster is done on aggregate resource consumption, the implication is you need to ensure the workload is balanced across the nodes of the cluster. This sizing methodology assumes work will be done to distribute the workload across nodes of the resulting cluster. This can be done manually for applications with asymmetric workloads, or it can be handled automatically by Oracle Database.

Manually Distributing Workloads

For targets that will run multiple consolidated databases, workloads can be more evenly distributed by assignment of connections to instances in a more evenly distributed fashion. The approach is to first reduce the number of instances see on the source to the number of instances that will be configured on the target. the second step is to bin-pack database instances into server nodes starting with the most heavily loaded instances first. The following diagram shows a conceptual view of this approach using 2 servers and 2 RAC databases that are evenly distributed across them. Remember this is the END-STATE architecture running 2 servers, while the source system might have been running more servers/nodes in the cluster.

The above diagram shows a simple example of evenly balancing the workload of two databases across two servers. In this example, both databases have a 75/25 split of work across instances (assume these databases have similarly sized workloads). Application connections that generate 75% of the workload for RAC database 1 are routed to the first server, with 25% of the workload routed to the second sever. The opposite is done for the second database, resulting in an even workload distribution across the two servers.

With larger deployments, this will be dramatically more complex. For example, consider a deployment of 50 databases running on a single on-premises Exadata system with 8-nodes (400 instances), moving to an Exadata Cloud system with 4 nodes (200 database instances). That means consolidating from 400 RAC instances to 200 instances, then evenly distributing the work across those 200 instances. Any collected metrics will come from the existing 400 RAC instances.

Automatically Distributing Workloads

Whether databases are consolidated or not, workloads can be automatically distributed across nodes on Exadata targets. The high performance cluster interconnect runs high performance network hardware, but also uses higher performance protocols in software that are unique to Exadata. While it might be possible to deploy physical hardware that performs as well as Exadata, the software is another matter. Exadata software uses Remote Direct Memory Access (RDMA) on the cluster interconnect in a feature called Exafusion (short for Exadata Cache Fusion) that provides latencies as low as 6 µsec (microseconds). You can see the effects of this performance boosting feature in AWR (Automatic Workload Repository) reports on Exadata systems.

The most common approach to automatic distribution of workloads across instances of a RAC database involves connecting via the SCAN (Single Client Access Network) address rather than connecting to specific nodes or instances. There are several methods to accomplish load balancing. The best choice depends on the existing application/database architecture.

Conclusion

Asymmetric workloads can result in under-sized OCPU/ECPU in Exadata Cloud. Asymmetric workloads are more common on asymmetric clusters, but may also exist on symmetric clusters as well. Properly distributing workloads on the target will avoid cluster asymmetry, resulting in OCPU/ECPU savings.

AI Comes to Exadata Sizing

The dramatically increased data volume we use for database system sizing has enabled us to use Artificial Intelligence to further improve the process. We previously relied on just a few data points for each database, but now we collect thousands! Read on to learn how we are integrating AI into the process of sizing Exadata for on-premises and Cloud.

Data Explosion

We injected an explosion of data into the sizing process starting in 2019, removing guesswork from the sizing process by making it much more data-driven. Rather than relying on “rule of thumb” approaches, we let the data show us the way forward. We now collect thousands of datapoints for each database and correlate those datapoints across databases to enable virtually ANY consolidation scheme users can imagine.

Oracle Cloud Capacity Analytics

We developed a tool called Oracle Cloud Capacity Analytics (OCCA) that is used internally to size systems for running Oracle Databases. The majority of the databases we size are moving to Oracle Cloud or one of the multi-cloud targets we support, but we also see a stead-stream of databases moving to Exadata on-premises. The OCCA tool gives estimates for the amount of system capacity required to run a set of databases. OCCA relies on data analytics to determine system size required to run those databases now and projecting into the future. The entire process is even more analytical and mathematical than ever before.

We use data science and analytics to size systems that will handle your databases in both Cloud and on-premises. We have also implemented Artificial Intelligence in strategic ways to make the process more effective than ever before.

AI Context Engineering

Our sizing tool begins with careful AI context engineering to ensure the AI is providing accurate answers to questions our users ask, but it also ensures the AI only provides answers in the area of expertise it’s trained on. If you ask the AI questions about things it doesn’t know about, the AI will tell you. For example, here’s what happens when a user asks OCCA why the sky is blue…

Carefully engineering the context the AI works within allows us to focus on training the AI for specific areas of expertise, increasing the quality of answers it gives to users. Context engineering also ensures the AI won’t even try to answer questions it knows nothing about, so we don’t have to worry about it telling users how to rule the world or otherwise do things the AI simply doesn’t handle. The AI we use has great information about sizing such as the definition of a cohort…

The AI we’re using (called “Ask OCCA”) is trained on the OCCA User’s Guide and related content to ensure it gives quality answers that are relevant to our users. It’s interesting that the AI will explain concepts from a different perspective than our technical writers might take on the same topic. Explaining the same topic from a different angle can help users understand the topic even better than before.

AI for Automating Cohorts

Databases are grouped together for consolidation, which saves system resources and reduces labor costs for managing databases. We call these groups of databases “cohorts” because there wasn’t a universally accepted term for this. In short, cohorts are groups of databases that can be consolidated together. The opposite of consolidation is isolation, and it’s important to consider these two ends of the spectrum. Cohorts capture the following:

  • Location of databases
  • Administrative Isolation
  • Security Isolation
  • Blast Radius

Most customers have databases in at least 2 locations such as production and disaster recovery locations. Many cloud vendors also have regions and availability domains for an additional layer of isolation by location, and that scheme is becoming more common in on-premises data centers. Administrative isolation is nearly ubiquitous because most organizations prefer to isolate production from development and test systems. Financial services, banking, and government customers often have specially isolated systems for security purposes, sometimes including multiple separate (and secure) networks. It’s the digital equivalent of security gate, security desk, and locks at each floor of a high rise building. It’s also common for customers to isolate databases from each other to reduce the impact of a failure or what is known as the “blast radius” of a failure.

There are sometimes markers in the data that we collect that suggests what cohort assignments should be. Customers use a wide variety of schemes for organizing their databases including some of the attributes shown below:

  • Database naming conventions
  • Server naming conventions
  • Cluster names
  • Exadata Database Machines used for consolidation
  • Enterprise Manager attributes (cost center, department, lifecycle status, line of business, location)

Artificial Intelligence is an ideal technology to determine what scheme a customer might be using, making cohort assignments much easier to divine from the data. The relationships between Primary and Standby databases can also give us clues about what the cohorts should be, or at least which databases can reside in each cohort. We are looking to AI to help us understand a population of databases and how they should be grouped together before we ask customers how they want to organize their databases. The opposite approach is to simply isolate everything onto dedicated Virtual Machines, but the cost and operational advantages of database consolation are too great to ignore.

AI-Driven Sizing Report Comparisons

Each target platform (Exadata on-premises, Exadata Cloud@Customer, Autonomous Database, etc.) has multiple options, versions, and other settings that affect sizing output. Thousands of data points are ingested for each sizing, and each platform has different characteristics that affect the capacity required to handle the workload. These factors make report comparison extremely complex, so we turned to Artificial Intelligence to make these comparisons useful. AI is able to evaluate the various factors that make one report different from another, allowing us to build a comparison tool that users find helpful.

AI-Driven Gap Handling (AI Code Assist)

Handling gaps in data is a great example of how AI Code Assist has helped us make our data-driven sizing even more effective. Gaps in data occur simply because Oracle Enterprise Manager (OEM) deployments are operational systems used. OEM is used for keeping databases running and helping administrators care for those databases on a day-to-day or even minute-by-minute basis. OEM captures performance metrics for the purpose of viewing and addressing performance problems. However, we use that information for a dramatically different purpose, which is capacity planning and sizing. We often find gaps in the data or “missing metrics” because some databases might be down, might be undergoing an upgrade, or whatever operational state the database could be in when the metrics are extracted.

These missing metrics or data gaps occur in the most recent, most fine-grained data, but we can sometimes use other metrics from prior periods to fill those gaps. The code required to fill these gaps is extremely complex, examining over 10’s or 100’s of thousands of rows with many attributes per row and more than a year of historical metrics for each database. We used AI Code Assist technology to develop the algorithms needed to fill these gaps in the collected metrics. This allowed our relatively small team of developers to deploy the necessary code that is used by hundreds of engineers to build sizing estimates for thousands of customers.

Conclusion

This article should give you a good idea of how we are putting AI to work in the area of Exadata sizing. Whether you are moving to Cloud or deploying Exadata on-premises, AI is helping us do an even more effective job of sizing systems to meet your needs using the inputs and guidance you provide.

Exadata Patching

This article covers Exadata Patching for on-premises Exadata systems (non-Cloud) or what is also known as software maintenance.

Oracle Platinum Services

Oracle will patch Exadata systems if you are unable to do it yourself. The platinum services team has done a great job tailoring the service to meet the ever-expanding capabilities of Exadata and resulting needs of customers. However, there are cases where customers need more flexibility than Platinum Services can provide, so some customers do Exadata patching themselves or contract for Oracle or 3rd party services to provide that flexibility. Be sure to review the Platinum Services Data Sheet (here), review the Scope of Services (linked from the Data Sheet) and talk to your Oracle Sales Representative to see if Platinum Services is right for you.

Advantages of Exadata in the Cloud

Exadata in the Cloud includes Cloud@Customer, OCI (Oracle Cloud Infrastructure), and Exadata multi-cloud (@AWS, @Azure, and @GCP). While this article covers on-premises Exadata systems, it’s important to note that Exadata in the Cloud offers some distinct advantages when it comes to patching. Some of the complexity outlined in this article is addressed in the Cloud by…

  • Standardization
  • Oracle Cloud Operations (division of labor)
  • Automated Tools

As you will see in this article, patching Exadata includes a great deal of flexibility in how it’s implemented. Oracle Cloud addresses this by first standardizing on the most common approaches which are then automated via the Cloud Control Plane software. This standardized framework is then evolved to include additional options needed by customers. Oracle Cloud includes a division of labor and responsibilities, assigning some of those responsibilities to Oracle Cloud Operations. All hardware maintenance is the responsibility of Oracle Cloud Operations, as well as the lower layers of the Exadata System Software stack (including the Hypervisor). The upper layers (Virtual Machine, GI, and Database software) are the responsibility of the customer, but automated tools are provided to simplify these tasks.

Frequency of Patching

Oracle allows customers to patch Exadata systems according to the frequency that best meets their business needs. In general, any of the following

  • Yearly
  • Quarterly
  • Monthly

Oracle recommends patching Exadata systems at least yearly, while the majority of Oracle customers patch Exadata systems on a quarterly basis. Heightened security concerns (due to the rise of Artificial Intelligence) are pushing more customers toward monthly patching, especially for applying security patches. Few customers patch Exadata systems on a monthly basis due to challenges with reboots and business impact (even with rolling patching). See the “Live Update” section of this article for more information.

Patch Bundles

Oracle provides patch bundles for all layers of the Exadata infrastructure, from O/S and drivers through database software. These bundles are separated into 5 areas as follows:

  • Exadata Database Servers (Bare Metal and Virtual Machines)
  • Exadata Storage Servers
  • Exadata network switches (InfiniBand or RoCE)
  • Oracle Grid Infrastructure
  • Oracle Database software (binaries & data patch)

The first three on the list above involving patching Exadata components and including drivers, O/S, and Exadata specific software. The included drivers are only those drivers necessary for Exadata systems. Exadata runs a distribution of Oracle Linux with a specialized kernel and minimal set of Linux packages needed for running Oracle Database. Compatibility of Exadata Release with GI and Database software are documented in MOS note KB153930.

Quarterly Full Stack Download Patch (QFSDP)

Oracle combines all patches into a single patch bundle known as the Quarterly Full Stack Download Patch (QFSDP) for the most common combination of versions. The advantage of QFSDP is it simplifies the task of evaluating software version dependencies, since all of those dependencies are built into the QFSDP. However, there might be additional combinations of software versions that ARE compatible beyond what is included in QFSDP. You get a bit more flexibility by not using QFSDP, but at the expense of a bit more work evaluating all of the combinations and permutations of software versions (outlined in MOS note KB153930).

Exadata Software Versions & Releases

All Exadata systems should be patched according to an established schedule as outlined above. It is important to stay current on Exadata software for the most stable and secure system possible, regardless of the hardware age. Exadata System Software includes the following levels:

  • Major Version
  • Release
  • Maintenance Release

Changes in Major Version involve changing the Oracle Linux version, which will sometimes involve more extensive changes. For example, the reserved UID/GID (User ID and Group ID) ranges changed in Oracle Linux 8, which sometimes forced changes to customer systems. Likewise, more extensive changes are expected when moving to Oracle Linux 9 (in Exadata System Software 26.2). One change that can affect customers is the minimum length of SSH keys to deliver greater security in Linux 9.

Recommended Exadata Releases

Oracle documents recommended releases in KB153930, and it is not always the latest release. Customers should use the RECOMMENDED release unless there is a feature or hardware compatibility requirement that dictates a later release than what is documented as recommended. Oracle performs extensive certification testing of each software release and hardware generation to ensure quality of the release. The recommended release will change as experience is gained in the field on real-world customer systems.

Current vs. Target Release

The amount of work required during the patching process is at least partially dependent on the current vs. target release. Waiting too long between patching events will often increase the amount of effort involved. Systems need to be at the minimum release level to upgrade directly to certain target releases. If the patching involves a change in Major Version (Linux version change), additional work is often required.

Exadata Database Server: Rolling vs. Non-Rolling Patching

The “state of the art” method for Exadata maintenance is to use rolling (or RAC rolling) patching, where each Database Server gets shutdown individually. RAC (Real Application Clusters) keeps the database running and accessible while nodes are being rebooted. Transparent Application Continuity (TAC) allows connections to move between nodes so users are not impacted. However, some applications don’t support RAC rolling updates.

Be aware that rolling patching of Database Servers will reduce the aggregate CPU, Memory, and I/O bandwidth when each Database Server goes down. Connections should be moved over to a node that’s running if the application is properly configured to use TAC. However, the aggregate capacity will be reduced and won’t be available to run workloads.

Exadata Storage: Rolling vs. Non-Rolling Patching

As with Database Servers, the “state of the art” for Exadata Storage Servers is to perform rolling updates. However, Exadata storage needs to be configured in High Redundancy to use rolling updates. Be aware that rolling patching of Exadata Storage Servers will reduce the aggregate XRMEM and Flash Cache available when each Storage Server is brought down. Exadata software (as of 26.1) pre-populates Flash Cache on partner Storage Servers to avoid a Flash Cache miss during patching. However, the aggregate Flash Cache is still being reduced during the patching process.

Exadata Switches – Always Rolling Patching

The internal switch fabric of Exadata (InfiniBand for older Exadata models, and RoCE in newer Exadata models) is designed to always be patched in a rolling fashion. The network fabric is fully redundant, with multiple network connections for each Database Server, multiple network connections for each Storage Server, and redundant switches as well.

Exadata Documentation

Each generation of the Exadata Documentation is cumulative, so the latest version includes instructions that apply to earlier versions.

MOS Articles

My Oracle Support (MOS) includes Knowledge Base articles that are critical to Exadata patching. Those notes are updated at least monthly, so be sure to review the latest version prior to any Exadata patching. The key KB articles are these:

  • KB153930 – Exadata Database Machine and Exadata Storage Server Supported Versions
  • KB467937 – Exadata System Software Certification

KB153930 detailed which versions of the various software stack layers are compatible with each other, while KB467937 details which Exadata Software versions have been certified with which Exadata Hardware generations.

Patch Readme Files

Each set of Exadata patches include readme files. These contain details of what is being applied in the update/patch, but may also contain other important information. It’s highly important to review any of the readme files within each download.

Exadata Configuration Check Utility (exachk)

Your Exadata system should generally be in a “healthy” state before you start patching. All Exadata systems should be configured to run the Exadata Configuration Checker (exachk) on a regular basis. The exachk tool will check the health of the system. Oracle periodically updates exachk to include new checks when issues are discovered, so exachk itself should be updated periodically to ensure your system has the latest checks included.

Local Customizations

You should avoid customizing your Exadata system as much as possible. That includes adding software that isn’t explicitly necessary for operating the system. Some 3rd party software or local customizations can stall the upgrade/patching process. If you can’t avoid installing 3rd party software, it’s best to install software in the form of RPM packages because software installed as packages are tracked by Linux in a standard manner. It is possible to customize the system in such a manner that will cause patching to fail. Customers have “root” (super user) authority on Exadata, and that level of access can be used to make changes that cause failures. When Oracle development becomes aware of these issues, checks are added to the exachk utility to detect those problems prior to patching. However, it is not possible to check every possible local customization that will impact patching.

Handling 3rd Party Packages

Oracle introduced features in the 25.1 release that provide much better handling of 3rd party packages. The Exadata software installer (patchmgr) is able to check dependencies among software packages to determine if 3rd party packages need to be upgraded in conjunction with a given Exadata software update. The traditional solution was to de-install the 3rd party software prior to the Exadata software update, then re-install the 3rd party software when Exadata patching is complete. That process is rather inconvenient and it’s undesirable in some cases (especially with security software).

Exadata Live Update

Starting with Exadata System Software 24.1, Oracle provides the Live Update feature which is designed to avoid reboot (keeping the system “live”) when critical updates and especially security updates (CVEs) are applied. Live Update is designed to be deployed as follows:

  • Quarterly Full Patching
  • Monthly Live Updates for CVEs

The most common Exadata patching interval used by customers is quarterly full patching (QFSDP or not). The vast majority of customers say that more frequent patching would be a burden and isn’t necessarily feasible. The amount of work would be a burden, and coordination with the business and application teams would make that difficult. Oracle designed the Live Update feature to allow customers to apply critical fixes and security updates (CVEs) without a system reboot. For more information about Exadata Live Update, see here.

Exadata Hardware Lifetimes

It’s important to realize that Exadata software should be updated on a regular basis throughout its lifecycle. Oracle anticipates that customers will refresh Exadata hardware after approximately 7 years of use or even more frequently in some cases. MOS Knowledge Base document KB467937 documents which Exadata System Software Releases have been certified on which Exadata Hardware Generations. The matrix in KB467937 shows the earliest certified release on a hardware generation, and will show the final certified release when that occurs. Certification is the process of testing Exadata System Software on a given Exadata Hardware Generation. Oracle runs approximately 25 million tests during each certification run, ensuring customers will not encounter issues. Oracle runs certification tests on hardware until Last Ship Date plus 7 years (LSD+7) or longer as outlined in KB467937.

Data Guard Considerations

Oracle Data Guard is an important component in minimizing downtime in high availability environments. However, it’s important to understand that REDO apply in Data Guard doesn’t serve to warm the Flash Cache. Changed blocks will be pulled into Flash Cache, but read activity is not propagated from primary to standby.

Standby-First Patching (Database patches)

Standby-First Patching is a concept that applies to the Database layer, and has been around since Oracle11g. For patches (RU or Interim) that are flagged as standby-first installable in the patch readme, the binaries (but not the data patch) can be patched on the standby side first.

  • Check pre-requisites
  • Patch standby binaries
  • Switch roles
  • Patch primary binaries
  • Run Datapatch

The Standby-First Patching process needs to be integrated with the Exadata System Software patching process. For more information on patching databases that use Data Guard, please refer to the manual here.

Database Patching

Exadata System Software releases require certain minimum Database patching. As of this writing, all Oracle Databases should be running one of the following versions:

  • Oracle Database 19c
  • Oracle AI Database 26ai

Although earlier database versions (11g, 12cR1, 12cR2, 18c, 21c, and 23c) can run on Exadata, special support agreements are required and Oracle prefers everyone upgrade instead. If you have 3rd party applications that require one of these older versions, you should put pressure on the application vendor to upgrade. If it’s an internally developed (custom) application, pressure needs to be applied on the internal application owner.

Minimum database software patches for Exadata Releases are documented in KB153930.

Virtualized Exadata

Exadata machines can run bare-metal or virtualized. For virtualized systems, the hypervisor (on the physical host) will be patched, and the virtual machine(s) on the host will be patched separately. Virtual machines on a physical server can be members in RAC clusters residing on different physical server. Physical servers (and hypervisors) are not members of clusters. If using Exascale storage, Virtual Machines can move between physical servers and they can be moved without a restart using Live Migration.

Exadata Patching Process

The process of patching Exadata can vary according to the criteria discussed above. However, the general process is essentially as follows:

  • Preparation
  • Setup Driving System
  • Run pre-checks
  • Backup
  • Apply patches
  • Post-Update Verification

Let’s dive into each of these steps in detail.

1) Preparation

You can begin by looking at the basic contours of the patching process. Preparation includes all of the pre-work required to determine what needs to be done. This is the planning, so some of these will lead to steps that are executed during the patching process.

  • Exadata Hardware Generation(s)
  • Current Exadata System Software Release
  • Target Exadata System Software Release & prerequisites
  • 3rd Party Software Installed
  • Current System Health
  • Check for Customizations
  • High Availability Requirements & Options for HA

Exadata Hardware Generation(s) involved will dictate which Exadata System Software releases can be applied on that hardware. Watch for older hardware that is approaching End of Certification and needs to be refreshed. Every Exadata System Software release will allow direct upgrade from certain other releases with a minimum Maintenance Release for each. If you’re following the minimum upgrade interval (upgrading at least yearly), you should already be meeting the minimums. However, if you are delaying your patching for a longer interval, it might be necessary to apply a Maintenance Release before applying the target Exadata System Software Release.

2) Setup Driving System

Exadata System Software patching can be done from a separate system or from one of the Exadata Database Servers. The advantage of using a separate system is that particular system doesn’t need to be upgraded as part of the process. If you’re running the upgrade from an Exadata Database Server, you’ll need to switch the driving server at some point in the process. In other words, you cannot use a driving system to update itself.

3) Run Pre-Checks

The overall plan built during the preparation phase will involve some steps that are executed during pre-check. For example, the preparation/planning process might identify certain 3rd party software is installed on the system, which will mean that software will be either uninstalled or RPM dependencies will have to be addressed during the upgrade. In other words, running of pre-checks happens during the execution phase of the patching process, whereas preparation is the planning phase.

4) Backup

A full system backup and database backup need to be completed during the execution phase of patching. This provides a fallback in the event patching fails.

5) Apply Patches

Calling these “patches” is somewhat of a misnomer, since Major Versions, Releases, and Maintenance Releases might be involved in the process. It really depends on the current vs. target release involved and what boundaries of Major Versions and Releases are being crossed.

6) Post-Update Verification

The final step of any system patching or upgrade should always involve verification. Databases need to be running and accessible, applications need connectivity. It’s often important to TEST the application to ensure everything is working properly before declaring an end of the patching process.

Debugging

Technicians executing the patching need to be adept at debugging the process. Fast response to problems during patching will mitigate downtime and business impact. Debugging starts with examining any log files produced by the driving process. For example, the Exadata patchmgr tool generates logs that are helpful in debugging the process.