Exadata Exascale was first introduced in 2024, and brings some big advantages over the ASM (Automatic Storage Management) architecture that we’ve had since Exadata was first introduced in 2008. You need to run Oracle Database 23ai to use Exascale, but that simply means you get access to over 300 new database features like AI Vector Search and JSON Relational Duality in 23ai.

More Efficient Space Usage

Exascale puts all storage in a single pool rather than separate Disk Groups for DATA and RECO (and the optional SPARSE Disk Group) we have with ASM. Those separate silos of storage end up wasting space and it’s not easy to change the allocations. Exascale storage capacity is simpler to manage because it’s pooled storage.

Eliminates Duplicate Management

You create an ASM Disk Group for RECO, then you create an FRA (Fast Recovery Area) in each database and put it in RECO. That’s a bit of duplicate effort that gets eliminated because Exascale doesn’t need a separate RECO area to hold recovery data like your FRA. You size RECO, then you size FRA for each database with ASM. With Exascale, you just size the FRA.

Vaults for Security and Resource Management

Within a Exascale Storage Pool, you have Vaults, which are logical constructs that provide security isolation and resource controls. You can have arbitrarily large numbers of Vaults, and it’s fast and easy to add more of them and change their resource limits. Each Vault can include limits on:

• Storage space
• IOPS
• XRMEM Cache
• Flash Cache

These controls let administrators keep one tenant from impacting another, meaning they prevent “noisy neighbor” problems. When I say “tenant” here, I’m talking about a Virtual Machine cluster, which contains one or more databases (more about that later).

Scale Extra Small to Extra Large

Exascale allows storage for a tenant to scale from Gigabytes to Petabytes, giving it a much wider range than Exadata with ASM storage. The smallest Exadata system gives you about 52 TB of storage for DATA for each Storage Server (with triple mirroring) and there is a minimum of 3 Storage Servers. You can easily see how the number of tenants (VM Clusters) gets large very quickly if you set the smallest storage for a tenant at 300 GB or even 1 TB in size.

Separation of Compute & Storage

ASM uses software running on each compute server to control allocation of and access to storage. This architecture creates a tight coupling between compute and storage. Exascale eliminates this coupling and provides better separation between compute and storage, making management simpler and more flexible.

Database Aware Thin Clones

Thin Clones are copies of databases that don’t consume space until changes are made. Being “database-aware” means there is a single DDL command to create them rather than using a bolt-on utility. Easy, fast, and space efficient copies of databases.

Multi-Generation – Mixed Disk Sizes

Disks often get larger from on hardware generation to the next. ASM requires equally sized disk slices. You could expand an older system with newer generation storage, but you would either sacrifice the extra space or create extra Disk Groups with that space. Exascale lets you just put the storage into the same pool and it handles it without sacrificing space.

VM Images on Shared Storage

The storage for a Virtual Machine includes the boot volume, Oracle Database homes (software) and other contents of a running VM. Exadata has always put VM images on each Database Server. Exascale lets you put VM storage on the Exascale shared storage, which brings multiple advantages as we will see.

Effectively Unlimited VM Storage

The amount of space each VM needs is relatively small compared to the shared storage of Exadata. Moving VM storage off the Database Servers to the Exascale storage side gives you access to more VM space than you’ll ever need, regardless of how many VMs you run.

Massive Numbers of VMs (tenants)

Exadata has been limited to 4, 8, or 12 Virtual Machines for each Database Server, but Exascale dramatically increases that limit. The highest number any customer has asked for was 25, so we doubled that to 50 as our initial limit. We want to make sure it’s rock-solid, so we put a stake in the ground and called it 50. One point about being “rock solid” is they ALL need to boot in a reasonable amount of time on server startup.

RDMA Enabled Storage – High Performance

All of the storage in Exascale is RDMA enabled, whether it’s being used for databases or VM images. The advantage is performance. RDMA is a protocol used by software for extremely low latency and fastest access. It’s completely unique to Exadata. The Oracle Database software makes RDMA calls, and that is only implemented on Exadata.

Live Migration is Coming

We announced at Oracle Cloud World last year that Live Migration would be coming to Exascale. This means that you’ll be able to move a VM from one Database Server to another without shutting it down. What makes this impressive is the fact that Oracle Database uses RDMA for communication and IO on Exadata. RDMA is extremely fast, but it turns Live Migration into rocket science.

Conclusion

Exascale is the next generation software architecture for Exadata. It provides advantages for databases and virtual machines, allowing you to scale from XS to XL in virtually any dimension.

I ran across this photo from 12 years ago and got to thinking it’s still more than 4X larger than the largest system you can construct for an Oracle database on Amazon’s RDS (Relational Database Service). This is a single Exadata Version 1 system spanning 3 data center racks.

Exadata Version 1 – Triple Rack System

This was one of the first systems I had seen at a customer site and it was a triple-rack configuration.

• 3 Data Center racks wide
• 528 CPU cores
• 504 disk drives
• 226 Terabytes of storage

Of course the latest Exadata systems are much more powerful than this, but it’s stunning when you realize that Amazon cannot match even 1/4 of this machine from 12 years ago. The largest Oracle database you can run in Amazon is around 50 Terabytes. The largest number of CPU cores you can have in Amazon is 64 cores, but that system (db.x1.32xlarge) will cost MORE than Exadata Cloud Service!

As the #1 database leader overall, Oracle is also the best database for microservices architectures and the best database for sharding. Microservices and Sharding are application development and data modeling decisions. Whether to consolidate those databases and how to consolidate them is a “macro-level” infrastructure management decision. In this article, we will explore these topics:

• Microservices Design Pattern: Database Per Service
• Sharding: Subdividing a Large Database into Shards
• Why Isolate Databases?
• How to Isolate Databases
• Don’t Over-Isolate: Limit Virtual Sprawl
• Stop Noisy Neighbors: Use Resource Manager
• Use Resource Shapes for Simplicity
• Reducing Blast Area Still Means a Blast!
• HA is Still Important With Microservices & Sharding
• Data Driven Apps: Benefits of Converged Database

Oracle is a Converged Database, providing the full range of database capabilities for modern Data Driven Applications. Oracle Database provides industry leading capabilities that exceed what point-solution databases can deliver.  You can deploy Oracle Databases as separate, dedicated databases supporting specific applications or microservices, or consolidate them to make more efficient use of system resources and administrative staff. Oracle Database can scale much larger than other databases, but can also scale-down to handle even the smallest microservices or sharding needs.

Microservices Design Pattern: Database Per Service

The Database Per Service Design Pattern says that each microservice has it’s own “database”, which means a collection of data and does not necessarily refer to a dedicated Oracle Database. In fact, (as outlined by Chris Richardson here) there are 3 ways for each microservice to have it’s own set of data as follows:

• Private Tables per Service
• Schema per Service
• Oracle Database per Service

Each microservice has it’s own set of data, and that data is only accessed through API’s belonging to the microservice. Whether a particular microservice has a set of tables, a dedicated schema, or an entire dedicated database generally depends on the SIZE of data, volume of activity and need for isolation. Microservices with larger amounts of data will tend to have their own databases, whereas smaller amounts of data and activity would argue for a dedicated schema or even simply a number of tables within a schema. Of course an Oracle Database can be a stand-alone database, but multiple databases can also be combined together as Pluggable Databases within a Container Database. We will look at this and other isolation topics in later sections of this post.

Sharding: Sub-Dividing a Large Database into Shards

I have worked on multiple applications where we sub-divided a massive logical database into many physical Oracle databases going back to the 1990s. The sharding concept is really nothing new, but Oracle Database now offers Sharding as built-in, out-of-the-box services that make building sharded databases and applications much simpler. Sharding features are built into the Oracle Database so that application developers don’t need to implement those features in application code.

At the end of the day, when you shard a database into multiple smaller pieces, you still have the same amount of data. If you shard a 70TB database into 10 shards of 7TB each, it’s still 70TB. If you shard that data into 100 shards, it’s still 70TB. Oracle is the KING of managing massive amounts of data, and it doesn’t matter if that’s 1 database or 100.

The point of sharding is that each database will be smaller and therefore easier to operate. You can then isolate those databases in order to improve operations.  Let’s look at WHY it’s sometimes necessary to isolate databases, then we will look at HOW to isolate them.

Why to Isolate Databases

Multiple microservices can reside within a single database, or might have dedicated databases (or Pluggable Databases within Containers), while shards are always spread across databases. Once microservice databases and shards are defined at the database layer, there are 5 factors that govern why those databases need to be isolated vs. consolidated as follows:

• Administrative Separation
• Security Separation
• Maintenance (patching and upgrades)
• Blast Radius (scope of failure impact)
• Resource Management

Administrative Separation refers to organizations that have multiple DBA teams or other needs to separate databases from an administrative standpoint. This is also common in SaaS (Software as a Service) providers, where customers of the service provider have administrative access, or the service provider has separate teams that administer databases. In either case, the multiple administrative teams means those databases need to be isolated from each other.

Security Separation refers to use of dedicated networking infrastructure to service specific databases (quarantine LAN, etc.), and subjecting certain databases to higher security standards such as for regulatory reasons (HIPAA, PCI, PII, etc.). Databases must be isolated in cases where security requirements differ widely between them.

Maintenance considerations include patching and upgrades of servers, databases, and any supporting infrastructure. As microservices are consolidated together more closely, any maintenance must be coordinated among those microservices and testing is often required to ensure compatibility. The biggest concern with maintenance is major version upgrades at the infrastructure layer (O/S, database, etc.), where there is greater potential for functional changes that might impact how services operate.

Blast radius (or fault isolation) considerations refer to the scope of impact any failure has. An outage of any single microservice can jeopardize the function of the application as a whole. However, smaller databases can be repaired more quickly than larger database, so limiting the scope of impact can result in faster service restoration during a failure.

Resource Management refers to ensuring each database receives the resources it requires, as well as guarding against the “noisy neighbor” problem. One approach to resource management is to isolate databases onto dedicated physical or virtual machines, but we recommend to simply use Oracle Resource Manager instead.

How to Isolate Databases

In addition to the reasons why to isolate databases, there is also the question of how the Database per Service Design Pattern and sharded databases are deployed at the physical infrastructure level. With Oracle, we have 4 options for how to isolate databases as follows:

• Dedicated Physical Servers per Database
• Dedicated Virtual Machine per Database
• Multiple Databases per Physical/Virtual Machine
• Multiple Pluggable Databases Per Container
• Oracle Resource Manager (DBRM & IORM)

We need to make sure developers have the autonomy they’re looking for, but then balance how they are deployed in the infrastructure. Managers of the infrastructure must determine how and when to isolate vs. consolidate databases. Excessive isolation such as the “one database per virtual machine” approach results in high cost of operation, so we recommend a more balanced approach.

Don’t Over-Isolate: Limit Virtual Sprawl

Data Centers in the past often suffered from sprawl (physical sprawl), where each application and it’s database(s) ran on their own physical servers. Virtual Machine technology has allowed IT organizations to stem the tide of physical sprawl, but this has often resulted in virtual sprawl. Increasing compute and storage density has also allowed more workload to be laid on top of the same physical footprint, using virtualization to maintain the same (or even greater) isolation between workloads.

You will quickly build an administrative nightmare if each microservice uses a dedicated database, and if each database uses a dedicated Virtual Machine. You can easily meet the needs of your application developers by taking a more judicious approach to deploying databases by using the full range of options at your disposal, using the right tool for the task such as using Oracle Resource Manager to manage resources.

Stop Noisy Neighbors: Use Resource Manager

There is no reason to isolate Oracle Databases simply for the purposes of resource management. We can easily ensure each database receives the appropriate amount of resources using Oracle Database Resource Manager (DBRM).  There are 4 primary resources that need to be managed in any system:

• CPU
• Memory
• Processes
• I/O

CPU is managed using Oracle Database Resource Manager (DBRM), using shares & limits inside of a Container Database, Instance Caging (CPU_COUNT) across containers and non-pluggable databases, and within databases using Consumer Groups.  DBRM gives us control of ALL types of databases and workloads, and it’s integrated with IORM (see below for detail).

Memory usage for Oracle databases is managed by controlling SGA (System Global Area) and PGA (Process Global Area) settings.

Processes within Oracle databases are managed by controls for sessions and parallel query servers.

I/O resources are much simpler to manage on Exadata because we have IORM (I/O Resource Manager). Exadata is integrated hardware and software, spanning both compute and storage. Using IORM Objective “auto” allows IORM to inherit the same resource ratios set for CPU in DBRM, so you have one set of controls that handles everything.

See our demo series on Resource Management (starting here) that shows how you can manage resources between pluggable databases, between container databases and non-pluggable databases, and even within databases using Consumer Groups.

Use Resource Shapes for Simplicity

Of course we recommend establishing a standardized set of “resource shapes” to choose from rather than making each database a unique and special snowflake. It’s much easier to manage a large population of databases if they all follow some standards, including the resources assigned to them. Using Resource Shapes means you establish shapes with the same ratios for the following resources:

• CPU
• Memory (SGA & PGA)
• Processes (Sessions & PQ Servers)
• I/O

I covered this in detail in the MAA Best Practices white paper on Database Consolidation available here. The white paper includes different allocations for DW vs. OLTP databases, so you’ll see 2 different tables of Resource Shapes. DW systems need larger PGA memory and more processes are devoted to parallel processing than on OLTP systems.  You will see those differences in the resource shape tables.

Reducing Blast Area Still Means a BLAST!

Reducing the “blast area” by dividing the data by each microservice and sharding large databases doesn’t ELIMINATE the blast (meaning “failure”). What a smaller blast area gives you is pieces that are smaller and easier to fix (because they are smaller). It’s also important to realize that a piece of an application dead in the water is still dead in the water.  If that piece of the application (a microservice) is critical, the entire application might also be dead in the water.

Sharding is another design that reduces the “blast area” of a failure, but an outage of a shard is still an outage.  It’s just easier to fix because it’s smaller, but it still needs to be fixed! In the best case scenario, you have SOME of your users dead in the water because the data they need (and the microservice) isn’t available.  In the worst case scenario, you will have some microservices or portions of your application that need ALL of the shards available for it to function, and those portions of the application are dead in the water.

In short, reducing the blast area doesn’t necessarily increase availability, but might help in reducing the duration of an outage.

HA is still Important with Microservices & Sharding

High Availability (HA) systems are still important, even for microservices and sharded databases. There can still be a “blast” that takes down a database, which takes down a microservice or shard, which will impact at least some of your users if not all of them depending on how your application uses that data.

Just because you have things broken down into small pieces doesn’t mean there’s no impact of an outage on any piece. It’s still important to provide HA, which we do with these tools:

• Core HA Features of Oracle Database
• Oracle Real Application Clusters
• Oracle Active Data Guard
• Oracle Application Continuity

Core HA features of Oracle Database have become something that we almost don’t even consider these days, but are still huge differentiators compared to other databases on the market. Oracle experts often take for granted things like online index rebuilds, online table move, and other features that have been developed over the years. I am often surprised to find other databases haven’t caught up, and some of the newer database engines are still years behind.

Oracle Real Application Clusters (RAC) is still unique in the market after all of these years. We often think of RAC for scalability, but it’s a huge part of the Oracle Database availability solution as well.

Oracle Active Data Guard is the Oracle solution for providing Read Replicas  (or “reader farms”), but it’s also a critical part of the Oracle Database availability solution. Active Data Guard provides for availability when a “blast” (failure) occurs, but it’s also used to provide availability during proactive maintenance.

Oracle Application Continuity hasn’t gotten enough press over the past 5+ years, but it’s yet another completely unique capability that Oracle offers. People seem to remember TAF (Transparent Application Failover), but Application Continuity is a completely different animal.  TAF required application code changes, while AC has essentially pushed much of that complexity into the SQL driver for Oracle.

Oracle Database has long exceeded the capabilities of applications, and Application Continuity closes the gap. Applications need to tolerate rolling outages of a RAC Cluster, and Application Continuity is what makes this happen. Application Continuity is implemented through CONFIGURATION changes, not within application code.

Data Driven Apps: Benefits of Converged Database

The Oracle Converged Database brings a number of critical advantages for data driven applications compared to point-solution databases.  See Juan Loaiza’s talk on Data Driven Apps at OOW London 2019 for a great talk on this topic. Although point-solution databases are sometimes referred to as “best of breed”, many of them don’t live up to that claim. You would suppose a database with a narrow focus might do that job better, but that’s not necessarily the case. The Oracle Converged Database advantages include:

• Data Modeling Flexibility
• Simplified Data Movement
• Portable Developer Skills
• Increased Developer Productivity
• Any Workload, Any Data
• Simplified Consolidation & Isolation
• Any Size, Any Scale

Data Modeling Flexibility comes from having a single database that provides the full range of data modeling needs. While developers might make the perfect decision and choose the exact point-solution database needed, requirements often change. A development team might choose a relational database, only to find that some data really needs a different modeling approach such as JSON Documents.

While Oracle Databases can be deployed using a SINGLE modeling technique (Relational, Document, Star Schema, Property Graph, etc.) you can also use MULTIPLE modeling approaches within the same Oracle Database. I have worked with data that didn’t fit easily into the relational model, so it’s great to be able to use JSON Documents or another technique that fits better for certain portions of your data model. It would be a huge pain (show-stopper really) if you had to switch databases or use another database for a small component like a single microservice.

Simplified Data Movement comes from using the same Oracle Database on both the sending and receiving side. That data movement is made vastly easier if the data resides with a single shared database, but is easier between Oracle Databases because both databases use the same tools, same drivers, same datatypes, etc. Data movement can be completely eliminated in cases where applications share the same data, such as combined OTLP and Reporting in a single database (yes, this is possible and quite common with Oracle). Data Movement can also be quite costly in Cloud environments, which is another benefit of using Oracle Converged Database rather than multiple point-solution databases.

Portable Developer Skills are a key benefit of the Oracle Converged Database, allowing developers to work on ANY microservice or application that uses the same database without extensive re-training. Developers who write analytic or Data Mining code can easily move into a development team working on transactional applications or vice-versa.

Increased Developer Productivity comes from having a common set of interfaces for all databases, regardless of what features are used within each database. Any feature provided by the database represents code that DOES NOT have to be written by a developer.

Any Workload, Any Data includes traditional OLTP and Analytic applications, but also includes Machine Learning, block chain, property graphs, Time Series analysis, Spatial, Internet of Things, and Event Processing. Oracle Database includes robust capabilities in these areas that have been proven over many deployments at customer sites worldwide.

Simplified Consolidation & Isolation comes by having databases that all run the same Converged Database. Administrators have the flexibility to isolate databases where needed, or consolidate them to ease the administrative burden as well as to improve resource efficiency. Oracle Databases can even be consolidated into a Container Database, for much greater density and lower cost of operation. Yes, this also applies to Cloud environments, where costs can easily escalate. The bottom line is you simply can’t do this with multiple divergent database engines such as one for Relational OLTP, one for Relational Star Schema DW, one for JSON Documents, etc.

Any Size, Any Scale database or workload can be handled by Oracle Database. It’s amazing to me that other databases haven’t caught up to Oracle after all these years. We still encounter customers who are faced with a database migration (to Oracle) because their chosen database engine simply can’t keep up. Oracle Database is able to easily support databases at least 10X larger than our competitors can handle, and the Exadata platform simply takes that number higher.

Microservices and Sharding increases the number of databases, while also reducing the maximum size of databases, but it’s great to know your can scale a database without worry if it’s running Oracle Database. You have time to adjust if you need to re-shard your databases or if your microservices databases grow unexpectedly. Having size limits that are 10X or even 100X higher than other databases gives you flexibility.

Microservices and Sharding DO NOT reduce the overall amount of data! In fact, the volume of data can increase somewhat if you’re duplicating any data across microservices or shards. In any case, the volume of data certainly isn’t less.

Summary

Oracle Converged Database gives you flexibility to deploy databases as needed, using microservices and sharding. Oracle is the #1 most widely used database overall, but also the #1 most widely used for microservices and sharding. Oracle Databases can be easily consolidated or isolated as needed to meet the needs of modern Data Driven Applications.

NoSQL databases (known as key/value, key/JSON, or “document” database) provide data that is self-describing, which is a huge improvement compared to older “record-based” databases. NoSQL databases are often compared to (purely) relational databases, but this is primarily because relational databases hold the biggest market. Record-based databases are really a non-entity these days, so there wouldn’t be much benefit in trying to “compete” against them.

The best comparison of NoSQL is against record-based databases.

Record Based Databases

Old fashioned “record-based” databases contained the data, while the structure of data was contained within the application. The database would contain a set of data values such as the following:

123, Chris, Craft, 123-555-1212

The application would know that these data attributes were as follows:

Record Key, First Name, Last Name, Phone Number

The application would typically contain a structure definition for the language the application was written in. If you go back far enough in history, this would have been Assembler or possibly COBOL language. The C/C++ language(s) would have used a struct contained in a header file.

This was fine in the days when ALL of the application code that touched the data was written in the same language. It was a huge pain when multiple applications accessed a common record-based database using different languages, such as a COBOL application that created data, and a C/C++ or Java application needed to access the same data. Someone needed to maintain structure definitions that were in-sync with each other, and it was a huge problem when changes were made to the structure.

Record-based databases included things like VSAM and ISAM. Neither of those databases used SQL, so they would certainly fit the definition of a NoSQL database. I realize the NoSQL people in this world aren’t thinking of VSAM and ISAM to be part of their orbit. Of course that’s a basic problem with the “NoSQL” term itself. Anything defined by what it’s not is very poorly defined.

Self-Describing Data

NoSQL databases typically contain self-describing data. JSON (Java Script Object Notation) is the most commonly used form of self-describing data, but there are other forms as well including XML. Rather than describing data in a struct within the application code, those structure definitions are kept within the data. The database provides a single source of truth for data structure, but also allows for much simpler modification of the data structure and code that access the data as the application and database evolve over time.

A JSON structure matching the previous example might look like this:

{
    "recordkey":"123",
    "firstname":"Chris",
    "lastname":"Craft",
    "phonenumber":"123-555-1212"
}

Of course every single record contains the field names, which consume more storage than the data in this example. Storage is cheap these days and we have much greater processing power than years ago, so there isn’t as much “data pressure” that would force us to define the structure inside the application. The reduction of “data pressure” allows us to use a method that’s less storage efficient to gain advantages in the application development side.

Data Pressure Considerations

The term “data pressure” essentially refers to the cost of storage and the cost of processing larger and larger volumes of data.  Record-based database were extremely efficient in terms of the space they required and compute power to process that data. Less data volume meant less data store and less data to process, so nobody would have considered making data self-describing.

Record-based databases were never self-describing because that simply was not practical decades ago. The cost of storage was simply too high and putting data descriptions into the data itself would have resulted in a huge amount of data duplication (duplication of the data definitions stored within each record), and there would not have been enough compute power on earth to work with that data.

Comparing NoSQL to Relational

NoSQL databases are more about the structure and representation of data than about the  absence of SQL language. Part of this is the same issue of object/relational impedance mismatch that has been around for decades. The rise of object-oriented programming resulted in application code that simply didn’t fit well with data modeled relationally.

Oracle Database supports Key/JSON

The SQL language does not work only with relational data. We can certainly use SQL to access a 2 column table containing a KEY (column #1) and JSON structure (column #2). Oracle Database includes the ability to model data in key/JSON format. Oracle databases might contain nothing but key/JSON data, some relational and some key/JSON, or JSON can be used to extend a relational model.

Oracle is a Converged Database

Oracle Database is a called a “converged” database because it supports virtually any data modeling approach and supports virtually any workload, all within a single database or deployed as separate databases.  Oracle is a multi-model database that supports relational, graph, time series, key/JSON, row-oriented, column-oriented (on Exadata), Geospatial data, normalized models, dimensional models (star schema, snowflake schema, etc.) and others.  Oracle Database supports all workload types including OLTP, DW, and IoT within a single database.

Oracle database uses the SQL language to process what some consider to be “NoSQL” format data (like JSON).

All data has relationships to other data. You can easily see this by simply looking at any piece of data.  All data has relationships to other data such as:

• Customers place Orders
• Orders are for Products
• Products are in Warehouses
• Warehouses have Employees

The list of relationships goes on and on, ad infinitum. Ensuring relationships are tracked is critical to data integrity. What happens if a customer orders a product that doesn’t exist? What happens if products go missing from a warehouse?

Databases Simplify Applications

The raison d’être for databases has always been to simplify application code. Anything done by the database is something the application code doesn’t have to handle. The basic functions of a database through the history of IT has been these:

• Preventing Data Loss
• Controlling multiple user access to data
• Enforcing data values are correct
• Ensuring data relationships are valid
• Data Security

These functions have been done by databases for decades because it simplifies application code.

Preventing Data Loss

Databases protect data, which is the lifeblood of an organization. Preventing loss of data has been key to survival even in the days of paper records and stone tablets. Electronic records are subjected to loss from a wide variety of failures, and databases are responsible for providing protection against these failures. Data loss might mean loss of the ENTIRE database, but also means changes from one user getting destroyed by another user.

Controlling Multiple User Access to Data

Without databases, applications would face extreme complexity in controlling access to data by multiple users. Access to data includes reading data, but also includes changing that data. Applications would be horribly complex without the concurrency controls provided by databases. Users also shouldn’t have access to changes made by others until those changes are complete.

Enforcing Data Values are Correct

The correctness of data values includes enforcing number attributes contain numeric values, date attributes contain valid dates, JSON strings contain valid JSON, GPS locations are valid locations, countries are valid countries, etc. Without this enforcement by the database, errors in application code will introduce corrupt data, which will have a cascading impact on other application code. If the database contains an order date of February 30th, that’s clearly wrong and it’s difficult to determine how that value got inserted into the database. It’s much better to catch that when it happens and prevent the change.

Ensuring Data Relationships are Valid

Relationships represent a special class of data values that are enforced by the database. This goes to the heart of why databases were created, and why the relational database became the leader. Hierarchical databases could only enforce certain data relationships. Network databases were a little better, but also had limitations. Relational databases (invented by Edgar F. Codd) provided much greater capability to model data relationships and ensure those relationships were reflected in the data.

Data Security

Of course databases play a central role in the security of data, including who can view data as well as who can make changes to data. This includes not simply access to the entire database, but down to individual attributes as well. There are different access controls over who can view customers, who can change customer data, and who can view or change customer addresses and credit cards.

All of this complexity can be pushed into the application code, but we often want separate data access controls enforced by the database. The principal is known as “defense in depth”. Fort Knox has the gold inside a vault, inside a building, surrounded by a fence, patrolled by guards. You certainly wouldn’t want a Fort Knox with the gold sitting in the middle of a field with guards running around trying to protect it. Defense. In. Depth.

Normalization & Denormalization

Data often begins in a denormalized form or at least we often start to think about data that’s not normalized, so we typically “normalize” the data when designing a relational data model. We then “de-normalize” data primarily for performance reasons. One example might be phone numbers. Customers often have multiple phone numbers such as home, work, and cellular phones.

In a normalized data model, each of a customers’ phone numbers would be stored in separate rows of a phone number table related to the customer. However, that means accessing extra rows of data when you want to show them all. To improve performance of accessing ALL of the rows, we might simply put separate home, work, and cellular phone numbers on the customer table. However, what happens when a customer has 2 cell phones?

Phone numbers are a great case for the combination of JSON within an otherwise relational database.  Rather than modeling N number of phone numbers in the customer table, we might create a PHONE_NUMBERS column as a JSON structure.  That design allows a variable number of phone numbers for each customer.

Database Optimization

I heard someone say recently that relational databases aren’t optimized for anything, which is a complete misrepresentation of reality. Every table in a normalized relational data model should have a primary key. That primary key will have an index. Access to a row of data through the index is “optimized” and will give good performance.

The database “engine” does not provide the logical structure of data. Database “engines” (historically) have not automatically generated indexes beyond those needed to enforce constraints such as primary keys and unique keys. Building of secondary indexes (those beyond what’s needed to enforce constraints) has always been the responsibility of the database administrator, data modeler, or application developer. So yes, in a sense the database is not “optimized” out of the box if you don’t optimize it!

Oracle Autonomous Database is the first on the market that is doing this sort of optimization and automatically handling what has traditionally been the responsibility of the DBA, data modeler, or application developer.

Does the emergence of the Cloud mean that “purpose-built” databases make more sense, or is an All-Inclusive database still the best choice?

There are multiple purpose-built databases on the market that handle specific types of data, specific workloads, or have some specialized functionality.  Examples of purpose-built databases include:

• Relational OLTP Databases
• Relational Data Warehouse Databases
• Key/Value Databases
• Time Series Databases
• In-Memory Databases
• Graph Databases

Of course there are other examples, but these are the most common.  The majority of OLTP databases are relational, and relational data modeling is the norm for Data Warehouse databases as well.  Key/value databases have gotten more popular in OLTP applications that involve high frequency “store & retrieve” operations on relatively simple and often dynamic structures.  Key/Value databases (using JSON or XML structures) have the advantage of a non-relational data model that can be quite dynamic.  Time Series databases have emerged with a focus on forecasting.  In-Memory databases deliver high performance, and Property Graph databases are focused on relationship attributes.

Easy Deployment

In the Cloud, organizations can easily deploy multiple divergent technologies, so does it make more sense to use purpose-built databases in the Cloud?  In an on-premise environment, deploying another database means buying and deploying yet another server.  Some Cloud vendors will also provide operational support or a “managed service” for each database, so why not?

Of course it’s not only a matter of deploying the database itself.  Each divergent database engine will require it’s own client-side drivers to establish connectivity.  Each of those clients will require a tie-in to any centralized Identity Management service as well.

Operational Burden

In years past (and with on-premise data center environments currently), OPERATING multiple purpose-built databases was always a big disadvantage.  Each database engine requires an IT organization to build expertise in that engine, deploy servers and storage to run those databases, as well as secure and maintain them all.  Moving to a Cloud environment, some of the operational burden still falls on the IT organization, even if the database is operated as a “managed service” provided by the Cloud vendor.

Security Burden

Each separate database technology represents yet another technology that needs to be secured.  Each database has a different security model, with it’s own unique capabilities and quirks.  Each database can be expected to have it’s own set of security flaws and bugs that need to be tracked and managed for proper compliance.  Even if the Cloud vendor is responsible for applying security patches, the IT organization needs to be aware of the flaws and making sure vulnerabilities are addressed.  Trust your Cloud vendor, but VERIFY they are taking care of security!

Building Developer Skills

Each divergent database technology requires specialized developer skills.  Developers need to build expertise in the data modeling capabilities, the functional capabilities, and the business theories that are involved.  In the case of analytics, this often includes expertise in statistical analysis such as auto-regressive modeling for time-series forecasting.

Islands of Data

Multiple divergent database technologies results in islands of data that often needs to be integrated.  Movement of data between databases takes time and induces lag.  If operational reports are executed only from a Data Warehouse, those reports will be incorrect until data from the OLTP system reaches the Data Warehouse.

Islands of data become a big problem if it’s sub-divided too much, becoming a series of  data islands.  Deploying multiple read-replicas of data also isn’t always a solution to every situation, since those replicas need to be kept updated.  Read Replicas are a great solution for data with effectivity dates, but a very poor solution for data with frequent updates.

Multiple islands of data ultimately induces delays in data updates across applications, modules, or even micro-services if each is using a separate data store.  The most obvious case is the time-delay induced by ETL processing between OLTP systems and Data Warehouse systems.  The time-delay is fine for long-term analytics, but unacceptable for operational reporting.

Islands of Functionality

Of course each purpose-built database includes a set of specific functionality.  If a key/value database is used for the CUSTOMER ORDERS database, it’s a problem if some relational modeling capabilities are needed in that database.

Key/Value databases have been presented as if they were something new, but are decades old. Key/value databases have always been FAST for ingesting data because they are simply storing a structure in a single “write” (or insert) operation.  Some of the earliest databases used a key/value modeling approach, which becomes extremely cumbersome with highly complex data relationships.  What’s new in key/value is the ability to model data structures using JSON or XML, including the ability to ENFORCE the JSON or XML structural integrity.

All-Inclusive Database Approach

The All-Inclusive database approach puts all of this functionality into a single database engine.  Developers can use ANY of the available functionality within ANY database instead of having to move the data to a place where the functionality resides.  Developers simply use the functionality in-place.

As the following diagram illustrates, Oracle includes ALL of this functionality in a single database engine.  Developers working on an OLTP database can use capabilities of Data Warehouse databases within the same database.  The data doesn’t have to be moved to take advantage of such features.  If data nee

One Engine, Not One Database!

Notice that this means ONE ENGINE, not one database!  Just because you could put all of your data into a single database doesn’t mean you should.  In fact, you could use separate Oracle databases for each type of data & functionality as follows:

Of course this approach doesn’t necessarily make sense either.  Nobody does data architecture like this.  Each database is created to handle specific applications or sub-components of an application.  Developers then use whatever functionality makes sense to handle the business needs of that application or sub-component.

Data Architecture

Data Architecture for an organization approaches the problem from the standpoint of business processes and data ownership.  Data architecture should not include limits on data modeling capabilities or limits on how the data can be processed.  Consider the following sample data architecture:

Notice that Active Data Guard is used for most databases to provide availability, as well as for Read Replicas in the case of the Product Catalog.  Active Data Guard is even used for the ORDERS database, even though it’s a sharded database.  While sharding reduces the failure impact (or “blast area”) of the database, each shard still represents a single source of truth for the range data it contains.

If this data architecture included limitations in functionality, the entire architecture gets much more complex.  For example, the ORDERS database mainly uses key/value data modeling to increase agility into the data model, as well as improving response time.  Submitting an order involves a single WRITE (or INSERT) of a single record that contains a complex JSON structure.  Processing each order means a single READ (or SELECT) of a single row of data containing that JSON structure.  However, if we need a bit of relational functionality within this database, that’s certainly possible.  The main function of the database uses key/value, but developers aren’t limited to just that single data modeling capability.

All-Inclusive in the Cloud

The Oracle database has always been a rather LARGE chunk of code.  It can be hard to install and takes up lots of space.  However, once it’s up and running, Oracle has enormous depth in capability.  My original assumption was that Cloud might tend to favor purpose-built databases.  Of course you will hear that from vendors who lack an all-inclusive database engine of their own.  However, it turns out that Cloud still does favor the All-Inclusive approach.

Islands of data & functionality is still a disadvantage, even in the Cloud.