About

I am a Performance Architect in Oracle Corporation’s Systems Technology Group which is a part of Server Technologies. I am a member of the Oracle Exadata Storage Server development team.

Short BIO

Kevin is a Performance Architect in Oracle Corporation’s Server Technology Group. His 20 year career has included Engineering, Competitive Benchmarking, Support, and Application Development on Oracle SMP and Clustered platforms. His work prior to Oracle at HP/PolyServe, Veritas, IBM/Sequent and Altos was focused on scalability and availability of the Oracle server. His Oracle port-level work at Sequent led to his U.S. patents in SMP/NUMA locking and database caching methods. Kevin speaks frequently at Oracle conferences and is a member of The OakTable Network. Kevin is also an Oracle Employee Ace. In addition to book collaborations, his written works have appeared in Oracle Magazine, Oracle Internals Magazine, IBM Redbooks and SELECT.

This blog is called “Kevin Closson’s Oracle Blog: Platform, Storage, and Clustering Topics for Oracle Databases.” I blog about a wide array of topics that should be of interest to data center professionals interested in:

• Oracle Exadata Storage Server and the HP Oracle Database Machine
• Oracle on Intel and AMD based NUMA Servers
• Oracle on Linux
• Oracle Internals Topics Relating to Storage and I/O
• Clustering
• Storage Architecture
• Real Application Clusters
• General Oracle Internals Topics
• High Availability

Long Introduction
My experience with Oracle dates back roughly 19 years. I worked on Informix applications on Intel 310 and Altos systems before that. Not to mention the “weird stuff” like Control Data IPF2 on Cyber 180, Wang PACE on VS 100 and Honeywell TPS on DPS6  running GCOS.

The first position I had that allowed me to focus on Oracle was at Altos Computer Systems with Version 5 Oracle on 3 different Unix variants. Altos offered systems based on both Motorola and Intel processors. That seems like an eternity past, but even then my focus was on the platform.

Sequent Computer Systems recruited me in 1990. I spent the next 10 years there focused on Oracle. In Sequent Advanced Oracle Engineering I worked in the team that devised and implemented platform-specific enhancements to exploit Sequent’s SMP and later, NUMA architectures. I worked on all the high-end benchmark efforts, to include world record TPC-C and TPC-D, vendor benchmarks like BaaN BRU, and customer proofs of concept such as—what was at that time—the worlds largest open systems database (80 TB with NUMA-Q 2000, Oracle, EMC and British Telecom).

A lot of the interesting things we implemented at the port-level in Oracle were covered in my contribution to James Morle’s book Scaling Oracle8i: Building Highly Scalable OLTP System Architectures (James is a good friend and a founding member of the Oak Table network). During that time I also got to meet and work with other really good people in the Oracle community, people I respect highly such as Graham Wood, Anjo Kolk, Jeff Needham, Glenn Fawcett, Raghupathi Malige, Juan Tellez, Paul McKenney,  Ken Dove and so many others.

The bulk of my work in that group was the low-level analysis of Oracle workloads with a keen eye for optimization. Yes, we did propeller-head stuff like hooking up bus analyzers to collect address traces, CPI analysis, instruction trace analysis and code ordering for locality using many workloads.

While in Advanced Oracle Engineering at Sequent, I architected most of the NUMA optimizations in the Sequent port of Oracle such as the NUMA-optimized local buffer preference method (QLBP), Extended Cache (48 GB SGA on 32-bit x86 based systems aka Indirect Data Buffers) and others. I came up with a few good ideas worthy of US Patents. Implementing one of those patents in Oracle8i gained us about 50% improved throughput in TPC-C at 64 processors. Like I say on occasion, that period of time for me was spent scrutinizing really little things that moved really fast, very frequently. Because Sequent was the development platform for Oracle Parallel Server on Unix (7.0.9) and Oracle Intra-node Parallel Query Option, I had the opportunity to get deep into the internals of those features leading to my interest in clustering.

I shouldn’t fail to mention that Sequent was the development platform for Informix Parallel Data Query as well and I spent quite a bit of time doing performance benchmarking with the Dynamic Server Architecture and Parallel Data Query (PDQ) products. However, clustered systems for Oracle became one of my deepest interests.

However, along the way I took a detour from clustered systems.

In 1999, IBM bought Sequent Computer Systems. I stayed in the Oracle Engineering group I was in, but immediately began to focus on high-end RS/6000 systems with AIX. Bill Maron and Bret Olszewski of IBM’s Center for Advanced Studies were nice enough to sponsor my way onto some very formidable RS/6000 gear for performance analysis. One of the first projects I did was a side-by-side benchmark comparison of the RS/6000 S70 and the NUMA-Q 2000 using a very contentious Oracle workload that simulated insurance quote generation. That was interesting work, and I learned a great deal about the difference between “fast” systems and “big” systems.

Soon after IBM bought Sequent, I got tied up in the effort to port Oracle to what IBM and SCO and others hoped would be the standard Unix offering for IA64—Project Monterey. Since my focus would have been the performance acceptance of that port, I faced hurdles that I could overcome. For example, the port was being done using IA64 systems called Software Development Vehicles (not to be confused with the infamous Software SDVs of the time). No, the SDVs we had were little white boxes (that threw off a tremendous amount of heat) with very early IA64 processors in them. These SDVs made porting Oracle difficult since they crashed a lot. Moreover, there was no compiler or debugger! Nope, the compiles were done on SCO Unix boxes and the only debugger we could use required an AIX workstation. If I recall correctly, that debugger was an early rendition of IDEBUG, or at least that would make sense since we talked to guys in Toronto about problems with the debugger. No matter, the point was that the port was never going to finish given those conditions. And if it did, Oracle on Monterey would have been a lame duck since Monterey itself was a bit untimely–given the rapid acceptance of Linux at the time. Since Linux trumped Monterey, I moved on to Veritas—and thus my departure from clustering. You see, Sequent had a fully functional cluster-aware OEMed version of Veritas Volume Manager in the mid 1990s and a cluster filesystem based on Veritas VxFS in about 1997 yet Veritas themselves had neither. So leaving for Veritas at that time meant leaving clustering as I knew it.

My time at Veritas (in the Database Editions Group) was focused on the Oracle Disk Manager (ODM) Library implementation. Having an extensive background in Oracle internals, high-end SMPs and I/O performance enabled me to ensure the performance of the Veritas ODM library implementation. Most of my performance analysis was done on a 20-CPU Sun UE10K with 4 Fibre Channel HBAs. Those were pretty powerful systems and my responsibility was to hunt down opportunities for software optimization in the library and Solaris kernel side of the implementation. Generally that means critical code analysis. The problem is that locking issues are difficult to spot unless the processors are saturated—and it is very difficult to saturate processors with an I/O library without a formidable disk subsystem. Some may recall that this period of time in California’s Silicon Valley was being plagued by the “energy crisis” of the time. Companies were dealing with power cut-backs and rolling black-outs. It turned out that acquiring the 300+ hard disk drives to keep the 20-processor UE10K busy was just not going to happen. So, I got clever and partnered with a provider of Solid State Disk technology to augment the bit of regular SAN storage I had. Using this sort of storage in order to drive I/O-related code paths to the point of processor saturation proved to be a valuable methodology so I wrote a paper about the practice. The technology was fun to work with there, but I always appreciate even more the opportunity to get to know really good people and I am pleased to say that Peter Bestel was one of them! Veritas was rapidly changing with a new CEO on board, but even more so since the NASDAQ had just split open like a coconut. I was pleased to get a call from a group of former Sequent colleagues who were starting up PolyServe.

In 2001, I joined PolyServe as the Chief Architect of Oracle Solutions leading the effort to get the PolyServe clustering product suite fit for Oracle. That meant getting direct I/O worked out, implementing an ODM library for optimized Oracle I/O and later a failover HA solution for PolyServe customers not running Real Application Clusters—all with correctness, and high performance.

That brings us to now…