You Buy a NUMA System, Oracle Says Disable NUMA! What Gives? Part II.

BLOG UPDATE ( 19-JUN-2009): I need to point out that ML 759565.1 has been significantly revised. The message regarding test before enabling NUMA persists. Not that it matters much,  I concur with that advice. The original post follows:

Cart Before Horse?
Yes, in this mini-series of posts Part II will precede Part I. I’ll explain…eventually.

In the comment thread of  my recent blog post entitled Oracle Database 11g Automatic Memory Management – Part IV. Don’t Use PRE_PAGE_SGA, OK?, a reader asked if I’d comment on Metalink note 759565.1 (759565.1). The reader’s specific interest was in the late breaking stance Oracle Support has taken regarding software NUMA optimizations. Just what is that support position? Run with all NUMA software optimizations disabled. Full stop.

I have been in contact with the Support Engineer that “owns” ML 759565.1 and have given some advice on how to change that note. I’ve been informed that a re-write is underway and that I will be on the review for that revision. That’s good, but I still think this topic is worthy of your time (feeling especially presumptuous today I guess).

The current rendition of the support note reads:

Oracle Support recommends turning off NUMA at the OS level or at the database level. If NUMA is disabled at the OS level then it will also be disabled at Oracle level. It does not matter how NUMA is disabled as long as it is disabled either at the OS layer or the Database layer.

What’s that? Run a NUMA system with all software optimizations disabled? Gasp! But wait, given how much I’ve rambled on about NUMA and NUMA software optimization principles over the years you will surely be flabbergasted to find out that, in principle, I agree with this stance.

What Is It Exactly That I Agree With?
Herein lies the rub. I have to offer a lengthy diatribe about NUMA in order to explain what I agree with. Not all NUMA systems are created equal. NUMA system fall into one three camps:

• Pioneer, Proprietary NUMA Implementations (PPNI).
• Modern, Proprietary NUMA Implementations (MPNI).
• Commodity NUMA Implementations (CNI).

 

Different NUMA Implementations == Differences!
Let’s see if I can make some sense of these differences. And, trust me, this does relate to ML 759565.1.

1)      Pioneer, Proprietary NUMA Implementations (PPNI). The first commercial cache-coherent NUMA system was the Sequent NUMAQ-2000. Within a couple of years of that hardware release there were several other pioneer implementations brought to market by DG, DEC, SGI, Unisys and others. The implementation details of these pioneer NUMA systems varied hugely (e.g., interconnect technology, levels of OS NUMA awareness, etc). One thing these pioneer implementations all shared in common was the fact that they suffered huge ratios between local and remote memory latency. When I say huge, I’m talking as much as 50 to 1 for highly contended multiple-hop remote memory. The only reason these pioneer systems were brought to market was because they offered tremendous advancements in system bandwidth. The cost, however,  was lumpy memory and thus software NUMA-awareness was of utmost importance.  I would consider systems like the Sun E25K to be “second-generation” pioneer systems. Sure, the E25K suffered memory locality costs, but not as badly as the true pioneer systems. Few would argue that even the “second-generation” pioneer systems relied heavily on software NUMA-awareness.

2)      Modern, Proprietary NUMA Implementations (MPNI). I’m not going to cite many systems here as cases in point. I don’t aim to wound the tender sensibilities of any hardware supplier. I can define what I mean by MPNI by simply stating that MPNI systems differ from PPNI in terms of remote to local memory latency ratios. In short, MPNI systems have very favorable L:R latency ratios. By very favorable, I mean significantly less than 2 to 1. An example of an MPNI system would be the Sun SPARC Enterprise M9000 Server which, according to my good friend Glenn Fawcett, sports an approximate local to remote latency ratio of 1.3:1. In my opinion, it is not worth the complexities necessary to do proper, effective software  NUMA awareness when there is only 30% disparity between the local and remote memory (at least not Oracle NUMA-awareness). Now, having said that, I know the M9000 supports scaling to multiple cabinets. I don’t know enough about the crossbar (Juniper Interconnect) to say whether it requires any “hop” overhead in a multiple-cabinet configuration. Sun literature states point-to-point without caveats so the L:R ratio might remain constant as one adds cabinets. Nonetheless, the point being made here is that there exist today modern, proprietary NUMA implementations and concerns over Oracle NUMA awareness should be weighed according to MPNI capabilities—not arcane, PPNI capabilities.

3)      Commodity NUMA Implementations (CNI). I don’t feel compelled to hide my exuberance for modern NUMA implementations such as Intel QuickPath Interconnect and the HyperTransport (HT) used by AMD. The points I want to make about CNI are as follows:

1. Memory Latency Ratios. While I’ve not stayed as up to speed on local-remote ratios with HT 3.0, I know that the Intel QPI-based systems offer very pleasant L:R ratios (e.g., 1.4:1 or better). More importantly, I should point out that even remote memory references in Nehalem-based Servers (Xeon 5500) are faster than all memory references in the previous generation Xeon-based systems (e.g., “Harpertown” Xeon 5400)!
2. BIOS-Enabled NUMA. Commodity NUMA systems support the concept of boot-time NUMA enablement. When booted with NUMA disabled at the BIOS, the resultant memory architecture is commonly referred to as Sufficiently Uniform Memory Access (SUMA) or Sufficiently Uniform Memory Organization (SUMO).

Does All This Really Relate to ML 759565.1?
Yes. While I haven’t seen the re-write of that note, I’ll say what I think it needs to say:

1. Regarding Commodity NUMA Implementations.
   1. If you are running Oracle on a CNI system you should test before you even bother enabling NUMA in the BIOS—not the other way around. If you deploy a CNI system, such as a two-socket Intel Xeon 5500 (Nehalem) server to run Oracle, I assert that you would have to do significant testing to find much of a performance difference between disabling NUMA in the BIOS and a fully-NUMA-aware configuration (i.e., NUMA BIOS=on + OS NUMA + on Oracle enable_NUMA_optimization=TRUE). That is, you will likely have to go to significant effort to find a performance delta in the range of anything greater than about 10%. It will be extremely workload dependent (in reality I’m aware of test results that show 10% improvement with NUMA disabled in the BIOS). Having said that, I’m not sure a 10% improvement would be worth the Linux-specific issues associated with setting enable_NUMA_optimization to TRUE. I’ll gladly take my medicine from anyone that can show me more than, say, 10% with all the bells and whistles (and associated thorns and barbs) of a fully NUMA-aware Oracle Database 11g deployment. Remember, I’m speaking specifically about CNI systems and let’s keep track of the publish date of this blog entry because the number of sockets can bear significance on this topic as I’ll point out later in this series. When I talk about a SUMA/SUMO approach to Oracle on CNI, I mean single-hop memory, which should be the case up to at least 4 sockets, but I don’t know since QPI systems are limited to two sockets today.
      1. What are the “thorns and barbs” I allude to? Well, it’s all about OS NUMA sophistication with specific regard to when, or if, to remotely execute a process, when an imbalance warrants a process migration and how often such things should occur. The best NUMA-aware Operating System ever developed (DYNIX/ptx) had these sorts of issues iron out flat. We (Sequent) had these sorts of issues directly in our fully focused rifle scopes and we weren’t messing around. Who is scrutinizing such issues in today’s world with CNI systems? After all, the “thorns and barbs” I allude to are dealt with in the Operating System. Do you know many Linux guys that are scrutinizing low-level systems performance issues specific to Oracle on CNI with bus traces  and all? That’s not exactly a saturated research field 🙂 There is a reason the term SUMA/SUMO exists!

1. Regarding Modern Proprietary NUMA Implementations.
   1. If you are running Oracle on a MPNI system, take heed to Oracle Support Services’ recommendation to run with Oracle NUMA optimizations disabled (enable_NUMA_optimization = FALSE) as is the default with patch 8199533 applied. Based on testing and analysis, it may make sense to enable NUMA awareness in Oracle. That will depend on the workload and the degree of non-uniformity of your MPNI system. Talk of disabling NUMA in the OS will likely not relate to your platform. More on this latter in this post. Talk of disabling NUMA at the hardware level (akin to the BIOS CNI approach) is most likely irrelevant.
2. Regarding Pioneer Proprietary NUMA Implementation.
   1. If you are running Oracle on a PPNI system, I’d like to buy you a beer! That would be a pretty old system with very old software. But, vintage notwithstanding, it would likely be a pretty stable system.

That Middle Ground: MPNI with enable_NUMA_optimization = FALSE
It’s true. Running Oracle Database with its NUMA-awareness disabled on a MPNI system is a sort of middle ground. It so happens that this particular middle ground makes a lot of sense—and you likely have no choice either way. After all, I know of no MPNI system that allows you to run in a hardware-SUMA mode. It would be a bit of a chore to interleave memory across motherboards.

The Operating Systems that support MPNI systems generally interleave the memory allocations that back IPC shared memory and mmap()s. That is a good thing since the result is a fairness of access to the shared resource. More importantly, however, on really large NUMA systems (as most MPNI are) is the fact that private memory allocations (e.g., stack and heap) are allocated from the memory local to the NUMA node the process is executing on. Likewise, kernel structures affiliated with the process are allocated from local memory. So, the blend is a good thing on these types of systems. Having said that, however, I cannot vouch for what happens when there is CPU saturation on a NUMA node and the process scheduler is faced with a decision to remotely execute a process or just let idle cycles blow by. Moreover, I cannot vouch for these Operating Systems’ intelligence regarding when to just migrate a process over to a less-saturated NUMA node re-homing all the physical memory backing the process. Getting that stuff right is pretty hard work. I’ve got the T-Shirt to prove it.

I’m going to call this Part II and follow up with Part I soon. When you read Part II that move will make sense.