Oracle on Opteron with Linux-The NUMA Angle (Part VI). Introducing Cyclops.

This is part 6 in a series about Oracle on Opteron-based NUMA servers running Linux. The list of prior installments can be found through my index of NUMA-related posts.

In part 5 of the series I discussed using Opteron-based servers with NUMA features disabled in the BIOS. Running an Opteron server (e.g., HP Proliant DL585) in this fashion is sometimes called SUMA (Sufficiently Uniform Memory Access) or SUMO (Sufficiently Uniform Memory Organization). At the risk of being controversial, I pointed out that in the Oracle Validated Configuration listing for Proliant, the recommendation is given to configure Opteron-based servers as SUMO/SUMA. In my experience, most folks do not change the BIOS and are therefore running a NUMA system since that is the default. However, if steps are taken to disable NUMA on an Opteron system, there are subtleties that warrant deeper understanding. How subtle are the subtleties? That question is the main theme of this blog series.

Memory Latencies with SUMA/SUMO vs NUMA
In part 5 of the series, I used the SLB memory latency workload to show how memory writes differ in NUMA versus SUMA/SUMO. I wrote:

Writing memory on the SUMA configuration in the 8 concurrent memhammer case demonstrated latencies on order of 156ns but dropped 38% to 97ns by switching to NUMA and using the Linux 2.6 NUMA API.

But What About Oracle?
What is the cost of running Oracle on SUMA? The simple answer is, it depends. More architectural background is needed before I go into that.

SUMA, NUMA and CYCLOPS
OK, so SUMA is what you get when you tweak a Proliant Opteron-based server so that memory is interleaved at the low level. Accompanying this with the setting of numa=off in the grub.conf file gets you a completely non-NUMA setup.

Cyclops
NUMA enabled in the BIOS, however, is the default. If the Oracle ports to Linux were NUMA-aware, that would be just fine. However, if the server isn’t configured as a SUMA and you boot Oracle without any consideration for the fact that you are on a NUMA system, you get what I call Cyclops. Let’s take a look at what I mean.

In the following screen shot I have booted an Oracle10g SGA of 7584MB on my Proliant DL585. The system is configured with 32GB physical memory which is, of course, 4 banks of 8GB each attached to one of the 4 dual-core Opterons (nodes). Before booting this SGA, I had between roughly 7.6GB and 7.7GB free memory on each of the memory banks. In the following figure it’s clear that after booting this 7584MB SGA I am left with all but 116MB of memory consumed from node 0 (socket 0)—Cyclops!

NOTE: You may need to right click->view the image

cyclops1

Right, so really bad things can happen if processes that are memory-resident on node 0 try to allocate more memory. In the 2.4 Kernel timeframe Red Hat points out such ill affect as OOM process termination in this web page. I haven’t spent much time researching how 2.6 responds to it because the point of this blog entry to not get into such a situation.

Let’s consider what things are like on a Cyclops even if there are no process or memory allocation failures. Let’s say, for instance, there is a listener with soft node affinity to node 2. All the sessions it forks off will have node affinity to node 2 where they will be granted pages for their kernel structures, page tables, stack, heap and so on. However, the entire SGA is remote memory since as you can see all the memory for the SGA was allocated from node 0. That is, um, not good.

Hugepages Are More Attractive Than Cyclops
Cyclops pops up its ugly single-eyed head only when you are running NUMA (not SUMA/SOMA) and fail to allocate/use hugepages. Whether you allocate hugepages off the grub boot line or out of sysctl.conf, memory for hugepages is allocated in a distributed fashion from the varying memory banks. Did I say round-robin? No. Because I don’t yet know whether it is round-robin or segmented. I have to leave something to blog about in the future.

The following is a screen shot of a session where I allocated 3800 2MB hugepages after the system was booted by echoing that value into /proc/sys/vm/nr_hugepages. Notice that unlike Cyclops, the pages are allocated for Oracle’s future use in a more distributed fashion from the various memory banks. I then booted Oracle. No Cyclops here.

hugepages

Interleaving NUMA Memory Allocation
The numactl(8) command supports the notion of pushing memory allocation preferences down to its children. Until such time as the Linux port of Oracle is NUMA-aware internally—as was done in the Sequent DYNIX/ptx, SGI, DG, and to a lesser degree the Solaris Oracle10g port with MPO—the best hopes for efficient memory usage on a commodity NUMA system is to interleave the placement of shared memory via numactl(8). With the SGA allocated in this fashion on a 4-socket NUMA system, Oracle’s memory accesses for the variable and buffer pool components will have locality of up to 25%–generally speaking. Yes, I’m sure some session could go crazy with logical reads of 2 buffers 20,000 times per second or some pathological situation, but I am trying to cover the topic in more general terms. You might wonder how this differs from SUMA/SOMA though.

With SUMA, all memory is interleaved. That means even the NUMA-aware Linux 2.6 kernel cannot exploit the hardware architecture by allocating structures with respect to the memory hierarchies. That is a pure waste. Moreover, with SUMA, 100% of your Oracle memory accesses will hit interleaved memory. That includes PGA. In contrast, properly allocated NUMA-interleaved hugepages results in fairness in the SGA placement, but allocation in the PGA (heap) and stack for the sessions are 100% local memory! That is a good thing. In the following screen shot I coupled numactl(8) memory interleaving with hugepages.

interleave

Validated Oracle Configuration
As I pointed out, this Oracle Validated Configuration listing for Proliant recommends turning off NUMA. Now that I’m an HP employee, I’ll have to pursue that a bit because I don’t agree with it at all. You’ll see why when I post my performance measurements contrasting NUMA (with interleave hugepages) to SUMA/SOMA. Look at that Validated Configuration web page closely and you’ll see a recommendation to allow Oracle to use hugepages by tuning /etc/security/limits.conf, but neither allocation of hugepages from the grub boot line nor via the sysctl.conf file!

Could it be that the recommendations in this Validated Configuration were a knee-jerk reaction to Cyclops? I’m not much of a betting man, but I’d wager $5.00 that was the case. Like I said, I’m in HP now…I’ll have to see what all that was about.

Up Next
In my next installment, I will provide Oracle measurements contrasting SUMA and NUMA. I know I’ve said this would be the installment with Oracle performance numbers, but I had to lay too much ground work in this post. The mind can only absorb what the seat can endure.

Patent Infringement
For all you folks that hate the concept of software patents, here’s a good one. When my Sequent colleagues and I were working out the OS-requirements to support our NUMA-optimizations of the Oracle 8 port to Sequent’s NUMA-Q system, we knew early on we’d need a very rich set of enhancements to shmget() for memory region placement. So we specified the requirements to our OS developers. Lo and behold U.S. Patent 6,505,286 plopped out. So, for extra credit, can someone explain to me how the Linux 2.6 libnuma call numa_alloc_onnode() (described here) is not in complete violation of that patent? Hmmm…

Now for a real taste of NUMA-Oracle history, read the following: Sequent_NUMA_Oracle8i

11 Responses to “Oracle on Opteron with Linux-The NUMA Angle (Part VI). Introducing Cyclops.”


  1. 1 Herta April 19, 2007 at 1:03 pm

    Good series of articles. Thanks. Looking forward to the next one. Any chance of adding RAC to the picture? I like the way in which you start up a single database, but don’t see how I might apply that to a RAC environment running multiple databases.

    As to your concern about Oracle/HP recommending turning NUMA off: the page you refer to (http://www.oracle.com/technology/tech/linux/validated-configurations/html/vc_dl585_4node_fas9xx.html?rssid=valconfig) simply shows a “*validated* configuration”. It’s not a recommendation.

    Kind regards,

    Herta

  2. 2 Geoff April 23, 2007 at 11:38 pm

    We have 3 DL585 machines each with 4 2.4 Ghz opterons and 32GB ram. We had no idea of the recommendation to turn off the NUMA setting at the bios level and have therefore been running in standard NUMA mode. Apparently using huge pages has saved us. The machine most heavily burdened is running along quite happily.

    Issues: One database instance on a separate machine will simply not use the huge pages. Another one was using it and stopped after reboot. We are not sure why. During research, we stumbled here (luckily). We have been looking at the “validated configuration” on Oracle’s site and will not disable NUMA after reading your blog. Thanks!

  3. 3 Geoff April 23, 2007 at 11:43 pm

    Forgot to add: we’re running Oracle 10.2.0.2 on SUSE Linux 9 SP2.

  4. 4 Charles C. Figueiredo April 28, 2008 at 9:44 pm

    This is an excellent and very informative thread of blog, Kevin. Thanks!

    I’m curious, have you done any testing or have any insights into vm.numa_memory_allocator? There is little info available on this tunable that I’ve found so far, and before I start sifting through kernel source, I wanted to throw this out there. This was added to Update 7 of RHEL 3 (yes, I know, a bit dated at this point), but since Oracle recommends disabling NUMA on DL 585 gear, and I’m using that with RHEL 3 U8+, I’m curious about no NUMA vs. leaving it enabled in the BIOS and using this tunable. I’m wondering if it helps avoid the cyclops scenario…

    Thanks,

    -charles.

  5. 5 kevinclosson April 29, 2008 at 4:42 pm

    Hi Charles,

    It turns out that mention of a particular setting in a Validated Configuration doesn’t imply a support requirement. That aside, I really can’t comment on anything NUMA related in RHEL3 since I abandoned that long before I even got that DL585…which I no longer have since it is in an HP lab and I’ve moved on to Oracle…

  6. 6 kevinclosson April 29, 2008 at 4:44 pm

    Hi Charles,

    It turns out that mention of a particular setting in a Validated Configuration doesn’t imply a support requirement. That aside, I really can’t comment on anything NUMA related in RHEL3 since I abandoned that long before I even got that DL585…which I no longer have since it is in an HP lab and I’ve moved on to Oracle…

  7. 7 Dan May 19, 2010 at 2:44 am

    Patents are always fun to argue. I’m neither totally for or against them. If someone patents a new hardware technology
    and 50% of software engineers think of the same obvious ways to
    use it then I don’t think the first person should get a patent.

    One could say that cpu registers are a form of non-uniform
    memory access. They are certainly fast. Hence, would the ‘C’
    specifier ‘register’ be in violation of the patent because it is
    a local allocation of “memory”? :-)

    The gall of the Linux folks actually wanting to allow
    the use of “local faster memory” as local faster memory!
    Somebody patents a hammer and then patents pounding things with it.

    Was NUMA itself been patented? Or do all implementations
    pay a royalty/licensed?

    • 8 kevinclosson May 20, 2010 at 4:18 pm

      >>The gall of the Linux folks actually wanting to allow the use of “local faster memory” as local faster memory!

      There is nothing wrong with wanting to do any particular optimization. The question is about the implementation method. I’m not here to argue whether any of my patents or patents of former Sequent colleagues were sufficiently novel to warrant a patent because it’s a bit too late for that. It is my opinion, however, that if you are the first to do something novel, useful and clear of prior art you’ve probably got something worthy of a patent. There needs to be incentive for being the first.

      >>If someone patents a new hardware technology and 50% of software engineers think of the same obvious ways to use it then I don’t think the first person should get a patent.

      …and neither do I. A patent needs to cover something non-obvious. That is one of the criteria. Everything is obvious with 20-20 hindsight.

      >>Somebody patents a hammer and then patents pounding things with it.

      …The hammer and pounding analogy comes up all the time. I tend to look at it more like someone invents the hammer then someone later invents an improvement that in a single embodiment makes it 1) physically impossible for the user to hit his thumb, 2) physically impossible to bend a nail, 3) physically impossible to drive the nail in crooked and amplifies kinetic energy 3-fold during usage.

      NUMA itself is not patentable. If someone invented a NUMA system that automagically smoothed out all topology side-effects (e.g., latency, unfairness, etc) then I think that would be worthy of a patent and would make this specific patent topic conversation moot :-)


  1. 1 Oracle11g Automatic Memory Management - Part III. A NUMA Issue. « Kevin Closson’s Oracle Blog: Platform, Storage & Clustering Topics Related to Oracle Databases Trackback on August 24, 2007 at 10:53 pm
  2. 2 Xeon E5-2600 OS CPU To Core / SMT Thread Mapping On Linux. It Matters. « Kevin Closson's Blog: Platforms, Databases and Storage Trackback on April 25, 2012 at 8:58 pm
  3. 3 Can Oracle Database Release 2 (11.2.0.3) Properly Count Cores? No. Does It Matter All That Much? Not Really.. | Kevin Closson's Blog: Platforms, Databases and Storage Trackback on March 5, 2013 at 5:45 pm

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




EMC Employee Disclaimer

The opinions and interests expressed on EMC employee blogs are the employees' own and do not necessarily represent EMC's positions, strategies or views. EMC makes no representation or warranties about employee blogs or the accuracy or reliability of such blogs. When you access employee blogs, even though they may contain the EMC logo and content regarding EMC products and services, employee blogs are independent of EMC and EMC does not control their content or operation. In addition, a link to a blog does not mean that EMC endorses that blog or has responsibility for its content or use.

This disclaimer was put into place on March 23, 2011.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,027 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2013. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

Follow

Get every new post delivered to your Inbox.

Join 2,027 other followers

%d bloggers like this: