Configuring Linux Hugepages for Oracle Database Is Just Too Difficult! Isn’t It? Part – II.

After my recent blog entry entitled   Configuring Linux Hugepages for Oracle Is Just Too Difficult! Isnt It? Part I, I engaged in a couple of email threads and a thread on oracle-l about when to employ hugepages.  In those exchanges I was amazed to find that it is still a borderline issue for folks. I feel it is very cut and dried and thus I prepared the following guidelines that more or less spell it out.

  1. Reasons for Using Hugepages
    1. Use hugepages if OLTP or ERP. Full stop.
    2. Use hugepages if DW/BI with large numbers of dedicated connections or a large SGA. Full stop.
    3. Use hugepages if you don’t like the amount of memory page tables are costing you (/proc/meminfo). Full stop.
  2. SGA Memory Management Models
    1. AMM does not support hugepages. Full stop.
    2. ASMM supports hugepages.
  3. Instance Type
    1. ASM uses AMM by default. ASM instances do not need hugepages. Full stop.
    2. All non-ASM instances should be considered candidate for hugepages. See 1.1->1.3 above.
  4. Configuration
    1. Limits (multiple layers)
      1. /etc/security/limits.conf establishes limits for hugepages for processes. Note, setting these values does not pre-allocate any resources.
      2. Ulimit also establishes hugepages limits for processes.
  5. Allocation
    1. /etc/sysctl.conf vm.nr_hugepages allocates memory to the hugepages pool.
  6. Sizing
    1. Read MOS 401749.1 for information on tools available to aid in the configuration of vm/nr_hugepages

To make the point of how urgently  Oracle DBAs need to qualify their situation against list items 1.1 through 1.3 above, please consider the following quote from an internal email I received. The email is real and the screen output came from a real customer system. Yes, 120+ gigabytes of memory wasted in page tables. Fact is often stranger than fiction!

And here is an example of kernel pagetables usage, with a 24GB SGA, and 6000+ connections ..  with no hugepages in use .

# grep PageT /proc/meminfo

PageTables:   123731372 kB

Link to Part III in this series:

Configuring Linux Hugepages for Oracle Database is Just Too Difficult! Isn’t It? Part – III.

28 Responses to “Configuring Linux Hugepages for Oracle Database Is Just Too Difficult! Isn’t It? Part – II.”


  1. 1 dombrooks October 21, 2010 at 7:56 am

    Hi Kevin,

    I think there is a lot of confusion around this issue. I was reviewing an AMM versus Hugepages decision and the advice (from everywhere) was less than clearcut.

    I just had a conversation last week when I was asking how the decision came about that the standard 11g builds within my client were AMM not Hugepages – which I can understand from a generic systems build perspective. But in fact it turns out that none of their 11g systems use hugepages.

    In fact, one of the things quoted back to me about the decision in favour of AMM was your article:
    http://kevinclosson.wordpress.com/2007/08/23/oracle11g-automatic-memory-management-and-linux-hugepages-support/

    The conclusion from the engineering team team was that for 10g, hugepages is better for large SGA + large session count but for 11g, AMM was better.

    At the end of the day, people should do their own tests, come to their own conclusions for their own circumstances.

    However, just to be more cut and dried, is your advice above the same regardless of 10g, 11gR1 or 11gR2?

    Thanks.

    Cheers,
    Dominic

    • 2 kevinclosson December 3, 2010 at 12:55 am

      Dom,

      First, I need to apologize because I think your comment got trapped up in SPAM protection. Anyway, I think I like the following quote:

      “At the end of the day, people should do their own tests, come to their own conclusions for their own circumstances.”

      Here, here!

      >>However, just to be more cut and dried, is your advice above the same regardless of 10g, 11gR1 or 11gR2?

      The answer to that would be a resounding yes.

  2. 3 maclean October 21, 2010 at 8:01 am

    Nice advise. I also don’t recommend to configure Linux Hugepages;
    Full stop.

  3. 5 dombrooks October 21, 2010 at 8:02 am

    Actually, to be even more cut and dried, do you have guideline definitions for “large numbers of dedicated connections” and “large SGA”?

    Cheers,
    Dominic

    • 6 kevinclosson October 21, 2010 at 3:34 pm

      dombrooks,

      I should revisit that Aug 2007 post because it does sound like an endorsement of AMM. That was really old 11g software and the topic deserves another visit. Perhaps on X2-8 so we get a good sprinkling of NUMA in the mix as well.

      Here is how I’d like to put this:

      1. If 1.3 (in the list above) is the most important criteria and the performance you see with AMM is otherwise acceptable then stay with AMM. Indeed, ASM instances do just fine with AMM.
      2. If AMM is sufficient in your deployment then you either have a small SGA and few connections or a large SGA and fewer connections.

    • 7 kevinclosson October 21, 2010 at 3:36 pm

      dombrooks,

      “Large number of dedicated connections” and “large SGA” in the context of hugepages can only be quantified by the amount of memory wasted in page tables and whether the administrator is satisfied with that cost.

  4. 8 Tim Hall October 21, 2010 at 5:21 pm

    Hi.

    I was just going to ask about your recommendation of hugepages when your previous article on AMM implied it’s acceptable compared to hugepages. Now you’ve gone an answered my question before I’ve asked it I can’t think of anything intelligent to say. :)

    Cheers

    Tim…

    • 9 kevinclosson October 21, 2010 at 5:30 pm

      Hi Tim,

      Thanks for stopping by. I’m about to blog a fresh piece about AMM + hugepages. In short, however, the whole topic boils down to how much RAM people are willing to spend on overhead.

  5. 11 Anindya October 22, 2010 at 6:56 pm

    Hi Kevin,

    Do your recommendations hold good for Linux or for Unix flavors also? Just asking the question since I have been working at multiple customer locations running AIX and have not seen any of them use “Large Pages” as such. Most of them do have heavily used OLTPs running with 100+ GB SGAs also.

    thanks,
    Anindya

    • 12 kevinclosson October 22, 2010 at 11:20 pm

      Anindya:

      Most Unix offerings (count them on one hand these days) have large page support and fewer (if any) administrative difficulties involved. But, yes, the answer to your question is large page table entries for shared memory between a lot of processes is generally a good thing.

  6. 13 orainternals November 5, 2010 at 2:14 pm

    Kevin

    I am glad to see this post from you.

    It is surprising to see how many performance problems / server hang situations can be resolved by configuring HugePages in Linux. I blogged about one such issue in this blog entry:

    http://orainternals.wordpress.com/2008/11/13/performance-tuning-hugepages-and-linux/

    As you pointed out, it should be a rule: If you use Oracle on Linux and care about performance or scalability, use HugePages.
    Surprisingly there is hesitation from Linux Admins and DBAs to configure HugePages. Your post should alleviate some of that hesitation.

    I have a question though: In a fully virtualized environment (such as Linux clients running on IBM Z/OS host), do you see any issues setting up HugePages?

    Thanks,
    Cheers
    Riyaj

    • 14 kevinclosson November 5, 2010 at 6:29 pm

      Hello orainternals,

      I do see an issue using hugepages on virtualized linux clients in a z/OS environment. That issue, however, is simply that I don’t know anything about virtualized linux clients running on z/OS :-) Sorry. Thanks for stopping by!

      • 15 John Hurley November 8, 2010 at 12:53 am

        We are running hugepages … and also using ASM.

        Does ASM use/allocate hugepages by default or not? My ASM instances are pretty vanilla no customization of init parameters.

        It sounds like it would be bad if ASM instance did use hugepages under default config.

        Currently on 11.1.0.7.5 on OEL 5.5 64 bit.

        Thanks in advance.

  7. 17 John Hurley November 8, 2010 at 9:09 pm

    So the Oracle ASM software is smart enough to know when running in an AMM configuration to not use hugepages? Sorry that’s the part I as apparently missing.

    How about a regular database instance? If you try to run it using AMM ( mine are all configured fixed sizing ) will it ignore completely the hugepages?

    Guess I should do the testing eh?

  8. 19 Scott November 9, 2010 at 4:06 am

    Kevin.
    Can you comment if future versions of Oracle will support hugepages and AMM?

  9. 21 Andre M November 12, 2010 at 8:17 pm

    Hey Kevin

    Have you had a look at transparent largepages?
    http://lwn.net/Articles/359158/

    Looks like you’d basically get huge/large pages automatically?

  10. 22 beitinsw May 8, 2012 at 4:10 am

    Hi Kevin, just a maddening observation in Centos 6.2 (I assume this didn’t happen earlier): setting memlock in limits.conf results then starting Oracle in a “full crash report” being sent to root. This is actually just a warning stating that “Using mlock ulimits for SHM_HUGETLB deprecated”.

    This is listed as a known issue in Oracle Linux (http://oss.oracle.com/el6/docs/RELEASE-NOTES-GA-en.html) with the workaround being 1) configure the application to use CAP_IPC_LOCK or 2) add the process to hugetlb_shm_group. This is really out of my element here, but it appears I got it working properly by granting CAP_IPC_LOCK to $ORACLE_HOME/bin/oracle thusly:

    [root@server bin]# setcap CAP_IPC_LOCK+ep oracle

    Oracle starts, Hugepages are used, and no “crash report” appears. I’m not positive that it’s all “proper” however. It’ll be fun when we go into production with this machine.


  1. 1 eagle’s home » Linux Hugepages Trackback on October 24, 2010 at 11:08 am
  2. 2 eBay DBA Blog » Linux Hugepages Trackback on October 24, 2010 at 11:20 am
  3. 3 HugePages Overhead « OraStory Trackback on May 30, 2012 at 12:08 pm
  4. 4 Log Buffer #205, A Carnival of the Vanities for DBAs Trackback on February 13, 2013 at 9:55 am
  5. 5 .:. Teracomp IT Consulting .:. » Blog Archive » Otimizando o desempenho do Oracle usando Hugepages no Linux Trackback on June 19, 2013 at 3:37 pm

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




EMC Employee Disclaimer

The opinions and interests expressed on EMC employee blogs are the employees' own and do not necessarily represent EMC's positions, strategies or views. EMC makes no representation or warranties about employee blogs or the accuracy or reliability of such blogs. When you access employee blogs, even though they may contain the EMC logo and content regarding EMC products and services, employee blogs are independent of EMC and EMC does not control their content or operation. In addition, a link to a blog does not mean that EMC endorses that blog or has responsibility for its content or use.

This disclaimer was put into place on March 23, 2011.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 1,981 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2013. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

Follow

Get every new post delivered to your Inbox.

Join 1,981 other followers

%d bloggers like this: