Quantifying Hugepages Memory Savings with Oracle Database 11g

In my recent post about physical memory consumed by page tables when hugepages are not in use I showed an example of 500 dedicated connections to an Oracle Database 11g instance with 8000M SGA consuming roughly 7 gigabytes of physical memory just for page tables. A reader emailed me to point out that it would be informational to show page table consumption with hugepages employed. How right.

The following script was running while I invoked sqlplus 500 times to connect to the same instance discussed in this post. As the script shows, page table cost peaked at roughly 265 MB compared to the 7 GB lost to page tables in the non-hugepages case.

I’m scratching my head and thinking of what else could possibly be said on the matter…

$ while true
> do
> ps -ef | grep oracletest | wc -l
> grep PageTables /proc/meminfo
> sleep 30
> done
1
PageTables:      23484 kB
501
PageTables:     264432 kB
501
PageTables:     264900 kB
501
PageTables:     265400 kB
501
PageTables:     265944 kB
130
PageTables:      83112 kB
120
PageTables:      78672 kB
110
PageTables:      74188 kB
100
PageTables:      69712 kB
90
PageTables:      65264 kB
80
PageTables:      60804 kB
70
PageTables:      56332 kB
60
PageTables:      51872 kB
50
PageTables:      47376 kB
40
PageTables:      42848 kB
30
PageTables:      37904 kB
20
PageTables:      32976 kB
10
PageTables:      28024 kB
1
PageTables:      23532 kB
1
PageTables:      23528 kB

14 Responses to “Quantifying Hugepages Memory Savings with Oracle Database 11g”


  1. 1 Bernd Eckenfels July 29, 2009 at 2:36 am

    So it does not keep the page table memory allocated after the process has quit.

    Oh BTW, I pointed to your numbers here:
    http://stackoverflow.com/questions/853736/how-to-find-or-calculate-a-linux-processs-page-table-size-and-other-kernel-accou/1182989

    Bernd

  2. 2 yafang gao July 29, 2009 at 8:05 am

    In our db machine,
    everytime when the db startup ,the HugePages_Rsvd may increased.
    Now it is 5963,and can not to release and cant not used by oracle.
    Only todo is reboot machine to release it.
    Are their other method to decrease it except reboot ?

    thanks a bunch!

    [root@dgpubdb3 ~]# cat /proc/meminfo
    MemTotal: 32959992 kB
    MemFree: 10257644 kB
    Buffers: 137896 kB
    Cached: 9728592 kB
    SwapCached: 159328 kB
    Active: 8431064 kB
    Inactive: 1781244 kB
    HighTotal: 0 kB
    HighFree: 0 kB
    LowTotal: 32959992 kB
    LowFree: 10257644 kB
    SwapTotal: 16386292 kB
    SwapFree: 13313700 kB
    Dirty: 96 kB
    Writeback: 0 kB
    AnonPages: 186480 kB
    Mapped: 2391716 kB
    Slab: 134276 kB
    PageTables: 80524 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    CommitLimit: 26760176 kB
    Committed_AS: 16327048 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed: 271316 kB
    VmallocChunk: 34359466727 kB
    HugePages_Total: 5963
    HugePages_Free: 5963
    HugePages_Rsvd: 5963
    Hugepagesize: 2048 kB

    • 3 kevinclosson July 29, 2009 at 2:07 pm

      Yafang,

      What does the output of ipcs -m look like? Also, I presume there are no other apps on the box calling shmget with the SHM_HUGETLB (or no usage of hugetlbfs/libhugetlbfs for that matter). The pages are only in reserved status until processes attached to the segment start to fault the pages in. If you set the init.ora parameter pre_page_sga to TRUE you should notice that the pages come out of reserved and are deducted from HugePages_Free as soon as the instance boot process is complete.

      If the hugepages are not being used by Oracle it is probably because it is an insufficient amount (as I blogged about). So, what does show sga report? You might care to run the sizing script.

  3. 4 yafang gao July 30, 2009 at 6:08 am

    this machine only run oracle database.
    I has not set parameter “pre_page_sga”

    The problem is HugePages_Rsvd always increase when ths db startup.
    when you shutdown db,HugePages_Rsvd is not retrun to zero.
    when it increase to a number,the Hugepage that can allocated for sga less than SGA,then the SGA can not use Hugepage at all and the HugePages_Rsvd can reach a very high value ,as 5963 in example.
    This memory(5963*2MB) can not release to system.
    Only reboot can free the reserved Hugepage and use Hugepage normally.
    But if you shutdown db and restart,maybe can not use Hugepage and HugePages_Rsvd*2MB amount memory is wasted ,can not use by any application,and can not relesed to system until reboot.

    Thanks for your response.
    I like your blog.

  4. 6 Steve August 19, 2009 at 7:22 pm

    thanks for the information Kevin. I have been testing huge pages and noticed the page table size differences as well, on a 2 node NUMA configured IBM3950 with 256GB memory, without huge pages the page table can really expand under load. Overall I am seeing 10% or greater performance gains on several platforms for SGA sizes 6G and 10G for my test databases.

  5. 8 Christo Kutrovsky September 21, 2009 at 6:20 pm

    Hi Kevin,

    I’ve given a presentation on Oracle and memory a few times. I talk a lot about hugepages there.

    Slides with synced audio can be found here:

    http://www.pythian.com/news/741/pythian-goodies-free-memory-swap-oracle-and-everything

  6. 9 Narendra Prabhudesai December 20, 2013 at 5:13 am

    Hello Kevin,

    First, thanks for this series of posts on Hugepages with detail explanations and test cases.
    However, I have a doubt about the test case used above. In your test case, you are only creating DB sessions (in dedicated server mode) but those DB sessions do nothing (and in particular do not “use” SGA). In that case, how does using/not using HugePgaes help? Are Hugepages used by SGA as well as PGA or only SGA?

    Thanks

    • 10 kevinclosson December 20, 2013 at 12:25 pm

      “you are only creating DB sessions (in dedicated server mode) but those DB sessions do nothing (and in particular do not “use” SGA).”

      Hello Narenda,

      If you get a session via a dedicated connection your foreground process maps the entire SGA–immediately. That is sufficient use to see the cost in page tables as I show via meminfo in the post.

      • 11 Narendra Prabhudesai December 22, 2013 at 5:55 am

        Hello Kevin,

        Thanks for your response.
        Apologies if this sounds like a naive question but what exactly does it mean by “foreground process maps the entire SGA-immediately.” ? If process “mapping” memory does not mean process “using” memory, why would page tables be need to contain details about it?


  1. 1 Oracle Exadata Performance series – Part 1: Should I use Hugepages on Linux Database Nodes? | Tanel Poder's blog: IT & Mobile for Geeks and Pros Trackback on March 13, 2011 at 6:55 pm
  2. 2 HugePages Overhead « OraStory Trackback on May 30, 2012 at 12:08 pm

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




EMC Employee Disclaimer

The opinions and interests expressed on EMC employee blogs are the employees' own and do not necessarily represent EMC's positions, strategies or views. EMC makes no representation or warranties about employee blogs or the accuracy or reliability of such blogs. When you access employee blogs, even though they may contain the EMC logo and content regarding EMC products and services, employee blogs are independent of EMC and EMC does not control their content or operation. In addition, a link to a blog does not mean that EMC endorses that blog or has responsibility for its content or use.

This disclaimer was put into place on March 23, 2011.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 1,962 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2013. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

Follow

Get every new post delivered to your Inbox.

Join 1,962 other followers

%d bloggers like this: