BBQ Reviews - Charcoal BBQ Reviews - Kamado Smoker Reviews - Wood Pellet Grill Reviews - Charcoal Kamado Reviews - Pellet Grill Reviews - Ceramic Grill Reviews

RHEL 6 Configuring SHMMAX and SHMALL Settings

Looking for other tech tips? I have some blogs on LDAP Tech Tips  as well as NetApp Tech Tips and Configuring Puppet Master Servers. Also my BBQ and fine food blog might turn some heads too.  See All my  Consolidated Solaris Technical Blog Entries, as well as all my Consolidated Linux Technical Blog Entries

RHEL 6 Configuring SHMMAX and SHMALL Settings

RHEL 6 Configuring SHMMAX and SHMALL kernel paramaters are the key to your oracle databases performance. These two parameters are pretty important paramaters and they affect how Oracle creates and manages SGA’s.  Your operating system has to provide an application access to a certain type of memory with certain types of controls.  It’s the Unix kernel that allows applications to access shared memory (IPC), and how much memory, in multiple big chunks or small chunks etc..

centosSometimes, portions of memory get used leaving memory segments out of order.  So these chunks of memory, from now on called segments, can get used inefficiently leaving unused memory segments.  This can be caused by many reasons which I won’t get into now.  So when Orale wants to access some shared memory on startup, it knows how much memory it needs for it’s SGA and has to figure out the best and most efficient way to access the resource.  If Oracle needs 8GB or shared memory but can only find 4 seperate 2GB segments that are out of order, it will then have no choice but to use this memory model.  This non-contiguous memory model isn’t the best.  Oracle will use shared memory in 3 different ways or memory models.  The first memory model as just described, the non-contiguous model where the memory segments are scattered over the place.  Another memory model being multiple memory segments that are in fact contiguous, so this is better, less overhead than the non-contiguous model.  But the ultimate is one big segment that is the size of the SGA. So in this case an 8 GB segment.  And that is where the tuning comes into play.  How to tune the kernel and it’s paramaters to meet your Oracle applications shared memory needs.  As I mentioned in the opening paragraph, its the SHMMAX that is the key kernel paramater in Linux to tune the SGA shared memory requirement.

RHEL 6 and Shared memory and SHMMAX and SHMALL

Just to clarify a couple of definitions, I will go over some terminology.  The term shared memory is a term used to describe a type of memory management in the Unix kernel.  It is a memory region that can shared between different processes.  Oracle uses this shared memory when using and implementing the SGA, which has to be visible to all database sessions.  Using a shared memory model, this avoids creating duplicate copies of memory.  So programs can eaily share data.  These regions of memory, because they are shared need to have controls on who can access them and who can’t. A locking mechanism is required. This is where semaphores come into play. Basically, flags that are either on or off. If the memory is being used, the process turns on the flag and other processes have to wait until the semaphore is freed or the flag is turned off.  So shared memory needs semaphores, they go hand in hand.  I should probably mention that an application can access shared memory programatically through a set of POSIX C routines, common on most Unix like operating systems including Linux.  The IPC or shared memory API’s have common IPC calls like shmget() and shmat() which allow the programmer to get a shared memory segment, or attach to a segment.  There are many more.  Now the kernel paramaters SHMMAX and SHMALL need to be defined now. SHMMAX is really just the maximum size of a single shared memory segment.  It’s size is represented in bytes”.  And SHMALL is the sum of all shared memory segments on the whole system.  But it is measured in number of pages.  Will leave it at that for now.

At times on very busy transactional databases, you can run out of semaphores. This seriously limits the performance of a database. You can look at your semaphores as follows:

# ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 65537      root       600        1         
0x00000000 229378     oracle     660        300       
0x00000000 265667     oracle     660        300      
0x00000000 247100     oracle     660        360

Enough of a semaphore tangent. Linux being a type of Unix, there are 300 different ways of finding the value of a system paramaters. The traditional way is calling ipcs to list the shared memory limits:

# ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096                     <--- this is SHMMNI
max seg size (kbytes) = 67108864                  <--- this is SHMMAX
max total shared memory (kbytes) = 17179869184    <--- this is SHMALL
min seg size (bytes) = 1

Or cat’ing the /proc/* data structures

#  cat /proc/sys/kernel/shmall

4294967296

#  cat /proc/sys/kernel/shmmax

68719476736

RHEL 6: how to configure SHMALL for an Oracle database

Right off the top, since SHMALL is the sum of all the shared memory segments on your system, you had better make it smaller than your total system memory.   If I remember correctly, back in the 90′s, when memory was more expensive than the US space program,  on HP-UX systems we used to define shared memory SHMALL equivalent to much larger than physical memory.  Swap disks were busy then. I don’t know if it’s possible to do this now, but it’s not recommended. I’ll have to test that some day for fun. SHMALL should also be less than the sum of all the SGA’s configured on the system.  If you allow more shared memory than the physical memory on the system, then your asking for a world of hurt.  Obviously your system has to be sized appropriately for the work to be done.  If you create a new database instance and start Oracle which will then want to create a new SGA, shmget() -> shmat() etc… it will fail with the much feared “out of memory” etc…  This just means that SHMALL is doing it’s job properly and preventing Oracle from taking what it isn’t allowed to take.

A typical error message caused by there being insufficient memory when an Oracle instance is starting would be something like:

ORA-27102: out of memory Linux-x86_64 Error: 28: No space left on device

This means SHMALL is doing it’s job. As shared memory is a device, ie. /dev/shm, the error message above makes sense.

The logic in finding the SHMALL value for your RHEL 6 system

Configuring the kernel parameter SHMALL is mostly just basic math.  It’s a decision that involves finding the amount of memory you want to give to Oracle, or how much you can give away.  This memory I am talking about can be talked about in terms of physical memory.  Physical memory shouldn’t be confused with virtual memory which has swap space included.  From the physical memory we take away the memory that the operating system needs.  Linux will use a lot of memory for disk caching and other assorted data structures.  So lets be moderately generous and say we will give the Unix kernel 1GB, this is a good rule.

Working example

System memory 12GB

Linux Kernel  1GB

Memory for Oracle  11GB (System memory – Linux Kernel)

So we can assign SHMALL the value of 11GB. But… SHMALL is measured in memory pages not bytes.  We need to divide the 11GB by the systems page size value.  It’s almost always 4K which is the recommended size, but to be safe, you run the following command:

# getconf PAGE_SIZE
4096

We have to convert the 11GB byte value into a page value because as I mentioned above, SHMALL is measured in page size.  I don’t make the rules, sorry.  I use bc, the unix calculator and I don’t need decimal places.

# bc
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
11*1024*1024*1024   <--- 11 x 1K x 1K x 1K gives me 11GB value in bytes
11811160064
11811160064/4096    <--- 11GB value in bytes divided by 4k (page size)
2883584             <--- page size value to assign to SHMALL

So this is a tad conveluted, but easy math.  Lets make this SHMALL kernel paramater permanent by adding it to the /etc/sysctl.conf file.

# vi /etc/sysctl.conf
kernel.shmall = 2883584

The logic in finding the SHMMAX value for your RHEL 6 system

If you read the release and install and configuration documentation for Oracle 11g, Oracle recommends you set shmmax to half of your physical memory or 4 GB less 1 byte. Choose whichever is lower.  The interesting 4 GB less 1byte memory setting is based on the virtual memory addressing limitations of the 32 bit (x86) processors. All 64 bit processors can address much larger memory addresse space and is not limited like it’s 32 bit brethren. So much larger shared memory segments can be permitted and created and used.

When Oracle first starts up, after having read it’s configuration files it will know how much memory it needs for it’s assorted SGA’s.  The best fit for an Oracle SGA shared memory segment is when the SGA fits into one shared memory segment.  As I mentioned earlier, Oracle can use non-contiguous shared memory segments, but its not Oracle’s chosen memory model.  Therefore by defenition, if the SGA is larger than the value of SHMMAX, then Oracle has to use one of the two other remaining shared memory models.  Contiguous or non-contiguous memory segments.

I try to set the SHMMAX to the size of the largest SGA defined on your database server.  If your system has 6 Oracle instances configured, 2 require 2 GB SGA’s, 2 require 1.5GB SGA’s and 2 require 1 GB SGA’s, you would set the SHMMAX to 2GB.

Edit your sysctl.conf and add the SHMMAX value to 2 GB in bytes

# vi /etc/sysctl.conf
kernel.shmmax = 2147483648

Or, as this is Unix and there are many ways of doing things, use sysctl -w argument=xyz which writes to the sysctl.conf file.

# sysctl -w kernel.shmmax=2147483648

Now reload the parameters by asking sysctl politely to to reread it’s configuration

# sysctl –p

Now that you have configured shmmax and shmall, and all the other steps required to configure you system for Oracle, I will jump ahead very fast here and install Oracle and create a test db instance.

# su - oracle
$ cd /tmp/2wherever/install/files/are
$ ./runInstaller

….

# service oracle start

Create a test databse

$ dbca

etc…..

# ipcs -a
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x6c6c6536 0          root       600        4096       0
0x2270894c 2359297    oracle     660        1512046592 30
------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x00000000 0          root       600        1
0x00000000 65537      root       600        1
0x00000000 294914     apache     600        1
0x00000000 327683     apache     600        1
0x396b9144 983044     oracle     660        154
------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

The last few steps have been exceedingly oversimplied, but the point here is to show the shared memory use.  You will also see that user oracle has grabbed 154 semaphores.  This isn’t a huge number, and in this case it’s only a test db instance so no real traffic.

 

See All Solaris BlogsSee All Linux Blogs  Netapp BlogsSee Puppet Master Blogs  See All OpenLDAP Blogs

,

12 Responses to RHEL 6 Configuring SHMMAX and SHMALL Settings

  1. kutts November 13, 2013 at 1:58 am #

    Thats a great arcticle about how to set the sh values. I understand now what would happen if the shmall is set to a higher value.

    How would the system behave in case if I set the shmmax to a very high value. In this case, if the total system physical memory is 12 GB and I set shmmax to 12 GB itself rather than the recommended 6 or 4 GB – 1 byte, then what would be the problem ? Would I cause a situation where the OS consumes the 12 GB during startup and thereby starving Oracle of any shared memory ?

  2. rocker November 13, 2013 at 7:10 am #

    Kutts, thanks for the nice comment.

    Oracle only uses as much memory as it’s told to use. Oracle’s SGA is a configurable startup value, meaning you could ask it to use 6 GB, and Oracle will use it all and take up as many shared memory segments as are needed to fulfill it’s memory requirement. Leaving you the remaining 6. I’ve never tried setting shmmax to physical memory size, but would be a fun test. We used to run our HP-UX 10.x boxes with massive shmmax settings equal to physical memory or even more if I remember correctly. Back then, we could configure shared memory and ask it to use virtual memory as opposed to physical memory. It worked, but heavily transactional databases stalled and that service was moved to dedicated hpux boxes.

    Oracle is such a big beast.

    Cheers

  3. kamal November 13, 2013 at 9:19 am #

    what do you set SHMMNI to?

    • vlad November 13, 2013 at 11:25 pm #

      shmmni or number of shared memory segments on most unix’s including linux is 4k. That’s more than enough even for a big database. The shmmax is the most important one, with semaphores.

  4. Spiros85 April 10, 2014 at 8:31 am #

    By following your article I wanted to calculated the default amount of RAM my Oracle Linux uses in order to calculate shmall and shmmax

    The first step was to take a look at the values contained at /etc/sysctl.conf
    [root@server ~]# sysctl -a | grep shm
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    kernel.shmmni = 4096

    Second I used ‘ipcs -lm’ command
    [root@server ~]# ipcs -lm

    —— Shared Memory Limits ——–
    max number of segments = 4096 —–> shmmni (It is expressed in Pages)
    max seg size (kbytes) = 67108864 —–> shmmax (It is expressed in Kbytes)
    max total shared memory (kbytes) = 17179869184 —–> shmall (It is exressed in Kbytes)
    min seg size (bytes) = 1

    According to your article, if you want to find shmall for 11GBs of memory you have to do the following mathematical equation:
    (11*1024*1024*1024)/4096

    By given the values of ‘ipcs -lm’ command, I wanted to verify the values contained to ‘/etc/sysctl.conf’
    shmall = (17179869184/4096)*1024 = 4294967296
    shmmax = 67108864*1024 = 68719476736

    Let’s find the value of ‘shmmax’ from ‘shmall’
    shmmax = (17179869184/1024)*4096 = 68719476736 (This is the value in ‘/etc/sysctl.conf’)
    68719476736/1024 = 67108864 —-> (This is the value in Kbytes displayed by ‘ipcs -lm’)

    So based on the upon information to find out the initial amount of RAM used to calculate those values, I did:
    ((4294967296*4096)/1024*1024*1024) = 16384

    That means my Oracle Linux does this calculations for 16384 Gbs?

    Thanks In advance

    • Bernard April 10, 2014 at 5:32 pm #

      Hi Spiros

      Sorry, I’m exhausted today, so I won’t do math for you.

      To find the amount of memory you want to give to oracle, if your oracle database server is a straight up database server, you usually want to give it as much memory as it needs based on SGA requirements. The userland processes will need some memory as well. So you need to find a magic number that you think will work for you. I’d take your physical memory, minus a couple of gig for kernel & caches and userland processes, db writer etc…

      Take the resulting physical memory number, turn it into bytes using the math I have shown above. Divide that by 4K chunks which is your page size, that resulting number is what you can set the shmall to, as I’ve shown above too.

      Another way to sort this out is to look in /usr/include/linux/shm.h You will see:

      #define SHMALL (SHMMAX/PAGE_SIZE*(SHMMNI/16)) /* max shm system wide (pages) */

      There are many ways of doing this. SHMMNI is usually defaulted to 4K as well. So 4096/16 should be multiplied against your shmmax / 4096. And this is actually very helpful. Shows the basic logic you can use to find the maximum size.

      I know that doesn’t answer your question directly, but this shows you the logic, in different words.

      Now, off to get some sleep.

      • Spiros85 April 11, 2014 at 3:24 am #

        Thank you very much dear Bernard for your reply. Actually the reason I did all these maths was to find out they way my Operating System sets the ‘shmmax’ by calculating the ‘shmall’. To be more specific I will try to explain this via an example based on your article.

        I wanted to calculate the amount of shall for 2GiB
        shmall=((2*1024*1024*1024)/4096)=524288

        After this I added this value to ‘/etc/sysctl.conf’

        When I did ‘ipcs -lm’ I got the following output
        [root@serverX ~]# ipcs -lm

        —— Shared Memory Limits ——–
        max number of segments = 4096
        max seg size (kbytes) = 67108864
        max total shared memory (kbytes) = 20971552
        min seg size (bytes) = 1

        The problem was that this ‘max seg size (kbytes) = 67108864′ value did not respond to 2GiB allocated memory.

        So for that reason, I wanted to find out how to calculate the corresponding amount of shmmax. By doing some maths I came to the following conclusion by given the amount of ‘shmall’:
        shmmax = (20971552/1024)*4096 = 83886080

        ’83886080′ is the value that I have to insert at ‘/etc/sysctl.conf’

        After this with ‘ipcs -lm’ I got:
        [root@serverX ~]# ipcs -lm

        —— Shared Memory Limits ——–
        max number of segments = 4096
        max seg size (kbytes) = 81920
        max total shared memory (kbytes) = 20971552
        min seg size (bytes) = 1

        Is that approach correct?

        Thanks in advance :)

        • Bernard April 11, 2014 at 8:43 am #

          Spiros, you can’t set SHMALL to all you physical memory:

          shmall=((2*1024*1024*1024)/4096)=524288

          If you have a 2 GB system, you will only hurt yourself if you are using the entire physical memory on the system for database. On a 2 GB system, I’d only use 1 GB for shared mem. Old HPUX 10.x systems let you expand share mem beyond physical into virtual memory. Worked ok, but had its moments.

          Re. SHMMAX in many cases can be set to half your physical memory. But I doubt you would have a 1 GB SGA requirement, so chop up your memory carefully here. Your SHMMAX setting above looks reasonable, or better yet, have the DBA tell you the size of your SGA, and set it to that.

          I’d change your SHMALL math to look like:

          SHMALL = ((1*1024*1024*1024)/4096) = 262144

          Oracle Linux I believe is a red hat derivative / fedora. On a 32 bit system, SHMMAX will default to under 4 GB, but you don’t have that much memory. A 64 bit system gets more :) Your Oracle install docs should tell you exactly how and what to do. I might add that a system with 2 GB of RAM for Oracle is pretty slim pickings. It will work of course, but doesn’t leave you with much room if the system is busy. Best advice, match SHMMAX to what that version of oracle wants, and what the dba says the SGA is. Also, as I said, don’t use all your physical memory for SHMALL, leave some for the kernel to breath, so something like my formula above should work.

        • Spiros85 April 11, 2014 at 9:15 am #

          I would like to correct something

          It is not 81920 but 8192

          Thanks

Leave a Reply


Hit Counter provided by Skylight