Setting Network Parameters
Setting of network parameters in a RAC environment is important because of its reliance on high performance interconnect between the Oracle Instances.
net.ipv4.ip_local_port_range
This parameter sets the range for outgoing connections. The Oracle validated rpm changes the lower port value to 1024 and the upper port value to 65000.
Setting up net.core* parameters
net.core.rmem_default, net.core.wmem_default, net.core.rmem_max, net.core.wmem_max
These parameters are necessary for UDP, the protocol for the Oracle RAC interconnect communication with Gigabit Ethernet. You can set these values to the Oracle validated rpm recommended values.
net.core.rmem_default - A kernel parameter that sets the default size of receive buffers used by sockets.
net.core.rmem_max - The maximum socket receive buffer size which may be set by using the SO_RCVBUF socket option.
net.core.wmem_default - the default value in bytes of the socket send buffer
net.core.wmem_max - the maximum socket send buffer size which may be set by the SO_SNDBUF socket option. From linux 2.6 kernel version, you no longer strictly need to set the values of these parameters as they are tuned automatically.
So if net.core.rmem_max is set to 5 MB and rmem_default is set to 1MB and there are 100 processes running, each with a socket open for communication on the interconnect, then minimum 100 MB (1 MB per socket) will be allocated for inter node communication. Anything more than that will be allocated on a need basis upto a maximum value set by rmem_max parameter. So the total memory usage depends on rmem_default, rmem_max, the number of open sockets available and the amount of buffer space the process is actually using.
To get the total number of Oracle related UDP sockets, you can execute the following command
netstat -anp -udp | grep ora | wc -l.
For tcp auto tuning the values of tcp_rmem and tcp_wmem specify the minimum, default and maximum values of the socket send and receive buffers.
A sample setting as per Oracle validated rpm would look as follows
net.ipv4.tcp_rmem - 4096 262144 4194304
net.ipv4.tcp_wmem - 4096 262144 4194304
For the default values, the auto tuned parameters take precedence, so these must the set to the oracle recommended value of 262144. For maximum values, the static settings take precedence, overriding the auto-tune parameter values.
Message Queues
Communication between Oracle processes happens in an asynchronous manner with Inter Process Communication (IPC) messages. The messages are placed in a message queue where they are read by other processes. An example is the communication that takes place between the Oracle foreground and background processes visible through Oracle wait events 'rdbms ipc message' and 'rdbms ipc reply'. 'rdbms ipc message' is sent by an Oracle background process when idle and waiting for a foreground process to send a message.
You can see the message queue limits with the command ipcs -lq.
$ipcs -lq
--------------------------Messages : Limits -----------------------------
max queues system wide = 16
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536
This provides information for 3 message queue parameters
msgmni - the number of message queues. the default is 16
msgmax - the maximum message size
msgmnb - the maximum combined value of all message queues at any one time.
Number of open files
The parameter that affects the number of open files for all processed on the system is fs.file-max and the Oracle validated rpm sets this parameter to 327679 which is lower than the default value. The number of open and available file handles on the system can be seen in /proc/sys/fs/file-nr.
Oracle recommends the number of file descriptors to be a value of 512 * PROCESSES. Thus the oracle recommended value of 327k is ok for upto 640 Oracle processes. Beyond this number of processes will result in
ORA-27123 : unable to attach to shared memory segment. There is also a per user limit on the number of open files in addition to the system wide value, which by default is set to 1024. The oracle validated rpm modifies this value per user value in the file /etc/security/limits.conf.
Configuring asynchronous I/O
fs.aio-max-nr kernel parameter sets the maximum number of asynchronous I/O operations permitted on the system. The default for this parameter is 65536 and the oracle validated rpm increases it to 3145828. The actual value of this can be seen in /proc/sys/fs/aio-nr. If this value is equal to fs.aio-max-nr then asynchronous I/O operations can be hampered causing the system to slow down. If ASMLIB is used, setting the value of this parameter has no effect and so default value can be maintained.
kernel.sysrq - The value of this parameter can be 0, 1 or a number greater than 1 which is a bitmask indicating which features to allow. This setting allows the user to perform various low level commands regardless of the system's state.It is often used to recover from freezes or to reboot a computer without corrupting the file system. Its effect is similar to a computer's hardware reset button but with many more options and more control.
References
Thought for the day
Focus on where you want to go, not on what you fear
--Anthony Robbins
No comments:
Post a Comment