前阵子，Infra报告无法透过putty以及SecureCRT连接到数据库服务器，提示的错误为Resource temporarily unavailable。由于该服务器上有差不多有20个nstance，应该是超出了系统当前设置的值。关于超出资源限制的事之前有碰到过，只不过不是这个错误，而是open files: cannot modify limit: Operation not permitted。下面描述一下关于Resource temporarily unavailable这个错误及其相关信息。

一、故障现象
installer@linux_02:~> ps -U oracle |wc -l
2015
installer@linux_02:~> lsof | grep oracle|wc -l
83646
installer@linux_02:~> su - oracle
Password:
su: cannot set user id: Resource temporarily unavailable

#从Putty或者SecureCRT连接则会出现如下类似的错误：
Server sent disconnect message
type 2(protocol error):
"fork failed: Resource temporarily unavailable"

#系统日志如下：
Aug 19 09:25:01 linux_02 /usr/sbin/cron[6736]: (oracle) MAIL (mailed 25 bytes of output butgot status 0x0001 )
Aug 19 09:26:29 linux_02 sshd[6864]: Accepted keyboard-interactive/pam for oracle from 192.168.9.1 port 1253 ssh2
Aug 19 09:26:29 linux_02 sshd[6875]: fatal: setresuid 2000: Resource temporarily unavailable
Aug 19 09:27:15 linux_02 sshd[6922]: Accepted keyboard-interactive/pam for oracle from 192.168.9.1 port 1312 ssh2
Aug 19 09:27:15 linux_02 sshd[6985]: fatal: setresuid 2000: Resource temporarily unavailable

二、故障分析
#以下是Metalink上788064.1对Resource temporarily unavailable的错误描述：
The error is different when it  is reaching the limit ‘open files‘ and ‘max user processes‘ in /etc/profile .

A). Error on reaching the limit ‘open files‘:

  [oracle@mydesk~]$ssh rac2
  oracle@rac2‘s password:
  -bash: ulimit: max user processes: cannot modify limit: Operation not permitted
  -bash: /home/oracle/.bash_profile: Too many open files

B). Error on reaching the limit ‘max user processes‘:

  [oracle@mydesk~]$ssh oracle@rac2
  oracle@rac2‘s password:
  -bash: ulimit: open files: cannot modify limit: Operation not permitted
  -bash: fork: Resource temporarily unavailable

#也就是说我们当前的错误是由于进程数的限制而引起的   #Author : Leshami
#下面检查当前limits.conf配置                   #Blog    : http://blog.csdn.net/leshami
linux_02:/etc/security # grep -v ^# /etc/security/limits.conf

oracle  soft    nproc   2047
oracle  hard    nproc   16384
oracle  soft    nofile  1024
oracle  hard    nofile  65536

#下面是来自Suse官方的Resolution：
The affected user has reached the maximum number of process specified for him into the file /etc/security/limits.conf .

These commands, executed as root, can give you the number of process and of open files for the given user:

ps -U username | wc -l

lsof | grep username | wc -l

To fix the issue increase the NPROC Soft limit according to the user and applications needs; please look at the note below about the /etc/security/limits.conf file.
#上面的描述也是要增加nproc的值，
#从之前的运用的ps以及lsof命令来看，ps命令得到的oracle用户的进程数为2015，从这个值来看，当前的配置符合要求。
#反而是打开文件数超出了设定的hard值，为83682。比较纳闷的是不是文件数错误，而是进程数错误。

#ID 788064.1对此给出的solution 如下：
a). Modify /etc/security/limits.conf manually
Increase the value of ‘soft nofile‘ until it is equal to ‘hard nofile‘ value. Increase the value of ‘soft nproc‘ until it is equal to ‘hard nproc‘ value.
#增加值到soft nofile的值到等于hard nofile的值；增加soft nproc的值到等于hard nproc的值。
#从这个来看的话，应该是系统一旦启动后会分配soft的值，但是最大值不能超过hard值。不管系统是否空闲，都会分配soft值。

b). Install oracle-validated package to modify /etc/security/limits.conf #安装oracle-validated来自动修改limits.conf(RHEL4 to OL5适用)
After install the oracle-validated  package, the content of /etc/security/limits.conf  is modified.

#Metalink ID 1239915.1上对此的描述也是要调整limits.conf

SOLUTION

The file /etc/security/limits.conf controls the resource limits for each user.
The nproc value determines how many processes that user is allowed and the nofile value limits the total number of files which may be opened at once.

Increase both the soft and hard limits for the destination user ("oracle" in our example) and save the file.
Activate The Changed User Limits

These changes take effect only upon login, so out and re-login to activate the changes.

三、故障解决
#根据上面的Solution调整limits.conf文件
linux_02:/etc/security # cp limits.conf limits.conf.bk
linux_02:/etc/security # vi /etc/security/limits.conf
linux_02:/etc/security # grep -v ^# /etc/security/limits.conf
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft nofile 65536
oracle hard nofile 65536

oracle@linux_02:/users/oracle> ps -U oracle |wc -l
2714
oracle@linux_02:/users/oracle> lsof | grep oracle|wc -l
110694

四、相关参考
ulimit: open files: cannot modify limit: Operation not permitted
DocID: 1239915.1
DocID: 788064.1

cannot set user id: Resource temporarily unavailable