（片手落ちまくりだったので修正）
どうやら 1024 のままでそれ以上開けない -> エラーになってる？疑惑をもち、
TokyoTyrant と memcached それぞれが動いているサーバでごそごそと調べる。
TokyoTyrant ... -le オプションでエラーログはでるようになっている、そのエラーログにはファイルディスクリプタ不足で開けない旨は書かれておらず。
ttacceptsock failed というのがでるみたいなんだなこれが。
memcached ... -v オプションなしで起動してるのでログはない
カーネルがやや古めで /proc/<PID>/limits は無いため、すぐわからず。。
でまぁ TokyoTyrant からみてみると、
SYSTEM maximum connection: 1048575
というログはあり、むちゃくちゃ大きい値をとって起動している。ソースみてみると ttserver.c でそれっぽいコードが。ChangeLog にも
* ttserver.c (proc): the system connection limit is now reset.
と書かれていた、ので TokyoTyrant はシロの模様。
ためしに Amazon Linux AMI でインスタンス起動、TokyoCabinet と TokyoTyrant を入れて
ttserver を起動し /proc/<PID>/limits をみてみると・・・
[ec2-user@ip-10-150-131-110 tokyo]$ cat run.sh
/usr/local/bin/ttserver -port 1978 -dmn -le -pid /var/run/ttserver.pid \
-log /var/log/ttserver/ttserver.log -ulim 1024m \
/ttdata/casket.tch#bnum=100000000
[ec2-user@ip-10-150-131-110 tokyo]$ sudo sh ./run.sh
[ec2-user@ip-10-150-131-110 tokyo]$ cat /var/run/ttserver.pid
1095
[ec2-user@ip-10-150-131-110 tokyo]$ sudo cat /proc/1095/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 1048575 1048575 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 59759 59759 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
やはり TokyoTyrant はよろしくやってくれてる。
memcached は、
やはり 1024 だった <= これはまちがい。-c の値を、疑惑をもったサーバと同じ設定にしてなかっただけ。-c で指定すればよい。
[ec2-user@ip-10-150-131-110 tokyo]$ sudo /etc/init.d/memcached start
Starting memcached: [ OK ]
[ec2-user@ip-10-150-131-110 tokyo]$ cat /var/run/memcached/memcached.pid
1117
[ec2-user@ip-10-150-131-110 tokyo]$ sudo cat /proc/1117/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 1024 1024 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 59759 59759 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ので、、/etc/init.d/memcached 内で ulimit を実行するようにした。
[ec2-user@ip-10-150-131-110 tokyo]$ sudo /etc/init.d/memcached restart
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
[ec2-user@ip-10-150-131-110 tokyo]$ cat /var/run/memcached/memcached.pid
1137
[ec2-user@ip-10-150-131-110 tokyo]$ sudo cat /proc/1137/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 32768 32768 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 59759 59759 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
これでよろしい。。。確認。
[ec2-user@ip-10-150-118-101 ~]$ sudo /etc/init.d/memcached restart
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
[ec2-user@ip-10-150-118-101 ~]$ pgrep memcached
1147
[ec2-user@ip-10-150-118-101 ~]$ sudo cat /proc/1147/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 10240 10240 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 59759 59759 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
[ec2-user@ip-10-150-118-101 ~]$ cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="10240"
CACHESIZE="64"
OPTIONS=""
=> Max open files は MAXCONN での値になった
MAXCONN を 10240 -> 256 に変更して再起動 (Max open files が デフォルトの 1024 になることを確認)
[ec2-user@ip-10-150-118-101 ~]$ sudo vi /etc/sysconfig/memcached
[ec2-user@ip-10-150-118-101 ~]$ cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="256"
CACHESIZE="64"
OPTIONS=""
[ec2-user@ip-10-150-118-101 ~]$ sudo /etc/init.d/memcached restart
Stopping memcached: [ OK ]
Starting memcached: [ OK ]
[ec2-user@ip-10-150-118-101 ~]$ pgrep memcached
1171
[ec2-user@ip-10-150-118-101 ~]$ sudo cat /proc/1171/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 unlimited processes
Max open files 1024 1024 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 59759 59759 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ソースの該当部分は 4512 - 4539 行目か。 (memcached 1.4.4 の場合)
4521 /*
4522 * If needed, increase rlimits to allow as many connections
4523 * as needed.
4524 */
4525
4526 if (getrlimit(RLIMIT_NOFILE, &rlim) != 0) {
4527 fprintf(stderr, "failed to getrlimit number of files\n");
4528 exit(EX_OSERR);
4529 } else {
4530 int maxfiles = settings.maxconns;
4531 if (rlim.rlim_cur < maxfiles)
4532 rlim.rlim_cur = maxfiles;
4533 if (rlim.rlim_max < rlim.rlim_cur)
4534 rlim.rlim_max = rlim.rlim_cur;
4535 if (setrlimit(RLIMIT_NOFILE, &rlim) != 0) {
4536 fprintf(stderr, "failed to set rlimit for open files. Try running as root or r
equesting smaller maxconns value.\n");
4537 exit(EX_OSERR);
4538 }
4539 }
めんどうがらずにソースを見ましょう・・。
バックログが大量に必要になるときがあるなら -b で渡す値を決めるのと、net.core.somaxconn の値をみておくのも忘れずに。
