Monday, March 9, 2009

Hbase: Too many open files

While running a job that was inserting a lot of records in HBase (~7,000,000 records), I was getting this exception (in datanode log file):

2009-03-09 20:26:21,072 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(, storageID=DS-1758498865-
, infoPort=50075, ipcPort=50020):DataXceiveServer: Too many open files
at Method)

The error is verbose, but I had problem raising this limit until I found this post. It seems like you have to modify two files to raise the limit on ubuntu. First of all, you need to add the following line to /etc/pam.d/common-session (as root):
  • session required
After enabling the pam limits module, you can simply edit the limits.conf file located at /etc/security/limits.conf to add a specific number of opened files limit to the hadoop user (in my case, hadoop). Add the following lines according to your installation:
  • hadoop hard nofile 65000
  • hadoop soft nofile 30000
Logout and login back into your hadoop user account and your limit should be raised. To check if it's really the case, execute the following command: ulimit -n. It should return 30000 (the soft limit we set)