What actually causing java.lang.OutOfMemoryError: unable to create new native thread

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

What actually causing java.lang.OutOfMemoryError: unable to create new native thread

Jason Wee
Hello people, below is an extraction from cassandra system log.

ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
        at org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)


  if (native_thread->osthread() == NULL) {
    // No one should hold a reference to the 'native_thread'.
    delete native_thread;
    if (JvmtiExport::should_post_resource_exhausted()) {
      JvmtiExport::post_resource_exhausted(
        JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
        "unable to create new native thread");
    }
    THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
              "unable to create new native thread");
  }

Question. Is that out of memory error due to native os memory or java heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor max user processes 260000. open files capped at 65536

Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?

Thank you.

Jason
Reply | Threaded
Open this post in threaded view
|

Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread

graham sanderson
First question are you running 32bit or 64bit… on 32bit you can easily run out of virtual address space for thread stacks.

On Nov 10, 2014, at 8:25 AM, Jason Wee <[hidden email]> wrote:

Hello people, below is an extraction from cassandra system log.

ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
        at org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)


  if (native_thread->osthread() == NULL) {
    // No one should hold a reference to the 'native_thread'.
    delete native_thread;
    if (JvmtiExport::should_post_resource_exhausted()) {
      JvmtiExport::post_resource_exhausted(
        JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
        "unable to create new native thread");
    }
    THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
              "unable to create new native thread");
  }

Question. Is that out of memory error due to native os memory or java heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor max user processes 260000. open files capped at 65536

Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?

Thank you.

Jason


smime.p7s (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread

Chris Lohfink-3
if your using 64 bit, check output of:

cat /proc/{cassandra pid}/limits

some older linux kernels wont work with above so if it doesnt exist check the ulimit -a output for the cassandra user. max processes per user may be your issue as well.

---
Chris Lohfink


On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson <[hidden email]> wrote:
First question are you running 32bit or 64bit… on 32bit you can easily run out of virtual address space for thread stacks.

On Nov 10, 2014, at 8:25 AM, Jason Wee <[hidden email]> wrote:

Hello people, below is an extraction from cassandra system log.

ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
        at org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)


  if (native_thread->osthread() == NULL) {
    // No one should hold a reference to the 'native_thread'.
    delete native_thread;
    if (JvmtiExport::should_post_resource_exhausted()) {
      JvmtiExport::post_resource_exhausted(
        JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
        "unable to create new native thread");
    }
    THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
              "unable to create new native thread");
  }

Question. Is that out of memory error due to native os memory or java heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor max user processes 260000. open files capped at 65536

Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?

Thank you.

Jason


Reply | Threaded
Open this post in threaded view
|

Re: What actually causing java.lang.OutOfMemoryError: unable to create new native thread

Jason Wee
Hi, thank you for response.

using 64bit and kernel 2.6.32-358.18.1.el6.x86_64.


# cat /proc/13405/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            10485760             unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             260000               260000               processes 
Max open files            65536                65536                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       255762               255762               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        



Is the stack size and / or Max open files is sufficient? The rest fd pretty much infinite. 

Jason

On Tue, Nov 11, 2014 at 4:09 AM, Chris Lohfink <[hidden email]> wrote:
if your using 64 bit, check output of:

cat /proc/{cassandra pid}/limits

some older linux kernels wont work with above so if it doesnt exist check the ulimit -a output for the cassandra user. max processes per user may be your issue as well.

---
Chris Lohfink


On Mon, Nov 10, 2014 at 11:21 AM, graham sanderson <[hidden email]> wrote:
First question are you running 32bit or 64bit… on 32bit you can easily run out of virtual address space for thread stacks.

On Nov 10, 2014, at 8:25 AM, Jason Wee <[hidden email]> wrote:

Hello people, below is an extraction from cassandra system log.

ERROR [Thread-273] 2012-04-10 16:33:18,328 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
        at org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)


  if (native_thread->osthread() == NULL) {
    // No one should hold a reference to the 'native_thread'.
    delete native_thread;
    if (JvmtiExport::should_post_resource_exhausted()) {
      JvmtiExport::post_resource_exhausted(
        JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
        "unable to create new native thread");
    }
    THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
              "unable to create new native thread");
  }

Question. Is that out of memory error due to native os memory or java heap? Stacked size to the jvm is -Xss128k. Operating system file descriptor max user processes 260000. open files capped at 65536

Can any java/cpp expert pin point what JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and  JVMTI_RESOURCE_EXHAUSTED_THREADS means too?

Thank you.

Jason