cassandra slows down after inserts

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

cassandra slows down after inserts

rkmr.em@gmail.com
hi
im using the default config from the svn trunk for everything: tables, column, supercolumn, single node etc.

im inserting rows like this

for u in users:
        client.insert('Table1', str(u.uid), 'Super1:attrs:key', str(u.key), timestamp, False)

after about 100k rows, cassandra completely slows down taking for ever to insert or get_slice_super. the commitlog is about 70MB in size.
when i stop and start cassandra the logs are replayed and it is fast again, till it reaches the same point when it becomes really slow.

is this expected? is there something in the config that can be changed so this doesnt happen?

thanks



    <Tables>
        <Table Name="Table1">
            <!-- The fraction of keys per sstable whose locations we
                 keep in memory in "mostly LRU" order.  (JUST the key
                 locations, NOT any column values.)

                 The amount of memory used by the default setting of
                 0.01 is comparable to the amount used by the internal
                 per-sstable key index. Consider increasing this is
                 fine if you have fewer, wider rows.  Set to 0 to
                 disable entirely.
            -->
            <KeysCachedFraction>0.01</KeysCachedFraction>
            <!-- if FlushPeriodInMinutes is configured and positive, it will be
                 flushed to disk with that period whether it is dirty or not.
                 This is intended for lightly-used columnfamilies so that they
                 do not prevent commitlog segments from being purged. -->
            <ColumnFamily ColumnSort="Name" Name="Standard1" FlushPeriodInMinutes="60"/>
            <ColumnFamily ColumnSort="Name" Name="Standard2"/>
            <ColumnFamily ColumnSort="Time" Name="StandardByTime1"/>
            <ColumnFamily ColumnType="Super" Name="Super1"/>
        </Table>
    </Tables>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

Sandeep Tata
What hardware are you running one? How long does the slowdown last ?
 There are a few reasons for temporary slowdowns ... perhaps the JVM
started GCing? Cassandra spends time cleaning up the on-disk SSTables
in a process called compaction. This could cause the client to observe
a slowdown.

Things you could try --
Reduce the Memtable size in the config files. (If GCing was the problem)
Increasing the number of SSTables written before compaction kicks in.


On Sun, Jul 12, 2009 at 9:16 PM, [hidden email]<[hidden email]> wrote:

> hi
> im using the default config from the svn trunk for everything: tables,
> column, supercolumn, single node etc.
>
> im inserting rows like this
>
> for u in users:
>         client.insert('Table1', str(u.uid), 'Super1:attrs:key', str(u.key),
> timestamp, False)
>
> after about 100k rows, cassandra completely slows down taking for ever to
> insert or get_slice_super. the commitlog is about 70MB in size.
> when i stop and start cassandra the logs are replayed and it is fast again,
> till it reaches the same point when it becomes really slow.
>
> is this expected? is there something in the config that can be changed so
> this doesnt happen?
>
> thanks
>
>
>
>     <Tables>
>         <Table Name="Table1">
>             <!-- The fraction of keys per sstable whose locations we
>                  keep in memory in "mostly LRU" order.  (JUST the key
>                  locations, NOT any column values.)
>
>                  The amount of memory used by the default setting of
>                  0.01 is comparable to the amount used by the internal
>                  per-sstable key index. Consider increasing this is
>                  fine if you have fewer, wider rows.  Set to 0 to
>                  disable entirely.
>             -->
>             <KeysCachedFraction>0.01</KeysCachedFraction>
>             <!-- if FlushPeriodInMinutes is configured and positive, it will
> be
>                  flushed to disk with that period whether it is dirty or
> not.
>                  This is intended for lightly-used columnfamilies so that
> they
>                  do not prevent commitlog segments from being purged. -->
>             <ColumnFamily ColumnSort="Name" Name="Standard1"
> FlushPeriodInMinutes="60"/>
>             <ColumnFamily ColumnSort="Name" Name="Standard2"/>
>             <ColumnFamily ColumnSort="Time" Name="StandardByTime1"/>
>             <ColumnFamily ColumnType="Super" Name="Super1"/>
>         </Table>
>     </Tables>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

Jonathan Ellis-3
On Mon, Jul 13, 2009 at 12:37 AM, Sandeep Tata<[hidden email]> wrote:
> What hardware are you running one? How long does the slowdown last ?
>  There are a few reasons for temporary slowdowns ... perhaps the JVM
> started GCing?

Every time someone has reported this symptom, that has been the problem.

The object count tunable is the most direct way to ameliorate this.

http://wiki.apache.org/cassandra/MemtableThresholds
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

rkmr.em@gmail.com
In reply to this post by Sandeep Tata

how do i find out if JVM is GCing?

On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <[hidden email]> wrote:
What hardware are you running one?

 dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
operating system is fedora core 9


How long does the slowdown last ?

i stopped inserting data after slowdown starts, and it is still slow now after over 10 hours.
however if i stop cassandra and start it, it becomes super fast immediately. till i insert another 100k or so rows when it becomes really slow again.

 

 There are a few reasons for temporary slowdowns ... perhaps the JVM
started GCing?
how do i find out if this is the cause?

 
Cassandra spends time cleaning up the on-disk SSTables
in a process called compaction. This could cause the client to observe
a slowdown.

Things you could try --
Reduce the Memtable size in the config files. (If GCing was the problem)
Increasing the number of SSTables written before compaction kicks in.

can you tell me what numbers i should use?

thanks!

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

rkmr.em@gmail.com
when i stop cassandra and start it again, this is what is printed. it takes just a couple of seconds for this to run.
and after that it becomes really fast.


Listening for transport dt_socket at address: 8888
DEBUG - Loading settings from ./../conf/storage-conf.xml
DEBUG - adding Super1 as 0
DEBUG - adding Standard2 as 1
DEBUG - adding Standard1 as 2
DEBUG - adding StandardByTime1 as 3
DEBUG - adding LocationInfo as 4
DEBUG - adding HintsColumnFamily as 5
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
INFO - Compacting [/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
DEBUG - INDEX LOAD TIME for /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
INFO - Replaying /home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db   : 73600
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db   : 84224
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db   : 94848
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db   : 105472
DEBUG - Expected bloom filter size : 105472
INFO - Compacted to /home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db.  0/28831084 bytes for 104856/104860 keys read/written.  Time: 8119ms.
INFO - Flushing Memtable(Super1)@552364977
DEBUG - Submitting Super1 for compaction
INFO - Completed flushing Memtable(Super1)@552364977
INFO - Flushing Memtable(Standard1)@1290243769
DEBUG - Submitting Standard1 for compaction
INFO - Completed flushing Memtable(Standard1)@1290243769
INFO - Compacting [/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db   : 256
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db   : 512
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db   : 768
DEBUG - index size for bloom filter calc for file  : /home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db   : 1024
DEBUG - Expected bloom filter size : 1024
INFO - Compacted to /home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db.  0/210 bytes for 0/1 keys read/written.  Time: 301ms.
DEBUG - Starting to listen on 127.0.0.1:7001
INFO - Cassandra starting up...



On Mon, Jul 13, 2009 at 6:06 AM, [hidden email] <[hidden email]> wrote:

how do i find out if JVM is GCing?

On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <[hidden email]> wrote:
What hardware are you running one?

 dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
operating system is fedora core 9


How long does the slowdown last ?

i stopped inserting data after slowdown starts, and it is still slow now after over 10 hours.
however if i stop cassandra and start it, it becomes super fast immediately. till i insert another 100k or so rows when it becomes really slow again.

 

 There are a few reasons for temporary slowdowns ... perhaps the JVM
started GCing?
how do i find out if this is the cause?

 
Cassandra spends time cleaning up the on-disk SSTables
in a process called compaction. This could cause the client to observe
a slowdown.

Things you could try --
Reduce the Memtable size in the config files. (If GCing was the problem)
Increasing the number of SSTables written before compaction kicks in.

can you tell me what numbers i should use?

thanks!


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

Jonathan Ellis-3
In reply to this post by rkmr.em@gmail.com
See the wiki page I linked.

On Mon, Jul 13, 2009 at 8:06 AM, [hidden email]<[hidden email]> wrote:

>
> how do i find out if JVM is GCing?
>
> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <[hidden email]>
> wrote:
>>
>> What hardware are you running one?
>
>  dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
> operating system is fedora core 9
>
>
>> How long does the slowdown last ?
>
> i stopped inserting data after slowdown starts, and it is still slow now
> after over 10 hours.
> however if i stop cassandra and start it, it becomes super fast immediately.
> till i insert another 100k or so rows when it becomes really slow again.
>
>
>>
>>  There are a few reasons for temporary slowdowns ... perhaps the JVM
>> started GCing?
>
> how do i find out if this is the cause?
>
>
>>
>> Cassandra spends time cleaning up the on-disk SSTables
>> in a process called compaction. This could cause the client to observe
>> a slowdown.
>>
>> Things you could try --
>> Reduce the Memtable size in the config files. (If GCing was the problem)
>> Increasing the number of SSTables written before compaction kicks in.
>
> can you tell me what numbers i should use?
>
> thanks!
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

Jonathan Ellis-3
In reply to this post by rkmr.em@gmail.com
Cassandra is replaying the transaction log and preloading SSTable
indexes.  This is normal.

On Mon, Jul 13, 2009 at 8:10 AM, [hidden email]<[hidden email]> wrote:

> when i stop cassandra and start it again, this is what is printed. it takes
> just a couple of seconds for this to run.
> and after that it becomes really fast.
>
>
> Listening for transport dt_socket at address: 8888
> DEBUG - Loading settings from ./../conf/storage-conf.xml
> DEBUG - adding Super1 as 0
> DEBUG - adding Standard2 as 1
> DEBUG - adding Standard1 as 2
> DEBUG - adding StandardByTime1 as 3
> DEBUG - adding LocationInfo as 4
> DEBUG - adding HintsColumnFamily as 5
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
> INFO - Replaying
> /home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db   : 73600
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db   : 84224
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db   : 94848
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db   : 105472
> DEBUG - Expected bloom filter size : 105472
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db.  0/28831084
> bytes for 104856/104860 keys read/written.  Time: 8119ms.
> INFO - Flushing Memtable(Super1)@552364977
> DEBUG - Submitting Super1 for compaction
> INFO - Completed flushing Memtable(Super1)@552364977
> INFO - Flushing Memtable(Standard1)@1290243769
> DEBUG - Submitting Standard1 for compaction
> INFO - Completed flushing Memtable(Standard1)@1290243769
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db   : 256
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db   : 512
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db   : 768
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db   : 1024
> DEBUG - Expected bloom filter size : 1024
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db.  0/210 bytes
> for 0/1 keys read/written.  Time: 301ms.
> DEBUG - Starting to listen on 127.0.0.1:7001
> INFO - Cassandra starting up...
>
>
>
> On Mon, Jul 13, 2009 at 6:06 AM, [hidden email] <[hidden email]>
> wrote:
>>
>> how do i find out if JVM is GCing?
>>
>> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <[hidden email]>
>> wrote:
>>>
>>> What hardware are you running one?
>>
>>  dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
>> operating system is fedora core 9
>>
>>
>>> How long does the slowdown last ?
>>
>> i stopped inserting data after slowdown starts, and it is still slow now
>> after over 10 hours.
>> however if i stop cassandra and start it, it becomes super fast
>> immediately. till i insert another 100k or so rows when it becomes really
>> slow again.
>>
>>
>>>
>>>  There are a few reasons for temporary slowdowns ... perhaps the JVM
>>> started GCing?
>>
>> how do i find out if this is the cause?
>>
>>
>>>
>>> Cassandra spends time cleaning up the on-disk SSTables
>>> in a process called compaction. This could cause the client to observe
>>> a slowdown.
>>>
>>> Things you could try --
>>> Reduce the Memtable size in the config files. (If GCing was the problem)
>>> Increasing the number of SSTables written before compaction kicks in.
>>
>> can you tell me what numbers i should use?
>>
>> thanks!
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

Kevin Castiglione
Jonathan,
good to know this is normal. but the reason i sent this is , once i stop and start cassandra it gets ready in few seconds and insertion and get_slices are super fast like few milliseconds.

but after it starts to slow down even after 10 hours it is still slow. why is this the case?

On Mon, Jul 13, 2009 at 6:14 AM, Jonathan Ellis <[hidden email]> wrote:
Cassandra is replaying the transaction log and preloading SSTable
indexes.  This is normal.

On Mon, Jul 13, 2009 at 8:10 AM, [hidden email]<[hidden email]> wrote:
> when i stop cassandra and start it again, this is what is printed. it takes
> just a couple of seconds for this to run.
> and after that it becomes really fast.
>
>
> Listening for transport dt_socket at address: 8888
> DEBUG - Loading settings from ./../conf/storage-conf.xml
> DEBUG - adding Super1 as 0
> DEBUG - adding Standard2 as 1
> DEBUG - adding Standard1 as 2
> DEBUG - adding StandardByTime1 as 3
> DEBUG - adding LocationInfo as 4
> DEBUG - adding HintsColumnFamily as 5
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
> INFO - Replaying
> /home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db   : 73600
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db   : 84224
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db   : 94848
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db   : 105472
> DEBUG - Expected bloom filter size : 105472
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db.  0/28831084
> bytes for 104856/104860 keys read/written.  Time: 8119ms.
> INFO - Flushing Memtable(Super1)@552364977
> DEBUG - Submitting Super1 for compaction
> INFO - Completed flushing Memtable(Super1)@552364977
> INFO - Flushing Memtable(Standard1)@1290243769
> DEBUG - Submitting Standard1 for compaction
> INFO - Completed flushing Memtable(Standard1)@1290243769
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db   : 256
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db   : 512
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db   : 768
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db   : 1024
> DEBUG - Expected bloom filter size : 1024
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db.  0/210 bytes
> for 0/1 keys read/written.  Time: 301ms.
> DEBUG - Starting to listen on 127.0.0.1:7001
> INFO - Cassandra starting up...
>
>
>
> On Mon, Jul 13, 2009 at 6:06 AM, [hidden email] <[hidden email]>
> wrote:
>>
>> how do i find out if JVM is GCing?
>>
>> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <[hidden email]>
>> wrote:
>>>
>>> What hardware are you running one?
>>
>>  dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
>> operating system is fedora core 9
>>
>>
>>> How long does the slowdown last ?
>>
>> i stopped inserting data after slowdown starts, and it is still slow now
>> after over 10 hours.
>> however if i stop cassandra and start it, it becomes super fast
>> immediately. till i insert another 100k or so rows when it becomes really
>> slow again.
>>
>>
>>>
>>>  There are a few reasons for temporary slowdowns ... perhaps the JVM
>>> started GCing?
>>
>> how do i find out if this is the cause?
>>
>>
>>>
>>> Cassandra spends time cleaning up the on-disk SSTables
>>> in a process called compaction. This could cause the client to observe
>>> a slowdown.
>>>
>>> Things you could try --
>>> Reduce the Memtable size in the config files. (If GCing was the problem)
>>> Increasing the number of SSTables written before compaction kicks in.
>>
>> can you tell me what numbers i should use?
>>
>> thanks!
>>
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

rkmr.em@gmail.com
In reply to this post by Jonathan Ellis-3
Jonathan,
good to know this is normal. but the reason i sent this is , once i stop and start cassandra it gets ready in few seconds and insertion and get_slices are super fast like few milliseconds.

but after it starts to slow down even after 10 hours it is still slow. why is this the case?

On Mon, Jul 13, 2009 at 6:14 AM, Jonathan Ellis <[hidden email]> wrote:
Cassandra is replaying the transaction log and preloading SSTable
indexes.  This is normal.

On Mon, Jul 13, 2009 at 8:10 AM, [hidden email]<[hidden email]> wrote:
> when i stop cassandra and start it again, this is what is printed. it takes
> just a couple of seconds for this to run.
> and after that it becomes really fast.
>
>
> Listening for transport dt_socket at address: 8888
> DEBUG - Loading settings from ./../conf/storage-conf.xml
> DEBUG - adding Super1 as 0
> DEBUG - adding Standard2 as 1
> DEBUG - adding Standard1 as 2
> DEBUG - adding StandardByTime1 as 3
> DEBUG - adding LocationInfo as 4
> DEBUG - adding HintsColumnFamily as 5
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
> INFO - Replaying
> /home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db   : 73600
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db   : 84224
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db   : 94848
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db   : 105472
> DEBUG - Expected bloom filter size : 105472
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db.  0/28831084
> bytes for 104856/104860 keys read/written.  Time: 8119ms.
> INFO - Flushing Memtable(Super1)@552364977
> DEBUG - Submitting Super1 for compaction
> INFO - Completed flushing Memtable(Super1)@552364977
> INFO - Flushing Memtable(Standard1)@1290243769
> DEBUG - Submitting Standard1 for compaction
> INFO - Completed flushing Memtable(Standard1)@1290243769
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db   : 256
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db   : 512
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db   : 768
> DEBUG - index size for bloom filter calc for file  :
> /home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db   : 1024
> DEBUG - Expected bloom filter size : 1024
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db.  0/210 bytes
> for 0/1 keys read/written.  Time: 301ms.
> DEBUG - Starting to listen on 127.0.0.1:7001
> INFO - Cassandra starting up...
>
>
>
> On Mon, Jul 13, 2009 at 6:06 AM, [hidden email] <[hidden email]>
> wrote:
>>
>> how do i find out if JVM is GCing?
>>
>> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <[hidden email]>
>> wrote:
>>>
>>> What hardware are you running one?
>>
>>  dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
>> operating system is fedora core 9
>>
>>
>>> How long does the slowdown last ?
>>
>> i stopped inserting data after slowdown starts, and it is still slow now
>> after over 10 hours.
>> however if i stop cassandra and start it, it becomes super fast
>> immediately. till i insert another 100k or so rows when it becomes really
>> slow again.
>>
>>
>>>
>>>  There are a few reasons for temporary slowdowns ... perhaps the JVM
>>> started GCing?
>>
>> how do i find out if this is the cause?
>>
>>
>>>
>>> Cassandra spends time cleaning up the on-disk SSTables
>>> in a process called compaction. This could cause the client to observe
>>> a slowdown.
>>>
>>> Things you could try --
>>> Reduce the Memtable size in the config files. (If GCing was the problem)
>>> Increasing the number of SSTables written before compaction kicks in.
>>
>> can you tell me what numbers i should use?
>>
>> thanks!
>>
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

rkmr.em@gmail.com
In reply to this post by Jonathan Ellis-3
On Mon, Jul 13, 2009 at 6:03 AM, Jonathan Ellis <[hidden email]> wrote:
On Mon, Jul 13, 2009 at 12:37 AM, Sandeep Tata<[hidden email]> wrote:
> What hardware are you running one? How long does the slowdown last ?
>  There are a few reasons for temporary slowdowns ... perhaps the JVM
> started GCing?

Every time someone has reported this symptom, that has been the problem.

The object count tunable is the most direct way to ameliorate this.

http://wiki.apache.org/cassandra/MemtableThresholds

this is my current setting:

    <MemtableObjectCountInMillions>0.02</MemtableObjectCountInMillions>

should i increase or decrease it?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cassandra slows down after inserts

Jonathan Ellis-3
decrease

On Mon, Jul 13, 2009 at 8:53 AM, [hidden email]<[hidden email]> wrote:

> On Mon, Jul 13, 2009 at 6:03 AM, Jonathan Ellis <[hidden email]> wrote:
>>
>> On Mon, Jul 13, 2009 at 12:37 AM, Sandeep Tata<[hidden email]>
>> wrote:
>> > What hardware are you running one? How long does the slowdown last ?
>> >  There are a few reasons for temporary slowdowns ... perhaps the JVM
>> > started GCing?
>>
>> Every time someone has reported this symptom, that has been the problem.
>>
>> The object count tunable is the most direct way to ameliorate this.
>>
>> http://wiki.apache.org/cassandra/MemtableThresholds
>
> this is my current setting:
>
>     <MemtableObjectCountInMillions>0.02</MemtableObjectCountInMillions>
>
> should i increase or decrease it?
>
Loading...