bulk data load

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

bulk data load

koray mehmet
Hi, 

I have to add a column and value to current data. I tried with batch statement but got warning at cassandra as 

WARN [Native-Transport-Requests:1423504] 2014-10-22 16:49:11,426 BatchStatement.java (line 223) Batch of prepared statements for [keyspace.table] is of size 26600, exceeding specified threshold of 5120 by 21480.

After that messages, could not write to cassandra also.

How should i go on?

Thanks
Koray Sariteke
Reply | Threaded
Open this post in threaded view
|

Re: bulk data load

Juho Mäkinen
You should split your batch statements into smaller batches, say 100 operations per batch (or less if you keep getting those errors). You can also grow the batch_size_warn_threshold_in_kb in your cassandra.yaml a bit, I'm using 20kb in my cluster. You can read more from the relevant Jira: https://issues.apache.org/jira/browse/CASSANDRA-6487

On Thu, Oct 30, 2014 at 11:36 AM, koray mehmet <[hidden email]> wrote:
Hi, 

I have to add a column and value to current data. I tried with batch statement but got warning at cassandra as 

WARN [Native-Transport-Requests:1423504] 2014-10-22 16:49:11,426 BatchStatement.java (line 223) Batch of prepared statements for [keyspace.table] is of size 26600, exceeding specified threshold of 5120 by 21480.

After that messages, could not write to cassandra also.

How should i go on?

Thanks
Koray Sariteke

Reply | Threaded
Open this post in threaded view
|

Re: bulk data load

hnahak
This post has NOT been accepted by the mailing list yet.
Hi

I have 500 MB csv file to load using java, but the  data load  is very slow.

it took 6096.61 seconds for 10,000,000 rows. are there any best way to load it fast, if  I have restriction to load  it from program? i.e.. without using COPY or sstable ingestion.

From java  code I connect to keyspace and table/column family and read a line from csv and INSERT into the table..

any help guys !!