sstable loader

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

sstable loader

Rahul Bhardwaj
Hi All,

 Can we use sstable loader for loading external flat file or csv file.
If yes , kindly share the steps or manual.

I need to put 40 million data into a table of around 70 columns



Regards:
Rahul Bhardwaj





Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!
Reply | Threaded
Open this post in threaded view
|

Re: sstable loader

Amila Paranawithana
Hi,

This post[1] may be useful. But note that this was done with cassandra older version. So there may be new way to do this.

[1]. http://amilaparanawithana.blogspot.com/2012/06/bulk-loading-external-data-to-cassandra.html

Thanks,


On Fri, Mar 27, 2015 at 11:40 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi All,

 Can we use sstable loader for loading external flat file or csv file.
If yes , kindly share the steps or manual.

I need to put 40 million data into a table of around 70 columns



Regards:
Rahul Bhardwaj





Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!



--

Amila Iroshani Paranawithana , Senior Software Engineer , AdroitLogic
| ☎: +94779747398
| ✍: http://amilaparanawithana.blogspot.com

Facebook Twitter LinkedIn Skype amila.paranawithana
Reply | Threaded
Open this post in threaded view
|

Re: sstable loader

Rahul Bhardwaj
Hi Amila ,

I tried your code. In which we made some modification accordingly like:

public class DataImportExample
{
    static String filename;

    public static void main(String[] args) throws IOException
    {
        filename = "/root/perl_work/abc.csv";
        BufferedReader reader = new BufferedReader(new FileReader(filename));
        String keyspace = "mesh_glusr";
        File directory = new File(keyspace);
        if (!directory.exists()){
          directory.mkdir();}

        // random partitioner is created, u can give the partitioner as u want
        IPartitioner partitioner = new RandomPartitioner();

        SSTableSimpleUnsortedWriter usersWriter = new SSTableSimpleUnsortedWriter(
                directory,partitioner,keyspace,"test1",AsciiType.instance,null,64);


after compiling when we executed it with below command :
"java  -cp /root/perl_work DataImportExample"

it throws below error:

Expecting URI in variable: [cassandra.config].  Please prefix the file with file:/// for local files or file://<server>/ for remote files.  Aborting.
Fatal configuration error; unable to start. See log for stacktrace.

We are not able to find what went wrong , actually not good java developer, so plz guide.


Regards:
Rahul Bhardwaj

On Fri, Mar 27, 2015 at 2:55 PM, Amila Paranawithana <[hidden email]> wrote:
Hi,

This post[1] may be useful. But note that this was done with cassandra older version. So there may be new way to do this.

[1]. http://amilaparanawithana.blogspot.com/2012/06/bulk-loading-external-data-to-cassandra.html

Thanks,


On Fri, Mar 27, 2015 at 11:40 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi All,

 Can we use sstable loader for loading external flat file or csv file.
If yes , kindly share the steps or manual.

I need to put 40 million data into a table of around 70 columns



Regards:
Rahul Bhardwaj





Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!



--

Amila Iroshani Paranawithana , Senior Software Engineer , AdroitLogic
| ☎: +94779747398
| ✍: http://amilaparanawithana.blogspot.com

Facebook Twitter LinkedIn Skype amila.paranawithana



Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!
Reply | Threaded
Open this post in threaded view
|

Re: sstable loader

Vanessa Gligor
Hi,

I used this https://github.com/yukim/cassandra-bulkload-example/ (I have modified BulkLoad.java for my needs) for the sstable loader and it works ok. You can take a look, maybe it will help you.

Regards,
Vanessa.

On Mon, Mar 30, 2015 at 10:04 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi Amila ,

I tried your code. In which we made some modification accordingly like:

public class DataImportExample
{
    static String filename;

    public static void main(String[] args) throws IOException
    {
        filename = "/root/perl_work/abc.csv";
        BufferedReader reader = new BufferedReader(new FileReader(filename));
        String keyspace = "mesh_glusr";
        File directory = new File(keyspace);
        if (!directory.exists()){
          directory.mkdir();}

        // random partitioner is created, u can give the partitioner as u want
        IPartitioner partitioner = new RandomPartitioner();

        SSTableSimpleUnsortedWriter usersWriter = new SSTableSimpleUnsortedWriter(
                directory,partitioner,keyspace,"test1",AsciiType.instance,null,64);


after compiling when we executed it with below command :
"java  -cp /root/perl_work DataImportExample"

it throws below error:

Expecting URI in variable: [cassandra.config].  Please prefix the file with file:/// for local files or file://<server>/ for remote files.  Aborting.
Fatal configuration error; unable to start. See log for stacktrace.

We are not able to find what went wrong , actually not good java developer, so plz guide.


Regards:
Rahul Bhardwaj

On Fri, Mar 27, 2015 at 2:55 PM, Amila Paranawithana <[hidden email]> wrote:
Hi,

This post[1] may be useful. But note that this was done with cassandra older version. So there may be new way to do this.

[1]. http://amilaparanawithana.blogspot.com/2012/06/bulk-loading-external-data-to-cassandra.html

Thanks,


On Fri, Mar 27, 2015 at 11:40 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi All,

 Can we use sstable loader for loading external flat file or csv file.
If yes , kindly share the steps or manual.

I need to put 40 million data into a table of around 70 columns



Regards:
Rahul Bhardwaj





Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!



--

Amila Iroshani Paranawithana , Senior Software Engineer , AdroitLogic
| ☎: <a href="tel:%2B94779747398" value="+94779747398" target="_blank">+94779747398
| ✍: http://amilaparanawithana.blogspot.com

Facebook Twitter LinkedIn Skype amila.paranawithana



Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!

Reply | Threaded
Open this post in threaded view
|

Re: sstable loader

Rahul Bhardwaj
Hi Venessa,

Thanks for sharing.

But when after compiling BulkLoad.java, on executing it is returning 

Exception in thread "main" java.lang.NoClassDefFoundError: BulkLoad (wrong name: bulkload/BulkLoad)

had you also seen this.


Regards:
Rahul Bhardwaj

On Mon, Mar 30, 2015 at 12:57 PM, Vanessa Gligor <[hidden email]> wrote:
Hi,

I used this https://github.com/yukim/cassandra-bulkload-example/ (I have modified BulkLoad.java for my needs) for the sstable loader and it works ok. You can take a look, maybe it will help you.

Regards,
Vanessa.

On Mon, Mar 30, 2015 at 10:04 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi Amila ,

I tried your code. In which we made some modification accordingly like:

public class DataImportExample
{
    static String filename;

    public static void main(String[] args) throws IOException
    {
        filename = "/root/perl_work/abc.csv";
        BufferedReader reader = new BufferedReader(new FileReader(filename));
        String keyspace = "mesh_glusr";
        File directory = new File(keyspace);
        if (!directory.exists()){
          directory.mkdir();}

        // random partitioner is created, u can give the partitioner as u want
        IPartitioner partitioner = new RandomPartitioner();

        SSTableSimpleUnsortedWriter usersWriter = new SSTableSimpleUnsortedWriter(
                directory,partitioner,keyspace,"test1",AsciiType.instance,null,64);


after compiling when we executed it with below command :
"java  -cp /root/perl_work DataImportExample"

it throws below error:

Expecting URI in variable: [cassandra.config].  Please prefix the file with file:/// for local files or file://<server>/ for remote files.  Aborting.
Fatal configuration error; unable to start. See log for stacktrace.

We are not able to find what went wrong , actually not good java developer, so plz guide.


Regards:
Rahul Bhardwaj

On Fri, Mar 27, 2015 at 2:55 PM, Amila Paranawithana <[hidden email]> wrote:
Hi,

This post[1] may be useful. But note that this was done with cassandra older version. So there may be new way to do this.

[1]. http://amilaparanawithana.blogspot.com/2012/06/bulk-loading-external-data-to-cassandra.html

Thanks,


On Fri, Mar 27, 2015 at 11:40 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi All,

 Can we use sstable loader for loading external flat file or csv file.
If yes , kindly share the steps or manual.

I need to put 40 million data into a table of around 70 columns



Regards:
Rahul Bhardwaj





Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!



--

Amila Iroshani Paranawithana , Senior Software Engineer , AdroitLogic
| ☎: <a href="tel:%2B94779747398" value="+94779747398" target="_blank">+94779747398
| ✍: http://amilaparanawithana.blogspot.com

Facebook Twitter LinkedIn Skype amila.paranawithana



Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!




Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!
Reply | Threaded
Open this post in threaded view
|

Re: sstable loader

Amila Paranawithana
Hi Rahul,

By just seeing the error I guess the file name you have given need to be changed to to format file:///.... .Not sure anyway.

Cheers!
Amila

On Mon, Mar 30, 2015 at 1:46 PM, Rahul Bhardwaj <[hidden email]> wrote:
Hi Venessa,

Thanks for sharing.

But when after compiling BulkLoad.java, on executing it is returning 

Exception in thread "main" java.lang.NoClassDefFoundError: BulkLoad (wrong name: bulkload/BulkLoad)

had you also seen this.


Regards:
Rahul Bhardwaj

On Mon, Mar 30, 2015 at 12:57 PM, Vanessa Gligor <[hidden email]> wrote:
Hi,

I used this https://github.com/yukim/cassandra-bulkload-example/ (I have modified BulkLoad.java for my needs) for the sstable loader and it works ok. You can take a look, maybe it will help you.

Regards,
Vanessa.

On Mon, Mar 30, 2015 at 10:04 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi Amila ,

I tried your code. In which we made some modification accordingly like:

public class DataImportExample
{
    static String filename;

    public static void main(String[] args) throws IOException
    {
        filename = "/root/perl_work/abc.csv";
        BufferedReader reader = new BufferedReader(new FileReader(filename));
        String keyspace = "mesh_glusr";
        File directory = new File(keyspace);
        if (!directory.exists()){
          directory.mkdir();}

        // random partitioner is created, u can give the partitioner as u want
        IPartitioner partitioner = new RandomPartitioner();

        SSTableSimpleUnsortedWriter usersWriter = new SSTableSimpleUnsortedWriter(
                directory,partitioner,keyspace,"test1",AsciiType.instance,null,64);


after compiling when we executed it with below command :
"java  -cp /root/perl_work DataImportExample"

it throws below error:

Expecting URI in variable: [cassandra.config].  Please prefix the file with file:/// for local files or file://<server>/ for remote files.  Aborting.
Fatal configuration error; unable to start. See log for stacktrace.

We are not able to find what went wrong , actually not good java developer, so plz guide.


Regards:
Rahul Bhardwaj

On Fri, Mar 27, 2015 at 2:55 PM, Amila Paranawithana <[hidden email]> wrote:
Hi,

This post[1] may be useful. But note that this was done with cassandra older version. So there may be new way to do this.

[1]. http://amilaparanawithana.blogspot.com/2012/06/bulk-loading-external-data-to-cassandra.html

Thanks,


On Fri, Mar 27, 2015 at 11:40 AM, Rahul Bhardwaj <[hidden email]> wrote:
Hi All,

 Can we use sstable loader for loading external flat file or csv file.
If yes , kindly share the steps or manual.

I need to put 40 million data into a table of around 70 columns



Regards:
Rahul Bhardwaj





Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!



--

Amila Iroshani Paranawithana , Senior Software Engineer , AdroitLogic
| ☎: <a href="tel:%2B94779747398" value="+94779747398" target="_blank">+94779747398
| ✍: http://amilaparanawithana.blogspot.com

Facebook Twitter LinkedIn Skype amila.paranawithana



Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!




Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how IndiaMART Maximiser helped Mr. Khanna expand his business. kyunki Kaam Yahin Banta Hai!!!



--

Amila Iroshani Paranawithana , Senior Software Engineer , AdroitLogic
| ☎: +94779747398
| ✍: http://amilaparanawithana.blogspot.com

Facebook Twitter LinkedIn Skype amila.paranawithana