Replacing Redis

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Replacing Redis

Benson Margulies
I'm about to launch off on replacing redis with cassandra. I wonder if
anyone else has ever been there and done that.
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Redis

Joshua Partogi
Any reason why you want to do that?

On Sat, Feb 19, 2011 at 1:32 AM, Benson Margulies <[hidden email]> wrote:
> I'm about to launch off on replacing redis with cassandra. I wonder if
> anyone else has ever been there and done that.
>



--
http://twitter.com/jpartogi
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Redis

Benson Margulies
redis times out at random regardless of what we configure for client
timeouts; the platform-sensitive binaries are painful for us since we
support many platform; just to name two reasons.

On Fri, Feb 18, 2011 at 10:04 AM, Joshua Partogi <[hidden email]> wrote:

> Any reason why you want to do that?
>
> On Sat, Feb 19, 2011 at 1:32 AM, Benson Margulies <[hidden email]> wrote:
>> I'm about to launch off on replacing redis with cassandra. I wonder if
>> anyone else has ever been there and done that.
>>
>
>
>
> --
> http://twitter.com/jpartogi
>
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Redis

Jonathan Shook
Benson,
I was considering using Redis for a specific project. Can you
elaborate a bit on your problem with it? What were the circumstances,
loading factors, etc?

On Fri, Feb 18, 2011 at 9:19 AM, Benson Margulies <[hidden email]> wrote:

> redis times out at random regardless of what we configure for client
> timeouts; the platform-sensitive binaries are painful for us since we
> support many platform; just to name two reasons.
>
> On Fri, Feb 18, 2011 at 10:04 AM, Joshua Partogi <[hidden email]> wrote:
>> Any reason why you want to do that?
>>
>> On Sat, Feb 19, 2011 at 1:32 AM, Benson Margulies <[hidden email]> wrote:
>>> I'm about to launch off on replacing redis with cassandra. I wonder if
>>> anyone else has ever been there and done that.
>>>
>>
>>
>>
>> --
>> http://twitter.com/jpartogi
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Redis

Benson Margulies
typical experiment.

Redis 2.0.4 deployed on my macbook pro.

Saves enabled.

appendfsync off.

vm enabled, 1g max memory.

72 databases. Each database asked to store 13*N key-value pairs with
lpush, bucket size not very big, N -> 500,000.

Client jredis.

Start running against a stream of inputs. Sooner or later, client
times out 'last operation not performed'.

These could be jredis bugs, but I don't care to find out.
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Redis

Gaurav Sharma
Try Jedis or better-still JOhm if you would like to have automatic object mapping to Redis. You can select a connection pool of your choice even though we recommend you use Apache Commons Pool.

Cassandra and Redis are very different beasts and work best at their respective scales. If you just want a simple cache, Redis can do the job - it is really fast and operations are atomic and with AOF, data-loss window can be reduced to max transactions occurring in 1 second :)

All that being said, why are you doing an lpush (list-op) on your KV pairs and not hset (with pipelining if possible)?

Also, try the v2.2 by directly building off of the redis main branch. It is much faster than v2.0 and Jedis/JOhm are compatible with 2.2


On Fri, Feb 18, 2011 at 12:00 PM, Benson Margulies <[hidden email]> wrote:
typical experiment.

Redis 2.0.4 deployed on my macbook pro.

Saves enabled.

appendfsync off.

vm enabled, 1g max memory.

72 databases. Each database asked to store 13*N key-value pairs with
lpush, bucket size not very big, N -> 500,000.

Client jredis.

Start running against a stream of inputs. Sooner or later, client
times out 'last operation not performed'.

These could be jredis bugs, but I don't care to find out.

Reply | Threaded
Open this post in threaded view
|

Re: Replacing Redis

Benson Margulies
Thanks for replying, but it is not at all clear to me that redis was
ever the right solution to the problem at hand. I'm still working out
whether, in fact, Cassandra is.

Up to a fairly large scale, this particular problem needs no 'cache'
at all. It needs a write-behind persistence mechanism -- a log. The
simplest form of which is, of course, just a file, though that
solution poses problems for backup and some failure scenarios.

There are two parts to the data footprint, one growing more rapidly
than the other. The faster growing part has a natural 'redis-friendly'
data model (keys map to values); but it would map to EHCache as well.
However, the initial redis/jredis implementation used redis, rather
than local ordinary JVM objects, for the slower growing part. The
lpush/ltrim usage was in here.

At really gigantic scales, which I don't claim to have entirely
thought through, even the slow-growing part will stop fitting into
memory. However, it has a natural architecture for distribution, so
there is a dilemma in my head between following the logic of the
algorithm out to implement distribution, or letting something like
cassandra do that for me.
Reply | Threaded
Open this post in threaded view
|

Re: Replacing Redis

David Strauss-2
On Sun, 2011-02-20 at 09:04 -0500, Benson Margulies wrote:
> There are two parts to the data footprint, one growing more rapidly
> than the other. The faster growing part has a natural 'redis-friendly'
> data model (keys map to values); but it would map to EHCache as well.

If you're coming into the non-relational world with the perspective that
the main consideration is your "data model," you're approaching the
problem from the wrong direction. The non-relational world isn't about
BCNF or foreign keys. Storing and retrieving individual items is the
easy part; we need to understand how you intend to *use* the data to
help you understand if Cassandra is a good fit.

      * How do you add new data?
      * Other than accessing individual items by their primary key (or
        equivalent), how do you read it?
      * What needs to be real-time? Asynchronous?
      * What correctness or consistency constraints can you relax?
      * What availability needs do you have?
      * What programming languages are you working in?

> However, the initial redis/jredis implementation used redis, rather
> than local ordinary JVM objects, for the slower growing part. The
> lpush/ltrim usage was in here.

It's not clear just from your use of lpush/ltrim what you're doing. Some
uses can map to Cassandra's APIs with some creative reinterpretation;
others can't.

--
David Strauss
   | [hidden email]
   | +1 512 577 5827 [mobile]


signature.asc (853 bytes) Download Attachment