Delayed events processing / queue (anti-)pattern

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Delayed events processing / queue (anti-)pattern

RobinUs2
Hi there,

I'm working on a project that needs delayed event processing. Example:

Today a certain set of conditions X is met, and in 7 days and 12 hours (delay Y) we want to take action Z.

The set of conditions X is out of the scope of this problem. 
The delay Y will be at least 1 minute, and at most 90 days with a resolution per minute. 
The action Z will be describe by a JSON string.

Other details:
- for every point in the future there are probably hundreds of actions which have to be processed
- all actions for a point in time will be processed at once (thus not removing action by action as a typical queue would do)
- once all actions have been processed we remove the entire row (by key, not the individual columns)

Now I'm aware of the fact that Cassandra is not suited well for queues (which is basically what we want, but then hundreds / thousands of queues for a specific point in time in the future). See http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets for more details

Basically I the columnfamily to contain around 130K (90*1440) rows at most. Every record will have between 1 and a few million (average expected around few hundred thousand) columns where key is a timeuuid and the value is a byte array / string of a few kilobytes at most.

Why not use real queues? We currently have Kafka and Cassandra in our infra that would be suitable to this, however creating a queue for every possible moment in time would need a lot (up to ~ 130K) in queues, which is not what Kafka is meant for. 

Why not a different datastore? We could use redis which I think might be better suited for this, but we don't have that running. 

I think this should work well given the details and usage pattern. 

Do you have any thoughts on using Cassandra for this? Or should we use a different system, like Redis?

Thanks.

Best regards, 

Robin Verlangen
Chief Data Architect




Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

Robert Coli-3
On Tue, Mar 24, 2015 at 5:05 AM, Robin Verlangen <[hidden email]> wrote:
- for every point in the future there are probably hundreds of actions which have to be processed
- all actions for a point in time will be processed at once (thus not removing action by action as a typical queue would do)
- once all actions have been processed we remove the entire row (by key, not the individual columns)

I've used Cassandra for similar queue-like things, and it's "fine." Not ideal, but number of objects and access patterns are "fine."


This design never truncates history, but if you can tolerate throwing away history, that problem goes away..

=Rob
 
Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

DuyHai Doan
Some ideas I throw in here:

"The delay Y will be at least 1 minute, and at most 90 days with a resolution per minute" --> Use the delay (with format YYYYMMDDHHMM as integer) as your partition key.

Example: today March 24th at 12:00 (201502241200) you need to delay 3 actions, action A in exact 3 days, action B in 10 hours and action C in 5 minutes. Thus you will create 3 partitions:

- for A, partition key = 201503271200
- for B, partition key = 201503242200
- for C, partition key = 201503241205

In each partition, you'll need to create as many clustering columns as there are actions to execute. According to your estimate, the average is a few hundred thousands and the max is a few millions so it's fine. Also, you would have a pool of worker which will load the whole partition (with paging when necessary) every minute and process the actions.

Once all the actions have been executed, you can either remove the complete partition or keep them for archiving.

Duy Hai DOAN 

On Tue, Mar 24, 2015 at 9:19 PM, Robert Coli <[hidden email]> wrote:
On Tue, Mar 24, 2015 at 5:05 AM, Robin Verlangen <[hidden email]> wrote:
- for every point in the future there are probably hundreds of actions which have to be processed
- all actions for a point in time will be processed at once (thus not removing action by action as a typical queue would do)
- once all actions have been processed we remove the entire row (by key, not the individual columns)

I've used Cassandra for similar queue-like things, and it's "fine." Not ideal, but number of objects and access patterns are "fine."


This design never truncates history, but if you can tolerate throwing away history, that problem goes away..

=Rob
 

Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

RobinUs2
Hi there,

@Robert: can you elaborate a bit more on the "not ideal" parts? In my case I will be throwing away the rows (thus the points in time that are "now in the past"), which will create tombstones which are compacted away.

@DuyHai: that was exactly what I had in mind and from a C* point of view this should work as it's write heavy. I add hundreds of thousands of columns to a key, and then read them all at once (or maybe a few times with pagination), and then remove the entire row by it's primary key.

Any other thoughts on this?

Best regards, 

Robin Verlangen
Chief Data Architect




Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.

On Tue, Mar 24, 2015 at 10:01 PM, DuyHai Doan <[hidden email]> wrote:
Some ideas I throw in here:

"The delay Y will be at least 1 minute, and at most 90 days with a resolution per minute" --> Use the delay (with format YYYYMMDDHHMM as integer) as your partition key.

Example: today March 24th at 12:00 (201502241200) you need to delay 3 actions, action A in exact 3 days, action B in 10 hours and action C in 5 minutes. Thus you will create 3 partitions:

- for A, partition key = 201503271200
- for B, partition key = 201503242200
- for C, partition key = 201503241205

In each partition, you'll need to create as many clustering columns as there are actions to execute. According to your estimate, the average is a few hundred thousands and the max is a few millions so it's fine. Also, you would have a pool of worker which will load the whole partition (with paging when necessary) every minute and process the actions.

Once all the actions have been executed, you can either remove the complete partition or keep them for archiving.

Duy Hai DOAN 

On Tue, Mar 24, 2015 at 9:19 PM, Robert Coli <[hidden email]> wrote:
On Tue, Mar 24, 2015 at 5:05 AM, Robin Verlangen <[hidden email]> wrote:
- for every point in the future there are probably hundreds of actions which have to be processed
- all actions for a point in time will be processed at once (thus not removing action by action as a typical queue would do)
- once all actions have been processed we remove the entire row (by key, not the individual columns)

I've used Cassandra for similar queue-like things, and it's "fine." Not ideal, but number of objects and access patterns are "fine."


This design never truncates history, but if you can tolerate throwing away history, that problem goes away..

=Rob
 


Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

Robert Coli-3
On Wed, Mar 25, 2015 at 12:45 AM, Robin Verlangen <[hidden email]> wrote:
@Robert: can you elaborate a bit more on the "not ideal" parts? In my case I will be throwing away the rows (thus the points in time that are "now in the past"), which will create tombstones which are compacted away.

"Not ideal" is what I mean... Cassandra has immutable data files, use cases which do DELETE pay an obvious penalty. Some percentage of tombstones will exist continuously, and you have to store them and seek past them.

=Rob
 
Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

Thunder Stumpges

Would it help here to not actually issue a delete statement but instead use date based compaction and a dynamically calculated ttl that is some safe distance in the future from your key?

Just a thought.
-Thunder

On Mar 25, 2015 11:07 AM, "Robert Coli" <[hidden email]> wrote:
On Wed, Mar 25, 2015 at 12:45 AM, Robin Verlangen <[hidden email]> wrote:
@Robert: can you elaborate a bit more on the "not ideal" parts? In my case I will be throwing away the rows (thus the points in time that are "now in the past"), which will create tombstones which are compacted away.

"Not ideal" is what I mean... Cassandra has immutable data files, use cases which do DELETE pay an obvious penalty. Some percentage of tombstones will exist continuously, and you have to store them and seek past them.

=Rob
 
Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

RobinUs2
Interesting thought, that should work indeed, I'll evaluate both options and provide an update here once I have results.

Best regards, 

Robin Verlangen
Chief Data Architect




Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.

On Thu, Mar 26, 2015 at 7:09 AM, Thunder Stumpges <[hidden email]> wrote:

Would it help here to not actually issue a delete statement but instead use date based compaction and a dynamically calculated ttl that is some safe distance in the future from your key?

Just a thought.
-Thunder

On Mar 25, 2015 11:07 AM, "Robert Coli" <[hidden email]> wrote:
On Wed, Mar 25, 2015 at 12:45 AM, Robin Verlangen <[hidden email]> wrote:
@Robert: can you elaborate a bit more on the "not ideal" parts? In my case I will be throwing away the rows (thus the points in time that are "now in the past"), which will create tombstones which are compacted away.

"Not ideal" is what I mean... Cassandra has immutable data files, use cases which do DELETE pay an obvious penalty. Some percentage of tombstones will exist continuously, and you have to store them and seek past them.

=Rob
 

Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

Brice Dutheil

Would it help here to not actually issue a delete statement but instead use date based compaction and a dynamically calculated ttl that is some safe distance in the future from your key?

I’m not sure about about this part date based compaction, do you mean DateTieredCompationStrategy ?

Anyway we achieved something like that without this strategy with a TTL + date in partition key based approach. The thing however to watch is the size of the partition (one should avoid too long partitions (in thrift wide rows)), so care must be taken on the date increment to be correctly adjusted.


-- Brice

On Thu, Mar 26, 2015 at 5:23 PM, Robin Verlangen <[hidden email]> wrote:
Interesting thought, that should work indeed, I'll evaluate both options and provide an update here once I have results.

Best regards, 

Robin Verlangen
Chief Data Architect




Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.

On Thu, Mar 26, 2015 at 7:09 AM, Thunder Stumpges <[hidden email]> wrote:

Would it help here to not actually issue a delete statement but instead use date based compaction and a dynamically calculated ttl that is some safe distance in the future from your key?

Just a thought.
-Thunder

On Mar 25, 2015 11:07 AM, "Robert Coli" <[hidden email]> wrote:
On Wed, Mar 25, 2015 at 12:45 AM, Robin Verlangen <[hidden email]> wrote:
@Robert: can you elaborate a bit more on the "not ideal" parts? In my case I will be throwing away the rows (thus the points in time that are "now in the past"), which will create tombstones which are compacted away.

"Not ideal" is what I mean... Cassandra has immutable data files, use cases which do DELETE pay an obvious penalty. Some percentage of tombstones will exist continuously, and you have to store them and seek past them.

=Rob
 


Reply | Threaded
Open this post in threaded view
|

Re: Delayed events processing / queue (anti-)pattern

Thunder Stumpges

Yeah that's the one :) sorry, was on my phone and didn't want to look up the exact name.

Cheers,
Thunder

On Mar 27, 2015 6:17 AM, "Brice Dutheil" <[hidden email]> wrote:

Would it help here to not actually issue a delete statement but instead use date based compaction and a dynamically calculated ttl that is some safe distance in the future from your key?

I’m not sure about about this part date based compaction, do you mean DateTieredCompationStrategy ?

Anyway we achieved something like that without this strategy with a TTL + date in partition key based approach. The thing however to watch is the size of the partition (one should avoid too long partitions (in thrift wide rows)), so care must be taken on the date increment to be correctly adjusted.


-- Brice

On Thu, Mar 26, 2015 at 5:23 PM, Robin Verlangen <[hidden email]> wrote:
Interesting thought, that should work indeed, I'll evaluate both options and provide an update here once I have results.

Best regards, 

Robin Verlangen
Chief Data Architect




Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.

On Thu, Mar 26, 2015 at 7:09 AM, Thunder Stumpges <[hidden email]> wrote:

Would it help here to not actually issue a delete statement but instead use date based compaction and a dynamically calculated ttl that is some safe distance in the future from your key?

Just a thought.
-Thunder

On Mar 25, 2015 11:07 AM, "Robert Coli" <[hidden email]> wrote:
On Wed, Mar 25, 2015 at 12:45 AM, Robin Verlangen <[hidden email]> wrote:
@Robert: can you elaborate a bit more on the "not ideal" parts? In my case I will be throwing away the rows (thus the points in time that are "now in the past"), which will create tombstones which are compacted away.

"Not ideal" is what I mean... Cassandra has immutable data files, use cases which do DELETE pay an obvious penalty. Some percentage of tombstones will exist continuously, and you have to store them and seek past them.

=Rob