Backup strategy

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Backup strategy

Sridhar Chellappa
We are running into problems where Backup jobs are taking away a huge bandwidth out of the C* nodes. While we can schedule a particular timing window for backups alone, the request pattern is so random; there is no particular time where we can schedule backups, periodically.

My current thinking is to run backups against a replica that does not serve requests. Questions:

  1. Is it the right strategy?
  2. if it is - how do I pull a replica out from serving requests ?
  3. If not, what is the right  backup strategy ?
Reply | Threaded
Open this post in threaded view
|

Re: Backup strategy

Aaron Turner
Why not just have a small DC/ring of nodes which you just do your
snapshots/backups from?

I wouldn't take nodes offline from the ring just to back them up.

The other option is to add sufficient nodes to handle your existing
request I/O + backups.  Sounds like you might be already I/O
constrained.
--
Aaron Turner
http://synfin.net/         Twitter: @synfinatic
https://github.com/synfinatic/tcpreplay - Pcap editing and replay
tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
    -- Benjamin Franklin


On Tue, Nov 5, 2013 at 4:36 PM, Sridhar Chellappa
<[hidden email]> wrote:

> We are running into problems where Backup jobs are taking away a huge
> bandwidth out of the C* nodes. While we can schedule a particular timing
> window for backups alone, the request pattern is so random; there is no
> particular time where we can schedule backups, periodically.
>
> My current thinking is to run backups against a replica that does not serve
> requests. Questions:
>
> Is it the right strategy?
> if it is - how do I pull a replica out from serving requests ?
> If not, what is the right  backup strategy ?
Reply | Threaded
Open this post in threaded view
|

Re: Backup strategy

Ray Sutton
I don't understand how the creation of a snapshot causes any load whatsoever. By definition, a snapshot is a hard link of an existing SSTable. The SSTable is not being physically copied so there is no disk I/O, it's just a reference to an inode. 

--
Ray  //o-o\\



On Tue, Nov 5, 2013 at 8:09 PM, Aaron Turner <[hidden email]> wrote:
Why not just have a small DC/ring of nodes which you just do your
snapshots/backups from?

I wouldn't take nodes offline from the ring just to back them up.

The other option is to add sufficient nodes to handle your existing
request I/O + backups.  Sounds like you might be already I/O
constrained.
--
Aaron Turner
http://synfin.net/         Twitter: @synfinatic
https://github.com/synfinatic/tcpreplay - Pcap editing and replay
tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
    -- Benjamin Franklin


On Tue, Nov 5, 2013 at 4:36 PM, Sridhar Chellappa
<[hidden email]> wrote:
> We are running into problems where Backup jobs are taking away a huge
> bandwidth out of the C* nodes. While we can schedule a particular timing
> window for backups alone, the request pattern is so random; there is no
> particular time where we can schedule backups, periodically.
>
> My current thinking is to run backups against a replica that does not serve
> requests. Questions:
>
> Is it the right strategy?
> if it is - how do I pull a replica out from serving requests ?
> If not, what is the right  backup strategy ?

Reply | Threaded
Open this post in threaded view
|

Re: Backup strategy

Robert Coli-3
In reply to this post by Sridhar Chellappa
On Tue, Nov 5, 2013 at 4:36 PM, Sridhar Chellappa <[hidden email]> wrote: 
  1. If not, what is the right  backup strategy ?
You didn't specify, but it sounds like you are doing a snapshot and then a full offhost backup of the sstables?

Perhaps instead of point in time full backups, you could continuously back up with something like tablesnap [1]? tablesnap watches the datadir for newly created SSTables and then writes the file and a manifest containing which other files were also in the directory at the time. These two pieces of information allow you to restore to the point in time when any given (immutable) data file was created.

=Rob

[1] https://github.com/synack/tablesnap
Reply | Threaded
Open this post in threaded view
|

Re: Backup strategy

Sridhar Chellappa
Yes. I am taking a Snapshot and then offloading the full data into S3.  How will Table Snap help?


On Wed, Nov 6, 2013 at 6:57 AM, Robert Coli <[hidden email]> wrote:
On Tue, Nov 5, 2013 at 4:36 PM, Sridhar Chellappa <[hidden email]> wrote: 
  1. If not, what is the right  backup strategy ?
You didn't specify, but it sounds like you are doing a snapshot and then a full offhost backup of the sstables?

Perhaps instead of point in time full backups, you could continuously back up with something like tablesnap [1]? tablesnap watches the datadir for newly created SSTables and then writes the file and a manifest containing which other files were also in the directory at the time. These two pieces of information allow you to restore to the point in time when any given (immutable) data file was created.

=Rob

[1] https://github.com/synack/tablesnap

Reply | Threaded
Open this post in threaded view
|

Re: Backup strategy

Robert Coli-3
On Thu, Nov 7, 2013 at 6:28 AM, Sridhar Chellappa <[hidden email]> wrote:
Yes. I am taking a Snapshot and then offloading the full data into S3.  How will Table Snap help?

As I detailed in my previous mail :

1) incremental style backup, instead of snapshot + full
2) tracks meta information about backup sets
3) comes with tools to expire old sets
4) also comes with restore tool to restore sets.

More detail at :


=Rob

Reply | Threaded
Open this post in threaded view
|

Re: Backup strategy

Dan Simpson
Thanks for sharing tablesnap.  It's just what I have been looking for.


On Thu, Nov 7, 2013 at 5:10 PM, Robert Coli <[hidden email]> wrote:
On Thu, Nov 7, 2013 at 6:28 AM, Sridhar Chellappa <[hidden email]> wrote:
Yes. I am taking a Snapshot and then offloading the full data into S3.  How will Table Snap help?

As I detailed in my previous mail :

1) incremental style backup, instead of snapshot + full
2) tracks meta information about backup sets
3) comes with tools to expire old sets
4) also comes with restore tool to restore sets.

More detail at :


=Rob