Unknown CF / Schema OK

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Unknown CF / Schema OK

Tim Olson
​After upgrading a schema, I'm getting lots of UnknownColumnFamilyException in the logs.  However, all nodes have the same schema as reported by nodetool describecluster.   I queried the system tables for the given column family UUID, but it doesn't appear in any of the schemas on any of the nodes.  I restarted all clients, but that didn't help either.

The cluster was running 2.1.2 but I recently upgraded to 2.1.3.

Any ideas?  This is basically making our production cluster highly unresponsive.

Tim
Reply | Threaded
Open this post in threaded view
|

Re: Unknown CF / Schema OK

Tim Olson
I did figure this out:

When adding a columnfamily, the query timed out before all nodes replied, and I sent the schema out again.  Half the nodes ended up with the CF having UUID A and half the nodes ended up with the new CF but UUID B.  UnknownColumnFamilyExceptions were thrown until the enqueued data exceeded memory.  Eventually one half of the nodes crashed, with the other half having a consistent view of the CF.  At this point I just dropped the offending CF schema in the active cluster, then the downed nodes could be re-added successfully.  We lost some data.  :(



On Sun, Mar 22, 2015 at 11:39 AM, Tim Olson <[hidden email]> wrote:
​After upgrading a schema, I'm getting lots of UnknownColumnFamilyException in the logs.  However, all nodes have the same schema as reported by nodetool describecluster.   I queried the system tables for the given column family UUID, but it doesn't appear in any of the schemas on any of the nodes.  I restarted all clients, but that didn't help either.

The cluster was running 2.1.2 but I recently upgraded to 2.1.3.

Any ideas?  This is basically making our production cluster highly unresponsive.

Tim