Onboarding to Accord

Accord supports all existing CQL and can be enabled on a per table and per token range within that table basis. Enabling Accord on existing tables requires a migration process that can be done on this same per table and per range basis that safely transitions data from being managed by Cassandra + Paxos to Cassandra + Accord without downtime.

A migration is required because Accord can’t safely read data written by non-SERIAL writes. Accord requires deterministic reads in order to have deterministic transaction recovery and non-SERIAL writes can’t be read deterministically while still being highly available.

This guide describes how to enable Accord and what differences to expect when migrating your existing CQL workload to Accord.

This guide does not cover the new transaction syntax.

Repair can now include an optional Accord repair that nodetool repair will enable by default like Paxos repair. This repair doesn’t actually synchronize any data it just runs a transaction that checks that Accord has resolved the state of all transactions in the repaired range up to the point the transaction was created and that the transactions are applied at ALL.

Accord is normally doing this in the background anyways this just ensures that it has occurred at ALL and hasn’t experienced any delays.

Migrating an existing table to run on Accord starts by altering the table:

After the table is altered it is required to run nodetool consensus_admin begin-migration on ranges in the table unless accord.range_migration=auto.

When a range is initially marked migrating to Accord all non-SERIAL writes will execute on Accord while SERIAL writes will continue to execute on Paxos. non-SERIAL writes include regular writes, logged and unlogged batches, hints, and read repair. Accord will perform synchronous commit the specified consistency level requiring 2x WAN RTT.

Tables that are migrating or are partially migrated to Accord (or back to Paxos) can be listed using nodetool consensus_admin list or the sytem table system_accord_debug.migration_state.

Migration to Accord consists of two phases with the first phase starting when a range is marked migrating, and the second phase starting after a full or incremental data repair, and then the migration completing after a second repair which must be a full data repair + Paxos repair. While marking the range as migrating can be done automatically with accord.range_migration=auto, there is not automation for triggering the repairs. If you regularly run compatible repairs then the migration will eventually complete, but if you don’t run them or want the migration to complete sooner then you will need to either trigger them manually or invoke nodetool consensus_admin finish-migration to trigger them.

Any repair that is compatible will drive migration forward whether it only covers part of the migrating range or whether is started via nodetool consensus_admin finish-migration or some other external process that initiates repair. Force repair with down nodes will not be eligible to drive any type or phase of migration forward. Force repair with all nodes up will still work.

Migration from Accord to Paxos occurs in a single phase and begins by altering the table’s transactional_mode to off and then optionally marking ranges as migrating as discussed above.

Once a range is marked migrating all operations in the migrating range will stop executing on Accord. Before each operation occurs they will have to run an Accord key repair similar to the Paxos key repair to ensure Accord transactions for that key have committed at QUORUM.

An Accord repair needs to be run on the migrating range, triggered manually or via nodetool finish-migration, and once that completes non-SERIAL operations will run using the usual eventually consistent path and SERIAL operations will execute on Paxos.

All the nodetool migration commands are based on new StorageServiceMBean JMX methods. These methods are migrateConsensusProtocol, finishConsensusMigration, listConsensusMigrations, getAccordManagedKeyspaces, and getAccordManagedTables and can be used by external management tools to manage consensus migration. The existing methods for starting repairs can also be used to start the repairs that are needed to complete migration.

Migration requires support for read and write consistency levels because Accord ends up being required to read Paxos writes at QUORUM and Accord needs to execute non-SERIAL writes while Paxos is still being used for SERIAL writes and thus needs to perform synchronous commit at the requested consistency level.

Once migration is complete the read and write consistency levels will be ignored with transactional mode full . With transactional mode mixed_reads Accord will continue to do synchronous commit and honor the requested commit/write consistency level.

Accord will always reject any requests to execute at unsupported consistency levels to ensure that migration to/from Accord is always possible.

Supported read consistency levels are ONE, QUORUM, SERIAL, and ALL. Supported write consistency levels are ANY, ONE, QUORUM, SERIAL, and ALL. LOCAL, TWO, and THREE are not supported. ANY is executed as an asynchronous commit similar to Paxos.

non-SERIAL operations are linearizable by default when migrated to Accord, but during migration they might not be linearizable due to timestamp handling changes. During migration, normal CQL operations will use the Accord timestamp once a range starts migration, but will fall back to server timestamp when migrating away from Accord.

Hints and batch log replay will use server timestamp and they only come into play during migration as hints and batch log replay won’t be generated when a range is on Accord.

USING TIMESTAMP is allowed and the application of the operations will occur in a linearizable order, but from the perspective of a reader the merged result may not appear linearizable.

For BATCH operations, timestamp handling is more complex: - If the batch uses USING TIMESTAMP, the user timestamp will be used - If all mutations in a BATCH use USING TIMESTAMP, the user timestamp will be used - If not using USING TIMESTAMP and all partition keys are on Accord, the Accord timestamp is used - If the BATCH has some partitions on Accord and others not on Accord, the server timestamp will be used (writes to the Accord table will not be linearizable for multi-table batches where one table is not migrating to Accord) - If the batch mixes server timestamp and USING TIMESTAMP mutations, the default behavior is to reject the batch, configurable via accord.mixed_time_source_handling with values: * reject (default) - reject the CQL operation * log - accept the operation using server timestamp but log operations where writes to the Accord table will not be linearizable * ignore - accept the operation silently

Logged batches that only touch Accord tables will not be written to the batch log because that functionality is redundant with Accord. Batches that touch both Accord and non-Accord tables will be written to the batch log.

Paging runs a separate transaction per page and does not produce a linearizable result.

Partition range reads are split into multiple transactions during execution and will not produce a strict serializable result. Additionally during migration there are no barriers/repairs executed before partition range reads. When migrating from Accord to Paxos the effective commit CL for Accord writes as viewed from partition range reads will be ANY. Adding barriers/repairs before partition range reads would cause them to time out so they are not done.

Pre-existing batchlog entries and hints will be processed during and after migration until they are completed. If they need to be executed through Accord they will be routed through Accord automatically.

Logged batches that only touch Accord data will not be written to the batch log because that functionality is redundant with Accord. Batches that touch both Accord and non-Accord data are split: the Accord mutations are replayed via the Accord transaction protocol, while non-Accord mutations follow the standard batch log path. If Accord replay fails, hint-based fallback is used.

Hints are not written for Accord writes although the batch log may result in new hints because batch log entries are converted to hints after the first retry.

Various operations can access both Accord and non-Accord managed data. These are transparently split into parts that execute on Accord and parts that execute outside of Accord and the results are merged. If the splitting process races with migration then the operations is re-split and retried without surfacing an error to the client.

Partition range reads with a limit use more memory and CPU at the nodes being read from and at the coordinator. Accord splits the ranges owned by each node into smaller subranges and each subrange is owned by a command store. The partition range read will execute at every intersecting command store on a node and each will return LIMIT N results which are sent back to the coordinator. The coordinator then merges them and re-applies the limit.

The additional memory and CPU will be amplified proportional to the number of command stores which defaults to DatabaseDescriptor.getAvailableProcessors().

Accord’s read and write metrics are counted under the existing Read and Write scope along with eventually consistent operations. To see Accord specific metrics you can look at the AccordRead and AccordWrite scope. CASRead and CASWrite will not track CAS or SERIAL read operations that end up running on Accord and they will instead show up in AccordRead/AccordWrite and Read/Write.

If a single request ends up running on both systems due to misrouting it will show up as multiple requests. Misrouted requests are counted under the RetryDifferentSystem meter and will show up in AccordRead and AccordWrite if Accord was the system the request was misrouted to as well as Read and Write. If the request was misrouted to non-Accord code then it will show up under Read and Write metrics or CASRead and CASWrite metrics.

Hints can be misrouted and this is tracked in HintsServiceMetrics under the HintsRetryDifferentSystem meter.

Partition range reads can also potentially generate additional Accord transactions depending on how the reads end up having to be split due to intersection with migrating ranges.