Read Path & Query Providers

Read Path & Query Providers

Status: Proposed direction.

The query-provider enhancement issues are one family: keep the read core honest and reusable, then expose it through Flight/Trino and DataFusion without duplicating Cassandra-specific rules in each consumer.

Public summary

One read core

Snapshot discovery, partition reads, token pruning, predicate evaluation, and statistics should be shared below provider-specific APIs.

Provider adapters

DataFusion should use the same snapshot, split, and predicate model that already feeds Flight and Trino.

RF-correct stats

Optimizer row counts need token-range scope so replicated tables do not look larger than the logical data set.

Provider diagram

flowchart LR
  subgraph Core["Shared read core"]
    direction TB
    Snapshot["Snapshot / data-dir discovery"]
    FastPath["Partition read fast path"]
    Stats["Token-range scoped stats"]
    Snapshot --> FastPath --> Stats
  end

  subgraph Providers["Provider surfaces"]
    direction TB
    Flight["Flight / Trino"]
    DataFusion["DataFusion table provider"]
  end

  Stats --> Flight
  FastPath --> DataFusion

The proposed direction is a shared read core with provider-specific surfaces above it.

Mermaid source for the provider diagram

flowchart LR
  subgraph Core["Shared read core"]
    direction TB
    Snapshot["Snapshot / data-dir discovery"]
    FastPath["Partition read fast path"]
    Stats["Token-range scoped stats"]
    Snapshot --> FastPath --> Stats
  end

  subgraph Providers["Provider surfaces"]
    direction TB
    Flight["Flight / Trino"]
    DataFusion["DataFusion table provider"]
  end

  Stats --> Flight
  FastPath --> DataFusion

Statistics path

flowchart LR
  Splits["Token-range splits"] --> Replica["One replica per logical range"]
  Replica --> Stats["RF-correct row counts"]
  Stats --> Optimizer["Trino / provider optimizer"]

Logical optimizer statistics should follow the same token-range split model used for replica-correct reads.

Mermaid source for the statistics diagram

flowchart LR
  Splits["Token-range splits"] --> Replica["One replica per logical range"]
  Replica --> Stats["RF-correct row counts"]
  Stats --> Optimizer["Trino / provider optimizer"]

Issue	Coverage	Public interpretation
#941 DataFusion table provider	Direct	Add a DataFusion provider after the reusable read core is stable.
#942 partition read path	Direct	Route single-partition reads through the cheaper read path instead of compacting to memory unnecessarily.
#1336 RF-correct optimizer row counts	Direct	Stats must be token-range scoped rather than summed across replicas.

Research sources

Flight / Trino plan The existing plan for snapshot reads, token ranges, predicate pushdown, and Trino splits.

Fast analytics design The broader Arrow Flight and query-provider design context.

Point-read fast path Design notes for avoiding unnecessary full merge work on point and partition reads.

DataFusion provider council Architecture analysis for the DataFusion provider and the shared scan contract.

Primary issue links:

• #941 DataFusion table provider
• #942 partition read path
• #1336 RF-correct optimizer row counts