# Dataflow

> How Flownode selects and runs its batching and legacy streaming execution paths.

# Dataflow

Flownode has two internal execution paths:

- **Batching mode** is the primary path for aggregation and TQL workloads. It evaluates queries over persisted source data and writes materialized results to a sink table.
- **Streaming mode** is the legacy path retained for compatibility and deprecated for new workloads. It incrementally processes rows mirrored from Frontend as they arrive.

Users do not select the mode directly. When a Flow is created, GreptimeDB derives the path from the source tables and then the query, in this order:

```mermaid
flowchart LR
    MISS{"any source tablemissing?"}
    MISS -->|"yes, deferred"| PEND["pendingbatching Flow"]
    MISS -->|"yes, otherwise"| ERR["rejected"]
    MISS -->|"no"| TTL{"any source tablettl = 'instant'?"}
    TTL -->|"yes"| STREAM["Streaming mode"]
    TTL -->|"no"| TQL{"TQL query?"}
    TQL -->|"yes"| BATCH["Batching mode"]
    TQL -->|"no"| AGG{"plan has Aggregateor Distinct?"}
    AGG -->|"yes"| BATCH
    AGG -->|"no"| STREAM
```

The source-table checks come first, so a Flow that aggregates a source table with `ttl = 'instant'` runs in streaming mode rather than batching mode. A Flow whose source table does not exist yet is rejected unless it is created with `WITH (defer_on_missing_source = true)`, which produces a pending batching Flow.

## Batching mode

Batching mode reuses GreptimeDB's query engine instead of maintaining an operator graph for every incoming row. For a time-windowed Flow, its main loop is:

1. A source-table write marks the affected time windows as dirty.
2. A `BatchingTask` runs on its evaluation schedule or adaptive polling cadence and collects the pending dirty windows at that evaluation. Marking a window dirty does not wake the task.
3. The task adds time predicates for those windows to the Flow query and asks Frontend to execute it against the source tables.
4. The query result is inserted into the sink table, updating the materialized result for windows that were evaluated.
5. Successfully processed windows are removed from the dirty set. Failed work remains available for a later evaluation.

TQL Flows, and evaluation-interval Flows whose plan cannot be pruned safely by dirty windows, run the complete query rather than a time-filtered one. On that path the dirty set is only a scheduling signal. This path also lets Flow use query-engine features that the streaming renderer does not implement. See [Batching Mode](./batching_mode.md) for the task and dirty-window components.

## Streaming mode

The `dataflow` module (see `flow::compute` module) is the core computing module of `flow`.
It takes a SQL query and transforms it into flow's internal execution plan.
This execution plan is then rendered into an actual dataflow, which is essentially a directed acyclic graph (DAG) of functions with input and output ports.
New row changes drive the graph incrementally.

The renderer supports map/filter/project and reduce operations. Join and union plan nodes exist, but their streaming renderers are not implemented.

Internally, the dataflow handles data in row format, using a tuple `(row, time, diff)`. Here, `row` represents the actual data being passed, which may contain multiple `Value` objects.
`time` is the system time which tracks the progress of the dataflow, and `diff` typically represents the insertion or deletion of the row (+1 or -1).
Therefore, the tuple represents the insert/delete operation of the `row` at a given system `time`. Stateful operators keep indexed traces of these changes in an [Arrangement](./arrangement.md).