Abstract

Modern data processing workflows frequently encounter ragged data: multidimensional collections whose inner lengths vary and are revealed only at runtime. Ragged data is prevalent in domains throughout science and computing, yet existing workflow engines treat it as nested containers. This representation leaves shape semantics ambiguous, obscures task dependencies when upstream coordinates are not yet enumerated, and imposes coarse synchronization barriers that stall independent work across tasks. We argue that ragged data is fundamentally a flat function over a coordinate space, and that making this view explicit resolves all three difficulties at once. We realize this insight in Operon, a statically verifiable workflow engine built around a formalism of named dimensions with explicit dependency relations. Users declare pipelines in the Operon DSL by annotating data flow with dimension names, where the static checker verifies that every task does not include faulty iterations regardless of runtime lengths. At runtime, the system models in-progress arrays as partial functions over downward-closed coordinate domains, which provides a natural representation for partially-unknown data states, an exact notion of per-task indegree before all upstream coordinates are enumerated, and a sound basis for fault recovery without programmer intervention.

• Publication: https://arxiv.org/abs/2511.16080