Papers › Online Analysis of Distributed Dataflows with Timely Dataflow
Online Analysis of Distributed Dataflows with Timely Dataflow
Malte Sandstede
The archive published only this paper's code-link row. Authors, date and abstract are from arXiv's metadata (CC0), read from the Kaggle arXiv metadata snapshot of 2026-09-12 where its title matched the archive's; the title is the archive's.
We present ST2, an end-to-end solution to analyze distributed dataflows in an online setting. It is powered by Timely Dataflow, a low-latency, distributed data-parallel dataflow computational framework, and expands on its predecessor SnailTrail 1, a system to run online critical path analysis on program activity graphs derived from dataflow execution traces. ST2 connects to a running Timely computation, creates the program activity graph representation, and runs multiple analyses on top of it. Analyses include aggregate metrics, progress and temporal invariant checking, and graph pattern matching. Through a command-line interface and a real-time dashboard, users are able to interact with and visualize ST2's analysis results. For ST2's implementation, we discuss Differential Dataflow, a framework that uses differential computation to incrementalize even complex relational dataflow operators, as an alternative to Timely Dataflow, but ultimately settle on using Timely. In our performance evaluations, we are able to show that ST2 is able to comfortably keep up with common streaming computations in offline and online settings, even exceeding SnailTrail 1's performance. We also showcase and evaluate ST2 from a functional standpoint in a case study. Using the dashboard to profile a faulty source computation, we manage to successfully detect the issues' root cause. We argue that ST2 is an extendable system that paves the way for users to debug, monitor, and optimize online distributed dataflows.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections