Papers › Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent...
Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability
Shafiuddin Rehan Ahmed, Sourabh Deshpande
Title, abstract, authors and date from arXiv's metadata (CC0); this paper is not in the Papers with Code archive (frozen 2025-07-28).
Multi-agent frameworks (MAFs) promise to simplify LLM-driven software development, yet no principled metric captures how well AI coding assistants can generate correct, framework-specific code. We introduce \textit{AI-assistability} (𝒜ℐ), a composite metric that quantifies a framework's amenability to AI-assisted development by combining structural alignment ($\barσ$) with functional correctness (pass@1). To evaluate this metric in a controlled setting, we design DDL2PropBank, a novel benchmark task that maps relational database schemas to PropBank semantic rolesets, and implement identical agent logic across ten frameworks using the Agent-as-a-Tool pattern. Our results challenge the intuition that declarative framework design guarantees AI-assistability: Agno, with a single canonical pattern and convention-aligned API, achieves the highest 𝒜ℐ score (0.55), while DSPy -- the most declarative framework by design -- scores lowest (0.07), as its novel abstractions are insufficiently represented in AI training data. We find that convention alignment, not declarative design alone, is the primary driver of AI-assistability (r = 0.576 between $\barσ$ and pass@1). All artifacts -- DDL2PropBank, PropBank MCP server, and all implementations -- are available at https://github.com/ahmeshaf/ddl2propbank
Code
Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Syntology holds the repository link but has not harvested or run code from it.
Results from the paper
The Papers with Code archive ends with its 2025-07-28 snapshot. This paper's arXiv identifier, 2602.11198, was issued in February 2026, after that date, so the archive has no leaderboard rows for it.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections