{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-sifo-benchmark-investigating-the","title":"The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models","arxiv_id":"2406.19999","date":"2024-06-28","proceeding":null,"authors":["Xinyi Chen","Baohao Liao","Jirui Qi","Panagiotis Eustratiadis","Christof Monz","Arianna Bisazza","Maarten de Rijke"],"abstract":"Following multiple instructions is a crucial ability for large language models (LLMs). Evaluating this ability comes with significant challenges: (i) limited coherence between multiple instructions, (ii) positional bias where the order of instructions affects model performance, and (iii) a lack of objectively verifiable tasks. To address these issues, we introduce a benchmark designed to evaluate models' abilities to follow multiple instructions through sequential instruction following (SIFo) tasks. In SIFo, the successful completion of multiple instructions is verifiable by examining only the final instruction. Our benchmark evaluates instruction following using four tasks (text modification, question answering, mathematics, and security rules), each assessing different aspects of sequential instruction following. Our evaluation of popular LLMs, both closed-source and open-source, shows that more recent and larger models significantly outperform their older and smaller counterparts on the SIFo tasks, validating the benchmark's effectiveness. All models struggle with following sequences of instructions, hinting at an important lack of robustness of today's language models.","url_abs":"https://arxiv.org/abs/2406.19999v2","url_pdf":"https://arxiv.org/pdf/2406.19999v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-sifo-benchmark-investigating-the","repo_url":"https://github.com/shin-ee-chen/SIFo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.19999","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.19999"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shin-ee-chen/SIFo","reach":null}],"summary":{"ran_draft_wrong":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"f2a96da09d0ad98f","entry":"create_prompt","repo":"shin-ee-chen/SIFo","repo_kind":"official","path":"llm_inference/get_responses_from_vllm_models.py","file_url":"https://github.com/shin-ee-chen/SIFo/blob/HEAD/llm_inference/get_responses_from_vllm_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2a96da09d0ad98f"}},{"code_sha256_prefix":"4fbcc77551b875d9","entry":"get_config_from_input_path","repo":"shin-ee-chen/SIFo","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/shin-ee-chen/SIFo/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4fbcc77551b875d9"}},{"code_sha256_prefix":"05e68d490c85117c","entry":"get_shown_model_name","repo":"shin-ee-chen/SIFo","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/shin-ee-chen/SIFo/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05e68d490c85117c"}},{"code_sha256_prefix":"718b0b411c6d8542","entry":"input_preprocess","repo":"shin-ee-chen/SIFo","repo_kind":"official","path":"llm_inference/get_responses_from_vllm_models.py","file_url":"https://github.com/shin-ee-chen/SIFo/blob/HEAD/llm_inference/get_responses_from_vllm_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"718b0b411c6d8542"}},{"code_sha256_prefix":"76af8fb53c8ff838","entry":"preprocess_example","repo":"shin-ee-chen/SIFo","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/shin-ee-chen/SIFo/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76af8fb53c8ff838"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}