{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2309-06363","title":"Learning to Predict Concept Ordering for Common Sense Generation","arxiv_id":"2309.06363","date":"2023-09-12","proceeding":null,"authors":["Tianhui Zhang","Danushka Bollegala","Bei Peng"],"abstract":"Prior work has shown that the ordering in which concepts are shown to a commonsense generator plays an important role, affecting the quality of the generated sentence. However, it remains a challenge to determine the optimal ordering of a given set of concepts such that a natural sentence covering all the concepts could be generated from a pretrained generator. To understand the relationship between the ordering of the input concepts and the quality of the generated sentences, we conduct a systematic study considering multiple language models (LMs) and concept ordering strategies. We find that BART-large model consistently outperforms all other LMs considered in this study when fine-tuned using the ordering of concepts as they appear in CommonGen training data as measured using multiple evaluation metrics. Moreover, the larger GPT3-based large language models (LLMs) variants do not necessarily outperform much smaller LMs on this task, even when fine-tuned on task-specific training data. Interestingly, human annotators significantly reorder input concept sets when manually writing sentences covering those concepts, and this ordering provides the best sentence generations independently of the LM used for the generation, outperforming a probabilistic concept ordering baseline","url_abs":"https://arxiv.org/abs/2309.06363v1","url_pdf":"https://arxiv.org/pdf/2309.06363v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2309-06363","repo_url":"https://github.com/tianhuizhang/concept_ordering","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.06363","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.06363"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tianhuizhang/concept_ordering","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":4,"ran":7,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"bert-gen/s2s_ft/modeling_decoding.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/bert-gen/s2s_ft/modeling_decoding.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"d9778dbf9748ada8","entry":"asMinutes","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"Probabilistic/utils.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/Probabilistic/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9778dbf9748ada8"}},{"code_sha256_prefix":"633d1e424a09eb70","entry":"detokenize","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"bert-gen/decode_seq2seq.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/bert-gen/decode_seq2seq.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"633d1e424a09eb70"}},{"code_sha256_prefix":"40e9fee2e0b7e278","entry":"gelu","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"bert-gen/s2s_ft/modeling_decoding.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/bert-gen/s2s_ft/modeling_decoding.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40e9fee2e0b7e278"}},{"code_sha256_prefix":"a4475703ff58ecf9","entry":"gelu_fast","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"bart/src/model/activations.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/bart/src/model/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4475703ff58ecf9"}},{"code_sha256_prefix":"dc9ffc0f29e7fa3d","entry":"gelu_new","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"bart/src/model/activations.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/bart/src/model/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc9ffc0f29e7fa3d"}},{"code_sha256_prefix":"8130ddbbba17f308","entry":"get_best_sequence","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"bert-gen/gen_seq_from_trace.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/bert-gen/gen_seq_from_trace.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8130ddbbba17f308"}},{"code_sha256_prefix":"1cb1e5b36b0a5bc0","entry":"gpt35_reply","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"LLM/askllm.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/LLM/askllm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cb1e5b36b0a5bc0"}},{"code_sha256_prefix":"75177cb8ad871724","entry":"gpt3_reply","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"LLM/askllm.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/LLM/askllm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"75177cb8ad871724"}},{"code_sha256_prefix":"ba6e66dc83d560e4","entry":"read_traces_from_file","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"bert-gen/gen_seq_from_trace.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/bert-gen/gen_seq_from_trace.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba6e66dc83d560e4"}},{"code_sha256_prefix":"41c4b118c0a5c704","entry":"timeSince","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"Probabilistic/utils.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/Probabilistic/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41c4b118c0a5c704"}},{"code_sha256_prefix":"d539451884031f80","entry":"topk_permutation","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"Probabilistic/utils.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/Probabilistic/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d539451884031f80"}},{"code_sha256_prefix":"7f1978d0498f6aea","entry":"gpt3_zeroshot_reply","repo":"tianhuizhang/concept_ordering","repo_kind":"official","path":"LLM/askllm.py","file_url":"https://github.com/tianhuizhang/concept_ordering/blob/HEAD/LLM/askllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f1978d0498f6aea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}