{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rlip-relational-language-image-pre-training","title":"RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection","arxiv_id":"2209.01814","date":"2022-09-05","proceeding":null,"authors":["Hangjie Yuan","Jianwen Jiang","Samuel Albanie","Tao Feng","Ziyuan Huang","Dong Ni","Mingqian Tang"],"abstract":"The task of Human-Object Interaction (HOI) detection targets fine-grained visual parsing of humans interacting with their environment, enabling a broad range of applications. Prior work has demonstrated the benefits of effective architecture design and integration of relevant cues for more accurate HOI detection. However, the design of an appropriate pre-training strategy for this task remains underexplored by existing approaches. To address this gap, we propose Relational Language-Image Pre-training (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descriptions. To make effective use of such pre-training, we make three technical contributions: (1) a new Parallel entity detection and Sequential relation inference (ParSe) architecture that enables the use of both entity and relation descriptions during holistically optimized pre-training; (2) a synthetic data generation framework, Label Sequence Extension, that expands the scale of language data available within each minibatch; (3) mechanisms to account for ambiguity, Relation Quality Labels and Relation Pseudo-Labels, to mitigate the influence of ambiguous/noisy samples in the pre-training data. Through extensive experiments, we demonstrate the benefits of these contributions, collectively termed RLIP-ParSe, for improved zero-shot, few-shot and fine-tuning HOI detection performance as well as increased robustness to learning from noisy annotations. Code will be available at https://github.com/JacobYuan7/RLIP.","url_abs":"https://arxiv.org/abs/2209.01814v3","url_pdf":"https://arxiv.org/pdf/2209.01814v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rlip-relational-language-image-pre-training","repo_url":"https://github.com/jacobyuan7/rlip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"rlip-relational-language-image-pre-training","repo_url":"https://github.com/jacobyuan7/ocn-hoi-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"rlip-relational-language-image-pre-training","repo_url":"https://github.com/jacobyuan7/rlipv2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"human-object-interaction-detection","task_name":"Human-Object Interaction Detection"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"synthetic-data-generation","task_name":"Synthetic Data Generation"}],"methods":[{"method_slug":"visual-parsing","method_name":"Visual Parsing"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/human-object-interaction-detection-on-hico","task":"Human-Object Interaction Detection","dataset":"HICO-DET","model":"RLIP-ParSe (ResNet-50)","rank_in_archive_order":18,"of":55,"metrics":{"mAP":"32.84"},"uses_additional_data":false},{"leaderboard":"/sota/human-object-interaction-detection-on-hico","task":"Human-Object Interaction Detection","dataset":"HICO-DET","model":"ParSe (ResNet-101)","rank_in_archive_order":19,"of":55,"metrics":{"mAP":"32.76"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.01814","atlas_url":"https://app.syntology.ai/?focus=2209.01814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.01814"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/JacobYuan7/RLIP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jacobyuan7/ocn-hoi-benchmark","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jacobyuan7/rlipv2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jacobyuan7/rlip","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1db67394391c79fa","entry":"build_transformer","repo":"JacobYuan7/RLIP","repo_kind":"official","path":"models/ParSetransformer.py","file_url":"https://github.com/JacobYuan7/RLIP/blob/HEAD/models/ParSetransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1db67394391c79fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}