{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/socialgpt-prompting-llms-for-social-relation","title":"SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization","arxiv_id":"2410.21411","date":"2024-10-28","proceeding":null,"authors":["Wanhua Li","Zibin Meng","Jiawei Zhou","Donglai Wei","Chuang Gan","Hanspeter Pfister"],"abstract":"Social relation reasoning aims to identify relation categories such as friends, spouses, and colleagues from images. While current methods adopt the paradigm of training a dedicated network end-to-end using labeled image data, they are limited in terms of generalizability and interpretability. To address these issues, we first present a simple yet well-crafted framework named {\\name}, which combines the perception capability of Vision Foundation Models (VFMs) and the reasoning capability of Large Language Models (LLMs) within a modular framework, providing a strong baseline for social relation recognition. Specifically, we instruct VFMs to translate image content into a textual social story, and then utilize LLMs for text-based reasoning. {\\name} introduces systematic design principles to adapt VFMs and LLMs separately and bridge their gaps. Without additional model training, it achieves competitive zero-shot results on two databases while offering interpretable answers, as LLMs can generate language-based explanations for the decisions. The manual prompt design process for LLMs at the reasoning phase is tedious and an automated prompt optimization method is desired. As we essentially convert a visual classification task into a generative task of LLMs, automatic prompt optimization encounters a unique long prompt optimization issue. To address this issue, we further propose the Greedy Segment Prompt Optimization (GSPO), which performs a greedy search by utilizing gradient information at the segment level. Experimental results show that GSPO significantly improves performance, and our method also generalizes to different image styles. The code is available at https://github.com/Mengzibin/SocialGPT.","url_abs":"https://arxiv.org/abs/2410.21411v1","url_pdf":"https://arxiv.org/pdf/2410.21411v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"socialgpt-prompting-llms-for-social-relation","repo_url":"https://github.com/mengzibin/socialgpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"Relation"},{"task_slug":"visual-social-relationship-recognition","task_name":"Visual Social Relationship Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-social-relationship-recognition-on-1","task":"Visual Social Relationship Recognition","dataset":"PIPA","model":"SocialGPT","rank_in_archive_order":1,"of":6,"metrics":{"Accuracy":"66.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.21411","atlas_url":"https://app.syntology.ai/?focus=2410.21411","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.21411"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Mengzibin/SocialGPT","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mengzibin/socialgpt","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"90d108caba3c8524","entry":"ProgressiveMultiPromptStory","repo":"mengzibin/socialgpt","repo_kind":"official","path":"GSPO-PIPA/llm_prompt/base/prompt_manager.py","file_url":"https://github.com/mengzibin/socialgpt/blob/HEAD/GSPO-PIPA/llm_prompt/base/prompt_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90d108caba3c8524"}},{"code_sha256_prefix":"a404f3d6373002f0","entry":"bbox_xywh_to_xyxy","repo":"Mengzibin/SocialGPT","repo_kind":"official","path":"social-story-main-PIPA/models/region_semantic.py","file_url":"https://github.com/Mengzibin/SocialGPT/blob/HEAD/social-story-main-PIPA/models/region_semantic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a404f3d6373002f0"}},{"code_sha256_prefix":"a134d4aa43dba9df","entry":"pil_image_to_base64","repo":"Mengzibin/SocialGPT","repo_kind":"official","path":"social-story-main-PIPA/models/image_text_transformation.py","file_url":"https://github.com/Mengzibin/SocialGPT/blob/HEAD/social-story-main-PIPA/models/image_text_transformation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a134d4aa43dba9df"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}