{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hulk-a-universal-knowledge-translator-for","title":"Hulk: A Universal Knowledge Translator for Human-Centric Tasks","arxiv_id":"2312.01697","date":"2023-12-04","proceeding":null,"authors":["Yizhou Wang","Yixuan Wu","Shixiang Tang","Weizhen He","Xun Guo","Feng Zhu","Lei Bai","Rui Zhao","Jian Wu","Tong He","Wanli Ouyang"],"abstract":"Human-centric perception tasks, e.g., pedestrian detection, skeleton-based action recognition, and pose estimation, have wide industrial applications, such as metaverse and sports analysis. There is a recent surge to develop human-centric foundation models that can benefit a broad range of human-centric perception tasks. While many human-centric foundation models have achieved success, they did not explore 3D and vision-language tasks for human-centric and required task-specific finetuning. These limitations restrict their application to more downstream tasks and situations. To tackle these problems, we present Hulk, the first multimodal human-centric generalist model, capable of addressing 2D vision, 3D vision, skeleton-based, and vision-language tasks without task-specific finetuning. The key to achieving this is condensing various task-specific heads into two general heads, one for discrete representations, e.g., languages, and the other for continuous representations, e.g., location coordinates. The outputs of two heads can be further stacked into four distinct input and output modalities. This uniform representation enables Hulk to treat diverse human-centric tasks as modality translation, integrating knowledge across a wide range of tasks. Comprehensive evaluations of Hulk on 12 benchmarks covering 8 human-centric tasks demonstrate the superiority of our proposed method, achieving state-of-the-art performance in 11 benchmarks. The code is available on https://github.com/OpenGVLab/Hulk.","url_abs":"https://arxiv.org/abs/2312.01697v4","url_pdf":"https://arxiv.org/pdf/2312.01697v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hulk-a-universal-knowledge-translator-for","repo_url":"https://github.com/opengvlab/hulk","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"hulk-a-universal-knowledge-translator-for","repo_url":"https://github.com/opengvlab/humanbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-human-pose-estimation","task_name":"3D Human Pose Estimation"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"human-mesh-recovery","task_name":"Human Mesh Recovery"},{"task_slug":"human-part-segmentation","task_name":"Human Part Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"pedestrian-attribute-recognition","task_name":"Pedestrian Attribute Recognition"},{"task_slug":"pedestrian-detection","task_name":"Pedestrian Detection"},{"task_slug":"pedestrian-image-caption","task_name":"Pedestrian Image Caption"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-human-pose-estimation-on-3dpw","task":"3D Human Pose Estimation","dataset":"3DPW","model":"Hulk(ViT-L)","rank_in_archive_order":8,"of":119,"metrics":{"MPJPE":"66.3","MPVPE":"77.4","PA-MPJPE":"38.5"},"uses_additional_data":false},{"leaderboard":"/sota/3d-human-pose-estimation-on-3dpw","task":"3D Human Pose Estimation","dataset":"3DPW","model":"Hulk(ViT-B)","rank_in_archive_order":17,"of":119,"metrics":{"MPJPE":"67","MPVPE":"79.8","PA-MPJPE":"39.9"},"uses_additional_data":false},{"leaderboard":"/sota/human-part-segmentation-on-cihp","task":"Human Part Segmentation","dataset":"CIHP","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":1,"of":6,"metrics":{"Mean IoU":"72.68"},"uses_additional_data":false},{"leaderboard":"/sota/human-part-segmentation-on-cihp","task":"Human Part Segmentation","dataset":"CIHP","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":2,"of":6,"metrics":{"Mean IoU":"71.26"},"uses_additional_data":false},{"leaderboard":"/sota/human-part-segmentation-on-human3-6m","task":"Human Part Segmentation","dataset":"Human3.6M","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":1,"of":3,"metrics":{"mIoU":"69.89"},"uses_additional_data":false},{"leaderboard":"/sota/human-part-segmentation-on-human3-6m","task":"Human Part Segmentation","dataset":"Human3.6M","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":2,"of":3,"metrics":{"mIoU":"68.56"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-crowdhuman-full-body","task":"Object Detection","dataset":"CrowdHuman (full body)","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":6,"of":19,"metrics":{"AP":"93","mMR":"36.5"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-crowdhuman-full-body","task":"Object Detection","dataset":"CrowdHuman (full body)","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":9,"of":19,"metrics":{"AP":"92.4","mMR":"40.7"},"uses_additional_data":false},{"leaderboard":"/sota/pedestrian-attribute-recognition-on-pa-100k","task":"Pedestrian Attribute Recognition","dataset":"PA-100K","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":2,"of":13,"metrics":{"Accuracy":"88.97"},"uses_additional_data":true},{"leaderboard":"/sota/pedestrian-attribute-recognition-on-pa-100k","task":"Pedestrian Attribute Recognition","dataset":"PA-100K","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":3,"of":13,"metrics":{"Accuracy":"87.85"},"uses_additional_data":true},{"leaderboard":"/sota/pedestrian-attribute-recognition-on-rapv2","task":"Pedestrian Attribute Recognition","dataset":"RAPv2","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"85.86"},"uses_additional_data":false},{"leaderboard":"/sota/pedestrian-attribute-recognition-on-rapv2","task":"Pedestrian Attribute Recognition","dataset":"RAPv2","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy":"85.26"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-aic","task":"Pose Estimation","dataset":"AIC","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":1,"of":10,"metrics":{"AP":"37.1"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-aic","task":"Pose Estimation","dataset":"AIC","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":2,"of":10,"metrics":{"AP":"35.6"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-coco","task":"Pose Estimation","dataset":"COCO (Common Objects in Context)","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":3,"of":10,"metrics":{"AP":"78.7"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-coco","task":"Pose Estimation","dataset":"COCO (Common Objects in Context)","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":5,"of":10,"metrics":{"AP":"77.5"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-lip-val","task":"Semantic Segmentation","dataset":"LIP val","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":1,"of":13,"metrics":{"mIoU":"66.02%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-lip-val","task":"Semantic Segmentation","dataset":"LIP val","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":2,"of":13,"metrics":{"mIoU":"63.98%"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-ntu-rgbd","task":"Skeleton Based Action Recognition","dataset":"NTU RGB+D","model":"Hulk(Finetune, ViT-L)","rank_in_archive_order":1,"of":135,"metrics":{"Accuracy (CS)":"94.3"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-ntu-rgbd","task":"Skeleton Based Action Recognition","dataset":"NTU RGB+D","model":"Hulk(Finetune, ViT-B)","rank_in_archive_order":3,"of":135,"metrics":{"Accuracy (CS)":"94"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.01697","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.01697"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/hulk","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/humanbench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":12,"unverified":12},"by_repo_kind":{"official":{"samples":24,"ran":12,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8a63c5da44344ba6","entry":"accuracy_multi","repo":"opengvlab/hulk","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a63c5da44344ba6"}},{"code_sha256_prefix":"41c46b0d5713eb14","entry":"all_gather","repo":"opengvlab/hulk","repo_kind":"official","path":"core/comm_.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/comm_.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41c46b0d5713eb14"}},{"code_sha256_prefix":"ae5f2eed69658ac0","entry":"allreduce","repo":"opengvlab/hulk","repo_kind":"official","path":"core/distributed_utils.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae5f2eed69658ac0"}},{"code_sha256_prefix":"c632684a9726a77a","entry":"broadcast_object","repo":"opengvlab/hulk","repo_kind":"official","path":"core/comm_.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/comm_.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c632684a9726a77a"}},{"code_sha256_prefix":"5d01dded5497144b","entry":"dist_init","repo":"opengvlab/humanbench","repo_kind":"official","path":"PATH/core/distributed_utils.py","file_url":"https://github.com/opengvlab/humanbench/blob/HEAD/PATH/core/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d01dded5497144b"}},{"code_sha256_prefix":"4570d6d70bccdfb6","entry":"flat","repo":"opengvlab/hulk","repo_kind":"official","path":"core/config.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/config.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4570d6d70bccdfb6"}},{"code_sha256_prefix":"00a9f7918ac39329","entry":"gather","repo":"opengvlab/hulk","repo_kind":"official","path":"core/comm_.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/comm_.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00a9f7918ac39329"}},{"code_sha256_prefix":"eb8a2600afc1de95","entry":"init","repo":"opengvlab/hulk","repo_kind":"official","path":"core/fp16/amp.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/fp16/amp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb8a2600afc1de95"}},{"code_sha256_prefix":"70165e0988ce9571","entry":"param_group_fc","repo":"opengvlab/hulk","repo_kind":"official","path":"core/make_param_group.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/make_param_group.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70165e0988ce9571"}},{"code_sha256_prefix":"e4a7663d686ce91a","entry":"param_group_multitask","repo":"opengvlab/hulk","repo_kind":"official","path":"core/make_param_group.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/make_param_group.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4a7663d686ce91a"}},{"code_sha256_prefix":"e4f5737bd720668d","entry":"param_group_no_wd","repo":"opengvlab/hulk","repo_kind":"official","path":"core/make_param_group.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/make_param_group.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4f5737bd720668d"}},{"code_sha256_prefix":"ff1253be17efe1ae","entry":"retry_if_cuda_oom","repo":"opengvlab/hulk","repo_kind":"official","path":"core/memory.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/memory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff1253be17efe1ae"}},{"code_sha256_prefix":"78d1f9f766be74b7","entry":"accuracy","repo":"opengvlab/hulk","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78d1f9f766be74b7"}},{"code_sha256_prefix":"02640c5911ca6b56","entry":"allreduce_async","repo":"opengvlab/hulk","repo_kind":"official","path":"core/distributed_utils.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02640c5911ca6b56"}},{"code_sha256_prefix":"305673e9acaebcc6","entry":"broadcast","repo":"opengvlab/hulk","repo_kind":"official","path":"core/distributed_utils.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"305673e9acaebcc6"}},{"code_sha256_prefix":"f27c7ab614e20039","entry":"float_function","repo":"opengvlab/hulk","repo_kind":"official","path":"core/fp16/amp.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/fp16/amp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f27c7ab614e20039"}},{"code_sha256_prefix":"6b4eee4b4bb1f238","entry":"half_function","repo":"opengvlab/hulk","repo_kind":"official","path":"core/fp16/amp.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/fp16/amp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b4eee4b4bb1f238"}},{"code_sha256_prefix":"7c26a4ba6aca54f0","entry":"load_last_iter","repo":"opengvlab/hulk","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c26a4ba6aca54f0"}},{"code_sha256_prefix":"b14254d606cb6719","entry":"param_group_no_wd","repo":"opengvlab/humanbench","repo_kind":"official","path":"PATH/core/make_param_group.py","file_url":"https://github.com/opengvlab/humanbench/blob/HEAD/PATH/core/make_param_group.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b14254d606cb6719"}},{"code_sha256_prefix":"b1471eccec180d87","entry":"simple_group_split","repo":"opengvlab/humanbench","repo_kind":"official","path":"PATH/core/distributed_utils.py","file_url":"https://github.com/opengvlab/humanbench/blob/HEAD/PATH/core/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1471eccec180d87"}},{"code_sha256_prefix":"4e36f197c125d4d3","entry":"specific_group_split","repo":"opengvlab/humanbench","repo_kind":"official","path":"PATH/core/distributed_utils.py","file_url":"https://github.com/opengvlab/humanbench/blob/HEAD/PATH/core/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e36f197c125d4d3"}},{"code_sha256_prefix":"831beea1b4db57e3","entry":"specific_group_split","repo":"opengvlab/hulk","repo_kind":"official","path":"core/config.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"831beea1b4db57e3"}},{"code_sha256_prefix":"62ad1cdb3bc504fe","entry":"specific_group_split","repo":"opengvlab/humanbench","repo_kind":"official","path":"PATH/core/config.py","file_url":"https://github.com/opengvlab/humanbench/blob/HEAD/PATH/core/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62ad1cdb3bc504fe"}},{"code_sha256_prefix":"6f5d8cf806cbe77c","entry":"specific_group_split_modality_groups","repo":"opengvlab/hulk","repo_kind":"official","path":"core/config.py","file_url":"https://github.com/opengvlab/hulk/blob/HEAD/core/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f5d8cf806cbe77c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}