{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/himtok-learning-hierarchical-mask-tokens-for","title":"HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model","arxiv_id":"2503.13026","date":"2025-03-17","proceeding":null,"authors":["Tao Wang","Changxu Cheng","Lingfeng Wang","Senda Chen","Wuyue Zhao"],"abstract":"The remarkable performance of large multimodal models (LMMs) has attracted significant interest from the image segmentation community. To align with the next-token-prediction paradigm, current LMM-driven segmentation methods either use object boundary points to represent masks or introduce special segmentation tokens, whose hidden states are decoded by a segmentation model requiring the original image as input. However, these approaches often suffer from inadequate mask representation and complex architectures, limiting the potential of LMMs. In this work, we propose the Hierarchical Mask Tokenizer (HiMTok), which represents segmentation masks with up to 32 tokens and eliminates the need for the original image during mask de-tokenization. HiMTok allows for compact and coarse-to-fine mask representations, aligning well with the LLM next-token-prediction paradigm and facilitating the direct acquisition of segmentation capabilities. We develop a 3-stage training recipe for progressive learning of segmentation and visual capabilities, featuring a hierarchical mask loss for effective coarse-to-fine learning. Additionally, we enable bidirectional information flow, allowing conversion between bounding boxes and mask tokens to fully leverage multi-task training potential. Extensive experiments demonstrate that our method achieves state-of-the-art performance across various segmentation tasks,while also enhancing visual grounding and maintaining overall visual understanding.","url_abs":"https://arxiv.org/abs/2503.13026v1","url_pdf":"https://arxiv.org/pdf/2503.13026v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"himtok-learning-hierarchical-mask-tokens-for","repo_url":"https://github.com/yayafengzi/LMM-HiMTok","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2503.13026","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.13026"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yayafengzi/LMM-HiMTok","reach":null}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10cffbe1e3498007","entry":"Conv2dSame","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10cffbe1e3498007"}},{"code_sha256_prefix":"8d70fe4156ce8962","entry":"ResidualAttentionBlock","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d70fe4156ce8962"}},{"code_sha256_prefix":"9142492bb73c2243","entry":"TiTokDecoder","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9142492bb73c2243"}},{"code_sha256_prefix":"9d5d0b05c060bba6","entry":"VectorQuantizer","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d5d0b05c060bba6"}},{"code_sha256_prefix":"8a18fd0e92a888ed","entry":"_expand_token","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a18fd0e92a888ed"}},{"code_sha256_prefix":"67105525709960da","entry":"BaseModel","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67105525709960da"}},{"code_sha256_prefix":"e4549d7f35e76f6f","entry":"HiMT","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4549d7f35e76f6f"}},{"code_sha256_prefix":"948ed9d3fca0a0b0","entry":"ResnetBlock","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"948ed9d3fca0a0b0"}},{"code_sha256_prefix":"17e460a2bb81e496","entry":"TiTokEncoder","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17e460a2bb81e496"}},{"code_sha256_prefix":"a743a53b959dc7db","entry":"UpsamplingBlock","repo":"yayafengzi/LMM-HiMTok","repo_kind":"official","path":"himt/himt.py","file_url":"https://github.com/yayafengzi/LMM-HiMTok/blob/HEAD/himt/himt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a743a53b959dc7db"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}