{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/polyformer-referring-image-segmentation-as","title":"PolyFormer: Referring Image Segmentation as Sequential Polygon Generation","arxiv_id":"2302.07387","date":"2023-02-14","proceeding":"CVPR 2023 1","authors":["Jiang Liu","Hui Ding","Zhaowei Cai","Yuting Zhang","Ravi Kumar Satzoda","Vijay Mahadevan","R. Manmatha"],"abstract":"In this work, instead of directly predicting the pixel-level segmentation masks, the problem of referring image segmentation is formulated as sequential polygon generation, and the predicted polygons can be later converted into segmentation masks. This is enabled by a new sequence-to-sequence framework, Polygon Transformer (PolyFormer), which takes a sequence of image patches and text query tokens as input, and outputs a sequence of polygon vertices autoregressively. For more accurate geometric localization, we propose a regression-based decoder, which predicts the precise floating-point coordinates directly, without any coordinate quantization error. In the experiments, PolyFormer outperforms the prior art by a clear margin, e.g., 5.40% and 4.52% absolute improvements on the challenging RefCOCO+ and RefCOCOg datasets. It also shows strong generalization ability when evaluated on the referring video segmentation task without fine-tuning, e.g., achieving competitive 61.5% J&F on the Ref-DAVIS17 dataset.","url_abs":"https://arxiv.org/abs/2302.07387v2","url_pdf":"https://arxiv.org/pdf/2302.07387v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"polyformer-referring-image-segmentation-as","repo_url":"https://github.com/amazon-science/polygon-transformer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"referring-expression-comprehension","task_name":"Referring Expression Comprehension"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-segmentation","task_name":"Video Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/referring-expression-segmentation-on-davis","task":"Referring Expression Segmentation","dataset":"DAVIS 2017 (val)","model":"PolyFormer-B","rank_in_archive_order":8,"of":18,"metrics":{"J&F 1st frame":"60.9","Zero-Shot Transfer":"true"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-5","task":"Referring Expression Segmentation","dataset":"RefCOCO+ test B","model":"PolyFormer-L","rank_in_archive_order":12,"of":30,"metrics":{"Mean IoU":"66.73","Overall IoU":"61.87"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-5","task":"Referring Expression Segmentation","dataset":"RefCOCO+ test B","model":"PolyFormer-B","rank_in_archive_order":14,"of":30,"metrics":{"Mean IoU":"64.64","Overall IoU":"59.33"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-4","task":"Referring Expression Segmentation","dataset":"RefCOCO+ testA","model":"PolyFormer-L","rank_in_archive_order":12,"of":30,"metrics":{"Mean IoU":"75.71","Overall IoU":"74.56"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-4","task":"Referring Expression Segmentation","dataset":"RefCOCO+ testA","model":"PolyFormer-B","rank_in_archive_order":15,"of":30,"metrics":{"Mean IoU":"74.51","Overall IoU":"72.89"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-3","task":"Referring Expression Segmentation","dataset":"RefCOCO+ val","model":"PolyFormer-L","rank_in_archive_order":16,"of":33,"metrics":{"Mean IoU":"72.15","Overall IoU":"69.33"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco-3","task":"Referring Expression Segmentation","dataset":"RefCOCO+ val","model":"PolyFormer-B","rank_in_archive_order":17,"of":33,"metrics":{"Mean IoU":"70.65","Overall IoU":"67.64"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog-1","task":"Referring Expression Segmentation","dataset":"RefCOCOg-test","model":"PolyFormer-L","rank_in_archive_order":11,"of":18,"metrics":{"Mean IoU":"71.17","Overall IoU":"70.19"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog-1","task":"Referring Expression Segmentation","dataset":"RefCOCOg-test","model":"PolyFormer-B","rank_in_archive_order":12,"of":18,"metrics":{"Mean IoU":"69.88","Overall IoU":"69.05"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog","task":"Referring Expression Segmentation","dataset":"RefCOCOg-val","model":"PolyFormer-L","rank_in_archive_order":12,"of":23,"metrics":{"Mean IoU":"71.15","Overall IoU":"69.2"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcocog","task":"Referring Expression Segmentation","dataset":"RefCOCOg-val","model":"PolyFormer-B","rank_in_archive_order":14,"of":23,"metrics":{"Mean IoU":"69.36","Overall IoU":"67.76"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco","task":"Referring Expression Segmentation","dataset":"RefCoCo val","model":"PolyFormer-L","rank_in_archive_order":17,"of":37,"metrics":{"Mean IoU":"76.94","Overall IoU":"75.96"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refcoco","task":"Referring Expression Segmentation","dataset":"RefCoCo val","model":"PolyFormer-B","rank_in_archive_order":19,"of":37,"metrics":{"Overall IoU":"74.82"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-referit","task":"Referring Expression Segmentation","dataset":"ReferIt","model":"PolyFormer-L","rank_in_archive_order":1,"of":3,"metrics":{"Mean IoU":"67.22","Overall IoU":"72.6"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-referit","task":"Referring Expression Segmentation","dataset":"ReferIt","model":"PolyFormer-B","rank_in_archive_order":2,"of":3,"metrics":{"Mean IoU":"65.98","Overall IoU":"71.91"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.07387","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.07387"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/polygon-transformer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"f18199d037514305","entry":"BatchNorm2d","repo":"amazon-science/polygon-transformer","repo_kind":"official","path":"models/polyformer/unify_transformer.py","file_url":"https://github.com/amazon-science/polygon-transformer/blob/HEAD/models/polyformer/unify_transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f18199d037514305"}},{"code_sha256_prefix":"90fa02b8cae75fe7","entry":"drop_path","repo":"amazon-science/polygon-transformer","repo_kind":"official","path":"models/polyformer/unify_transformer_layer.py","file_url":"https://github.com/amazon-science/polygon-transformer/blob/HEAD/models/polyformer/unify_transformer_layer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"90fa02b8cae75fe7"}},{"code_sha256_prefix":"fba339cb809b27fe","entry":"make_image_bucket_position","repo":"amazon-science/polygon-transformer","repo_kind":"official","path":"models/polyformer/unify_transformer.py","file_url":"https://github.com/amazon-science/polygon-transformer/blob/HEAD/models/polyformer/unify_transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fba339cb809b27fe"}},{"code_sha256_prefix":"c0b56d6f87906655","entry":"make_token_bucket_position","repo":"amazon-science/polygon-transformer","repo_kind":"official","path":"models/polyformer/unify_transformer.py","file_url":"https://github.com/amazon-science/polygon-transformer/blob/HEAD/models/polyformer/unify_transformer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c0b56d6f87906655"}},{"code_sha256_prefix":"22b41155e451683b","entry":"mish","repo":"amazon-science/polygon-transformer","repo_kind":"official","path":"bert/modeling_bert.py","file_url":"https://github.com/amazon-science/polygon-transformer/blob/HEAD/bert/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"22b41155e451683b"}},{"code_sha256_prefix":"fb9398de2966d6de","entry":"load_tf_weights_in_bert","repo":"amazon-science/polygon-transformer","repo_kind":"official","path":"bert/modeling_bert.py","file_url":"https://github.com/amazon-science/polygon-transformer/blob/HEAD/bert/modeling_bert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fb9398de2966d6de"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}