{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/interpreting-transformers-for-jet-tagging","title":"Interpreting Transformers for Jet Tagging","arxiv_id":"2412.03673","date":"2024-12-04","proceeding":null,"authors":["Aaron Wang","Abhijith Gandrakota","Jennifer Ngadiuba","Vivekanand Sahu","Priyansh Bhatnagar","Elham E Khoda","Javier Duarte"],"abstract":"Machine learning (ML) algorithms, particularly attention-based transformer models, have become indispensable for analyzing the vast data generated by particle physics experiments like ATLAS and CMS at the CERN LHC. Particle Transformer (ParT), a state-of-the-art model, leverages particle-level attention to improve jet-tagging tasks, which are critical for identifying particles resulting from proton collisions. This study focuses on interpreting ParT by analyzing attention heat maps and particle-pair correlations on the $\\eta$-$\\phi$ plane, revealing a binary attention pattern where each particle attends to at most one other particle. At the same time, we observe that ParT shows varying focus on important particles and subjets depending on decay, indicating that the model learns traditional jet substructure observables. These insights enhance our understanding of the model's internal workings and learning process, offering potential avenues for improving the efficiency of transformer architectures in future high-energy physics applications.","url_abs":"https://arxiv.org/abs/2412.03673v2","url_pdf":"https://arxiv.org/pdf/2412.03673v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"interpreting-transformers-for-jet-tagging","repo_url":"https://github.com/aaronw5/Interpreting-Transformers-for-Jet-Tagging","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"jet-tagging","task_name":"Jet Tagging"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.03673","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.03673"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aaronw5/Interpreting-Transformers-for-Jet-Tagging","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"3373564a9570e6db","entry":"get_loss","repo":"aaronw5/Interpreting-Transformers-for-Jet-Tagging","repo_kind":"official","path":"networks/ParT_drop.py","file_url":"https://github.com/aaronw5/Interpreting-Transformers-for-Jet-Tagging/blob/HEAD/networks/ParT_drop.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3373564a9570e6db"}},{"code_sha256_prefix":"bb686b06414f8ac0","entry":"extract_particle_features","repo":"aaronw5/Interpreting-Transformers-for-Jet-Tagging","repo_kind":"official","path":"preprocessing/convert_h5_to_parquet.py","file_url":"https://github.com/aaronw5/Interpreting-Transformers-for-Jet-Tagging/blob/HEAD/preprocessing/convert_h5_to_parquet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb686b06414f8ac0"}},{"code_sha256_prefix":"0a3fe61e095e5a19","entry":"get_model","repo":"aaronw5/Interpreting-Transformers-for-Jet-Tagging","repo_kind":"official","path":"networks/example_PCNN.py","file_url":"https://github.com/aaronw5/Interpreting-Transformers-for-Jet-Tagging/blob/HEAD/networks/example_PCNN.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a3fe61e095e5a19"}},{"code_sha256_prefix":"d974984c61c03742","entry":"get_model","repo":"aaronw5/Interpreting-Transformers-for-Jet-Tagging","repo_kind":"official","path":"networks/example_PFN.py","file_url":"https://github.com/aaronw5/Interpreting-Transformers-for-Jet-Tagging/blob/HEAD/networks/example_PFN.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d974984c61c03742"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}