{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gram-fast-fine-tuning-of-pre-trained-language","title":"GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering","arxiv_id":"2204.04179","date":"2022-04-08","proceeding":"NAACL 2022 7","authors":["Yoonseok Yang","Kyu Seok Kim","Minsam Kim","Juneyoung Park"],"abstract":"Content-based collaborative filtering (CCF) predicts user-item interactions based on both users' interaction history and items' content information. Recently, pre-trained language models (PLM) have been used to extract high-quality item encodings for CCF. However, it is resource-intensive to train a PLM-based CCF model in an end-to-end (E2E) manner, since optimization involves back-propagating through every content encoding within a given user interaction sequence. To tackle this issue, we propose GRAM (GRadient Accumulation for Multi-modality in CCF), which exploits the fact that a given item often appears multiple times within a batch of interaction histories. Specifically, Single-step GRAM aggregates each item encoding's gradients for back-propagation, with theoretic equivalence to the standard E2E training. As an extension of Single-step GRAM, we propose Multi-step GRAM, which increases the gradient update latency, achieving a further speedup with drastically less GPU memory. GRAM significantly improves training efficiency (up to 146x) on five datasets from two task domains of Knowledge Tracing and News Recommendation. Our code is available at https://github.com/yoonseok312/GRAM.","url_abs":"https://arxiv.org/abs/2204.04179v2","url_pdf":"https://arxiv.org/pdf/2204.04179v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gram-fast-fine-tuning-of-pre-trained-language","repo_url":"https://github.com/yoonseok312/gram","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"collaborative-filtering","task_name":"Collaborative Filtering"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"knowledge-tracing","task_name":"Knowledge Tracing"},{"task_slug":"news-recommendation","task_name":"News Recommendation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.04179","atlas_url":"https://app.syntology.ai/?focus=2204.04179","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.04179"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yoonseok312/GRAM","reach":null}],"summary":{"ran":5,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"cae4f1b8ddc3713c","entry":"AdditiveAttention","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cae4f1b8ddc3713c"}},{"code_sha256_prefix":"11413dcf5f8a30c0","entry":"ElementEncoder","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11413dcf5f8a30c0"}},{"code_sha256_prefix":"a8fe4d06671859f3","entry":"MultiHeadAttention","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8fe4d06671859f3"}},{"code_sha256_prefix":"ee47e6c01a7302ed","entry":"ScaledDotProductAttention","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee47e6c01a7302ed"}},{"code_sha256_prefix":"4c3ddcb374fd904d","entry":"TextEncoder","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c3ddcb374fd904d"}},{"code_sha256_prefix":"1dc11117180ddb6d","entry":"ModelBert","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1dc11117180ddb6d"}},{"code_sha256_prefix":"bb00e2907229e014","entry":"NewsEncoder","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb00e2907229e014"}},{"code_sha256_prefix":"8bf18760aea7dc62","entry":"UserEncoder","repo":"yoonseok312/GRAM","repo_kind":"official","path":"gram/news_recommendation/model_bert.py","file_url":"https://github.com/yoonseok312/GRAM/blob/HEAD/gram/news_recommendation/model_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bf18760aea7dc62"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}