{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learn-to-preserve-and-diversify-parameter","title":"Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization","arxiv_id":"2407.15085","date":"2024-07-21","proceeding":null,"authors":["Jiajun Hu","Jian Zhang","Lei Qi","Yinghuan Shi","Yang Gao"],"abstract":"Domain generalization (DG) aims to avoid the performance degradation of the model when the distribution shift between the limited training data and unseen test data occurs. Recently, foundation models with enormous parameters have been pre-trained with huge datasets, demonstrating strong generalization ability and showing promising direction for solving the DG problem. However, fully Fine-Tuning (FT) the foundation models results in unsatisfactory out-of-distribution accuracy due to the destroyed pre-trained generalized features. Recently, Parameter-Efficient Fine-Tuning (PEFT) alleviates the above problem by fine-tuning a small portion of the model parameters while keeping the rest frozen, which achieves better generalization performance compared to FT. Nevertheless, PEFT still suffers from the issue of overfitting to the training domains. To address the above issue, we propose Parameter-Efficient Group with Orthogonal regularization (PEGO) for vision transformers, which effectively preserves the generalization ability of the pre-trained network and learns more diverse knowledge compared with conventional PEFT. Specifically, we inject a group of trainable Low-Rank Adaptation (LoRA) modules into the pre-trained model and propose an orthogonal regularization loss to enhance the generalization ability of the model. Our framework achieves SOTA performance on five DG benchmarks, while only requiring training a small number of parameters without adding additional testing cost.","url_abs":"https://arxiv.org/abs/2407.15085v1","url_pdf":"https://arxiv.org/pdf/2407.15085v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learn-to-preserve-and-diversify-parameter","repo_url":"https://github.com/JudgingH/PEGO","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"orthogonal-regularization","method_name":"Orthogonal Regularization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.15085","atlas_url":"https://app.syntology.ai/?focus=2407.15085","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.15085"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JudgingH/PEGO","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":8,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"00e569acd6b45ef0","entry":"conv3x3","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/wide_resnet.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/wide_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00e569acd6b45ef0"}},{"code_sha256_prefix":"02a21c87d500ef9d","entry":"default_hparams","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/hparams_registry.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/hparams_registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02a21c87d500ef9d"}},{"code_sha256_prefix":"c5919b96db289dbd","entry":"get_shapes","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/algorithms/miro.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/algorithms/miro.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5919b96db289dbd"}},{"code_sha256_prefix":"71a3a61ceed99bf3","entry":"hashable","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/query.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/query.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71a3a61ceed99bf3"}},{"code_sha256_prefix":"2f597a2077a1c95d","entry":"levelize","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/logger.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f597a2077a1c95d"}},{"code_sha256_prefix":"2f8af5779e1edcb6","entry":"make_selector_fn","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/query.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/query.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f8af5779e1edcb6"}},{"code_sha256_prefix":"955c4020424d8ac1","entry":"make_weights_for_balanced_classes","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/misc.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"955c4020424d8ac1"}},{"code_sha256_prefix":"58e06fd6cfa6a889","entry":"random_hparams","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/hparams_registry.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/hparams_registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58e06fd6cfa6a889"}},{"code_sha256_prefix":"4fa2b54178fcc1df","entry":"random_pairs_of_minibatches","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/misc.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/misc.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fa2b54178fcc1df"}},{"code_sha256_prefix":"12c23ce22097acd0","entry":"to_minibatch","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/algorithms/algorithms.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/algorithms/algorithms.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12c23ce22097acd0"}},{"code_sha256_prefix":"5dd6a674fbe45a89","entry":"to_row","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/misc.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dd6a674fbe45a89"}},{"code_sha256_prefix":"687efdf3db9c44df","entry":"accuracy_from_loader","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/evaluator.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"687efdf3db9c44df"}},{"code_sha256_prefix":"aa6558d41d5c4673","entry":"get_writer","repo":"JudgingH/PEGO","repo_kind":"official","path":"domainbed/lib/writers.py","file_url":"https://github.com/JudgingH/PEGO/blob/HEAD/domainbed/lib/writers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa6558d41d5c4673"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}