{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/realistic-evaluation-of-transductive-few-shot-1","title":"Realistic Evaluation of Transductive Few-Shot Learning","arxiv_id":"2204.11181","date":"2022-04-24","proceeding":"NeurIPS 2021 12","authors":["Olivier Veilleux","Malik Boudiaf","Pablo Piantanida","Ismail Ben Ayed"],"abstract":"Transductive inference is widely used in few-shot learning, as it leverages the statistics of the unlabeled query set of a few-shot task, typically yielding substantially better performances than its inductive counterpart. The current few-shot benchmarks use perfectly class-balanced tasks at inference. We argue that such an artificial regularity is unrealistic, as it assumes that the marginal label probability of the testing samples is known and fixed to the uniform distribution. In fact, in realistic scenarios, the unlabeled query sets come with arbitrary and unknown label marginals. We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference, removing the class-balance artefact. Specifically, we model the marginal probabilities of the classes as Dirichlet-distributed random variables, which yields a principled and realistic sampling within the simplex. This leverages the current few-shot benchmarks, building testing tasks with arbitrary class distributions. We evaluate experimentally state-of-the-art transductive methods over 3 widely used data sets, and observe, surprisingly, substantial performance drops, even below inductive methods in some cases. Furthermore, we propose a generalization of the mutual-information loss, based on $\\alpha$-divergences, which can handle effectively class-distribution variations. Empirically, we show that our transductive $\\alpha$-divergence optimization outperforms state-of-the-art methods across several data sets, models and few-shot settings. Our code is publicly available at https://github.com/oveilleux/Realistic_Transductive_Few_Shot.","url_abs":"https://arxiv.org/abs/2204.11181v1","url_pdf":"https://arxiv.org/pdf/2204.11181v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"realistic-evaluation-of-transductive-few-shot-1","repo_url":"https://github.com/oveilleux/realistic_transductive_few_shot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/few-shot-image-classification-on-dirichlet-4","task":"Few-Shot Image Classification","dataset":"Dirichlet CUB-200 (5-way, 1-shot)","model":"\\alpha-TIM","rank_in_archive_order":2,"of":8,"metrics":{"1:1 Accuracy":"75.7"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-image-classification-on-dirichlet-5","task":"Few-Shot Image Classification","dataset":"Dirichlet CUB-200 (5-way, 5-shot)","model":"\\alpha-TIM","rank_in_archive_order":2,"of":8,"metrics":{"1:1 Accuracy":"89.8"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-image-classification-on-dirichlet","task":"Few-Shot Image Classification","dataset":"Dirichlet Mini-Imagenet (5-way, 1-shot)","model":"\\alpha-TIM","rank_in_archive_order":2,"of":12,"metrics":{"1:1 Accuracy":"67.4"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-image-classification-on-dirichlet-1","task":"Few-Shot Image Classification","dataset":"Dirichlet Mini-Imagenet (5-way, 5-shot)","model":"\\alpha-TIM","rank_in_archive_order":2,"of":12,"metrics":{"1:1 Accuracy":"82.5"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-image-classification-on-dirichlet-2","task":"Few-Shot Image Classification","dataset":"Dirichlet Tiered-Imagenet (5-way, 1-shot)","model":"\\alpha-TIM","rank_in_archive_order":3,"of":9,"metrics":{"1:1 Accuracy":"74.4"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-image-classification-on-dirichlet-3","task":"Few-Shot Image Classification","dataset":"Dirichlet Tiered-Imagenet (5-way, 5-shot)","model":"\\alpha-TIM","rank_in_archive_order":1,"of":9,"metrics":{"1:1 Accuracy":"86.6"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.11181","atlas_url":"https://app.syntology.ai/?focus=2204.11181","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.11181"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/oveilleux/Realistic_Transductive_Few_Shot","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/oveilleux/realistic_transductive_few_shot","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":4,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"c2bb0431f66349c4","entry":"Logger","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2bb0431f66349c4"}},{"code_sha256_prefix":"9612cc426ca84971","entry":"TIM","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9612cc426ca84971"}},{"code_sha256_prefix":"1e25337f3b8c7d05","entry":"extract_features","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e25337f3b8c7d05"}},{"code_sha256_prefix":"a611fe8fce609022","entry":"get_cond_entropy","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a611fe8fce609022"}},{"code_sha256_prefix":"da0a1da72474e584","entry":"get_mi","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da0a1da72474e584"}},{"code_sha256_prefix":"a03c85ae21c3b667","entry":"get_one_hot","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a03c85ae21c3b667"}},{"code_sha256_prefix":"e01d7f2363beb45a","entry":"ALPHA_TIM","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e01d7f2363beb45a"}},{"code_sha256_prefix":"6a236e22aa59cecc","entry":"convert_prob_to_samples","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/datasets/sampler.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/datasets/sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a236e22aa59cecc"}},{"code_sha256_prefix":"fdf1fa9384ed9dcb","entry":"get_dirichlet_query_dist","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/datasets/sampler.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/datasets/sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdf1fa9384ed9dcb"}},{"code_sha256_prefix":"e6a5040deda27fed","entry":"get_entropy","repo":"oveilleux/realistic_transductive_few_shot","repo_kind":"official","path":"src/methods/tim.py","file_url":"https://github.com/oveilleux/realistic_transductive_few_shot/blob/HEAD/src/methods/tim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6a5040deda27fed"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}