{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/calibrated-prediction-in-and-out-of-domain","title":"DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling","arxiv_id":"2105.12441","date":"2021-05-26","proceeding":"ICCV 2021 10","authors":["Akis Linardos","Matthias Kümmerer","Ori Press","Matthias Bethge"],"abstract":"Since 2014 transfer learning has become the key driver for the improvement of spatial saliency prediction; however, with stagnant progress in the last 3-5 years. We conduct a large-scale transfer learning study which tests different ImageNet backbones, always using the same read out architecture and learning protocol adopted from DeepGaze II. By replacing the VGG19 backbone of DeepGaze II with ResNet50 features we improve the performance on saliency prediction from 78% to 85%. However, as we continue to test better ImageNet models as backbones (such as EfficientNetB5) we observe no additional improvement on saliency prediction. By analyzing the backbones further, we find that generalization to other datasets differs substantially, with models being consistently overconfident in their fixation predictions. We show that by combining multiple backbones in a principled manner a good confidence calibration on unseen datasets can be achieved. This new model, \"DeepGaze IIE\", yields a significant leap in benchmark performance in and out-of-domain with a 15 percent point improvement over DeepGaze II to 93% on MIT1003, marking a new state of the art on the MIT/Tuebingen Saliency Benchmark in all available metrics (AUC: 88.3%, sAUC: 79.4%, CC: 82.4%).","url_abs":"https://arxiv.org/abs/2105.12441v3","url_pdf":"https://arxiv.org/pdf/2105.12441v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"calibrated-prediction-in-and-out-of-domain","repo_url":"https://github.com/matthias-k/DeepGaze","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"calibrated-prediction-in-and-out-of-domain","repo_url":"https://github.com/islab-sophia/odisal","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"saliency-prediction","task_name":"Saliency Prediction"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2105.12441","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.12441"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/matthias-k/DeepGaze","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/islab-sophia/odisal","reach":null}],"summary":{"ran":9,"ran_draft_wrong":3,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":16,"ran":13,"repositories":1},"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"782fbc775dc6b910","entry":"Bias","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"782fbc775dc6b910"}},{"code_sha256_prefix":"b459556b97135112","entry":"Conv2dMultiInput","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b459556b97135112"}},{"code_sha256_prefix":"81c3534e413bef0d","entry":"DeepGazeIIIMixture","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81c3534e413bef0d"}},{"code_sha256_prefix":"05a7be491393286d","entry":"FeatureExtractor","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05a7be491393286d"}},{"code_sha256_prefix":"89450c83c8ee18d4","entry":"Finalizer","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89450c83c8ee18d4"}},{"code_sha256_prefix":"9ed0d69c62c3b5dc","entry":"GaussianFilterNd","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ed0d69c62c3b5dc"}},{"code_sha256_prefix":"17f528f7b814b15f","entry":"LayerNorm","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17f528f7b814b15f"}},{"code_sha256_prefix":"4581eabb812cb3ff","entry":"LayerNormMultiInput","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4581eabb812cb3ff"}},{"code_sha256_prefix":"9be88f282c69b1b4","entry":"MixtureModel","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9be88f282c69b1b4"}},{"code_sha256_prefix":"4b5054af2d648043","entry":"build_saliency_network","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b5054af2d648043"}},{"code_sha256_prefix":"325af44c6cd8829e","entry":"encode_scanpath_features","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"325af44c6cd8829e"}},{"code_sha256_prefix":"27f99b6bedd193fb","entry":"gaussian_filter_1d","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27f99b6bedd193fb"}},{"code_sha256_prefix":"fb1d63f8099f907a","entry":"import_class","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb1d63f8099f907a"}},{"code_sha256_prefix":"070fdd717ed94582","entry":"DeepGazeIIE","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"070fdd717ed94582"}},{"code_sha256_prefix":"7d46769a9a82cd86","entry":"build_deepgaze_mixture","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d46769a9a82cd86"}},{"code_sha256_prefix":"4e411ff28ffe758d","entry":"build_deepgaze_mixture","repo":"islab-sophia/odisal","repo_kind":"listed","path":"program/models/deepgaze2e.py","file_url":"https://github.com/islab-sophia/odisal/blob/HEAD/program/models/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e411ff28ffe758d"}},{"code_sha256_prefix":"2d1db431509c80ca","entry":"build_fixation_selection_network","repo":"matthias-k/DeepGaze","repo_kind":"official","path":"deepgaze_pytorch/deepgaze2e.py","file_url":"https://github.com/matthias-k/DeepGaze/blob/HEAD/deepgaze_pytorch/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d1db431509c80ca"}},{"code_sha256_prefix":"633b7efb12453cdd","entry":"build_fixation_selection_network","repo":"islab-sophia/odisal","repo_kind":"listed","path":"program/models/deepgaze2e.py","file_url":"https://github.com/islab-sophia/odisal/blob/HEAD/program/models/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"633b7efb12453cdd"}},{"code_sha256_prefix":"9141b2bb636a8297","entry":"build_saliency_network","repo":"islab-sophia/odisal","repo_kind":"listed","path":"program/models/deepgaze2e.py","file_url":"https://github.com/islab-sophia/odisal/blob/HEAD/program/models/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9141b2bb636a8297"}},{"code_sha256_prefix":"fa03a00d81540b0f","entry":"deepgaze2e","repo":"islab-sophia/odisal","repo_kind":"listed","path":"program/models/deepgaze2e.py","file_url":"https://github.com/islab-sophia/odisal/blob/HEAD/program/models/deepgaze2e.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa03a00d81540b0f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}