{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/object-proposal-evaluation-protocol-is","title":"Object-Proposal Evaluation Protocol is 'Gameable'","arxiv_id":"1505.05836","date":"2015-05-21","proceeding":"CVPR 2016 6","authors":["Neelima Chavali","Harsh Agrawal","Aroma Mahendru","Dhruv Batra"],"abstract":"Object proposals have quickly become the de-facto pre-processing step in a\nnumber of vision pipelines (for object detection, object discovery, and other\ntasks). Their performance is usually evaluated on partially annotated datasets.\nIn this paper, we argue that the choice of using a partially annotated dataset\nfor evaluation of object proposals is problematic -- as we demonstrate via a\nthought experiment, the evaluation protocol is 'gameable', in the sense that\nprogress under this protocol does not necessarily correspond to a \"better\"\ncategory independent object proposal algorithm.\n  To alleviate this problem, we: (1) Introduce a nearly-fully annotated version\nof PASCAL VOC dataset, which serves as a test-bed to check if object proposal\ntechniques are overfitting to a particular list of categories. (2) Perform an\nexhaustive evaluation of object proposal methods on our introduced nearly-fully\nannotated PASCAL dataset and perform cross-dataset generalization experiments;\nand (3) Introduce a diagnostic experiment to detect the bias capacity in an\nobject proposal algorithm. This tool circumvents the need to collect a densely\nannotated dataset, which can be expensive and cumbersome to collect. Finally,\nwe plan to release an easy-to-use toolbox which combines various publicly\navailable implementations of object proposal algorithms which standardizes the\nproposal generation and evaluation so that new methods can be added and\nevaluated on different datasets. We hope that the results presented in the\npaper will motivate the community to test the category independence of various\nobject proposal methods by carefully choosing the evaluation protocol.","url_abs":"http://arxiv.org/abs/1505.05836v4","url_pdf":"http://arxiv.org/pdf/1505.05836v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"object-proposal-evaluation-protocol-is","repo_url":"https://github.com/Cloud-CV/object-proposals","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-discovery","task_name":"Object Discovery"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1505.05836","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}