{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-s-in-the-box-an-analysis-of-undesirable","title":"What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus","arxiv_id":"2105.02732","date":"2021-05-06","proceeding":null,"authors":["Alexandra Sasha Luccioni","Joseph D. Viviano"],"abstract":"Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.","url_abs":"https://arxiv.org/abs/2105.02732v3","url_pdf":"https://arxiv.org/pdf/2105.02732v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-s-in-the-box-an-analysis-of-undesirable","repo_url":"https://github.com/josephdviviano/whatsinthebox","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.02732","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.02732"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/josephdviviano/whatsinthebox","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1169853f491c612b","entry":"count_ngram_matches","repo":"josephdviviano/whatsinthebox","repo_kind":"official","path":"witb/utils/nlp.py","file_url":"https://github.com/josephdviviano/whatsinthebox/blob/HEAD/witb/utils/nlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1169853f491c612b"}},{"code_sha256_prefix":"14d492da37c493cd","entry":"get_ngrams","repo":"josephdviviano/whatsinthebox","repo_kind":"official","path":"witb/utils/nlp.py","file_url":"https://github.com/josephdviviano/whatsinthebox/blob/HEAD/witb/utils/nlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14d492da37c493cd"}},{"code_sha256_prefix":"7422d8302575aee0","entry":"read_wet_file","repo":"josephdviviano/whatsinthebox","repo_kind":"official","path":"witb/utils/ioutils.py","file_url":"https://github.com/josephdviviano/whatsinthebox/blob/HEAD/witb/utils/ioutils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7422d8302575aee0"}},{"code_sha256_prefix":"fea3aa5fb762a346","entry":"read_yaml","repo":"josephdviviano/whatsinthebox","repo_kind":"official","path":"witb/utils/ioutils.py","file_url":"https://github.com/josephdviviano/whatsinthebox/blob/HEAD/witb/utils/ioutils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fea3aa5fb762a346"}},{"code_sha256_prefix":"ba77e5c34c34d989","entry":"remove_unicode_punct","repo":"josephdviviano/whatsinthebox","repo_kind":"official","path":"witb/utils/textutils.py","file_url":"https://github.com/josephdviviano/whatsinthebox/blob/HEAD/witb/utils/textutils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba77e5c34c34d989"}},{"code_sha256_prefix":"2f7a4b9a8b78c0fd","entry":"replace_unicode_punct","repo":"josephdviviano/whatsinthebox","repo_kind":"official","path":"witb/utils/textutils.py","file_url":"https://github.com/josephdviviano/whatsinthebox/blob/HEAD/witb/utils/textutils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f7a4b9a8b78c0fd"}},{"code_sha256_prefix":"c2e55f8f1a1f1444","entry":"strip_accents","repo":"josephdviviano/whatsinthebox","repo_kind":"official","path":"witb/utils/textutils.py","file_url":"https://github.com/josephdviviano/whatsinthebox/blob/HEAD/witb/utils/textutils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2e55f8f1a1f1444"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}