{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-models-have-a-moral-dimension","title":"Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do","arxiv_id":"2103.11790","date":"2021-03-08","proceeding":null,"authors":["Patrick Schramowski","Cigdem Turan","Nico Andersen","Constantin A. Rothkopf","Kristian Kersting"],"abstract":"Artificial writing is permeating our lives due to recent advances in large-scale, transformer-based language models (LMs) such as BERT, its variants, GPT-2/3, and others. Using them as pre-trained models and fine-tuning them for specific tasks, researchers have extended state of the art for many NLP tasks and shown that they capture not only linguistic knowledge but also retain general knowledge implicitly present in the data. Unfortunately, LMs trained on unfiltered text corpora suffer from degenerated and biased behaviour. While this is well established, we show that recent LMs also contain human-like biases of what is right and wrong to do, some form of ethical and moral norms of the society -- they bring a \"moral direction\" to surface. That is, we show that these norms can be captured geometrically by a direction, which can be computed, e.g., by a PCA, in the embedding space, reflecting well the agreement of phrases to social norms implicitly expressed in the training texts and providing a path for attenuating or even preventing toxic degeneration in LMs. Being able to rate the (non-)normativity of arbitrary phrases without explicitly training the LM for this task, we demonstrate the capabilities of the \"moral direction\" for guiding (even other) LMs towards producing normative text and showcase it on RealToxicityPrompts testbed, preventing the neural toxic degeneration in GPT-2.","url_abs":"https://arxiv.org/abs/2103.11790v3","url_pdf":"https://arxiv.org/pdf/2103.11790v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-models-have-a-moral-dimension","repo_url":"https://github.com/ml-research/MoRT_NMI","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"general-knowledge","task_name":"General Knowledge"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"pca","method_name":"PCA"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.11790","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.11790"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-research/MoRT_NMI","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7c98160760473f83","entry":"best_and_worst_question","repo":"ml-research/MoRT_NMI","repo_kind":"official","path":"MoRT/mort/funcs_mcm.py","file_url":"https://github.com/ml-research/MoRT_NMI/blob/HEAD/MoRT/mort/funcs_mcm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c98160760473f83"}},{"code_sha256_prefix":"bf60ebe2c2b77444","entry":"calc_banned_ngram_tokens","repo":"ml-research/MoRT_NMI","repo_kind":"official","path":"MoRT/mcm_textgeneration/mcm_models.py","file_url":"https://github.com/ml-research/MoRT_NMI/blob/HEAD/MoRT/mcm_textgeneration/mcm_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf60ebe2c2b77444"}},{"code_sha256_prefix":"53df4c9a654254cf","entry":"chunks","repo":"ml-research/MoRT_NMI","repo_kind":"official","path":"MoRT/mort/funcs_mcm.py","file_url":"https://github.com/ml-research/MoRT_NMI/blob/HEAD/MoRT/mort/funcs_mcm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53df4c9a654254cf"}},{"code_sha256_prefix":"a76d890b6f0424e9","entry":"get_actions_temporal","repo":"ml-research/MoRT_NMI","repo_kind":"official","path":"MoRT/mort/dataMoral.py","file_url":"https://github.com/ml-research/MoRT_NMI/blob/HEAD/MoRT/mort/dataMoral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a76d890b6f0424e9"}},{"code_sha256_prefix":"0b42b3fc90731551","entry":"top_k_top_p_filtering","repo":"ml-research/MoRT_NMI","repo_kind":"official","path":"MoRT/mcm_textgeneration/mcm_models.py","file_url":"https://github.com/ml-research/MoRT_NMI/blob/HEAD/MoRT/mcm_textgeneration/mcm_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b42b3fc90731551"}},{"code_sha256_prefix":"022cbd43faa014bc","entry":"top_k_top_p_filtering_dialogpt","repo":"ml-research/MoRT_NMI","repo_kind":"official","path":"MoRT/mcm_textgeneration/mcm_models.py","file_url":"https://github.com/ml-research/MoRT_NMI/blob/HEAD/MoRT/mcm_textgeneration/mcm_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"022cbd43faa014bc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}