{"url":"/method/electric","slug":"electric","name":"Electric","full_name":"Electric","full_name_withheld":false,"description_markdown":"**Electric** is an energy-based cloze model for representation learning over text. Like BERT, it is a conditional generative model of tokens given their contexts. However, Electric does not use masking or output a full distribution over tokens that could occur in a context. Instead, it assigns a scalar energy score to each input token indicating how likely it is given its context.\r\n\r\nSpecifically, like BERT, Electric also models $p\\_{\\text {data }}\\left(x\\_{t} \\mid \\mathbf{x}\\_{\\backslash t}\\right)$, but does not use masking or a softmax layer. Electric first maps the unmasked input $\\mathbf{x}=\\left[x\\_{1}, \\ldots, x\\_{n}\\right]$ into contextualized vector representations $\\mathbf{h}(\\mathbf{x})=\\left[\\mathbf{h}\\_{1}, \\ldots, \\mathbf{h}\\_{n}\\right]$ using a transformer network. The model assigns a given position $t$ an energy score\r\n\r\n$$\r\nE(\\mathbf{x})\\_{t}=\\mathbf{w}^{T} \\mathbf{h}(\\mathbf{x})\\_{t}\r\n$$\r\n\r\nusing a learned weight vector $w$. The energy function defines a distribution over the possible tokens at position $t$ as\r\n\r\n$$\r\np\\_{\\theta}\\left(x\\_{t} \\mid \\mathbf{x}_{\\backslash t}\\right)=\\exp \\left(-E(\\mathbf{x})\\_{t}\\right) / Z\\left(\\mathbf{x}\\_{\\backslash t}\\right) \r\n$$\r\n\r\n$$\r\n=\\frac{\\exp \\left(-E(\\mathbf{x})\\_{t}\\right)}{\\sum\\_{x^{\\prime} \\in \\mathcal{V}} \\exp \\left(-E\\left(\\operatorname{REPLACE}\\left(\\mathbf{x}, t, x^{\\prime}\\right)\\right)\\_{t}\\right)}\r\n$$\r\n\r\nwhere $\\text{REPLACE}\\left(\\mathbf{x}, t, x^{\\prime}\\right)$ denotes replacing the token at position $t$ with $x^{\\prime}$ and $\\mathcal{V}$ is the vocabulary, in practice usually word pieces. Unlike with BERT, which produces the probabilities for all possible tokens $x^{\\prime}$ using a softmax layer, a candidate $x^{\\prime}$ is passed in as input to the transformer. As a result, computing $p_{\\theta}$ is prohibitively expensive because the partition function $Z\\_{\\theta}\\left(\\mathbf{x}\\_{\\backslash t}\\right)$ requires running the transformer $|\\mathcal{V}|$ times; unlike most EBMs, the intractability of $Z\\_{\\theta}(\\mathbf{x} \\backslash t)$ is more due to the expensive scoring function rather than having a large sample space.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Pre-Training Transformers as Energy-Based Cloze Models","paper":"/paper/pre-training-transformers-as-energy-based-1","first_author":"Kevin Clark","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/pre-training-transformers-as-energy-based-1"},"source":{"url":"https://arxiv.org/abs/2012.08561v1","title":"Pre-Training Transformers as Energy-Based Cloze Models","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Autoencoding Transformers","url":"/methods/category/autoencoding-transformers","pwc_aliases":[]},{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]},{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Language Models","url":"/methods/category/language-models","pwc_aliases":[]}],"n_papers_tagged":167,"archive_num_papers":167,"papers_newest_first":[{"paper":null,"title":"Integrated Switched Capacitor Array and Synchronous Charge Extraction with Adaptive Hybrid MPPT for Piezoelectric Harvesters","date":"2025-07-16","arxiv_id":"2507.12163","n_code_links":0,"syntology":null},{"paper":null,"title":"Smart Ride and Delivery Services with Electric Vehicles: Leveraging Bidirectional Charging for Profit Optimisation","date":"2025-06-25","arxiv_id":"2506.20401","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep reinforcement learning-based joint real-time energy scheduling for green buildings with heterogeneous battery energy storage devices","date":"2025-06-07","arxiv_id":"2506.06824","n_code_links":0,"syntology":null},{"paper":null,"title":"Agent-Based Decentralized Energy Management of EV Charging Station with Solar Photovoltaics via Multi-Agent Reinforcement Learning","date":"2025-05-24","arxiv_id":"2505.18750","n_code_links":0,"syntology":null},{"paper":null,"title":"Control of Renewable Energy Communities using AI and Real-World Data","date":"2025-05-22","arxiv_id":"2505.17321","n_code_links":0,"syntology":null},{"paper":null,"title":"Electric Bus Charging Schedules Relying on Real Data-Driven Targets Based on Hierarchical Deep Reinforcement Learning","date":"2025-05-15","arxiv_id":"2505.10262","n_code_links":0,"syntology":null},{"paper":null,"title":"Optimizing Electric Bus Charging Scheduling with Uncertainties Using Hierarchical Deep Reinforcement Learning","date":"2025-05-15","arxiv_id":"2505.10296","n_code_links":0,"syntology":null},{"paper":null,"title":"Leveraging Surplus Electricity: Profitability of Bitcoin Mining as a National Strategy in South Korea","date":"2025-05-01","arxiv_id":"2505.00303","n_code_links":0,"syntology":null},{"paper":null,"title":"LLM-Enabled EV Charging Stations Recommendation","date":"2025-04-29","arxiv_id":"2505.01447","n_code_links":0,"syntology":null},{"paper":null,"title":"Sensitivity Analysis of State Space Models for Scrap Composition Estimation in EAF and BOF","date":"2025-04-15","arxiv_id":"2504.11319","n_code_links":0,"syntology":null},{"paper":null,"title":"Vehicle Dynamics Control for Simultaneous Optimization of Tire Emissions and Performance in EVs","date":"2025-04-14","arxiv_id":"2504.10709","n_code_links":0,"syntology":null},{"paper":null,"title":"Modeling Scrap Composition in Electric Arc and Basic Oxygen Furnaces","date":"2025-04-13","arxiv_id":"2504.09382","n_code_links":0,"syntology":null},{"paper":null,"title":"Embedding Reliability Verification Constraints into Generation Expansion Planning","date":"2025-04-06","arxiv_id":"2504.07131","n_code_links":0,"syntology":null},{"paper":null,"title":"Methodology for Detecting Energy Anomalies due to Multi-Replay Attacks on Electric Vehicle Charging Infrastructure","date":"2025-04-01","arxiv_id":"2504.00319","n_code_links":0,"syntology":null},{"paper":null,"title":"EVOLVE: a Value-Added Services Platform for Electric Vehicle Charging Stations","date":"2025-03-24","arxiv_id":"2503.18687","n_code_links":0,"syntology":null},{"paper":"/paper/pruning-based-tinyml-optimization-of-machine","title":"Pruning-Based TinyML Optimization of Machine Learning Models for Anomaly Detection in Electric Vehicle Charging Infrastructure","date":"2025-03-19","arxiv_id":"2503.14799","n_code_links":1,"syntology":null},{"paper":null,"title":"Large Neighborhood Search and Bitmask Dynamic Programming for Wireless Mobile Charging Electric Vehicle Routing Problems in Medical Transportation","date":"2025-03-11","arxiv_id":"2503.08752","n_code_links":0,"syntology":null},{"paper":null,"title":"The Untapped Potential of Smart Charging: How EV Owners Can Save Money and Reduce Emissions Without Behavioral Change","date":"2025-03-05","arxiv_id":"2503.03167","n_code_links":0,"syntology":null},{"paper":null,"title":"A Kolmogorov-Arnold Network for Explainable Detection of Cyberattacks on EV Chargers","date":"2025-03-04","arxiv_id":"2503.02281","n_code_links":0,"syntology":null},{"paper":null,"title":"H-FLTN: A Privacy-Preserving Hierarchical Framework for Electric Vehicle Spatio-Temporal Charge Prediction","date":"2025-02-25","arxiv_id":"2502.18697","n_code_links":0,"syntology":null},{"paper":null,"title":"Impact Analysis of Utility-Scale Energy Storage on the ERCOT Grid in Reducing Renewable Generation Curtailments and Emissions","date":"2025-02-21","arxiv_id":"2502.15958","n_code_links":0,"syntology":null},{"paper":null,"title":"TSS GAZ PTP: Towards Improving Gumbel AlphaZero with Two-stage Self-play for Multi-constrained Electric Vehicle Routing Problems","date":"2025-02-17","arxiv_id":"2502.15777","n_code_links":0,"syntology":null},{"paper":null,"title":"InterGridNet: An Electric Network Frequency Approach for Audio Source Location Classification Using Convolutional Neural Networks","date":"2025-02-14","arxiv_id":"2502.10011","n_code_links":0,"syntology":null},{"paper":null,"title":"Impact of Lead Time on Aggregate EV Flexibility for Congestion Management Services","date":"2025-01-27","arxiv_id":"2501.15946","n_code_links":0,"syntology":null},{"paper":null,"title":"Intelligent Electric Power Steering: Artificial Intelligence Integration Enhances Vehicle Safety and Performance","date":"2024-12-11","arxiv_id":"2412.08133","n_code_links":0,"syntology":null},{"paper":"/paper/evaluating-the-relationship-of-ev-charging","title":"Evaluating the Relationship of EV Charging Station on the Uptake of Electric Vehicles -- Implication of the NEVI Formula Program","date":"2024-12-04","arxiv_id":"2412.03608","n_code_links":1,"syntology":null},{"paper":"/paper/a-bandpass-twin-t-active-filter-used-in-the","title":"A Bandpass Twin-T Active Filter Used in the Buchla 200 Electric Music Box Synthesizer","date":"2024-11-18","arxiv_id":"2411.11358","n_code_links":1,"syntology":null},{"paper":null,"title":"Knowledge-enhanced Transformer for Multivariate Long Sequence Time-series Forecasting","date":"2024-11-17","arxiv_id":"2411.11046","n_code_links":0,"syntology":null},{"paper":null,"title":"Integrated Power and Thermal Management for Enhancing Energy Efficiency and Battery Life in Connected and Automated Electric Vehicles","date":"2024-11-08","arxiv_id":"2411.05298","n_code_links":0,"syntology":null},{"paper":null,"title":"A Scientific Machine Learning Approach for Predicting and Forecasting Battery Degradation in Electric Vehicles","date":"2024-10-18","arxiv_id":"2410.14347","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/management","name":"Management","papers":27},{"task":"/task/scheduling","name":"Scheduling","papers":13},{"task":"/task/energy-management","name":"energy management","papers":13},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":11},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":9},{"task":"/task/time-series-1","name":"Time Series","papers":7},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":6},{"task":"/task/anomaly-detection","name":"Anomaly Detection","papers":5},{"task":"/task/model-predictive-control","name":"Model Predictive Control","papers":4},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":4},{"task":"/task/arc","name":"ARC","papers":3},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":3},{"task":"/task/combinatorial-optimization","name":"Combinatorial Optimization","papers":3},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":3},{"task":"/task/decision-making","name":"Decision Making","papers":3},{"task":"/task/intrusion-detection","name":"Intrusion Detection","papers":3},{"task":"/task/load-forecasting","name":"Load Forecasting","papers":3},{"task":"/task/time-series","name":"Time Series Analysis","papers":3},{"task":"/task/energy-trading","name":"energy trading","papers":3},{"task":"/task/regression-1","name":"regression","papers":3}],"tasks_shown":20,"n_tasks":83,"usage_by_year":[{"year":"2020","papers":1},{"year":"2021","papers":25},{"year":"2022","papers":32},{"year":"2023","papers":42},{"year":"2024","papers":43},{"year":"2025","papers":24}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/electric"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}