{"url":"/sota/multimodal-text-prediction-on-multisubs","task":{"name":"Multimodal Text Prediction","url":"/task/multimodal-text-prediction","note":null},"dataset":{"name":"MultiSubs","url":"/dataset/multisubs"},"category":"Natural Language Processing","categories":["Natural Language Processing"],"category_note":null,"description":"**Multimodal text prediction** is a type of natural language processing that involves predicting the next word or sequence of words in a sentence, given multiple modalities or types of input. In traditional text prediction, the prediction is based solely on the context of the sentence, such as the words that precede the target word. In multimodal text prediction, additional modalities, such as images, audio, or user behavior, are also used to inform the prediction.\r\n\r\nFor example, in a multimodal text prediction system for captioning images, the system may use both the content of the image and the words that have been typed so far to generate the next word in the caption. The image may provide additional context or information about the content of the caption, while the typed words may provide information about the style or tone of the caption.\r\n\r\nMultimodal text prediction can be achieved using a variety of techniques, including deep learning models and statistical models. These models can be trained on large datasets of text and multimodal inputs to learn the relationships between the different types of data and improve the accuracy of the predictions.\r\n\r\nMultimodal text prediction has many applications, including chatbots, virtual assistants, and predictive text input for mobile devices. By incorporating additional modalities into the prediction process, multimodal text prediction systems can provide more accurate and useful predictions, improving the overall user experience.","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["Accuracy","Word similarity"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction); null = not inferred, chart draws points only","by_metric":{"Accuracy":"higher","Word similarity":null}},"counts":{"rows":1,"rows_with_code":1,"rows_with_paper_page":1,"rows_dated":1,"rows_using_additional_data":0},"rows":[{"rank_in_archive_order":1,"model":"9-gram LM with back-off","metrics":{"Accuracy":"30.35","Word similarity":"0.44"},"uses_additional_data":false,"paper_date":"2021-03-02","paper":"/paper/multisubs-a-large-scale-multimodal-and","paper_url":"https://arxiv.org/abs/2103.01910v3","paper_title":"MultiSubs: A Large-scale Multimodal and Multilingual Dataset","code":"https://github.com/josiahwang/multisubs-eval","n_code_links":1,"syntology":null}],"since_archive":{"claim":"Results that newer papers report for their own method, placed here by Syntology. A model pointed at the cell in the paper's own table; the number was read from that cell and checked against this leaderboard's metric, dataset, split and scale; an independent check that saw this leaderboard's other rows and every other leaderboard on the same dataset accepted it. Not reviewed by the paper's authors or by the archive's editors, and not ranked against the archive rows.","extraction_file_present":true,"measurement":{"test_papers":883,"papers_with_output":881,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"measured_on":"2026-09-24","frozen_commit":"0e3de0df94"},"measurement_note":"blind adjudication of accepted entries on a held-out split of archive papers, rules frozen before the test","coverage":{"sentence":"Syntology has checked 6,795 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","complete":false,"papers_newer_than_archive":9581,"papers_checked":6795,"papers_extracted_not_yet_verified":0,"boards_without_verdict":2,"papers_not_yet_extracted":2785},"order":"newest first by month (arXiv date, else the arXiv-id month), then arXiv id descending","columns":[],"entries":[]},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per row: N of M harvested code samples from that row's paper executed on a synthesized fixture; the other M-N are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","rows_with_graph_line":0,"rows_with_any_sample_ran":0,"distinct_papers_with_graph_line":0,"distinct_papers_with_any_sample_ran":0,"samples_over_distinct_papers":{"n_ran":0,"n_unverified":0,"n_samples":0,"n_pointer_only_licence":0,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":0,"n_unverified":0,"n_samples":0,"n_pointer_only_licence":0,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}}}