{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-language-beat-numerical-regression","title":"Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction","arxiv_id":"2403.18447","date":"2024-03-27","proceeding":"CVPR 2024 1","authors":["Inhwan Bae","Junoh Lee","Hae-Gon Jeon"],"abstract":"Language models have demonstrated impressive ability in context understanding and generative performance. Inspired by the recent success of language foundation models, in this paper, we propose LMTraj (Language-based Multimodal Trajectory predictor), which recasts the trajectory prediction task into a sort of question-answering problem. Departing from traditional numerical regression models, which treat the trajectory coordinate sequence as continuous signals, we consider them as discrete signals like text prompts. Specially, we first transform an input space for the trajectory coordinate into the natural language space. Here, the entire time-series trajectories of pedestrians are converted into a text prompt, and scene images are described as text information through image captioning. The transformed numerical and image data are then wrapped into the question-answering template for use in a language model. Next, to guide the language model in understanding and reasoning high-level knowledge, such as scene context and social relationships between pedestrians, we introduce an auxiliary multi-task question and answering. We then train a numerical tokenizer with the prompt data. We encourage the tokenizer to separate the integer and decimal parts well, and leverage it to capture correlations between the consecutive numbers in the language model. Lastly, we train the language model using the numerical tokenizer and all of the question-answer prompts. Here, we propose a beam-search-based most-likely prediction and a temperature-based multimodal prediction to implement both deterministic and stochastic inferences. Applying our LMTraj, we show that the language-based model can be a powerful pedestrian trajectory predictor, and outperforms existing numerical-based predictor methods. Code is publicly available at https://github.com/inhwanbae/LMTrajectory .","url_abs":"https://arxiv.org/abs/2403.18447v1","url_pdf":"https://arxiv.org/pdf/2403.18447v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-language-beat-numerical-regression","repo_url":"https://github.com/inhwanbae/lmtrajectory","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"trajectory-prediction","task_name":"Trajectory Prediction"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.18447","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.18447"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/inhwanbae/LMTrajectory","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":10,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"b54533a035f69b78","entry":"batch_traj2txt","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/converter.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/converter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b54533a035f69b78"}},{"code_sha256_prefix":"6c9d8e97f8976cab","entry":"generate_homography","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/homography.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/homography.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6c9d8e97f8976cab"}},{"code_sha256_prefix":"2612d67abdd01825","entry":"get_exp_config","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/config.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2612d67abdd01825"}},{"code_sha256_prefix":"35ea3f457179e4fd","entry":"image2world","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/homography.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/homography.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"35ea3f457179e4fd"}},{"code_sha256_prefix":"f65d09762517789d","entry":"preprocess_function","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"model/nltoolkit.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/model/nltoolkit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f65d09762517789d"}},{"code_sha256_prefix":"123c2031d3127cc9","entry":"read_file","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/dataloader.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"123c2031d3127cc9"}},{"code_sha256_prefix":"0553d4e98692f456","entry":"text2traj","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/converter.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/converter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0553d4e98692f456"}},{"code_sha256_prefix":"f1fee667db180729","entry":"traj2text","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/converter.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/converter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f1fee667db180729"}},{"code_sha256_prefix":"131c4aa12088ef21","entry":"traj_collate_fn","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/dataloader.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"131c4aa12088ef21"}},{"code_sha256_prefix":"c73cb6c875b73dca","entry":"world2image","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/homography.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/homography.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c73cb6c875b73dca"}},{"code_sha256_prefix":"38b6b384edfe5cc7","entry":"get_dataloader","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"utils/dataloader.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/utils/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"38b6b384edfe5cc7"}},{"code_sha256_prefix":"e2522a4d74f0b769","entry":"image_description_generator","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"model/imagemodel.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/model/imagemodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e2522a4d74f0b769"}},{"code_sha256_prefix":"eff19094fe0ec70e","entry":"postprocess_text","repo":"inhwanbae/LMTrajectory","repo_kind":"official","path":"model/nltoolkit.py","file_url":"https://github.com/inhwanbae/LMTrajectory/blob/HEAD/model/nltoolkit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"eff19094fe0ec70e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}