{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/taylor-videos-for-action-recognition","title":"Taylor Videos for Action Recognition","arxiv_id":"2402.03019","date":"2024-02-05","proceeding":null,"authors":["Lei Wang","Xiuyuan Yuan","Tom Gedeon","Liang Zheng"],"abstract":"Effectively extracting motions from video is a critical and long-standing problem for action recognition. This problem is very challenging because motions (i) do not have an explicit form, (ii) have various concepts such as displacement, velocity, and acceleration, and (iii) often contain noise caused by unstable pixels. Addressing these challenges, we propose the Taylor video, a new video format that highlights the dominate motions (e.g., a waving hand) in each of its frames named the Taylor frame. Taylor video is named after Taylor series, which approximates a function at a given point using important terms. In the scenario of videos, we define an implicit motion-extraction function which aims to extract motions from video temporal block. In this block, using the frames, the difference frames, and higher-order difference frames, we perform Taylor expansion to approximate this function at the starting frame. We show the summation of the higher-order terms in the Taylor series gives us dominant motion patterns, where static objects, small and unstable motions are removed. Experimentally we show that Taylor videos are effective inputs to popular architectures including 2D CNNs, 3D CNNs, and transformers. When used individually, Taylor videos yield competitive action recognition accuracy compared to RGB videos and optical flow. When fused with RGB or optical flow videos, further accuracy improvement is achieved. Additionally, we apply Taylor video computation to human skeleton sequences, resulting in Taylor skeleton sequences that outperform the use of original skeletons for skeleton-based action recognition.","url_abs":"https://arxiv.org/abs/2402.03019v4","url_pdf":"https://arxiv.org/pdf/2402.03019v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"taylor-videos-for-action-recognition","repo_url":"https://github.com/leiwangr/video-ar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.03019","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.03019"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LeiWangR/video-ar","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leiwangr/video-ar","reach":{"status":"ok"}}],"summary":{"ran":8,"ran_draft_wrong":2,"ran_violates":2,"unverified":3},"by_repo_kind":{"official":{"samples":15,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"a7af8c079621d2b7","entry":"ConvTemporalGraphical","repo":"leiwangr/video-ar","repo_kind":"official","path":"backbone/net/st_gcn.py","file_url":"https://github.com/leiwangr/video-ar/blob/HEAD/backbone/net/st_gcn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a7af8c079621d2b7"}},{"code_sha256_prefix":"67d47de5b5e3cd38","entry":"Graph","repo":"leiwangr/video-ar","repo_kind":"official","path":"backbone/net/st_gcn.py","file_url":"https://github.com/leiwangr/video-ar/blob/HEAD/backbone/net/st_gcn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67d47de5b5e3cd38"}},{"code_sha256_prefix":"8b14974e8aa7320f","entry":"downsample","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/feeder/tools.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/feeder/tools.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b14974e8aa7320f"}},{"code_sha256_prefix":"c257ac36cba643a2","entry":"get_hop_distance","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/net/utils/graph.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/net/utils/graph.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c257ac36cba643a2"}},{"code_sha256_prefix":"4e946d7c92be9d41","entry":"import_class","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/torchlight/torchlight/io.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/torchlight/torchlight/io.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e946d7c92be9d41"}},{"code_sha256_prefix":"83ed4b272b1e38d0","entry":"ngpu","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/torchlight/torchlight/gpu.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/torchlight/torchlight/gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83ed4b272b1e38d0"}},{"code_sha256_prefix":"8158f4435afa26b7","entry":"normalize_digraph","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/net/utils/graph.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/net/utils/graph.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8158f4435afa26b7"}},{"code_sha256_prefix":"f3dfe2628c96a73d","entry":"normalize_undigraph","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/net/utils/graph.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/net/utils/graph.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3dfe2628c96a73d"}},{"code_sha256_prefix":"1b4ec459a91bb8fd","entry":"st_gcn","repo":"leiwangr/video-ar","repo_kind":"official","path":"backbone/net/st_gcn.py","file_url":"https://github.com/leiwangr/video-ar/blob/HEAD/backbone/net/st_gcn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b4ec459a91bb8fd"}},{"code_sha256_prefix":"f017532fc389cbfe","entry":"str2bool","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/torchlight/torchlight/io.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/torchlight/torchlight/io.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f017532fc389cbfe"}},{"code_sha256_prefix":"9f4ed8535e8abbeb","entry":"str2dict","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/torchlight/torchlight/io.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/torchlight/torchlight/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f4ed8535e8abbeb"}},{"code_sha256_prefix":"621e08e84928c5c3","entry":"visible_gpu","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/torchlight/torchlight/gpu.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/torchlight/torchlight/gpu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"621e08e84928c5c3"}},{"code_sha256_prefix":"ee1cc73a20e6085c","entry":"Model","repo":"leiwangr/video-ar","repo_kind":"official","path":"backbone/net/st_gcn.py","file_url":"https://github.com/leiwangr/video-ar/blob/HEAD/backbone/net/st_gcn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee1cc73a20e6085c"}},{"code_sha256_prefix":"8c9b61ef46b62157","entry":"mean_subtractor","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/feeder/tools.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/feeder/tools.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c9b61ef46b62157"}},{"code_sha256_prefix":"2f7f389f0584c97e","entry":"temporal_slice","repo":"LeiWangR/video-ar","repo_kind":"official","path":"backbone/feeder/tools.py","file_url":"https://github.com/LeiWangR/video-ar/blob/HEAD/backbone/feeder/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f7f389f0584c97e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}