{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-01025","title":"How Much Do RF Drone Benchmarks Overstate? A Controlled Study and Theory of Data Leakage in UAV Signal Identification","arxiv_id":"2607.01025","date":"2026-07-01","proceeding":null,"authors":["David Shulman"],"abstract":"Radio-frequency (RF) sensing is a central modality for counter-unmanned-aerial-system (counter-UAS) defence because it exploits the control, telemetry, and video links between a drone and its operator. Reported accuracies for RF-based drone detection and identification are often very high, but many are obtained using cross-validation that splits a small number of continuous recordings into short segments. This can place near-duplicate slices of the same recording in both training and test partitions, creating data leakage. We study this leakage problem through theory and measurement. We formalise the optimism of segment-level cross-validation and show, using Cover's function-counting theorem, that a classifier can exactly memorise the recording-to-label map when the number of independent recordings, R, is small relative to the feature dimension, d. In particular, this can occur when 2R is less than or approximately equal to d. Under these conditions, naive accuracy approaches 1, and the inflation gap approaches 1 - ACC*, where ACC* is the Bayes accuracy. The inflation eases only once R grows beyond this separability threshold. A controlled synthetic experiment with 10 seeds confirms the predicted curves: naive balanced accuracy rises from the Bayes level toward 1.0 as recording-specific nuisance variation grows, while honest recording-grouped evaluation declines to chance, with a gap reaching about 0.5. On the public DroneRF dataset, pooled leave-one-recording-out cross-validation shows drone type identification, AR versus Bebop, collapsing from a naive macro-F1 of 0.74 to 0.46, the two-class chance level. A leakage-pathway ablation attributes essentially all of the inflation to segment-level leakage.","url_abs":"https://arxiv.org/abs/2607.01025","url_pdf":"https://arxiv.org/pdf/2607.01025","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.01025","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.01025"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/shulm/spectrahawk","reach":null}],"summary":{"ran":9,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e71904d9dcc35dc1","entry":"build_baseline","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/models.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e71904d9dcc35dc1"}},{"code_sha256_prefix":"2683b118856fca64","entry":"feature_vector","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/features.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2683b118856fca64"}},{"code_sha256_prefix":"4fd8b3ef91e1dede","entry":"features_matrix","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/models.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fd8b3ef91e1dede"}},{"code_sha256_prefix":"1e4aabfec7a90d68","entry":"generate_background_rf","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/synth_rf.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/synth_rf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e4aabfec7a90d68"}},{"code_sha256_prefix":"f0752986593a253c","entry":"generate_drone_rf","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/synth_rf.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/synth_rf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0752986593a253c"}},{"code_sha256_prefix":"c49486fb4480ce9f","entry":"make_synthetic_dataset","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/synth_rf.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/synth_rf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c49486fb4480ce9f"}},{"code_sha256_prefix":"a6d64f26583660b9","entry":"pd_at_pfa","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/evaluate.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6d64f26583660b9"}},{"code_sha256_prefix":"0a3617ff521128e9","entry":"plot_roc","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/evaluate.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a3617ff521128e9"}},{"code_sha256_prefix":"df2142aa1a2b1881","entry":"psd_rf","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/features.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df2142aa1a2b1881"}},{"code_sha256_prefix":"aa2f72226ceeacb4","entry":"load_dronerf","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/data_io.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/data_io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa2f72226ceeacb4"}},{"code_sha256_prefix":"029adcf4a737a323","entry":"load_dronerf_features","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/data_io.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/data_io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"029adcf4a737a323"}},{"code_sha256_prefix":"4a685e67520977a3","entry":"stft_rf","repo":"shulm/spectrahawk","repo_kind":"found_in_text","path":"spectrahawk/features.py","file_url":"https://github.com/shulm/spectrahawk/blob/HEAD/spectrahawk/features.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a685e67520977a3"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}