{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/twawler-a-lightweight-twitter-crawler","title":"twAwler: A lightweight twitter crawler","arxiv_id":"1804.07748","date":"2018-04-20","proceeding":null,"authors":["Polyvios Pratikakis"],"abstract":"This paper presents twAwler, a lightweight twitter crawler that targets\nlanguage-specific communities of users. twAwler takes advantage of multiple\nendpoints of the twitter API to explore user relations and quickly recognize\nusers belonging to the targetted set. It performs a complete crawl for all\nusers, discovering many standard user relations, including the retweet graph,\nmention graph, reply graph, quote graph, follow graph, etc. twAwler respects\nall twitter policies and rate limits, while able to monitor large communities\nof active users.\n  twAwler was used between August 2016 and March 2018 to generate an extensive\ndataset of close to all Greek-speaking twitter accounts (about 330 thousand)\nand their tweets and relations. In total, the crawler has gathered 750 million\ntweets of which 424 million are in Greek; 750 million follow relations;\ninformation about 300 thousand lists, their members (119 million member\nrelations) and subscribers (27 thousand subscription relations); 705 thousand\ntrending topics; information on 52 million users in total of which 292 thousand\nhave been since suspended, 141 thousand have deleted their account, and 3.5\nmillion are protected and cannot be crawled. twAwler mines the collected tweets\nfor the retweet, quote, reply, and mention graphs, which, in addition to the\nfollow relation crawled, offer vast opportunities for analysis and further\nresearch.","url_abs":"http://arxiv.org/abs/1804.07748v1","url_pdf":"http://arxiv.org/pdf/1804.07748v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"twawler-a-lightweight-twitter-crawler","repo_url":"https://github.com/polyvios/twAwler","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}