We’ve written about Twitter a lot here at ProfHacker. One of the best ways to archive tweets, particularly around a particular search term or hashtag is Martin Hawksey’s TAGS archive, which George recently wrote about. But the TAGS archive works best if you already know you want to archive a hashtag or search term; the archives can only go as far back as the Twitter API will allow, which isn’t all the way back.
I ran into this problem when I wanted to go back and collect all of the #FYCchat archives, a Twitter chat that I co-founded and ran for a lot of years (and that has been resurrected!). Because of the continued changes in Twitter’s API, I had lost a number of early chats (and I sometimes forgot to set up a TAGS archive). Twitter changed the interface so that you could infinitely scroll to the first tweet of anything that you search in their search bar, but it’s a long process, and while I can see the tweets, I can’t really do much with them.
Enter Pat Lockley, an Academic Technologist (you can see his work at http://pgogywebstuff.com/) who saw my plea on Twitter for a tool that would scrape a hashtag. He created Twitter Recorder (if you click on this link, it’ll download), a Chrome extension that will create CSV files of the tweets it finds on a Twitter search result page.
He has a pretty good how-to page, but basically: once you install the extension, you go to the Twitter search page, click “Latest”, and the hit plugin icon, and it will prompt you to start. The page will scroll automatically and download a file every time it hits 100 scraped tweets. You can then merge and mine the archive as you see fit!
I’ve been able to pull together an archive for #FYCchat, as well as a couple of other hashtags I hadn’t set up an automatic archive for, and it’s been a huge help. I can see being able to use this, especially if a hashtag becomes a “thing” before anyone noticed.
What is your favorite way to archive tweets?