Today we released a simple robot which scrapes follower information from any Twitter user. This will be useful for anyone who is doing competitor analysis or doing research on who follows particular topics. Robot is placed in Demo space on Web Robots portal for anyone to use.
How to use it:
- Sign in to our portal here.
- Download our scraping extension from here.
- Find robot named twitter_followers in the dropdown.
- Modify start URL to your target’s follower list. For example: https://twitter.com/werobots/followers
- Click Run.
- Let robot finish it’s job and download data from portal.
In case you want to create your own version of such robot robot, here is it’s full code:
// start page must be set to follower page of twitter user // Example: https://twitter.com/csc/followers steps.start = function(bottomCount) { if (bottomCount === undefined) bottomCount = 0; if (checkBottom()) { bottomCount++; if (bottomCount > 10) { next("", "start", bottomCount); } else { next("", "scrapeFollowers"); } } else { next("", "start", 0); } $('html, body').scrollTop( $(document).height()); setTimeout(done, 1000); }; steps.scrapeFollowers = function() { var profiles = []; $(".ProfileCard").each(function(i,v) { var profile = { id : v.dataset.userId, name : $("a.ProfileNameTruncated-link", v).text().trim(), screen_name : v.dataset.screenName, link : $("a.ProfileNameTruncated-link", v)[0].href, bio : $(".ProfileCard-bio", v).text().trim(), avatar : $("img.ProfileCard-avatarImage", v)[0].src }; profiles.push(profile); }); emit(document.URL.split("/").slice(-2,-1) +"_followers", profiles); done(); }; function checkBottom() { if($(window).scrollTop() + $(window).height() == $(document).height()) { return true; } else { return false; } }
Hi there,
it does not work for me, it opens a new window.
Log:
2015-11-02T20:55:53.833Z UI: INFO: next: execute: steps.start();
2015-11-02T20:55:53.834Z UI: INFO: next: openUrl: https://twitter.com/showpad/followers
2015-11-02T20:55:53.837Z UI: INFO: Run started: twitter_followers_2015-11-02T20_55_53_833Z
2015-11-02T20:55:54.081Z UI: DEBUG: Run save ok: undefined
2015-11-02T20:55:54.084Z UI: DEBUG: Run status sent ok: [object Object]
2015-11-02T20:56:03.948Z UI: DEBUG: Run status sent ok: undefined
Thanks!
Someone was playing around with the robot code. Remember, it is public – anyone can use it.
Robot works now – I just ran it on your link. Result with about 2.5k followers is here: http://portal.webrobots.io/runs/twitter_followers_2015-12-03T13_50_01_993Z
Web Robots Team
This works, but only outputs 6 followers out of 10k. What is wrong
It may be flaky depending on how responsive Twitter’s infinite scroll data loading is.
You may want to play with variables on these two lines:
if (bottomCount < 5) { <-- increase from 5 to let's say 10 setTimeout(done, 1000); <-- change 1000 to 3000
Great tool :-)
Is there a way to extend this scraper in order to automatically get “followers of followers”?
I would like to generate ego-networks starting from one account (=start URL) and including this accounts followers and their followers and so on (up to a predefined number of steps).
The output should then also include a collumn for the user currently in focus (in order to reconstruct the social network).