Today we released a simple robot which scrapes follower information from any Twitter user. This will be useful for anyone who is doing competitor analysis or doing research on who follows particular topics. Robot is placed in Demo space on Web Robots portal for anyone to use.

Twitter Scraper

Easy Twitter Scraping

How to use it:

  1. Sign in to our portal here.
  2. Download our scraping extension from here.
  3. Find robot named twitter_followers in the dropdown.
  4. Modify start URL to your target’s follower list. For example: https://twitter.com/werobots/followers
  5. Click Run.
  6. Let robot finish it’s job and download data from portal.

In case you want to create your own version of such robot robot, here is it’s full code:

// start page must be set to follower page of twitter user
// Example: https://twitter.com/csc/followers

steps.start = function(bottomCount) {
    
    if (bottomCount === undefined) bottomCount = 0;
    
    if (checkBottom()) {
        
        bottomCount++;
        
        if (bottomCount > 10) {
            next("", "start", bottomCount);
        } else {
            next("", "scrapeFollowers");
        }
        
    } else {
        next("", "start", 0);
    }
    
    $('html, body').scrollTop( $(document).height());
    setTimeout(done, 1000);
};

steps.scrapeFollowers = function() {
    var profiles = [];
    
    $(".ProfileCard").each(function(i,v) {
        var profile = {
            id : v.dataset.userId,
            name : $("a.ProfileNameTruncated-link", v).text().trim(),
            screen_name : v.dataset.screenName,
            link : $("a.ProfileNameTruncated-link", v)[0].href,
            bio : $(".ProfileCard-bio", v).text().trim(),
            avatar : $("img.ProfileCard-avatarImage", v)[0].src
        };
        
        profiles.push(profile);
    });
    
    emit(document.URL.split("/").slice(-2,-1) +"_followers", profiles);
    done();
};

function checkBottom() {
       if($(window).scrollTop() + $(window).height() == $(document).height()) {
           return true;
       } else {
           return false;
       }
}