Scrape Twitter Followers

Scrape Twitter Followers

Today we released a simple robot which scrapes follower information from any Twitter user. This will be useful for anyone who is doing competitor analysis or doing research on who follows particular topics. Robot is placed in Demo space on Web Robots portal for anyone to use.

Twitter Scraper

Easy Twitter Scraping

How to use it:

  1. Sign in to our portal here.
  2. Download our scraping extension from here.
  3. Find robot named twitter_followers in the dropdown.
  4. Modify start URL to your target’s follower list. For example: https://twitter.com/werobots/followers
  5. Click Run.
  6. Let robot finish it’s job and download data from portal.

In case you want to create your own version of such robot robot, here is it’s full code:

// start page must be set to follower page of twitter user
// Example: https://twitter.com/csc/followers

steps.start = function(bottomCount) {
    
    if (bottomCount === undefined) bottomCount = 0;
    
    if (checkBottom()) {
        
        bottomCount++;
        
        if (bottomCount > 10) {
            next("", "start", bottomCount);
        } else {
            next("", "scrapeFollowers");
        }
        
    } else {
        next("", "start", 0);
    }
    
    $('html, body').scrollTop( $(document).height());
    setTimeout(done, 1000);
};

steps.scrapeFollowers = function() {
    var profiles = [];
    
    $(".ProfileCard").each(function(i,v) {
        var profile = {
            id : v.dataset.userId,
            name : $("a.ProfileNameTruncated-link", v).text().trim(),
            screen_name : v.dataset.screenName,
            link : $("a.ProfileNameTruncated-link", v)[0].href,
            bio : $(".ProfileCard-bio", v).text().trim(),
            avatar : $("img.ProfileCard-avatarImage", v)[0].src
        };
        
        profiles.push(profile);
    });
    
    emit(document.URL.split("/").slice(-2,-1) +"_followers", profiles);
    done();
};

function checkBottom() {
       if($(window).scrollTop() + $(window).height() == $(document).height()) {
           return true;
       } else {
           return false;
       }
}
By | 2017-01-09T16:20:31+00:00 August 5th, 2015|Uncategorized|4 Comments

4 Comments

  1. michael November 2, 2015 at 10:56 pm - Reply

    Hi there,

    it does not work for me, it opens a new window.

    Log:

    2015-11-02T20:55:53.833Z UI: INFO: next: execute: steps.start();
    2015-11-02T20:55:53.834Z UI: INFO: next: openUrl: https://twitter.com/showpad/followers
    2015-11-02T20:55:53.837Z UI: INFO: Run started: twitter_followers_2015-11-02T20_55_53_833Z
    2015-11-02T20:55:54.081Z UI: DEBUG: Run save ok: undefined
    2015-11-02T20:55:54.084Z UI: DEBUG: Run status sent ok: [object Object]
    2015-11-02T20:56:03.948Z UI: DEBUG: Run status sent ok: undefined

    Thanks!

  2. robin December 22, 2015 at 2:38 am - Reply

    This works, but only outputs 6 followers out of 10k. What is wrong

    • nicerobot December 29, 2015 at 5:03 pm - Reply

      It may be flaky depending on how responsive Twitter’s infinite scroll data loading is.

      You may want to play with variables on these two lines:
      if (bottomCount < 5) { <-- increase from 5 to let's say 10 setTimeout(done, 1000); <-- change 1000 to 3000

Leave A Comment