web scraping

3 Dec, 2015

New Features

By |2019-03-04T16:16:49+02:00December 3rd, 2015|Web Scraping|0 Comments

We are happy to announce some new features in our robot writing framework. These features are:

  • Fork() – split robot into many parallel robots and run them simultaneously. This feature shortens long scraping jobs by parallelising them. Cloud autoscaling handles necessary instance capacity so our customers can run 100s of instances on-demand.
  • skipVisited – allows robot to intelligently skip steps to links that were already visited. Avoid data duplication, save robot running time.
  • respectRobotsTxt – crawl target sources with compliance to their robots.txt file.

These features are explained in detail and examples added to our framework documentation page.

Read More
5 Aug, 2015

Scrape Twitter Followers

By |2019-03-04T16:22:14+02:00August 5th, 2015|Web Scraping|5 Comments

Today we released a simple robot which scrapes follower information from any Twitter user. This will be useful for anyone who is doing competitor analysis or doing research on who follows particular topics. Robot is placed in Demo space on Web Robots portal for anyone to use.

Twitter Scraper Easy Twitter Scraping

How to use it:

  1. Sign in to our portal here.
  2. Download our scraping extension from here.
  3. Find robot named twitter_followers in the dropdown.
  4. Modify start URL to your target’s follower list. For example: https://twitter.com/werobots/followers
  5. Click Run.
  6. Let robot finish it’s job and download data from portal.

In case you want to create your […]

Read More
28 Nov, 2014

Insights from Google event “Always Ahead”

By |2019-03-04T16:34:38+02:00November 28th, 2014|Web Scraping|0 Comments

Yesterday weRobots attended an event sponsored by Google and Enterprise Lithuania. We went there with an expectation of a small event (maybe a workshop) for geeks, but it actually was huge with big crowd, full main hall at LITEXPO conference center and three breakout sessions in the afternoon. The organisers were nice enough to let both weRobots co-founders to enter with only one ticket, plus we were selected for the “red room” break out sessions which was designated for the most advanced crowd.

Google Data Conference

The most interesting part of the conference was a strong presentation by Pawel Matkowski (Google IE). He teased the audience with a promise that we will see something that […]

Read More
16 Oct, 2014

Random Proxy Switcher

By |2019-03-04T16:33:52+02:00October 16th, 2014|Web Scraping|0 Comments

We decided to release a side product of our internal systems. It is a free Chrome extension which allows users to browse the web while randomly changing proxy every minute. Of course the catch is that we do not give out proxy servers themselves, but there are plenty services that provide them. Or anyone can launch their own for free by following this tutorial.

proxy extension

Read More
13 Oct, 2014

How to Setup a Free Proxy Server on Amazon EC2

By |2019-03-04T16:33:18+02:00October 13th, 2014|Web Scraping|1 Comment

Open Amazon Web Services (AWS) Account

Go to the AWS Portal and sign up. Credit card will be required, but Amazon will not charge anything. Amazon will also ask for your phone number and verify it. Amazon EC2 offers free Micro Instances which are good enough for proxy server setup. They remain free for the first year of AWS usage.

Creating an EC2 Instance

Once you have the account login to AWS Management Console and from the EC2 Dashboard click the Launch Instance. Follow the steps and launch the instance of Ubuntu Server which is marked as Free tier eligible. Make sure you download SSH key file (.pem) as it will be needed to connect to server.

Free tier eligible Ubuntu Server

Read More
13 Oct, 2014

Smart CSS Selectors

By |2019-03-04T16:37:21+02:00October 13th, 2014|Web Scraping|0 Comments

Our web scraping business requires that we develop scraper robots quickly and efficiently. We can offer competitive pricing only if we are most efficient at creating robots for each source. Old saying “time is money” means a lot here and we always look for ways to do things better and faster.

In scraper development process everyone uses either Xpaths or CSS selectors to parse DOM for data to be extracted or links to crawl through. One can inspect DOM elements (via Google Chrome) for classes, IDs or other attributes. Then solve a small or big puzzle to write a selector. It requires knowing powerful CSS Selector syntax, detective work inside DOM, and some trial and error.

There is a nifty tool […]

Read More