Today we are releasing an update to our main extension – Web Robots Scraper IDE. This release has a version number 2017.6.20 and has several improvements in UI, proxy settings control, handling hash symbols in URLs.

Version 2017.6.20 RELEASE NOTES

• UI: Robot run statistics is displayed in the same place and no longer “jumping”
• UI: when robot finishes it’s status is a direct link to robot run list on portal. Run link is a direct link to data preview and download on portal.
• setProxy() functionality has been expanded. See documentation for details.
• Bugfix: fixed a bug where subsequent steps with URLs having identical address before # symbol were not loading correctly (Example: http://foobar.com#a and after that go to http://foobar.com#b).
• Other internal engine improvements and bugfixes.

Recently we rolled out an updated version of our main web scraping extension which contains several important updates and new features. This update allows our users to develop and debug robots even faster than before. So what exactly is new?

1. jQuery has been upgraded from version 1.10.2 to 2.2.4
2. done() now can take a milliseconds delay parameter. For example done(1000); will delay step finish by 1 second.
3. New tab Selectors which allows testing selectors inline and generates robot code. Selectors are immediately tested on browser’s active tab so developer can see if they work correctly. Copy code button copies Javascript code to clipboard which can be pasted directly into robot’s step.
   
4. Output tab now displays data in table format. This makes easy to monitor quality of data that is emitted. As robot is running table displays 200 most recently emitted rows.
   

 

We are happy to announce some new features in our robot writing framework. These features are:

• Fork() – split robot into many parallel robots and run them simultaneously. This feature shortens long scraping jobs by parallelising them. Cloud autoscaling handles necessary instance capacity so our customers can run 100s of instances on-demand.
• skipVisited – allows robot to intelligently skip steps to links that were already visited. Avoid data duplication, save robot running time.
• respectRobotsTxt – crawl target sources with compliance to their robots.txt file.

These features are explained in detail and examples added to our framework documentation page.