Manage Taxonomy Links through Portal

Robots often need to have a list of taxonomy URLs as the starting point for the workflow. Web Robots portal has a feature to manage such links centrally without altering robot code itself. This feature can be accessed through “Taxo” button under any robot.

Taxonomy URLs and additional parameters can be edited in Excel like sheet (Note: right-click on the sheet has insert rows, remove rows functionality). URL column is for the link itself. A, B and C columns are for any optional addition information. Crawl column allows to turn on or off the particular row (useful for debugging).

Once taxonomy is added to portal and saved, it is available inside robot through $taxo variable. Taxonomy from the example above will be available inside robot as the following array:

[{
	"url": "https://www.amazon.com/s?bbn=16225007011&rh=n:16225007011,n:1292110011&dc",
	"a": "Storage",
	"b": null,
	"c": null
}, {
	"url": "https://www.amazon.com/s?bbn=16225007011&rh=n:16225007011,n:1292115011",
	"a": "Monitors",
	"b": null,
	"c": null
}]

A typical $taxo usage example in the robot’s start step could look like this:

steps.start = function() {
	$taxo.forEach(category =>
		next(category.url, "enter_category", category.a)
	);
	done();
};