Managing Similar Robots with Tags

We have introduced robot TAG feature to our platform. With tags you can create multiple scheduled instances of the same robot, and automatically pass different tags to them as parameters.

TAG Concept

Robot with a TAG shares the code with its parent robot and is considered a child robot. Child robot has an independent schedule, QC metrics, and is displayed as a separate robot in the portal runs list.

The TAG is a short string (up to 20 characters) which will be passed as part of params object to the start step of the robot. TAG can consist of: Latin upper and lower case letters, numbers and characters “-” or “_”. Other characters are forbidden.

Use cases for TAG

TAGs are useful when separate websites share identical layout but need to be scraped separately. For example a retailer might have similar websites in multiple countries and the country code could be passed as a TAG.

They can also be used to direct specific robot behavior for specific schedules. For example TAGs could be used to set up a different robot behavior for the weekends, or perform a deeper more detailed scrape once a week or month.

Walkthrough

In the Edit Robot page multiple independent schedules can be created by pressing the “Add”  button. They will use the same robot code, but practically function as separate robots. Each schedule will create its own runs and QC and will be identified by the assigned TAG. The TAG string can be accessed in the start step from a passed params object.  This provides additional functionality as different schedules of a single robot can use the TAG to direct the behavior of a robot appropriately for different use cases.

TAG schedule creation menu:

Robots with TAGs in portal robots page:

Example of TAG used in a robot:

// Defaulting params to an empty object in case a TAG is not passed
steps.start = function( params = {} ){
    let tag = params.tag;
    if ( tag === 'ADVANCED') {
        next('', 'advancedScrape');
    } else if( tag === 'MONTHLY') {
        next('', 'monthlyScrape');
    } else {
        next('', 'defaultScrape');
    }
    done();
}