Extractor SDK/CLI Release Notes

Release v1.3.0 - April 1, 2021

  • Mixed Proxy CLI support (Documentation Reference)

  • generate2FAToken CLI support (Documentation Reference)

  • Removal of Policy Support

    • As of this version, policies are no longer created or updated as part of deploying an Extractor from the CLI. Policies can still be managed via the SAAS Platform.

  • Limiting Residential Proxy Usage

    • If a user tries to deploy an extractor from the CLI, and the proxy type is explicitly set to RESIDENTIAL in the extractor file, it will fail. In order to successfully deploy an extractor with this setting requires the proxy type to be removed from the CLI code, and added through the SAAS Platform with permission from an Organization or Import.io Administrator.

    • Proxy type can no longer be set for an extractor via the CLI. Existing extractors will no longer include proxy type as part of an update, and any proxy type changes need to be set through the SAAS Platform. A warning message will appear to users who have proxy type set.

Release v1.2.3

Release v1.1.0 - November 24, 2020

extractionGlobals can be added to a robot.yaml or extractor.yaml. Properties in extractionGlobals will be applied to all extraction dependencies in the extractor.

Supported properties:

  • screenCapture: boolean, if true will take a screenshot when data is extracted

  • screenCaptureType: 'png'|'pdf' default: png

Release v1.0.0 - October 28, 2020

Workbench Source Deployment

New import-io source:deploy commands which allow deploying Sources directly to Workbench.

Release v0.9.0 - August 26, 2020

  • Support for checkAuthentication entry points in robot.yaml

    • If this value is present checkAuthInteractions will be populated on runtime configuration for extractor

    • At runtime this sequence will execute before each input, checking that the user is logged in. If error is thrown will force log in sequence to be re run

    • extractor:run:local and extractor:run:remote will automatically run this when running an extractor.yaml file locally

Release v0.8.0 - August 25, 2020

  • New APIs

  • Bug fix for extractor:deploy command failing eslint

Release v0.7.0 - August 20, 2020

  • new reportWrongGeocoding API available Docs here

    • Report that the site is reporting a different country from expected

  • Extractors now enable network debugger by default, to ensure all requests are captured.

Release v0.6.0 - August 10, 2020

  • Support for new module: type dependencies

    • Allows reusable JavaScript modules to be injected as dependencies into actions

  • Run Extractor Commands

    • import-io extractor:run:local and import-io extractor:run:remote

    • Runs an entire extractor according to the extractor.yaml file

    • Will run an Authenticated Extractor end to end, without the need to explicitly run authentication then data actions separately.

Release v0.5.0 - August 6, 2020

Release v0.4.3 - August 6, 2020

Bug Fixes

  • Fixes issue where all extractors where trying to be uploaded as authenticated extractors

Release v0.4.0 - July 28, 2020

  • Authenticated Extractor support

    • authentication Entry points can now be specified in robot.yaml. This serves as the action used to log in the user before the data is extracted.

    • An extractor can use the authentication entrypoint by specifying authenticated: true in extractor.yaml

  • Browser sessions are stored by default for 15 minutes both in local and remote

  • New --incognito flag can force a new browser session

Release v0.3.5 - July 25, 2020

  • Bug fix for data.json output

Release v0.3.3 - July 20, 2020

  • Bug fix where for parameter values defined in extractor.yaml were not available in code entry point.

Release v0.3.0

  • New options when calling goto. Click here for more

    • firstRequestTimeout

    • antiFingerPrint

    • downloadLimit

    • downloadTimeout

    • downloadRetries

  • New APIs extractor context to set these properties globally

    • setFirstRequestTimeout

    • setAntiFingerprint

    • setDownloadLimit

    • setDownloadTimeout

    • setDownloadRetries