Release Notes

A new destination type, AZURE allows pushing Snapshot assets to Azure Blob Storage

An extractors list at the Organzation level. Allows users to see which extractors are used by an org, which versions of those extractors are in use, which versions of those extractors are available, and which sources are using a given extractor.

You can now label Snapshots in the same way that you can label Sources allowing users to quickly identify and group snapshots which are exhibiting similar behavior.

Destinations can be linked/unlinked on a flow in the event all collection destinations do not need to be included in a delivery.

Chained Flows can link/unlink destinations by segment, based on each segment’s collection’s destinations.

All NEW projects will now opt-in to HTML Requester Pays, this can be toggled via Project Settings

UI Support for Requester Pays is now available via Project Settings. The Settings page can be found in the sidebar on a Project’s home page.

Requester Pays support has been extended to Screen Captures. In order to access files via the appropriate headers, please reach out to Ops to ensure a customer’s information is properly configured.

A readme section has been added to the organization homepage for tracking information relevant to an entire organization.

The Product API Backend has been updated from HTTP to gRPC

A minor bug fix has been issued for the Public Delivery and Delivery Drilldown pages

Projects now have a setting that can be enabled that allows customers to retrieve HTML assets using a "Requester Pays" header

Project Settings can be managed via tha API, UI support is in development. For API information, review our API docs: https://workbench.import.io/api/docs/#/Project

More information on configuring this can be found here: Enable Requester Pays (HTML)

Project Flows list shows all Flows, not just the first 100

Aggregation error message will now correctly say "No results to aggregate" when there is no crawl run data to aggregate.

Flow edit page now properly loads aggregation options when navigating directly to edit page.

Query gap for crawl runs will not exceed 1 hour

Fix for CopyFiles stage during Snapshot import, allowing larger downloaded file sets to be pushed to destinations

Performance improvement during Snapshot QA Tests generation

Allow users other than ADMIN to deploy authenticated extractors

Performance improvement for Snapshot Aggregation

Improvement when starting Snapshots with large number of inputs

Snapshots will not run after cancelling a delivery, even if they have not been chunked yet

The S3 Configuration is now being cleared from state, if you change a new Flow type from SIMPLE to CHAINED

After editing the Collection Parameters or Assignee on the Collection Home page, if you then click on the Edit button, the README is still present on the form. The same behavior has been fixed on a Source Home page

The Save button is enabled after the README has been changed on a Flow

A source engineer can no longer edit a destination from the Org Home page

ZEUS-1544 (reported by Delivery) addresses a minor bug where updating a parameter on the Source Home was not always saving

Destination form improvements, including name validation

Collection Settings save validation improvements

Corrected an error message associated with re-extracting a PUSHED snapshot

Proper handling for single quotes in JQ expressions

Snapshots that are incomplete, scheduled, or awaiting inputs will be cancelled when a delivery is closed by the system

Fix missing Input JQ Transforms for first segment of CHAINED Flow form

Add pagination ("Load More" button) for Parent Snapshot Chunks list

Validate Destination name when creating new Destinations

Retain navigation history when navigating from Snapshot to a Delivery

Properly mark timed out IMPORT jobs as FAILED

Display "highest" user role on user table

Show Input JQ Transform field when configuring a SIMPLE Flow

Accurate TIMED_OUT input reporting for paginated extractors built with SaaS user interface

Removed extra file generation step during GENERATING_ASSETS stage of Snapshot Import

Proper Html Extraction importing and reporting for CLI extractors using MERGE_ROWS merge type

Fixed disappearing expanded rows after scrolling on Snapshot QA Checks page

Allow removing ADMIN role from Org Users

Improved error message for FAILED Snapshots that failed to import due to no json data on the crawl run

Fixed scenario where second segment snapshots in a CHAINED Flow were receiving the wrong collection windows. Resulting in an incorrect query gap being set for crawl runs.

Retain parent/chunk relationships when a Snapshot aggregation is retried.

Fix email integration form displaying wrong information

Fixed issue where navigating to Deliveries from a selected Delivery resulted in deliveries missing from the list

Destination Push fixes

Removed eliding for breadcrumbs

Source list can now be filtered by parameter values for bulk editing

Fixed double locale conversion during import stage where column typing was getting applied twice, causing incorrect data.

Creating and Editing Sources

Snapshot operations

Editing Source Filters on a flow

Performance improvements around Snapshot Stats Generation and Re Extract

Failing to get snapshot inputs from S3 will fail immediately instead of retrying. Allowing users to quickly know the inputs file is not available.

Fix sort order on "History" views

Faster re-extract performance

Fix for Snapshots not timing out properly on non chained Flows

Snapshot Home download button is now disabled when files are not available

Required checkbox available on Collection Checks table

If a check is optional, its result will still be evaluated, but will not prevent a snapshot from pushing if it fails

Metrics for the count of failed optional checks and failed required checks is available on the snapshot health, and are included on the snapshot tables

Performance improvement for failed chunk aggregation when files are very large

HEALTH() - References snapshot health values.

METRIC() - References values contained in the stats file

PUSHED Snapshots can now be re-extracted

PUSHED Snapshots can also be transitioned back to QA

Fix for applicataion displaying duplicate Deliveries

Increased default limit of displayed collection checks to 1000, previous was 100.

Create/Edit Source API now returns 400 instead of 500 when a bad extractor Id is passed.

Better handling of chunk splitting

SQL Query optimization

Proper handling of input files with bad encoding

UI performance improvements on delivery pages

Various bug fixes on user interface

Typed column fixes for locale de-ch

Not all parameters visible on Source home page

API error and UI failing to display when a single snapshot’s page count is 0

Snapshot row count displayed - instead of 0 after importing

IMPORT performance issues. "Essential container in task exited"

Collection Check for swing % fails if a column consistently returns no data

Bug fix for Chained Flow Segments failing to start due to input lookup error

Bug fix for “re-extract” on Snapshots

Bug fix for aggregated push variables

Flow source filtering

Destinations accept Source Parameter values in your output template

Added support for import.io CLI extractor output

Setting QPS for one chunk is now supported.

“First seen” column - A collection setting is now available for collections whose schema has a primary key, and generates the date when an item was "first seen" and includes it as a meta-data timestamp column "_firstSeen" in the snapshot data.

“FILE” Type column - This new column type on schemas helps include the metadata information about files extracted and downloaded during extraction time.

Include downloaded files in destination pushes - You can now include any files and images that were downloaded during your extraction in pushes to your destinations.

SFTP Destinations Type - You can now configure SFTP servers as destinations for your data.

Severity on Alarms - Flow alarms now have a “severity” option which can be set on the alarm and included in any alarm notifications.

APIs and UI to Resolve or Trigger Alarms - Now when a delivery is open you have the option to manually trigger or resolve alarms for the selected delivery alarm page.

Importing cancelled snapshots - now if you cancel a snapshot you have an option in the UI to manually import the data that was collected.

Max chunk crawl time - When configuring a flow you can set the chunkTimeoutHours which is the max time a chunk should run for. If the chunk times out the data collected will be automatically imported.