Release Notes
Release v7.12.0 - September 22, 2021
Release Notes: https://import-io.atlassian.net/wiki/spaces/PRODUCT/blog/2021/08/16/1659273242/Release+Blog+2021-09-22+DOC+7.12.0
Adhoc Flow Delivery Dialog
Users now have two options for running a flow in DOC - Default Run (reflects existing behavior) or Custom Run. Default Run will navigate to a confirmation screen that shows a detailed summary of the settings for the flow. Custom Run will walk the user through customizing the options, with the flow’s default settings pre-filled. We hope that this will ease the use of Adhoc Flows and make it easier for users to ensure deliveries are started with the proper parameters. Additionally, the "Flow Configuration" for a particular delivery will be visible from a delivery’s dashboard.
Auth Credentials Bank
Auth Credentials can be managed and linked to Sources within DOC. Credentials/Logins can be grouped with Login Pools. Login Pools are linked to a Source that has an authenticated extractor; at runtime, a login from the pool will be used. If more than one login exists in the pool, credentials are rotated to reduce freqeunecy of use.
Enhancements
-
Project History
-
deliverByis now available as a variable for destinations -
Removed unnecessary views from UI to clean up menus, including Organization Deliveries.
Re-run Transforms
Re-run Transforms is now available as a Snapshot option. This enables users to re-run a their transform functions against data rows without having to re-extract or re-run. For more information, read the docs.
| Re-run Transforms requires v3.0.1 of the CLI, which is pending release. |
Release v7.10.0 - July 29th, 2021
Release Blog: https://import-io.atlassian.net/wiki/spaces/PRODUCT/blog/2021/07/22/1617264641/2021-07-29+DOC+7.10.0
Azure Destination
-
A new destination type,
AZUREallows pushing Snapshot assets to Azure Blob Storage
Release v7.9.3 - July 22nd, 2021
Bug fix for large file uploads. Increases max part size for multipart uploads during CopyFiles Import pipeline stage.
Release v7.9.1 - July 14th, 2021
Bug fix for Snapshots occasionally failing import due to Out of Memory issues.
Release v7.9.0 - June 30, 2021
Release Blog: https://import-io.atlassian.net/wiki/spaces/PRODUCT/blog/2021/06/29/1554645260/2020-06-30+-+DOC+Release+7.9 Jira Release: https://import-io.atlassian.net/projects/ZEUS/versions/15840
Custom Flow Collection Destinations
-
Destinations can be linked/unlinked on a flow in the event all collection destinations do not need to be included in a delivery.
-
Chained Flows can link/unlink destinations by segment, based on each segment’s collection’s destinations.
Requester Pays Enhancements
-
All NEW projects will now opt-in to HTML Requester Pays, this can be toggled via Project Settings
-
UI Support for Requester Pays is now available via Project Settings. The Settings page can be found in the sidebar on a Project’s home page.
-
Requester Pays support has been extended to Screen Captures. In order to access files via the appropriate headers, please reach out to Ops to ensure a customer’s information is properly configured.
Release v7.8.0 - June 3, 2021
Requester Pays HTML
-
Projects now have a setting that can be enabled that allows customers to retrieve HTML assets using a "Requester Pays" header
-
Project Settings can be managed via tha API, UI support is in development. For API information, review our API docs: https://workbench.import.io/api/docs/#/Project
-
More information on configuring this can be found here: Enable Requester Pays (HTML)
Source Input Trimming
When running a flow, if the number of inputs for a Source breaches "Maximum Inputs" the inputs file will be trimmed to allow the Snapshot to continue to run.
The number of inputs that breached the threshold are captured on the Snapshot, and whether or not the threshold has been breached is displayed on the Snapshots list.
Bug Fixes
-
Project Flows list shows all Flows, not just the first 100
-
Aggregation error message will now correctly say "No results to aggregate" when there is no crawl run data to aggregate.
-
Flow edit page now properly loads aggregation options when navigating directly to edit page.
-
Query gap for crawl runs will not exceed 1 hour
-
Fix for CopyFiles stage during Snapshot import, allowing larger downloaded file sets to be pushed to destinations
Release v7.6.0 - May 5, 2021
Extractor Version Control
DOC Extractors
Sources with an extractor now have the option to transition their extractor and its latest version (Runtime Configuration) to DOC. A new section has been added to the Source Home called "Extractor Information". If a Source’s extractor has not been transitioned, information and a button to migrate an extractor and its latest version can be found on the Source Home.
Example:
After an extractor has been transitioned, its latest version (RTC) and other metadata will be available in the "Extractor Information" section of Source Home.
Example:
If an extractor has multiple versions, a user may view versions of an extractor or change the version of the extractor that a Source is using by clicking the icon on the top right of the "Extractor Information" section. Clicking the icon should redirect the user to the "Extractor Versions" page for a Source. A list of versions will be displayed in chronological order, each relating to a RTC. Versions that are created from a CLI deployment will include information for finding the code reference in a project’s GitHub repository.
Example:
Extractor Versions
An Extractor Version is captured whenever an extractor is transitioned to DOC (via the Source Home or API) or if an extractor is deployed to a Source that has a DOC extractor. Extractor Versions can be changed on a Source, and have crawl runs started with that version’s Runtime Configuration. This can be accomplished by changing the Extractor Version based on the instructions above, and running a Source or Flow. Sources using an older version of an extractor will have an informational alert at the top of the Source Home to indicate that the latest version is not being used.
Creating Extractor Versions with relevant git information require CLI v2.1.0 or higher (CLI Downloads). The deploy command has not changed, except that deploying an extractor to a Source will automatically create an Extractor (if Source does not have an Extractor already) and Extractor Version in DOC. More information can be found in the Extractor Studio Release Notes.
Snapshot Chunk Aggregation Improvements
New option available for Snapshot aggregation in Flows, "Ignore Duplicate Rows". If this option is checked, Snapshot Aggregation skips MergeExtractions and AvroStage during import, allowing faster aggregation time.
Import Crawl Run file
A Snapshot can now be imported using any ndjson file. See the API docs for more information
Release v7.5.0 - April 21, 2021
Project Improvements
Environment
Projects now have an "Environment" property which can be either PRODUCTION, DEV, or STAGING.
Projects can now share a name, as long as they have a different "Environment"s.
Flows and Collections can now be moved between projects, by patching the projectId via the API
Lock
Projects can now be locked by clicking on the "lock" icon from the Project home page.
Once a Project is locked, a user can operate on a Project’s assets if:
-
The user has
ADMINorOPSrole for the Project’s Organization -
The Snapshot, Source, or Collection assigned to them
Newly created "Production" Projects are locked by default
OPS Role
OPS is a new role available to users of an Organization.
Users with the OPS role are able to operate on "locked" projects and create and maintain projects.
Flow Collection Settings
Collection settings can now be overridden at the Flow level, including Custom Output Settings, Global Output Settings, and Screen Capture Settings
Group By Transform
"Group by" is a feature introduced to run before Custom Transform (current JQ Transform) and allows flexible collapsing into a single line entries that share a common parameter
(i.e. a Product that appears under more than one category: the group by "Category Name - Rank" would output "Home Goods | Spring > Flowers > Daffodils | Mothers Day > Flowers - 10 | 5 | 7").
More information and examples available here
Archive
Archive is a feature that allows delivery of a single compressed file containing multiple destination assets, supporting customer needs downstream the delivery pipeline.
Release v7.4.0 - March 25, 2021
Query Usage - Extractions
Snapshots now have a usage field which is used to track more granular platform usage for billing purposes. Currently the only supported metric within this field is extractions
usage.extractions serves as a count of data extractions calls which were used during the data extraction. This includes paginated data.
View Failed Inputs Count
Snapshot Home page will now display the number of failed inputs for a crawl run in addition to the successful inputs count.
Release v7.3.0 - March 4, 2021
User Tokens
This feature will allow users to create and view their user tokens in the UI. A user token is utilized when working with API requests or when deploying and editing Sources from an extractor library.
On the User Account page, all of the tokens that belong to a user will be displayed (https://workbench.import.io/users). If a user does not have any tokens, they will be prompted to generate one.
If a new user is created, a token will automatically be generated for them. Also, the user tokens are no longer connected to a specific organization.
Bug Fixes
-
Snapshots will not run after cancelling a delivery, even if they have not been chunked yet
-
The S3 Configuration is now being cleared from state, if you change a new Flow type from SIMPLE to CHAINED
-
After editing the Collection Parameters or Assignee on the Collection Home page, if you then click on the Edit button, the README is still present on the form. The same behavior has been fixed on a Source Home page
-
The Save button is enabled after the README has been changed on a Flow
-
A source engineer can no longer edit a destination from the Org Home page
Release v7.2.0 - Feb 11, 2021
Snapshot Tables
Users are currently only able to view 100 snapshots at a time in the application, and have to load anything beyond the limit via API, which is not ideal for projects that have > 100 snapshots per delivery. All snapshot tables have been updated so that users can load more than the limit. The page numbers at the bottom of tables (for snapshots only) have been removed, and a "Load More" is available to view as many as needed (50 will load at a time and be added to the table). Additionally, child snapshots will now all be loaded, instead of being restricted to 100.
Test Plan and Overview: https://docs.google.com/document/d/1OLPsWGGFvUdfPUXwBqfLRoQEyAlo0KhrIEKll5CvSTg/edit?usp=sharing
CLI Updates
A new version (1.2.3) of the CLI is now available for download, new features found in Extractor Studio Release Notes
Bug Fixes
-
ZEUS-1544 (reported by Delivery) addresses a minor bug where updating a parameter on the Source Home was not always saving
-
Destination form improvements, including name validation
-
Collection Settings save validation improvements
-
Corrected an error message associated with re-extracting a PUSHED snapshot
Release v7.1.1 - Feb 4, 2021
Bug Fixes
-
Proper handling for single quotes in JQ expressions
-
Snapshots that are incomplete, scheduled, or awaiting inputs will be cancelled when a delivery is closed by the system
-
Fix missing Input JQ Transforms for first segment of CHAINED Flow form
-
Add pagination ("Load More" button) for Parent Snapshot Chunks list
-
Validate Destination name when creating new Destinations
-
Retain navigation history when navigating from Snapshot to a Delivery
Release v7.1.0 - Jan 26, 2021
Release v7.0.4 - Jan 21, 2021
Bug Fixes/Improvements
-
Show Input JQ Transform field when configuring a SIMPLE Flow
-
Accurate TIMED_OUT input reporting for paginated extractors built with SaaS user interface
-
Removed extra file generation step during GENERATING_ASSETS stage of Snapshot Import
-
Proper Html Extraction importing and reporting for CLI extractors using
MERGE_ROWSmerge type -
Fixed disappearing expanded rows after scrolling on Snapshot QA Checks page
-
Allow removing ADMIN role from Org Users
-
Improved error message for FAILED Snapshots that failed to import due to no json data on the crawl run
Release v7.0.3 - Jan 11, 2021
Bug Fixes
-
Fixed scenario where second segment snapshots in a CHAINED Flow were receiving the wrong collection windows. Resulting in an incorrect query gap being set for crawl runs.
-
Retain parent/chunk relationships when a Snapshot aggregation is retried.
-
Fix email integration form displaying wrong information
-
Fixed issue where navigating to Deliveries from a selected Delivery resulted in deliveries missing from the list
-
Destination Push fixes
-
Make sure that failures are properly marked with
FAILEDinstead ofINTERNAL_FAILURE -
Ensure that failed revert attempts receive status of
REVERT_FAILED -
Allow retrying a revert that failed
-
Release v7 - Jan 4, 2021
Generate Assets Stage
“Generate Assets” is a new stage for the import pipeline, and will be the final stage before the snapshot is ready to be reviewed by a user, or automatically pushed. If a custom destination is designated for any of the snapshot’s destinations, a custom file will be created and copied to DOC’s internal assets. Moving the custom asset generation to the import pipeline allows for the “Destination Push” process to be more efficient, and also allows the user to download a custom file (similar to downloading the JSON on a snapshot now from the home page) without a push taking place. A link will be made available in the “Download Snapshot” window for this.
IMPORTANT: Note that if you need to apply changes to a custom file after it has been imported you will need to "Regenerate Custom Assets" from the Snapshot Home page.
For more see Snapshot Retry Options
Delivery Alarm Severities
An Alarm Group’s and Delivery’s state will reflect the most severe Alarm state.
The priority is as follows (most severe first):
CRITICAL → ERROR → WARNING → INFO
Pass/Revert all QA Checks
A checkbox is available on the QA Checks page which will allow you to PASS all QA checks or revert to their initial calculated value.
View History of Metric values
When performing QA Checks you now have the option to "Show Historical Values" which will fetch and display a history of values for that particular field.
Release v6.1.0 - Dec 9, 2020
SOURCE_ENGINEER Role
SOURCE_ENGINEER role is now available to assign users of an Organization. This role restricts the creation and editing of most resources but allows:
-
Creating and Editing Sources
-
Snapshot operations
-
Running, re-running, re-importing, re-extracting
-
Passing/Failing QA Checks
-
Reverting/Retrying Destination Pushes
-
Commenting
-
-
Editing Source Filters on a flow
Byte Order Mark on Custom Output
You can now include a BOM header in your custom output. This is configured via a checkbox on the Collection Settings page.
Release v5.5.0 - Nov 9 2020
Chunk Collect Hours on chained segments
"Chunk Collect Hours" can now be configured at the segment level, allowing more granular control of your "Collected on Time" metrics.
Delivery Alarms for Snapshot QA Checks
Alarms can now be configured using failedRequiredTests and failedOptionalTests as metrics
Release v5.4.0 - Nov 6 2020
Release v5.2.0 - Oct 27 2020
Optional Collection Checks
Collection checks can now be marked as required or optional to allow snapshots to "auto-push"
-
Required checkbox available on Collection Checks table
-
If a check is optional, its result will still be evaluated, but will not prevent a snapshot from pushing if it fails
-
Metrics for the count of failed optional checks and failed required checks is available on the snapshot health, and are included on the snapshot tables
Warning for saving flows
Usability improvement to provide a warning to users who are editing a flow, and prevent removal of source filters by accident. If a user tries to edit a flow that has source filters, and the source filters are empty, they will be prompted to confirm that this is intentional.
Support for positive/negative swing checks
PCT checks are only capable of testing the deviation of a value between two snapshots. With the addition of this feature placing a sign (+/-) before the target number will permit the check to take the direction of the deviation into account. For example a PCT check of % Error ⇐ +10% will no longer trigger if the change is -20%.
Release v5.1.0 - Oct 22 2020
New Collection Check Functions
Checks can now reference stats and health metrics
-
HEALTH()- References snapshothealthvalues.-
Example
%{ IF(HEALTH("rows") > 100000, 2, 10) }%
-
-
METRIC()- References values contained in thestatsfile-
Example
%{ METRIC("data:field/name/stringPunctRatio/p05") }
-
Re-extract improvements
-
PUSHED Snapshots can now be re-extracted
-
Removes data from Redshift Datalake
-
Supercedes snapshot and re-extracts
-
-
PUSHED Snapshots can also be transitioned back to QA
-
Removes data from the Redshift Datalake
-
Allows for re-extraction
-
Reverting pushed files from destinations is still a manual operation. Can be achieved by "reverting" the destination push.
Release v5.0.1 Oct 12 2020
-
Increased memory for Import jobs
-
Reduce likelihood of Snapshot Imports failing during stats generation
-
-
Remove POST_PROCESSING from allowed Snapshot transitions
Release v5.0.0 - Sep 29 2020
-
Deduplication of inputs across chained chunks
-
As a chunked and chained Flow executes, inputs are guaranteed to be unique in each chunk.
-
For example: In a chained & chunked flow, segment 2 chunk 2 will not run any inputs that are in any of the sister chunks in segment 2.
-
A separate JQ Deduplication Transform can be provided which will allow you to configure what should be considered unique.
-
-
Source Input Thresholds
-
Users can now specify the maximum number of inputs a Source should can run with. If the number of inputs on a Snapshot exceeds the maximum then the Snapsot will fail to start.
-
-
Read-only view of Collection Destinations for a flow.
-
This includes showing destinations by segment (if chained flow) and a link to the collection destinations page for easy access to update when necessary.
-
-
flow(flow slug) now available as a template parameter for destination paths -
Source Validations
-
Users cannot mark a Source as
ACTIVEif it has no extractorId -
"Play" button is disabled on Source Home if there is no extractorId
-
Release v4.1.0 - Sep 16, 2020
-
Optimized chunk scheduling in CHAINED flows
-
Chained segments will wait for all snapshots to reach a finished state before starting the next segment.
-
-
Clicking on an Organization/Delivery navigates to Flow/Delivery to provide user with Flow context.
Release v4.0.0 - Sep 8, 2020
-
Delivery Snapshots UI
-
If a delivery is CHAINED, the Snapshot table will be grouped by segment/collection
-
-
Improved Destination Pushes UI
-
Destination Pushes are now available under the "Health Metrics" section of Snapshot Home
-
Push errors are now displayed in the UI
-
Successful pushes are now displayed in Destination Push Spotlight
-
-
Maintenance/Outage message in UI
-
If the platform undergoes scheduled maintenance or is experiencing an outage, a message will be displayed at the top of each page
-
-
Bulk Edit Source Parameters
-
Users can now edit parameters for multiple sources at once
-
Bug fixes
-
Not all parameters visible on Source home page
-
API error and UI failing to display when a single snapshot’s page count is 0
-
Snapshot row count displayed - instead of 0 after importing
-
IMPORT performance issues. "Essential container in task exited"
-
Collection Check for swing % fails if a column consistently returns no data
Release v3.6.0 - Aug 31, 2020
-
Input transforms on SIMPLE and CHAINED Flows
-
For Flows whose inputs are stored on S3, a JQ expression can be declared which will transform the inputs before running snapshots.
-
-
Cancel Parent/Chunked Snapshots
-
It is now possible to cancel a CHUNKED snapshot which will cancel all running chunked children
-
-
New Collection Checks Page
-
A redesigned Collection Checks page which allows text to be inserted to support check parameterization. See Release v3.4.0 below.
-
Release v3.5.0 - Aug 24, 2020
-
Custom parquet output
-
A custom Parquet file can be generated from the custom output using a predefined Schema. Configured on Collection Settings.
-
-
Configurable
dataHourson each Segment for CHAINED Flows-
Specify how long each segment in your chain should take to collect data.
-
-
Re-Extract improvements
-
Now CLI output and Paginated Extractors from SaaS platform can be re-extracted using saved HTML.
-
Release v3.4.0 - Aug 18, 2020
-
Initial support for parameterizing Collection Checks (API)
-
Collection checks can now be written with syntax
%{ PARAM("foo") }%'and%{ DEFAULT(PARAM("zip"), 400) }%to substitute using parameters specified on sources.
-
-
Ability to cancel Snapshots in
START_SCHEDULED
Release v3.2.0 - August 3rd, 2020
-
Zip compression available on destinations for
custom,stats,pagesandjsonfiles. -
Delivery metrics and speed enhancements on CHAINED flows
-
stopBytimestamps on snapshots will be dynamically generated as the delivery progresses based on time left in “dataHours”, thus improving speed and metric reporting.
-
Release 3.1.0 - July 23rd, 2020
-
Start delivery API enhancements.
-
API now accepts a payload to override the saved flow configuration, negating the need to PATCH a flow before running it.
-
A new FlowDelivery relationship was added to track what configuration a delivery ran with.
-
Release v3.0 - July 8th, 2020
-
CHAINED Flows
-
New type of flow which allows chaining the output from one collection into the inputs of another. Data can be transformed across collection segments using
jqsyntax.
-
-
Support for transform failures and metrics
-
New Extractors support embedded transforms within them. These transforms are executed during the Snapshot import pipeline. If a transform fails it will mark it on the Page Summary and track the percentage of failed transforms as a metric on the snapshot health.
-
-
Multipart upload support when pushing to s3 destinations.
-
Allows files > 5gb to be pushed to buckets
-
Release v2.6.1 - July 14th, 2020
-
New option to
skipQaTestson a flow -
Application now displays status of chunk aggregation.
-
Crawl run id is now clickable and navigates to crawl run debugger
-
New links under “Download Snapshot”
-
Inputs- the inputs that the crawl run ran with -
Crawl Run (Log)- log file from crawl run output -
Crawl Run (JSON)raw crawl run output before DOC transformation
-
-
delivery_idavailable as a variable for destination output path
Release v2.6.0 - June 30, 2020
-
Chunk aggregation
-
Option on Flows to wait until all child "chunks" of a snapshot are PASSED_QA before aggregating data and pushing to destinations.
-
Release v2.5.0 - June 22, 2020
-
Application now runs snapshots as the
legacyPlatformIduser (SaaS Platform) if this property is set on the organization. Allowing for proper billing and to support authenticated extractors
Release v2.4.0 - June 19, 2020
-
Support for fractional hours in flow configuration.
-
Comments thread on Deliveries
Release v2.3.0 - June 10, 2020
What’s New
-
Flow source filtering
-
Now you can select/filter which Sources get run in a SIMPLE Flow by any of the parameters in your Source.
-
-
Destinations accept Source Parameter values in your output template
-
Now you can use any Source Parameter as destination template variable. For example, if you have a Parameter locale. this can be set in your destination path/filename as :source.locale. When your Source is run with locale set to UK the destination will output with sourcename-UK .
-
-
Added support for import.io CLI extractor output
-
Now extractors created by the import.io CLI can be imported into workbench.
-
Release v1.8.2 - April 24, 2020
-
Fixed latest delivery + destination pushes bugs.
-
Added assigned user to snapshot tables
-
Added seconds to the destination and simple flow timestamps
-
Updated the user docs
-
Health metrics are open on the selected snapshot home page
-
Added a snapshot link in delivery sidebar, instead of delivery home dropdown
-
Added ‘All’ button to snapshot and pushes spotlight
-
Added revert option to pushes spotlight table
-
Fixed Delivery Push and Bull Queue metrics
Release v1.8.0 - April 10, 2020
Collections
-
“First seen” column - A collection setting is now available for collections whose schema has a primary key, and generates the date when an item was "first seen" and includes it as a meta-data timestamp column "_firstSeen" in the snapshot data.
Schemas
-
“FILE” Type column - This new column type on schemas helps include the metadata information about files extracted and downloaded during extraction time.
Destinations
-
Include downloaded files in destination pushes - You can now include any files and images that were downloaded during your extraction in pushes to your destinations.
-
SFTP Destinations Type - You can now configure SFTP servers as destinations for your data.
Alarms
-
Severity on Alarms - Flow alarms now have a “severity” option which can be set on the alarm and included in any alarm notifications.
-
APIs and UI to Resolve or Trigger Alarms - Now when a delivery is open you have the option to manually trigger or resolve alarms for the selected delivery alarm page.