Release Notes

Reduce likelihood of Snapshot Imports failing during stats generation

As a chunked and chained Flow executes, inputs are guaranteed to be unique in each chunk.

For example: In a chained & chunked flow, segment 2 chunk 2 will not run any inputs that are in any of the sister chunks in segment 2.

A separate JQ Deduplication Transform can be provided which will allow you to configure what should be considered unique.

Users can now specify the maximum number of inputs a Source should can run with. If the number of inputs on a Snapshot exceeds the maximum then the Snapsot will fail to start.

This includes showing destinations by segment (if chained flow) and a link to the collection destinations page for easy access to update when necessary.

Users cannot mark a Source as ACTIVE if it has no extractorId

"Play" button is disabled on Source Home if there is no extractorId

Chained segments will wait for all snapshots to reach a finished state before starting the next segment.

If a delivery is CHAINED, the Snapshot table will be grouped by segment/collection

Destination Pushes are now available under the "Health Metrics" section of Snapshot Home

Push errors are now displayed in the UI

Successful pushes are now displayed in Destination Push Spotlight

If the platform undergoes scheduled maintenance or is experiencing an outage, a message will be displayed at the top of each page

Users can now edit parameters for multiple sources at once

For Flows whose inputs are stored on S3, a JQ expression can be declared which will transform the inputs before running snapshots.

It is now possible to cancel a CHUNKED snapshot which will cancel all running chunked children

A redesigned Collection Checks page which allows text to be inserted to support check parameterization. See Release v3.4.0 below.

A custom Parquet file can be generated from the custom output using a predefined Schema. Configured on Collection Settings.

Specify how long each segment in your chain should take to collect data.

Now CLI output and Paginated Extractors from SaaS platform can be re-extracted using saved HTML.

Collection checks can now be written with syntax %{ PARAM("foo") }%' and %{ DEFAULT(PARAM("zip"), 400) }% to substitute using parameters specified on sources.

In CHAINED flows, parent snapshots in the last segment of a chain will aggregate chunks before pushing if “aggregateChunks” is turned on.

stopBy timestamps on snapshots will be dynamically generated as the delivery progresses based on time left in “dataHours”, thus improving speed and metric reporting.

API now accepts a payload to override the saved flow configuration, negating the need to PATCH a flow before running it.

A new FlowDelivery relationship was added to track what configuration a delivery ran with.

New type of flow which allows chaining the output from one collection into the inputs of another. Data can be transformed across collection segments using jq syntax.

New Extractors support embedded transforms within them. These transforms are executed during the Snapshot import pipeline. If a transform fails it will mark it on the Page Summary and track the percentage of failed transforms as a metric on the snapshot health.

Allows files > 5gb to be pushed to buckets

Inputs - the inputs that the crawl run ran with

Crawl Run (Log) - log file from crawl run output

Crawl Run (JSON) raw crawl run output before DOC transformation

Option on Flows to wait until all child "chunks" of a snapshot are PASSED_QA before aggregating data and pushing to destinations.