Snapshot Retry Options
In the event that a snapshot import fails, the JQ tranform for custom files changes or fails, or the runtime configuration of a snapshot changes, there could be a need to retry certain stages of a snapshot’s lifecycle.
The options provided to handle these scenarios are:
Regenerate Custom Assets
The above options can be found by clicking the ellipsis on the top-right corner of a snapshot’s home page. These options are only available in certain circumstances, relying on a snapshot’s status and other factors. More information about each option can be found below.
Generate Assets is the final stage of the import pipeline for a snapshot. If the snapshot’s collection has any linked destinations with a custom file type selected (custom text or custom parquet), they will be created during this stage.
In the event that the JQ transform associated with a snapshot fails for a custom file, the snapshot will fail to import (ie. status will be FAILED) with a relevant error message. If a snapshot fails because of the "Generate Assets" stage, this stage can be retried with the "Regenerate Custom Assets" option. This will only retry the "Generate Assets" stage. If the regeneration is successful, a snapshot can continue to QA to be pushed.
|If a snapshot fails because of a JQ transform error, the JQ transform should be revised before trying to regenerate assets.|
In the event that a snapshot fails to import prior to the "Generate Assets" stage, a snapshot can be re-imported. A re-import can be used if there are schema incompatibilities between the extractor’s schema and DOC’s schema. Additionally, a re-import can be tried if the import fails for any other reason i(ie. a system failure or timeout).
Re-importing a snapshot will put the original snapshot into a status of SUPERCEDED and a new snapshot will be created using the crawl run and a new import.
If an entirely new crawl run is necessary without a revision to an extractor’s runtime configuration, an option is available to re-run. The original snapshot will be moved to a status of SUPERCEDED and a new snapshot will be created, along with a new crawl run.
If an extractor needs to be updated before being re-imported, a snapshot can be re-extracted after the extractor is updated. The original snapshot will be SUPERCEDED and a new snapshot will be created to capture the re-extraction. The "Re-extract" stage will be added to the new snapshot’s import pipeline.
In the event that the final snapshot output is incorrect due to an issue in the transform function (CLI extractors), extractors that have been migrated to DOC and are deployed using v3.0.0 or higher of the CLI can re-run the import pipeline with an extractor version (Runtime Configuration) that includes the "fixed" version of the transform. This re-evaluation requires that the transforms are stored on the Runtime Configuration for both the version used on the crawl run, and the version being used to re-run transforms. Information about updating an extractor to use this functionality can be found under the "Extractor Studio" section of the docs.
|If a snapshot is SUPERCEDED, its new snapshot will have a link to the original visible on a snapshot’s home page.|