Schemas

An Organization is the company or entity requesting the data extraction. In the DOC environment, certain
information is defined at the Organization level. These items are global and impact the entire
Organization. At this level, Schemas (similar to Destinations and Integrations – for example) have a
global impact and may be used in different Projects and Collections.

Schemas, essentially, are the Extractor column names. As you define your Schema, the fields that you
add and the data types you assign must align with the columns in the related Extractor. The Schemas
page allows you to construct the shape of the extracted data.

When you initially create a Schema, it is in a Draft state. You can make modifications to the Schema
while it is in a Draft state. Once you publish the Schema (an act you must perform before data is pushed
to the customer Destination), you cannot make any breaking changes. For example, after you publish a
Schema, you can add columns; however, you cannot change the data type of a column from text
to numeric. This is a breaking change.

Add a Schema

To add a new Schema:

  1. From the left navigation pane, click Schemas.

  2. From the top right of the Schemas page, click the Add a Schema icon or plus (+) symbol.

  3. From the New Schema page that now appears, enter content in the Name field.

    The name you enter will autofill the Slug/ID field. You can associate Slug/IDs
    with many DOC platform objects, which serve as self-defined identifiers. Slug/IDs
    can be useful as you reference APIs or create variable names. You cannot change Slug/IDs.
    That noted, ensure they are meaningful.

Add a Field

schemas new field
Field Description

Single Value

Select this checkbox if you must consider whether or
not the value is an array.

Filter

If the value is a falsy value (zero, false, blank), mark this
row as filtered and do not include it in the data pushed
to the customer Destination.

Internal

If this is true, the data is excluded from the data
pushed to the customer Destination.

Primary Key

The fields that are part of the composite primary
key give the rows the _id metadata column –
a generated UUID from the hash of the column values. The data pushed to Destinations is deduplicated on this ID.

Type

The type gives the avro/parquet data types and also
controls how the system turns extracted text values
into typed values. The locale parameter on a Source
is used when performing this conversion.

Default Value

A textual default value for the column.
Note that this should be in ISO format for date/time and
JSON format for numbers and Booleans.

Formula

https://github.com/handsontable/formula.js/tree+ /master/test[Supported functions].
Plus PARAM('name') to get a Source parameter. Plus INPUT('name') to get an input parameter.
Plus OUTPUT('name') to get the output value.

Validation Rules

These settings contribute towards the validation error statistics for Snapshots.

Field Description

Required

Select this checkbox if you want to enact validation rules.

Regex Pattern

RegEX or regular expressions allow you to check a string
of characters for matches. You might need to check
fields on a form to ensure they meet certain
specifications. For example, you might create
an expression to check the Email Address
field to ensure it has an @ symbol and an
extension such as .com or .org. In another example,
you might create an expression to validate passwords
to ensure they satisfy requirements.

Min Length

This field requires that you specify the minimum length
of the value associated with the RegEX or regular expression.

Max Length

This field requires that you specify the maximum length
of the value associated with the RegEX or regular expression.

These rules are soft indicators. They do NOT filter data. You must select the Filter checkbox
(available in the Add a Field view) to invoke this setting.

To add a new Field:

  1. Enter a Field Name.

    The name must be unique and cannot contain spaces.

  2. Move the Deprecate toggle switch to the right or ON position if you want to deem this field inactive.

    The Deprecate toggle switch defaults to the left or OFF position, as most fields are active.

  3. Enter a Description.

  4. Select from the Single Value, Filter, Internal, and Primary Key checkboxes (as defined above).

  5. Enter content in the Type field.

    This field denotes a data type. The default entry is Text. Boolean, Currency, Whole Number, Decimal,
    Date, Date & Time, Image, URL, and File represent data types.

  6. Enter a Default Value.

  7. Enter a Formula (as defined above).

  8. Select the Required checkbox if you need to enter Validation Rules.

  9. Enter a Regex Pattern.

  10. Enter a Min Length.

  11. Enter a Max Length.

Viewing a Schema

The entries in the table on the primary page contain existing Schemas. You can identify an entry then
click to select.

schemas view

As you view a Schema, certain icons might be visible:

Icon Description

🔑

Part of the primary key

[ ]

Array field

⭐️

Required field

The top right of this page displays the Edit, Publish, and Download the Avro Schema icons.

Edit a Schema

To edit a Schema:

  1. From the left navigation pane, click Schemas.

  2. From the Schemas table, identify an entry then click to select.

  3. From the top right of the new page that now appears, click the Edit Schema or pencil-like icon.

  4. From the Edit Schema page, make any necessary updates.

    You cannot modify the Slug/ID field. In addition, if this Schema has been published, you cannot
    make any breaking changes. For example, after you publish a Schema, you can add columns;
    however, you cannot change the data type of a column from text to numeric. This is a
    breaking change.

Publish a Schema

To publish a Schema:

  1. From the left navigation pane, click Schemas.

  2. From the Schemas table, identify an entry then click to select.

  3. From the top right of the new page that now appears, click the Publish Schema icon.

After you publish a Schema, it moves from a Draft state to Published. Once published, you cannot
make any breaking changes. For example, after you publish a Schema, you can add columns; however,
you cannot change the data type of a column from text to numeric. This is a breaking change.

If you click the Save button after adding new fields, you will be able to review all Schema changes.
You can choose to cancel out of this view, which will return you to the Edit Schema page.

If you want to stop using a field, you can mark a field Deprecated (by sliding this toggle
switch to the right or ON position). Deprecating a field will remove it from the Schema view.

Download the Avro Schema

To download the Avro Schema:

  1. From the left navigation pane, click Schemas.

  2. From the Schemas table, identify an entry then click to select.

  3. From the top right of the new page that now appears, click the Download the Avro Schema icon.

The downloaded Avro Schema provides a JSON file inclusive of the record definition of the Schema you
created along with its associated fields.

Schema History

To view the Schema History:

  1. From the left navigation pane, click Schemas.

  2. From the Schemas table, identify an entry then click to select.

  3. From this new page which provides a view of the Schema, click the View Changes button.

    Here, you can view updates to the Schema fields for both Draft and Published Schemas.

Delete a Schema

You cannot delete Schemas.