Large file processing

This functionality is available in a Stedi module.

The large file processing module allows you to translate single EDI files that are more than 50MB in size.

Since the translation ratio of an EDI file to JSON typically approaches 1:10, a translated JSON artifact will be multiple times the size of the original EDI file. To avoid overwhelming downstream systems, you can use fragments to split translated transactions from Stedi into smaller, more manageable chunks.

How fragments work

Large files are often the result of transaction sets containing many repeated loops or segments. For example, a healthcare provider may send an 837 Health Care Claim containing many individual claims, or a brand may send an 846 Inventory Inquiry/Advice file containing millions of SKUs.

You can enable fragments for one repeated EDI segment in each transaction set. Once enabled, Stedi splits the translated transaction payload into chunks based on the selected segment.

For example, if you enable fragments on the LIN loop in an 846 Inventory Inquiry/Advice, Stedi will fragment a translated 846 file and emit the following events:

One transaction.processed.v2 event for each unique transaction set that was included in the original file.
One or more fragment.processed.v2 events containing several iterations of the LIN loop (inventory items)

When you receive a transaction.processed.v2 event through a Destination webhook, the transaction payload contains an empty array in the position of the fragmented segment. Adjacent to that segment, the payload also includes an x-stedi-fragment-detail section with general information about the fragments that Stedi created.

{
  "event": {
    "version": "0",
    "id": "1c2ee29e-f640-7b65-c8f5-297fb753ceaf",
    "detail-type": "transaction.processed.v2",
    "source": "stedi.core",
    "account": "001690149382",
    "time": "2023-11-13T15:47:09Z",
    "region": "us-east-1",
    "resources": [
      "https://core.us.stedi.com/2023-08-01/transactions/1b1d2424-72ba-4157-bcfa-3e1620430a3f"
    ],
    "detail": {
      "transactionId": "1b1d2424-72ba-4157-bcfa-3e1620430a3f",
      "direction": "INBOUND",
      "mode": "production",
      "fileExecutionId": "bb141a6f-79f8-9c88-9b91-37609ddd90f9",
      "processedAt": "2023-11-13T15:47:09.231Z",
      "fragments": {
        "batchSize": 800,
        "fragmentCount": 1,
        "keyName": "item_identification_LIN_loop"
      },
      "artifacts": [
        {
          "artifactType": "application/edi-x12",
          "usage": "input",
          "url": "https://core.us.stedi.com/2023-08-01/transactions/1b1d2424-72ba-4157-bcfa-3e1620430a3f/input",
          "sizeBytes": 665,
          "model": "transaction"
        },
        {
          "artifactType": "application/json",
          "usage": "output",
          "url": "https://core.us.stedi.com/2023-08-01/transactions/1b1d2424-72ba-4157-bcfa-3e1620430a3f/output",
          "sizeBytes": 825,
          "model": "transaction"
        }
      ],
      "partnership": {
        "partnershipId": "senderid_acme",
        "partnershipType": "x12",
        "sender": { "profileId": "senderid" },
        "receiver": { "profileId": "acme" }
      },
      "x12": {
        "transactionSetting": {
          "guideId": "01H9JMMG4839VQG9QQVSZ6X29G",
          "transactionSettingId": "01HF4N77F5YWA2RXEDMMF5FF6J"
        },
        "metadata": {
          "interchange": {
            "acknowledgmentRequestedCode": "0",
            "controlNumber": 76
          },
          "functionalGroup": {
            "controlNumber": 76,
            "release": "004010",
            "date": "2022-09-24",
            "time": "20:01",
            "functionalIdentifierCode": "IB"
          },
          "transaction": {
            "controlNumber": "319101",
            "transactionSetIdentifier": "846"
          },
          "receiver": {
            "applicationCode": "ACME",
            "isa": { "qualifier": "ZZ", "id": "ACME" }
          },
          "sender": {
            "applicationCode": "SENDERID",
            "isa": { "qualifier": "ZZ", "id": "SENDERID" }
          }
        }
      }
    }
  },
  "artifact": {
    "artifactType": "application/json",
    "usage": "output",
    "attachedReason": "WITHIN_SIZE_LIMIT",
    "detail": {
      "heading": {
        "transaction_set_header_ST": {
          "transaction_set_identifier_code_01": "846",
          "transaction_set_control_number_02": 319101
        },
        "beginning_segment_for_inventory_inquiry_advice_BIA": {
          "transaction_set_purpose_code_01": "00",
          "report_type_code_02": "DD",
          "reference_identification_03": "37",
          "date_04": "2022-09-12",
          "time_05": "12:30"
        },
        "name_N1_loop": [
          {
            "name_N1": {
              "entity_identifier_code_01": "ZZ",
              "name_02": "WHSE",
              "identification_code_qualifier_03": "92",
              "identification_code_04": "WHSE"
            }
          }
        ]
      },
      "detail": {
        "item_identification_LIN_loop": [],
        "x-stedi-fragment-detail": {
          "batchSize": 800,
          "fragmentCount": 1,
          "keyName": "item_identification_LIN_loop"
        }
      },
      "summary": {
        "transaction_totals_CTT": {
          "number_of_line_items_01": 16,
          "hash_total_02": 196
        },
        "transaction_set_trailer_SE": {
          "number_of_included_segments_01": 37,
          "transaction_set_control_number_02": 319101
        }
      }
    }
  }
}

When you receive fragment.processed.v2 event through a Destination webhook, the payload contains the fragment as well as general information about the transaction.

{
  "event": {
    "version": "0",
    "id": "de2f11fc-1726-a17f-c9af-cc191cb0f7a0",
    "detail-type": "fragment.processed.v2",
    "source": "stedi.core",
    "account": "001690149382",
    "time": "2023-11-13T15:47:09Z",
    "region": "us-east-1",
    "resources": [
      "https://core.us.stedi.com/2023-08-01/transactions/1b1d2424-72ba-4157-bcfa-3e1620430a3f"
    ],
    "detail": {
      "fileExecutionId": "bb141a6f-79f8-9c88-9b91-37609ddd90f9",
      "transactionId": "1b1d2424-72ba-4157-bcfa-3e1620430a3f",
      "direction": "INBOUND",
      "mode": "production",
      "processedAt": "2023-11-13T15:47:09.231Z",
      "fragments": {
        "batchSize": 800,
        "fragmentCount": 1,
        "keyName": "item_identification_LIN_loop"
      },
      "fragmentIndex": 0,
      "artifacts": [
        {
          "artifactType": "application/json",
          "usage": "output",
          "url": "https://core.us.stedi.com/2023-08-01/transactions/1b1d2424-72ba-4157-bcfa-3e1620430a3f/fragments/0/output",
          "model": "fragment",
          "sizeBytes": 4802
        }
      ],
      "partnership": {
        "partnershipId": "wormpotato_amazonds",
        "partnershipType": "x12",
        "sender": { "profileId": "amazonds" },
        "receiver": { "profileId": "wormpotato" }
      },
      "x12": {
        "transactionSetting": {
          "guideId": "01H9JMMG4839VQG9QQVSZ6X29G",
          "transactionSettingId": "01HF4N77F5YWA2RXEDMMF5FF6J"
        },
        "metadata": {
          "interchange": {
            "acknowledgmentRequestedCode": "0",
            "controlNumber": 76
          },
          "functionalGroup": {
            "controlNumber": 76,
            "release": "004010",
            "date": "2022-09-24",
            "time": "20:01",
            "functionalIdentifierCode": "IB"
          },
          "transaction": {
            "controlNumber": "319101",
            "transactionSetIdentifier": "846"
          },
          "receiver": {
            "applicationCode": "ACME",
            "isa": { "qualifier": "ZZ", "id": "WORMPOTATO" }
          },
          "sender": {
            "applicationCode": "SENDERID",
            "isa": { "qualifier": "ZZ", "id": "AMAZONDS" }
          }
        }
      }
    }
  },
  "artifact": {
    "artifactType": "application/json",
    "usage": "output",
    "attachedReason": "WITHIN_SIZE_LIMIT",
    "detail": {
      "item_identification_LIN_loop": [
        {
          "item_identification_LIN": {
            "assigned_identification_01": "1",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "M3199W"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 9
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "2",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "M8010A"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 52
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "3",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "M4799V"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 3
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "4",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "M4899Q"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 0
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "5",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5155C"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 0
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "6",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5152N"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 16
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "7",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5101O"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 4
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "8",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5101C"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 10
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "9",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5152O"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 9
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "10",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5101N"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 24
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "11",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5155O"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 11
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "12",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5155N"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 15
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "13",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5152C"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 10
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "14",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5101W"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 13
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "15",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5152W"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 15
            }
          ]
        },
        {
          "item_identification_LIN": {
            "assigned_identification_01": "16",
            "product_service_id_qualifier_02": "SK",
            "product_service_id_03": "K5155W"
          },
          "destination_quantity_SDQ": [
            {
              "unit_or_basis_for_measurement_code_01": "EA",
              "identification_code_qualifier_02": "92",
              "identification_code_03": "WHSE",
              "quantity_04": 15
            }
          ]
        }
      ]
    }
  }
}

Configure fragments

You must enable fragments within the transaction’s guide and the corresponding inbound transaction setting.

To enable fragments within a Stedi guide:

Go to the Guides page and edit the guide you want to update.
Click the node (segment) in the guide that you want to use to split the file. The node must be a looping data structure, such as an HL loop containing inventory items.
Toggle Set as fragment to ON.
Publish the changes to your guide.

Once you have enabled fragments on the Stedi guide, create an inbound transaction setting with the guide and toggle Enable fragments to ON. You can use the default batch size (800 segments) or set a custom batch size (up to 50,000 segments).

Within this partnership, Stedi will now split inbound transactions of this type into fragments.

View fragments

To view fragments, go to the Transactions page and click a transaction. On the transaction’s details page, you can review each fragment’s metadata and output JSON.

Ingest fragments

You have two options for ingesting fragments into your downstream system:

Push-based: Configure a destination webhook for fragment.processed.v2 events and ingest fragments as Stedi emits these events.
Pull-based: Configure a destination webhook for transaction.processed.v2 events only. After you receive a transaction.processed.v2 event, you can use the API to retrieve associated fragments in batches according to your system’s requirements.

Get started

Configure

Operate

Modules

Large file processing

How fragments work

Configure fragments

View fragments

Ingest fragments

Get started

Configure

Operate

Modules

​How fragments work

​Configure fragments

​View fragments

​Ingest fragments

How fragments work

Configure fragments

View fragments

Ingest fragments