Convert EDI to JSON

Introduction

In this tutorial, we demonstrate how you can build an end to end EDI processing solution on the Stedi platform. You could use this solution to ingest any EDI from your trading partners over FTP. Once the EDI file is uploaded, Stedi will automatically convert it to a JSON document and store the results in a Stedi Bucket. From there, you can download the file or submit it to an external webhook.

The Stedi platform is event-driven and charges you only for the operations that are involved to process your business transactions. You can find an overview of all Stedi services here.

In this tutorial, we will show you how to make this conversion with an EDI 810 Invoice document. The EDI 810 is commonly used to send invoices between trading partners. You can find more information about this EDI format here.

What we will build

At a high level, this implementation does the following;

  1. A trading partner uploads a EDI 810 document to a Stedi managed SFTP server, using their own personal SFTP login credentials.

  2. The EDI 810 document gets automatically placed in a Stedi Bucket. This is where we can persist the document for a longer term and build event-driven integrations with Stedi or external systems.

  3. From the bucket, we trigger a Stedi Function which does the actual file conversion. It will convert the EDI 810 document to a JSON document using Stedi EDI Core.

  4. The converted JSON document is written back into the bucket. From there, you can retrieve the converted document using the Stedi Bucket SDK or SFTP.

Under normal conditions, the end-to-end process should take under 10 seconds to complete the conversion. You do not need to scale or configure any infrastructure, Stedi will automatically do that.

Prerequisites for tutorial

A few of the configuration steps can be completed using the Stedi UI in your browser. For creating the Stedi Function, we will require some tools locally installed on your machine.

  • git
  • node.js
  • npm
  • jq
  • local Bash terminal
  • any SFTP client
  • any text editor (for example VS Code)

You also need a Stedi account and a Stedi API key. You can create the key in the Stedi UI and find more information about API authentication in our documentation.

Deploying to Stedi

Now let's get started deploying this solution to Stedi.

1. Creating a Stedi SFTP user and Stedi Bucket

We can create a new SFTP user through the Stedi UI. First, navigate to the Stedi UI using this link.

Next, click on "Create User" and fill in a description and file path for the user. We recommend putting a meaningful description for the user, so that you can easily identify it later. For the file path, we recommend creating a dedicated folder per trading partner. In this case, we can set the path to "/inbound".

After clicking "save", the username and credentials for the new SFTP user will be shown. Please take note of the password for the SFTP user as it will only be shown once. We currently do not offer a solution to retrieve the password of a user later on, but you can always create a new user in the worst case.

We should now be able to log in to the SFTP server using an SFTP client. You can use any conventional SFTP client for this purpose. In this tutorial, we are using the free Forklift SFTP client on a Mac.

If the credentials were entered correctly, you will now see an empty folder in your SFTP client. You can try to upload a test document to it to see if this works.

2. Stedi Bucket

As you saw in the reference architecture, we will need a bucket to persist the data going in and out of SFTP. We don't have to do anything special to provision this bucket - Stedi automatically did that when the SFTP user was created in the previous step.

We will need the bucket name for the following steps. In order to retrieve it, we can take a look at the SFTP users overview.

3. Creating the Stedi Function

Now let's create the Stedi Function. First, navigate to the Stedi UI using this link. First off, click on "Create Function" to create a new Stedi Function. Provide the function with a meaningful name, such as "ConverterFunction".

While Stedi Functions has an online code editor, we will need to use the Stedi API to create this function. Our function will require the Stedi SDK to be included and this can only be achieved through the API.

To make the deployment easier, we recommend cloning our Stedi Function template from GitHub. The repository contains the source code, helper scripts and sample files that you can use to test the solution end to end.

Go ahead and clone the git repository to your local machine using the following command:

git clone https://github.com/Stedi/starter-kit.git
cd starter-kit/tutorials/edi-to-json

Now you should have the following files in your local folder.

📂edi-to-json
┣ 📂build
┣ 📂samplefiles
┃ ┗ 📜810.edi
┣ 📜deploy.sh
┣ 📜event.json
┣ 📜handler.ts
┣ 📜package.json
┗ 📜readme.md

Let's take a look at the 'handler.ts' file first, which contains the source code for the Stedi Function we will deploy. When a function is invoked, it will execute the code under the 'export async function handler()' section. From there, it will follow these steps:

  1. Check if the input Bucket event is in the "input/" path prefix. If not, throw an error and exit the function. We only want to process new EDI documents that are uploaded to the SFTP server and this way we can prevent infinite invocation loops.
  2. Next, the function will download the EDI document from Buckets to the Function.
  3. Make an API request to EDI core to convert the EDI document to a JSON document.
  4. Write the result JSON back to buckets in the "output/" folder.

The Function will retrieve the Stedi API key and bucket names from the functions environment variables, so we do not need to set these or provide them in the function. After the tutorial is completed, you can modify and extend the function code as you wish to meet your business needs.

Now we will use the deploy.sh script to create our Function to Stedi. There are two things we should do within this script before we can successfully execute it.

  • On line 4, set the Stedi API key to your own.
  • On line 5, set the Stedi Function name that you used before (i.e. ConverterFunction).
  • On line 6, set the Stedi Bucket name that you retrieved before.

After this is done, you can create the new Stedi Function in your account. This deployment step should typically take less than a minute;

bash deploy.sh update

Your code is now deployed to Functions! In one of the next steps, we will create a bucket to function trigger, which will automatically submit a JSON payload with details about the file to be converted to the function.

Before we set that up, we should test manually if the Function can perform the EDI conversion on an SFTP document. We can do this by submitting a JSON payload to the function with the bucket and path of the object.

An example of such a JSON payload is shown below.

{
   "Records": [
       {
           "s3": {
               "bucket": {
                   "name": "bucket" // replace this with your Bucket name
               },
               "object": {
                   "key": "input/850.txt" // replace this with your file name
               }
           }
       }
   ]
}

There are two things we should try now;

  • Upload an EDI document from the repository to your SFTP server. Ensure it is placed in the "input/" folder of the bucket, or the conversion process by the function will be skipped. The name and file extension of the file does not matter, as long as it is placed in the right folder.

  • Next, edit 'event.json' in your local folder and set the 'source' property to the path of the file you uploaded. This is an input payload for our Function which we will use for testing.

You can now try to invoke the Function from CLI, using bash deploy.sh invoke. This should result in a response of the converted EDI document, together with some metadata. If you specified the wrong bucket or path, the response will contain an error.


Click to view the JSON response for a successful request
{
 "headers": {
   "content-type": "application/json"
 },
 "bucketObjectKey": "input/850.txt",
 "bucketObjectBody": "ISA*00*          *00*          *01*040132628ESPR  *12*8002228811     *220423*1754*U*00501*000000529*0*P*>~\nGS*PO*040132628ESPR*8002228811*20220423*1754*000000529*X*005010~\nST*850*0001~\nBEG*00*DS*2697447**20220423~\nPER*OC*JOHN DOE*TE*1111111111~\nTD5****ZZ*FHD~\nN1*SF*ABCD*92*ABCD~\nN1*ST*JOHN DOE*92*DROPSHIP CUSTOMER~\nN3*123 MAIN ST~\nN4*Havre*MT*59501*US~\nPO1*1*1.00*EA*114.17**VC*207940*SK*207940~\nPID*F****Colorful Rug~\nCTT*1~\nAMT*TT*114.17~\nSE*12*0001~\nGE*1*000000529~\nIEA*1*000000529~",
 "statusCode": 200,
 "statusError": null,
 "jediResponse": {
   "interchanges": [
     {
       "interchange_control_header_ISA": {
         "authorization_information_qualifier_01": "no_authorization_information_present_no_meaningful_information_in_i02_00",
         "authorization_information_02": "",
         "security_information_qualifier_03": "no_security_information_present_no_meaningful_information_in_i04_00",
         "security_information_04": "",
         "interchange_id_qualifier_05": "duns_dun_bradstreet_01",
         "interchange_sender_id_06": "040132628ESPR",
         "interchange_id_qualifier_07": "phone_telephone_companies_12",
         "interchange_receiver_id_08": "8002228811",
         "interchange_date_09": "220423",
         "interchange_time_10": "1754",
         "repetition_separator_11": "U",
         "interchange_control_version_number_12": "00501",
         "interchange_control_number_13": "000000529",
         "acknowledgment_requested_14": "no_interchange_acknowledgment_requested_0",
         "interchange_usage_indicator_15": "production_data_P",
         "component_element_separator_16": ">"
       },
       "groups": [
         {
           "functional_group_header_GS": {
             "functional_identifier_code_01": "purchase_order_850_PO",
             "application_senders_code_02": "040132628ESPR",
             "application_receivers_code_03": "8002228811",
             "date_04": "20220423",
             "time_05": "1754",
             "group_control_number_06": "000000529",
             "responsible_agency_code_07": "accredited_standards_committee_x12_X",
             "version_release_industry_identifier_code_08": "005010"
           },
           "transaction_sets": [
             {
               "heading": {
                 "transaction_set_header_ST": {
                   "transaction_set_identifier_code_01": "850",
                   "transaction_set_control_number_02": "0001"
                 },
                 "beginning_segment_for_purchase_order_BEG": {
                   "transaction_set_purpose_code_01": "original_00",
                   "purchase_order_type_code_02": "dropship_DS",
                   "purchase_order_number_03": "2697447",
                   "date_05": "20220423"
                 },
                 "administrative_communications_contact_PER": [
                   {
                     "contact_function_code_01": "order_contact_OC",
                     "name_02": "JOHN DOE",
                     "communication_number_qualifier_03": "telephone_TE",
                     "communication_number_04": "1111111111"
                   }
                 ],
                 "carrier_details_routing_sequence_transit_time_TD5": [
                   {
                     "transportation_method_type_code_04": "mutually_defined_ZZ",
                     "routing_05": "FHD"
                   }
                 ],
                 "party_identification_N1_loop": [
                   {
                     "party_identification_N1": {
                       "entity_identifier_code_01": "ship_from_SF",
                       "name_02": "ABCD",
                       "identification_code_qualifier_03": "assigned_by_buyer_or_buyers_agent_92",
                       "identification_code_04": "ABCD"
                     }
                   },
                   {
                     "party_identification_N1": {
                       "entity_identifier_code_01": "ship_to_ST",
                       "name_02": "JOHN DOE",
                       "identification_code_qualifier_03": "assigned_by_buyer_or_buyers_agent_92",
                       "identification_code_04": "DROPSHIP CUSTOMER"
                     },
                     "party_location_N3": [
                       {
                         "address_information_01": "123 MAIN ST"
                       }
                     ],
                     "geographic_location_N4": [
                       {
                         "city_name_01": "Havre",
                         "state_or_province_code_02": "MT",
                         "postal_code_03": "59501",
                         "country_code_04": "US"
                       }
                     ]
                   }
                 ]
               },
               "detail": {
                 "baseline_item_data_PO1_loop": [
                   {
                     "baseline_item_data_PO1": {
                       "assigned_identification_01": "1",
                       "quantity_02": "1.00",
                       "unit_or_basis_for_measurement_code_03": "each_EA",
                       "unit_price_04": "114.17",
                       "product_service_id_qualifier_06": "vendors_sellers_catalog_number_VC",
                       "product_service_id_07": "207940",
                       "product_service_id_qualifier_08": "stock_keeping_unit_sku_SK",
                       "product_service_id_09": "207940"
                     },
                     "product_item_description_PID_loop": [
                       {
                         "product_item_description_PID": {
                           "item_description_type_01": "free_form_F",
                           "description_05": "Colorful Rug"
                         }
                       }
                     ]
                   }
                 ]
               },
               "summary": {
                 "transaction_totals_CTT_loop": [
                   {
                     "transaction_totals_CTT": {
                       "number_of_line_items_01": "1"
                     },
                     "monetary_amount_information_AMT": {
                       "amount_qualifier_code_01": "total_transaction_amount_TT",
                       "monetary_amount_02": "114.17"
                     }
                   }
                 ],
                 "transaction_set_trailer_SE": {
                   "number_of_included_segments_01": "12",
                   "transaction_set_control_number_02": "0001"
                 }
               },
               "set": "850"
             }
           ],
           "functional_group_trailer_GE": {
             "number_of_transaction_sets_included_01": "1",
             "group_control_number_02": "000000529"
           },
           "release": "005010"
         }
       ],
       "interchange_control_trailer_IEA": {
         "number_of_included_functional_groups_01": "1",
         "interchange_control_number_02": "000000529"
       },
       "delimiters": {
         "element": "*",
         "segment": "~",
         "sub_element": ">"
       }
     },
     {
       "interchange_control_header_ISA": {},
       "groups": [
         {
           "functional_group_header_GS": {},
           "transaction_sets": [
             {
               "detail": {
                 "not_in_spec_1_200_200": {}
               },
               "set": "810"
             }
           ],
           "functional_group_trailer_GE": {},
           "release": "005010"
         }
       ],
       "interchange_control_trailer_IEA": {},
       "delimiters": {
         "element": "*",
         "segment": "~",
         "sub_element": ">"
       }
     }
   ],
   "__version": "jedi@2.0"
 }
}

Besides the JSON output on your terminal, you should also see a new file appear in your SFTP folder under the "output/" directory. Here, we will store the results from the conversion. You or your trading partner could retrieve this file from a bucket directly or through SFTP.

For testing purposes, we triggered the function manually to perform a conversion. In the next step, we will show you how to automate this process every time a new bucket object is uploaded.

4. Connecting the Bucket to the Function

We provide a simple command to connect the bucket to the Function. By running 'bash deploy.sh bucket_enable', we can create a binding between a new bucket upload and a function trigger. This trigger will start when a new function invocation once a file is uploaded to the "input/" directory on SFTP. For any other operation or directory, it will skip the conversion.

NOTE: It is very important that you avoid creating an infinite loop of bucket to Function invocations.

Stedi creates a new function invocation every time a file is uploaded to the bucket. If this invocation writes back a file to the same directory, this can lead to an infinite loop of invocations.

We strongly recommend to do one of the following;

  • Write the results from your invocation to a different bucket than the source bucket.
  • In the Function code, check whether the file is written to the "input/" folder and only write results to the "output/" folder.

We have implemented the second option in this tutorial.

Please go ahead and run the following command to set up the trigger.

bash deploy.sh bucket_enable

To stop the trigger or temporarily pause it, run the following command.

bash deploy.sh bucket_disable

5. Testing the end to end flow

We're nearing the finish line and can create an end to end test now. Instead of having to run a manual invoke like before, we can simply upload a file to SFTP in the inbound folder. A few seconds later, we should see the converted output in the "output/" folder.

Try doing this now with a new EDI file and see if it works as expected. We provide several EDI files for testing in the "samples" folder that you can use. You can of course also pick your own EDI documents and see if this works.

Final thoughts

Congratulations on building your first Stedi workflow! We hope you now have a better idea how to build event-driven workflows on our platform.

IntroductionPrerequisites for tutorialDeploying to StediFinal thoughts