StediDOCS

Inbound EDI using Terraform

Updated June 22, 2022

Introduction

The goal of this tutorial is to deploy a serverless system that ingests an EDI file and outputs a transformed JSON. Luckily we have Stedi to do the transforming from EDI into a JSON. For a deeper overview, visit Introduction to Stedi.

What we will build

In this tutorial, we provide a reference implementation that deploys to AWS cloud using Terraform.

At a high level, this implementation...

  1. Gets an EDI file as input data.

  2. Translate the EDI to JSON EDI (JEDI).

  3. Maps that JEDI to a specific JSON shape for an API.

  4. Write that JSON back into S3.

You'll need:

  • The Terraform CLI (1.0.1+) installed [^1]
  • An AWS account - you may incur minor charges (less than $1) for this exercise.
  • The AWS CLI (2.0+) configured to your account
  • A Stedi account and API key

Breakdown of Stedi services we will use

  1. EDI Core (the service that translates EDI files to JSON EDI (JEDI))
  2. Mappings (the service that maps one JSON into another JSON).

Our system is set up so when an EDI file loads in the S3 bucket, S3 will trigger the Lambda that will get the file, transform it by making HTTP requests to Stedi services, and places it back into S3.

Architecture for tutorial

The step-by-step data flow

AWS can be intimidating. Let’s make it easier by illustrating what we’re going to accomplish.

Flow chart of requests through architecture

We have several things to create in AWS. We’ll use Terraform to define them. They can be found in the Github repo under main.tf

For more details see the AWS Component breakdown in the Appendix.

Deploying

Setup infrastructure using Terraform

We are starting with a skeleton repository. Download or clone it from: https://github.com/Stedi/starter-kit

The project structure is:

├── scripts
│   └── setup.sh    // Shell script for creating lambda .zip
├── index.js        // Main Lambda code that does all the magic
│
├── main.tf     // Core terraform infrastructure is written here
├── variables.tf    // String variables for terraform
│
├── 850.edi     // Our X12 850 EDI file
│
└── package.json    // (To download @axios when we run npm install)
A. Detailed breakdown of Terraform code

Let’s focus on `main.tf`.

In section I, we’re defining some providers that help terraform run. The required_provider aws lets us deploy to AWS, and the random provider is a helper to create random text.

# I. Terraform specific
terraform {
 required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "= 3.48.0"
   }
   random = { # This gives us random strings, it's useful below.
     source  = "hashicorp/random"
     version = "= 3.1.0"
   }
 }

 required_version = "~> 1.0"
}

provider "aws" {
 region = var.aws_region
}

In section II, we define the main role for the Lambda function. We attach two policies to that role, one for writing to CW log groups (declared below), and for read/write access to S3. Last, we give the S3 principal the ability to trigger our Lambda (we’re permitting the S3 service itself).

# II. Lambda

# Create a role for lambda
resource "aws_iam_role" "iam_for_lambda" {
 name = "iam_for_lambda"

 assume_role_policy = jsonencode({
   Version = "2012-10-17"
   Statement = [{
     Action = "sts:AssumeRole"
     Effect = "Allow"
     Sid    = ""
     Principal = {
       Service = "lambda.amazonaws.com"
     }
     }
   ]
 })
}

# Attaches a policy that allows writing to CW Logs, for the role
resource "aws_iam_role_policy_attachment" "lambda_policy" {
 role       = aws_iam_role.iam_for_lambda.name
 policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

# Ataches a policy that allows read/write to S3, for the role
resource "aws_iam_role_policy_attachment" "lambda_s3_policy" {
 role       = aws_iam_role.iam_for_lambda.name
 policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}

# The CW Log group itself
resource "aws_cloudwatch_log_group" "stedi_lambda" {
 name = "/aws/lambda/${aws_lambda_function.stedi_lambda.function_name}"
}

# Another permission so the S3 bucket can trigger the Lambda.
resource "aws_lambda_permission" "allow_bucket" {
 statement_id  = "AllowExecutionFromS3Bucket"
 action        = "lambda:InvokeFunction"
 function_name = aws_lambda_function.stedi_lambda.arn
 principal     = "s3.amazonaws.com"
 source_arn    = aws_s3_bucket.bucket.arn
}

Then we define our core Lambda. We define our runtime as nodejs. The handler needs to be the name of your main code file. If our main file was named “bob”, we’d set this to be handler = “bob.handler”. In our case, it’s index.js so we set handler = “index.handler.

Next, we’ve uploaded our Lambda code as a .zip file. There are other ways (using terraform archives), but this is a simpler. We’ll make sure our Lambda code in index.js is .zipped up and in the /tmp/ directory for this to run. The scripts/setup.sh script does this for us.

We also define a source_code_hash which checks if the .zip file is has changed, requiring a redeployment.

Last, our Lambda calls Stedi APIs. Those calls need some configuration as environment variables so we can pass secrets and strings to the Lambda runtime.

# MAIN LAMBDA - Resources defines lambda using source code uploaded to S3 in .zip.
resource "aws_lambda_function" "stedi_lambda" {
 function_name = "StediLambda"

 runtime = "nodejs12.x"
 handler = "index.handler"

 # Where the setup.sh puts the Lambda file .zip
 filename = "/tmp/index.zip"

 # This auto-trigger Lambda updates whenever we change the code!
 source_code_hash = filebase64sha256("/tmp/index.zip")

 timeout = 30 # lambda timeout to 30 seconds.

 # Important environment variables for calling Stedi APIs
 environment {
   variables = {
     stedi_api_key = var.stedi_api_key,
     stedi_mapping_id = var.stedi_mapping_id
   }
 }

 role = aws_iam_role.iam_for_lambda.arn
}

In section III, we define a bucket to be created with a random (and sometimes comical) name such as panda-obviously-moved-zebra. Because S3 names are global the name must be unique.

# III. S3
resource "random_pet" "random_bucket_name" {
 prefix = var.project_name
 length = 3
}

resource "aws_s3_bucket" "bucket" {
 bucket = random_pet.random_bucket_name.id

 acl           = "private"
 force_destroy = true
}

resource "aws_s3_bucket_notification" "bucket_notification" {
 bucket = aws_s3_bucket.bucket.id

 lambda_function {
   lambda_function_arn = aws_lambda_function.stedi_lambda.arn
   events              = ["s3:ObjectCreated:*"]
   filter_prefix       = "inbound/"
   filter_suffix       = ".edi"
 }

 depends_on = [aws_lambda_permission.allow_bucket]
}

Most importantly, we define an S3 bucket notification to trigger our Lambda. ["s3:ObjectCreated:*"] means it will trigger only on object creation events. And this notification triggers only in the inbound/ folder for files with the .edi ending. So a file in inbound/my-850-file.edi will invoke our Lambda.

Note: when S3 notifications trigger a Lambda, it also passes the following object:

"Records": [
  {
    "eventVersion": "2.1",
    "eventSource": "aws:s3",
    "awsRegion": "us-east-1",
    "eventTime": "2021-12-17T19:04:15.899Z",
    "eventName": "ObjectCreated:Put",
    ...
    "s3": {
      "s3SchemaVersion": "...",
      "configurationId": "trigger-lambda",
      "bucket": {
        "name": "panda-obviously-moved-zebra",          // <- Bucket name
        "ownerIdentity": {
          "principalId": "..."
        },
        "arn": "arn:aws:s3::panda-obviously-moved-zebra"
      },
      "object": {
        "key": "inbound/my-850-file.edi",           // <- File name
        "size": 393,
        "eTag": "...",
        "sequencer": "..."
      }
    }
  }
]

This makes it easy. The Lambda gets the name the file when it’s triggered so it knows what filename to read from S3.


B. Detailed breakdown of Lambda code

Let’s focus on `index.js`.

This is the main Lambda function. It has four main sections.

const AWS = require("aws-sdk");
const s3 = new AWS.S3();
const axios = require("axios");

// Get environment variables
const apiKey = process.env.stedi_api_key;
const mapId = process.env.stedi_mapping_id;

// Create axios client to Stedi services
const axiosClient = axios.create({
  headers: {
    Authorization: `Key ${apiKey}`,
    "Content-Type": "application/json",
  },
});

module.exports.handler = async (event, context) => {
  console.log("Event: ", JSON.stringify(event));

  try {
    // 1. Get file from S3 bucket
    const Bucket = event.Records[0].s3.bucket.name;
    const data = await getFileFromS3(Bucket, event);

    // 2. Call /translate to transform 850 EDI -> JEDI 850
    const translateResponse = await translateEDI(data.Body.toString("ascii"));

    // 3. Call /map to transform JEDI 850 -> Purchase Order JSON
    const mappingResponse = await mapJEDI(translateResponse.jedi);

    // 4. Put the Purchase Order JSON into S3 under the folder orders/
    const writeResult = await putFileIntoS3(
      Bucket,
      mappingResponse.purchase_order
    );

    // 5. End lambda
    return {
      statusCode: 200,
      body: JSON.stringify({
        message: "Stedi Lambda succeeded!",
        result: writeResult,
      }),
    };
  } catch (e) {
    return {
      statusCode: 500,
      body: JSON.stringify({
        message: e,
      }),
    };
  }
};

Section 1. uses the Bucket and Filename to get the data inside the file from S3 using the S3 SDK call getObject()

async function getFileFromS3(Bucket, event) {
  try {
    const Key = decodeURIComponent(
      event.Records[0].s3.object.key.replace(/\+/g, " ")
    );
    return await s3.getObject({ Bucket, Key }).promise();
  } catch (e) {
    console.log(e);
  }
}

Section 2. The file is read, and the below request is created to the shape of the /translate API from Stedi.

Note: the input field is the X12 850 EDI.

{
 "input_format": "edi",
 "input": "ISA*00*          *00*          *ZZ*STEDI          *ZZ*ACME           *210901*1234*U*00801*000000001*0*T*>~GS*PO*SENDERGS*007326879*20210901*1234*1*X*008020~ST*850*000000001~BEG*24*SP*PO-00001**20210901~N1*2L*STEDI INC.~REF*K6*A composable platform for building flexible EDI systems~PER*SR**EA*team@stedi.com~PO1**1*2P*0.0001*PE*GE*EDI Core~PO1**1*C0*0.05*PE*GE*Mappings~CTT*2~SE*9*000000001~GE*1*1~IEA*1*000000001~",
 "output_format": "jedi@2.0-beta"
}

This POST request is sent to the /translate endpoint.

async function translateEDI(edi_data) {
  try {
    const response = await axiosClient.request({
      method: "POST",
      url: "https://edi-core.stedi.com/2021-06-05/translate",
      data: {
        input_format: "edi",
        input: edi_data,
        output_format: "jedi@2.0-beta",
      },
    });
    const jedi = response.data;
    console.log("/translate RESP", JSON.stringify(jedi, null, 2));

    return {
      statusCode: 200,
      jedi,
    };
  } catch (e) {
    console.log(e);
  }
}

Section 3, the response from /translate is received (see below). This is an X12 850 JEDI; a lossless representation of the X12 850 EDI data.

Click to see the 850 JEDI
```json
{
  "code": "valid",
  "output": {
    "interchanges": [
      {
        "interchange_control_header_ISA": {
          "authorization_information_qualifier_01": "no_authorization_information_present_no_meaningful_information_in_i02_00",
          "authorization_information_02": "",
          "security_information_qualifier_03": "no_security_information_present_no_meaningful_information_in_i04_00",
          "security_information_04": "",
          "interchange_id_qualifier_05": "mutually_defined_ZZ",
          "interchange_sender_id_06": "STEDI",
          "interchange_id_qualifier_07": "mutually_defined_ZZ",
          "interchange_receiver_id_08": "ACME",
          "interchange_date_09": "210901",
          "interchange_time_10": "1234",
          "repetition_separator_11": "U",
          "interchange_control_version_number_code_12": "00801",
          "interchange_control_number_13": "000000001",
          "acknowledgment_requested_code_14": "no_interchange_acknowledgment_requested_0",
          "interchange_usage_indicator_code_15": "test_data_T",
          "component_element_separator_16": ">"
        },
        "groups": [
          {
            "functional_group_header_GS": {
              "functional_identifier_code_01": "purchase_order_850_PO",
              "application_senders_code_02": "SENDERGS",
              "application_receivers_code_03": "007326879",
              "date_04": "20210901",
              "time_05": "1234",
              "group_control_number_06": "1",
              "responsible_agency_code_07": "accredited_standards_committee_x12_X",
              "version_release_industry_identifier_code_08": "008020"
            },
            "transaction_sets": [
              {
                "heading": {
                  "transaction_set_header_ST": {
                    "transaction_set_identifier_code_01": "850",
                    "transaction_set_control_number_02": "000000001"
                  },
                  "beginning_segment_for_purchase_order_BEG": {
                    "transaction_set_purpose_code_01": "draft_24",
                    "purchase_order_type_code_02": "sample_SP",
                    "purchase_order_number_03": "PO-00001",
                    "date_05": "20210901"
                  },
                  "party_identification_N1_loop": [
                    {
                      "party_identification_N1": {
                        "entity_identifier_code_01": "corporation_2L",
                        "name_02": "STEDI INC."
                      },
                      "reference_information_REF": [
                        {
                          "reference_identification_qualifier_01": "purchase_description_K6",
                          "reference_identification_02": "A composable platform for building flexible EDI systems"
                        }
                      ],
                      "administrative_communications_contact_PER": [
                        {
                          "contact_function_code_01": "sales_representative_or_department_SR",
                          "communication_number_qualifier_03": "internet_email_address_EA",
                          "communication_number_04": "team@stedi.com"
                        }
                      ]
                    }
                  ]
                },
                "detail": {
                  "baseline_item_data_PO1_loop": [
                    {
                      "baseline_item_data_PO1": {
                        "quantity_02": "1",
                        "unit_or_basis_for_measurement_code_03": "kilobyte_2P",
                        "unit_price_04": "0.0001",
                        "basis_of_unit_price_code_05": "price_per_each_PE",
                        "product_service_id_qualifier_06": "generic_name_description_GE",
                        "product_service_id_07": "EDI Core"
                      }
                    },
                    {
                      "baseline_item_data_PO1": {
                        "quantity_02": "1",
                        "unit_or_basis_for_measurement_code_03": "calls_C0",
                        "unit_price_04": "0.05",
                        "basis_of_unit_price_code_05": "price_per_each_PE",
                        "product_service_id_qualifier_06": "generic_name_description_GE",
                        "product_service_id_07": "Mappings"
                      }
                    }
                  ]
                },
                "summary": {
                  "transaction_totals_CTT_loop": [
                    {
                      "transaction_totals_CTT": {
                        "number_of_line_items_01": "2"
                      }
                    }
                  ],
                  "transaction_set_trailer_SE": {
                    "number_of_included_segments_01": "9",
                    "transaction_set_control_number_02": "000000001"
                  }
                },
                "set": "850"
              }
            ],
            "functional_group_trailer_GE": {
              "number_of_transaction_sets_included_01": "1",
              "group_control_number_02": "1"
            },
            "release": "008020"
          }
        ],
        "interchange_control_trailer_IEA": {
          "number_of_included_functional_groups_01": "1",
          "interchange_control_number_02": "000000001"
        },
        "delimiters": {
          "element": "*",
          "segment": "~",
          "sub_element": ">"
        }
      }
    ],
    "__version": "jedi@2.0-beta"
  }
}
```

Then the mapJEDI(jedi_data) function receives and sends a second POST request to the /map API from Stedi. \ Note: the mapId is taken from the environment variables. This means the mapping must already be created in the Stedi Terminal.

async function mapJEDI(jedi_data) {
  try {
    const response = await axiosClient.request({
      method: "POST",
      url: "https://mappings.stedi.com/2021-06-01/mappings/" + mapId + "/map",
      data: jedi_data,
    });

    const purchase_order = response.data;
    console.log("/map RESP", JSON.stringify(purchase_order, null, 2));

    return {
      statusCode: 200,
      purchase_order,
    };
  } catch (e) {
    console.log(e);
  }
}

And in return, we get back a much simpler JSON that we’ll call our purchase_order.

{
 "po_number": "PO-00001",
 "sender_id": "STEDI",
 "interchange_receiver_id_08": "ACME"
}

Finally, in Section 4 we will write the purchase_order JSON back into S3 in a folder named orders/. The name of the file will be the po_number joined with a random number. E.g. PO-00001-98.json. This is written to S3 using the S3 SDK call putObject().

async function putFileIntoS3(Bucket, purchase_order) {
  try {
    const objectName = `orders/${await purchase_order.po_number}-${Math.floor(
      Math.random() * 100
    )}.json`;

    const stringifiedData = JSON.stringify(purchase_order, null, 2); // Need this because putObject only accepts string type
    const s3PutObjectParams = {
      Bucket,
      Key: objectName,
      Body: stringifiedData,
      ContentType: `application/json`,
    };
    const result = await s3.putObject({ ...s3PutObjectParams }).promise();
    console.log(
      `File uploaded successfully at https:/` +
        Bucket +
        `.s3.amazonaws.com/` +
        objectName
    );
  } catch (e) {
    console.log(e);
  }
}

Steps

In Stedi

First, create a Stedi API Key by following Stedi Authentication Guide. It'll look like

abcd1234.XXXXXXXXXXXXXXXXXXXXXXXX

Then, create a Stedi Mapping. You can create a pre-built Mapping for this tutorial by clicking:

Run on Stedi

Record the Mapping ID, it'll look like:

image11

In your CLI

Run

$ git clone https://github.com/Stedi/starter-kit.git

Then change directory and execute the setup script which runs neccessary Terraform steps.

$ cd integration-tutorials/stedi-inbound-transformation-aws-terraform
$ bash /scripts/setup.sh

Now to deploy the stack First, confirm your AWS profile and IAM access keys are configured for the AWS CLI, then run

$ terraform apply

Enter your Stedi API Key, and Stedi Mapping ID when prompted. You'll see

prompt

Enter “yes”. This will deploy to your AWS account. If you encounter any failures, they are likely due to a misconfiguration of your AWS profile in the CLI.

On success, you'll see:

image4

Note: your S3 bucket name. It will be different than our example panda-smoothly-open-teal.

In AWS Console

Navigate to the S3 Console and create a folder called inbound/ within the S3 bucket. Inside the inbound/ folder upload the 850.edi file. Get this file from inside the cloned project repo.

image1

Go back to the root of the bucket and click Refresh. You'll see a new folder called orders/.

image9

Inside that folder, you'll find a JSON file that's been created by our Lambda.

image3

Opening that file, we can confirm it looks like the JSON payload we were expecting.

image8

Congratulations, you completed the tutorial succesfully!

Final thoughts

In order to monitor if the solution is running correctly, you can take a look at the CloudWatch Metrics of the Lambda function. Here, you can see how many successful invocations happened and inspect the duration of these requests.

This concludes our workshop, but please feel free to extend and modify this stack further. You can try to process your own EDI files with the solution and explore different mappings that can be made.

Cleaning up the resources on AWS

If you’d like to tear down and clean up your AWS account, run

$ terraform destroy

and it will remove everything including the data in the files.

We hope you enjoyed this tutorial on how to implement a workflow on AWS and calling Stedi to do data transformations, and we hope this makes the journey towards implementation clearer.

As always, Stedi is evolving, so if you have any thoughts or feedback we would love to hear from you!

Appendix A

Infrastructure components needed

[^1]: Why Terraform and not CDK for deploying to AWS? Terraform is an Infrastructure-as-Code (IaC) tool that allows you to manage infrastructure with configuration files rather than a GUI. While these IaC tools exist for specific cloud providers, Terraform is cloud provider agnostic and with minor changes can be modified to deploy to Azure, GCP, Alibaba Cloud, Heroku, and Oracle Cloud.