Automated employee data upload playbook with AWS Lambda & S3

This provides a step-by-step guide to deploying an AWS Lambda function that automatically reads entitlement data from an S3 bucket and uploads it to BalkanID using our employee upload API.

Note: To get your tenant's BalkanID API key and Secret, please follow steps here.

Overview

When a CSV file is uploaded to a specified Amazon S3 bucket, the Lambda function is triggered automatically. The function:

1. Retrieves the uploaded file from S3.

2. Transforms the uploaded file to BalkanID employee CSV format specified in this page.

3. Requests a pre-signed upload URL from BalkanID Employee Upload API.

4. Uploads the employee data CSV to BalkanID.

5. Logs the process and handles errors.

This ensures seamless integration and real-time entitlement updates without manual intervention.

Prerequisites

Before setting up the Lambda function, ensure you have:

• An AWS S3 bucket to store employee data.

• An IAM role with necessary permissions.

• BalkanID API credentials (X-Api-Key-ID, X-Api-Key-Secret).

• Basic knowledge of AWS Lambda, IAM, and S3.

Create an S3 Bucket

If you do not have an S3 bucket, create one using the AWS CLI or AWS Console.

aws s3 mb s3://your-bucket-name

Create an IAM Role for Lambda

Lambda needs permission to read files from S3 and write logs to CloudWatch.

Create an IAM Role

1. Go to AWS IAM Console → Roles → Create Role.

2. Select AWS Service → Lambda.

3. Attach the following policies:

   • AmazonS3ReadOnlyAccess (allows Lambda to read from S3)

   • AWSLambdaBasicExecutionRole (for logging to CloudWatch)

4. Click Create Role and copy the Role ARN.

Deploy the AWS Lambda Function

Create the Lambda Function

1. Go to AWS Lambda Console → Create Function.

2. Select Author from scratch.

3. Configure:

   • Function Name: s3-to-balkanid

   • Runtime: Python 3.x

   • Role: Select the IAM role you created earlier.

4. Click Create Function.

Add an S3 Trigger

Configure S3 Event Notification

1. Open your S3 bucket in the AWS Console.

2. Go to Properties → Event Notifications.

3. Click Create Event Notification:

   • Name: TriggerLambdaOnUpload

   • Event types: Select PUT

   • Prefix: (Optional - If you only want to trigger on specific files/folders)

   • Destination: Select Lambda Function

   • Choose your Lambda function (s3-to-balkanid)

4. Click Save.

Lambda Function Code

Replace this code in the Lambda Function Code section. Include any transformation necessary to transform your CSV format to BalkanID employee CSV format as part of the lambda_handler function.

Python

import json
import os
import boto3
import requests

# Initialize S3 client
s3_client = boto3.client("s3")

# BalkanID API credentials (set in AWS Lambda environment variables)
BALKANID_API_KEY_ID = os.getenv("BALKANID_API_KEY_ID")
BALKANID_API_KEY_SECRET = os.getenv("BALKANID_API_KEY_SECRET")
BALKANID_UPLOAD_URL = "https://app.balkan.id/api/rest/v0/employees/upload-url"


def get_presigned_url():
    """
    Fetches a pre-signed upload URL from BalkanID API.
    """
    headers = {
        "X-Api-Key-ID": BALKANID_API_KEY_ID,
        "X-Api-Key-Secret": BALKANID_API_KEY_SECRET,
        "Content-Type": "application/json",
    }

    response = requests.get(BALKANID_UPLOAD_URL, headers=headers)

    if response.status_code == 200:
        return response.json().get("upload_url")
    else:
        raise Exception(f"Failed to get upload URL: {response.text}")


def upload_to_balkanid(pre_signed_url, file_data):
    """
    Uploads the CSV file to the pre-signed URL.
    """
    headers = {"Content-Type": "text/csv"}

    response = requests.put(pre_signed_url, data=file_data, headers=headers)

    if response.status_code == 200 or response.status_code == 204:
        return "Upload successful"
    else:
        raise Exception(f"Upload failed: {response.text}")


def lambda_handler(event, context):
    """
    AWS Lambda function triggered by an S3 event.
    """
    try:
        # Extract bucket and file info from S3 event
        bucket_name = event["Records"][0]["s3"]["bucket"]["name"]
        file_key = event["Records"][0]["s3"]["object"]["key"]

        # Download the CSV file from S3
        s3_object = s3_client.get_object(Bucket=bucket_name, Key=file_key)
        file_data = s3_object["Body"].read()

        # Get pre-signed upload URL from BalkanID
        pre_signed_url = get_presigned_url()

        # Upload the file to BalkanID
        upload_status = upload_to_balkanid(pre_signed_url, file_data)

        return {
            "statusCode": 200,
            "body": json.dumps({"message": upload_status}),
        }

    except Exception as e:
        return {
            "statusCode": 500,
            "body": json.dumps({"error": str(e)}),
        }

Set Environment Variables

In AWS Lambda Console, navigate to:

• Configuration → Environment variables → Edit → Add variables:

Key	Value
BALKANID_API_KEY_ID	Your API Key ID
BALKANID_API_KEY_SECRET	Your API Key Secret

Test the Setup

Upload a File to S3

Run this command to manually upload a test employee CSV file:

aws s3 cp employee.csv s3://your-bucket-name/

This should automatically trigger the Lambda function, processing the file and uploading it to BalkanID.

Check CloudWatch Logs

If debugging is required, view logs in AWS CloudWatch:

1. Go to AWS CloudWatch Console.

2. Select Logs → Lambda → Find your Lambda function logs.

3. Check for any errors or confirmation messages.