Data Science in Drilling - Episode 11

Create and Deploy Customized AWS Lambda Functions Through Docker

written by Zeyu Yan, Ph.D., Head of Data Science from Nvicta AI

Data Science in Drilling is a multi-episode series written by the technical team members in Nvicta AI. Nvicta AI is a startup company who helps drilling service companies increase their value offering by providing them with advanced AI and automation technologies and services. The goal of this Data Science in Drilling series is to provide both data engineers and drilling engineers an insight of the state-of-art techniques combining both drilling engineering and data science.

This is another episode about Docker and AWS Lambda function. Hope you enjoy it. :)

Enjoying great knowledge is just like enjoying delicious ramen.

Introduction

In the second episode of this series, we have already covered how to deploy a simple AWS Lambda function. In this episode, we will cover how to deploy Lambda functions through customized Docker images, which significantly improves the customizability of Lambda functions. With this method, any Lambda functions with complex dependencies could be easily deployed.

What We'll Cover Today

1. How to create and deploy customized Lambda functions through Docker.

2. How to test the customized Lambda functions locally.

3. How to test the deployed customized Lambda functions.

Create Customized Lambda Function Through Docker

First, create an empty Python project with virtual environment and series of empty files as follows:

Let's install numpy as the only dependency of this project as follows:

pip install numpy

Then generate the requirements.txt file through the following command:

pip freeze > requirements.txt

The updated project structure is as follows:

Now let's fill in the empty files we created earlier. Fill in the .dockerignore file with the following contents:

.gitignore
.git
README.md
.idea/
__pycache__/
.DS_Store
venv/
.ipynb_checkpoints/

Fill in the Dockerfile file with the following contents:

FROM python:3.8.6

RUN apt-get update && \
    apt-get install -y \
    autoconf \
    libtool \
    g++ \
    make \
    cmake \
    unzip \
    libcurl4-openssl-dev

ARG FUNCTION_DIR="/function"
WORKDIR ${FUNCTION_DIR}
COPY . .

RUN pip install awslambdaric

RUN pip install --upgrade pip setuptools wheel
RUN pip install -r requirements.txt

ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
COPY entry.sh /
RUN chmod 755 /usr/bin/aws-lambda-rie /entry.sh
ENTRYPOINT [ "/entry.sh" ]
CMD [ "app.handler" ]

Fill in the entry.sh file with the following contents:

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
    exec /usr/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $1
else
    exec /usr/local/bin/python -m awslambdaric $1
fi

Finally, fill in the app.py file:

import traceback
import numpy as np


def handler(event, context):
    try:
        count = int(event['count'])
        result = int(np.random.randint(10, size=1)[0])
        count += result
        print(f'count: {count}')
        return {
            'flag': 1,
            'result': result,
            'current_count': count
        }
    except Exception as e:
        traceback.print_exc()
        return {
            'flag': 0,
            'message': f'Error happened due to {str(e)}'
        }

What our Lambda function does is quite simple. It receives the count from the request, add it with a random integer between 0 and 9, and finally returns a response contains both the random integer and the updated count.

Test the Lambda Function Locally

We have finished creating our customized Lambda function. Let’s first test it locally to make sure it's bug free before deployment. Inside the project root folder, use the following command to build the Docker container:

docker build -t lambda_docker:latest .

Then use the following command to run the Docker container on your local machine:

docker run -p 9000:8080  lambda_docker:latest

Check the status of your Docker containers using the following command in a separate terminal:

docker ps

One should see something like this:

Run the following Python script to test the Lambda function locally:

import requests

url = 'http://localhost:9000/2015-03-31/functions/function/invocations'
data = {'count': 1}
r = requests.post(url, json=data)
print(r.json())

The result is:

{'flag': 1, 'result': 3, 'current_count': 4}

Deploy the Lambda Function

To deploy the Lambda function, we first need to create a repository using the AWS Elastic Container Registry service. To start, select the Elastic Container Registry service from AWS Console Home:

Then click on the Create repository button on the top right corner:

Provide a name for your repository and leave all the other settings as default. Create the repository.

Go to the created repository and click on the View push commands button on the top right corner:

This will show you all the necessary commands to follow to push your customized Docker image to this repository:

After pushing and refreshing the page, our customized Docker image will appear in the repository. The image URI cannot be copied.

Next, select the Lambda service from AWS Console Home:

Then click on the Create function button on the top right corner:

Choose the Container image option:

Then provide a name for the Lambda function and paste the container image URI. Leave all the other settings as default and create the function.

Your customized Lambda function should be up running now.

Test the Deployed Lambda Function

To test the deployed Lambda function, select the Test option in the Lambda function page:

Create an new test event, edit the event json and click on the Test button.

Here is the execution result:

We have also covered how to invoke Lambda functions through Python scripts in the first and second episodes of this series. Please refer to these two episodes for more details.

Conclusions

In this article, we went through how to create, deploy and test customized Lambda functions built through Docker. Hope you enjoy it! More skills about Docker and other AWS services will be covered in the future episodes. Stay tuned!

Get in Touch

Thank you for reading! Please let us know if you like this series or if you have critiques. If this series was helpful to you, please follow us and share this series to your friends.

If you or your company needs any help on projects related to drilling automation and optimization, AI, and data science, please get in touch with us Nvicta AI. We are here to help. Cheers!