Hosting websites in Amazon - Setting up URLs and certificates

Developing an application is the part that takes all the interest from any developer; I mean, the name kind of implies that right?. However, a complete development process does not finish when the team write the last line of code, we still have to make the deployment.

During most of the process, we can use simpler configurations. For example, we can use the default links provided when we host in a S3 bucket, if we are using an EC2 instance we can use the direct IP, or even use a load balancer that points to a set of Docker instances. However, in all of these cases, the URLs are not friendly since they are autogenerated by Amazon.

This post will explain how to do a proper deployment of an application for a production-level environment. For this, we will use the following technologies:

  1. S3 bucket: where the application code will be hosted. We won’t run through this during this post, but you can check my Amplify posts to see how to do it.

  2. Cloudfront: Content delivery network service

  3. Route53: domain registration service

  4. Certificate Manager: SSL certificates provider

Requesting the certificate

As a prerequisite for this part, we will need to create a hosted zone in Route 53 where the certificate will be associated. For that, go to the Route 53 service, in the left menu select hosted zones, and create one if you don’t have already.

Route 53 - hosted zone

The first step that we will cover is requesting the SSL certificate. For this, we will go to the Certificate Manager service in Amazon and click on “Request a Certificate” and select a public certificate.

Certificate - Step 1

The next step will ask us to register all the domain names that we want the certificate to be used for.

Certificate - step 2

Important: we can use a wildcard to use multiple subdomains. For example, you can type *.mydomain.com, and it will cover cases like admin.mydomain.com.

After setting the domain names, it will ask us to select the verification method of the certificate. Since we are going to use only Amazon services, let’s select DNS validation.

Certificate - step 3

Finally, we will review all the selections and continue with the wizard. After the request has been created, we need to validate and create the domain for the certificate in Route 53, however, this is an automated step that the Certificate Manager can do.

Certificate - step 4

In the dashboard for the certificate manager, you will see the certificate that you just created, and inside of it, it will prompt you to create the DNS records in Route 53.

Certificate - step 5

Creating the Cloudfront distribution

Now that we have our certificate set up and ready to be used, we need to create the cloud front distribution that will point to our S3 bucket. For that, we will go to the Cloud front service dashboard and click on “Create distribution”.

For our use case, we will select “Web” as the delivery method (this is the first screen that will show). From there, we will be presented a big form with a lot of options, but for most of them we will leave the default values except in 3 specific areas.

First, we need to select the S3 bucket where the application is hosted as the “Origin Domain Name” (this field is a dropdown, even though it doesn’t look like it).

Cloudfront - step 1

The second change that we will make to the form is under the “Default Cache Behavior Settings” section. In there, we will select “Redirect HTTP to HTTPS” in the viewer protocol policy option. This will automatically redirect the user to the HTTPS site, even if they don’t specify it in the address bar.

Cloudfront - step 2

Finally, under the “Distribution Settings” section, we will select the certificate created previously.

Cloudfront - step 3

After all of that, click on “Create Distribution" button at the end of the screen and wait some time while Amazon provision all the resources for it.

Important: when you are using React or Angular applications, we need to set the default root object to be the index.html file. For that, in the same form, a couple of fields after selecting the certificate, it can be specified. If you didn’t set it up the first time, you can modify the distribution after its creation.

Create the actual URL

Previously, we used briefly Route53 to validate the certificate request, however this time we will create the real URL that our application will use. So, we go to the Route53 service in amazon and in the left bar menu click on Hosted zones.

From the hosted zone that you created before, we will add a new record set. If all the previous steps were executed successfully, there should already be one record set for the SSL certificate.

In the new record set, we will type the name of the URL that we want to use, for example: admin.mydomain.com, set the record set to be an alias and from the input that will appear, select the cloudfront distribution created earlier.

Route 53 - record set

After finishing creating it, we will need to wait some time while Amazon register the domain in the DNS servers and propagates the changes. Shortly after, you will be able to use your new URL to access your website.

Bonus track: what about APIs?

As a small tip, if you want to do this for an API that is being hosted in ECS or Beanstalk, instead of creating a Cloudfront distribution, you can create the record set in Route 53 directly and point to the load balancer that is associated to those instances. The first two steps are still required, :)

Summary

Now that all sections have been covered, I hope that your application has been deployed successfully, and your customers will be happy by having a pretty URL that is easy to remember (hopefully). If you have any comment, don't hesitate in contacting me or leaving a comment below. And remember to follow me on twitter to get updated on every new post.

Deploying a REST API using Express to Elastic Beanstalk

A couple of weeks ago, I was trying to deploy a small service, but I didn’t wanted to start creating the full infrastructure suite on Fargate yet; hence, the best option is Elastic Beanstalk. For a couple of days I did the deployment manually, but after reading a little bit more about Beanstalk, I found a CLI that simplified a lot of the deployment process.

Here is some documentation if you want to read about it, but for the mean time, I will write a little bit about how I used it. It might be useful to you, if like me, you are looking for ways to do a quick deployment and don’t want to invest time in configuring the CI/CD pipeline or a more complex setup.

Prerequisites

The express application was built using:

  1. Node version 10.15.0

  2. generator-express-no-stress-typescript version 4.2.1

Installing the CLI

To install the CLI, there are two approaches, using HomeBrew or the other one using PIP. In my case, I used Brew because it’s so much simpler (and also because the other way failed for me :).

Just type the following and brew should take care of everything for you:

brew install awsebcli

Configuring the CLI

I’m going to assume you are already using AWS profiles in your machine, if not, I recommend reading about them here.

To start the configuration in your project, navigate to the root folder of it, and type the following command:

eb init —profile [YOUR PROFILE NAME]

It will start a series of questions in the next sequence:

  1. Region: in our case, we selected us-east-1

  2. Elastic Beanstalk Application: it will prompt you create a new app if none is already created.

  3. Platform: in our case, we selected node.js

  4. SSH configuration

  5. Keypair: to connect to the instance, you will need a key pair to use via SSH. The CLI will help you to create one as well.

Important: there is a step asking you if you want to use CodeCommit, this will help you create a pipeline, however, since we are using other source control tools, we ignored it.

Once the CLI finishes, you will see a new folder in your project called: .elasticbeanstalk (notice the dot at the beginning). If you wanna read more about configuration options, go here and here.

First deployment

Now that our app is configured (at least for starters), we need to do a deployment. For that, we need to create an environment with the command:

eb create [ENVIRONMENT NAME]

We used DEV, and after it finishes, we need to update some configuration in the AWS console. We will update the node command, this command is the one nginx will use whenever a new deployment is done.

Elastic Beanstalk environment configuration

Just one more thing, as you can see, the environment uses nginx as base server, so make sure that the application is listening on port 8081 (look at the documentation here). Finally, we can run:

eb deploy

Important: I had an issue, a really weird one when I started doing changes and deploying them, and it seemed like the deployment never grabbed the latest changes. After reading in internet for some time, I found this post; and in one of his IMPORTANT notes, Jared mentioned that the changes must be committed to Git for them to be deployed. I don’t have the answer to why this happens, but seems important to note.

If you are using plain javascript, this could be all, it will give you the URL to use to connect and whatever you have built, should be ready for usage. However, as many of my other posts, I like typescript, and this isn’t the end of our journey.

Deploying Typescript

Using typescript, the deployment is not as straightforward, so we need to add one more step to our config.yml file and in the package.json.

Important: these steps are necessary because Elastic Beanstalk do not install dev dependencies in their machines, so we need to overcome that issue with this new process.

First, we need to add something that helps us to do the compilation of Typescript to Javascript. For that, we will add a shell script in the root of our app with the following contents:

zip dist/$npm_package_name.zip -r dist package.json package-lock.json .env

Then, we will modify the package.json’s scripts with the following:

{
  ....
  "scripts": {
      "compile": "ts-node build.ts && tsc && sh dist.sh",
       ...
  },
  ...
}

And finally, in our config.yml file under the .elasticbeanstalk folder, we will add the following content before the global declarations:

deploy:
  artifact: dist/npm_package_name.zip

Now, let’s explain a little bit about what we are doing.

The scripts in the package.json file were updated to run the shell script after the compilation is finished.

The shell file grabs everything that was compiled and move it into a zip file.

Important: The “npm_package_name” in the shell file will refer to the package.json name attribute. Make sure that in the config.yml file, you type the same name

Next, in the config.yml, we specify the file that we will deploying to the environment in Elastic Beanstalk. So, the ebcli will only grab the zip file and send it. Under the hoods, elastic beanstalk will unzip it and run the command specified in the beanstalk environment.

Finally, run again the eb deploy command, and now our Typescript Express API will run in the cloud.

Summary

Now that all sections have been covered, I hope that the application has been a success and you have deployed your app into the cloud. If you have any comment, don't hesitate in contacting me or leaving a comment below. And remember to follow me on twitter to get updated on every new post.

Google Authentication using AWS-Amplify (+ deployment)

Authentication using AWS is a process I covered in a previous post, however, this time we are going to use a tool provided by Amazon called Amplify.

For this tutorial, we are going to create a simple application using Facebook’s create-react-app. Then, we will add the authentication layer by using AWS-amplify, and finally, add the hosting in S3 buckets. But before that, let’s provide some basic concepts.

What is AWS-Amplify?

According to their own site, AWS-amplify is defined as a library for frontend and mobile developers that are building cloud-based applications, and facilitates them the necessary tools to add multiple cloud features. For this tutorial, we will focus on two main features: storage and authentication, but Amplify provide many more like:

  • Analytics

  • API integration

  • Push notifications

  • Cache

  • Among others

What is create-react-app and how to install it?

Create-react-app is the best tool to use whenever you want to start creating a web application with React, and for someone like me that likes Typescript, it has now the built-in capability to create apps using it.

Installing it into your machine is like installing any global package from npm. Just type “npm install -g create-react-app“ and voliá!

There are some dependencies needed although, for example, you must have at least node version 6. This library also allows you to focus on creating your application instead of dealing with webpack or babel configuration.

Now, let’s start with the real deal, and work on our Google authenticated application.

Create the app

For this tutorial, I will use the following versions:

  • node: 10.15.0

  • npm: 6.4.1

  • create-react-app: 2.1.3

  • aws-amplify: 1.1.19

To create the app, we will run the following line in your preferred terminal: “create-react-app google-auth-tuto --typescript“. This will generate all the necessary code you need to start working.

Running the app

To start using the application, run “npm install” in your terminal to verify that you have all the necessary packages installed. Then, in the package.json file generated, some scripts have been created by default, this time we will use the “start” script; so simply run “npm start” and it will open a tab in your browser after the application finishes compiling your code.

npm start

Now that our application is running, we can start using AWS-amplify to add some cloud functions, but first, we will need to configure amplify in your machine. For that, you can follow the next video that explains how to do it (taken from aws-amplify main site).

Configuring amplify in the project

Now that amplify is configured in your machine, we can add it to our application by running: “amplify init” in your project root folder. It will prompt you several questions and after them it will start creating some resources in your account, here is an example of what you will see in your terminal.

amplify init configuration

At the end, if this is the first time you are running aws-amplify, it will create a new profile instead of using an existing one. In this example, I’ve used my profile named ljapp-amplify, so this section might be different for you.

Important: always create different profiles for your AWS accounts, in my case, I have to use multiple accounts for my companies’ clients, so it facilitates a lot my work.

After the AWS resources have been created, let’s add the authentication layer to our app. AWS-amplify have different categories of resources, authentication is one of them. So, let’s add it by running: “amplify auth add“. Same as before, you will see some configurations asked by amplify, here is a summary of what you will receive.

amplify auth add

The only information that you might be wondering how to get is the Google Web Client Id. For that, please follow the instructions found here, under the “Create a client ID and client secret” section.

Finally, run “amplify push” and this will start creating all the authentication resources in your account.

amplify push

Important: AWS-amplify uses Identity Pools for 3rd party integration instead of user pools. Since identity pools doesn’t manage groups, we can only authenticate them. So, if we need to provide specific permissions or roles, we need to use claims (or switch to user pools manually) and configure it manually in AWS console.

Modifying React code

Up till now, we have setup all the foundation in the AWS account via amplify, but we still need to add logic in our react application. For that, we will install two npm packages:

  • npm install aws-amplify

  • npm install aws-amplify-react

Then, we will modify our App.ts file with the following code.

import React, { Component } from 'react';
import Amplify from 'aws-amplify';
import { withAuthenticator } from 'aws-amplify-react';

import logo from './logo.svg';
import aws_exports from './aws-exports';
import './App.css';

Amplify.configure(aws_exports);

class App extends Component {
  render() {
    return (
      <div className="App">
        <header className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          <p>
            Edit <code>src/App.tsx</code> and save to reload.
          </p>
          <a
            className="App-link"
            href="https://reactjs.org"
            target="_blank"
            rel="noopener noreferrer"
          >
            Learn React
          </a>
        </header>
      </div>
    );
  }
}

const federated = {
  google_client_id: 'SOME_NUMBER_HERE.apps.googleusercontent.com',
};

export default withAuthenticator(App, true, [], federated);

The second parameter in the “withAuthenticator” high-order component, will create a header for our application with some minimal information like the name of the user logged in, and also, renders the log out button.

Important: By default, aws-amplify provides some default screens that can be customized, but also, it allows for creating our own components for login, register, among others. This will not be covered in today’s tutorial and we will be using the default screens.

As of today, the package aws-amplify-react hasn’t been updated with a typescript definition, so we will need to add a file that declares it as a module (with the name aws-amplify-react.d.ts), to avoid typescript errors during development. The contents of the file are:

declare module 'aws-amplify-react';

Now that everything is set, we can run our application again and we will be seeing the following screen.

Amplify login screen

And then, we can log in using google’s button and after verifying our account, we will get into the application.

User logged into the application

Hosting the application

Now that everything is setup, we can host our application in the cloud with amplify. For that, we will add the hosting feature by running the next command: “amplify hosting add“, and same as before, some configuration is required.

amplify hosting add

Shortly, it will ask you to run amplify publish, and this will create the S3 bucket if it doesn’t exist, and open right away a browser tab with the application hosted on that bucket.

Summary

Now that all sections have been covered, I hope that the application has been a success and you have created a React application that can use google authentication, and hosted easily in S3 buckets in AWS. In an upcoming tutorial, I will talk about using Cognito User Pools to do 3rd party authentication.

If you have any comment, don't hesitate in contacting me or leaving a comment below. And remember to follow me on twitter to get updated on every new post.

Use Athena To Process Data From DynamoDB

Have you ever thought that it would be so good to have SQL-like queries in DynamoDB?

I worked in companies that have always used Microsoft's SQL Server as database, and it's so easy for me to do things like COUNT, AVG, among others to calculate metrics and other indicators. However, now that I've been using Dynamo for some projects, I miss that.

This week, I found Athena (not the Greek Goddess), a tool provided by Amazon to query big data information stored in S3 using standard SQL queries. To use it, you don't need to configure any infrastructure, it runs serverless and executes the queries directly into the datasource in S3. It supports a variety of data formats like CSV, JSON, among others (today, we will be doing some examples using JSON).

But then, how do I use it to query from Dynamo? well, Athena still does not support queries directly into Dynamo, but we can mimic this functionality by using AWS Glue.

AWS Glue, is another tool that allows developers to create ETL jobs that can perform many tasks, and it's completely integrated with Athena. AWS Glue uses something called Crawlers that create schemas from the datasources that are analyzed, so for example, creating a crawler from a dynamo table, will enumerate all the columns that the table can have and a possible type of the data it contains.

So combining everything, we can do the following:

  1. Create a crawler that reads the Dynamo table
  2. Create a Glue job that reads the information and stores it in an S3 bucket
  3. Create a new crawler that reads the S3 bucket
  4. Use Athena to query information using the crawler created in the previous step.
  5. Expose information using API Gateway and Lambda functions

Here you have a visual representation of those steps:

And here is an example of the queries that you can create:

SELECT a.openCount, b.openWACount, c.solvedYTD
FROM (
   SELECT COUNT(*) AS openCount
   FROM caseview_metrics
   WHERE status='open'
) a
CROSS JOIN (
   SELECT COUNT(*) AS openWACount
   FROM caseview_metrics
   WHERE status='open' AND CARDINALITY(abductorsids) >= 1
) b
CROSS JOIN (
   SELECT COUNT(*) as solvedYTD
   FROM caseview_metrics
   WHERE status='close' AND closeddate BETWEEN 1514764800000 AND 1546300799000
) c

In this example, you can see functions as COUNT or CARDINALITY, expressions like BETWEEN or comparisons, and finally you can also apply JOIN clauses like the CROSS JOIN.

Summary

AWS Athena is a tool that you can add to your toolkit, and if you are using Dynamo, it can enhance greatly the experience for the users, as well as facilitate development. 

If you have any comment, don't hesitate in contacting me or leaving a comment below. And remember to follow me on twitter to get updated on every new post.

How to provide temporary access to S3 Buckets?

There are times when we store assets like images or videos in S3 buckets to be displayed in our websites. But what happens when we want to secure those assets so that only authenticated users can see them?

Well, there are many ways to provide security, one of the most common is used the "Referer" header but this can be spoofed, so we lose the security we wanted before. Another one is using Cloudfront and create signed URLs but that requires a lot of development work, and the last option was to use API Gateway to return binary data. After analyzing all this options, I determined that none of them provided the security we needed nor satisfied all of our use cases. Finally, I came up with another solution using a little bit of all the approaches mentioned before.

In order to provide security to the S3 folder, we are going to use signed URLs to provide temporary access to the bucket where the assets are hosted. To create the signed URLs we are using 2 lambda functions, the first one will be running under a IAM role that will create the signed URLs and the second one will be an authorizer function for the first one that will verify if the user making the request have the proper credentials. Here is a diagram of how the security flow for the S3 bucket works:

S3 Security Architecture

S3 Security Architecture

The first step to accomplish this is to remove the public policy that the bucket has, we want the bucket to be as closed as possible.

The second step will be to create a lambda function that will generate the signed URLs. For that, we need to create a lambda function called resolver and type the code provided below:

const AWS = require('aws-sdk');
 
exports.handler = (event, context, callback) => {
    AWS.config.update({
        region: "us-east-2"
    });
 
    const s3 = new AWS.S3({signatureVersion: 'v4', signatureCache: false});
    var key = event["queryStringParameters"]["key"];
    s3.getSignedUrl('getObject', {
        Bucket: "owi-trainer-assets",
        Key: key,
        Expires: 7200
    }, function(error, data){
        if(error) {
            context.done(error);
        }else{
            var response = {
                statusCode: 301,
                headers: {
                    "Location" : data
                },
                body: null
            };
            callback(null, response);
        }
    })
};

The getSignedUrl function from the SDK receives 3 parameters, the name of the operation that will be allowed from the URL created, an object containing the configuration (bucket, key of the object in the bucket and the expiration time in seconds), and lastly, the callback that will be executed once the URL is generated. As you can see, we are returning a code 301 in the response to force the client to redirect the request to the generated URL.

The third step is create an API Gateway endpoint that works as a proxy to the lambda function. The only important aspect here is to grab the ID of the API endpoint because we will need it for the next step. The ID can be obtained from the UI when the endpoint is created, in the next image, the text highlighted in yellow is the ID we need.

Gateway ID

Gateway ID

The fourth step is to create the validator lambda function that will verify that the client requesting an asset is a valid client. For that, we will follow the following steps.

  1. The validator function requires 2 NPM packages that not provided by default in the lambda ecosystem. So we will need to upload a zip file that contains all the necessary libraries.
  2. To accomplish that, create a folder named validator and navigate to it in a command window. In there, type "npm init" to create a package.json file and install these two components:
    1. aws-auth-policy: contains the AuthPolicy class that is required for a Gateway authorizer to perform actions.
    2. jsonwebtoken: this library is going to be used to validate the JWT tokens sent in the query string from the client.
  3. Inside of the validator folder created before, add an index.js file that will contain the logic to validate the tokens. The code will be provided below.
  4. Finally, create a lambda function named validator and upload the folder in a zip file.
var jwt = require('jsonwebtoken');
var AuthPolicy = require("aws-auth-policy");
 
exports.handler = (event, context) => {
    jwt.verify(event.queryStringParameters.token, "<SECRET TOKEN TO AUTHENTICATE JWT>",
    function(err, decoded){
        if(err) {
            console.log(err);
            context.fail("Unable to load encryption key");
        }
        else{
            console.log("Decoded: " + JSON.stringify(decoded));
 
            var policy = new AuthPolicy(decoded.sub, "<AWS-ACCOUNT-ID>", {
                region: "<REGION>",
                restApiId: "<API GATEWAY ID>",
                stage: "<STAGE>"
            });
            policy.allowMethod(AuthPolicy.HttpVerb.GET, "*");
 
            context.succeed(policy.build());
        }
    });
};

Finally, the fifth and last step is to add the authorizer in the API Gateway, for that, go to the Authorizers section in the Gateway you created and click on  "Create New Authorizer". Follow the details as follows:

Authorizer Configuration

Authorizer Configuration

As you can see, the token will be sent as part of the query string, other options are to send the token as a header or a stage variable.

If you have any comment, don't hesitate in contacting me or leaving a comment below. And remember to follow me on @cannyengineer to get updated on every new post.