← back to the blog

Setting Up Your New Blog in Keystone.js - Part II

Posted on May 2nd, 2016 in automation, ansible, continuous, integration, deployment, ssl, certificates

In this post I will continue where I left off in my previous post. I will show how to automate the Kesytone.js deployment and how to generate new and free SSL certificates.

In the last post I explained how to customize Keystone.js and marked but to reach our goal we had to fork, check out, modify, and push the code (among others) so that our changes would become effective in our Keystone.js application. This maintenance task is downright underwhelming and error-prone and because of it's repetitive nature it can be perfectly automated.

So in the next section I will show how to set up continuos integration and deployment so that all our local code changes would cascade through all their inter-connected upstream projects and get deployed into production by running a single command in the local command line.

Note that in the following paragraphs I will use Jenkins as a CI server, but it really doesn't matter which application you choose (e.g. Bamboo, CircleCI, CIsimple/, Codeship, Jenkins/Hudson, Semaphoreapp, Shippable, Solano CI, TravisCI or Wercker among others) since the underlying principles are all the same.

Continuous Deployment with Ansible

So the goal will be to deploy the latest code changes into production after all the make files, build scripts and tests were running successfully. To achieve this we will employ Ansible, which has good support for the technology stack Keystone is gonna use in production (mongodb, Node.js).

Ansible is a configuration management tool, which is basically trying to help the administrative tasks by turning imperative operational tasks into a set of declarative configuration files which in turn the Ansible engine converts into actions.

Because of Ansible's modular nature it is very easy to configure it and set it up. What you are gonna do, you will create an inventory file, create new roles and a playbook which will contain the deployment tasks. You will also create a new template for the environment variables which are required when Keystone.js starts up.

First of all follow the Getting Started Guide and make sure you have a valid inventory file.

  1. Create a new folder ansible-tasks and navigate into it
    mkdir ansible-tasks
    cd ansible-tasks
  2. Create a new file inventory and add the following content:
    The first section should list all the servers where you are planning to deploy using ansible. This is the core feature of Ansible that it just takes a list and assumes that all servers should look and behave the same. The prod_blog_servers:vars section should contain those configuration values which you want to be the same across the whole server park.
  3. While it is possible to write a playbook in one very large file (and you might start out learning playbooks this way), eventually you’ll want to reuse files and start to organize things. So create a new file deploy.yml and add the following content:
    - hosts: "{{ env }}_blog_servers"
       - common
       - blog
    As you can see the file contains a reference to the server listing which we created in the previous steps, and two roles: common and blog. The {{ env }} code is an example of ansible variable resolution. Env can be provided through the command line to ansible-playbook.
  4. Create a new folder called roles and navigate into it
    mkdir roles
    cd roles
  5. Create a new folder called common and navigate into it. This folder will contain sub-tasks, files and templates.
  6. Create a new folder called defaults and create a new file in it main.yml and create all those constants or defaults here which are persistent across different playbook runs. E.g.

    app_name: "%your-app-name-here%"
    root_dir: "/home/%your-username-here%"
    blog_dir: "{{ root_dir }}/services/{{ app_name }}"
    blog_bin_dir: "{{ blog_dir }}"
    # Blog app general config
    blog_install_cmd: "{{ blog_bin_dir }}/node install"
    blog_start_cmd: "supervisorctl start {{ app_name }}"
    blog_stop_cmd: "supervisorctl stop {{ app_name }}"
    blog_restart_cmd: "supervisorctl restart {{ app_name }}"
  7. Go back to the roles folder and create a new sub-folder called blog and navigate into it. This folder will contain the main deployment tasks. Create a new folder tasks and navigate into it.
  8. Create a new file deploy.yml and add the following content:

    - name: ensure blog folder exists
     file: path={{ blog_dir }} state=directory
    - name: Get the blog code from git
     git: dest={{ blog_dir }} force=yes
    - name: Install the latest Keystone changes
     npm: name=keystone path={{ blog_dir }}
    - name: generate the '.env' file
     template: src=env.j2 dest={{ blog_dir }}/.env
    - name: restart the blog
     command: "{{ blog_restart_cmd }}"
  9. Create a new file main.yml and add the following content:

    - include: deploy.yml
  10. Navigate back to the parent folder blog and create a new sub-folder templates and navigate into it. Then create a new file called env.j2 and add the following content:
    COOKIE_SECRET={{ cookie_secret }}
    SESSION_STORE={{ session_store }}
    ADMIN_NAME_FIRST={{ admin_name_first }}
    ADMIN_NAME_LAST={{ admin_name_last }}
    ADMIN_NAME_EMAIL={{ admin_name_email }}
    ADMIN_NAME_PASSWORD={{ admin_name_password }}
    NODE_ENV={{ node_env }}
    PORT={{ port }}
    HOST={{ host }}
    COMPRESS={{ compress }}
    SSL={{ ssl }}
    SSL_KEY={{ ssl_key }}
    SSL_CERT={{ ssl_cert }}
    SSL_CA={{ ssl_ca }}
    SSL_PORT={{ ssl_port }}
    SSL_HOST={{ ssl_host }}
    POST_IMAGE_UPLOAD_PATH={{ post_image_upload_path }}
    As you can see this last step basically just a template full of variables waiting for resolution. This is the key to make our application fully customizable. Every time when we add a new feature that we want to customize from time to time, we have to list it here and Ansible will generate a .env file with the proper values. Than Keystone.js will read the generated .env file in turn and our application will use those properties.

Now that everything is set up, push your project (ansible-tasks) to github and go to the next chapter.

Deployment with Jenkins

The goal is that every time we make changes in the code locally we test them and push those changes into production. It turns out that Jenkins is perfect for this purpose.

I will talk about Jenkins throughout the chapter but you can go ahead and achieve the same goal by using your preferred continuous integration server, it really doesn't matter which one you use at all.

First of all install Jenkins and make sure that Jenkins can clone source code from Github, it can receive Github webhook events and it can execute Ansible.

Make sure that the Jenkins user (the OS user which is executing the Jenkins war file) can log in to the production servers. One good practice is to generate SSH keys on the Jenkins server for the Jenkins user and upload the public keys to the production servers. You can follow this guide: how to create an authentication key-pair.

To make Jenkins aware of our code changes (either about the Keystone application or the Ansible tasks) we will have to integrate Jenkins with Github.

  1. Open your Github project in a browser and click on Settings
  2. Select Webhooks & services from the left side menu items
  3. Click on Add webhook
  4. Paste the following into the Payload URL field: http://#{your-jenkins-hostname-here}:8080/github-webhook/
  5. Change Content type to application/x-www-form-urlencoded
  6. Leave the Secret token empty (in case of Jenkins because it cannot handle it)
  7. Select Just the push event.
  8. Tick the Active checkbox
  9. Repeat these steps for other projects also

Now every time you push changes to this Github project, Github will notify Jenkins. And Jenkins will trigger specific listening projects in turn. Let's create these listening projects next.

Basically you will need 2 projects, one which listens for the ansible task code changes, and the other one which listens to the Keystone code changes and executes Ansible providing proper configuration values.

  1. On the Jenkins dashboard click New item
  2. Provide an Item name e.g. blog-build and select Freestyle project
  3. Click OK
  4. On the project configuration page in the Source Code Management select Git and provide the Keystone application Github project URL that you created earlier and provide the proper login credentials.
  5. In the Build Triggers section select Build when a change is pushed to GitHub
  6. In the Build section select Execute shell from the drop-down list
  7. Enter those commands which trigger the build of your project. Example:
    npm install
    npm run-script build
    npm run-script lint
    npm run-script pre-test
    npm run-script test-all
    npm run-script test-cov
  8. In the Post-build Actions section select Build other projects from the drop-down list
  9. Type blog-deployment in the Projects to build text field
  10. Save the project

Let's create another Jenkins job for deploying the code changes using Ansible. This job will eventually push the code changes into production.

  1. On the Jenkins dashboard click New item
  2. Provide the Item name blog-deployment and select Freestyle project
  3. Click OK
  4. On the project configuration page in the Source Code Management select Git and provide the ansible-tasks Github project URL that you created earlier and provide the proper login credentials.
  5. In the Build section select Invoke Ansible Playbook
  6. In the Playbook path field enter ${WORKSPACE}/deploy.yml
  7. In the Inventory section select Inline content and provide those configuration values that you earlier set up in the env.j2 file. E.g.:

  8. Click on Advanced...
  9. Enter the following into Additional parameters: -e env=prod
  10. Save the project

That's it. Now you can make some changes locally in your Keystone application code, and push that to Github. From then, everything will cascade through the earlier steps and the Jenkins jobs will get executed automatically. When those jobs are done, you can refresh your browser and you will see the latest changes applied to the production code.

The main takeaway of this chapter is that Github and Jenkins can be integrated through an HTTP API which facilitates inter-connected communication between those two servers.

The other important takeaway is that you can organize the Jenkins projects so that they form a downstream-upstream hierarchy and downstream changes can kick-in upstream changes when they finished.

In this simple example we only criss-crossed 2 projects, but you can see that this can be (and should be) easily extended to trigger auto-deployment when you make changes in the Keystone.js admin application. Which can be (and should be) extended in turn to trigger auto deployment when the you make changes in the marked project. So eventually you end up with pipelines which are responsible for a stream of administrative maintenance tasks running each after the other. This will free up tremendous amount of development time and improve code quality beyond unexpected levels so it's good practice to do this early when the project starts.

Generating Free SSL Certificates

Your code is deployed into production but there is one big problem. The plaintext HTTP data is still open for sniffing. To solve this problem you can install SSL certificates.

In general when you want to secure your HTTP server with SSL certificates you need to pay for that. But this is not truly necessary since there are free alternatives to that. I will show how to integrate your Keystone.js application with SSL certificates generated through the Let's Encrypt service. Run the following steps on your production server.

  1. Follow the guidelines in the Getting Started guide
  2. Navigate to the letsencrypt folder
  3. Execute the following command:
    ./letsencrypt-auto certonly --standalone -d #{your-hostname} -d #{your-hostname-starting-with-www} --config-dir . --work-dir . --logs-dir .
    This will create a new sub-folder in the live folder. The sub-folder will have the same name as #{your-hostname} and it will contain four files:
  4. Open your blog-deployment JENKINS project and add the following configurations to the Ansible playbook inline content:
    Basically we have to provide the certificates to the Keystone.js application through the .env file, so even if you don't use Jenkins, you have to add these lines to .env.
  5. Navigate to your LOCAL blog application code
  6. Open keystone.js and add the following code:

     'name': '#{your-application-name-here}',
     'brand': '#{your-application-name-here}',
     'post image upload path': process.env.POST_IMAGE_UPLOAD_PATH,
     'ssl ca': process.env.SSL_CA

    As you can see we have to tell Keystone.js to use the SSL CA certificate which is available through the SSL_CA environment variable.

Now you can commit and push the LOCAL keystone.js changes to Github. This will kick in the Jenkins jobs we created earlier which will deploy the code into production and start up the server using our brand new SSL certificates.

Open your browser and navigate to your domain name. https://your-domain-name:3001

Enjoy your encrypted HTTP packages.