Fogetti

← back to the blog


Setting Up Your New Blog in Keystone.js - Part I

Posted on April 29th, 2016 in admin, customization, disqus, integration, keystone, keystonejs, setup, tutorial

In this post I will explain how to set up a Keystone.js webapp, how to switch the default admin interface to the shiny new React based application, how to get rid of the unnecessary parts of the admin UI, how to switch the Bootstrap theme of the front-end, how to modify the Cloudinary based post model to save the uploaded images in the file system instead of the cloud, how to modify the same post model to use markdown syntax instead of HTML and how to integrate the Disqus comment system with those blog posts.

In the coming post I will explain how to deploy the application and make it production ready. Along the way I will also give some tips on how to speed up development by enabling continuous deployment, generate valid and free SSL certificates and how to speed up the front-end performance a bit.


Some background on why I chose Keystone: I am coming from the enterprise Java world and as much as I love Java and the power it represents in the back-end, it just sometimes feel clumsy when I have to adhere to all the J2EE non-sense. This is especially true about Java Servlet containers and J2EE application servers. They just unnecessarily overcomplicate things sometimes. By the time the requests arrive to our code they become an unrecongnizable bloated monster and to support this monster we have to navigate through a very rigid project structure which is intimidating and unflexible at best and we have to get familiar with virtual hosts, root and other contexts, filters and listeners, resource references, security roles, servlet mappings, taglibs and session configs to begin with.

I guess I was not the only one who felt that the Java way is a big no-no. That's why so many simplified Java web frameworks sprung up from nowhere (e.g. Jetty, Spark, Dropwizard, Play or JHipster) which were all breaking with the traditional approach. What is the common denominator in these projects? That they are all inspired by other languages: Spark is inspired by Sinatra, Grails is inspired by (you guessed) Ruby on Rails and Play is relying on the Akka asynchronous framework which draws its inspiration from Erlang.

So I needed something lightweight. One thing that makes Node.js appealing is that the Node.js community is extremely productive in making small, usable and meaningful modules. Mongoose is one excellent example of that kind and it really lifts the burden of the programmer from carefully inspecting the data model, anticipating all the possible schema changes ahead of time.

Well the good news is that Keystone comes with a blog data model by default, so configuring the web-app is rather about customizing than implementing new features, which is a good thing in case somebody is looking for a light-weight blogging app and not familiar with Drupal, Wordpress or Joomla or has never used PHP before and doesn't want to spend a lot of time implementing the full MVC stack.

Keystone Set-Up

So first you have to follow the instructions on the Keystone.js website: 'Getting Started'


While you can install mongodb in a breeze, mongodb assumes some defaults during start-up. One problem might be that the `/data/db` folder doesn't exist or it doesn't have the correct access rights to write into that folder it will fail during start-up.

Another thing to keep in mind is that mongodb comes with absolutely no authentication by default. So basically anyone who can connect to your mongodb port has open access to all your data. This is something which is well worth to keep in mind. As the hacker who pwned the Hacking Team put it: "NoSQL, or rather NoAuthentication, has been a huge gift to the hacker community. Just when I was worried that they'd finally patched all of the authentication bypass bugs in MySQL, new databases came into style that lack authentication by design."

Please refer to the specific sections of the mongodb documentation how to solve these problems.

Bootstrap Theme Change

So, time to make some changes. I will be using SASS in the following steps, but using LESS requires the same steps.

  1. Navigate to Bootswatch to find a new theme. You will see that every theme has a button below with an integrated drop-down component. Click the drop-down to see the 3 different options. If you want you can download the CSS files or the minified CSS files only, but I strongly recommend using LESS or SASS instead, since they are modular and it can help to keep the code organized.

  2. Create a new folder public/styles/themes

  3. Copy the files into the public/styles/themes folder

    1. Modify public/styles/site.scss to @import the newly downloaded theme files.

      @import "themes/variables";
      @import "themes/bootswatch";
      

      As an alternative to saving and maintaining two files (variables and bootswatch) you can just append the bootswatch file after the variable declarations and save those together in one file. Or you can do it the other way around

    2. Merge the two downloaded files (variables and bootswatch) by appending bootswatch at the end of the variables, and put a new @import "../site"; instruction at the very beginning of the file. (Actually this is what the Keystone.js team is doing in the demo application).

It doesn't really matter which structure you use (nr. 3.1 or nr. 3.2), what really matters is that the @imports should come in a specific order so that bootswatch overrides the original bootstrap variables and style definitions.

That's it. Now every time when you modify the SCSS files and refresh your browser in development mode, Keystone will recompile those SCSS files and generate a new CSS. This is the CSS that you will have to reference in your default.jade file.

Removing Unnecessary Menu Items

Well this will be easy. Keystone.js is basically just a fancy Express.js application. Everything which is available in Express.js is also available in Keystone.js. Actually we can even get a reference to the original Express.js application by calling keystone.get('app') after Keystone got initialized. In my case I didn't need the gallery section and I also wanted to remove the Sign-in menu point.

  1. Remove the unnecessary model file from the models folder

  2. Remove the unnecessary view file from the routes/views folder

  3. Remove the unnecessary route binding from routes/index.js

  4. Remove the unnecessary navlink from routes/middleware.js

  5. Remove the unnecessary template file from the templates/views folder

  6. Remove the unnecessary navlink if-else-if condition from templates/layouts/default.jade

That's it. Simple enough. Now you have to restart the server to make these changes take effect. Adding a new menu point and a related DB model is similarly simple, you just have to take these steps the other way around.

Now there are four more things left to do. Modifying the admin interface to use the new React based architecture. Then modifying the blog model to avoid using Cloudinary. Then modifying the post model to use the Markdown field instead of the TinyMCE field. And finally integrating Disqus.

Modifying the Admin Interface Architecture

We will use the new React based layout. Nothing surprising here. We will just stand on the shoulders of giants and use NPM to get control of the desired codebase.


Using the git master branch as the codebase of your live application means that it might contain bugs and since usually the master branch is the main development branch it might easily get out-of-date. Also every time you run npm install, you will get a different codebase (depending on the last commits in the master branch). You can select a git tag instead, since tagged branches are considered as releases in most cases.
  1. Open package.json in your project
  2. Edit the dependencies section to link to the Github codebase of keystone:
    ...
    "dependencies": {
     "keystone": "git+https://github.com/keystonejs/keystone.git",
     "async": "^1.5.0",
     "underscore": "^1.8.3",
     "node-sass": "^3.3.2",
     "node-sass-middleware": "^0.9.7", 
     "dotenv": "^1.1.0"
    },
    ...
    
  3. Run npm install keystone

By default this will download and install the master branch of the github project, which might or might not be what you want. In case you want to be more specific you can also provide a parameter to the remote URL. The remote url syntax looks like this:

<protocol>://[<user>[:<password>]@]<hostname>[:<port>][:][/]<path>[#<commit-ish>]

#<commit-ish> can be a branch, a tag or a commit specifier.

Now you have to restart the server again and navigate to:

http://localhost:3000/keystone

After logging in your should see your brand new React based application.

Using the File System Instead Of Cloudinary

While there are legitimate reasons to use an image hosting service like Cloudinary (e.g. they are serving your images through a CDN like Akamai) my small blog application is not big enough to justify paying the extra bucks. And even though Cloudinary has a good set of API methods, I don't need any of those at the moment or in the near future and a blog might generate more than 500 MB of images which is the upper limit of the free storage.

Let's modify Keystone to use the file system in the mongoose model instead of Cloudinary.

  1. Open models/post.js
  2. Edit the image field as below:
    var moment = require('moment');
    ...
    Post.add({
     title: { type: String, required: true },
     state: { type: Types.Select, options: 'draft, published, archived', default: 'draft', index: true },
     author: { type: Types.Relationship, ref: 'User', index: true },
     publishedDate: { type: Types.Date, index: true, dependsOn: { state: 'published' } },
     image: {
         type: Types.LocalFile,
         dest: keystone.get('post image upload path'),
         prefix: '#{your-prefix-here}',
         allowedTypes: 'image/jpeg,image/svg+xml,image/png' ,
         filename: function(item, file){
             return moment().format('x') + '-' + item.id + '.' + file.extension
         }
     },
     content: {
         brief: { type: Types.Html, wysiwyg: true, height: 150 },
         extended: { type: Types.Html, wysiwyg: true, height: 400 }
     },
     categories: { type: Types.Relationship, ref: 'PostCategory', many: true }
    });
    

Simple enough. In this case the dest field contains an absolute path in the file system but it can be a relative path also. Since I definitely need a different URL path in the browser I also set the prefix property here.

The filename will consist of a timestamp and the item id in this case, where item.id is generated by mongoose.

That's the brilliant part in using mongoose, after modifying the model we can choose the strategy of not to migrate the data , we can just keep working instead. The only thing we have to do is to modify the application and update the expectations it is assuming.

So let's modify the blog and post view templates since these are the only two places where we use the images.

  1. Open templates/views/post.jade and replace
    .image-wrap: img(src=data.post._.image.fit(750,450)).img-responsive
    
    with
    .image-wrap: img(src='#{your-prefix-here}' + data.post.image.filename).img-responsive
    
  2. Similarly open templates/views/blog.jade and replace
    img(src=post._.image.fit(160,160)).img.pull-right
    
    with
    img(src='../../images/post/' + post.image.filename).img.pull-right
    

Well, it's done. So we can move on.

After restarting Keystone again you can start uploading your images to the file system.


Keystone will need the proper file system access rights to upload those files to the file system. Make sure that you set the access rights properly before starting uploading.

Using Markdown instead of TinyMCE

OK, this will be a little bit more tricky. Since Keystone doesn't have any well designed plug-in architecture to plug in our custom Fields, neither it allows customizing existing admin UI fields easily, we will have to fork some of the required github projects.


Forking will detach your codebase from the upstream implementation if you don't import those changes frequently. Also the upstream changes might be conflicting with your codebase. This might or might not be what you want, depending on your use case. If you are planning to provide a customized version of Keystone.js to clients or customers who are familiar how the Keystone.js codebase works, then they might raise impracticable expectations regarding your application (since your version might deviate from upstream). Make sure that you integrate upstream changes frequently.

So the goal is to use custom Markdown syntax to create notes and cautions in the blog posts, and to preview and display those notes and cautions in HTML using custom CSS styles applied.

Well it turns out that Keystone.js is using the marked npm module to do it's job. But unfortunately marked doesn't have a proper extension point just like Keystone, so to make things worse we will also have to modify marked.

At this point we will make everything manually, but in a follow-up post I will show it how to automate these steps so that modifications in one project will cascade through all the dependent projects automatically.

Let's modify marked first.

  1. Fork https://github.com/chjj/marked.git
  2. Clone the marked project:
    git clone https://github.com/#{your-username-here}/marked.git
    
  3. Modify 'lib/marked.js' by adding a new line to the block like this:
    var block = {
     newline: /^\n+/,
     code: /^( {4}[^\n]+\n*)+/,
     fences: noop,
     hr: /^( *[-*_]){3,} *(?:\n+|$)/,
     heading: /^ *(#{1,6}) *([^\n]+?) *#* *(?:\n+|$)/,
     nptable: noop,
     lheading: /^([^\n]+)\n *(=|-){2,} *(?:\n+|$)/,
     blockquote: /^( *>[^\n]+(\n(?!def)[^\n]+)*\n*)+/,
     list: /^( *)(bull) [\s\S]+?(?:hr|def|\n{2,}(?! )(?!\1bull )\n*|\s*$)/,
     html: /^ *(?:comment *(?:\n|\s*$)|closed *(?:\n{2,}|\s*$)|closing *(?:\n{2,}|\s*$))/,
     def: /^ *\[([^\]]+)\]: *<?([^\s>]+)>?(?: +["(]([^\n]+)[")])? *(?:\n+|$)/,
     plugin: /^ *\[([^\:\]]+):([^\]]+)\] *\n*/,
     table: noop,
     paragraph: /^((?:[^\n]+\n?(?!hr|heading|lheading|blockquote|tag|def))+)\n*/,
     text: /^[^\n]+/
    };
    
  4. Modify the tokenizer part by adding a new if condition:

     // html
     ...
    
     // plugin
     if (cap = this.rules.plugin.exec(src)) {
       src = src.substring(cap[0].length);
       this.tokens.push({
         type: 'plugin',
         plugin: cap[1],
         arg: cap[2]
       });
       continue;
     }
    
     // def
     ...
    
  5. Add a new switch case to the Parser:
     case 'plugin': {
       try {
         return marked.plugins[this.token.plugin](this.token.arg)
           + '\n';
       } catch(e) {
         return '<p><strong>Plugin error: ' + token.plugin + '</strong></p>\n';
       }
     }
    

Save the file and run make which will minimize the javascript file. Now you can commit your changes and push them to your fork.

Next you are gonna customize Keystone. Those steps are somewhat similar to that of marked.

  1. Fork https://github.com/keystonejs/keystone
  2. Clone the Keystone project:
    git clone https://github.com/#{your-username-here}/keystone
    
  3. Open the package.json file
  4. Edit the dependencies section to link to the Github codebase of marked:
    ...
    "dependencies": {
     ....
     "mandrill-api": "1.0.45",
     "marked": "git+https://github.com/#{your-username-here}/marked.git#plugin-branch",
     "method-override": "2.3.5",
     ....
    },
    ...
    
  5. Run the following command to install the customizations you made in marked:
    npm install marked
    
  6. Modify fields/types/markdown/MarkdownType.js by adding a new property to the 'marked' variable:
    marked.plugins = {
    note: function(arg) {
     return '<blockquote class="note">' + arg + '</blockquote>';
    },
    caution: function(arg) {
     return '<blockquote class="caution">' + arg + '</blockquote>';
    }
    };
    
  7. Modify fields/types/markdown/lib/bootstrap-markdown.js by adding a new property to the 'marked' variable:
    marked.plugins = {
    note: function(arg) {
     return '<blockquote class="note">' + arg + '</blockquote>';
    },
    caution: function(arg) {
     return '<blockquote class="caution">' + arg + '</blockquote>';
    }
    };
    
  8. Add new @import to fields/types/markdown/less/boostrap-markdown.less
    @import "bootstrap-md-preview.less";
    
  9. Create the new file bootstrap-md-preview.less. You can put your custom note and caution styling in this file.
  10. Save all files.
  11. Run the npm build:
    npm run-script build
    

Now you can commit your changes and push them to your fork. Of course to make these changes look nice in the blog post also, you will have to add the same styling (which you used in bootstrap-md-preview.less) to the front-end also.

Now the last step is to install all these changes (both Keystone admin UI and marked) in your Keystone application.

  1. Navigate to your Keystone application
  2. Open the package.json file
  3. Edit the dependencies section to link to your forked codebase of Keystone:
    ...
    "dependencies": {
     "keystone": "git+https://github.com/#{your-username-here}/keystone.git",
     "async": "^1.5.0",
     "underscore": "^1.8.3",
     "node-sass": "^3.3.2",
     "node-sass-middleware": "^0.9.7", 
     "dotenv": "^1.1.0"
    },
    ...
    
  4. Run the following command to install the customizations you made in Keystone (and marked also in a transitive way):
    npm install keystone
    
  5. Apply the same style that you used in bootstrap-md-preview.less
  6. Modify models/Post.js similarly to the image earlier
    Post.add({
     title: { type: String, required: true },
     state: { type: Types.Select, options: 'draft, published, archived', default: 'draft', index: true },
     author: { type: Types.Relationship, ref: 'User', index: true },
     publishedDate: { type: Types.Date, index: true, dependsOn: { state: 'published' } },
     image: {
         type: Types.LocalFile,
         dest: keystone.get('post image upload path'),
         prefix: '#{your-prefix-here}',
         allowedTypes: 'image/jpeg,image/svg+xml,image/png' ,
         filename: function(item, file){
             return moment().format('x') + '-' + item.id + '.' + file.extension
         }
     },
     content: {
         brief: { type: Types.Markdown , wysiwyg: true, height: 150 },
         extended: { type: Types.Markdown, wysiwyg: true, height: 400 }
     },
     categories: { type: Types.Relationship, ref: 'PostCategory', many: true }
    });
    
  7. Modify templates/views/blog.jade to use the new brief and extended markdown fields. Replace:
         p!= post.content.brief
         if post.content.extended
    
    with
         p!= post.content.brief.html
         if post.content.extended.html
    
  8. Modify templates/views/post.jade to use the new brief and extended markdown fields. Replace:
    != data.post.content.full
    
    with
    != data.post.content.full.html
    

Now after restarting your application, you can start using your new Markdown syntax ([note:#{text}] and [caution:#{text}]) in the blog posts fields. When you click the preview button you will see the styles applied which you defined in bootstrap-md-preview.less earlier. And when you open the blog post in the front-end you will see the same html rendered that you can see in the preview mode.

Integration with Disqus

The last thing is to integrate Disqus into your blog posts. The goal is to let people leave comments at the end of every post.

First you will have to visit the following URL and set up a new Disqus site (which basically requires two things, selecting a Disqus site name and providing the URL of your website): https://fogettiblog.disqus.com/admin/settings/universalcode/

After setting up the new site Disqus will show an inline HTML code similar to this:

<div id="disqus_thread"></div>
<script>
/**
* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables
*/
/*
var disqus_config = function () {
this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable
this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
};
*/
(function() { // DON'T EDIT BELOW THIS LINE
var d = document, s = d.createElement('script');

s.src = '//#{your-disqus-site-name}.disqus.com/embed.js';

s.setAttribute('data-timestamp', +new Date());
(d.head || d.body).appendChild(s);
})();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript>

It's quite self explanatory, we will somehow have to generate a proper page URL and a page identifier on every page where we include the Disqus module.

Well it turns out that the identifier part is easy, since mongoose automatically generates a unique ID and a nice looking slug for every blog post. So let's make the changes.

  1. Open templates/views/post.jade and append the following at the end of the file:

         div#disqus_thread
         script.
             var disqus_config = function () {
                 this.page.url = '#{req.protocol}://#{req.hostname}:#{port}#{req.originalUrl}';
                 this.page.identifier = '#{data.post.id}';
                 this.page.title = '#{data.post.title}';
             };
             (function() {
                 var d = document, s = d.createElement('script');
                 s.src = '//#{your-disqus-site-name}.disqus.com/embed.js';
                 s.setAttribute('data-timestamp', +new Date());
                 (d.head || d.body).appendChild(s);
             })();
         noscript.
             Please enable JavaScript to view the 
             #[a(href="https://disqus.com/?ref_noscript", rel="nofollow") comments powered by Disqus.]
    

    This is basically the same inline HTML code in Jade syntax. The only difficulty is to get the req object which is the request object passed around by Express.js. Now it turns out that neither the req (which is the HTTP request) nor the res (which is the HTTP response) is provided by default in Keystone to the Jade templates, so we need a bit of extra code.

  2. Open the routes/middleware.js and add the following to the initLocals method:

         ....
     locals.port = process.env.PORT || 3000;
     locals.req = req;
     locals.res = res;
    
     next();
    

That's it. Restart your Keystone server, and voila, you have your new Disqus system integrated with Keystone!

So that's it for today. I hope you enjoyed it. Next I will write about how to automate the application development, speed up the front-end a bit and generate SSL certificates.