Deploy your Node/Couchbase application to the cloud with Clever Cloud

Introduction

Clever Cloud is the first PaaS to provide Couchbase as a service allowing developers to run applications in a fully managed environment. This article shows how to deploy an existing application to Clever Cloud.

I am using a very simple Node application that I have documented in a previous article: “Easy application development with Couchbase, Angular and Node”.

Clever Cloud provides support for various databases MySQL, PostgreSQL, but also and this is most important for me Couchbase. No only Clever Cloud allows you to use database services but also you can deploy and host your application that could be developed in the language/technology of your choice : Java, Node, Scala, Python, PHP, … and all this in a secure, scalable and managed environment.

Setting up your Clever Cloud environment

Create your account

1. Go to the Clever Cloud site : http://www.clever-cloud.com/
2. Click on “Login” link and follow the steps to create your account.
3. After few seconds you will received an email and be redirected to the Clever Cloud Console.

Create a Couchbase instance

The Clever Cloud Console allows you to create your Couchbase Bucket in few clicks:

1. Cick on “Services” in the left menu

2.  Click on “Add a Service” in the left menu 

3. Click on “Couchbase” button.

4. Select the size of the RAM quota for your bucket

The size of the RAM Quota for your bucket will have an impact on performance but also on the pricing.

5. Click “Add this Service”

You are done, you should receive an email with all the information to access your newly created bucket.

The mail from Clever Cloud contains the following information:

db_host = xxxxxxxx.couchbase.clvrcld.net	Location of the database, this is where the endpoint is located.
db_name = yyyyyyyy	Name of the Couchbase bucket
db_username = xxxxxxxx	Not used in Couchbase context
db_password = zzzzzzzz	Password to connect to the Couchbase Bucket

So you are now ready to use your bucket.

Note: In the current version of the Clever Cloud Couchbase Service you do not have access to a management console. If you want to get some information about the database or create views you need to do it from you application code.

Connect your Application to Couchbase@Clever-Cloud

The first step is to get some code, so let’s clone the “Couchbase Ideas Sample Application”, and install the dependencies, using the following commands:

git clone -b 03-vote-with-value https://github.com/tgrall/couchbase-node-ideas.git

cd couchbase-node-ideas

git branch mybranch

git checkout mybranch

npm install

Open the app.js and edit the connection info to point your application to the Couchbase instance and modify the HTTP port of your application to 8080 - this is a mandatory step documented here :

dbConfiguration = {
 "hosts": ["xxxxxxxxxxx.couchbase.clvrcld.net:8091"],
 "bucket": "xxxxxxxxxxx",
 "user": "xxxxxxxxxx",
 "password": "yyyyyyyyyyyyyyyyyyyyyyyyy"

};
...
...

  appServer = app.listen(8080, function() {
 console.log("Express server listening on port %d in %s mode", appServer.address().port, app.settings.env);
  });

Launch your application using

node app.js

Go to http://localhost:8080

Your application is now using Couchbase on the cloud powered by Clever Cloud. Let’s now deploy the application itself on Clever Cloud

Deploy your application on Clever Cloud

The easiest way to deploy an application to Clever Cloud is using git. The first thing to do is to add your SSH public key into Clever Cloud Console. If you do not have any SSH yet, follow the steps described on Github : “Generating SSH Keys”.

Add your SSH key

Note: As you can guess this should be done only once
Open the id_rsa.pub file with a text editor. This is your SSH key. Select all and copy to your clipboard.

1. Go to the Clever Cloud Console
2. Click on “Profile” entry in the left menu
3. Click on “SSH Keys”
4. Click on “Add a SSH Key”
5. Enter a name (anything you want) and paste your key
6. Click “Add” button

You are now ready to deploy applications to Clever Cloud. The next thing to do, is to create a new node application in Clever Cloud.

Create your Application

1. Click “Add an app” in the Application menu in the top menu.
2. Give a name and description to this application
3. Select the Instance type, in this case “Node.js”
4. Configure your instances, you can keep the default values for now, click “Next”
5. Check the configuration, and click “Create”

Your application is created, you are redirected to the generic information page, where you can find a Git URL that we will use to deploy the application.
You can navigate into the entries in the left menu to see more information about your application. In addition to the Information page, you can look at the following entries:

1. “Domain Names” to configure the URL to access your application
2. “Logs” to view the application logs

Deploy the Application

So we are almost there!
The deployment to Clever Cloud is done using a Git push command, so you need to add the deployment URL as a remote repository to your application, using the following command:

git remote add clever git+ssh://git@push.clever-cloud.com/app_[your_app_id].git

git commit -a -m “Couchbase on Clever Cloud connection”

git push clever mybranch:master

Once you have added the application as remote repository you can commit and push your application.

The last command pushes the application  to Clever Cloud. It is important to note that Clever Cloud will always deploy the application on the “master” branch on the remote repository. The notation mybranch:master is used to mention it. If you work locally on your master branch just use “master”.

You can now go to the Clever Cloud console and look in the log and click on the URL in the “Domain Names” section to test your application.

You should be able to see your application, that is running on the Clever Cloud PaaS.

When you update your application, you just need to do a  git push and git commit.

Conclusion

In this tutorial you have learned how to:

• Create your Clever Cloud account
• Create a Couchbase instance
• Create and deploye a Node.js application

Feel free to test this yourself, with Node or other technology, as you can see it is quite easy to setup.

Getting started with Couchbase and node.js on Windows

In a previous post I have explained how to use Couchbase and Node.js on OS X. Since it is quite different on Windows here another article about it.

Install Couchbase Server 2.0

If you have not installed Couchbase Server already, do the following :

1. Download Couchbase Server from here
2. Run the installer
3. Configure the database at http://localhost:8091 (if you have issue take a look to this article)

These steps are documented in the Couchbase Server Manual.

Install Node 

Install latest version of node

It is quite easy to install Node.js using the Windows installer provided at http://nodejs.org.

Once you have installed node, you can test is using the command line interface:

c:\Users\tgrall>node
> console.log(process.version);
v0.8.16

Node is installed. So far so good !

Install Couchnode

Couchnode, the Couchbase Client Library for Node.js, is a native module. The tool used to install native modules is node-gyp.  So to install couchnode you need to install :

• node-gyp
• python
• Visual Studio to have access to a C/C++ compiler

Install node-gyp

The node-gyp module is easy to install and you can install it using npm using the following command:

npm install -g node-gyp

The -g parameter indicates that this module will be installed globally and added to your %PATH%.

Install Python

GYP uses Python to generate the project, so you need to install it on your environment. I have installed Python 2.7.3 using the Windows installer.

Install Visual Studio

Finally you need a C/C++ compiler, the best way to get it is to install Visual Studio. As you probably know I am not a Windows expert and I do not know a lot about Microsoft development tools. I have downloaded Visual Studio Express the free development tools from here; it was sufficient.

Install Libcouchbase for Windows

Couchnode uses libcouchbase the C client library, so before running the npm install for Couchbase, you need to install libcouchbase itself.

You can download it from Couchbase site. The Windows versions are located in the left menu of the page. Download the zip file, that matches your environment. I have downloaded the "Windows, 64-bit MSVC 10".

Node-gyp will look for all the dependencies (DLL, library headers) into c:\couchbase directory, so you need to unzip the file in this folder. This location comes from the binding.gyp file of the couchnode project.

Install and Test Couchnode itself!

Let's check what we have done so far; we have installed:

• Node
• node-gyp
• Python
• Visual Studio
• Libcouchbase

We are now ready to install and use couchnode itself. For this we can create a new node project.

mkdir my-app
cd my-app
npm install couchbase

The install command will:

• Create a node_modules folder and put couchbase client library in it
• When installing/building couchnode on Windows I had the following warning :

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Microsoft.CppBuild.targets(1138,5): warning MSB8012: TargetExt(.dll) does not match the Linker's Output

File property value (.node). This may cause your project to build incorrectly.

To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Link.OutputFile).

[C:\Users\tgrall\node\node_modules\couchbase\build\couchbase_impl.vcxproj]

This is only a warning and as far as I know, it is not a blocker. At the end of the log you should see:

couchbase@0.0.10 node_modules\couchbase
├── bindings@1.0.0
└── request@2.11.4


You have successfully installed couchnode.

Let's now write a small test. Create a test.js file with the following content:

var  driver = require('couchbase');

driver.connect({
  "username": "",
  "password": "",
  "hostname": "localhost:8091",
  "bucket": "default"}, 
  function(err, cb) {
    if (err) {
      throw (err)
    }
 
 var key = 'foo';
 cb.set(key, '{"server" : "couchbase", "version" : 2 }' , function (err, meta) {
  if (err) { console.log(err); }
        cb.get(key, function(err, doc) {
   if (err){ console.log(err);}
   console.log(doc);
        });  
 });
});

Run this with the command:

node test.js

You should see the following text in your console :

{ server: 'couchbase', version: 2 }

Conclusion

In this article you have learned how to:

• Install Couchbase
• Install Node
• Install and configure node-gyp
• Install and use Couchbase and Node

all this on Windows 7.

Building a chat application using Node.js and Couchbase

After some basic articles about Couchbase installation, Node.js integration. Let's now dive into a more complete  example: a chat application.

The first version of the chat should be compliant with the following requirements:

• web based
• single room
• user just needs to enter a login and he can start to interact with other connected users
• user should be able to navigate into the chat history

The Couchbase Chat application is build using the following components

• Node.js for the application
• with express and socket.io
• Couchbase to persist all the messages

I won't go in all the detail of the design of the Node.js application. You can find many example of Node based chat application. I prefere to focus on how I have design the persistence using Couchbase more than the application itself. If you want me to give more detail about the complete application feel free to drop me a message/comment and I will do it.

What are the challenges with persisting the messages?
Storing the information is quite easy, just "dump" the message information in your database. The challenge is more around the fact that user want to access the history of the messages. So the key point here is how to store the information in a way that it is easy to get back in a sorted fashion.

You will find many different ways of achieving that depending of the technology you are using and they query capabilities of your persistence engine. Using Couchbase you have two ways to access/find the data:

• Using Views that allows you to query and secondary level index and do advanced operation such as sorting, query on key range, ...
• Directly access the data using its key

In this post I will show you how you can use the the two options to build your application and retrieved information that are stored and retrieved in a specific order:

• First Options: using a view to get the message history
• Second Options: using a counter as a key for the messages

The source code of the application is available in Github : https://github.com/tgrall/couchbase-chat


Get the Couchbase connection

The following code is used to connect to Couchbase, once it is done, the Web server is started:

var express = require('express');
var app = express();
var http = require('http');
var server = http.createServer(app);
var io = require('socket.io').listen(server);

var driver = require('couchbase');

driver.connect({
 "username": "",
 "password": "",
 "hostname": "localhost:8091",
 "bucket": "default"}, 
 function(err, couchbase) {
  if (err) {
   throw (err)
  }

  server.listen(8080);

  app.get('/', function(req, res) {
   res.sendfile(__dirname + '/index.html');
  });
...
// Application code
// Socket.io events
...
});

Let's now see how Couchbase is used in the chat application.

First Option : Using views to get the message history

Post a new message

In this example messages are formatted using the following information:

{
  "type": "message",
  "user": "Tug",
  "message": "Hello all !",
  "timestamp": 1349836768909
}

The key is based on the timestamp and  the user name : 1349836768909-Tug. I am adding the user name to be sure that the key is unique. Like that I do not have to manage conflicts.

The insertion of the message :

  socket.on('postMessage', function(data) {
    // create a new message
    var message = {
      type: "message",
      user: socket.username,
      message: data,
      timestamp: Date.now()
    }
    var messageKey = message.timestamp +"-"+ message.user;
    io.sockets.emit('updateChatWindow', message);
    couchbase.set(messageKey, JSON.stringify(message),function(err) {  }); 
  });

• The postMessage event is called by the client when the user post a new message. 
• A new message object is created with : a type, the user, the message itself and a timestamp.
• The message is sent to the different clients using the io.sockets.emit() function (line 10)
• Finally the message is saved into Couchbase (line 11). As you can see the only thing you have to do is to send the Javascript object as a simple JSON String.

At this point your application work perfectly, all the connected user will see the new messages since they are sent by the server as soon as they are created. But it is not possible for a user to navigate in the chat history and see older messages.

Retrieve messages from Couchase 

As explained earlier, it is possible to use a view to retrieve the message from the database in a proper order.  The view looks like that:

function (doc, meta) { 
  if ( meta.type == "json" && doc.type == "message" ) {
   emit(doc.timestamp, null);
  }
}

Each time a new document is inserted in the database, if this is a JSON document and the type of this document is "message" the index will be updated. When this view is called the result looks like :

{"id":"1352733392477-JOHN","key":1352733392477,"value":null},

As you can see the id of the document (timestamp-username) is automatically inserted in the response.


You can use the following command to insert the view in your Couchbase Server: (configure the server address, port and bucket accordingly to your environment)

curl -X PUT -H 'Content-Type: application/json' http://127.0.0.1:8092/default/_design/chat -d '{"views":{"message_hisory":{"map":"function (doc, meta) {\n  \n  if ( meta.type == \"json\" && doc.type == \"message\" ) {\n   emit(doc.timestamp, null);\n  }\n}"}}}'

The application now calls the view using the following code

socket.on('showhistory', function(limit,startkey) {
  limit = (limit == undefined || limit == 0)? 5 : limit;
  var options = {"descending": "true", "limit" : limit, "stale" : "false"};
  if (startkey > 0) {
    options.startkey = startkey-1;
  }
  couchbase.view("chat","message_hisory", options , function(err, resp, view) {
    var rows = view.rows;
    var keys = new Array();
    for( var i = 0; i < rows.length ; i++  ) {
      keys.push( rows[i].id );
    }
    couchbase.get(keys,function(err, doc, meta) {
      socket.emit('updateChatWindow', doc, true);
    });
  });
});

When the client send a showHistory event the application capture this event and call the view with proper parameters to send back the list of messages to the client.

The options object contains the different parameters that will be used to call the view:

• Use descending order to return the messages from the newest to the oldest
• The number of message to return (limit)
• Ask the view to update the index before returning the rows using the stale=false parameter.
• Use startkey parameter if the client send a specific starting point. 

On line 7, the view "chat", "message_history" is called using the Node.js SDK, with the options object.

In the callback function, the application creates an array containing the document id (the keys of the document itself), then on line 13 the messages are retrieved from Couchbase using the get() function. (note: in this function I may have a small issue when multiple messages are sent in the same milliseconds and are just on the edge of the offset)

We have an interesting point to discuss, the view is used only to return the list of keys, and then do a multiple get call with the list of keys. This is most of the time better than returning too much data in the view.

In this first option, the application is using a view to get the message history. This is great, the only thing to look at closely is the fact that this approach uses index and the indexes are stored on the disk. So you need to be sure that the message is saved and the index updated before printing the message in the history, this is why the stale=false is required in this specific scenario.

Second Option : Using a counter as document Key

Let's see now how it is possible, with few changes in the the application, to do the same without using a view and only use the in memory keys. Using this approach the application only use the keys that are all in the memory of the server (memcached).

The application logic stays the same:

1. When user connects to the server the system returns the last 5 messages from the database
2. Each time the user posts a message it should be persisted
3. The user can manually load older messages from the database to view the complete chat history

Post a new message

The key associated to the message is now a counter, and the application use the increment feature of Couchbase:

socket.on('postMessage', function(data) {
  // create a new message
  var message = {
    type: "message",
    user: socket.username,
    message: data,
    timestamp: Date.now()
  }
  couchbase.incr("chat:msg_count", function (data, error, key, cas, value ) { 
    var messageKey = "chat:"+ value;
    message.id = value;
    io.sockets.emit('updateChatWindow', message);
    couchbase.set(messageKey, JSON.stringify(message),function(err) {  }); 
    });
});

Once the message object is created (line 3), the application increments a value chat:msg_count that will be used as message counter (line 9). Note that the Node Couchbase SDK will automatically create the key if it is not present when the incr() method is called.

When the server has returned the new value, increment by 1 with a default value of 0, the callback function is call :

• The value is used to create a new key for the message (line 10)
• The message is push to the different users  (line 12)
• Then the message is saved into Couchbase (line 13)

So what we have here:

• a new item that contains the counter, associated to the key : chat:msg_count
• each message will have a key that looks like chat:0, chat:1, chat:2, ... 

Retrieve messages from Couchase 

Retrieving the older messages from Couchbase is very easy since all the message contains a unique and sequencial id. The showHistory event just need to create a list of keys based on the correct number and get them from Couchbase.

socket.on('showHistory', function(limit,startkey) {
  var keys = new Array();
  for (i = startkey; i > (startkey-limit) && i >= 0 ; i--) {
    keys.push("chat:"+i);
  }
  couchbase.get(keys,function(err, doc, meta) {
    socket.emit('updateChatWindow', doc, true);
  });
});

The line 3-5 are used to create an array of keys, and then in line 6 this array is used to do a multiple get and send the messages to the client using socket.emit.

Here the logic is almost the same that the one used in the previous example. The only difference is the fact that we do not call Couchbase server to create the list of keys to use to print the message history.

Conclusion

As you can see when working with a NoSQL database like any other persistence store you often different ways of achieving the same thing. In this example I used two approaches, one using a view, the other one using the key directly.

The important thing here is to take some time when designing your application to see which approach will be the best for your application. In this example of the chat application I would probably stay with the "Key/Counter" approach that will be the most efficient in term of performance and scalability since it does not use secondary index.