How to implement Document Versioning with Couchbase

Introduction

Developers are often asking me how to "version" documents with Couchbase 2.0. The short answer is: the clients and server do not expose such feature, but it is quite easy to implement.

In this article I will use a basic approach, and you will be able to extend it depending of your business requirements. 

Design

The first thing to do is to select how to "store/organize" the versions of your document, and for this you have different designs:

• copy the versions the document into new documents
• copy the versions of the document into a list of embedded documents
• store the list of attributes that have been changed into a embedded element (or new documents)
• store the "delta"
• …

You will have to chose the design based on your application requirements (business logic, size of the dataset, ...).  For this article, let's use one of the most simplistic approach: create new document for each version with the following rules for the keys:

1. The current version is is a simple Key/Document, no change to the key.
2. The version is a copy of the document, and the version number is added to the key.

This looks like:

Current Version	mykey
Version 1	mykey::v1
Version 2	mykey::v2
...	...

With this approach, existing applications will always use the current version of the document, since the key is not changed. But this approach creates new documents that will be indexed by existing views.

For example, in the Beer Sample application, the following view is used to list the beer by name:

function (doc, meta) {
    if(doc.type && doc.type == "beer") {
        emit(doc.name);
    }
}

It is quite simple to "support" versioning without impacting the existing code, except the view itself. The new view needs to emit keys,value only for the current version of the document. This is the new view code:

function (doc, meta) {
    if(doc.type && doc.type == "beer" && (meta.id).indexOf("::v") == -1   ) {
        emit(doc.name);
    }
}

With this change the existing applications that are using this view will continue to work with the same behavior.

Implementing the versioning

Based on this design, when the application needs to version the document, the following logic should happen:

1. Get the current version of the document
2. Increment the version number (for example using another key that maintains the version number for each document)
3. Create the version with the new key  "mykey::v1"
4. Save the document current version

Let's look at the code in Java

  Object obj = client.get(key);
  if (obj != null) {
    // get the next version, create or use the key: mykey_version
    long version = client.incr(key + "_version", 1, 1); 
    String keyForVersion = key + "::v" + version; // mykey::v1
    try {
        client.set(keyForVersion, obj).get();
    } catch (Exception e) {
        logger.severe("Cannot save version "+ version + " for key "+ key +" - Error:"+ e.getMessage() );
    }
   }
   client.set(key, value);

Quite simple isn't?

The application can access the document using the key, but also get one version or the list of all versions, this is one of the reasons why it is interesting to create a key (mykey_version), and use it also to delete documents and related versions.

Based on the previous comment, the delete operation looks like:

  Object obj = client.get(key);
  // need to delete all the version first
  Object vObject = this.get(key + "_version");
  if (vObject != null) {
    long biggerVersion = Long.parseLong((String) vObject);
    try {
        // delete all the versions
        for (int i = 1; i <= biggerVersion; i++) {
            String versionKey = key + "::v" + i;
            client.delete(versionKey).get();
        }
        // delete the counter
        client.delete(key + "_version").get();
    } catch (InterruptedException e) {
      e.printStackTrace();
    } catch (ExecutionException e) {
      e.printStackTrace();
    }
  }
  client.delete(key);

Use versioning

As an example, I have created a small library available on GitHub https://github.com/tgrall/couchbase-how-to-versioning, this library extends the Couchbase Client and overrides some of the operations : set, replace and delete. (the basic one: no TLL, no durability) As I said before this is just an example.

Build and Install 

git clone https://github.com/tgrall/couchbase-how-to-versioning.git
cd how-to-versioning
mvn clean install

Then add this library to your project in addition to Couchbase Java Client, for example in your pom.xml

...
  
      com.couchbase.howtos
      couchbase-how-to-versioning
      1.0-SNAPSHOT
  
  
      couchbase
      couchbase-client
      1.1.8
  

...

Code your application

Create a document and version it:

 List uris = new LinkedList();
 uris.add(URI.create("http://127.0.0.1:8091/pools"));
 CouchbaseClientWithVersioning client = null
 try {
  client = new CouchbaseClientWithVersioning(uris, "default", "");
  String key = "key-001";
  client.set(key, "This is the original version");
  System.out.printf("Original '%s' .\n", client.get(key));
  client.set(key, "This is a new version", true); // create a new version
  System.out.printf("Current Version '%s' .\n", client.get(key));
  System.out.printf("Version 1 '%s' .\n", client.get(key, 1));
  client.set(key, "This is another version", true); // create a new version
  System.out.printf("All versions %s .\n", client.getAllVersions(key));
  client.deleteVersion(key, 1); // create a new version
  System.out.printf("All versions %s (after delete 1 version).\n", client.getAllVersions(key));
  client.delete(key); // create a new version
  System.out.printf("All versions %s (after delete the main key).\n", client.getAllVersions(key));
 } catch (Exception e) {
  e.printStackTrace();
 }
 if (client !=null) {
  client.shutdown();
 }

Quick explanation:

• Line 5: instead of using the CouchbaseClient, the application uses the extended  CouchbaseClientWithVersioning class.
• Line 7: create a new entry
• Line 9: create a new version, the boolean value to "true" force the versioning of the document
• The application use other methods such as get a specific version (line 11), get all versions (line 13), delete a specific version (line 14), and finally delete the key and all versions (line 16).

So using this approach the developer controls explicitly when to create a version, since he has to add the boolean parameter in the set operation. In this small sample library it is also possible to do auto versioning, in this case all set and replace calls will create a version, to achieve that the developer just needs to call the setAutoVersioning(true) method. Something like:

    client = new CouchbaseClientWithVersioning(uris, "default", "");
    client.setAutomaticVersionning(true);

With this approach you can provide versioning to your application with minimal code change. You can test it in the Beer Sample application, just do not forget to change the views as documenter above to only return current version of the documents.

Conclusion

As you can see doing versioning in Couchbase is not that complicated, but it is something that must be done by your application based on its requirements and constraints. You have many different solution and none of these options is perfect for all use cases.

In this specific sample code, I am working with a simple design where I create a copy of the documents for each version. With this approach also, it is interesting to mention that you can version "anything", not only JSON document but also any values.  As I said before, this is one possible approach, and like any design, it has some impact on the application or database, in this case most the database:

• Increase the number of keys and documents
• Double - or more- the number of operations, for example when updating a document, the application needs to get the current value, create a version, save the current version.
• Consistency management when adding new version and incrementing the version number (need to deal with errors when creating a new version, deleting the versions and counter....)

Many features could be added to this easily, for example:

• Limit to a specific number of version,
• Enable the versioning only of replace() operation
• Add specific attribute about versions in JSON document (for example date of the version)
• ....

If you are using versioning in your Couchbase application feel free to comment or write a small article that describes the way your are doing it.

Deploy your Node/Couchbase application to the cloud with Clever Cloud

Introduction

Clever Cloud is the first PaaS to provide Couchbase as a service allowing developers to run applications in a fully managed environment. This article shows how to deploy an existing application to Clever Cloud.

I am using a very simple Node application that I have documented in a previous article: “Easy application development with Couchbase, Angular and Node”.

Clever Cloud provides support for various databases MySQL, PostgreSQL, but also and this is most important for me Couchbase. No only Clever Cloud allows you to use database services but also you can deploy and host your application that could be developed in the language/technology of your choice : Java, Node, Scala, Python, PHP, … and all this in a secure, scalable and managed environment.

Setting up your Clever Cloud environment

Create your account

1. Go to the Clever Cloud site : http://www.clever-cloud.com/
2. Click on “Login” link and follow the steps to create your account.
3. After few seconds you will received an email and be redirected to the Clever Cloud Console.

Create a Couchbase instance

The Clever Cloud Console allows you to create your Couchbase Bucket in few clicks:

1. Cick on “Services” in the left menu

2.  Click on “Add a Service” in the left menu 

3. Click on “Couchbase” button.

4. Select the size of the RAM quota for your bucket

The size of the RAM Quota for your bucket will have an impact on performance but also on the pricing.

5. Click “Add this Service”

You are done, you should receive an email with all the information to access your newly created bucket.

The mail from Clever Cloud contains the following information:

db_host = xxxxxxxx.couchbase.clvrcld.net	Location of the database, this is where the endpoint is located.
db_name = yyyyyyyy	Name of the Couchbase bucket
db_username = xxxxxxxx	Not used in Couchbase context
db_password = zzzzzzzz	Password to connect to the Couchbase Bucket

So you are now ready to use your bucket.

Note: In the current version of the Clever Cloud Couchbase Service you do not have access to a management console. If you want to get some information about the database or create views you need to do it from you application code.

Connect your Application to Couchbase@Clever-Cloud

The first step is to get some code, so let’s clone the “Couchbase Ideas Sample Application”, and install the dependencies, using the following commands:

git clone -b 03-vote-with-value https://github.com/tgrall/couchbase-node-ideas.git

cd couchbase-node-ideas

git branch mybranch

git checkout mybranch

npm install

Open the app.js and edit the connection info to point your application to the Couchbase instance and modify the HTTP port of your application to 8080 - this is a mandatory step documented here :

dbConfiguration = {
 "hosts": ["xxxxxxxxxxx.couchbase.clvrcld.net:8091"],
 "bucket": "xxxxxxxxxxx",
 "user": "xxxxxxxxxx",
 "password": "yyyyyyyyyyyyyyyyyyyyyyyyy"

};
...
...

  appServer = app.listen(8080, function() {
 console.log("Express server listening on port %d in %s mode", appServer.address().port, app.settings.env);
  });

Launch your application using

node app.js

Go to http://localhost:8080

Your application is now using Couchbase on the cloud powered by Clever Cloud. Let’s now deploy the application itself on Clever Cloud

Deploy your application on Clever Cloud

The easiest way to deploy an application to Clever Cloud is using git. The first thing to do is to add your SSH public key into Clever Cloud Console. If you do not have any SSH yet, follow the steps described on Github : “Generating SSH Keys”.

Add your SSH key

Note: As you can guess this should be done only once
Open the id_rsa.pub file with a text editor. This is your SSH key. Select all and copy to your clipboard.

1. Go to the Clever Cloud Console
2. Click on “Profile” entry in the left menu
3. Click on “SSH Keys”
4. Click on “Add a SSH Key”
5. Enter a name (anything you want) and paste your key
6. Click “Add” button

You are now ready to deploy applications to Clever Cloud. The next thing to do, is to create a new node application in Clever Cloud.

Create your Application

1. Click “Add an app” in the Application menu in the top menu.
2. Give a name and description to this application
3. Select the Instance type, in this case “Node.js”
4. Configure your instances, you can keep the default values for now, click “Next”
5. Check the configuration, and click “Create”

Your application is created, you are redirected to the generic information page, where you can find a Git URL that we will use to deploy the application.
You can navigate into the entries in the left menu to see more information about your application. In addition to the Information page, you can look at the following entries:

1. “Domain Names” to configure the URL to access your application
2. “Logs” to view the application logs

Deploy the Application

So we are almost there!
The deployment to Clever Cloud is done using a Git push command, so you need to add the deployment URL as a remote repository to your application, using the following command:

git remote add clever git+ssh://git@push.clever-cloud.com/app_[your_app_id].git

git commit -a -m “Couchbase on Clever Cloud connection”

git push clever mybranch:master

Once you have added the application as remote repository you can commit and push your application.

The last command pushes the application  to Clever Cloud. It is important to note that Clever Cloud will always deploy the application on the “master” branch on the remote repository. The notation mybranch:master is used to mention it. If you work locally on your master branch just use “master”.

You can now go to the Clever Cloud console and look in the log and click on the URL in the “Domain Names” section to test your application.

You should be able to see your application, that is running on the Clever Cloud PaaS.

When you update your application, you just need to do a  git push and git commit.

Conclusion

In this tutorial you have learned how to:

• Create your Clever Cloud account
• Create a Couchbase instance
• Create and deploye a Node.js application

Feel free to test this yourself, with Node or other technology, as you can see it is quite easy to setup.

SQL to NoSQL : Copy your data from MySQL to Couchbase

TL;DR: Look at the project on Github.

Introduction

During my last interactions with the Couchbase community, I had the question how can I easily import my data from my current database into Couchbase. And my answer was always the same:

• Take an ETL such as Talend to do it
• Just write a small program to copy the data from your RDBMS to Couchbase...

So I have written this small program that allows you to import the content of a RDBMS into Couchbase. This tools could be used as it is, or you can look at the code to adapt it to your application.

The Tool: Couchbase SQL Importer

The Couchbase SQL Importer, available here, allows you with a simple command line to copy all -or part of- your SQL schema into Couchbase. Before explaining how to run this command, let's see how the data are stored into Couchbase when they are imported:

• Each table row is imported a single JSON document
• where each table column becomes a JSON attribute
• Each document as a key made of the name of the table and a counter (increment)

The following concrete example, based on the MySQL World sample database, will help you to understand how it works. This database contains 3 tables : City, Country, CountryLanguage. The City table looks like:

+-------------+----------+------+-----+---------+----------------+
| Field       | Type     | Null | Key | Default | Extra          |
+-------------+----------+------+-----+---------+----------------+
| ID          | int(11)  | NO   | PRI | NULL    | auto_increment |
| Name        | char(35) | NO   |     |         |                |
| CountryCode | char(3)  | NO   |     |         |                |
| District    | char(20) | NO   |     |         |                |
| Population  | int(11)  | NO   |     | 0       |                |
+-------------+----------+------+-----+---------+----------------+

The JSON document that matches this table looks like the following:

city:3805
{ 
  "Name": "San Francisco",
  "District": "California",
  "ID": 3805,
  "Population": 776733,
  "CountryCode": "USA"
}

You see that here I am simply taking all the rows and "moving" them into Couchbase. This is a good first step to play with your dataset into Couchbase, but it is probably not the final model you want to use for your application; most of the time you will have to see when to use embedded documents, list of values, .. into your JSON documents.

In addition to the JSON document the tool create views based on the following logic:

• a view that list all imported documents with the name of the "table" (aka type) as key
• a view for each table with the primary key columns

View: all/by_type

{
  "rows": [
    {"key": "city", "value": 4079}, 
    {"key": "country", "value": 239}, 
    {"key": "countrylanguage", "value": 984}
   ]
}

As you can see this view allows you to get with a single Couchbase query the number of document by type. 

Also for each table/document type, a view is created where the key of the index is built from the table primary key. Let's for example query the "City" documents.

View: city/by_pk?reduce=false&limit=5

{
  "total_rows": 4079,
  "rows": [
    {"id": "city:1", "key": 1, "value": null}, 
    {"id": "city:2", "key": 2, "value": null}, 
    {"id": "city:3", "key": 3, "value": null}, 
    {"id": "city:4", "key": 4, "value": null},
    {"id": "city:5", "key": 5, "value": null}
  ]
}

The index key matches the value of the City.ID column.  When the primary key is made of multiple columns the key looks like:

View: CountryLanguage/by_pk?reduce=false&limit=5

{
  "total_rows": 984,
  "rows": [
    {"id": "countrylanguage:1", "key": ["ABW", "Dutch"], "value": null}, 
    {"id": "countrylanguage:2", "key": ["ABW", "English"], "value": null}, 
    {"id": "countrylanguage:3", "key": ["ABW", "Papiamento"], "value": null},
    {"id": "countrylanguage:4", "key": ["ABW", "Spanish"], "value": null},
    {"id": "countrylanguage:5", "key": ["AFG", "Balochi"], "value": null}
  ]
}

This view is built from the CountryLanguage table primary key made of CountryLanguage.CountryCode and CountryLanguage.Language columns.

+-------------+---------------+------+-----+---------+-------+
| Field       | Type          | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+-------+
| CountryCode | char(3)       | NO   | PRI |         |       |
| Language    | char(30)      | NO   | PRI |         |       |
| IsOfficial  | enum('T','F') | NO   |     | F       |       |
| Percentage  | float(4,1)    | NO   |     | 0.0     |       |
+-------------+---------------+------+-----+---------+-------+

How to use Couchbase SQL Importer tool? 

The importer is a simple Java based command line utility, quite simple to use:

1. Download the CouchbaseSqlImporter.jar file from here. This file is contains all the dependencies to work with Couchbase: the Java Couchbase Client, and GSON.

2. Download the JDBC driver for the database you are using as data source. For this example I am using MySQL and I have download the driver for MySQL Site.

3. Configure the import using a properties file.

## SQL Information ##
sql.connection=jdbc:mysql://192.168.99.19:3306/world
sql.username=root
sql.password=password

## Couchbase Information ##
cb.uris=http://localhost:8091/pools
cb.bucket=default
cb.password=

## Import information
import.tables=ALL
import.createViews=true
import.typefield=type
import.fieldcase=lower

This sample properties file contains three sections :

• The two first sections are used to configure the connections to your SQL database and Couchbase cluster (note that the bucket must be created first)
• The third section allow you to configure the import itself
• import.tables : ALL to import all tables, or a the list of tables you want to import, for example City, Country
• import.createViews : true or false, to force the creation of the views.
• import.typefield : this is use to add a new attribute in all documents that contains the "type".
• import.fieldcase : null, lower, upper : this will force the case of the attributes name and the value of the type (City or city or CITY for example).

4. Run the tool !

java -cp "./CouchbaseSqlImporter.jar:./mysql-connector-java-5.1.25-bin.jar" com.couchbase.util.SqlImporter import.properties 

So you run the Java command with the proper classpath (-cp parameter).

And you are done, you can get your data from your SQL database into Couchbase.

If you are interested to see how it is working internally, you can take a look to the next paragraph.

The Code: How it works?

The main class of the tool is really simple  com.couchbase.util.SqlImporter, the process is:

1. Connect to the SQL database

2. Connect to Couchbase

3. Get the list of tables

4. For each tables execute a "select * from table"

  4.1. Analyze the ResultSetMetadata to get the list of columns

  

  4.2. Create a Java map for each rows where the key is the name of the columns and the value…is the value

  4.3. Serialize this Map into a GSON document and save it into Couchbase

The code is available in the ImportTable(String table) Java method.

One interesting point is that you can use and extend the code to deal with your application.

Conclusion

I have created this tool quickly to help some people in the community, if you are using it and need new features, let me know, using comment or pull request.