Sunday, December 30, 2012

Couchbase 101: Create views (MapReduce) from your Java application

When you are developing a new applications with Couchbase 2.0, you sometimes need to create view dynamically from your code. For example you may need this when you are installing your application, writing some test, or you can also use that when you are building frameworks, and wants to dynamically create views to query data. This post shows how to do it.

Prerequisites

If you are using Maven you can use the following information in your pom.xml to add the Java Client library:

  
    
      couchbase
      Couchbase Maven Repository
      default
      http://files.couchbase.com/maven2/
      
        false
      
    
  
  
  
    
      couchbase
      couchbase-client
      1.1.0
      jar
    
  
See online at https://gist.github.com/4337172

Create and Manage Views From Java 

The full Maven project is available on Github.

Connect to Couchbase Cluster

The first thing to do when you want to create a view from Java is obviously to connect to the cluster.

import com.couchbase.client.CouchbaseClient;
...
...

    List uris = new LinkedList();
    uris.add(URI.create("http://127.0.0.1:8091/pools"));
    CouchbaseClient client = null;
    try {
        client = new CouchbaseClient(uris, "beer-sample", "");

        // put your code here

        client.shutdown();      

    } catch (Exception e) {
        System.err.println("Error connecting to Couchbase: " + e.getMessage());
        System.exit(0);
    }

...
...

  1. Create a list of URIs to different nodes of the cluster - lines 5-6. (In this example I am working on a single node)
  2. Connect to the bucket, in our case beer-sample -line 9. You can include the password if the bucket is protected ( this is not the case here so I am sending an empty string)
If you are looking for more information about Couchbase and Java, you can read this article from DZone : Hello World with Couchbase and Java.

Let's now talk about Couchbase views. You use views/map-reduce functions to index and query data from Couchbase Server based on the content of the JSON document you store inside Couchbase. For more information about views you can look at the "view basics" chapter of the Couchbase Server Manual.

Create Views from Java

Creating a view from Java is really easy : the Java Client Library contains all the classes and methods to do it. As a concrete use case we will use the Application that is described in the Couchbase Java Tutorial.

When you follow this tutorial, you need to manually create some views, as you can see here. In this example, we will create our map function and directly in our Java code and then store it to Couchbase Server. The tutorial asks you to create the following artifacts:
  • a view named "by_name
  • in the design document named "dev_beer" (development mode)
  • and the map function which looks like the following :
 function (doc, meta) {
   if(doc.type && doc.type == "beer") {
     emit(doc.name, null);
   }
 }


The following code allows you to do it from Java:

import com.couchbase.client.protocol.views.DesignDocument;
import com.couchbase.client.protocol.views.ViewDesign;
...
    DesignDocument designDoc = new DesignDocument("dev_beer");

    String viewName = "by_name";
    String mapFunction =
            "function (doc, meta) {\n" +
            "  if(doc.type && doc.type == \"beer\") {\n" +
            "    emit(doc.name);\n" +
            "  }\n" +
            "}";

    ViewDesign viewDesign = new ViewDesign(viewName,mapFunction);
    designDoc.getViews().add(viewDesign);
    client.createDesignDoc( designDoc );
...


  • Create a design document using the com.couchbase.client.protocol.views.DesignDocument class - line 4.
  • Create a view using com.couchbase.client.protocol.views.ViewDesign class with a name and the map function - line 14.
  • You can add this view to a design document - line 15
  • Finally save the document into the cluster using the CouchbaseClient.createDesignDoc method.
If you need to use a reduce function (built-in or custom) you just need to pass to the ViewDesign constructor as 3rd parameter.

When developing view, from Java or from any other tool/language be sure you understand what are the best practices, and the life cycle of the index. This is why I am inviting you to take a look to the following chapters in the Couchbase documentation:

Using the view

First of all, the view that you just created is in "development mode", and by default the Java client SDK will only access the view when it is in "production mode". This means that when you are calling a view from your application it will search it into the production environment. So before connecting to Couchbase cluster you need to setup the viewmode to development.

This is done using the viewmode environment variable from the Java SDK, that could be set using the following methods:
  • In your code, add this line before the client connects to the cluster : System.setProperty("viewmode", "development");
  • At the command line -Dviewmode=development
  • In a properties file viewmode=development
Once it is done you can call the view using the following code:
import import com.couchbase.client.protocol.views.*;

...
   System.setProperty("viewmode", "development"); // before the connection to Couchbase
...
   View view = client.getView("beer", "by_name");
   Query query = new Query();
   query.setIncludeDocs(true).setLimit(20);
   query.setStale( Stale.FALSE );
   ViewResponse result = client.query(view, query);
   for(ViewRow row : result) {
     row.getDocument(); // deal with the document/data
   }
...

This code queries the view you just created. This means Couchbase Server will generate an index based on your map function, will query the server for results. In this case, we specifically want to set a limit of 20 results and also get the most current results by setting Stale.FALSE.
  • Set the viewmode to development - line 4
  • Get the view using the CouchbaseClient.getView() method -line 6-. As you can see I just use the name beer for the design document (and not dev_beer, Couchbase will know where to search since I am in development mode)
  • Create a query and set a limit (20) and ask the SDK to return the document itself
    setIncludeDocs(true) -line 8- The document will be returned from Couchbase server in the most efficient way
  • Ask the system to update the index before returning the result using query.setStale( Stale.FALSE ); -line 9-. Once again be careful when you use setStale method. Just to be sure here is the documentation about it : Index Updates and the stale Parameter
  • Execute the query - line 10
  • And use the result - lines 11-13

Conclusion

In this article, you have learned:
  • How to create Couchbase views from Java
  • Call this view from Java
  • Configure development/production mode views from Couchbase Java Client Library
This example is limited to the creation of a view, you can take a look to the other methods related to design documents and views if you want to manage your design documents : getDesignDocument(), deleteDesignDocument(), ... .


Wednesday, December 26, 2012

What to do if your Couchbase Server does not start?

Working with the Couchbase 2.0.0 release you may have issues when trying to access the Web Admin Console or simply starting the server. This is due to the way Couchbase Server uses the IP address/hostname during the installation process. So when you have one of the following errors :
  • On Windows, Server is not working at all, even after installation. You can access the sever on port 8092 (Couchbase API port), but cannot on port 8091
  • You have the following error when accessing the console
    "[10:02:02] IP address seems to have changed. Unable to listen on 'ns_1@10.0.2.15'" 


  • When you try to restart the server it does not start and you have the following error message in the error log :
    "Configured address '10.0.2.15' seems to be invalid. Will refuse to start for safety reasons" 
Some of these issues are related to a known issue on Windows ( see MB-7417 that will be fixed in 2.0.1) or the fact that Couchbase server does not support change of the IP address after installation.  This is documented in the section “Using Couchbase in the Cloud: Handling Changes in IP Addresses” of the Couchbase Server Manual. This article explains what should be done when configuring Couchbase Server on Windows, but you can do equivalent steps on any platform using the shell scripts available on Linux and/or Mac OS X.

Once you have installed Couchbase, you can see in the console that the IP address of your server is used :


Typically the address 192.168.0.97 is stored in the configuration of Couchbase. If your server receives a new address from the DHCP server, Couchbase will not work anymore. In this article you will see how you can configure Couchbase to use another IP address or Hostname.

Important: The steps that follow will completely destroy any data and configuration from the node, so it is best to start with a fresh Couchbase install. If you can not, you should backup your data using the file based backup-restore documented here.

Setting up a hostname in hosts file
The best practice is to register Couchbase using a hostname instead of an IP Address. For this you will need to associate this hostname to an IP address in the hosts file.

Since the hosts file is part of the system, you need to edit it as administrator. You have different approaches to achieve this:

The following steps explain the easiest way to do it: 
  1. Click “Start Menu”
  2. Navigate to “All Programs > Accessories”
  3. “Right Click” on “Notepad” (or your favorite text editor)
  4. Click “Run as Administrator”

You can now open the file C:\Windows\System21\drivers\etc\hosts

Add a new entry with a host name that will be used by Couchbase Server for example something like :
127.0.0.1    couchbase-node1.mycompany.com

Here I am using the local address (127.0.0.1) like that I won't have to change it even when my IP address changes. (This is useful when you are working in a single node mode)

Configure Couchbase to use the hostname or new IP address

During the installation Couchbase has been registered as a Windows Service. To be able to associate Couchbase to the new hostname (or IP address) the service needs to re-configured and reinstalled.

This could be done using the scripts provided with the product. To run the scripts you need to do it as administrators, you can do it with one of the following methods:
  • Search for the file and right click and select  Run as Administrator  (documented below)
  • Run the terminal as administrator and run all the command from there (documented below)
  • Search for the file and run it using Ctrl+Shift+Enter

Option 1 : Using the Start Menu and Search Program

Stop Couchbase Server Windows Service
The first thing to do, is to stop this service that is automatically started after the installation:
  1. Click Start Menu
  2. Type Services in the Search Program form
  3. Click on Services
  4. In the Services Application navigate to CouchbaseServer
  5. Right Click and Click on Stop 
  6. Couchbase is now stopped. 
Edit the Service Register script
Note: Due to a small formatting issue (See MB-7322), Notepad could not be used, a solution is to take Notepad++ or any other advanced editing tool.
  1. As Administrator, open the
    C:\Program Files\Couchbase\Server\bin\service_register.bat file with your favorite editor. To open the editor as Administrator you can use the approach described in the previous step.
  2. Edit the line 9 to replace %IP_ADDR% by your hostname, the line should look like:
    set NS_NAME=ns_1@couchbase-node1.mycompany.com
  3. Save the file
Delete existing configuration and logs
  1. Using the file explorer, go into:
     C:\Program Files\Couchbase\Server\var\lib\couchbase\mnesia
  2. Delete its content (Select All and Right Click)
Register the new Configuration as Service
  1. Using the file explorer, go into:
    C:\Program Files\Couchbase\Server\bin
  2. Right Click on service_reregister.bat
  3. Click on Run as Administrator
This script recreates the Couchbase Server Windows Service and starts it automatically.

Check the configuration
  1. Launch your Internet Browser
  2. Go to http://localhost:8091
  3. Follow the Couchbase Installation Steps
  4. Once install connect to the console
  5. Go to Server Nodes tab
  6. Check that the server name is now couchbase-node1.mycompany.com
Your Couchbase node is now configured to use the hostname of your server.



Option 2 : Using the Command Line

Launch Command Prompt as Administrator
  1. Click Start Menu
  2. Type Command Prompt in the Search program form
  3. Type Ctrl+Shift+Enter
  4. Go to C:\Program Files\Couchbase\Server\bin (or other location if you have chosen another location during installation)
You are now ready to do the administration tasks.
  1. Execute the service_stop.bat
  2. Edit the Service Register script
    1. Open service_register.bat
    2. Edit the line 9 to replace %IP_ADDR% by your hostname (or your IP address), the line should look like:
      set NS_NAME=ns_1@couchbase-node1.mycompany.com
    3. Save the file
  3. Delete the content of:
     C:\Program Files\Couchbase\Server\var\lib\couchbase\mnesia
  4. Execute the service_reregister.bat
    This script recreates the Couchbase Server Windows Service and starts it automatically.

Check the configuration
  1. Launch your Internet Browser
  2. Go to http://localhost:8091
  3. Follow the Couchbase Installation Steps
  4. Once install connect to the console
  5. Go to Server Nodes tab
  6. Check that the server name is now couchbase-node1.mycompany.com

Your Couchbase node is now configured to use the hostname of your server.


Saturday, December 1, 2012

New Adventure...

Yesterday was my last day at eXo... I have been working at eXo since 2008, and we have achieved many exciting things such as building eXo Platform, the open source social platform, and the Cloud-IDE allowing developers to build, test, and deploy applications online.

It was a great experience for me, but it is time for me to jump into a new adventure...

I am joining Couchbase as a Technical Evangelist for the EMEA Region. When I have started to work with NoSQL engines (starting with Google BigTable for Resultri) I really enjoyed the experience and the flexibility that it gives to the developers. Then I choose to work on a documented oriented database because it looks very natural to me, and evaluate the clustering capabilities. This is how I discovered Couchbase and its community.

This new job is a great opportunity for me to do the things that I really like about software:

  • Coding applications using different languages and frameworks
  • Understanding the sysops/devops related challenges when using a new product/technology
  • And finally probably the most important part; sharing it with others!
I look forward to sharing what I like about NoSQL and Couchbase and discuss with others about their experience or needs around NoSQL. 

See you soon online and in real life during conferences and meetups!

Couchbase Logo

Tuesday, November 13, 2012

Building a chat application using Node.js and Couchbase

After some basic articles about Couchbase installation, Node.js integration. Let's now dive into a more complete  example: a chat application.

The first version of the chat should be compliant with the following requirements:

  • web based
  • single room
  • user just needs to enter a login and he can start to interact with other connected users
  • user should be able to navigate into the chat history

The Couchbase Chat application is build using the following components
  • Node.js for the application
  • Couchbase to persist all the messages

I won't go in all the detail of the design of the Node.js application. You can find many example of Node based chat application. I prefere to focus on how I have design the persistence using Couchbase more than the application itself. If you want me to give more detail about the complete application feel free to drop me a message/comment and I will do it.

What are the challenges with persisting the messages?
Storing the information is quite easy, just "dump" the message information in your database. The challenge is more around the fact that user want to access the history of the messages. So the key point here is how to store the information in a way that it is easy to get back in a sorted fashion.

You will find many different ways of achieving that depending of the technology you are using and they query capabilities of your persistence engine. Using Couchbase you have two ways to access/find the data:

  • Using Views that allows you to query and secondary level index and do advanced operation such as sorting, query on key range, ...
  • Directly access the data using its key

In this post I will show you how you can use the the two options to build your application and retrieved information that are stored and retrieved in a specific order:

  • First Options: using a view to get the message history
  • Second Options: using a counter as a key for the messages

The source code of the application is available in Github : https://github.com/tgrall/couchbase-chat


Get the Couchbase connection

The following code is used to connect to Couchbase, once it is done, the Web server is started:

var express = require('express');
var app = express();
var http = require('http');
var server = http.createServer(app);
var io = require('socket.io').listen(server);

var driver = require('couchbase');

driver.connect({
 "username": "",
 "password": "",
 "hostname": "localhost:8091",
 "bucket": "default"}, 
 function(err, couchbase) {
  if (err) {
   throw (err)
  }

  server.listen(8080);

  app.get('/', function(req, res) {
   res.sendfile(__dirname + '/index.html');
  });
...
// Application code
// Socket.io events
...
});

Let's now see how Couchbase is used in the chat application.

First Option : Using views to get the message history

Post a new message
In this example messages are formatted using the following information:

{
  "type": "message",
  "user": "Tug",
  "message": "Hello all !",
  "timestamp": 1349836768909
}

The key is based on the timestamp and  the user name : 1349836768909-Tug. I am adding the user name to be sure that the key is unique. Like that I do not have to manage conflicts.

The insertion of the message :

  socket.on('postMessage', function(data) {
    // create a new message
    var message = {
      type: "message",
      user: socket.username,
      message: data,
      timestamp: Date.now()
    }
    var messageKey = message.timestamp +"-"+ message.user;
    io.sockets.emit('updateChatWindow', message);
    couchbase.set(messageKey, JSON.stringify(message),function(err) {  }); 
  });

  • The postMessage event is called by the client when the user post a new message. 
  • A new message object is created with : a type, the user, the message itself and a timestamp.
  • The message is sent to the different clients using the io.sockets.emit() function (line 10)
  • Finally the message is saved into Couchbase (line 11). As you can see the only thing you have to do is to send the Javascript object as a simple JSON String.
At this point your application work perfectly, all the connected user will see the new messages since they are sent by the server as soon as they are created. But it is not possible for a user to navigate in the chat history and see older messages.


Retrieve messages from Couchase 

As explained earlier, it is possible to use a view to retrieve the message from the database in a proper order.  The view looks like that:

function (doc, meta) { 
  if ( meta.type == "json" && doc.type == "message" ) {
   emit(doc.timestamp, null);
  }
}

Each time a new document is inserted in the database, if this is a JSON document and the type of this document is "message" the index will be updated. When this view is called the result looks like :

{"id":"1352733392477-JOHN","key":1352733392477,"value":null},

As you can see the id of the document (timestamp-username) is automatically inserted in the response.


You can use the following command to insert the view in your Couchbase Server: (configure the server address, port and bucket accordingly to your environment)
curl -X PUT -H 'Content-Type: application/json' http://127.0.0.1:8092/default/_design/chat -d '{"views":{"message_hisory":{"map":"function (doc, meta) {\n  \n  if ( meta.type == \"json\" && doc.type == \"message\" ) {\n   emit(doc.timestamp, null);\n  }\n}"}}}'



The application now calls the view using the following code

socket.on('showhistory', function(limit,startkey) {
  limit = (limit == undefined || limit == 0)? 5 : limit;
  var options = {"descending": "true", "limit" : limit, "stale" : "false"};
  if (startkey > 0) {
    options.startkey = startkey-1;
  }
  couchbase.view("chat","message_hisory", options , function(err, resp, view) {
    var rows = view.rows;
    var keys = new Array();
    for( var i = 0; i < rows.length ; i++  ) {
      keys.push( rows[i].id );
    }
    couchbase.get(keys,function(err, doc, meta) {
      socket.emit('updateChatWindow', doc, true);
    });
  });
});

When the client send a showHistory event the application capture this event and call the view with proper parameters to send back the list of messages to the client.

The options object contains the different parameters that will be used to call the view:
  • Use descending order to return the messages from the newest to the oldest
  • The number of message to return (limit)
  • Ask the view to update the index before returning the rows using the stale=false parameter.
  • Use startkey parameter if the client send a specific starting point. 
On line 7, the view "chat", "message_history" is called using the Node.js SDK, with the options object.

In the callback function, the application creates an array containing the document id (the keys of the document itself), then on line 13 the messages are retrieved from Couchbase using the get() function. (note: in this function I may have a small issue when multiple messages are sent in the same milliseconds and are just on the edge of the offset)

We have an interesting point to discuss, the view is used only to return the list of keys, and then do a multiple get call with the list of keys. This is most of the time better than returning too much data in the view.

In this first option, the application is using a view to get the message history. This is great, the only thing to look at closely is the fact that this approach uses index and the indexes are stored on the disk. So you need to be sure that the message is saved and the index updated before printing the message in the history, this is why the stale=false is required in this specific scenario.


Second Option : Using a counter as document Key

Let's see now how it is possible, with few changes in the the application, to do the same without using a view and only use the in memory keys. Using this approach the application only use the keys that are all in the memory of the server (memcached).

The application logic stays the same:
  1. When user connects to the server the system returns the last 5 messages from the database
  2. Each time the user posts a message it should be persisted
  3. The user can manually load older messages from the database to view the complete chat history

Post a new message
The key associated to the message is now a counter, and the application use the increment feature of Couchbase:
socket.on('postMessage', function(data) {
  // create a new message
  var message = {
    type: "message",
    user: socket.username,
    message: data,
    timestamp: Date.now()
  }
  couchbase.incr("chat:msg_count", function (data, error, key, cas, value ) { 
    var messageKey = "chat:"+ value;
    message.id = value;
    io.sockets.emit('updateChatWindow', message);
    couchbase.set(messageKey, JSON.stringify(message),function(err) {  }); 
    });
});

Once the message object is created (line 3), the application increments a value chat:msg_count that will be used as message counter (line 9). Note that the Node Couchbase SDK will automatically create the key if it is not present when the incr() method is called.

When the server has returned the new value, increment by 1 with a default value of 0, the callback function is call :

  • The value is used to create a new key for the message (line 10)
  • The message is push to the different users  (line 12)
  • Then the message is saved into Couchbase (line 13)


So what we have here:
  • a new item that contains the counter, associated to the key : chat:msg_count
  • each message will have a key that looks like chat:0, chat:1, chat:2, ... 

Retrieve messages from Couchase 
Retrieving the older messages from Couchbase is very easy since all the message contains a unique and sequencial id. The showHistory event just need to create a list of keys based on the correct number and get them from Couchbase.

socket.on('showHistory', function(limit,startkey) {
  var keys = new Array();
  for (i = startkey; i > (startkey-limit) && i >= 0 ; i--) {
    keys.push("chat:"+i);
  }
  couchbase.get(keys,function(err, doc, meta) {
    socket.emit('updateChatWindow', doc, true);
  });
});

The line 3-5 are used to create an array of keys, and then in line 6 this array is used to do a multiple get and send the messages to the client using socket.emit.

Here the logic is almost the same that the one used in the previous example. The only difference is the fact that we do not call Couchbase server to create the list of keys to use to print the message history.

Conclusion

As you can see when working with a NoSQL database like any other persistence store you often different ways of achieving the same thing. In this example I used two approaches, one using a view, the other one using the key directly.

The important thing here is to take some time when designing your application to see which approach will be the best for your application. In this example of the chat application I would probably stay with the "Key/Counter" approach that will be the most efficient in term of performance and scalability since it does not use secondary index.







Monday, November 5, 2012

Couchbase : Create a large dataset using Twitter and Java

An easy way to create large dataset when playing/demonstrating Couchbase -or any other NoSQL engine- is to inject Twitter feed into your database.

For this small application I am using:

In this example I am using Java to inject Tweets into Couchbase, you can obviously use another langage if you want to.

The sources of this project are available on my Github repository  Twitter Injector for Couchbase you can also download the Binary version here, and execute the application from the command line, see Run The Application paragraph. Do not forget to create your Twitter oAuth keys (see next paragraph) 


Create oAuth Keys

The first thing to do to be able to use the Twitter API is to create a set of keys. If you want to learn more about all these keys/tokens take a look to the oAuth protocol : http://oauth.net/


1. Log in into the Twitter Development Portal : https://dev.twitter.com/

2. Create a new Application 
Click on the "Create an App" link or go into the "User Menu > My Applications > Create a new application"

3. Enter the Application Details information



4. Click "Create Your Twitter Application" button

Your application's OAuth settings are now available :


5- Go down on the Application Settings page and click on the "Create My Access Token" button



You have now all the necessary information to create your application:
  • Consumer key 
  • Consumer secret
  • Access token
  • Access token secret

These keys will be uses in the twitter4j.properties file when running the Java application from the command line see



Create the Java Application

The following code is the main code of the application:

Some basic explanation:

  • The setUp() method simply reads the twitter4j.properties file from the classpath to build the Couchbase connection string.
  • The injectTweets opens the Couchbase connection -line 76- and calls the TwitterStream API. 
  • A Listener is created and will receive all the onStatus(Status status) from Twitter. The most important method is onStatus() that receive the message and save it into Couchbase. 
  • One interesting thing : since Couchbase is a JSON Document database it allows your to just take the JSON String and save it directly.
    cbClient.add(idStr,0 ,twitterMessage);

Packaging
To be able to execute the application directly from the Jar file, I am using the assembly plugin with the following informations from the pom.xml :




  ... 
  
    
     com.couchbase.demo.TwitterInjector
    
    
     .
    
  
  ...

Some information:

  • The mainClass entry allows you to set which class to execute when running java -jar command.
  • The Class-Path entry allows you to set the current directory as part of the classpath where the program will search for the twitter4j.properties file.
  • The assembly file is also configure to include all the dependencies (Twitter4J, Couchbase client  SDK, ...)
If you do want to build it from the sources, simply run :

mvn clean package

This will create the following Jar file ./target/CouchbaseTwitterInjector.jar



Run the Java Application

Before running the application you must create a twitter4j.properties file with the following information :

twitter4j.jsonStoreEnabled=true

oauth.consumerKey=[YOUR CONSUMER KEY]
oauth.consumerSecret=[YOUR CONSUMER SECRET KEY]
oauth.accessToken=[YOUR ACCESS TOKEN]
oauth.accessTokenSecret=[YOUR ACCESS TOKEN SECRET]

couchbase.uri.list=http://127.0.0.1:8091/pools
couchbase.bucket=default
couchbase.password=

Save the properties file and from the same location run:


jar -jar [path-to-jar]/CouchbaseTwitterInjector.jar


This will inject Tweets into your Couchbase Server. Enjoy !

Monday, September 24, 2012

Create a Simple Node.js and Couchbase application... on OS X

NOTE: The Couchbase Node.js Client Library is currently changing. I will update this article and source code once the API is stable.

I am currently playing a little bit with Node.js . It is quite fun! In this article I won't go in a a very complex application but just give you the basic steps to create your first Node.js+Couchbase application... on Mac OS X.

Installation

Couchbase 2.0 Beta:
You can take a look the first steps of my previous article to install Couchbase. The basics steps are:

  • Download Couchbase 2. 0 Beta 
  • Start the server (Run the "Couchbase Server" application)
  • Configure the server

Other Components :
  • Node.js
  • Couchbase Client Library (C version)
To install these two components I am using homebrew (aka brew).

Hmmm, what is homebrew?

Homebrew is a package manager for OS X that allows you to install, update and uninstall unix tools using very simple commands. You can find more information on the homebrew site. So let's start by installing homebrew itself.

From a terminal window:
ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"

Then let's install node
brew install node

and finally install Couchbase Client Library
brew install https://github.com/couchbase/homebrew/raw/preview/Library/Formula/libcouchbase.rb

Coding

You are now ready to start the development of your application. 

Create a very simple application

1. Create a folder for your application
mkdir sample-app
cd sample-app
2. Create an app.js file and copy the following code:
var http = require("http");

function onRequest(request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.write("Hello World");
  response.end();
}

var server = http.createServer(onRequest);
server.listen(8080);

console.log("> SERVER STARTED");
I won't go in all the details about a node application; for this I invite you to read The Node Beginnger Book.

3. Start your server
node app.js
You should be able to access your application using http://localhost:8080
To stop your server just use ctrl+c.

Install and use Couchbase client for node.js

npm is the node package manager, that allows you to easily install node.js modules and manage dependencies for your application (I will present the dependency management in another post)

1. Install the Couchbase Client for node.js, using the following command
npm install couchbase
Couldn't be simpler! You can find more information about Couchbase Client for node on its npm page.

2. Lets now insert some data into Couchbase

Create a simple function that inserts some documents, and call it after server startup:
var http = require("http");

// load the Couchbase driver and connect to the cluster
var driver = require('couchbase');
var cb = new driver.Couchbase("localhost:8091", null, null, "default");

...
...

function insertData() {
 // insert employees in Couchbase
 var emps = [{
  "type": "employee",
  "id": 100,
  "name": "Thomas",
  "dept": "Sales",
  "salary": 5000
 }, {
  "type": "employee",
  "id": 200,
  "name": "John",
  "dept": "Development",
  "salary": 4500
 }, {
  "type": "employee",
  "id": 300,
  "name": "Jane",
  "dept": "Marketing",
  "salary": 5000
 }]

 // Insert the data in Couchbase using the add method ()
 for (index = 0; index < emps.length; index++) {
  cb.add(JSON.stringify(emps[index].id), JSON.stringify(emps[index]), 0, undefined, function(data, err, key, cas) {
   if (err && err != 12) { // 12 : LCB_KEY_EEXISTS  
    console.log("Failed to store object:\n" + err);
   }
  });
 }
}

server.listen(8080, insertData());

  • Lines 4-5 : load the Couchbase driver and connect to the server. I am using the complete list of parameter connect("server:port", "username", "password", "bucket"). But you can use the short version connect("server:port")
  • Lines 12-30 : just create JSON object that will be pushed in Couchbase.
  • Line 33 : the application just read each element of the array and insert them into Couchbase using the couchbase.add() function.
  • Lines 34-38 : the couchbase.add() function set the value only if it does not exist. If the value exists the error code LCB_KEY_EEXISTS (12) is return by the callback function

3. Start your server - node app.js - and check using the Admin Console that employees are inserted into your Couchbase instance. Note that node.js applications do not support hot deployment, so you need to bounce your application when changing the code.


Create and use Couchbase View

Let's now create and use a view to return the employee list.

1. All Couchbase views are accessible using a simple REST API, but you can also use the node.js plugin : baseview; so let's install this module:

npm install baseview


2. Create a new view from your application

You can use the Admin Console to create the view, but it is also possible to do it from your node.js code. So let's add the view programmatically in the insertData function.
var http = require("http");

// load the Couchbase driver and connect to the cluster
var driver = require('couchbase');
var cb = new driver.Couchbase("localhost:8091", null, null, "default");
// load the baseview module
var baseview = require('baseview')({url: 'http://localhost:8092', bucket: 'default'});


...
...

function insertData() {

 //create a new view
  baseview.setDesign('design_employees', {
     'basic_list': {
        'map': "function (doc, meta) { if(doc.type == 'employee') {emit(meta.id, doc.name);}}"
      }
    },
    function(err, res){
  if (err != null) console.log(err);
    }
  ); 

...

}
...
 
This new code, create a new view in Couchbase server:
  • line 7 : load the module and set the properties to call the view services
  • line 16-17 : call the basevie.setDesign() method, that create a view.
  • line 18 : set the map function that list all the employees

3. Let's now call the view in the onRequest function.

function onRequest(request, response) {
 response.writeHead(200, {
  "Content-Type": "text/plain"
 });
 response.write("See list of employees in the console");
 var params = 
  { 'descending'  : 'true'
  , 'include_docs' : 'true'
  };
 baseview.view('design_employees', 'basic_list', params, function(error, data) {
    for( var i = data.rows.length-1,length = data.rows.length ; i >= 0; i-- ) {
    var employee = data.rows[i].doc.json;
    console.log(employee);   
   } 
  });   
 response.end();
}


Calling the view is quite simple :

  • lines 6-8 : create an object to send view parameters. In this example I am just using descending, and include_docs to get the full document as part of the response. You can find the list of all the parameters you can use in the Couchbase documentation : Querying Using the REST API (The baseview module is using REST API to call the views).
  • line 10-14 : just loop on the result content, returned in the data variable, and print the employee information in the console.

Note: Because of the asynchronous nature of node.js, and my lack of experience with node, I was not able to send the list of employee to the HTTP response.

In another article I will explain how to integrate Couchbase with an node.js application based on Express and Socket.io, where I list the Employee in my the Web page.

Below, the complete code of this simple node.js application:
var http = require("http");
var driver = require('couchbase');
var cb = new driver.Couchbase("localhost:8091", null, null, "default");
var baseview = require('baseview')({url: 'http://127.0.0.1:8092',bucket: 'default'});

function onRequest(request, response) {
 response.writeHead(200, {
  "Content-Type": "text/plain"
 });
 response.write("See list of employees in the console");
 var params = {
  'descending': 'true',
  'include_docs': 'true'
 };
 baseview.view('design_employees', 'basic_list', params, function(error, data) {
  for (var i = data.rows.length - 1, length = data.rows.length; i >= 0; i--) {
   var employee = data.rows[i].doc.json;
   console.log(employee);
  }
 });
 response.end();
}

function insertData() {
 //create a new view
 baseview.setDesign('design_employees', {
  'basic_list': {
   'map': "function (doc, meta) { if(doc.type == 'employee') {emit(meta.id, doc.name);}}"
  }
 }, function(err, res) {
  if (err != null) console.log(err);
 });

 // insert employees in Couchbase
 var emps = [{
  "type": "employee",
  "id": 100,
  "name": "Thomas",
  "dept": "Sales",
  "salary": 5000
 }, {
  "type": "employee",
  "id": 200,
  "name": "John",
  "dept": "Development",
  "salary": 4500
 }, {
  "type": "employee",
  "id": 300,
  "name": "Jane",
  "dept": "Marketing",
  "salary": 5000
 }]


 // Insert the data in Couchbase using the add method ()
 for (index = 0; index < emps.length; index++) {
  cb.add(JSON.stringify(emps[index].id), JSON.stringify(emps[index]), 0, undefined, function(data, err, key, cas) {
   if (err && err != 12) { // 12 : LCB_KEY_EEXISTS  
    console.log("Failed to store object:\n" + err);
   }
  });
 }
}

var server = http.createServer(onRequest);

server.listen(8080, insertData());

console.log("> SERVER STARTED");

Friday, July 6, 2012

Couchbase 101 : install, store and query data

Introduction

In this post I just want to show how easily is to get started with Couchbase, and also explain how to “query” the data. The basic steps of this tutorial are:
  1. Install Couchbase 
  2. Create Data 
  3. Query Data 

I will try to post more articles, if I have time to show how to use Couchbase from your applications (starting with Java).

Prerequisites :
  •  Could not be simpler : Couchbase 2.0 available here. (Currently in Developer Preview)

Couchbase 101 : Insert and Query data

Installation

I am using Couchbase on Mac OS X, so let me describe the installation in this environment. If you are using other operating system just take a look to the Couchbase documentation.

Couchbase installation is very (very!) fast:
  1. Download the Mac OS X Zip file.
  2. Double-click the downloaded Zip installation file to extract the contents. This will create a single file, the Couchbase.app application.
  3. Drag and Drop the Couchbase.app to your chosen installation folder, such as the system Applications folder.

Start and Configure Couchbase Server


To start Couchbase Server, just double click on the Couchbase Server. Once the server is started, a new icon is added in the OS X Menu to indicate that the Server is up and running.

You can now configure your Couchbase instance, for this you just need to access the Admin Console, available at the following location http://127.0.0.1:8091 (change the IP address if needed) or simply by going in the Couchbase menu and click on Open Admin Console entry.



  1. Welcome Screen : Click Setup
  2. Set the disk and cluster configuration. On my instance I keep the default location for the on disk storage. Just configure the size of the memory usage for your instance, for example 800Mb. So far, we have a single instance, so no need to join a cluster.
  3. Choose to generate sample data. This will be interesting to learn more about data and views.
  4. Create the default bucket (use for testing only). A bucket is used by Couchbase to store data. It could be compared to a “database” in RDBMS world.
  5. Configure update notifications to be alerted when new version of Couchbase is released
  6. Configure the server with a final step with the administrator username and password
  7. When this is done you are automatically redirected to the Admin Console.
This is it! You are ready to use your Couchbase server.

Couchbase has many interesting features, especially around scalability and elasticity but for not in this article let's focus on the basics :
  • Insert some data and query them

Insert Data

Couchbase has many ways to manipulate data from you favorite programming language using the different client libraries : Java, Python, PHP, Ruby, .Net, C. For now let's use the Admin Console to create and query data.

Couchbase can store any type of data, but when you need to manipulate some data with a structure the best way is to use JSON Documents. So let's use the console and create documents.

To create new documents in your database, click on the "Data Buckets" tab. If you have installed the sample you see 2 buckets: default and gamesim-sample.

Let's create a new documents in the default bucket:
  1. Click on Documents button
  2. Click on Create Document
  3. Since each document must have an id for example 100.
  4. Couchbase save the document and add some metadata such as _rev, $flags, expiration
  5. Add new attributes to the document that describe an employee : Name, Departement and Salary, then save it. You just need to update the JSON object with values
    {
      "_id": "100",
      "name": "Thomas",
      "dept": "Sales",
      "salary": 5000
    }

Repeat the operation with some other employees :
200,Jason,Technology,5500
300,Mayla,Technology,7000
400,Nisha,Marketing,9500
500,Randy,Technology,6000
501,Ritu,Accounting,5400


You have now a list of employees in your database. That was easy isn't? Let's now query them.

Query Data

Access document directly from its ID

First of all you can quickly access a document using a simple HTTP request using its id. For example to access the Mayla with the id 300 just enter the following URL:
  • http://127.0.0.1:8092/default/300  
In this URL you have :
  • 8092 is the Couch API REST port used to access data (where 8091 is the port for the Admin console)
  • default is the bucket in which the document is stored
  • 300 is the id of the document

Search your data with queries

So we have seen how you can access one document. But what if my need is :
  •  "Give me all the employee of the Technology department"
To achieve such query it is necessary to create views. The views are used by Couchbase server to index and search your data. A view is a Map function written in JavaScript, that will generate a key value list that is compliant with logic you put in the Map function. Note that this key,value is now indexed and sorted by key. This is important when you query your data.
So let's create a new view from the Admin Console:
  1. Click on the Views tab (be sure you are on the default bucket)
  2. Click on the "Create Development View"
  3. Enter the Document and View name:
    • Document Name : _design/dev_dept
    • View Name : dept
  4. Cick Save
  5. Click on your View to edit it
Since we need to provide the list of employees that are part of a the Technology department, we need to create a view that use the department as key, so the map function looks like :

function (doc) {
  emit(doc.dept, null);
}

Save the view

This function takes the document and create a list that contains the "dept" as key and null as value. The value itself is not that important in our case. A simple rule will be : do not put too much data in the value since at the end Couchbase server creates an index with this map. Will see that Couchbase allows developer to easily get the document information when accessing a view.

Click on the "Show Results" button, the result will look like:
{"total_rows":6,"rows":[
  {"id":"501","key":"Accounting","value":null},
  {"id":"400","key":"Marketing","value":null},
  {"id":"100","key":"Sales","value":null},
  {"id":"200","key":"Technology","value":null},
  {"id":"300","key":"Technology","value":null},
  {"id":"500","key":"Technology","value":null}
]
}

As we have seen in earlier it is possible to access the document using a single URL, it is the same for views. You can for example access the view we have just created using the following URL:

Now it is possible to use query parameter to filter the results using the key parameter with the value entered using a JSON String :
The result of this query is now :

{"total_rows":6,"rows":[
  {"id":"200","key":"Technology","value":null},
  {"id":"300","key":"Technology","value":null},
  {"id":"500","key":"Technology","value":null}
]
}

You have many other parameters you can use when accessing a view to control the size, the time out, .... One of them is quite interesting is include_docs that ask Couchbase to include the full content of the document in the result. So if you call :

The result is :

{"total_rows":6,"rows":[
  {"id":"200","key":"Technology","value":null,"doc":  {"_id":"200","_rev":"1-1de6e6751608eada0000003200000000","$flags":0,"$expiration":0,"name":"Jason","dept":"Technology","salary":5500}},
  {"id":"300","key":"Technology","value":null,"doc":{"_id":"300","_rev":"1-f3e44cee742bfae10000003200000000","$flags":0,"$expiration":0,"name":"Mayla","dept":"Technology","salary":7000}},
  {"id":"500","key":"Technology","value":null,"doc":  {"_id":"500","_rev":"1-05780359aac8f3790000003200000000","$flags":0,"$expiration":0,"name":"Randy","dept":"Technology","salary":6000}}
]
}



Let's now create a little more complicated view to answer the following business requirement: "Give me all the employee with a salary between 5000 and 6000"
So now you know that you need to create a new view with the salary as key let's with the following Map function:

function (doc) {
  emit(doc.salary, null);
}

Couchbase is automatically sorting the key when creating/updating the index so, let's use the startkey and endkey parameter when calling the view. So let's call the view with from the following URL:
The result is :
{"total_rows":6,"rows":[
  {"id":"100","key":5000,"value":null,"doc":{"_id":"100","_rev":"1-0f33580d780014060000002e00000000","$flags":0,"$expiration":0,"name":"Thomas","dept":"Sales","salary":5000}},
  {"id":"501","key":5400,"value":null,"doc":{"_id":"501","_rev":"1-b1fe5bc79637720e0000003100000000","$flags":0,"$expiration":0,"name":"Ritu","dept":"Accounting","salary":5400}},
  {"id":"200","key":5500,"value":null,"doc":{"_id":"200","_rev":"1-1de6e6751608eada0000003200000000","$flags":0,"$expiration":0,"name":"Jason","dept":"Technology","salary":5500}},
  {"id":"500","key":6000,"value":null,"doc":{"_id":"500","_rev":"1-05780359aac8f3790000003200000000","$flags":0,"$expiration":0,"name":"Randy","dept":"Technology","salary":6000}}
]
}


Conclusion

In this short article you have learn how to:
  • Install Couchbase
  • Create data using the Admin Console
  • Query data with views
When I get more time I will write another article that do the same from Java, and other languages.

Friday, May 11, 2012

eXo Platform: Internationalization of content

In this screencast I explain the support of multi-lingual content of eXo Platform 3.5. The different features that you can see in this video are:

  • Support of multiple languages from URL 
  • Configure the eXo File Explorer to add support for multi-lingual content (new button)
  • Content translation
  • Add new language to the platform


Wednesday, April 11, 2012

Twitter Boostrap 2 and Google Maps

Like many developers I am using Twitter Boostrap for my Web applications. Using this framework has been very helpful for me, since I am really not a good HTML/CSS developer. For now, on my site Resultri I am using the default look and feel, will customize it later.

Lately, I wanted to integrate Google Map to my application, and when testing it, I had the bad surprise to see that the Controls and WindowInfo are not printed correctly as you can see in the screen shot below:



This is not a big issue at all, just a conflict on the img tag and its style (max-width) coming from Twitter Bootstrap. The quick fix :
  • override the style of the img tag for the div that contains your map.
For example in my case the div for my map is define as:



You just need to add a new style to your page with the following definition:




After adding this to my page the map is correctly printed as you can see in the following screenshot :




Sunday, February 19, 2012

SAP Cloud Inside : Build and Run Applications in the Cloud

Last week I was invited to present eXo Cloud IDE during the SAP Cloud Inside. This SAP Community event was a great opportunity to discuss about the cloud with an interesting point of view: the impact of the cloud for SAP customers (especially administrators and developers).

During this presentation I have introduced the eXo Cloud IDE, and I did a demonstration in which O have built and deployed applications : Open Social Gadgets, Ruby on Rails and Java/Spring, and explain how it could be extended to SAP business services.

Here the slides that I have used during this presentation:


Remember that you can register yourself to the eXo Cloud IDE Service and start develop application from your browser.

Saturday, January 21, 2012

Google AppEngine Full Text Search with Cloud SQL

Introduction

Many Google AppEngine developers have been waiting for the Full Text Search feature, especially coming from Google the biggest search engine on the Web. I was quite happy to see that Google team is working on it as you can check in the Google I/O 2011 session : Full Text Search by Bo Majewski, Ged Ellis . As far as I know the very promising indexing service is not yet available.

In this article I will explain how you can provide some kind of full text search in your application using services available App Engine services.

In my specific use case I do not ask for a lot of feature, I just need to have simple search a string in various attributes of my entities independently of the case, and possible special characters (such as è,é, ... ). I am far of being an expert of Google Datastore API but I did not find any simple way to achieve this directly using the Java API. What I have done to solve this issue is to duplicate a part of my data into the Google Cloud SQL to use the MySQL fulltext search capabilities.

Prerequisites
To achieve the following tasks you need to :


Content

In the following paragraphs I will explain the basics of the integration of Cloud SQL for full text search, but you can, if you want, jump to :




1. Create Articles Entities

Start by creating some simple entities with some attributes for example, an entity name Article, with title and body attributes.


import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;

//...
//...

  Entity article = new Entity("Article");
  article.setProperty("title", "MySQL Tutorial");
  article.setProperty("body", "DBMS stands for DataBase ...");
  datastore.put(article);

  article = new Entity("Article");
  article.setProperty("title", "Datastore Index Selection and Advanced Search");
  article.setProperty("body", "Learn how recent improvements to the query planner ... function in your application");
  datastore.put(article); 

If you look in the Datastore API, or even JDO or JPA you have no simple way to look for all the articles that are related to Triathlon, or Database, or Entities. Google DataStore does not support  clause where with a "OR" between different fields; and I do not want to mention the fact that it is not possible to ignore the text case in a simple way.

This is why we need to have some full text features. Some of you are surely thinking about using Apache Lucene to do the trick, and yes it is possible. You can use for example the GAELucene project : http://code.google.com/p/gaelucene/. I use another approach, may be less advanced in term of "indexing/searching" options but sufficient for my use case:
  • I store the text values on which I want to do some search in Google Cloud SQL and use the Full Text features of MySQL.


2. Create a SQL Table to store Text values (in development environment)

When using Google AppEngine, the Cloud SQL instances are accessed using a specific driver and configuration that we will see later. For now, we are still in development environment, this is where you have to use your local MySQL instance.

In this specific use case we will copy in a table the two fields and add a new unique key based on the entity key.  So here the SQL to create this:

CREATE SCHEMA search_values DEFAULT CHARACTER SET utf8 ;

USE search_values;


CREATE TABLE articles  (
  entity_key varchar(250),
  title text,
  body text,
  PRIMARY KEY RESULTS_PK (entity_key),
  FULLTEXT (title,body)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;


Lines 1 and 3 are here to create the database schema and use it; then the script create a table that will contain a copy of the title and body from the entity. 

3. Configure your development environment 

This section is a short explanation of the Cloud SQL Documentation : Getting Started: Java

  1. Copy the MySQL JDBC driver into your Google App Engine SDK directory, under /lib/impl/. You can download the MySQL JDBC driver here.
  2. In Eclipse, select your Java package.
  3. Click Run > Run Configurations.
  4. Expand the Web Application menu item.
  5. Add the following lines into the VM Arguments pane:
    -Drdbms.server=local
    -Drdbms.driver=com.mysql.jdbc.Driver
    -Drdbms.url=jdbc:mysql://localhost:3306/search_values?user=username&password=password
    
  6. Click the Classpath tab.
  7. Select your project and click Add External JARs...
  8. Navigate to the Google App Engine SDK directory, then lib/impl, and select the JDBC driver JAR file. Click Open. The driver JAR is listed under User Entries.
  9. Click Apply.
Your development environment is now ready to use your local MySQL database. Let's now, use this database.

4. Use your MySQL table and copy the text values from Google Datastore to MySQL Table

Copying the data from Datastore entity to the table is quite easy:

  Connection conn = null;
  try {
   DriverManager.registerDriver(new AppEngineDriver());
   conn = DriverManager.getConnection("jdbc:google:rdbms://[your db instance]/search_values");
   conn.setAutoCommit(false);  
   String statement = "REPLACE INTO articles (entity_key, title, body) VALUES( ? , ? , ? )";
   PreparedStatement stmt = conn.prepareStatement(statement);

   DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
   Query q = new Query("Article");   
   PreparedQuery pq = datastore.prepare(q);

   // loop on each entity and insert the values in the SQL Table
   for (Entity result : pq.asIterable()) {
    stmt.setString(1,  KeyFactory.keyToString(result.getKey())   );
    stmt.setString(2,  result.getProperty("title").toString() );
    stmt.setString(3,  result.getProperty("body").toString() );
    stmt.executeUpdate();
    conn.commit();
   }



  } catch (SQLException e) {
   e.printStackTrace();
  } finally {
   if (conn != null)
    try {
     conn.close();
    } catch (SQLException ignore) {}
  }



Some specials things here, compare to standard Java Web Development:
  • I manage the connection directly in my code (I have not looked yet if I can use datasources/connection pool in the context of Google AppEngine)
  • Line #3: registering the AppEngine driver that is responsible of managing the connection, expecially work in development -local MySQL- or production mode -CloudSQL-.
  • Line #4 : Get the connection. It is interesting to mention that in development the connection URL is grabbed from the environment variable Drdbms.url you have set previously. We will see later how we move this to the cloud. This is the magical part of the AppEngineDriver that manages different connection types Local MySQL or CloudSQL depending of the context
  • After these lines, the code is quite simple :
    • Get all the Articles entities from the datastore and loop
    • "Upsert" the database record (REPLACE INTO syntax)
  • Line #15 is storing the Key of the entity in a safe string using the KeyFactory.keyToString() method.

If you want to test this code just put this lines in a servlet to "sycnhronize" the data from datastore into the MySQL table. Obviously this code is just here for learning propose and should be integrated in a better way in a real application; starting with pushing the data in the database when entities are created/updated (and deleted ;) ). The sample code available from GitHub contains these methods.


5. Implement a search method

The goal is simple return a list of entities returned by a simple search criteria :
  • public Iterable searchEntities(String query)

The logic is here quite simple:
  1. Execute a SQL query
  2. For each result, get the Entity using the Key
  3. Return the list of Entities

 public Iterable searchEntity(String query) {
  List  results = new ArrayList();
  Connection conn = null;
  try {
   DriverManager.registerDriver(new AppEngineDriver());
   conn = DriverManager.getConnection("jdbc:google:rdbms://[your db instance]/search_values");
   String statement = "SELECT entity_key FROM articles WHERE MATCH (title,body) AGAINST (? WITH QUERY EXPANSION);";
   PreparedStatement stmt = conn.prepareStatement(statement);
   stmt.setString(1, query);
   ResultSet rs = stmt.executeQuery();
   while (rs.next()) {
    String keyAsString = rs.getString(1);    
    Entity article = DatastoreServiceFactory.getDatastoreService().get( KeyFactory.stringToKey(keyAsString)  );
    results.add(article);
   }

  } catch (SQLException e) {
   e.printStackTrace();
  } catch (EntityNotFoundException e) {
   e.printStackTrace();
  } finally {
   if (conn != null)
    try {
     conn.close();
    } catch (SQLException ignore) {}
  }
  return results;
 }


In this method, the system connect to the database and then execute a query to search data using any type of SQL/MySQL query. In this exampe I am using the full text function with the "WITH QUERY EXPANSION". You can obviously use any type of SQL queries for example simple LIKE statement if this is enough four your application.

With this approach when I search for :
  • "database" : the method returns all the articles concerning database, mysql, RDBMS independently of the case.
  • "index" " the method returns all the articles talking about indexing/indexes or search.

6.  Deploy to GAE 


Once you have created your application, and activated and configure your CloudSQL instance (here), you can deploy your application and enjoy an easy way of using Full Text Search with GAE.


Conclusion

In this article I explained how you can use Google Cloud SQL to easily supports Full Text Search queries, based on the Full Text support of MySQL.

The code snippets that I have shared in this article are really basic and not ready for real life usage but still a good starting point. For example I have been using this in my application with GAE Queues to manage my indexes on larger volume of data.

As said before, you can test the application online at http://gae-fulltext-search.appspot.com/ and the source code is available on GitHub : https://github.com/tgrall/gae-full-text-search