Sunday, December 30, 2012

Couchbase 101: Create views (MapReduce) from your Java application

When you are developing a new applications with Couchbase 2.0, you sometimes need to create view dynamically from your code. For example you may need this when you are installing your application, writing some test, or you can also use that when you are building frameworks, and wants to dynamically create views to query data. This post shows how to do it.

Prerequisites

If you are using Maven you can use the following information in your pom.xml to add the Java Client library:

  
    
      couchbase
      Couchbase Maven Repository
      default
      http://files.couchbase.com/maven2/
      
        false
      
    
  
  
  
    
      couchbase
      couchbase-client
      1.1.0
      jar
    
  
See online at https://gist.github.com/4337172

Create and Manage Views From Java 

The full Maven project is available on Github.

Connect to Couchbase Cluster

The first thing to do when you want to create a view from Java is obviously to connect to the cluster.

import com.couchbase.client.CouchbaseClient;
...
...

    List uris = new LinkedList();
    uris.add(URI.create("http://127.0.0.1:8091/pools"));
    CouchbaseClient client = null;
    try {
        client = new CouchbaseClient(uris, "beer-sample", "");

        // put your code here

        client.shutdown();      

    } catch (Exception e) {
        System.err.println("Error connecting to Couchbase: " + e.getMessage());
        System.exit(0);
    }

...
...

  1. Create a list of URIs to different nodes of the cluster - lines 5-6. (In this example I am working on a single node)
  2. Connect to the bucket, in our case beer-sample -line 9. You can include the password if the bucket is protected ( this is not the case here so I am sending an empty string)
If you are looking for more information about Couchbase and Java, you can read this article from DZone : Hello World with Couchbase and Java.

Let's now talk about Couchbase views. You use views/map-reduce functions to index and query data from Couchbase Server based on the content of the JSON document you store inside Couchbase. For more information about views you can look at the "view basics" chapter of the Couchbase Server Manual.

Create Views from Java

Creating a view from Java is really easy : the Java Client Library contains all the classes and methods to do it. As a concrete use case we will use the Application that is described in the Couchbase Java Tutorial.

When you follow this tutorial, you need to manually create some views, as you can see here. In this example, we will create our map function and directly in our Java code and then store it to Couchbase Server. The tutorial asks you to create the following artifacts:
  • a view named "by_name
  • in the design document named "dev_beer" (development mode)
  • and the map function which looks like the following :
 function (doc, meta) {
   if(doc.type && doc.type == "beer") {
     emit(doc.name, null);
   }
 }


The following code allows you to do it from Java:

import com.couchbase.client.protocol.views.DesignDocument;
import com.couchbase.client.protocol.views.ViewDesign;
...
    DesignDocument designDoc = new DesignDocument("dev_beer");

    String viewName = "by_name";
    String mapFunction =
            "function (doc, meta) {\n" +
            "  if(doc.type && doc.type == \"beer\") {\n" +
            "    emit(doc.name);\n" +
            "  }\n" +
            "}";

    ViewDesign viewDesign = new ViewDesign(viewName,mapFunction);
    designDoc.getViews().add(viewDesign);
    client.createDesignDoc( designDoc );
...


  • Create a design document using the com.couchbase.client.protocol.views.DesignDocument class - line 4.
  • Create a view using com.couchbase.client.protocol.views.ViewDesign class with a name and the map function - line 14.
  • You can add this view to a design document - line 15
  • Finally save the document into the cluster using the CouchbaseClient.createDesignDoc method.
If you need to use a reduce function (built-in or custom) you just need to pass to the ViewDesign constructor as 3rd parameter.

When developing view, from Java or from any other tool/language be sure you understand what are the best practices, and the life cycle of the index. This is why I am inviting you to take a look to the following chapters in the Couchbase documentation:

Using the view

First of all, the view that you just created is in "development mode", and by default the Java client SDK will only access the view when it is in "production mode". This means that when you are calling a view from your application it will search it into the production environment. So before connecting to Couchbase cluster you need to setup the viewmode to development.

This is done using the viewmode environment variable from the Java SDK, that could be set using the following methods:
  • In your code, add this line before the client connects to the cluster : System.setProperty("viewmode", "development");
  • At the command line -Dviewmode=development
  • In a properties file viewmode=development
Once it is done you can call the view using the following code:
import import com.couchbase.client.protocol.views.*;

...
   System.setProperty("viewmode", "development"); // before the connection to Couchbase
...
   View view = client.getView("beer", "by_name");
   Query query = new Query();
   query.setIncludeDocs(true).setLimit(20);
   query.setStale( Stale.FALSE );
   ViewResponse result = client.query(view, query);
   for(ViewRow row : result) {
     row.getDocument(); // deal with the document/data
   }
...

This code queries the view you just created. This means Couchbase Server will generate an index based on your map function, will query the server for results. In this case, we specifically want to set a limit of 20 results and also get the most current results by setting Stale.FALSE.
  • Set the viewmode to development - line 4
  • Get the view using the CouchbaseClient.getView() method -line 6-. As you can see I just use the name beer for the design document (and not dev_beer, Couchbase will know where to search since I am in development mode)
  • Create a query and set a limit (20) and ask the SDK to return the document itself
    setIncludeDocs(true) -line 8- The document will be returned from Couchbase server in the most efficient way
  • Ask the system to update the index before returning the result using query.setStale( Stale.FALSE ); -line 9-. Once again be careful when you use setStale method. Just to be sure here is the documentation about it : Index Updates and the stale Parameter
  • Execute the query - line 10
  • And use the result - lines 11-13

Conclusion

In this article, you have learned:
  • How to create Couchbase views from Java
  • Call this view from Java
  • Configure development/production mode views from Couchbase Java Client Library
This example is limited to the creation of a view, you can take a look to the other methods related to design documents and views if you want to manage your design documents : getDesignDocument(), deleteDesignDocument(), ... .


12 comments:

man4j said...

Great post! But will be nice if couchbase client has a more type-safe and generificated API as all java-world apis. F.e. as http://code.google.com/p/async-couchdb-client/source/browse/trunk/acc/src/test/java/com/n1global/acc/examples/booksdb/. And how can I create map/reduce function in pure Java? F.e. http://code.google.com/p/async-couchdb-client/source/browse/trunk/acc/src/test/java/com/n1global/acc/examples/javamr ?

Tug Grall said...

Hello,

Today Couchbase does not have a supported API with Java "strong types/mapping". We let the developer chose how he wants to acheeve this
You can use any JSON/Java Mapping tool to achieve this, for example GSON, Jackson, ...

I have started to work on the Hibernate OGM Data Provider for Couchbase:
https://github.com/tgrall/hibernate-ogm/tree/couchbase-support

I am currently stucked on the association part and need more time to finish this.

Also I am working on a more simple (without class mapping) API to manipulate Document easily:
https://github.com/tgrall/couchbase-document-api


Finally to answer your question about MapReduce in Java, same we do not have that yet.

Obviously the framework that you are exposing could be easily moved to Couchbase API.

Unknown said...
This comment has been removed by a blog administrator.
Brad Wood said...

Thank you for the article. How do you add a new view to an existing DesignDoc? There is a createDesignDoc() method in the CouchBase Client, but not an updateDesignDoc() method.

Tug Grall said...

Brad,

To add a new view, you just replace the design document. So you have to do:

1. Get the DesignDocument using
dd = client.getDesignDocument("name");

3. Add a view to this document using
dd.getViews().add(viewDesign);

4. Then push the design document
client.createDesignDoc(dd);


Tug

Unknown said...

Tug,

I am doing the same thing on my machine, but the design document is not being created. In fact, client.createDesignDoc() is returning false (well, since it is HttpFuture, that value doesn't mean too much).

Is there any special settings to go along with design document creation?

The following is my connection message,

2013-09-12 11:14:45.985 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa={OUR_SERVER}, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-09-12 11:14:45.991 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@31eaca2
2013-09-12 11:14:46.022 INFO com.couchbase.client.ViewConnection: Added node19.hadoop.nmg to connect queue
2013-09-12 11:14:46.023 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2013-09-12 11:14:46.073 INFO net.spy.memcached.auth.AuthThread: Authenticated to {OUR_SERVER}
2013-09-12 11:14:47.565 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2013-09-12 11:14:47.570 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: {OUR_SERVER} - Session request successful
2013-09-12 11:14:47.620 INFO com.couchbase.client.CouchbaseClient: Creating Design Document:dev_views

Tug Grall said...

Hello Seungtack,

Nothing special here, what you can do is check the status of the OperationFuture object.

Tug

InfraMouse said...

I get this error:
java.lang.NoSuchMethodError: com.couchbase.client.CouchbaseClient.getView(Ljava/lang/String;Ljava/lang/String;)Lcom/couchbase/client/protocol/views/View;


At this line:
View view = client.getView(DESIGN_DOC, VIEW_NAME);

How do I fix this? :/

InfraMouse said...

I get this error:
java.lang.NoSuchMethodError: com.couchbase.client.CouchbaseClient.getView(Ljava/lang/String;Ljava/lang/String;)Lcom/couchbase/client/protocol/views/View;


On this line:
View view = client.getView(DESIGN_DOC, VIEW_NAME);


How do I fix this? :/

endymyon said...

Hello
Your post helped me a lot for integration.
Where can i find methods that can be called on docs or variables of docs?

Unknown said...
This comment has been removed by a blog administrator.
CricketUpdate said...

Super tutorial
How to do wild search character. Suppose if I want to search row contaning substring then how to do it.