Friday, May 11, 2012

eXo Platform: Internationalization of content

In this screencast I explain the support of multi-lingual content of eXo Platform 3.5. The different features that you can see in this video are:

  • Support of multiple languages from URL 
  • Configure the eXo File Explorer to add support for multi-lingual content (new button)
  • Content translation
  • Add new language to the platform


Wednesday, April 11, 2012

Twitter Boostrap 2 and Google Maps

Like many developers I am using Twitter Boostrap for my Web applications. Using this framework has been very helpful for me, since I am really not a good HTML/CSS developer. For now, on my site Resultri I am using the default look and feel, will customize it later.

Lately, I wanted to integrate Google Map to my application, and when testing it, I had the bad surprise to see that the Controls and WindowInfo are not printed correctly as you can see in the screen shot below:



This is not a big issue at all, just a conflict on the img tag and its style (max-width) coming from Twitter Bootstrap. The quick fix :
  • override the style of the img tag for the div that contains your map.
For example in my case the div for my map is define as:



You just need to add a new style to your page with the following definition:




After adding this to my page the map is correctly printed as you can see in the following screenshot :




Sunday, February 19, 2012

SAP Cloud Inside : Build and Run Applications in the Cloud

Last week I was invited to present eXo Cloud IDE during the SAP Cloud Inside. This SAP Community event was a great opportunity to discuss about the cloud with an interesting point of view: the impact of the cloud for SAP customers (especially administrators and developers).

During this presentation I have introduced the eXo Cloud IDE, and I did a demonstration in which O have built and deployed applications : Open Social Gadgets, Ruby on Rails and Java/Spring, and explain how it could be extended to SAP business services.

Here the slides that I have used during this presentation:


Remember that you can register yourself to the eXo Cloud IDE Service and start develop application from your browser.

Saturday, January 21, 2012

Google AppEngine Full Text Search with Cloud SQL

Introduction

Many Google AppEngine developers have been waiting for the Full Text Search feature, especially coming from Google the biggest search engine on the Web. I was quite happy to see that Google team is working on it as you can check in the Google I/O 2011 session : Full Text Search by Bo Majewski, Ged Ellis . As far as I know the very promising indexing service is not yet available.

In this article I will explain how you can provide some kind of full text search in your application using services available App Engine services.

In my specific use case I do not ask for a lot of feature, I just need to have simple search a string in various attributes of my entities independently of the case, and possible special characters (such as è,é, ... ). I am far of being an expert of Google Datastore API but I did not find any simple way to achieve this directly using the Java API. What I have done to solve this issue is to duplicate a part of my data into the Google Cloud SQL to use the MySQL fulltext search capabilities.

Prerequisites
To achieve the following tasks you need to :


Content

In the following paragraphs I will explain the basics of the integration of Cloud SQL for full text search, but you can, if you want, jump to :




1. Create Articles Entities

Start by creating some simple entities with some attributes for example, an entity name Article, with title and body attributes.


import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;

//...
//...

  Entity article = new Entity("Article");
  article.setProperty("title", "MySQL Tutorial");
  article.setProperty("body", "DBMS stands for DataBase ...");
  datastore.put(article);

  article = new Entity("Article");
  article.setProperty("title", "Datastore Index Selection and Advanced Search");
  article.setProperty("body", "Learn how recent improvements to the query planner ... function in your application");
  datastore.put(article); 

If you look in the Datastore API, or even JDO or JPA you have no simple way to look for all the articles that are related to Triathlon, or Database, or Entities. Google DataStore does not support  clause where with a "OR" between different fields; and I do not want to mention the fact that it is not possible to ignore the text case in a simple way.

This is why we need to have some full text features. Some of you are surely thinking about using Apache Lucene to do the trick, and yes it is possible. You can use for example the GAELucene project : http://code.google.com/p/gaelucene/. I use another approach, may be less advanced in term of "indexing/searching" options but sufficient for my use case:
  • I store the text values on which I want to do some search in Google Cloud SQL and use the Full Text features of MySQL.


2. Create a SQL Table to store Text values (in development environment)

When using Google AppEngine, the Cloud SQL instances are accessed using a specific driver and configuration that we will see later. For now, we are still in development environment, this is where you have to use your local MySQL instance.

In this specific use case we will copy in a table the two fields and add a new unique key based on the entity key.  So here the SQL to create this:

CREATE SCHEMA search_values DEFAULT CHARACTER SET utf8 ;

USE search_values;


CREATE TABLE articles  (
  entity_key varchar(250),
  title text,
  body text,
  PRIMARY KEY RESULTS_PK (entity_key),
  FULLTEXT (title,body)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;


Lines 1 and 3 are here to create the database schema and use it; then the script create a table that will contain a copy of the title and body from the entity. 

3. Configure your development environment 

This section is a short explanation of the Cloud SQL Documentation : Getting Started: Java

  1. Copy the MySQL JDBC driver into your Google App Engine SDK directory, under /lib/impl/. You can download the MySQL JDBC driver here.
  2. In Eclipse, select your Java package.
  3. Click Run > Run Configurations.
  4. Expand the Web Application menu item.
  5. Add the following lines into the VM Arguments pane:
    -Drdbms.server=local
    -Drdbms.driver=com.mysql.jdbc.Driver
    -Drdbms.url=jdbc:mysql://localhost:3306/search_values?user=username&password=password
    
  6. Click the Classpath tab.
  7. Select your project and click Add External JARs...
  8. Navigate to the Google App Engine SDK directory, then lib/impl, and select the JDBC driver JAR file. Click Open. The driver JAR is listed under User Entries.
  9. Click Apply.
Your development environment is now ready to use your local MySQL database. Let's now, use this database.

4. Use your MySQL table and copy the text values from Google Datastore to MySQL Table

Copying the data from Datastore entity to the table is quite easy:

  Connection conn = null;
  try {
   DriverManager.registerDriver(new AppEngineDriver());
   conn = DriverManager.getConnection("jdbc:google:rdbms://[your db instance]/search_values");
   conn.setAutoCommit(false);  
   String statement = "REPLACE INTO articles (entity_key, title, body) VALUES( ? , ? , ? )";
   PreparedStatement stmt = conn.prepareStatement(statement);

   DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
   Query q = new Query("Article");   
   PreparedQuery pq = datastore.prepare(q);

   // loop on each entity and insert the values in the SQL Table
   for (Entity result : pq.asIterable()) {
    stmt.setString(1,  KeyFactory.keyToString(result.getKey())   );
    stmt.setString(2,  result.getProperty("title").toString() );
    stmt.setString(3,  result.getProperty("body").toString() );
    stmt.executeUpdate();
    conn.commit();
   }



  } catch (SQLException e) {
   e.printStackTrace();
  } finally {
   if (conn != null)
    try {
     conn.close();
    } catch (SQLException ignore) {}
  }



Some specials things here, compare to standard Java Web Development:
  • I manage the connection directly in my code (I have not looked yet if I can use datasources/connection pool in the context of Google AppEngine)
  • Line #3: registering the AppEngine driver that is responsible of managing the connection, expecially work in development -local MySQL- or production mode -CloudSQL-.
  • Line #4 : Get the connection. It is interesting to mention that in development the connection URL is grabbed from the environment variable Drdbms.url you have set previously. We will see later how we move this to the cloud. This is the magical part of the AppEngineDriver that manages different connection types Local MySQL or CloudSQL depending of the context
  • After these lines, the code is quite simple :
    • Get all the Articles entities from the datastore and loop
    • "Upsert" the database record (REPLACE INTO syntax)
  • Line #15 is storing the Key of the entity in a safe string using the KeyFactory.keyToString() method.

If you want to test this code just put this lines in a servlet to "sycnhronize" the data from datastore into the MySQL table. Obviously this code is just here for learning propose and should be integrated in a better way in a real application; starting with pushing the data in the database when entities are created/updated (and deleted ;) ). The sample code available from GitHub contains these methods.


5. Implement a search method

The goal is simple return a list of entities returned by a simple search criteria :
  • public Iterable searchEntities(String query)

The logic is here quite simple:
  1. Execute a SQL query
  2. For each result, get the Entity using the Key
  3. Return the list of Entities

 public Iterable searchEntity(String query) {
  List  results = new ArrayList();
  Connection conn = null;
  try {
   DriverManager.registerDriver(new AppEngineDriver());
   conn = DriverManager.getConnection("jdbc:google:rdbms://[your db instance]/search_values");
   String statement = "SELECT entity_key FROM articles WHERE MATCH (title,body) AGAINST (? WITH QUERY EXPANSION);";
   PreparedStatement stmt = conn.prepareStatement(statement);
   stmt.setString(1, query);
   ResultSet rs = stmt.executeQuery();
   while (rs.next()) {
    String keyAsString = rs.getString(1);    
    Entity article = DatastoreServiceFactory.getDatastoreService().get( KeyFactory.stringToKey(keyAsString)  );
    results.add(article);
   }

  } catch (SQLException e) {
   e.printStackTrace();
  } catch (EntityNotFoundException e) {
   e.printStackTrace();
  } finally {
   if (conn != null)
    try {
     conn.close();
    } catch (SQLException ignore) {}
  }
  return results;
 }


In this method, the system connect to the database and then execute a query to search data using any type of SQL/MySQL query. In this exampe I am using the full text function with the "WITH QUERY EXPANSION". You can obviously use any type of SQL queries for example simple LIKE statement if this is enough four your application.

With this approach when I search for :
  • "database" : the method returns all the articles concerning database, mysql, RDBMS independently of the case.
  • "index" " the method returns all the articles talking about indexing/indexes or search.

6.  Deploy to GAE 


Once you have created your application, and activated and configure your CloudSQL instance (here), you can deploy your application and enjoy an easy way of using Full Text Search with GAE.


Conclusion

In this article I explained how you can use Google Cloud SQL to easily supports Full Text Search queries, based on the Full Text support of MySQL.

The code snippets that I have shared in this article are really basic and not ready for real life usage but still a good starting point. For example I have been using this in my application with GAE Queues to manage my indexes on larger volume of data.

As said before, you can test the application online at http://gae-fulltext-search.appspot.com/ and the source code is available on GitHub : https://github.com/tgrall/gae-full-text-search

Thursday, January 19, 2012

eXo Platform : Integrate Twitter and eXo Activity Stream

eXo Platform 3.5 provides many extension points and API for developers, allowing them to create very cool stuff.

I have developed a small extension that allows any user to associate his Twitter account to his eXo Platform account. This extension simply post on your Twitter account when you write a message with a special hashtag (#tw).

This is a very quick development that I have done while waiting for my kids, so I still have things to integrate to provide complete feature, but this is a good example to show how you can extend the platform.

You can view it in action in this video:




You can download the source code and the binaries from this GitHub project. I hope to find some time to complete the feature, and document this use case.

Enjoy!


Monday, December 5, 2011

How to watch YouTube videos offline (on OS X)?

Lately I have been traveling a lot, and I was not able to access the internet all the time; but I still want to look at some YouTube video, for example the greate Google I/O Sessions...

My needs are simple :

  • running on OS X (Lion)
  • download the video easily
  • no need to convert the file (to be able to read the file as soon as possible)
  • free
After some basic research, I found an easy way to achieve this using the following softwares:
  Enjoy!

Sunday, November 20, 2011

Installing Memcached on Mac OS X and using it in Java


Introduction

In this article I will explain how you can:

  1. Install and Configure Memcached on Mac OS X
  2. Use Memcached in your Java Application

I won't go in too much detail about the benefits of using a distributed cache in your applications, but let's at least provide some use cases for applications that are running in the context of an enterprise portal, eXo Platform in my case - surprising isn't? And I will show this in another post.

We have many reasons to use a cache (distributed or not), in the context of enterprise portal, let's take a look to some of these reasons:

  • A portal is used to aggregate data in a single page. These data could come from different sources : Web Services, Database, ERP, ..... and accessing the data in real time could be costly. So it will be quite interesting to cache the result of the call when possible.
  • If the portal is used to aggregate many data from many sources, it is sometime necessary to jump into another application to continue some operation. A distributed and shared cache could be used to manage some context between different applications running in different processes (JVM or even technologies)
These are two example where a shared cache could be interesting for your portal based applications, we can find many other reason.


Note that the Portlet API (JSR-286) contains already a cache mechanism that cache the HTML fragment, and that eXo Platform also provide a low level cache, based on JBoss Cache.


Installation and Configuration

Installing Memcached from sources

You can find some information about Memcached installation on the Memcached Wiki. The following steps are the steps that I have used on my environment.

As far as I know, Memached is not available as package for Mac OS X. I am still on Snow Leopard (10.6.8), and I have installed XCode and all development tools. I have use the article "Installing memcached 1.4.1 on Mac OS X 10.6 Snow Leopard" from wincent.com. For simplicity reason I have duplicate the content and updated to the latest releases.

1. Create a working directory :

$ mkdir memcachedbuild
$ cd memcachebuild

 2. Install libevent that is mandatory for memcached

$ curl -O http://www.monkey.org/~provos/libevent-1.4.14-stable.tar.gz
$ tar xzvf libevent-1.4.14-stable.tar.gz
$ cd libevent-1.4.14-stable
$ ./configure
$ make
$ make verify
$ sudo make install  

3. Install memcached

Go back to your install directory (memcachedbuild)

$ curl -O http://memcached.googlecode.com/files/memcached-1.4.10.tar.gz
$ tar xzvf memcached-1.4.10.tar.gz
$ cd memcached-1.4.10
$ ./configure
$ make
$ make test
$ sudo make install 
You are now ready to use memcached that is available at /usr/local/bin/memcached

This allows you to avoid changing to the pre-installed memcached located in /usr/bin, if you want to replace it instead of having you own install, just run the configure command with the following parameter:  ./configure --prefix=/usr

Starting and testing Memcached

Start the memcached server, using the following command line:

$ /usr/local/bin/memcached -d -p 11211

This command starts the memcached server as demon (-d parameter), on the TCP port 11211 (this is the default value). You can find more about the memcached command using man memcached.

It is possible to connect and test your server using a telnet connection. Once connected you can set and get object in the cache, take a look to the following paragraph.

$ telnet 127.0.0.1 11211
Trying 127.0.0.1...
Connected to tgrall-server.
Escape character is '^]'.
set KEY 0 600 16
This is my value
STORED
get KEY
VALUE KEY 0 16
This is my value
END

 
The set command allows you to put a new value in the cache using the following syntax:

set <key>  <flags> <expiration_time>  <number_of_bytes> [noreply] \n\n

<value>
  • key : the key used to store the data in the cache
  • flags : a 32 bits unsigned integer that memcached stored with the data
  • expiration_time : expiration time in seconds, if you put 0 this means no delay
  • number_if_bytes : number of bytes in the data block
  • noreply : option to tell the server to not return any value
  • value : the value to store and associate to the key.
    This is a short view of the documentation located in your source directory /memcachedbuild/memcached-1.4.10/doc/protocol.txt .


    The get command allows you to access the value that is associated with the key.

    You can check the version of memcahed you are running by calling the stats command in your telnet session.


    Your memcached server is up and running, you can now start to use it inside your applications.


    Simple Java Application with Memcached

    The easiest way to use memcached from your Java applications is to use a client library. You can find many client libraries. In this example I am using spymemcached developped by the people from Couchbase.

    1. Adding SpyMemcached to your Maven project

    Add the repository to you pom.xml (or you setting.xml)

    <repository>
        <id>spy</id>
        <name>Spy Repository</name>
        <layout>default</layout>
        <url>http://files.couchbase.com/maven2/</url>
    </repository> 

    then the dependency to your pom.xml

    <dependency>
        <groupid>spy</groupid>
        <artifactid>spymemcached</artifactid>
        <version>2.7.3</version>
    </dependency>
    
    
    

    2. Use SpyMemcache client in your application

    The following code is a simple Java class that allows you to enter the key and the value and set it in the cache.


    package com.grallandco.blog;
    
    import java.io.BufferedReader;
    import java.io.IOException;
    import java.io.Console;
    import java.io.InputStreamReader;
    import java.util.Date;
    import java.util.logging.Level;
    import java.util.logging.Logger;
    import net.spy.memcached.AddrUtil;
    import net.spy.memcached.MemcachedClient;
    
    public class Test {
    
        public static void main(String[] args) {
           try {
               
               System.out.print("Enter the new key : ");
               BufferedReader reader = new BufferedReader( new InputStreamReader(System.in));
               String key = null;
               key = reader.readLine();
               
               System.out.print("Enter the new value : ");
               String value = null;
               value = reader.readLine();
               
                MemcachedClient cache = new MemcachedClient(AddrUtil.getAddresses("127.0.0.1:11211"));
                
                // read the object from memory
                System.out.println("Get Object before set :"+ cache.get(key)  );
    
                // set a new object            
                cache.set(key, 0, value );
    
                System.out.println("Get Object after set :"+ cache.get(key)  );
                
    
            } catch (IOException ex) {
                Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
                System.exit(0);
            }
    
           
            System.exit(0);
           
        }
    }
    
    
    

    So when executing the application you will see something like :

    Enter the new key : CITY
    Enter the new value : Paris, France
    2011-11-16 15:22:09.928 INFO net.spy.memcached.MemcachedConnection:  Added {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
    2011-11-16 15:22:09.932 INFO net.spy.memcached.MemcachedConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@5b40c281
    Get Object before set :null
    Get Object after set :Paris, France
    

    You can also access the object from a Telnet session:
    get CITY
    VALUE CITY 0 13
    Paris, France
    END
    


    You can use any Java class in your application, the only thing to do is to make this class serializable.

    This is it for the first post about memcached and Java,  I am currently working on a small example integrating Web Services call, Portlets and memcached.