Showing posts with label google. Show all posts
Showing posts with label google. Show all posts

Wednesday, April 11, 2012

Twitter Boostrap 2 and Google Maps

Like many developers I am using Twitter Boostrap for my Web applications. Using this framework has been very helpful for me, since I am really not a good HTML/CSS developer. For now, on my site Resultri I am using the default look and feel, will customize it later.

Lately, I wanted to integrate Google Map to my application, and when testing it, I had the bad surprise to see that the Controls and WindowInfo are not printed correctly as you can see in the screen shot below:



This is not a big issue at all, just a conflict on the img tag and its style (max-width) coming from Twitter Bootstrap. The quick fix :
  • override the style of the img tag for the div that contains your map.
For example in my case the div for my map is define as:



You just need to add a new style to your page with the following definition:




After adding this to my page the map is correctly printed as you can see in the following screenshot :




Saturday, January 21, 2012

Google AppEngine Full Text Search with Cloud SQL

Introduction

Many Google AppEngine developers have been waiting for the Full Text Search feature, especially coming from Google the biggest search engine on the Web. I was quite happy to see that Google team is working on it as you can check in the Google I/O 2011 session : Full Text Search by Bo Majewski, Ged Ellis . As far as I know the very promising indexing service is not yet available.

In this article I will explain how you can provide some kind of full text search in your application using services available App Engine services.

In my specific use case I do not ask for a lot of feature, I just need to have simple search a string in various attributes of my entities independently of the case, and possible special characters (such as è,é, ... ). I am far of being an expert of Google Datastore API but I did not find any simple way to achieve this directly using the Java API. What I have done to solve this issue is to duplicate a part of my data into the Google Cloud SQL to use the MySQL fulltext search capabilities.

Prerequisites
To achieve the following tasks you need to :


Content

In the following paragraphs I will explain the basics of the integration of Cloud SQL for full text search, but you can, if you want, jump to :




1. Create Articles Entities

Start by creating some simple entities with some attributes for example, an entity name Article, with title and body attributes.


import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;

//...
//...

  Entity article = new Entity("Article");
  article.setProperty("title", "MySQL Tutorial");
  article.setProperty("body", "DBMS stands for DataBase ...");
  datastore.put(article);

  article = new Entity("Article");
  article.setProperty("title", "Datastore Index Selection and Advanced Search");
  article.setProperty("body", "Learn how recent improvements to the query planner ... function in your application");
  datastore.put(article); 

If you look in the Datastore API, or even JDO or JPA you have no simple way to look for all the articles that are related to Triathlon, or Database, or Entities. Google DataStore does not support  clause where with a "OR" between different fields; and I do not want to mention the fact that it is not possible to ignore the text case in a simple way.

This is why we need to have some full text features. Some of you are surely thinking about using Apache Lucene to do the trick, and yes it is possible. You can use for example the GAELucene project : http://code.google.com/p/gaelucene/. I use another approach, may be less advanced in term of "indexing/searching" options but sufficient for my use case:
  • I store the text values on which I want to do some search in Google Cloud SQL and use the Full Text features of MySQL.


2. Create a SQL Table to store Text values (in development environment)

When using Google AppEngine, the Cloud SQL instances are accessed using a specific driver and configuration that we will see later. For now, we are still in development environment, this is where you have to use your local MySQL instance.

In this specific use case we will copy in a table the two fields and add a new unique key based on the entity key.  So here the SQL to create this:

CREATE SCHEMA search_values DEFAULT CHARACTER SET utf8 ;

USE search_values;


CREATE TABLE articles  (
  entity_key varchar(250),
  title text,
  body text,
  PRIMARY KEY RESULTS_PK (entity_key),
  FULLTEXT (title,body)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;


Lines 1 and 3 are here to create the database schema and use it; then the script create a table that will contain a copy of the title and body from the entity. 

3. Configure your development environment 

This section is a short explanation of the Cloud SQL Documentation : Getting Started: Java

  1. Copy the MySQL JDBC driver into your Google App Engine SDK directory, under /lib/impl/. You can download the MySQL JDBC driver here.
  2. In Eclipse, select your Java package.
  3. Click Run > Run Configurations.
  4. Expand the Web Application menu item.
  5. Add the following lines into the VM Arguments pane:
    -Drdbms.server=local
    -Drdbms.driver=com.mysql.jdbc.Driver
    -Drdbms.url=jdbc:mysql://localhost:3306/search_values?user=username&password=password
    
  6. Click the Classpath tab.
  7. Select your project and click Add External JARs...
  8. Navigate to the Google App Engine SDK directory, then lib/impl, and select the JDBC driver JAR file. Click Open. The driver JAR is listed under User Entries.
  9. Click Apply.
Your development environment is now ready to use your local MySQL database. Let's now, use this database.

4. Use your MySQL table and copy the text values from Google Datastore to MySQL Table

Copying the data from Datastore entity to the table is quite easy:

  Connection conn = null;
  try {
   DriverManager.registerDriver(new AppEngineDriver());
   conn = DriverManager.getConnection("jdbc:google:rdbms://[your db instance]/search_values");
   conn.setAutoCommit(false);  
   String statement = "REPLACE INTO articles (entity_key, title, body) VALUES( ? , ? , ? )";
   PreparedStatement stmt = conn.prepareStatement(statement);

   DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
   Query q = new Query("Article");   
   PreparedQuery pq = datastore.prepare(q);

   // loop on each entity and insert the values in the SQL Table
   for (Entity result : pq.asIterable()) {
    stmt.setString(1,  KeyFactory.keyToString(result.getKey())   );
    stmt.setString(2,  result.getProperty("title").toString() );
    stmt.setString(3,  result.getProperty("body").toString() );
    stmt.executeUpdate();
    conn.commit();
   }



  } catch (SQLException e) {
   e.printStackTrace();
  } finally {
   if (conn != null)
    try {
     conn.close();
    } catch (SQLException ignore) {}
  }



Some specials things here, compare to standard Java Web Development:
  • I manage the connection directly in my code (I have not looked yet if I can use datasources/connection pool in the context of Google AppEngine)
  • Line #3: registering the AppEngine driver that is responsible of managing the connection, expecially work in development -local MySQL- or production mode -CloudSQL-.
  • Line #4 : Get the connection. It is interesting to mention that in development the connection URL is grabbed from the environment variable Drdbms.url you have set previously. We will see later how we move this to the cloud. This is the magical part of the AppEngineDriver that manages different connection types Local MySQL or CloudSQL depending of the context
  • After these lines, the code is quite simple :
    • Get all the Articles entities from the datastore and loop
    • "Upsert" the database record (REPLACE INTO syntax)
  • Line #15 is storing the Key of the entity in a safe string using the KeyFactory.keyToString() method.

If you want to test this code just put this lines in a servlet to "sycnhronize" the data from datastore into the MySQL table. Obviously this code is just here for learning propose and should be integrated in a better way in a real application; starting with pushing the data in the database when entities are created/updated (and deleted ;) ). The sample code available from GitHub contains these methods.


5. Implement a search method

The goal is simple return a list of entities returned by a simple search criteria :
  • public Iterable searchEntities(String query)

The logic is here quite simple:
  1. Execute a SQL query
  2. For each result, get the Entity using the Key
  3. Return the list of Entities

 public Iterable searchEntity(String query) {
  List  results = new ArrayList();
  Connection conn = null;
  try {
   DriverManager.registerDriver(new AppEngineDriver());
   conn = DriverManager.getConnection("jdbc:google:rdbms://[your db instance]/search_values");
   String statement = "SELECT entity_key FROM articles WHERE MATCH (title,body) AGAINST (? WITH QUERY EXPANSION);";
   PreparedStatement stmt = conn.prepareStatement(statement);
   stmt.setString(1, query);
   ResultSet rs = stmt.executeQuery();
   while (rs.next()) {
    String keyAsString = rs.getString(1);    
    Entity article = DatastoreServiceFactory.getDatastoreService().get( KeyFactory.stringToKey(keyAsString)  );
    results.add(article);
   }

  } catch (SQLException e) {
   e.printStackTrace();
  } catch (EntityNotFoundException e) {
   e.printStackTrace();
  } finally {
   if (conn != null)
    try {
     conn.close();
    } catch (SQLException ignore) {}
  }
  return results;
 }


In this method, the system connect to the database and then execute a query to search data using any type of SQL/MySQL query. In this exampe I am using the full text function with the "WITH QUERY EXPANSION". You can obviously use any type of SQL queries for example simple LIKE statement if this is enough four your application.

With this approach when I search for :
  • "database" : the method returns all the articles concerning database, mysql, RDBMS independently of the case.
  • "index" " the method returns all the articles talking about indexing/indexes or search.

6.  Deploy to GAE 


Once you have created your application, and activated and configure your CloudSQL instance (here), you can deploy your application and enjoy an easy way of using Full Text Search with GAE.


Conclusion

In this article I explained how you can use Google Cloud SQL to easily supports Full Text Search queries, based on the Full Text support of MySQL.

The code snippets that I have shared in this article are really basic and not ready for real life usage but still a good starting point. For example I have been using this in my application with GAE Queues to manage my indexes on larger volume of data.

As said before, you can test the application online at http://gae-fulltext-search.appspot.com/ and the source code is available on GitHub : https://github.com/tgrall/gae-full-text-search

Sunday, February 28, 2010

Create and Deploy a JAX-RS REST service on Google App Engine

In this article you will learn how to create a REST service using JAX-RS reference implementation (Jersey) and deploy it on Google AppEngine.

Prerequisites

For this tutorial you will need:
  • a Google AppEngine account : http://code.google.com/appengine/
  • Eclipse Galileo (3.5.x)
  • Google App Engine SDK for Java

    • Install the Google Plugin for Eclipse as documented here (Check that you are using the release 1.3.1 of the GAE Java SDK, if not download it and configure the plugin to use it)
    • it is also useful to have the AppEngine documentation locally, you can download it from here.
  • JAX-RS Reference Implementation, be sure you take the Jersey 1.1.5 release. You can download it from here.

    • Unzip the file in a directory that we will call $JERSEY_HOME
  • JAXB 2.2 Implementation to simplify the marshalling/unmarshalling of the XML, and also facilitate the JSON support. Download it from here 


    • Install it using thejava -jar JAXB2_20091104.jar command.  The installation directory of JAXB will be called $JAXB_HOME

Creating new application

To create a new App Engine project in Eclipse:
  1. Click on the "New Web Application Project" button in the toolbar . It is also possible to do it using the menu File > Web Application Project

  2. The "Create a Web Application Project" wizard opens:
  • Project Name: EmployeeService
  • Package : com.grallandco.employee.service
  • Uncheck "Use Google Web Toolkit"
  • Check that the SDK version your are using is "App Engine 1.3.0"; if not configure the project to use it.
  • The screen should look like the following screen :


  • Click Finish

  • The project should look like the following screen :



Running the application

The App Egine SDK, installed with the Eclipse plugin contains a Web server (based on Jetty), that could be used for testing and debugging. To test that your application has been created correctly select the menu Run > Run As > Web Application. I personnaly most of the time run my server using the debug command Run > DebugAs > Web Application. In debug mode you can change source code and test is without restarting the server.

The web server is starting automatically, you should see the following message in the Eclipse console

The server is running at http://localhost:8080/

You can access the application, and the sample servlet that has been created using the URL: http://localhost:8080/employeeservice

To stop the server, click on the terminate button in the Eclipse console.

Configuring the REST support in the application

To be able to create and run REST services in your application you need to:
  • Add the JAX-RS, JAXB Jars in your project and application
  • Configure the web application (web.xml) to handle REST requests
Add JAX-RS, JAXB to your project
  1. Right click on the project and select menu entry Build Path > Configure Build Path...
  2. Click on the Add External JARs button

  3. Select all the JARs located in $JERSEY_HOME/lib  and $JAXB_HOME/lib folders. You can for better visibility and reuse create a user library with all these JARs
  4. You also need to copy the JARs in the web-inf/lib directory of your application, this step is mandatory to be sure that the JARs are included in the application when deployed to App Engine.
    Note: I do not like this step. I would prefer to do that by configuration of the build path, to automatically add the JARs to the WEB-INF/lib directory when executing/deploying the application. Unfortunately I did not find the way to do it, so if you know it, feel free to post a comment and I will update the article.

Configure the web application

In this step you will register a new URI to handle REST requests. To do that you need to register a new servlet that is using the Jersey API and configure it to a specific URI (eg: /ressources  and/or /rest) and configure what are the Java packages that contain the REST implementation classes. So you need to modify the web.xml of your application with the following entries:

  <servlet>
    <servlet-name>Jersey Web Application</servlet-name>
     <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
     <init-param>
     <param-name>com.sun.jersey.config.property.packages</param-name>
     <param-value>com.grallandco.employee.service.rest.impl</param-value>
     </init-param>
     <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>Jersey Web Application</servlet-name>
    <url-pattern>/resources/*</url-pattern>
  </servlet-mapping>
   <servlet-mapping>
    <servlet-name>Jersey Web Application</servlet-name>
    <url-pattern>/rest/*</url-pattern>
  </servlet-mapping>


This servlet that will answer to the /resources/ and /rest/ URL. The configuration parameter com.sun.jersey.config.property.packages is used by Jersey to list the packages where REST services implementation are located.Note that you can put as many package as you need to, you just need to separate the package names by a ; .


Creating a simple REST Service to test the environment

The project is now ready to contain REST service. It is time to create one.Create for example the class com.grallandco.employee.service.rest.impl.HelloWorldResource, be sure to use the package name that you have configured in the web.xml for the Jersey servlet, based on the configuration we have made in previous step the package is com.grallandco.employee.service.rest.impl

Here a sample class with the JAX-RS annotations:
package com.grallandco.employee.service.rest.impl;
import javax.ws.rs.Path;
import javax.ws.rs.GET;
import javax.ws.rs.Produces;
@Path("/hr/")
public class EmployeeResource {
 
 @GET
 @Produces("text/plain")
 @Path("/employee") 
 public String getEmployee() {
        return "Hello World!";
    }
}
You should be able to test it, stop the server and run it again, enter the following URL in your browser:
http://localhost:8080/resources/hr/employee
or
http://localhost:8080/rest/hr/employee


Deploying the application to Google App Engine

Before deploying the application you need to register a new application in Google App Engine using the Administartion Console, see the documentation here. In my example I have used "tugdual" as Application ID.
You can easily now deploy the application to Google App Engine by clicking on the  "Deploy App Engine Project" button Deploy App Engine Project Button available in the Eclipse toolbar.
To be able to deploy your application to Google App Engine, you need to check that your application can be registered, the application ID is stored in the WEB-INF/lib/appengine-web.xml.
<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
 <application>[your-application-id]</application>   
 <version>1</version>    
 <!-- Configure java.util.logging -->
  <system-properties>
   <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
 </system-properties>    
</appengine-web-app>

The App Engine deploy button prompts you for multiple informations: username (your Google account) and password.

When the deployment is complete you can access your application using the following URL:
http://[your-application-id].appspot.com/resources/hr/employee
or
http://[your-application-id].appspot.com/rest/hr/employee

Ading XML and JSON support to the service

Let's now add new method to manipulate an "Employee" object using the service, and the data format should be based on JSON and XML. This is where JAXB is useful, since it allows easily to transform marshall/unmarshall Java objects in XML -obviously- and JSON (cool isn't!)

Creating an Employee Class

Start with the creation of a new class to manipulate Employee data, this is a very simple Java class that could look like the following code:

package com.grallandco.employee.service.model;
import java.util.Date;

public class Employee {
    private String firstName;
    private String lastName;
    private Date hireDate;
    private String email;   
    public Employee(String firstName, String lastName, Date hireDate, String email) {
        this.firstName = firstName;
        this.lastName = lastName;
        this.hireDate = hireDate;
        this.email = email;
    }
    public Employee() {}
    public String getFirstName() {
 return firstName;
 }
    public void setFirstName(String firstName) {
 this.firstName = firstName;
 }
    public String getLastName() {
 return lastName;
 }
    public void setLastName(String lastName) {
     this.lastName = lastName;
 }
    public Date getHireDate() {
 return hireDate;
 }
    public void setHireDate(Date hireDate) {
 this.hireDate = hireDate;
 }
    public String getEmail() {
        return email;
 }
    public void setEmail(String email) {
        this.email = email;
   }
    public String toString() {
        StringBuffer sb = new StringBuffer();
        sb.append("First: ").append(getFirstName());
        sb.append(" - Last: ").append(getLastName());
        sb.append(" - Date: ").append(getHireDate());
        sb.append(" - Email: ").append(getEmail());
        return sb.toString();
    }
}
When implementing your "real" application with some persistence layer this POJO is the one as JDO/JPA entity.

Create a Converter class for your entity

I usually encapsulate all the transformation in some converter class, like that I do not directly couple my business class to the serialisation mechanism. (So I do that for classes and lists of classes). So instead of adding the JAXB annotations to the Employee class itself, let's create an EmployeeConverter class that will be responsible of the transformation and used by your REST service.

package com.grallandco.employee.service.converter;

import java.util.Date;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import com.grallandco.employee.service.model.Employee;

@XmlRootElement(name = "employee")
public class EmployeeConverter {
 private Employee entity = null;
 public EmployeeConverter() {
 entity = new Employee();
 }

 public EmployeeConverter(Employee entity) {
 this.entity = entity;
 }

 @XmlElement
 public String getFirstName() {
 return entity.getFirstName();
 }

 @XmlElement
 public String getLastName() {
 return entity.getLastName();
 }

 @XmlElement
 public Date getHireDate() {
 return entity.getHireDate();
 }

 @XmlElement
 public String getEmail() {
 return entity.getEmail();
 }

 public Employee getEmployee() {
 return entity;
 }

 public void setFirstName(String firstName) {
 entity.setFirstName(firstName);
 }

 public void setHireDate(Date hireDate) {
 entity.setHireDate(hireDate);
 }

 public void setLastName(String email) {
 entity.setEmail(email);
 }

 public void setEmail(String lastName) {
 entity.setLastName(lastName);
 }
}
You can now update your service to use this utility/converter class to return XML or JSON ojbect based on the content type of the request.

Add support to JSON and XML to your REST service

You need to change the EmployeeRessource class, to change the signature and add new annotations of the getEmployee() method.
The annotation you are adding:
  • @Produces({"application/xml", "application/json"}) : indicates which type of content will be produced by the service. Based on the type of the request.
  • @Path("/employee/{employeeEmail}/") :    change the Path to indicate a Path parameter, here for example the URL can accept an email in the URI - not the best example, but you get the point...
  • public EmployeeConverter getEmployee( @PathParam ("employeeEmail") String email)  : change the type returned by the method and take a parameter as String that match the Path param defined in the @Path annotation
Here the complete class code:
package com.grallandco.employee.service.rest.impl;

import javax.ws.rs.Path;
import javax.ws.rs.GET;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import com.grallandco.employee.service.converter.EmployeeConverter;
import com.grallandco.employee.service.model.Employee;

@Path("/hr/")
public class EmployeeRessource {

 
 @GET
 @Produces({"application/xml", "application/json"})
 @Path("/employee/{employeeEmail}/") 
 public EmployeeConverter getEmployee( @PathParam ("employeeEmail") String email) {
 //dummy code
 Employee emp = new Employee();
 emp.setEmail(email);
 emp.setFirstName("John");
 emp.setLastName("Doe");
 EmployeeConverter converter = new EmployeeConverter(emp);
 return converter;
 } 
}


Test the service

You can now run the server locally and test the service
http://localhost:8080/resources/hr/employee/tug@grallandco.com
This will return an XML document.
If you want to test the JSON call you have multiple choice:
  • Using following command
tgrall$ curl -H "Accept: application/json" http://localhost:8080/resources/hr/employee/tug@grallandco.com
{"email":"tug@grallandco.com","firstName":"John","lastName":"Doe"}
  • Using an HTTP client that allows your to configure/set the HTTP request completely, I am using the Poster Firefox Plugin
  • Using some Javascript code in an application
You can repeat the test on your deployed application on Google App Engine.

Conclusion

In this article you have learned how to create and deploy a new REST Service on Google App Engine. This service has been created with the JAX-RS Reference Implementation the Jersey project. In the next article you will learn how to add persistence and create a CRUD Rest service on Google App Engine.

Sunday, November 8, 2009

NantesJUG: Next Meeting Friday November 13th about "Google Technologies"

This Friday , the Nantes JUG is pleased to organize a conference about "Google Technologies". Didier Girard, http://twitter.com/dgirard , will be presenting and demoing cool stuff from Google:

  • GWT 2
  • Android
  • Wave
  • App Engine
Want to come? Register for free here

The event will occur at the "Ecole des Mines de Nantes" one of our sponsor from 7 to 9 pm.

Thursday, March 29, 2007

Online Public Libraries: Google Books and now.. Europeana

I am sure that you know about Google Books: http://books.google.com/

Even if I truly believe that Google wants the good by publishing all this information for free - We all remember the sentence: "Don't be evil" that is part of the Google code of Conduct-. I think it is not a good idea to have only one service that provide access to the "world culture"... Competition is always good for consumer...

The French National Library (BNF) and some other public libraries (Hungarian, Portugal) have created a new site to offer a similar service in beta: http://www.europeana.eu. It is true that currently the list of books is really small compare to Google but I am inviting you to use both of then.