Percona PAM authentication in JDBC

Percona's MySQL distribution supports PAM based authentication.  Unfortunately this currently confuses the heck out of the latest MySQL JDBC driver.  First of all, the percona server asks the client to use the "dialog" authentication plugin.  Unless percona binaries are installed on the machine, this plugin is not available.  One can substitute the clear text password plugin, but it depends on SSL being present.

What can you do if your Percona server is configured to use pam authentication and does not support secure connections?  Write your own MySQL connector authentication plugin.  Assuming the connector jar distribution is on your classpath, simply compile the below:

public class DialogPasswordPlugin extends MysqlClearPasswordPlugin {
    @Override
    public String getProtocolPluginName() {
        return "dialog";
    }

    @Override
    public boolean requiresConfidentiality() {
        return false;
    }
}

put the compiled class on your classpath and add "authenticationPlugins=com.rubiconproject.mysql.percona.DialogPasswordPlugin" to your JDBC URL.
You should now be able to authenticate with the Percona server.

Resolving generic method return types

Java supports generic/parameterized return types:

public <t> T anyType();

A method with a generic return type like above is now expected to return different types depending on its caller. For example -

String string = anyType();

expects "anyType()" to return a String. Whereas

Integer i = anyType();

expects "anyType()" to return an Integer.  There does not appear to ber a way of determining a method's expected return type at runtime.  To get around this, many APIs tend to specify the expected return type by describing it in method parameters - for example:

public <t> T anyType(Class<T> type);
public <T> T getBean(String beanName,Class<T> beanType);

I put together a library that provides an alternative -

public <T> T anyType(){
  Class<?> myReturnType = Resolver.getConcreteReturnType();
  ..
}

The library works by inspecting a method's call stack. The call stack typically includes class names, method names, and line numbers. Using these, the library locates the code surrounding method invocations using the ASM bytecode manipulation library and inspects it for "CHECKCAST" operations. The found type is the expected return type of the parameterized method. Extracting parameterized return types (e.g. Map<String,List<Integer>>) is a little trickier but also possible.

Instead of going with the google code route to share the project like I'm used to, I've decided to give github a try. I've also set up continuous integration using drone.io - so far, it's been an excellent experience. Read more over at https://github.com/ProfessorEugene/rrd-generics-inspector !

Extending static methods in java

Java doesn't allow extending static methods.  I put together a library with an

@OverrideStatic

annotation that allows one to change this behavior over at http://code.google.com/p/rrd-static-extender/.

Having recently done a bit of dirty copy/pasting to extend a third party library with a bunch of static methods at work, I've given some thought to what it would take to allow one to extend static methods.

Take the below hierarchy for example:

class Base{
  void instanceMethod(){
    staticMethod();
  }
  static void staticMethod(){
    ...
  }
}
class Subclass extends Base{
   static void staticMethod(){
    ...
   }
}     

When one creates an instance of Subclass and invokes "instanceMethod" , Base#staticMethod() instead of Subclass#staticMethod() is invoked.  Other than using something like aspectj or maybe powermock there isn't really a good way to change this behavior.  

In some cases, one might want all instances of Subclass to invoke Subclass#staticMethod() instead of Base#staticMethod when "instanceMethod" is invoked.  The desired behavior seems pretty simple to explain:

• subclasses should have the ability to override their super-class' static methods
• instance methods of such classes should invoke the overriding method
• any inner and local classes of the super-class who's enclosing instance is a subclass should also invoke the overriding method.
• instance methods of such classes' super classes should invoke the overridden method
• in order to not interfere with expected behavior one should be able to mark an overriding method with an annotation

Starting with an @OverridesStatic annotation, there's a couple of approaches one could take to implement this behavior.  At the end of the day, the bytecode of the base class needs to be instrumented to dispatch method calls to "staticMethod" to the appropriate place depending on the context of such calls.  

Having intercepted a static method call, the instance of the caller needs to be inspected to check whether it is assignable from a class which contains the overriding method.  If said instance is an inner class, it's enclosing class reference (something like this$1) needs to be recursively inspected.  If the instance is a type of a subclass with the overriding method, the overriding method should be called in place of the overridden method.

Using the ASM bytecode manipulation framework to detect and modify any invocations of static methods in a class is a breeze.  The problem is to how to load the modified bytecode.  There's a few alternatives:

• Use your own class loader that overrides the "defineClass" method for loading any classes that might need to have their static methods overridden.  The problem here is that portions of your application now needs to be loaded through a custom class loader - something like TomcastInstrumentableClassLoader.
• Rename classes (eg. the popular CGLib enhancer approach) as @OverrideStatic annotations are detected and load them in a separate class loader.  Create a deep-proxy loaded in the application class loader that recursively copies fields  and invokes methods on instances in the separate object hierarchy.  This path is laden with traps - your proxy gateway won't work on final classes and the amount of reflection that has to happen is going to take a toll on performance.
• Instrument the entire virtual machine using a java agent.  While this is the cleanest way to go, it typically requires one to start the JVM with some funky looking command line arguments.

As anyone that's used jconsole before must be suspicious of, Java provides an way to allow one to instrument running virtual machines.  Launching jconsole shows you a list of java virtual machines currently executing on your machine and allows you to 'attach' to one.

Presumably, jconsole achieves this via the java attach API.  This API lets you list any running virtual machines and attach to them and load agents.  While the implementation of the API is proprietary, the "Instrumentation" framework used by such agents is not.  Indeed, coding against the java.lang.instrument.Instrumentation interface seems to be a portable endeavor.

In light of this, I came up with the rrd-attach-util library to allow one to attach to and instrument the virtual machine in which it is executed.  The library provides a very simple API to obtain an Instrumentation instance for the running VM without having to jump through hoops packaging an actual agent:

LocalInstrumentationFactory.getLocalInstrumentation();

Once an Instrumentation object is obtained, all loaded classes can be inspected and instrumented.  Since attach API implementations are proprietary, as far as I know rrd-attach-util only works with Sun/Oracle JDK6 and later.  This doesn't seem like a big deal since most folks seem to be running Oracle JREs these days.

I used this API to create a simple implementation of a "static extender"  To use it, one annotates static methods with an @OverrideStatic annotation, calls OverrideStatics.enable(), and all such annotations are picked up and static method invokation changes to the behavior described above until such time that OverrideStatics.disable() is invoked.

A unit test that demonstrates this behavior in action:

static class BaseClass{  
 public String callStaticMethod() throws Exception{      
  return staticMethod();   
 }
 public static String staticMethod(){
  return "BaseClass";
 }
}
static class SubClass extends BaseClass{
 @OverrideStatic
 public static String staticMethod(){
  return "Subclass";
 }
}
@Test
public void demonstrateOverrideStatic() throws Exception{
 /* instantiate a base and sub class */
 BaseClass baseClass = new BaseClass();
 SubClass subClass = new SubClass();
 /* BaseClass#callStaticMethod will always route to BaseClas#staticMethod */
 assertEquals("BaseClass",baseClass.callStaticMethod());
 /* SubClass#callStaticMethod will always route to BaseClas#staticMethod */
 assertEquals("BaseClass",subClass.callStaticMethod());
 /* lets see what happens when we enable static overrides */  
 OverrideStatics.enable();  
 /* SubClass#callStaticMethod will now route to SubClass#staticMethod */
 assertEquals("Subclass",subClass.callStaticMethod());
 /* BaseClass#callStaticMethod will still route to BaseClass#staticMethod */
 assertEquals("BaseClass",baseClass.callStaticMethod());
}

My implementation works great for my needs and doesn't seem to cause a big performance hit.  There is however a lot of room for improvement:

• The actual code could use a cleanup.
• Calls to overridden static methods are intercepted and routed to a utility method that uses reflection to decide which method to invoke.  It seems like this can be done through bytecode manipulation like the actual interception of these calls.

In general though, this is a great example of what one can do in a few days to totally modify Java behavior through bytecode manipulation.  The library also serves as a functional showcase of the utility provided by the rrd-attach-util API.  Until I've got the time to clean up, use at your own risk!

Spring: Force a bean to be the first to initialize

Spring provides the "depends-on" bean attribute to control bean initialization order.  Unfortunately, there are some rare cases where you really want to run some code before the application context initializes.  An example scenario might involve messing around with the thread class loader before something like "PropertiesPlaceholderConfigurer" has a chance to initialize.

There are a number of ways to go about this, but the easiest might be through implementing a PriorityOrdered BeanFactoryPostProcessor.  After all spring beans are defined, spring calls all BeanFactoryPostProcessor s defined in the application context.  Post processors that implement PriorityOrdered, and have the highest precedence are called first.

To execute some code before any other spring beans or processors have a chance to initialize, one can therefore use something like the following:

        
/**
 * An example spring pre-initializer.  Referncing this class as a bean in an 
 * application context will cause it to be the first executed PostProcessor 
 * after all beans in the application context are defined.
 */
public class PreInitializer implements BeanFactoryPostProcessor, PriorityOrdered {
 @Override
 public int getOrder() {
  return Ordered.HIGHEST_PRECEDENCE;
 }
 @Override
 public void postProcessBeanFactory(
   ConfigurableListableBeanFactory beanFactory) throws BeansException {
  /* put initialization code here */
 }
}

This is an easy hack to perform initialization tasks that might affect the application context before anything else - including property resolution has a chance to run.

Computing path to maven module base directory

Having spent a bunch of time writing functional tests for maven projects with many modules, I tend to end up needing access to the project base path (typically the path with the pom.xml file in it).  This isn't much of a problem when running a single module - new File(".") pretty much does it.  On the other hand when attempting to access modules outside of the jvm base path, things start to get tricky.

There's a couple ways to skin this cat - probably one of the more portable ones is a utility method that accepts a Class and a path and returns a File by checking the supplied Class' classloader:

        public static File getPathInModule(Class<?> classInModule,String ... items){
                String tcPath = classInModule.getProtectionDomain().getCodeSource().getLocation().getFile();
                String[] tcPathParts = StringUtils.split(tcPath,File.separator);
                StringBuffer pomPath = new StringBuffer();
                for(String part:tcPathParts){
                        pomPath.append(File.separator);
                        if("target".equals(part)){
                                break;
                        }
                        pomPath.append(part);
                }
                pomPath.append(StringUtils.join(items,File.separator));
                return new File(pomPath.toString());
        }

This works fairly well as long as there is a target directory present, which is for my purposes - always. It could be modified to look for a pom.xml file instead though.

Hadoop 3.0.0 maven repository

Recent maven hadoop artifacts are pretty hard to come by.  The central repository contains stable and 2.0.0-alpha releases, CDH repositories have 2.0.X releases, and the apache repository has 3.X snapshots.  That means if you want 2.3.X or 3.X functionality, you're either stuck pulling in snapshots or building/hosting your own releases.

The apache snapshots are problematic because individual build snapshots don't appear to be retained for very long.  Building/hosting your own artifacts seems like a pain.  Hopefully, this is a temporary issue until hadoop 3.X is released.  Until then, here's a maven repository that has (unofficial) 3.0.0 release artifacts with source jars.

It is pretty easy to put in pretty much any hadoop version into the repository.  Visit the project page or shoot me a email to request one!

First Project in Central Maven Repository - spring-event-router

I've started a few open source java projects built with maven before, but have always ended up using a private repository to host binaries.  One project of note was the CampusEAI portlet-archetype.  Though under a GPLv3 licence, the CampusEAI repository was not publicly accessible until very recently.  I am delighted that CampusEAI appears to have opened up their previously closed maven repository to the world.  The CampusEAI portlet-archetype is an exciting project that deserves a post of its own and is now publicly available at http://svn.campuseai.org/site/portlet-archetype/

Unfortunately, the CampusEAI repository has not been publicly accessible for almost a year after I left the company.  Noticing this, I have committed to publishing any future OSS Java work into the central maven repository.

I've been working a bit with Spring Framework ApplicationEvents at work and thought it would be neat to write a simple dispatch mechanism by which listeners could subscribe to specific types of events without tons of instanceof operations.  I saw a couple of posts on velocityreviews that contained snippets of code tackling similar issues.  There's even a google code proejct over at http://code.google.com/p/spring-custom-annotations/ that attempts to solve a similar problem with their "@EventListener" annotation.  That project appears to no longer be maintained and has some warnings against usage in production code.  In light of this, I decided to hack up some code myself and try out google code's new git source control as well as using Sonatype to publish my first artifacts into the Central Maven repository.

I put together a tiny library called "spring-event-router" hosted over at http://code.google.com/p/spring-event-router/ and followed the Sonatype OSS Maven Repository Usage Guide to try and publish my binaries to the Central Maven Repository.  Setting up a project with Sonatype was a (necessarily) manual process but it only took around 24h to get my first binaries into the repository.  The gist of the ordeal is creating a JIRA issue and having Sonatype review your project to validate some requirements before granting you the ability to stage and publish your artifact.  Once your artifact has been published, a secondary review is performed; upon completion any releases into the Sonatype repository are synced with Central on an hourly basis without further Sonatype oversight.

Though admittedly a very simplistic project, I was very happy with the ease of being able to use "mvn release:prepare; mvn release:perform" to publish artifacts into the central maven repository.  The binaries are available here and should resolve as valid dependencies for any maven project without the need of a custom section!