Tuesday 19 July 2011

Rules Engine on mobile platform : Getting JRuleEngine to do what it promised ..

During my work on rule engine on mobile platform, I came across this JRuleEngine which I briefly touched upon, in my last blog here. In there, I mentioned that I extended JRuleEngine to support multiple "facts" of same type. I thought it would be worth-while to publish what and how I did it, so this blog.


The last released ( 2008 ? ) version of JRuleEngine said in the documentation that it supports multiple facts of same type, but it actually didn't. All the facts which are loaded in the working memory are stored in a map with the getClass().getName() as key, which obviously means that only one fact of a "type" can be stored since the key in the map must be unique. So the last fact that was fed into working memory always wins, and is considered for rules execution. This is not what ( and most of us ) really wanted. 


To overcome this issue, there were a few considerations that were needed. 
(a)  Working memory should be capable of loading multiple facts of same type
(b)  Rules engine should be able to query for and receive an array of matching facts
(c)  The action/s should be able to act on all the matching facts
(d)  The result should be able to return all the facts
(e)  This behavior should be followed by all rules in the rule file


To take care of this issue, first thing required was to store the facts with a unique identifier, and so the engine would need a method to be called upon the facts which returned a unique identifier.  To achieve this, I created an abstract class named "ParentObject" which defined a "getId()" method and inherited all the facts from this ParentObject. So all the facts had to provide implementation for the getId() method. In my case, this method just returns the unique id field ( primary key in database ). But this id alone may not be unique across different types of facts, so while adding these facts into the working memory map, I appended getClass().getName() with the returned value from getId() to make this instance entry really unique.


Fact object : 
-----------------
public class FactObject extends ParentObject {
...
...
    public int getId() 
    {
         return this.id;
    }
...
...
}


Rule Engine :
 -----------------

public class StatefulRuleSessionImpl implements StatefulRuleSession {
...
...
if (object instanceof ParentObject)
{
    workingMemory.put(object.getClass().getName()+"_"+((ParentObject)object).getId(),object);
}

...
...

}


I then wrote a method to retrieve all the matching facts from the working memory. So whenever executeRules() is called, the engine now calls this new method that returns all the facts matching the "if" condition/s in the rule/s.


To see if the fact passes the constraint in the rule, the rule engine first gets the matching facts from working memory, executes the LHS in the constraint, gets the return value and then parses the RHS of the constraint in the rule. It then compares the return value from the method call on fact with the result of parsing RHS, and decides if the facts pass the constraint. So on the same lines, my code gets all the matching facts from the working memory, stores the fact id and the result for each fact from LHS expression in to a map, and then later loops through the map to filter out the facts which do not pass the first constraint. 


This check is then recursively done for each constraint, one by one, leaving behind only the facts which match all the constraints in the rule.


Then comes the "actions" part.  The action needs to be executed on all the facts which matched all the constraints in last step, so my extended code now evaluates the LHS part of the action once and then in loop, executes the RHS part of the action for all the facts that passed last step.


The rule engine then repeats this process for all actions within the rule, for all rules in the rules file. 


The result, by default, gathers all facts anyway, so didn't have to do anything special there.

Tuesday 28 June 2011

Rules Engine on mobile platform : Getting Rule engine to work on Mobile Platform


One fine morning I started googling about rules engines, mostly open source. I came across this Drools rule engine. Sounded exciting. Getting rules engine to work on windows wasn’t a huge challenge, but since I was totally new to Drools concept, RETE algorithm, it was a bit challenging. Within a matter of couple of days, I was able to do a proof of concept with Drools. The next challenge was to do something that was never done before or very few folks out there have tried, was to get Drools to work on a mobile platform. Since Android’s getting very popular these days and since it has good java support ( and of-course I had recently bought a new Android phone ), Android was the mobile platform of choice !

Getting Drools to work on Android, wasn’t as easy as it appeared initially. My thought ( which was later found to be too optimistic to be true ) was that Drools is java based and android supports java, so just port drools library and application to Android and run the application. The assumption was fare and simple, I needed to run a java app on a platform that supports java. The ride was fun but wasn’t as easy as it sounds.

I had two strategies to begin with. “A” – to compile and load rules at run time on android and “B” – to pre-compile the rules and package the precompiled/serialized rules and de-serialize and load on android. Option “B” appeared to be more suitable for a “mobile” type environment, at least at this initial stage, but option “A” was better from application/rule upgrade standpoint, so I started with option “A”.

After getting a basic Hello World android app built using Eclipse and Android SDK, I pushed it out to my android device and boom! It just worked !! During this small ( ? ) exercise I learnt a lot of good android stuff which proved to be very useful in later part of this journey– how to root an android device, adb tools, pushing and pulling files to and from the device, packaging application and so on.  Later, upgrading the app to use SQLlite database wasn’t a huge deal.

Main challenge (and head-ache) started when I starting writing the Drools part of my proof-of-concept app. I started off with a minimal set of drools runtime libraries ( to keep the app size to minimum), viz. drools-core, drools-compiler, drools-api , antlr and the mvel jar. Using compatible versions of these components is very important. I chose the drools 5.1.0 version and obtained compatible versions of the dependencies as and when required through out this exercise. In fact, I had switched to drools 5.2 but had to switch back to 5.1 since the packaging tool did not like drools-compiler jar from 5.2 and failed with stack-overflow error and I didn’t have time to figure out why. So stepped back to 5.1 since my aim was to get any version of drools to work on android anyway. When I compiled the first drools-android app, it started complaining about eclipse jdt jar not being in classpath which I added thereafter. This first compilation gave me little more insight of what happens when you compile/package an android app. 

My initial idea of android’s support for java had a new dimension now, when I found out that the packing tool re-compiles the sun java compiled classes into another format called DEX ( Dalvik Executable ) format, which could be interpreted by the DVM ( Dalvik virtual machine ) that runs on Android. This led to another discovery that Android’s java is not sun’s java, but it is the Google implementation of java. Obviously to keep the vm lightweight, they have only partially implemented the java spec ( This was going to add another obstacle in my journey down the stream, which I had no knowledge about, at this point of time ).

Ok. So when I packaged and build my first small drools android app and ported on to android phone, it installed successfully ( woo hoo ! ) , but crashed when I opened up the installed app. I enabled the debugging and “adb logcat” showed me why it crashed. The DVM crashed with an error saying “parent class loader may not be null”. When I opened up the drools source, org.drools.util.CompositeClassLoader, I found the constructor that was the culprit.

    public CompositeClassLoader(final ClassLoader parentClassLoader) {
        super(null);
        loader.set( new DefaultLoader() );
    }

Upon further investigation and googling, I found a blog post by Kris Verlaenen (http://blog.athico.com/2011/03/jbpm5-lightweight-running-on-android.html) and he had faced the same issue. Thanks to Kris, this was a quick fix for me, to replace super(null) call by super(CompositeClassLoader.class.getClassLoader()). After re-compiling the source and redeploying the app on android, the app again crashed while opening up and this time it failed compiling the rule ( .drl ) file saying that eclise jdt jar is not in the classpath. This is a huge file ( @4mb alone) for a mobile environment. Post some help from the mailing list, I decided to use janino, something like this. To get this to work, I had to add “janino-2.5.15.jar” to the application and classpath.

        …
        Properties properties = new Properties();
        properties.put("drools.dialect.java.compiler", "JANINO")
        KnowledgeBuilderConfiguration kbConfig =
        KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration( properties );
        KnowledgeBuilder kBuilder = KnowledgeBuilderFactory.newKnowledgeBuilder(kbConfig);
        kBuilder.add( ResourceFactory.newFileResource( drlPath ), ResourceType.DRL );
        KnowledgeBase kBase = KnowledgeBaseFactory.newKnowledgeBase();
        kBase.addKnowledgePackages( kBuilder.getKnowledgePackages() );
        …

I then started getting StringIndexOutOfBounds errors and I had to comment out the java version check (dangerous ! ) from AbstractParser.java and ParseTools.java from mvel compiler jar to get past this issue. Moving ahead, I packaged this app ( @ 1.7 mb – packaged ) and ported on the android device. Now it started throwing null pointer exceptions from “build” method of org.drools.rule.builder.RuleBuilder class indicating problem with the compilation. At this time I had spent reasonable amount of time getting this option “A” to work, So I decided to start following my plan “B”.  Here I serialized my knowledge base and wrote it to a .pkg file. ( all referenced classes need to be serializable )

        …
        OutputStream os = new FileOutputStream( rbPath );
        ObjectOutputStream oos = new ObjectOutputStream( os );
        oos.writeObject( kBase );
        …

This way, I got past the compilation problem. Planted ahead of me was the issue that I touched upon briefly earlier – the issue of partial implementation of java by Google. Google has not implemented entire java spec, but left out some packages including java beans package which my app required on android. In its absence, it started throwing “java.lang.NoClassDefFoundError: java.beans.Introspector” errors indicating missing package. On further research and help from mailing list ( thanks to Michael Anstis ) I found out that Apache have developed full java implementation named “harmony” and have a jar named ‘beans.jar’ as a part of their runtime that was what I needed ! I downloaded the jar and re-packaged it using “jarjar” utility and included in the app and it took care of this missing package problem !

The next one standing in the line was issue of de-serializing classes serialized in JVM environment into the DVM environment. KnowledgeBase that I had serialized was in the sun’s java format and what the DVM was expecting when de-serialized was the dalvik executable format. I spent some amount of time on it with almost no success .. so I decided to give up on option B and went back to Option A.

I picked it up where I had left it out. Null pointer exception during rule build. I tried using package builder instead of knowledgebuilder. Mean while I also figured out that the package builder configuration is not able to load the default JavaDialectConfiguration ( didn’t spend enough time to understand why ), so I created one myself and set it to the package builder configuration.

         Properties properties = new Properties();
         properties.setProperty( "drools.dialect.java.compiler","JANINO" );
         PackageBuilderConfiguration cfg = new PackageBuilderConfiguration( properties);
         JavaDialectConfiguration javaConf =  new JavaDialectConfiguration( );
         javaConf.init(cfg);
         cfg.setDialectConfiguration( "java", javaConf );
         PackageBuilder builder = new PackageBuilder( cfg );
         builder.addPackageFromDrl(new InputStreamReader(is));
         RuleBase ruleBase  = RuleBaseFactory.newRuleBase();
         ruleBase.addPackage( builder.getPackage() ); 

It still led to null pointer exceptions, where I was creating the PackageBuilderConfigurator instance. With a few deep dives into drools source, I figured out what the problem was. While creating the package builder configuration, along with properties, I passed “null” as second argument, which did the trick.

         PackageBuilderConfiguration cfg = new PackageBuilderConfiguration( properties, null);

With this done, I was all set to run drools on android , knowing a little about challenges which were going to be set before me along the journey !  All compiled, packaged well, deployed on android, it was now able to go past the rule building ( and the null pointer exception ) and the application crashed with HEAP error .. that was the next challenge – heap / cpu management !

I spent some good amount of time getting over the heap error, but third party libraries being involved, and unavailability of rich profiling tools for android, I decided to give up on Drools and decided to go with something lightweight, like JRuleEngine (http://jruleengine.sourceforge.net/). JRuleEngine being another java implementation and my java - android workbench already set, it was a relatively easy deal to get the vanilla JRuleEngine to work.

JRuleEngine looked like an abandoned work ( no updates post 2008 ), so I had to extend it to get it to do what I needed. The last sourced version of JRules does not support ( eventhough the documentation says, to some extend, it does ! ) inserting multiple facts of same “type” into working memory, since it accesses the facts with getClass().getName(). I extended the JRuleEngine to consider multiple facts of same “type” while evaluating rules ( assumptions ) and executing rules ( actions ), since rules need to be evaluated for every instance of the fact “type” and actions need to be taken on “all” matching facts.

Again, I had to download the Apache Harmony’s beans.jar and awt.jar and re-package it for android’s DVM using jarjar utility. With this jar , extended jrules.jar from JRulesEngine and jsr94.jar in place, following snippet worked very well. ( based on example1 packaged in JRuleEngine. )

            Class.forName( "org.jruleengine.RuleServiceProviderImpl" );
RuleServiceProvider serviceProvider = RuleServiceProviderManager.getRuleServiceProvider( "org.jruleengine" );
RuleExecutionSet res1 = ruleAdministrator.getLocalRuleExecutionSetProvider( null ).createRuleExecutionSet( inStream, null );
// instream is the input stream for the xml based rules file.
            RuleAdministrator ruleAdministrator = serviceProvider.getRuleAdministrator();
            RuleRuntime ruleRuntime = serviceProvider.getRuleRuntime();
String uri = res1.getName();
StatefulRuleSession statefulRuleSession =  (StatefulRuleSession) ruleRuntime.createRuleSession( uri, new HashMap(),RuleRuntime.STATEFUL_SESSION_TYPE );
statefulRuleSession.addObjects( input ); // input is ArrayList of facts
statefulRuleSession.executeRules();
List results = statefulRuleSession.getObjects();

I later modified this basic app and added a clickable list view in recursion which let me implement what I wanted – a rules based application on mobile platform with a guided selling UI J

This marked the final milestone of my journey, a journey that gave me a deep insight of what I would otherwise have always wondered about.