Tuesday 28 June 2011

Rules Engine on mobile platform : Getting Rule engine to work on Mobile Platform


One fine morning I started googling about rules engines, mostly open source. I came across this Drools rule engine. Sounded exciting. Getting rules engine to work on windows wasn’t a huge challenge, but since I was totally new to Drools concept, RETE algorithm, it was a bit challenging. Within a matter of couple of days, I was able to do a proof of concept with Drools. The next challenge was to do something that was never done before or very few folks out there have tried, was to get Drools to work on a mobile platform. Since Android’s getting very popular these days and since it has good java support ( and of-course I had recently bought a new Android phone ), Android was the mobile platform of choice !

Getting Drools to work on Android, wasn’t as easy as it appeared initially. My thought ( which was later found to be too optimistic to be true ) was that Drools is java based and android supports java, so just port drools library and application to Android and run the application. The assumption was fare and simple, I needed to run a java app on a platform that supports java. The ride was fun but wasn’t as easy as it sounds.

I had two strategies to begin with. “A” – to compile and load rules at run time on android and “B” – to pre-compile the rules and package the precompiled/serialized rules and de-serialize and load on android. Option “B” appeared to be more suitable for a “mobile” type environment, at least at this initial stage, but option “A” was better from application/rule upgrade standpoint, so I started with option “A”.

After getting a basic Hello World android app built using Eclipse and Android SDK, I pushed it out to my android device and boom! It just worked !! During this small ( ? ) exercise I learnt a lot of good android stuff which proved to be very useful in later part of this journey– how to root an android device, adb tools, pushing and pulling files to and from the device, packaging application and so on.  Later, upgrading the app to use SQLlite database wasn’t a huge deal.

Main challenge (and head-ache) started when I starting writing the Drools part of my proof-of-concept app. I started off with a minimal set of drools runtime libraries ( to keep the app size to minimum), viz. drools-core, drools-compiler, drools-api , antlr and the mvel jar. Using compatible versions of these components is very important. I chose the drools 5.1.0 version and obtained compatible versions of the dependencies as and when required through out this exercise. In fact, I had switched to drools 5.2 but had to switch back to 5.1 since the packaging tool did not like drools-compiler jar from 5.2 and failed with stack-overflow error and I didn’t have time to figure out why. So stepped back to 5.1 since my aim was to get any version of drools to work on android anyway. When I compiled the first drools-android app, it started complaining about eclipse jdt jar not being in classpath which I added thereafter. This first compilation gave me little more insight of what happens when you compile/package an android app. 

My initial idea of android’s support for java had a new dimension now, when I found out that the packing tool re-compiles the sun java compiled classes into another format called DEX ( Dalvik Executable ) format, which could be interpreted by the DVM ( Dalvik virtual machine ) that runs on Android. This led to another discovery that Android’s java is not sun’s java, but it is the Google implementation of java. Obviously to keep the vm lightweight, they have only partially implemented the java spec ( This was going to add another obstacle in my journey down the stream, which I had no knowledge about, at this point of time ).

Ok. So when I packaged and build my first small drools android app and ported on to android phone, it installed successfully ( woo hoo ! ) , but crashed when I opened up the installed app. I enabled the debugging and “adb logcat” showed me why it crashed. The DVM crashed with an error saying “parent class loader may not be null”. When I opened up the drools source, org.drools.util.CompositeClassLoader, I found the constructor that was the culprit.

    public CompositeClassLoader(final ClassLoader parentClassLoader) {
        super(null);
        loader.set( new DefaultLoader() );
    }

Upon further investigation and googling, I found a blog post by Kris Verlaenen (http://blog.athico.com/2011/03/jbpm5-lightweight-running-on-android.html) and he had faced the same issue. Thanks to Kris, this was a quick fix for me, to replace super(null) call by super(CompositeClassLoader.class.getClassLoader()). After re-compiling the source and redeploying the app on android, the app again crashed while opening up and this time it failed compiling the rule ( .drl ) file saying that eclise jdt jar is not in the classpath. This is a huge file ( @4mb alone) for a mobile environment. Post some help from the mailing list, I decided to use janino, something like this. To get this to work, I had to add “janino-2.5.15.jar” to the application and classpath.

        …
        Properties properties = new Properties();
        properties.put("drools.dialect.java.compiler", "JANINO")
        KnowledgeBuilderConfiguration kbConfig =
        KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration( properties );
        KnowledgeBuilder kBuilder = KnowledgeBuilderFactory.newKnowledgeBuilder(kbConfig);
        kBuilder.add( ResourceFactory.newFileResource( drlPath ), ResourceType.DRL );
        KnowledgeBase kBase = KnowledgeBaseFactory.newKnowledgeBase();
        kBase.addKnowledgePackages( kBuilder.getKnowledgePackages() );
        …

I then started getting StringIndexOutOfBounds errors and I had to comment out the java version check (dangerous ! ) from AbstractParser.java and ParseTools.java from mvel compiler jar to get past this issue. Moving ahead, I packaged this app ( @ 1.7 mb – packaged ) and ported on the android device. Now it started throwing null pointer exceptions from “build” method of org.drools.rule.builder.RuleBuilder class indicating problem with the compilation. At this time I had spent reasonable amount of time getting this option “A” to work, So I decided to start following my plan “B”.  Here I serialized my knowledge base and wrote it to a .pkg file. ( all referenced classes need to be serializable )

        …
        OutputStream os = new FileOutputStream( rbPath );
        ObjectOutputStream oos = new ObjectOutputStream( os );
        oos.writeObject( kBase );
        …

This way, I got past the compilation problem. Planted ahead of me was the issue that I touched upon briefly earlier – the issue of partial implementation of java by Google. Google has not implemented entire java spec, but left out some packages including java beans package which my app required on android. In its absence, it started throwing “java.lang.NoClassDefFoundError: java.beans.Introspector” errors indicating missing package. On further research and help from mailing list ( thanks to Michael Anstis ) I found out that Apache have developed full java implementation named “harmony” and have a jar named ‘beans.jar’ as a part of their runtime that was what I needed ! I downloaded the jar and re-packaged it using “jarjar” utility and included in the app and it took care of this missing package problem !

The next one standing in the line was issue of de-serializing classes serialized in JVM environment into the DVM environment. KnowledgeBase that I had serialized was in the sun’s java format and what the DVM was expecting when de-serialized was the dalvik executable format. I spent some amount of time on it with almost no success .. so I decided to give up on option B and went back to Option A.

I picked it up where I had left it out. Null pointer exception during rule build. I tried using package builder instead of knowledgebuilder. Mean while I also figured out that the package builder configuration is not able to load the default JavaDialectConfiguration ( didn’t spend enough time to understand why ), so I created one myself and set it to the package builder configuration.

         Properties properties = new Properties();
         properties.setProperty( "drools.dialect.java.compiler","JANINO" );
         PackageBuilderConfiguration cfg = new PackageBuilderConfiguration( properties);
         JavaDialectConfiguration javaConf =  new JavaDialectConfiguration( );
         javaConf.init(cfg);
         cfg.setDialectConfiguration( "java", javaConf );
         PackageBuilder builder = new PackageBuilder( cfg );
         builder.addPackageFromDrl(new InputStreamReader(is));
         RuleBase ruleBase  = RuleBaseFactory.newRuleBase();
         ruleBase.addPackage( builder.getPackage() ); 

It still led to null pointer exceptions, where I was creating the PackageBuilderConfigurator instance. With a few deep dives into drools source, I figured out what the problem was. While creating the package builder configuration, along with properties, I passed “null” as second argument, which did the trick.

         PackageBuilderConfiguration cfg = new PackageBuilderConfiguration( properties, null);

With this done, I was all set to run drools on android , knowing a little about challenges which were going to be set before me along the journey !  All compiled, packaged well, deployed on android, it was now able to go past the rule building ( and the null pointer exception ) and the application crashed with HEAP error .. that was the next challenge – heap / cpu management !

I spent some good amount of time getting over the heap error, but third party libraries being involved, and unavailability of rich profiling tools for android, I decided to give up on Drools and decided to go with something lightweight, like JRuleEngine (http://jruleengine.sourceforge.net/). JRuleEngine being another java implementation and my java - android workbench already set, it was a relatively easy deal to get the vanilla JRuleEngine to work.

JRuleEngine looked like an abandoned work ( no updates post 2008 ), so I had to extend it to get it to do what I needed. The last sourced version of JRules does not support ( eventhough the documentation says, to some extend, it does ! ) inserting multiple facts of same “type” into working memory, since it accesses the facts with getClass().getName(). I extended the JRuleEngine to consider multiple facts of same “type” while evaluating rules ( assumptions ) and executing rules ( actions ), since rules need to be evaluated for every instance of the fact “type” and actions need to be taken on “all” matching facts.

Again, I had to download the Apache Harmony’s beans.jar and awt.jar and re-package it for android’s DVM using jarjar utility. With this jar , extended jrules.jar from JRulesEngine and jsr94.jar in place, following snippet worked very well. ( based on example1 packaged in JRuleEngine. )

            Class.forName( "org.jruleengine.RuleServiceProviderImpl" );
RuleServiceProvider serviceProvider = RuleServiceProviderManager.getRuleServiceProvider( "org.jruleengine" );
RuleExecutionSet res1 = ruleAdministrator.getLocalRuleExecutionSetProvider( null ).createRuleExecutionSet( inStream, null );
// instream is the input stream for the xml based rules file.
            RuleAdministrator ruleAdministrator = serviceProvider.getRuleAdministrator();
            RuleRuntime ruleRuntime = serviceProvider.getRuleRuntime();
String uri = res1.getName();
StatefulRuleSession statefulRuleSession =  (StatefulRuleSession) ruleRuntime.createRuleSession( uri, new HashMap(),RuleRuntime.STATEFUL_SESSION_TYPE );
statefulRuleSession.addObjects( input ); // input is ArrayList of facts
statefulRuleSession.executeRules();
List results = statefulRuleSession.getObjects();

I later modified this basic app and added a clickable list view in recursion which let me implement what I wanted – a rules based application on mobile platform with a guided selling UI J

This marked the final milestone of my journey, a journey that gave me a deep insight of what I would otherwise have always wondered about.

22 comments:

  1. I want to use JRuleEngine on android platform.

    Does beans.jar and awt.jar re-package simply using android jarjar utility??

    Could you upload jsr94.jar and JRuleEngine.jar for android??

    ReplyDelete
  2. Yes. You need to repackage those tow jars using jarjar. Standard jsr94.jar and jruleengine.jar need to be added to the eclipse project. Eclipse build / packager would automatically convert these to dalvik (android) jars.

    ReplyDelete
  3. Hi Abhay,

    I was trying to use JBPM on Android these past couple of weeks and came across your blog post yesterday. I found Kris Verlaenen's blog post which has an example of using the process api to construct a process and instantiate it, but found I ran into many of the same issues as you did when trying to load a process from a .bpmn file.

    There are a couple things I'd like to comment on:
    1) The IndexOutofBounds exceptions go away with
    System.setProperty("java.version", "1.6");
    2) I had to add the following as well or I got an error about no validating saxparser implementation available
    System.setProperty("drools.schema.validating", "false");
    3) Also, there is no mention of how to load a file in your post, I did it by creating an assets folder and adding my bpmn file to it. I loaded it like this:
    InputStream is = o.getResources().getAssets().open("sample.bpmn");
    where o is an instance of my activity (since I was loading the file in a static function).

    In the end, I could not get past all the NPEs and gave up, figuring I would have heap space problems as well. I also defined my own service task nodes in the drools definitions; I imagine that would be really painful to get working also.

    For anyone else contemplating using jbpm on Android, I suggest either implementing the process engine server side, using the process api, or another workflow engine entirely.

    My next attempt will be to construct my process using the process api. It's not ideal but I don't know what else I can do. I suppose it might be possible to parse a bpmn xml file and then use the process api to construct it, then I can load different processes. Luckily I only need a subset of bpmn.

    ReplyDelete
    Replies
    1. Hi Peter
      I did not try setting up jbpm on android. I did come across the Kris's blog post that you mentioned about. To fix the NPE, I did try setting the java version to 1.6, but I believe there were multiple places in the code where I had to add this line of code .. so I preferred the CompositeClassLoader constructor option.

      And yes, I used assets in my poc. The assets is the most appropriate place to store the XMLs in. Thank you for bringing that out.

      Delete
  4. I am stubborn and made some more progress :) Thanks Abhay for getting the ball rolling!

    I was able to load a BPMN file using the process api (also called Fluent, apparently). The code is very ugly and makes many assumptions (as mentioned, I only need a small subset of functionality) but it works. For anyone trying the same thing, there were only two gotchas:
    1) MVEL index out of bounds. I put the system property that the java version is 1.6, and that went away (see above).
    2) With version 5.2, constraints cause a classcastexception at runtime. I posted my fix on the JBoss community forum. https://community.jboss.org/message/629564

    For the next version, I'll probably start by extracting some of the source from the official bpmn xml loader, and cutting out the bits that don't play well with Android. See the following for example:
    https://github.com/droolsjbpm/jbpm/blob/master/jbpm-bpmn2/src/main/java/org/jbpm/bpmn2/xml/ProcessHandler.java

    ReplyDelete
  5. Thanks for posting updates here, Peter. Just wondering how further did you go on this one.

    ReplyDelete
  6. Peter, were you able to achieve what you had in mind ?

    ReplyDelete
  7. Hi Abbay, after following your steps in your post, I am having problems running jruleengine. As you advised, I repackaged beans.jar and awt.jar into a different package name, updated some jruleengine classes to reference the repackaged classes, and put the repackaged jar in the classpath. I also had to add rmi.jar in my classpath because jsr94 depeneded on it (i.e java.rmi.RemoteException). When I try to run example1.java from an Android Java Activity class, I get the error below.

    ....
    [2012-09-06 05:43:24 - RulesEnginePoc] Dx 1 error; aborting
    [2012-09-06 05:43:24 - RulesEnginePoc] Conversion to Dalvik format failed with error 1
    [2012-09-06 06:39:24 - RulesEnginePoc] Dx
    trouble processing "java/awt/AWTError.class":
    ...


    Did I miss something? Please assist me here

    ReplyDelete
  8. Hi Abhay, i think you did a great job trying to get drools to run on android. i am a student at the technical university in vienna and doing my master thesis about a context aware workflow engine on android. i would love to cite your blog entry but i think i would need your full name so that it is accepted by the university.

    Would be great if you send me your name, my email adress is harald.hofstaetter[ a t ]gmail.com. If you do not want to, i absolutely understand your decision.

    In return i would like to send you my thesis if you are interested. Thanks in advance. harald

    ReplyDelete
  9. I recently ported Esper - an open source, Complex Event Engine to Android.
    Simple benchmarks show 70.000 events per second and it is possible to re-configure it on the fly through push notifications from anywhere in the world. No rooting needed. Every dependency is resolved - by hand from the OpenJDK project.

    https://github.com/plingpling/Asper

    Hope this can be of interest to you.

    ReplyDelete
  10. please show me each step you have used. i really need this thing to work on mobile.

    Thanks

    ReplyDelete
  11. Awesome! Education is the extreme motivation that open the new doors of data and material. So we always need to study around the things and the new part of educations with that we are not mindful.
    python training institute in marathahalli
    python training institute in btm
    Python training course in Chennai

    ReplyDelete

  12. Thank you for such a sweet tutorial - all this time later, I've found it and love the end result. I appreciate the time you spent sharing your skills. devops training in chennai | devops training in anna nagar | devops training in omr | devops training in porur | devops training in tambaram | devops training in velachery

    ReplyDelete