An Erlang-Java Interop Demo

Erlang is designed and used for distributed systems. As such, it is quite good at talking to itself. And as I'll show in the following, it is — at least in some cases — reasonably good at talking to other languages as well.

When would you want to do this?

(You can skip this section if you know.)

Interoperability between languages is useful when there's value to be had in the software in both ends. You might prefer to keep some functionality in Erlang because it's already written in that language; because it has better libraries for the task; or simply because the language supports that task better. The same goes for the other language: there are C libraries around for almost anything (and plenty of C/C++ legacy code), and for GUIs you might prefer e.g Java.

Or you have the same functionality implemented in both languages, and wish to do comparison tests on the two implementations.

Or the architecture is inherently distributed — e.g. a client-server configuration — so that there is no advantage to be had in building the two parts using the same language anyway.

In the following demonstration, we have a Java side with some freshly written software with interesting bugs, and an Erlang side with an interesting tool for discovering bugs and a nice-ish language for specifying tests.

The demo

What I intend to do:

• Define an interface;
• Write a Java function;
• Make it accessible from Erlang;
• Test it using Erlang.

Sounds like a lot of work? It needn't be. Not for you, anyway; most of the heavy lifting has already been done.

(If you don't care about exposition, and are just interested in the technical end result, here is the executable summary.)

An interface

First, we'll need to define an interface. For this, we use the IDL — Interface Description Language:

// File "foo.idl"
interface Foo {
  string quuxinate(in string s);
};

We translate this into Java using the Erlang compiler:

erlc '+{be,java}' foo.idl

This results in a Java interface in Foo.java, as well as some stub code we'll be using.

A function in Java

For an implementation, let's say that quuxinate() is to reverse the string. To add a twist, let's say that it breaks down under some rare, complex-ish circumstances, such as when there's a letter which occurs thrice in the input string — that'll be a bug we can find later, when we start testing it:

// File "FooImpl.java"
public class FooImpl extends _FooImplBase /* which implements Foo */ {
  public String quuxinate(String s) {
    int[] stats = new int[256];
    for (int i=0; i<s.length(); i++) {
      char c =  s.charAt(i);
      if (c<255 && ++stats[c] >= 3) throw new RuntimeException("WTF");
    }
    return new StringBuilder(s).reverse().toString();
  }
}

Making the function accessible from Erlang

You may have noticed that we don't implement Foo directly, but rather extend an stub class which implements it. This means that the class already knows how to talk to Erlang — specifically, it knows how to handle requests like a gen_server.

What is left is to establish connection to an Erlang node. There are two frameworks for this: CORBA and Erlang inter-node communication. CORBA is, I believe, the heavy-weight option, and I haven't worked with it; in the following, we'll use Erlang inter-node communication.

A runnning Erlang program (OS-level process) is often referred to as a 'node', because Erlang is designed for having multiple such programs be connected, in what is known as a 'cluster' of nodes.

It's not an exclusive club, either: nodes can be, and usually are, Erlang nodes, but other kinds can enter the mix — a C or Java node, for instance.

A bit of code is needed to run a Java program as a node; don't worry though, this is the only lengthy bit, and it can be generalized so that it doesn't have to refer to Foo:

// File "FooServer.java"
public class FooServer {
  // The following is based on the program in lib/ic/examples/java-client-server/server.java
  static java.lang.String snode = "javaserver";
  static java.lang.String cookie = "xyz";

  public static void main(String[] args) throws java.io.IOException, com.ericsson.otp.erlang.OtpAuthException {

    com.ericsson.otp.erlang.OtpServer self = new com.ericsson.otp.erlang.OtpServer(snode, cookie);

    System.err.print("Registering with EPMD...");
    boolean res = self.publishPort();
    if (!res) throw new RuntimeException("Node name was already taken.");
    System.err.println("done");

    do {
      try {
        com.ericsson.otp.erlang.OtpConnection connection = self.accept();
        System.err.println("Incoming connection.");
        try {
          handleConnection(connection);
        } catch (Exception e) {
          System.err.println("Server terminated: "+e);
        } finally {
          connection.close();
          System.err.println("Connection terminated.");
        }
      } catch (Exception e) {
        System.err.println("Error accepting connection: "+e);
      }
    } while (true);
  }

  static void handleConnection(com.ericsson.otp.erlang.OtpConnection connection) throws Exception {
    while (connection.isConnected() == true) {
      FooImpl srv = new FooImpl();
      com.ericsson.otp.erlang.OtpInputStream request= connection.receiveBuf();
      try {
        com.ericsson.otp.erlang.OtpOutputStream reply = srv.invoke(request);
        if (reply != null) {
          connection.sendBuf(srv.__getCallerPid(), reply);
        }
      } catch (Exception e) {
        System.err.println("Server exception: "+e);
        e.printStackTrace(System.err);
        handleException(e, connection, null);
      }
    }
  }

  static void handleException(Exception e, com.ericsson.otp.erlang.OtpConnection connection, com.ericsson.otp.ic.Environment env) throws Exception {
    // We'll improve on this later...
    throw e;
  }
}

Time to build and try out:

ERL_ROOT=`erl -noshell -eval 'io:format("~s\n", [code:root_dir()]), init:stop().'` # Or simply where Erlang is.
IC_JAR=`ls -1 $ERL_ROOT/lib/ic-*/priv/ic.jar`
JI_JAR=`ls -1 $ERL_ROOT/lib/jinterface-*/priv/OtpErlang.jar`
CLASSPATH=".:$IC_JAR:$JI_JAR"
javac -classpath "$CLASSPATH" *.java

epmd # Start Erlang port mapper daemon if it isn't already.
java -classpath "$CLASSPATH" FooServer

The Java node should now be running and registered as javaserver — ready to accept connections with the right cookie.

Let's test that:

erl -sname tester -setcookie xyz
> {ok,Host}=inet:gethostname().
> JavaServer = {dummy, list_to_atom("javaserver@"++Host)}.
> gen_server:call(JavaServer, {quuxinate, "Testing, 1-2-3"}).

The reply should be the reverse string:

"3-2-1 ,gnitseT"

And the bug we put there is working too:

> gen_server:call(JavaServer, {quuxinate, "Testing, 1 2 3"}).
** exception exit: {{nodedown,javaserver@flitwick},
                    {gen_server,call,
                                [{dummy,javaserver@flitwick},
                                 {quuxinate,"Testing, 1 2 3"}]}}
     in function  gen_server:call/2

Property testing of Java code

Now then, what can we do with this setup?

One interesting thing that we can do is to apply one of the property-based testing tools to our Java function. In the following, I'll be using Triq, but Quviq QuickCheck or PropEr could be substituted with only minor changes. First, assuming that you haven't got Triq, but have git:

git clone git://github.com/krestenkrab/triq.git
(cd triq && ./rebar compile)

Then we are ready to write our test — namely, that given any (ASCII) string, quuxinate() should return the reverse string:

// File "test.erl"
-module(test).
-include_lib("triq/include/triq.hrl").
-export([main/0]).

prop_reverse(JavaServer) ->                  % The property
  ?FORALL(S, ascii_string(),
      gen_server:call(JavaServer, {quuxinate, S})
      == lists:reverse(S)).

ascii_string() ->                            % A data generator
  list(choose(0,127)).

main() ->
  {ok,Host}=inet:gethostname(),
  JavaServer = {dummy, list_to_atom("javaserver@"++Host)},

  triq:check(prop_reverse(JavaServer), 100), % Do the magic
  init:stop().                               % Shut down cleanly

Compile and run the test:

erlc -I triq/include -pa triq/ebin test.erl

erl -noshell -sname tester -setcookie xyz -pa triq/ebin -run test main

Triq will now generate a hundred random test cases, and verify the property for each case. After a dozen such tests, the bug is triggered:

...........Failed with: {exit,
                 {{nodedown,javaserver@flitwick},
                  {gen_server,call,
                      [{dummy,javaserver@flitwick},
                       {quuxinate,
                           [79,84,75,110,3,42,73,14,1,53,76,42,126,40,118,122,
                            74,2,58,34,42,98]}]}},
                 [{gen_server,call,2},
                  {test,'-prop_reverse/1-fun-0-',2},
                  {triq,check_input,4},
                  {triq,check_forall,6},
                  {triq,check,3},
                  {test,main,0},
                  {init,start_it,1},
                  {init,start_em,1}]}

Failed after 12 tests with {'EXIT',
                            {{nodedown,javaserver@flitwick},
                             {gen_server,call,
                              [{dummy,javaserver@flitwick},
                               {quuxinate,
                                [79,84,75,110,3,42,73,14,1,53,76,42,126,40,
                                 118,122,74,2,58,34,42,98]}]}}}

The 22-character string has three '*'s (ascii value 42) in it, which was what triggered the bug.

Triq then proceeds to simplify the test case:

Simplified:
        S = [33,33,33]

concluding that the string "!!!" is a locally-minimal failing test case.

Quite useful, isn't it? — and the amount of non-reusable code has been quite manageable (3 lines of IDL, 14 lines of implementation, 14 lines of test code, 1 line of Foo-specific code in FooServer).
(Speaking of which: you're free to use these snippets as you see fit; provided as-is and with no guarantees of anything, of course.)

I should mention at this point that property testing tools exist within the Java world as well — there exists at least one for Scala. I didn't find it particularly satifying to use, though, although I may have been unlucky; in any case, this is an alternative.

Erlang, IDL and exceptions

The Guide (that is, the Erlang IC (IDL compiler) User Guide) has this to say about handling of Java exceptions:

While exception mapping is not implemented, the stubs will generate some Java exceptions in case of operation failure. No exceptions are propagated through the communication.

Which means that, out of the box, these are our options for handling Java-side exceptions:

1. Convert exceptions into values within quuxinate().
2. Return no result on exceptions — in which case the Erlang-side call will time out (after, as a default, 5 seconds).
3. Close the connection — in which case the Erlang-side call will receive an error result immediately.

None of these options are especially satisfying.

So let's look at how to do this:

4. Report exceptions to the caller as an {error, Reason} reply.

It's not too difficult. What we need to do (according to the gen_server call protocol) is send a {RequestRef, Reply} tuple to the caller, where RequestRef is a reference which was included in the request and must be included in the response as well.

The main difficulty is one of access: at the point of error handling, we will need access to (1) the caller's PID and (2) the request reference. I'd like to keep FooImpl and the connection-managing FooServer separate, so we need to add an accessor:

// File "FooImpl.java"
public class FooImpl extends _FooImplBase /* which implements Foo */ {
  ...
  /** The request environment is exposed for error handling reasons. */
  public com.ericsson.otp.ic.Environment getEnv() {
     return _env;
  }
}

That's it. We could instead have added one accessor for the caller PID and one for the request reference, but let's keep the complexity in the server class.

In the server class, instead of throwing an exception which causes the connection to be terminated, we will build and send an error reply:

// File "FooServer.java"
public class FooServer {
  ...
// in handleConnection(), replace
        // handleException(e, connection, null);
// with:
        handleException(e, connection, srv.getEnv());
  ...
// and replace handleException() with:
  static void handleException(Exception e, com.ericsson.otp.erlang.OtpConnection connection, com.ericsson.otp.ic.Environment env) throws Exception {
    // Write exception reply:
    com.ericsson.otp.erlang.OtpOutputStream err_reply = new com.ericsson.otp.erlang.OtpOutputStream();
    err_reply.write_tuple_head(2);
    err_reply.write_any(env.getSref());
    err_reply.write_tuple_head(2);  // Construct return value {error, ErrorText}
    err_reply.write_atom("error");
    err_reply.write_string(e.toString());
    connection.sendBuf(env.getScaller(), err_reply);
  }
}

There; that's all.

If we test it again, we get a nicer behaviour:

> gen_server:call(JavaServer, {quuxinate, "Testing, 1 2 3"}).
{error,"java.lang.RuntimeException: WTF"}

Disclaimer

I ought to mention that I have only just discovered this interoperability option (two days ago, in fact, when I stumbled upon this article); it is only a few months ago that I wrote a custom socket server test program in Java, and corresponding driver code in Erlang, just to achieve what could be had far more easily using Erlang's IDL compiler. It may therefore not be so easy to use in practice as the above make it seem. (For one thing, the only type that I've used is string.)

Even so, I hope to have inspired others to try this out.
One way to do so is to have a look at the executable summary referred to earlier; it is the demo as a shell script.

Happy inter-language hacking!

---

Further technical notes (and speculation)

The IDL compiler supports generating code for Java, C and Erlang. Here is its documentation.

If you're writing Erlang port programs (like an Erlang driver, but running in a separate process), but have an interface which is evolving quicḱly enough, or you yourself are lazy enough, that writing and maiintaining the serialization and deserialization code has quite lost its attraction, then it might be tempting to look into using IDL for that interface, and have the boring bits code-generated for you.

The IDL compiler and the code it generates does not seem to have been designed for this, but as far as I can tell it appears to be achieveable.

Testing distributed-store algorithms

This is a follow-up to my post on a datastructure for storing collections in Riak.
While I have been planning this follow-up, Kresten Krab Thorup has actually gone ahead and implemented the algorithm (on GitHub; see src/riak_column.erl) — which suites me nicely, as I haven't written a single line of implementation and am not really past the thinking stage yet :-)
To wit:

Shortly after writing the post, I discovered a couple of issues or finer points, both of which have to do with the fact that different rows in Riak live independent lives —  they are independently versioned, and no cross-row guarantees are given — specifically, nothing can be concluded from the read order: if one client A updates row X, then row Y, then another client B may see the Y update and later see the old version of row X.

(If Riak is set up with appropriate consistency settings, such that the majority of the replicas of an object are written synchronously, then you do have indirectly some sort of guarantee. Below a certain threshold of machine and/or network problems, that is.
But even so, nothing can be concluded from the read order if there several clusters set up with cluster-to-cluster replication: changes on different keys in one cluster arrive at the other clusters in arbitrary order.)

This leads to at least the following issues:

1. Firstly, when splitting a row into two, I included this step: "Write an empty row back under the original auxiliary row key".
   In fact, doing that would be wrong. Even writing some tombstone dummy value in the old row would be a bad idea; instead, one should simply mark the row as obsolete while keeping the old data. This is necessary because a client accessing the collection later on may see from the master row that the old row has been split, but not see the new rows. Or it may see the old row, but not the change in the main row. In the former case, it is necessary that at least the values in the old row be available (or the collection would suddenly have shrunk); in the latter case, it would not be apparent that the values in the old row are obsolete.
   
   
2. Secondly, after modifying an auxiliary row, the main row should be updated — even nothing in it has changed. This may sound silly, but is the easiest way to ensure that, in the event of a concurrent row split, the auxiliary row in question is taken into account at a subsequent merge (indeed, that the merge is triggered at all).
   
   
3. And even that is not enough if there are more than one cluster: At the time of the read repair of the main row, the update for the auxiliary row may not have arrived - and it may therefore be lost silently. It appears that some kind of versioning of the auxiliary rows is necessary; with such versioning, we can tell when repairing the main row that we're missing an update on the pre-split auxiliary row, and that the reference to it should therefore be kept in the main row, so that it can be repaired later when all information is available.

Ah, the perils and challenges of incomplete information.

Managing concurrent subtlety
As the saying goes,

Meddle not in the affairs of concurrent systems for their ways are subtle, and are quick to anger.
-- Tolkie(error: timeout)n

In a domain as subtle as this — with gotchas in the style of the above mentioned issues, how can we ever convince ourselves that a scheme like the one for Riak collections are implemented robustly and correctly?

As always, there are two ways: Formal verification — proving that the program is correct; and thorough testing — gaining confidence by exercising the program.
Both have their advantages and disadvantages.

If I were Dijkstra, I'd develop a formal proof (as he did in quite a few of his blog postings), and probably modify the algorithm appropriately along the way. I, however, am no Dijkstra, do not believe to have the time to develop a formal proof — nor do I have any delusion of being able to write anything worth reading in the process. Luckily, going the other way, through testing, have something going for it as well:

• It is not always clear which guarantees the components we build on actually provide. That mean that, regarding formal proof, the set of axioms may be uncertain.
• The same test can be applied to different implementations of a component.
• The same test can be applied to different versions of a component — i.e. if we modify the code, we can cheaply gain some confidence in the modified version.

For both formal verification and testing goes that the answers you get depend on the questions you ask.
When proving a property, you prove only that property; when you test a concrete call sequence with concrete values, you test with only those values.
Which is of course a good reason to get familiar with property-based testing — which lies somewhere in between the two in that it tests a property on a number of concrete call sequences — typically a few hundred, and typically different instances for each time a test is run.

This can be a nice compromise between formal verification and hand-rolled test cases — provided of course that the properties and the instance generators are chosen well.
As always with testing, paranoia and imagination is key. But you get more value for your paranoia when it's used to power random test case generation.

For the problem in question, and assuming an Erlang implementation like KKT's, property testing can be done with relative ease using a tool like Quviq QuickCheck or Proper. Their support for testing state machines comes in handy; I've not tried using it before, but this seems a good occasion.

How to test randomly
Randomized testing involves abstracting over usage scenarios.
How the abstraction is done determines what will be tested.
Knowledge of the problem domain is the primary guide; paranoia together with perhaps knowledge of implementation details should provide additional input.
What must be considered is:

• What is tested:
  Which API functions to exercise?
  Which invariants to verify?
• How it is tested:
  Test concurrent use to check for thread safety?
  Test with invalid inputs?
  Should certain special circumstances be simulated - e.g., file I/O errors, disk-full conditions, network delays, packet loss?
• With what it is tested:
  Test with abnormal (very long/short/high/low) inputs?
  Key collisions?
  Many values for the same key?
  Keys which are nearly identical?

This is of course where the imagination and paranoia enter the picture.
You can't assume that "since we generate the values randomly over a large domain, we exercise all code paths" — exercising e.g. a dictionary with a million randomly generated keys is of little use if the keys never collide.

How to test distributed collections
OK then, how to test an implementation of the distributed collections scheme?
Using the state machine testing support of Proper, we can maintain a "model" collection aside the subject-under-scrutiny collection (shortened to SUS in the following).

What to test:
This is the easiest part: We will test the usual collection functions: insert, delete, lookup, list keys.
The invariant is that the result is consistent with the same operation performed on the model collection — as defined below.

How to test:
We'll certainly need to test concurrent use. Let's say that there are three simultaneous users of the collection; we know that there are subtleties involving two, and there may be additional ones involving at least three.
The users won't be really simultaneous, though — We will test concurrent use in a way where the concurrency is made explicit. This provides repeatability and insight into why things fail, which is indispensable for this problem domain.
So, instead of having an actual underlying Riak store, we'll mock it up, in such a way that the mock provides just the guarantees we actually expect the real thing to provide.


Model representation
The structure I have in mind is: the mock remembers all versions of all values put into it. This is what the entire simulated multi-cluster store — let's call it the "cloud" — has ever seen.
The mock furthermore has a number of "views" of the store, corresponding to what you would see if you accessed the cloud at different points. It has a number of these; we'll make the assumption that in any of these views, for any given key, the version will only increase with time. I.e. we do not expect the version of any value in any view to evolve backwards. The mock keeps track of the view-to-version mapping for all keys.

One thing that can happen, then, beside operations on collections, is that value versions find their way from one view to another. Also, version merging can happen in transit — the versions being merged are not necessarily the latest value in any view, but may be any versions that have ever existed in the cloud.
For simplicity, let's say that this is modelled with views also — if we allow enough views, this won't reduce the generality.

The model thus consists, not just of a simple collection, but of an abstract model of the storage cloud.
Beside the externally visible events — the calls to the collection API — there are internal events: a given version of a given key finds its way to one view to another. Both kind of events go into the randomly generated test scenario specification.

The test
It was originally my intention to end with putting all of this together in a Proper-based Erlang unit test — it would then be a good occasion for me to get experience with the state-machine modelling support (link: state machine part is  from p.26).
However, it appears that I have a latency-vs.-completeness tradeoff to make, so I'd better stop here, publish what I have, and hope to return to the subject soon, hopefully with some concrete code. (While this posting has been under way, naturally other topics have come up which I'd like to write about; the order in which further postings arrive here is undetermined...)