How to glue JAVA & C++ together with SWIG & Type Maps
Anyline Engineering Team
Engineers at Anyline
Write a blog post they said. Talk about SWIG they said. Make it short they said.
Well, for those of you that don’t know, the latter two are mutually exclusive which results in a blog post of excess length. But, on the upside, you can just skip our gibberish and move to the essential code parts.
This blog post is all about SWIG (the Simplified Wrapper and Interface Generator). Unfortunately, for those of you that like pretty pictures, there won’t be any.
Now, having lost half of the readers with this previous sentence, if this blog post is not about images and computer vision, what is it about anyway, and why is it posted here? After all, this is supposed to be a computer vision blog.
SWIG & JNI
SWIG is (almost) the solution to problems that many companies and developers developing for more than one mobile platform face, namely how to hold a shared core code (C++ in this case), and access it from different host languages.
While it is fairly easy to access C++ code on iOS, doing the same from Android proves to be a more complicated task. Luckily, we are going to give an answer to the question: How to glue JAVA and C++ together?
- Create your own cross-platform OCR app / mobile scanning app with our free demo & OCR SDK trial!
We are going to do so by making use of the Java Native Interface (JNI), together with SWIG. SWIG wraps and connects C++ code via the JNI to Java code. (For those of you reading carefully, yes, this is a SEO sentence. It will be the only one. Promise!)
Something about SWIG
In general, SWIG needs a C++ input file (holding the code you want to be wrapped), and an interface file, telling SWIG how to wrap things. The output of SWIG is JAVA code, with which you are able to access C++ objects and functions.
The interface file consists of many directives, some more complicated, some less, some better documented, some less. You can find a lot of them here at SWIG.
Commanding SWIG
If you want to implement the example in this blog post yourself, you need to download and install the swig tool.
The SWIG command looks like this
swig -java -c++ -package io.anyline.swig.example -outdir src/io/anyline/swig/example -o wrap/swig_example_wrap.cpp swig_example.i
–java and -c++ mean that we want to wrap C++ code to JAVA
–package defines the resulting package name
–outdir the output directory for the JAVA files
–o the output file for the C++ wrapper code
and finally your interface file.
Note: SWIG is available for many programming languages. We will focus on JAVA for this blog post.
The Scenario
We will start with an easy example, providing you with the basics of
- How to wrap your classes
- Wrapping input and output parameters for functions
- How to bind the parameters to existing JAVA implementations
Our example scenario will be the following. Suppose we have a C++ core (called MySwigExample), which receives an image as an input (an OpenCV Mat), performs some operation, and returns a C++ result (conveniently called MySwigResult), holding a thresholded image (OpenCV Mat) and a detected region of interest (OpenCV Rect) in the image.
The best part is: we don’t have to care about any actual C++ implementation. There is none. All we need are the header files with the function declarations. Because from the JAVA point of view, we only need a mechanism to call the functions and retrieve the results.
Okay, let’s start with the C++ header of our example
(https://gist.github.com/anonymous/76973e1bb916f9eefad6ad60dade947d#file-gist-rbx)
For simplicity (some would call it laziness), the code is stripped to its pure essentials.
What we want to achieve is to create an empty MySwigResult object and a MySwigExample object, call the run function, and retrieve the cv::Mat and cv::Rect of the MySwigResult object. And, of course, we want to do all that from JAVA.
To tell SWIG what to do, we need the SWIG interface file. The first simple interface version, just defines a module name (which is used for the generated JNI class names), and tells SWIG to include our C++ header.
(https://gist.github.com/anonymous/4dbc2846cf45a31218c6aaccd0103e71#file-gist-rbx)
Running this with the swig command will produce six JAVA files. The two JAVA implementations MySwigExample.java and MySwigResult.java, as well as the wrapper code in swig_example.java (yep – empty) and swig_exampleJNI.java.
There are two JAVA files left which are called SWIGTYPE_p_cv__Mat.java and SWIGTYPE_p_cv__Rect.java. That is because SWIG does not know how to handle the OpenCV classes in JAVA, and just creates its own wrapper for them. But don’t worry, we’ll get there.
If you take a look inside the generated JAVA classes, you will see that they call methods in swig_exampleJNI, which are native methods. These methods actually call the C++ code, residing in wrap/swig_example_wrap.cpp. In there, the C++ objects are created, and a pointer to them is returned to the JAVA side. So all the created objects on the JAVA side are really just wrappers, not implementing any logic, but calling their respective counterparts on the C++ side.
Long story short, if we call a JAVA constructor for MySwigResult, a C++ MySwigResult object is created, and the pointer is stored in the JAVA wrapper.
Connect the OpenCV Classes with their JAVA Counterparts
So far so good, but it gets better. Now that we have our wrapped classes, it would be nice to connect the OpenCV classes that are used in the C++ part with their JAVA counterparts. One option would be to have SWIG wrap the C++ implementation of OpenCV by adding the respective header files to the SWIG interface file, but we don’t need all of the classes.
Fortunately, there is an elegant way to establish this connection.
First, let’s have a look at the JAVA implementation of Mat.
After taking a closer look, we discover that a Mat is actually also a C++ object, where the JAVA part is simply a wrapper, holding a public final long nativeObj. This is the same as swigCPtr field in our wrapped classes. So simply a pointer to the C++ object it wraps.
This means, that whenever we want to pass a Mat from JAVA to C++, or the other way around, we can simply pass this pointer because the object already exists in the C++ environment.
To tell SWIG how to do that, we use typemaps. A type map, the name kind of gives it away, is a directive telling SWIG how one type (in C++) should be mapped to another type (in JAVA).
And the following is the typemap for Mat:
(https://gist.github.com/anonymous/a7ff7589486e4d1dd50643f81519c521#file-gist-rbx)
The jstype typemap tells SWIG that we want to use org.opencv.core.Mat for the cv::Mat type.
The javain typemap tells SWIG how to pass the JAVA object to its intermediary JNI class. In this case, we call the getNativeObjAddr() function on the JAVA Mat, to obtain the pointer address for the C++ cv::Mat and pass it along.
Next up, the jtype typemap tells SWIG that a long will be passed to the swig_exampleJNI.java. This long is the pointer we obtained in the previous step.
This long becomes a jlong (the corresponding JNI native type – just don’t mind, we think of it as a long) in the jni typemap when passed on to the SWIG C++ function (located in wrap/swig_example_wrap.cpp).
And finally, we have to tell SWIG how to create the cv::Mat in the C++ function out of the pointer we just passed along. That is done via the in typemap, where we simply tell SWIG to cast that pointer to a cv::Mat.
And… done! We are now able to pass a Mat from JAVA to C++. This is it. All the “magic”. We’re aware of the fact that this is not easy to understand when you try it the first time. But give it some time. Get a coffee and think about it for a day or two. Trust us, it will make sense after a while!
To make it even better, passing a cv::Mat back to JAVA is a piece of cake! Since we already have all the jstype, jtype and jni typemaps, all we have to do is tell SWIG how to handle the pointer that is returned from the swig_exampleJNI.java methods. And OpenCV does us a real favour there, since we can simply call the constructor of the JAVA Mat and pass the pointer.
Done. If you told your colleagues that it would take quite some time to create a wrapper to pass a Mat from C++ to JAVA, you just earned a lot of free time with this one-liner.
If you add the typemaps to the interface file (be aware to add it before the %include statement – otherwise it will be useless), and run the swig command again, you will see that the SWIGTYPE_p_cv__Mat.java is gone and the JAVA parameters are replaced by a neat org.opencv.core.Mat.
But, quite frankly, OpenCV made it easy for us to create the Mat typemap. We will now create typemaps for the OpenCV Rect, which does not have a pointer to C++ anywhere near its code.
The Rect (a working title heading)
By now, you have studied the JAVA code of a OpenCV Rect to find a loophole and be able to skip this part of the blog post. If you found one, great, and please tell us about it. If you didn’t, I am afraid you are stuck with me here. Don’t worry, I promise it gets “fun” (as fun as wrapping a cv::Rect from C++ to JAVA can be…)
What we are going to do is, unlike in the previous part about Mat, copy the cv::Rect from C++ to JAVA, rather than sending pointers to the object around. This also means, that the wrapped Rect in JAVA will not be linked to the C++ object. So if the C++ object is modified, the JAVA version will not know anything about it, and live happily in its JAVA world, not caring about this C++ everyone keeps talking about.
So let’s have a look at the corresponding typemaps:
(https://gist.github.com/anonymous/e50639738d97d4e6b1cd551cb3ce1fc6#file-gist-rbx)
The C++ part of the JNI (swig_example_wrap.cpp) is going to return a jobject this time, instead of a pointer.
What we need to do now, is tell SWIG how to create that JAVA object within its C++ code.
The answer to that lies within the out typemap. Simply put, we ask the JNI environment jenv for the JAVA class of the Rect, ask what the constructor is, and call the constructor with the x, y, width and height parameters of our C++ cv::Rect. The so obtained object is then simply returned, and we just pass it on in the javaout typemap.
Note: The JNIEnv is provided as a parameter to all JNI C++ functions. It is a struct holding pointers to all JNI functions. Feel free to read more about it here (https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html)
Note2: Yes, this code does not include any security checks. So if a method does not exist, you might run into a problem – see “laziness” before.
Run the swig command again, and you will see that the getRegionOfInterest() method now returns a org.opencv.core.Rect. Tada!
But, the setter for regionOfInterest and the constructor still require the swig generated SWIGTYPE_p_cv__Rect as parameters. That is because we didn’t define the in typemap in our interface file.
We could either write the in typemap, or just ignore these methods. We decided democratically that we are going to ignore them.
Ignoring Functions
Well, why can we ignore these methods anyway?
First, the main part of our program runs in C++, so we only need the JAVA side to be able to access the results of the program. We are never going to set any parameter on MySwigResult from JAVA. Neither are we going to construct a MySwigResult object with a thresholded image and a region of interest in the constructor – simply because you should take care of that part in C++.
So much for the why part, now we’ll look at the how part. And that is easy as well:
Thankfully the SWIG guys were pretty straight forward regarding their naming conventions. %ignore followed by the class and the function signature tells SWIG skip this functions when creating the wrapper code.
(https://gist.github.com/anonymous/410223fd3507e70c2f787fb98a49310a#file-gist-rbx)
Run the swig command again, and you will see that all the auto-generated SWIG classes for our OpenCV objects are gone.
Final Steps or “Please Wrap It Up Already, I’m Tired”
That’s the last part, then we are done. You might have realized by now that the generated wrappers for MySwigExample and MySwigResult lack the import statements for the OpenCV classes. You could just add them manually, but once you run the swig command again, the changes would be lost inevitably. There might be some of you saying now “I’m going to remember that, don’t worry. Bye!”
For all others – and the ones that read the comment at the top of the SWIG generated classes – there is a simple SWIG directive that adds the imports to your generated JAVA classes. Believe it or not, the naming convention is straightforward:
(https://gist.github.com/anonymous/f557c518ef10e75dcb2812e7ff5881d7#file-gist-rbx)
This adds the import statements to the beginning of your generated JAVA classes. Actually there is nothing more to say about this. And that also concludes our first blog post about SWIG and its typemaps.
Summary: SWIG is a Real Benefit in Many Situations
SWIG is pretty powerful if you know how to use it, and what to use when, and respect the sequence of your directives, and so on. And there are a couple of real experts on SWIG out there (looking in your direction, Flexo from stack overflow!) and this is just a simple example, trying to give you an introduction to SWIG and how to benefit from it.
But despite its complexity, SWIG can be a real benefit in many situations. If you look at the code that SWIG created, you will get the point. Imagine to write and maintain this code yourself.
