Mobile Development in .NET – Part 2 of 2147483648

Consuming OpenCV in a Windows Phone App

I’ve had my troubles getting OpenCV to work properly in our Anyline Windows SDK and decided to write a little blog post about the issues and how I solved them. More precisely, I’m talking about our use-case where OpenCV is consumed via WinRT in a C# Class Library, which is then consumed in a Windows Phone App. Hopefully, this will help someone struggling with similar problems.

To keep an overview of the scenario I’m going to describe, let’s picture the following project structure:

1. C++ OpenCV (and other 3rd party libraries)
2. C++/CX Library as Windows Runtime Component
3. C# Class Library
4. C# Windows Phone App (WP 8.1 with WinRT)

Building OpenCV for Winrt

Depending on which OpenCV modules you use, OpenCV depends on 3rd party libraries like libjpeg, libpng, zlib, etc.  – which should be statically linked into the library. Now there are GitHub repositories from MS OpenTech and Itseez that provide CMake scripts for Windows Store Apps and Windows Phone Apps so you can create and build the whole OpenCV solution for the platform/version/architecture of your choice by following this guide.

By default, most of these OpenCV modules are built as a dynamic link library (the equivalent of a shared library in linux or macOS). The dynamic library consists of a .lib and a .dll file (plus other files like .pdb for program database or .exp for linker inputs):

• .lib contains the methods as entry points that are exposed.
• .dll contains the actual implementation and non-exposable methods

You don’t actually need the .lib file if you choose to link at run-time instead of load-time and know the (probably mangled) function names – find out more in this MS article if you’re curious.

Let’s have a look at the Windows Runtime Component (C++/CX) layer:

When properly including the OpenCV header files and telling the linker where the .lib files reside, the Runtime Component builds successfully. When wrapping OpenCV functionality in C++/CX via “public ref” wrappers, it enables us to communicate between C# (and any other .NET language) and OpenCV.

Ok, so everything builds. But when debugging the Windows Phone App and (indirectly) calling code the first time from the OpenCV library, the following exception is thrown:

The specified module could not be found. (Exception from HRESULT: 0x8007007E)

The problem with this error is, that you don’t know which module can’t be found. To find out where this error is coming from, you have to find out which module is actually loaded for the first time at run-time. In my case, there were a bunch of 3rd party libraries involved and it could be any of them because there was only one “call” visible from the C++ layer. So what I did is: I deployed the .dll files as content onto the Windows Phone and voilá – it worked. So the “module” had to be one of the .dlls I copied. But that’s not a very elegant solution. You shouldn’t expose those libraries “above” the C# Class Library layer – no one should need to include some unfamiliar 3rd party .dll files in order to use the Windows SDK. Also, what if someone would already use OpenCV as part of their app?

So the first workaround was to at least embed the library files into the SDK assembly and extract them at run-time before loading them. Not very elegant either, but it did the trick, and it “hid” the files from the user.

But there were some other issues that occurred and held me up again: Consuming OpenCV through the Class Library via WinRT in a Windows Phone App in this way fails to pass Windows Store App Certification. There were a bunch of errors that looked like this:

This API is not supported for this application type -
Api=?setTo@Mat@cv@@QAAAAV12@ABV_InputArray@2@0@Z.
Module=opencv_core300.dll. File=AnylineWinRT.dll.
This API is not
supported for this application type -
Api=?sum@cv@@YA?AV?$Scalar_@N@1@ABV_InputArray@1@@Z.
Module=opencv_core300.dll. File=AnylineWinRT.dll.
This API is not
supported for this application type -
Api=?zeros@Mat@cv@@SA?AVMatExpr@2@HHH@Z. Module=opencv_core300.dll.
File=AnylineWinRT.dll.
...


You can see the mangled function names, the module in which the code is found (opencv_core300.dll), and the module that calls them (AnylineWinRT.dll). It took a while to figure out what the actual cause was and how to fix it, and for this case, it meant that I could not include OpenCV as a dynamic link library because the certification test figured that the assembly would make external API calls that are not explicitly supported. This makes sense for security reasons. So the fix for this was: Go back to step 1 and build every 3rd library statically and configuring the linkers so the same dependencies would not be linked in multiple times.

After many coffees and probably a few dozen Stackoverflow / MSDN threads, I managed to compile and build everything statically and voilá – the Store Certification for the App passed.. well, almost. There were still some unsupported API errors, but those were not regarding the linker setup and it turned out that I flipped some #ifdef switch in a 3rd party library that enabled methods like SetFileAttributesA or DeleteFileA. Those errors occurred because of other 3rd party libraries that had some wrong compiler/linker flags set. I fixed those errors by indirectly correcting the #ifdef directives through the library settings respectively.

Conclusion

The lessons learned are: If you require 3rd party libraries within your own Windows Phone library, link them statically. Yes, it will make your own library bigger but it might save you from a lot of trouble. Don’t mess around with preprocessor directives unless you know what you’re doing – most issues relate to the project configuration directly. Also, finding bugs that are related to linking can be a pain, and fixing them can cost a lot of time. Always test your Windows App with the App Certification Kit early enough. This concludes today’s blog post about my issues with OpenCV on Windows Phone.

Try our Demo App