Posts

Image Storage and Indexing for Machine Vision Images

Every Software Engineer needs a hobby – to this end I have been toying with an idea for the last while.

There are many machine vision and computer vision applications that capture images from cameras and store them on disk. These applications can generate so many images that working with them can be quite difficult. For example consider an application that acquires from two cameras each acquiring at 30 frames per second – this application will save 216K images per hour, a 5 hour run would generate 1 million images!

Very often the images will be stored on a file system (local or networked) in some sort of hierarchical directory structure. Using a file system is a very efficient way of storing images, database systems (Relational or NoSQL) don’t offer many advantages and indeed can have associated disadvantages.

But how can we effectively work with so many images, we have possibly millions of images sitting in a set of directories, how can we interact with them and efficiently and query them based on attributes that are interest to us so the we can perform more analysis?

For example consider this set of (contrived) image queries:

Give me all of the images:

+ from camera 1
+ from camera 1 acquired on Sunday between 13:00 and 13:10
+ whose file size > 1MB
+ acquired within 100 meters of this GPS location
+ that have an average brightness > 63 Grey levels

Some people have attacked this image query problem by using a relational database to store image meta-data, if designed well this can allow for efficient image retrieval, however it seems to me that a schema-less approach is a better fit for images with dynamic attributes and I like the idea of not being tied down to any particular database technology and all of the baggage that comes with it.

So my idea is to start out on the road of implementing (for fun) a simple image indexing system for rather large sets of images, it will have an associated tool set, API and maybe even a query language in the future.

The system will:

Allow indexing of large numbers of images in arbitrary hierarchical directory structures

Index images based on standard attributes such as:

+ Acquisition Date/Time
+ Name
+ Source (e.g. camera)
+ Type
+ Size
+ Bit Depth
+ Exif Data, e.g.:
–> Location
–> Author
–> Acquisition parameters (aperture, exposure time etc.)
+ Etc.

Index images optionally based on Computer Vision metrics, e.g.
+ Brightness
+ Sharpness
+ Etc.

Allow users to define their own attributes for indexing, e.g.:
+ Define image attributes based on an OpenCV algorithm
+ Define attributes based on the contents of the image fie name.

The system will:

+ Have no dependencies on technologies such as Database systems etc.
+ Be cross platform

To get the ball rolling and so that we can say the first sod has been turned, here is some (naive) python which scans a directory tree of images and creates a flat CSV file of the image name, path and size:

Run it like this:

Once the directory tree has been walked and the CSV file generated we can use the following script to query images:

This allows us to run queries like this:

This will quickly list the images whose file size > 76000 bytes and whose name contains ‘_43’

This is a really simple first step but it does demonstrate how even a flat ‘index’ of attributes can be of great use.

Next Step:

+ Add more image attributes to the CSV file

A Machine Vision Engineer’s take on Fifty Shades of Grey by James Mahon

A Machine Vision Engineer’s take on Fifty Shades of Grey by James Mahon (generated by pure C source code):

50_shades_of_grey_machine_vision

And the source code:

Computer Vision in the Cloud? – Amazon makes Nvidia GPUs available

I just came across this article from The Register that mentions that Amazon is making Nvidia GPU cores available:

http://www.theregister.co.uk/2013/11/05/aws_gpu_grid_service/

Makes me think again about the possibilities of doing Computer Vision / Machine Vision in the cloud (taking advantage of GPGPU techniques via OpenCL etc.), however like most of Amazon’s stuff it would probably be way too expensive, interesting though – I must look into it if I ever get the time…

Machine Vision on the Raspberry Pi anybody?

Here’s is a blog post that has some pictures of the (hopefully) soon to arrive camera for the Pi.

 

http://www.raspberrypi.org/archives/3224

 

It will be interesting to see what cone be done from a machine vision or computer vision point of view with the Pi once cameras are available.

 

Python & Web2py for Smart Camera user interface?

I am currently in the planning phase of the software build for a new machine vision smart camera. The camera will have an ARM soft core and will run linux. Part of the planning involves thinking about a user interface to control and configure the camera – at the moment I am leaning towards an HTML5 based interface built using python, web2py (or flask) along with a twitter bootstrap based theme.

 

I did briefly toy with the idea of bringing node.js into the mix, but I just can’t bring myself around to the idea, I have a love hate relationship with javascript, that sometimes leans more to the ‘hate’ end of the spectrum!

 

Now here’s the fun bit – I don’t yet have my hands on the camera hardware so I think I will try to run up a proof of concept on my raspberry-pi!

 

Although the pi will be a good bit less powerful than the smart cam it should provide a useful reference point…

 

Put this little fella to work!

 

Using OpenCV from .NET C++/CLI

I am happy to report that I have had a reasonably easy run of using OpenCV from a .NET C++/CLI project, I was initially worried that the two might not play well together, but so far – so good! I am wrapping some machine vision code that uses OpenCV in a C++/CLI assembly so that I can call it from C# code. I have kept the rather creepy CLI stuff to a minimum it just implements the interface, everything else is vanilla C++ using the standard library.

 

The only thing I had to do to get rid of build errors was to make sure that each _proper_ C++ file in the project had the following value for the ‘Common Language RunTime Support’ build setting -> ‘No Common Language RunTime Support’!

 

So if you are toying with the idea of developing some OpenCV based software from within C++/CLI, I say give it twirl it makes interacting with .NET much easier than (say) using COM or similar!

 

Auxiliary Display with Matrox Solios & Millenium P650

I ran into a problem while attempting to get an Auxiliary display to work on a Machine Vision system I was developing with a Matrox Solios Frame Grabber and a (dual head) Matrox Millenium P650 graphics card.  The plan was to have each vision station (in a Parmacutical AOI setup) output all of the images it acquired from its cameras to a massive 15 input digital video recorder (DVR) via its Auxiliary display port – in this way the client would have a record of all image acquired and processed on each of their 11 inspection stations   Anyway, while trying to initialise the display from code via a call to:

Matrix Mil always reported that no system capable of supporting an Auxiliary display could be found – no matter how I changed the millions of possible settings, the auxiliary display would always fail to initialise – loads of hours later I became seriously worried that the Millenium P-Series graphics cards do not provide Auxiliary display support to Matrox Mil!

Mil Auxiliary Display Error

Well, it turns out that the P-Series graphics cards do provide Auxiliary display support to Mil, but in order to get it to work (in my case) an update needed to be applied to the Matrox graphics card driver – In my case I installed the update contained within m800du09.zip, once applied the Auxiliary display worked just fine!

So anyway the moral of this story is that if you’re trying to get an Auxiliary display to work with Mil and you’re getting this error message no matter what you do – get on to Matrox and ask if there are any updates for your graphics card drivers and install them straight away!

PS As the Mil documentation says, in order to get the Auxiliary display to work you will first have to disable the ‘dual-head’ functionality of your Matrox graphics card.  Now, with the newer Matrox graphics software ‘dual-head support’ is no longer called ‘dual-head support’, instead go into the Matrox ‘Power Desktop’ settings, and set the ‘Multi-Display Setup’ option to ‘1 Display (no feature display)’.

PPS It seems that where the Auxiliary display is concerned, the Matrox drivers are not very good at cleaning up their resources when the calling process ends – it is very important that you call MdispFree() on the Aux. display when your process is shutting down otherwise when you try to re-launch the process you will get the above error message every time you try to allocate the display until you reboot your system!

Matrox Mil & Firewire 1394 Image Acquisition

With newer versions of Matrox Mil (>= Mil 8 ) it is possible to grab images from firewire cameras without the need for any Matrox hardware, Mil will just use your PC’s normal Firewire/1394 adaptor(s) to acquire the images. When setup correctly you should be able to grab from either code or from intellicam.

However in typical Matrox fashion finding out how to set Mil up this way is quite difficult as there isn’t much information ‘out there’ or in the Mil documentation on how to do it,  and (again in typical Matrox fashion) the procedure is a tad non-intuitive.

So to add 1394 support – Install Mil, and when you get to the stage where it asks you which drivers it should install for the various possible frame grabber cards choose ‘Meteor II 1394’ remember though that you don’t  actually need a Meteor II frame grabber – Mil will use your existing Firewire adaptor(s) instead!

The next thing to note is that whenever you want to see or use any of the Mil 1394 features from Intellicam or from code you must have your camera(s) plugged in – otherwise Mil will deny all knlowedge!

For example if you start up Intellicam without your firewire camera plugged in then you may get an error message like the following, plugging the camera in and restarting Intellicam will fix things:

ooops camera is unplugged!

To grab images from Intellicam, first plug in your camera and then launch Intellicam and make sure the ‘Meteor-II /1394’ system is selected.

 Choose the 1394 System

With firewire Mil does not use DCF files like it does for other frame-grabber and camera combinations, instead the required camera and digitiser format is specified by a string.  To see the available formats hit the ‘New DCF’ toolbar button or choose the ‘File/New…’ menu option,  this will display a list of format strings:

Choose your digitiser format

Once you select a suitable format you should be able to grab from the camera!