Tracking fast moving targets with a Gadgeteer camera…

…is completely impractical.

In my last post, I briefly described a simple hack which allowed the Gadgeteer system to control the bearing and azimuth of an old USB missile launcher by using a joystick.  Manual control like this is all very well, but it would be much more entertaining if the systems could track a target automatically before raining down atomic retribution on it.  I use the words ‘would be’ advisedly, because as we shall see this is fairly impractical with the Gadgeteer kit I have to hand.  Even so, there’s a lot to be learned from the process of trying.

The sensor I’m going to use for trying to track targets is a small camera.  There are a number of Gadgeteer cameras available, but the one I have is a Sytech serial camera module.  This captures images at selectable resolutions and returns them in jpeg format.

P1020277

In addition to the camera, I’m going to use the Sytech 4.3″ LCD touch screen as a user interface.  I’ve mentioned this one before in a previous post.  To start with, create a new project in Visual Studio, and connect up the LCD, the main board (I’m using a FEZ Hydra) and the camera on the designer.  In your program.cs file, you should now have an object called ‘camera’ and another called ‘lCDTouch’.  All that remains is to write the code to make these work together.  I’ll leave that as a simple exercise for the reader.

I’m kidding, of course.  There are three basic parts to the code: the user interface, the logic controlling the camera, and the image processing code for acquiring targets.  We’ll start with the user interface.

The user interface

The Micro Framework supports a cut-down version of WPF, which makes it relatively easy to create a useful user interface.  The functionality we need is relatively simple: a few buttons to control the picture taking, and an area to display the picture.  To create this, we will use a couple of StackPanels, because they are a very easy way of containing other controls such as buttons and images.  Incidentally, if you are not familiar with WPF, you might want to look at a tutorial or two.  What we are doing is not horribly complex, but if you are unfamiliar with the basic concepts you may find it a bit confusing.  The great thing about StackPanels is that you can nest them.  So I can have a horizontally-oriented StackPanel, which contains two other StackPanels.  The first of these is vertically-aligned, and contains any buttons I want to use.  The other is also vertical, and contains only a test box for status messages and an image (when the camera has taken it).  The descriptioin is perhaps a little confusing, but (in the current economic climate), a picture is worth six hundred and forty-three words:

ScreenLayout

It won’t win any design awards, but it’s functional.  It is pretty easy to create this, with one small exception: the Micro Framework’s version of WPF doesn’t have a Button control.  This seems a bit of an oversight to me, but maybe that’s because I’m so used to working with PC applications – in the Micro Framework world, a Button is a real, physical lump of plastic and metal.  Fortunately, it’s not too difficult to create a class of our own which does what we need from a button.  After all, what is a Button?  At its simplest, it is a rectangle drawn on the screen, with some text in it.  It generates an event when the user clicks on it.  We can create our own button by deriving a class from ContentControl, which is a WPF control which can hold other controls:

public class WpfButton : ContentControl { public event EventHandler ButtonClicked; private string _Caption; public string Caption { get {return _Caption;} set { _Caption = value; _text.TextContent = value; } } private Color _BackColor; public Color BackColor { get { return _BackColor; } set { _BackColor = value; _rectangle.Fill = new SolidColorBrush(_BackColor); } } private Text _text; private Panel _panel; private Rectangle _rectangle; public WpfButton(int width, int height, string caption, Font font) { this.Width = width; this.Height = height; _Caption = caption; _panel = new Panel(); _rectangle = new Rectangle(this.Width, this.Height); _text = new Text(); _text.TextContent = caption; _text.Font = font; _text.TextAlignment = TextAlignment.Center; _text.VerticalAlignment = VerticalAlignment.Center; _text.TextWrap = true; BackColor = Colors.LightGray; base.Child = _panel; _panel.Children.Add(_rectangle); _panel.Children.Add(_text); _rectangle.Fill = new SolidColorBrush(Colors.LightGray); _rectangle.Stroke = new Pen(Colors.Black); } protected virtual void OnClick() { if (ButtonClicked != null) { ButtonClicked(this, new EventArgs()); } } protected override void OnTouchDown(TouchEventArgs e) { base.OnTouchDown(e); _rectangle.Fill = new SolidColorBrush(Colors.Purple); TouchCapture.Capture(this); } protected override void OnTouchUp(TouchEventArgs e) { base.OnTouchUp(e); TouchCapture.Capture(this, CaptureMode.None); _rectangle.Fill = new SolidColorBrush(_BackColor); OnClick(); } }

This probably deserves a little explanation.  The WpfButton class is descended from ContentControlContentControl, like many controls, has a Child property, whch can be any WPF user interface object.  We want our button to have two children: a Rectangle and a Text control.  In order to do this, we use a Panel control (which can have multiple child controls) as the child of the ContentControl, and the add the Rectangle and the Text to it.  the Text will appear in front of the Rectangle, because it is added later.  There are two properties defined: Caption, which is the text to go in the button (which changes the TextContent property of the Text control in its setter) and BackColor (which set the Fill property of the Rectangle in its setter) to allow us to dynamically modify the WpfButton at runtime.  We can also set these in the constructor, along with the button’s size and its font.  All that remains is to create the button’s behaviour.  In a previous post, we looked at the TouchDown and TouchUp events that the touch screen generates.  All we need to do is override them for our control, and then we can create a new ButtonClicked event.  There is a little bit of sneakiness here, because of how we want the WpfButton to behave.  To give feedback to the user, we want the WpfButton to change colour when it is pressed down.  This is easy – we just change the Fill property of the Rectangle when the touchDown message is received – but what happens when the user slides his finger off the button before the TouchUp message is generated?  A different control will receive the TouchUp, and our WpfButton will remain pressed.  Fortunately, there is a way around this.  Calling the TouchCapture.Capture method means that all touch messages will go to the selected control until further notice.  Of course we need to rescind this as soon as we have received a TouchUp, and that’s what happens in the OnTouchUp handler.

With the WpfButton defined, we can now create our user interface.  First, we’ll declare the UI controls we want to use:

Panel imagePanel;
Font buttonFont;
Text textProgress;
WpfButton buttonTakePicture;
WpfButton buttonSvgaImage;
WpfButton buttonVgaImage;
WpfButton buttonThumbImage;
WpfButton buttonToggleProcessing;

Then in the ProgramStarted() method, create them and assemble the UI:

buttonFont = Resources.GetFont(Resources.FontResources.NinaB); StackPanel mainPanel = new StackPanel(Orientation.Horizontal); lcdTouch.WPFWindow.Child = mainPanel; StackPanel buttonPanel = new StackPanel(Orientation.Vertical); buttonPanel.SetMargin(2); mainPanel.Children.Add(buttonPanel); buttonTakePicture = new WpfButton(80, 50, "Wait", buttonFont); buttonTakePicture.ButtonClicked +=

new EventHandler(buttonTakePicture_ButtonClicked); buttonPanel.Children.Add(buttonTakePicture); buttonSvgaImage = new WpfButton(80, 50, "SVGA", buttonFont); buttonSvgaImage.ButtonClicked +=

new EventHandler(buttonSvgaImage_ButtonClicked); buttonPanel.Children.Add(buttonSvgaImage); buttonVgaImage = new WpfButton(80, 50, "VGA", buttonFont); buttonVgaImage.ButtonClicked +=

new EventHandler(buttonVgaImage_ButtonClicked); buttonPanel.Children.Add(buttonVgaImage); buttonThumbImage = new WpfButton(80, 50, "Thumb", buttonFont); buttonThumbImage.ButtonClicked +=

new EventHandler(buttonThumbImage_ButtonClicked); buttonPanel.Children.Add(buttonThumbImage); buttonToggleProcessing = new WpfButton(80, 50, "Track", buttonFont); buttonToggleProcessing.ButtonClicked +=

new EventHandler(buttonToggleProcessing_ButtonClicked); buttonPanel.Children.Add(buttonToggleProcessing); StackPanel rightHandPanel = new StackPanel(Orientation.Vertical); rightHandPanel.SetMargin(2); mainPanel.Children.Add(rightHandPanel); textProgress = new Text(buttonFont,"Initialising camera"); rightHandPanel.Children.Add(textProgress); imagePanel = new Panel(); rightHandPanel.Children.Add(imagePanel);

If this seems more complex that you were expecting, then bear in mind that we are aiming for something like this:

P1020280-001

You’ll have noticed that there are five buttons, and only one of them says ‘Take picture’.  the three in the middle allow you to choose the resolution of the pictures the camera takes, and the final one will toggle the movement tracking mode.   We now need to write the ButtonClick handlers for the WpfButtons we have defined, and to do this we need to know a little about the camera. 

The camera

Firstly, the camera communicates with the main board over a serial interface.  In one way, we don’t care about this – it’s hidden from us by the software – but this has an impact on performance, because images can be quite big.  The camera supports three different image resolutions: 160×120, 320×240 and 640×320 pixels.  We can set the resolution with the Camera.Resolution property, and this is done in the ButtonClicked handlers for the middle three buttons.  At full resolution 16-bit colour, an image could be 400k bytes big, and that can take several seconds to transfer over the serial connection.  Secondly, the camera does quite a bit of processing itself, so isn’t always available to respond to  requests to take a picture.  We are therefore wise to respond to status messages from the camera, and there are three that we should care about:  

camera.CameraReady += 
  new SerialCamera.CameraEnabledEventHandler(camera_CameraReady);
camera.CameraPictureReady += 
  new SerialCamera.CameraEventHandler(camera_CameraPictureReady);
camera.OnPictureProgess += 
  new GTM.Sytech.Camera.PictureProgressDel(camera_OnPictureProgess);

CameraReady is fired when the camera is first available for taking pictures after it is initialised. CameraPictureReady is triggered when a photo has been taken and is available for download from the camera. OnPictureProgress is fired repeatedly while a picture is being downloaded from the camera, and its arguments contain the progress of the transfer, so that you can give feedback to the user if required.

void camera_CameraReady(SerialCamera sender,

GTM.Sytech.CameraProtocol.ImageSize resolution)
{
    textProgress.TextContent = "Camera is ready";
    buttonTakePicture.BackColor = Colors.Green;
    buttonTakePicture.Caption = "Take picture";
    isProcessing = false;
}


 

 

void camera_OnPictureProgess(object sender,

GTM.Sytech.Camera.ProgressEventArg arg) { textProgress.TextContent = "Retrieving picture: "+ ((arg.blockReceived * 100) / arg.Blocks).ToString() + "%"; }

The most complex method is missing, of course.  That’s the one which is called to process the image when it is returned from the camera.  Before we get to that, remember that we want to be processing images continuously, rather than just pressing a button to take a picture.  The code therefore contains a timer which checks every half second to see if (1) the camera is ready to take another picture and (2) the user wants it to be doing this.  The latter is controlled by a boolean variable called ‘isMovementDetecting’, toggled by the so-far unexplained ‘Track’ button.

Image processing

Calling what this program does ‘Image Processing’ makes it sound very grand.  In reality, the program takes the latest image from the camera, and compares it with the previous one to see if there are any differences.  It does this in a very dumb way, one pixel at a time.  It uses Bitmap’s GetPixel method to get the colour of each pixel in the image, and then converts the Color value to a Gadgeteer.Color value (because that way it’s easier to access the RGB values).  For each pixel, the difference between its RGB values and the those of the previous image are calculated.   If the sum of these is above a defined threshold, the pixel in the resulting image is coloured orange.  The final image thus shows areas of the image representing movement.  Or a change in lighting conditions.  Or a movement of the camera.  Or… well, you get the idea.  This is not a robust target tracking algorithm.  Plus, it takes several seconds to execute.  But it does produce results like this:

P1020286

P1020285

P1020284

Those three images are sequential, and show the introduction of an object into the camera’s field of view.  It’s orange in the middle frame, because it’s new.  In the final frame, the object is its natural colour, because it’s the same as the previous frame.  The code to do this is here:

void camera_CameraPictureReady(SerialCamera sender, GTM.Sytech.Camera.ImageEventArg CameraImage)
{
    buttonTakePicture.Caption = "Take picture";

    if (bitmapCurrent !=null) bitmapPrevious = bitmapCurrent;
    bitmapCurrent=CameraImage.GetBitmap();
    Bitmap bitmapModified = new Bitmap(bitmapCurrent.Width, bitmapCurrent.Height);
    bitmapModified.DrawImage(0,0,bitmapCurrent,0,0,bitmapCurrent.Width, bitmapCurrent.Height);

    if (isMovementDetecting && bitmapPrevious!=null)
    {
        textProgress.TextContent = "Processing image for movement";
                
        int width = (int)bitmapCurrent.Width;
        int height = (int)bitmapCurrent.Height;
        int threshold = 40;

        for (int x = 0; x < width; x++)
        {
            for (int y = 0; y < height; y++)
            {
                Gadgeteer.Color oldColour = bitmapPrevious.GetPixel(x, y);
                Gadgeteer.Color newColour = bitmapCurrent.GetPixel(x, y);
                int deltaRed = AbsDifference(oldColour.R,newColour.R);
                int deltaGreen = AbsDifference(oldColour.G,newColour.G);
                int deltaBlue = AbsDifference(oldColour.B, newColour.B);
                int deltaTotal = deltaRed + deltaGreen + deltaBlue;
                if (deltaTotal > threshold)
                    bitmapModified.SetPixel(x, y, Colors.Orange);
                        
                }
        }
    }

    imagePanel.Children.Clear();
    Image newImage;
    if (isMovementDetecting)
    {
        newImage = new Image(bitmapModified);
        imagePanel.Children.Add(newImage);
    }
    else
    {
        newImage = new Image(bitmapModified);
        imagePanel.Children.Add(newImage);
    }

    buttonTakePicture.BackColor = Colors.Green;
    textProgress.TextContent = "Camera is ready";
    isProcessing = false;
}

The code maintains three bitmaps: the previous image from the camera, the current image and a new image with the current image overlaid by orange pixels where they are sufficiently different from the old image.  It then shoves the new bitmap into a WPF Image control, and puts this into the children of the Panel control of the UI.

Having processed an image and found movement, the logical thing to do would be to direct the missile launcher (that’s where we started, remember) at the orange area and fire a missile.  A problem with that is that it takes about fifteen seconds per frame to do this level of processing, and no self-respecting moving target is going to stand still for that length of time just for the privilege of being shot.  These are also the lowest resolution images the camera can produce.  The biggest images contain twelve times as many pixels.  Even assuming that the main board had enough memory to hold the uncompressed images, it would take several minutes to process each frame.

So what can we learn from this?  Obviously, don’t expect to do complex real-time image processing with a vanilla Gadgeteer system.  I think it’s pretty amazing that it can be done at all, even if real time is not possible.  There are things that can be done to improve the performance of the code, of course – a simple speedup would be to only check a grid of pixels, for example, rather than every single one.  Another would be to use some assembler-level code in the system.  But that’s cheating.

Next time

There’s another reason why we can’t target the missile launcher and fire it.  Two, actually.  Firstly, we’ve got no control over the firing motor.  Secondly, the launcher gives no feedback to the system about its aiming direction, or even if it has any ammunition.  Next time, I’ll look at addressing these issues, by interfacing non-Gadgeteer equipment to the Gadgeteer system.  Golly, it’s getting exciting.

The full monty

The entire code for the project is shown below.  It’s entirely possible that I’ve skipped some critical bits in my explanation, so if you want to simply copy and paste it into your solution, go right ahead.

using System;
using System.Collections;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Presentation;
using Microsoft.SPOT.Presentation.Controls;
using Microsoft.SPOT.Presentation.Media;
using Microsoft.SPOT.Touch;

using Gadgeteer.Networking;
using GT = Gadgeteer;
using GTM = Gadgeteer.Modules;
using Gadgeteer.Modules.Sytech;
using Microsoft.SPOT.Presentation.Shapes;
using Microsoft.SPOT.Input;

namespace MotionTracker
{
    public partial class Program
    {
        Panel imagePanel;
        Font buttonFont;
        Text textProgress;
        WpfButton buttonTakePicture;
        WpfButton buttonSvgaImage;
        WpfButton buttonVgaImage;
        WpfButton buttonThumbImage;
        WpfButton buttonToggleProcessing;

        GT.Timer pictureTimer = new GT.Timer(500);
        bool isProcessing = true;
        Bitmap bitmapCurrent;
        Bitmap bitmapPrevious;
   
        void ProgramStarted()
        {
            buttonFont = Resources.GetFont(Resources.FontResources.NinaB);

            StackPanel mainPanel = new StackPanel(Orientation.Horizontal);
            lcdTouch.WPFWindow.Child = mainPanel;
            StackPanel buttonPanel = new StackPanel(Orientation.Vertical);
            buttonPanel.SetMargin(2);
            mainPanel.Children.Add(buttonPanel);

            buttonTakePicture = new WpfButton(80, 50, "Wait", buttonFont);
            buttonTakePicture.ButtonClicked += new EventHandler(buttonTakePicture_ButtonClicked);
            buttonPanel.Children.Add(buttonTakePicture);

            buttonSvgaImage = new WpfButton(80, 50, "SVGA", buttonFont);
            buttonSvgaImage.ButtonClicked += new EventHandler(buttonSvgaImage_ButtonClicked);
            buttonPanel.Children.Add(buttonSvgaImage);

            buttonVgaImage = new WpfButton(80, 50, "VGA", buttonFont);
            buttonVgaImage.ButtonClicked += new EventHandler(buttonVgaImage_ButtonClicked);
            buttonPanel.Children.Add(buttonVgaImage);

            buttonThumbImage = new WpfButton(80, 50, "Thumb", buttonFont);
            buttonThumbImage.ButtonClicked += new EventHandler(buttonThumbImage_ButtonClicked);
            buttonPanel.Children.Add(buttonThumbImage);

            buttonToggleProcessing = new WpfButton(80, 50, "Movement", buttonFont);
            buttonToggleProcessing.ButtonClicked += new EventHandler(buttonToggleProcessing_ButtonClicked);
            buttonPanel.Children.Add(buttonToggleProcessing);

            StackPanel rightHandPanel = new StackPanel(Orientation.Vertical);
            rightHandPanel.SetMargin(2);
            mainPanel.Children.Add(rightHandPanel);
            textProgress = new Text(buttonFont,"Initialising camera");
            rightHandPanel.Children.Add(textProgress);
            imagePanel = new Panel();
            rightHandPanel.Children.Add(imagePanel);

            camera.CameraReady += new SerialCamera.CameraEnabledEventHandler(camera_CameraReady);
            camera.CameraPictureReady += new SerialCamera.CameraEventHandler(camera_CameraPictureReady);
            camera.OnPictureProgess += new GTM.Sytech.Camera.PictureProgressDel(camera_OnPictureProgess);
            camera.EnableCamera();

            pictureTimer.Tick += new GT.Timer.TickEventHandler(pictureTimer_Tick);
            isProcessing = true;
            pictureTimer.Start();
        }

        void buttonThumbImage_ButtonClicked(object sender, EventArgs e)
        {
            camera.Resolution = GTM.Sytech.CameraProtocol.ImageSize.thumb;
            textProgress.TextContent = "Resolution set to thumb";
        }

        void buttonVgaImage_ButtonClicked(object sender, EventArgs e)
        {
            camera.Resolution = GTM.Sytech.CameraProtocol.ImageSize.vga;
            textProgress.TextContent = "Resolution set to vga";
        }
        void buttonSvgaImage_ButtonClicked(object sender, EventArgs e)
        {
            camera.Resolution = GTM.Sytech.CameraProtocol.ImageSize.svga;
            textProgress.TextContent = "Resolution set to svga";
        }        
        
        void buttonTakePicture_ButtonClicked(object sender, EventArgs e)
        {
            camera.TakePicture();
            buttonTakePicture.BackColor = Colors.Orange;
            buttonTakePicture.Caption = "Wait";
        }    
    
        bool isMovementDetecting = false;
        void buttonToggleProcessing_ButtonClicked(object sender, EventArgs e)
        {
            isMovementDetecting = !isMovementDetecting;
        }

        void pictureTimer_Tick(GT.Timer timer)
        {
            if (isMovementDetecting && !isProcessing)
            {
                isProcessing = true;
                camera.TakePicture();
            }
        }



        void camera_OnPictureProgess(object sender, GTM.Sytech.Camera.ProgressEventArg arg)
        {
            textProgress.TextContent = "Retrieving picture: "+((arg.blockReceived * 100) / arg.Blocks).ToString() + "%";
        }


        

        void camera_CameraPictureReady(SerialCamera sender, GTM.Sytech.Camera.ImageEventArg CameraImage)
        {
            buttonTakePicture.Caption = "Take picture";

            if (bitmapCurrent !=null) bitmapPrevious = bitmapCurrent;
            bitmapCurrent=CameraImage.GetBitmap();
            Bitmap bitmapModified = new Bitmap(bitmapCurrent.Width, bitmapCurrent.Height);
            bitmapModified.DrawImage(0,0,bitmapCurrent,0,0,bitmapCurrent.Width, bitmapCurrent.Height);

            if (isMovementDetecting && bitmapPrevious!=null)
            {
                textProgress.TextContent = "Processing image for movement";
                
                int width = (int)bitmapCurrent.Width;
                int height = (int)bitmapCurrent.Height;
                int threshold = 40;

                for (int x = 0; x < width; x++)
                {
                    for (int y = 0; y < height; y++)
                    {
                        Gadgeteer.Color oldColour = bitmapPrevious.GetPixel(x, y);
                        Gadgeteer.Color newColour = bitmapCurrent.GetPixel(x, y);
                        int deltaRed = AbsDifference(oldColour.R,newColour.R);
                        int deltaGreen = AbsDifference(oldColour.G,newColour.G);
                        int deltaBlue = AbsDifference(oldColour.B, newColour.B);
                        int deltaTotal = deltaRed + deltaGreen + deltaBlue;
                        if (deltaTotal > threshold)
                            bitmapModified.SetPixel(x, y, Colors.Orange);
                        
                        }
                }
            }

            imagePanel.Children.Clear();
            Image newImage;
            if (isMovementDetecting)
            {
                newImage = new Image(bitmapModified);
                imagePanel.Children.Add(newImage);
            }
            else
            {
                newImage = new Image(bitmapModified);
                imagePanel.Children.Add(newImage);
            }

            buttonTakePicture.BackColor = Colors.Green;
            textProgress.TextContent = "Camera is ready";
            isProcessing = false;
        }

        private int AbsDifference(int a, int b)
        {
            if (a >= b)
                return a - b;
            else
                return b - a;
        }

        void camera_CameraReady(SerialCamera sender, GTM.Sytech.CameraProtocol.ImageSize resolution)
        {
            textProgress.TextContent = "Camera is ready";
            buttonTakePicture.BackColor = Colors.Green;
            buttonTakePicture.Caption = "Take picture";
            isProcessing = false;
        }
    }

    public class WpfButton : ContentControl
    {
        public event EventHandler ButtonClicked; 
        
        private string _Caption;
        public string Caption 
        { 
            get {return _Caption;}
            set
            {
                _Caption = value;
                _text.TextContent = value;
            }
        }

        private Color _BackColor;
        public Color BackColor 
        {
            get { return _BackColor; }
            set 
            { 
                _BackColor = value; 
                _rectangle.Fill = new SolidColorBrush(_BackColor); 
            }
        }

        private Text _text;
        private Panel _panel;
        private Rectangle _rectangle;

        public WpfButton(int width, int height, string caption, Font font)
        {
            this.Width = width;
            this.Height = height;
            _Caption = caption;
            _panel = new Panel();
            _rectangle = new Rectangle(this.Width, this.Height);
            _text = new Text();
            _text.TextContent = caption;
            _text.Font = font;
            _text.TextAlignment = TextAlignment.Center;
            _text.VerticalAlignment = VerticalAlignment.Center;
            _text.TextWrap = true;
            BackColor = Colors.LightGray;

            base.Child = _panel;
            _panel.Children.Add(_rectangle);
            _panel.Children.Add(_text);
            _rectangle.Fill = new SolidColorBrush(Colors.LightGray);
            _rectangle.Stroke = new Pen(Colors.Black);
        }

        protected virtual void OnClick()
        {
            if (ButtonClicked != null)
            {
                ButtonClicked(this, new EventArgs());
            }
        }

        protected override void OnTouchDown(TouchEventArgs e)
        {
            base.OnTouchDown(e);
            _rectangle.Fill = new SolidColorBrush(Colors.Purple);
            TouchCapture.Capture(this);        }

        protected override void OnTouchUp(TouchEventArgs e)
        {
            base.OnTouchUp(e);
            TouchCapture.Capture(this, CaptureMode.None);
            _rectangle.Fill = new SolidColorBrush(_BackColor);
            OnClick();        
        }
    }
}

Leave a Reply

Your email address will not be published. Required fields are marked *