VizualBusiness: Abusing the Microsoft Research's Touch Mouse Sensor API SDK with a Console-based Heat-map

Abusing the Microsoft Research's Touch Mouse Sensor API SDK with a Console-based Heat-map:

In August I purchased and reviewed the Microsoft Touch Mouse. I still use my Microsoft Arc Mouse more than the touch, initially due to what I felt was dodgy scrolling performance on the Touch Mouse, as I mentioned in my review. Still, I've kept it in my backpack and I use the Touch Mouse perhaps a few times a month and have kept the software up to date in case there's some software changes made to improve performance.

I can happily say that they've changed something and the scrolling performance is WAY better. I can finally get 1 to 2 pixel precision with it while scrolling in my browsers. The other nice feature is the "three finger swipe up" which gives you effectively a Windows version of the Mac Expose window switcher view.

Today I noticed while catching up on Long Zheng's excellent blog that the Touch Mouse Sensor SDK is available for download. Per their site:

"The Microsoft Touch Mouse supports multitouch gestures via an integrated sensor. The Touch Mouse Sensor API SDK is a small library intended to enable students and researchers to experiment directly with Touch Mouse sensor output.

Using this SDK, you can create your own applications consuming the 13×15 sensor image. The SDK includes C# and C++ samples demonstrating how to read and manipulate sensor data"

OK, so what can we do with this thing? It comes with a number of samples to get the bitmap from the sensor and process it. It includes examples in both C# and C++. Because I don't have C++ on this machine I got an error opening that solution, but you can just remove that project and still build the samples.

You don't need to do your work in a graphical application, although it's nice to visualize this data of course. First sample is a console app, that tells you were the center of mass of your big fat finger is.

The smarts are in the Microsoft.Research.TouchMouseSensor namespace. You implement a callback function that the mouse effectively calls whenever something interesting happens. It looks like this:

/// <summary>
/// Function receiving callback from mouse.
/// </summary>
/// <param name="pTouchMouseStatus">Values indicating status of mouse.</param>
/// <param name="pabImage">Bytes forming image, 13 rows of 15 columns.</param>
/// <param name="dwImageSize">Size of image, assumed to always be 195 (13x15).</param>
static void TouchMouseCallbackFunction(ref TOUCHMOUSESTATUS pTouchMouseStatus,
   byte[] pabImage,
   int dwImageSize)
{
...
}

As you can see, the image size is 13x15, always. You get the raw bytes that represent the image, 195 bytes long.

The first thing I was surprised to see was just HOW MUCH of the surface area of the mouse is covered and how high the resolution is. I was thinking, "13x15? That's so small, how could it tell me anything useful with such a limited structure?" Well, that's how clueless I was.

Of course, each of these bytes is giving a number between 0 and 255 so a lot of information can be extrapolated between two pixels with different values and you can get a pretty detailed picture of what's going on. How much? Well, before I move on to a graphical example, why not abuse ASCII as much as we can?

First, I'll output just the hex values with this code, then show what it looks like when I move my figures around the surface of the mouse.

Console.SetWindowSize(50,16);
Console.SetCursorPosition(0, 0);
StringBuilder sb = new StringBuilder(300);

for (Int32 y = 0; y < pTouchMouseStatus.m_dwImageHeight; y++)
{
   // Iterate over columns.
   for (Int32 x = 0; x < pTouchMouseStatus.m_dwImageWidth; x++)
   {
       // Get the pixel value at current position.
       int pixel = pabImage[pTouchMouseStatus.m_dwImageWidth * y + x];
       if (pixel > 0)
           sb.AppendFormat("#{0:X2}", pixel);
       else
           sb.Append("   ");
   }
   sb.Append("\n");
  
}
Console.Write(sb.ToString());

Here's the result, animated.

Cool. But what about with color? I could use the default Console stuff and set the colors, or I could get direct access to the Console, use the Win32 APIs and what not, but I could also use Tim Sneath's ConsoleEx class that's on NuGet thanks to Anthony Mastrean! First, I'll "install-package ConsoleEx" to bring it in.

How about a mouse touch heat-map with color? I thought about building a whole "find closest color" map and trying to map 255 different colors, but then I just did this.

if (pixel > 230)
   Console.BackgroundColor = ConsoleColor.Red;
else if (pixel > 128)
   Console.BackgroundColor = ConsoleColor.Yellow;
else if (pixel > 90)
   Console.BackgroundColor = ConsoleColor.Green;
else if (pixel > 45)
   Console.BackgroundColor = ConsoleColor.Blue;
else if(pixel > 15)
   Console.BackgroundColor = ConsoleColor.DarkBlue;
else
   Console.BackgroundColor = ConsoleColor.Black;

ConsoleEx.WriteAt(x*2, y, "  ");

And the result is this:

You can do the same thing in WPF, of course, with better bitmap support, but seriously, with a Console-based heatmap, who needs graphics? ;)

OK, fine, here's the sample code that takes the results of the TouchMouseSensorEventArgs and creates a grayscale image.

void TouchMouseSensorHandler(object sender, TouchMouseSensorEventArgs e)
{
   // We're in a thread belonging to the mouse, not the user interface
   // thread. Need to dispatch to the user interface thread.
   Dispatcher.Invoke((Action<TouchMouseSensorEventArgs>)SetSource, e);
}

void SetSource(TouchMouseSensorEventArgs e)
{
   // Convert bitmap from memory to graphic form.
   BitmapSource source =
       BitmapSource.Create(e.Status.m_dwImageWidth, e.Status.m_dwImageHeight,
       105, 96,
       PixelFormats.Gray8, null, e.Image, e.Status.m_dwImageWidth);

   // Show bitmap in user interface.
   SensorImage.Source = source;
}

And the result:

256 color grayscale heatmap of the Touch Mouse

One way to look at this is as if the surface of the mouse is a tiny Xbox Kinect. What kinds of gestures could I recognize and hook up to do stuff? Control WIndows, do custom stuff inside my application, browser, or whatever. Any ideas?

The downloads are here: