MVVM and Speech using the Kinect-Pt. II

In the last post I talked about what I recently done with Speech recognition and tying it in with MVVM’s concepts of Commands.

In this post, I want to walkthrough, step by step of how I set things up.  To get everything installed I just followed the directions for setting up the Kinect SDK, which also included the direction on setting up the Speech API.  Google that and you’ll be well on your way.

After getting it setup, I recommend you give the Kinect SDK samples a try to make sure everything installed correctly.  From there I took a look at what the Kinect speech sample was doing and modified it to work with the default audio source instead of the Kinect.  Mostly, because my Kinect needs to pull double duty between my hacking and actually letting me play on the Xbox.  Not sure how I can convince the wife we need a second one just yet.

Note that some of the code examples use some extensions methods in a little library of mine.  So you might not be able to directly copy/paste. Hit up the continue reading link for the rest…

Continue reading “MVVM and Speech using the Kinect-Pt. II”

MVVM and Speech using the Kinect–Pt. I

I have always been fascinated by interacting with a computer in ways beyond the traditional keyboard and mouse. The problem is, as I learned while getting my degree in computer science, some problems are really hard.

I didn’t want to dedicate years to obtaining PhD in computer vision or spend my life doing statistics. But, like any other good developer, I’ll just take the toolkit someone else came up with to solve the hard problems.

Enter the Kinect.

In reality, Microsoft has had a Speech API for quite some time and I’ve played with it in the past, but with the Kinect they’ve produced a beautiful piece of hardware that can do 3D depth sensing, speech recognition and directional detection, live skeletal tracking and more all in package that doesn’t cost a lot more than a high-end web cam. This first example doesn’t actually require the Kinect, in fact, the code itself just uses the default audio input. But it can be easily changed to use the Kinect audio stream. Later projects I’m working on will use the Kinect camera’s to do some hopefully neat things.

Since the Kinect hacking started my brain has been churning with ideas. Of the most pragmatic, was the thought to tie in speech recognition to an MVVM application. The nice thing about a well implemented screen using MVVM is that you have your UI described seperately in XAML on the front end and a class (ViewModel) containing your library of commands that can be executed. Using a Command object you can tie a specific element, like a button, to perform a specific command like Save very cleanly and easily.

This clean seperation of concerns mean you don’t really care how a command is invoked, whether it’s a button press, keyboard shortcut, or voice command it all works the same. Your ViewModel executes the command and the UI happily updates through the powerful databinding of XAML.

Aside from the obvious scifi references that this brings to mind, it could also help by making programs more accessible to the vision or mobility impaired. Also, it could be just plain more efficient in some scenarios.

Most of the work is done for us by the commanding infrastructure in WPF. So first I’d like ta take a look at how this implementation will be used. Below is a standard button declaration with a command attached.

The other great thing about XAML is the extensibility, so by the time we’ve implemented this speech API the only thing that will change is this:

One simple property added and that’s pretty much all the end developer needs to do. The only other thing we need for using the speech recognition is something I call a SpeechCommand, which is basically just an implementation of the standard DelegateCommand found in MVVM frameworks. The SpeechCommand acts exactly like the standard commands, but it is also the place for the Phrase AttachedProperty to live and is the glue that bridges the application to my wrapper around the Speech API.

In the next post I’ll walkthrough how I built the app and post some source code. Until then, I leave you with a screenshot. Please note, that no mouse or keyboards were harmed (or used Smile) in the taking of this screenshot.



Custom Markup Extension To Replace IValueConverter

Something that’s been around in WPF for a long time but is just seeing the light of day in the upcoming Silverlight 5 release is the concept of a custom MarkupExtension. For those unfamiliar with the concept, a MarkupExtension is anything within the curly braces “{}” of your xaml.

Some examples include, Binding, Static- and DynamicResource, Type, Static, etc. You can look at the code below and see some examples:


Basically, the xaml parser can interpret an attribute as a string literal value or convert it to an object through some means. Markup extensions allow you to do the deciding about how the value you’re setting should be interpreted by the xaml parser. For a much more in depth look at how this all works, a decent place to start is this link on MSDN:

One interesting use of MarkupExtensions I’ve been playing with is a reimplementation of the Binding markup extension to provide an alternative to the IValueConverter interface. Sometimes you really only need to do a conversion once or a couple times on a specific screen. When you implement a value converter it feels like you are taking a tiny bit of view or business logic and stuffing it in an unrelated portion of your application. That being said, a standardized library of value converters like boolean-to-visibility and so on can make developing on a project much easier to use. But for the one off scenarios, it’d be nice to keep all that logic contained in your ViewModel and not have to worry about spinning up a new class for such a simple thing.

I used the post here as a starting point for my custom binding because working with Binding or BindingBase was not going so well. From there I created my own Binding class and added a ConverterDelegate property. This property looks for a method on the same DataContext of the binding and uses a generic IValueConverter behind the scenes to call that method of the DataContext. This helps get rid of all those little one-off classes that you have to create for specific conversion scenarios.

The code is far from production ready but the gist of it is that a custom MarkupExtension overrides the ProvideValue method of the abstract MarkupExtension class. In the background I look up the method on my DataContext and use a IValueConverter that calls that method to do the conversion. So to wire up my converter I can simply do this:


You can see I have a converter delegate called ToUpper and in my code behind or ViewModel I can simply write whatever methods I need to do conversions like this:

public partial class Window1 : Window
	public Window1()
		DataContext = this;

	// our converter method
	public object ToUpper(object value)
		return (object)value.ToString().ToUpper();

	public string ButtonText { get; set;}

I can post the sample project if I get interest but the real point is to give a small showing of what’s possible with custom MarkupExtensions in WPF and SL5.

Wii Remote Force Calculator

Recently, I’ve been doing a fun side project. It’s always fun when you can come up with an excuse to combine your hobbies or interests. So I came up with the idea of using a Wii remote to calculate the force of a strike on a heavy bag.


I’ve had to dredge up old memories from my college physics classes, and I’m not entirely sure what I’ve got is hundred percent accurate, but I feel with proper setup it’s pretty good. The calculation is easy enough, to get the force you just multiply the mass times the acceleration of the object. The accelerometers in the Wii remote are accurate to about +/- 3 Gs with about a ten percent range of error. Not too terrible for a fifty dollar bluetooth enable accelerometer you can pick up at any store in town. The problem is, if you try to hold it in your hand you can generate massive amounts of acceleration well above 3 Gs. So instead, I tied the remote onto a heavy bag. Much more mass, means you get the same amount of force with acceleration levels well within the remotes acceptable range.

Force = mass * acceleration

Because I’m not punching any harder, the force stays the same. The heavy bag has a lot more mass, so the acceleration has to be a lot less to equal the same force.

The thing I’m not sure of is whether I have to take into account the fact the bag is on a chain so it actually swings up in an arc. I don’t know how much that affects the calculations, but I don’t think it would be a lot with the small amount of time range we’re talking about.

My next steps are to try and rig up some testing system to try to determine the accuracy of the measurements. I also want to get the opinion of some physics people to get my calculations even more accurate. It’s been a fun project so far and I’d like to get it to the point where I could bring it into the gym and have my students play around with it a little bit.

The application itself is written in C#, using WPF, MVVM, and Wiimote Lib

I’ll try to post any updates here as they develop.

Overlaying Controls in WPF with Adorners

One of the common things that comes up on multiple projects using WPF is the ability to overlay the screen or a certain portion of it. Either to create a richer modal-type experience than a message box provides or to block access to a certain portion of the screen while an asynchronous or long running operation is happening.

There are a number of ways to do this but the one I’ve settled on after tackling it on a few projects is an adorner that automatically overlays and control with any content you want.

Other options include using the Popup control, which is problematic because popups are not part of the normal visual layout. They are always on top of all other content and don’t move when you resize or move the window, at least not automatically. Another way you can do it is put everything inside a grid, and add the content you want to overlay with at the end of the Grid’s content with no Row or Column specification. You can set the visibility to collapsed and show or hide based on databinding or triggers, etc. This works better than the popup for resizing, but is not as reusable. Even though the adorner is a bit more code, I think it’s more reusable and better than the Popup option.

The way I use it is I create a UserControl that will be my overlay, let’s call it ProgressMessage. I’ve got a Grid I want to overlay called LayoutRoot. I can then call OverlayAdorner<ProgressMessage>.Overlay(LayoutRoot). Now my grid will be overlaid with the ProgressMessage user control. I’ve also provided an override of the Overlay method so you can actually pass in an instance of the content you want to overlay with.

I use a factory pattern and how IDisposable/using statements work to automatically create/remove the adorner. You could also store the IDisposable that’s returned and call Dispose later to remove the AdornerLayer

using (OverlayAdorner.Overlay(LayoutRoot)) 
   // do some stuff here while overlaid 

A couple of quick notes, because of the way WPF layout and hit-testing works, you should not have any height or width set on your overlay content, and the background needs to be non-transparent. To get a semi-transparent background use the alpha-portion of the aRGB color format on your background. So instead of Black, use #44000000 and that gives you a semi-transparent gray background. Additionally, all these methods block mouse input, but the keyboard navigation remains active. I’ve started playing with lost focus events and other methods to intercept losing focus and retain that. Otherwise the user can tab through the controls underneath the overlay and activate them using arrow keys, enter and space bar. You can either solve this, or once I straighten it out I’ll post what I come up with


Here is the rest of the class, OverlayAdorner.cs

    /// Overlays a control with the specified content 
    /// The type of content to create the overlay from 
    public class OverlayAdorner : Adorner, IDisposable where TOverlay : UIElement, new()
        private UIElement _adorningElement; private AdornerLayer _layer; ///  /// Overlay the specified element ///  /// The element to overlay ///  public static IDisposable Overlay(UIElement elementToAdorn) { return Overlay(elementToAdorn, new TOverlay()); } 
        /// Overlays the element with the specified instance of TOverlay 
        /// Element to overlay 
        /// The content of the overlay 
        public static IDisposable Overlay(UIElement elementToAdorn, TOverlay adorningElement)
            var adorner = new OverlayAdorner(elementToAdorn, adorningElement);
            adorner._layer = AdornerLayer.GetAdornerLayer(elementToAdorn);
            return adorner as IDisposable;

        private OverlayAdorner(UIElement elementToAdorn, UIElement adorningElement)
            : base(elementToAdorn)
            this._adorningElement = adorningElement;
            if (adorningElement != null)
            Focusable = true;

        protected override int VisualChildrenCount
            get { return _adorningElement == null ? 0 : 1; }

        protected override Size ArrangeOverride(Size finalSize)
            if (_adorningElement != null)
                Point adorningPoint = new Point(0, 0);
                _adorningElement.Arrange(new Rect(adorningPoint, this.AdornedElement.DesiredSize));
            return finalSize;

        protected override Visual GetVisualChild(int index)
            if (index == 0 && _adorningElement != null)
                return _adorningElement;
            return base.GetVisualChild(index);
        public void Dispose()

Frame-based animation in WPF

I recently was working on the ubiquitous photo/slideshow app in WPF.  This is something I’ve been tinkering with off and on for last 6 months.  The original intention was to create a photo slideshow application for my upcoming wedding.  Being the nerd, a static video slideshow just wasn’t going to cut it.

Along the way I learned quite a bit about keeping performance up and memory usage low while working with tons of images.  It’s finally in a position where it’s almost done and I wanted to add a few tweaks.  The photos zoom in and randomly arrange like they were dropped on a table.  Once there, they show one by one. 

I wanted to add a little random “drift” while the image was showing to make it more interesting to the eye.  I started originally by creating random storyboards and listening to the Storyboard.Completed event.  When the event fired I created a new storyboard to animate my photo’s Canvas.Left and Top properties.  This worked, but there was an annoying lag between the stop and start of the animations. 

I wanted to move to a frame-based animation rather than WPF’s built-in time-based animation style.  I could have used a timer to update my properties, but I wanted to work more within the constraints of WPF.  I found articles for Silverlight that indicate an empty storyboard with no duration will fire it’s completed event on the next frame, there you can update your properties and restart the storyboard to update your properties every frame.  Although this may work in Silverlight, I could not get it to work in WPF.  Though as I look back, I didn’t try setting the Duration to “0:0:0”.  I wonder if that would work?

Regardless, the technique I ended up using was listening to CompositionTarget.Rendering event it code-behind.  The event fires before your UI renders each frame, allowing you to hook in and do frame based animation.

For more information, you can go here:

WPF ListBox Virtualization

Virtualization in WPF is amazing when you get it, annoying when you lose it.  I’m working on a form where the user can dynamically filter and sort a ListBox.  I’m using a DataTemplate to give a nice card-like view to each object, which makes it take up a  lot of space on the screen.  To fix this I used the WrapPanel as the ListBox’s ItemsPanel.


WrapPanel is not virtualized, I can’t find one that is and don’t have any time to write one myself.  So I came up with a work around to give the user a fixed number of column view using the standard panel for a ListBox.  This got me virtualization back.  Then I wanted to smoothly scroll so I turned ScrollViewer.CanContentScroll to false.  Smooth scrolling, no virtualization.  Because this box may have thousands of records in it, performance is much more important than smooth scrolling. 

The moral of the story is, if you’re working on something in WPF and performance is important and you might have a large dataset coming in.  Double check your changes to controls to make sure you’re maintaining virtualization.  If anybody has a good list of what disables virtualization (grouping, custom panels, smooth scrolling), I’d like to see it.

Hosting WPF in WinForms

I’ve done a lot of WPF and Silverlight, but always as standalone apps. In one project, we actually embedded Silverlight inside a WPF application using a browser control.

I’ve been firmly convinced that XAML is the way of the future for some time now. But recently I’ve gotten to see a very real, tangible benefit of hosting WPF inside a windows form application.

A lot of time businesses will want some fancy visualization or way to view their data. In Windows Forms if you want anything “cool” out of the box you’ve got one of these options: paying for a 3rd party set of controls, finding a half-baked control online at CodePlex or CodeProject, or writing it yourself.

Most of the time the project budget and plan don’t include time or money for options 1 and 3. Option 2 is Russian roulette. Sometimes you get some really cool stuff, sometimes the control you find is garbage. In WPF, it becomes almost trivial to do the sort of complex control customization that was such a pain in WinForms. At the same time, most business probably don’t want to start off by rewriting their existing app in WPF.

Using the ElementHost control and some custom WPF UserControls you can embed some very cool UI functionality with minimal effort and impact to your existing application. My WPF UserControls don’t look like “WPF”. At least not all the LOB demos you see where there are fancy transitions and gradients slapped in everywhere. But it’s blowing away the end users. The integration is seamless, it looks like WinForms, but simply using some stylish DataTemplates and the WPF TreeView we’ve improved the customers user experience ten-fold. We’re using it 3 critical places of the app where the user needed some better visualization of the data that was coming in. With WPF we could do that much faster and better with minimal impact to the rest of the application.