Typemock Racer Evaluation (Part 2)

by Stephen Harrison 26. July 2009 23:50

The Blue Screen Of Death was traditionally how us developers demonstrated to customers that they had reached parts of our code that we hadn't tested all that well, however that is so 1990's and with the break down of Moores Law we are focusing on multicore processors to give improved performance.

What does this mean for developers, well it means that we will have to start to use more and more multi-threading in our applications to take advantage of this. Writing multi-threaded applications is hard, very hard (to do well!). It's all to easy to introduce race conditions and thread locking, try updating the UI on the wrong thread in a WinForms application and you will be punished.

This is where we introduce the White Screen Of Death or the White Box of Doom. When an application enters a dead lock and is unable to respond and give feedback to the user Windows creates a white ghost like image covering to it. The White Screen Of Death is our chance to show customers that we got it wrong again.

Don't let your customers suffer from a long loch! Loch Long: (Sorry, it's the best I could do!)
Autumn in Loch Long

Not only is it hard to create multi threaded applications it's also exceptionally hard to test them for scenarios where thread locking may occur. I'm sure their will be plenty of opportunities for us to show the user a WSOD or WBOD.

I'm a huge fan of tools, and having the write tool for the job. I recently put a curve on the edge of my kitchen worktop to make a nicer entrance to the kitchen.

There were a few ways to do this badly, I could have used a jigsaw, I could have done it by hand with a coping saw, I could have used my small router which would have gone through bits like crazy and burnt most of the wood. However for me their was only 1 way. A decent powerful big yellow router, some good sharp bits and a MDF template made with my smaller router – again right tool for the job. I'm really pleased with the result even if it did cost me about £300 (twice what all 3 worktops in the kitchen cost!), and I now have a really nice router for my next job.

Curve to worktop

Enter Typemock Racer to the rescue, the right tool for multi-thread deadlock testing of your classes.

In the past I've been involved a lot with connecting up instrumentation and lab automation. As soon as you have hardware you need to communicate with and data to collect as well as the need for user control and real time data display you create a situation crying out for multi-threads with locks, mutexs and correct invocation onto the UI.

When I started work at Akubio(1) on the RAPid 4 I picked up some code that had all sorts of threading going on with complex locking scenarios, combine this with a need to get the product out of the door quickly, real time data display and last minute instrument control specification changes the chances of a thread lock were all too high, at that time their was little we could do in the way of testing other than manually exercising the software for long periods of time trying all sorts of evil things.

Fortunately Typemock have recently released a product to help reduce the need for so much manual testing called Racer. Now developers can write unit and integration tests with test frameworks like NUnit, apply a [ParallelInspection] attribute or use the ThreadTest class and let Racer go and run through various scenarios weaving through the code to try and catch a dead lock. This can be run from within Visual Studio using test runners such as Resharper.

When I found out about the release of Racer I was really interested because of previous issues and being fully aware of how easy it is to get into a dead lock situation and how difficult it is to test for this so I've thrown together a very simple application that illustrates just how easy it is with a misplaced event to get into a thread lock.

I've included 4 ways to test for the thread lock.


  • A NUnit test using the ParallelInspection test.

  • A NUnit test using the ThreadTest class.

  • A Nunit test that creates 2 threads and starts these together and tests for timeout on Joins.

  • A WinForm UI where buttons can be clicked to test.



The application is available for download from GitHub. The current version is really just a shell with some basic functionality with the sole purpose of inducing a thread lock.

The application is intended to demonstrate a situation of having one or more external hardware devices that communicate with the application, sending temperature data in and allowing the PC to control cooling devices (i.e. fans) via the external hardware. A simple example of this would be a Arduino controller with a relay or driver to control a fan, a thermistor on an analog input and then communicating with the PC via the USB serial port.

I have introduced an interface IExternalDevice which allows the communication class for the hardware to be faked.

public interface IExternalDevice

    {

        event EventHandler DataReceived;

        void Open();

        void Close();

        float ReadTemperature();

        void EnableCooling();

        void DisableCooling();

        bool CheckStatus();

    }



The class FakeExternalDevice simulates an external device communication class and provides a way to simulate the data being received in tests.

namespace ThreadLockSimulator.Model

{

    /// <summary>

    /// Represents an external device that reads temperature

    /// and reports this back, leaving the manager to

    /// decide if heater should be enabled or disabled.

    /// </summary>

    public class FakeExternalDevice : IExternalDevice

    {

        public event EventHandler DataReceived;

 

        private object hardwareLockObject = new object();

 

        protected void RaiseDataReceived()

        {

            // Raise an event to pretend data has been received.

            if (DataReceived != null)

            {

                DataReceived(this, EventArgs.Empty);

            }

        }

 

        public void Open()

        {

            ////// Create a new thread to poll the hardware on.

            ////ThreadStart threadStart = new ThreadStart(PollHardware);

            ////Thread thread = new Thread(threadStart);

            ////thread.Start();

        }

 

        public void Close()

        {

            // TODO: Disable ticker.   

        }

 

        public void TriggerDataReceived()

        {

            // Simulate hardware event where data has been received.

            // in an ideal world the event needs to be outside of the lock

            // as this is what will cause our thread lock.

            lock (hardwareLockObject)

            {

                RaiseDataReceived();

            }

        }

 

        public float ReadTemperature()

        {

            Debug.WriteLine("Read Temperature");

 

            lock (hardwareLockObject)

            {

                return 100F;

            }

        }

 

        /// <summary>

        /// Turns the fan on.

        /// </summary>

        public void EnableCooling()

        {

            lock (hardwareLockObject)

            {

                // Write to hardware

                Debug.WriteLine("Enable cooling cooling", "Fake Device");

            }

        }

 

        /// <summary>

        /// Turns the fan off.

        /// </summary>

        public void DisableCooling()

        {

            lock (hardwareLockObject)

            {

                Debug.WriteLine("Disable cooling", "Fake Device");

            }

        }

 

        /// <summary>

        /// Checks the status of the hardware connection

        /// </summary>

        /// <returns>Returns true if the device is connected</returns>

        public bool CheckStatus()

        {

            lock (hardwareLockObject)

            {

                Debug.WriteLine("Check Status", "Fake Device");

                return true;

            }

        }

    }

}



The class DeviceManager is responsible for managing the external devices (1+), holding data received and passing through commands to the hardware communication classes.

namespace ThreadLockSimulator

{

    /// <summary>

    /// Responsible for hooking up the devices and reading the data

    /// </summary>

    public class DeviceManager

    {

        #region Events

 

        public event EventHandler DataReceived;

        public event EventHandler CurrentTemperatureChanged;

 

        #endregion

 

        #region fields

 

        private float currentTemperature;

        private List<IExternalDevice> externalDevices = new List<IExternalDevice>();

        private List<float> temperatures = new List<float>();

        private ITemperatureController temperatureController;

 

        /// <summary>

        /// Syncronisation object.

        /// </summary>

        private object syncObject = new object();

 

        #endregion

 

        #region Constructors

 

        public DeviceManager(IEnumerable<IExternalDevice> devices)

            : this(new SimpleTemperatureController(), devices)

        {

        }

 

        public DeviceManager(ITemperatureController temperatureController, IEnumerable<IExternalDevice> devices)

        {

            this.temperatureController = temperatureController;

            AddExternalDevices(devices);

        }

 

        #endregion

 

        #region Event Raisers

 

        protected void RaiseDataReceivedEvent(object sender, EventArgs empty)

        {

            System.Diagnostics.Debug.WriteLine("RaiseDataReceived, Thread:" + Thread.CurrentThread.GetHashCode(), "DeviceManager");

 

            EventHandler handler = DataReceived;

            if (handler != null)

            {

                handler(this, EventArgs.Empty);

            }

        }

 

        #endregion

 

        #region Public Properties

 

        public List<float> Temperatures

        {

            get

            {

                // Lock whilst getting the temperatures to prevent alteration.

                lock (syncObject)

                {

                    return temperatures;

                }

            }

        }

 

        public float CurrentTemperature

        {

            get

            {

                lock (syncObject)

                {

                    return currentTemperature;

                }

            }

            private set

            {

                lock (syncObject)

                {

                    currentTemperature = value;

                }

            }

        }

 

        #endregion

 

        #region Public Methods

 

        public void Start()

        {

            // Lock to prevent the colleciton being altered whilst itterating.

            lock (syncObject)

            {

                foreach (IExternalDevice externalDevice in externalDevices)

                {

                    externalDevice.Open();

                }

            }

        }

 

        #endregion

 

        #region Private implementation

 

        void device_DataReceived(object sender, EventArgs e)

        {

            System.Diagnostics.Debug.WriteLine("Data Received", "DeviceManager");

 

            // Lock whilst reading the data to prevent

            // other communications with the hardware

            // and internal data being read whilst it's being updated.

            lock (syncObject)

            {

                IExternalDevice device = (IExternalDevice)sender;

 

                GetTemperature(device);

 

                //CheckTemperature(device, CurrentTemperature);

            }

 

            RaiseDataReceivedEvent(this, EventArgs.Empty);

        }

 

        private void GetTemperature(IExternalDevice device)

        {

            float temperature = device.ReadTemperature();

            CurrentTemperature = temperature;

            temperatures.Add(temperature);

        }

 

        private void CheckTemperature(IExternalDevice device, float temperature)

        {

            if (temperatureController.ShouldEnableCooling(temperature))

            {

                EnableCooling();

            }

            else

            {

                DisableCooling();

            }

        }

 

        private void AddExternalDevices(IEnumerable<IExternalDevice> enumerable)

        {

            // Wire up the data received event and add to the collection of known devices.

            foreach (IExternalDevice externalDevice in enumerable)

            {

                AddExternalDevice(externalDevice);

            }

        }

 

        #endregion

 

        public void AddExternalDevice(IExternalDevice externalDevice)

        {

            lock (syncObject)

            {

                externalDevice.DataReceived += new EventHandler(device_DataReceived);

                externalDevices.Add(externalDevice);

            }

        }

 

        public void EnableCooling()

        {

            lock (syncObject)

            {

                foreach (IExternalDevice externalDevice in externalDevices)

                {

                    externalDevice.EnableCooling();

                }

            }

        }

 

        public void DisableCooling()

        {

            Debug.WriteLine("Disable Cooling", "Manager");

            lock (syncObject)

            {

                foreach (IExternalDevice externalDevice in externalDevices)

                {

                    externalDevice.DisableCooling();

                }

            }

        }

 

        public bool CheckDeviceStatus()

        {

            // Lock whilst accessing hardware, check all the external decivices

            // are ok.

            Debug.WriteLine("Check Status", "Manager");

            lock(syncObject)

            {

                foreach (IExternalDevice externalDevice in externalDevices)

                {

                    if (!externalDevice.CheckStatus())

                    {

                        return false;

                    }                  

                }

 

                return true;

            }

        }

    }

}



You will notice there are 2 flows going on, one the flow of data from the external device into the DeviceManager (and then ultimately to the UI), and the second the flow of control going from the DeviceManager class to the external device's.

I've been a little OTT with the thread locking, but not unrealistic of a real world class to protect the data and instrument from multiple threads accessing it all at once, I've not used anything special in the way of mutex's or other locking techniques.

I have intentionally induced a situation that's easy to do accidentally. I've raised an event from within a lock, this is a recipe for disaster and whilst it's really obvious it's happening here, there are all sorts of ways to do this accidentally (imaging a Dirty event being raised on a data object that you populate from within a lock, a side effect of populating it which you might not be expecting).

So what happens:

If you call EnableCooling on the manager you will get a lock from the manager and then a second lock from the external hardware class.

If on a separate thread the hardware has some data to push to the device manager this will lock on the external hardware class and then try to gain a lock in the device manager.

If these two happen at once you can get a dead lock between the two threads. Hello WSOD!

Here's the test for this:

[Test]

        public void TestUsingThreadTest()

        {

            FakeExternalDevice device = new FakeExternalDevice();

            device.Open();

 

            DeviceManager manager = new DeviceManager(new[] { device });

 

            ThreadTest

                .AddThreadAction(device.TriggerDataReceived)

                .AddThreadAction(manager.EnableCooling)

                .Start();

        }



Here's how Racer highlights this. Clear? Certainly not!.
ThreadLock Visualization

I also tried a test with the ParallelInspection attribute applied to the test:

        [Test]

        [ParallelInspection]

        public void TestSwitchOnCoolingWhilstReceivingData()

        {

            // Create an instance of a faked external hardware device

            FakeExternalDevice device = new FakeExternalDevice();

            device.Open();

 

            DeviceManager manager = new DeviceManager(new[] { device });

 

            // Force a data received from the device.

            // Locks external device lock then try to lock on syncObject in manager.

            device.TriggerDataReceived();

 

            // This should lock on Manager syncObject, try to lock on the external device lock.

            manager.EnableCooling();

 

            // Either the TriggerDataReceived call should fail to lock on the managers syncObject

            // or EnableCooling should fail to lock on the external Device's hardware lock as the other thread should

            // already have acquired the lock.

        }



Here's the result.

Parallel Inspection Test

That's right, Racer failed to detect the deadlock!

Creating 2 threads and running them at the same time actually worked detecting the deadlock, although it's unreliable and would probably fail on a single core processors (I'm running this on a quad core box).

/// <summary>

        /// This test should fail with thread timeouts due to locking but it's

        /// not reliable.

        /// </summary>

        [Test]

        public void TestManualThreadLockTest()

        {

            FakeExternalDevice device = new FakeExternalDevice();

            device.Open();

 

            DeviceManager manager = new DeviceManager(new[] { device });

 

            ThreadStart start1 = device.TriggerDataReceived;

            ThreadStart start2 = manager.EnableCooling;

 

            Thread thread1 = new Thread(start1);

            thread1.Name = "Thread 1";

 

            Thread thread2 = new Thread(start2);

            thread2.Name = "Thread 2";

 

            // Start both the threads.

            // Expect thread blocking as external device locks internally then calls manager which locks internally

            // call to manager locks internally then calles external device which tries to lock internally.

            thread2.Start();

            thread1.Start();

 

            if (!thread1.Join(5000))

            {

                Debug.WriteLine("t1 join");

                Assert.Fail("Thread 1 timeout");

            }

 

            if (!thread2.Join(5000))

            {

                Debug.WriteLine("t2 join");

                Assert.Fail("Thread 2 timeout");

            }

        }



What have I learnt from this:


  • Dead lock detection is difficult, really difficult. Even with good tooling it's still possible to not spot it. Just because your test passes doesn't mean you don't have a dead lock problem.

  • You need to understand what your application is doing and where the locking is likely to occur and write tests around that. It's very possible that you will end up with a lot of tests trying out the combinations.

  • I think Racer tests should be in their own project and run as a separate category, probably along side integration tests. They can be slow running numerous scenarios, the license cost is expensive so theirs a good chance that not every developer on the team would have a license, not having Racer enabled when running the tests will show a lot of incorrect failures.

  • Having a separate project is a visual reminder to add Racer tests which helps highlight that multi thread testing is important and should be included.

  • It's worth testing your classes for deadlocks in both a unit and integration test, just because you test your class in isolation for thread locking doesn't mean that it wont play a part in creating a situation where deadlock can occur.



Problems with Racer.

As you probably noticed from my Racer Part 1 post I'm all to happy to point out problems. I found a few more whilst learning how to use Racer.

Racer has 3 menu options. I think they are all broken.
Racer Menu


  • Racer is disabled by default when you start Visual Studio. I really don't understand why this should be. If you have tests that use Racer and you hit your “Run All Tests” button you get failures from your Racer tests because it's disabled, it's annoying.

  • In part 1 I covered the problems with "Configure Typemock Racer"

  • When I installed the real license Typemock were kind enough to send me (because I wrote the Part 1 blog entry about Racer) is tells me “Evaluation license expires in 3 day(s)”. Erm, I thought that was supposed to be a full license. (What happens in 3 days?)
    License expires

  • Visualize Threads doesn't do anything on my installation. Nothing! Roy Osherove's excellent Racer screencast highlights that the visualization is still experimental. BTW – if you are trying to learn Racer watch that screen cast! Also, if you value unit testing follow Roy's blog and read his book! - it's on my list.



The visualization is terrible, on so many grounds.

Here's what it looks like:
ThreadLock Visualization

It's a really small image, theirs no need for that, theirs no indication that the box's are threads, theirs no clue where in the code the locks/dead locks are. I appreciate that Racer is still very Version 1 and this really shows.

When using Resharper the output “link” for the image is plain text, so you need to copy and paste into something that can show images.

Take a look at CodeRush, the visualization in that is fantastic, they even do an Xpress version which is free, even the CodeRush and Refactor! Pro only costs $249, significantly less than Racer.

I would love to see some of the Code Rush visualization effects make their way into Racer. A big arrow for each thread sat at a lock or a bubble floating above the lock with the thread count in it as well as another big arrow pointing the the dead lock highlighting where it's all gone wrong.

I realise that Racer has two responsibilities, one to integrate with the IDE and provide developers feedback and the other to run as an automated NUnit test where the Visual Studio visualization wont help!

You can't run Racer tests in the NUnit IDE – not every body uses Resharper or TestDriven.NET - seriously some people still like to use NUnit IDE!

NUnit Runner with Racer Test

I've not tried to get these tests running as part of a CI build, but clearly theirs a few extra steps that need to be taken to get this running from MSBuild or CCNET rather than your regular NUnit test running – is this all down to Racer being disabled? – if so it's a real letdown.

I've got a few more things to try Racer with and the nice people at Typemock suggested I try Isolator in combination so I'm going to give that a try when time allows. As it happens I want to give Isolator a try on the .NET micro framework stuff - it's not on the supported list which probably means I won't get very far, just like what happened when I tried redgate's Ants performace and memory profilers with the micro framework which was really sad.


(1) Akubio are no longer in business and the RAPid 4 has been taken over by TTP Labtech.








Tags:

Evaluations

Typemock Racer – Part 1a

by Stephen Harrison 15. July 2009 21:09

I thought it only fair to add a post to stay thank you to Typemock who true to their word have been kind enough to send me a license for Typemock Racer because I blogged about it – even though its somewhat negative.

I wouldn't have bothered posting anything had I though the product was duff, I do think it's going to be one of those must have tools – I already wish it had been around a few years ago as I could have done with it for some evil thread locking I managed to get myself into, going forward with multi-core processors being the norm, more and more of us developers will consider it an essential part of our tool kit, just like NUnit, Resharper, Ants Profiler and many other killer tools.

Tags:

Evaluations

Typemock Racer Evaluation (Part 1) - FAIL

by Stephen Harrison 12. July 2009 20:03

It's taken me a lot of time to get Typemock Racer up and running to the point I can actually evaluate it and I have to say that if it wasn't for the carrot being dangled in front of me (the chance of a free copy for blogging about Racer) I would have given up long before now, sadly all because of a stupid error and a misleading error message.

Error

Now I know it's not very nice to publicly trash a product but Typemock have asked for us to blog about it ("All reviews are welcome") and having a reference on the net that others can find when they have similar issues is always a help, plus I do like to have a little moan about stuff as you might notice from my @BookSwapSteve Twitter account.

One important thing to remember is this product costs a lot of money €890.00 (US $ 1241 / GBP 765) - seriously big money for a tool – even one that works smoothly and sadly no sign or a personal edition. I would expect a much better experience than I've had so far. If this was $30 shareware Id still be disappointed but for non beta $1200 commercial offering it's terrible. Actually that's a little unfair as I've seen some great stuff from the videos and this looks to be a really promising product that I'm sure took a lot of bright people some serious hard work to pull off.

I also believe it's important to provide a balanced view, because Typemock are giving away an expensive license just because we write a bit of a review about it means the quality of the write ups may not be so in depth, for example, here's one that @Typemock mentioned earlier Review of Typemock Racer which whilst it sure is a review it looks like someone threw together some marketing talk and published it hardly touching the product (heck, even the price is quoted in $ not Euros so it's very wrong! - at the time of writing!) – given we have limited time to get the review done that's understandable (they may well have played with the product as far as I know and they do raise some good points about the Racer).

So here's part 1 of my experience of Typemock Racer:


  • Find download page after discovering the download link was broken on the Typemock website.

  • Download the evaluation (x64 version).

  • Install as per normal (Vista Ultimate box).

  • Install opens a web browser so you can request a evaluation license (21 days) and the help file.

  • Fire up Visual Studio.

  • FAIL!



First error:

Typemock failed to load error

The Addin 'Typemock Racer for VS2008' failed to load or cause an exception.

That's a REALLY bad start for a product, and for a product that's testing focused, erm, you'd hope their had been some testing done on it!

So I clicked No as I wanted Typemock Racer to load.
Then I get another error:

Can not open Typemock.Racer Registry key

It's also interesting to note how inconsistent they are in the use of "TypeMock/Typemock" is it M or m? And for added value theirs even a \n in their to try and get a new line - FAIL!.

So I removed the product, installed again (and repeated that a few times over a number of days).

Eventually deciding that today one way or the other I'm going to get Racer going (because I really do think it's a great idea and is something I would have found really useful in the past) I decided to battle on, I've written myself a little mock application simulating reading a hardware device with opportunities for thread locking and got to the point when I wanted to try out Racer.

The clue, as you would expect is in the error, Typemock Racer couldn't access the registry key.

This registry key:

Typemock Racer registry key that it can not open.

Then I realised, the configuration tool would be writing my license code as well as trying to read it so it would need write access to HKLM (local machine) – hence I needed to be an administrator!

Starting the Racer configuration utility from the Start menu by right clicking and using "Run as administrator" soon got the tool loaded up.

It wasn't long before the next problem. Copy and paste my "Company Name" and License from the email and the Set License button was still disabled. WTF!

Typemock Racer Configuration fail.

It turns out that in the copy and paste their was a space at the end of my email address (as you can see – it's not obvious!) and also the License.

So:


  • The application should request admin access through the application manifest thingy

  • The application needs a better error message or the option to install into the current user registry not local machine – “Can not write”, not “Can not open” – because you can open that registry key for read!

  • We should not be developing application with admin access unless we really have to, clearly the Typemock configuration from Visual Studio expects to be in admin mode.

  • The dialog box should do a trim of the entered values to remove spaces either side of the values.



Now back to my application...

Interestingly I can now enable Typemock Racer from the VS2008 Typemock menu when not an admin – I guess that doesn't write to the HKLM key.

Well my first "simple" test turned out to be a lot more impressive than I thought. All I wanted to do was to ensure that my manager class raised a DataReceived event when a (fake) hardware device it is managing raised it's DataReceived event and the manager had processed the data.

Here's the resultant output:

DeviceManagerTest.TestDataReceivedEventRaisedFromManagerWhenDeviceDataReceived : Passed
Typemock Racer Version 1.0.0.23
------------------------------
Total scenarios run: 1

No Issues found running test in single thread
Started running test on multiple threads
Starting new Test Run
..Total scenarios run: 252

Total scenarios run 252 – WOW, 252! where did they all come from.

Time for me to investigate and play with Racer some more. I'm intending to do a (less negative) part 2 post and make my code available so check back in a week or two (13 days of my trial license left!).

Interestingly running Racer didn't require Visual Studio to be run as an administrator (as you would hope) – BUT the stupid license configuration utility required admin access (and could easily have used HKCU – or given the option).

Times like this reminds me as a developer how important it is to get the licensing experience right for the user! as it's all too easy to do down the wrong path and bite the hand that feeds you.

Success of Failure

As developers it's important for us to realise that a step we would do once early on in development and probably not visit again can have a serious implication for the end user, it's all to easy for us to focus on parts of the code we run over and over again – but that one of step can have big implications.

Tags:

Evaluations