The Blue Screen Of Death was traditionally how us developers demonstrated to customers that they had reached parts of our code that we hadn't tested all that well, however that is so 1990's and with the break down of Moores Law we are focusing on multicore processors to give improved performance.
What does this mean for developers, well it means that we will have to start to use more and more multi-threading in our applications to take advantage of this. Writing multi-threaded applications is hard, very hard (to do well!). It's all to easy to introduce race conditions and thread locking, try updating the UI on the wrong thread in a WinForms application and you will be punished.
This is where we introduce the White Screen Of Death or the White Box of Doom. When an application enters a dead lock and is unable to respond and give feedback to the user Windows creates a white ghost like image covering to it. The White Screen Of Death is our chance to show customers that we got it wrong again.
Don't let your customers suffer from a long loch! Loch Long: (Sorry, it's the best I could do!)
Not only is it hard to create multi threaded applications it's also exceptionally hard to test them for scenarios where thread locking may occur. I'm sure their will be plenty of opportunities for us to show the user a WSOD or WBOD.
I'm a huge fan of tools, and having the write tool for the job. I recently put a curve on the edge of my kitchen worktop to make a nicer entrance to the kitchen.
There were a few ways to do this badly, I could have used a jigsaw, I could have done it by hand with a coping saw, I could have used my small router which would have gone through bits like crazy and burnt most of the wood. However for me their was only 1 way. A decent powerful big yellow router, some good sharp bits and a MDF template made with my smaller router again right tool for the job. I'm really pleased with the result even if it did cost me about £300 (twice what all 3 worktops in the kitchen cost!), and I now have a really nice router for my next job.
Enter Typemock Racer to the rescue, the right tool for multi-thread deadlock testing of your classes.
In the past I've been involved a lot with connecting up instrumentation and lab automation. As soon as you have hardware you need to communicate with and data to collect as well as the need for user control and real time data display you create a situation crying out for multi-threads with locks, mutexs and correct invocation onto the UI.
When I started work at Akubio(1) on the RAPid 4 I picked up some code that had all sorts of threading going on with complex locking scenarios, combine this with a need to get the product out of the door quickly, real time data display and last minute instrument control specification changes the chances of a thread lock were all too high, at that time their was little we could do in the way of testing other than manually exercising the software for long periods of time trying all sorts of evil things.
Fortunately Typemock have recently released a product to help reduce the need for so much manual testing called Racer. Now developers can write unit and integration tests with test frameworks like NUnit, apply a [ParallelInspection] attribute or use the ThreadTest class and let Racer go and run through various scenarios weaving through the code to try and catch a dead lock. This can be run from within Visual Studio using test runners such as Resharper.
When I found out about the release of Racer I was really interested because of previous issues and being fully aware of how easy it is to get into a dead lock situation and how difficult it is to test for this so I've thrown together a very simple application that illustrates just how easy it is with a misplaced event to get into a thread lock.
I've included 4 ways to test for the thread lock.
- A NUnit test using the ParallelInspection test.
- A NUnit test using the ThreadTest class.
- A Nunit test that creates 2 threads and starts these together and tests for timeout on Joins.
- A WinForm UI where buttons can be clicked to test.
The application is available for download from GitHub. The current version is really just a shell with some basic functionality with the sole purpose of inducing a thread lock.
The application is intended to demonstrate a situation of having one or more external hardware devices that communicate with the application, sending temperature data in and allowing the PC to control cooling devices (i.e. fans) via the external hardware. A simple example of this would be a Arduino controller with a relay or driver to control a fan, a thermistor on an analog input and then communicating with the PC via the USB serial port.
I have introduced an interface IExternalDevice which allows the communication class for the hardware to be faked.
public interface IExternalDevice
{
event EventHandler DataReceived;
void Open();
void Close();
float ReadTemperature();
void EnableCooling();
void DisableCooling();
bool CheckStatus();
}
The class FakeExternalDevice simulates an external device communication class and provides a way to simulate the data being received in tests.
namespace ThreadLockSimulator.Model
{
/// <summary>
/// Represents an external device that reads temperature
/// and reports this back, leaving the manager to
/// decide if heater should be enabled or disabled.
/// </summary>
public class FakeExternalDevice : IExternalDevice
{
public event EventHandler DataReceived;
private object hardwareLockObject = new object();
protected void RaiseDataReceived()
{
// Raise an event to pretend data has been received.
if (DataReceived != null)
{
DataReceived(this, EventArgs.Empty);
}
}
public void Open()
{
////// Create a new thread to poll the hardware on.
////ThreadStart threadStart = new ThreadStart(PollHardware);
////Thread thread = new Thread(threadStart);
////thread.Start();
}
public void Close()
{
// TODO: Disable ticker.
}
public void TriggerDataReceived()
{
// Simulate hardware event where data has been received.
// in an ideal world the event needs to be outside of the lock
// as this is what will cause our thread lock.
lock (hardwareLockObject)
{
RaiseDataReceived();
}
}
public float ReadTemperature()
{
Debug.WriteLine("Read Temperature");
lock (hardwareLockObject)
{
return 100F;
}
}
/// <summary>
/// Turns the fan on.
/// </summary>
public void EnableCooling()
{
lock (hardwareLockObject)
{
// Write to hardware
Debug.WriteLine("Enable cooling cooling", "Fake Device");
}
}
/// <summary>
/// Turns the fan off.
/// </summary>
public void DisableCooling()
{
lock (hardwareLockObject)
{
Debug.WriteLine("Disable cooling", "Fake Device");
}
}
/// <summary>
/// Checks the status of the hardware connection
/// </summary>
/// <returns>Returns true if the device is connected</returns>
public bool CheckStatus()
{
lock (hardwareLockObject)
{
Debug.WriteLine("Check Status", "Fake Device");
return true;
}
}
}
}
The class DeviceManager is responsible for managing the external devices (1+), holding data received and passing through commands to the hardware communication classes.
namespace ThreadLockSimulator
{
/// <summary>
/// Responsible for hooking up the devices and reading the data
/// </summary>
public class DeviceManager
{
#region Events
public event EventHandler DataReceived;
public event EventHandler CurrentTemperatureChanged;
#endregion
#region fields
private float currentTemperature;
private List<IExternalDevice> externalDevices = new List<IExternalDevice>();
private List<float> temperatures = new List<float>();
private ITemperatureController temperatureController;
/// <summary>
/// Syncronisation object.
/// </summary>
private object syncObject = new object();
#endregion
#region Constructors
public DeviceManager(IEnumerable<IExternalDevice> devices)
: this(new SimpleTemperatureController(), devices)
{
}
public DeviceManager(ITemperatureController temperatureController, IEnumerable<IExternalDevice> devices)
{
this.temperatureController = temperatureController;
AddExternalDevices(devices);
}
#endregion
#region Event Raisers
protected void RaiseDataReceivedEvent(object sender, EventArgs empty)
{
System.Diagnostics.Debug.WriteLine("RaiseDataReceived, Thread:" + Thread.CurrentThread.GetHashCode(), "DeviceManager");
EventHandler handler = DataReceived;
if (handler != null)
{
handler(this, EventArgs.Empty);
}
}
#endregion
#region Public Properties
public List<float> Temperatures
{
get
{
// Lock whilst getting the temperatures to prevent alteration.
lock (syncObject)
{
return temperatures;
}
}
}
public float CurrentTemperature
{
get
{
lock (syncObject)
{
return currentTemperature;
}
}
private set
{
lock (syncObject)
{
currentTemperature = value;
}
}
}
#endregion
#region Public Methods
public void Start()
{
// Lock to prevent the colleciton being altered whilst itterating.
lock (syncObject)
{
foreach (IExternalDevice externalDevice in externalDevices)
{
externalDevice.Open();
}
}
}
#endregion
#region Private implementation
void device_DataReceived(object sender, EventArgs e)
{
System.Diagnostics.Debug.WriteLine("Data Received", "DeviceManager");
// Lock whilst reading the data to prevent
// other communications with the hardware
// and internal data being read whilst it's being updated.
lock (syncObject)
{
IExternalDevice device = (IExternalDevice)sender;
GetTemperature(device);
//CheckTemperature(device, CurrentTemperature);
}
RaiseDataReceivedEvent(this, EventArgs.Empty);
}
private void GetTemperature(IExternalDevice device)
{
float temperature = device.ReadTemperature();
CurrentTemperature = temperature;
temperatures.Add(temperature);
}
private void CheckTemperature(IExternalDevice device, float temperature)
{
if (temperatureController.ShouldEnableCooling(temperature))
{
EnableCooling();
}
else
{
DisableCooling();
}
}
private void AddExternalDevices(IEnumerable<IExternalDevice> enumerable)
{
// Wire up the data received event and add to the collection of known devices.
foreach (IExternalDevice externalDevice in enumerable)
{
AddExternalDevice(externalDevice);
}
}
#endregion
public void AddExternalDevice(IExternalDevice externalDevice)
{
lock (syncObject)
{
externalDevice.DataReceived += new EventHandler(device_DataReceived);
externalDevices.Add(externalDevice);
}
}
public void EnableCooling()
{
lock (syncObject)
{
foreach (IExternalDevice externalDevice in externalDevices)
{
externalDevice.EnableCooling();
}
}
}
public void DisableCooling()
{
Debug.WriteLine("Disable Cooling", "Manager");
lock (syncObject)
{
foreach (IExternalDevice externalDevice in externalDevices)
{
externalDevice.DisableCooling();
}
}
}
public bool CheckDeviceStatus()
{
// Lock whilst accessing hardware, check all the external decivices
// are ok.
Debug.WriteLine("Check Status", "Manager");
lock(syncObject)
{
foreach (IExternalDevice externalDevice in externalDevices)
{
if (!externalDevice.CheckStatus())
{
return false;
}
}
return true;
}
}
}
}
You will notice there are 2 flows going on, one the flow of data from the external device into the DeviceManager (and then ultimately to the UI), and the second the flow of control going from the DeviceManager class to the external device's.
I've been a little OTT with the thread locking, but not unrealistic of a real world class to protect the data and instrument from multiple threads accessing it all at once, I've not used anything special in the way of mutex's or other locking techniques.
I have intentionally induced a situation that's easy to do accidentally. I've raised an event from within a lock, this is a recipe for disaster and whilst it's really obvious it's happening here, there are all sorts of ways to do this accidentally (imaging a Dirty event being raised on a data object that you populate from within a lock, a side effect of populating it which you might not be expecting).
So what happens:
If you call EnableCooling on the manager you will get a lock from the manager and then a second lock from the external hardware class.
If on a separate thread the hardware has some data to push to the device manager this will lock on the external hardware class and then try to gain a lock in the device manager.
If these two happen at once you can get a dead lock between the two threads. Hello WSOD!
Here's the test for this:
[Test]
public void TestUsingThreadTest()
{
FakeExternalDevice device = new FakeExternalDevice();
device.Open();
DeviceManager manager = new DeviceManager(new[] { device });
ThreadTest
.AddThreadAction(device.TriggerDataReceived)
.AddThreadAction(manager.EnableCooling)
.Start();
}
Here's how Racer highlights this. Clear? Certainly not!.
I also tried a test with the ParallelInspection attribute applied to the test:
[Test]
[ParallelInspection]
public void TestSwitchOnCoolingWhilstReceivingData()
{
// Create an instance of a faked external hardware device
FakeExternalDevice device = new FakeExternalDevice();
device.Open();
DeviceManager manager = new DeviceManager(new[] { device });
// Force a data received from the device.
// Locks external device lock then try to lock on syncObject in manager.
device.TriggerDataReceived();
// This should lock on Manager syncObject, try to lock on the external device lock.
manager.EnableCooling();
// Either the TriggerDataReceived call should fail to lock on the managers syncObject
// or EnableCooling should fail to lock on the external Device's hardware lock as the other thread should
// already have acquired the lock.
}
Here's the result.
That's right, Racer failed to detect the deadlock!
Creating 2 threads and running them at the same time actually worked detecting the deadlock, although it's unreliable and would probably fail on a single core processors (I'm running this on a quad core box).
/// <summary>
/// This test should fail with thread timeouts due to locking but it's
/// not reliable.
/// </summary>
[Test]
public void TestManualThreadLockTest()
{
FakeExternalDevice device = new FakeExternalDevice();
device.Open();
DeviceManager manager = new DeviceManager(new[] { device });
ThreadStart start1 = device.TriggerDataReceived;
ThreadStart start2 = manager.EnableCooling;
Thread thread1 = new Thread(start1);
thread1.Name = "Thread 1";
Thread thread2 = new Thread(start2);
thread2.Name = "Thread 2";
// Start both the threads.
// Expect thread blocking as external device locks internally then calls manager which locks internally
// call to manager locks internally then calles external device which tries to lock internally.
thread2.Start();
thread1.Start();
if (!thread1.Join(5000))
{
Debug.WriteLine("t1 join");
Assert.Fail("Thread 1 timeout");
}
if (!thread2.Join(5000))
{
Debug.WriteLine("t2 join");
Assert.Fail("Thread 2 timeout");
}
}
What have I learnt from this:
- Dead lock detection is difficult, really difficult. Even with good tooling it's still possible to not spot it. Just because your test passes doesn't mean you don't have a dead lock problem.
- You need to understand what your application is doing and where the locking is likely to occur and write tests around that. It's very possible that you will end up with a lot of tests trying out the combinations.
- I think Racer tests should be in their own project and run as a separate category, probably along side integration tests. They can be slow running numerous scenarios, the license cost is expensive so theirs a good chance that not every developer on the team would have a license, not having Racer enabled when running the tests will show a lot of incorrect failures.
- Having a separate project is a visual reminder to add Racer tests which helps highlight that multi thread testing is important and should be included.
- It's worth testing your classes for deadlocks in both a unit and integration test, just because you test your class in isolation for thread locking doesn't mean that it wont play a part in creating a situation where deadlock can occur.
Problems with Racer.
As you probably noticed from my Racer Part 1 post I'm all to happy to point out problems. I found a few more whilst learning how to use Racer.
Racer has 3 menu options. I think they are all broken.
- Racer is disabled by default when you start Visual Studio. I really don't understand why this should be. If you have tests that use Racer and you hit your Run All Tests button you get failures from your Racer tests because it's disabled, it's annoying.
- In part 1 I covered the problems with "Configure Typemock Racer"
- When I installed the real license Typemock were kind enough to send me (because I wrote the Part 1 blog entry about Racer) is tells me Evaluation license expires in 3 day(s). Erm, I thought that was supposed to be a full license. (What happens in 3 days?)
- Visualize Threads doesn't do anything on my installation. Nothing! Roy Osherove's excellent Racer screencast highlights that the visualization is still experimental. BTW if you are trying to learn Racer watch that screen cast! Also, if you value unit testing follow Roy's blog and read his book! - it's on my list.
The visualization is terrible, on so many grounds.
Here's what it looks like:
It's a really small image, theirs no need for that, theirs no indication that the box's are threads, theirs no clue where in the code the locks/dead locks are. I appreciate that Racer is still very Version 1 and this really shows.
When using Resharper the output link for the image is plain text, so you need to copy and paste into something that can show images.
Take a look at CodeRush, the visualization in that is fantastic, they even do an Xpress version which is free, even the CodeRush and Refactor! Pro only costs $249, significantly less than Racer.
I would love to see some of the Code Rush visualization effects make their way into Racer. A big arrow for each thread sat at a lock or a bubble floating above the lock with the thread count in it as well as another big arrow pointing the the dead lock highlighting where it's all gone wrong.
I realise that Racer has two responsibilities, one to integrate with the IDE and provide developers feedback and the other to run as an automated NUnit test where the Visual Studio visualization wont help!
You can't run Racer tests in the NUnit IDE not every body uses Resharper or TestDriven.NET - seriously some people still like to use NUnit IDE!
I've not tried to get these tests running as part of a CI build, but clearly theirs a few extra steps that need to be taken to get this running from MSBuild or CCNET rather than your regular NUnit test running is this all down to Racer being disabled? if so it's a real letdown.
I've got a few more things to try Racer with and the nice people at Typemock suggested I try Isolator in combination so I'm going to give that a try when time allows. As it happens I want to give Isolator a try on the .NET micro framework stuff - it's not on the supported list which probably means I won't get very far, just like what happened when I tried redgate's Ants performace and memory profilers with the micro framework which was really sad.
(1) Akubio are no longer in business and the RAPid 4 has been taken over by TTP Labtech.