Automating end-to-end NServiceBus tests with NServiceBus.AcceptanceTesting

Photo Credit: LoveInTheWinter via Compfight cc

Most of you will agree that automating software tests is a good idea. Writing unit tests is almost a no brainer nowadays, and I’m a big fan of Behavior Driven Development and the use of Cucumber to bring together system analysts, programmers and testers more closely. The closer your tests and documentation are to the actual software, the better, IMO.

Repeatable and automated functional tests are paramount to guarantee the quality of a constantly evolving software system. Especially when things become more and more complex, like in distributed systems. As you may know I’m a fan of NServiceBus, and testing our NServiceBus message based systems end-to-end has always been a bit cumbersome. The fine folks at Particular Software – creators of NServiceBus – have found a nice way to do their own integration and acceptance tests, and you can use that too!

The framework that supports this is somewhat of a hidden gem in the NServiceBus stack, and I know that the Particular team is still refining the ideas. Nonetheless, you can use it yourself. It’s called NServiceBus.AcceptanceTesting. Unfortunately it’s somewhat undocumented so it’s not easily discovered and not very easy to get started with. You’ll need to dive into the acceptance tests in the NServiceBus source code to find out how it works. This can be a little bit hairy because there’s a lot going on in these tests to validate all the different transports, persistence, behavior pipeline and messaging scenarios that NServiceBus supports. This means that there is a lot of infrastructure code in the NServiceBus acceptance test suite as well to facilitate all the different scenarios. How to distinguish between what’s in the AcceptanceTesting framework and what’s not?

As a sample, I created a simpler scenario with two services and added a couple of acceptance tests to offer a stripped down application of the AcceptanceTesting framework. You can find the full solution on GitHub, but I’ll give a step by step breakdown below.

The scenario
The sample scenario consists of two services: Sales and Shipping. When the Sales service receives a RegisterOrder command – say from a web front end – it does some business logic (e.g. validate if the amount <= 500) and decides whether the order is accepted or refused. Sales will publish an event accordingly: either OrderAccepted or OrderReceived. The Shipping service subscribes to the OrderAccepted event. It will ship the order as soon as it is accepted and publish an OrderShipped event. Like so:

I’m sure it won’t bring me the Nobel prize for software architecture, but that’s not the point. From a testing perspective, we’d like to know if a valid order actually gets shipped, and if an invalid order is refused (and not shipped).

Project setup
Once you have your solution set up with a Messages library, and the implementation projects for your message handlers, we’ll add a test project for our acceptance tests. You can use your favourite unit test framework, I chose MSTest in my sample.

Next, in your test project, add a reference to the NServiceBus.AcceptanceTesting package via the Package Manager Console:

Install-Package NServiceBus.AcceptanceTesting

This will pull down the necessary dependencies for you to start writing acceptance tests.

Writing a test
Let’s have a look at one of the tests I have implemented in my sample:

[TestMethod]
public void Order_of_500_should_be_accepted_and_shipped()
{
    Scenario.Define(() => new Context { })
        .WithEndpoint<Sales>(b => 
            b.Given((bus, context) =>
                // The SubscriptionBehavior will monitor for incoming subscription messages
                // Here we want to track if Shipping is subscribing to our the OrderAccepted event
                SubscriptionBehavior.OnEndpointSubscribed(s => 
                {
                    if (s.SubscriberReturnAddress.Queue.Contains("Shipping"))
                    {
                        context.ShippingIsSubscribed = true;
                    }
                }))
                // As soon as ShippingIsSubscribed (guarded by the first expression), we'll
                // fire off the test by sending a RegisterOrder command to the Sales endpoint
            .When(c => c.ShippingIsSubscribed, bus => bus.Send<RegisterOrder>(m =>
                {
                    m.Amount = 500;
                    m.CustomerName = "John";
                    m.OrderId = 1;
                }))
         )
        // No special actions for this endpoint, it just has to do its work
        .WithEndpoint<Shipping>() 
        // The test succeeds when the order is accepted by the Sales endpoint,
        // and subsequently the order is shipped by the Shipping endpoint
        .Done(context => context.OrderIsAccepted && context.OrderIsShipped && !context.OrderIsRefused)
        .Run();
}

Whoa, that’s a lot of fluent API shizzle! That’s just one statement with a bunch of lambda’s, mind you. Let’s break it down to see what we have here…

The AcceptanceTesting harness runs a scenario, as denoted by the Scenario class. The basic skeleton looks like this:

[TestMethod]
public void Order_of_500_should_be_accepted_and_shipped()
{
    Scenario.Define(() => new Context { })

        .WithEndpoint<Sales>()

        .WithEndpoint<Shipping>() 

        .Done(context => context.OrderIsAccepted && context.OrderIsShipped && !context.OrderIsRefused)

        .Run();
}

A scenario is defined using the Define method, which receives an instance of a class named Context. Next, the WithEndpoint() generic methods help us setup the different endpoints that participate in the current test scenario. In this case: Sales and Shipping. We’ll have a look at the types used here later.

Before the scenario is kicked off with the Run() method, we define a condition that indicates when the test has succeeded and pass that to the Done() method.

The expression looks like this:

context.OrderIsAccepted && context.OrderIsShipped && !context.OrderIsRefused

We’re evaluating a bunch of properties on an object named context. This is actually the instance of the Context class we saw being passed to the Scenario.Define() method. The context class looks like this:

class Context : ScenarioContext
{
  public bool OrderIsAccepted { get; set; }
  public bool OrderIsRefused { get; set; }
  public bool OrderIsShipped { get; set; }
  public bool ShippingIsSubscribed { get; set; }
}

It inherits from ScenarioContext, a base class in the NServiceBus.AcceptanceTesting framework, and it’s just a bunch of properties that get passed around throughout our test scenarios to keep track of the progress. The trick is to set these properties at specific moments as your test runs and as soon as the conditions are met, the test is considered a success.

In the example above, we expect that the order is accepted and shipped, and we also double check that it wasn’t refused. We can assert this by tracking the events being published.

The next piece of the puzzle is the definition of the endpoints that participate in the test:

.WithEndpoint<Sales>()

The type parameter in this case is a class called Sales. This class represents the Sales endpoint, but is actually defined in the test code. This is what it looks like:

public class Sales : EndpointConfigurationBuilder
{
  public Sales()
  {
    EndpointSetup<DefaultServer>()
    // Makes sure that the RegisterOrder command is mapped to the Sales endpoint
      .AddMapping<RegisterOrder>(typeof(Sales));
  }

  class SalesInspector : IMutateOutgoingMessages, INeedInitialization
  {
    // Will be injected via DI
    public Context TestContext { get; set; }

    public object MutateOutgoing(object message)
    {
      if (message is OrderAccepted)
      {
        TestContext.OrderIsAccepted = true;
      }

      if (message is OrderRefused)
      {
        TestContext.OrderIsRefused = true;
      }

      return message;
    }

    public void Customize(BusConfiguration configuration)
    {
       configuration.RegisterComponents(c => c.ConfigureComponent<SalesInspector>(DependencyLifecycle.InstancePerCall));
    }
  }
}

The Sales class derives from EndpointConfigurationBuilder, and is our bootstrap for this particular endpoint. The class itself doesn’t do much, except bootstrapping the endpoint by specifying an endpoint setup template – a class named DefaultServer – and making sure that the RegisterOrder message is mapped to its endpoint.

We also see a nested class called SalesInspector, which is an NServiceBus MessageMutator. We are using the extensibility of NServiceBus to plug in hooks that help us track the progress of the test. In this case, the mutator listens for outgoing messages – which would be OrderAccepted or OrderRefused for the Sales endpoint – and sets the flags on the scenario context accordingly.

This is all wired up through the magic of type scanning and the use of the INeedInitialization interface. This happens through the endpoint setup template class: DefaultServer. I actually borrowed most of this code from the original NServiceBus code base, but stripped it down to just use the default stuff:

/// <summary>
/// Serves as a template for the NServiceBus configuration of an endpoint.
/// You can do all sorts of fancy stuff here, such as support multiple transports, etc.
/// Here, I stripped it down to support just the defaults (MSMQ transport).
/// </summary>
public class DefaultServer : IEndpointSetupTemplate
{
  public BusConfiguration GetConfiguration(RunDescriptor runDescriptor, 
                                EndpointConfiguration endpointConfiguration,
                                IConfigurationSource configSource, 
                                Action<BusConfiguration> configurationBuilderCustomization)
  {
    var settings = runDescriptor.Settings;

    var types = GetTypesToUse(endpointConfiguration);

    var config = new BusConfiguration();
    config.EndpointName(endpointConfiguration.EndpointName);
    config.TypesToScan(types);
    config.CustomConfigurationSource(configSource);
    config.UsePersistence<InMemoryPersistence>();
    config.PurgeOnStartup(true);

    // Plugin a behavior that listens for subscription messages
    config.Pipeline.Register<SubscriptionBehavior.Registration>();
    config.RegisterComponents(c => c.ConfigureComponent<SubscriptionBehavior>(DependencyLifecycle.InstancePerCall));

    // Important: you need to make sure that the correct ScenarioContext class is available to your endpoints and tests
    config.RegisterComponents(r =>
    {
      r.RegisterSingleton(runDescriptor.ScenarioContext.GetType(), runDescriptor.ScenarioContext);
      r.RegisterSingleton(typeof(ScenarioContext), runDescriptor.ScenarioContext);
    });

    // Call extra custom action if provided
    if (configurationBuilderCustomization != null)
    {
      configurationBuilderCustomization(config);
    }

    return config;
  }

  static IEnumerable<Type> GetTypesToUse(EndpointConfiguration endpointConfiguration)
  {
    // Implementation details can be found on GitHub
  }
}

Most of this code will look familiar: it uses the BusConfiguration options to define the endpoint. In this case, the type scanner will look through all referenced assemblies to find handlers and other NServiceBus stuff that may participate in the tests.

Most notable is the use of the SubscriptionBehavior class, which is plugged into the NServiceBus pipeline that comes with NServiceBus 5.0 – watch the NServiceBus Lego Style talk by John and Indu at NSBCon London for more info. This behavior simply listens for subscription messages from endpoints and raises events that you can hook into. This is necessary for our tests to run successfully because the test can only start once all endpoints are running and subscribed to the correct events. The behavior class is not part of the NServiceBus.AcceptanceTesting framework though. IMO, it would be handy if Particular moved this one to the AcceptanceTesting framework as I think you’ll be needing this one a lot. Again, I borrowed the implementation from the NServiceBus code base:

class SubscriptionBehavior : IBehavior<IncomingContext>
{
  public void Invoke(IncomingContext context, Action next)
  {
    next();
    var subscriptionMessageType = GetSubscriptionMessageTypeFrom(context.PhysicalMessage);
    if (EndpointSubscribed != null && subscriptionMessageType != null)
    {
      EndpointSubscribed(new SubscriptionEventArgs
      {
        MessageType = subscriptionMessageType,
        SubscriberReturnAddress = context.PhysicalMessage.ReplyToAddress
      });
    }
  }

  static string GetSubscriptionMessageTypeFrom(TransportMessage msg)
  {
    return (from header in msg.Headers where header.Key == Headers.SubscriptionMessageType select header.Value).FirstOrDefault();
  }

  public static Action<SubscriptionEventArgs> EndpointSubscribed;

  public static void OnEndpointSubscribed(Action<SubscriptionEventArgs> action)
  {
    EndpointSubscribed = action;
  }

  internal class Registration : RegisterStep
  {
    public Registration()
      : base("SubscriptionBehavior", typeof(SubscriptionBehavior), "So we can get subscription events")
    {
      InsertBefore(WellKnownStep.CreateChildContainer);
    }
  }
}

Okay, almost done. We have our endpoint templates set up, message mutators listening to the relevant outgoing messages and SubscriptionBehavior to make sure the test is ready to run. Let’s get back to the part that actually makes the whole scenario go:

    Scenario.Define(() => new Context { })
        .WithEndpoint<Sales>(b => 
            b.Given((bus, context) =>
                // The SubscriptionBehavior will monitor for incoming subscription messages
                // Here we want to track if Shipping is subscribing to our the OrderAccepted event
                SubscriptionBehavior.OnEndpointSubscribed(s => 
                {
                    if (s.SubscriberReturnAddress.Queue.Contains("Shipping"))
                    {
                        context.ShippingIsSubscribed = true;
                    }
                }))
                // As soon as ShippingIsSubscribed (guarded by the first expression), we'll
                // fire off the test by sending a RegisterOrder command to the Sales endpoint
            .When(context => context.ShippingIsSubscribed, bus => bus.Send<RegisterOrder>(m =>
                {
                    m.Amount = 500;
                    m.CustomerName = "John";
                    m.OrderId = 1;
                }))
         )
   ...

For the Sales endpoint, we specified a whole bunch of extra stuff. First, there’s the event handler for the SubscriptionBehavior.OnEndpointSubscribed event. Here, the Sales endpoint basically waits for the Shipping endpoint to subscribe to the events. The context is available here as well, part of the lambda that’s passed to the Given() method, so we can flag the subscription by setting a boolean.

The final piece is the guard passed to the When() method. This is monitored by the AcceptanceTesting framework as the test runs and as soon as the specified condition is met, we can use the bus instance available there to send a message to the Sales endpoint: the RegisterOrder command will trigger the whole process we’re testing here. We’re sending an order of $500, which we expect to be accepted and shipped. There’s a test that checks the refusal of an order > 500 in the sample as well.

Some tips
For your end-to-end tests, you will be pulling together DLL’s from all of your endpoints and with all of your message definitions. So it makes sense to setup a separate solution or project structure for these tests instead of adding it to an existing solution.

If your handlers are in the same DLL as your EndpointConfig class, the assembly scanner will run into trouble, because it will find multiple classes that implement IConfigureThisEndpoint. While you can intervene in how the assembly scanner does its work (e.g. manually filtering out specific DLL’s per endpint definition), it might be better to keep your handlers in separate assemblies to make acceptance testing easier.

As you see, you need to add some infrastructural stuff to your tests, such as the EndpointConfigurationBuilder classes and the IEndpointSetupTemplate class for everything to work properly. You can implement this infrastructure stuff per test or per test suite, but you might want to consider creating some more generic implementations that you can reuse across different test suites. IMO the DefaultServer implementation from the NServiceBus tests is also a nice candidate for becoming a part of the NServiceBus.AcceptanceTesting package to simplify your test code as this is already a very flexible implementation.

Conclusion
As you see, the NServiceBus.AcceptanceTesting framework takes some time to get to know properly. There’s more advanced stuff in there which I didn’t cover in this introduction. Now if you dive into the scenarios implemented by Particular themselves, you’ll find inspiration for testing other bits of your system.

I like the model a lot, and I think this can save a lot of time retesting many end-to-end scenarios.