Automating end-to-end NServiceBus tests with NServiceBus.AcceptanceTesting

Photo Credit: LoveInTheWinter via Compfight cc
Photo Credit: LoveInTheWinter via Compfight cc
Most of you will agree that automating software tests is a good idea. Writing unit tests is almost a no brainer nowadays, and I’m a big fan of Behavior Driven Development and the use of Cucumber to bring together system analysts, programmers and testers more closely. The closer your tests and documentation are to the actual software, the better, IMO.

Repeatable and automated functional tests are paramount to guarantee the quality of a constantly evolving software system. Especially when things become more and more complex, like in distributed systems. As you may know I’m a fan of NServiceBus, and testing our NServiceBus message based systems end-to-end has always been a bit cumbersome. The fine folks at Particular Software – creators of NServiceBus – have found a nice way to do their own integration and acceptance tests, and you can use that too!

The framework that supports this is somewhat of a hidden gem in the NServiceBus stack, and I know that the Particular team is still refining the ideas. Nonetheless, you can use it yourself. It’s called NServiceBus.AcceptanceTesting. Unfortunately it’s somewhat undocumented so it’s not easily discovered and not very easy to get started with. You’ll need to dive into the acceptance tests in the NServiceBus source code to find out how it works. This can be a little bit hairy because there’s a lot going on in these tests to validate all the different transports, persistence, behavior pipeline and messaging scenarios that NServiceBus supports. This means that there is a lot of infrastructure code in the NServiceBus acceptance test suite as well to facilitate all the different scenarios. How to distinguish between what’s in the AcceptanceTesting framework and what’s not?

As a sample, I created a simpler scenario with two services and added a couple of acceptance tests to offer a stripped down application of the AcceptanceTesting framework. You can find the full solution on GitHub, but I’ll give a step by step breakdown below.

The scenario
The sample scenario consists of two services: Sales and Shipping. When the Sales service receives a RegisterOrder command – say from a web front end – it does some business logic (e.g. validate if the amount <= 500) and decides whether the order is accepted or refused. Sales will publish an event accordingly: either OrderAccepted or OrderReceived. The Shipping service subscribes to the OrderAccepted event. It will ship the order as soon as it is accepted and publish an OrderShipped event. Like so:

NServiceBusAcceptanceTestScenario

I’m sure it won’t bring me the Nobel prize for software architecture, but that’s not the point. From a testing perspective, we’d like to know if a valid order actually gets shipped, and if an invalid order is refused (and not shipped).

Project setup
Once you have your solution set up with a Messages library, and the implementation projects for your message handlers, we’ll add a test project for our acceptance tests. You can use your favourite unit test framework, I chose MSTest in my sample.

Next, in your test project, add a reference to the NServiceBus.AcceptanceTesting package via the Package Manager Console:

Install-Package NServiceBus.AcceptanceTesting

This will pull down the necessary dependencies for you to start writing acceptance tests.

Writing a test
Let’s have a look at one of the tests I have implemented in my sample:

[TestMethod]
public void Order_of_500_should_be_accepted_and_shipped()
{
    Scenario.Define(() => new Context { })
        .WithEndpoint<Sales>(b => 
            b.Given((bus, context) =>
                // The SubscriptionBehavior will monitor for incoming subscription messages
                // Here we want to track if Shipping is subscribing to our the OrderAccepted event
                SubscriptionBehavior.OnEndpointSubscribed(s => 
                {
                    if (s.SubscriberReturnAddress.Queue.Contains("Shipping"))
                    {
                        context.ShippingIsSubscribed = true;
                    }
                }))
                // As soon as ShippingIsSubscribed (guarded by the first expression), we'll
                // fire off the test by sending a RegisterOrder command to the Sales endpoint
            .When(c => c.ShippingIsSubscribed, bus => bus.Send<RegisterOrder>(m =>
                {
                    m.Amount = 500;
                    m.CustomerName = "John";
                    m.OrderId = 1;
                }))
         )
        // No special actions for this endpoint, it just has to do its work
        .WithEndpoint<Shipping>() 
        // The test succeeds when the order is accepted by the Sales endpoint,
        // and subsequently the order is shipped by the Shipping endpoint
        .Done(context => context.OrderIsAccepted && context.OrderIsShipped && !context.OrderIsRefused)
        .Run();
}

Whoa, that’s a lot of fluent API shizzle! That’s just one statement with a bunch of lambda’s, mind you. Let’s break it down to see what we have here…

The AcceptanceTesting harness runs a scenario, as denoted by the Scenario class. The basic skeleton looks like this:

[TestMethod]
public void Order_of_500_should_be_accepted_and_shipped()
{
    Scenario.Define(() => new Context { })

        .WithEndpoint<Sales>()

        .WithEndpoint<Shipping>() 

        .Done(context => context.OrderIsAccepted && context.OrderIsShipped && !context.OrderIsRefused)

        .Run();
}

A scenario is defined using the Define method, which receives an instance of a class named Context. Next, the WithEndpoint() generic methods help us setup the different endpoints that participate in the current test scenario. In this case: Sales and Shipping. We’ll have a look at the types used here later.

Before the scenario is kicked off with the Run() method, we define a condition that indicates when the test has succeeded and pass that to the Done() method.

The expression looks like this:

context.OrderIsAccepted && context.OrderIsShipped && !context.OrderIsRefused

We’re evaluating a bunch of properties on an object named context. This is actually the instance of the Context class we saw being passed to the Scenario.Define() method. The context class looks like this:

class Context : ScenarioContext
{
  public bool OrderIsAccepted { get; set; }
  public bool OrderIsRefused { get; set; }
  public bool OrderIsShipped { get; set; }
  public bool ShippingIsSubscribed { get; set; }
}

It inherits from ScenarioContext, a base class in the NServiceBus.AcceptanceTesting framework, and it’s just a bunch of properties that get passed around throughout our test scenarios to keep track of the progress. The trick is to set these properties at specific moments as your test runs and as soon as the conditions are met, the test is considered a success.

In the example above, we expect that the order is accepted and shipped, and we also double check that it wasn’t refused. We can assert this by tracking the events being published.

The next piece of the puzzle is the definition of the endpoints that participate in the test:

.WithEndpoint<Sales>()

The type parameter in this case is a class called Sales. This class represents the Sales endpoint, but is actually defined in the test code. This is what it looks like:

public class Sales : EndpointConfigurationBuilder
{
  public Sales()
  {
    EndpointSetup<DefaultServer>()
    // Makes sure that the RegisterOrder command is mapped to the Sales endpoint
      .AddMapping<RegisterOrder>(typeof(Sales));
  }

  class SalesInspector : IMutateOutgoingMessages, INeedInitialization
  {
    // Will be injected via DI
    public Context TestContext { get; set; }

    public object MutateOutgoing(object message)
    {
      if (message is OrderAccepted)
      {
        TestContext.OrderIsAccepted = true;
      }

      if (message is OrderRefused)
      {
        TestContext.OrderIsRefused = true;
      }

      return message;
    }

    public void Customize(BusConfiguration configuration)
    {
       configuration.RegisterComponents(c => c.ConfigureComponent<SalesInspector>(DependencyLifecycle.InstancePerCall));
    }
  }
}

The Sales class derives from EndpointConfigurationBuilder, and is our bootstrap for this particular endpoint. The class itself doesn’t do much, except bootstrapping the endpoint by specifying an endpoint setup template – a class named DefaultServer – and making sure that the RegisterOrder message is mapped to its endpoint.

We also see a nested class called SalesInspector, which is an NServiceBus MessageMutator. We are using the extensibility of NServiceBus to plug in hooks that help us track the progress of the test. In this case, the mutator listens for outgoing messages – which would be OrderAccepted or OrderRefused for the Sales endpoint – and sets the flags on the scenario context accordingly.

This is all wired up through the magic of type scanning and the use of the INeedInitialization interface. This happens through the endpoint setup template class: DefaultServer. I actually borrowed most of this code from the original NServiceBus code base, but stripped it down to just use the default stuff:

/// <summary>
/// Serves as a template for the NServiceBus configuration of an endpoint.
/// You can do all sorts of fancy stuff here, such as support multiple transports, etc.
/// Here, I stripped it down to support just the defaults (MSMQ transport).
/// </summary>
public class DefaultServer : IEndpointSetupTemplate
{
  public BusConfiguration GetConfiguration(RunDescriptor runDescriptor, 
                                EndpointConfiguration endpointConfiguration,
                                IConfigurationSource configSource, 
                                Action<BusConfiguration> configurationBuilderCustomization)
  {
    var settings = runDescriptor.Settings;

    var types = GetTypesToUse(endpointConfiguration);

    var config = new BusConfiguration();
    config.EndpointName(endpointConfiguration.EndpointName);
    config.TypesToScan(types);
    config.CustomConfigurationSource(configSource);
    config.UsePersistence<InMemoryPersistence>();
    config.PurgeOnStartup(true);

    // Plugin a behavior that listens for subscription messages
    config.Pipeline.Register<SubscriptionBehavior.Registration>();
    config.RegisterComponents(c => c.ConfigureComponent<SubscriptionBehavior>(DependencyLifecycle.InstancePerCall));

    // Important: you need to make sure that the correct ScenarioContext class is available to your endpoints and tests
    config.RegisterComponents(r =>
    {
      r.RegisterSingleton(runDescriptor.ScenarioContext.GetType(), runDescriptor.ScenarioContext);
      r.RegisterSingleton(typeof(ScenarioContext), runDescriptor.ScenarioContext);
    });

    // Call extra custom action if provided
    if (configurationBuilderCustomization != null)
    {
      configurationBuilderCustomization(config);
    }

    return config;
  }

  static IEnumerable<Type> GetTypesToUse(EndpointConfiguration endpointConfiguration)
  {
    // Implementation details can be found on GitHub
  }
}

Most of this code will look familiar: it uses the BusConfiguration options to define the endpoint. In this case, the type scanner will look through all referenced assemblies to find handlers and other NServiceBus stuff that may participate in the tests.

Most notable is the use of the SubscriptionBehavior class, which is plugged into the NServiceBus pipeline that comes with NServiceBus 5.0 – watch the NServiceBus Lego Style talk by John and Indu at NSBCon London for more info. This behavior simply listens for subscription messages from endpoints and raises events that you can hook into. This is necessary for our tests to run successfully because the test can only start once all endpoints are running and subscribed to the correct events. The behavior class is not part of the NServiceBus.AcceptanceTesting framework though. IMO, it would be handy if Particular moved this one to the AcceptanceTesting framework as I think you’ll be needing this one a lot. Again, I borrowed the implementation from the NServiceBus code base:

class SubscriptionBehavior : IBehavior<IncomingContext>
{
  public void Invoke(IncomingContext context, Action next)
  {
    next();
    var subscriptionMessageType = GetSubscriptionMessageTypeFrom(context.PhysicalMessage);
    if (EndpointSubscribed != null && subscriptionMessageType != null)
    {
      EndpointSubscribed(new SubscriptionEventArgs
      {
        MessageType = subscriptionMessageType,
        SubscriberReturnAddress = context.PhysicalMessage.ReplyToAddress
      });
    }
  }

  static string GetSubscriptionMessageTypeFrom(TransportMessage msg)
  {
    return (from header in msg.Headers where header.Key == Headers.SubscriptionMessageType select header.Value).FirstOrDefault();
  }

  public static Action<SubscriptionEventArgs> EndpointSubscribed;

  public static void OnEndpointSubscribed(Action<SubscriptionEventArgs> action)
  {
    EndpointSubscribed = action;
  }

  internal class Registration : RegisterStep
  {
    public Registration()
      : base("SubscriptionBehavior", typeof(SubscriptionBehavior), "So we can get subscription events")
    {
      InsertBefore(WellKnownStep.CreateChildContainer);
    }
  }
}

Okay, almost done. We have our endpoint templates set up, message mutators listening to the relevant outgoing messages and SubscriptionBehavior to make sure the test is ready to run. Let’s get back to the part that actually makes the whole scenario go:

    Scenario.Define(() => new Context { })
        .WithEndpoint<Sales>(b => 
            b.Given((bus, context) =>
                // The SubscriptionBehavior will monitor for incoming subscription messages
                // Here we want to track if Shipping is subscribing to our the OrderAccepted event
                SubscriptionBehavior.OnEndpointSubscribed(s => 
                {
                    if (s.SubscriberReturnAddress.Queue.Contains("Shipping"))
                    {
                        context.ShippingIsSubscribed = true;
                    }
                }))
                // As soon as ShippingIsSubscribed (guarded by the first expression), we'll
                // fire off the test by sending a RegisterOrder command to the Sales endpoint
            .When(context => context.ShippingIsSubscribed, bus => bus.Send<RegisterOrder>(m =>
                {
                    m.Amount = 500;
                    m.CustomerName = "John";
                    m.OrderId = 1;
                }))
         )
   ...

For the Sales endpoint, we specified a whole bunch of extra stuff. First, there’s the event handler for the SubscriptionBehavior.OnEndpointSubscribed event. Here, the Sales endpoint basically waits for the Shipping endpoint to subscribe to the events. The context is available here as well, part of the lambda that’s passed to the Given() method, so we can flag the subscription by setting a boolean.

The final piece is the guard passed to the When() method. This is monitored by the AcceptanceTesting framework as the test runs and as soon as the specified condition is met, we can use the bus instance available there to send a message to the Sales endpoint: the RegisterOrder command will trigger the whole process we’re testing here. We’re sending an order of $500, which we expect to be accepted and shipped. There’s a test that checks the refusal of an order > 500 in the sample as well.

Some tips
For your end-to-end tests, you will be pulling together DLL’s from all of your endpoints and with all of your message definitions. So it makes sense to setup a separate solution or project structure for these tests instead of adding it to an existing solution.

If your handlers are in the same DLL as your EndpointConfig class, the assembly scanner will run into trouble, because it will find multiple classes that implement IConfigureThisEndpoint. While you can intervene in how the assembly scanner does its work (e.g. manually filtering out specific DLL’s per endpint definition), it might be better to keep your handlers in separate assemblies to make acceptance testing easier.

As you see, you need to add some infrastructural stuff to your tests, such as the EndpointConfigurationBuilder classes and the IEndpointSetupTemplate class for everything to work properly. You can implement this infrastructure stuff per test or per test suite, but you might want to consider creating some more generic implementations that you can reuse across different test suites. IMO the DefaultServer implementation from the NServiceBus tests is also a nice candidate for becoming a part of the NServiceBus.AcceptanceTesting package to simplify your test code as this is already a very flexible implementation.

keep-calm-you-passed-the-test

Conclusion
As you see, the NServiceBus.AcceptanceTesting framework takes some time to get to know properly. There’s more advanced stuff in there which I didn’t cover in this introduction. Now if you dive into the scenarios implemented by Particular themselves, you’ll find inspiration for testing other bits of your system.

I like the model a lot, and I think this can save a lot of time retesting many end-to-end scenarios.

Advertisements

11 thoughts on “Automating end-to-end NServiceBus tests with NServiceBus.AcceptanceTesting”

  1. Hi Roy.

    Thanks for this article, it is pretty much the only resource on the web for the acceptance testing suite. However, I downloaded your code and couldn’t get the tests to complete successfully. Tests timeout after 90 seconds. This is consistent with what I am noticing when writting my own tests in my project. If I use a single endpoint for testing everything works fine but as soon as I introduce another one it stops working and times out. I tried only testing your Sales endpoint and it seems to work but as soon as I add the Shipping endpoint it just times out.

    Any help appreciated!

    1. Hi Alex,

      Thanks for your comment and my apologies for my belated reply. I’m lagging behind on my blog comments… 😦

      I hope you managed to sort out your timeout issues. I remember I was having them too in the beginning, but not sure anymore what was causing them. One thing you may need to have an extra look at is your “Done” expression, and making sure that all flags involved in the expression are set correctly, otherwise the framework will wait until it times out.

    1. Hi Riccardo,

      Thanks for your comment. Sorry I’m lagging so behind with my blog comments… I’m not sure but I’d say that adding Jimmy’s MessageRouting involves not much more than adding a reference to the libraries in your endpoints and in the EndpointConfigurationBuilder, call the setup methods that enable the routing features.

  2. Hi Roy, how have you found this approach when testing a large number of endpoints collaborating together? If we tested our overall system using NServiceBus.AcceptanceTesting then we’d be looking at approx. 30+ endpoints under test (and this is looking to grow).

    Have you found this has provided many benefits over deploying the endpoints to a test environment and then running tests via sending messages to them? This is what we’ve been doing so far – we just subscribe to endpoints we are interested in getting events from and then assert that they are published within a defined time frame. As part of the test tear down activities we unsubscribe.

    1. Hi Mike,

      Thanks for your comment.

      I think actually deploying the endpoints is the ultimate test since you have all of the infrastructure involved. TBH I have only done this with ~10 endpoints in total, and each test only involved around 3 endpoints on average. I think if you keep the individual tests to smaller scenarios, your tests stay more maintainable. I think the notation used for the tests is a plus because of their expressiveness. It allowed me to quickly come up with more scenarios and edge cases and write them as relatively short unit tests. I can imagine that your 30+ endpoint end-to-end test scripts become quite elaborate…? Or did you write some sort of framework to facilitate writing tests like these?

      Deploying the endpoints together with the front end app was part of our process, but we did that during end-to-end acceptance testing, and usually with manually performed test scenarios.

      1. Hi Roy,

        The approach I’ve been using can be utilized for small clusters of related endpoints and also whole end to end tests as you say. We get much more benefit focusing on the former of these – though there’s very few of them as NSB’s unit testing framework is really good.

        I wouldn’t try to verify 30+ endpoints in one go via a single test until end to end testing as you say (even then a single scenario wouldn’t test all of these in one go), but then we don’t need to verify all the events that are being generated, it’s more a case that the overall flows are reaching their natural destination (e.g. an order has been billed and shipped) as we’ve verified more fine grained service collaborations earlier on.

        The approach we’ve used can be found here:
        https://github.com/mikegore1000/NServiceBus.IntegrationTesting

        I wouldn’t call it a framework as such, it’s pretty simple and only involves 3 classes, but has covered all the scenarios we’ve needed to test so far. The main downside compared to the approach in your article is that you have to deploy the endpoints on a target environment, rather than simply starting everything in the test process so it wouldn’t be as quick to run.

        At present we haven’t tried this for full end to end tests yet, just smaller collaborations between 3 endpoints.

  3. Hi there Roy, I found that it might be possible to use this instead of the nested loop logic. I tried this locally and seems to have done the same trick.

    IExcludesBuilder builder = AllAssemblies.Except(“ScenarioTests”);
    config.AssembliesToScan(builder);

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s