NSBCon 2014 recap

Standard

Update: Particular has put up an awesome recap page for NSBCon London. All session videos are available there as well! And be sure to check out the excellent intro video.

NSBCon 2014, the first official NServiceBus conference at Skills Matter in London was a great success. I really enjoyed the sessions and hallway discussions with the participants about how they were using NServiceBus. Mark and I had the opportunity to share our experiences with NServiceBus in a session. Here are some of my personal highlights.

Barbecoa
Bear with me, I’ll get to technical NServiceBus stuff, but being a foodie, I can’t resist posting food pics as well. But if you’re really boring and don’t care about great food, you can skip the fun.

This was my second visit to London and we had a little bit of time to spend in the evenings to explore London. I arrived a day earlier than Mark to attend the ADSD Unconference on June 26th. My colleagues Marcel and Sander were also in town for a large ALM project they’re doing, so we decided to meet up and go for diner. Gordon Ramsay was out of our league, but luckily Jamie Oliver was close by with his excellent BBQ/grill restaurant Barbecoa. Nice ambience and the dishes were simple yet very refined and tasty.

Jamie Oliver's Barbecoa

Wood plank-smoked duck, cherries, maple dressing, red mizuna and pecans

Unpulled pork, Caraway slaw, jalopeno cornbread

Barbecoa brownie, Raspberry & Pink Peppercorn Sorbet & Aerated Chocolate

Barbecoa's butchery



Starter: Wood plank-smoked duck, cherries, maple dressing, red mizuna and pecans – excellent and playful taste combination between the smoked duck, sweet cherries and earthy pecans

Main: Unpulled pork, Caraway slaw, jalopeno cornbread – wow this pork butt was tender, and the combination with the spicy jalopeno cornbread was excellent

Dessert: Barbecoa brownie, Raspberry & Pink Peppercorn Sorbet & Aerated Chocolate – I love the use of pepper or other spices with fruit, made the sorbet really come alive; and chocolate brownies… no need to say more

Right around the corner was Barbecoa’s Butchery, where all the meat is dry-aging. Very nice. After dinner, I had a quick stroll along the Thames and St. Paul’s Cathedral.

And the conference hadn’t even started :)

ADSD Unconference 
In February 2012, I attended Udi Dahan’s Advanced Distributed Systems Design course. Most attendees will confirm that this course is quite mind twisting if you have been brainwashed with the “Service Oriented Architecture = Web Services” dogma the whole time. Nowadays I think messaging and asynchronous systems are a bit hipper with all these cloud platforms taking off, but chaining web services into a ball of mud together was still all the rage at the time.

The unconference was an interesting way to get ADSD alumni together to discuss their experiences applying the techniques from the course. An unconference is an interesting format, where topics are determined by the participants, and then discussed in free format sessions.

We saw an impressive example of a composite web UI implementation by Lars Corneliussen from Faktum Software – BTW, I checked, but he’s not some distant Scandinavian cousin :) Anyway, we discussed UI composition, whereby data from separate services is combined at the UI level. One of the interesting challenges I see is applying this pattern with mobile apps. Given the latency and low bandwidth that mobile apps have to deal with, I think it’s better to do the composition at the API level and prepare highly optimized resources for the app to communicate with.

Andreas OhlundThe Particular team also facilitated a couple of great discussions. A discussion about Ops got me interested in Splunk, for holistic and pro-active monitoring of all sorts of events across a distributed system. I definitely need to check that out. Also, Indu Alagarsamy had a nice discussion about Routing Slips vs. Process Orchestration with Saga’s. Have a look at Jimmy Bogard’s blog for a great description of different Saga / messaging patterns. Danny Cohen coined the term “Bolshevik” for centralized process orchestration. I’m going to use that term from now on :)

One of the insights was that – in a way – routing slip is “Bolshevik” as well, as a routing slip also has a predefined, sequential route. I tend to agree with that, though the Routing Slip pattern can be useful for having messages flow across specific endpoints only.

I really enjoyed the nice discussions at the ADSD Unconference and appreciated the willingness to share experiences and lessons learned by all participants. Having them face-to-face in a small group also really helps.

London
Mark arrived shortly after I finished the unconference. We went for a nice long walk along the Thames to visit some highlights in London. Might as well make good use of your time, right?

St. Paul's Cathedral and Millennium Bridge

London skyline

Tower Bridge

Southwark Bridge


NSBCon Day 1
IMG_5954The first day of NSBCon started off with a nice breakfast at Skills Matter. What a great location and a nice environment for tech conferences like NSBCon. Udi Dahan kicked off the day with a presentation about the past, present and future of NServiceBus. There were a couple more old timers in the room that also started using NServiceBus at version 1.x, just like me :) Udi reminded us how awful the website and logo used to look at the time, and he sincerely apologized for ILMerge-ing all the external dependencies into NServiceBus. Yep, it was bad, but the team has made NServiceBus a very slick and solid product over the past few years!

Info Support was prominently visible as well as one of the event’s sponsors.

The rest of the program was a nice mixture of case studies with NServiceBus, technology deep dives and theory. It was nice to see how NServiceBus is used at big companies like Wonga (Charlie Baker’s session) to handle large amounts of payments and Spotlight (Dylan Beattie’s session) where it even plays a role in video encoding.

Most prevalent from almost all of the session was how important it is to have decent monitoring across your entire system. Luckily the new tools from Particular can come a long way in giving insight in a message driven system, but I think that you can’t do without decent, holistic monitoring. Dashboards that give insight into both the technical stuff that goes on in your system and functional checks.

This was also one of the points that Mark and I highlighted on our Best Practices session. It was an honor to present at this first official NServiceBus conference. We got some nice feedback afterwards. The hallway discussions afterwards are always so valuable.

James Lewis gave a great talk about Managing Microservices and Yves Goeleven taught us about using NServiceBus in an Azure cloud. I really like using the Azure platform, cloud architecture brings a whole set of new challenges to the table.

The day ended for us with a nice speaker diner. It was great to spend some time with the whole Particular team and the other speakers to share experiences. You don’t get that many NServiceBus users from across Europe together in one room that easily.

NSBCon day 2
After a nice espresso at the Goswell Road Coffee Shop, the second day started with a deep dive into the Particular Service Platform by Danny Cohen. He explained how the separate components work together to monitor, diagnose and even design distributed systems with NServiceBus. I especially like the role of ServiceControl, which can serve as a nice extension point for your custom monitoring needs as well. New Relic feed, anyone? I have blogged about ServiceControl and ServicePulse before.

To use Danny’s words: “I’ll let you take in the coolness for a minute”

The other sessions were very enjoyable as well. A look at the new pipeline architecture in NServiceBus 5 by Indu Alagarsamy and John Simons was very nice. I really like the new model, based on the Russian Doll Pattern. Mark and I already saw some nice opportunities for replacing things in our implementation with these new style behaviors.

Greg Young and Szymon Pobiega showed an impressive integration of Event Store and NServiceBus. Event Store is another product I definitely want to check out. Greg is a great presenter as well.

Jan Ove Skogheim took us on the journey he made with his customer, migrating a complex, web service infested system to a message driven architecture with NServiceBus. Some nice insights there as well. In trying to get the team into the right mindset, Jan Ove was very keen on using the right terminology. “When someone said ‘service’, I slapped them in the face” :)

And Andreas Öhlund gave some nice insights into the internal development process at Particular. He quoted Netscape’s founder while explaining why releases are sometimes delayed: “Don’t ship crap”. I completely agree.

What struck me was that Andreas used this image in his presentation:

Slap!

So, both Jan Ove and Andreas use slapping… Perhaps some Scandinavian custom, but let me be the first to coin an official term for this methodology: Slap Driven Development. Interesting concept that I might try at work some time. You heard it first here!

In closing
NSBCon 2014 was a big success. Udi was visibly very proud to have a first official conference for his brainchild and rightly so. He has built a great company and community around NServiceBus. Looking forward to NSBCon 2015 already!

A lap around ServicePulse (2 of 3)

Standard

This is part 2 of my short series about the new Particular Service Platform… specifically about ServicePulse. In my first post about ServicePulse, you can read about the overall architecture and the problems that ServicePulse solves. Now I want to focus on one specific feature: the ability to add custom checks to your services.

After having set up ServiceControl and ServicePulse, you are able to monitor any NServiceBus service in your architecture. Just drop in the Heartbeat plugin and ServicePulse will see your service. The Heartbeat basically says: the service is there, it’s running and it’s able to send messages (since the Heartbeat is sent through a message to the ServiceControl input queue).

In general, this will not be sufficient, as you will also want to know how the service is behaving from a functional point of view. Or maybe you even want to monitor some deeper technical dependencies, such as whether it can reach its database, whether the config is OK, whether it can reach that external web service you rely upon, et cetera.

This is where custom checks come in, and they’re easy to implement. A custom check is very similar to the Heartbeat check: you just drop in a DLL that contains one or more checks, and the messages will be sent to ServiceControl. We end up with the following deployment overview:

ServiceControl-Deployment-CustomChecks

The custom check plugins will report through the same channel as the Heartbeat plugin.

Implementing a custom check
A custom check is pretty easy to implement, and you can choose between two scenarios: a periodic check and a one-time check.

Either way, you need to add a reference to the ServiceControl.Plugin.CustomChecks Nuget package and you’re good to go.

Startup check
A Startup custom check is executed when the service starts, so basically only once. A check like this is useful if you want to check some configuration settings after deployment, or if you want to check some environment dependencies that the service needs to run properly. You implement a one-time check by inheriting from ServiceControl.Plugin.CustomChecks.CustomCheck.

public class StartupCheck : ServiceControl.Plugin.CustomChecks.CustomCheck
{
    public StartupCheck()
        : base("StartupCheck", "Categoryname") // the name of the check and its category are specified here 
    {
        if (ConfigurationManager.AppSettings["TestSetting"] != "1")
        {
            ReportFailed("TestSetting must be 1!");
        }
        else
        {
            ReportPass();
        }
    }
}

The default constructor passes a check name and a category to the base constructor, which bootstraps the custom check. CustomCheck is an abstract base class, so you need to implement the StartupCheck method. All you have to do is put in your check, and call either ReportPass if all is well, or ReportFailed if there’s a problem. ReportFailed accepts a fault reason, which will be visible in ServicePulse.

Periodic check
A periodic check runs at a self defined interval, so it’s ideally suited to monitor things that might change over time, such as the availability of certain services or databases, or functional scenarios such as: “has all Point of Sale data arrived yet?”.

Again, implementing such a check is pretty easy. This time, inherit from ServicePulse.Plugin.CustomChecks.PeriodicCheck.

public class CheckHealth : PeriodicCheck
{
    public CheckHealth()
        : base("Healthcheck", "CategoryName", TimeSpan.FromMinutes(10))
    {

    }

    public override CheckResult PerformCheck()
    {
        // Fake a failure once in a while
        // TODO: think of a useful check to implement here.
        if (DateTime.Now.Second % 2 == 0)
        {
            return CheckResult.Failed("This is a sample failure report");
        }
         return CheckResult.Pass;
    }
}

Besides the check name and category name, the base constructor also excepts a TimeSpan. You can specify here at what interval the check runs. Each 10 minutes in the example.

Next, we override the PerformCheck method, which returns a CheckResult object. Case of success, report CheckResult.Pass, otherwise use CheckResult.Failed. Again, a reason or description for the feature must be supplied.

So what does admin/ops see?
After a custom check is deployed and activated, we can see the results in the ServicePulse front end. On the Dashboard, it shows:

Custom checks

And upon further inspection on the Custom Checks screen:

Custom checks - overview

In my current project, we make heavy use of custom checks to monitor the health of our system, and whether customers are efficiently using it. For now, we do this through a custom built monitoring service (based on NServiceBus), but I can see these migrating to ServiceControl plugins over time.

Next time: dealing with failed messages!

Meet the new Particular Service Platform… A lap around ServicePulse (1 of 3)

Standard

Most of you might have noticed that I’m a big fan of NServiceBus, a very nice .NET based framework for implementing service oriented systems, based on messaging. Particular Software, the company behind NServiceBus, has been working on building a platform around NServiceBus, which turns it into much more than just a developer’s framework.

The new additions to the platform are:

  • ServiceMatrix, a slick designer, embedded in Visual Studio to aid development of NServiceBus solutions
  • ServiceInsight, a debug/diagnostics tool that helps to track messages and message conversations
  • ServiceControl and ServicePulse, a dynamic duo that lets us monitor production environments

While these components can actually work together in a complete solution, I’ll focus on ServiceControl and ServicePulse in three consecutive blogposts. These two work together to provide overview and control over a running NServiceBus deployment in production. I’ll explain what each of these components do.

TL;DR – how to get up and running:

  1. Download and install ServiceControl
  2. Download and install ServicePulse
  3. Enable auditing on your NServiceBus endpoints
  4. Drop the Heartbeat ServiceControl plugin in your service’s bin folder and restart your service
  5. Start the ServicePulse front end
  6. Sit back and relax

Want more detail? First, let’s look at the problems ServiceControl and ServicePulse solve.

Distributed systems, distributed messages
The beauty – in my opinion – of NServiceBus is that it is a true implementation of the Service Bus architectural pattern, yielding a distributed system based on a decentralized messaging infrastructure. In essence, much like Ethernet, “the bus is everywhere”. With many independent, autonomous services running all over your data center, sending messages all over the place, how do you keep track of how it’s doing? How do you deal with messages that fail, and end up in an error queue in such a distributed architecture? With NServiceBus, there is no central broker through which all messages flow, which you could measure and monitor.

This has been a real challenge with NServiceBus up until now. I’ve seen many Roll Your Own solutions (including in my own project), and the use of 3rd party tools like QueueExplorer to deal with failed messages. Particular of course recognized this, resulting in the new tool suite, part of which is ServicePulse.

So what is ServicePulse?
ServicePulse is a system administrator’s best friend – a dashboard for monitoring the health and activity in a running NServiceBus ecosystem.

ServicePulse is basically a website, built with NancyFx that shows the status of your system in a nice dashboard overview:

ServicePulse-Dashboard-overview

What’s fun about this website, is that it updates in near realtime with the aid of SignalR. This is a nice example of the combination of NServiceBus and SignalR in action to provide “messaging” all the way into the front end. The two mix well together, I have blogged about it before as well.

ServicePulse architecture
Let’s dive into the architecture of this solution, and see how ServicePulse deals with the distributed nature of all our services.

As mentioned, there is no central broker in a distributed system that uses NServiceBus that you can easily stick a monitor on top. So, to get health information in a central dashboard, we need to somehow gather the information centrally.

Actually, ServicePulse is just a website that leverages another component in the Particular Service Platform as a backend: ServiceControl. ServiceControl is an NServiceBus service itself that exposes a REST API for consumers like ServicePulse. A typical deployment would look like this:

ServiceControl-Deployment-basic

One “best practice” with NServiceBus is to have a central error queue, so that you can monitor failed messages in one place. ServiceControl feeds off that error queue. Pretty smart, because this enables us to just “click on” ServiceControl like a LEGO piece.

Next, ServiceControl also wants to know about the activity in your system. It does this by feeding off the audit queue, much in the same fashion as the error messages. Having a centralized audit queue is also considered a good practice. And now we finally have a service that consumes all these messages and does something useful with it. :) Make sure to enable auditing on all services you want to monitor.

ServiceControl can listen to only one audit and one error queue and ServicePulse can only listen to one ServiceControl instance. This is OK if you follow these best practices, but if you for some reason decide to have a different deployment strategy, you need to account for this.

Note that ServiceControl consumes the messages off these queues, so both audit and failed messages are now under control of ServiceControl, which stores them in a RavenDB database. This means that you have to be careful with this data and make sure that it’s properly backed up. Remember, failed messages contain valuable business data! ServicePulse doesn’t do a lot with the audit messages, but ServiceInsight does. ServiceInsight is a different topic altogether. Also keep in mind that ServiceControl is not a long term storage solution. It will clean up the database with a configurable interval (e.g. 2 weeks). So if you need to adhere to compliancy rules that require you to keep an audit trail of messages for several years, make sure you regularly backup the database, or extract the messages from ServiceControl to store it in a durable place.

By the way, when I say “queue”, I don’t necessarily mean an MSMQ queue. Just like NServiceBus itself, ServiceControl is transport agnostic, so you can also use it when you do SQL Server transport, RabbitMQ or ZeroMQ, for example.

Of course you can also use the ServiceControl REST API yourself to get information from your system and report it to other monitoring environments. For example, I’m thinking of having some of this information published to our New Relic account. Something for another blogpost :)

Important to note: ServiceControl by default listens to the http://localhost:33333 endpoint. ServicePulse will try to access ServiceControl via that url directly from the browser. If you want to use ServicePulse from a remote browser, you will need to tweak these url’s and ports during installation. You can tweak this configuration as described on the Particular docs site.

Enabling monitoring on endpoints
So with this basic setup, ServiceControl can now pull in error and audit messages. If you were to fire up the ServicePulse dashboard now, you’d see messages along the lines of:

New ‘MyService.MyEndpoint’ endpoint detected at ‘HOSTNAME’. In order for this endpoint to be monitored the plugin needs to be installed.

“The plugin”? What plugin? Here comes the magic… :)

If you want your service to be monitored, it must send a heartbeat to ServiceControl. It does this by means of a heartbeat plugin, a component that you just drop in the bin directory of your service. It will automatically be picked up when the service is (re)started. It implements the IWantToRunWhenBusStartsAndStops interface, so it gets invoked by NServiceBus automatically.

The plugin is described on the Particular docs, but the gist is that you just download the ServiceControl.Plugin.Heartbeat Nuget component into your endpoint project. Alternatively, you can extract the DLL and just drop it in the deployment folder of your services.

The plugin reports to ServiceControl via messaging, just like your own services. It does so by reporting to the Particular.ServiceControl input queue. The deployment now looks like this:

ServiceControl-Deployment-heartbeat

Now, if a service goes down, ServicePulse will report this immediately:

Service down

The administrator can now react, fix the problem and bring the service back online.

Service back online

How cool!

Next up: Custom checks and Dealing with failed messages. Stay tuned!

Learning NServiceBus

Standard

Over the past two years, I’ve been architecting and building a system, applying design principles I’ve learned over the years doing software architecture at Info Support, and from the excellent “Advanced Distributed Systems Design” course by Udi Dahan. Now, I should say that these are no “one-size-fits-all” design principles; in fact there’s never a one-size-fits-all architecture for anything. But they had a natural fit with the problem domain that we are working on.

The two year timespan stems from the fact that we have been (and are still in the progress of) migrating the system from a classic ASP.NET three tier architecture to a more disconnected and distributed one, using messaging where applicable. This has helped us achieve more flexibility, better separation of concerns and an overall faster system for the end user. We’re not finished yet, as there’s a lot of legacy code left to migrate and all the while, the shop has to stay open, but we’re definitely getting there.

NServiceBus logoOne of the tools and frameworks that have been crucial in this re-architecture is NServiceBus. For those who don’t know NServiceBus, it is a light-weight, very elegant .NET framework, built around the principles of a distributed, service oriented architecture, using messaging. It’s designed by Udi Dahan and his team at Particular Software. In fact, NServiceBus is the direct result of the architectural lessons and experiences Udi has learned, architecting systems for customers over many years.

While NServiceBus is very light-weight, it still has a bit of a learning curve, to really understand the concepts and apply them correctly. Especially if you haven’t done the ADSD course mentioned earlier, some things might seem a bit abstract to a developer at first.

There’s always been documentation on the NServiceBus website, but not always up to date and a bit fragmented to my liking. There’s also the very helpful and thriving NServiceBus community group, where you can ask a great variety of questions to a great bunch of experts. There’s even a great NServiceBus course on PluralSight by Andreas Öhlund. But with NServiceBus 4.0 out, the course material is a bit stale on some of the technical stuff.

What was lacking most was a well structured, easy to follow guide on using and understanding NServiceBus, to be used as a companion when building your application with it. Fortunately, there’s one now: the book “Learning NServiceBus”, written by the awesome David Boike. David is an NServiceBus Champ, a very active contributor in the community forums and an allround knowledgeable software architect.

Learning NServiceBus book coverLearning NServiceBus is a great companion to building applications using NServiceBus, as well as administering and operating those applications. What I like best about the book is that it’s well structured, building up from the most basic “hello world” type scenario to the more complex capabilities, such as Sagas, and error handling & second level retries. All the while, it provides great background info on the architectural considerations behind NServiceBus. David has a pleasant, witty writing style which makes it easy to read. Not your average “dry” software development book, luckily.

You can also download the sample code used throughout the book, but I find it a better learning experience to build the samples from scratch as you read the book.

The book focuses on the all new NServiceBus 4.0 release, containing a bunch of important changes compared to 3.X, which makes it all the more worthwile to check out this book!

My congrats to David Boike on getting the book published. Great work and well recommended! You can get it from the publisher’s website.

Azure Service Bus Notification Hub for iOS + Bonus Xamarin Binding!

Standard

UPDATE: Xamarin now has a component in their component store for Azure Messaging, that uses the Notification Hubs described in this blogpost.

Recently I’ve been working with Edwin van Wijk on the brand new Azure Service Bus Notification Hub by Microsoft. It’s a unified messaging hub for delivering push notifications to mobile devices. It’s became GA in August 2013, and now supports all major mobile platforms, including iOS and Android.

Here’s a video of our presentation on this topic, at our weekly tech meetup at Info Support HQ (it’s in Dutch):

The service is quite nice, and is a cloud based contenter to products like PushSharp. You won’t need custom hosts for the integration, which is a benefit over PushSharp and the likes. However, in its current (Preview) version, it doesn’t support nearly as many platforms as PushSharp.

The Notification Hub builds upon the Azure Service Bus infrastructure for relaying messages in a massively scalable way. This means that it also leverages Service Bus’s topics and subscriptions feature for easily grouping messages using Tags. A nice way to filter messages and distinguish groups.

Edwin and I did a session on this. You can find the slides here. The slides are in Dutch by the way.

For iOS, Microsoft has released a native Objective-C library as part of their Azure SDK. There’s a tutorial on MSDN that nicely describes the steps you need to take to consume the library in XCode.

In order to test it, you’ll need to set up quite a lot of prerequisites, including a Windows Azure Service Bus instance, a Notification Hub, Apple AppID and an APNS certificate. A bit finicky to set up, but once it runs, it works nicely.

Xamarin.iOS binding
It’s quite logical for Microsoft to release an Objective-C library at first. Still, I felt a bit dissatisfied having to write my app in Objective-C. I have been wanting to experiment a bit more with creating a C# binding to Objective-C libraries, so I’ve decided to make one available for this library.

I have a working version up on GitHub, which includes a sample. It’s based on the latest API version – you’ll need a Notification Hub created after April 2013 for it to work. You’ll also need to setup your own certificates and fill in the Service Bus namespace, Notification Hub name and Shared Access Secret Listen key.

Basically, you’ll use the library in the RegisteredForRemoteNotifications method on your AppDelegate class. I have a write-up on that in an older blogpost here.

The current binding is a basic one. Completion handlers are mapped to delegates, and I think some of the type mappings could be more C# friendly. I’d also like to make it async/await friendly but I’ll have to experiment with that a bit more.

I’d love to hear your feedback on this!

[UPDATE August 13th, 2013: I've updated the Xamarin.iOS binding to the GA version!]