Meet the new Particular Service Platform… A lap around ServicePulse (1 of 3)

Standard

Most of you might have noticed that I’m a big fan of NServiceBus, a very nice .NET based framework for implementing service oriented systems, based on messaging. Particular Software, the company behind NServiceBus, has been working on building a platform around NServiceBus, which turns it into much more than just a developer’s framework.

The new additions to the platform are:

  • ServiceMatrix, a slick designer, embedded in Visual Studio to aid development of NServiceBus solutions
  • ServiceInsight, a debug/diagnostics tool that helps to track messages and message conversations
  • ServiceControl and ServicePulse, a dynamic duo that lets us monitor production environments

While these components can actually work together in a complete solution, I’ll focus on ServiceControl and ServicePulse in three consecutive blogposts. These two work together to provide overview and control over a running NServiceBus deployment in production. I’ll explain what each of these components do.

TL;DR – how to get up and running:

  1. Download and install ServiceControl
  2. Download and install ServicePulse
  3. Enable auditing on your NServiceBus endpoints
  4. Drop the Heartbeat ServiceControl plugin in your service’s bin folder and restart your service
  5. Start the ServicePulse front end
  6. Sit back and relax

Want more detail? First, let’s look at the problems ServiceControl and ServicePulse solve.

Distributed systems, distributed messages
The beauty – in my opinion – of NServiceBus is that it is a true implementation of the Service Bus architectural pattern, yielding a distributed system based on a decentralized messaging infrastructure. In essence, much like Ethernet, “the bus is everywhere”. With many independent, autonomous services running all over your data center, sending messages all over the place, how do you keep track of how it’s doing? How do you deal with messages that fail, and end up in an error queue in such a distributed architecture? With NServiceBus, there is no central broker through which all messages flow, which you could measure and monitor.

This has been a real challenge with NServiceBus up until now. I’ve seen many Roll Your Own solutions (including in my own project), and the use of 3rd party tools like QueueExplorer to deal with failed messages. Particular of course recognized this, resulting in the new tool suite, part of which is ServicePulse.

So what is ServicePulse?
ServicePulse is a system administrator’s best friend – a dashboard for monitoring the health and activity in a running NServiceBus ecosystem.

ServicePulse is basically a website, built with NancyFx that shows the status of your system in a nice dashboard overview:

ServicePulse-Dashboard-overview

What’s fun about this website, is that it updates in near realtime with the aid of SignalR. This is a nice example of the combination of NServiceBus and SignalR in action to provide “messaging” all the way into the front end. The two mix well together, I have blogged about it before as well.

ServicePulse architecture
Let’s dive into the architecture of this solution, and see how ServicePulse deals with the distributed nature of all our services.

As mentioned, there is no central broker in a distributed system that uses NServiceBus that you can easily stick a monitor on top. So, to get health information in a central dashboard, we need to somehow gather the information centrally.

Actually, ServicePulse is just a website that leverages another component in the Particular Service Platform as a backend: ServiceControl. ServiceControl is an NServiceBus service itself that exposes a REST API for consumers like ServicePulse. A typical deployment would look like this:

ServiceControl-Deployment-basic

One “best practice” with NServiceBus is to have a central error queue, so that you can monitor failed messages in one place. ServiceControl feeds off that error queue. Pretty smart, because this enables us to just “click on” ServiceControl like a LEGO piece.

Next, ServiceControl also wants to know about the activity in your system. It does this by feeding off the audit queue, much in the same fashion as the error messages. Having a centralized audit queue is also considered a good practice. And now we finally have a service that consumes all these messages and does something useful with it. :) Make sure to enable auditing on all services you want to monitor.

ServiceControl can listen to only one audit and one error queue and ServicePulse can only listen to one ServiceControl instance. This is OK if you follow these best practices, but if you for some reason decide to have a different deployment strategy, you need to account for this.

Note that ServiceControl consumes the messages off these queues, so both audit and failed messages are now under control of ServiceControl, which stores them in a RavenDB database. This means that you have to be careful with this data and make sure that it’s properly backed up. Remember, failed messages contain valuable business data! ServicePulse doesn’t do a lot with the audit messages, but ServiceInsight does. ServiceInsight is a different topic altogether. Also keep in mind that ServiceControl is not a long term storage solution. It will clean up the database with a configurable interval (e.g. 2 weeks). So if you need to adhere to compliancy rules that require you to keep an audit trail of messages for several years, make sure you regularly backup the database, or extract the messages from ServiceControl to store it in a durable place.

By the way, when I say “queue”, I don’t necessarily mean an MSMQ queue. Just like NServiceBus itself, ServiceControl is transport agnostic, so you can also use it when you do SQL Server transport, RabbitMQ or ZeroMQ, for example.

Of course you can also use the ServiceControl REST API yourself to get information from your system and report it to other monitoring environments. For example, I’m thinking of having some of this information published to our New Relic account. Something for another blogpost :)

Important to note: ServiceControl by default listens to the http://localhost:33333 endpoint. ServicePulse will try to access ServiceControl via that url directly from the browser. If you want to use ServicePulse from a remote browser, you will need to tweak these url’s and ports during installation. You can tweak this configuration as described on the Particular docs site.

Enabling monitoring on endpoints
So with this basic setup, ServiceControl can now pull in error and audit messages. If you were to fire up the ServicePulse dashboard now, you’d see messages along the lines of:

New ‘MyService.MyEndpoint’ endpoint detected at ‘HOSTNAME’. In order for this endpoint to be monitored the plugin needs to be installed.

“The plugin”? What plugin? Here comes the magic… :)

If you want your service to be monitored, it must send a heartbeat to ServiceControl. It does this by means of a heartbeat plugin, a component that you just drop in the bin directory of your service. It will automatically be picked up when the service is (re)started. It implements the IWantToRunWhenBusStartsAndStops interface, so it gets invoked by NServiceBus automatically.

The plugin is described on the Particular docs, but the gist is that you just download the ServiceControl.Plugin.Heartbeat Nuget component into your endpoint project. Alternatively, you can extract the DLL and just drop it in the deployment folder of your services.

The plugin reports to ServiceControl via messaging, just like your own services. It does so by reporting to the Particular.ServiceControl input queue. The deployment now looks like this:

ServiceControl-Deployment-heartbeat

Now, if a service goes down, ServicePulse will report this immediately:

Service down

The administrator can now react, fix the problem and bring the service back online.

Service back online

How cool!

Next up: Custom checks and Dealing with failed messages. Stay tuned!

Learning NServiceBus

Standard

Over the past two years, I’ve been architecting and building a system, applying design principles I’ve learned over the years doing software architecture at Info Support, and from the excellent “Advanced Distributed Systems Design” course by Udi Dahan. Now, I should say that these are no “one-size-fits-all” design principles; in fact there’s never a one-size-fits-all architecture for anything. But they had a natural fit with the problem domain that we are working on.

The two year timespan stems from the fact that we have been (and are still in the progress of) migrating the system from a classic ASP.NET three tier architecture to a more disconnected and distributed one, using messaging where applicable. This has helped us achieve more flexibility, better separation of concerns and an overall faster system for the end user. We’re not finished yet, as there’s a lot of legacy code left to migrate and all the while, the shop has to stay open, but we’re definitely getting there.

NServiceBus logoOne of the tools and frameworks that have been crucial in this re-architecture is NServiceBus. For those who don’t know NServiceBus, it is a light-weight, very elegant .NET framework, built around the principles of a distributed, service oriented architecture, using messaging. It’s designed by Udi Dahan and his team at Particular Software. In fact, NServiceBus is the direct result of the architectural lessons and experiences Udi has learned, architecting systems for customers over many years.

While NServiceBus is very light-weight, it still has a bit of a learning curve, to really understand the concepts and apply them correctly. Especially if you haven’t done the ADSD course mentioned earlier, some things might seem a bit abstract to a developer at first.

There’s always been documentation on the NServiceBus website, but not always up to date and a bit fragmented to my liking. There’s also the very helpful and thriving NServiceBus community group, where you can ask a great variety of questions to a great bunch of experts. There’s even a great NServiceBus course on PluralSight by Andreas Öhlund. But with NServiceBus 4.0 out, the course material is a bit stale on some of the technical stuff.

What was lacking most was a well structured, easy to follow guide on using and understanding NServiceBus, to be used as a companion when building your application with it. Fortunately, there’s one now: the book “Learning NServiceBus”, written by the awesome David Boike. David is an NServiceBus Champ, a very active contributor in the community forums and an allround knowledgeable software architect.

Learning NServiceBus book coverLearning NServiceBus is a great companion to building applications using NServiceBus, as well as administering and operating those applications. What I like best about the book is that it’s well structured, building up from the most basic “hello world” type scenario to the more complex capabilities, such as Sagas, and error handling & second level retries. All the while, it provides great background info on the architectural considerations behind NServiceBus. David has a pleasant, witty writing style which makes it easy to read. Not your average “dry” software development book, luckily.

You can also download the sample code used throughout the book, but I find it a better learning experience to build the samples from scratch as you read the book.

The book focuses on the all new NServiceBus 4.0 release, containing a bunch of important changes compared to 3.X, which makes it all the more worthwile to check out this book!

My congrats to David Boike on getting the book published. Great work and well recommended! You can get it from the publisher’s website.

Azure Service Bus Notification Hub for iOS + Bonus Xamarin Binding!

Standard

Recently I’ve been working with Edwin van Wijk on the brand new Azure Service Bus Notification Hub by Microsoft. It’s a unified messaging hub for delivering push notifications to mobile devices. It’s became GA in August 2013, and now supports all major mobile platforms, including iOS and Android.

Here’s a video of our presentation on this topic, at our weekly tech meetup at Info Support HQ (it’s in Dutch):

The service is quite nice, and is a cloud based contenter to products like PushSharp. You won’t need custom hosts for the integration, which is a benefit over PushSharp and the likes. However, in its current (Preview) version, it doesn’t support nearly as many platforms as PushSharp.

The Notification Hub builds upon the Azure Service Bus infrastructure for relaying messages in a massively scalable way. This means that it also leverages Service Bus’s topics and subscriptions feature for easily grouping messages using Tags. A nice way to filter messages and distinguish groups.

Edwin and I did a session on this. You can find the slides here. The slides are in Dutch by the way.

For iOS, Microsoft has released a native Objective-C library as part of their Azure SDK. There’s a tutorial on MSDN that nicely describes the steps you need to take to consume the library in XCode.

In order to test it, you’ll need to set up quite a lot of prerequisites, including a Windows Azure Service Bus instance, a Notification Hub, Apple AppID and an APNS certificate. A bit finicky to set up, but once it runs, it works nicely.

Xamarin.iOS binding
It’s quite logical for Microsoft to release an Objective-C library at first. Still, I felt a bit dissatisfied having to write my app in Objective-C. I have been wanting to experiment a bit more with creating a C# binding to Objective-C libraries, so I’ve decided to make one available for this library.

I have a working version up on GitHub, which includes a sample. It’s based on the latest API version – you’ll need a Notification Hub created after April 2013 for it to work. You’ll also need to setup your own certificates and fill in the Service Bus namespace, Notification Hub name and Shared Access Secret Listen key.

Basically, you’ll use the library in the RegisteredForRemoteNotifications method on your AppDelegate class. I have a write-up on that in an older blogpost here.

The current binding is a basic one. Completion handlers are mapped to delegates, and I think some of the type mappings could be more C# friendly. I’d also like to make it async/await friendly but I’ll have to experiment with that a bit more.

I’d love to hear your feedback on this!

[UPDATE August 13th, 2013: I've updated the Xamarin.iOS binding to the GA version!]

Xamarin Evolve 2013 – Day 4

Standard

And so, Evolve 2013 has come to an end. Time flies when you’re having fun. This was day 4 for us in Austin. Four days of Xamarin mobile goodness.

This was another interesting day. It started off with a session by Josh Clark from Global Moxie, and author of Tapworthy on user interface design and touch. While his message – have users directly interact with content instead of indirections, such as buttons – wasn’t revolutionary, he had a couple of nice and inspiring examples to show us. Apps that have implemented some innovative UI’s. A nice session to start the day with.

Oh, and he had a picture of an OCD Chef’s cutting board… man I gotta have this in my kitchen :)

After that, I went to Marek Safar‘s session on Using Async in Mobile Apps. He shared some good tips and pointed out some pitfalls around SynchronizationContext when using Async.

Most of the day was spent in the hallways of the Hilton though, talking to people, meeting all the faces I have in my Twitter stream in person. This was awesome. Great to share a passion for mobile development with Xamarin and talk face to face about our experiences. Thanks Greg, John, Wally, Ben, Nic, Stuart, Miguel, Nat and all those I forgot to mention for catching up.

Of course, Marcel (@marcelv) and I did our own session titled “Sharing 80% of Code on iOS, Android and Windows Phone”. We decided to give our own take on code sharing, and how to achieve this in a low barrier, simple way. We spent a lot of time during the conference, speaking to developers with all different levels of coding skills and we found that a lot of them were impressed by some of the frameworks and approaches that were being presented for achieving code reuse, but for whom all of this went a bit over their head. These frameworks tend to hide a lot of clever tricks, and you also run the risk of over-abstracting. So we took our lessons learned from a real life example app we built for a customer, and explained some small but clever architecture and coding tricks.

We got great response afterwards, and got some requests for sharing some of our code. Marcel and I will be working on a blogpost, showing all of the constructs we presented today. Have an eye out for Marcel’s blog. We’ll put it up there and I’ll cross post it here as well.

I visited Nic Wise‘s session about MonoTouch.Dialog as well. It’s tough to be the last session of the day, but he did a good job. I’m not entirely convinced that MonoTouch.Dialog is a solution in every scenario, but it does provide a neat way to quickly code up an iOS UI.

So after that, it was time for the closing keynote already… Nat and Miguel announced Evolve 2014, and in order to show their commitment, five lucky birds in the audience won a ticket. Awesome, can’t wait for the next installment of this awesome conference.

In closing, I’d just like to give my compliments to the whole Xamarin staff and the production company for putting together an inspiring conference. I loved the attention to detail that was visible everywhere. I enjoyed the vibe, the great food, and the fantastic content.

See ya’ll next year!

Xamarin Evolve 2013 – Day 3

Standard

Day 3! Or actually Day 1 of the Evolve conference, to be exact. And whew what a day!

Of course the conference kicked off with the keynote by Nat and Miguel. What a well produced and slick show! A big stage, with gorgeous graphics on the big projector screens, awesome. You should definitely check out the video of the keynote here. We were proud to see Info Support there as Silver Sponsor:

Xamarin blew everyone away with a couple of great product announcements today. Only six weeks after releasing Xamarin 2.0, four major new features were announced:

C# 5.0 async/await
This C# compiler feature enables much cleaner syntax for async programming, resulting in better readable and shorter code. Moreover, less chance of bugs due to event handlers dangling around. Async/await was announced by Xamarin a short time ago as a beta, but available as of today in a final version.

F#
For those who like functional programming, F# is now a first class citizen in the Xamarin stack as well. I’m still new to F# but I’ll definitely check it out. A nice addition.

But the biggest features, that got the whole audience exploding were…

iOS User Interface Designer
WOW! This was some rabbit Miguel pulled out of his hat. An iOS UI designer embedded in Xamarin Studio, to replace having to resort to XCode’s Interface Builder. And it does not only replace it very well, but goes way beyond what IB does. Besides looking gorgeous, the designer tightly integrates with the code generator in Xamarin Studio. Adding controls to a view automatically adds member variables to the class. Actually they’re still outlets, but you won’t have to worry about them. So dragging lines from UI elements to Objective-C code files is over. I used to get so many smirks from Visual Studio developers when I demo Xamarin / XCode integration in my talks. This is all over now. No more explaining what an outlet is, because… as Miguel mentioned in his talk:

I am glad though, that Xamarin stayed closed to the Apple structure when it comes to the generated code. As far as I can see, the designer generates code that is compatible with code coming from the XCode integration. Controls are still marked with an [Outlet] attributes, etc. I have yet to try the designer myself, but I assume that XIB files and storyboards are instantly usable with the Xamarin designer.

Furthermore, any UI component that you might have in your solution, be it from the Xamarin Component Store, from a referenced DLL or self coded in your project, automatically becomes available in the designer. And what’s more, it shows a live preview of the control at design time!

I really didn’t expect this designer to be available so soon. I had seen screenshots and ideas from Miguel but I thought that it was still some time away from becoming a real product.

Again, kudos to the Xamarin team!

Like Miguel mentioned… If you ever think “Xamarin should do UMMM…”

Xamarin Test Cloud
Again, WOW! After the iOS designer, people weren’t expecting any more stuff. But then Nat came on and introduced the Xamarin Test Cloud. A fantastic new product that does automatic UI testing for virtually every smart phone or tablet device available on the market. Xamarin actually acquired LessPainful solutions for their infrastructure. Basically, Test Cloud is a test service that uses real, physical devices to run UI tests on, based on exploratory testing (i.e. hitting every feature in the app) or a prerecorded test script, expressed in Given, When, Then construct.

Here’s Xamarin’s own introduction page on Test Cloud: link

TechCrunch has written a nice introduction article on this as well.

So that was a shocker for the audience. Awesome. This is a major step for Xamarin though. With this completeness of vision, I think Xamarin will win over a lot of customers that were reluctant to come along with the Xamarin tools. Customers, at least the ones we’re servicing, were a bit reluctant about the maturity and completeness of the holistic vision and approach of mobile development. I think today, Xamarin nailed that question!

The sessions following after that were mostly good. I enjoyed Stuart Lodge’s session on MvvmCross, Sebastien Pouliot’s session on Advanced Build Mechanics for iOS and of course Scott Hanselman’s session, which was basically a nice celebration of the C# language.

And of course I enjoyed doing our own talk about creating a uniform login experience across devices. We’ll put the accompanying slides and source code online as soon as we have a chance.

Then the day ended with a Food Truck party in Brush Square Park across the street. Austin is famous for its food trucks, all specialized in a certain type of food. And they’re good! I had a Tex Mex style wiener, which was really nice and spicy, a nice corn flour tortilla wrap with chicken, cilantro and chilli. As a desert, there was ice cream. Not just regular stuff, but… salted chocolate and red velvet cookies. Heaven, for a foodie. Again, a great vibe among the conference visitors.

After the foodtrucks were closed, we got together at Maggie Mae’s for more partying. Austin has a nice vibe, there’s live music all around.

So tomorrow is the last day of the conference already. How time flies. There’s one more session for Marcel and me to do. This one will be about code sharing. We’ll dive into some of the more detailed, sometimes basic type of tricks and architectural patterns you can apply to achieve more code sharing but still leave the app easily maintainable.

See ya in our session tomorrow!

Xamarin Evolve 2013 – Day 2

Standard

Time flies. Day 2 of the Evolve conference has come to an end. Today was actually the second training day, and the conference itself has yet to start. Exciting!

The weather was nice again. Spent some time outside on the balcony, enjoying the sun and the warmth and a nice chat.

Austin

Today, the technical difficulties with the Wifi were gone so the training sessions went a bit smoother. Marcel and I actually started out in the Beginner’s track, that started with a session on navigation patterns. Some nice tips from Craig Dunn, who pointed us at the excellent Flyout Navigation and Sattellite Menu components, which provide new and modern navigation concepts for iOS apps.

After that, we jumped track back on the Advanced schedule for a module on Collection Views in iOS. The UICollectionView is a new UI paradigm in iOS, which drastically simplifies layouting collections of items in all sorts of forms. I recommend you take a look at this tutorial on the Xamarin docs site.

Speaking of docs, these last two days were a triumph for the Xamarin docs team. They wrote all the training material, which looks awesome and thorough. It is amazing how much great products and material is cranked out by Xamarin. The training material and the demo’s are exemplary of this.

The session I enjoyed the most was the one on Backgrounding, by Nina Vyedin. While she battled with the sound system that wouldn’t cooperate, she did a good job explaining the concepts around background tasks in both iOS and Android. Highlighting the similarities and differences, do’s and don’ts, before actually diving into code. All that with a couple of funny analogies to explain the concepts. Kudos.

Marcel and I had some work to do on our session, putting on the final touches, so we took some time in the afternoon to work on that. We did finish with the two modules on Code Sharing and Memory Management. Interesting talks. We’ll be speaking about our take on code sharing on Wednesday. The memory management module had some interesting tips on performance, something that’s very important in mobile apps. Mobile devices are not as powerful as laptop computers, so you have to be careful what you do with respect to memory usage and processor cycles.

The day ended with a fun and laid back welcome reception for the conference attendees. We had a couple of beers and some nice food and met some of the Xamarin folks. We decided to skip the GitHub Drinkup for tonight, but I hope everyone had fun there.

The reception featured a nice animation, projected on the big screen, that featured all nationalities present at Evolve. Very impressive. We learned that there are 11 Dutchies and I’m pretty sure that we’ve met them all :)

I haven’t gotten around to making more photo’s today, so these two will have to do. I did see another exciting bit of news fly by on Twitter today:

That’s awesome!

So tomorrow is going to be big! 200 additional attendees will arrive and join the 400 already here. The keynote by Nat and Miguel promises to be very exciting… Nat teased us in his welcome talk that there are going to be some very nice announcements. Can’t wait to find out!

I hope to see you guys in our session about Creating Unified Login for Apps. 2.30PM in the Gould Salon! Oh, and say hi to our colleague Henk, who’s manning the Info Support sponsor booth!