And a CTP is available now.

I first heard of the .NET Complex Event Processing engine at this year's TechEd. Until lately, all we had to go on was Torsten Grabs' TechEd presentation and a whitepaper that gave a fairly high-level overview of its purpose and architecture. And, oh, it was to be released as part of SQL Server 2008 R2. So last week when SQL Server 2008 R2 CTP3 was released, there was also an announcement that a CTP of StreamInsight was available. The CEP engine has a name. And there's code and libraries to work with.

Downloaded it last week and installed it. It has no dependencies on SQL Server 2008 R2; in fact it doesn't even use SQL Server at this point. The metadata is stored in a SQL Server CE database, and the data it's processing as a stream isn't particularly stored anywhere unless your event target stores it. There's an example of consuming data from and storing data in SQL Server, but that's not always the point.

At a quick first glance, it is vaguely reminiscent of SQL Server Notification Services with event sources and event targets, but the similarity ends there. In the Notification Services architecture, events are stored in the database and matched against subscriptions to produce notifications. The focus in SQLNS is on storing events, matching, and formatting notifications. Until the event is stored, it's not visible to the infrastructure, only to the event provider.

In StreamInsight, the focus is on processing and querying the data in the input stream, in real-time. The stream processing architecture is lightweight, and almost all of the use cases presented in the docs name "the ability to handle up to 100000 events per second for a large number of devices" as a goal. Storing events is an option, but the interesting part is being able to query the event stream in-flight and generate output events based on aggregations, outliers, or other characteristics of the stream. In this respect, StreamInsight resembles TIBCO events much more than SQLNS.

There is an input adapter architecture and an output adapter architecture. You can write your own event server, use their server infrastructure or imbed it in an existing application. There's even a model for reporting state changes in the stream (event source) to interested parties in the event sink to implement a pub-sub application. The development model is .NET-based and at the center of it is a LINQ provider over the event stream. You register compiled LINQ queries that execute over to stream and provide selection, projection, joins, aggregation, and some additional operators like timestamp modification. You can also extend the LINQ provider's functionality with your own user-defined functions.

Rather than continue to regurgitate the documentation (I'm doing this here as an exercise to make sure I "get it", feel free to disagree with or correct me), I'll urge you to download the CTP, run the examples, get comfortable with the architecture, and try writing a simple adapter to get a feel for it. There's a team blog and a forum where the team members hang out (I recognize some of the folks' names there) if you have questions or problems.

There's also a nice event flow debugger to give you visibility over how stream flow through the processing engine, queries, and adapters. But beware…the debugger needs a later OS like Vista, Windows7, or Windows Server 2008 to work. It looks like the debugger is using event tracing for Windows (ETW) as its trace vehicle, and because it uses the "newer" ETW (ETW-vnext was known once as Crimson, and introduced in the latest OSes) it needs one of these OSes to function.

So, happy stream processing!…