StreamInsight & Silverlight

In August 2008 I started to work on a project that used Silverlight client to plot charts based on the real time data. On server the data were imported from third party application into an SQL database and using SOAP services with periodical polling we loaded selected data into the Silverlight client.

In February 2010 I got the opportunity to have a look at our architecture again and solve the same task using the new Microsoft StreamInsight platform. To compare the new approach with the old one I decided to create a simple test application.

Diagram of the test application

Current architecture

If you have a look at the blue part of the diagram above you can see the architecture of our current application.

The third party application is replaced with DataGenerator. That’s a console application that generates about five thousand random valued records every second. Each record consists of a text symbol and a numeric value.

Then there is the block named SqlImporter. It’s another console application that periodically queries the DataGenerator for new data and inserts them into SQL database.

The last piece of functionality executed on the server is the SqlService. This is a standard WCF service that uses a request/response approach. Through this service Silverlight clients periodically requests data for specified symbol, returned data contain time and numeric value.

New architecture

The new solution is featured in the red part of the diagram. It is based on Microsoft StreamInsight, a new platform for complex event processing applications that can work with large volumes of data with very little latency. I used the latest currently available version, November 2009 CTP.

Another important difference is the use of duplex WCF service that can push the data from server to the Silverlight client. The client subscribes for a symbol only once and whenever new item with this symbol is processed by StreamInsight server it is automatically send to the Silverlight client application.

Silverlight client

The client application uses both services to get the data and render them into two charts with some extra tracing information, like average delay and number of service calls. Once the user picks a symbol from a drop down list both charts start to render the values. The blue one plots data loaded through SQL server and the red one values that were processed using StreamInsight.

Screenshot of the client application

StreamInsight application

Let’s return back to the moment when user picks the symbol and see what is happening. Silverlight client application uses duplex WCF service to subscribe for all records with this symbol. This starts a new StreamInsight query that processes all generated data, filters appropriate records and returns them to the WCF service that can push them to the client.

I used explicit server development model with server running as a separate process. Input adapter loads data from DataGenerator as XML using its API, parses them into individual records and inserts them into queue.

The actual query is very simple, it only filters records with specified symbol and passes them to the output adapter.

Output adapter is responsible for sending the values to the client using the duplex service. As the StreamInsight server runs in a different process I used .NET Remoting with named pipes to pass the records to the WCF service.

Server screenshot with DataGenerator, SqlImporter and StreamInsight server console

Running the test application

I uploaded my application to the company server in the UK and tested it from my laptop while working from our Prague office. With both old and new technologies I was able to get the data into my Silverlight client within less than one second after it was generated. From my test I can’t tell that any of the approaches were significantly better.

The StreamInsight server usually needed more time to initialize the query and therefore when I picked a symbol from the drop down list I was able to see the first records earlier using the SQL chart. But after few seconds the StreamInsight chart started to work at full speed and new data appeared on the red chart slightly faster than on the blue one.

It’s important to state that unlike this experiment our real application plots charts not only from real time but also historical data. And for this part there is nothing that StreamInsight could help us with. It can give you immediate insight into stream of real time data, but it’s not storing this data once they are processed. So even if we used StreamInsight, we would have to do the SQL part for historical data anyway.

And also I have to mention that StreamInsight supports many complex processing algorithms, but for our case I needed only very simple filtering. Using such a potent technology for a simple task like this seems almost inappropriate.