System architecture opinions sought

[TheRefactornator]

..in the .NET development arena.

Consider a system that has many distributed nodes gathering data that needs to be sent to a single central enterprise level system ("the core").

These remote data gathering nodes have the potential to send up to 5 transactions per second to the core for processing, but it's likely to be less. Each single transaction will be small serialized xml data, probably around 50 bytes.

The number of remote nodes in this application is 1000 so the potential max tps is 5000.

As a transaction is received at the core, it will be processed (de-serialized) and it's contents re-broadcast out to connected clients (the UI) so these clients are aware of the data that was originally sent from a remote node. The number of connected clients that will be interested in recieving any single transaction broadcast from the core will be 20 max.

I have already developed a C++/C# IOCP socket server as a windows service which I believe will provide the best performance and highest possible tps processing capability but I don't think any single socket server instance will be able to handle 5000 tps. Therefore I think there will be the need for some load balancing hardware up front to spread the transactions received at the core to a number of back-end application servers. The load balancing hardware will need to be in an active passive arrangement to prevent single point of failure at this level and I think it will also need to be intelligent enough to distribute the load of transactions based on the CPU activity of the target app servers.

Has anyone got any experience of developing a system of this size/type?
Does anyone have any good suggestions for intelligent load balancing hardware?

[AtW]

These remote data gathering nodes have the potential to send up to 5 transactions per second to the core for processing, but it's likely to be less. Each single transaction will be small serialized xml data, probably around 50 bytes.

A few words of advice:
1) use compact binary data rather XML
2) cache transactions on nodes and send them every second (so one packet with 5 transactions inside it - this will yield big performance savings).
3) use async sockets
4) preprocess data on nodes if possible (this way central server will need to do less work)

Your most expensive part will be XML fluff, avoid it - maybe only push it to your final 20 clients, but keep to simple binary for the rest.

SKA is written in .NET and it handles 200+ nodes sending data at sustained rate of 10 Mbits: CPU usage is nearly 0.

[TheRefactornator]

A few words of advice:
1) use compact binary data rather XML
2) cache transactions on nodes and send them every second (so one packet with 5 transactions inside it - this will yield big performance savings).
3) use async sockets
4) preprocess data on nodes if possible (this way central server will need to do less work)

Your most expensive part will be XML fluff, avoid it - maybe only push it to your final 20 clients, but keep to simple binary for the rest.

SKA is written in .NET and it handles 200+ nodes sending data at sustained rate of 10 Mbits: CPU usage is nearly 0.

Thanks for the advice. In response:

1) Do you know of any c# object-> serialize to binary -> deserialize back to object code I can look at? I've always used xml for this type of work in the past but you're right it is an overhead I can do without here.
2) Good advice. I already considered this would be a good optimization and it means that the potential 5000tps is now reduced to 1000tps.
3) My IOCP sockets method will be even more efficient than standard .NET async sockets so I'm happy with this part of the app.
4) I might be able to do some pre-processing at the data gathering nodes but I'm not sure of the processing power I'll have there yet. The node's main task is going to be servicing a number of serial (RS232) data hiways and the platform could well be an embedded device running windows CE / compact framework so in that case it could be up to the job.

Any thoughts on load balancing hardware? I'm aware of the Barracuda 640 that is able to handle 2000tps and redundant configuration for high availability http://www.barracudanetworks.com/ns/...cer_models.php but I would be interested to learn about other possible options.

[AtW]

1) Do you know of any c# object-> serialize to binary -> deserialize back to object code I can look at? I've always used xml for this type of work in the past but you're right it is an overhead I can do without here.

Don't use serialization at all - it's too heavy: convenient but you pay cost of performance.

Create your class that will know itself how to write/read data from binary stream, you can't possibly keep lots of data in 50 bytes.

2) Good advice. I already considered this would be a good optimization and it means that the potential 5000tps is now reduced to 1000tps.

Real size of packets is likely to be around 1500 bytes (that MTU ethernet thingy), calls, interrupts etc are all best minimised, so bigger packets are preferable to high number of packets, it all depends how real time you want the system to work: for most people 1 second is pretty real time.

3) My IOCP sockets method will be even more efficient than standard .NET async sockets so I'm happy with this part of the app.

Asyn .NET sockets already use completion ports - don't try to reinvent the wheel in this area - first version of SKA was using .NET 1.1 and they improved sockets in .NET 2.0 and also in .NET 3.5.

Any thoughts on load balancing hardware? I'm aware of the Barracuda 640 that is able to handle 2000tps and redundant configuration for high availability http://www.barracudanetworks.com/ns/...cer_models.php but I would be interested to learn about other possible options.

Can't help you with that - I write efficient enough software to avoid the need for load balancing...

One more thing - if you intend to do heavy data processing on main server make sure you do it in a separate thread with lower CPU priority than threads that deal with communications - never try to do heavy processing in socket callback.

HTH

P.S. Avoid using MemoryStream objects - this garbage collection "guarantee" that you won't have memory leaks is a pack of tulips.

[Churchill]

Real size of packets is likely to be around 1500 bytes (that MTU ethernet thingy), calls, interrupts etc are all best minimised, so bigger packets are preferable to high number of packets, it all depends how real time you want the system to work: for most people 1 second is pretty real time.

What is your definition of "Realtime" aTw?

[AtW]

What is your definition of "Realtime" aTw?

Actual length of "real-time" depends on a particular situation, in this case 1 second should be pretty real time (0.2 sec updates are not real time in embedded sense)

[DimPrawn]

With regards to the data, what happens if data is lost, do you need guaranteed delivery of data to the server? Retries? Is the order the data is received important?

Would bog std Windows Server NLB give you the scalability you require?

[TheRefactornator]

Actual length of "real-time" depends on a particular situation, in this case 1 second should be pretty real time (0.2 sec updates are not real time in embedded sense)

In fact the system being discussed here is a large distributed model of an existing smaller localized system where UI latency actually is 1 second. 1 second has always been perceived by the users and the system vendor to be real-time in this application.

[AtW]

In fact the system being discussed here is a large distributed model of an existing smaller localized system where UI latency actually is 1 second. 1 second has always been perceived by the users and the system vendor to be real-time in this application.

So looks like I got the situation right (term "real-time" is different in length depending on context) and our local dawggy got it wrong...

[TheRefactornator]

With regards to the data, what happens if data is lost, do you need guaranteed delivery of data to the server? Retries? Is the order the data is received important?

Would bog std Windows Server NLB give you the scalability you require?

It is not too critical if data is lost occasionally because the data gathering nodes will be running a loop forever gathering data from remote field devices over RS232/485 network, but it will only be only delta change that a transaction will be created for transport to the core. Obviously data loss should be minimal but if it happens it won't be a disaster..order of data received is not important at all.

The serial comms layer will implement it's own application level retry mechanism, but to keep TCP/IP network traffic to a minimum the socket layer is unlikely to be implemented in a request/response type arrangement. It's more likely to simply be one where the data gatherer packages up a handful of transactions, sends and forgets until next time.

I really have no idea about Windows Server NLB and it's capabilities but thanks for the pointer - it is something to consider.