Multicast Addressing Pitfalls

The IP multicast model has been described as follows: “You put packets in at one end, and the network conspires to deliver them to anyone who asks”. That sounds pretty simple indeed, but as you spend more time with applications that deliver multicast services you find that in many cases you will ‘put packets in at one end, look at the network protocol analyzer, and see nothing that you expect’. The truth of the matter is that getting multicast applications to work as expected requires deep understanding of the network topology and the underlying routing devices.

This post reviews the multicast addressing technology along with in detail illustrations of some of the pitfalls that are often encountered when deploying multicast applications.

In my next post you can find walkthrough of C# sample application that allows sending and receiving multicast data over the network.

How does it Work?

In multicast applications, the sender defines IP multicast group address (from 224.0.0.0 to 239.255.255.255) to which is send data packets. Receivers inform the network that they are interested in receiving data packets sent to a certain group by sending Internet Group Management Protocol (IGMP) package to the closest network node (router, IGMP querier). The node is in charge of maintaining multicast distribution trees such that data packets sent to a multicast group reach all receivers which have joined the group.

Why Use Multicast?

The great thing about multicast addressing is that in order to distribute a message to multiple receivers - the sender have to dispatch only one message over the wire, while it’s the network (e.g. intermediary routers) responsibility to copy and distribute the message to all the receivers in the group. In case there aren’t any receivers that had joined the group – the network drops the message.  The fact that the sender is unaware of the receivers greatly improves the application ability to scale to a large receiver population

The Pitfalls

Developing application that sends and receives multicast streams over the network is considered a fairly simple task. You can get pretty fast to the point where everything seems to work fine in the integration labs, were all the PCs communicate through a single network card and connected through few well known switches. The problems start when the application first meet the client network that is often spitted by VLANs and filled with all kinds of Cisco goodness. Unavoidably, you’ll find yourself steering at the network sniffer trying to figure out why you don’t see packages from group X even though the IGMP package was sent to the router, or why you do see packages from group Y - even though your application had never requested to join this particular group (No IGMP package was sent).

Where is my Multicast?

Multihomed Network

In case a host is connected to two or more networks or have two or more network cards installed and enabled – senders must explicitly state to which network interface (represented by the unicast IP address of the network card) they want to send the multicast traffic, and receivers must explicitly state from which network interface they want to receive multicast traffic. If they don’t – the operation system takes the liberty to choose the network interface, and the multicast traffic has 50% chance of reaching to the intended destination.

A common mistake that developers make is not binding the socket used for sending multicast traffic to the appropriate network card endpoint, or binding the socket used for receiving multicast to the address Any (0.0.0.0).

This mistake is so common because developers often assume that UDP unicast socket and UDP multicast sockets that are being used to send data can be implemented alike, and that sockets that are being used to receive multicast data can bind to IPAddress.Any (0.0.0.0). This is true when the host has single network card, but not true for host that lives in multihomed network.

Firewall

In case the machine firewall is turned on, and the firewall is NOT configured to relay transports from a certain multicast group (and corresponding UDP port) – your application will not receive messages from that group.

Inter VLAN

In case the sender and the receivers don’t sit on the same VLAN, the receivers will not receive messages from the sender.

Switch doesn’t Support IGMP Snooping

In case the sender is connected to a switch that support IGMP snooping (which means that it relay multicast messages only through ports which sent IGMP message), and the receiver is connected to another switch that doesn’t support IGMP snooping - the multicast relaying mechanism will "breaks down" in the absence of an mrouter port, thus the receiver will not receive the messages from the sender.

If you want a fix for this solution, you must have the switches somehow learn or know of an mrouter port. When the switches know their mrouter port, the right Switch (that doesn’t support IGMP snooping) relays out the IGMP report that it receives through its mrouter port. From the perspective of the left Switch, it received merely another IGMP report. The left switch adds that port into its IGMP snooping table and begins sending out multicast traffic on that port as well. At this point, the right Switch receive the multicast traffic, and the application works as expected.

Why am I getting this Multicast?

Broadcasting Multicast Traffic

Switches that cannot understand multicast addresses usually broadcast multicast traffic to all the members of a LAN. Machines that are connected to such switches will see multicast traffic from groups that they’ve never registered to.

 

Port Mirroring

In case the machine is connected to a router port that is configured as ‘port mirroring’ – the machine will receive multicast streams form all the groups.

Port mirroring, also known as a roving analysis port, is a method of monitoring network traffic that forwards a copy of each incoming and outgoing packet from one port of a network switch to another port where the packet can be studied. A network administrator uses port mirroring as a diagnostic tool or debugging feature, especially when fending off an attack. It enables the administrator to keep close track of switch performance and alter it if necessary. Port mirroring can be managed locally or remotely.

Multicasting over WAN

Source-specific multicast (SSM) make multicasting packages over the WAN eligible by reducing the amount of multicast routing information that the network must maintain. With SSM receivers supply the sender’s source address to the routers as a part of joining the group (by setting the receiver socket option MCAST_JOIN_SOURCE_GROUP).

A great presentation (ppt) that discusses multicast addressing can be downloaded from here.

Links

http://www.netcraftsmen.net/welcher/papers/multicast01.html

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008059a9df.shtml

.NET CLR Thread Pool Internals

In the previous post we saw how applications that need to scale up can utilize the Task Parallel Library and the improved CLR 4.0 thread pool in order to optimize the scheduling of their massive amount of fine grained, short living tasks.

In this post we’ll dive through the implementation of the CLR thread pool in an attempt to understand the way in which it optimizes the creation and destruction of threads in order to execute asynchronous callbacks in a scalable fashion.

CLR Thread Pool v.s Win32 Thread Pool

The CLR thread pool is the managed equivalent of win32 thread pool (that is available since Windows 2000). Although they expose different APIs and differ in design, implementation and internal structure – the basic operation manner remained the same. Application code queue callbacks that the thread pool execute asynchronously in a scalable fashion through heuristic algorithm.

The main advantages of the CLR thread pool are that 1) it includes separate pool for ordinary callbacks (user callback, timers etc) and separate pool for asynchronous I/O (which maintains a process wide I/O completion port).  And 2) it is aware of managed beasts such as the GC, that blocks the application threads during collection which can cause the unaware pool to create redundant threads.

Managed applications should use the CLR thread pool even on top of the Vista thread pool - that exposes richer API and performs a lot better than Win32 legacy pool. Also because the CLR thread pool is already used by many components that are integrated parts of the .NET Framework, such as Sockets,  FileWriter/Reader, System/Threading timers and more.

How does the CLR Thread Pool Work? and Why should We Care?

There’s a lot of truth in the saying that most applications are better of being designed with no respect to the internal operation fashion of the thread pool (since it’s subject to change). However, it’s important for the developers to get familiar with the internals of the thread pool since 1) nobody likes black boxes managing the execution of their code, 2) it might convince some to neglect thoughts about creating a custom thread pool, and 3) wrong utilization of the thread pool can result in starvation (due to too many callbacks running together) or on extreme cases deadlocks (due to callbacks relying on each other to complete).   

So...

Since the main overhead of asynchronous execution is the creation and destruction of threads - a key feature of the thread pool is to optimize the creation and destruction of threads.

The basic operation manner of the thread pool is actually pretty easy to explain. The thread pool starts from 0 threads, it immediately creates new threads to serve work requests until the number of running threads reaches a configurable minimum (the default minimum is set to the number of CPUs on the machine).

While the number of running threads is equal or bigger than that minimum – the thread pool will create new threads at the rate of 1 thread per 0.5 second. Which means that if your application is running on a dual core machine, and 16 work requests that spans for 10 seconds are scheduled together, assuming the the thread pool is empty, the first two request will be served immediately, the third after 0.5 second, the forth after 1 second and the 16th after 7 seconds. In addition, to avoid unnecessary starvation a demons thread is running in the background and periodically monitors the CPU – in case of low CPU utilization it creates new threads as appropriate.

The thread pool will not create new threads after it reaches a configurable maximum. The default maximum is set to 250 * number of CPUs, which is 1o times more than it was in the 1.0 version of the .NET Framework. The default maximum was increased in order to reduce the chance for possible dead lock that can occur when callbacks rely on each other in order to complete.

After a thread finish its work it is not being destroyed immediately, rather it stays in the pool waiting for another work to arrive. Once new work arrive it is being served immediately by one of the waiting threads. The waiting threads are being destroyed only after spending 10 seconds (was 40 seconds) on the pool doing nothing.

The following figure shows how the ‘thread injection and retirement algorithm’ of the thread pool works.

What the Future Holds?

In the 4.0 version of the CLR thread pool - the ‘thread injection and retirement algorithm’ has changed. In cases where there are more running threads than CPUs - instead of creating new threads at the rate of 1 thread per 0.5 second – the new thread pool takes the liberty to ‘hold back’ threads (i.e. reduce the concurrency level) for a while. 

In addition, the maximum threads count is NOT being set to some arbitrary number (like 20-250) that is big enough to help preventing dead locks, instead it’s being set to the maximum that the machine can handle in the limitation of the process virtual address space.

Each thread consumes 1M of committed bytes. So applications which allow infinite number of parallel tasks will sooner or later run out of contiguous virtual address space. Thus, the thread pool doesn’t allow the introduction of a new thread that will be the cause of OutOfMemory exception.