Enhanced Reporting

Flow Plots – a New Technique of Network Visualisation

nMetrics has been delivering methods for representing network performance information for a number of years. Traditionally much of this data is represented as line plots, histograms or pie charts to provide an overall view of who and what is generating network traffic.

Whilst these traditional approaches are useful and will remain an important component in representing network performance, they do not provide an effective mechanism to capture the patterns of flows that traverse most networks. Many of the anomalous performance characteristics that are of most interest to network operators are missed by these traditional reporting techniques.

nMetrics has devised a new approach of representing whole of network flow patterns to provide a much more meaningful insight into network performance. The first of these representations is a Flow Plot diagram and provides a detailed picture of the conversations between disparate hosts and subnets and the associated TCP/UDP port usage.

Unlike many network behaviour analysis systems which rely on predefined algorithms to detect anomalous activity, the nMetrics Flow Plot diagrams provide a visual snap shot of whole of network flow patterns that allow network operators to identify strange behaviour almost instantly.

As an extension, most network reporting tools provide pictures of network behaviour as a snapshot in time. nMetrics now provide the ability to replay multiple Flow Plot snapshots to allow for flow patterns over a long period of time to be replayed similar to watching a movie. This makes for a much more powerful analysis tool as it allows network operators to identify changes in network behaviour over time that may go unnoticed when looking at a single discrete snapshot.

To illustrate the power of the flow plot visualisations, a brief discussion on usage and interpretation is provided below.

The Flow Plot effectively tracks each IP flow across the network. That is, every connection or conversation between two discrete hosts constitutes an IP flow and is mapped on the Flow Plot Diagram. On the left and right hand axes are the originating subnets.
(click on the image to enlarge)


(click to enlarge)

The diagram on the left shows an axis for a typical flow plot diagram with the various subnets and groups defined. The Flow Plot uses the same subnet and groupings as defined on the Application Monitor. All flows are grouped within their logical subnet or groups to provide a more intuitive perspective on network flow patterns.

The left hand axis on the Flow Plot diagram represents the originating side of each flow which normally represents the client side of the IP connection.  

Note that there is also an “External” subnet configured. Flows that originate with addresses not defined within the Application Monitor are placed in the “External” subnet on the Flow Plot diagram. This often represents Internet based traffic or, at the very least, traffic originating from potentially unknown sources, and as such, is usually of specific interest. 

The centre axis shows the TCP/UDP or Layer 4 port number used by the IP flow. Most IP conversations are bi-directional. That is, made up of two uni-directional flows, one from client to server and a corresponding connection from server to client. The Application Monitor correlates the two uni-direction flows that constitute a single IP conversation and can determine direction accordingly. Under these conditions, only the originating Destination Port number is retained as it is the most significant and typically represents the application to which the flow belongs. For example, in a normal web connection between 2 hosts, the destination port will be Port 80 representing an HTTP transfer.

Because the number of possible ports is high (65,536), it is not possible to display all of them on the centre axis. Instead, the ports are grouped into ranges. It is possible to zoom in on the centre port axis to see port detail for each specific flow.

By holding the mouse pointer over the specific flow, it is also possible to see the originating source and destination address and individual port number of the flow as shown on the diagram on the right.                        (click on the image to enlarge)

  

Obviously we expect to see many flows on the well known port numbers below 1024, and in fact most custom applications tend to use port numbers below 10,000. This usually results in Flow Plot diagrams with much activity toward the bottom of the centre axis. Some common applications, such as FTP or some VoIP protocols use dynamic port ranges above 1024, when this occurs we often see a distinctive pattern more like a ‘fan’ where the flows using these dynamic ports are displayed.

Observation of Flow Plots on real network data has shown that they tend to be reasonably static. That is, the range of ports, and hence the overall look of the Flow Plot, is consistent between consecutive time slices. This characteristic is very important, as it allows us to very easily identify anomalous behaviour by comparing the look of Flow Plots from different sample periods.

A good example is the flow plot provided to the right. Notice on this plot that we see a small number of flows with port numbers in the range 36,000-37,000. They are shown on the picture below along with a ‘zoom-in’ window for clarity.

This flow plot is for a single 5 minute sample period, and given the number of flows on any given network, on their own, these somewhat abnormal flows would not be cause for concern. They certainly would not normally trigger an alarm and require network operator intervention.                      (Click on the image to enlarge)

      

What is more interesting though, is if we look at Flow Plots from consecutive 5 minute samples, it becomes clear that the same small number of flows exist but the ports used increment slowly across each period. That is, there is a host external to the current network (shown here as 10.11.200.25) running a slow port scan against an internal host (10.20.12.25). The port scan looks to be generating connections very slowly at around one connection per minute. The whole port scan may take several days to complete but it is also likely to be undetected by most intrusion detection systems or firewalls. 

 

By replaying a whole days worth of consecutive Flow Plots a security anomaly such as this slow port scan becomes immediately obvious.

 

Another good example of the power of Flow Plots is to isolate flows to or from a specific site. Again, like whole of network traffic, most sites within a network exhibit a consistent Flow Plot signature so anything unusual becomes immediately obvious.

 

(click to enlarge)

 Here is a Flow Plot that isolates traffic to an individual group, in this instance the Queensland (QLD) group that contains all the subnets within the state of Queensland for the monitored network.

Even a quick glance at this picture shows something clearly unusual. There are hundreds of connections coming from outside the network (from the ‘External’ group) all directed toward the QLD group. Note that this represents flows for a single 5 minute sample.

Closer inspection shows that the origin is a range of external hosts, connecting to a single IP host within the QLD group using different port numbers. Given the size and rate of these connections, that is, all small flows sent within a single 5 minute period, it is clear that this picture represents a Denial of Service attack on a host in the QLD group. In fact, given that these flows use multiple source addresses it may actually be a Distributed Denial of Service attack).

This provides just a couple of examples of the power of Flow Plots in identifying anomalous behaviour within network data. These examples highlight some of the more obvious security issues but the concept translates just as effectively to identifying other spurious conditions that affect network performance. For example, Flow Plots have been used to effectively identify conditions such as backups that overrun into business hours thus causing performance degradation, policy violations where certain types of traffic exist which shouldn’t, or other unexpected events like unscheduled anti-virus updates.

The use of Flow Plots provide a very effective and easy mechanism to portray whole of network traffic patterns and will become a powerful tool to assist in the diagnosis and maintenance of network infrastructure and add to the improvement of overall end user experience.