Internet Traffic Analyzing System

Introduction

Internet Traffic Statistics Collection and Analyzing System provides for multi-dimensional analysis and control over a company's Internet traffic. The core of the system is built with help of MS OLAP, Oracle server 8i and Loading Control Service (proprietary technology). The system allows for regular collection of data on Internet traffic consumption (and production). The data take the following way through the system: loading from external sources (log files of applications that support Internet access); distribution into the Data Warehouse; processing into OLAP structures (dimensions, multidimensional cubes); presentation of data from MS OLAP as reports. Introducing ITAS reduces the company's Internet expenses by 20 to 50 percent. What is even more important, these savings are gained not by "forbidding as much as possible" but by "allowing everything if it is under control".
 

Basic system features

General control over incoming and outgoing traffic through the office-to-Internet connection point

ITAS is configured together with the company's proxy server. All the log files that are created by the applications that run with the proxy server are uploaded into the ITAS on a regular basis. The system takes the total traffic turnover numbers and generates the top-level data on incoming and outgoing traffic through the proxy server. Next this data is separated into traffic turnover through specific proxy server ports. These traffic consumption reports allow controlling the channel workload (by day, day of week, and hour of day) and compare loads on different ports and draw conclusions as to which services are used most by the company's employees.

Analysis and control over traffic consumed by specific employees

Next yet another OLAP dimension - the employees - is added to the traffic data. This dimension can be hierarchical according to the organizational structure of the company (divisions, departments, groups, employees). This way ITAS allows mining data about which employees or groups of employees access which Internet resources, how much traffic they create and when they do it. This data can also be sorted (by employees or groups thereof, by resources, and the peak load times).

Control over number and volume of mail messages

When mail server log files are uploaded, information about all messages that have passed through the system is uploaded into the data warehouse. OLAP is used to structure this data by sender and recipient addresses, time and size of messages, thus allowing the building of respective reports.

Control over time spent online

ITAS can operate not only with traffic volumes (MB), but also with numbers of certain transactions, like messages received/sent, Web pages browsed, etc. This provides grounds for conclusions on how much time this or that employee or department spends on working with mail and on browsing the Internet.

Information about the destinations of outbound traffic

All the recipients of outbound messages are stored in a special dictionary (dimension), which allows for future structuring of the number and volume of outgoing mail. Those who are interested not only in traffic consumption volumes, but also in where does the traffic produced by the own company's resources (e.g., the corporate site) go, can have reports on access to these resources from other Internet locations. The destinations report provides a view on who is most interested in the data provided on the company's site.
 

Basic System Reports:

1. Report on office traffic consumption/production per time unit.
2. Reports on types of traffic going through the proxy server (Web, mail, other ports, black hole). Black hole traffic is traffic that has passed through the proxy server's firewall, but did not go further, i.e. was consumed by the proxy server users (administrators, services, people who are entitled to use the server) plus the error margin.
3. Reports on workloads of other open ports (e.g. news, pop3, bink-d, ping&trace).
4. Report on traffic distribution by department.
5. Comparison of personified and general traffic. Control over the black hole traffic.
6. Report on the amount of traffic that was consumed by the employees while working over HTTP and FTP protocols.
7. Report on HTTP and FTP traffic consumed by separate employees, departments, or the company as a whole, distributed by the originating domain. This report allows for knowing what resources are used by a particular employee/department or the company as a whole, and to what extent.
8. Report on personal mail traffic: total volume of received, sent, and intra-office messages.
9. Same as above, but with numbers of messages instead of their size.
10. Report on accounted and personified traffic that is distributed by departments and traffic types (mail, Web pages, FTP, etc.).

Technologies used:

Software requirements: