Understanding the web log

Summary

The lecture introduces web logs as a source of data for decision support and for studying how websites are used. Because websites generate large volumes of interaction data, logs must be cleaned and analyzed before they can reveal useful patterns about visitors’ behavior.

How Web Logs Are Generated

• A browser sends an HTTP request to a web server, commonly using the GET or POST method.
• The server processes the request and returns an HTTP response. For dynamic pages, it may retrieve information from a database.
• A page may require separate requests for embedded resources such as images, audio, or video. As a result, loading one page can produce multiple log entries.
• Responses include status codes. For example, 200 indicates success, while 404 means that a page was not found.
• Servers record requests and related information in log files, including access logs and error logs.

What Access Logs Can Contain

Common fields include:

• Client IP address
• Authenticated username, if authentication is used
• Date and time of the request
• HTTP method, such as GET or POST
• Requested URL
• HTTP protocol version and response status code
• Number of bytes transferred
• Referrer: the page from which the request originated
• User agent: information about the browser or other requesting software

The recorded IP address may not identify the actual end user. Requests can pass through organizational proxy servers, ISPs, and other network intermediaries, which may change or obscure the address visible to the destination server.

Logs also include requests from automated crawlers or spiders, not just human visitors. Analysts studying human behavior therefore need to identify and remove or account for crawler activity.

Web Usage Mining and Clickstream Analysis

Analyzing log data to understand how a website is used is called web usage mining. Because the data is generated by users’ clicks and navigation, this analysis is also known as clickstream analysis.

Logs can support analysis of user behavior and other decision-making tasks, but their raw text format and large volume make direct manual interpretation impractical.

Sessions and Session Identification

HTTP is stateless: the server does not automatically remember a user between requests. Session identification is therefore a key preprocessing challenge.

The lecture distinguishes between:

• Client session: A person’s activity from opening a browser until closing it. It may span multiple websites and browser tabs.
• Server session: A sequence of requests made to a particular server, from the first request to the last.

Cookies and other methods can help associate requests with a session. Session boundaries may also need to be inferred using heuristics.

Web Usage Mining Workflow

1. Collect and integrate log data

   • Gather logs from relevant sources, such as web servers and application servers.
   • Combine data from multiple or mirrored servers where applicable.

2. Preprocess the data

   • Clean the logs, including handling irrelevant or automated requests.
   • Identify page views from individual resource requests.
   • Identify sessions and organize requests into meaningful sequences.
   • Use knowledge of the website’s structure and relevant subject area to guide preparation.

3. Transform the data

   • Convert the cleaned records into a form suitable for analysis, potentially storing them in a database.

4. Discover patterns

   • Apply techniques such as transaction clustering, page-view clustering, correlation analysis, association-rule mining, and sequential-pattern mining.

5. Analyze and model behavior

   • Interpret the discovered patterns alongside domain knowledge and information about the site’s structure to develop models of user behavior.

Speakers and Sources Featured

• Speaker: One lecturer or instructor introduces and explains web logs and web usage mining.
• Sources and systems discussed: HTTP requests and responses; web and application servers; access and error logs; browsers; proxies and ISPs; crawlers and spiders; cookies; and mirrored servers. No other speakers are featured.