Understanding the web log-II

Summary

The lecture explains how to clean and prepare web server access logs for web-usage analysis. Raw logs are incomplete and noisy, so careful preprocessing is necessary before drawing conclusions about visitor behavior.

Why access logs are incomplete or misleading

• Caching: Browsers, proxies, and other nearby caches may serve pages without contacting the website’s server. Those requests therefore do not appear in the server’s access log.
• Dynamic IP addresses: ISPs may assign different IP addresses to the same visitor, making it difficult to identify users reliably from IP addresses alone.
• HTTP is stateless: Individual requests do not inherently indicate which requests belong to the same visit or session. Cookies and other mechanisms can help, but none are universally reliable.
• Automated traffic: Search-engine and advertising crawlers generate requests that appear in logs alongside human activity.
• Embedded resources: A page load can generate separate requests for images, audio, video, and other resources in addition to the HTML page itself. These can obscure the user’s actual page-navigation path.
• Frames: When one HTML page is embedded inside another, identifying which page was actually viewed and reconstructing navigation can become more difficult.

Together, these limitations make preprocessing labor-intensive; the lecture notes that filtering can take up a large share of log-analysis work.

Main preprocessing workflow

1. Collect raw usage data from the access logs.
2. Clean the data by removing irrelevant or misleading entries, including:
   • Requests for embedded resources when the goal is to analyze page views.
   • Robot and crawler traffic.
   • Invalid references and other unwanted records.
3. Integrate data where available, such as:
   • Logs from multiple servers.
   • Semantic information about pages and services.
   • Customer registration or demographic data.
4. Identify users and sessions using available evidence and heuristics.
5. Recognize page views—the pages and associated objects that together produce a display in a browser.
6. Complete missing paths by inferring requests absent from the log, often because of caching.
7. Reduce the data when necessary, for example through sampling or by removing unrelated page views using site knowledge.
8. Create session and episode files for further analysis.

Filtering and identifying traffic

• Embedded requests: Filter out requests for non-HTML objects when analyzing page navigation. Site-specific knowledge can help distinguish actual pages from embedded content and address complications such as frames.
• Robots and crawlers: Use user-agent information and automated detection algorithms to identify bots. The lecture also describes robots.txt, which well-behaved crawlers consult for site-specific crawling instructions. A maintained list of real site pages can also help identify and remove other requests.
• User identification: Possible approaches include:
  • IP address plus browser/user-agent: Easy to obtain, but not guaranteed to identify one person consistently, especially when proxies or changing IPs are involved.
  • Embedded session IDs: IDs can be included in links or page requests and returned with subsequent requests. They can work independently of IP addresses, but may fail to connect activity across separate browser instances.
  • Registration: Can identify a user when they choose to register, but depends on user participation.
  • Cookies: Can help track activity within and across visits, but users can disable or reject them.
  • Software agents or browser scripts: Browser plugins or embedded JavaScript can collect and send usage information. Google Analytics JavaScript is given as an example.

Sessionization

Sessionization groups requests believed to come from the same visitor into a visit. This is essential because behavior analysis depends on having a meaningful sequence of a visitor’s actions.

The lecture describes two main heuristic approaches, often used together:

• Time-oriented heuristics: Group requests based on the time between them. A commonly used guideline is a 30-second gap: if no new request arrives within that period, the session may be considered over. The threshold is adjustable and is not a strict rule.
• Navigation-oriented heuristics: Use the site’s link structure. For example, requests for page A and a page linked from A may plausibly belong to the same visitor’s path.

Because no identification method is fully reliable, session boundaries and user identities are generally inferred rather than known with certainty.

Path completion and semantic modeling

• Path completion attempts to infer missing page references, particularly those omitted because a cached page did not generate a request to the server.
• Useful evidence includes the site’s link structure and referrer information in the log. Frames can make reconstruction harder because log entries may refer to a parent frame.
• The lecture also introduces semantic session modeling: associating requested pages or URLs with concepts about the site’s content or services. This can help interpret sequences as meaningful activities—for example, browsing by category, narrowing by title, and then viewing individual products.
• These concepts may be organized into an ontology, a hierarchy of related concepts. A session can then be represented as a set, sequence, tree, or graph of requests leading to a concept.

Output and behavioral analysis

After filtering and session identification, the prepared data can be stored in a flat-file or database format. Example fields include IP address, user ID, request time and method, requested resource, protocol, status code, bytes transferred, referrer, user agent, session ID, and object ID.

Sessions can then be grouped into episodes that represent different purposes or behavior patterns. For example, new visitors who browse by category may behave differently from returning customers who go directly to familiar products. Recognizing such groups supports analysis of navigation, search, and visitor behavior.

Speakers and sources featured

• Speaker: One unnamed lecturer; no other speaker is identifiable in the subtitles.
• Sources and systems mentioned: Web server access logs; browser and proxy caches; ISP-assigned IP addresses and DNS resolution; search-engine and advertising crawlers; robots.txt; site link structures and referrer fields; cookies, registration, and embedded session IDs; browser plugins and JavaScript, including Google Analytics scripts; and an example log format attributed generally to an unspecified source.