User behaviour modelling from web log

User Behaviour Modelling from Web Logs

The lecture explains how cleaned web access logs can be used to model visitors’ navigation through a website. The goal is to understand browsing patterns and support decisions such as improving site navigation or managing website content.

Core Ideas

• Browsing is uncertain: Visitors may enter through a search engine, a link from another site, or a directly typed address. They may visit different pages, spend varying amounts of time on them, and leave at different points. Navigation is therefore represented using a probabilistic model.
• Use a finite-state Markov model: Each state represents either a webpage or a group of pages with a shared function or theme. Grouping pages can make a large website easier to model.
• Add two dummy states:
  • Entry state: Represents the visitor’s position before entering the website.
  • Exit state: Represents the visitor after leaving the website.
• The model has two parts:
  • Static part: The states and the possible links or transitions between them.
  • Dynamic part: The observed transition probabilities and timing information estimated from access logs.

Method for Building the Model

1. Define the states.

   • Choose individual pages or group related pages into functional categories.
   • Include the dummy entry and exit states.

2. Map the site structure.

   • Identify which pages or categories are connected.
   • Use a site map if available, or create a website crawler to follow links and record connections.
   • This produces the model’s static structure.

3. Use cleaned access logs to identify observed transitions.

   • The requested page and the log’s referrer field indicate movement from one page to another.
   • For example, if the referrer is state S1 and the requested page belongs to S2, the log records a transition from S1 to S2.
   • Referrer information can also help identify entry into the site from a search engine, another website, or a direct visit.

4. Calculate transition probabilities.

   • For each state, count the observed transitions from it to every possible next state.
   • Divide each outgoing transition count by the total number of transitions out of that state.
   • For example, if a state has 20 outgoing transitions—8, 4, 7, and 1 to four possible destinations—the corresponding probabilities are 8/20, 4/20, 7/20, and 1/20.
   • Repeat this for all states to form the site’s transition-probability matrix.

5. Estimate time between transitions.

   • Use timestamps associated with successive page requests to estimate the time for a transition.
   • The lecture suggests averaging the observed times for each state-to-state transition.
   • These measured intervals are not pure “thinking time”: they may include the visitor’s decision time, network delays, server-side page construction, and browser processing. The approach described does not allow these components to be cleanly separated.

Scope and Cautions

• The lecture introduces the modelling approach at a conceptual level and does not cover the full mathematical or implementation detail.
• It focuses on a finite-state model and simplified average timing estimates; more complex time distributions are acknowledged but not developed.
• The resulting model is intended to help interpret navigation behaviour and inform website decisions.

Speakers and Sources

• Speaker: One lecturer or presenter; no other speakers are identifiable in the subtitles.
• Sources and inputs discussed: Cleaned web access logs, especially requested-page, referrer, and timestamp fields; website link information obtained from a site map or crawler.