Video summary

WebSphere Troubleshooting 22 JAN 25

Main summary

Key takeaways

Technology

Overview

This video is a WebSphere troubleshooting + admin “quick guide” covering common configuration areas, installation steps, runtime diagnostics, logging/trace controls, security/SSL setup, clustering concepts, and request-flow analysis. It’s heavily focused on where to look (files/logs/dumps), how to change settings (console + config), and what to verify when users can’t access applications.

Core WebSphere Concepts & Configuration Files

How to start components

  • Start DMGR via start manager.
  • Start node agent via start node.
  • Start an application server via start server (using a specified server name).

Main configuration files (WebSphere profile)

  • server.xml: Main configuration for the WebSphere server.
  • serverindex.xml: XML registry of internal items/definitions (auto-generated numbers/IDs are mentioned).
  • security.xml: Security-related configuration.
  • httpd.conf (for IHS/plugins context) and plugin configuration, including:
    • plugin-cfg.xml (key plugin config):
      • Cluster members
      • Retry intervals
      • Route/JSESSIONID/clone IDs
      • Virtual host + port mappings
      • Transport chains (HTTP/HTTPS)
      • Additional plugin/route mapping details

Logging & Failure Diagnosis Workflow

Log types mentioned

  • Logs in SystemOut / SystemErr style.
  • FFDC logs (used when “logs are not updating” / to capture failure details).
  • Process logs and server logs.

FFDC troubleshooting

  • Uses First Failure Data Capture (FFDC) to find the root cause when normal logs aren’t current.

Installation Manager (IM) & Repository Management

Installation methods via Installation Manager

  1. Root method (implied “root-to-install” approach)
  2. CLI mode
  3. GUI mode
  4. Response file method (scripted install)

IM repository handling

  • Mentions working with existing repositories and adding another repository (“better repositories” referenced).

Ports & Network/Security Basics

LDAP secure port

  • 636 (LDAPS)

Common HTTP/HTTPS port examples

  • WebSphere app access ports: HTTP 9080 / HTTPS 9443
  • Web server (IHS): 80 / 443
  • Plugin HTTP/HTTPS mappings referenced generally

Garbage Collection (GC) Tuning in WebSphere

Where to configure GC

  • Console → Servers → (select server) → Process management → Java Virtual Machine

Heap sizing (example)

  • Minimum/maximum heap around 1 GB (as demonstrated)

GC policy types listed

  • GenCon (“Gen Con”)
  • Opta(n) / optavgpas style (auto-text; described as “opt… / up to …”)
  • Balanced

Goal

  • Remove unwanted memory / reduce memory pressure.

Node Agent, DMGR, and Cell Structure

  • Node agent described as communication between DMGR and the app runtime.
  • Cell includes managed/unmanaged nodes.

Keystores, Certificates, and Session Settings

Keystores / certificates

  • Add/import certificates:
    • Personal or self-signed certificates
    • Third-party certificates
  • Trust stores referenced conceptually (e.g., “t consists of third party certs”).

Default session timeout

  • 30 minutes

Application server architecture components (high-level)

  • IHS/web server
  • Web container
  • EJB container
  • JMX
  • Message service/engines
  • Security
  • Naming
  • Related components

Request Flow & Routing Troubleshooting

High-level request flow

  • Client → DNS → Load balancer → Web server (IHS) → WebSphere web application → Application server → Database

When URL isn’t fetching: what to verify

  • URL + port
  • Server status (server down is a common cause)
  • URI
  • Plugin propagation / virtual host port numbers
  • Hostname mapping in /etc/hosts

Common HTTP errors mentioned

  • 503: Service unavailable
    • Verify request flow and network/plugin/misconfiguration.
  • 404: Page not found
    • May indicate an app/deployment issue (delete/rename temporarily mentioned).
  • 500: Server down / runtime problem
    • Verify plugins and upstream services.
  • Mentions checking packet loss/misconfiguration during 503 investigation.

Repositories in WebSphere / Security Repositories

Four repositories mentioned:

  1. Local OS
  2. Federated repository
  3. Standard LDAP registry
  4. Custom registry

Web Server vs Web Container vs EJB Container

  • Web server (IHS): serves static content
  • WebSphere web container: serves dynamic content using JSP/Servlets
  • EJB container: hosts EJBs, including:
    • Entity beans (data/entity)
    • Session beans

Sessions & Cookies Configuration

Console steps

  • Web container → session management → enable sessions
  • Enable session tracking
  • Enable cookies

Timeout

  • Default cookie/session timeout: 30 minutes (repeated)

Cluster behavior

  • Mentions distributed environments:
    • Distribute sessions for clusters (failover behavior described: if one app is down, another continues using session state).

Plugin Configuration and plugin-cfg.xml Usage

plugin-cfg.xml items listed include:

  • Cluster members
  • Retry/traversal intervals
  • JSESSIONID / clone IDs
  • Virtual host port numbers
  • Transport chains including HTTP/HTTPS
  • Plugin keys (file references like plugin-key.kdb / plugin-keydb, plus plugin identity key/fields)

Dumps, Heap/Thread Analysis, and OOM Troubleshooting

How to identify OOM cause

  • Check systemOut.log / systemError.log for patterns like:
    • java.lang.NullPointerException
    • java.lang.OutOfMemoryError

Use analyzers

  • Heap analyzer and thread analyzer
  • MAT mentioned for dump analysis

How to generate dumps

  • Thread dump via kill command / PID (example PID referenced)
  • Heap dump via WebSphere console:
    • Troubleshooting → Java dumps → core → select dump type → generate dump

Notes:

  • Dumps are not always “human readable” (binary format mentioned).
  • Suggests not deleting dumps (or managing/deleting carefully).

Tracing / Log Details Controls

Enable advanced traces (real-time scenario)

  • Diagnostics → traces → change log details → apply/save

Log size & rotation

  • Example default log size: 10 MB
  • Mentions rotation rules (24 hours / size-based rotation)

Log levels

  • WebSphere and plugin log levels referenced:
    • Info / Warning / Debug / Error
  • Plugin default log level shown as error by default
  • Mentions memory default log level is info

SSL/TLS and TLS 1.2 Change

How to change TLS

  • Through DMGR console → Security → SSL certificate and management → SSL configurations
  • Change Quality of protection to TLS 1.2
  • Update backend SSL properties (config/property file mentioned)

Production change process

  • Use change window + CAB approvals + schedule → deploy → test → validate app works

Installing WebSphere / Profiles / Profile Creation

Install steps include

  • Use Installation Manager commands
  • Find offering IDs (example offering ID pattern referenced)
  • Verify post-install.log for success/failure

Profiles

  • Profiles are sets of configurations (profile bin/log/temp/apps).
  • Creation methods:
    • CLI mode (common)
    • GUI mode

Types/templates referenced:

  • DMGR profile
  • Application server profile (default template)
  • Custom profile templates
  • Admin agent / Job manager / secure proxy / configuration template (mentioned in the list)

Clusters: Types and Purpose

Cluster purpose

  • A group of servers (managed servers) for failover, scalability, high availability.

Types

  • Horizontal cluster: across different machines; failover if one server fails (best for HA)
  • Vertical cluster: components installed on the same machine
  • Mixed/Hybrid mode: combination of both (“hybrid cluster” / “mixer mode” mentioned)

Database / JDBC / JNDI & Data Source Setup (High-Level)

Described steps:

  1. Create security realm/aliases (security alias)
  2. Create a JDBC provider
  3. Create a data source using the JDBC provider
  4. Verify Universal Drivers location:
    • Example path like /opt/IBM/websphere/.../UniversalJDBCDriver/...jar (checking jar presence is suggested)
  5. Test connection from DMGR:
    • Check errors in logs like SystemOut/SystemErr (or agent logs)

Misc Troubleshooting Tasks

  • Verify hostnames:
    • Check /etc/hosts for correct hostname-to-IP mapping and permissions.
  • Nullify/clean logs:
    • Check log size and clearing/log rollover (example shows log size reduced to a small size after action).
  • Server start/stop issue:
    • Verify hostname in /etc/hosts and permissions.

Main Speakers/Sources

  • Subtitles don’t clearly identify a specific person by name.
  • Source appears to be a single trainer/engineer delivering a WebSphere troubleshooting session (no other speakers explicitly named).

Original video