Ask an OpenShift Admin (Ep 26): Day 2 Operations, part 1

Summary of “Ask an OpenShift Admin (Ep 26): Day 2 Operations, part 1”

Format / purpose

• A live Q&A “office-hours” stream where attendees ask about OpenShift administration topics.
• Speakers also revisit earlier blog/documentation themes and capture audience feedback on what’s unclear or missing.

---

Day 2 Operations concept (Red Hat “day numbering”)

• Red Hat uses a day/phase model:
  • Day 0 / phase 0: architecture and design
  • Day 1 / phase 1: deployment
  • Day 2: everything after deployment (operations) until decommissioning
    • (Jokingly referred to as “day 10,000” due to the long lifecycle.)
• This episode focuses on cluster and node-level operations (part 1).
• Storage-focused and additional operational topics are planned for next week.

---

Key admin concepts and changes from OpenShift 3 to 4

Why OpenShift administration changed so much (Operators everywhere)

• The shift from OpenShift 3 to 4 is closely tied to widespread Operator adoption.
• In OpenShift 3, admins often used external Ansible playbooks/SSH to configure nodes and services.
• In OpenShift 4, Operators move that operational knowledge into the cluster using Kubernetes-native approaches.
  • Example: changes previously driven by Ansible are now commonly handled by the Machine Config Operator.
• Benefits highlighted:
  • Less manual node management (fewer “SSH/Ansible poke the nodes” workflows)
  • Declarative, in-cluster configuration control
  • Operators can often upgrade themselves, including broader component upgrades (“upgrade the whole cluster”)

Clarifying “Cluster API” vs OpenShift “Hive”

• Terminology was corrected during discussion:
  • Cluster API (generic concept) typically refers to APIs that orchestrate cluster creation programmatically, often tied to infrastructure providers.
  • OpenShift provisioning uses Hive to enable cluster provisioning based on Kubernetes objects.
• Hive has an operator in OperatorHub (noted as community currently), with expectations for improved Red Hat support later.

---

Subnetting guidance for OpenShift clusters (common doc confusion)

The team addresses repeated questions about subnet configuration for OpenShift networking and install settings.

Install config contains 3 subnet-related settings

1. Cluster network (pod network CIDR)
   • Used to allocate pod IPs per node.
   • Example: /14 is carved into per-node /23 ranges; pods receive IPs from those ranges.
2. Service network
   • Range for Kubernetes Service ClusterIPs.
3. Machine network
   • Used for network interface discovery during node setup.
   • OpenShift expects nodes to have an interface in that range to determine which network to use.
   • If multiple interfaces exist, interface ordering/range matters; the wrong interface choice can affect behaviors such as proxy/no_proxy.

Do you need dedicated subnets per cluster?

• No, you do not need a subnet dedicated exclusively to OpenShift.
• No, a dedicated subnet per cluster is not required.

Main risk when sharing a subnet: on-prem IPI + VRRP domain IDs

• When using on-prem IPI, keepalived-style VIPs rely on VRRP concepts.
• VRRP IDs must be unique across the subnet (or redundancy domain).
• The VIP generator may be based on cluster name; duplicate cluster names or domain IDs can cause cluster bring-up failures or confusing networking outcomes between new and existing clusters.
• Sharing the subnet with other infrastructure services (e.g., DNS/DHCP) is generally fine and has no inherent harm.

---

Guides / features highlighted: Quick Starts (in-cluster guided workflows)

Quick Starts for administrators

• Quick starts provide guided step-by-step flows inside the OpenShift UI (side panel/tour experience).
• Admins can create them to:
  • Teach workflows for junior admins and/or developers
  • Help users self-serve repeated tasks
• Use case emphasized: reducing recurring support questions by embedding operational guidance directly into the cluster UI.

---

Upgrade blocker: OpenShift 4.7 on VMware / kernel offload conflict

• A known upgrade blocker prevents OpenShift 4.6 → 4.7 upgrades in certain VMware + RHEL kernel combinations.
• Root cause described:
  • VMware (virtual hardware versions 14+) enables hardware offload for VXLAN
  • The RHEL 8.3 kernel has similar functionality
  • The conflict can lead to packet loss
• Notes:
  • Workarounds were mentioned (e.g., disabling offload/checksum features), but no fully supported workaround was claimed in the video.
  • Framed as “blocking upgrades,” emphasizing engineering coordination and the need for patience.
• Scope nuance:
  • Virtual hardware version 13 might not be affected, but could hit other constraints (e.g., VMware CSI provisioner requiring newer virtual hardware versions).

---

Dev/test cluster options and minimum footprint

CodeReady Containers (CRC) 1.25

• CRC 1.25 was mentioned as recently released.
• It’s based on OpenShift 4.5.7 and deploys a single-node cluster in a VM (on a developer workstation).

Single-node OpenShift (TECH preview direction)

• Positioned for edge scenarios (smaller deployments; less resiliency than multi-node).
• Compared to CRC:
  • CRC: for developers testing apps locally
  • Single Node OpenShift: closer to edge/remote deployment validation
• Expectations discussed for when it may be available (e.g., around OpenShift 4.8, potentially via tech preview/nightlies).

Minimum cluster size guidance

• Referenced guidance suggests the smallest currently deployable “real” OpenShift cluster is three nodes, with the caveat that docs are evolving.

---

Time synchronization + GitOps

Operational importance

• Correct time synchronization is critical because:
  • Certificates rely on accurate time
  • Logging/audit correctness depends on it

Automation approach

• When using GitOps (e.g., get ops / Argo CD operator), time sync/config can be made reproducible across clusters.
• A related future stream idea was mentioned: using GitOps not only for app deployment, but for administrative/operational tasks.

---

Day 2 administration: documentation starting points and examples covered

Documentation starting point

• The main place to start is the “post-install configuration” documentation section.
• Other areas referenced include stability/performance (example: CPU manager via operator).

Node and control plane sizing / machine management

• Use docs for:
  • Control plane and infrastructure node sizing
  • Worker sizing guidance
• Changing control plane size:
  • Safest approach described: replace control plane nodes one at a time with updated resources
  • (Caveat noted for certain non-integrated install scenarios.)
• Changing worker capacity:
  • Resize via new MachineSets:
    • adjust CPU/memory counts
    • potentially move to a different storage domain/datastore
• Emphasized flexibility:
  • multiple machine types
  • taints/node labels tied to machine sets
  • custom node roles for GPU / high-performance / local storage workloads

Infrastructure nodes and licensing/workload placement

• Infra nodes (logging/metrics/platform services) are not strictly required, but are recommended by Red Hat.
• Licensing implications should be considered if infra services run on nodes that also host application workloads.

Example: applying host-level storage settings (NetApp Trident via Machine Config)

• A deep example used NetApp Trident (iSCSI example) to illustrate node configuration modification with Machine Config:
  • install/configure required packages/settings
  • run required scripts via systemd on startup
• Key limitation noted:
  • On CoreOS-like hosts, RPM installs aren’t persistent, so Machine Config-based delivery (config + scripts) is preferred.

---

Product/tech comparisons mentioned

OpenShift SDN vs OVN-Kubernetes

• No “substantial difference” for baseline expectations, but the right choice depends on required features.
• OVN-kubernetes may become default SDN in a future timeframe (mentioned: OpenShift 4.9).
• Capabilities highlighted:
  • cross-OS support (e.g., Windows nodes)
  • overlay tech considerations (e.g., GENEVE vs VXLAN-like approaches)

---

Main speakers / sources (as stated)

• Andrew Sullivan — Technical Marketing Manager, Red Hat Cloud Platforms Business Unit (main speaker; admin background and answers)
• Chris Short — Executive Producer, OpenShift TV (host/moderator)