Summary of Reinforcement Learning in Action: Creating Arena Battle AI for 'Blade & Soul'

Summary of "Reinforcement Learning in Action: Creating Arena Battle AI for 'Blade & Soul'"

Overview:
This video presents a detailed case study on developing AI agents for the arena battle mode in the MMORPG Blade & Soul using Reinforcement Learning (RL). The project focuses on creating programmable, competitive AI agents capable of playing one-on-one battles with diverse fighting styles (aggressive, balanced, defensive) and performing at a professional gamer level.

---

Key Technological Concepts and Challenges:

1. Problem Setting:
   • Blade & Soul arena battle is a real-time, one-on-one fighting game where players compete to reduce the opponent’s HP to zero within three minutes.
   • The game has 11 character classes; the project focused on the "Destroyer" class due to its popularity and fixed skill settings for fairness.
   • The agent observes the environment (HP, skill points, distance, cooldowns, arena boundaries) and chooses actions every 100 milliseconds.
   • Actions include skill use, movement, and targeting, with complex strategic trade-offs (e.g., crowd control skills before damage skills, timing resistance or escape skills).
2. Challenges:
   • High Complexity: Extremely large state and action spaces (~10¹⁸⁰⁰ for the Destroyer), combining skill, movement, and targeting decisions.
   • Real-Time Response Constraint: Decisions must be made quickly, disallowing computationally expensive search methods; neural network policies are used for efficient inference.
   • Generalization: The AI must perform well against a wide variety of opponents, including human players with unpredictable styles.
   • Guiding Fighting Styles: Without hard-coded rules, the AI needs to exhibit diverse styles (aggressive, balanced, defensive) through reward shaping.

---

Reinforcement Learning Approach:

• Basic RL Concept: Agents learn through trial and error, modifying their policy to maximize cumulative rewards.
• Rewards:
  • Primary: Winning the match and dealing damage (reducing opponent HP).
  • Secondary: Shaped rewards to encourage different aggressiveness levels (damage dealt vs. HP preserved, distance maintained or closed).
• Features: The AI receives the same information as human players, such as HP, skill points, distance, position, and skill cooldowns.
• Training Setup:
  • 100 parallel simulations running combat games.
  • Used the ACER (Actor-Critic with Experience Replay) RL algorithm to update policies.
  • Employed self-play with a growing pool of past agents to ensure diverse and progressively stronger opponents.

---

Engineering Techniques:

• Movement Policy Learning: To overcome minimal impact of single move actions, agents maintain the same move decision for 10 consecutive steps, enabling meaningful movement strategy learning.
• Skill Usage Constraints: Skills with limited range are blocked when opponents are too far, preventing wasteful exploration and speeding up learning.
• Feature Discretization: Continuous distance features were discretized to help the agent better learn exact skill ranges, improving skill usage precision (e.g., for the "drag" skill).

---

Experimental Results:

• Initial training against a fixed built-in AI showed rapid improvement but the agent was still exploitable by humans.
• Self-play training against a diverse pool of agents led to stable performance increases and better generalization.
• The trained AI defeated both built-in AI and skilled human players it had never encountered before, demonstrating robustness.
• Demo videos showed progressive improvement from random actions to executing complex combos and strategic skill use.
• Different fighting styles were visually and behaviorally distinct: aggressive agents relentlessly attack, defensive agents maintain distance and counterattack.

---

Professional Evaluation:

• Prior to a live blind match event, professional gamers tested the AI with imposed reaction delays (~230 ms) to ensure fairness.
• Aggressive AI agents dominated most matches, overwhelming human players by not allowing breaks in combat.
• Balanced and defensive agents produced more competitive, human-like matches.
• Pro gamers acknowledged the aggressive AI's flawless and unique playstyle, differing significantly from human strategies.

---

Conclusions:

1. Successfully created pro-level AI for a complex real-time fighting game using Reinforcement Learning.
2. Demonstrated that diverse fighting styles can be guided via reward shaping without hard-coded rules.
3. Developed robust AI through self-play with a diverse opponent pool.
4. Employed engineering techniques to reduce problem complexity and accelerate learning.

---

Speakers / Sources:

• Jinyoung Zhang – Team Leader, introduced the problem, challenges, and overall project goals.
• Sunro (Sun Noor) – AI Research Engineer, explained Reinforcement Learning fundamentals, reward design, training process, engineering techniques, and experimental results.
• Additional insights from professional gamers who tested the AI in blind matches.

---

Additional Notes:

• The team announced a related session on gym learning-based universe kinematics.
• Q&A touched on fairness of AI’s aggressive style and tuning of neural network weights through trial and error.

Category

Technology

Video