ChaZIR/AIM-PIbd-32-Tyorkin-D-V
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
AIM-PIbd-32-Tyorkin-D-V/lab_6/lab6.ipynb

5125 lines
111 KiB
Plaintext
Raw Normal View History

lab6
2024-12-07 00:21:55 +04:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Лабораторная работа №6"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Задача:\n",
"\n",
"**Разработка и запуск проекта по обучению с подкреплением для игры \"Крестики-нолики\"**\n",
"\n",
"**Что нужно сделать:**\n",
"\n",
"1. **Перевести проект на библиотеку Gymnasium и современную версию Python.** Gymnasium — это современная альтернатива библиотеке Gym, предоставляющая удобные инструменты для создания и тестирования моделей обучения с подкреплением.\n",
"2. **Реализовать агента для игры \"Крестики-нолики\" в виде отдельного класса.** Следуя примеру из лекции, создайте класс агента, который будет отвечать за принятие решений и обучение.\n",
lab66
2024-12-07 00:32:06 +04:00
"3. **Основной цикл обучения для работы с отдельным классом агента.** Обучение происходит через взаимодействие с классом агента.\n",
"4. **Протестировать новую версию программы.** Убедиться, что программа работает корректно и агент успешно обучается играть в \"Крестики-нолики\".\n",
lab6
2024-12-07 00:21:55 +04:00
"\n",
"**Обучение с подкреплением (Reinforcement Learning, RL)** — это подход машинного обучения, при котором агент учится принимать решения, взаимодействуя с окружающей средой. Цель агента — выбирать действия, которые максимизируют накопленную награду в долгосрочной перспективе.\n",
"\n",
"**Ссылка на игру \"Крестики-нолики\":** [https://github.com/nczempin/gym-tic-tac-toe](https://github.com/nczempin/gym-tic-tac-toe)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Переход на среду Gymnasium\n",
"\n",
"**Gymnasium (прежнее название — OpenAI Gym)** — это библиотека, предназначенная для разработки и тестирования алгоритмов обучения с подкреплением. Она предлагает широкий набор стандартных сред для RL, включая игры, задачи управления и моделирование.\n",
"\n",
"Gymnasium предлагает обновленный API по сравнению с Gym, который:\n",
"\n",
"* **Упрощает** методы `reset` и `step`.\n",
"* **Добавляет** новые функции, такие как поддержка тайм-аутов и контроль количества шагов.\n",
"* **Улучшает** поддержку пользовательских сред.\n",
"\n",
"**Основные преимущества Gymnasium:**\n",
"\n",
"* **Стандартизированные интерфейсы** для взаимодействия со средой.\n",
"* **Простая интеграция** с популярными библиотеками RL.\n",
"* **Поддержка** создания собственных сред.\n",
"\n",
"**Ключевые функции Gymnasium:**\n",
"\n",
"* `env.reset()` — инициализация среды.\n",
"* `env.step(action)` — выполнение действия и переход в новое состояние.\n",
"* `env.render()` — визуализация текущего состояния среды."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import gymnasium as gym\n",
"from gymnasium import spaces\n",
"\n",
"class TicTacToeEnv(gym.Env):\n",
" metadata = {'render.modes': ['human']}\n",
" \n",
" symbols = ['O', ' ', 'X']\n",
"\n",
" def __init__(self):\n",
" super().__init__()\n",
" self.action_space = spaces.Discrete(9)\n",
" self.observation_space = spaces.Discrete(9 * 3 * 2)\n",
" self.reset()\n",
"\n",
" def step(self, action):\n",
" done = False\n",
" reward = 0\n",
"\n",
" p, square = action # p - игрок (1 или -1), square - номер клетки\n",
"\n",
" board = self.state['board']\n",
" proposed = board[square] \n",
" om = self.state['on_move'] \n",
" if proposed != 0: # Клетка уже занята\n",
" print(f\"Незаконный ход: Квадрат {square} уже занят.\")\n",
" done = True\n",
" reward = -1 * om \n",
" if p != om: # Не тот игрок на ходу\n",
" print(f\"Незаконный ход: игрок {p} не находится в движении\")\n",
" done = True\n",
" reward = -1 * om\n",
" else:\n",
" board[square] = p\n",
" self.state['on_move'] = -p\n",
"\n",
" for i in range(3):\n",
" # Горизонтали и вертикали\n",
" if (board[i * 3] == p and board[i * 3 + 1] == p and board[i * 3 + 2] == p) or \\\n",
" (board[i] == p and board[i + 3] == p and board[i + 6] == p):\n",
" reward = p\n",
" done = True\n",
" break\n",
"\n",
" # Диагонали\n",
" if (board[0] == p and board[4] == p and board[8] == p) or \\\n",
" (board[2] == p and board[4] == p and board[6] == p):\n",
" reward = p\n",
" done = True\n",
" \n",
" return self.state, reward, done, {}\n",
"\n",
" def reset(self):\n",
" self.state = {}\n",
" self.state['board'] = [0, 0, 0, 0, 0, 0, 0, 0, 0] \n",
" self.state['on_move'] = 1 \n",
" return self.state, {}\n",
"\n",
" def render(self, close=False):\n",
" if close:\n",
" return\n",
" print(\"on move: \" , self.symbols[self.state['on_move']+1])\n",
" for i in range (9):\n",
" print (self.symbols[self.state['board'][i]+1], end=\" \");\n",
" if ((i % 3) == 2):\n",
" print();\n",
"\n",
" def move_generator(self):\n",
" moves = []\n",
" for i in range(9):\n",
" if self.state['board'][i] == 0:\n",
" p = self.state['on_move']\n",
" m = [p, i]\n",
" moves.append(m)\n",
" return moves"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Разработка агента\n",
"\n",
"В рамках обучения с подкреплением (Reinforcement Learning, RL) **агент** — это система, которая взаимодействует с окружающей средой для достижения определенной цели. Основная задача агента — разработать стратегию, которая максимизирует накопленную награду в долгосрочной перспективе.\n",
"\n",
"**Роль агента:**\n",
"\n",
"* **Принятие решений:** Агент выбирает действие на основе текущего состояния среды.\n",
"* **Получение обратной связи:** После выполнения действия агент получает от среды награду и информацию о новом состоянии.\n",
"\n",
"**Основные функции агента:**\n",
"\n",
"* **Выбор действия:** Использует алгоритмы или стратегии для определения следующего шага.\n",
"* **Обучение:** Анализирует полученный опыт и обновляет свои знания или стратегию для повышения эффективности.\n",
"* **Адаптация:** Адаптируется к изменениям в окружающей среде."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"\n",
"# Реализация Агента, который в рамках обучения с подкреплением взаимодействует со средой и вырабатывает наилучшую стратегию \n",
"\n",
"class Agent:\n",
" def __init__(self, symbol):\n",
" self.symbol = symbol # Символ игрока (1 - X, -1 - O)\n",
" \n",
" def get_action(self, moves):\n",
" return random.choice(moves) # Выбираем случайный ход из доступных"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Основной цикл обучения\n",
"\n",
"Основной цикл обучения включает создание и взаимодействие со средой. После завершения игры агент получает очки, которые отражают его успехи и позволяют оценить, насколько успешно прошло обучение системы."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
" O \n",
" X X \n",
" \n",
"on move: X\n",
" O O \n",
" X X \n",
" \n",
"on move: O\n",
"X O O \n",
" X X \n",
" \n",
"on move: X\n",
"X O O \n",
" X X \n",
" O \n",
"on move: O\n",
"X O O \n",
"X X X \n",
" O \n",
"Episode 1, Total Reward: 1\n",
"Average Reward: 1.0\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" \n",
" O \n",
"X \n",
"on move: O\n",
" \n",
" O \n",
"X X \n",
"on move: X\n",
"O \n",
" O \n",
"X X \n",
"on move: O\n",
"O X \n",
" O \n",
"X X \n",
"on move: X\n",
"O X \n",
" O \n",
"X O X \n",
"on move: O\n",
"O X \n",
" X O \n",
"X O X \n",
"Episode 2, Total Reward: 1\n",
"Average Reward: 1.0\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
"O \n",
" X \n",
" \n",
"on move: O\n",
"O \n",
" X \n",
" X \n",
"on move: X\n",
"O \n",
"O X \n",
" X \n",
"on move: O\n",
"O X \n",
"O X \n",
" X \n",
"on move: X\n",
"O O X \n",
"O X \n",
" X \n",
"on move: O\n",
"O O X \n",
"O X \n",
"X X \n",
"Episode 3, Total Reward: 1\n",
"Average Reward: 1.0\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X O \n",
" \n",
"on move: O\n",
" \n",
"X X O \n",
" \n",
"on move: X\n",
" O \n",
"X X O \n",
" \n",
"on move: O\n",
" O \n",
"X X O \n",
" X \n",
"on move: X\n",
" O O \n",
"X X O \n",
" X \n",
"on move: O\n",
" O O \n",
"X X O \n",
" X X \n",
"on move: X\n",
"O O O \n",
"X X O \n",
" X X \n",
"Episode 4, Total Reward: -1\n",
"Average Reward: 0.5\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
"O X \n",
" \n",
" \n",
"on move: O\n",
"O X \n",
" X \n",
" \n",
"on move: X\n",
"O X \n",
" X \n",
"O \n",
"on move: O\n",
"O X \n",
" X X \n",
"O \n",
"on move: X\n",
"O X O \n",
" X X \n",
"O \n",
"on move: O\n",
"O X O \n",
" X X \n",
"O X \n",
"on move: X\n",
"O X O \n",
"O X X \n",
"O X \n",
"Episode 5, Total Reward: -1\n",
"Average Reward: 0.2\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" O \n",
" X \n",
"on move: O\n",
" \n",
" O \n",
"X X \n",
"on move: X\n",
" O \n",
" O \n",
"X X \n",
"on move: O\n",
" O \n",
" O X \n",
"X X \n",
"on move: X\n",
" O \n",
"O O X \n",
"X X \n",
"on move: O\n",
" O \n",
"O O X \n",
"X X X \n",
"Episode 6, Total Reward: 1\n",
"Average Reward: 0.3333333333333333\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" O X \n",
" \n",
" \n",
"on move: O\n",
" O X \n",
"X \n",
" \n",
"on move: X\n",
" O X \n",
"X \n",
" O \n",
"on move: O\n",
"X O X \n",
"X \n",
" O \n",
"on move: X\n",
"X O X \n",
"X \n",
"O O \n",
"on move: O\n",
"X O X \n",
"X X \n",
"O O \n",
"on move: X\n",
"X O X \n",
"X O X \n",
"O O \n",
"on move: O\n",
"X O X \n",
"X O X \n",
"O X O \n",
"Episode 7, Total Reward: 0\n",
"Average Reward: 0.2857142857142857\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
"O \n",
"X \n",
" \n",
"on move: O\n",
"O \n",
"X \n",
" X \n",
"on move: X\n",
"O O \n",
"X \n",
" X \n",
"on move: O\n",
"O O \n",
"X \n",
" X X \n",
"on move: X\n",
"O O \n",
"X \n",
"O X X \n",
"on move: O\n",
"O O \n",
"X X \n",
"O X X \n",
"on move: X\n",
"O O \n",
"X X O \n",
"O X X \n",
"on move: O\n",
"O X O \n",
"X X O \n",
"O X X \n",
"Episode 8, Total Reward: 1\n",
"Average Reward: 0.375\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
"O \n",
" \n",
"on move: O\n",
"X \n",
"O X \n",
" \n",
"on move: X\n",
"X \n",
"O X \n",
" O \n",
"on move: O\n",
"X \n",
"O X \n",
"X O \n",
"on move: X\n",
"X \n",
"O X \n",
"X O O \n",
"on move: O\n",
"X \n",
"O X X \n",
"X O O \n",
"on move: X\n",
"X O \n",
"O X X \n",
"X O O \n",
"on move: O\n",
"X X O \n",
"O X X \n",
"X O O \n",
"Episode 9, Total Reward: 0\n",
"Average Reward: 0.3333333333333333\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
"O X \n",
" \n",
"on move: O\n",
" \n",
"O X X \n",
" \n",
"on move: X\n",
" \n",
"O X X \n",
" O \n",
"on move: O\n",
" \n",
"O X X \n",
"X O \n",
"on move: X\n",
"O \n",
"O X X \n",
"X O \n",
"on move: O\n",
"O \n",
"O X X \n",
"X O X \n",
"on move: X\n",
"O O \n",
"O X X \n",
"X O X \n",
"on move: O\n",
"O X O \n",
"O X X \n",
"X O X \n",
"Episode 10, Total Reward: 0\n",
"Average Reward: 0.3\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
"O \n",
" \n",
"on move: O\n",
" X \n",
"O \n",
"X \n",
"on move: X\n",
" X \n",
"O \n",
"X O \n",
"on move: O\n",
" X \n",
"O \n",
"X X O \n",
"on move: X\n",
" X \n",
"O O \n",
"X X O \n",
"on move: O\n",
" X X \n",
"O O \n",
"X X O \n",
"on move: X\n",
" X X \n",
"O O O \n",
"X X O \n",
"Episode 11, Total Reward: -1\n",
"Average Reward: 0.18181818181818182\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X O \n",
" \n",
" \n",
"on move: O\n",
"X O \n",
"X \n",
" \n",
"on move: X\n",
"X O \n",
"X \n",
"O \n",
"on move: O\n",
"X O X \n",
"X \n",
"O \n",
"on move: X\n",
"X O X \n",
"X O \n",
"O \n",
"on move: O\n",
"X O X \n",
"X O \n",
"O X \n",
"on move: X\n",
"X O X \n",
"X O O \n",
"O X \n",
"on move: O\n",
"X O X \n",
"X O O \n",
"O X X \n",
"Episode 12, Total Reward: 0\n",
"Average Reward: 0.16666666666666666\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
"X \n",
" X \n",
"on move: X\n",
" O \n",
"X \n",
"O X \n",
"on move: O\n",
" O \n",
"X X \n",
"O X \n",
"on move: X\n",
" O O \n",
"X X \n",
"O X \n",
"on move: O\n",
"X O O \n",
"X X \n",
"O X \n",
"on move: X\n",
"X O O \n",
"X O X \n",
"O X \n",
"Episode 13, Total Reward: -1\n",
"Average Reward: 0.07692307692307693\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
"X \n",
" X \n",
" O \n",
"on move: X\n",
"X \n",
" X \n",
" O O \n",
"on move: O\n",
"X \n",
"X X \n",
" O O \n",
"on move: X\n",
"X \n",
"X X \n",
"O O O \n",
"Episode 14, Total Reward: -1\n",
"Average Reward: 0.0\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
"O X \n",
" \n",
" \n",
"on move: O\n",
"O X \n",
" \n",
"X \n",
"on move: X\n",
"O X \n",
" \n",
"X O \n",
"on move: O\n",
"O X \n",
" \n",
"X X O \n",
"on move: X\n",
"O X \n",
" O \n",
"X X O \n",
"Episode 15, Total Reward: -1\n",
"Average Reward: -0.06666666666666667\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
"O \n",
" X \n",
"on move: O\n",
" X \n",
"O \n",
" X \n",
"on move: X\n",
" X \n",
"O O \n",
" X \n",
"on move: O\n",
" X \n",
"O X O \n",
" X \n",
"on move: X\n",
" X \n",
"O X O \n",
" X O \n",
"on move: O\n",
" X \n",
"O X O \n",
"X X O \n",
"Episode 16, Total Reward: 1\n",
"Average Reward: 0.0\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
"X X \n",
" \n",
" O \n",
"on move: X\n",
"X X O \n",
" \n",
" O \n",
"on move: O\n",
"X X O \n",
"X \n",
" O \n",
"on move: X\n",
"X X O \n",
"X \n",
"O O \n",
"on move: O\n",
"X X O \n",
"X X \n",
"O O \n",
"on move: X\n",
"X X O \n",
"X X \n",
"O O O \n",
"Episode 17, Total Reward: -1\n",
"Average Reward: -0.058823529411764705\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
"X X \n",
" \n",
" O \n",
"on move: X\n",
"X X \n",
" O \n",
" O \n",
"on move: O\n",
"X X \n",
" O \n",
"X O \n",
"on move: X\n",
"X X \n",
"O O \n",
"X O \n",
"on move: O\n",
"X X \n",
"O O \n",
"X O X \n",
"on move: X\n",
"X X \n",
"O O O \n",
"X O X \n",
"Episode 18, Total Reward: -1\n",
"Average Reward: -0.1111111111111111\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" O \n",
" X \n",
"on move: O\n",
" \n",
" O \n",
"X X \n",
"on move: X\n",
" O \n",
" O \n",
"X X \n",
"on move: O\n",
" X O \n",
" O \n",
"X X \n",
"on move: X\n",
" X O \n",
" O \n",
"X X O \n",
"on move: O\n",
"X X O \n",
" O \n",
"X X O \n",
"on move: X\n",
"X X O \n",
" O O \n",
"X X O \n",
"Episode 19, Total Reward: -1\n",
"Average Reward: -0.15789473684210525\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" \n",
"X X \n",
"on move: X\n",
" O \n",
" \n",
"X O X \n",
"on move: O\n",
" X O \n",
" \n",
"X O X \n",
"on move: X\n",
" X O \n",
" O \n",
"X O X \n",
"on move: O\n",
" X O \n",
"X O \n",
"X O X \n",
"on move: X\n",
"O X O \n",
"X O \n",
"X O X \n",
"on move: O\n",
"O X O \n",
"X O X \n",
"X O X \n",
"Episode 20, Total Reward: 0\n",
"Average Reward: -0.15\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X \n",
" O \n",
"on move: O\n",
" \n",
"X X \n",
" O \n",
"on move: X\n",
" \n",
"X O X \n",
" O \n",
"on move: O\n",
"X \n",
"X O X \n",
" O \n",
"on move: X\n",
"X \n",
"X O X \n",
" O O \n",
"on move: O\n",
"X X \n",
"X O X \n",
" O O \n",
"on move: X\n",
"X X O \n",
"X O X \n",
" O O \n",
"on move: O\n",
"X X O \n",
"X O X \n",
"X O O \n",
"Episode 21, Total Reward: 1\n",
"Average Reward: -0.09523809523809523\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
"O \n",
"on move: O\n",
"X \n",
" X \n",
"O \n",
"on move: X\n",
"X \n",
"O X \n",
"O \n",
"on move: O\n",
"X \n",
"O X X \n",
"O \n",
"on move: X\n",
"X \n",
"O X X \n",
"O O \n",
"on move: O\n",
"X \n",
"O X X \n",
"O O X \n",
"Episode 22, Total Reward: 1\n",
"Average Reward: -0.045454545454545456\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" X \n",
" X \n",
"on move: X\n",
" O \n",
" X \n",
" O X \n",
"on move: O\n",
" O \n",
" X X \n",
" O X \n",
"on move: X\n",
" O O \n",
" X X \n",
" O X \n",
"on move: O\n",
"X O O \n",
" X X \n",
" O X \n",
"Episode 23, Total Reward: 1\n",
"Average Reward: 0.0\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X \n",
"O \n",
"on move: O\n",
" X \n",
"X \n",
"O \n",
"on move: X\n",
"O X \n",
"X \n",
"O \n",
"on move: O\n",
"O X \n",
"X \n",
"O X \n",
"on move: X\n",
"O X \n",
"X O \n",
"O X \n",
"on move: O\n",
"O X \n",
"X O \n",
"O X X \n",
"on move: X\n",
"O X \n",
"X O O \n",
"O X X \n",
"on move: O\n",
"O X X \n",
"X O O \n",
"O X X \n",
"Episode 24, Total Reward: 0\n",
"Average Reward: 0.0\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" O \n",
"X \n",
" \n",
"on move: O\n",
" O X \n",
"X \n",
" \n",
"on move: X\n",
" O X \n",
"X \n",
" O \n",
"on move: O\n",
" O X \n",
"X \n",
"X O \n",
"on move: X\n",
" O X \n",
"X \n",
"X O O \n",
"on move: O\n",
"X O X \n",
"X \n",
"X O O \n",
"Episode 25, Total Reward: 1\n",
"Average Reward: 0.04\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
" X \n",
" X \n",
" O \n",
"on move: X\n",
" X \n",
" X \n",
"O O \n",
"on move: O\n",
" X X \n",
" X \n",
"O O \n",
"on move: X\n",
" X X \n",
" X \n",
"O O O \n",
"Episode 26, Total Reward: -1\n",
"Average Reward: 0.0\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X O \n",
" \n",
" \n",
"on move: O\n",
"X O \n",
" X \n",
" \n",
"on move: X\n",
"X O \n",
" X \n",
"O \n",
"on move: O\n",
"X O \n",
" X X \n",
"O \n",
"on move: X\n",
"X O \n",
" X X \n",
"O O \n",
"on move: O\n",
"X O \n",
" X X \n",
"O O X \n",
"Episode 27, Total Reward: 1\n",
"Average Reward: 0.037037037037037035\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
"O \n",
" X \n",
" \n",
"on move: O\n",
"O \n",
" X \n",
"X \n",
"on move: X\n",
"O \n",
" X \n",
"X O \n",
"on move: O\n",
"O X \n",
" X \n",
"X O \n",
"on move: X\n",
"O X \n",
"O X \n",
"X O \n",
"on move: O\n",
"O X \n",
"O X \n",
"X X O \n",
"Episode 28, Total Reward: 1\n",
"Average Reward: 0.07142857142857142\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X O \n",
" \n",
"on move: O\n",
" X \n",
" X O \n",
" \n",
"on move: X\n",
" X \n",
"O X O \n",
" \n",
"on move: O\n",
" X \n",
"O X O \n",
" X \n",
"Episode 29, Total Reward: 1\n",
"Average Reward: 0.10344827586206896\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
" X \n",
" X \n",
" O \n",
"on move: X\n",
" X \n",
"O X \n",
" O \n",
"on move: O\n",
" X X \n",
"O X \n",
" O \n",
"on move: X\n",
"O X X \n",
"O X \n",
" O \n",
"on move: O\n",
"O X X \n",
"O X \n",
"X O \n",
"Episode 30, Total Reward: 1\n",
"Average Reward: 0.13333333333333333\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" \n",
" \n",
"X O \n",
"on move: O\n",
" X \n",
" \n",
"X O \n",
"on move: X\n",
" X \n",
" \n",
"X O O \n",
"on move: O\n",
" X \n",
" X \n",
"X O O \n",
"on move: X\n",
"O X \n",
" X \n",
"X O O \n",
"on move: O\n",
"O X \n",
"X X \n",
"X O O \n",
"on move: X\n",
"O X \n",
"X O X \n",
"X O O \n",
"Episode 31, Total Reward: -1\n",
"Average Reward: 0.0967741935483871\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" O X \n",
" \n",
"on move: O\n",
" \n",
" O X \n",
" X \n",
"on move: X\n",
" O \n",
" O X \n",
" X \n",
"on move: O\n",
"X O \n",
" O X \n",
" X \n",
"on move: X\n",
"X O O \n",
" O X \n",
" X \n",
"on move: O\n",
"X O O \n",
"X O X \n",
" X \n",
"on move: X\n",
"X O O \n",
"X O X \n",
" O X \n",
"Episode 32, Total Reward: -1\n",
"Average Reward: 0.0625\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
"O \n",
" \n",
"X \n",
"on move: O\n",
"O \n",
" X \n",
"X \n",
"on move: X\n",
"O O \n",
" X \n",
"X \n",
"on move: O\n",
"O O \n",
" X X \n",
"X \n",
"on move: X\n",
"O O \n",
" X X \n",
"X O \n",
"on move: O\n",
"O O \n",
" X X \n",
"X O X \n",
"on move: X\n",
"O O \n",
"O X X \n",
"X O X \n",
"on move: O\n",
"O O X \n",
"O X X \n",
"X O X \n",
"Episode 33, Total Reward: 1\n",
"Average Reward: 0.09090909090909091\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" \n",
" O \n",
"on move: O\n",
"X \n",
" X \n",
" O \n",
"on move: X\n",
"X O \n",
" X \n",
" O \n",
"on move: O\n",
"X O \n",
" X \n",
" X O \n",
"on move: X\n",
"X O \n",
"O X \n",
" X O \n",
"on move: O\n",
"X O \n",
"O X X \n",
" X O \n",
"on move: X\n",
"X O O \n",
"O X X \n",
" X O \n",
"on move: O\n",
"X O O \n",
"O X X \n",
"X X O \n",
"Episode 34, Total Reward: 0\n",
"Average Reward: 0.08823529411764706\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
"X X \n",
" \n",
" O \n",
"on move: X\n",
"X X \n",
"O \n",
" O \n",
"on move: O\n",
"X X \n",
"O \n",
"X O \n",
"on move: X\n",
"X X \n",
"O O \n",
"X O \n",
"on move: O\n",
"X X \n",
"O O \n",
"X X O \n",
"on move: X\n",
"X X \n",
"O O O \n",
"X X O \n",
"Episode 35, Total Reward: -1\n",
"Average Reward: 0.05714285714285714\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" O X \n",
" \n",
" \n",
"on move: O\n",
"X O X \n",
" \n",
" \n",
"on move: X\n",
"X O X \n",
" O \n",
" \n",
"on move: O\n",
"X O X \n",
" O \n",
" X \n",
"on move: X\n",
"X O X \n",
" O O \n",
" X \n",
"on move: O\n",
"X O X \n",
" O O \n",
"X X \n",
"on move: X\n",
"X O X \n",
" O O \n",
"X O X \n",
"Episode 36, Total Reward: -1\n",
"Average Reward: 0.027777777777777776\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" O \n",
"X \n",
" \n",
"on move: O\n",
" O \n",
"X \n",
" X \n",
"on move: X\n",
"O O \n",
"X \n",
" X \n",
"on move: O\n",
"O O X \n",
"X \n",
" X \n",
"on move: X\n",
"O O X \n",
"X \n",
"O X \n",
"on move: O\n",
"O O X \n",
"X X \n",
"O X \n",
"Episode 37, Total Reward: 1\n",
"Average Reward: 0.05405405405405406\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" O X \n",
" \n",
"on move: O\n",
"X \n",
" O X \n",
" \n",
"on move: X\n",
"X \n",
" O X \n",
"O \n",
"on move: O\n",
"X \n",
" O X \n",
"O X \n",
"on move: X\n",
"X O \n",
" O X \n",
"O X \n",
"on move: O\n",
"X O \n",
"X O X \n",
"O X \n",
"on move: X\n",
"X O \n",
"X O X \n",
"O O X \n",
"Episode 38, Total Reward: -1\n",
"Average Reward: 0.02631578947368421\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X O \n",
" \n",
" \n",
"on move: O\n",
"X O \n",
"X \n",
" \n",
"on move: X\n",
"X O O \n",
"X \n",
" \n",
"on move: O\n",
"X O O \n",
"X \n",
" X \n",
"on move: X\n",
"X O O \n",
"X \n",
" O X \n",
"on move: O\n",
"X O O \n",
"X X \n",
" O X \n",
"on move: X\n",
"X O O \n",
"X O X \n",
" O X \n",
"Episode 39, Total Reward: -1\n",
"Average Reward: 0.0\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
" X \n",
"X \n",
" O \n",
"on move: X\n",
" X \n",
"X \n",
" O O \n",
"on move: O\n",
" X \n",
"X \n",
"X O O \n",
"on move: X\n",
" X \n",
"X O \n",
"X O O \n",
"on move: O\n",
" X X \n",
"X O \n",
"X O O \n",
"on move: X\n",
"O X X \n",
"X O \n",
"X O O \n",
"Episode 40, Total Reward: -1\n",
"Average Reward: -0.025\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
"O X \n",
" \n",
"on move: O\n",
" \n",
"O X \n",
" X \n",
"on move: X\n",
" \n",
"O X \n",
"O X \n",
"on move: O\n",
" X \n",
"O X \n",
"O X \n",
"on move: X\n",
" X \n",
"O X O \n",
"O X \n",
"on move: O\n",
"X X \n",
"O X O \n",
"O X \n",
"Episode 41, Total Reward: 1\n",
"Average Reward: 0.0\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
" X \n",
"X \n",
" O \n",
"on move: X\n",
" X O \n",
"X \n",
" O \n",
"on move: O\n",
" X O \n",
"X \n",
" O X \n",
"on move: X\n",
" X O \n",
"X \n",
"O O X \n",
"on move: O\n",
" X O \n",
"X X \n",
"O O X \n",
"on move: X\n",
" X O \n",
"X O X \n",
"O O X \n",
"Episode 42, Total Reward: -1\n",
"Average Reward: -0.023809523809523808\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X O \n",
" \n",
" \n",
"on move: O\n",
"X O \n",
"X \n",
" \n",
"on move: X\n",
"X O \n",
"X \n",
"O \n",
"on move: O\n",
"X O \n",
"X \n",
"O X \n",
"on move: X\n",
"X O \n",
"X O \n",
"O X \n",
"on move: O\n",
"X O \n",
"X O \n",
"O X X \n",
"on move: X\n",
"X O \n",
"X O O \n",
"O X X \n",
"on move: O\n",
"X O X \n",
"X O O \n",
"O X X \n",
"Episode 43, Total Reward: 0\n",
"Average Reward: -0.023255813953488372\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" \n",
" O \n",
"on move: O\n",
"X \n",
" \n",
" O X \n",
"on move: X\n",
"X O \n",
" \n",
" O X \n",
"on move: O\n",
"X O X \n",
" \n",
" O X \n",
"on move: X\n",
"X O X \n",
" \n",
"O O X \n",
"on move: O\n",
"X O X \n",
" X \n",
"O O X \n",
"Episode 44, Total Reward: 1\n",
"Average Reward: 0.0\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
"X O \n",
" \n",
" X \n",
"on move: X\n",
"X O \n",
" O \n",
" X \n",
"on move: O\n",
"X O \n",
" O \n",
" X X \n",
"on move: X\n",
"X O \n",
"O O \n",
" X X \n",
"on move: O\n",
"X O \n",
"O O \n",
"X X X \n",
"Episode 45, Total Reward: 1\n",
"Average Reward: 0.022222222222222223\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X O \n",
" \n",
" \n",
"on move: O\n",
"X O \n",
" X \n",
" \n",
"on move: X\n",
"X O \n",
" X \n",
"O \n",
"on move: O\n",
"X O X \n",
" X \n",
"O \n",
"on move: X\n",
"X O X \n",
" X \n",
"O O \n",
"on move: O\n",
"X O X \n",
"X X \n",
"O O \n",
"on move: X\n",
"X O X \n",
"X X O \n",
"O O \n",
"on move: O\n",
"X O X \n",
"X X O \n",
"O X O \n",
"Episode 46, Total Reward: 0\n",
"Average Reward: 0.021739130434782608\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" O \n",
" \n",
"on move: O\n",
"X \n",
" O \n",
" X \n",
"on move: X\n",
"X \n",
" O \n",
" O X \n",
"on move: O\n",
"X \n",
" O \n",
"X O X \n",
"on move: X\n",
"X O \n",
" O \n",
"X O X \n",
"on move: O\n",
"X O \n",
"X O \n",
"X O X \n",
"Episode 47, Total Reward: 1\n",
"Average Reward: 0.0425531914893617\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" O \n",
" \n",
"X \n",
"on move: O\n",
" O \n",
" \n",
"X X \n",
"on move: X\n",
" O \n",
" \n",
"X X O \n",
"on move: O\n",
" O \n",
"X \n",
"X X O \n",
"on move: X\n",
" O \n",
"X O \n",
"X X O \n",
"on move: O\n",
"X O \n",
"X O \n",
"X X O \n",
"Episode 48, Total Reward: 1\n",
"Average Reward: 0.0625\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
"O \n",
" \n",
" X \n",
"on move: O\n",
"O \n",
" X \n",
" X \n",
"on move: X\n",
"O \n",
" X \n",
" X O \n",
"on move: O\n",
"O X \n",
" X \n",
" X O \n",
"on move: X\n",
"O X \n",
" X \n",
"O X O \n",
"on move: O\n",
"O X X \n",
" X \n",
"O X O \n",
"Episode 49, Total Reward: 1\n",
"Average Reward: 0.08163265306122448\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
"O \n",
"on move: O\n",
" \n",
" X X \n",
"O \n",
"on move: X\n",
" O \n",
" X X \n",
"O \n",
"on move: O\n",
" O \n",
" X X \n",
"O X \n",
"on move: X\n",
" O \n",
"O X X \n",
"O X \n",
"on move: O\n",
"X O \n",
"O X X \n",
"O X \n",
"Episode 50, Total Reward: 1\n",
"Average Reward: 0.1\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" \n",
"O X \n",
"on move: O\n",
" \n",
" X \n",
"O X \n",
"on move: X\n",
" \n",
" X \n",
"O O X \n",
"on move: O\n",
" X \n",
" X \n",
"O O X \n",
"Episode 51, Total Reward: 1\n",
"Average Reward: 0.11764705882352941\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
"O X \n",
" \n",
" \n",
"on move: O\n",
"O X X \n",
" \n",
" \n",
"on move: X\n",
"O X X \n",
" \n",
"O \n",
"on move: O\n",
"O X X \n",
" \n",
"O X \n",
"on move: X\n",
"O X X \n",
" \n",
"O X O \n",
"on move: O\n",
"O X X \n",
" X \n",
"O X O \n",
"Episode 52, Total Reward: 1\n",
"Average Reward: 0.1346153846153846\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" O \n",
" \n",
"on move: O\n",
" X \n",
" O X \n",
" \n",
"on move: X\n",
" X \n",
" O X \n",
"O \n",
"on move: O\n",
" X X \n",
" O X \n",
"O \n",
"on move: X\n",
" X X \n",
" O X \n",
"O O \n",
"on move: O\n",
" X X \n",
"X O X \n",
"O O \n",
"on move: X\n",
"O X X \n",
"X O X \n",
"O O \n",
"on move: O\n",
"O X X \n",
"X O X \n",
"O O X \n",
"Episode 53, Total Reward: 1\n",
"Average Reward: 0.1509433962264151\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
" X \n",
" \n",
"X O \n",
"on move: X\n",
" X O \n",
" \n",
"X O \n",
"on move: O\n",
" X O \n",
" \n",
"X O X \n",
"on move: X\n",
" X O \n",
"O \n",
"X O X \n",
"on move: O\n",
" X O \n",
"O X \n",
"X O X \n",
"on move: X\n",
" X O \n",
"O X O \n",
"X O X \n",
"on move: O\n",
"X X O \n",
"O X O \n",
"X O X \n",
"Episode 54, Total Reward: 1\n",
"Average Reward: 0.16666666666666666\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
" O \n",
" X \n",
" X \n",
"on move: X\n",
" O \n",
" O X \n",
" X \n",
"on move: O\n",
" X O \n",
" O X \n",
" X \n",
"on move: X\n",
" X O \n",
"O O X \n",
" X \n",
"on move: O\n",
" X O \n",
"O O X \n",
" X X \n",
"on move: X\n",
"O X O \n",
"O O X \n",
" X X \n",
"on move: O\n",
"O X O \n",
"O O X \n",
"X X X \n",
"Episode 55, Total Reward: 1\n",
"Average Reward: 0.18181818181818182\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
"O X \n",
" \n",
" \n",
"on move: O\n",
"O X X \n",
" \n",
" \n",
"on move: X\n",
"O X X \n",
" \n",
" O \n",
"on move: O\n",
"O X X \n",
" X \n",
" O \n",
"on move: X\n",
"O X X \n",
"O X \n",
" O \n",
"on move: O\n",
"O X X \n",
"O X \n",
" X O \n",
"on move: X\n",
"O X X \n",
"O X \n",
"O X O \n",
"Episode 56, Total Reward: -1\n",
"Average Reward: 0.16071428571428573\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
"O \n",
" \n",
"on move: O\n",
"X \n",
"O \n",
" X \n",
"on move: X\n",
"X \n",
"O O \n",
" X \n",
"on move: O\n",
"X \n",
"O O \n",
"X X \n",
"on move: X\n",
"X \n",
"O O \n",
"X O X \n",
"on move: O\n",
"X X \n",
"O O \n",
"X O X \n",
"on move: X\n",
"X X \n",
"O O O \n",
"X O X \n",
"Episode 57, Total Reward: -1\n",
"Average Reward: 0.14035087719298245\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" O X \n",
" \n",
" \n",
"on move: O\n",
" O X \n",
" \n",
" X \n",
"on move: X\n",
" O X \n",
" \n",
"O X \n",
"on move: O\n",
" O X \n",
" X \n",
"O X \n",
"Episode 58, Total Reward: 1\n",
"Average Reward: 0.15517241379310345\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X \n",
"O \n",
"on move: O\n",
" \n",
"X \n",
"O X \n",
"on move: X\n",
" \n",
"X O \n",
"O X \n",
"on move: O\n",
" \n",
"X O \n",
"O X X \n",
"on move: X\n",
" O \n",
"X O \n",
"O X X \n",
"on move: O\n",
" O X \n",
"X O \n",
"O X X \n",
"on move: X\n",
"O O X \n",
"X O \n",
"O X X \n",
"on move: O\n",
"O O X \n",
"X X O \n",
"O X X \n",
"Episode 59, Total Reward: 0\n",
"Average Reward: 0.15254237288135594\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
" X \n",
" \n",
"X O \n",
"on move: X\n",
" O X \n",
" \n",
"X O \n",
"on move: O\n",
" O X \n",
"X \n",
"X O \n",
"on move: X\n",
" O X \n",
"X \n",
"X O O \n",
"on move: O\n",
"X O X \n",
"X \n",
"X O O \n",
"Episode 60, Total Reward: 1\n",
"Average Reward: 0.16666666666666666\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" \n",
" O \n",
"on move: O\n",
"X \n",
" X \n",
" O \n",
"on move: X\n",
"X O \n",
" X \n",
" O \n",
"on move: O\n",
"X O \n",
" X X \n",
" O \n",
"on move: X\n",
"X O \n",
" X X \n",
"O O \n",
"on move: O\n",
"X O \n",
" X X \n",
"O X O \n",
"on move: X\n",
"X O O \n",
" X X \n",
"O X O \n",
"on move: O\n",
"X O O \n",
"X X X \n",
"O X O \n",
"Episode 61, Total Reward: 1\n",
"Average Reward: 0.18032786885245902\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
"O \n",
"on move: O\n",
" \n",
" X \n",
"O X \n",
"on move: X\n",
" O \n",
" X \n",
"O X \n",
"on move: O\n",
"X O \n",
" X \n",
"O X \n",
"on move: X\n",
"X O \n",
" X \n",
"O X O \n",
"on move: O\n",
"X O \n",
"X X \n",
"O X O \n",
"on move: X\n",
"X O \n",
"X O X \n",
"O X O \n",
"on move: O\n",
"X O X \n",
"X O X \n",
"O X O \n",
"Episode 62, Total Reward: 0\n",
"Average Reward: 0.1774193548387097\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" O \n",
" \n",
"on move: O\n",
"X \n",
" O \n",
" X \n",
"on move: X\n",
"X \n",
" O \n",
"O X \n",
"on move: O\n",
"X \n",
" O \n",
"O X X \n",
"on move: X\n",
"X \n",
"O O \n",
"O X X \n",
"on move: O\n",
"X X \n",
"O O \n",
"O X X \n",
"on move: X\n",
"X X \n",
"O O O \n",
"O X X \n",
"Episode 63, Total Reward: -1\n",
"Average Reward: 0.15873015873015872\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
"O \n",
" \n",
"X \n",
"on move: O\n",
"O \n",
"X \n",
"X \n",
"on move: X\n",
"O \n",
"X O \n",
"X \n",
"on move: O\n",
"O X \n",
"X O \n",
"X \n",
"on move: X\n",
"O O X \n",
"X O \n",
"X \n",
"on move: O\n",
"O O X \n",
"X X O \n",
"X \n",
"Episode 64, Total Reward: 1\n",
"Average Reward: 0.171875\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" X \n",
" X \n",
"on move: X\n",
" O \n",
" X \n",
"O X \n",
"on move: O\n",
" X O \n",
" X \n",
"O X \n",
"Episode 65, Total Reward: 1\n",
"Average Reward: 0.18461538461538463\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" O \n",
" \n",
"on move: O\n",
"X \n",
" O \n",
" X \n",
"on move: X\n",
"X \n",
" O O \n",
" X \n",
"on move: O\n",
"X X \n",
" O O \n",
" X \n",
"on move: X\n",
"X X \n",
" O O \n",
" X O \n",
"on move: O\n",
"X X X \n",
" O O \n",
" X O \n",
"Episode 66, Total Reward: 1\n",
"Average Reward: 0.19696969696969696\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
"O \n",
"on move: O\n",
" \n",
" X X \n",
"O \n",
"on move: X\n",
" O \n",
" X X \n",
"O \n",
"on move: O\n",
" X O \n",
" X X \n",
"O \n",
"on move: X\n",
" X O \n",
"O X X \n",
"O \n",
"on move: O\n",
" X O \n",
"O X X \n",
"O X \n",
"on move: X\n",
"O X O \n",
"O X X \n",
"O X \n",
"Episode 67, Total Reward: -1\n",
"Average Reward: 0.1791044776119403\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" X \n",
" X \n",
"on move: X\n",
" O \n",
"O X \n",
" X \n",
"on move: O\n",
" O X \n",
"O X \n",
" X \n",
"on move: X\n",
" O X \n",
"O X \n",
"O X \n",
"on move: O\n",
" O X \n",
"O X X \n",
"O X \n",
"on move: X\n",
" O X \n",
"O X X \n",
"O X O \n",
"on move: O\n",
"X O X \n",
"O X X \n",
"O X O \n",
"Episode 68, Total Reward: 0\n",
"Average Reward: 0.17647058823529413\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" \n",
"O X \n",
"on move: O\n",
" \n",
" X \n",
"O X \n",
"on move: X\n",
" \n",
" X \n",
"O O X \n",
"on move: O\n",
" X \n",
" X \n",
"O O X \n",
"on move: X\n",
"O X \n",
" X \n",
"O O X \n",
"on move: O\n",
"O X \n",
" X X \n",
"O O X \n",
"on move: X\n",
"O X O \n",
" X X \n",
"O O X \n",
"on move: O\n",
"O X O \n",
"X X X \n",
"O O X \n",
"Episode 69, Total Reward: 1\n",
"Average Reward: 0.18840579710144928\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
"X \n",
" X \n",
"on move: X\n",
" O O \n",
"X \n",
" X \n",
"on move: O\n",
" O O \n",
"X X \n",
" X \n",
"on move: X\n",
" O O \n",
"X X \n",
" O X \n",
"on move: O\n",
" O O \n",
"X X \n",
"X O X \n",
"on move: X\n",
" O O \n",
"X O X \n",
"X O X \n",
"Episode 70, Total Reward: -1\n",
"Average Reward: 0.17142857142857143\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" \n",
" O \n",
"X \n",
"on move: O\n",
"X \n",
" O \n",
"X \n",
"on move: X\n",
"X \n",
" O \n",
"X O \n",
"on move: O\n",
"X \n",
" O X \n",
"X O \n",
"on move: X\n",
"X O \n",
" O X \n",
"X O \n",
"Episode 71, Total Reward: -1\n",
"Average Reward: 0.15492957746478872\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" O \n",
" \n",
"on move: O\n",
"X \n",
"X O \n",
" \n",
"on move: X\n",
"X O \n",
"X O \n",
" \n",
"on move: O\n",
"X O \n",
"X O \n",
" X \n",
"on move: X\n",
"X O \n",
"X O \n",
" O X \n",
"on move: O\n",
"X O \n",
"X X O \n",
" O X \n",
"Episode 72, Total Reward: 1\n",
"Average Reward: 0.16666666666666666\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X O \n",
" \n",
"on move: O\n",
" \n",
" X O \n",
" X \n",
"on move: X\n",
" O \n",
" X O \n",
" X \n",
"on move: O\n",
" O \n",
" X O \n",
" X X \n",
"on move: X\n",
" O \n",
"O X O \n",
" X X \n",
"on move: O\n",
"X O \n",
"O X O \n",
" X X \n",
"Episode 73, Total Reward: 1\n",
"Average Reward: 0.1780821917808219\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X \n",
" O \n",
"on move: O\n",
" X \n",
"X \n",
" O \n",
"on move: X\n",
" X \n",
"X O \n",
" O \n",
"on move: O\n",
" X \n",
"X O X \n",
" O \n",
"on move: X\n",
" X \n",
"X O X \n",
"O O \n",
"on move: O\n",
"X X \n",
"X O X \n",
"O O \n",
"on move: X\n",
"X X O \n",
"X O X \n",
"O O \n",
"Episode 74, Total Reward: -1\n",
"Average Reward: 0.16216216216216217\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" \n",
"O X \n",
"on move: O\n",
" \n",
" \n",
"O X X \n",
"on move: X\n",
" O \n",
" \n",
"O X X \n",
"on move: O\n",
" X O \n",
" \n",
"O X X \n",
"on move: X\n",
" X O \n",
"O \n",
"O X X \n",
"on move: O\n",
" X O \n",
"O X \n",
"O X X \n",
"Episode 75, Total Reward: 1\n",
"Average Reward: 0.17333333333333334\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
"O \n",
" \n",
"X \n",
"on move: O\n",
"O \n",
" \n",
"X X \n",
"on move: X\n",
"O \n",
" O \n",
"X X \n",
"on move: O\n",
"O \n",
" O \n",
"X X X \n",
"Episode 76, Total Reward: 1\n",
"Average Reward: 0.18421052631578946\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
"O \n",
" X \n",
"on move: O\n",
" \n",
"O \n",
" X X \n",
"on move: X\n",
" O \n",
"O \n",
" X X \n",
"on move: O\n",
" O \n",
"O X \n",
" X X \n",
"on move: X\n",
" O \n",
"O X \n",
"O X X \n",
"on move: O\n",
" O \n",
"O X X \n",
"O X X \n",
"on move: X\n",
" O O \n",
"O X X \n",
"O X X \n",
"on move: O\n",
"X O O \n",
"O X X \n",
"O X X \n",
"Episode 77, Total Reward: 1\n",
"Average Reward: 0.19480519480519481\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
"O \n",
"on move: O\n",
" X \n",
" \n",
"O X \n",
"on move: X\n",
" X \n",
" O \n",
"O X \n",
"on move: O\n",
"X X \n",
" O \n",
"O X \n",
"on move: X\n",
"X X \n",
"O O \n",
"O X \n",
"on move: O\n",
"X X \n",
"O O \n",
"O X X \n",
"on move: X\n",
"X X \n",
"O O O \n",
"O X X \n",
"Episode 78, Total Reward: -1\n",
"Average Reward: 0.1794871794871795\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
" O X \n",
" X \n",
" \n",
"on move: X\n",
" O X \n",
" X \n",
" O \n",
"on move: O\n",
"X O X \n",
" X \n",
" O \n",
"on move: X\n",
"X O X \n",
" X \n",
" O O \n",
"on move: O\n",
"X O X \n",
"X X \n",
" O O \n",
"on move: X\n",
"X O X \n",
"X X O \n",
" O O \n",
"on move: O\n",
"X O X \n",
"X X O \n",
"X O O \n",
"Episode 79, Total Reward: 1\n",
"Average Reward: 0.189873417721519\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" \n",
" X X \n",
"on move: X\n",
"O O \n",
" \n",
" X X \n",
"on move: O\n",
"O O \n",
"X \n",
" X X \n",
"on move: X\n",
"O O O \n",
"X \n",
" X X \n",
"Episode 80, Total Reward: -1\n",
"Average Reward: 0.175\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
"O \n",
" \n",
"X \n",
"on move: O\n",
"O \n",
" X \n",
"X \n",
"on move: X\n",
"O \n",
" O X \n",
"X \n",
"on move: O\n",
"O \n",
" O X \n",
"X X \n",
"on move: X\n",
"O \n",
" O X \n",
"X O X \n",
"on move: O\n",
"O X \n",
" O X \n",
"X O X \n",
"Episode 81, Total Reward: 1\n",
"Average Reward: 0.18518518518518517\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" \n",
" \n",
"X O \n",
"on move: O\n",
" \n",
" X \n",
"X O \n",
"on move: X\n",
"O \n",
" X \n",
"X O \n",
"on move: O\n",
"O X \n",
" X \n",
"X O \n",
"Episode 82, Total Reward: 1\n",
"Average Reward: 0.1951219512195122\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
" \n",
" X \n",
" O X \n",
"on move: X\n",
" \n",
" O X \n",
" O X \n",
"on move: O\n",
" X \n",
" O X \n",
" O X \n",
"on move: X\n",
" X \n",
"O O X \n",
" O X \n",
"on move: O\n",
" X \n",
"O O X \n",
"X O X \n",
"on move: X\n",
" X O \n",
"O O X \n",
"X O X \n",
"on move: O\n",
"X X O \n",
"O O X \n",
"X O X \n",
"Episode 83, Total Reward: 0\n",
"Average Reward: 0.1927710843373494\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
"O X \n",
" \n",
"on move: O\n",
" X \n",
"O X \n",
" \n",
"on move: X\n",
" X \n",
"O X \n",
" O \n",
"on move: O\n",
" X X \n",
"O X \n",
" O \n",
"on move: X\n",
" X X \n",
"O X \n",
" O O \n",
"on move: O\n",
"X X X \n",
"O X \n",
" O O \n",
"Episode 84, Total Reward: 1\n",
"Average Reward: 0.20238095238095238\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" O \n",
" \n",
"X \n",
"on move: O\n",
" O \n",
" \n",
"X X \n",
"on move: X\n",
"O O \n",
" \n",
"X X \n",
"on move: O\n",
"O O \n",
"X \n",
"X X \n",
"on move: X\n",
"O O \n",
"X \n",
"X X O \n",
"on move: O\n",
"O O X \n",
"X \n",
"X X O \n",
"on move: X\n",
"O O X \n",
"X O \n",
"X X O \n",
"Episode 85, Total Reward: -1\n",
"Average Reward: 0.18823529411764706\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" O X \n",
" \n",
" \n",
"on move: O\n",
" O X \n",
" \n",
" X \n",
"on move: X\n",
" O X \n",
" O \n",
" X \n",
"on move: O\n",
" O X \n",
" O \n",
" X X \n",
"on move: X\n",
" O X \n",
" O O \n",
" X X \n",
"on move: O\n",
"X O X \n",
" O O \n",
" X X \n",
"on move: X\n",
"X O X \n",
"O O O \n",
" X X \n",
"Episode 86, Total Reward: -1\n",
"Average Reward: 0.1744186046511628\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" O \n",
" \n",
"on move: O\n",
" X X \n",
" O \n",
" \n",
"on move: X\n",
" X X \n",
" O O \n",
" \n",
"on move: O\n",
"X X X \n",
" O O \n",
" \n",
"Episode 87, Total Reward: 1\n",
"Average Reward: 0.1839080459770115\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X \n",
" O \n",
"on move: O\n",
" X \n",
"X \n",
" O \n",
"on move: X\n",
" X \n",
"X \n",
" O O \n",
"on move: O\n",
" X \n",
"X \n",
"X O O \n",
"on move: X\n",
" X \n",
"X O \n",
"X O O \n",
"on move: O\n",
" X X \n",
"X O \n",
"X O O \n",
"on move: X\n",
" X X \n",
"X O O \n",
"X O O \n",
"on move: O\n",
"X X X \n",
"X O O \n",
"X O O \n",
"Episode 88, Total Reward: 1\n",
"Average Reward: 0.19318181818181818\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
"O \n",
" \n",
" X \n",
"on move: O\n",
"O X \n",
" \n",
" X \n",
"on move: X\n",
"O X \n",
"O \n",
" X \n",
"on move: O\n",
"O X X \n",
"O \n",
" X \n",
"on move: X\n",
"O X X \n",
"O O \n",
" X \n",
"on move: O\n",
"O X X \n",
"O O \n",
"X X \n",
"on move: X\n",
"O X X \n",
"O O O \n",
"X X \n",
"Episode 89, Total Reward: -1\n",
"Average Reward: 0.1797752808988764\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" \n",
"X X \n",
"on move: X\n",
" O O \n",
" \n",
"X X \n",
"on move: O\n",
" O O \n",
" X \n",
"X X \n",
"on move: X\n",
" O O \n",
" O X \n",
"X X \n",
"on move: O\n",
" O O \n",
" O X \n",
"X X X \n",
"Episode 90, Total Reward: 1\n",
"Average Reward: 0.18888888888888888\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
"O X \n",
" \n",
"on move: O\n",
"X \n",
"O X \n",
" \n",
"on move: X\n",
"X \n",
"O O X \n",
" \n",
"on move: O\n",
"X \n",
"O O X \n",
"X \n",
"on move: X\n",
"X \n",
"O O X \n",
"X O \n",
"on move: O\n",
"X X \n",
"O O X \n",
"X O \n",
"on move: X\n",
"X X O \n",
"O O X \n",
"X O \n",
"on move: O\n",
"X X O \n",
"O O X \n",
"X X O \n",
"Episode 91, Total Reward: 0\n",
"Average Reward: 0.18681318681318682\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" O \n",
" \n",
"X \n",
"on move: O\n",
" O \n",
" \n",
"X X \n",
"on move: X\n",
" O \n",
" \n",
"X X O \n",
"on move: O\n",
" O \n",
"X \n",
"X X O \n",
"on move: X\n",
"O O \n",
"X \n",
"X X O \n",
"on move: O\n",
"O X O \n",
"X \n",
"X X O \n",
"on move: X\n",
"O X O \n",
"X O \n",
"X X O \n",
"Episode 92, Total Reward: -1\n",
"Average Reward: 0.17391304347826086\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" \n",
" X O \n",
"on move: O\n",
"X \n",
" \n",
" X O \n",
"on move: X\n",
"X \n",
"O \n",
" X O \n",
"on move: O\n",
"X X \n",
"O \n",
" X O \n",
"on move: X\n",
"X O X \n",
"O \n",
" X O \n",
"on move: O\n",
"X O X \n",
"O \n",
"X X O \n",
"on move: X\n",
"X O X \n",
"O O \n",
"X X O \n",
"on move: O\n",
"X O X \n",
"O O X \n",
"X X O \n",
"Episode 93, Total Reward: 0\n",
"Average Reward: 0.17204301075268819\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
"O \n",
"X \n",
" \n",
"on move: O\n",
"O \n",
"X \n",
" X \n",
"on move: X\n",
"O \n",
"X \n",
" O X \n",
"on move: O\n",
"O \n",
"X \n",
"X O X \n",
"on move: X\n",
"O O \n",
"X \n",
"X O X \n",
"on move: O\n",
"O O \n",
"X X \n",
"X O X \n",
"on move: X\n",
"O O \n",
"X X O \n",
"X O X \n",
"on move: O\n",
"O O X \n",
"X X O \n",
"X O X \n",
"Episode 94, Total Reward: 1\n",
"Average Reward: 0.18085106382978725\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" X O \n",
" \n",
" X \n",
"on move: X\n",
" X O \n",
" O \n",
" X \n",
"on move: O\n",
" X O \n",
" X O \n",
" X \n",
"on move: X\n",
" X O \n",
" X O \n",
"O X \n",
"on move: O\n",
" X O \n",
" X O \n",
"O X X \n",
"Episode 95, Total Reward: 1\n",
"Average Reward: 0.18947368421052632\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
" \n",
" X \n",
" O X \n",
"on move: X\n",
" O \n",
" X \n",
" O X \n",
"on move: O\n",
" X O \n",
" X \n",
" O X \n",
"on move: X\n",
"O X O \n",
" X \n",
" O X \n",
"on move: O\n",
"O X O \n",
" X X \n",
" O X \n",
"on move: X\n",
"O X O \n",
" X X \n",
"O O X \n",
"on move: O\n",
"O X O \n",
"X X X \n",
"O O X \n",
"Episode 96, Total Reward: 1\n",
"Average Reward: 0.19791666666666666\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
"X \n",
" X \n",
"on move: X\n",
" O \n",
"X O \n",
" X \n",
"on move: O\n",
" O \n",
"X O \n",
" X X \n",
"on move: X\n",
" O \n",
"X O O \n",
" X X \n",
"on move: O\n",
"X O \n",
"X O O \n",
" X X \n",
"on move: X\n",
"X O \n",
"X O O \n",
"O X X \n",
"on move: O\n",
"X O X \n",
"X O O \n",
"O X X \n",
"Episode 97, Total Reward: 0\n",
"Average Reward: 0.1958762886597938\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
"O \n",
" \n",
"on move: O\n",
" X \n",
"O X \n",
" \n",
"on move: X\n",
" O X \n",
"O X \n",
" \n",
"on move: O\n",
" O X \n",
"O X \n",
" X \n",
"on move: X\n",
"O O X \n",
"O X \n",
" X \n",
"on move: O\n",
"O O X \n",
"O X \n",
" X X \n",
"Episode 98, Total Reward: 1\n",
"Average Reward: 0.20408163265306123\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
"X \n",
" X \n",
"on move: X\n",
"O O \n",
"X \n",
" X \n",
"on move: O\n",
"O O \n",
"X \n",
" X X \n",
"on move: X\n",
"O O \n",
"X \n",
"O X X \n",
"on move: O\n",
"O X O \n",
"X \n",
"O X X \n",
"on move: X\n",
"O X O \n",
"X O \n",
"O X X \n",
"on move: O\n",
"O X O \n",
"X X O \n",
"O X X \n",
"Episode 99, Total Reward: 1\n",
"Average Reward: 0.21212121212121213\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
"O \n",
" X \n",
"on move: O\n",
" \n",
"O \n",
" X X \n",
"on move: X\n",
" \n",
"O \n",
"O X X \n",
"on move: O\n",
" \n",
"O X \n",
"O X X \n",
"on move: X\n",
" O \n",
"O X \n",
"O X X \n",
"on move: O\n",
"X O \n",
"O X \n",
"O X X \n",
"on move: X\n",
"X O \n",
"O O X \n",
"O X X \n",
"Episode 100, Total Reward: -1\n",
"Average Reward: 0.2\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
"X O \n",
" X \n",
" \n",
"on move: X\n",
"X O \n",
" O X \n",
" \n",
"on move: O\n",
"X O \n",
" O X \n",
"X \n",
"on move: X\n",
"X O \n",
" O X \n",
"X O \n",
"on move: O\n",
"X O \n",
" O X \n",
"X X O \n",
"on move: X\n",
"X O O \n",
" O X \n",
"X X O \n",
"on move: O\n",
"X O O \n",
"X O X \n",
"X X O \n",
"Episode 101, Total Reward: 1\n",
"Average Reward: 0.2079207920792079\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X O \n",
" \n",
"on move: O\n",
" X \n",
"X O \n",
" \n",
"on move: X\n",
" X \n",
"X O O \n",
" \n",
"on move: O\n",
" X \n",
"X O O \n",
" X \n",
"on move: X\n",
" X \n",
"X O O \n",
"O X \n",
"on move: O\n",
" X X \n",
"X O O \n",
"O X \n",
"on move: X\n",
"O X X \n",
"X O O \n",
"O X \n",
"on move: O\n",
"O X X \n",
"X O O \n",
"O X X \n",
"Episode 102, Total Reward: 0\n",
"Average Reward: 0.20588235294117646\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
"O \n",
" X \n",
" \n",
"on move: O\n",
"O X \n",
" X \n",
" \n",
"on move: X\n",
"O X \n",
" X \n",
" O \n",
"on move: O\n",
"O X \n",
" X \n",
" O X \n",
"on move: X\n",
"O X \n",
" X \n",
"O O X \n",
"on move: O\n",
"O X \n",
" X X \n",
"O O X \n",
"Episode 103, Total Reward: 1\n",
"Average Reward: 0.21359223300970873\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
"O \n",
" \n",
"on move: O\n",
"X X \n",
"O \n",
" \n",
"on move: X\n",
"X X \n",
"O O \n",
" \n",
"on move: O\n",
"X X X \n",
"O O \n",
" \n",
"Episode 104, Total Reward: 1\n",
"Average Reward: 0.22115384615384615\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
"O \n",
" \n",
"on move: O\n",
" X \n",
"O \n",
"X \n",
"on move: X\n",
" X \n",
"O \n",
"X O \n",
"on move: O\n",
" X X \n",
"O \n",
"X O \n",
"on move: X\n",
"O X X \n",
"O \n",
"X O \n",
"on move: O\n",
"O X X \n",
"O X \n",
"X O \n",
"on move: X\n",
"O X X \n",
"O X \n",
"X O O \n",
"on move: O\n",
"O X X \n",
"O X X \n",
"X O O \n",
"Episode 105, Total Reward: 1\n",
"Average Reward: 0.22857142857142856\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
"O X \n",
" \n",
"on move: O\n",
" X \n",
"O X \n",
" \n",
"on move: X\n",
" X \n",
"O X \n",
" O \n",
"on move: O\n",
"X X \n",
"O X \n",
" O \n",
"on move: X\n",
"X O X \n",
"O X \n",
" O \n",
"on move: O\n",
"X O X \n",
"O X \n",
"X O \n",
"on move: X\n",
"X O X \n",
"O O X \n",
"X O \n",
"on move: O\n",
"X O X \n",
"O O X \n",
"X X O \n",
"Episode 106, Total Reward: 0\n",
"Average Reward: 0.22641509433962265\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
" O \n",
" X X \n",
" \n",
"on move: X\n",
" O \n",
" X X \n",
" O \n",
"on move: O\n",
" O \n",
"X X X \n",
" O \n",
"Episode 107, Total Reward: 1\n",
"Average Reward: 0.2336448598130841\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
"X \n",
" X \n",
" O \n",
"on move: X\n",
"X \n",
" X \n",
" O O \n",
"on move: O\n",
"X X \n",
" X \n",
" O O \n",
"on move: X\n",
"X X \n",
"O X \n",
" O O \n",
"on move: O\n",
"X X X \n",
"O X \n",
" O O \n",
"Episode 108, Total Reward: 1\n",
"Average Reward: 0.24074074074074073\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
"O \n",
" X \n",
"on move: O\n",
" X \n",
"O \n",
" X \n",
"on move: X\n",
" X \n",
"O O \n",
" X \n",
"on move: O\n",
" X \n",
"O O \n",
"X X \n",
"on move: X\n",
" X \n",
"O O O \n",
"X X \n",
"Episode 109, Total Reward: -1\n",
"Average Reward: 0.22935779816513763\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
" \n",
" X \n",
"X O \n",
"on move: X\n",
" \n",
" X \n",
"X O O \n",
"on move: O\n",
" \n",
" X X \n",
"X O O \n",
"on move: X\n",
" O \n",
" X X \n",
"X O O \n",
"on move: O\n",
" O X \n",
" X X \n",
"X O O \n",
"Episode 110, Total Reward: 1\n",
"Average Reward: 0.23636363636363636\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" O \n",
" \n",
"on move: O\n",
" X \n",
" O \n",
" X \n",
"on move: X\n",
" X \n",
" O O \n",
" X \n",
"on move: O\n",
"X X \n",
" O O \n",
" X \n",
"on move: X\n",
"X X \n",
" O O \n",
" O X \n",
"on move: O\n",
"X X \n",
"X O O \n",
" O X \n",
"on move: X\n",
"X O X \n",
"X O O \n",
" O X \n",
"Episode 111, Total Reward: -1\n",
"Average Reward: 0.22522522522522523\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
"O \n",
" X \n",
"on move: O\n",
" \n",
"O \n",
"X X \n",
"on move: X\n",
" \n",
"O O \n",
"X X \n",
"on move: O\n",
" \n",
"O X O \n",
"X X \n",
"on move: X\n",
" O \n",
"O X O \n",
"X X \n",
"on move: O\n",
"X O \n",
"O X O \n",
"X X \n",
"on move: X\n",
"X O O \n",
"O X O \n",
"X X \n",
"on move: O\n",
"X O O \n",
"O X O \n",
"X X X \n",
"Episode 112, Total Reward: 1\n",
"Average Reward: 0.23214285714285715\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
"O X \n",
" \n",
" \n",
"on move: O\n",
"O X \n",
" \n",
"X \n",
"on move: X\n",
"O X \n",
"O \n",
"X \n",
"on move: O\n",
"O X X \n",
"O \n",
"X \n",
"on move: X\n",
"O X X \n",
"O \n",
"X O \n",
"on move: O\n",
"O X X \n",
"O \n",
"X O X \n",
"on move: X\n",
"O X X \n",
"O O \n",
"X O X \n",
"on move: O\n",
"O X X \n",
"O X O \n",
"X O X \n",
"Episode 113, Total Reward: 1\n",
"Average Reward: 0.23893805309734514\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" \n",
" X X \n",
"on move: X\n",
" O \n",
" O \n",
" X X \n",
"on move: O\n",
" O \n",
" O \n",
"X X X \n",
"Episode 114, Total Reward: 1\n",
"Average Reward: 0.24561403508771928\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
"X \n",
" X \n",
"on move: X\n",
" O \n",
"X \n",
"O X \n",
"on move: O\n",
"X O \n",
"X \n",
"O X \n",
"on move: X\n",
"X O \n",
"X O \n",
"O X \n",
"on move: O\n",
"X O \n",
"X O X \n",
"O X \n",
"on move: X\n",
"X O \n",
"X O X \n",
"O X O \n",
"on move: O\n",
"X O X \n",
"X O X \n",
"O X O \n",
"Episode 115, Total Reward: 0\n",
"Average Reward: 0.24347826086956523\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
"O \n",
"on move: O\n",
" X X \n",
" \n",
"O \n",
"on move: X\n",
"O X X \n",
" \n",
"O \n",
"on move: O\n",
"O X X \n",
" X \n",
"O \n",
"on move: X\n",
"O X X \n",
" X \n",
"O O \n",
"on move: O\n",
"O X X \n",
" X \n",
"O O X \n",
"Episode 116, Total Reward: 1\n",
"Average Reward: 0.25\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" \n",
" O \n",
"X \n",
"on move: O\n",
" \n",
" O \n",
"X X \n",
"on move: X\n",
" \n",
" O \n",
"X O X \n",
"on move: O\n",
" \n",
"X O \n",
"X O X \n",
"on move: X\n",
"O \n",
"X O \n",
"X O X \n",
"on move: O\n",
"O X \n",
"X O \n",
"X O X \n",
"on move: X\n",
"O X \n",
"X O O \n",
"X O X \n",
"on move: O\n",
"O X X \n",
"X O O \n",
"X O X \n",
"Episode 117, Total Reward: 0\n",
"Average Reward: 0.24786324786324787\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
" O \n",
" X X \n",
" \n",
"on move: X\n",
" O \n",
" X X \n",
" O \n",
"on move: O\n",
"X O \n",
" X X \n",
" O \n",
"on move: X\n",
"X O \n",
" X X \n",
"O O \n",
"on move: O\n",
"X O \n",
"X X X \n",
"O O \n",
"Episode 118, Total Reward: 1\n",
"Average Reward: 0.2542372881355932\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
"O X \n",
" \n",
" \n",
"on move: O\n",
"O X \n",
"X \n",
" \n",
"on move: X\n",
"O X \n",
"X \n",
" O \n",
"on move: O\n",
"O X \n",
"X X \n",
" O \n",
"on move: X\n",
"O O X \n",
"X X \n",
" O \n",
"on move: O\n",
"O O X \n",
"X X X \n",
" O \n",
"Episode 119, Total Reward: 1\n",
"Average Reward: 0.2605042016806723\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" O \n",
" X \n",
"on move: O\n",
" \n",
" O \n",
"X X \n",
"on move: X\n",
" O \n",
" O \n",
"X X \n",
"on move: O\n",
" O X \n",
" O \n",
"X X \n",
"on move: X\n",
" O X \n",
" O \n",
"X O X \n",
"on move: O\n",
" O X \n",
"X O \n",
"X O X \n",
"on move: X\n",
" O X \n",
"X O O \n",
"X O X \n",
"Episode 120, Total Reward: -1\n",
"Average Reward: 0.25\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
" O \n",
" X X \n",
" \n",
"on move: X\n",
"O O \n",
" X X \n",
" \n",
"on move: O\n",
"O O X \n",
" X X \n",
" \n",
"on move: X\n",
"O O X \n",
" X X \n",
"O \n",
"on move: O\n",
"O O X \n",
" X X \n",
"O X \n",
"Episode 121, Total Reward: 1\n",
"Average Reward: 0.256198347107438\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
"O \n",
"on move: O\n",
"X X \n",
" \n",
"O \n",
"on move: X\n",
"X X \n",
"O \n",
"O \n",
"on move: O\n",
"X X \n",
"O \n",
"O X \n",
"on move: X\n",
"X X O \n",
"O \n",
"O X \n",
"on move: O\n",
"X X O \n",
"O \n",
"O X X \n",
"on move: X\n",
"X X O \n",
"O O \n",
"O X X \n",
"Episode 122, Total Reward: -1\n",
"Average Reward: 0.2459016393442623\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" \n",
" \n",
"X O \n",
"on move: O\n",
" \n",
" \n",
"X X O \n",
"on move: X\n",
" \n",
"O \n",
"X X O \n",
"on move: O\n",
" X \n",
"O \n",
"X X O \n",
"on move: X\n",
" X \n",
"O O \n",
"X X O \n",
"on move: O\n",
" X X \n",
"O O \n",
"X X O \n",
"on move: X\n",
" X X \n",
"O O O \n",
"X X O \n",
"Episode 123, Total Reward: -1\n",
"Average Reward: 0.23577235772357724\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
"O \n",
" \n",
" X \n",
"on move: O\n",
"O X \n",
" \n",
" X \n",
"on move: X\n",
"O X \n",
" O \n",
" X \n",
"on move: O\n",
"O X \n",
" O \n",
" X X \n",
"on move: X\n",
"O O X \n",
" O \n",
" X X \n",
"on move: O\n",
"O O X \n",
"X O \n",
" X X \n",
"on move: X\n",
"O O X \n",
"X O O \n",
" X X \n",
"on move: O\n",
"O O X \n",
"X O O \n",
"X X X \n",
"Episode 124, Total Reward: 1\n",
"Average Reward: 0.24193548387096775\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
"O \n",
" X \n",
"on move: O\n",
" X \n",
"O \n",
" X \n",
"on move: X\n",
" X \n",
"O \n",
"O X \n",
"on move: O\n",
" X \n",
"O X \n",
"O X \n",
"on move: X\n",
" X \n",
"O X O \n",
"O X \n",
"on move: O\n",
" X X \n",
"O X O \n",
"O X \n",
"on move: X\n",
"O X X \n",
"O X O \n",
"O X \n",
"Episode 125, Total Reward: -1\n",
"Average Reward: 0.232\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
"O X \n",
" \n",
"on move: O\n",
" \n",
"O X \n",
"X \n",
"on move: X\n",
" \n",
"O O X \n",
"X \n",
"on move: O\n",
" \n",
"O O X \n",
"X X \n",
"on move: X\n",
" \n",
"O O X \n",
"X O X \n",
"on move: O\n",
"X \n",
"O O X \n",
"X O X \n",
"on move: X\n",
"X O \n",
"O O X \n",
"X O X \n",
"Episode 126, Total Reward: -1\n",
"Average Reward: 0.2222222222222222\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" O \n",
" \n",
"on move: O\n",
" X X \n",
" O \n",
" \n",
"on move: X\n",
" X X \n",
" O O \n",
" \n",
"on move: O\n",
" X X \n",
" O O \n",
" X \n",
"on move: X\n",
" X X \n",
" O O \n",
"O X \n",
"on move: O\n",
" X X \n",
"X O O \n",
"O X \n",
"on move: X\n",
" X X \n",
"X O O \n",
"O X O \n",
"on move: O\n",
"X X X \n",
"X O O \n",
"O X O \n",
"Episode 127, Total Reward: 1\n",
"Average Reward: 0.2283464566929134\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
" \n",
"O \n",
"on move: O\n",
"X \n",
"X \n",
"O \n",
"on move: X\n",
"X \n",
"X O \n",
"O \n",
"on move: O\n",
"X \n",
"X O X \n",
"O \n",
"on move: X\n",
"X \n",
"X O X \n",
"O O \n",
"on move: O\n",
"X X \n",
"X O X \n",
"O O \n",
"on move: X\n",
"X X \n",
"X O X \n",
"O O O \n",
"Episode 128, Total Reward: -1\n",
"Average Reward: 0.21875\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" O \n",
" X \n",
" \n",
"on move: O\n",
" O \n",
" X X \n",
" \n",
"on move: X\n",
"O O \n",
" X X \n",
" \n",
"on move: O\n",
"O O \n",
" X X \n",
" X \n",
"on move: X\n",
"O O O \n",
" X X \n",
" X \n",
"Episode 129, Total Reward: -1\n",
"Average Reward: 0.20930232558139536\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" O \n",
" \n",
"on move: O\n",
" X \n",
" O \n",
"X \n",
"on move: X\n",
"O X \n",
" O \n",
"X \n",
"on move: O\n",
"O X \n",
" O \n",
"X X \n",
"on move: X\n",
"O X \n",
" O O \n",
"X X \n",
"on move: O\n",
"O X \n",
" O O \n",
"X X X \n",
"Episode 130, Total Reward: 1\n",
"Average Reward: 0.2153846153846154\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
" X \n",
" \n",
"X O \n",
"on move: X\n",
"O X \n",
" \n",
"X O \n",
"on move: O\n",
"O X \n",
" \n",
"X X O \n",
"on move: X\n",
"O X \n",
" O \n",
"X X O \n",
"Episode 131, Total Reward: -1\n",
"Average Reward: 0.20610687022900764\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
"O X \n",
" \n",
"on move: O\n",
" X \n",
"O X \n",
" \n",
"on move: X\n",
" O X \n",
"O X \n",
" \n",
"on move: O\n",
" O X \n",
"O X \n",
" X \n",
"Episode 132, Total Reward: 1\n",
"Average Reward: 0.21212121212121213\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
"O \n",
" X \n",
" \n",
"on move: O\n",
"O \n",
" X \n",
" X \n",
"on move: X\n",
"O \n",
" X O \n",
" X \n",
"on move: O\n",
"O \n",
" X O \n",
"X X \n",
"on move: X\n",
"O O \n",
" X O \n",
"X X \n",
"on move: O\n",
"O O \n",
" X O \n",
"X X X \n",
"Episode 133, Total Reward: 1\n",
"Average Reward: 0.21804511278195488\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" O X \n",
" \n",
" \n",
"on move: O\n",
"X O X \n",
" \n",
" \n",
"on move: X\n",
"X O X \n",
" O \n",
" \n",
"on move: O\n",
"X O X \n",
" O \n",
"X \n",
"on move: X\n",
"X O X \n",
"O O \n",
"X \n",
"on move: O\n",
"X O X \n",
"O X O \n",
"X \n",
"Episode 134, Total Reward: 1\n",
"Average Reward: 0.22388059701492538\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
" \n",
" X \n",
"X O \n",
"on move: X\n",
"O \n",
" X \n",
"X O \n",
"on move: O\n",
"O \n",
"X X \n",
"X O \n",
"on move: X\n",
"O O \n",
"X X \n",
"X O \n",
"on move: O\n",
"O O \n",
"X X X \n",
"X O \n",
"Episode 135, Total Reward: 1\n",
"Average Reward: 0.22962962962962963\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" \n",
"X \n",
"O \n",
"on move: O\n",
" X \n",
"X \n",
"O \n",
"on move: X\n",
"O X \n",
"X \n",
"O \n",
"on move: O\n",
"O X \n",
"X X \n",
"O \n",
"on move: X\n",
"O X \n",
"X X \n",
"O O \n",
"on move: O\n",
"O X \n",
"X X \n",
"O X O \n",
"on move: X\n",
"O X \n",
"X O X \n",
"O X O \n",
"Episode 136, Total Reward: -1\n",
"Average Reward: 0.22058823529411764\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
"O \n",
" \n",
" X \n",
"on move: O\n",
"O \n",
" X \n",
" X \n",
"on move: X\n",
"O \n",
" X \n",
"O X \n",
"on move: O\n",
"O \n",
"X X \n",
"O X \n",
"on move: X\n",
"O \n",
"X X \n",
"O O X \n",
"on move: O\n",
"O \n",
"X X X \n",
"O O X \n",
"Episode 137, Total Reward: 1\n",
"Average Reward: 0.22627737226277372\n",
"on move: O\n",
" \n",
"X \n",
" \n",
"on move: X\n",
" O \n",
"X \n",
" \n",
"on move: O\n",
"X O \n",
"X \n",
" \n",
"on move: X\n",
"X O \n",
"X \n",
"O \n",
"on move: O\n",
"X O \n",
"X X \n",
"O \n",
"on move: X\n",
"X O O \n",
"X X \n",
"O \n",
"on move: O\n",
"X O O \n",
"X X X \n",
"O \n",
"Episode 138, Total Reward: 1\n",
"Average Reward: 0.2318840579710145\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" O \n",
" X \n",
" X \n",
"on move: X\n",
"O O \n",
" X \n",
" X \n",
"on move: O\n",
"O X O \n",
" X \n",
" X \n",
"Episode 139, Total Reward: 1\n",
"Average Reward: 0.23741007194244604\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" \n",
"O X \n",
"on move: O\n",
" X \n",
" \n",
"O X \n",
"on move: X\n",
" X \n",
" O \n",
"O X \n",
"on move: O\n",
" X \n",
"X O \n",
"O X \n",
"on move: X\n",
" X O \n",
"X O \n",
"O X \n",
"Episode 140, Total Reward: -1\n",
"Average Reward: 0.22857142857142856\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" O \n",
" \n",
" X \n",
"on move: O\n",
" X O \n",
" \n",
" X \n",
"on move: X\n",
" X O \n",
" \n",
" O X \n",
"on move: O\n",
" X O \n",
"X \n",
" O X \n",
"on move: X\n",
" X O \n",
"X O \n",
" O X \n",
"on move: O\n",
" X O \n",
"X X O \n",
" O X \n",
"on move: X\n",
" X O \n",
"X X O \n",
"O O X \n",
"on move: O\n",
"X X O \n",
"X X O \n",
"O O X \n",
"Episode 141, Total Reward: 1\n",
"Average Reward: 0.23404255319148937\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X \n",
"O \n",
" \n",
"on move: O\n",
"X \n",
"O X \n",
" \n",
"on move: X\n",
"X \n",
"O X \n",
" O \n",
"on move: O\n",
"X \n",
"O X \n",
" O X \n",
"Episode 142, Total Reward: 1\n",
"Average Reward: 0.23943661971830985\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
"O \n",
" \n",
" X \n",
"on move: O\n",
"O \n",
" X \n",
" X \n",
"on move: X\n",
"O \n",
" X \n",
" O X \n",
"on move: O\n",
"O \n",
"X X \n",
" O X \n",
"on move: X\n",
"O O \n",
"X X \n",
" O X \n",
"on move: O\n",
"O X O \n",
"X X \n",
" O X \n",
"on move: X\n",
"O X O \n",
"X X O \n",
" O X \n",
"on move: O\n",
"O X O \n",
"X X O \n",
"X O X \n",
"Episode 143, Total Reward: 0\n",
"Average Reward: 0.23776223776223776\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
" \n",
" O \n",
"on move: O\n",
" X \n",
" \n",
"X O \n",
"on move: X\n",
"O X \n",
" \n",
"X O \n",
"on move: O\n",
"O X \n",
" X \n",
"X O \n",
"Episode 144, Total Reward: 1\n",
"Average Reward: 0.24305555555555555\n",
"on move: O\n",
" \n",
" \n",
"X \n",
"on move: X\n",
" O \n",
" \n",
"X \n",
"on move: O\n",
" O X \n",
" \n",
"X \n",
"on move: X\n",
"O O X \n",
" \n",
"X \n",
"on move: O\n",
"O O X \n",
" \n",
"X X \n",
"on move: X\n",
"O O X \n",
"O \n",
"X X \n",
"on move: O\n",
"O O X \n",
"O \n",
"X X X \n",
"Episode 145, Total Reward: 1\n",
"Average Reward: 0.2482758620689655\n",
"on move: O\n",
"X \n",
" \n",
" \n",
"on move: X\n",
"X O \n",
" \n",
" \n",
"on move: O\n",
"X O \n",
" \n",
"X \n",
"on move: X\n",
"X O \n",
"O \n",
"X \n",
"on move: O\n",
"X O \n",
"O \n",
"X X \n",
"on move: X\n",
"X O \n",
"O O \n",
"X X \n",
"on move: O\n",
"X O \n",
"O O \n",
"X X X \n",
"Episode 146, Total Reward: 1\n",
"Average Reward: 0.2534246575342466\n",
"on move: O\n",
" \n",
" \n",
" X \n",
"on move: X\n",
" \n",
" O \n",
" X \n",
"on move: O\n",
" \n",
" O X \n",
" X \n",
"on move: X\n",
" \n",
"O O X \n",
" X \n",
"on move: O\n",
" \n",
"O O X \n",
"X X \n",
"on move: X\n",
" O \n",
"O O X \n",
"X X \n",
"on move: O\n",
"X O \n",
"O O X \n",
"X X \n",
"on move: X\n",
"X O \n",
"O O X \n",
"X O X \n",
"on move: O\n",
"X X O \n",
"O O X \n",
"X O X \n",
"Episode 147, Total Reward: 0\n",
"Average Reward: 0.25170068027210885\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" O X \n",
" \n",
" \n",
"on move: O\n",
"X O X \n",
" \n",
" \n",
"on move: X\n",
"X O X \n",
" \n",
" O \n",
"on move: O\n",
"X O X \n",
" X \n",
" O \n",
"on move: X\n",
"X O X \n",
" X \n",
"O O \n",
"on move: O\n",
"X O X \n",
"X X \n",
"O O \n",
"on move: X\n",
"X O X \n",
"X X O \n",
"O O \n",
"on move: O\n",
"X O X \n",
"X X O \n",
"O X O \n",
"Episode 148, Total Reward: 0\n",
"Average Reward: 0.25\n",
"on move: O\n",
" X \n",
" \n",
" \n",
"on move: X\n",
" X \n",
"O \n",
" \n",
"on move: O\n",
" X \n",
"O \n",
" X \n",
"on move: X\n",
"O X \n",
"O \n",
" X \n",
"on move: O\n",
"O X \n",
"O X \n",
" X \n",
"Episode 149, Total Reward: 1\n",
"Average Reward: 0.2550335570469799\n",
"on move: O\n",
" \n",
" X \n",
" \n",
"on move: X\n",
" \n",
" X \n",
" O \n",
"on move: O\n",
" X \n",
" X \n",
" O \n",
"on move: X\n",
"O X \n",
" X \n",
" O \n",
"on move: O\n",
"O X X \n",
" X \n",
" O \n",
"on move: X\n",
"O X X \n",
" X \n",
" O O \n",
"on move: O\n",
"O X X \n",
" X X \n",
" O O \n",
"on move: X\n",
"O X X \n",
"O X X \n",
" O O \n",
"on move: O\n",
"O X X \n",
"O X X \n",
"X O O \n",
"Episode 150, Total Reward: 1\n",
"Average Reward: 0.26\n"
]
}
],
"source": [
"# Основной цикл обучения (работа с отдельным классом агента)\n",
"\n",
"# Создание среды для игры в крестики-нолики\n",
"environment = TicTacToeEnv()\n",
"\n",
"# Создание агента (играющего крестиками)\n",
"agent = Agent(symbol=1)\n",
"\n",
"num_episodes = 150 # Количество эпизодов (игр) для обучения\n",
"collected_rewards = [] # Список для хранения наград/побед в каждом эпизоде \n",
"\n",
"# Переменная для отслеживания символа и текущего игрока\n",
"oom = 1\n",
"\n",
"for i in range(num_episodes):\n",
" # Сброс среды и начало нового эпизода\n",
" state, _ = environment.reset() \n",
"\n",
" # Общая награда за эпизод\n",
" total_reward = 0\n",
"\n",
" # Флаг завершения игры\n",
" done = False\n",
" om = oom \n",
"\n",
" # Максимум 9 ходов, поскольку поле 3x3 \n",
" for j in range(9): \n",
" moves = environment.move_generator() \n",
"\n",
" # Ходов нет, заканчиваем игру\n",
" if not moves:\n",
" break\n",
"\n",
" \n",
" if len(moves) == 1:\n",
" move = moves[0] # Если остался один ход на основе стратегии\n",
" else:\n",
" move = agent.get_action(moves) # Агент выбирает ход на основе стратегии\n",
"\n",
" # Выполнение хода и обновление состояния игры\n",
" next_state, reward, done, info = environment.step(move)\n",
" total_reward += reward\n",
" state = next_state\n",
"\n",
" # Отображаем текущее состояние игры\n",
" environment.render()\n",
"\n",
" if done:\n",
" break\n",
"\n",
" om = -om # Смена игрока\n",
"\n",
" collected_rewards.append(total_reward)\n",
"\n",
" print(f\"Episode {i+1}, Total Reward: {total_reward}\")\n",
" average_reward = sum(collected_rewards) / len(collected_rewards)\n",
" print(f\"Average Reward: {average_reward}\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue Copy Permalink