110 KiB
110 KiB
Начинаем работу...¶
Необходимо развернуть и запустить проект по реализации обучения с подкреплением для игры "Крестики-нолики"
Перевести проект на библиотеку gymnasium и современную версию Python. Реализовать агена для игры "Крестики-нолики" ы ымдк отдельного класса (как в лекции). Переписать основной цикл обучения для работы с отдельным классом агента (как в лекции). Протестировать новую версию программы.
Обучение с подкреплением (Reinforcement Learning, RL) — это метод машинного обучения, при котором агент обучается путем взаимодействия с окружающей средой. Цель агента — максимизировать получаемую награду через последовательность действий.
Крестики-нолики: https://github.com/nczempin/gym-tic-tac-toe
Перевод среды на Gymnasium¶
Gymnasium (ранее OpenAI Gym) — это библиотека для создания и тестирования алгоритмов обучения с подкреплением. Она предоставляет набор стандартных сред для RL, таких как игры, задачи управления и симуляции.
Gymnasium предлагает обновленную API по сравнению с Gym:
Упростились методы reset и step. Добавлены новые возможности, такие как поддержка тайм-аутов и контроль шагов. Улучшена поддержка пользовательских сред.
Основные возможности Gymnasium:
Стандартизированные интерфейсы для взаимодействия со средой. Простая интеграция с популярными библиотеками RL. Поддержка пользовательских сред.
Ключевые функции Gymnasium:
env.reset() — инициализация среды.
env.step(action) — выполнение действия и переход в новое состояние.
env.render() — визуализация текущего состояния среды.
In [4]:
import gymnasium as gym
from gymnasium import spaces
class TicTacToeEnv(gym.Env):
metadata = {'render.modes': ['human']}
symbols = ['O', ' ', 'X']
def __init__(self):
super().__init__()
self.action_space = spaces.Discrete(9)
self.observation_space = spaces.Discrete(9 * 3 * 2)
self.reset()
def step(self, action):
done = False
reward = 0
p, square = action # p - игрок (1 или -1), square - номер клетки
board = self.state['board']
proposed = board[square]
om = self.state['on_move']
if proposed != 0: # Клетка уже занята
print(f"Незаконный ход: Квадрат {square} уже занят.")
done = True
reward = -1 * om
if p != om: # Не тот игрок на ходу
print(f"Незаконный ход: игрок {p} не находится в движении")
done = True
reward = -1 * om
else:
board[square] = p
self.state['on_move'] = -p
for i in range(3):
# Горизонтали и вертикали
if (board[i * 3] == p and board[i * 3 + 1] == p and board[i * 3 + 2] == p) or \
(board[i] == p and board[i + 3] == p and board[i + 6] == p):
reward = p
done = True
break
# Диагонали
if (board[0] == p and board[4] == p and board[8] == p) or \
(board[2] == p and board[4] == p and board[6] == p):
reward = p
done = True
return self.state, reward, done, {}
def reset(self):
self.state = {}
self.state['board'] = [0, 0, 0, 0, 0, 0, 0, 0, 0]
self.state['on_move'] = 1
return self.state, {}
def render(self, close=False):
if close:
return
print("on move: " , self.symbols[self.state['on_move']+1])
for i in range (9):
print (self.symbols[self.state['board'][i]+1], end=" ");
if ((i % 3) == 2):
print();
def move_generator(self):
moves = []
for i in range(9):
if self.state['board'][i] == 0:
p = self.state['on_move']
m = [p, i]
moves.append(m)
return moves
Реализация агента¶
Агент в контексте обучения с подкреплением (Reinforcement Learning, RL) — это система, которая взаимодействует с окружающей средой, чтобы достичь определенной цели. Его задача — выработать стратегию, которая максимизирует получаемую награду в долгосрочной перспективе.
Роль агента:
Агент принимает решение (выбирает действие), основываясь на текущем состоянии среды, и затем получает обратную связь (награду и новое состояние) от среды.
Функционал агента:
Выбор действия: Использует алгоритмы или стратегии, чтобы определить, что делать дальше.
Обучение: Обновляет свои знания или стратегию на основе опыта, чтобы лучше справляться с задачей.
Интерактивность: Адаптируется к изменениям в среде.
In [5]:
import random
# Реализация Агента, который в рамках обучения с подкреплением взаимодействует со средой и вырабатывает наилучшую стратегию
class Agent:
def __init__(self, symbol):
self.symbol = symbol # Символ игрока (1 - X, -1 - O)
def get_action(self, moves):
return random.choice(moves) # Выбираем случайный ход из доступных
Основной цикл обучения¶
Основной цикл включает в себя создание среды и работу с ней. По истечению выполнения игры, начисляются очки, которые будут свидетельствовать о том, какие результаты достиг наш агент, обучилась ли наша система.
In [6]:
# Основной цикл обучения (работа с отдельным классом агента)
# Создание среды для игры в крестики-нолики
environment = TicTacToeEnv()
# Создание агента (играющего крестиками)
agent = Agent(symbol=1)
num_episodes = 150 # Количество эпизодов (игр) для обучения
collected_rewards = [] # Список для хранения наград/побед в каждом эпизоде
# Переменная для отслеживания символа и текущего игрока
oom = 1
for i in range(num_episodes):
# Сброс среды и начало нового эпизода
state, _ = environment.reset()
# Общая награда за эпизод
total_reward = 0
# Флаг завершения игры
done = False
om = oom
# Максимум 9 ходов, поскольку поле 3x3
for j in range(9):
moves = environment.move_generator()
# Ходов нет, заканчиваем игру
if not moves:
break
if len(moves) == 1:
move = moves[0] # Если остался один ход на основе стратегии
else:
move = agent.get_action(moves) # Агент выбирает ход на основе стратегии
# Выполнение хода и обновление состояния игры
next_state, reward, done, info = environment.step(move)
total_reward += reward
state = next_state
# Отображаем текущее состояние игры
environment.render()
if done:
break
om = -om # Смена игрока
collected_rewards.append(total_reward)
print(f"Episode {i+1}, Total Reward: {total_reward}")
average_reward = sum(collected_rewards) / len(collected_rewards)
print(f"Average Reward: {average_reward}")
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O
O
on move: O
X X
O
X O
on move: X
O X X
O
X O
on move: O
O X X
X O
X O
Episode 1, Total Reward: 1
Average Reward: 1.0
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O O
on move: O
X X
X
O O
on move: X
X O X
X
O O
on move: O
X O X
X
O X O
on move: X
X O X
X O
O X O
on move: O
X O X
X O X
O X O
Episode 2, Total Reward: 0
Average Reward: 0.5
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
O
X X
on move: O
O
O X
X X
on move: X
O
O X
X X O
on move: O
O X
O X
X X O
Episode 3, Total Reward: 1
Average Reward: 0.6666666666666666
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
X X
O
on move: O
O X
X X
O
on move: X
O X
X X
O O
on move: O
O X X
X X
O O
on move: X
O X X
X X
O O O
Episode 4, Total Reward: -1
Average Reward: 0.25
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
O X
O
on move: O
X
O X
O X
on move: X
X
O O X
O X
on move: O
X
O O X
X O X
on move: X
O X
O O X
X O X
on move: O
O X X
O O X
X O X
Episode 5, Total Reward: 1
Average Reward: 0.4
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
O X
X
on move: O
O X
O X
X
Episode 6, Total Reward: 1
Average Reward: 0.5
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O O
X X
on move: O
X
O O
X X
on move: X
X
O O O
X X
Episode 7, Total Reward: -1
Average Reward: 0.2857142857142857
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O O
on move: O
X X
O O X
on move: X
X O X
O O X
on move: O
X
X O X
O O X
Episode 8, Total Reward: 1
Average Reward: 0.375
on move: O
X
on move: X
O
X
on move: O
X
O
X
on move: X
O X
O
X
on move: O
O X
O X
X
Episode 9, Total Reward: 1
Average Reward: 0.4444444444444444
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
X O
X
O
on move: O
X O
X
X O
on move: X
X O
O X
X O
on move: O
X O
O X
X X O
on move: X
X O
O O X
X X O
on move: O
X X O
O O X
X X O
Episode 10, Total Reward: 0
Average Reward: 0.4
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
O
X X
on move: O
O
O
X X X
Episode 11, Total Reward: 1
Average Reward: 0.45454545454545453
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
X O
O
X
on move: O
X O
O X
X
on move: X
X O O
O X
X
on move: O
X O O
O X X
X
on move: X
X O O
O X X
X O
on move: O
X O O
O X X
X X O
Episode 12, Total Reward: 0
Average Reward: 0.4166666666666667
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O O
X
X
on move: O
X O O
X
X
on move: X
X O O
X O
X
on move: O
X O O
X O
X X
Episode 13, Total Reward: 1
Average Reward: 0.46153846153846156
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
O X
X O
on move: O
O X
X
X O
on move: X
O X
X
O X O
on move: O
O X
X X
O X O
on move: X
O O X
X X
O X O
on move: O
O O X
X X X
O X O
Episode 14, Total Reward: 1
Average Reward: 0.5
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X
O O
on move: O
X
X
X O O
on move: X
O X
X
X O O
on move: O
O X
X X
X O O
on move: X
O X
X X O
X O O
on move: O
O X X
X X O
X O O
Episode 15, Total Reward: 1
Average Reward: 0.5333333333333333
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X O
X
on move: O
X O
X O
X
on move: X
X O
X O O
X
on move: O
X O
X O O
X X
Episode 16, Total Reward: 1
Average Reward: 0.5625
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
X
O
X O
on move: O
X X
O
X O
on move: X
X X
O O
X O
on move: O
X X X
O O
X O
Episode 17, Total Reward: 1
Average Reward: 0.5882352941176471
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
X
O
on move: O
O X
X
X O
on move: X
O O X
X
X O
on move: O
O O X
X
X O X
Episode 18, Total Reward: 1
Average Reward: 0.6111111111111112
on move: O
X
on move: X
O
X
on move: O
X
O
X
on move: X
X
O O
X
on move: O
X X
O O
X
on move: X
X X
O O
X O
on move: O
X X X
O O
X O
Episode 19, Total Reward: 1
Average Reward: 0.631578947368421
on move: O
X
on move: X
O X
on move: O
X
O X
on move: X
X
O X O
on move: O
X
O X O
X
on move: X
O X
O X O
X
on move: O
O X
O X O
X X
on move: X
O X
O X O
X O X
on move: O
O X X
O X O
X O X
Episode 20, Total Reward: 1
Average Reward: 0.65
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
O
X X
on move: O
O X
O
X X
on move: X
O X
O
O X X
on move: O
O X X
O
O X X
on move: X
O X X
O O
O X X
on move: O
O X X
X O O
O X X
Episode 21, Total Reward: 0
Average Reward: 0.6190476190476191
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
X O
on move: O
X
O X
X O
on move: X
X
O X
X O O
on move: O
X
X O X
X O O
on move: X
O X
X O X
X O O
Episode 22, Total Reward: -1
Average Reward: 0.5454545454545454
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X O
O
on move: O
X X
X O
O
on move: X
X X
X O O
O
on move: O
X X X
X O O
O
Episode 23, Total Reward: 1
Average Reward: 0.5652173913043478
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
O
X
on move: O
O X
O
X X
on move: X
O X
O O
X X
on move: O
O X X
O O
X X
on move: X
O X X
O O O
X X
Episode 24, Total Reward: -1
Average Reward: 0.5
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
X O
O
X
on move: O
X O
X O
X
Episode 25, Total Reward: 1
Average Reward: 0.52
on move: O
X
on move: X
O
X
on move: O
X
O
X
on move: X
X
O
X O
on move: O
X X
O
X O
on move: X
X O X
O
X O
on move: O
X O X
O
X X O
on move: X
X O X
O O
X X O
on move: O
X O X
O X O
X X O
Episode 26, Total Reward: 1
Average Reward: 0.5384615384615384
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
O
X O
X
on move: O
O
X O
X X
on move: X
O
X O O
X X
on move: O
O X
X O O
X X
on move: X
O X
X O O
O X X
on move: O
O X X
X O O
O X X
Episode 27, Total Reward: 0
Average Reward: 0.5185185185185185
on move: O
X
on move: X
O X
on move: O
X
O X
on move: X
O
X
O X
on move: O
O
X
X O X
on move: X
O
O X
X O X
on move: O
O X
O X
X O X
Episode 28, Total Reward: 1
Average Reward: 0.5357142857142857
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X O
O
on move: O
X
X X O
O
on move: X
X O
X X O
O
on move: O
X X O
X X O
O
on move: X
X X O
X X O
O O
Episode 29, Total Reward: -1
Average Reward: 0.4827586206896552
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
X O
on move: O
O X
X X O
on move: X
O O X
X X O
on move: O
O O X
X
X X O
Episode 30, Total Reward: 1
Average Reward: 0.5
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
O
X O
X
on move: O
O
X O
X X
on move: X
O
X O
O X X
on move: O
O X
X O
O X X
on move: X
O X
O X O
O X X
Episode 31, Total Reward: -1
Average Reward: 0.45161290322580644
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
O X
on move: O
X
O X
O X
Episode 32, Total Reward: 1
Average Reward: 0.46875
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
O
X X
O
on move: O
O X
X X
O
on move: X
O X
O X X
O
Episode 33, Total Reward: -1
Average Reward: 0.42424242424242425
on move: O
X
on move: X
X
O
on move: O
X
O
X
on move: X
X
O
X O
on move: O
X
X O
X O
on move: X
O X
X O
X O
Episode 34, Total Reward: -1
Average Reward: 0.38235294117647056
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X O
X
O
on move: O
X O
X
O X
on move: X
X O
O X
O X
on move: O
X O
O X X
O X
on move: X
X O
O X X
O X O
on move: O
X X O
O X X
O X O
Episode 35, Total Reward: 1
Average Reward: 0.4
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
X O
O
X
on move: O
X O X
O
X
on move: X
X O X
O O
X
on move: O
X O X
O O
X X
on move: X
X O X
O O
X O X
Episode 36, Total Reward: -1
Average Reward: 0.3611111111111111
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
X O
X O
on move: O
X O
X O
X
on move: X
X O
O X O
X
on move: O
X O X
O X O
X
on move: X
X O X
O X O
X O
on move: O
X O X
O X O
X X O
Episode 37, Total Reward: 1
Average Reward: 0.3783783783783784
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
X O
X O
on move: O
X O
X X O
on move: X
O X O
X X O
on move: O
O X O
X X O
X
Episode 38, Total Reward: 1
Average Reward: 0.39473684210526316
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
X O
X
O
on move: O
X O
X
O X
Episode 39, Total Reward: 1
Average Reward: 0.41025641025641024
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
X
X O O
on move: O
X
X
X O O
on move: X
X
X O
X O O
on move: O
X
X O X
X O O
on move: X
O X
X O X
X O O
Episode 40, Total Reward: -1
Average Reward: 0.375
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O O
X
X
on move: O
O O
X
X X
on move: X
O O
X O
X X
on move: O
X O O
X O
X X
Episode 41, Total Reward: 1
Average Reward: 0.3902439024390244
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
O X
on move: O
X
O X
O X
Episode 42, Total Reward: 1
Average Reward: 0.40476190476190477
on move: O
X
on move: X
X
O
on move: O
X
O
X
on move: X
X
O
X O
on move: O
X X
O
X O
on move: X
O X X
O
X O
Episode 43, Total Reward: -1
Average Reward: 0.37209302325581395
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
O X
X
on move: O
O
O X
X X
on move: X
O
O X O
X X
on move: O
O X
O X O
X X
on move: X
O X
O X O
O X X
on move: O
X O X
O X O
O X X
Episode 44, Total Reward: 1
Average Reward: 0.38636363636363635
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
O X X
O
on move: O
X
O X X
O
on move: X
O X
O X X
O
Episode 45, Total Reward: -1
Average Reward: 0.35555555555555557
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
X
X O
O
on move: O
X X
X O
O
on move: X
X X
X O O
O
on move: O
X X
X O O
X O
Episode 46, Total Reward: 1
Average Reward: 0.3695652173913043
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
X O
X O
on move: O
X
X O
X O
on move: X
X O
X O
X O
on move: O
X O X
X O
X O
Episode 47, Total Reward: 1
Average Reward: 0.3829787234042553
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O O
X X
on move: O
O O
X
X X
on move: X
O O
X O
X X
on move: O
X O O
X O
X X
Episode 48, Total Reward: 1
Average Reward: 0.3958333333333333
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O
O X
X
on move: O
O X
O X
X
Episode 49, Total Reward: 1
Average Reward: 0.40816326530612246
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
X
X O O
on move: O
X X
X O O
on move: X
X X
O
X O O
on move: O
X X
X O
X O O
Episode 50, Total Reward: 1
Average Reward: 0.42
on move: O
X
on move: X
X
O
on move: O
X
X O
on move: X
X
X O O
on move: O
X X
X O O
on move: X
O X X
X O O
on move: O
O X X
X
X O O
on move: X
O X X
X O
X O O
Episode 51, Total Reward: -1
Average Reward: 0.39215686274509803
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
O X
on move: O
X
O X
O X
Episode 52, Total Reward: 1
Average Reward: 0.40384615384615385
on move: O
X
on move: X
O X
on move: O
O X X
on move: X
O X X
O
on move: O
O X X
O
X
on move: X
O X X
O O
X
on move: O
O X X
O O X
X
on move: X
O X X
O O X
O X
Episode 53, Total Reward: -1
Average Reward: 0.37735849056603776
on move: O
X
on move: X
X
O
on move: O
X
O
X
on move: X
X
O O
X
on move: O
X X
O O
X
on move: X
O X X
O O
X
on move: O
O X X
O O
X X
on move: X
O X X
O O
O X X
Episode 54, Total Reward: -1
Average Reward: 0.35185185185185186
on move: O
X
on move: X
O X
on move: O
X O X
on move: X
O
X O X
on move: O
O X
X O X
on move: X
O X
O
X O X
on move: O
O X
X O
X O X
on move: X
O O X
X O
X O X
on move: O
O O X
X X O
X O X
Episode 55, Total Reward: 1
Average Reward: 0.36363636363636365
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O
O
on move: O
X X
O X
O
on move: X
O X X
O X
O
Episode 56, Total Reward: -1
Average Reward: 0.3392857142857143
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O O
on move: O
X
X X
O O
on move: X
X O
X X
O O
on move: O
X O X
X X
O O
on move: X
X O X
X X
O O O
Episode 57, Total Reward: -1
Average Reward: 0.3157894736842105
on move: O
X
on move: X
X
O
on move: O
X
X O
on move: X
X
O
X O
on move: O
X
O
X O X
on move: X
O X
O
X O X
on move: O
O X X
O
X O X
on move: X
O X X
O O
X O X
on move: O
O X X
O X O
X O X
Episode 58, Total Reward: 1
Average Reward: 0.3275862068965517
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O O X
X
on move: O
O O X
X X
on move: X
O
O O X
X X
on move: O
X O
O O X
X X
on move: X
O X O
O O X
X X
on move: O
O X O
O O X
X X X
Episode 59, Total Reward: 1
Average Reward: 0.3389830508474576
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
X O
on move: O
X O X
X O
on move: X
X O X
X O
O
on move: O
X O X
X O
O X
on move: X
X O X
X O O
O X
on move: O
X O X
X O O
O X X
Episode 60, Total Reward: 0
Average Reward: 0.3333333333333333
on move: O
X
on move: X
X
O
on move: O
X
O X
on move: X
X
O X
O
on move: O
X
O X
X O
on move: X
O X
O X
X O
on move: O
X O X
O X
X O
on move: X
X O X
O X
X O O
Episode 61, Total Reward: -1
Average Reward: 0.3114754098360656
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
O X X
on move: O
X
O
O X X
on move: X
O X
O
O X X
Episode 62, Total Reward: -1
Average Reward: 0.2903225806451613
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
X
X O O
on move: O
X
X O O
X
Episode 63, Total Reward: 1
Average Reward: 0.30158730158730157
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O O
X X
on move: O
X
O O
X X
on move: X
X
O O O
X X
Episode 64, Total Reward: -1
Average Reward: 0.28125
on move: O
X
on move: X
O
X
on move: O
X
O
X
on move: X
X
O O
X
on move: O
X
O O
X X
on move: X
X O
O O
X X
on move: O
X O
O X O
X X
Episode 65, Total Reward: 1
Average Reward: 0.2923076923076923
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X
X O
on move: O
O
X X
X O
on move: X
O
X X O
X O
on move: O
O X
X X O
X O
Episode 66, Total Reward: 1
Average Reward: 0.30303030303030304
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
X X
O
on move: O
O X
X X
O
on move: X
O X
X X
O O
on move: O
O X
X X X
O O
Episode 67, Total Reward: 1
Average Reward: 0.31343283582089554
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
O X
X
on move: O
O
O X X
X
on move: X
O
O X X
O X
on move: O
O X
O X X
O X
on move: X
O X O
O X X
O X
on move: O
O X O
O X X
X O X
Episode 68, Total Reward: 0
Average Reward: 0.3088235294117647
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
X O
on move: O
X
O X
X O
on move: X
X O
O X
X O
on move: O
X O
O X
X X O
on move: X
X O
O O X
X X O
on move: O
X X O
O O X
X X O
Episode 69, Total Reward: 0
Average Reward: 0.30434782608695654
on move: O
X
on move: X
X
O
on move: O
X
X O
on move: X
X
X O
O
on move: O
X X
X O
O
on move: X
X X
X O
O O
on move: O
X X X
X O
O O
Episode 70, Total Reward: 1
Average Reward: 0.3142857142857143
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O O
X
X
on move: O
O O
X
X X
on move: X
O O
X O
X X
on move: O
O O
X O
X X X
Episode 71, Total Reward: 1
Average Reward: 0.323943661971831
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
O X
X
on move: O
O
O X
X X
on move: X
O
O X
X O X
on move: O
O
X O X
X O X
on move: X
O O
X O X
X O X
Episode 72, Total Reward: -1
Average Reward: 0.3055555555555556
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
O X
X
O
on move: O
O X
X X
O
on move: X
O X
X O X
O
on move: O
O X
X O X
O X
Episode 73, Total Reward: 1
Average Reward: 0.3150684931506849
on move: O
X
on move: X
O
X
on move: O
X
O
X
on move: X
X
O
O X
on move: O
X
X O
O X
on move: X
X
X O O
O X
on move: O
X X
X O O
O X
on move: X
O X X
X O O
O X
on move: O
O X X
X O O
O X X
Episode 74, Total Reward: 0
Average Reward: 0.3108108108108108
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X O
X
on move: O
O X O
X X
on move: X
O X O
X O X
on move: O
O X O
X
X O X
on move: X
O X O
O X
X O X
on move: O
O X O
O X X
X O X
Episode 75, Total Reward: 0
Average Reward: 0.30666666666666664
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
O X
O
on move: O
X
O X X
O
on move: X
X
O X X
O O
on move: O
X X
O X X
O O
on move: X
X X
O X X
O O O
Episode 76, Total Reward: -1
Average Reward: 0.2894736842105263
on move: O
X
on move: X
X
O
on move: O
X
O
X
on move: X
X
O O
X
on move: O
X X
O O
X
on move: X
X X O
O O
X
on move: O
X X O
O O
X X
on move: X
X X O
O O
X X O
Episode 77, Total Reward: -1
Average Reward: 0.2727272727272727
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
O X
X O
on move: O
X
O X
X O
on move: X
O X
O X
X O
on move: O
O X X
O X
X O
on move: X
O X X
O X
O X O
Episode 78, Total Reward: -1
Average Reward: 0.2564102564102564
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
X X O
on move: O
O
X
X X O
on move: X
O
X O
X X O
Episode 79, Total Reward: -1
Average Reward: 0.24050632911392406
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O
O
on move: O
X X
O X
O
on move: X
X X
O X O
O
on move: O
X X
O X O
O X
Episode 80, Total Reward: 1
Average Reward: 0.25
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
X O
X
O
on move: O
X O
X
X O
Episode 81, Total Reward: 1
Average Reward: 0.25925925925925924
on move: O
X
on move: X
O X
on move: O
X
O X
on move: X
O X
O X
on move: O
O X
O X X
on move: X
O X
O
O X X
on move: O
O X X
O
O X X
on move: X
O X X
O O
O X X
on move: O
O X X
X O O
O X X
Episode 82, Total Reward: 0
Average Reward: 0.25609756097560976
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O O
X X
on move: O
O O
X
X X
on move: X
O O
X O
X X
on move: O
X O O
X O
X X
on move: X
X O O
X O O
X X
on move: O
X O O
X O O
X X X
Episode 83, Total Reward: 1
Average Reward: 0.26506024096385544
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X O
X
O
on move: O
X O
X X
O
on move: X
X O O
X X
O
on move: O
X O O
X X
O X
Episode 84, Total Reward: 1
Average Reward: 0.27380952380952384
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X O
O
on move: O
X X
X O
O
on move: X
X X
X O
O O
on move: O
X X
X O
O X O
on move: X
X O X
X O
O X O
on move: O
X O X
X X O
O X O
Episode 85, Total Reward: 0
Average Reward: 0.27058823529411763
on move: O
X
on move: X
O
X
on move: O
X
O
X
on move: X
X
O O
X
on move: O
X
O O
X X
on move: X
X O
O O
X X
on move: O
X X O
O O
X X
on move: X
X X O
O O O
X X
Episode 86, Total Reward: -1
Average Reward: 0.2558139534883721
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O O
on move: O
X X
O O
X
on move: X
X X
O O
X O
on move: O
X X X
O O
X O
Episode 87, Total Reward: 1
Average Reward: 0.26436781609195403
on move: O
X
on move: X
X
O
on move: O
X
O
X
on move: X
X
O
X O
on move: O
X X
O
X O
on move: X
X X
O O
X O
on move: O
X X
O O X
X O
on move: X
X X O
O O X
X O
on move: O
X X O
O O X
X O X
Episode 88, Total Reward: 0
Average Reward: 0.26136363636363635
on move: O
X
on move: X
X
O
on move: O
X
X O
on move: X
X
X O O
on move: O
X
X
X O O
on move: X
O X
X
X O O
on move: O
O X
X X
X O O
on move: X
O X
X X O
X O O
on move: O
O X X
X X O
X O O
Episode 89, Total Reward: 1
Average Reward: 0.2696629213483146
on move: O
X
on move: X
O X
on move: O
X
O X
on move: X
O
X
O X
on move: O
O
X
O X X
on move: X
O
O X
O X X
on move: O
X O
O X
O X X
on move: X
X O O
O X
O X X
on move: O
X O O
O X X
O X X
Episode 90, Total Reward: 1
Average Reward: 0.2777777777777778
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
O
X
on move: O
O X
O
X X
on move: X
O X
O O
X X
on move: O
O X X
O O
X X
on move: X
O X X
O O
X O X
on move: O
O X X
X O O
X O X
Episode 91, Total Reward: 0
Average Reward: 0.27472527472527475
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
O X
on move: O
O X
X
O X
on move: X
O X O
X
O X
on move: O
O X O
X X
O X
on move: X
O X O
O X X
O X
Episode 92, Total Reward: -1
Average Reward: 0.2608695652173913
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O O
X X
on move: O
X
O O
X X
on move: X
X O
O O
X X
on move: O
X O
O X O
X X
on move: X
O X O
O X O
X X
on move: O
O X O
O X O
X X X
Episode 93, Total Reward: 1
Average Reward: 0.26881720430107525
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O
O
on move: O
X X
O
O X
on move: X
X X O
O
O X
on move: O
X X O
O X
O X
on move: X
X X O
O X
O X O
on move: O
X X O
O X X
O X O
Episode 94, Total Reward: 1
Average Reward: 0.2765957446808511
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
X O
X O
on move: O
X O
X
X O
on move: X
X O
X
O X O
on move: O
X O
X X
O X O
on move: X
X O O
X X
O X O
on move: O
X O O
X X X
O X O
Episode 95, Total Reward: 1
Average Reward: 0.28421052631578947
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X
X O
on move: O
O X
X
X O
Episode 96, Total Reward: 1
Average Reward: 0.2916666666666667
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
O X
X
on move: O
O
X O X
X
on move: X
O
X O X
X O
Episode 97, Total Reward: -1
Average Reward: 0.27835051546391754
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X
X O
on move: O
O X
X
X O
on move: X
O O X
X
X O
on move: O
O O X
X X
X O
on move: X
O O X
X O X
X O
Episode 98, Total Reward: -1
Average Reward: 0.2653061224489796
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O O
X
X
on move: O
O X O
X
X
on move: X
O X O
O X
X
on move: O
O X O
O X
X X
on move: X
O X O
O X
X X O
Episode 99, Total Reward: -1
Average Reward: 0.25252525252525254
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
O
X O
X
on move: O
X O
X O
X
on move: X
X O
X O
O X
on move: O
X O
X O
O X X
on move: X
O X O
X O
O X X
on move: O
O X O
X X O
O X X
Episode 100, Total Reward: 1
Average Reward: 0.26
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
O
X O
X
on move: O
O
X X O
X
on move: X
O O
X X O
X
on move: O
O O
X X O
X X
on move: X
O O O
X X O
X X
Episode 101, Total Reward: -1
Average Reward: 0.24752475247524752
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
X X O
on move: O
X
O
X X O
on move: X
X
O O
X X O
on move: O
X X
O O
X X O
on move: X
X O X
O O
X X O
on move: O
X O X
O O X
X X O
Episode 102, Total Reward: 0
Average Reward: 0.24509803921568626
on move: O
X
on move: X
O
X
on move: O
X
O
X
on move: X
X
O O
X
on move: O
X
O O
X X
on move: X
X O
O O
X X
on move: O
X O X
O O
X X
on move: X
X O X
O O
X O X
Episode 103, Total Reward: -1
Average Reward: 0.23300970873786409
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O O
on move: O
X X
O O X
on move: X
X X
O O X
O
on move: O
X X
O O X
X O
on move: X
X X
O O X
O X O
on move: O
X X X
O O X
O X O
Episode 104, Total Reward: 1
Average Reward: 0.2403846153846154
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O O
on move: O
X
X X
O O
on move: X
X O
X X
O O
on move: O
X O
X X
X O O
on move: X
X O
O X X
X O O
on move: O
X X O
O X X
X O O
Episode 105, Total Reward: 0
Average Reward: 0.23809523809523808
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X
O O
on move: O
X X
X
O O
on move: X
X X O
X
O O
on move: O
X X O
X
O X O
on move: X
X X O
O X
O X O
on move: O
X X O
O X X
O X O
Episode 106, Total Reward: 1
Average Reward: 0.24528301886792453
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X O X
O
on move: O
X O X
X O
on move: X
X O X
X O
O
on move: O
X O X
X O
O X
on move: X
X O X
O X O
O X
on move: O
X O X
O X O
O X X
Episode 107, Total Reward: 1
Average Reward: 0.2523364485981308
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
X O
X O
on move: O
X O
X
X O
on move: X
X O
X O
X O
Episode 108, Total Reward: -1
Average Reward: 0.24074074074074073
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X
X O
on move: O
O
X X
X O
on move: X
O
X X O
X O
on move: O
O X
X X O
X O
on move: X
O O X
X X O
X O
on move: O
O O X
X X O
X X O
Episode 109, Total Reward: 1
Average Reward: 0.24770642201834864
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O O
X X
on move: O
X
O O
X X
on move: X
O X
O O
X X
on move: O
O X
O X O
X X
on move: X
O X O
O X O
X X
on move: O
O X O
O X O
X X X
Episode 110, Total Reward: 1
Average Reward: 0.2545454545454545
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O O X
X
on move: O
O O X
X
X
on move: X
O O X
X
O X
on move: O
O O X
X X
O X
on move: X
O O X
X X
O O X
on move: O
O O X
X X X
O O X
Episode 111, Total Reward: 1
Average Reward: 0.26126126126126126
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X
X O
on move: O
O
X X
X O
on move: X
O O
X X
X O
on move: O
O O
X X X
X O
Episode 112, Total Reward: 1
Average Reward: 0.26785714285714285
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
X O
O
X
on move: O
X O
O X
X
on move: X
O X O
O X
X
on move: O
O X O
O X X
X
Episode 113, Total Reward: 1
Average Reward: 0.2743362831858407
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X O
O
on move: O
X X
X O
O
on move: X
X X
X O O
O
on move: O
X X X
X O O
O
Episode 114, Total Reward: 1
Average Reward: 0.2807017543859649
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O
O X
X
on move: O
X O
O X
X
Episode 115, Total Reward: 1
Average Reward: 0.28695652173913044
on move: O
X
on move: X
O X
on move: O
X
O X
on move: X
X O
O X
on move: O
X O
X
O X
on move: X
X O O
X
O X
on move: O
X O O
X
O X X
on move: X
X O O
O X
O X X
on move: O
X O O
O X X
O X X
Episode 116, Total Reward: 1
Average Reward: 0.29310344827586204
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X
O O
on move: O
X X
X
O O
on move: X
X X
O X
O O
on move: O
X X
O X X
O O
on move: X
X X
O X X
O O O
Episode 117, Total Reward: -1
Average Reward: 0.28205128205128205
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
X O
on move: O
O X
X O
X
on move: X
O X
X O
O X
on move: O
O X X
X O
O X
on move: X
O X X
X O O
O X
on move: O
O X X
X O O
O X X
Episode 118, Total Reward: 0
Average Reward: 0.2796610169491525
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
O X X
O
on move: O
O X X
O
X
on move: X
O X X
O O
X
on move: O
O X X
X O O
X
on move: X
O X X
X O O
X O
Episode 119, Total Reward: -1
Average Reward: 0.2689075630252101
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X
X O
on move: O
X O
X
X O
on move: X
X O
X O
X O
on move: O
X X O
X O
X O
on move: X
X X O
O X O
X O
on move: O
X X O
O X O
X O X
Episode 120, Total Reward: 1
Average Reward: 0.275
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O O
X
X
on move: O
O O
X X
X
on move: X
O O
O X X
X
on move: O
O O
O X X
X X
on move: X
O O
O X X
O X X
Episode 121, Total Reward: -1
Average Reward: 0.2644628099173554
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
X O
on move: O
O X
X
X O
on move: X
O X
O X
X O
on move: O
O X X
O X
X O
on move: X
O X X
O O X
X O
Episode 122, Total Reward: -1
Average Reward: 0.2540983606557377
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O
O
on move: O
X X
O X
O
on move: X
X X
O O X
O
on move: O
X X
O O X
X O
on move: X
X X
O O X
O X O
on move: O
X X X
O O X
O X O
Episode 123, Total Reward: 1
Average Reward: 0.2601626016260163
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O O
X
X
on move: O
O O
X X
X
on move: X
O O
O X X
X
on move: O
O O
O X X
X X
on move: X
O O
O X X
X O X
on move: O
O X O
O X X
X O X
Episode 124, Total Reward: 0
Average Reward: 0.25806451612903225
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
X
O
on move: O
O X
X
X O
on move: X
O X
X
X O O
on move: O
O X X
X
X O O
Episode 125, Total Reward: 1
Average Reward: 0.264
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
O X
X
O
on move: O
O X
X X
O
on move: X
O O X
X X
O
on move: O
O O X
X X
O X
Episode 126, Total Reward: 1
Average Reward: 0.2698412698412698
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O
O
on move: O
X X
O
X O
on move: X
X X O
O
X O
on move: O
X X O
O
X X O
on move: X
X X O
O O
X X O
on move: O
X X O
O O X
X X O
Episode 127, Total Reward: 0
Average Reward: 0.2677165354330709
on move: O
X
on move: X
X
O
on move: O
X
O X
on move: X
X
O
O X
on move: O
X X
O
O X
on move: X
X X
O O
O X
on move: O
X X
O O X
O X
on move: X
X X O
O O X
O X
Episode 128, Total Reward: -1
Average Reward: 0.2578125
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
X O
X
O
on move: O
X O
X X
O
on move: X
X O
X X
O O
on move: O
X O
X X
X O O
on move: X
O X O
X X
X O O
on move: O
O X O
X X X
X O O
Episode 129, Total Reward: 1
Average Reward: 0.26356589147286824
on move: O
X
on move: X
X O
on move: O
X O
X
on move: X
O
X O
X
on move: O
O
X O
X X
on move: X
O O
X O
X X
on move: O
O O X
X O
X X
on move: X
O O X
X O
O X X
on move: O
O O X
X X O
O X X
Episode 130, Total Reward: 0
Average Reward: 0.26153846153846155
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O X
O X
on move: O
O X
X
O X
on move: X
O X O
X
O X
on move: O
O X O
X
X O X
on move: X
O X O
O X
X O X
on move: O
O X O
O X X
X O X
Episode 131, Total Reward: 0
Average Reward: 0.2595419847328244
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
O X
O
on move: O
X
O X
X O
on move: X
O X
O X
X O
on move: O
O X
O X X
X O
on move: X
O X
O X X
X O O
on move: O
O X X
O X X
X O O
Episode 132, Total Reward: 1
Average Reward: 0.26515151515151514
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O O
X
X
on move: O
O O
X X
X
on move: X
O O
X X O
X
on move: O
O O
X X O
X X
on move: X
O O O
X X O
X X
Episode 133, Total Reward: -1
Average Reward: 0.2556390977443609
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
O
X X
O
on move: O
O
X X
X O
on move: X
O O
X X
X O
on move: O
O O
X X X
X O
Episode 134, Total Reward: 1
Average Reward: 0.26119402985074625
on move: O
X
on move: X
O
X
on move: O
O X
X
on move: X
O X
O
X
on move: O
O X X
O
X
on move: X
O X X
O
X O
on move: O
O X X
X O
X O
Episode 135, Total Reward: 1
Average Reward: 0.26666666666666666
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
X
X O O
on move: O
X
X O O
X
on move: X
X
X O O
X O
on move: O
X
X O O
X O X
on move: X
O X
X O O
X O X
Episode 136, Total Reward: -1
Average Reward: 0.25735294117647056
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
O X
X
on move: O
O
O X
X X
on move: X
O
O O X
X X
on move: O
O X
O O X
X X
Episode 137, Total Reward: 1
Average Reward: 0.26277372262773724
on move: O
X
on move: X
O
X
on move: O
O
X
X
on move: X
O
X
X O
on move: O
X O
X
X O
on move: X
X O
X O
X O
on move: O
X O X
X O
X O
on move: X
X O X
X O
O X O
on move: O
X O X
X X O
O X O
Episode 138, Total Reward: 0
Average Reward: 0.2608695652173913
on move: O
X
on move: X
X
O
on move: O
X
X
O
on move: X
X
X O
O
on move: O
X
X O X
O
on move: X
O X
X O X
O
on move: O
O X
X O X
X O
on move: X
O X
X O X
X O O
Episode 139, Total Reward: -1
Average Reward: 0.2517985611510791
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O O
X X
on move: O
O O
X
X X
on move: X
O O
O X
X X
on move: O
O O
O X X
X X
on move: X
O O O
O X X
X X
Episode 140, Total Reward: -1
Average Reward: 0.24285714285714285
on move: O
X
on move: X
X
O
on move: O
X
O
X
on move: X
X
O
O X
on move: O
X
O X
O X
Episode 141, Total Reward: 1
Average Reward: 0.24822695035460993
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
O
X X
O
on move: O
X O
X X
O
on move: X
X O
X X
O O
on move: O
X O
X X X
O O
Episode 142, Total Reward: 1
Average Reward: 0.2535211267605634
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
O
X
X O
on move: O
O
X X
X O
on move: X
O
X X
X O O
on move: O
O
X X X
X O O
Episode 143, Total Reward: 1
Average Reward: 0.25874125874125875
on move: O
X
on move: X
X
O
on move: O
X
X O
on move: X
O
X
X O
on move: O
O X
X
X O
on move: X
O X O
X
X O
on move: O
O X O
X X
X O
on move: X
O X O
X X
X O O
on move: O
O X O
X X X
X O O
Episode 144, Total Reward: 1
Average Reward: 0.2638888888888889
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X O
O
on move: O
X X O
X O
on move: X
X X O
O
X O
on move: O
X X O
O
X X O
on move: X
X X O
O O
X X O
on move: O
X X O
O O X
X X O
Episode 145, Total Reward: 0
Average Reward: 0.2620689655172414
on move: O
X
on move: X
O
X
on move: O
O
X X
on move: X
O
X X
O
on move: O
O
X X
X O
on move: X
O
X O X
X O
Episode 146, Total Reward: -1
Average Reward: 0.2534246575342466
on move: O
X
on move: X
X O
on move: O
X
X O
on move: X
O
X
X O
on move: O
X O
X
X O
on move: X
X O
X O
X O
Episode 147, Total Reward: -1
Average Reward: 0.24489795918367346
on move: O
X
on move: X
X
O
on move: O
X X
O
on move: X
X X
O
O
on move: O
X X X
O
O
Episode 148, Total Reward: 1
Average Reward: 0.25
on move: O
X
on move: X
O
X
on move: O
X O
X
on move: X
X O O
X
on move: O
X O O
X X
on move: X
X O O
O
X X
on move: O
X O O
X O
X X
on move: X
X O O
X O O
X X
on move: O
X O O
X O O
X X X
Episode 149, Total Reward: 1
Average Reward: 0.2550335570469799
on move: O
X
on move: X
O X
on move: O
O X
X
on move: X
O O X
X
on move: O
X
O O X
X
on move: X
X
O O X
O X
on move: O
X X
O O X
O X
Episode 150, Total Reward: 1
Average Reward: 0.26