AIM-PIbd-32-Tyorkin-D-V/lab_6/lab6.ipynb

Лабораторная работа №6¶

Задача:¶

Разработка и запуск проекта по обучению с подкреплением для игры "Крестики-нолики"

Что нужно сделать:

1. Перевести проект на библиотеку Gymnasium и современную версию Python. Gymnasium — это современная альтернатива библиотеке Gym, предоставляющая удобные инструменты для создания и тестирования моделей обучения с подкреплением.
2. Реализовать агента для игры "Крестики-нолики" в виде отдельного класса. Следуя примеру из лекции, создайте класс агента, который будет отвечать за принятие решений и обучение.
3. Основной цикл обучения для работы с отдельным классом агента. Обучение происходит через взаимодействие с классом агента.
4. Протестировать новую версию программы. Убедиться, что программа работает корректно и агент успешно обучается играть в "Крестики-нолики".

Обучение с подкреплением (Reinforcement Learning, RL) — это подход машинного обучения, при котором агент учится принимать решения, взаимодействуя с окружающей средой. Цель агента — выбирать действия, которые максимизируют накопленную награду в долгосрочной перспективе.

Ссылка на игру "Крестики-нолики": https://github.com/nczempin/gym-tic-tac-toe

Переход на среду Gymnasium¶

Gymnasium (прежнее название — OpenAI Gym) — это библиотека, предназначенная для разработки и тестирования алгоритмов обучения с подкреплением. Она предлагает широкий набор стандартных сред для RL, включая игры, задачи управления и моделирование.

Gymnasium предлагает обновленный API по сравнению с Gym, который:

• Упрощает методы reset и step.
• Добавляет новые функции, такие как поддержка тайм-аутов и контроль количества шагов.
• Улучшает поддержку пользовательских сред.

Основные преимущества Gymnasium:

• Стандартизированные интерфейсы для взаимодействия со средой.
• Простая интеграция с популярными библиотеками RL.
• Поддержка создания собственных сред.

Ключевые функции Gymnasium:

• env.reset() — инициализация среды.
• env.step(action) — выполнение действия и переход в новое состояние.
• env.render() — визуализация текущего состояния среды.

In [2]:

import gymnasium as gym
from gymnasium import spaces

class TicTacToeEnv(gym.Env):
    metadata = {'render.modes': ['human']}
    
    symbols = ['O', ' ', 'X']

    def __init__(self):
        super().__init__()
        self.action_space = spaces.Discrete(9)
        self.observation_space = spaces.Discrete(9 * 3 * 2)
        self.reset()

    def step(self, action):
        done = False
        reward = 0

        p, square = action  # p - игрок (1 или -1), square - номер клетки

        board = self.state['board']
        proposed = board[square] 
        om = self.state['on_move'] 
        if proposed != 0:  # Клетка уже занята
            print(f"Незаконный ход: Квадрат {square} уже занят.")
            done = True
            reward = -1 * om 
        if p != om:  # Не тот игрок на ходу
            print(f"Незаконный ход: игрок {p} не находится в движении")
            done = True
            reward = -1 * om
        else:
            board[square] = p
            self.state['on_move'] = -p

        for i in range(3):
            # Горизонтали и вертикали
            if (board[i * 3] == p and board[i * 3 + 1] == p and board[i * 3 + 2] == p) or \
               (board[i] == p and board[i + 3] == p and board[i + 6] == p):
                reward = p
                done = True
                break

        # Диагонали
        if (board[0] == p and board[4] == p and board[8] == p) or \
           (board[2] == p and board[4] == p and board[6] == p):
            reward = p
            done = True
                
        return self.state, reward, done, {}

    def reset(self):
        self.state = {}
        self.state['board'] = [0, 0, 0, 0, 0, 0, 0, 0, 0] 
        self.state['on_move'] = 1 
        return self.state, {}

    def render(self, close=False):
        if close:
            return
        print("on move: " , self.symbols[self.state['on_move']+1])
        for i in range (9):
            print (self.symbols[self.state['board'][i]+1], end=" ");
            if ((i % 3) == 2):
                print();

    def move_generator(self):
        moves = []
        for i in range(9):
            if self.state['board'][i] == 0:
                p = self.state['on_move']
                m = [p, i]
                moves.append(m)
        return moves

Разработка агента¶

В рамках обучения с подкреплением (Reinforcement Learning, RL) агент — это система, которая взаимодействует с окружающей средой для достижения определенной цели. Основная задача агента — разработать стратегию, которая максимизирует накопленную награду в долгосрочной перспективе.

Роль агента:

• Принятие решений: Агент выбирает действие на основе текущего состояния среды.
• Получение обратной связи: После выполнения действия агент получает от среды награду и информацию о новом состоянии.

Основные функции агента:

• Выбор действия: Использует алгоритмы или стратегии для определения следующего шага.
• Обучение: Анализирует полученный опыт и обновляет свои знания или стратегию для повышения эффективности.
• Адаптация: Адаптируется к изменениям в окружающей среде.

In [3]:

import random

# Реализация Агента, который в рамках обучения с подкреплением взаимодействует со средой и вырабатывает наилучшую стратегию  

class Agent:
    def __init__(self, symbol):
        self.symbol = symbol  # Символ игрока (1 - X, -1 - O)
    
    def get_action(self, moves):
        return random.choice(moves)  # Выбираем случайный ход из доступных

Основной цикл обучения¶

Основной цикл обучения включает создание и взаимодействие со средой. После завершения игры агент получает очки, которые отражают его успехи и позволяют оценить, насколько успешно прошло обучение системы.

In [4]:

# Основной цикл обучения (работа с отдельным классом агента)

# Создание среды для игры в крестики-нолики
environment = TicTacToeEnv()

# Создание агента (играющего крестиками)
agent = Agent(symbol=1)

num_episodes = 150  # Количество эпизодов (игр) для обучения
collected_rewards = []      # Список для хранения наград/побед в каждом эпизоде 

# Переменная для отслеживания символа и текущего игрока
oom = 1

for i in range(num_episodes):
    # Сброс среды и начало нового эпизода
    state, _ = environment.reset() 

    # Общая награда за эпизод
    total_reward = 0

    # Флаг завершения игры
    done = False
    om = oom 

    # Максимум 9 ходов, поскольку поле 3x3 
    for j in range(9): 
        moves = environment.move_generator() 

        # Ходов нет, заканчиваем игру
        if not moves:
            break

        
        if len(moves) == 1:
            move = moves[0]    # Если остался один ход на основе стратегии
        else:
            move = agent.get_action(moves)   # Агент выбирает ход на основе стратегии

        # Выполнение хода и обновление состояния игры
        next_state, reward, done, info = environment.step(move)
        total_reward += reward
        state = next_state

        # Отображаем текущее состояние игры
        environment.render()

        if done:
            break

        om = -om    # Смена игрока

    collected_rewards.append(total_reward)

    print(f"Episode {i+1}, Total Reward: {total_reward}")
    average_reward = sum(collected_rewards) / len(collected_rewards)
    print(f"Average Reward: {average_reward}")

on move:  O
      
  X   
      
on move:  X
    O 
  X   
      
on move:  O
    O 
  X X 
      
on move:  X
  O O 
  X X 
      
on move:  O
X O O 
  X X 
      
on move:  X
X O O 
  X X 
    O 
on move:  O
X O O 
X X X 
    O 
Episode 1, Total Reward: 1
Average Reward: 1.0
on move:  O
      
      
X     
on move:  X
      
    O 
X     
on move:  O
      
    O 
X   X 
on move:  X
O     
    O 
X   X 
on move:  O
O   X 
    O 
X   X 
on move:  X
O   X 
    O 
X O X 
on move:  O
O   X 
  X O 
X O X 
Episode 2, Total Reward: 1
Average Reward: 1.0
on move:  O
      
  X   
      
on move:  X
O     
  X   
      
on move:  O
O     
  X   
  X   
on move:  X
O     
O X   
  X   
on move:  O
O   X 
O X   
  X   
on move:  X
O O X 
O X   
  X   
on move:  O
O O X 
O X   
X X   
Episode 3, Total Reward: 1
Average Reward: 1.0
on move:  O
      
X     
      
on move:  X
      
X   O 
      
on move:  O
      
X X O 
      
on move:  X
  O   
X X O 
      
on move:  O
  O   
X X O 
  X   
on move:  X
  O O 
X X O 
  X   
on move:  O
  O O 
X X O 
  X X 
on move:  X
O O O 
X X O 
  X X 
Episode 4, Total Reward: -1
Average Reward: 0.5
on move:  O
  X   
      
      
on move:  X
O X   
      
      
on move:  O
O X   
  X   
      
on move:  X
O X   
  X   
O     
on move:  O
O X   
  X X 
O     
on move:  X
O X O 
  X X 
O     
on move:  O
O X O 
  X X 
O   X 
on move:  X
O X O 
O X X 
O   X 
Episode 5, Total Reward: -1
Average Reward: 0.2
on move:  O
      
      
  X   
on move:  X
      
  O   
  X   
on move:  O
      
  O   
X X   
on move:  X
    O 
  O   
X X   
on move:  O
    O 
  O X 
X X   
on move:  X
    O 
O O X 
X X   
on move:  O
    O 
O O X 
X X X 
Episode 6, Total Reward: 1
Average Reward: 0.3333333333333333
on move:  O
    X 
      
      
on move:  X
  O X 
      
      
on move:  O
  O X 
X     
      
on move:  X
  O X 
X     
    O 
on move:  O
X O X 
X     
    O 
on move:  X
X O X 
X     
O   O 
on move:  O
X O X 
X   X 
O   O 
on move:  X
X O X 
X O X 
O   O 
on move:  O
X O X 
X O X 
O X O 
Episode 7, Total Reward: 0
Average Reward: 0.2857142857142857
on move:  O
      
X     
      
on move:  X
O     
X     
      
on move:  O
O     
X     
    X 
on move:  X
O   O 
X     
    X 
on move:  O
O   O 
X     
  X X 
on move:  X
O   O 
X     
O X X 
on move:  O
O   O 
X X   
O X X 
on move:  X
O   O 
X X O 
O X X 
on move:  O
O X O 
X X O 
O X X 
Episode 8, Total Reward: 1
Average Reward: 0.375
on move:  O
X     
      
      
on move:  X
X     
O     
      
on move:  O
X     
O   X 
      
on move:  X
X     
O   X 
    O 
on move:  O
X     
O   X 
X   O 
on move:  X
X     
O   X 
X O O 
on move:  O
X     
O X X 
X O O 
on move:  X
X   O 
O X X 
X O O 
on move:  O
X X O 
O X X 
X O O 
Episode 9, Total Reward: 0
Average Reward: 0.3333333333333333
on move:  O
      
  X   
      
on move:  X
      
O X   
      
on move:  O
      
O X X 
      
on move:  X
      
O X X 
  O   
on move:  O
      
O X X 
X O   
on move:  X
O     
O X X 
X O   
on move:  O
O     
O X X 
X O X 
on move:  X
O   O 
O X X 
X O X 
on move:  O
O X O 
O X X 
X O X 
Episode 10, Total Reward: 0
Average Reward: 0.3
on move:  O
  X   
      
      
on move:  X
  X   
O     
      
on move:  O
  X   
O     
X     
on move:  X
  X   
O     
X   O 
on move:  O
  X   
O     
X X O 
on move:  X
  X   
O O   
X X O 
on move:  O
  X X 
O O   
X X O 
on move:  X
  X X 
O O O 
X X O 
Episode 11, Total Reward: -1
Average Reward: 0.18181818181818182
on move:  O
X     
      
      
on move:  X
X O   
      
      
on move:  O
X O   
X     
      
on move:  X
X O   
X     
O     
on move:  O
X O X 
X     
O     
on move:  X
X O X 
X   O 
O     
on move:  O
X O X 
X   O 
O X   
on move:  X
X O X 
X O O 
O X   
on move:  O
X O X 
X O O 
O X X 
Episode 12, Total Reward: 0
Average Reward: 0.16666666666666666
on move:  O
      
      
    X 
on move:  X
    O 
      
    X 
on move:  O
    O 
X     
    X 
on move:  X
    O 
X     
O   X 
on move:  O
    O 
X   X 
O   X 
on move:  X
  O O 
X   X 
O   X 
on move:  O
X O O 
X   X 
O   X 
on move:  X
X O O 
X O X 
O   X 
Episode 13, Total Reward: -1
Average Reward: 0.07692307692307693
on move:  O
      
    X 
      
on move:  X
      
    X 
  O   
on move:  O
X     
    X 
  O   
on move:  X
X     
    X 
  O O 
on move:  O
X     
X   X 
  O O 
on move:  X
X     
X   X 
O O O 
Episode 14, Total Reward: -1
Average Reward: 0.0
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O   X 
      
X     
on move:  X
O   X 
      
X   O 
on move:  O
O   X 
      
X X O 
on move:  X
O   X 
  O   
X X O 
Episode 15, Total Reward: -1
Average Reward: -0.06666666666666667
on move:  O
      
      
  X   
on move:  X
      
O     
  X   
on move:  O
    X 
O     
  X   
on move:  X
    X 
O   O 
  X   
on move:  O
    X 
O X O 
  X   
on move:  X
    X 
O X O 
  X O 
on move:  O
    X 
O X O 
X X O 
Episode 16, Total Reward: 1
Average Reward: 0.0
on move:  O
  X   
      
      
on move:  X
  X   
      
  O   
on move:  O
X X   
      
  O   
on move:  X
X X O 
      
  O   
on move:  O
X X O 
X     
  O   
on move:  X
X X O 
X     
O O   
on move:  O
X X O 
X   X 
O O   
on move:  X
X X O 
X   X 
O O O 
Episode 17, Total Reward: -1
Average Reward: -0.058823529411764705
on move:  O
    X 
      
      
on move:  X
    X 
      
  O   
on move:  O
X   X 
      
  O   
on move:  X
X   X 
  O   
  O   
on move:  O
X   X 
  O   
X O   
on move:  X
X   X 
O O   
X O   
on move:  O
X   X 
O O   
X O X 
on move:  X
X   X 
O O O 
X O X 
Episode 18, Total Reward: -1
Average Reward: -0.1111111111111111
on move:  O
      
      
  X   
on move:  X
      
  O   
  X   
on move:  O
      
  O   
X X   
on move:  X
    O 
  O   
X X   
on move:  O
  X O 
  O   
X X   
on move:  X
  X O 
  O   
X X O 
on move:  O
X X O 
  O   
X X O 
on move:  X
X X O 
  O O 
X X O 
Episode 19, Total Reward: -1
Average Reward: -0.15789473684210525
on move:  O
      
      
    X 
on move:  X
    O 
      
    X 
on move:  O
    O 
      
X   X 
on move:  X
    O 
      
X O X 
on move:  O
  X O 
      
X O X 
on move:  X
  X O 
  O   
X O X 
on move:  O
  X O 
X O   
X O X 
on move:  X
O X O 
X O   
X O X 
on move:  O
O X O 
X O X 
X O X 
Episode 20, Total Reward: 0
Average Reward: -0.15
on move:  O
      
X     
      
on move:  X
      
X     
    O 
on move:  O
      
X   X 
    O 
on move:  X
      
X O X 
    O 
on move:  O
X     
X O X 
    O 
on move:  X
X     
X O X 
  O O 
on move:  O
X X   
X O X 
  O O 
on move:  X
X X O 
X O X 
  O O 
on move:  O
X X O 
X O X 
X O O 
Episode 21, Total Reward: 1
Average Reward: -0.09523809523809523
on move:  O
      
  X   
      
on move:  X
      
  X   
O     
on move:  O
X     
  X   
O     
on move:  X
X     
O X   
O     
on move:  O
X     
O X X 
O     
on move:  X
X     
O X X 
O O   
on move:  O
X     
O X X 
O O X 
Episode 22, Total Reward: 1
Average Reward: -0.045454545454545456
on move:  O
      
      
    X 
on move:  X
  O   
      
    X 
on move:  O
  O   
    X 
    X 
on move:  X
  O   
    X 
  O X 
on move:  O
  O   
  X X 
  O X 
on move:  X
  O O 
  X X 
  O X 
on move:  O
X O O 
  X X 
  O X 
Episode 23, Total Reward: 1
Average Reward: 0.0
on move:  O
      
X     
      
on move:  X
      
X     
O     
on move:  O
  X   
X     
O     
on move:  X
O X   
X     
O     
on move:  O
O X   
X     
O X   
on move:  X
O X   
X   O 
O X   
on move:  O
O X   
X   O 
O X X 
on move:  X
O X   
X O O 
O X X 
on move:  O
O X X 
X O O 
O X X 
Episode 24, Total Reward: 0
Average Reward: 0.0
on move:  O
      
X     
      
on move:  X
  O   
X     
      
on move:  O
  O X 
X     
      
on move:  X
  O X 
X     
    O 
on move:  O
  O X 
X     
X   O 
on move:  X
  O X 
X     
X O O 
on move:  O
X O X 
X     
X O O 
Episode 25, Total Reward: 1
Average Reward: 0.04
on move:  O
      
    X 
      
on move:  X
      
    X 
    O 
on move:  O
    X 
    X 
    O 
on move:  X
    X 
    X 
O   O 
on move:  O
  X X 
    X 
O   O 
on move:  X
  X X 
    X 
O O O 
Episode 26, Total Reward: -1
Average Reward: 0.0
on move:  O
X     
      
      
on move:  X
X   O 
      
      
on move:  O
X   O 
  X   
      
on move:  X
X   O 
  X   
O     
on move:  O
X   O 
  X X 
O     
on move:  X
X   O 
  X X 
O O   
on move:  O
X   O 
  X X 
O O X 
Episode 27, Total Reward: 1
Average Reward: 0.037037037037037035
on move:  O
      
  X   
      
on move:  X
O     
  X   
      
on move:  O
O     
  X   
X     
on move:  X
O     
  X   
X   O 
on move:  O
O X   
  X   
X   O 
on move:  X
O X   
O X   
X   O 
on move:  O
O X   
O X   
X X O 
Episode 28, Total Reward: 1
Average Reward: 0.07142857142857142
on move:  O
      
  X   
      
on move:  X
      
  X O 
      
on move:  O
  X   
  X O 
      
on move:  X
  X   
O X O 
      
on move:  O
  X   
O X O 
  X   
Episode 29, Total Reward: 1
Average Reward: 0.10344827586206896
on move:  O
  X   
      
      
on move:  X
  X   
      
    O 
on move:  O
  X   
  X   
    O 
on move:  X
  X   
O X   
    O 
on move:  O
  X X 
O X   
    O 
on move:  X
O X X 
O X   
    O 
on move:  O
O X X 
O X   
X   O 
Episode 30, Total Reward: 1
Average Reward: 0.13333333333333333
on move:  O
      
      
X     
on move:  X
      
      
X   O 
on move:  O
    X 
      
X   O 
on move:  X
    X 
      
X O O 
on move:  O
    X 
    X 
X O O 
on move:  X
O   X 
    X 
X O O 
on move:  O
O   X 
X   X 
X O O 
on move:  X
O   X 
X O X 
X O O 
Episode 31, Total Reward: -1
Average Reward: 0.0967741935483871
on move:  O
      
    X 
      
on move:  X
      
  O X 
      
on move:  O
      
  O X 
    X 
on move:  X
    O 
  O X 
    X 
on move:  O
X   O 
  O X 
    X 
on move:  X
X O O 
  O X 
    X 
on move:  O
X O O 
X O X 
    X 
on move:  X
X O O 
X O X 
  O X 
Episode 32, Total Reward: -1
Average Reward: 0.0625
on move:  O
      
      
X     
on move:  X
O     
      
X     
on move:  O
O     
    X 
X     
on move:  X
O O   
    X 
X     
on move:  O
O O   
  X X 
X     
on move:  X
O O   
  X X 
X O   
on move:  O
O O   
  X X 
X O X 
on move:  X
O O   
O X X 
X O X 
on move:  O
O O X 
O X X 
X O X 
Episode 33, Total Reward: 1
Average Reward: 0.09090909090909091
on move:  O
X     
      
      
on move:  X
X     
      
    O 
on move:  O
X     
    X 
    O 
on move:  X
X O   
    X 
    O 
on move:  O
X O   
    X 
  X O 
on move:  X
X O   
O   X 
  X O 
on move:  O
X O   
O X X 
  X O 
on move:  X
X O O 
O X X 
  X O 
on move:  O
X O O 
O X X 
X X O 
Episode 34, Total Reward: 0
Average Reward: 0.08823529411764706
on move:  O
  X   
      
      
on move:  X
  X   
      
    O 
on move:  O
X X   
      
    O 
on move:  X
X X   
O     
    O 
on move:  O
X X   
O     
X   O 
on move:  X
X X   
O O   
X   O 
on move:  O
X X   
O O   
X X O 
on move:  X
X X   
O O O 
X X O 
Episode 35, Total Reward: -1
Average Reward: 0.05714285714285714
on move:  O
    X 
      
      
on move:  X
  O X 
      
      
on move:  O
X O X 
      
      
on move:  X
X O X 
  O   
      
on move:  O
X O X 
  O   
    X 
on move:  X
X O X 
  O O 
    X 
on move:  O
X O X 
  O O 
X   X 
on move:  X
X O X 
  O O 
X O X 
Episode 36, Total Reward: -1
Average Reward: 0.027777777777777776
on move:  O
      
X     
      
on move:  X
  O   
X     
      
on move:  O
  O   
X     
    X 
on move:  X
O O   
X     
    X 
on move:  O
O O X 
X     
    X 
on move:  X
O O X 
X     
O   X 
on move:  O
O O X 
X   X 
O   X 
Episode 37, Total Reward: 1
Average Reward: 0.05405405405405406
on move:  O
      
    X 
      
on move:  X
      
  O X 
      
on move:  O
X     
  O X 
      
on move:  X
X     
  O X 
O     
on move:  O
X     
  O X 
O   X 
on move:  X
X O   
  O X 
O   X 
on move:  O
X O   
X O X 
O   X 
on move:  X
X O   
X O X 
O O X 
Episode 38, Total Reward: -1
Average Reward: 0.02631578947368421
on move:  O
X     
      
      
on move:  X
X O   
      
      
on move:  O
X O   
X     
      
on move:  X
X O O 
X     
      
on move:  O
X O O 
X     
    X 
on move:  X
X O O 
X     
  O X 
on move:  O
X O O 
X   X 
  O X 
on move:  X
X O O 
X O X 
  O X 
Episode 39, Total Reward: -1
Average Reward: 0.0
on move:  O
  X   
      
      
on move:  X
  X   
      
  O   
on move:  O
  X   
X     
  O   
on move:  X
  X   
X     
  O O 
on move:  O
  X   
X     
X O O 
on move:  X
  X   
X O   
X O O 
on move:  O
  X X 
X O   
X O O 
on move:  X
O X X 
X O   
X O O 
Episode 40, Total Reward: -1
Average Reward: -0.025
on move:  O
      
  X   
      
on move:  X
      
O X   
      
on move:  O
      
O X   
    X 
on move:  X
      
O X   
O   X 
on move:  O
    X 
O X   
O   X 
on move:  X
    X 
O X O 
O   X 
on move:  O
X   X 
O X O 
O   X 
Episode 41, Total Reward: 1
Average Reward: 0.0
on move:  O
  X   
      
      
on move:  X
  X   
      
  O   
on move:  O
  X   
X     
  O   
on move:  X
  X O 
X     
  O   
on move:  O
  X O 
X     
  O X 
on move:  X
  X O 
X     
O O X 
on move:  O
  X O 
X   X 
O O X 
on move:  X
  X O 
X O X 
O O X 
Episode 42, Total Reward: -1
Average Reward: -0.023809523809523808
on move:  O
X     
      
      
on move:  X
X O   
      
      
on move:  O
X O   
X     
      
on move:  X
X O   
X     
O     
on move:  O
X O   
X     
O   X 
on move:  X
X O   
X O   
O   X 
on move:  O
X O   
X O   
O X X 
on move:  X
X O   
X O O 
O X X 
on move:  O
X O X 
X O O 
O X X 
Episode 43, Total Reward: 0
Average Reward: -0.023255813953488372
on move:  O
X     
      
      
on move:  X
X     
      
  O   
on move:  O
X     
      
  O X 
on move:  X
X O   
      
  O X 
on move:  O
X O X 
      
  O X 
on move:  X
X O X 
      
O O X 
on move:  O
X O X 
  X   
O O X 
Episode 44, Total Reward: 1
Average Reward: 0.0
on move:  O
      
      
  X   
on move:  X
    O 
      
  X   
on move:  O
X   O 
      
  X   
on move:  X
X   O 
  O   
  X   
on move:  O
X   O 
  O   
  X X 
on move:  X
X   O 
O O   
  X X 
on move:  O
X   O 
O O   
X X X 
Episode 45, Total Reward: 1
Average Reward: 0.022222222222222223
on move:  O
X     
      
      
on move:  X
X O   
      
      
on move:  O
X O   
  X   
      
on move:  X
X O   
  X   
O     
on move:  O
X O X 
  X   
O     
on move:  X
X O X 
  X   
O   O 
on move:  O
X O X 
X X   
O   O 
on move:  X
X O X 
X X O 
O   O 
on move:  O
X O X 
X X O 
O X O 
Episode 46, Total Reward: 0
Average Reward: 0.021739130434782608
on move:  O
X     
      
      
on move:  X
X     
  O   
      
on move:  O
X     
  O   
    X 
on move:  X
X     
  O   
  O X 
on move:  O
X     
  O   
X O X 
on move:  X
X   O 
  O   
X O X 
on move:  O
X   O 
X O   
X O X 
Episode 47, Total Reward: 1
Average Reward: 0.0425531914893617
on move:  O
      
      
X     
on move:  X
  O   
      
X     
on move:  O
  O   
      
X X   
on move:  X
  O   
      
X X O 
on move:  O
  O   
X     
X X O 
on move:  X
  O   
X O   
X X O 
on move:  O
X O   
X O   
X X O 
Episode 48, Total Reward: 1
Average Reward: 0.0625
on move:  O
      
      
  X   
on move:  X
O     
      
  X   
on move:  O
O     
  X   
  X   
on move:  X
O     
  X   
  X O 
on move:  O
O   X 
  X   
  X O 
on move:  X
O   X 
  X   
O X O 
on move:  O
O X X 
  X   
O X O 
Episode 49, Total Reward: 1
Average Reward: 0.08163265306122448
on move:  O
      
  X   
      
on move:  X
      
  X   
O     
on move:  O
      
  X X 
O     
on move:  X
  O   
  X X 
O     
on move:  O
  O   
  X X 
O   X 
on move:  X
  O   
O X X 
O   X 
on move:  O
X O   
O X X 
O   X 
Episode 50, Total Reward: 1
Average Reward: 0.1
on move:  O
      
      
    X 
on move:  X
      
      
O   X 
on move:  O
      
    X 
O   X 
on move:  X
      
    X 
O O X 
on move:  O
    X 
    X 
O O X 
Episode 51, Total Reward: 1
Average Reward: 0.11764705882352941
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O X X 
      
      
on move:  X
O X X 
      
O     
on move:  O
O X X 
      
O X   
on move:  X
O X X 
      
O X O 
on move:  O
O X X 
  X   
O X O 
Episode 52, Total Reward: 1
Average Reward: 0.1346153846153846
on move:  O
    X 
      
      
on move:  X
    X 
  O   
      
on move:  O
    X 
  O X 
      
on move:  X
    X 
  O X 
O     
on move:  O
  X X 
  O X 
O     
on move:  X
  X X 
  O X 
O O   
on move:  O
  X X 
X O X 
O O   
on move:  X
O X X 
X O X 
O O   
on move:  O
O X X 
X O X 
O O X 
Episode 53, Total Reward: 1
Average Reward: 0.1509433962264151
on move:  O
  X   
      
      
on move:  X
  X   
      
  O   
on move:  O
  X   
      
X O   
on move:  X
  X O 
      
X O   
on move:  O
  X O 
      
X O X 
on move:  X
  X O 
O     
X O X 
on move:  O
  X O 
O X   
X O X 
on move:  X
  X O 
O X O 
X O X 
on move:  O
X X O 
O X O 
X O X 
Episode 54, Total Reward: 1
Average Reward: 0.16666666666666666
on move:  O
      
    X 
      
on move:  X
    O 
    X 
      
on move:  O
    O 
    X 
    X 
on move:  X
    O 
  O X 
    X 
on move:  O
  X O 
  O X 
    X 
on move:  X
  X O 
O O X 
    X 
on move:  O
  X O 
O O X 
  X X 
on move:  X
O X O 
O O X 
  X X 
on move:  O
O X O 
O O X 
X X X 
Episode 55, Total Reward: 1
Average Reward: 0.18181818181818182
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O X X 
      
      
on move:  X
O X X 
      
    O 
on move:  O
O X X 
    X 
    O 
on move:  X
O X X 
O   X 
    O 
on move:  O
O X X 
O   X 
  X O 
on move:  X
O X X 
O   X 
O X O 
Episode 56, Total Reward: -1
Average Reward: 0.16071428571428573
on move:  O
X     
      
      
on move:  X
X     
O     
      
on move:  O
X     
O     
    X 
on move:  X
X     
O O   
    X 
on move:  O
X     
O O   
X   X 
on move:  X
X     
O O   
X O X 
on move:  O
X X   
O O   
X O X 
on move:  X
X X   
O O O 
X O X 
Episode 57, Total Reward: -1
Average Reward: 0.14035087719298245
on move:  O
    X 
      
      
on move:  X
  O X 
      
      
on move:  O
  O X 
      
    X 
on move:  X
  O X 
      
O   X 
on move:  O
  O X 
    X 
O   X 
Episode 58, Total Reward: 1
Average Reward: 0.15517241379310345
on move:  O
      
X     
      
on move:  X
      
X     
O     
on move:  O
      
X     
O X   
on move:  X
      
X   O 
O X   
on move:  O
      
X   O 
O X X 
on move:  X
  O   
X   O 
O X X 
on move:  O
  O X 
X   O 
O X X 
on move:  X
O O X 
X   O 
O X X 
on move:  O
O O X 
X X O 
O X X 
Episode 59, Total Reward: 0
Average Reward: 0.15254237288135594
on move:  O
    X 
      
      
on move:  X
    X 
      
    O 
on move:  O
    X 
      
X   O 
on move:  X
  O X 
      
X   O 
on move:  O
  O X 
X     
X   O 
on move:  X
  O X 
X     
X O O 
on move:  O
X O X 
X     
X O O 
Episode 60, Total Reward: 1
Average Reward: 0.16666666666666666
on move:  O
X     
      
      
on move:  X
X     
      
    O 
on move:  O
X     
    X 
    O 
on move:  X
X   O 
    X 
    O 
on move:  O
X   O 
  X X 
    O 
on move:  X
X   O 
  X X 
O   O 
on move:  O
X   O 
  X X 
O X O 
on move:  X
X O O 
  X X 
O X O 
on move:  O
X O O 
X X X 
O X O 
Episode 61, Total Reward: 1
Average Reward: 0.18032786885245902
on move:  O
      
    X 
      
on move:  X
      
    X 
O     
on move:  O
      
    X 
O X   
on move:  X
  O   
    X 
O X   
on move:  O
X O   
    X 
O X   
on move:  X
X O   
    X 
O X O 
on move:  O
X O   
X   X 
O X O 
on move:  X
X O   
X O X 
O X O 
on move:  O
X O X 
X O X 
O X O 
Episode 62, Total Reward: 0
Average Reward: 0.1774193548387097
on move:  O
X     
      
      
on move:  X
X     
    O 
      
on move:  O
X     
    O 
  X   
on move:  X
X     
    O 
O X   
on move:  O
X     
    O 
O X X 
on move:  X
X     
O   O 
O X X 
on move:  O
X X   
O   O 
O X X 
on move:  X
X X   
O O O 
O X X 
Episode 63, Total Reward: -1
Average Reward: 0.15873015873015872
on move:  O
      
      
X     
on move:  X
O     
      
X     
on move:  O
O     
X     
X     
on move:  X
O     
X   O 
X     
on move:  O
O   X 
X   O 
X     
on move:  X
O O X 
X   O 
X     
on move:  O
O O X 
X X O 
X     
Episode 64, Total Reward: 1
Average Reward: 0.171875
on move:  O
      
      
  X   
on move:  X
    O 
      
  X   
on move:  O
    O 
  X   
  X   
on move:  X
    O 
  X   
O X   
on move:  O
  X O 
  X   
O X   
Episode 65, Total Reward: 1
Average Reward: 0.18461538461538463
on move:  O
X     
      
      
on move:  X
X     
  O   
      
on move:  O
X     
  O   
  X   
on move:  X
X     
  O O 
  X   
on move:  O
X X   
  O O 
  X   
on move:  X
X X   
  O O 
  X O 
on move:  O
X X X 
  O O 
  X O 
Episode 66, Total Reward: 1
Average Reward: 0.19696969696969696
on move:  O
      
    X 
      
on move:  X
      
    X 
O     
on move:  O
      
  X X 
O     
on move:  X
    O 
  X X 
O     
on move:  O
  X O 
  X X 
O     
on move:  X
  X O 
O X X 
O     
on move:  O
  X O 
O X X 
O   X 
on move:  X
O X O 
O X X 
O   X 
Episode 67, Total Reward: -1
Average Reward: 0.1791044776119403
on move:  O
      
      
  X   
on move:  X
  O   
      
  X   
on move:  O
  O   
    X 
  X   
on move:  X
  O   
O   X 
  X   
on move:  O
  O X 
O   X 
  X   
on move:  X
  O X 
O   X 
O X   
on move:  O
  O X 
O X X 
O X   
on move:  X
  O X 
O X X 
O X O 
on move:  O
X O X 
O X X 
O X O 
Episode 68, Total Reward: 0
Average Reward: 0.17647058823529413
on move:  O
      
      
    X 
on move:  X
      
      
O   X 
on move:  O
      
  X   
O   X 
on move:  X
      
  X   
O O X 
on move:  O
  X   
  X   
O O X 
on move:  X
O X   
  X   
O O X 
on move:  O
O X   
  X X 
O O X 
on move:  X
O X O 
  X X 
O O X 
on move:  O
O X O 
X X X 
O O X 
Episode 69, Total Reward: 1
Average Reward: 0.18840579710144928
on move:  O
      
      
    X 
on move:  X
    O 
      
    X 
on move:  O
    O 
X     
    X 
on move:  X
  O O 
X     
    X 
on move:  O
  O O 
X   X 
    X 
on move:  X
  O O 
X   X 
  O X 
on move:  O
  O O 
X   X 
X O X 
on move:  X
  O O 
X O X 
X O X 
Episode 70, Total Reward: -1
Average Reward: 0.17142857142857143
on move:  O
      
      
X     
on move:  X
      
  O   
X     
on move:  O
X     
  O   
X     
on move:  X
X     
  O   
X O   
on move:  O
X     
  O X 
X O   
on move:  X
X O   
  O X 
X O   
Episode 71, Total Reward: -1
Average Reward: 0.15492957746478872
on move:  O
X     
      
      
on move:  X
X     
    O 
      
on move:  O
X     
X   O 
      
on move:  X
X O   
X   O 
      
on move:  O
X O   
X   O 
    X 
on move:  X
X O   
X   O 
  O X 
on move:  O
X O   
X X O 
  O X 
Episode 72, Total Reward: 1
Average Reward: 0.16666666666666666
on move:  O
      
  X   
      
on move:  X
      
  X O 
      
on move:  O
      
  X O 
  X   
on move:  X
    O 
  X O 
  X   
on move:  O
    O 
  X O 
  X X 
on move:  X
    O 
O X O 
  X X 
on move:  O
X   O 
O X O 
  X X 
Episode 73, Total Reward: 1
Average Reward: 0.1780821917808219
on move:  O
      
X     
      
on move:  X
      
X     
    O 
on move:  O
  X   
X     
    O 
on move:  X
  X   
X O   
    O 
on move:  O
  X   
X O X 
    O 
on move:  X
  X   
X O X 
O   O 
on move:  O
X X   
X O X 
O   O 
on move:  X
X X O 
X O X 
O   O 
Episode 74, Total Reward: -1
Average Reward: 0.16216216216216217
on move:  O
      
      
  X   
on move:  X
      
      
O X   
on move:  O
      
      
O X X 
on move:  X
    O 
      
O X X 
on move:  O
  X O 
      
O X X 
on move:  X
  X O 
O     
O X X 
on move:  O
  X O 
O X   
O X X 
Episode 75, Total Reward: 1
Average Reward: 0.17333333333333334
on move:  O
      
      
X     
on move:  X
O     
      
X     
on move:  O
O     
      
X X   
on move:  X
O     
    O 
X X   
on move:  O
O     
    O 
X X X 
Episode 76, Total Reward: 1
Average Reward: 0.18421052631578946
on move:  O
      
      
    X 
on move:  X
      
O     
    X 
on move:  O
      
O     
  X X 
on move:  X
  O   
O     
  X X 
on move:  O
  O   
O X   
  X X 
on move:  X
  O   
O X   
O X X 
on move:  O
  O   
O X X 
O X X 
on move:  X
  O O 
O X X 
O X X 
on move:  O
X O O 
O X X 
O X X 
Episode 77, Total Reward: 1
Average Reward: 0.19480519480519481
on move:  O
  X   
      
      
on move:  X
  X   
      
O     
on move:  O
  X   
      
O X   
on move:  X
  X   
    O 
O X   
on move:  O
X X   
    O 
O X   
on move:  X
X X   
O   O 
O X   
on move:  O
X X   
O   O 
O X X 
on move:  X
X X   
O O O 
O X X 
Episode 78, Total Reward: -1
Average Reward: 0.1794871794871795
on move:  O
      
  X   
      
on move:  X
  O   
  X   
      
on move:  O
  O X 
  X   
      
on move:  X
  O X 
  X   
    O 
on move:  O
X O X 
  X   
    O 
on move:  X
X O X 
  X   
  O O 
on move:  O
X O X 
X X   
  O O 
on move:  X
X O X 
X X O 
  O O 
on move:  O
X O X 
X X O 
X O O 
Episode 79, Total Reward: 1
Average Reward: 0.189873417721519
on move:  O
      
      
  X   
on move:  X
    O 
      
  X   
on move:  O
    O 
      
  X X 
on move:  X
O   O 
      
  X X 
on move:  O
O   O 
X     
  X X 
on move:  X
O O O 
X     
  X X 
Episode 80, Total Reward: -1
Average Reward: 0.175
on move:  O
      
      
X     
on move:  X
O     
      
X     
on move:  O
O     
    X 
X     
on move:  X
O     
  O X 
X     
on move:  O
O     
  O X 
X   X 
on move:  X
O     
  O X 
X O X 
on move:  O
O   X 
  O X 
X O X 
Episode 81, Total Reward: 1
Average Reward: 0.18518518518518517
on move:  O
      
      
X     
on move:  X
      
      
X O   
on move:  O
      
  X   
X O   
on move:  X
O     
  X   
X O   
on move:  O
O   X 
  X   
X O   
Episode 82, Total Reward: 1
Average Reward: 0.1951219512195122
on move:  O
      
    X 
      
on move:  X
      
    X 
  O   
on move:  O
      
    X 
  O X 
on move:  X
      
  O X 
  O X 
on move:  O
  X   
  O X 
  O X 
on move:  X
  X   
O O X 
  O X 
on move:  O
  X   
O O X 
X O X 
on move:  X
  X O 
O O X 
X O X 
on move:  O
X X O 
O O X 
X O X 
Episode 83, Total Reward: 0
Average Reward: 0.1927710843373494
on move:  O
      
  X   
      
on move:  X
      
O X   
      
on move:  O
    X 
O X   
      
on move:  X
    X 
O X   
    O 
on move:  O
  X X 
O X   
    O 
on move:  X
  X X 
O X   
  O O 
on move:  O
X X X 
O X   
  O O 
Episode 84, Total Reward: 1
Average Reward: 0.20238095238095238
on move:  O
      
      
X     
on move:  X
  O   
      
X     
on move:  O
  O   
      
X X   
on move:  X
O O   
      
X X   
on move:  O
O O   
X     
X X   
on move:  X
O O   
X     
X X O 
on move:  O
O O X 
X     
X X O 
on move:  X
O O X 
X O   
X X O 
Episode 85, Total Reward: -1
Average Reward: 0.18823529411764706
on move:  O
    X 
      
      
on move:  X
  O X 
      
      
on move:  O
  O X 
      
    X 
on move:  X
  O X 
  O   
    X 
on move:  O
  O X 
  O   
  X X 
on move:  X
  O X 
  O O 
  X X 
on move:  O
X O X 
  O O 
  X X 
on move:  X
X O X 
O O O 
  X X 
Episode 86, Total Reward: -1
Average Reward: 0.1744186046511628
on move:  O
  X   
      
      
on move:  X
  X   
  O   
      
on move:  O
  X X 
  O   
      
on move:  X
  X X 
  O O 
      
on move:  O
X X X 
  O O 
      
Episode 87, Total Reward: 1
Average Reward: 0.1839080459770115
on move:  O
      
X     
      
on move:  X
      
X     
  O   
on move:  O
    X 
X     
  O   
on move:  X
    X 
X     
  O O 
on move:  O
    X 
X     
X O O 
on move:  X
    X 
X   O 
X O O 
on move:  O
  X X 
X   O 
X O O 
on move:  X
  X X 
X O O 
X O O 
on move:  O
X X X 
X O O 
X O O 
Episode 88, Total Reward: 1
Average Reward: 0.19318181818181818
on move:  O
      
      
  X   
on move:  X
O     
      
  X   
on move:  O
O X   
      
  X   
on move:  X
O X   
O     
  X   
on move:  O
O X X 
O     
  X   
on move:  X
O X X 
O O   
  X   
on move:  O
O X X 
O O   
X X   
on move:  X
O X X 
O O O 
X X   
Episode 89, Total Reward: -1
Average Reward: 0.1797752808988764
on move:  O
      
      
  X   
on move:  X
  O   
      
  X   
on move:  O
  O   
      
X X   
on move:  X
  O O 
      
X X   
on move:  O
  O O 
    X 
X X   
on move:  X
  O O 
  O X 
X X   
on move:  O
  O O 
  O X 
X X X 
Episode 90, Total Reward: 1
Average Reward: 0.18888888888888888
on move:  O
      
    X 
      
on move:  X
      
O   X 
      
on move:  O
X     
O   X 
      
on move:  X
X     
O O X 
      
on move:  O
X     
O O X 
X     
on move:  X
X     
O O X 
X   O 
on move:  O
X X   
O O X 
X   O 
on move:  X
X X O 
O O X 
X   O 
on move:  O
X X O 
O O X 
X X O 
Episode 91, Total Reward: 0
Average Reward: 0.18681318681318682
on move:  O
      
      
X     
on move:  X
    O 
      
X     
on move:  O
    O 
      
X X   
on move:  X
    O 
      
X X O 
on move:  O
    O 
X     
X X O 
on move:  X
O   O 
X     
X X O 
on move:  O
O X O 
X     
X X O 
on move:  X
O X O 
X   O 
X X O 
Episode 92, Total Reward: -1
Average Reward: 0.17391304347826086
on move:  O
      
      
  X   
on move:  X
      
      
  X O 
on move:  O
X     
      
  X O 
on move:  X
X     
O     
  X O 
on move:  O
X   X 
O     
  X O 
on move:  X
X O X 
O     
  X O 
on move:  O
X O X 
O     
X X O 
on move:  X
X O X 
O O   
X X O 
on move:  O
X O X 
O O X 
X X O 
Episode 93, Total Reward: 0
Average Reward: 0.17204301075268819
on move:  O
      
X     
      
on move:  X
O     
X     
      
on move:  O
O     
X     
    X 
on move:  X
O     
X     
  O X 
on move:  O
O     
X     
X O X 
on move:  X
O O   
X     
X O X 
on move:  O
O O   
X X   
X O X 
on move:  X
O O   
X X O 
X O X 
on move:  O
O O X 
X X O 
X O X 
Episode 94, Total Reward: 1
Average Reward: 0.18085106382978725
on move:  O
      
      
    X 
on move:  X
    O 
      
    X 
on move:  O
  X O 
      
    X 
on move:  X
  X O 
    O 
    X 
on move:  O
  X O 
  X O 
    X 
on move:  X
  X O 
  X O 
O   X 
on move:  O
  X O 
  X O 
O X X 
Episode 95, Total Reward: 1
Average Reward: 0.18947368421052632
on move:  O
      
  X   
      
on move:  X
      
  X   
  O   
on move:  O
      
  X   
  O X 
on move:  X
    O 
  X   
  O X 
on move:  O
  X O 
  X   
  O X 
on move:  X
O X O 
  X   
  O X 
on move:  O
O X O 
  X X 
  O X 
on move:  X
O X O 
  X X 
O O X 
on move:  O
O X O 
X X X 
O O X 
Episode 96, Total Reward: 1
Average Reward: 0.19791666666666666
on move:  O
      
      
  X   
on move:  X
  O   
      
  X   
on move:  O
  O   
X     
  X   
on move:  X
  O   
X O   
  X   
on move:  O
  O   
X O   
  X X 
on move:  X
  O   
X O O 
  X X 
on move:  O
X O   
X O O 
  X X 
on move:  X
X O   
X O O 
O X X 
on move:  O
X O X 
X O O 
O X X 
Episode 97, Total Reward: 0
Average Reward: 0.1958762886597938
on move:  O
    X 
      
      
on move:  X
    X 
O     
      
on move:  O
    X 
O   X 
      
on move:  X
  O X 
O   X 
      
on move:  O
  O X 
O   X 
  X   
on move:  X
O O X 
O   X 
  X   
on move:  O
O O X 
O   X 
  X X 
Episode 98, Total Reward: 1
Average Reward: 0.20408163265306123
on move:  O
      
      
    X 
on move:  X
    O 
      
    X 
on move:  O
    O 
X     
    X 
on move:  X
O   O 
X     
    X 
on move:  O
O   O 
X     
  X X 
on move:  X
O   O 
X     
O X X 
on move:  O
O X O 
X     
O X X 
on move:  X
O X O 
X   O 
O X X 
on move:  O
O X O 
X X O 
O X X 
Episode 99, Total Reward: 1
Average Reward: 0.21212121212121213
on move:  O
      
      
  X   
on move:  X
      
O     
  X   
on move:  O
      
O     
  X X 
on move:  X
      
O     
O X X 
on move:  O
      
O   X 
O X X 
on move:  X
    O 
O   X 
O X X 
on move:  O
X   O 
O   X 
O X X 
on move:  X
X   O 
O O X 
O X X 
Episode 100, Total Reward: -1
Average Reward: 0.2
on move:  O
      
    X 
      
on move:  X
    O 
    X 
      
on move:  O
X   O 
    X 
      
on move:  X
X   O 
  O X 
      
on move:  O
X   O 
  O X 
X     
on move:  X
X   O 
  O X 
X   O 
on move:  O
X   O 
  O X 
X X O 
on move:  X
X O O 
  O X 
X X O 
on move:  O
X O O 
X O X 
X X O 
Episode 101, Total Reward: 1
Average Reward: 0.2079207920792079
on move:  O
      
X     
      
on move:  X
      
X O   
      
on move:  O
    X 
X O   
      
on move:  X
    X 
X O O 
      
on move:  O
    X 
X O O 
    X 
on move:  X
    X 
X O O 
O   X 
on move:  O
  X X 
X O O 
O   X 
on move:  X
O X X 
X O O 
O   X 
on move:  O
O X X 
X O O 
O X X 
Episode 102, Total Reward: 0
Average Reward: 0.20588235294117646
on move:  O
      
  X   
      
on move:  X
O     
  X   
      
on move:  O
O   X 
  X   
      
on move:  X
O   X 
  X   
  O   
on move:  O
O   X 
  X   
  O X 
on move:  X
O   X 
  X   
O O X 
on move:  O
O   X 
  X X 
O O X 
Episode 103, Total Reward: 1
Average Reward: 0.21359223300970873
on move:  O
X     
      
      
on move:  X
X     
O     
      
on move:  O
X X   
O     
      
on move:  X
X X   
O   O 
      
on move:  O
X X X 
O   O 
      
Episode 104, Total Reward: 1
Average Reward: 0.22115384615384615
on move:  O
  X   
      
      
on move:  X
  X   
O     
      
on move:  O
  X   
O     
X     
on move:  X
  X   
O     
X O   
on move:  O
  X X 
O     
X O   
on move:  X
O X X 
O     
X O   
on move:  O
O X X 
O   X 
X O   
on move:  X
O X X 
O   X 
X O O 
on move:  O
O X X 
O X X 
X O O 
Episode 105, Total Reward: 1
Average Reward: 0.22857142857142856
on move:  O
      
    X 
      
on move:  X
      
O   X 
      
on move:  O
    X 
O   X 
      
on move:  X
    X 
O   X 
    O 
on move:  O
X   X 
O   X 
    O 
on move:  X
X O X 
O   X 
    O 
on move:  O
X O X 
O   X 
X   O 
on move:  X
X O X 
O O X 
X   O 
on move:  O
X O X 
O O X 
X X O 
Episode 106, Total Reward: 0
Average Reward: 0.22641509433962265
on move:  O
      
    X 
      
on move:  X
  O   
    X 
      
on move:  O
  O   
  X X 
      
on move:  X
  O   
  X X 
  O   
on move:  O
  O   
X X X 
  O   
Episode 107, Total Reward: 1
Average Reward: 0.2336448598130841
on move:  O
      
    X 
      
on move:  X
      
    X 
    O 
on move:  O
X     
    X 
    O 
on move:  X
X     
    X 
  O O 
on move:  O
X   X 
    X 
  O O 
on move:  X
X   X 
O   X 
  O O 
on move:  O
X X X 
O   X 
  O O 
Episode 108, Total Reward: 1
Average Reward: 0.24074074074074073
on move:  O
      
      
    X 
on move:  X
      
O     
    X 
on move:  O
  X   
O     
    X 
on move:  X
  X   
O O   
    X 
on move:  O
  X   
O O   
X   X 
on move:  X
  X   
O O O 
X   X 
Episode 109, Total Reward: -1
Average Reward: 0.22935779816513763
on move:  O
      
  X   
      
on move:  X
      
  X   
  O   
on move:  O
      
  X   
X O   
on move:  X
      
  X   
X O O 
on move:  O
      
  X X 
X O O 
on move:  X
  O   
  X X 
X O O 
on move:  O
  O X 
  X X 
X O O 
Episode 110, Total Reward: 1
Average Reward: 0.23636363636363636
on move:  O
    X 
      
      
on move:  X
    X 
  O   
      
on move:  O
    X 
  O   
    X 
on move:  X
    X 
  O O 
    X 
on move:  O
X   X 
  O O 
    X 
on move:  X
X   X 
  O O 
  O X 
on move:  O
X   X 
X O O 
  O X 
on move:  X
X O X 
X O O 
  O X 
Episode 111, Total Reward: -1
Average Reward: 0.22522522522522523
on move:  O
      
      
  X   
on move:  X
      
O     
  X   
on move:  O
      
O     
X X   
on move:  X
      
O   O 
X X   
on move:  O
      
O X O 
X X   
on move:  X
    O 
O X O 
X X   
on move:  O
X   O 
O X O 
X X   
on move:  X
X O O 
O X O 
X X   
on move:  O
X O O 
O X O 
X X X 
Episode 112, Total Reward: 1
Average Reward: 0.23214285714285715
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O   X 
      
X     
on move:  X
O   X 
O     
X     
on move:  O
O X X 
O     
X     
on move:  X
O X X 
O     
X O   
on move:  O
O X X 
O     
X O X 
on move:  X
O X X 
O   O 
X O X 
on move:  O
O X X 
O X O 
X O X 
Episode 113, Total Reward: 1
Average Reward: 0.23893805309734514
on move:  O
      
      
    X 
on move:  X
    O 
      
    X 
on move:  O
    O 
      
  X X 
on move:  X
    O 
    O 
  X X 
on move:  O
    O 
    O 
X X X 
Episode 114, Total Reward: 1
Average Reward: 0.24561403508771928
on move:  O
      
      
  X   
on move:  X
  O   
      
  X   
on move:  O
  O   
X     
  X   
on move:  X
  O   
X     
O X   
on move:  O
X O   
X     
O X   
on move:  X
X O   
X O   
O X   
on move:  O
X O   
X O X 
O X   
on move:  X
X O   
X O X 
O X O 
on move:  O
X O X 
X O X 
O X O 
Episode 115, Total Reward: 0
Average Reward: 0.24347826086956523
on move:  O
    X 
      
      
on move:  X
    X 
      
O     
on move:  O
  X X 
      
O     
on move:  X
O X X 
      
O     
on move:  O
O X X 
    X 
O     
on move:  X
O X X 
    X 
O O   
on move:  O
O X X 
    X 
O O X 
Episode 116, Total Reward: 1
Average Reward: 0.25
on move:  O
      
      
X     
on move:  X
      
  O   
X     
on move:  O
      
  O   
X   X 
on move:  X
      
  O   
X O X 
on move:  O
      
X O   
X O X 
on move:  X
O     
X O   
X O X 
on move:  O
O   X 
X O   
X O X 
on move:  X
O   X 
X O O 
X O X 
on move:  O
O X X 
X O O 
X O X 
Episode 117, Total Reward: 0
Average Reward: 0.24786324786324787
on move:  O
      
    X 
      
on move:  X
  O   
    X 
      
on move:  O
  O   
  X X 
      
on move:  X
  O   
  X X 
  O   
on move:  O
X O   
  X X 
  O   
on move:  X
X O   
  X X 
O O   
on move:  O
X O   
X X X 
O O   
Episode 118, Total Reward: 1
Average Reward: 0.2542372881355932
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O   X 
X     
      
on move:  X
O   X 
X     
    O 
on move:  O
O   X 
X   X 
    O 
on move:  X
O O X 
X   X 
    O 
on move:  O
O O X 
X X X 
    O 
Episode 119, Total Reward: 1
Average Reward: 0.2605042016806723
on move:  O
      
      
    X 
on move:  X
      
    O 
    X 
on move:  O
      
    O 
X   X 
on move:  X
  O   
    O 
X   X 
on move:  O
  O X 
    O 
X   X 
on move:  X
  O X 
    O 
X O X 
on move:  O
  O X 
X   O 
X O X 
on move:  X
  O X 
X O O 
X O X 
Episode 120, Total Reward: -1
Average Reward: 0.25
on move:  O
      
    X 
      
on move:  X
  O   
    X 
      
on move:  O
  O   
  X X 
      
on move:  X
O O   
  X X 
      
on move:  O
O O X 
  X X 
      
on move:  X
O O X 
  X X 
O     
on move:  O
O O X 
  X X 
O   X 
Episode 121, Total Reward: 1
Average Reward: 0.256198347107438
on move:  O
  X   
      
      
on move:  X
  X   
      
O     
on move:  O
X X   
      
O     
on move:  X
X X   
O     
O     
on move:  O
X X   
O     
O X   
on move:  X
X X O 
O     
O X   
on move:  O
X X O 
O     
O X X 
on move:  X
X X O 
O O   
O X X 
Episode 122, Total Reward: -1
Average Reward: 0.2459016393442623
on move:  O
      
      
X     
on move:  X
      
      
X   O 
on move:  O
      
      
X X O 
on move:  X
      
O     
X X O 
on move:  O
    X 
O     
X X O 
on move:  X
    X 
O O   
X X O 
on move:  O
  X X 
O O   
X X O 
on move:  X
  X X 
O O O 
X X O 
Episode 123, Total Reward: -1
Average Reward: 0.23577235772357724
on move:  O
      
      
    X 
on move:  X
O     
      
    X 
on move:  O
O   X 
      
    X 
on move:  X
O   X 
    O 
    X 
on move:  O
O   X 
    O 
  X X 
on move:  X
O O X 
    O 
  X X 
on move:  O
O O X 
X   O 
  X X 
on move:  X
O O X 
X O O 
  X X 
on move:  O
O O X 
X O O 
X X X 
Episode 124, Total Reward: 1
Average Reward: 0.24193548387096775
on move:  O
      
      
    X 
on move:  X
      
O     
    X 
on move:  O
    X 
O     
    X 
on move:  X
    X 
O     
O   X 
on move:  O
    X 
O X   
O   X 
on move:  X
    X 
O X O 
O   X 
on move:  O
  X X 
O X O 
O   X 
on move:  X
O X X 
O X O 
O   X 
Episode 125, Total Reward: -1
Average Reward: 0.232
on move:  O
      
    X 
      
on move:  X
      
O   X 
      
on move:  O
      
O   X 
X     
on move:  X
      
O O X 
X     
on move:  O
      
O O X 
X   X 
on move:  X
      
O O X 
X O X 
on move:  O
X     
O O X 
X O X 
on move:  X
X O   
O O X 
X O X 
Episode 126, Total Reward: -1
Average Reward: 0.2222222222222222
on move:  O
    X 
      
      
on move:  X
    X 
  O   
      
on move:  O
  X X 
  O   
      
on move:  X
  X X 
  O O 
      
on move:  O
  X X 
  O O 
  X   
on move:  X
  X X 
  O O 
O X   
on move:  O
  X X 
X O O 
O X   
on move:  X
  X X 
X O O 
O X O 
on move:  O
X X X 
X O O 
O X O 
Episode 127, Total Reward: 1
Average Reward: 0.2283464566929134
on move:  O
X     
      
      
on move:  X
X     
      
O     
on move:  O
X     
X     
O     
on move:  X
X     
X O   
O     
on move:  O
X     
X O X 
O     
on move:  X
X     
X O X 
O   O 
on move:  O
X   X 
X O X 
O   O 
on move:  X
X   X 
X O X 
O O O 
Episode 128, Total Reward: -1
Average Reward: 0.21875
on move:  O
      
  X   
      
on move:  X
    O 
  X   
      
on move:  O
    O 
  X X 
      
on move:  X
O   O 
  X X 
      
on move:  O
O   O 
  X X 
  X   
on move:  X
O O O 
  X X 
  X   
Episode 129, Total Reward: -1
Average Reward: 0.20930232558139536
on move:  O
    X 
      
      
on move:  X
    X 
    O 
      
on move:  O
    X 
    O 
X     
on move:  X
O   X 
    O 
X     
on move:  O
O   X 
    O 
X   X 
on move:  X
O   X 
  O O 
X   X 
on move:  O
O   X 
  O O 
X X X 
Episode 130, Total Reward: 1
Average Reward: 0.2153846153846154
on move:  O
    X 
      
      
on move:  X
    X 
      
    O 
on move:  O
    X 
      
X   O 
on move:  X
O   X 
      
X   O 
on move:  O
O   X 
      
X X O 
on move:  X
O   X 
  O   
X X O 
Episode 131, Total Reward: -1
Average Reward: 0.20610687022900764
on move:  O
      
    X 
      
on move:  X
      
O   X 
      
on move:  O
    X 
O   X 
      
on move:  X
  O X 
O   X 
      
on move:  O
  O X 
O   X 
    X 
Episode 132, Total Reward: 1
Average Reward: 0.21212121212121213
on move:  O
      
  X   
      
on move:  X
O     
  X   
      
on move:  O
O     
  X   
  X   
on move:  X
O     
  X O 
  X   
on move:  O
O     
  X O 
X X   
on move:  X
O   O 
  X O 
X X   
on move:  O
O   O 
  X O 
X X X 
Episode 133, Total Reward: 1
Average Reward: 0.21804511278195488
on move:  O
    X 
      
      
on move:  X
  O X 
      
      
on move:  O
X O X 
      
      
on move:  X
X O X 
    O 
      
on move:  O
X O X 
    O 
X     
on move:  X
X O X 
O   O 
X     
on move:  O
X O X 
O X O 
X     
Episode 134, Total Reward: 1
Average Reward: 0.22388059701492538
on move:  O
      
  X   
      
on move:  X
      
  X   
  O   
on move:  O
      
  X   
X O   
on move:  X
O     
  X   
X O   
on move:  O
O     
X X   
X O   
on move:  X
O   O 
X X   
X O   
on move:  O
O   O 
X X X 
X O   
Episode 135, Total Reward: 1
Average Reward: 0.22962962962962963
on move:  O
      
X     
      
on move:  X
      
X     
O     
on move:  O
    X 
X     
O     
on move:  X
O   X 
X     
O     
on move:  O
O   X 
X   X 
O     
on move:  X
O   X 
X   X 
O   O 
on move:  O
O   X 
X   X 
O X O 
on move:  X
O   X 
X O X 
O X O 
Episode 136, Total Reward: -1
Average Reward: 0.22058823529411764
on move:  O
      
      
    X 
on move:  X
O     
      
    X 
on move:  O
O     
  X   
    X 
on move:  X
O     
  X   
O   X 
on move:  O
O     
X X   
O   X 
on move:  X
O     
X X   
O O X 
on move:  O
O     
X X X 
O O X 
Episode 137, Total Reward: 1
Average Reward: 0.22627737226277372
on move:  O
      
X     
      
on move:  X
    O 
X     
      
on move:  O
X   O 
X     
      
on move:  X
X   O 
X     
O     
on move:  O
X   O 
X   X 
O     
on move:  X
X O O 
X   X 
O     
on move:  O
X O O 
X X X 
O     
Episode 138, Total Reward: 1
Average Reward: 0.2318840579710145
on move:  O
      
      
  X   
on move:  X
    O 
      
  X   
on move:  O
    O 
  X   
  X   
on move:  X
O   O 
  X   
  X   
on move:  O
O X O 
  X   
  X   
Episode 139, Total Reward: 1
Average Reward: 0.23741007194244604
on move:  O
      
      
  X   
on move:  X
      
      
O X   
on move:  O
  X   
      
O X   
on move:  X
  X   
  O   
O X   
on move:  O
  X   
X O   
O X   
on move:  X
  X O 
X O   
O X   
Episode 140, Total Reward: -1
Average Reward: 0.22857142857142856
on move:  O
      
      
    X 
on move:  X
    O 
      
    X 
on move:  O
  X O 
      
    X 
on move:  X
  X O 
      
  O X 
on move:  O
  X O 
X     
  O X 
on move:  X
  X O 
X   O 
  O X 
on move:  O
  X O 
X X O 
  O X 
on move:  X
  X O 
X X O 
O O X 
on move:  O
X X O 
X X O 
O O X 
Episode 141, Total Reward: 1
Average Reward: 0.23404255319148937
on move:  O
X     
      
      
on move:  X
X     
O     
      
on move:  O
X     
O X   
      
on move:  X
X     
O X   
  O   
on move:  O
X     
O X   
  O X 
Episode 142, Total Reward: 1
Average Reward: 0.23943661971830985
on move:  O
      
      
    X 
on move:  X
O     
      
    X 
on move:  O
O     
  X   
    X 
on move:  X
O     
  X   
  O X 
on move:  O
O     
X X   
  O X 
on move:  X
O   O 
X X   
  O X 
on move:  O
O X O 
X X   
  O X 
on move:  X
O X O 
X X O 
  O X 
on move:  O
O X O 
X X O 
X O X 
Episode 143, Total Reward: 0
Average Reward: 0.23776223776223776
on move:  O
    X 
      
      
on move:  X
    X 
      
  O   
on move:  O
    X 
      
X O   
on move:  X
O   X 
      
X O   
on move:  O
O   X 
  X   
X O   
Episode 144, Total Reward: 1
Average Reward: 0.24305555555555555
on move:  O
      
      
X     
on move:  X
  O   
      
X     
on move:  O
  O X 
      
X     
on move:  X
O O X 
      
X     
on move:  O
O O X 
      
X X   
on move:  X
O O X 
O     
X X   
on move:  O
O O X 
O     
X X X 
Episode 145, Total Reward: 1
Average Reward: 0.2482758620689655
on move:  O
X     
      
      
on move:  X
X   O 
      
      
on move:  O
X   O 
      
X     
on move:  X
X   O 
O     
X     
on move:  O
X   O 
O     
X   X 
on move:  X
X   O 
O O   
X   X 
on move:  O
X   O 
O O   
X X X 
Episode 146, Total Reward: 1
Average Reward: 0.2534246575342466
on move:  O
      
      
    X 
on move:  X
      
  O   
    X 
on move:  O
      
  O X 
    X 
on move:  X
      
O O X 
    X 
on move:  O
      
O O X 
X   X 
on move:  X
    O 
O O X 
X   X 
on move:  O
X   O 
O O X 
X   X 
on move:  X
X   O 
O O X 
X O X 
on move:  O
X X O 
O O X 
X O X 
Episode 147, Total Reward: 0
Average Reward: 0.25170068027210885
on move:  O
    X 
      
      
on move:  X
  O X 
      
      
on move:  O
X O X 
      
      
on move:  X
X O X 
      
    O 
on move:  O
X O X 
  X   
    O 
on move:  X
X O X 
  X   
O   O 
on move:  O
X O X 
X X   
O   O 
on move:  X
X O X 
X X O 
O   O 
on move:  O
X O X 
X X O 
O X O 
Episode 148, Total Reward: 0
Average Reward: 0.25
on move:  O
  X   
      
      
on move:  X
  X   
O     
      
on move:  O
  X   
O     
  X   
on move:  X
O X   
O     
  X   
on move:  O
O X   
O X   
  X   
Episode 149, Total Reward: 1
Average Reward: 0.2550335570469799
on move:  O
      
  X   
      
on move:  X
      
  X   
  O   
on move:  O
  X   
  X   
  O   
on move:  X
O X   
  X   
  O   
on move:  O
O X X 
  X   
  O   
on move:  X
O X X 
  X   
  O O 
on move:  O
O X X 
  X X 
  O O 
on move:  X
O X X 
O X X 
  O O 
on move:  O
O X X 
O X X 
X O O 
Episode 150, Total Reward: 1
Average Reward: 0.26