AIM-PIbd-31-Afanasev-S-S/lab6.ipynb at main

2024-12-07 00:40:57 +04:00

76 KiB

Raw Permalink Blame History

Начало лабораторной работы¶

Что необходимо сделать:¶

Развернуть и запустить проект по реализации обучения с подкреплением для игры "Крестики-нолики". Перевести проект на библиотеку gymnasium и современную версию Python. Реализовать агента для игры "Крестики-нолики" в виде отдельного класса (по примеру из лекции). Переписать основной цикл обучения для работы с отдельным классом агента (по примеру из лекции). Выполнить тестирование новой версии программы.

Наши крестики-нолики: https://github.com/nczempin/gym-tic-tac-toe

Перевод проекта на библиотеку gymnasium¶

Gymnasium — это открытая библиотека Python, которая предоставляет стандартизированные среды для разработки и тестирования алгоритмов обучения с подкреплением (Reinforcement Learning, RL). Ранее была известна как OpenAI Gym (до 2022 года), но теперь развивается под новым именем.

Библиотека позволяет разработчикам RL-агентов взаимодействовать с различными симуляциями — от простых игровых задач до сложных физических моделей. Gymnasium упрощает процесс тестирования и сравнения алгоритмов благодаря унифицированному интерфейсу.

Основные возможности Gymnasium:

Стандартизированные интерфейсы для взаимодействия со средой.
Простая интеграция с популярными библиотеками RL.
Поддержка пользовательских сред.

Основные функции Gymnasium:

Инициализация среды (gymnasium.make()): позволяет создавать экземпляр среды по её имени.
Сброс среды (reset()): возвращает начальное состояние среды и другую информацию.
Выполнение действия (step(action)): передаёт действие агенту и возвращает результат.
Закрытие среды (close()): очищает ресурсы, связанные со средой.
Режим рендеринга (render()): позволяет визуализировать работу среды.

In [2]:

import gymnasium as gym
from gymnasium import spaces

class TicTacToeEnv(gym.Env):
    metadata = {'render.modes': ['human']}
    
    symbols = ['O', ' ', 'X']

    def __init__(self):
        super().__init__()
        self.action_space = spaces.Discrete(9) # Дискретное пространство действий (0–8), соответствующее номерам клеток на игровом поле (от 0 до 8).
        self.observation_space = spaces.Discrete(9 * 3 * 2) # Дискретное пространство состояний. Расчёт: 9 клеток × 3 состояния (пустая, крестик, нолик) × 2 игрока (на чей ход).
        self.reset()

    def step(self, action):
        done = False
        reward = 0

        p, square = action  # p - игрок (1 или -1), square - номер клетки

        board = self.state['board']
        proposed = board[square] 
        om = self.state['on_move'] 
        if proposed != 0:  # Клетка уже занята
            print(f"Незаконный ход: Квадрат {square} уже занят.")
            done = True
            reward = -1 * om 
        if p != om:  # Не тот игрок на ходу
            print(f"Незаконный ход: игрок {p} не находится в движении")
            done = True
            reward = -1 * om
        else:
            board[square] = p
            self.state['on_move'] = -p

        for i in range(3):
            # Горизонтали и вертикали
            if (board[i * 3] == p and board[i * 3 + 1] == p and board[i * 3 + 2] == p) or \
               (board[i] == p and board[i + 3] == p and board[i + 6] == p):
                reward = p
                done = True
                break

        # Диагонали
        if (board[0] == p and board[4] == p and board[8] == p) or \
           (board[2] == p and board[4] == p and board[6] == p):
            reward = p
            done = True
                
        return self.state, reward, done, {}

    def reset(self):
        self.state = {}
        self.state['board'] = [0, 0, 0, 0, 0, 0, 0, 0, 0] 
        self.state['on_move'] = 1 
        return self.state, {}

    def render(self, close=False):
        if close:
            return
        print("on move: " , self.symbols[self.state['on_move']+1])
        for i in range (9):
            print (self.symbols[self.state['board'][i]+1], end=" ")
            if ((i % 3) == 2):
                print()

    def move_generator(self):
        moves = []
        for i in range(9):
            if self.state['board'][i] == 0:
                p = self.state['on_move']
                m = [p, i]
                moves.append(m)
        return moves

Реализация агента¶

В контексте обучения с подкреплением (Reinforcement Learning, RL) агент — это программный компонент, который взаимодействует с окружающей средой (environment) с целью научиться выбирать оптимальные действия для достижения своей цели.

Роль агента:¶

Агент принимает решение (выбирает действие), основываясь на текущем состоянии среды, и затем получает обратную связь (награду и новое состояние) от среды.

Функционал агента:¶

Выбор действия: Использует алгоритмы или стратегии, чтобы определить, что делать дальше.
Обучение: Обновляет свои знания или стратегию на основе опыта, чтобы лучше справляться с задачей.
Интерактивность: Адаптируется к изменениям в среде.

In [3]:

import random

class Agent:
    def __init__(self, symbol):
        self.symbol = symbol  # Символ игрока (1 - X, -1 - O)
    
    def get_action(self, moves):
        return random.choice(moves)  # Выбираем случайный ход из доступных

Основной цикл обучения¶

In [4]:

# Основной цикл обучения (работа с отдельным классом агента)

# Создание среды для игры в крестики-нолики
environment = TicTacToeEnv()

# Создание агента (играющего крестиками)
agent = Agent(symbol=1)

num_episodes = 100  # Количество эпизодов (игр) для обучения
collected_rewards = [] # Список для хранения наград/побед в каждом эпизоде 

# Переменная для отслеживания символа и текущего игрока
oom = 1

for i in range(num_episodes):
    # Сброс среды и начало нового эпизода
    state, _ = environment.reset() 

    # Общая награда за эпизод
    total_reward = 0

    # Флаг завершения игры
    done = False
    om = oom 

    # Максимум 9 ходов, поскольку поле 3x3 
    for j in range(9): 
        moves = environment.move_generator() 

        # Ходов нет, заканчиваем игру
        if not moves:
            break

        
        if len(moves) == 1:
            move = moves[0]    # Если остался один ход на основе стратегии
        else:
            move = agent.get_action(moves)   # Агент выбирает ход на основе стратегии

        # Выполнение хода и обновление состояния игры
        next_state, reward, done, info = environment.step(move)
        total_reward += reward
        state = next_state

        # Отображаем текущее состояние игры
        environment.render()

        if done:
            break

        om = -om    # Смена игрока

    collected_rewards.append(total_reward)

    print(f"Episode {i+1}, Total Reward: {total_reward}")
    average_reward = sum(collected_rewards) / len(collected_rewards)
    print(f"Average Reward: {average_reward}")

on move:  O
      
      
    X 
on move:  X
      
      
O   X 
on move:  O
      
      
O X X 
on move:  X
      
    O 
O X X 
on move:  O
      
  X O 
O X X 
on move:  X
    O 
  X O 
O X X 
on move:  O
  X O 
  X O 
O X X 
Episode 1, Total Reward: 1
Average Reward: 1.0
on move:  O
      
      
    X 
on move:  X
      
O     
    X 
on move:  O
      
O   X 
    X 
on move:  X
      
O O X 
    X 
on move:  O
  X   
O O X 
    X 
on move:  X
  X O 
O O X 
    X 
on move:  O
  X O 
O O X 
X   X 
on move:  X
O X O 
O O X 
X   X 
on move:  O
O X O 
O O X 
X X X 
Episode 2, Total Reward: 1
Average Reward: 1.0
on move:  O
      
      
  X   
on move:  X
      
    O 
  X   
on move:  O
  X   
    O 
  X   
on move:  X
  X   
  O O 
  X   
on move:  O
  X   
  O O 
X X   
on move:  X
  X   
  O O 
X X O 
on move:  O
  X X 
  O O 
X X O 
on move:  X
  X X 
O O O 
X X O 
Episode 3, Total Reward: -1
Average Reward: 0.3333333333333333
on move:  O
      
      
  X   
on move:  X
  O   
      
  X   
on move:  O
  O   
      
X X   
on move:  X
  O   
      
X X O 
on move:  O
  O X 
      
X X O 
on move:  X
  O X 
O     
X X O 
on move:  O
  O X 
O X   
X X O 
Episode 4, Total Reward: 1
Average Reward: 0.5
on move:  O
      
      
X     
on move:  X
      
      
X   O 
on move:  O
    X 
      
X   O 
on move:  X
  O X 
      
X   O 
on move:  O
  O X 
  X   
X   O 
Episode 5, Total Reward: 1
Average Reward: 0.6
on move:  O
X     
      
      
on move:  X
X O   
      
      
on move:  O
X O   
X     
      
on move:  X
X O   
X     
O     
on move:  O
X O   
X     
O X   
on move:  X
X O   
X     
O X O 
on move:  O
X O   
X   X 
O X O 
on move:  X
X O   
X O X 
O X O 
on move:  O
X O X 
X O X 
O X O 
Episode 6, Total Reward: 0
Average Reward: 0.5
on move:  O
      
      
X     
on move:  X
      
      
X   O 
on move:  O
      
    X 
X   O 
on move:  X
      
    X 
X O O 
on move:  O
X     
    X 
X O O 
on move:  X
X     
  O X 
X O O 
on move:  O
X   X 
  O X 
X O O 
on move:  X
X   X 
O O X 
X O O 
on move:  O
X X X 
O O X 
X O O 
Episode 7, Total Reward: 1
Average Reward: 0.5714285714285714
on move:  O
      
      
X     
on move:  X
      
      
X   O 
on move:  O
      
  X   
X   O 
on move:  X
      
  X O 
X   O 
on move:  O
X     
  X O 
X   O 
on move:  X
X O   
  X O 
X   O 
on move:  O
X O   
X X O 
X   O 
Episode 8, Total Reward: 1
Average Reward: 0.625
on move:  O
    X 
      
      
on move:  X
    X 
      
O     
on move:  O
    X 
    X 
O     
on move:  X
    X 
    X 
O   O 
on move:  O
    X 
    X 
O X O 
on move:  X
    X 
O   X 
O X O 
on move:  O
X   X 
O   X 
O X O 
on move:  X
X O X 
O   X 
O X O 
on move:  O
X O X 
O X X 
O X O 
Episode 9, Total Reward: 0
Average Reward: 0.5555555555555556
on move:  O
      
      
  X   
on move:  X
      
      
  X O 
on move:  O
X     
      
  X O 
on move:  X
X     
O     
  X O 
on move:  O
X X   
O     
  X O 
on move:  X
X X   
O   O 
  X O 
on move:  O
X X   
O X O 
  X O 
Episode 10, Total Reward: 1
Average Reward: 0.6
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O   X 
      
    X 
on move:  X
O O X 
      
    X 
on move:  O
O O X 
  X   
    X 
on move:  X
O O X 
  X   
  O X 
on move:  O
O O X 
  X   
X O X 
Episode 11, Total Reward: 1
Average Reward: 0.6363636363636364
on move:  O
      
    X 
      
on move:  X
  O   
    X 
      
on move:  O
  O   
  X X 
      
on move:  X
  O   
  X X 
O     
on move:  O
  O   
  X X 
O   X 
on move:  X
  O   
O X X 
O   X 
on move:  O
X O   
O X X 
O   X 
Episode 12, Total Reward: 1
Average Reward: 0.6666666666666666
on move:  O
      
    X 
      
on move:  X
      
    X 
O     
on move:  O
      
  X X 
O     
on move:  X
    O 
  X X 
O     
on move:  O
    O 
X X X 
O     
Episode 13, Total Reward: 1
Average Reward: 0.6923076923076923
on move:  O
      
X     
      
on move:  X
      
X   O 
      
on move:  O
      
X X O 
      
on move:  X
    O 
X X O 
      
on move:  O
    O 
X X O 
    X 
on move:  X
  O O 
X X O 
    X 
on move:  O
X O O 
X X O 
    X 
Episode 14, Total Reward: 1
Average Reward: 0.7142857142857143
on move:  O
      
    X 
      
on move:  X
O     
    X 
      
on move:  O
O     
    X 
  X   
on move:  X
O     
    X 
  X O 
on move:  O
O X   
    X 
  X O 
on move:  X
O X   
    X 
O X O 
on move:  O
O X   
X   X 
O X O 
on move:  X
O X   
X O X 
O X O 
Episode 15, Total Reward: -1
Average Reward: 0.6
on move:  O
      
      
  X   
on move:  X
      
O     
  X   
on move:  O
    X 
O     
  X   
on move:  X
    X 
O     
O X   
on move:  O
X   X 
O     
O X   
on move:  X
X   X 
O   O 
O X   
on move:  O
X   X 
O X O 
O X   
on move:  X
X O X 
O X O 
O X   
on move:  O
X O X 
O X O 
O X X 
Episode 16, Total Reward: 1
Average Reward: 0.625
on move:  O
X     
      
      
on move:  X
X     
  O   
      
on move:  O
X X   
  O   
      
on move:  X
X X   
  O   
O     
on move:  O
X X   
  O   
O X   
on move:  X
X X O 
  O   
O X   
Episode 17, Total Reward: -1
Average Reward: 0.5294117647058824
on move:  O
X     
      
      
on move:  X
X     
      
O     
on move:  O
X X   
      
O     
on move:  X
X X O 
      
O     
on move:  O
X X O 
      
O   X 
on move:  X
X X O 
  O   
O   X 
Episode 18, Total Reward: -1
Average Reward: 0.4444444444444444
on move:  O
      
      
    X 
on move:  X
      
  O   
    X 
on move:  O
      
X O   
    X 
on move:  X
O     
X O   
    X 
on move:  O
O     
X O X 
    X 
on move:  X
O     
X O X 
O   X 
on move:  O
O     
X O X 
O X X 
on move:  X
O   O 
X O X 
O X X 
Episode 19, Total Reward: -1
Average Reward: 0.3684210526315789
on move:  O
      
  X   
      
on move:  X
  O   
  X   
      
on move:  O
  O   
  X   
  X   
on move:  X
  O   
O X   
  X   
on move:  O
  O   
O X   
X X   
on move:  X
O O   
O X   
X X   
on move:  O
O O   
O X   
X X X 
Episode 20, Total Reward: 1
Average Reward: 0.4
on move:  O
      
  X   
      
on move:  X
O     
  X   
      
on move:  O
O     
X X   
      
on move:  X
O     
X X   
O     
on move:  O
O     
X X X 
O     
Episode 21, Total Reward: 1
Average Reward: 0.42857142857142855
on move:  O
      
  X   
      
on move:  X
      
  X   
  O   
on move:  O
X     
  X   
  O   
on move:  X
X     
O X   
  O   
on move:  O
X   X 
O X   
  O   
on move:  X
X   X 
O X O 
  O   
on move:  O
X X X 
O X O 
  O   
Episode 22, Total Reward: 1
Average Reward: 0.45454545454545453
on move:  O
      
      
  X   
on move:  X
    O 
      
  X   
on move:  O
    O 
      
  X X 
on move:  X
    O 
  O   
  X X 
on move:  O
    O 
X O   
  X X 
on move:  X
O   O 
X O   
  X X 
on move:  O
O X O 
X O   
  X X 
on move:  X
O X O 
X O   
O X X 
Episode 23, Total Reward: -1
Average Reward: 0.391304347826087
on move:  O
      
    X 
      
on move:  X
      
    X 
  O   
on move:  O
      
    X 
X O   
on move:  X
      
    X 
X O O 
on move:  O
      
  X X 
X O O 
on move:  X
  O   
  X X 
X O O 
on move:  O
  O   
X X X 
X O O 
Episode 24, Total Reward: 1
Average Reward: 0.4166666666666667
on move:  O
  X   
      
      
on move:  X
  X   
O     
      
on move:  O
X X   
O     
      
on move:  X
X X   
O     
  O   
on move:  O
X X   
O     
  O X 
on move:  X
X X   
O O   
  O X 
on move:  O
X X   
O O   
X O X 
on move:  X
X X O 
O O   
X O X 
on move:  O
X X O 
O O X 
X O X 
Episode 25, Total Reward: 0
Average Reward: 0.4
on move:  O
  X   
      
      
on move:  X
  X   
      
    O 
on move:  O
  X X 
      
    O 
on move:  X
  X X 
      
O   O 
on move:  O
  X X 
  X   
O   O 
on move:  X
O X X 
  X   
O   O 
on move:  O
O X X 
  X   
O X O 
Episode 26, Total Reward: 1
Average Reward: 0.4230769230769231
on move:  O
      
      
X     
on move:  X
      
O     
X     
on move:  O
X     
O     
X     
on move:  X
X O   
O     
X     
on move:  O
X O X 
O     
X     
on move:  X
X O X 
O     
X   O 
on move:  O
X O X 
O   X 
X   O 
on move:  X
X O X 
O   X 
X O O 
on move:  O
X O X 
O X X 
X O O 
Episode 27, Total Reward: 1
Average Reward: 0.4444444444444444
on move:  O
X     
      
      
on move:  X
X     
O     
      
on move:  O
X     
O X   
      
on move:  X
X     
O X   
    O 
on move:  O
X     
O X   
  X O 
on move:  X
X     
O X   
O X O 
on move:  O
X X   
O X   
O X O 
Episode 28, Total Reward: 1
Average Reward: 0.4642857142857143
on move:  O
      
      
  X   
on move:  X
      
  O   
  X   
on move:  O
    X 
  O   
  X   
on move:  X
    X 
  O   
O X   
on move:  O
  X X 
  O   
O X   
on move:  X
  X X 
  O   
O X O 
on move:  O
  X X 
  O X 
O X O 
on move:  X
  X X 
O O X 
O X O 
on move:  O
X X X 
O O X 
O X O 
Episode 29, Total Reward: 1
Average Reward: 0.4827586206896552
on move:  O
      
X     
      
on move:  X
      
X     
O     
on move:  O
    X 
X     
O     
on move:  X
    X 
X   O 
O     
on move:  O
X   X 
X   O 
O     
on move:  X
X   X 
X O O 
O     
on move:  O
X   X 
X O O 
O   X 
on move:  X
X   X 
X O O 
O O X 
on move:  O
X X X 
X O O 
O O X 
Episode 30, Total Reward: 1
Average Reward: 0.5
on move:  O
      
      
  X   
on move:  X
O     
      
  X   
on move:  O
O     
X     
  X   
on move:  X
O     
X O   
  X   
on move:  O
O     
X O X 
  X   
on move:  X
O     
X O X 
  X O 
Episode 31, Total Reward: -1
Average Reward: 0.45161290322580644
on move:  O
X     
      
      
on move:  X
X   O 
      
      
on move:  O
X   O 
      
  X   
on move:  X
X O O 
      
  X   
on move:  O
X O O 
    X 
  X   
on move:  X
X O O 
    X 
  X O 
on move:  O
X O O 
  X X 
  X O 
on move:  X
X O O 
  X X 
O X O 
on move:  O
X O O 
X X X 
O X O 
Episode 32, Total Reward: 1
Average Reward: 0.46875
on move:  O
      
      
  X   
on move:  X
    O 
      
  X   
on move:  O
    O 
      
X X   
on move:  X
    O 
  O   
X X   
on move:  O
    O 
  O   
X X X 
Episode 33, Total Reward: 1
Average Reward: 0.48484848484848486
on move:  O
  X   
      
      
on move:  X
  X   
      
O     
on move:  O
  X   
    X 
O     
on move:  X
  X O 
    X 
O     
on move:  O
  X O 
    X 
O   X 
on move:  X
  X O 
O   X 
O   X 
on move:  O
  X O 
O   X 
O X X 
on move:  X
  X O 
O O X 
O X X 
Episode 34, Total Reward: -1
Average Reward: 0.4411764705882353
on move:  O
X     
      
      
on move:  X
X     
  O   
      
on move:  O
X     
  O   
    X 
on move:  X
X O   
  O   
    X 
on move:  O
X O   
  O   
X   X 
on move:  X
X O   
  O O 
X   X 
on move:  O
X O X 
  O O 
X   X 
on move:  X
X O X 
O O O 
X   X 
Episode 35, Total Reward: -1
Average Reward: 0.4
on move:  O
    X 
      
      
on move:  X
    X 
      
  O   
on move:  O
    X 
  X   
  O   
on move:  X
    X 
  X   
  O O 
on move:  O
  X X 
  X   
  O O 
on move:  X
  X X 
  X   
O O O 
Episode 36, Total Reward: -1
Average Reward: 0.3611111111111111
on move:  O
      
      
    X 
on move:  X
      
    O 
    X 
on move:  O
      
X   O 
    X 
on move:  X
    O 
X   O 
    X 
on move:  O
X   O 
X   O 
    X 
on move:  X
X O O 
X   O 
    X 
on move:  O
X O O 
X   O 
X   X 
Episode 37, Total Reward: 1
Average Reward: 0.3783783783783784
on move:  O
X     
      
      
on move:  X
X     
      
O     
on move:  O
X     
X     
O     
on move:  X
X     
X   O 
O     
on move:  O
X     
X X O 
O     
on move:  X
X O   
X X O 
O     
on move:  O
X O X 
X X O 
O     
on move:  X
X O X 
X X O 
O   O 
on move:  O
X O X 
X X O 
O X O 
Episode 38, Total Reward: 0
Average Reward: 0.3684210526315789
on move:  O
    X 
      
      
on move:  X
    X 
    O 
      
on move:  O
X   X 
    O 
      
on move:  X
X   X 
    O 
O     
on move:  O
X X X 
    O 
O     
Episode 39, Total Reward: 1
Average Reward: 0.38461538461538464
on move:  O
X     
      
      
on move:  X
X     
      
    O 
on move:  O
X     
    X 
    O 
on move:  X
X     
    X 
  O O 
on move:  O
X   X 
    X 
  O O 
on move:  X
X   X 
  O X 
  O O 
on move:  O
X X X 
  O X 
  O O 
Episode 40, Total Reward: 1
Average Reward: 0.4
on move:  O
      
      
    X 
on move:  X
      
  O   
    X 
on move:  O
      
  O   
X   X 
on move:  X
  O   
  O   
X   X 
on move:  O
  O   
X O   
X   X 
on move:  X
O O   
X O   
X   X 
on move:  O
O O   
X O   
X X X 
Episode 41, Total Reward: 1
Average Reward: 0.4146341463414634
on move:  O
      
  X   
      
on move:  X
  O   
  X   
      
on move:  O
  O   
X X   
      
on move:  X
  O   
X X   
    O 
on move:  O
  O X 
X X   
    O 
on move:  X
  O X 
X X   
O   O 
on move:  O
  O X 
X X   
O X O 
on move:  X
  O X 
X X O 
O X O 
on move:  O
X O X 
X X O 
O X O 
Episode 42, Total Reward: 0
Average Reward: 0.40476190476190477
on move:  O
      
      
  X   
on move:  X
    O 
      
  X   
on move:  O
  X O 
      
  X   
on move:  X
  X O 
  O   
  X   
on move:  O
  X O 
  O X 
  X   
on move:  X
O X O 
  O X 
  X   
on move:  O
O X O 
  O X 
  X X 
on move:  X
O X O 
O O X 
  X X 
on move:  O
O X O 
O O X 
X X X 
Episode 43, Total Reward: 1
Average Reward: 0.4186046511627907
on move:  O
      
      
X     
on move:  X
      
      
X O   
on move:  O
      
      
X O X 
on move:  X
      
    O 
X O X 
on move:  O
  X   
    O 
X O X 
on move:  X
O X   
    O 
X O X 
on move:  O
O X   
X   O 
X O X 
on move:  X
O X O 
X   O 
X O X 
on move:  O
O X O 
X X O 
X O X 
Episode 44, Total Reward: 0
Average Reward: 0.4090909090909091
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O   X 
X     
      
on move:  X
O   X 
X     
  O   
on move:  O
O   X 
X   X 
  O   
on move:  X
O   X 
X   X 
O O   
on move:  O
O   X 
X X X 
O O   
Episode 45, Total Reward: 1
Average Reward: 0.4222222222222222
on move:  O
      
X     
      
on move:  X
      
X O   
      
on move:  O
      
X O   
    X 
on move:  X
      
X O O 
    X 
on move:  O
      
X O O 
  X X 
on move:  X
    O 
X O O 
  X X 
on move:  O
    O 
X O O 
X X X 
Episode 46, Total Reward: 1
Average Reward: 0.43478260869565216
on move:  O
      
      
X     
on move:  X
      
O     
X     
on move:  O
      
O   X 
X     
on move:  X
O     
O   X 
X     
on move:  O
O     
O   X 
X   X 
on move:  X
O O   
O   X 
X   X 
on move:  O
O O   
O   X 
X X X 
Episode 47, Total Reward: 1
Average Reward: 0.44680851063829785
on move:  O
      
      
  X   
on move:  X
O     
      
  X   
on move:  O
O X   
      
  X   
on move:  X
O X   
O     
  X   
on move:  O
O X X 
O     
  X   
on move:  X
O X X 
O     
O X   
Episode 48, Total Reward: -1
Average Reward: 0.4166666666666667
on move:  O
      
X     
      
on move:  X
O     
X     
      
on move:  O
O   X 
X     
      
on move:  X
O O X 
X     
      
on move:  O
O O X 
X     
  X   
on move:  X
O O X 
X   O 
  X   
on move:  O
O O X 
X   O 
  X X 
on move:  X
O O X 
X O O 
  X X 
on move:  O
O O X 
X O O 
X X X 
Episode 49, Total Reward: 1
Average Reward: 0.42857142857142855
on move:  O
  X   
      
      
on move:  X
  X   
      
O     
on move:  O
X X   
      
O     
on move:  X
X X   
    O 
O     
on move:  O
X X   
  X O 
O     
on move:  X
X X   
O X O 
O     
on move:  O
X X X 
O X O 
O     
Episode 50, Total Reward: 1
Average Reward: 0.44
on move:  O
      
      
X     
on move:  X
      
      
X O   
on move:  O
X     
      
X O   
on move:  X
X O   
      
X O   
on move:  O
X O   
      
X O X 
on move:  X
X O   
  O   
X O X 
Episode 51, Total Reward: -1
Average Reward: 0.4117647058823529
on move:  O
      
    X 
      
on move:  X
  O   
    X 
      
on move:  O
  O   
    X 
    X 
on move:  X
  O   
    X 
O   X 
on move:  O
  O   
X   X 
O   X 
on move:  X
  O   
X O X 
O   X 
on move:  O
  O X 
X O X 
O   X 
Episode 52, Total Reward: 1
Average Reward: 0.4230769230769231
on move:  O
      
      
  X   
on move:  X
      
O     
  X   
on move:  O
      
O     
  X X 
on move:  X
      
O   O 
  X X 
on move:  O
  X   
O   O 
  X X 
on move:  X
  X   
O O O 
  X X 
Episode 53, Total Reward: -1
Average Reward: 0.39622641509433965
on move:  O
X     
      
      
on move:  X
X O   
      
      
on move:  O
X O   
  X   
      
on move:  X
X O   
  X O 
      
on move:  O
X O   
  X O 
X     
on move:  X
X O   
O X O 
X     
on move:  O
X O X 
O X O 
X     
Episode 54, Total Reward: 1
Average Reward: 0.4074074074074074
on move:  O
      
      
X     
on move:  X
    O 
      
X     
on move:  O
X   O 
      
X     
on move:  X
X   O 
      
X   O 
on move:  O
X   O 
      
X X O 
on move:  X
X   O 
    O 
X X O 
Episode 55, Total Reward: -1
Average Reward: 0.38181818181818183
on move:  O
  X   
      
      
on move:  X
  X   
  O   
      
on move:  O
  X   
  O X 
      
on move:  X
  X   
O O X 
      
on move:  O
  X X 
O O X 
      
on move:  X
  X X 
O O X 
    O 
on move:  O
  X X 
O O X 
  X O 
on move:  X
  X X 
O O X 
O X O 
on move:  O
X X X 
O O X 
O X O 
Episode 56, Total Reward: 1
Average Reward: 0.39285714285714285
on move:  O
      
    X 
      
on move:  X
O     
    X 
      
on move:  O
O     
    X 
    X 
on move:  X
O O   
    X 
    X 
on move:  O
O O   
    X 
X   X 
on move:  X
O O   
O   X 
X   X 
on move:  O
O O X 
O   X 
X   X 
Episode 57, Total Reward: 1
Average Reward: 0.40350877192982454
on move:  O
      
X     
      
on move:  X
      
X     
    O 
on move:  O
    X 
X     
    O 
on move:  X
    X 
X     
O   O 
on move:  O
    X 
X     
O X O 
on move:  X
  O X 
X     
O X O 
on move:  O
X O X 
X     
O X O 
on move:  X
X O X 
X O   
O X O 
on move:  O
X O X 
X O X 
O X O 
Episode 58, Total Reward: 0
Average Reward: 0.39655172413793105
on move:  O
      
      
X     
on move:  X
      
O     
X     
on move:  O
  X   
O     
X     
on move:  X
  X   
O     
X   O 
on move:  O
  X   
O   X 
X   O 
on move:  X
  X   
O O X 
X   O 
on move:  O
X X   
O O X 
X   O 
on move:  X
X X   
O O X 
X O O 
on move:  O
X X X 
O O X 
X O O 
Episode 59, Total Reward: 1
Average Reward: 0.4067796610169492
on move:  O
      
      
  X   
on move:  X
      
  O   
  X   
on move:  O
  X   
  O   
  X   
on move:  X
O X   
  O   
  X   
on move:  O
O X   
  O   
  X X 
on move:  X
O X   
O O   
  X X 
on move:  O
O X   
O O X 
  X X 
on move:  X
O X   
O O X 
O X X 
Episode 60, Total Reward: -1
Average Reward: 0.38333333333333336
on move:  O
X     
      
      
on move:  X
X     
      
O     
on move:  O
X     
  X   
O     
on move:  X
X     
  X   
O O   
on move:  O
X     
  X X 
O O   
on move:  X
X     
O X X 
O O   
on move:  O
X X   
O X X 
O O   
on move:  X
X X O 
O X X 
O O   
on move:  O
X X O 
O X X 
O O X 
Episode 61, Total Reward: 1
Average Reward: 0.39344262295081966
on move:  O
X     
      
      
on move:  X
X     
O     
      
on move:  O
X     
O     
  X   
on move:  X
X     
O     
  X O 
on move:  O
X     
O X   
  X O 
on move:  X
X O   
O X   
  X O 
on move:  O
X O   
O X X 
  X O 
on move:  X
X O   
O X X 
O X O 
on move:  O
X O X 
O X X 
O X O 
Episode 62, Total Reward: 0
Average Reward: 0.3870967741935484
on move:  O
      
      
X     
on move:  X
    O 
      
X     
on move:  O
    O 
      
X X   
on move:  X
    O 
O     
X X   
on move:  O
    O 
O X   
X X   
on move:  X
    O 
O X O 
X X   
on move:  O
  X O 
O X O 
X X   
Episode 63, Total Reward: 1
Average Reward: 0.3968253968253968
on move:  O
      
    X 
      
on move:  X
O     
    X 
      
on move:  O
O     
    X 
X     
on move:  X
O     
  O X 
X     
on move:  O
O X   
  O X 
X     
on move:  X
O X O 
  O X 
X     
on move:  O
O X O 
  O X 
X   X 
on move:  X
O X O 
  O X 
X O X 
on move:  O
O X O 
X O X 
X O X 
Episode 64, Total Reward: 0
Average Reward: 0.390625
on move:  O
      
  X   
      
on move:  X
O     
  X   
      
on move:  O
O     
  X   
X     
on move:  X
O   O 
  X   
X     
on move:  O
O   O 
X X   
X     
on move:  X
O O O 
X X   
X     
Episode 65, Total Reward: -1
Average Reward: 0.36923076923076925
on move:  O
    X 
      
      
on move:  X
    X 
  O   
      
on move:  O
    X 
  O X 
      
on move:  X
  O X 
  O X 
      
on move:  O
  O X 
  O X 
    X 
Episode 66, Total Reward: 1
Average Reward: 0.3787878787878788
on move:  O
    X 
      
      
on move:  X
    X 
      
  O   
on move:  O
X   X 
      
  O   
on move:  X
X   X 
    O 
  O   
on move:  O
X X X 
    O 
  O   
Episode 67, Total Reward: 1
Average Reward: 0.3880597014925373
on move:  O
      
    X 
      
on move:  X
      
    X 
  O   
on move:  O
      
X   X 
  O   
on move:  X
    O 
X   X 
  O   
on move:  O
    O 
X   X 
X O   
on move:  X
    O 
X O X 
X O   
on move:  O
    O 
X O X 
X O X 
on move:  X
O   O 
X O X 
X O X 
on move:  O
O X O 
X O X 
X O X 
Episode 68, Total Reward: 0
Average Reward: 0.38235294117647056
on move:  O
      
    X 
      
on move:  X
O     
    X 
      
on move:  O
O     
    X 
  X   
on move:  X
O     
  O X 
  X   
on move:  O
O     
  O X 
X X   
on move:  X
O O   
  O X 
X X   
on move:  O
O O X 
  O X 
X X   
on move:  X
O O X 
O O X 
X X   
on move:  O
O O X 
O O X 
X X X 
Episode 69, Total Reward: 1
Average Reward: 0.391304347826087
on move:  O
  X   
      
      
on move:  X
  X O 
      
      
on move:  O
  X O 
  X   
      
on move:  X
  X O 
  X O 
      
on move:  O
  X O 
  X O 
  X   
Episode 70, Total Reward: 1
Average Reward: 0.4
on move:  O
      
  X   
      
on move:  X
      
  X O 
      
on move:  O
      
X X O 
      
on move:  X
      
X X O 
  O   
on move:  O
      
X X O 
X O   
on move:  X
    O 
X X O 
X O   
on move:  O
X   O 
X X O 
X O   
Episode 71, Total Reward: 1
Average Reward: 0.4084507042253521
on move:  O
      
  X   
      
on move:  X
      
O X   
      
on move:  O
      
O X   
  X   
on move:  X
      
O X O 
  X   
on move:  O
    X 
O X O 
  X   
on move:  X
  O X 
O X O 
  X   
on move:  O
X O X 
O X O 
  X   
on move:  X
X O X 
O X O 
O X   
on move:  O
X O X 
O X O 
O X X 
Episode 72, Total Reward: 1
Average Reward: 0.4166666666666667
on move:  O
      
      
  X   
on move:  X
  O   
      
  X   
on move:  O
  O X 
      
  X   
on move:  X
O O X 
      
  X   
on move:  O
O O X 
  X   
  X   
on move:  X
O O X 
  X   
O X   
on move:  O
O O X 
X X   
O X   
on move:  X
O O X 
X X O 
O X   
on move:  O
O O X 
X X O 
O X X 
Episode 73, Total Reward: 0
Average Reward: 0.410958904109589
on move:  O
X     
      
      
on move:  X
X O   
      
      
on move:  O
X O X 
      
      
on move:  X
X O X 
O     
      
on move:  O
X O X 
O     
    X 
on move:  X
X O X 
O     
  O X 
on move:  O
X O X 
O X   
  O X 
Episode 74, Total Reward: 1
Average Reward: 0.4189189189189189
on move:  O
      
      
    X 
on move:  X
      
      
  O X 
on move:  O
      
      
X O X 
on move:  X
    O 
      
X O X 
on move:  O
X   O 
      
X O X 
on move:  X
X   O 
  O   
X O X 
on move:  O
X   O 
X O   
X O X 
Episode 75, Total Reward: 1
Average Reward: 0.4266666666666667
on move:  O
      
      
X     
on move:  X
      
      
X   O 
on move:  O
      
  X   
X   O 
on move:  X
      
  X O 
X   O 
on move:  O
    X 
  X O 
X   O 
Episode 76, Total Reward: 1
Average Reward: 0.4342105263157895
on move:  O
    X 
      
      
on move:  X
    X 
O     
      
on move:  O
  X X 
O     
      
on move:  X
  X X 
O     
    O 
on move:  O
X X X 
O     
    O 
Episode 77, Total Reward: 1
Average Reward: 0.44155844155844154
on move:  O
      
      
X     
on move:  X
      
  O   
X     
on move:  O
  X   
  O   
X     
on move:  X
  X   
  O   
X O   
on move:  O
  X X 
  O   
X O   
on move:  X
  X X 
O O   
X O   
on move:  O
  X X 
O O X 
X O   
on move:  X
O X X 
O O X 
X O   
on move:  O
O X X 
O O X 
X O X 
Episode 78, Total Reward: 1
Average Reward: 0.44871794871794873
on move:  O
    X 
      
      
on move:  X
    X 
  O   
      
on move:  O
    X 
  O X 
      
on move:  X
  O X 
  O X 
      
on move:  O
  O X 
  O X 
    X 
Episode 79, Total Reward: 1
Average Reward: 0.45569620253164556
on move:  O
      
X     
      
on move:  X
    O 
X     
      
on move:  O
X   O 
X     
      
on move:  X
X   O 
X   O 
      
on move:  O
X X O 
X   O 
      
on move:  X
X X O 
X   O 
O     
on move:  O
X X O 
X   O 
O   X 
on move:  X
X X O 
X O O 
O   X 
Episode 80, Total Reward: -1
Average Reward: 0.4375
on move:  O
X     
      
      
on move:  X
X     
      
O     
on move:  O
X     
  X   
O     
on move:  X
X     
  X O 
O     
on move:  O
X     
X X O 
O     
on move:  X
X O   
X X O 
O     
on move:  O
X O   
X X O 
O   X 
Episode 81, Total Reward: 1
Average Reward: 0.4444444444444444
on move:  O
      
    X 
      
on move:  X
      
  O X 
      
on move:  O
      
  O X 
  X   
on move:  X
      
  O X 
O X   
on move:  O
      
X O X 
O X   
on move:  X
      
X O X 
O X O 
on move:  O
    X 
X O X 
O X O 
on move:  X
  O X 
X O X 
O X O 
on move:  O
X O X 
X O X 
O X O 
Episode 82, Total Reward: 0
Average Reward: 0.43902439024390244
on move:  O
      
    X 
      
on move:  X
      
    X 
    O 
on move:  O
      
X   X 
    O 
on move:  X
O     
X   X 
    O 
on move:  O
O     
X   X 
  X O 
on move:  X
O   O 
X   X 
  X O 
on move:  O
O   O 
X X X 
  X O 
Episode 83, Total Reward: 1
Average Reward: 0.4457831325301205
on move:  O
    X 
      
      
on move:  X
    X 
      
    O 
on move:  O
    X 
  X   
    O 
on move:  X
O   X 
  X   
    O 
on move:  O
O   X 
  X X 
    O 
on move:  X
O   X 
O X X 
    O 
on move:  O
O   X 
O X X 
  X O 
on move:  X
O O X 
O X X 
  X O 
on move:  O
O O X 
O X X 
X X O 
Episode 84, Total Reward: 1
Average Reward: 0.4523809523809524
on move:  O
      
    X 
      
on move:  X
      
    X 
  O   
on move:  O
  X   
    X 
  O   
on move:  X
  X   
    X 
  O O 
on move:  O
  X   
    X 
X O O 
on move:  X
  X   
O   X 
X O O 
on move:  O
  X   
O X X 
X O O 
on move:  X
  X O 
O X X 
X O O 
on move:  O
X X O 
O X X 
X O O 
Episode 85, Total Reward: 0
Average Reward: 0.4470588235294118
on move:  O
      
  X   
      
on move:  X
O     
  X   
      
on move:  O
O   X 
  X   
      
on move:  X
O O X 
  X   
      
on move:  O
O O X 
  X X 
      
on move:  X
O O X 
  X X 
  O   
on move:  O
O O X 
  X X 
X O   
Episode 86, Total Reward: 1
Average Reward: 0.45348837209302323
on move:  O
X     
      
      
on move:  X
X   O 
      
      
on move:  O
X   O 
  X   
      
on move:  X
X   O 
O X   
      
on move:  O
X   O 
O X X 
      
on move:  X
X   O 
O X X 
  O   
on move:  O
X   O 
O X X 
  O X 
Episode 87, Total Reward: 1
Average Reward: 0.45977011494252873
on move:  O
      
  X   
      
on move:  X
      
  X O 
      
on move:  O
  X   
  X O 
      
on move:  X
  X O 
  X O 
      
on move:  O
X X O 
  X O 
      
on move:  X
X X O 
  X O 
O     
on move:  O
X X O 
  X O 
O X   
Episode 88, Total Reward: 1
Average Reward: 0.4659090909090909
on move:  O
      
      
X     
on move:  X
      
      
X   O 
on move:  O
    X 
      
X   O 
on move:  X
    X 
      
X O O 
on move:  O
X   X 
      
X O O 
on move:  X
X   X 
    O 
X O O 
on move:  O
X X X 
    O 
X O O 
Episode 89, Total Reward: 1
Average Reward: 0.47191011235955055
on move:  O
    X 
      
      
on move:  X
    X 
      
  O   
on move:  O
    X 
      
X O   
on move:  X
    X 
  O   
X O   
on move:  O
X   X 
  O   
X O   
on move:  X
X O X 
  O   
X O   
Episode 90, Total Reward: -1
Average Reward: 0.45555555555555555
on move:  O
X     
      
      
on move:  X
X     
      
    O 
on move:  O
X     
  X   
    O 
on move:  X
X     
O X   
    O 
on move:  O
X   X 
O X   
    O 
on move:  X
X   X 
O X O 
    O 
on move:  O
X X X 
O X O 
    O 
Episode 91, Total Reward: 1
Average Reward: 0.46153846153846156
on move:  O
    X 
      
      
on move:  X
    X 
    O 
      
on move:  O
    X 
    O 
    X 
on move:  X
    X 
    O 
O   X 
on move:  O
  X X 
    O 
O   X 
on move:  X
  X X 
O   O 
O   X 
on move:  O
  X X 
O X O 
O   X 
on move:  X
  X X 
O X O 
O O X 
on move:  O
X X X 
O X O 
O O X 
Episode 92, Total Reward: 1
Average Reward: 0.4673913043478261
on move:  O
      
      
  X   
on move:  X
      
      
  X O 
on move:  O
  X   
      
  X O 
on move:  X
  X   
    O 
  X O 
on move:  O
  X   
  X O 
  X O 
Episode 93, Total Reward: 1
Average Reward: 0.4731182795698925
on move:  O
      
  X   
      
on move:  X
      
  X   
  O   
on move:  O
    X 
  X   
  O   
on move:  X
    X 
  X   
  O O 
on move:  O
X   X 
  X   
  O O 
on move:  X
X   X 
O X   
  O O 
on move:  O
X   X 
O X X 
  O O 
on move:  X
X   X 
O X X 
O O O 
Episode 94, Total Reward: -1
Average Reward: 0.4574468085106383
on move:  O
    X 
      
      
on move:  X
O   X 
      
      
on move:  O
O   X 
      
X     
on move:  X
O   X 
      
X O   
on move:  O
O   X 
  X   
X O   
Episode 95, Total Reward: 1
Average Reward: 0.4631578947368421
on move:  O
      
X     
      
on move:  X
      
X O   
      
on move:  O
      
X O   
X     
on move:  X
O     
X O   
X     
on move:  O
O     
X O   
X X   
on move:  X
O O   
X O   
X X   
on move:  O
O O   
X O X 
X X   
on move:  X
O O O 
X O X 
X X   
Episode 96, Total Reward: -1
Average Reward: 0.4479166666666667
on move:  O
      
  X   
      
on move:  X
      
  X   
    O 
on move:  O
      
  X   
  X O 
on move:  X
O     
  X   
  X O 
on move:  O
O     
  X   
X X O 
on move:  X
O     
O X   
X X O 
on move:  O
O X   
O X   
X X O 
Episode 97, Total Reward: 1
Average Reward: 0.4536082474226804
on move:  O
      
X     
      
on move:  X
      
X     
  O   
on move:  O
      
X     
X O   
on move:  X
      
X     
X O O 
on move:  O
      
X X   
X O O 
on move:  X
O     
X X   
X O O 
on move:  O
O   X 
X X   
X O O 
Episode 98, Total Reward: 1
Average Reward: 0.45918367346938777
on move:  O
      
      
  X   
on move:  X
      
      
  X O 
on move:  O
X     
      
  X O 
on move:  X
X O   
      
  X O 
on move:  O
X O   
    X 
  X O 
on move:  X
X O O 
    X 
  X O 
on move:  O
X O O 
X   X 
  X O 
on move:  X
X O O 
X   X 
O X O 
on move:  O
X O O 
X X X 
O X O 
Episode 99, Total Reward: 1
Average Reward: 0.46464646464646464
on move:  O
      
X     
      
on move:  X
      
X     
O     
on move:  O
      
X     
O   X 
on move:  X
O     
X     
O   X 
on move:  O
O     
X   X 
O   X 
on move:  X
O   O 
X   X 
O   X 
on move:  O
O   O 
X   X 
O X X 
on move:  X
O O O 
X   X 
O X X 
Episode 100, Total Reward: -1
Average Reward: 0.45

76 KiB Raw Permalink Blame History Unescape Escape