2438 lines
448 KiB
Plaintext
2438 lines
448 KiB
Plaintext
|
{
|
|||
|
|
"cells": [
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"## Датасет: [Tesla Insider Trading](https://www.kaggle.com/datasets/ilyaryabov/tesla-insider-trading).\n",
|
|||
|
|
"\n",
|
|||
|
|
"### Описание датасета:\n",
|
|||
|
|
"Датасет представляет собой выборку операций с ценными бумагами компании Tesla, совершённых инсайдерами, и является частью более крупного проекта \"Insider Trading S&P500 – Inside Info\". Данные охватывают транзакции с участием крупных акционеров и должностных лиц компании, включая такие операции, как покупка, продажа и опционы, начиная с 10 ноября 2021 года и до 27 июля 2022 года.\n",
|
|||
|
|
"\n",
|
|||
|
|
"---\n",
|
|||
|
|
"\n",
|
|||
|
|
"### Анализ сведений:\n",
|
|||
|
|
"**Проблемная область:**\n",
|
|||
|
|
"Проблемная область данного датасета касается анализа инсайдерских сделок в публичных компаниях, а также их влияния на ценообразование акций. Инсайдерские транзакции, совершаемые людьми с доступом к непубличной информации (такими как руководители, крупные акционеры или члены совета директоров), могут быть индикаторами будущих изменений стоимости акций. Исследование таких транзакций помогает понять, как информация внутри компании отражается в действиях ключевых участников, и может выявить паттерны поведения, которые влияют на рынки.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Актуальность:**\n",
|
|||
|
|
"Анализ инсайдерских сделок становится особенно важным в условиях высокой волатильности рынка и неопределенности. Инвесторы, аналитики и компании используют такие данные, чтобы лучше понимать сигналы от крупных акционеров и должностных лиц. Действия инсайдеров, такие как покупки и продажи акций, нередко рассматриваются как индикаторы доверия к компании, что может оказывать значительное влияние на рыночные ожидания и прогнозы.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Объекты наблюдений:**\n",
|
|||
|
|
"Объектами наблюдений в датасете являются инсайдеры компании Tesla — лица, имеющие значительное влияние на управление и информацию компании. Каждый объект характеризуется различными параметрами, включая должность, тип транзакции, количество акций и общую стоимость сделок.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Атрибуты объектов:**\n",
|
|||
|
|
"- Insider Trading: ФИО лица, совершившего транзакцию.\n",
|
|||
|
|
"- Relationship: Должность или статус данного лица в компании Tesla.\n",
|
|||
|
|
"- Date: Дата завершения транзакции.\n",
|
|||
|
|
"- Transaction: Тип транзакции.\n",
|
|||
|
|
"- Cost: Цена одной акции на момент совершения транзакции.\n",
|
|||
|
|
"- Shares: Количество акций, участвующих в транзакции.\n",
|
|||
|
|
"- Value ($): Общая стоимость транзакции в долларах США.\n",
|
|||
|
|
"- Shares Total: Общее количество акций, принадлежащих этому лицу после завершения данной транзакции.\n",
|
|||
|
|
"- SEC Form 4: Дата записи транзакции в форме SEC Form 4, обязательной для отчётности о сделках инсайдеров.\n",
|
|||
|
|
"\n",
|
|||
|
|
"---\n",
|
|||
|
|
"\n",
|
|||
|
|
"## Бизнес-цели:\n",
|
|||
|
|
"1. **Прогнозирование цен на акции Tesla на основе действий инсайдеров:**\n",
|
|||
|
|
"Одна из ключевых бизнес-целей состоит в создании модели для прогнозирования динамики акций Tesla, используя данные о транзакциях инсайдеров. Поскольку инсайдеры обладают глубоким знанием внутреннего состояния компании, их действия могут предсказывать изменения в стоимости акций. На основе анализа паттернов и частоты инсайдерских покупок и продаж можно разработать предсказательную модель, которая поможет инвесторам и аналитикам принимать более обоснованные решения.\n",
|
|||
|
|
"2. **Анализ влияния транзакций инсайдеров на динамику цены акций Tesla для оценки краткосрочных и долгосрочных рисков:**\n",
|
|||
|
|
"Цель – исследовать, как действия инсайдеров (особенно крупных акционеров и ключевых лиц) влияют на цену акций Tesla. Выявление корреляций между объёмом, типом и частотой инсайдерских сделок и изменениями цены акций позволит оценить риски и тенденции в динамике акций.\n",
|
|||
|
|
"---\n",
|
|||
|
|
"\n",
|
|||
|
|
"## Технические цели:\n",
|
|||
|
|
"**Для бизнес-цели №1:**\n",
|
|||
|
|
"- Сбор и подготовка данных:\n",
|
|||
|
|
" - Очистка данных от пропусков, выбросов и дубликатов, приведение данных к единому формату.\n",
|
|||
|
|
" - Преобразование категориальных переменных (тип транзакции, должность инсайдера и др.) в числовые значения с использованием кодирования (one-hot encoding).\n",
|
|||
|
|
" - Добавление временных характеристик, таких как сезонные факторы и тренды.\n",
|
|||
|
|
" - Разделение данных на обучающую и тестовую выборки для создания и тестирования прогнозной модели.\n",
|
|||
|
|
"- Разработка и обучение модели:\n",
|
|||
|
|
" - Исследование различных алгоритмов прогнозирования, таких как линейная регрессия, деревья решений, градиентный бустинг и рекуррентные нейронные сети (RNN).\n",
|
|||
|
|
" - Проведение кросс-валидации для подбора гиперпараметров и выбора наиболее эффективного алгоритма.\n",
|
|||
|
|
" - Обучение моделей на обучающей выборке и оценка качества на тестовой выборке с помощью метрик (RMSE, MAE и др.) для измерения точности предсказаний.\n",
|
|||
|
|
"- Развертывание и мониторинг модели:\n",
|
|||
|
|
" - Разработка и настройка дашборда для отслеживания прогноза стоимости акций в реальном времени на основе новых инсайдерских данных.\n",
|
|||
|
|
" - Настройка системы уведомлений и мониторинга для оповещения о значимых транзакциях инсайдеров.\n",
|
|||
|
|
" - Регулярное обновление модели и тестирование её точности по мере поступления новых данных.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Для бизнес-цели №2:**\n",
|
|||
|
|
"- Сбор и подготовка данных:\n",
|
|||
|
|
" - Очистка данных от пропусков и выбросов, приведение категориальных признаков (типы транзакций, роли инсайдеров и др.) к числовым форматам с помощью методов кодирования.\n",
|
|||
|
|
" - Интеграция с данными по историческим ценам акций Tesla, чтобы включить изменения цен в различные периоды после инсайдерских сделок.\n",
|
|||
|
|
" - Создание дополнительных признаков, например, накопительного объема акций и частоты транзакций для каждого инсайдера, чтобы выявить более глубокие взаимосвязи с ценами.\n",
|
|||
|
|
"- Разработка и обучение модели:\n",
|
|||
|
|
" - Использование методов временных рядов (ARIMA, Prophet) и регрессионных моделей для анализа корреляций между объёмом и типом инсайдерских транзакций и изменением цены акций.\n",
|
|||
|
|
" - Применение методов машинного обучения (случайный лес, градиентный бустинг) для определения ключевых факторов транзакций, наиболее значимых для изменений цен на акции.\n",
|
|||
|
|
" - Оценка моделей на тестовой выборке с использованием метрик (MAE, RMSE и др.) для точного измерения влияния и предсказательной способности.\n",
|
|||
|
|
"- Визуализация и мониторинг рисков:\n",
|
|||
|
|
" - Разработка интерактивного дашборда, отображающего связь между действиями инсайдеров и изменениями в цене акций Tesla с возможностью фильтрации по типу транзакций и роли инсайдера.\n",
|
|||
|
|
" - Настройка автоматических уведомлений о потенциальных рисках — например, при значительном росте объемов продажи акций инсайдерами и заметном снижении цены акций в краткосрочной перспективе.\n",
|
|||
|
|
" - Регулярное обновление и мониторинг модели для повышения точности прогнозов и своевременного обнаружения новых трендов.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Входные данные:**\n",
|
|||
|
|
"Инсайдер, должность, дата, тип транзакции, количество акций, общая стоимость сделки, общее количество акций у инсайдера, дата регистрации транзакции.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Целевой признак:**\n",
|
|||
|
|
"Cost – цена одной акции на момент транзакции.\n",
|
|||
|
|
"\n",
|
|||
|
|
"---"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Выгрузка данных из файла в DataFrame:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 639,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"from typing import Any\n",
|
|||
|
|
"from math import ceil\n",
|
|||
|
|
"import time\n",
|
|||
|
|
"\n",
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"from pandas import DataFrame, Series\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split, cross_val_score\n",
|
|||
|
|
"from sklearn.linear_model import LinearRegression\n",
|
|||
|
|
"from sklearn.metrics import root_mean_squared_error, r2_score, mean_absolute_error\n",
|
|||
|
|
"from sklearn.ensemble import RandomForestRegressor\n",
|
|||
|
|
"from imblearn.over_sampling import SMOTE\n",
|
|||
|
|
"import featuretools as ft\n",
|
|||
|
|
"from featuretools.entityset.entityset import EntitySet\n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"df: DataFrame = pd.read_csv('..//static//csv//TSLA.csv')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Краткая информация о DataFrame:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 640,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"<class 'pandas.core.frame.DataFrame'>\n",
|
|||
|
|
"RangeIndex: 156 entries, 0 to 155\n",
|
|||
|
|
"Data columns (total 9 columns):\n",
|
|||
|
|
" # Column Non-Null Count Dtype \n",
|
|||
|
|
"--- ------ -------------- ----- \n",
|
|||
|
|
" 0 Insider Trading 156 non-null object \n",
|
|||
|
|
" 1 Relationship 156 non-null object \n",
|
|||
|
|
" 2 Date 156 non-null object \n",
|
|||
|
|
" 3 Transaction 156 non-null object \n",
|
|||
|
|
" 4 Cost 156 non-null float64\n",
|
|||
|
|
" 5 Shares 156 non-null object \n",
|
|||
|
|
" 6 Value ($) 156 non-null object \n",
|
|||
|
|
" 7 Shares Total 156 non-null object \n",
|
|||
|
|
" 8 SEC Form 4 156 non-null object \n",
|
|||
|
|
"dtypes: float64(1), object(8)\n",
|
|||
|
|
"memory usage: 11.1+ KB\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/html": [
|
|||
|
|
"<div>\n",
|
|||
|
|
"<style scoped>\n",
|
|||
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|||
|
|
" vertical-align: middle;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe tbody tr th {\n",
|
|||
|
|
" vertical-align: top;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe thead th {\n",
|
|||
|
|
" text-align: right;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"</style>\n",
|
|||
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|||
|
|
" <thead>\n",
|
|||
|
|
" <tr style=\"text-align: right;\">\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th>count</th>\n",
|
|||
|
|
" <th>mean</th>\n",
|
|||
|
|
" <th>std</th>\n",
|
|||
|
|
" <th>min</th>\n",
|
|||
|
|
" <th>25%</th>\n",
|
|||
|
|
" <th>50%</th>\n",
|
|||
|
|
" <th>75%</th>\n",
|
|||
|
|
" <th>max</th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </thead>\n",
|
|||
|
|
" <tbody>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>Cost</th>\n",
|
|||
|
|
" <td>156.0</td>\n",
|
|||
|
|
" <td>478.785641</td>\n",
|
|||
|
|
" <td>448.922903</td>\n",
|
|||
|
|
" <td>0.0</td>\n",
|
|||
|
|
" <td>50.5225</td>\n",
|
|||
|
|
" <td>240.225</td>\n",
|
|||
|
|
" <td>934.1075</td>\n",
|
|||
|
|
" <td>1171.04</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </tbody>\n",
|
|||
|
|
"</table>\n",
|
|||
|
|
"</div>"
|
|||
|
|
],
|
|||
|
|
"text/plain": [
|
|||
|
|
" count mean std min 25% 50% 75% max\n",
|
|||
|
|
"Cost 156.0 478.785641 448.922903 0.0 50.5225 240.225 934.1075 1171.04"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 640,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Краткая информация о DataFrame\n",
|
|||
|
|
"df.info()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Статистическое описание числовых столбцов\n",
|
|||
|
|
"df.describe().transpose()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Конвертация данных:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 641,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Выборка данных:\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/html": [
|
|||
|
|
"<div>\n",
|
|||
|
|
"<style scoped>\n",
|
|||
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|||
|
|
" vertical-align: middle;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe tbody tr th {\n",
|
|||
|
|
" vertical-align: top;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe thead th {\n",
|
|||
|
|
" text-align: right;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"</style>\n",
|
|||
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|||
|
|
" <thead>\n",
|
|||
|
|
" <tr style=\"text-align: right;\">\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th>Insider Trading</th>\n",
|
|||
|
|
" <th>Relationship</th>\n",
|
|||
|
|
" <th>Date</th>\n",
|
|||
|
|
" <th>Transaction</th>\n",
|
|||
|
|
" <th>Cost</th>\n",
|
|||
|
|
" <th>Shares</th>\n",
|
|||
|
|
" <th>Value ($)</th>\n",
|
|||
|
|
" <th>Shares Total</th>\n",
|
|||
|
|
" <th>SEC Form 4</th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </thead>\n",
|
|||
|
|
" <tbody>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>0</th>\n",
|
|||
|
|
" <td>Kirkhorn Zachary</td>\n",
|
|||
|
|
" <td>Chief Financial Officer</td>\n",
|
|||
|
|
" <td>2022-03-06</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>196.72</td>\n",
|
|||
|
|
" <td>10455</td>\n",
|
|||
|
|
" <td>2056775</td>\n",
|
|||
|
|
" <td>203073</td>\n",
|
|||
|
|
" <td>Mar 07 07:58 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>1</th>\n",
|
|||
|
|
" <td>Taneja Vaibhav</td>\n",
|
|||
|
|
" <td>Chief Accounting Officer</td>\n",
|
|||
|
|
" <td>2022-03-06</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>195.79</td>\n",
|
|||
|
|
" <td>2466</td>\n",
|
|||
|
|
" <td>482718</td>\n",
|
|||
|
|
" <td>100458</td>\n",
|
|||
|
|
" <td>Mar 07 07:57 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>2</th>\n",
|
|||
|
|
" <td>Baglino Andrew D</td>\n",
|
|||
|
|
" <td>SVP Powertrain and Energy Eng.</td>\n",
|
|||
|
|
" <td>2022-03-06</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>195.79</td>\n",
|
|||
|
|
" <td>1298</td>\n",
|
|||
|
|
" <td>254232</td>\n",
|
|||
|
|
" <td>65547</td>\n",
|
|||
|
|
" <td>Mar 07 08:01 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>3</th>\n",
|
|||
|
|
" <td>Taneja Vaibhav</td>\n",
|
|||
|
|
" <td>Chief Accounting Officer</td>\n",
|
|||
|
|
" <td>2022-03-05</td>\n",
|
|||
|
|
" <td>Option Exercise</td>\n",
|
|||
|
|
" <td>0.00</td>\n",
|
|||
|
|
" <td>7138</td>\n",
|
|||
|
|
" <td>0</td>\n",
|
|||
|
|
" <td>102923</td>\n",
|
|||
|
|
" <td>Mar 07 07:57 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>4</th>\n",
|
|||
|
|
" <td>Baglino Andrew D</td>\n",
|
|||
|
|
" <td>SVP Powertrain and Energy Eng.</td>\n",
|
|||
|
|
" <td>2022-03-05</td>\n",
|
|||
|
|
" <td>Option Exercise</td>\n",
|
|||
|
|
" <td>0.00</td>\n",
|
|||
|
|
" <td>2586</td>\n",
|
|||
|
|
" <td>0</td>\n",
|
|||
|
|
" <td>66845</td>\n",
|
|||
|
|
" <td>Mar 07 08:01 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>5</th>\n",
|
|||
|
|
" <td>Kirkhorn Zachary</td>\n",
|
|||
|
|
" <td>Chief Financial Officer</td>\n",
|
|||
|
|
" <td>2022-03-05</td>\n",
|
|||
|
|
" <td>Option Exercise</td>\n",
|
|||
|
|
" <td>0.00</td>\n",
|
|||
|
|
" <td>16867</td>\n",
|
|||
|
|
" <td>0</td>\n",
|
|||
|
|
" <td>213528</td>\n",
|
|||
|
|
" <td>Mar 07 07:58 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>6</th>\n",
|
|||
|
|
" <td>Baglino Andrew D</td>\n",
|
|||
|
|
" <td>SVP Powertrain and Energy Eng.</td>\n",
|
|||
|
|
" <td>2022-02-27</td>\n",
|
|||
|
|
" <td>Option Exercise</td>\n",
|
|||
|
|
" <td>20.91</td>\n",
|
|||
|
|
" <td>10500</td>\n",
|
|||
|
|
" <td>219555</td>\n",
|
|||
|
|
" <td>74759</td>\n",
|
|||
|
|
" <td>Mar 01 07:29 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>7</th>\n",
|
|||
|
|
" <td>Baglino Andrew D</td>\n",
|
|||
|
|
" <td>SVP Powertrain and Energy Eng.</td>\n",
|
|||
|
|
" <td>2022-02-27</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>202.00</td>\n",
|
|||
|
|
" <td>10500</td>\n",
|
|||
|
|
" <td>2121000</td>\n",
|
|||
|
|
" <td>64259</td>\n",
|
|||
|
|
" <td>Mar 01 07:29 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>8</th>\n",
|
|||
|
|
" <td>Kirkhorn Zachary</td>\n",
|
|||
|
|
" <td>Chief Financial Officer</td>\n",
|
|||
|
|
" <td>2022-02-06</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>193.00</td>\n",
|
|||
|
|
" <td>3750</td>\n",
|
|||
|
|
" <td>723750</td>\n",
|
|||
|
|
" <td>196661</td>\n",
|
|||
|
|
" <td>Feb 08 06:14 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>9</th>\n",
|
|||
|
|
" <td>Baglino Andrew D</td>\n",
|
|||
|
|
" <td>SVP Powertrain and Energy Eng.</td>\n",
|
|||
|
|
" <td>2022-01-27</td>\n",
|
|||
|
|
" <td>Option Exercise</td>\n",
|
|||
|
|
" <td>20.91</td>\n",
|
|||
|
|
" <td>10500</td>\n",
|
|||
|
|
" <td>219555</td>\n",
|
|||
|
|
" <td>74759</td>\n",
|
|||
|
|
" <td>Jan 31 07:34 PM</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </tbody>\n",
|
|||
|
|
"</table>\n",
|
|||
|
|
"</div>"
|
|||
|
|
],
|
|||
|
|
"text/plain": [
|
|||
|
|
" Insider Trading Relationship Date \\\n",
|
|||
|
|
"0 Kirkhorn Zachary Chief Financial Officer 2022-03-06 \n",
|
|||
|
|
"1 Taneja Vaibhav Chief Accounting Officer 2022-03-06 \n",
|
|||
|
|
"2 Baglino Andrew D SVP Powertrain and Energy Eng. 2022-03-06 \n",
|
|||
|
|
"3 Taneja Vaibhav Chief Accounting Officer 2022-03-05 \n",
|
|||
|
|
"4 Baglino Andrew D SVP Powertrain and Energy Eng. 2022-03-05 \n",
|
|||
|
|
"5 Kirkhorn Zachary Chief Financial Officer 2022-03-05 \n",
|
|||
|
|
"6 Baglino Andrew D SVP Powertrain and Energy Eng. 2022-02-27 \n",
|
|||
|
|
"7 Baglino Andrew D SVP Powertrain and Energy Eng. 2022-02-27 \n",
|
|||
|
|
"8 Kirkhorn Zachary Chief Financial Officer 2022-02-06 \n",
|
|||
|
|
"9 Baglino Andrew D SVP Powertrain and Energy Eng. 2022-01-27 \n",
|
|||
|
|
"\n",
|
|||
|
|
" Transaction Cost Shares Value ($) Shares Total SEC Form 4 \n",
|
|||
|
|
"0 Sale 196.72 10455 2056775 203073 Mar 07 07:58 PM \n",
|
|||
|
|
"1 Sale 195.79 2466 482718 100458 Mar 07 07:57 PM \n",
|
|||
|
|
"2 Sale 195.79 1298 254232 65547 Mar 07 08:01 PM \n",
|
|||
|
|
"3 Option Exercise 0.00 7138 0 102923 Mar 07 07:57 PM \n",
|
|||
|
|
"4 Option Exercise 0.00 2586 0 66845 Mar 07 08:01 PM \n",
|
|||
|
|
"5 Option Exercise 0.00 16867 0 213528 Mar 07 07:58 PM \n",
|
|||
|
|
"6 Option Exercise 20.91 10500 219555 74759 Mar 01 07:29 PM \n",
|
|||
|
|
"7 Sale 202.00 10500 2121000 64259 Mar 01 07:29 PM \n",
|
|||
|
|
"8 Sale 193.00 3750 723750 196661 Feb 08 06:14 PM \n",
|
|||
|
|
"9 Option Exercise 20.91 10500 219555 74759 Jan 31 07:34 PM "
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 641,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Преобразование типов данных\n",
|
|||
|
|
"df['Insider Trading'] = df['Insider Trading'].astype('category') # Преобразование в категорию\n",
|
|||
|
|
"df['Relationship'] = df['Relationship'].astype('category') # Преобразование в категорию\n",
|
|||
|
|
"df['Transaction'] = df['Transaction'].astype('category') # Преобразование в категорию\n",
|
|||
|
|
"df['Cost'] = pd.to_numeric(df['Cost'], errors='coerce') # Преобразование в float\n",
|
|||
|
|
"df['Shares'] = pd.to_numeric(df['Shares'].str.replace(',', ''), errors='coerce') # Преобразование в float с удалением запятых\n",
|
|||
|
|
"df['Value ($)'] = pd.to_numeric(df['Value ($)'].str.replace(',', ''), errors='coerce') # Преобразование в float с удалением запятых\n",
|
|||
|
|
"df['Shares Total'] = pd.to_numeric(df['Shares Total'].str.replace(',', ''), errors='coerce') # Преобразование в float с удалением запятых\n",
|
|||
|
|
"\n",
|
|||
|
|
"print('Выборка данных:')\n",
|
|||
|
|
"df.head(10)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Проблема пропущенных данных:\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Проблема пропущенных данных** — это отсутствие значений в наборе данных, что может искажать результаты анализа и статистические выводы.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Проверка на отсутствие значений, представленная ниже, показала, что DataFrame не имеет пустых значений признаков. Нет необходимости использовать методы заполнения пропущенных данных."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 642,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Присутствуют ли пустые значения признаков в колонке:\n",
|
|||
|
|
"Insider Trading False\n",
|
|||
|
|
"Relationship False\n",
|
|||
|
|
"Date False\n",
|
|||
|
|
"Transaction False\n",
|
|||
|
|
"Cost False\n",
|
|||
|
|
"Shares False\n",
|
|||
|
|
"Value ($) False\n",
|
|||
|
|
"Shares Total False\n",
|
|||
|
|
"SEC Form 4 False\n",
|
|||
|
|
"dtype: bool \n",
|
|||
|
|
"\n",
|
|||
|
|
"Количество пустых значений признаков в колонке:\n",
|
|||
|
|
"Insider Trading 0\n",
|
|||
|
|
"Relationship 0\n",
|
|||
|
|
"Date 0\n",
|
|||
|
|
"Transaction 0\n",
|
|||
|
|
"Cost 0\n",
|
|||
|
|
"Shares 0\n",
|
|||
|
|
"Value ($) 0\n",
|
|||
|
|
"Shares Total 0\n",
|
|||
|
|
"SEC Form 4 0\n",
|
|||
|
|
"dtype: int64 \n",
|
|||
|
|
"\n",
|
|||
|
|
"Процент пустых значений признаков в колонке:\n",
|
|||
|
|
"\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Проверка пропущенных данных\n",
|
|||
|
|
"def check_null_columns(dataframe: DataFrame) -> None:\n",
|
|||
|
|
" # Присутствуют ли пустые значения признаков\n",
|
|||
|
|
" print('Присутствуют ли пустые значения признаков в колонке:')\n",
|
|||
|
|
" print(dataframe.isnull().any(), '\\n')\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Количество пустых значений признаков\n",
|
|||
|
|
" print('Количество пустых значений признаков в колонке:')\n",
|
|||
|
|
" print(dataframe.isnull().sum(), '\\n')\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Процент пустых значений признаков\n",
|
|||
|
|
" print('Процент пустых значений признаков в колонке:')\n",
|
|||
|
|
" for column in dataframe.columns:\n",
|
|||
|
|
" null_rate: float = dataframe[column].isnull().sum() / len(dataframe) * 100\n",
|
|||
|
|
" if null_rate > 0:\n",
|
|||
|
|
" print(f\"{column} процент пустых значений: {null_rate:.2f}%\")\n",
|
|||
|
|
" print()\n",
|
|||
|
|
" \n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка пропущенных данных\n",
|
|||
|
|
"check_null_columns(df)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Проблема зашумленности данных:\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Зашумленность** – это наличие случайных ошибок или вариаций в данных, которые могут затруднить выявление истинных закономерностей. Шум может возникать из-за ошибок измерений, неправильных записей или других факторов.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Выбросы** – это значения, которые значительно отличаются от остальных наблюдений в наборе данных. Выбросы могут указывать на ошибки в данных или на редкие, но важные события. Их наличие может повлиять на статистические методы анализа.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Представленный ниже код помогает определить наличие выбросов в наборе данных и устранить их (при наличии), заменив значения ниже нижней границы (рассматриваемого минимума) на значения нижней границы, а значения выше верхней границы (рассматриваемого максимума) – на значения верхней границы."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 643,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Проверка наличия выбросов в колонках:\n",
|
|||
|
|
"Колонка Cost:\n",
|
|||
|
|
"\tЕсть выбросы: Нет\n",
|
|||
|
|
"\tКоличество выбросов: 0\n",
|
|||
|
|
"\tМинимальное значение: 0.0\n",
|
|||
|
|
"\tМаксимальное значение: 1171.04\n",
|
|||
|
|
"\t1-й квартиль (Q1): 50.5225\n",
|
|||
|
|
"\t3-й квартиль (Q3): 934.1075\n",
|
|||
|
|
"\n",
|
|||
|
|
"Колонка Shares:\n",
|
|||
|
|
"\tЕсть выбросы: Да\n",
|
|||
|
|
"\tКоличество выбросов: 25\n",
|
|||
|
|
"\tМинимальное значение: 121\n",
|
|||
|
|
"\tМаксимальное значение: 11920000\n",
|
|||
|
|
"\t1-й квартиль (Q1): 3500.0\n",
|
|||
|
|
"\t3-й квартиль (Q3): 301797.75\n",
|
|||
|
|
"\n",
|
|||
|
|
"Колонка Value ($):\n",
|
|||
|
|
"\tЕсть выбросы: Да\n",
|
|||
|
|
"\tКоличество выбросов: 23\n",
|
|||
|
|
"\tМинимальное значение: 0\n",
|
|||
|
|
"\tМаксимальное значение: 2278695421\n",
|
|||
|
|
"\t1-й квартиль (Q1): 271008.0\n",
|
|||
|
|
"\t3-й квартиль (Q3): 148713213.25\n",
|
|||
|
|
"\n",
|
|||
|
|
"Колонка Shares Total:\n",
|
|||
|
|
"\tЕсть выбросы: Да\n",
|
|||
|
|
"\tКоличество выбросов: 21\n",
|
|||
|
|
"\tМинимальное значение: 49\n",
|
|||
|
|
"\tМаксимальное значение: 455467432\n",
|
|||
|
|
"\t1-й квартиль (Q1): 25103.5\n",
|
|||
|
|
"\t3-й квартиль (Q3): 1507273.75\n",
|
|||
|
|
"\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABdEAAAPOCAYAAADgBVF+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADQjElEQVR4nOzdeVyUVfvH8S8MCSqBKwilQlq5gFr6JFBTkJSaGIS0aWVm2aI9ubVg5VbJo2HaYpmWS0+iFhEVlmUqOdVkhVliahtkJWBmApqCzty/P3xmfo4wLoUMwuf9es2r5pzrnvuaceAw15z7HC/DMAwBAAAAAAAAAIAqvD2dAAAAAAAAAAAAdRVFdAAAAAAAAAAA3KCIDgAAAAAAAACAGxTRAQAAAAAAAABwgyI6AAAAAAAAAABuUEQHAAAAAAAAAMANiugAAAAAAAAAALhBER0AAAAAAAAAADcoogMAAAAAAAAA4AZFdAAAAACoRbfeeqv8/f09nQYAADiG2NhYxcbGeuz8n3/+uRo1aqSff/652v5bb71Vubm51fZ9++238vHxUX5+/inMsGGhiI4TlpmZKS8vr2pvERERnk4PaDBiY2N16623Sjo8aFY3qB84cECzZs1S7969FRgYKD8/P5133nkaNWqUvvvuu1OSV0ZGhmbPnl2lvbCwUF5eXs7B3cvLS4sWLTolOQD1FWMwUDccbwzeu3evJk2apIiICDVt2lQtW7ZUjx49dN9992nHjh21n3ANW7Rokby8vCRJubm58vLyUmFhoWeTAmoB4zBQN7gbh7OysuTl5aWXXnrJ7bGrVq2Sl5eXnnnmmVrItGY8/PDDuvHGG9W+ffuTPrZLly4aMGCAJk6cWKXvyNdu8uTJCgsL+4eZNgw+nk4Ap58JEyaoc+fOzvtPPPGEB7MBcLRdu3apX79+ysvLU0JCggYPHix/f39t27ZNy5Yt07x581RZWVnj583IyFB+fr5Gjx5d448N4DDGYKDuOnjwoC699FJt3bpVQ4cO1b333qu9e/dq8+bNysjI0DXXXKPQ0FBPpwngH2AcBuqmAQMGKDAwUBkZGbr99turjcnIyJDJZNINN9xQy9n9PRs3btSHH36oTz/91G3MoUOHVFFR4bb/rrvu0lVXXaUff/xRHTp0OBVpNigU0XHSrrjiCpdZNy+99JJ27drluYQAuLj11lv11VdfKTMzU4MGDXLpe+yxx/Twww97KDMA/xRjMFB3ZWdn66uvvtKSJUs0ePBgl74DBw6cki+wj8Vut6uyslJ+fn61el6gPmMcBuomX19fpaSkaOHChdqxY0eVL60PHDigN998U1dccYWCgoI8lOXJWbhwodq1a6eoqCiX9tLSUo0ePVqZmZnau3evlixZIn9/f8XExGj58uVq1qyZMzY+Pl7NmzfX4sWLNXXq1Fp+BvUPy7nghDn+8Pf2Pv7bxnGp55GXd9rtdnXr1q3KUg7ffPONbr31Vp1zzjny8/NTmzZtdNttt+mPP/5weczJkydXe/mcj8//fxcUGxuriIgI5eXlKSYmRo0bN1Z4eLjmzp1b5blMnDhRPXv2VGBgoJo2bSqz2ay1a9e6xDmWofDy8lJ2drZL34EDB9S8eXN5eXkpPT29Sp5BQUE6ePCgyzFLly51Pt6Rf2y99dZbGjBggEJDQ+Xr66sOHTrosccek81mO+5r7Tjf1q1bdd111ykgIEAtW7bUfffdpwMHDrjELly4UJdffrmCgoLk6+urLl266IUXXqjymImJiQoLC5Ofn5+CgoJ09dVXa9OmTS4xjudR3fIdnTp1kpeXl0aNGuVs2717t8aPH6/IyEj5+/srICBA/fv319dff+1y7NChQ+Xn56ctW7a4tPft21fNmzd3uRT6p59+0rXXXqsWLVqoSZMmioqK0ooVK1yOc1xq7Lj5+vrqvPPOU1pamgzDOPaL+z/u3nvVLaNy5Hvm6NuRdu7cqeHDh6tdu3YymUzOmH+6Pur69eu1YsUKDR8+vEoBXTr8x8WR71dJWrNmjcxms5o2bapmzZopMTGxyutfXl6u0aNHKywsTL6+vgoKCtIVV1yhDRs2SDr8s7dixQr9/PPPzufCJWFAzWEMznbpYwxmDK6LY/CPP/4oSbr44our9Pn5+SkgIKBK+2+//aakpCT5+/urdevWGj9+fJX3Xnp6umJiYtSyZUs1btxYPXv2VGZmZpXHcvy7L1myRF27dpWvr69WrlzpPM9tt92m4OBg+fr6qmvXrlqwYEGVx3j22WfVtWtXNWnSRM2bN1evXr2UkZHxt14PoD5hHM526WMcZhyui+PwTTfdJLvdrmXLllXpW7FihUpLSzVkyBBJJ/5+OFp1P9/S/7/WR69Pvn79evXr10+BgYFq0qSJLrvsMn3yyScn9Hyys7N1+eWXV3n97rvvPi1ZskRjx47VFVdcoalTp2ry5Mnau3ev9u3b5xJ7xhlnKDY2Vm+99dYJnRPHxkx0nDDHHw6+vr5/6/j//ve/VQYf6fC6VD/99JOGDRumNm3aaPPmzZo3b542b96szz77rMovjBdeeMHll+vRf8j8+eefuuqqq3Tdddfpxhtv1Guvvaa7775bjRo10m233SZJKisr00svvaQbb7xRd9xxh8rLy/Xyyy+rb9+++vzzz9WjRw+Xx/Tz89PChQuVlJTkbMvKyqoyMB+pvLxcOTk5uuaaa5xtCxculJ+fX5XjFi1aJH9/f40dO1b+/v5as2aNJk6cqLKyMj355JNuz3Gk6667TmFhYUpLS9Nnn32mZ555Rn/++adeeeUVl9eua9euuvrqq+Xj46N33nlH99xzj+x2u0aOHOnyeCNGjFCbNm20Y8cOPffcc4qPj1dBQYGaNGlS5XU5cvmOTz/9tNpNL3766SdlZ2fr2muvVXh4uEpKSvTiiy/qsssu07fffuv8pvjpp5/WmjVrNHToUFmtVplMJr344ov64IMP9N///tcZV1JSopiYGP3111/697//rZYtW2rx4sW6+uqrlZmZ6fK6S/9/6eX+/fu1fPlyTZgwQUFBQRo+fPgJvb6O18/x3ktNTT1m7IgRI2Q2myUdfq+8+eabLv1Dhw7Vhx9+qHvvvVfdu3eXyWTSvHnznEXpv+vtt9+WJN18880nFP/hhx+qf//+OuecczR58mTt379fzz77rC6++GJt2LDBWQi/6667lJmZqVGjRqlLly76448/9PHHH2vLli268MIL9fDDD6u0tFS//vqrZs2aJUlsmAbUIMZgxmDG4Lo/BjvWK33llVf0yCOPVPn5OZrNZlPfvn3Vu3dvpaen68MPP9TMmTPVoUMH3X333c64p59+WldffbWGDBmiyspKLVu2TNdee61ycnI0YMAAl8dcs2aNXnvtNY0aNUqtWrVSWFiYSkpKFBUV5SzqtG7dWu+9956GDx+usrIy53to/vz5+ve//62UlBRnAeqbb77R+vXrq8ysBxoaxmHGYcbhuj8OX3rppTr77LOVkZGhsWPHuvRlZGSoSZMmzvfxybwf/q41a9aof//+6tmzpyZNmiRvb29n8d5iseiiiy5ye+xvv/2m7du368ILL6zSt2LFCt16662aMmWKbr31VpnNZsXGxmrcuHHVPlbPnj311ltvqaysrNov9HESDOAEzZ4925BkfP311y7tl112mdG1a1eXtoULFxqSjIKCAsMwDOPAgQNGu3btjP79+xuSjIULFzpj//rrryrnWrp0qSHJWLdunbNt0qRJhiTj999/d5vjZZddZkgyZs6c6WyrqKgwevToYQQFBRmVlZWGYRjGoUOHjIqKCpdj//zzTyM4ONi47bbbnG0FBQWGJOPGG280fHx8jOLiYmdfnz59jMGDBxuSjCeffLJKnjfeeKORkJDgbP/5558Nb29v48Ybb6zyPKp7De68806jSZMmxoEDB9w+3yPPd/XVV7u033PPPVX+vao7T9++fY1zzjnnmOd47bXXDEnGl19+6WyTZKSkpBg+Pj4u7cOHD3e+LiNHjnS2HzhwwLDZbC6PW1BQYPj6+hpTp051aX///fcNScbjjz9u/PTTT4a/v7+
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x1000 with 4 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Проверка выбросов в DataFrame\n",
|
|||
|
|
"def check_outliers(dataframe: DataFrame, columns: list[str]) -> None:\n",
|
|||
|
|
" for column in columns:\n",
|
|||
|
|
" if not pd.api.types.is_numeric_dtype(dataframe[column]): # Проверяем, является ли колонка числовой\n",
|
|||
|
|
" continue\n",
|
|||
|
|
" \n",
|
|||
|
|
" Q1: float = dataframe[column].quantile(0.25) # 1-й квартиль (25%)\n",
|
|||
|
|
" Q3: float = dataframe[column].quantile(0.75) # 3-й квартиль (75%)\n",
|
|||
|
|
" IQR: float = Q3 - Q1 # Вычисляем межквартильный размах\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Определяем границы для выбросов\n",
|
|||
|
|
" lower_bound: float = Q1 - 1.5 * IQR # Нижняя граница\n",
|
|||
|
|
" upper_bound: float = Q3 + 1.5 * IQR # Верхняя граница\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Подсчитываем количество выбросов\n",
|
|||
|
|
" outliers: DataFrame = dataframe[(dataframe[column] < lower_bound) | (dataframe[column] > upper_bound)]\n",
|
|||
|
|
" outlier_count: int = outliers.shape[0]\n",
|
|||
|
|
"\n",
|
|||
|
|
" print(f\"Колонка {column}:\")\n",
|
|||
|
|
" print(f\"\\tЕсть выбросы: {'Да' if outlier_count > 0 else 'Нет'}\")\n",
|
|||
|
|
" print(f\"\\tКоличество выбросов: {outlier_count}\")\n",
|
|||
|
|
" print(f\"\\tМинимальное значение: {dataframe[column].min()}\")\n",
|
|||
|
|
" print(f\"\\tМаксимальное значение: {dataframe[column].max()}\")\n",
|
|||
|
|
" print(f\"\\t1-й квартиль (Q1): {Q1}\")\n",
|
|||
|
|
" print(f\"\\t3-й квартиль (Q3): {Q3}\\n\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Визуализация выбросов\n",
|
|||
|
|
"def visualize_outliers(dataframe: DataFrame, columns: list[str]) -> None:\n",
|
|||
|
|
" # Диаграммы размахов\n",
|
|||
|
|
" plt.figure(figsize=(15, 10))\n",
|
|||
|
|
" rows: int = ceil(len(columns) / 3)\n",
|
|||
|
|
" for index, column in enumerate(columns, 1):\n",
|
|||
|
|
" plt.subplot(rows, 3, index)\n",
|
|||
|
|
" plt.boxplot(dataframe[column], vert=True, patch_artist=True)\n",
|
|||
|
|
" plt.title(f\"Диаграмма размахов для \\\"{column}\\\"\")\n",
|
|||
|
|
" plt.xlabel(column)\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Отображение графиков\n",
|
|||
|
|
" plt.tight_layout()\n",
|
|||
|
|
" plt.show()\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Числовые столбцы DataFrame\n",
|
|||
|
|
"numeric_columns: list[str] = [\n",
|
|||
|
|
" 'Cost',\n",
|
|||
|
|
" 'Shares',\n",
|
|||
|
|
" 'Value ($)',\n",
|
|||
|
|
" 'Shares Total'\n",
|
|||
|
|
"]\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка наличия выбросов в колонках\n",
|
|||
|
|
"print('Проверка наличия выбросов в колонках:')\n",
|
|||
|
|
"check_outliers(df, numeric_columns)\n",
|
|||
|
|
"visualize_outliers(df, numeric_columns)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 644,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Проверка наличия выбросов в колонках после их устранения:\n",
|
|||
|
|
"Колонка Cost:\n",
|
|||
|
|
"\tЕсть выбросы: Нет\n",
|
|||
|
|
"\tКоличество выбросов: 0\n",
|
|||
|
|
"\tМинимальное значение: 0.0\n",
|
|||
|
|
"\tМаксимальное значение: 1171.04\n",
|
|||
|
|
"\t1-й квартиль (Q1): 50.5225\n",
|
|||
|
|
"\t3-й квартиль (Q3): 934.1075\n",
|
|||
|
|
"\n",
|
|||
|
|
"Колонка Shares:\n",
|
|||
|
|
"\tЕсть выбросы: Нет\n",
|
|||
|
|
"\tКоличество выбросов: 0\n",
|
|||
|
|
"\tМинимальное значение: 121.0\n",
|
|||
|
|
"\tМаксимальное значение: 749244.375\n",
|
|||
|
|
"\t1-й квартиль (Q1): 3500.0\n",
|
|||
|
|
"\t3-й квартиль (Q3): 301797.75\n",
|
|||
|
|
"\n",
|
|||
|
|
"Колонка Value ($):\n",
|
|||
|
|
"\tЕсть выбросы: Нет\n",
|
|||
|
|
"\tКоличество выбросов: 0\n",
|
|||
|
|
"\tМинимальное значение: 0.0\n",
|
|||
|
|
"\tМаксимальное значение: 371376521.125\n",
|
|||
|
|
"\t1-й квартиль (Q1): 271008.0\n",
|
|||
|
|
"\t3-й квартиль (Q3): 148713213.25\n",
|
|||
|
|
"\n",
|
|||
|
|
"Колонка Shares Total:\n",
|
|||
|
|
"\tЕсть выбросы: Нет\n",
|
|||
|
|
"\tКоличество выбросов: 0\n",
|
|||
|
|
"\tМинимальное значение: 49.0\n",
|
|||
|
|
"\tМаксимальное значение: 3730529.125\n",
|
|||
|
|
"\t1-й квартиль (Q1): 25103.5\n",
|
|||
|
|
"\t3-й квартиль (Q3): 1507273.75\n",
|
|||
|
|
"\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABdIAAAPOCAYAAAALMup9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADOqElEQVR4nOzdeVxWdfr/8TeIgIqAqICMiFSm4C6mormkJBqZli2aJi5pNWApkxaNqWHFZOWWpJkLNspotlhpoaipTeKGMbmPjSaW3pCpkKSAcH5/+ON8vQNu0ZBFX8/H4zzqPp/rnHPdtzdc3Nd9zufYGYZhCAAAAAAAAAAAFMu+ohMAAAAAAAAAAKAyo5EOAAAAAAAAAIANNNIBAAAAAAAAALCBRjoAAAAAAAAAADbQSAcAAAAAAAAAwAYa6QAAAAAAAAAA2EAjHQAAAAAAAAAAG2ikAwAAAAAAAABgA410AAAAAAAAAABsoJEOAAAAAOVo+PDhcnFxqeg0AABAGejRo4d69OhRYcffuXOnHB0ddfz48WLHhw8frs2bNxc7duDAATk4OGjfvn03MMObB4106KOPPpKdnV2xS4sWLSo6PeCW0aNHDw0fPlzS5UJXXCG+ePGiZs6cqY4dO8rNzU3Ozs668847FRkZqf/+9783JK+EhATNmjWryPoff/xRdnZ2ZkG2s7NTfHz8DckBuNVRq4HK4Wq1+vz585oyZYpatGihWrVqqW7dumrTpo2ee+45nTx5svwTLmPx8fGys7OTJG3evFl2dnb68ccfKzYpoBKhXgOVQ0n1+pNPPpGdnZ0WLlxY4rZJSUmys7PTnDlzyiHTsvH3v/9dgwcPlp+f3zVvGxgYqLCwME2ePLnI2JWv3dSpU9W4ceM/mWnV51DRCaDyeOmllxQQEGA+fu211yowGwB/dPr0afXp00cpKSm6//779fjjj8vFxUWHDx/WihUrtGDBAuXm5pb5cRMSErRv3z6NGzeuzPcN4NpQq4HKKy8vT926ddOhQ4cUHh6usWPH6vz589q/f78SEhL04IMPysfHp6LTBFAOqNdA5RQWFiY3NzclJCToySefLDYmISFB1apV06BBg8o5u+uTmpqqDRs2aNu2bSXGXLp0STk5OSWOP/3007rvvvv0v//9T7fffvuNSPOmQSMdpnvvvdfqrJqFCxfq9OnTFZcQACvDhw/Xd999p48++kgDBw60Gps2bZr+/ve/V1BmAMoLtRqovFavXq3vvvtOy5cv1+OPP241dvHixRvyZbctBQUFys3NlbOzc7keFwD1GqisnJyc9PDDD2vJkiU6efJkkS+4L168qE8//VT33nuvPD09KyjLa7NkyRI1atRInTp1slqfmZmpcePG6aOPPtL58+e1fPlyubi4qHPnzlq5cqXc3d3N2JCQENWpU0dLly5VTExMOT+DqoWpXWD+UW9vf/W3Q+HlnFdewllQUKBWrVoVmdbh+++/1/Dhw3XbbbfJ2dlZ3t7eGjlypH799VerfU6dOrXYS98cHP7ve54ePXqoRYsWSklJUefOnVWjRg35+/tr/vz5RZ7L5MmTFRQUJDc3N9WqVUtdu3bV119/bRVXOCWFnZ2dVq9ebTV28eJF1alTR3Z2dnrrrbeK5Onp6am8vDyrbf71r3+Z+7vyD6TPPvtMYWFh8vHxkZOTk26//XZNmzZN+fn5V32tC4936NAhPfroo3J1dVXdunX13HPP6eLFi1axS5YsUc+ePeXp6SknJycFBgZq3rx5RfbZv39/NW7cWM7OzvL09NQDDzygvXv3WsUUPo/ipvJo1qyZ7OzsFBkZaa47c+aMnn/+ebVs2VIuLi5ydXVV37599Z///Mdq2/DwcDk7O+vgwYNW60NDQ1WnTh2ry52PHj2qRx55RB4eHqpZs6Y6deqktWvXWm1XeDlx4eLk5KQ777xTsbGxMgzD9ov7/5X03ituSpUr3zN/XK6UkZGhUaNGqVGjRqpWrZoZ82fnQd2xY4fWrl2rUaNGFWmiS5f/ILjy/SpJmzZtUteuXVWrVi25u7urf//+RV7/3377TePGjVPjxo3l5OQkT09P3XvvvdqzZ4+kyz97a9eu1fHjx83nwuVcQPmjVq+2GqNWU6srY63+3//+J0nq0qVLkTFnZ2e5uroWWf/zzz9rwIABcnFxUf369fX8888Xee+99dZb6ty5s+rWrasaNWooKChIH330UZF9Ff67L1++XM2bN5eTk5MSExPN44wcOVJeXl5ycnJS8+bNtXjx4iL7eOedd9S8eXPVrFlTderUUfv27ZWQkHBdrwdwK6Jer7Yao15TrytjvR46dKgKCgq0YsWKImNr165VZmamhgwZIqn074c/Ku7nW/q/1/qP85Xv2LFDffr0kZubm2rWrKnu3bvr22+/LdXzWb16tXr27Fnk9Xvuuee0fPlyRUVF6d5771VMTIymTp2q8+fPKzs72yq2evXq6tGjhz777LNSHfNWxhnpMIu9k5PTdW3/z3/+s0jBkC7PK3X06FGNGDFC3t7e2r9/vxYsWKD9+/dr+/btRX7I582bZ/UL8Y9/fJw9e1b33XefHn30UQ0ePFgffvihnnnmGTk6OmrkyJGSpKysLC1cuFCDBw/W6NGj9dtvv2nRokUKDQ3Vzp071aZNG6t9Ojs7a8mSJRowYIC57pNPPilSTK/022+/ac2aNXrwwQfNdUuWLJGzs3OR7eLj4+Xi4qKoqCi5uLho06ZNmjx5srKysvTmm2+WeIwrPfroo2rcuLFiY2O1fft2zZkzR2fPntUHH3xg9do1b95cDzzwgBwcHPTFF1/or3/9qwoKChQREWG1vzFjxsjb21snT57U3LlzFRISomPHjqlmzZpFXpcrp/LYtm1bsTeuOHr0qFavXq1HHnlE/v7+Sk9P13vvvafu3bvrwIED5je8s2fP1qZNmxQeHq7k5GRVq1ZN7733ntavX69//vOfZlx6ero6d+6s33//Xc8++6zq1q2rpUuX6oEHHtBHH31k9bpL/3fZ5IULF7Ry5Uq99NJL8vT01KhRo0r1+ha+foXvvejoaJuxY8aMUdeuXSVdfq98+umnVuPh4eHasGGDxo4dq9atW6tatWpasGCB2Zi+Xp9//rkk6YknnihV/IYNG9S3b1/ddtttmjp1qi5cuKB33nlHXbp00Z49e8xm+NNPP62PPvpIkZGRCgwM1K+//qp///vfOnjwoNq1a6e///3vyszM1E8//aSZM2dKEjdHAyoAtZpaTa2u/LW6cF7SDz74QJMmTSry8/NH+fn5Cg0NVceOHfXWW29pw4YNevvtt3X77bfrmWeeMeNmz56tBx54QEOGDFFubq5WrFihRx55RGvWrFFYWJjVPjdt2qQPP/xQkZGRqlevnho3bqz09HR16tTJbNjUr19fX331lUaNGqWsrCzzPfT+++/r2Wef1cMPP2w2l77//nvt2LGjyBn2AIpHvaZeU68rf73u1q2bGjZsqISEBEVFRVmNJSQkqGbNmub7+FreD9dr06ZN6tu3r4KCgjRlyhTZ29ubDfxvvvlGHTp0KHHbn3/+WWlpaWrXrl2RsbVr12r48OF65ZVXNHz4cHXt2lU9evTQ3/72t2L3FRQUpM8++0xZWVnFfvmP/8/ALW/WrFmGJOM///mP1fru3bsbzZs3t1q3ZMkSQ5Jx7NgxwzAM4+LFi0ajRo2Mvn37GpKMJUuWmLG///57kWP961//MiQZW7duNddNmTLFkGT88ssvJebYvXt3Q5Lx9ttvm+tycnKMNm3aGJ6enkZubq5hGIZx6dIlIycnx2rbs2fPGl5eXsbIkSPNdceOHTMkGYMHDzYcHBwMi8VijvXq1ct4/PHHDUnGm2++WSTPwYMHG/fff7+5/vjx44a9vb0xePDgIs+juNfgqaeeMmrWrGlcvHixxOd75fEeeOABq/V//etfi/x7FXec0NBQ47bbbrN5jA8//NCQZOzevdtcJ8l4+OGHDQcHB6v1o0aNMl+XiIgIc/3FixeN/Px8q/0eO3bMcHJyMmJiYqzWr1u3zpBkvPrqq8bRo0cNFxcXY8CAAVY
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x1000 with 4 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Устранить выборсы в DataFrame\n",
|
|||
|
|
"def remove_outliers(dataframe: DataFrame, columns: list[str]) -> DataFrame:\n",
|
|||
|
|
" for column in columns:\n",
|
|||
|
|
" if not pd.api.types.is_numeric_dtype(dataframe[column]): # Проверяем, является ли колонка числовой\n",
|
|||
|
|
" continue\n",
|
|||
|
|
" \n",
|
|||
|
|
" Q1: float = dataframe[column].quantile(0.25) # 1-й квартиль (25%)\n",
|
|||
|
|
" Q3: float = dataframe[column].quantile(0.75) # 3-й квартиль (75%)\n",
|
|||
|
|
" IQR: float = Q3 - Q1 # Вычисляем межквартильный размах\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Определяем границы для выбросов\n",
|
|||
|
|
" lower_bound: float = Q1 - 1.5 * IQR # Нижняя граница\n",
|
|||
|
|
" upper_bound: float = Q3 + 1.5 * IQR # Верхняя граница\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Устраняем выбросы:\n",
|
|||
|
|
" # Заменяем значения ниже нижней границы на нижнюю границу\n",
|
|||
|
|
" # А значения выше верхней границы – на верхнюю\n",
|
|||
|
|
" dataframe[column] = dataframe[column].apply(lambda x: lower_bound if x < lower_bound else upper_bound if x > upper_bound else x)\n",
|
|||
|
|
" \n",
|
|||
|
|
" return dataframe\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Устраняем выборсы\n",
|
|||
|
|
"df: DataFrame = remove_outliers(df, numeric_columns)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка наличия выбросов в колонках\n",
|
|||
|
|
"print('Проверка наличия выбросов в колонках после их устранения:')\n",
|
|||
|
|
"check_outliers(df, numeric_columns)\n",
|
|||
|
|
"visualize_outliers(df, numeric_columns)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Разбиение набора данных на выборки:\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Групповое разбиение данных** – это метод разделения данных на несколько групп или подмножеств на основе определенного признака или характеристики. При этом наблюдения для одного объекта должны попасть только в одну выборку.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Основные виды выборки данных**:\n",
|
|||
|
|
"1. Обучающая выборка (60-80%). Обучение модели (подбор коэффициентов некоторой математической функции для аппроксимации).\n",
|
|||
|
|
"2. Контрольная выборка (10-20%). Выбор метода обучения, настройка гиперпараметров.\n",
|
|||
|
|
"3. Тестовая выборка (10-20% или 20-30%). Оценка качества модели перед передачей заказчику.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Разделим выборку данных на 3 группы и проанализируем качество распределения данных.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Стратифицированное разбиение требует, чтобы в каждом классе, по которому происходит стратификация, было минимум по два элемента, иначе метод не сможет корректно разделить данные на тренировочные, валидационные и тестовые наборы.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Чтобы решить эту проблему введём категории для значения стоимости акций. Вместо того, чтобы использовать точные значения цен для стратификации, мы создадим категории цен, основываясь на квартилях (25%, 50%, 75%) и минимальном и максимальном значении стоимости акций. Это позволит создать более крупные классы, что устранит проблему с редкими значениями.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Категории для разбиения цен:\n",
|
|||
|
|
"- Низкая цена: цены ниже первого квартиля (25%) — это значения меньше 50.52.\n",
|
|||
|
|
"- Средняя цена: цены между первым квартилем (25%) и третьим квартилем (75%) — от 50.52 до 934.11.\n",
|
|||
|
|
"- Высокая цена: цены выше третьего квартиля (75%) и до максимального значения — это значения выше 934.11.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Весь набор данных состоит из 156 объектов, из которых 78 (около 50%) относятся к категории средней цены (medium), 39 (около 25%) — к низкой цене (low), и 39 (около 25%) — к высокой цене (high).\n",
|
|||
|
|
"\n",
|
|||
|
|
"Все выборки показывают одинаковое распределение классов, что свидетельствует о том, что данные были отобраны случайным образом и не содержат явного смещения.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Однако, несмотря на сбалансированность при разбиении данных, в целом данные обладают значительным дисбалансом между классами. Это может быть проблемой при обучении модели, так как она может иметь тенденцию игнорировать низкие или высокие цены (low или high), что следует учитывать при дальнейшем анализе и выборе методов обработки данных.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Для получения более сбалансированных выборок данных необходимо воспользоваться методами приращения (аугментации) данных. В данном случае воспользуемся методом oversampling."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 645,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Функция для создания выборок\n",
|
|||
|
|
"def split_stratified_into_train_val_test(\n",
|
|||
|
|
" df_input,\n",
|
|||
|
|
" stratify_colname=\"y\",\n",
|
|||
|
|
" frac_train=0.6,\n",
|
|||
|
|
" frac_val=0.15,\n",
|
|||
|
|
" frac_test=0.25,\n",
|
|||
|
|
" random_state=None,\n",
|
|||
|
|
") -> tuple[Any, Any, Any]:\n",
|
|||
|
|
" \"\"\"\n",
|
|||
|
|
" Splits a Pandas dataframe into three subsets (train, val, and test)\n",
|
|||
|
|
" following fractional ratios provided by the user, where each subset is\n",
|
|||
|
|
" stratified by the values in a specific column (that is, each subset has\n",
|
|||
|
|
" the same relative frequency of the values in the column). It performs this\n",
|
|||
|
|
" splitting by running train_test_split() twice.\n",
|
|||
|
|
"\n",
|
|||
|
|
" Parameters\n",
|
|||
|
|
" ----------\n",
|
|||
|
|
" df_input : Pandas dataframe\n",
|
|||
|
|
" Input dataframe to be split.\n",
|
|||
|
|
" stratify_colname : str\n",
|
|||
|
|
" The name of the column that will be used for stratification. Usually\n",
|
|||
|
|
" this column would be for the label.\n",
|
|||
|
|
" frac_train : float\n",
|
|||
|
|
" frac_val : float\n",
|
|||
|
|
" frac_test : float\n",
|
|||
|
|
" The ratios with which the dataframe will be split into train, val, and\n",
|
|||
|
|
" test data. The values should be expressed as float fractions and should\n",
|
|||
|
|
" sum to 1.0.\n",
|
|||
|
|
" random_state : int, None, or RandomStateInstance\n",
|
|||
|
|
" Value to be passed to train_test_split().\n",
|
|||
|
|
"\n",
|
|||
|
|
" Returns\n",
|
|||
|
|
" -------\n",
|
|||
|
|
" df_train, df_val, df_test :\n",
|
|||
|
|
" Dataframes containing the three splits.\n",
|
|||
|
|
" \"\"\"\n",
|
|||
|
|
"\n",
|
|||
|
|
" if frac_train + frac_val + frac_test != 1.0:\n",
|
|||
|
|
" raise ValueError(\n",
|
|||
|
|
" \"fractions %f, %f, %f do not add up to 1.0\"\n",
|
|||
|
|
" % (frac_train, frac_val, frac_test)\n",
|
|||
|
|
" )\n",
|
|||
|
|
"\n",
|
|||
|
|
" if stratify_colname not in df_input.columns:\n",
|
|||
|
|
" raise ValueError(\"%s is not a column in the dataframe\" % (stratify_colname))\n",
|
|||
|
|
"\n",
|
|||
|
|
" X: DataFrame = df_input # Contains all columns.\n",
|
|||
|
|
" y: DataFrame = df_input[\n",
|
|||
|
|
" [stratify_colname]\n",
|
|||
|
|
" ] # Dataframe of just the column on which to stratify.\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Split original dataframe into train and temp dataframes.\n",
|
|||
|
|
" df_train, df_temp, y_train, y_temp = train_test_split(\n",
|
|||
|
|
" X, y, \n",
|
|||
|
|
" stratify=y, \n",
|
|||
|
|
" test_size=(1.0 - frac_train), \n",
|
|||
|
|
" random_state=random_state\n",
|
|||
|
|
" )\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Split the temp dataframe into val and test dataframes.\n",
|
|||
|
|
" relative_frac_test: float = frac_test / (frac_val + frac_test)\n",
|
|||
|
|
" df_val, df_test, y_val, y_test = train_test_split(\n",
|
|||
|
|
" df_temp,\n",
|
|||
|
|
" y_temp,\n",
|
|||
|
|
" stratify=y_temp,\n",
|
|||
|
|
" test_size=relative_frac_test,\n",
|
|||
|
|
" random_state=random_state,\n",
|
|||
|
|
" )\n",
|
|||
|
|
"\n",
|
|||
|
|
" assert len(df_input) == len(df_train) + len(df_val) + len(df_test)\n",
|
|||
|
|
"\n",
|
|||
|
|
" return df_train, df_val, df_test"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 646,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Распределение количества наблюдений по меткам (классам):\n",
|
|||
|
|
"Cost\n",
|
|||
|
|
"0.00 18\n",
|
|||
|
|
"6.24 10\n",
|
|||
|
|
"62.72 8\n",
|
|||
|
|
"20.91 7\n",
|
|||
|
|
"52.38 4\n",
|
|||
|
|
" ..\n",
|
|||
|
|
"1098.24 1\n",
|
|||
|
|
"1072.22 1\n",
|
|||
|
|
"1019.03 1\n",
|
|||
|
|
"1048.46 1\n",
|
|||
|
|
"1068.09 1\n",
|
|||
|
|
"Name: count, Length: 101, dtype: int64 \n",
|
|||
|
|
"\n",
|
|||
|
|
"Статистическое описание целевого признака:\n",
|
|||
|
|
"count 156.000000\n",
|
|||
|
|
"mean 478.785641\n",
|
|||
|
|
"std 448.922903\n",
|
|||
|
|
"min 0.000000\n",
|
|||
|
|
"25% 50.522500\n",
|
|||
|
|
"50% 240.225000\n",
|
|||
|
|
"75% 934.107500\n",
|
|||
|
|
"max 1171.040000\n",
|
|||
|
|
"Name: Cost, dtype: float64 \n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение количества наблюдений по меткам (классам):\n",
|
|||
|
|
"Cost_category\n",
|
|||
|
|
"medium 78\n",
|
|||
|
|
"low 39\n",
|
|||
|
|
"high 39\n",
|
|||
|
|
"Name: count, dtype: int64 \n",
|
|||
|
|
"\n",
|
|||
|
|
"Проверка сбалансированности выборок:\n",
|
|||
|
|
"Обучающая выборка: (93, 184)\n",
|
|||
|
|
"Распределение выборки данных по классам в колонке \"Cost_category\":\n",
|
|||
|
|
" Cost_category\n",
|
|||
|
|
"medium 47\n",
|
|||
|
|
"low 23\n",
|
|||
|
|
"high 23\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"Процент объектов класса \"medium\": 50.54%\n",
|
|||
|
|
"Процент объектов класса \"low\": 24.73%\n",
|
|||
|
|
"Процент объектов класса \"high\": 24.73%\n",
|
|||
|
|
"\n",
|
|||
|
|
"Контрольная выборка: (31, 184)\n",
|
|||
|
|
"Распределение выборки данных по классам в колонке \"Cost_category\":\n",
|
|||
|
|
" Cost_category\n",
|
|||
|
|
"medium 15\n",
|
|||
|
|
"low 8\n",
|
|||
|
|
"high 8\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"Процент объектов класса \"medium\": 48.39%\n",
|
|||
|
|
"Процент объектов класса \"low\": 25.81%\n",
|
|||
|
|
"Процент объектов класса \"high\": 25.81%\n",
|
|||
|
|
"\n",
|
|||
|
|
"Тестовая выборка: (32, 184)\n",
|
|||
|
|
"Распределение выборки данных по классам в колонке \"Cost_category\":\n",
|
|||
|
|
" Cost_category\n",
|
|||
|
|
"medium 16\n",
|
|||
|
|
"low 8\n",
|
|||
|
|
"high 8\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"Процент объектов класса \"medium\": 50.00%\n",
|
|||
|
|
"Процент объектов класса \"low\": 25.00%\n",
|
|||
|
|
"Процент объектов класса \"high\": 25.00%\n",
|
|||
|
|
"\n",
|
|||
|
|
"Проверка необходимости аугментации выборок:\n",
|
|||
|
|
"Для обучающей выборки аугментация данных требуется\n",
|
|||
|
|
"Для контрольной выборки аугментация данных требуется\n",
|
|||
|
|
"Для тестовой выборки аугментация данных требуется\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABigAAAHmCAYAAADp3gZeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADrEklEQVR4nOzdd1xV9f8H8NdlbwRRQTFw71GalgszNUepqZmaq1Ir07Lh/uZKM9PcmqNyNxy5NfcG90JBRZbIHrK5cMfn94e/e+NyAQHhHu7l9Xw8eJSXw7mvu877nvP+nM+RCSEEiIiIiIiIiIiIiIiIDMhM6gBERERERERERERERFTxsEFBREREREREREREREQGxwYFEREREREREREREREZHBsURERERERERERERERkcGxQEBERERERERERERGRwbFBQUREREREREREREREBscGBRERERERERERERERGRwbFEREREREREREREREZHBsUBAR0XMplUrExcXh8ePHUkehUpaTk4OYmBhERUVJHYWIiIheUFpaGsLCwpCRkSF1FCIiIqIiYYOCiIjyFRQUhDFjxsDDwwNWVlaoVq0aXn/9dQghpI5mFLZt24awsDDtvzdt2oTIyEjpAuVy7do1DB06FG5ubrC2toaHhwcGDBggdSwiIqJyKT09HcuWLdP+Ozk5GatXr5YuUC5CCKxfvx6vvfYa7Ozs4OTkhFq1amHbtm1SRyMiIqJSIoRAUlISgoKCpI5SJorVoNi0aRNkMpn2x8bGBvXr18f48eMRGxtbVhmJTN7s2bPh7e0N4L/PWX727NmDnj17ws3NDVZWVqhevToGDRqEU6dOlUkuX19fzJ49G8nJyWWy/qIKCAjA7NmzdQ72moKwsDDIZDKcOXMGACCTybBp0yZJM2lcunQJbdq0walTpzB16lQcPXoUx48fx969ewt8f5Ku8+fPY/LkyQgLC8PRo0fx+eefw8ys7MYFFLVG79u3Dx06dEBAQADmz5+P48eP4/jx41i3bl2ZZSMyZqzRhWONporA1tYW//vf/7B9+3ZERERg9uzZOHDggNSxAABDhw7Fp59+ikaNGmHr1q04fvw4Tpw4gf79+0sdjYiIqNy7e/cu9u7dq/33rVu3cOjQIekC5ZKWlob//e9/aNCgAaysrFC5cmXUr18fDx48kDpaqbMoyR/NnTsXtWrVglwux4ULF/DLL7/g8OHDuHv3Luzs7Eo7I1GFJ4TARx99hE2bNuHll1/G119/DXd3d0RHR2PPnj148803cfHiRbRr165U79fX1xdz5szBqFGjUKlSpVJdd3EEBARgzpw56Ny5s/YgEZWdnJwcfPjhh6hfvz6OHTsGZ2dnqSMZpa+++gqdO3dGrVq1AABff/01PDw8yvx+C6vRcrkco0ePxltvvYWdO3fCysqqzPMQmTrWaNZoMn3m5uaYM2cORowYAbVaDScnpwIPXmzatAkffvih9t/W1tZ46aWX0L17d3z33XeoVq1aqeXasmUL/v77b2zbtg1Dhw4ttfUSlUezZ8/Gpk2bEBYWpv2c5Xdm9549e7B+/XpcvXoVqampcHNzQ4cOHfDpp5+iS5cupZ7L19cXx44dw8SJEyWvxzt27MCoUaNMqh6HhYWhVq1aOH36NDp37gyZTIaNGzdi1KhRUkcjE5GWloZPPvkE7u7uqFy5Mr788kv07NkTvXv3ljRXYmIifHx88PjxY0yYMAHt27eHlZUVLC0tTeozrlGiBkXPnj3RunVrAMDo0aNRuXJlLFmyBPv27cOQIUNKNSARAT///DM2bdqEiRMnYsmSJTqjN2fMmIGtW7fCwqJEH2cqRWq1Gjk5ObCxsZE6ygs5cOAAHjx4gPv377M58QIaNmyI4OBg3L17F25ubqhTp45B7rewGh0VFQW5XI5NmzaxOUFUSlijjYOp1GiSzjfffIP3338fERERaNSo0XMPRBpiUN+iRYswZMgQNieIwAEDHDBAVDKvv/669gcA6tevjzFjxkicCpg0aRKio6Ph5+eHJk2aSB2nzJXKXBOaLnRoaCgAICkpCd9++y2aNWsGBwcHODk5oWfPnrh9+7be38rlcsyePRv169eHjY0NPDw80L9/fwQHBwP47/Tqgn46d+6sXdeZM2cgk8nw999/Y/r06XB3d4e9vT369OmDiIgIvfu+fPkyevToAWdnZ9jZ2cHHxwcXL17M9zFqOrV5f2bPnq237LZt29CqVSvY2trC1dUVgwcPzvf+C3tsuanVaixbtgxNmjSBjY0NqlWrhk8++QRPnz7VWc7b2xtvv/223v2MHz9eb535ZV+0aJHecwoA2dnZmDVrFurWrQtra2vUrFkTkydPRnZ2dr7PVW6dO3fWW9/8+fNhZmaGP/74o0TPx+LFi9GuXTtUrlwZtra2aNWqFXbt2pXv/W/btg1t2rSBnZ0dXFxc0KlTJxw7dkxnmSNHjsDHxweOjo5wcnLCq6++qpdt586d2tfUzc0Nw4YN05tLftSoUTqZXVxc0LlzZ5w/f/65z1NhsrKysGDBAjRs2BCLFy/Od2qJ4cOHo02bNtp/h4SE4L333oOrqyvs7Ozw2muv5TvKa+XKlWjSpIn2+WndurX2sc+ePRuTJk0CANSqVUv7uIozhcP9+/cxaNAgVKlSBba2tmjQoAFmzJih/X14eDjGjRuHBg0awNbWFpUrV8Z7772nN2//e++9BwB44403tDk0Uy4Az17Djh07wt7eHo6Ojujduzfu3bunl2fnzp1o3LgxbGxs0LRpU+zZsyffESYZGRn45ptvULNmTVhbW6NBgwZYvHix3ggdmUyG8ePHY/v27WjSpAmsra1x5MgReHt7o2/fvnr3L5fL4ezsjE8++aTIz2Femu2c5sfa2hr169fHggULinRtiLi4OHz88ceoVq0abGxs0KJFC2zevFlnmUuXLqFWrVrYvXs36tSpAysrK7z00kuYPHkysrKytMuNHDkSbm5uUCgUevfTvXt3NGjQQCdz7tcMQL7PfVE/397e3jqjZtLS0jB+/HjUqFED1tbWqFevHn788Ueo1Wqdv9O8Zrm9/fbbejl27dqVb+bk5GRMnDhR+96oW7cuFi5cqHM/mm3Zpk2bYG9vj7Zt26JOnTr4/PPPIZPJnjvaJ++2UDNCYtKkScjJydEup5lq5tq1awWuq3PnztptXmhoKC5duoQmTZqge/fusLS0hEwmg5mZGRo0aICbN2/q/K1SqcTMmTPh4uKizeLg4IB+/fqVqEZPnToVMpkMq1at0tZoOzs7WFpa4s0334RSqdR5ngcPHgwbGxttxtq1axe4PWWNZo1mjWaNNvYanV+tjIqKgre3N1q3bo309HTt7UWp5ZrP7OLFi/Xuq2nTptrPft7MhW1LZ8+eDZlMpn3vODk5aUcayuVynftQKpX4/vvvUadOHVhbW8Pb2xvTp0/Pd9tUUIbcr71mmYK2JxqajAkJCTq3X7t2Ld9puk6dOqV9f1aqVAl9+/ZFYGBgvusEAE9PT7z++uuwsLCAu7t7vt8VNHr27Ilhw4bhyZMn2Lx5M9RqNUJDQ+Hq6oo2bdroTCcBPNsmN23atMDHlvv7BfDss3D37l3UrFkTvXv3hpOTE+zt7QvcthVl21Oc/eni1JDi7HcTlVTuAQPXr1/H9OnT8dFHH2HGjBm4du0atmzZwgED5YBardarGURS27t3L+7du4dr167B398flStXljRPXFwcNm/ejB9//LFCNCeAUmpQaA5UaF7AkJAQ7N27F2+//TaWLFmCSZMmwd/fHz4+PoiKitL+nUqlwttvv405c+agVatW+Pnnn/Hll18iJSUFd+/e1bmPIUOGYOvWrTo/np6e+eaZP38+Dh06hClTpuCLL77A8ePH0bVrV50Da6dOnUKnTp2QmpqKWbNm4YcffkBycjK6dOmCK1eu5LteT09P7X3/8ssvBd73iBEjUK9ePSxZsgQTJ07EyZMn0alTpwLnCB47dqx2ve+++67e7z/55BNMmjQJ7du3x/Lly/Hhhx9i+/bteOutt/I9MFgSycnJWLBggd7tarUaffr0weL
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x500 with 3 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Оценка сбалансированности\n",
|
|||
|
|
"def check_balance(dataframe: DataFrame, dataframe_name: str, column: str) -> None:\n",
|
|||
|
|
" counts: Series[int] = dataframe[column].value_counts()\n",
|
|||
|
|
" print(dataframe_name + \": \", dataframe.shape)\n",
|
|||
|
|
" print(f\"Распределение выборки данных по классам в колонке \\\"{column}\\\":\\n\", counts)\n",
|
|||
|
|
" total_count: int = len(dataframe)\n",
|
|||
|
|
" for value in counts.index:\n",
|
|||
|
|
" percentage: float = counts[value] / total_count * 100\n",
|
|||
|
|
" print(f\"Процент объектов класса \\\"{value}\\\": {percentage:.2f}%\")\n",
|
|||
|
|
" print()\n",
|
|||
|
|
" \n",
|
|||
|
|
"# Определение необходимости аугментации данных\n",
|
|||
|
|
"def need_augmentation(dataframe: DataFrame,\n",
|
|||
|
|
" column: str, \n",
|
|||
|
|
" first_value: Any, second_value: Any) -> bool:\n",
|
|||
|
|
" counts: Series[int] = dataframe[column].value_counts()\n",
|
|||
|
|
" ratio: float = counts[first_value] / counts[second_value]\n",
|
|||
|
|
" return ratio > 1.5 or ratio < 0.67\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Визуализация сбалансированности классов\n",
|
|||
|
|
"def visualize_balance(dataframe_train: DataFrame,\n",
|
|||
|
|
" dataframe_val: DataFrame,\n",
|
|||
|
|
" dataframe_test: DataFrame, \n",
|
|||
|
|
" column: str) -> None:\n",
|
|||
|
|
" fig, axes = plt.subplots(1, 3, figsize=(15, 5))\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Обучающая выборка\n",
|
|||
|
|
" counts_train: Series[int] = dataframe_train[column].value_counts()\n",
|
|||
|
|
" axes[0].pie(counts_train, labels=counts_train.index, autopct='%1.1f%%', startangle=90)\n",
|
|||
|
|
" axes[0].set_title(f\"Распределение классов \\\"{column}\\\" в обучающей выборке\")\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Контрольная выборка\n",
|
|||
|
|
" counts_val: Series[int] = dataframe_val[column].value_counts()\n",
|
|||
|
|
" axes[1].pie(counts_val, labels=counts_val.index, autopct='%1.1f%%', startangle=90)\n",
|
|||
|
|
" axes[1].set_title(f\"Распределение классов \\\"{column}\\\" в контрольной выборке\")\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Тестовая выборка\n",
|
|||
|
|
" counts_test: Series[int] = dataframe_test[column].value_counts()\n",
|
|||
|
|
" axes[2].pie(counts_test, labels=counts_test.index, autopct='%1.1f%%', startangle=90)\n",
|
|||
|
|
" axes[2].set_title(f\"Распределение классов \\\"{column}\\\" в тренировочной выборке\")\n",
|
|||
|
|
"\n",
|
|||
|
|
" # Отображение графиков\n",
|
|||
|
|
" plt.tight_layout()\n",
|
|||
|
|
" plt.show()\n",
|
|||
|
|
" \n",
|
|||
|
|
"\n",
|
|||
|
|
"# Унитарное кодирование категориальных признаков (one-hot encoding)\n",
|
|||
|
|
"df_encoded: DataFrame = pd.get_dummies(df)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Вывод распределения количества наблюдений по меткам (классам)\n",
|
|||
|
|
"print('Распределение количества наблюдений по меткам (классам):')\n",
|
|||
|
|
"print(df_encoded['Cost'].value_counts(), '\\n')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Статистическое описание целевого признака\n",
|
|||
|
|
"print('Статистическое описание целевого признака:')\n",
|
|||
|
|
"print(df_encoded['Cost'].describe().transpose(), '\\n')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Определим границы для каждой категории стоимости акций\n",
|
|||
|
|
"bins: list[float] = [df_encoded['Cost'].min() - 1, \n",
|
|||
|
|
" df_encoded['Cost'].quantile(0.25), \n",
|
|||
|
|
" df_encoded['Cost'].quantile(0.75), \n",
|
|||
|
|
" df_encoded['Cost'].max() + 1]\n",
|
|||
|
|
"labels: list[str] = ['low', 'medium', 'high']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Создаем новую колонку с категориями стоимости акций\n",
|
|||
|
|
"df_encoded['Cost_category'] = pd.cut(df_encoded['Cost'], bins=bins, labels=labels)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Вывод распределения количества наблюдений по меткам (классам)\n",
|
|||
|
|
"print('Распределение количества наблюдений по меткам (классам):')\n",
|
|||
|
|
"print(df_encoded['Cost_category'].value_counts(), '\\n')\n",
|
|||
|
|
"\n",
|
|||
|
|
"df_train, df_val, df_test = split_stratified_into_train_val_test(\n",
|
|||
|
|
" df_encoded, \n",
|
|||
|
|
" stratify_colname=\"Cost_category\", \n",
|
|||
|
|
" frac_train=0.60, \n",
|
|||
|
|
" frac_val=0.20, \n",
|
|||
|
|
" frac_test=0.20\n",
|
|||
|
|
")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка сбалансированности выборок\n",
|
|||
|
|
"print('Проверка сбалансированности выборок:')\n",
|
|||
|
|
"check_balance(df_train, 'Обучающая выборка', 'Cost_category')\n",
|
|||
|
|
"check_balance(df_val, 'Контрольная выборка', 'Cost_category')\n",
|
|||
|
|
"check_balance(df_test, 'Тестовая выборка', 'Cost_category')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка необходимости аугментации выборок\n",
|
|||
|
|
"print('Проверка необходимости аугментации выборок:')\n",
|
|||
|
|
"print(f\"Для обучающей выборки аугментация данных {'не ' if not need_augmentation(df_train, 'Cost_category', 'low', 'medium') else ''}требуется\")\n",
|
|||
|
|
"print(f\"Для контрольной выборки аугментация данных {'не ' if not need_augmentation(df_val, 'Cost_category', 'low', 'medium') else ''}требуется\")\n",
|
|||
|
|
"print(f\"Для тестовой выборки аугментация данных {'не ' if not need_augmentation(df_test, 'Cost_category', 'low', 'medium') else ''}требуется\")\n",
|
|||
|
|
" \n",
|
|||
|
|
"# Визуализация сбалансированности классов\n",
|
|||
|
|
"visualize_balance(df_train, df_val, df_test, 'Cost_category')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Приращение данных:\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Аугментация данных** может быть полезна в том случае, когда имеется недостаточное количество данных и мы хотим сгенерировать новые данные на основе имеющихся, слегка модифицировав их.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Метод решения:**\n",
|
|||
|
|
"**Выборка с избытком (oversampling)** – копирование наблюдений или генерация новых наблюдений на основе существующих с помощью алгоритмов SMOTE и ADASYN (нахождение k-ближайших соседей)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 647,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Проверка сбалансированности выборок после применения метода oversampling:\n",
|
|||
|
|
"Обучающая выборка: (141, 184)\n",
|
|||
|
|
"Распределение выборки данных по классам в колонке \"Cost_category\":\n",
|
|||
|
|
" Cost_category\n",
|
|||
|
|
"low 47\n",
|
|||
|
|
"medium 47\n",
|
|||
|
|
"high 47\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"Процент объектов класса \"low\": 33.33%\n",
|
|||
|
|
"Процент объектов класса \"medium\": 33.33%\n",
|
|||
|
|
"Процент объектов класса \"high\": 33.33%\n",
|
|||
|
|
"\n",
|
|||
|
|
"Контрольная выборка: (45, 184)\n",
|
|||
|
|
"Распределение выборки данных по классам в колонке \"Cost_category\":\n",
|
|||
|
|
" Cost_category\n",
|
|||
|
|
"low 15\n",
|
|||
|
|
"medium 15\n",
|
|||
|
|
"high 15\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"Процент объектов класса \"low\": 33.33%\n",
|
|||
|
|
"Процент объектов класса \"medium\": 33.33%\n",
|
|||
|
|
"Процент объектов класса \"high\": 33.33%\n",
|
|||
|
|
"\n",
|
|||
|
|
"Тестовая выборка: (48, 184)\n",
|
|||
|
|
"Распределение выборки данных по классам в колонке \"Cost_category\":\n",
|
|||
|
|
" Cost_category\n",
|
|||
|
|
"low 16\n",
|
|||
|
|
"medium 16\n",
|
|||
|
|
"high 16\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"Процент объектов класса \"low\": 33.33%\n",
|
|||
|
|
"Процент объектов класса \"medium\": 33.33%\n",
|
|||
|
|
"Процент объектов класса \"high\": 33.33%\n",
|
|||
|
|
"\n",
|
|||
|
|
"Проверка необходимости аугментации выборок после применения метода oversampling:\n",
|
|||
|
|
"Для обучающей выборки аугментация данных не требуется\n",
|
|||
|
|
"Для контрольной выборки аугментация данных не требуется\n",
|
|||
|
|
"Для тестовой выборки аугментация данных не требуется\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABigAAAH/CAYAAADNB1UNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACm90lEQVR4nOzdd3hUdd6G8WfSO4QamhB6h11YFJSiooKoWLFTLLirrqKIihWxsAo27LoKCNiAFZQiRYo0KdJ7kRpKEkJ6z5z3D94ZMySBJCRzZs7cn+vy2iWZzHwzmZz7TH5nztgMwzAEAAAAAAAAAADgRn5mDwAAAAAAAAAAAHwPCxQAAAAAAAAAAMDtWKAAAAAAAAAAAABuxwIFAAAAAAAAAABwOxYoAAAAAAAAAACA27FAAQAAAAAAAAAA3I4FCgAAAAAAAAAA4HYsUAAAAAAAAAAAALdjgQIAcF75+fmKj4/X4cOHzR4FFSw3N1cnTpzQsWPHzB4FAABcoLS0NB08eFAZGRlmjwIAACqIYRhKSkrS3r17zR6lUrBAAQAo1t69e/Xggw+qTp06CgoKUu3atdW1a1cZhmH2aF5hypQpOnjwoPPfEydOVFxcnHkDFbJ+/XrdddddqlGjhoKDg1WnTh3dcsstZo8FAIBHSk9P13vvvef8d3Jysj766CPzBirEMAx9/vnnuuSSSxQWFqaoqCjFxsZqypQpZo8GAIDH27Ztm2bOnOn896ZNmzRnzhzzBiokLS1NL7zwglq0aKGgoCBVr15dzZs31+7du80ercKVaYFi4sSJstlszv9CQkLUvHlzPfroozp58mRlzQhY3qhRo9SoUSNJf/2eFefHH39U3759VaNGDQUFBalu3boaMGCAFi9eXClzrVq1SqNGjVJycnKlXH9p7dixQ6NGjXL5Y68VHDx4UDabTUuXLpUk2Ww2TZw40dSZHH7//Xd16dJFixcv1rPPPqv58+dr4cKFmjlzZomPT7havny5nn76aR08eFDz58/XI488Ij+/yjsuoLSNnjVrli677DLt2LFDr7/+uhYuXKiFCxfqs88+q7TZAG9Go8+NRsMXhIaG6oUXXtDUqVN15MgRjRo1Sj///HOxl3X3c+a77rpL//znP9WqVStNnjxZCxcu1KJFi3TzzTdX+G0BZqLH50aPgfJJS0vTQw89pN9//1179+7V448/rq1bt5o9lk6dOqWuXbtq/PjxuvXWWzVr1iwtXLhQS5cudW4LrSSgPF80evRoxcbGKjs7WytWrNAnn3yiuXPnatu2bQoLC6voGQGfZxiG7rvvPk2cOFF/+9vf9OSTTyomJkbHjx/Xjz/+qCuvvFIrV65Ut27dKvR2V61apVdeeUWDBw9W1apVK/S6y2LHjh165ZVX1KtXL0tuiD1Nbm6uhgwZoubNm2vBggWqUqWK2SN5pSeeeEK9evVSbGysJOnJJ59UnTp1Kv12z9Xo7OxsPfDAA7rmmms0bdo0BQUFVfo8gNXRaBoN6/P399crr7yigQMHym63Kyoq6rxHV7rjOfPXX3+t77//XlOmTNFdd91VIdcJeCt6TI+B8ujatavzP0lq3ry5HnzwQZOnkkaMGKHjx49r9erVatOmjdnjVLpyLVD07dtXnTt3liQ98MADql69ut555x3NmjVLd955Z4UOCEB6++23NXHiRA0bNkzvvPOOy9Eizz//vCZPnqyAgHL9OqMC2e125ebmKiQkxOxRLsjPP/+s3bt3a9euXSxOXICWLVtq//792rZtm2rUqKEmTZq45XbP1ehjx44pOztbEydOZHECqCA02jtYpdEwz/Dhw3X77bfryJEjatWq1Xn/EOmO58xjx47VnXfeyeIEIHrsLegxPNHMmTO1Y8cOZWVlqV27dqY/V46Pj9ekSZP06aef+sTihFRB70FxxRVXSJIOHDggSUpKStJTTz2ldu3aKSIiQlFRUerbt682b95c5Guzs7M1atQoNW/eXCEhIapTp45uvvlm7d+/X9JfL+cq6b9evXo5r2vp0qWy2Wz6/vvv9dxzzykmJkbh4eG64YYbdOTIkSK3vWbNGvXp00dVqlRRWFiYevbsqZUrVxb7Pfbq1avY2x81alSRy06ZMkWdOnVSaGioqlWrpjvuuKPY2z/X91aY3W7Xe++9pzZt2igkJES1a9fWQw89pNOnT7tcrlGjRrruuuuK3M6jjz5a5DqLm33s2LFF7lNJysnJ0csvv6ymTZsqODhYDRo00NNPP62cnJxi76vCevXqVeT6Xn/9dfn5+embb74p1/0xbtw4devWTdWrV1doaKg6deqk6dOnF3v7U6ZMUZcuXRQWFqbo6Gj16NFDCxYscLnMvHnz1LNnT0VGRioqKkr/+Mc/isw2bdo058+0Ro0auueee4qcS37w4MEuM0dHR6tXr15avnz5ee+nc8nKytKYMWPUsmVLjRs3rtiXst57773q0qWL899//vmnbrvtNlWrVk1hYWG65JJLij3K64MPPlCbNm2c90/nzp2d3/uoUaM0YsQISVJsbKzz+yrLS0Z37dqlAQMGqGbNmgoNDVWLFi30/PPPOz9/6NAhPfzww2rRooVCQ0NVvXp13XbbbUXO23/bbbdJki6//HLnHI6XeEpnfobdu3dXeHi4IiMj1a9fP23fvr3IPNOmTVPr1q0VEhKitm3b6scff9TgwYOLHGGSkZGh4cOHq0GDBgoODlaLFi00bty4Iu+9YLPZ9Oijj2rq1Klq06aNgoODNW/ePDVq1Ej9+/cvcvvZ2dmqUqWKHnrooVLfh2dzbOcc/wUHB6t58+YaM2ZMqd4bIj4+Xvfff79q166tkJAQdejQQZMmTXK5zO+//67Y2FjNmDFDTZo0UVBQkC666CI9/fTTysrKcl5u0KBBqlGjhvLy8orcztVXX60WLVq4zFz4Zyap2Pu+tL/fjRo10uDBg53/TktL06OPPqp69eopODhYzZo103/+8x/Z7XaXr3P8zAq77rrriswxffr0YmdOTk7WsGHDnI+Npk2b6s0333S5Hce2bOLEiQoPD9fFF1+sJk2a6JFHHpHNZnOZuzhnbwsDAwPVqFEjjRgxQrm5uc7LOV7avn79+hKvq1evXs5t3oEDB/T777+rTZs2uvrqqxUYGCibzSY/Pz+1aNFCGzdudPna/Px8vfTSS4qOjnbOEhERoRtvvLFcjX722Wdls9n04YcfOhsdFhamwMBAXXnllcrPz3e5n++44w6FhIQ4Z2zcuHGJ21MaTaNpNI329kYX18pjx46pUaNG6ty5s9LT050fL03LHb+z48aNK3Jbbdu2df7unz3zubalo0aNks1mcz52oqKiVL16dT3++OPKzs52uY38/Hy9+uqratKkiYKDg9WoUSM999xzxW6bSpqh8M/ecZmSticOjhkTExNdPr5+/fpiTwuyePFi5+OzatWq6t+/v3bu3FnsdUpS/fr11bVrVwUEBCgmJqbYfYWSZnrnnXckndkedenSRZMnT3Z5zhwQEKDIyMgSnzMPGzbMuV9Qp04d3XDDDdq2bZsaNGjg8vtXUo8d257IyEjZbDY1a9ZMAwYMcHnO/MMPPxR5Ph0SEqJatWopKirK5TlzcQ0pvD0q/N8DDzxQ5Hn3Sy+9RI//Hz2mx/T4L2b2uLKaKJXuOUivXr3Utm3bIl/reEwX/pmf/XxYOvOzO7ufhfcH3n33XTVs2FChoaHq2bOntm3bVuS2ytJFx3+RkZHq0qWLy/s4FJ7pfNuF4r6X4vaLynL/SNLHH3/sfMzVrVtXjzzySJFTsRXe/rZu3VqdOnXS5s2bi92uFqfXWc9Da9SooX79+hW5bx2/ByVxPLd3fA/r1q1zLuZ17txZISEhql69uu68804dPny4yNeX5edWmsfs2f3Lz8/Xtddeq2rVqmnHjh0uly3t8+vzqZDlY8cfKqpXry7pzIZ+5syZuu222xQbG6uTJ0/qs88+U8+ePbVjxw7VrVtXklRQUKDrrrtOv/76q+644w49/vjjSktL08KFC7Vt2zaXo03vvPNOXXvttS63O3LkyGLnef3112Wz2fTMM88oPj5
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x500 with 3 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Метод приращения с избытком (oversampling)\n",
|
|||
|
|
"def oversample(df: DataFrame, column: str) -> DataFrame:\n",
|
|||
|
|
" X: DataFrame = pd.get_dummies(df.drop(column, axis=1))\n",
|
|||
|
|
" y: DataFrame = df[column] # type: ignore\n",
|
|||
|
|
" \n",
|
|||
|
|
" smote = SMOTE()\n",
|
|||
|
|
" X_resampled, y_resampled = smote.fit_resample(X, y) # type: ignore\n",
|
|||
|
|
" \n",
|
|||
|
|
" df_resampled: DataFrame = pd.concat([X_resampled, y_resampled], axis=1)\n",
|
|||
|
|
" return df_resampled\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Приращение данных (oversampling)\n",
|
|||
|
|
"df_train_oversampled: DataFrame = oversample(df_train, 'Cost_category')\n",
|
|||
|
|
"df_val_oversampled: DataFrame = oversample(df_val, 'Cost_category')\n",
|
|||
|
|
"df_test_oversampled: DataFrame = oversample(df_test, 'Cost_category')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка сбалансированности выборок\n",
|
|||
|
|
"print('Проверка сбалансированности выборок после применения метода oversampling:')\n",
|
|||
|
|
"check_balance(df_train_oversampled, 'Обучающая выборка', 'Cost_category')\n",
|
|||
|
|
"check_balance(df_val_oversampled, 'Контрольная выборка', 'Cost_category')\n",
|
|||
|
|
"check_balance(df_test_oversampled, 'Тестовая выборка', 'Cost_category')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка необходимости аугментации выборок\n",
|
|||
|
|
"print('Проверка необходимости аугментации выборок после применения метода oversampling:')\n",
|
|||
|
|
"print(f\"Для обучающей выборки аугментация данных {'не ' if not need_augmentation(df_train_oversampled, 'Cost_category', 'low', 'medium') else ''}требуется\")\n",
|
|||
|
|
"print(f\"Для контрольной выборки аугментация данных {'не ' if not need_augmentation(df_val_oversampled, 'Cost_category', 'low', 'medium') else ''}требуется\")\n",
|
|||
|
|
"print(f\"Для тестовой выборки аугментация данных {'не ' if not need_augmentation(df_test_oversampled, 'Cost_category', 'low', 'medium') else ''}требуется\")\n",
|
|||
|
|
" \n",
|
|||
|
|
"# Визуализация сбалансированности классов\n",
|
|||
|
|
"visualize_balance(df_train_oversampled, df_val_oversampled, df_test_oversampled, 'Cost_category')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Конструирование признаков:\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Конструирование признаков (feature engineering)** – процесс использования знаний об особенностях решаемой задачи и предметной области для определения признаков, которые будут использованы для обучения статистической модели.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Методы конструирования признаков:**\n",
|
|||
|
|
"1. **Унитарное кодирование категориальных признаков (one-hot encoding)** – метод, который применяется для преобразования категориальных переменных в числовой формат. Каждая характеристика представляется в виде бинарного вектора, где для каждой категории выделяется отдельный признак (столбец) со значением 1 (True), если объект принадлежит этой категории, и 0 (False) в противном случае.\n",
|
|||
|
|
"\n",
|
|||
|
|
"В данном случае преобразование категориальных признаков уже было выполнено на этапе разбиения данных на выборки. Для этого использовался метод \"get_dummies\" из библиотеки \"pandas\"."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 648,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/html": [
|
|||
|
|
"<div>\n",
|
|||
|
|
"<style scoped>\n",
|
|||
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|||
|
|
" vertical-align: middle;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe tbody tr th {\n",
|
|||
|
|
" vertical-align: top;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe thead th {\n",
|
|||
|
|
" text-align: right;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"</style>\n",
|
|||
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|||
|
|
" <thead>\n",
|
|||
|
|
" <tr style=\"text-align: right;\">\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th>Cost</th>\n",
|
|||
|
|
" <th>Shares</th>\n",
|
|||
|
|
" <th>Value ($)</th>\n",
|
|||
|
|
" <th>Shares Total</th>\n",
|
|||
|
|
" <th>Insider Trading_Baglino Andrew D</th>\n",
|
|||
|
|
" <th>Insider Trading_DENHOLM ROBYN M</th>\n",
|
|||
|
|
" <th>Insider Trading_Kirkhorn Zachary</th>\n",
|
|||
|
|
" <th>Insider Trading_Musk Elon</th>\n",
|
|||
|
|
" <th>Insider Trading_Musk Kimbal</th>\n",
|
|||
|
|
" <th>Insider Trading_Taneja Vaibhav</th>\n",
|
|||
|
|
" <th>...</th>\n",
|
|||
|
|
" <th>SEC Form 4_Nov 30 04:42 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Oct 05 07:35 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Oct 31 07:06 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Sep 07 08:29 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Sep 07 08:33 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Sep 07 09:04 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Sep 12 09:44 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Sep 14 07:47 PM</th>\n",
|
|||
|
|
" <th>SEC Form 4_Sep 30 07:03 PM</th>\n",
|
|||
|
|
" <th>Cost_category</th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </thead>\n",
|
|||
|
|
" <tbody>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>0</th>\n",
|
|||
|
|
" <td>196.72</td>\n",
|
|||
|
|
" <td>10455.0</td>\n",
|
|||
|
|
" <td>2056775.0</td>\n",
|
|||
|
|
" <td>203073.0</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>1</th>\n",
|
|||
|
|
" <td>195.79</td>\n",
|
|||
|
|
" <td>2466.0</td>\n",
|
|||
|
|
" <td>482718.0</td>\n",
|
|||
|
|
" <td>100458.0</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>2</th>\n",
|
|||
|
|
" <td>195.79</td>\n",
|
|||
|
|
" <td>1298.0</td>\n",
|
|||
|
|
" <td>254232.0</td>\n",
|
|||
|
|
" <td>65547.0</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>3</th>\n",
|
|||
|
|
" <td>0.00</td>\n",
|
|||
|
|
" <td>7138.0</td>\n",
|
|||
|
|
" <td>0.0</td>\n",
|
|||
|
|
" <td>102923.0</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>low</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>4</th>\n",
|
|||
|
|
" <td>0.00</td>\n",
|
|||
|
|
" <td>2586.0</td>\n",
|
|||
|
|
" <td>0.0</td>\n",
|
|||
|
|
" <td>66845.0</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>low</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>5</th>\n",
|
|||
|
|
" <td>0.00</td>\n",
|
|||
|
|
" <td>16867.0</td>\n",
|
|||
|
|
" <td>0.0</td>\n",
|
|||
|
|
" <td>213528.0</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>low</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>6</th>\n",
|
|||
|
|
" <td>20.91</td>\n",
|
|||
|
|
" <td>10500.0</td>\n",
|
|||
|
|
" <td>219555.0</td>\n",
|
|||
|
|
" <td>74759.0</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>low</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>7</th>\n",
|
|||
|
|
" <td>202.00</td>\n",
|
|||
|
|
" <td>10500.0</td>\n",
|
|||
|
|
" <td>2121000.0</td>\n",
|
|||
|
|
" <td>64259.0</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>8</th>\n",
|
|||
|
|
" <td>193.00</td>\n",
|
|||
|
|
" <td>3750.0</td>\n",
|
|||
|
|
" <td>723750.0</td>\n",
|
|||
|
|
" <td>196661.0</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>9</th>\n",
|
|||
|
|
" <td>20.91</td>\n",
|
|||
|
|
" <td>10500.0</td>\n",
|
|||
|
|
" <td>219555.0</td>\n",
|
|||
|
|
" <td>74759.0</td>\n",
|
|||
|
|
" <td>True</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>...</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>False</td>\n",
|
|||
|
|
" <td>low</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </tbody>\n",
|
|||
|
|
"</table>\n",
|
|||
|
|
"<p>10 rows × 184 columns</p>\n",
|
|||
|
|
"</div>"
|
|||
|
|
],
|
|||
|
|
"text/plain": [
|
|||
|
|
" Cost Shares Value ($) Shares Total Insider Trading_Baglino Andrew D \\\n",
|
|||
|
|
"0 196.72 10455.0 2056775.0 203073.0 False \n",
|
|||
|
|
"1 195.79 2466.0 482718.0 100458.0 False \n",
|
|||
|
|
"2 195.79 1298.0 254232.0 65547.0 True \n",
|
|||
|
|
"3 0.00 7138.0 0.0 102923.0 False \n",
|
|||
|
|
"4 0.00 2586.0 0.0 66845.0 True \n",
|
|||
|
|
"5 0.00 16867.0 0.0 213528.0 False \n",
|
|||
|
|
"6 20.91 10500.0 219555.0 74759.0 True \n",
|
|||
|
|
"7 202.00 10500.0 2121000.0 64259.0 True \n",
|
|||
|
|
"8 193.00 3750.0 723750.0 196661.0 False \n",
|
|||
|
|
"9 20.91 10500.0 219555.0 74759.0 True \n",
|
|||
|
|
"\n",
|
|||
|
|
" Insider Trading_DENHOLM ROBYN M Insider Trading_Kirkhorn Zachary \\\n",
|
|||
|
|
"0 False True \n",
|
|||
|
|
"1 False False \n",
|
|||
|
|
"2 False False \n",
|
|||
|
|
"3 False False \n",
|
|||
|
|
"4 False False \n",
|
|||
|
|
"5 False True \n",
|
|||
|
|
"6 False False \n",
|
|||
|
|
"7 False False \n",
|
|||
|
|
"8 False True \n",
|
|||
|
|
"9 False False \n",
|
|||
|
|
"\n",
|
|||
|
|
" Insider Trading_Musk Elon Insider Trading_Musk Kimbal \\\n",
|
|||
|
|
"0 False False \n",
|
|||
|
|
"1 False False \n",
|
|||
|
|
"2 False False \n",
|
|||
|
|
"3 False False \n",
|
|||
|
|
"4 False False \n",
|
|||
|
|
"5 False False \n",
|
|||
|
|
"6 False False \n",
|
|||
|
|
"7 False False \n",
|
|||
|
|
"8 False False \n",
|
|||
|
|
"9 False False \n",
|
|||
|
|
"\n",
|
|||
|
|
" Insider Trading_Taneja Vaibhav ... SEC Form 4_Nov 30 04:42 PM \\\n",
|
|||
|
|
"0 False ... False \n",
|
|||
|
|
"1 True ... False \n",
|
|||
|
|
"2 False ... False \n",
|
|||
|
|
"3 True ... False \n",
|
|||
|
|
"4 False ... False \n",
|
|||
|
|
"5 False ... False \n",
|
|||
|
|
"6 False ... False \n",
|
|||
|
|
"7 False ... False \n",
|
|||
|
|
"8 False ... False \n",
|
|||
|
|
"9 False ... False \n",
|
|||
|
|
"\n",
|
|||
|
|
" SEC Form 4_Oct 05 07:35 PM SEC Form 4_Oct 31 07:06 PM \\\n",
|
|||
|
|
"0 False False \n",
|
|||
|
|
"1 False False \n",
|
|||
|
|
"2 False False \n",
|
|||
|
|
"3 False False \n",
|
|||
|
|
"4 False False \n",
|
|||
|
|
"5 False False \n",
|
|||
|
|
"6 False False \n",
|
|||
|
|
"7 False False \n",
|
|||
|
|
"8 False False \n",
|
|||
|
|
"9 False False \n",
|
|||
|
|
"\n",
|
|||
|
|
" SEC Form 4_Sep 07 08:29 PM SEC Form 4_Sep 07 08:33 PM \\\n",
|
|||
|
|
"0 False False \n",
|
|||
|
|
"1 False False \n",
|
|||
|
|
"2 False False \n",
|
|||
|
|
"3 False False \n",
|
|||
|
|
"4 False False \n",
|
|||
|
|
"5 False False \n",
|
|||
|
|
"6 False False \n",
|
|||
|
|
"7 False False \n",
|
|||
|
|
"8 False False \n",
|
|||
|
|
"9 False False \n",
|
|||
|
|
"\n",
|
|||
|
|
" SEC Form 4_Sep 07 09:04 PM SEC Form 4_Sep 12 09:44 PM \\\n",
|
|||
|
|
"0 False False \n",
|
|||
|
|
"1 False False \n",
|
|||
|
|
"2 False False \n",
|
|||
|
|
"3 False False \n",
|
|||
|
|
"4 False False \n",
|
|||
|
|
"5 False False \n",
|
|||
|
|
"6 False False \n",
|
|||
|
|
"7 False False \n",
|
|||
|
|
"8 False False \n",
|
|||
|
|
"9 False False \n",
|
|||
|
|
"\n",
|
|||
|
|
" SEC Form 4_Sep 14 07:47 PM SEC Form 4_Sep 30 07:03 PM Cost_category \n",
|
|||
|
|
"0 False False medium \n",
|
|||
|
|
"1 False False medium \n",
|
|||
|
|
"2 False False medium \n",
|
|||
|
|
"3 False False low \n",
|
|||
|
|
"4 False False low \n",
|
|||
|
|
"5 False False low \n",
|
|||
|
|
"6 False False low \n",
|
|||
|
|
"7 False False medium \n",
|
|||
|
|
"8 False False medium \n",
|
|||
|
|
"9 False False low \n",
|
|||
|
|
"\n",
|
|||
|
|
"[10 rows x 184 columns]"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 648,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"df_encoded.head(10)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"2. **Дискретизация числовых признаков** – процесс преобразования непрерывных числовых значений в категориальные группы или интервалы (дискретные значения).\n",
|
|||
|
|
"\n",
|
|||
|
|
"В данном случае преобразование числовой колонки \"Cost\" уже было выполнено ранее для стратифицированного разбиения исходных данных на выборки (обучающую, контрольную и тестовую). Для этого использовался метод квартильной группировки.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Границы интервалов определены следующим образом:\n",
|
|||
|
|
"- Минус 1 от минимального значения стоимости акций (чтобы включить все значения в первый интервал).\n",
|
|||
|
|
"- 25-й перцентиль (quantile(0.25)), разделяющий самый низкий 25% стоимостей от остальных.\n",
|
|||
|
|
"- 75-й перцентиль (quantile(0.75)), разделяющий 25% с наиболее высокими стоимостью акций.\n",
|
|||
|
|
"- Плюс 1 от максимального значения стоимости акций (чтобы включить максимальное значение в последний интервал)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 649,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Обучающая выборка:\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/html": [
|
|||
|
|
"<div>\n",
|
|||
|
|
"<style scoped>\n",
|
|||
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|||
|
|
" vertical-align: middle;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe tbody tr th {\n",
|
|||
|
|
" vertical-align: top;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe thead th {\n",
|
|||
|
|
" text-align: right;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"</style>\n",
|
|||
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|||
|
|
" <thead>\n",
|
|||
|
|
" <tr style=\"text-align: right;\">\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th>Cost</th>\n",
|
|||
|
|
" <th>Cost_category</th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </thead>\n",
|
|||
|
|
" <tbody>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>0</th>\n",
|
|||
|
|
" <td>889.08</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>1</th>\n",
|
|||
|
|
" <td>20.91</td>\n",
|
|||
|
|
" <td>low</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>2</th>\n",
|
|||
|
|
" <td>62.72</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>3</th>\n",
|
|||
|
|
" <td>189.50</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>4</th>\n",
|
|||
|
|
" <td>1026.75</td>\n",
|
|||
|
|
" <td>high</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>5</th>\n",
|
|||
|
|
" <td>935.75</td>\n",
|
|||
|
|
" <td>high</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>6</th>\n",
|
|||
|
|
" <td>913.26</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>7</th>\n",
|
|||
|
|
" <td>62.72</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>8</th>\n",
|
|||
|
|
" <td>0.00</td>\n",
|
|||
|
|
" <td>low</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>9</th>\n",
|
|||
|
|
" <td>269.39</td>\n",
|
|||
|
|
" <td>medium</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </tbody>\n",
|
|||
|
|
"</table>\n",
|
|||
|
|
"</div>"
|
|||
|
|
],
|
|||
|
|
"text/plain": [
|
|||
|
|
" Cost Cost_category\n",
|
|||
|
|
"0 889.08 medium\n",
|
|||
|
|
"1 20.91 low\n",
|
|||
|
|
"2 62.72 medium\n",
|
|||
|
|
"3 189.50 medium\n",
|
|||
|
|
"4 1026.75 high\n",
|
|||
|
|
"5 935.75 high\n",
|
|||
|
|
"6 913.26 medium\n",
|
|||
|
|
"7 62.72 medium\n",
|
|||
|
|
"8 0.00 low\n",
|
|||
|
|
"9 269.39 medium"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 649,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"print('Обучающая выборка:')\n",
|
|||
|
|
"df_train_oversampled[['Cost', 'Cost_category']].head(10)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"3. **«Ручной» синтез признаков** – процесс создания новых признаков на основе существующих данных. Это может включать в себя комбинирование нескольких признаков, использование математических операций (например, сложение, вычитание), а также создание полиномиальных или логарифмических признаков."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 650,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/html": [
|
|||
|
|
"<div>\n",
|
|||
|
|
"<style scoped>\n",
|
|||
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|||
|
|
" vertical-align: middle;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe tbody tr th {\n",
|
|||
|
|
" vertical-align: top;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe thead th {\n",
|
|||
|
|
" text-align: right;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"</style>\n",
|
|||
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|||
|
|
" <thead>\n",
|
|||
|
|
" <tr style=\"text-align: right;\">\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th>Date</th>\n",
|
|||
|
|
" <th>Year</th>\n",
|
|||
|
|
" <th>Quarter</th>\n",
|
|||
|
|
" <th>Month</th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </thead>\n",
|
|||
|
|
" <tbody>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>0</th>\n",
|
|||
|
|
" <td>2022-03-06</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>1</th>\n",
|
|||
|
|
" <td>2022-03-06</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>2</th>\n",
|
|||
|
|
" <td>2022-03-06</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>3</th>\n",
|
|||
|
|
" <td>2022-03-05</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>4</th>\n",
|
|||
|
|
" <td>2022-03-05</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>5</th>\n",
|
|||
|
|
" <td>2022-03-05</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>6</th>\n",
|
|||
|
|
" <td>2022-02-27</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>2</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>7</th>\n",
|
|||
|
|
" <td>2022-02-27</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>2</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>8</th>\n",
|
|||
|
|
" <td>2022-02-06</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>2</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>9</th>\n",
|
|||
|
|
" <td>2022-01-27</td>\n",
|
|||
|
|
" <td>2022</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </tbody>\n",
|
|||
|
|
"</table>\n",
|
|||
|
|
"</div>"
|
|||
|
|
],
|
|||
|
|
"text/plain": [
|
|||
|
|
" Date Year Quarter Month\n",
|
|||
|
|
"0 2022-03-06 2022 1 3\n",
|
|||
|
|
"1 2022-03-06 2022 1 3\n",
|
|||
|
|
"2 2022-03-06 2022 1 3\n",
|
|||
|
|
"3 2022-03-05 2022 1 3\n",
|
|||
|
|
"4 2022-03-05 2022 1 3\n",
|
|||
|
|
"5 2022-03-05 2022 1 3\n",
|
|||
|
|
"6 2022-02-27 2022 1 2\n",
|
|||
|
|
"7 2022-02-27 2022 1 2\n",
|
|||
|
|
"8 2022-02-06 2022 1 2\n",
|
|||
|
|
"9 2022-01-27 2022 1 1"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 650,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"df['Date'] = pd.to_datetime(df['Date']) # Преобразование в datetime\n",
|
|||
|
|
"df['Year'] = df['Date'].dt.year # Год\n",
|
|||
|
|
"df['Quarter'] = df['Date'].dt.quarter # Квартал\n",
|
|||
|
|
"df['Month'] = df['Date'].dt.month # Месяц\n",
|
|||
|
|
"\n",
|
|||
|
|
"df[['Date', 'Year', 'Quarter', 'Month']].head(10)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"4. **Масштабирование признаков на основе нормировки и стандартизации** – метод, который позволяет привести все числовые признаки к одинаковым или очень похожим диапазонам значений либо распределениям.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Методы масштабирования признаков:**\n",
|
|||
|
|
"- **Нормировка** – обычно применяется для равномерного распределения;\n",
|
|||
|
|
"- **Стандартизация** – обычно применяется для нормального распределения."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 651,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/html": [
|
|||
|
|
"<div>\n",
|
|||
|
|
"<style scoped>\n",
|
|||
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|||
|
|
" vertical-align: middle;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe tbody tr th {\n",
|
|||
|
|
" vertical-align: top;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe thead th {\n",
|
|||
|
|
" text-align: right;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"</style>\n",
|
|||
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|||
|
|
" <thead>\n",
|
|||
|
|
" <tr style=\"text-align: right;\">\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th>Cost</th>\n",
|
|||
|
|
" <th>Shares</th>\n",
|
|||
|
|
" <th>Value ($)</th>\n",
|
|||
|
|
" <th>Shares Total</th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </thead>\n",
|
|||
|
|
" <tbody>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>0</th>\n",
|
|||
|
|
" <td>-0.630340</td>\n",
|
|||
|
|
" <td>-0.607179</td>\n",
|
|||
|
|
" <td>-0.583446</td>\n",
|
|||
|
|
" <td>-0.528366</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>1</th>\n",
|
|||
|
|
" <td>-0.632418</td>\n",
|
|||
|
|
" <td>-0.635043</td>\n",
|
|||
|
|
" <td>-0.594307</td>\n",
|
|||
|
|
" <td>-0.604905</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>2</th>\n",
|
|||
|
|
" <td>-0.632418</td>\n",
|
|||
|
|
" <td>-0.639117</td>\n",
|
|||
|
|
" <td>-0.595883</td>\n",
|
|||
|
|
" <td>-0.630945</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>3</th>\n",
|
|||
|
|
" <td>-1.069956</td>\n",
|
|||
|
|
" <td>-0.618748</td>\n",
|
|||
|
|
" <td>-0.597637</td>\n",
|
|||
|
|
" <td>-0.603067</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>4</th>\n",
|
|||
|
|
" <td>-1.069956</td>\n",
|
|||
|
|
" <td>-0.634624</td>\n",
|
|||
|
|
" <td>-0.597637</td>\n",
|
|||
|
|
" <td>-0.629977</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>5</th>\n",
|
|||
|
|
" <td>-1.069956</td>\n",
|
|||
|
|
" <td>-0.584816</td>\n",
|
|||
|
|
" <td>-0.597637</td>\n",
|
|||
|
|
" <td>-0.520567</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>6</th>\n",
|
|||
|
|
" <td>-1.023228</td>\n",
|
|||
|
|
" <td>-0.607022</td>\n",
|
|||
|
|
" <td>-0.596122</td>\n",
|
|||
|
|
" <td>-0.624074</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>7</th>\n",
|
|||
|
|
" <td>-0.618541</td>\n",
|
|||
|
|
" <td>-0.607022</td>\n",
|
|||
|
|
" <td>-0.583003</td>\n",
|
|||
|
|
" <td>-0.631906</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>8</th>\n",
|
|||
|
|
" <td>-0.638653</td>\n",
|
|||
|
|
" <td>-0.630565</td>\n",
|
|||
|
|
" <td>-0.592643</td>\n",
|
|||
|
|
" <td>-0.533148</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>9</th>\n",
|
|||
|
|
" <td>-1.023228</td>\n",
|
|||
|
|
" <td>-0.607022</td>\n",
|
|||
|
|
" <td>-0.596122</td>\n",
|
|||
|
|
" <td>-0.624074</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </tbody>\n",
|
|||
|
|
"</table>\n",
|
|||
|
|
"</div>"
|
|||
|
|
],
|
|||
|
|
"text/plain": [
|
|||
|
|
" Cost Shares Value ($) Shares Total\n",
|
|||
|
|
"0 -0.630340 -0.607179 -0.583446 -0.528366\n",
|
|||
|
|
"1 -0.632418 -0.635043 -0.594307 -0.604905\n",
|
|||
|
|
"2 -0.632418 -0.639117 -0.595883 -0.630945\n",
|
|||
|
|
"3 -1.069956 -0.618748 -0.597637 -0.603067\n",
|
|||
|
|
"4 -1.069956 -0.634624 -0.597637 -0.629977\n",
|
|||
|
|
"5 -1.069956 -0.584816 -0.597637 -0.520567\n",
|
|||
|
|
"6 -1.023228 -0.607022 -0.596122 -0.624074\n",
|
|||
|
|
"7 -0.618541 -0.607022 -0.583003 -0.631906\n",
|
|||
|
|
"8 -0.638653 -0.630565 -0.592643 -0.533148\n",
|
|||
|
|
"9 -1.023228 -0.607022 -0.596122 -0.624074"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 651,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# scaler = MinMaxScaler()\n",
|
|||
|
|
"scaler = StandardScaler()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Применяем масштабирование к выбранным признакам\n",
|
|||
|
|
"df[numeric_columns] = scaler.fit_transform(df[numeric_columns])\n",
|
|||
|
|
"\n",
|
|||
|
|
"df[numeric_columns].head(10)\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"5. **С применением фреймворка FeatureTools** – библиотека для автоматизированного создания признаков (features) из структурированных данных. Подходит для задач машинного обучения, когда нужно быстро извлекать полезные признаки из больших объемов данных."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 664,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stderr",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"d:\\ULSTU\\Семестр 5\\AIM-PIbd-31-Masenkin-M-S\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/html": [
|
|||
|
|
"<div>\n",
|
|||
|
|
"<style scoped>\n",
|
|||
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|||
|
|
" vertical-align: middle;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe tbody tr th {\n",
|
|||
|
|
" vertical-align: top;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"\n",
|
|||
|
|
" .dataframe thead th {\n",
|
|||
|
|
" text-align: right;\n",
|
|||
|
|
" }\n",
|
|||
|
|
"</style>\n",
|
|||
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|||
|
|
" <thead>\n",
|
|||
|
|
" <tr style=\"text-align: right;\">\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th>Insider Trading</th>\n",
|
|||
|
|
" <th>Relationship</th>\n",
|
|||
|
|
" <th>Transaction</th>\n",
|
|||
|
|
" <th>Cost</th>\n",
|
|||
|
|
" <th>DAY(Date)</th>\n",
|
|||
|
|
" <th>MONTH(Date)</th>\n",
|
|||
|
|
" <th>WEEKDAY(Date)</th>\n",
|
|||
|
|
" <th>YEAR(Date)</th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>Id</th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" <th></th>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </thead>\n",
|
|||
|
|
" <tbody>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>154</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>1019.03</td>\n",
|
|||
|
|
" <td>10</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>2</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>155</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>1048.46</td>\n",
|
|||
|
|
" <td>10</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>2</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>156</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>1068.09</td>\n",
|
|||
|
|
" <td>10</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>2</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>152</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>1098.24</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>153</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>1072.22</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>3</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>151</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>1029.67</td>\n",
|
|||
|
|
" <td>12</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>4</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>148</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Option Exercise</td>\n",
|
|||
|
|
" <td>6.24</td>\n",
|
|||
|
|
" <td>15</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>0</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>149</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>992.72</td>\n",
|
|||
|
|
" <td>15</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>0</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>150</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Sale</td>\n",
|
|||
|
|
" <td>1015.85</td>\n",
|
|||
|
|
" <td>15</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>0</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" <tr>\n",
|
|||
|
|
" <th>145</th>\n",
|
|||
|
|
" <td>Musk Elon</td>\n",
|
|||
|
|
" <td>CEO</td>\n",
|
|||
|
|
" <td>Option Exercise</td>\n",
|
|||
|
|
" <td>6.24</td>\n",
|
|||
|
|
" <td>16</td>\n",
|
|||
|
|
" <td>11</td>\n",
|
|||
|
|
" <td>1</td>\n",
|
|||
|
|
" <td>2021</td>\n",
|
|||
|
|
" </tr>\n",
|
|||
|
|
" </tbody>\n",
|
|||
|
|
"</table>\n",
|
|||
|
|
"</div>"
|
|||
|
|
],
|
|||
|
|
"text/plain": [
|
|||
|
|
" Insider Trading Relationship Transaction Cost DAY(Date) \\\n",
|
|||
|
|
"Id \n",
|
|||
|
|
"154 Musk Elon CEO Sale 1019.03 10 \n",
|
|||
|
|
"155 Musk Elon CEO Sale 1048.46 10 \n",
|
|||
|
|
"156 Musk Elon CEO Sale 1068.09 10 \n",
|
|||
|
|
"152 Musk Elon CEO Sale 1098.24 11 \n",
|
|||
|
|
"153 Musk Elon CEO Sale 1072.22 11 \n",
|
|||
|
|
"151 Musk Elon CEO Sale 1029.67 12 \n",
|
|||
|
|
"148 Musk Elon CEO Option Exercise 6.24 15 \n",
|
|||
|
|
"149 Musk Elon CEO Sale 992.72 15 \n",
|
|||
|
|
"150 Musk Elon CEO Sale 1015.85 15 \n",
|
|||
|
|
"145 Musk Elon CEO Option Exercise 6.24 16 \n",
|
|||
|
|
"\n",
|
|||
|
|
" MONTH(Date) WEEKDAY(Date) YEAR(Date) \n",
|
|||
|
|
"Id \n",
|
|||
|
|
"154 11 2 2021 \n",
|
|||
|
|
"155 11 2 2021 \n",
|
|||
|
|
"156 11 2 2021 \n",
|
|||
|
|
"152 11 3 2021 \n",
|
|||
|
|
"153 11 3 2021 \n",
|
|||
|
|
"151 11 4 2021 \n",
|
|||
|
|
"148 11 0 2021 \n",
|
|||
|
|
"149 11 0 2021 \n",
|
|||
|
|
"150 11 0 2021 \n",
|
|||
|
|
"145 11 1 2021 "
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 664,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"df: DataFrame = pd.read_csv('..//static//csv//TSLA.csv')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Создание уникального идентификатора для каждой строки\n",
|
|||
|
|
"df['Id'] = range(1, len(df) + 1)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Создание EntitySet\n",
|
|||
|
|
"es = ft.EntitySet(id=\"Id\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Добавляем таблицу с индексом\n",
|
|||
|
|
"es: EntitySet = es.add_dataframe(\n",
|
|||
|
|
" dataframe_name=\"trades\", \n",
|
|||
|
|
" dataframe=df, \n",
|
|||
|
|
" index=\"Id\", \n",
|
|||
|
|
" time_index=\"Date\"\n",
|
|||
|
|
")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Генерация признаков с помощью глубокого синтеза признаков\n",
|
|||
|
|
"feature_matrix, feature_defs = ft.dfs(\n",
|
|||
|
|
" entityset=es, \n",
|
|||
|
|
" target_dataframe_name='trades', \n",
|
|||
|
|
" max_depth=1\n",
|
|||
|
|
")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выводим первые 10 строк сгенерированного набора признаков\n",
|
|||
|
|
"feature_matrix.head(10)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Оценка качества набора признаков:\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Предсказательная способность**: Способность набора признаков успешно прогнозировать целевую переменную. Это определяется через метрики, такие как RMSE, MAE, R², которые показывают, насколько хорошо модель использует признаки для достижения точных результатов. Для определения качества необходимо провести обучение модели на обучающей выборке и сравнить с оценкой прогнозирования на контрольной и тестовой выборках.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Скорость вычисления**: Время, необходимое для обработки данных и выполнения алгоритмов машинного обучения. Признаки должны быть вычисляемыми за разумный срок, чтобы обеспечить эффективность модели, особенно при работе с большими наборами данных. Для оценки качества необходимо провести измерение времени выполнения генерации признаков и обучения модели.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Надежность**: Устойчивость и воспроизводимость результатов при изменении входных данных. Надежные признаки должны давать схожие результаты независимо от случайных факторов или незначительных изменений в данных. Методы оценки: Кросс-валидация, анализ чувствительности модели к изменениям в данных.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Корреляция**: Степень взаимосвязи между признаками и целевой переменной, а также между самими признаками. Высокая корреляция с целевой переменной указывает на потенциальную предсказательную силу, тогда как высокая взаимосвязь между самими признаками может приводить к многоколлинеарности и снижению эффективности модели. Методы оценки: Анализ корреляционной матрицы признаков, удаление мультиколлинеарных признаков.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Цельность**: Не является производным от других признаков. Методы оценки: Проверка логической связи между признаками и целевой переменной, интерпретация результатов модели."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 653,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Время обучения модели: 0.30 секунд\n",
|
|||
|
|
"Среднеквадратичная ошибка: 195.01\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Разбить выборку на входные данные и целевой признак\n",
|
|||
|
|
"def split_dataframe(dataframe: DataFrame, column: str) -> tuple[DataFrame, DataFrame]:\n",
|
|||
|
|
" X_dataframe: DataFrame = dataframe.drop(columns=column, axis=1)\n",
|
|||
|
|
" y_dataframe: DataFrame = dataframe[column]\n",
|
|||
|
|
" \n",
|
|||
|
|
" return X_dataframe, y_dataframe\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение обучающей выборки на входные данные и целевой признак\n",
|
|||
|
|
"df_train_oversampled: DataFrame = pd.get_dummies(df_train_oversampled)\n",
|
|||
|
|
"X_df_train, y_df_train = split_dataframe(df_train_oversampled, \"Cost\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение контрольной выборки на входные данные и целевой признак\n",
|
|||
|
|
"df_val_oversampled: DataFrame = pd.get_dummies(df_val_oversampled)\n",
|
|||
|
|
"X_df_val, y_df_val = split_dataframe(df_val_oversampled, \"Cost\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение тестовой выборки на входные данные и целевой признак\n",
|
|||
|
|
"df_test_oversampled: DataFrame = pd.get_dummies(df_test_oversampled)\n",
|
|||
|
|
"X_df_test, y_df_test = split_dataframe(df_test_oversampled, \"Cost\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Модель линейной регрессии для обучения\n",
|
|||
|
|
"model = LinearRegression()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Начинаем отсчет времени\n",
|
|||
|
|
"start_time: float = time.time()\n",
|
|||
|
|
"model.fit(X_df_train, y_df_train)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Время обучения модели\n",
|
|||
|
|
"train_time: float = time.time() - start_time\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Предсказания и оценка модели\n",
|
|||
|
|
"predictions = model.predict(X_df_val)\n",
|
|||
|
|
"mse = root_mean_squared_error(y_df_val, predictions)\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f'Время обучения модели: {train_time:.2f} секунд')\n",
|
|||
|
|
"print(f'Среднеквадратичная ошибка: {mse:.2f}')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 654,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"RMSE: 89.7060467247578\n",
|
|||
|
|
"R²: 0.9632863955859148\n",
|
|||
|
|
"MAE: 51.21015516441333\n",
|
|||
|
|
"\n",
|
|||
|
|
"Кросс-валидация RMSE: 148.73333849768954\n",
|
|||
|
|
"\n",
|
|||
|
|
"Train RMSE: 49.74051197142329\n",
|
|||
|
|
"Train R²: 0.9887155168427046\n",
|
|||
|
|
"Train MAE: 22.617172692464216\n",
|
|||
|
|
"\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/plain": [
|
|||
|
|
"Text(0.5, 1.0, 'Фактическая стоимость по сравнению с прогнозируемой')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 654,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1sAAAIjCAYAAAD1OgEdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACoMklEQVR4nOzdd1iT59cH8G8GIcwgyhJRVBDFvetuXWjdtfpz7y1ubaXWbdW6Zx0daqvW1lHraJ11Fa2rWgUH4JYpRVYYIcnz/uHLUyOgBINhfD/XxVVznifJyaKc3Pd9bokgCAKIiIiIiIjIpKTmToCIiIiIiKgoYrFFRERERESUD1hsERERERER5QMWW0RERERERPmAxRYREREREVE+YLFFRERERESUD1hsERERERER5QMWW0RERERERPmAxRYREREREVE+YLFFRERERMXO9u3b8fDhQ/Hy1q1bER4ebr6EqEhisUWUjwYNGgRbW1tzp0FERESvOHfuHD755BM8fPgQR48exdixYyGV8k9jMi25uRMgKmr+/fdf7NixA+fOncPZs2eRmpqKdu3aoXbt2ujZsydq165t7hSJiIiKvUmTJuH9999H+fLlAQCTJ0+Gm5ubmbOiokYiCIJg7iSIiopdu3Zh+PDhSE5OhqenJzIyMhAVFYXatWvjn3/+QUZGBgYOHIjNmzdDoVCYO10iIqJiTa1WIygoCKVKlULFihXNnQ4VQRwrJTKRwMBA9OvXD66urggMDMSDBw/QunVrKJVKXL58GREREejduze2bduGSZMmGVx32bJlaNy4MUqWLAkrKyvUrVsXe/bsyXIfEokEc+bMES9rtVp8+OGHcHR0xK1bt8RzXvfz/vvvAwBOnz4NiUSC06dPG9xHhw4dstzP+++/L14v08OHDyGRSLB161aD+J07d/Dxxx/D0dERSqUS9erVw4EDB7I8lvj4eEyaNAmenp6wtLREmTJlMGDAAMTGxuaYX0REBDw9PVGvXj0kJycDADQaDWbNmoW6detCpVLBxsYGzZo1w6lTp7LcZ0xMDIYOHYqyZctCJpOJz0lup3r+/vvvaNGiBezs7GBvb4/69etj586d4nP0puc+k1arxfz581GxYkVYWlrC09MTn332GdLT0w3uz9PTE4MGDTKI7d69GxKJBJ6enmIs87WQSCTYv3+/wflpaWkoUaIEJBIJli1bZnDs2rVraN++Pezt7WFra4tWrVrhr7/+yvK4X/daZb5Or/vJfC/NmTMHEolEfI2NkXndnH5efR/+8ccfaNasGWxsbODg4IAuXbrg9u3bubqvtLQ0zJkzB5UqVYJSqYSbmxs++ugj3Lt3D8B/z/eyZcuwcuVKlCtXDlZWVmjRogWCgoIMbuvGjRsYNGgQKlSoAKVSCVdXVwwZMgT//vvvax+fnZ0dGjRokOX1fP/991GtWrUsOS9btgwSicRg/Qnw4j2b+TzY2dmhQ4cOCA4ONjgnp+nOe/bsyfIZzO53weXLl7O8x4Gs79+kpCT4+/vD3d0dlpaW8Pb2xuLFi6HX67Pcd3YuXryIDz/8ECVKlICNjQ1q1KiB1atXv/Y6W7duzdV7E/jvNbhz5w569uwJe3t7lCxZEhMmTEBaWprB7RrzGc7ufocNGwbA8L30qmrVqmV5rjN/h7m4uECpVKJmzZrYtm2bwTl3795Fy5Yt4erqCktLS3h4eGDUqFGIi4sTzzHm939un5cWLVqgZs2a2b4OPj4+8PPzEy/r9XqsWrUKVatWhVKphIuLC0aOHInnz59n+/xNnDgxy236+flBIpGgY8eORj9HL///y8bGBg0bNkTFihUxduxYSCSSLL93id4GpxESmUjmHw27du1C3bp1sxwvVaoUvv/+e9y6dQubNm3C7Nmz4ezsDABYvXo1OnfujL59+0Kj0WDXrl3o0aMHDh06hA4dOuR4n8OGDcPp06dx/Phx+Pr6AgB++OEH8fi5c+ewefNmrFy5EqVKlQIAuLi45Hh7Z8+exW+//Zanxw8AwcHBaNKkCdzd3TF9+nTY2Njg559/RteuXbF3715069YNAJCcnIxmzZrh9u3bGDJkCOrUqYPY2FgcOHAAT58+FXN9WUJCAtq3bw8LCwv89ttv4h+HiYmJ+Oabb9C7d28MHz4cSUlJ+Pbbb+Hn54dLly6hVq1a4m0MHDgQJ06cwLhx41CzZk3IZDJs3rwZf//99xsf29atWzFkyBBUrVoVAQEBcHBwwLVr13DkyBH06dMHM2bMEP+Aio2NxaRJkzBixAg0a9Ysy20NGzYM27Ztw8cff4wpU6bg4sWLWLRoEW7fvo1ffvklxxy0Wi1mzJiR43GlUoktW7aga9euYmzfvn1Z/lAEXrxWzZo1g729PT755BNYWFhg06ZNeP/993HmzBk0bNgQwJtfqypVqhi85zZv3ozbt29j5cqVYqxGjRo5P7FG2rBhg0Fh8ODBA8yaNcvgnBMnTqB9+/aoUKEC5syZg9TUVKxduxZNmjTB33//bVCovkqn06Fjx444efIkevXqhQkTJiApKQnHjx9HUFCQwTff33//PZKSkjB27FikpaVh9erVaNmyJW7evCl+zo4fP4779+9j8ODBcHV1RXBwMDZv3ozg4GD89ddfWQqUzOcyNjYWX331FXr06IGgoCD4+PgY/Vz98MMPGDhwIPz8/PDll18iJSUFGzZsQNOmTXHt2rXXPg/G+PTTT3N1Xvfu3XH8+HEMGDAADRo0wKlTpxAQEICHDx9i48aNr73u8ePH0bFjR7i5uWHChAlwdXXF7du3cejQIUyYMOGN9z1v3jxxqhjw4n09evTobM/t2bMnPD09sWjRIvz1119Ys2YNnj9/ju+//148x5jPcK1atTBlyhSDmJeX1xtzflVqairef/99hIWFwd/fH+XLl8fu3bsxaNAgxMfHi8+DWq1GmTJl0KlTJ9jb2yMoKAjr169HeHg4Dh48mOPtv+n3/5uel/79+2P48OEICgoy+ELg8uXLCAkJweeffy7GRo4cia1bt2Lw4MEYP348Hjx4gHXr1uHatWsIDAyEhYWFeK5SqcSOHTuwdOlSMf706VOcPHkSSqUyT89RdsLCwvD111/neJwozwQiMglHR0ehXLlyBrGBAwcKNjY2BrGZM2cKAISDBw+KsZSUFINzNBqNUK1aNaFly5YGcQDC7NmzBUEQhICAAEEmkwn79+/PMactW7YIAIQHDx5kOXbq1CkBgHDq1Ckx1rBhQ6F9+/YG9yMIgvDBBx8IzZs3N7j+gwcPBADCli1bxFirVq2E6tWrC2lpaWJMr9cLjRs3Fry9vcXYrFmzBADCvn37suSl1+uz5JeWlia8//77grOzsxAWFmZwvlarFdLT0w1iz58/F1xcXIQhQ4aIsdTUVEEqlQojR440ODe71+hV8fHxgp2dndCwYUMhNTU123xflt1zk+n69esCAGHYsGEG8alTpwoAhD/++EOMlStXThg4cKB4+auvvhIsLS2FDz74wOC9lnl/vXv3FuRyuRAVFSUea9WqldCnTx8BgLB06VIx3rVrV0GhUAj37t0TYxEREYKdnZ3Ba52b1+plAwcOzPI5yDR79mwBgPDs2bNsj79OTte9fPlylue6Vq1agrOzs/Dvv/+KsX/++UeQSqXCgAEDXns/3333nQBAWLFiRZZjmY838/m2srISnj59Kh6/ePGiAECYNGmSGHv1sy0IgvDjjz8KAISzZ89meXwvO3bsmABA+Pnnn8VYixYthKpVq2a5zaVLlxp81pOSkgQHBwdh+PDhBudFRUUJKpXKIJ7TZ2D37t1Zfke0aNFCaNGihXj5t99+EwAI7dq1y5L/y+/fgwcPCgCE6dOnG5wzaNAgAYBw8+bNLPefSavVCuXLlxfKlSsnPH/+3OBYdu/Bl2X+Drx8+bJB/NmzZ1l+z2W+Bp07dzY4d8yYMQIA4Z9//hEEwfjPcIcOHXLML/O99PJnM1PVqlUNnutVq1YJAITt27eLMY1GIzRq1EiwtbUVEhMTc7yfMWPGCLa2tuJlY37/5/Z5iY+PF5RKpfDpp58anDd+/HjBxsZ
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1000x600 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Модель случайного леса для обучения\n",
|
|||
|
|
"model = RandomForestRegressor()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Обучение модели\n",
|
|||
|
|
"model.fit(X_df_train, y_df_train)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Предсказание и оценка\n",
|
|||
|
|
"y_predictions = model.predict(X_df_test)\n",
|
|||
|
|
"\n",
|
|||
|
|
"rmse = root_mean_squared_error(y_df_test, y_predictions)\n",
|
|||
|
|
"r2 = r2_score(y_df_test, y_predictions)\n",
|
|||
|
|
"mae = mean_absolute_error(y_df_test, y_predictions)\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f\"RMSE: {rmse}\")\n",
|
|||
|
|
"print(f\"R²: {r2}\")\n",
|
|||
|
|
"print(f\"MAE: {mae}\\n\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Кросс-валидация\n",
|
|||
|
|
"scores = cross_val_score(model, X_df_train, y_df_train, cv=5, scoring='neg_mean_squared_error')\n",
|
|||
|
|
"rmse_cv = (-scores.mean())**0.5\n",
|
|||
|
|
"print(f\"Кросс-валидация RMSE: {rmse_cv}\\n\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Анализ важности признаков\n",
|
|||
|
|
"feature_importances = model.feature_importances_\n",
|
|||
|
|
"feature_names = X_df_train.columns\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка на переобучение\n",
|
|||
|
|
"y_train_predictions = model.predict(X_df_train)\n",
|
|||
|
|
"\n",
|
|||
|
|
"rmse_train = root_mean_squared_error(y_df_train, y_train_predictions)\n",
|
|||
|
|
"r2_train = r2_score(y_df_train, y_train_predictions)\n",
|
|||
|
|
"mae_train = mean_absolute_error(y_df_train, y_train_predictions)\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f\"Train RMSE: {rmse_train}\")\n",
|
|||
|
|
"print(f\"Train R²: {r2_train}\")\n",
|
|||
|
|
"print(f\"Train MAE: {mae_train}\\n\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Визуализация результатов\n",
|
|||
|
|
"plt.figure(figsize=(10, 6))\n",
|
|||
|
|
"plt.scatter(y_df_test, y_predictions, alpha=0.5)\n",
|
|||
|
|
"plt.plot([y_df_test.min(), y_df_test.max()], [y_df_test.min(), y_df_test.max()], 'k--', lw=2)\n",
|
|||
|
|
"plt.xlabel('Фактическая стоимость')\n",
|
|||
|
|
"plt.ylabel('Прогнозируемая стоимость')\n",
|
|||
|
|
"plt.title('Фактическая стоимость по сравнению с прогнозируемой')\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"### Вывод:\n",
|
|||
|
|
"\n",
|
|||
|
|
"1. **Оценка качества модели на тестовой выборке:**\n",
|
|||
|
|
" - **RMSE** (Корень из среднеквадратичной ошибки) на тестовой выборке составил 89.71, что указывает на среднюю ошибку в прогнозах.\n",
|
|||
|
|
" - **R²** (Коэффициент детерминации) равен 0.96, что означает, что модель объясняет 96% дисперсии данных. Это хороший показатель, указывающий на высокую объяснительную способность модели.\n",
|
|||
|
|
" - **MAE** (Средняя абсолютная ошибка) составила 51.21, показывая среднее абсолютное отклонение предсказаний от фактических значений.\n",
|
|||
|
|
"\n",
|
|||
|
|
"2. **Результаты кросс-валидации:**\n",
|
|||
|
|
" - **RMSE кросс-валидации** равен 148.73, что заметно выше значения RMSE на тестовой выборке. Это может свидетельствовать о том, что модель может быть подвержена колебаниям в зависимости от данных и, возможно, склонна к некоторому переобучению.\n",
|
|||
|
|
"\n",
|
|||
|
|
"3. **Проверка на переобучение:**\n",
|
|||
|
|
" - **Метрики на обучающей выборке** (RMSE = 49.74, R² = 0.99, MAE = 22.62) значительно лучше, чем на тестовой, что указывает на высокую точность на обучающих данных.\n",
|
|||
|
|
" - Разница в ошибках между обучающей и тестовой выборками говорит о возможном переобучении. Модель хорошо подходит к обучающим данным, но на тестовой выборке дает несколько более слабые результаты, что типично для моделей случайного леса, когда модель выучивает мелкие детали обучающего набора.\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Итог:**\n",
|
|||
|
|
"Модель случайного леса демонстрирует хорошее качество прогнозов с высоким коэффициентом детерминации на тестовых данных. Однако разница в ошибках на обучающей и тестовой выборках вместе с результатами кросс-валидации указывает на возможное переобучение."
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"metadata": {
|
|||
|
|
"kernelspec": {
|
|||
|
|
"display_name": "aimenv",
|
|||
|
|
"language": "python",
|
|||
|
|
"name": "python3"
|
|||
|
|
},
|
|||
|
|
"language_info": {
|
|||
|
|
"codemirror_mode": {
|
|||
|
|
"name": "ipython",
|
|||
|
|
"version": 3
|
|||
|
|
},
|
|||
|
|
"file_extension": ".py",
|
|||
|
|
"mimetype": "text/x-python",
|
|||
|
|
"name": "python",
|
|||
|
|
"nbconvert_exporter": "python",
|
|||
|
|
"pygments_lexer": "ipython3",
|
|||
|
|
"version": "3.12.5"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"nbformat": 4,
|
|||
|
|
"nbformat_minor": 2
|
|||
|
|
}
|