844 lines
215 KiB
Plaintext
844 lines
215 KiB
Plaintext
|
{
|
|||
|
|
"cells": [
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"## **СТОЛБЦЫ ДАТАСЕТА:**\n",
|
|||
|
|
"\n",
|
|||
|
|
"**Id**\n",
|
|||
|
|
"\n",
|
|||
|
|
"**name**\n",
|
|||
|
|
"\n",
|
|||
|
|
"**est_diameter_min** – минимальный диаметр косм. объекта (астероид, комета) рядом с Землёй (км)\n",
|
|||
|
|
"\n",
|
|||
|
|
"**est_diameter_max** – максимальный диаметр косм. объекта\n",
|
|||
|
|
"\n",
|
|||
|
|
"**relative_velocity** – скорость относительно Земли (км/с)\n",
|
|||
|
|
"\n",
|
|||
|
|
"**miss_distance** – расстояние, на кот. проходит рядом с Землёй (км)\n",
|
|||
|
|
"\n",
|
|||
|
|
"**orbiting_body** – тело, вокруг которого вращается (везде Земля)\n",
|
|||
|
|
"\n",
|
|||
|
|
"**sentry_object** – ведётся ли за ним авто мониторинг (везде false)\n",
|
|||
|
|
"\n",
|
|||
|
|
"**absolute_magnitude** – звёздная величина (яркость)\n",
|
|||
|
|
"\n",
|
|||
|
|
"**hazardous** – опасный для Земли / нет"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 254,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Index(['id', 'name', 'est_diameter_min', 'est_diameter_max',\n",
|
|||
|
|
" 'relative_velocity', 'miss_distance', 'orbiting_body', 'sentry_object',\n",
|
|||
|
|
" 'absolute_magnitude', 'hazardous'],\n",
|
|||
|
|
" dtype='object')\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"import numpy as np\n",
|
|||
|
|
"\n",
|
|||
|
|
"df = pd.read_csv(\"..//static//csv//neo.csv\")\n",
|
|||
|
|
"print(df.columns)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Бизнес-цели:\n",
|
|||
|
|
"\n",
|
|||
|
|
"1) Увеличение безопасности Земли\n",
|
|||
|
|
"\n",
|
|||
|
|
"2) Увеличение информации о безопасных космических объектах, на поверхности которых можно проводить исследования \n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"Цели технического проекта:\n",
|
|||
|
|
"\n",
|
|||
|
|
"1) Построить модель машинного обучения, которая сможет предсказать, нужно ли вести наблюдения за космическим объектом, как за опасным для Земли. Вход: est_diameter_min, est_diameter_max, relative_velocity, miss_distance, absolute_magnitude, hazardous. Целевой признак: hazardous\n",
|
|||
|
|
"\n",
|
|||
|
|
"2) Построить модель машинного обучения, которая сможет предсказать, безопасный ли объект для проведения исследований на его поверхности. Вход: est_diameter_min, est_diameter_max, relative_velocity, miss_distance, absolute_magnitude, hazardous. Целевой признак: hazardous"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**ПОДГОТОВКА ДАННЫХ**\n",
|
|||
|
|
"\n",
|
|||
|
|
"Т.к. hazardous является целевым признаком, то необходимо, чтобы данных об опасных и безопасных объектах было примерно равное количество. Безопасных объектов гораздо больше, поэтому можно произвести undersampling для того, чтобы количество безопасных и опасных объектов стало равным "
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 255,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"НАЧАЛЬНОЕ РАСПРЕДЕЛЕНИЕ\n",
|
|||
|
|
"Кол-во объектов с hazardous = true: 8840\n",
|
|||
|
|
"Кол-во объектов с hazardous = false: 81996\n",
|
|||
|
|
"\n",
|
|||
|
|
"СБАЛАНСИРОВАННОЕ РАСПРЕДЕЛЕНИЕ\n",
|
|||
|
|
"Кол-во объектов с hazardous = true: 8840\n",
|
|||
|
|
"Кол-во объектов с hazardous = false: 8840\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"\n",
|
|||
|
|
"print('НАЧАЛЬНОЕ РАСПРЕДЕЛЕНИЕ')\n",
|
|||
|
|
"true_hazardous = df[df['hazardous'] == True].shape[0]\n",
|
|||
|
|
"print(f'Кол-во объектов с hazardous = true: {true_hazardous}')\n",
|
|||
|
|
"\n",
|
|||
|
|
"false_hazardous = df[df['hazardous'] == False].shape[0]\n",
|
|||
|
|
"print(f'Кол-во объектов с hazardous = false: {false_hazardous}')\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"true_count = df[df['hazardous'] == True].shape[0]\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"false_count = df[df['hazardous'] == False].shape[0]\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# разделение датасета на 2 части по hazardous\n",
|
|||
|
|
"from sklearn.utils import resample\n",
|
|||
|
|
"df_hazardous_true = df[df['hazardous'] == True]\n",
|
|||
|
|
"df_hazardous_false = df[df['hazardous'] == False]\n",
|
|||
|
|
"\n",
|
|||
|
|
"# undersampling к части с Hazardous=false\n",
|
|||
|
|
"df_hazardous_false_undersampled = resample(df_hazardous_false, \n",
|
|||
|
|
" replace=False, \n",
|
|||
|
|
" n_samples=len(df_hazardous_true), #столько записей, сколько в df_hazardous_true\n",
|
|||
|
|
" random_state=123) \n",
|
|||
|
|
"\n",
|
|||
|
|
"# Объединение 2 частей\n",
|
|||
|
|
"df_balanced = pd.concat([df_hazardous_true, df_hazardous_false_undersampled])\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Перемешивание записей\n",
|
|||
|
|
"df = df_balanced.sample(frac=1, random_state=123).reset_index(drop=True)\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"print('\\nСБАЛАНСИРОВАННОЕ РАСПРЕДЕЛЕНИЕ')\n",
|
|||
|
|
"true_hazardous = df[df['hazardous'] == True].shape[0]\n",
|
|||
|
|
"print(f'Кол-во объектов с hazardous = true: {true_hazardous}')\n",
|
|||
|
|
"\n",
|
|||
|
|
"false_hazardous = df[df['hazardous'] == False].shape[0]\n",
|
|||
|
|
"print(f'Кол-во объектов с hazardous = false: {false_hazardous}')\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"\n",
|
|||
|
|
"Проверка на выбросы. В столбцах est_diameter_min, est_diameter_max, relative_velocity есть единичные выбросы, которые значительно больше основного распределения (est_diameter_min - которые больше примерно 3; est_diameter_max - которые больше примерно 5; relative_velocity - которые больше примерно 175000), и в столбце absolute_magnitude есть выбросы, которые значительно ниже основного распределния (примерно меньше 13). Эти единичные выбросы можно удалить. Остальные же значения, находящиеся за рамками основного распределения стоит оставить, т.к. они представляют из себя полезный шум\n",
|
|||
|
|
"\n",
|
|||
|
|
"После удаления единичных выбросов видно, что количество записей уменьшилось очень незначительно"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 256,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество записей до устранения выбросов: 17680\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABKUAAAMWCAYAAAAgRDUeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAClFUlEQVR4nOzdeVhU5fvH8Q8MssjmLvB1IzHBPdEUExU1ydQitdwqNcsWtcytrL6mZllaqZVLfiu1xTLNqLRcc6uwhbLU1MgkKwVLBZQQFJ7fH16cn8OioDAD+n5d11w6z3nmnHsO8Nwz9znnOS7GGCMAAAAAAADAgVydHQAAAAAAAACuPBSlAAAAAAAA4HAUpQAAAAAAAOBwFKUAAAAAAADgcBSlAAAAAAAA4HAUpQAAAAAAAOBwFKUAAAAAAADgcBSlAAAAAAAA4HAUpQAAAAAAAOBwFKWuIC4uLpo8ebJDtrVmzRq1aNFCnp6ecnFxUUpKikO2WxYNGTJE9erVs2tz5M/iSjV58mS5uLg4OwzgskQ+cQ7yCYArGbmncAXlh0u1ePFiubi4KDExsUTX60yJiYlycXHR4sWLS2X9mzdvlouLizZv3lwq679cUZQqAbl/sOc+atSooaioKH322WfODu+S/fzzz5o8eXKRB6SjR4/qtttuk5eXl+bOnau33npL3t7epRtkCfnqq680efLkMp94HGHevHmlNmADKBj5xB755PJAPgHKNnKPvfKcey7GM888o9jYWGeHcdlaunSpZs+e7ewwyjQ3ZwdwOZk6daqCg4NljFFycrIWL16sG2+8UZ988ol69uzp7PAu2s8//6wpU6aoU6dORarAf/vttzpx4oSeeuopde3atfQDLEFfffWVpkyZoiFDhqhSpUqltp2MjAy5uZXtP7958+apWrVqGjJkiLNDuShPPPGEHn30UWeHAVwU8slZ5JMLI58AKCnknrPKc+65GM8884z69u2rmJgYu/Y77rhD/fv3l4eHh3MCK4c6dOigjIwMubu7W21Lly7Vrl27NHr0aOcFVsaV7U8x5Uz37t3VqlUr6/mwYcNUs2ZNvfvuu+V6IC+uI0eOSFKJfghPT0+/rI5QeHp6OjsEpzhz5oxycnLsBurS4ubmVua/qAGFIZ+cRT65MPJJ6ecT4EpB7jmrLOSespCrbDabbDabU2Mob1xdXa/YvHwpuHyvFFWqVEleXl75vhinp6dr7Nixql27tjw8PNSwYUM9//zzMsZIOnvUMzQ0VKGhocrIyLBed+zYMQUGBqpdu3bKzs6WdPb6YR8fH/3222+Kjo6Wt7e3goKCNHXqVGt95/PDDz+oe/fu8vPzk4+Pj7p06aLt27dbyxcvXqxbb71VkhQVFWWd0lvYdbKdOnXS4MGDJUmtW7eWi4uL3ZHR5cuXKzw8XF5eXqpWrZpuv/12/fXXX3bryH1P+/fv14033ihfX18NGjTovO/jr7/+0l133aWaNWvKw8NDjRs31htvvJGv38svv6zGjRurYsWKqly5slq1aqWlS5dKOjsH0fjx4yVJwcHB1nstznXUsbGxatKkiTw9PdWkSRN9+OGHBfbLe03877//rgceeEANGzaUl5eXqlatqltvvTXftnNPr/7iiy/04IMPqnr16qpUqZLuvfdeZWVlKSUlRXfeeacqV66sypUra8KECfl+D3JycjR79mw1btxYnp6eqlmzpu69914dP37c6lOvXj3t3r1bW7ZssfZDp06drOUpKSkaPXq09TscEhKi5557Tjk5OVaf3Gu2n3/+ec2ePVv169eXh4eHfv755yLty3r16qlnz57avHmzWrVqJS8vLzVt2tT63Vu5cqWaNm0qT09PhYeH64cffrB7fUFzSrm4uGjkyJHWzyn3d2XNmjVFiglwFvIJ+YR84rx88tNPP2nIkCG66qqr5OnpqYCAAN111106evSo1ac4f2tAeUHucUzuyf3M+vPPP2vgwIGqXLmy2rdvby1/++23rW1WqVJF/fv31x9//HHBffP888+rXbt2qlq1qry8vBQeHq4VK1bY9XFxcVF6erqWLFli7Zvc95t3TqmePXvqqquuKnBbERERdgXNS4k713fffScXFxctWbIk37K1a9fKxcVFq1atstqKmr8L8vnnnysyMlLe3t6qVKmSbr75Zu3Zsydfv7/++kvDhg1TUFCQPDw8FBwcrPvvv19ZWVmS8s8p1alTJ61evVq///67tX/r1aunkydPytvbWw899FC+bfz555+y2WyaPn16kWK/LBhcskWLFhlJZsOGDebvv/82R44cMbt27TL33nuvcXV1NevWrbP65uTkmM6dOxsXFxdz9913m1deecX06tXLSDKjR4+2+m3fvt3YbDbz8MMPW239+/c3Xl5eZt++fVbb4MGDjaenp2nQoIG54447zCuvvGJ69uxpJJn//ve/dnFKMk8++aT1fNeuXcbb29sEBgaap556yjz77LMmODjYeHh4mO3btxtjjNm/f7958MEHjSTz2GOPmbfeesu89dZbJikpqcB9sW7dOjN8+HAjyUydOtW89dZb5quvvrLbT61btzazZs0yjz76qPHy8jL16tUzx48ft3tPHh4epn79+mbw4MFmwYIF5s033yx0/yclJZlatWqZ2rVrm6lTp5r58+ebm266yUgys2bNsvotXLjQSDJ9+/Y1r776qpkzZ44ZNmyYefDBB40xxvz4449mwIAB1uty3+vJkycL3fa51q5da1xdXU2TJk3Miy++aB5//HHj7+9vGjdubOrWrXven8Xy5ctN8+bNzaRJk8zChQvNY489ZipXrmzq1q1r0tPTrX65+7BFixbmhhtuMHPnzjV33HGHkWQmTJhg2rdvbwYOHGjmzZtn/R4sWbLEbtt33323cXNzM/fcc49ZsGCBeeSRR4y3t7dp3bq1ycrKMsYY8+GHH5patWqZ0NBQaz/k/h6np6ebZs2amapVq5rHHnvMLFiwwNx5553GxcXFPPTQQ9Z2Dhw4YCSZRo0amauuuso8++yzZtasWeb3338v0v6sW7euadiwoQkMDDSTJ082s2bNMv/5z3+Mj4+Pefvtt02dOnXMs88+a5599lnj7+9vQkJCTHZ2tvX6J5980uQd4iSZ5s2bW7/zs2fPNldddZWpWLGi+eeff4oUF1CayCf/j3xCPsnl7Hzy/PPPm8jISDN16lSzcOFC89BDDxkvLy9z7bXXmpycHKtfUf/WgLKG3PP/nJF7cj+zNmrUyNx8881m3rx5Zu7cucYYY6ZNm2ZcXFxMv379zLx588yUKVNMtWrVCtxm3vxQq1Yt88ADD5hXXnnFvPjii+baa681ksyqVausPm+99Zbx8PAwkZGR1r7J+34PHDhgjDHmzTffNJLMN998Y7edxMREI8nMnDnTaitq3Bdy1VVXmRtvvDFf+9ChQ03lypWtXFPU/J2bTxYtWmS1rV+/3ri5uZmrr77azJgxw4q1cuXK1ns3xpi//vrLBAUFmYoVK5rRo0ebBQsWmP/+978mLCzMek+bNm0yksymTZuMMWd/n1q0aGGqVatm7d8PP/zQGGPMoEGDTM2aNc2ZM2fs3tuMGTOMi4tLkXPc5YCiVAnI/YPN+/Dw8DCLFy+26xsbG2skmWnTptm19+3b17i4uJhff/3Vaps4caJxdXU1W7duNcuXLzeSzOzZs+1eN3jwYCPJjBo1ymrLyckxPXr0MO7u7ubvv/+22vMO5DExMcbd3d3s37/fajt06JDx9fU1HTp0sNpyt537x1XU/fHtt99abVlZWaZGjRqmSZMmJiMjw2pftWqVkWQmTZqU7z09+uijRdresGHDTGBgYL6iQv/+/Y2/v7/5999/jTHG3HzzzaZx48bnXdfMmTPtBt/iaNGihQkMDDQpKSlW27p164ykC36JyI3xXHFxcUaSXRLL3bfR0dF2H4QjIiKMi4uLue+++6y2M2fOmFq1apmOHTtabdu2bTOSzDvvvGO3rTVr1uRrb9y4sd1rcz311FPG29vb/PLLL3btjz76qLHZbObgwYPGmP8f9P38/MyRI0fyredC6tatayR
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1200x800 with 5 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABKUAAAMWCAYAAAAgRDUeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADTJklEQVR4nOzdeVwV9f7H8fcB5LAIuCCbopKSK25kiqnpzSRTr9w200zzalnpLTVTafGqdaOruZVb3lvaZpll2E/TNMulpEXUXFJDcy1Qc2GTQGF+f/hgrkeWAwbngLyej8d5eGa+n5n5zAjz5XzOzHcshmEYAgAAAAAAABzIxdkJAAAAAAAAoOqhKAUAAAAAAACHoygFAAAAAAAAh6MoBQAAAAAAAIejKAUAAAAAAACHoygFAAAAAAAAh6MoBQAAAAAAAIejKAUAAAAAAACHoygFAAAAAAAAh6MoVUVZLBZNnjzZIdtau3at2rRpIw8PD1ksFp0/f94h262IHnroITVs2NBmniP/L6qqyZMny2KxODsN4LpDX+Ic9CUAqir6naIV1jf8WUuWLJHFYtGRI0fKdL3OdOTIEVksFi1ZsqRc1r9x40ZZLBZt3LixXNZ/PaIoVcbyf3GvfAUEBKh79+5as2aNs9P703766SdNnjy5xCemM2fO6L777pOnp6fmzZund955R97e3uWbZBnZunWrJk+eXOE7IEeYP39+uZ24ARREX2KLvuT6QF8CVFz0O7Yqc79zLV566SXFx8c7O43r1tKlSzV79mxnp1FhuTk7gevV1KlTFRYWJsMwdPLkSS1ZskR33nmn/u///k99+vRxdnrX7KefftKUKVPUrVu3ElXif/jhB6Wnp+uFF15Qjx49yj/BMrR161ZNmTJFDz30kGrUqFFu28nKypKbW8X+VZw/f778/f310EMPOTuVa/Lcc89p4sSJzk4DKDX6ksvoS+yjLwFQFuh3LqvM/c61eOmll3TPPfcoJibGZv6DDz6o+++/X1ar1TmJVUJdu3ZVVlaW3N3dzXlLly7Vnj17NHr0aOclVoFV7L9eKrFevXrppptuMqeHDRumwMBAvf/++5X6hF5ap06dkqQy/UM8MzPzuvqmwsPDw9kpOMWlS5eUl5dnc8IuL25ubhX+wxpQGPqSy+hL7KMvKf++BKgK6Hcuqwj9TkXop1xdXeXq6urUHCobFxeXKtsnXytu33OQGjVqyNPTs8AH48zMTD311FMKDQ2V1WpVkyZN9Morr8gwDEmXv/ls2rSpmjZtqqysLHO5s2fPKjg4WJ06dVJubq6ky/cRV69eXb/88ouio6Pl7e2tkJAQTZ061VxfcXbs2KFevXrJ19dX1atX12233aZvv/3WbF+yZInuvfdeSVL37t3NS3uLul+2W7duGjJkiCSpffv2slgsNt+OLl++XJGRkfL09JS/v78GDRqkX3/91WYd+ft06NAh3XnnnfLx8dEDDzxQ7H78+uuv+vvf/67AwEBZrVa1aNFCb775ZoG41157TS1atJCXl5dq1qypm266SUuXLpV0eQyip59+WpIUFhZm7mtp7qeOj49Xy5Yt5eHhoZYtW+qTTz4pNO7qe+OPHj2qxx9/XE2aNJGnp6dq166te++9t8C28y+z/vrrr/XEE0+oTp06qlGjhkaMGKGcnBydP39egwcPVs2aNVWzZk2NHz++wM9BXl6eZs+erRYtWsjDw0OBgYEaMWKEzp07Z8Y0bNhQe/fu1aZNm8zj0K1bN7P9/PnzGj16tPkz3LhxY/373/9WXl6eGZN/7/Yrr7yi2bNnq1GjRrJarfrpp59KdCwbNmyoPn36aOPGjbrpppvk6empiIgI82dvxYoVioiIkIeHhyIjI7Vjxw6b5QsbU8pisWjUqFHm/1P+z8ratWtLlBPgDPQl9CX0Jc7rS3bt2qWHHnpIN9xwgzw8PBQUFKS///3vOnPmjBlTmt81oDKg33FMv5P/t+pPP/2kgQMHqmbNmurcubPZ/u6775rbrFWrlu6//34dP37c7rF55ZVX1KlTJ9WuXVuenp6KjIzURx99ZBNjsViUmZmpt956yzw2+ft79ZhSffr00Q033FDotqKiomwKmn8m73zbtm2TxWLRW2+9VaDt888/l8Vi0apVq8x5Je27C/Pll1+qS5cu8vb2Vo0aNdSvXz/t27evQNyvv/6qYcOGKSQkRFarVWFhYXrssceUk5MjqeCYUt26ddPq1at19OhR8/g2bNhQGRkZ8vb21pNPPllgGydOnJCrq6vi4uJKlHulZ6BMLV682JBkfPHFF8bp06eNU6dOGXv27DFGjBhhuLi4GOvWrTNj8/LyjL/85S+GxWIxhg8fbsydO9fo27evIckYPXq0Gfftt98arq6uxpgxY8x5999/v+Hp6WkcOHDAnDdkyBDDw8PDCA8PNx588EFj7ty5Rp8+fQxJxvPPP2+TpyTjn//8pzm9Z88ew9vb2wgODjZeeOEF4+WXXzbCwsIMq9VqfPvtt4ZhGMahQ4eMJ554wpBkPPPMM8Y777xjvPPOO0ZKSkqhx2LdunXGI488Ykgypk6darzzzjvG1q1bbY5T+/btjVmzZhkTJ040PD09jYYNGxrnzp2z2Ser1Wo0atTIGDJkiLFw4ULj7bffLvL4p6SkGPXq1TNCQ0ONqVOnGgsWLDD++te/GpKMWbNmmXGLFi0yJBn33HOP8frrrxtz5swxhg0bZjzxxBOGYRjGjz/+aAwYMMBcLn9fMzIyitz2lT7//HPDxcXFaNmypTFz5kzj2WefNfz8/IwWLVoYDRo0KPb/Yvny5Ubr1q2NSZMmGYsWLTKeeeYZo2bNmkaDBg2MzMxMMy7/GLZp08a44447jHnz5hkPPvigIckYP3680blzZ2PgwIHG/PnzzZ+Dt956y2bbw4cPN9zc3IyHH37YWLhwoTFhwgTD29vbaN++vZGTk2MYhmF88sknRr169YymTZuaxyH/5zgzM9No1aqVUbt2beOZZ54xFi5caAwePNiwWCzGk08+aW7n8OHDhiSjefPmxg033GC8/PLLxqxZs4yjR4+W6Hg2aNDAaNKkiREcHGxMnjzZmDVrllG3bl2jevXqxrvvvmvUr1/fePnll42XX37Z8PPzMxo3bmzk5uaay//zn/80rj7dSTJat25t/szPnj3buOGGGwwvLy/j999/L1FeQHmhL/kf+hL6knzO7kteeeUVo0uXLsbUqVONRYsWGU8++aTh6elp3HzzzUZeXp4ZV9LfNaAiod/5H2f0O/l/qzZv3tzo16+fMX/+fGPevHmGYRjGiy++aFgsFqN///7G/PnzjSlTphj+/v6FbvPqvqFevXrG448/bsydO9eYOXOmcfPNNxuSjFWrVpkx77zzjmG1Wo0uXbqYx+bq/T18+LBhGIbx9ttvG5KM77//3mY7R44cMSQZ06dPN+eVNG97brjhBuPOO+8sMH/o0KFGzZo1zX6mpH13fl+yePFic9769esNNzc348YbbzSmTZtm5lqzZk1z3w3DMH799VcjJCTE8PLyMkaPHm0sXLjQeP75541mzZqZ+/TVV18ZkoyvvvrKMIzLP09t2rQx/P39zeP7ySefGIZhGA888IARGBhoXLp0yWbfpk2bZlgslhL3b5UdRakylv+Le/XLarUaS5YssYmNj483JBkvvviizfx77rnHsFgsxsGDB815sbGxhouLi7F582Zj+fLlhiRj9uzZNssNGTLEkGT84x//MOfl5eUZvXv3Ntzd3Y3Tp0+b868+ocfExBju7u7GoUOHzHm//fab4ePjY3Tt2tWcl7/t/F+ykh6PH374wZyXk5NjBAQEGC1btjSysrLM+atWrTIkGZMmTSqwTxMnTizR9oYNG2YEBwcXKCrcf//9hp+fn3HhwgXDMAyjX79+RosWLYpd1/Tp021OwqXRpk0bIzg42Dh//rw5b926dYYkux8k8nO8UkJCgiHJpjPLP7bR0dE2fwxHRUUZFovFePTRR815ly5dMurVq2fceuut5rwtW7YYkoz33nvPZltr164tML9FixY2y+Z74YUXDG9vb+Pnn3+
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1200x800 with 5 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество записей после устранения выбросов: 17638\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"numeric_cols = df.select_dtypes(include=['number']).columns\n",
|
|||
|
|
"\n",
|
|||
|
|
"#все столбцы, кроме id\n",
|
|||
|
|
"numeric_cols = [col for col in numeric_cols if col != 'id']\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f'Количество записей до устранения выбросов: {len(df)}')\n",
|
|||
|
|
"\n",
|
|||
|
|
"plt.figure(figsize=(12, 8))\n",
|
|||
|
|
" \n",
|
|||
|
|
"\n",
|
|||
|
|
"for i, col in enumerate(numeric_cols, 1):\n",
|
|||
|
|
" if col == 'id':\n",
|
|||
|
|
" continue\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" outliers = df[col][(df[col] < lower_bound) | (df[col] > upper_bound)]\n",
|
|||
|
|
" plt.subplot(len(numeric_cols) // 3 + 1, 3, i) \n",
|
|||
|
|
" plt.boxplot(x=df[col])\n",
|
|||
|
|
" plt.title(f'Boxplot for {col}')\n",
|
|||
|
|
"\n",
|
|||
|
|
"plt.tight_layout()\n",
|
|||
|
|
"plt.show()\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"#Удаление единичных выбросов\n",
|
|||
|
|
"plt.figure(figsize=(12, 8))\n",
|
|||
|
|
"for i, col in enumerate(numeric_cols, 1):\n",
|
|||
|
|
" if col == 'id':\n",
|
|||
|
|
" continue\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" \n",
|
|||
|
|
" if (col=='est_diameter_min'):\n",
|
|||
|
|
" df = df[~((df[col] > 3))]\n",
|
|||
|
|
" if (col=='est_diameter_max'):\n",
|
|||
|
|
" df = df[~((df[col] > 5.5))]\n",
|
|||
|
|
" if (col=='relative_velocity'):\n",
|
|||
|
|
" df = df[~((df[col] > 175000))]\n",
|
|||
|
|
" if (col=='absolute_magnitude'):\n",
|
|||
|
|
" df = df[~((df[col] < 13))]\n",
|
|||
|
|
" \n",
|
|||
|
|
" plt.subplot(len(numeric_cols) // 3 + 1, 3, i)\n",
|
|||
|
|
" plt.boxplot(x=df[col].dropna()) \n",
|
|||
|
|
" plt.title(f'Boxplot for {col}')\n",
|
|||
|
|
"\n",
|
|||
|
|
"plt.tight_layout()\n",
|
|||
|
|
"plt.show()\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f'Количество записей после устранения выбросов: {len(df)}')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Пропущенных значений нет, значит решать данную проблему не надо"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 257,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Столбцы с null: []\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"#проверка на пропущенные значения\n",
|
|||
|
|
"columns_with_nulls = []\n",
|
|||
|
|
"for col in df.columns:\n",
|
|||
|
|
" if df[col].isnull().sum() > 0: \n",
|
|||
|
|
" columns_with_nulls.append(col)\n",
|
|||
|
|
"print(f\"Столбцы с null: {columns_with_nulls}\")"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**РАЗБИЕНИЕ НА ВЫБОРКИ**"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"После разбиения в обучающую выборку попало практически равное количество безопасных и опасных объектов, поэтому применять методы аугментации данных нет необходимости"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 258,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Размер обучающей выборки: 14110\n",
|
|||
|
|
"Размер контрольной выборки: 1764\n",
|
|||
|
|
"Размер тестовой выборки: 1764\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkQAAAGwCAYAAABIC3rIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA3zElEQVR4nO3deXxU5d3//3dIMiEsM2FLQmqAUGQJBQTUMHVBJE3A4I03sRWLgMhSaECBVpBHkc3egiigIkirQvAuFKFfVCSyhLBVCaBRFMJSqKFBIQmCyQBCAsn1+8Nfzs3IIglJJnBez8fjPMpc1+ecuT6MQ949c87EzxhjBAAAYGM1fL0AAAAAXyMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2yMQAQAA2wvw9QJuBCUlJTp69Kjq1q0rPz8/Xy8HAABcA2OMTp06pYiICNWocfVzQASia3D06FFFRkb6ehkAAKAcjhw5oltuueWqNQSia1C3bl1JP/yFOp1OH68GAABcC4/Ho8jISOvn+NUQiK5B6cdkTqeTQAQAwA3mWi534aJqAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewQiAABgewG+XgAA2EGzZ1J8vQSgWjs8I8Gnz+/TM0TNmjWTn5/fJVtSUpIk6dy5c0pKSlKDBg1Up04dJSYmKjc31+sY2dnZSkhIUK1atRQaGqqnn35aFy5c8KrZvHmzOnXqpKCgILVo0ULJyclV1SIAALgB+DQQffLJJzp27Ji1paamSpJ+/etfS5LGjBmjDz74QCtWrNCWLVt09OhR9enTx9q/uLhYCQkJKioq0rZt27R48WIlJydr0qRJVk1WVpYSEhLUrVs37dq1S6NHj9aQIUO0bt26qm0WAABUW37GGOPrRZQaPXq0Vq9erYMHD8rj8ahRo0ZaunSpHn74YUnS/v371aZNG6Wnp6tLly5as2aNevXqpaNHjyosLEyStGDBAo0fP17Hjx+Xw+HQ+PHjlZKSoj179ljP07dvX+Xn52vt2rWXXUdhYaEKCwutxx6PR5GRkSooKJDT6azEvwEANys+MgOurjI+MvN4PHK5XNf087vaXENUVFSkv/3tbxo7dqz8/PyUkZGh8+fPKzY21qpp3bq1mjRpYgWi9PR0tWvXzgpDkhQfH68RI0YoMzNTHTt2VHp6utcxSmtGjx59xbVMnz5dU6dOrfAer4R/KIEr8/V1BQDsodrcZfbee+8pPz9fjz/+uCQpJydHDodDISEhXnVhYWHKycmxai4OQ6XzpXNXq/F4PDp79uxl1zJhwgQVFBRY25EjR663PQAAUI1VmzNEb731lnr27KmIiAhfL0VBQUEKCgry9TIAAEAVqRZniP7zn/9ow4YNGjJkiDUWHh6uoqIi5efne9Xm5uYqPDzcqvnxXWelj3+qxul0Kjg4uKJbAQAAN6BqEYgWLVqk0NBQJST837UCnTt3VmBgoNLS0qyxAwcOKDs7W263W5Lkdru1e/du5eXlWTWpqalyOp2Kjo62ai4+RmlN6TEAAAB8HohKSkq0aNEiDRw4UAEB//cJnsvl0uDBgzV27Fht2rRJGRkZGjRokNxut7p06SJJiouLU3R0tPr3768vvvhC69at08SJE5WUlGR95DV8+HB99dVXGjdunPbv36/58+dr+fLlGjNmjE/6BQAA1Y/PryHasGGDsrOz9cQTT1wyN2fOHNWoUUOJiYkqLCxUfHy85s+fb837+/tr9erVGjFihNxut2rXrq2BAwdq2rRpVk1UVJRSUlI0ZswYvfLKK7rlllv05ptvKj4+vkr6AwAA1V+1+h6i6qos32NQHtx2D1zZzXLbPe9z4Op8/T1EPv/IDAAAwNcIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPYIRAAAwPZ8Hoi++eYbPfbYY2rQoIGCg4PVrl07ffrpp9a8MUaTJk1S48aNFRwcrNjYWB08eNDrGCdPnlS/fv3kdDoVEhKiwYMH6/Tp0141X375pe655x7VrFlTkZGRmjlzZpX0BwAAqj+fBqLvvvtOd911lwIDA7VmzRrt3btXs2bNUr169ayamTNn6tVXX9WCBQu0Y8cO1a5dW/Hx8Tp37pxV069fP2VmZio1NVWrV6/W1q1bNWzYMGve4/EoLi5OTZs2VUZGhl588UVNmTJFf/3rX6u0XwAAUD0F+PLJX3jhBUVGRmrRokXWWFRUlPVnY4xefvllTZw4Ub1795Ykvf322woLC9N7772nvn37at++fVq7dq0++eQT3X777ZKkuXPn6oEHHtBLL72kiIgILVmyREVFRVq4cKEcDofatm2rXbt2afbs2V7BCQAA2JNPzxCtWrVKt99+u379618rNDRUHTt21BtvvGHNZ2VlKScnR7GxsdaYy+VSTEyM0tPTJUnp6ekKCQmxwpAkxcbGqkaNGtqxY4dVc++998rhcFg18fHxOnDggL777rtL1lVYWCiPx+O1AQCAm5dPA9FXX32l119/XbfeeqvWrVunESNG6Mknn9TixYslSTk5OZKksLAwr/3CwsKsuZycHIWGhnrNBwQEqH79+l41lzvGxc9xsenTp8vlcllbZGRkBXQLAACqK58GopKSEnXq1EnPP/+8OnbsqGHDhmno0KFasGCBL5elCRMmqKCgwNqOHDni0/UAAIDK5dNA1LhxY0VHR3uNtWnTRtnZ2ZKk8PBwSVJubq5XTW5urjUXHh6uvLw8r/kLFy7o5MmTXjWXO8bFz3GxoKAgOZ1Orw0AANy8fBqI7rrrLh04cMBr7F//+peaNm0q6YcLrMPDw5WWlmbNezwe7dixQ263W5LkdruVn5+vjIwMq2bjxo0qKSlRTEyMVbN161adP3/eqklNTVWrVq287mgDAAD25NNANGbMGG3fvl3PP/+8Dh06pKVLl+qvf/2rkpKSJEl+fn4aPXq0/vznP2vVqlXavXu3BgwYoIiICD300EOSfjij1KNHDw0dOlQ7d+7Uxx9/rJEjR6pv376KiIiQJP32t7+Vw+HQ4MGDlZmZqXfeeUevvPKKxo4d66vWAQBANeLT2+7vuOMOvfvuu5owYYKmTZumqKgovfzyy+rXr59VM27cOJ05c0bDhg1Tfn6+7r77bq1du1Y1a9a0apYsWaKRI0eqe/fuqlGjhhITE/Xqq69a8y6XS+vXr1dSUpI6d+6shg0batKkSdxyDwAAJEl+xhjj60VUdx6PRy6XSwUFBZVyPVGzZ1Iq/JjAzeLwjARfL6FC8D4Hrq4y3utl+fnt81/dAQAA4GsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHsEIgAAYHs+DURTpkyRn5+f19a6dWtr/ty5c0pKSlKDBg1Up04dJSYmKjc31+sY2dnZSkhIUK1atRQaGqqnn35aFy5c8KrZvHmzOnXqpKCgILVo0ULJyclV0R4AALhB+PwMUdu2bXXs2DFr++ijj6y5MWPG6IM
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 640x480 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"hazardous\n",
|
|||
|
|
"True 7094\n",
|
|||
|
|
"False 7016\n",
|
|||
|
|
"Name: count, dtype: int64\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"data=df[['est_diameter_min', 'est_diameter_max', 'relative_velocity', 'miss_distance', 'absolute_magnitude', 'hazardous']].copy()\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# сначала разделение записей на 80% и 20%, где 80% - обучающая выборка\n",
|
|||
|
|
"train_data, temp_data = train_test_split(data, test_size=0.2, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# потом разделение остальных 20% поровну на контрольную и тестовую выборки\n",
|
|||
|
|
"val_data, test_data = train_test_split(temp_data, test_size=0.5, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка размеров выборок\n",
|
|||
|
|
"print(\"Размер обучающей выборки:\", len(train_data))\n",
|
|||
|
|
"print(\"Размер контрольной выборки:\", len(val_data))\n",
|
|||
|
|
"print(\"Размер тестовой выборки:\", len(test_data))\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# построение столбчатой диаграммы по столбцу rating_stars (сбалансированность обучающей выборки)\n",
|
|||
|
|
"hazardous_counts = train_data['hazardous'].value_counts()\n",
|
|||
|
|
"\n",
|
|||
|
|
"plt.bar(hazardous_counts.index, hazardous_counts.values)\n",
|
|||
|
|
"plt.xlabel('hazardous')\n",
|
|||
|
|
"plt.ylabel('Count')\n",
|
|||
|
|
"plt.show()\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(train_data[\"hazardous\"].value_counts())"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"## Конструирование признаков"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Унитарное кодирование категориальных признаков (для всех категориальных признаков в датасете)**\n",
|
|||
|
|
"\n",
|
|||
|
|
"т.к. в столбцах orbiting_body и sentry_object есть только 1 различное категориальное значение, то при использовании encoder = OneHotEncoder(sparse_output=False, drop=\"first\") срабатывает условие drop=\"first\", который удаляет первое категориальное значение для предотвращения линейной зависимости между столбцами. Поэтому в случае с hazardous использован OneHotEncoder с параметром drop=\"first\", а для orbiting_body и sentry_object без него.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Для решения обеих задач (бизнес-целей и технических целей) нужен только признак hazardous_True"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 259,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
" orbiting_body_Earth sentry_object_False hazardous_True\n",
|
|||
|
|
"0 1.0 1.0 1.0\n",
|
|||
|
|
"1 1.0 1.0 1.0\n",
|
|||
|
|
"2 1.0 1.0 0.0\n",
|
|||
|
|
"3 1.0 1.0 1.0\n",
|
|||
|
|
"4 1.0 1.0 0.0\n",
|
|||
|
|
"... ... ... ...\n",
|
|||
|
|
"17633 1.0 1.0 1.0\n",
|
|||
|
|
"17634 1.0 1.0 0.0\n",
|
|||
|
|
"17635 1.0 1.0 1.0\n",
|
|||
|
|
"17636 1.0 1.0 0.0\n",
|
|||
|
|
"17637 1.0 1.0 0.0\n",
|
|||
|
|
"\n",
|
|||
|
|
"[17638 rows x 3 columns]\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"from sklearn.preprocessing import OneHotEncoder\n",
|
|||
|
|
"import numpy as np\n",
|
|||
|
|
"\n",
|
|||
|
|
"encoder = OneHotEncoder(sparse_output=False, drop=\"first\")\n",
|
|||
|
|
"encoder1 = OneHotEncoder(sparse_output=False)\n",
|
|||
|
|
"\n",
|
|||
|
|
"encoded_values = encoder.fit_transform(df[[\"hazardous\"]])\n",
|
|||
|
|
"encoded_columns = encoder.get_feature_names_out([\"hazardous\"])\n",
|
|||
|
|
"\n",
|
|||
|
|
"encoded_values1 = encoder1.fit_transform(df[[\"orbiting_body\", \"sentry_object\"]])\n",
|
|||
|
|
"encoded_columns1 = encoder1.get_feature_names_out([\"orbiting_body\", \"sentry_object\"])\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"encoded_values_df1 = pd.DataFrame(encoded_values1, columns=encoded_columns1)\n",
|
|||
|
|
"encoded_values_df2 = pd.DataFrame(encoded_values, columns=encoded_columns)\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"encoded_values_df = pd.concat([encoded_values_df1, encoded_values_df2], axis=1)\n",
|
|||
|
|
"print(encoded_values_df)\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Унитарное кодирование для самой обучающей выборки"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 260,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
" est_diameter_min est_diameter_max relative_velocity miss_distance \\\n",
|
|||
|
|
"1882 0.056566 0.126485 53181.770308 1.608163e+07 \n",
|
|||
|
|
"10162 0.108282 0.242125 71641.177993 6.633209e+07 \n",
|
|||
|
|
"13664 0.186447 0.416908 72150.829803 7.266875e+07 \n",
|
|||
|
|
"17556 0.003052 0.006824 13472.804594 2.411481e+07 \n",
|
|||
|
|
"13523 0.048368 0.108153 31281.647283 2.217165e+06 \n",
|
|||
|
|
"... ... ... ... ... \n",
|
|||
|
|
"11308 0.069913 0.156329 38271.512861 5.439281e+07 \n",
|
|||
|
|
"11990 0.152249 0.340440 58488.870909 7.232920e+07 \n",
|
|||
|
|
"5402 0.008801 0.019681 20926.865447 2.458771e+07 \n",
|
|||
|
|
"861 0.183889 0.411188 40232.968565 3.188090e+07 \n",
|
|||
|
|
"15832 0.183889 0.411188 89521.116646 5.187941e+07 \n",
|
|||
|
|
"\n",
|
|||
|
|
" absolute_magnitude hazardous_True \n",
|
|||
|
|
"1882 23.36 0.0 \n",
|
|||
|
|
"10162 21.95 0.0 \n",
|
|||
|
|
"13664 20.77 1.0 \n",
|
|||
|
|
"17556 29.70 0.0 \n",
|
|||
|
|
"13523 23.70 0.0 \n",
|
|||
|
|
"... ... ... \n",
|
|||
|
|
"11308 22.90 0.0 \n",
|
|||
|
|
"11990 21.21 1.0 \n",
|
|||
|
|
"5402 27.40 0.0 \n",
|
|||
|
|
"861 20.80 1.0 \n",
|
|||
|
|
"15832 20.80 1.0 \n",
|
|||
|
|
"\n",
|
|||
|
|
"[14110 rows x 6 columns]\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"#обучающая выборка\n",
|
|||
|
|
"train_data_encoded = pd.get_dummies(train_data, columns=['hazardous'], drop_first=True, dtype=float)\n",
|
|||
|
|
"train_data = train_data_encoded\n",
|
|||
|
|
"\n",
|
|||
|
|
"#валидационная\n",
|
|||
|
|
"val_data_encoded = pd.get_dummies(val_data, columns=['hazardous'], drop_first=True, dtype=float)\n",
|
|||
|
|
"val_data = val_data_encoded\n",
|
|||
|
|
"\n",
|
|||
|
|
"#тестовая\n",
|
|||
|
|
"test_data_encoded = pd.get_dummies(test_data, columns=['hazardous'], drop_first=True, dtype=float)\n",
|
|||
|
|
"test_data = test_data_encoded\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(train_data)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Дискретизация числовых признаков**\n",
|
|||
|
|
"\n",
|
|||
|
|
"дискретизируется скорость relative_velocity\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 261,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
" est_diameter_min est_diameter_max relative_velocity miss_distance \\\n",
|
|||
|
|
"1882 0.056566 0.126485 Небольшая скорость 1.608163e+07 \n",
|
|||
|
|
"10162 0.108282 0.242125 Средняя скорость 6.633209e+07 \n",
|
|||
|
|
"13664 0.186447 0.416908 Средняя скорость 7.266875e+07 \n",
|
|||
|
|
"17556 0.003052 0.006824 Небольшая скорость 2.411481e+07 \n",
|
|||
|
|
"13523 0.048368 0.108153 Небольшая скорость 2.217165e+06 \n",
|
|||
|
|
"... ... ... ... ... \n",
|
|||
|
|
"11308 0.069913 0.156329 Небольшая скорость 5.439281e+07 \n",
|
|||
|
|
"11990 0.152249 0.340440 Небольшая скорость 7.232920e+07 \n",
|
|||
|
|
"5402 0.008801 0.019681 Небольшая скорость 2.458771e+07 \n",
|
|||
|
|
"861 0.183889 0.411188 Небольшая скорость 3.188090e+07 \n",
|
|||
|
|
"15832 0.183889 0.411188 Средняя скорость 5.187941e+07 \n",
|
|||
|
|
"\n",
|
|||
|
|
" absolute_magnitude hazardous_True \n",
|
|||
|
|
"1882 23.36 0.0 \n",
|
|||
|
|
"10162 21.95 0.0 \n",
|
|||
|
|
"13664 20.77 1.0 \n",
|
|||
|
|
"17556 29.70 0.0 \n",
|
|||
|
|
"13523 23.70 0.0 \n",
|
|||
|
|
"... ... ... \n",
|
|||
|
|
"11308 22.90 0.0 \n",
|
|||
|
|
"11990 21.21 1.0 \n",
|
|||
|
|
"5402 27.40 0.0 \n",
|
|||
|
|
"861 20.80 1.0 \n",
|
|||
|
|
"15832 20.80 1.0 \n",
|
|||
|
|
"\n",
|
|||
|
|
"[14110 rows x 6 columns]\n",
|
|||
|
|
"relative_velocity\n",
|
|||
|
|
"Небольшая скорость 8685\n",
|
|||
|
|
"Средняя скорость 5039\n",
|
|||
|
|
"Большая скорость 386\n",
|
|||
|
|
"Name: count, dtype: int64\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"velocity_labels=['Небольшая скорость', 'Средняя скорость', 'Большая скорость']\n",
|
|||
|
|
"\n",
|
|||
|
|
"#обучающая выборка\n",
|
|||
|
|
"train_data_discretized = train_data\n",
|
|||
|
|
"#равномерная группировка (для каждой категории диапазон чисел одинаковой длины)\n",
|
|||
|
|
"train_data_discretized['relative_velocity'] = pd.cut(train_data['relative_velocity'], bins=3, labels=velocity_labels)\n",
|
|||
|
|
"train_data = train_data_discretized\n",
|
|||
|
|
"\n",
|
|||
|
|
"#валидационная выборка\n",
|
|||
|
|
"val_data_discretized = val_data\n",
|
|||
|
|
"val_data_discretized['relative_velocity'] = pd.cut(val_data['relative_velocity'], bins=3, labels=velocity_labels)\n",
|
|||
|
|
"val_data = val_data_discretized\n",
|
|||
|
|
"\n",
|
|||
|
|
"#тестовая выборка\n",
|
|||
|
|
"test_data_discretized = test_data\n",
|
|||
|
|
"test_data_discretized['relative_velocity'] = pd.cut(test_data['relative_velocity'], bins=3, labels=velocity_labels)\n",
|
|||
|
|
"test_data = test_data_discretized\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(train_data_discretized)\n",
|
|||
|
|
"print(train_data[\"relative_velocity\"].value_counts())"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Ручной синтез признаков**\n",
|
|||
|
|
"\n",
|
|||
|
|
"На основании максимального и минимального радиуса объекта можно найти среднее значение диаметра"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 262,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"train_data['average_diameter'] = (train_data['est_diameter_min'] + train_data['est_diameter_max']) / 2\n",
|
|||
|
|
"val_data['average_diameter'] = (val_data['est_diameter_min'] + val_data['est_diameter_max']) / 2\n",
|
|||
|
|
"test_data['average_diameter'] = (test_data['est_diameter_min'] + test_data['est_diameter_max']) / 2"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Масштабирование признаков**"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 263,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
|||
|
|
"\n",
|
|||
|
|
"scaler = StandardScaler()\n",
|
|||
|
|
"\n",
|
|||
|
|
"numeric_cols = train_data.select_dtypes(include=['number']).columns\n",
|
|||
|
|
"\n",
|
|||
|
|
"numeric_cols = numeric_cols.drop('hazardous_True')\n",
|
|||
|
|
"\n",
|
|||
|
|
"train_data[numeric_cols] = scaler.fit_transform(train_data[numeric_cols])\n",
|
|||
|
|
"val_data[numeric_cols] = scaler.fit_transform(val_data[numeric_cols])\n",
|
|||
|
|
"test_data[numeric_cols] = scaler.fit_transform(test_data[numeric_cols])"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Featuretools**\n",
|
|||
|
|
"\n",
|
|||
|
|
"Featuretools не смог выявить дополнительных признаков по признакам, которые необходимы для решения поставленных задач"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 264,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
" est_diameter_min est_diameter_max relative_velocity miss_distance \\\n",
|
|||
|
|
"id \n",
|
|||
|
|
"0 -0.572328 -0.572328 Небольшая скорость -1.014078 \n",
|
|||
|
|
"1 -0.361244 -0.361244 Средняя скорость 1.278171 \n",
|
|||
|
|
"2 -0.042204 -0.042204 Средняя скорость 1.567227 \n",
|
|||
|
|
"3 -0.790752 -0.790752 Небольшая скорость -0.647632 \n",
|
|||
|
|
"4 -0.605790 -0.605790 Небольшая скорость -1.646526 \n",
|
|||
|
|
"\n",
|
|||
|
|
" absolute_magnitude hazardous_True average_diameter \n",
|
|||
|
|
"id \n",
|
|||
|
|
"0 0.447447 0.0 -0.572328 \n",
|
|||
|
|
"1 -0.051566 0.0 -0.361244 \n",
|
|||
|
|
"2 -0.469179 1.0 -0.042204 \n",
|
|||
|
|
"3 2.691234 0.0 -0.790752 \n",
|
|||
|
|
"4 0.567776 0.0 -0.605790 \n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"name": "stderr",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\featuretools\\entityset\\entityset.py:1733: UserWarning: index id not found in dataframe, creating new integer column\n",
|
|||
|
|
" warnings.warn(\n",
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
|
|||
|
|
" pd.to_datetime(\n",
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\featuretools\\synthesis\\deep_feature_synthesis.py:169: UserWarning: Only one dataframe in entityset, changing max_depth to 1 since deeper features cannot be created\n",
|
|||
|
|
" warnings.warn(\n",
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\featuretools\\synthesis\\dfs.py:321: UnusedPrimitiveWarning: Some specified primitives were not used during DFS:\n",
|
|||
|
|
" agg_primitives: ['any', 'count', 'mean', 'mode']\n",
|
|||
|
|
"This may be caused by a using a value of max_depth that is too small, not setting interesting values, or it may indicate no compatible columns for the primitive were found in the data. If the DFS call contained multiple instances of a primitive in the list above, none of them were used.\n",
|
|||
|
|
" warnings.warn(warning_msg, UnusedPrimitiveWarning)\n",
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\featuretools\\computational_backends\\feature_set_calculator.py:143: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n",
|
|||
|
|
" df = pd.concat([df, default_df], sort=True)\n",
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\featuretools\\computational_backends\\feature_set_calculator.py:143: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n",
|
|||
|
|
" df = pd.concat([df, default_df], sort=True)\n",
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\featuretools\\computational_backends\\feature_set_calculator.py:143: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n",
|
|||
|
|
" df = pd.concat([df, default_df], sort=True)\n",
|
|||
|
|
"c:\\AI labs\\aimenv\\Lib\\site-packages\\featuretools\\computational_backends\\feature_set_calculator.py:143: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n",
|
|||
|
|
" df = pd.concat([df, default_df], sort=True)\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import featuretools as ft\n",
|
|||
|
|
"\n",
|
|||
|
|
"es = ft.EntitySet(id='space_objects_data')\n",
|
|||
|
|
"es = es.add_dataframe(dataframe_name='space_objects', dataframe=train_data, index='id')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Генерация признаков\n",
|
|||
|
|
"feature_matrix, feature_defs = ft.dfs(\n",
|
|||
|
|
" entityset=es,\n",
|
|||
|
|
" target_dataframe_name=\"space_objects\",\n",
|
|||
|
|
" agg_primitives=[\"mean\", \"count\", \"mode\", \"any\"],\n",
|
|||
|
|
" max_depth=2,\n",
|
|||
|
|
")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Преобразование признаков для контрольной и тестовой выборок\n",
|
|||
|
|
"val_feature_matrix = ft.calculate_feature_matrix(features=feature_defs, entityset=es, instance_ids=val_data_encoded.index)\n",
|
|||
|
|
"test_feature_matrix = ft.calculate_feature_matrix(features=feature_defs, entityset=es, instance_ids=test_data_encoded.index)\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(feature_matrix.head())"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Оценка качества наборов признаков**"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"У модели, построенной по логической регрессии, точность (accuracy) составляет почти 85%, что является приемлемым значением. При этом, модель верно относит опасные объекты к опасным примерно в 80% случаев (precision), что может быть недостаточным значением \n",
|
|||
|
|
"\n",
|
|||
|
|
"Модель имеет показатель показатель полноты примерно 93%\n",
|
|||
|
|
"\n",
|
|||
|
|
"Значение метрики f1 примерно 86%, что означает, что между точностью и полнотой модели хороший баланс\n",
|
|||
|
|
"\n",
|
|||
|
|
"Среднее значение точности по кросс-валидации (Cross-Validation Mean) составляет около 85%, что близко к точности на тестовой выборке. Это указывает на то, что модель стабильна\n",
|
|||
|
|
"\n",
|
|||
|
|
"Дисперсия (Cross-Validation Std) составляет меньше 1%, что говорит о том, что модель качественно выполняет предсказания на\n",
|
|||
|
|
"тестовой выборке. Модель можно считать надёжной\n",
|
|||
|
|
"\n",
|
|||
|
|
"У модели очень большая скорость вычислений и очень низкий показатель среднеквадратичной ошибки, что говорит о том, что модель обучилась хорошо\n",
|
|||
|
|
"\n",
|
|||
|
|
"По диаграмме (матрице ошибок) можно увидеть, что модель большинство космических объектов оценила также, как есть на самом деле. Модель очень часто верно определяет опасные объекты, чуть хуже верно определяет неопасные объекты. Однако у неё есть склонность относить неопасные объекты к опасным"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 265,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Accuracy: 0.8479801559177887\n",
|
|||
|
|
"Precision: 0.8024464831804281\n",
|
|||
|
|
"Полнота: 0.92524682651622\n",
|
|||
|
|
"F1: 0.859482476252866\n",
|
|||
|
|
"Cross-Validation Mean: 0.8499645641389085\n",
|
|||
|
|
"Cross-Validation Std: 0.0053722448357894316\n",
|
|||
|
|
"Время обучения модели: 0.01 секунд\n",
|
|||
|
|
"Скорость вычисления: 0.00 секунд\n",
|
|||
|
|
"Среднеквадратичная ошибка: 0.15\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAApIAAAIjCAYAAACwHvu2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjGUlEQVR4nO3dd3gU1fv38c8GSCEhCS0JQQgBQu89dDRK74oISlUECUgTjNJRo4D0JqCACkoVUL8iCAgCkR5AQESKgBJaaKGElH3+4GF/rAnIDtnskrxfXnO5O+fM7D2rITf3OXPGZDabzQIAAABs5OLoAAAAAPBkIpEEAACAISSSAAAAMIREEgAAAIaQSAIAAMAQEkkAAAAYQiIJAAAAQ0gkAQAAYEhWRwcAAJnRnTt3FBsbq+TkZAUGBjo6HAAwhIokAKSTXbt2qUOHDsqTJ4/c3NyUL18+tW3b1tFhAYBhJJKAE5k/f75MJpNMJpO2bNmSot1sNqtAgQIymUxq1qyZAyKEUatWrVLt2rV16NAhvf/++1q3bp3WrVunTz75xNGhAYBhDG0DTsjd3V2LFi1S7dq1rfZv2rRJZ86ckZubm4MigxGxsbF69dVX1bBhQy1dulSurq6ODgkA0gQVScAJNWnSREuXLlViYqLV/kWLFqly5coKCAhwUGQwYt68ebp9+7bmz59PEgkgQyGRBJzQSy+9pEuXLmndunWWfXfu3NGyZcvUoUOHVI8ZP368atasqdy5c8vDw0OVK1fWsmXLrPrcGzZ/0Fa/fn1J0s8//yyTyaTFixfrnXfeUUBAgDw9PdWiRQudPn3a6pz169e3HHfPzp07Lef89+eHh4eniL1Zs2YqVKiQ1b79+/erS5cuKly4sNzd3RUQEKBu3brp0qVLD/vqLM6fP6/u3bvL399f7u7uKl++vBYsWGDV5+TJkzKZTBo/frzV/jJlyqS4pqFDh8pkMikuLs7qekaOHGnVb9y4cVbfpST9+uuvqlChgj744AMVKFBAbm5uCgkJ0Ycffqjk5GSr4xMTEzVmzBgVKVJEbm5uKlSokN555x3Fx8db9StUqJC6dOlita9Hjx5yd3fXzz///N9fEACkAYa2ASdUqFAhhYaG6quvvlLjxo0lST/88IOuXr2q9u3ba8qUKSmOmTx5slq0aKGOHTvqzp07+vrrr/XCCy/ou+++U9OmTSVJX3zxhaX/L7/8otmzZ2vixInKkyePJMnf39/qnO+//75MJpOGDBmi8+fPa9KkSQoLC1N0dLQ8PDweGP+QIUMe+ztYt26djh8/rq5duyogIEAHDx7U7NmzdfDgQf36668pktT73bp1S/Xr19eff/6p8PBwBQcHa+nSperSpYuuXLmiN99887HjS82VK1cUGRmZYv+lS5e0ZcsWbdmyRd26dVPlypW1fv16RURE6OTJk5o1a5al76uvvqoFCxbo+eef18CBA7V9+3ZFRkbq8OHD+uabbx742SNGjNCnn36qxYsXp0iCAcBuzACcxrx588ySzDt37jRPmzbNnCNHDvPNmzfNZrPZ/MILL5gbNGhgNpvN5qCgIHPTpk2tjr3X7547d+6Yy5QpY3766acf+lknTpxI0bZx40azJHP+/PnN165ds+xfsmSJWZJ58uTJln316tUz16tXz/L+f//7n1mSuVGjRuZ//xEjydy7d+8Un9e0aVNzUFDQQ6/HbDabv/rqK7Mk8+bNm1O9pnsmTZpklmT+8ssvLfvu3LljDg0NNXt5eVmu6cSJE2ZJ5nHjxlkdX7p0aatrMpvN5nfffdcsyXz9+nWr6xkxYoTl/eDBg81+fn7mypUrWx1fr149syTzyJEjrc7ZpUsXsyTzgQMHzGaz2RwdHW2WZH711Vet+g0aNMgsybxhwwbLvqCgIHPnzp3NZrPZ/Mknn5glmadOnfrQ7wUA0hpD24CTateunW7duqXvvvtO169f13fffffAYW1JVhXCy5cv6+rVq6pTp4727NljOIZOnTopR44clvfPP/+88uXLp//973+p9jebzYqIiFDbtm1VvXp1w58rWV/P7du3dfHiRdWoUUOS/vOa/ve//ykgIEAvvfSSZV+2bNnUt29fxcXFadOmTY8VW2r+/vtvTZ06VcOGDZOXl1eK9ixZsqh///5W+wYOHChJ+v777y1xS9KAAQMe2u9+q1at0htvvKG33nor1WkDAGBPJJKAk8qbN6/CwsK0aNEirVixQklJSXr++ecf2P+7775TjRo15O7urly5cilv3ryaOXOmrl69ajiGkJAQq/cmk0lFixbVyZMnU+2/cOFCHTx4UB988IHhz7wnNjZWb775pvz9/eXh4aG8efMqODhYkv7zmv766y+FhITIxcX6j7iSJUta2tPaiBEjFBgYqNdffz1Fm8lkUmBgoLy9va32Fy9eXC4uLpbv86+//pKLi4uKFi1q1S8gIEC+vr4p4o6OjtZLL72kpKQkxcbGpu0FAcAjYI4k4MQ6dOig1157TTExMWrcuLF8fX1T7ffLL7+oRYsWqlu3rmbMmKF8+fIpW7ZsmjdvnhYtWpQusd65c0fDhg1T9+7dVaxYscc+X7t27bRt2za99dZbqlChgry8vJScnKxGjRqluEHF0Q4fPqz58+fryy+/VLZs2VK0P2w+aWoeNv/zfvv27VPjxo31zDPP6K233tLLL7/M/EgA6YpEEnBirVu31uuvv65ff/1VixcvfmC/5cuXy93dXT/++KPVGpPz5s17rM8/evSo1Xuz2aw///xT5cqVS9F3xowZOn/+fIq7mI24fPmy1q9fr1GjRmn48OEPjOdBgoKCtH//fiUnJ1tVJX///XdLe1qKiIhQhQoV9OKLL6baHhwcrLVr1+r69etWUwX++OMPJScnW+5YDwoKUnJyso4ePWqpnkrSuXPndOXKlRRxly1bVkuXLpWHh4eWLl2qHj16aP/+/XJ3d0/T6wOAB2FoG3BiXl5emjlzpkaOHKnmzZs/sF+WLFlkMpmUlJRk2Xfy5EmtXLnysT7/888/1/Xr1y3vly1bprNnz1ruJL/n+vXrev/999W/f/80WeMyS5Ysku4mrvebNGnSIx3fpEkTxcTEWCXfiYmJmjp1qry8vFSvXr3HjvGeqKgorVq1Sh9++OEDK4lNmjRRUlKSpk2bZrV/woQJkmS5q75JkyaSUl7nv/vdU6lSJXl6esrFxUVz587VyZMnNXr06Me+JgB4VFQkASfXuXPn/+zTtGlTTZgwQY0aNVKHDh10/vx5TZ8+XUWLFtX+/fsNf3auXLlUu3Ztde3aVefOndOkSZNUtGhRvfbaa1b99uzZozx58mjw4MH/ec5Tp05pzZo1VvsuXLigW7duac2aNapXr568vb1Vt25djR07VgkJCcqfP7/Wrl2rEydOPFLcPXr00CeffKIuXbpo9+7dKlSokJYtW6atW7dq0qRJVlVBSTpy5IhVTHFxcXJxcbHad/z48VQ/a+3atXr22WcVFhb2wHiaNGmisLAwvfvuuzpx4oQqVKigDRs2aPny5erZs6fKlCkjSSpfvrw6d+6s2bNn68qVK6pXr5527NihBQsWqFWrVmrQoMEDP6NMmTIaMmSIPvzwQ7Vv3z7VqjEApDkH3zUO4D73L//zMKkt//Ppp5+aQ0JCzG5ubuYSJUqY582bZx4xYkSKJXj+/VkPW/7nq6++MkdERJj9/PzMHh4e5qZNm5r/+usvq773lraZOHGi1f7UPlvSf2734jlz5oy5devWZl9fX7OPj4/5hRdeMP/zzz8pltx5kHPnzpm7du1qzpMnj9nV1dVctmxZ87x586z63Fv+x5bt38v/mEwm8+7du1N8J/9ePiguLs7cv39/c2BgoDlbtmzmokWLmj/88ENzUlKSVb+EhATzqFGjzMHBweZs2bKZCxQoYI6IiDDfvn3bqt/9y//cc/v2bXOJEiXMVatWNScmJv7ndwQAj8tkNv9r7AhApvfzzz+rQYMGWrp06UPvFE9LJ0+eVHBwsE6cOJHiKTcAAOfEHEkAAAAYQiIJwCl4eHioYcOGNi+VAwBwHG62AeAU/P39U9y
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 800x600 with 2 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import time\n",
|
|||
|
|
"import seaborn as sns\n",
|
|||
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
|||
|
|
"from sklearn.metrics import mean_squared_error\n",
|
|||
|
|
"from sklearn.metrics import r2_score, accuracy_score, precision_recall_fscore_support, confusion_matrix\n",
|
|||
|
|
"from sklearn.model_selection import cross_val_score\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение данных на обучающую и валидационную выборки. Удаляем целевую переменную\n",
|
|||
|
|
"X = feature_matrix.drop('hazardous_True', axis=1)\n",
|
|||
|
|
"y = feature_matrix['hazardous_True']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# One-hot encoding для категориальных переменных (преобразование категориальных объектов в числовые)\n",
|
|||
|
|
"X = pd.get_dummies(X, drop_first=True)\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Обучение модели\n",
|
|||
|
|
"model = LogisticRegression()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Начинаем отсчет времени\n",
|
|||
|
|
"start_time = time.time()\n",
|
|||
|
|
"model.fit(X_train, y_train)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Время обучения модели\n",
|
|||
|
|
"train_time = time.time() - start_time\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Предсказания и оценка модели\n",
|
|||
|
|
"start_time = time.time()\n",
|
|||
|
|
"predictions = model.predict(X_test)\n",
|
|||
|
|
"pred_time = time.time() - start_time\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Предсказательная способность\n",
|
|||
|
|
"accuracy = accuracy_score(y_test, predictions)\n",
|
|||
|
|
"precision, recall, f1, _ = precision_recall_fscore_support(y_test, predictions, average='binary')\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f\"Accuracy: {accuracy}\")\n",
|
|||
|
|
"print(f\"Precision: {precision}\")\n",
|
|||
|
|
"print(f\"Полнота: {recall}\")\n",
|
|||
|
|
"print(f\"F1: {f1}\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Надежность\n",
|
|||
|
|
"cv_scores = cross_val_score(model, X, y, cv=5)\n",
|
|||
|
|
"cv_mean = cv_scores.mean()\n",
|
|||
|
|
"cv_std = cv_scores.std()\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f\"Cross-Validation Mean: {cv_mean}\")\n",
|
|||
|
|
"print(f\"Cross-Validation Std: {cv_std}\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(f'Время обучения модели: {train_time:.2f} секунд')\n",
|
|||
|
|
"print(f'Скорость вычисления: {pred_time:.2f} секунд')\n",
|
|||
|
|
"\n",
|
|||
|
|
"\n",
|
|||
|
|
"mse = mean_squared_error(y_test, predictions)\n",
|
|||
|
|
"print(f'Среднеквадратичная ошибка: {mse:.2f}')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Визуализация предсказаний\n",
|
|||
|
|
"confusions = confusion_matrix(y_test, predictions)\n",
|
|||
|
|
"plt.figure(figsize=(8, 6))\n",
|
|||
|
|
"sns.heatmap(confusions, annot=True, fmt='d', cmap='Blues', xticklabels=['Неопасный', 'Опасный'], yticklabels=['Неопасный', 'Опасный'])\n",
|
|||
|
|
"plt.xlabel('Предсказанное значение')\n",
|
|||
|
|
"plt.ylabel('Истинное значение')\n",
|
|||
|
|
"plt.title('Матрица ошибок')\n",
|
|||
|
|
"plt.show()"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"metadata": {
|
|||
|
|
"kernelspec": {
|
|||
|
|
"display_name": "aimenv",
|
|||
|
|
"language": "python",
|
|||
|
|
"name": "python3"
|
|||
|
|
},
|
|||
|
|
"language_info": {
|
|||
|
|
"codemirror_mode": {
|
|||
|
|
"name": "ipython",
|
|||
|
|
"version": 3
|
|||
|
|
},
|
|||
|
|
"file_extension": ".py",
|
|||
|
|
"mimetype": "text/x-python",
|
|||
|
|
"name": "python",
|
|||
|
|
"nbconvert_exporter": "python",
|
|||
|
|
"pygments_lexer": "ipython3",
|
|||
|
|
"version": "3.12.5"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"nbformat": 4,
|
|||
|
|
"nbformat_minor": 2
|
|||
|
|
}
|