{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Лабораторная работа №3\n",
    "\n",
    "### Набор данных \"Наблюдения НЛО в США\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Для набора данных \"Наблюдения НЛО в США\" можно выделить несколько бизнес-целей и соответствующие технические задачи. Давайте рассмотрим этот процесс поэтапно.\n",
    "#  \n",
    "#  1. Определение бизнес-целей\n",
    "#      Бизнес-цель 1: Прогнозирование местоположения и частоты наблюдений НЛО.\n",
    "#          Задача заключается в анализе географического распределения и времени наблюдений НЛО, чтобы определить, в каких местах и когда чаще всего происходят наблюдения.\n",
    "#      Бизнес-цель 2: Анализ факторов, влияющих на восприятие НЛО (например, форма, продолжительность, описание).\n",
    "#          Цель — понять, какие признаки, такие как форма НЛО, длительность наблюдения, могут быть связаны с более подробными или более эмоционально окрашенными отчетами.\n",
    "#  2. Цели технического проекта для каждой бизнес-цели\n",
    "#      Цель для бизнес-цели 1: Создать модель, которая предскажет вероятное местоположение и время наблюдений на основе данных о предыдущих наблюдениях.\n",
    "#          Технические задачи:\n",
    "#              Прогнозирование местоположения и времени (классификация или регрессия).\n",
    "#              Кластеризация по географическому положению.\n",
    "#              Анализ временных рядов для выявления сезонных колебаний.\n",
    "#      Цель для бизнес-цели 2: Анализировать текстовые описания наблюдений НЛО для выявления ключевых паттернов и факторов.\n",
    "#          Технические задачи:\n",
    "#              Анализ текста с использованием методов обработки естественного языка (NLP).\n",
    "#              Классификация описаний по типам объектов или возможным объяснениям (например, возможный самолет или атмосферное явление)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['summary', 'city', 'state', 'date_time', 'shape', 'duration', 'stats',\n",
      "       'report_link', 'text', 'posted', 'city_latitude', 'city_longitude'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import seaborn as sns\n",
    "\n",
    "# Загрузка данных\n",
    "df = pd.read_csv(\"../../datasets/nuforc_reports.csv\")\n",
    "\n",
    "# Срез данных, первые 15000 строк\n",
    "df = df.iloc[:15000]\n",
    "\n",
    "# Вывод\n",
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>summary</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>date_time</th>\n",
       "      <th>shape</th>\n",
       "      <th>duration</th>\n",
       "      <th>stats</th>\n",
       "      <th>report_link</th>\n",
       "      <th>text</th>\n",
       "      <th>posted</th>\n",
       "      <th>city_latitude</th>\n",
       "      <th>city_longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Viewed some red lights in the sky appearing to...</td>\n",
       "      <td>Visalia</td>\n",
       "      <td>CA</td>\n",
       "      <td>2021-12-15T21:45:00</td>\n",
       "      <td>light</td>\n",
       "      <td>2 minutes</td>\n",
       "      <td>Occurred : 12/15/2021 21:45  (Entered as : 12/...</td>\n",
       "      <td>http://www.nuforc.org/webreports/165/S165881.html</td>\n",
       "      <td>Viewed some red lights in the sky appearing to...</td>\n",
       "      <td>2021-12-19T00:00:00</td>\n",
       "      <td>36.356650</td>\n",
       "      <td>-119.347937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Look like 1 or 3 crafts from North traveling s...</td>\n",
       "      <td>Cincinnati</td>\n",
       "      <td>OH</td>\n",
       "      <td>2021-12-16T09:45:00</td>\n",
       "      <td>triangle</td>\n",
       "      <td>14 seconds</td>\n",
       "      <td>Occurred : 12/16/2021 09:45  (Entered as : 12/...</td>\n",
       "      <td>http://www.nuforc.org/webreports/165/S165888.html</td>\n",
       "      <td>Look like 1 or 3 crafts from North traveling s...</td>\n",
       "      <td>2021-12-19T00:00:00</td>\n",
       "      <td>39.174503</td>\n",
       "      <td>-84.481363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>seen dark rectangle moving slowly thru the sky...</td>\n",
       "      <td>Tecopa</td>\n",
       "      <td>CA</td>\n",
       "      <td>2021-12-10T00:00:00</td>\n",
       "      <td>rectangle</td>\n",
       "      <td>Several minutes</td>\n",
       "      <td>Occurred : 12/10/2021 00:00  (Entered as : 12/...</td>\n",
       "      <td>http://www.nuforc.org/webreports/165/S165810.html</td>\n",
       "      <td>seen dark rectangle moving slowly thru the sky...</td>\n",
       "      <td>2021-12-19T00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>One red light moving switly west to east, beco...</td>\n",
       "      <td>Knoxville</td>\n",
       "      <td>TN</td>\n",
       "      <td>2021-12-10T19:30:00</td>\n",
       "      <td>triangle</td>\n",
       "      <td>20-30 seconds</td>\n",
       "      <td>Occurred : 12/10/2021 19:30  (Entered as : 12/...</td>\n",
       "      <td>http://www.nuforc.org/webreports/165/S165825.html</td>\n",
       "      <td>One red light moving switly west to east, beco...</td>\n",
       "      <td>2021-12-19T00:00:00</td>\n",
       "      <td>35.961561</td>\n",
       "      <td>-83.980115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bright, circular Fresnel-lens shaped light sev...</td>\n",
       "      <td>Alexandria</td>\n",
       "      <td>VA</td>\n",
       "      <td>2021-12-07T08:00:00</td>\n",
       "      <td>circle</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Occurred : 12/7/2021 08:00  (Entered as : 12/0...</td>\n",
       "      <td>http://www.nuforc.org/webreports/165/S165754.html</td>\n",
       "      <td>Bright, circular Fresnel-lens shaped light sev...</td>\n",
       "      <td>2021-12-19T00:00:00</td>\n",
       "      <td>38.798958</td>\n",
       "      <td>-77.095133</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             summary        city state  \\\n",
       "0  Viewed some red lights in the sky appearing to...     Visalia    CA   \n",
       "1  Look like 1 or 3 crafts from North traveling s...  Cincinnati    OH   \n",
       "2  seen dark rectangle moving slowly thru the sky...      Tecopa    CA   \n",
       "3  One red light moving switly west to east, beco...   Knoxville    TN   \n",
       "4  Bright, circular Fresnel-lens shaped light sev...  Alexandria    VA   \n",
       "\n",
       "             date_time      shape         duration  \\\n",
       "0  2021-12-15T21:45:00      light        2 minutes   \n",
       "1  2021-12-16T09:45:00   triangle       14 seconds   \n",
       "2  2021-12-10T00:00:00  rectangle  Several minutes   \n",
       "3  2021-12-10T19:30:00   triangle    20-30 seconds   \n",
       "4  2021-12-07T08:00:00     circle              NaN   \n",
       "\n",
       "                                               stats  \\\n",
       "0  Occurred : 12/15/2021 21:45  (Entered as : 12/...   \n",
       "1  Occurred : 12/16/2021 09:45  (Entered as : 12/...   \n",
       "2  Occurred : 12/10/2021 00:00  (Entered as : 12/...   \n",
       "3  Occurred : 12/10/2021 19:30  (Entered as : 12/...   \n",
       "4  Occurred : 12/7/2021 08:00  (Entered as : 12/0...   \n",
       "\n",
       "                                         report_link  \\\n",
       "0  http://www.nuforc.org/webreports/165/S165881.html   \n",
       "1  http://www.nuforc.org/webreports/165/S165888.html   \n",
       "2  http://www.nuforc.org/webreports/165/S165810.html   \n",
       "3  http://www.nuforc.org/webreports/165/S165825.html   \n",
       "4  http://www.nuforc.org/webreports/165/S165754.html   \n",
       "\n",
       "                                                text               posted  \\\n",
       "0  Viewed some red lights in the sky appearing to...  2021-12-19T00:00:00   \n",
       "1  Look like 1 or 3 crafts from North traveling s...  2021-12-19T00:00:00   \n",
       "2  seen dark rectangle moving slowly thru the sky...  2021-12-19T00:00:00   \n",
       "3  One red light moving switly west to east, beco...  2021-12-19T00:00:00   \n",
       "4  Bright, circular Fresnel-lens shaped light sev...  2021-12-19T00:00:00   \n",
       "\n",
       "   city_latitude  city_longitude  \n",
       "0      36.356650     -119.347937  \n",
       "1      39.174503      -84.481363  \n",
       "2            NaN             NaN  \n",
       "3      35.961561      -83.980115  \n",
       "4      38.798958      -77.095133  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Для наглядности\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 15000 entries, 0 to 14999\n",
      "Data columns (total 12 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   summary         14998 non-null  object \n",
      " 1   city            14961 non-null  object \n",
      " 2   state           14235 non-null  object \n",
      " 3   date_time       14560 non-null  object \n",
      " 4   shape           13082 non-null  object \n",
      " 5   duration        13598 non-null  object \n",
      " 6   stats           15000 non-null  object \n",
      " 7   report_link     15000 non-null  object \n",
      " 8   text            14999 non-null  object \n",
      " 9   posted          14560 non-null  object \n",
      " 10  city_latitude   12002 non-null  float64\n",
      " 11  city_longitude  12002 non-null  float64\n",
      "dtypes: float64(2), object(10)\n",
      "memory usage: 1.4+ MB\n"
     ]
    }
   ],
   "source": [
    "# Описание данных (основные статистические показатели)\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Количество пропущенных значений в каждом столбце:\n",
      "summary              74\n",
      "city                382\n",
      "state              9345\n",
      "date_time          2668\n",
      "shape              5922\n",
      "duration           6492\n",
      "stats                 0\n",
      "report_link           0\n",
      "text                 38\n",
      "posted             2668\n",
      "city_latitude     26804\n",
      "city_longitude    26804\n",
      "dtype: int64\n",
      "summary Процент пустых значений: %0.05\n",
      "city Процент пустых значений: %0.28\n",
      "state Процент пустых значений: %6.82\n",
      "date_time Процент пустых значений: %1.95\n",
      "shape Процент пустых значений: %4.32\n",
      "duration Процент пустых значений: %4.74\n",
      "text Процент пустых значений: %0.03\n",
      "posted Процент пустых значений: %1.95\n",
      "city_latitude Процент пустых значений: %19.57\n",
      "city_longitude Процент пустых значений: %19.57\n",
      "Количество выбросов в столбце 'city_latitude': 1025\n",
      "Количество выбросов в столбце 'city_longitude': 23\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 62120 characters]",
      "text/plain": [
       "<Figure size 1500x1000 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Количество удаленных строк: 27832\n",
      "Количество выбросов в столбце 'city_latitude': 38\n",
      "Количество выбросов в столбце 'city_longitude': 0\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 76348 characters]",
      "text/plain": [
       "<Figure size 1500x1000 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Средняя цена в обучающей выборке: 38.655686510239775\n",
      "Средняя цена в контрольной выборке: 38.6586177634688\n",
      "Средняя цена в тестовой выборке: 38.61544768163524\n",
      "\n",
      "Стандартное отклонение цены в обучающей выборке: 5.380551235399826\n",
      "Стандартное отклонение цены в контрольной выборке: 5.34170765011401\n",
      "Стандартное отклонение цены в тестовой выборке: 5.3932492782181525\n",
      "\n",
      "Распределение по квартилам (обучающая):\n",
      "0.25    34.269424\n",
      "0.50    39.222500\n",
      "0.75    42.284678\n",
      "Name: city_latitude, dtype: float64\n",
      "\n",
      "Распределение по квартилам (контрольная):\n",
      "0.25    34.286571\n",
      "0.50    39.302247\n",
      "0.75    42.277381\n",
      "Name: city_latitude, dtype: float64\n",
      "\n",
      "Распределение по квартилам (тестовая):\n",
      "0.25    34.194501\n",
      "0.50    39.165900\n",
      "0.75    42.286500\n",
      "Name: city_latitude, dtype: float64\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 89172 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "# Загрузка данных\n",
    "df = pd.read_csv(\"../../datasets/nuforc_reports.csv\")\n",
    "\n",
    "\n",
    "#5. Устранение пропущенных данных\n",
    "    \n",
    "#Сведения о пропущенных данных\n",
    "print(\"Количество пропущенных значений в каждом столбце:\")\n",
    "print(df.isnull().sum())\n",
    "\n",
    "# Процент пропущенных значений признаков\n",
    "for i in df.columns:\n",
    "    null_rate = df[i].isnull().sum() / len(df) * 100\n",
    "    if null_rate > 0:\n",
    "        print(f'{i} Процент пустых значений: %{null_rate:.2f}')\n",
    "\n",
    "\n",
    "\n",
    "#6. Проблемы набора данных\n",
    "    #5.1Выбросы: Возможны аномалии в значениях скорости или расстояния.\n",
    "    #Смещение: Данные могут быть смещены в сторону объектов, которые легче обнаружить (крупные, близкие).\n",
    "\n",
    "#7. Решения для обнаруженных проблем\n",
    "    #Выбросы: Идентификация и обработка выбросов через методы (например, IQR или Z-оценка).\n",
    "    #Смещение: Использование методов балансировки данных, таких как oversampling.\n",
    "\n",
    "#7.1 Проверка набора данных на выбросы\n",
    "# Выбираем столбцы для анализа\n",
    "columns_to_check = df.select_dtypes(include=np.number).columns.tolist()#['city_latitude'  , 'sqft_living', 'bathrooms', 'yr_built']\n",
    "def Emissions(columns_to_check):\n",
    "\n",
    "    # Функция для подсчета выбросов\n",
    "    def count_outliers(df, columns):\n",
    "        outliers_count = {}\n",
    "        for col in columns:\n",
    "            Q1 = df[col].quantile(0.25)\n",
    "            Q3 = df[col].quantile(0.75)\n",
    "            IQR = Q3 - Q1\n",
    "            lower_bound = Q1 - 1.5 * IQR\n",
    "            upper_bound = Q3 + 1.5 * IQR\n",
    "            \n",
    "            # Считаем количество выбросов\n",
    "            outliers = df[(df[col] < lower_bound) | (df[col] > upper_bound)]\n",
    "            outliers_count[col] = len(outliers)\n",
    "        \n",
    "        return outliers_count\n",
    "\n",
    "    # Подсчитываем выбросы\n",
    "    outliers_count = count_outliers(df, columns_to_check)\n",
    "\n",
    "    # Выводим количество выбросов для каждого столбца\n",
    "    for col, count in outliers_count.items():\n",
    "        print(f\"Количество выбросов в столбце '{col}': {count}\")\n",
    "        \n",
    "    # Создаем гистограммы\n",
    "    plt.figure(figsize=(15, 10))\n",
    "    for i, col in enumerate(columns_to_check, 1):\n",
    "        plt.subplot(2, 3, i)\n",
    "        sns.histplot(df[col], kde=True)\n",
    "        plt.title(f'Histogram of {col}')\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "Emissions(columns_to_check)\n",
    "\n",
    "#Признак miss_distance не имеет выбросов, \n",
    "#признак absolute_magnitude имеет количество выбросов в приемлемом диапазоне\n",
    "#для признаков est_diameter_min, est_diameter_max и relative_velocity необходимо использовать метод решения проблемы выбросов. \n",
    "#Воспользуемся методом удаления наблюдений с такими выбросами:\n",
    "# Выбираем столбцы для очистки\n",
    "columns_to_clean = ['city_latitude']\n",
    "\n",
    "# Функция для удаления выбросов\n",
    "def remove_outliers(df, columns):\n",
    "    for col in columns:\n",
    "        Q1 = df[col].quantile(0.25)\n",
    "        Q3 = df[col].quantile(0.75)\n",
    "        IQR = Q3 - Q1\n",
    "        lower_bound = Q1 - 1.5 * IQR\n",
    "        upper_bound = Q3 + 1.5 * IQR\n",
    "        \n",
    "        # Удаляем строки, содержащие выбросы\n",
    "        df = df[(df[col] >= lower_bound) & (df[col] <= upper_bound)]\n",
    "    \n",
    "    return df\n",
    "\n",
    "# Удаляем выбросы\n",
    "df_cleaned = remove_outliers(df, columns_to_clean)\n",
    "\n",
    "# Выводим количество удаленных строк\n",
    "print(f\"Количество удаленных строк: {len(df) - len(df_cleaned)}\")\n",
    "\n",
    "df = df_cleaned\n",
    "\n",
    "#Оценим выбросы в выборке после усреднения:\n",
    "Emissions(columns_to_clean)\n",
    "\n",
    "#Удалось избавиться от выбросов в соответствующих признаках как видно на диаграммах.\n",
    "\n",
    "\n",
    "\n",
    "#8. Разбиение данных на выборки\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_data, temp_data = train_test_split(df, test_size=0.3, random_state=42)\n",
    "val_data, test_data = train_test_split(temp_data, test_size=0.5, random_state=42)\n",
    "\n",
    "# Средние значения цены\n",
    "print(\"Средняя цена в обучающей выборке:\", train_data['city_latitude'  ].mean())\n",
    "print(\"Средняя цена в контрольной выборке:\", val_data['city_latitude'  ].mean())\n",
    "print(\"Средняя цена в тестовой выборке:\", test_data['city_latitude'  ].mean())\n",
    "print()\n",
    "\n",
    "# Стандартное отклонение цены\n",
    "print(\"Стандартное отклонение цены в обучающей выборке:\", train_data['city_latitude'  ].std())\n",
    "print(\"Стандартное отклонение цены в контрольной выборке:\", val_data['city_latitude'  ].std())\n",
    "print(\"Стандартное отклонение цены в тестовой выборке:\", test_data['city_latitude'  ].std())\n",
    "print()\n",
    "\n",
    "# Проверка распределений по количеству объектов в диапазонах\n",
    "print(\"Распределение по квартилам (обучающая):\")\n",
    "print(train_data['city_latitude'  ].quantile([0.25, 0.5, 0.75]))\n",
    "print()\n",
    "print(\"Распределение по квартилам (контрольная):\")\n",
    "print(val_data['city_latitude'  ].quantile([0.25, 0.5, 0.75]))\n",
    "print()\n",
    "print(\"Распределение по квартилам (тестовая):\")\n",
    "print(test_data['city_latitude'  ].quantile([0.25, 0.5, 0.75]))\n",
    "\n",
    "# Построение гистограмм для каждой выборки\n",
    "plt.figure(figsize=(12, 6))\n",
    "\n",
    "sns.histplot(train_data['city_latitude'  ], color='blue', label='Train', kde=True)\n",
    "sns.histplot(val_data['city_latitude'  ], color='green', label='Validation', kde=True)\n",
    "sns.histplot(test_data['city_latitude'  ], color='red', label='Test', kde=True)\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('city_latitude'  )\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Распределение цены в обучающей, контрольной и тестовой выборках')\n",
    "plt.show()\n",
    "\n",
    "\n",
    "#9. Оценить  сбалансированность  выборок  для  каждого  набора  данных.  Оценить необходимость использования методов приращения (аугментации) данных. \n",
    "#Выводы по сбалансированности\n",
    "#Если распределение классов примерно равно (например, 50%/50%), выборка считается сбалансированной, и аугментация данных не требуется.\n",
    "#Если один из классов сильно доминирует (например, 90%/10%), выборка несбалансированная, и может потребоваться аугментация данных.\n",
    "\n",
    "#Выборки оказались недостаточно сбалансированными. Используем методы приращения данных с избытком и с недостатком:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Нет пропущенных данных."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Разбиваем на выборки (обучающую, тестовую, контрольную)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Размеры выборок:\n",
      "Обучающая выборка: 65464 записей\n",
      "Валидационная выборка: 21822 записей\n",
      "Тестовая выборка: 21822 записей\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 53292 characters]",
      "text/plain": [
       "<Figure size 1800x600 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Разделение признаков (features) и целевой переменной (target)\n",
    "X = df.drop(columns=['city_latitude'])  # Признаки (все столбцы, кроме 'city_latitude')\n",
    "y = df['city_latitude'] \n",
    "# Целевая переменная (price)\n",
    "\n",
    "# Разбиение на обучающую (60%), валидационную (20%) и тестовую (20%) выборки\n",
    "X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)\n",
    "X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)\n",
    "\n",
    "# Проверка размеров выборок\n",
    "print(f\"Размеры выборок:\")\n",
    "print(f\"Обучающая выборка: {X_train.shape[0]} записей\")\n",
    "print(f\"Валидационная выборка: {X_val.shape[0]} записей\")\n",
    "print(f\"Тестовая выборка: {X_test.shape[0]} записей\")\n",
    "\n",
    "# Визуализация распределения цен в каждой выборке\n",
    "plt.figure(figsize=(18, 6))\n",
    "\n",
    "plt.subplot(1, 3, 1)\n",
    "plt.hist(y_train, bins=30, color='blue', alpha=0.7)\n",
    "plt.title('Обучающая выборка')\n",
    "plt.xlabel('Цена')\n",
    "plt.ylabel('Количество')\n",
    "\n",
    "plt.subplot(1, 3, 2)\n",
    "plt.hist(y_val, bins=30, color='green', alpha=0.7)\n",
    "plt.title('Валидационная выборка')\n",
    "plt.xlabel('Цена')\n",
    "plt.ylabel('Количество')\n",
    "\n",
    "plt.subplot(1, 3, 3)\n",
    "plt.hist(y_test, bins=30, color='red', alpha=0.7)\n",
    "plt.title('Тестовая выборка')\n",
    "plt.xlabel('Цена')\n",
    "plt.ylabel('Количество')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Балансировка выборок**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Размеры выборок:\n",
      "Обучающая выборка: 6000 записей\n",
      "Валидационная выборка: 2000 записей\n",
      "Тестовая выборка: 2000 записей\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 33884 characters]",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Разделение признаков (features) и целевой переменной (target)\n",
    "X = df.drop(columns=['city_latitude']).head(10000)  # Признаки (все столбцы, кроме 'city_latitude')\n",
    "y = df['city_latitude'].head(10000)  # Целевая переменная (цена)\n",
    "\n",
    "# Применение one-hot encoding для категориальных признаков\n",
    "X = pd.get_dummies(X, drop_first=True)\n",
    "\n",
    "# Разбиение на обучающую (60%), валидационную (20%) и тестовую (20%) выборки\n",
    "X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)\n",
    "X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)\n",
    "\n",
    "# Проверка размеров выборок\n",
    "print(f\"Размеры выборок:\")\n",
    "print(f\"Обучающая выборка: {X_train.shape[0]} записей\")\n",
    "print(f\"Валидационная выборка: {X_val.shape[0]} записей\")\n",
    "print(f\"Тестовая выборка: {X_test.shape[0]} записей\")\n",
    "\n",
    "# Удаление выбросов (цены выше 95-го процентиля)\n",
    "upper_limit = y_train.quantile(0.95)\n",
    "X_train = X_train[y_train <= upper_limit]\n",
    "y_train = y_train[y_train <= upper_limit]\n",
    "\n",
    "# Логарифмическое преобразование целевой переменной\n",
    "y_train_log = np.log1p(y_train)\n",
    "y_val_log = np.log1p(y_val)\n",
    "y_test_log = np.log1p(y_test)\n",
    "\n",
    "# Стандартизация признаков\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_val_scaled = scaler.transform(X_val)\n",
    "X_test_scaled = scaler.transform(X_test)\n",
    "\n",
    "# Визуализация распределения цен в сбалансированной выборке\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.hist(y_train_log, bins=30, color='orange', alpha=0.7)\n",
    "plt.title('Сбалансированная обучающая выборка (логарифмическое преобразование)')\n",
    "plt.xlabel('Логарифм цены')\n",
    "plt.ylabel('Количество')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Унитарное кодирование категориальных признаков**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Данные до унитарного кодирования:\n",
      "                                                 summary           city state  \\\n",
      "0      Viewed some red lights in the sky appearing to...        Visalia    CA   \n",
      "1      Look like 1 or 3 crafts from North traveling s...     Cincinnati    OH   \n",
      "3      One red light moving switly west to east, beco...      Knoxville    TN   \n",
      "4      Bright, circular Fresnel-lens shaped light sev...     Alexandria    VA   \n",
      "5      I'm familiar with all the fakery and UFO sight...      Fullerton    CA   \n",
      "...                                                  ...            ...   ...   \n",
      "12494                    star like stop and go satellite  San Francisco    CA   \n",
      "12495           Two Balls of Light sighted in Tulsa, Ok.          Tulsa    OK   \n",
      "12496  Highley reflective silver oval/disk seen in sk...       St Louis    MO   \n",
      "12497  While walking on broadwalk on south west corne...          Tempe    AZ   \n",
      "12498  was sitting on my front porch with granddaught...        Elkhart    KS   \n",
      "\n",
      "                 date_time     shape         duration  \\\n",
      "0      2021-12-15T21:45:00     light        2 minutes   \n",
      "1      2021-12-16T09:45:00  triangle       14 seconds   \n",
      "3      2021-12-10T19:30:00  triangle    20-30 seconds   \n",
      "4      2021-12-07T08:00:00    circle              NaN   \n",
      "5      2020-07-07T23:00:00   unknown        2 minutes   \n",
      "...                    ...       ...              ...   \n",
      "12494  2000-06-05T23:30:00       NaN         1 minute   \n",
      "12495  2000-06-06T08:15:00    circle  about 5 minutes   \n",
      "12496  2000-06-06T17:22:00      oval       one minute   \n",
      "12497  2000-06-07T22:00:00   unknown        4 seconds   \n",
      "12498  2000-06-07T23:48:00    circle     1to2 minutes   \n",
      "\n",
      "                                                   stats  \\\n",
      "0      Occurred : 12/15/2021 21:45  (Entered as : 12/...   \n",
      "1      Occurred : 12/16/2021 09:45  (Entered as : 12/...   \n",
      "3      Occurred : 12/10/2021 19:30  (Entered as : 12/...   \n",
      "4      Occurred : 12/7/2021 08:00  (Entered as : 12/0...   \n",
      "5      Occurred : 7/7/2020 23:00  (Entered as : 07/07...   \n",
      "...                                                  ...   \n",
      "12494  Occurred : 6/5/2000 23:30  (Entered as : 06/05...   \n",
      "12495  Occurred : 6/6/2000 08:15  (Entered as : 06/06...   \n",
      "12496  Occurred : 6/6/2000 17:22  (Entered as : 06/06...   \n",
      "12497  Occurred : 6/7/2000 22:00  (Entered as : 060/7...   \n",
      "12498  Occurred : 6/7/2000 23:48  (Entered as : 6/7/2...   \n",
      "\n",
      "                                             report_link  \\\n",
      "0      http://www.nuforc.org/webreports/165/S165881.html   \n",
      "1      http://www.nuforc.org/webreports/165/S165888.html   \n",
      "3      http://www.nuforc.org/webreports/165/S165825.html   \n",
      "4      http://www.nuforc.org/webreports/165/S165754.html   \n",
      "5      http://www.nuforc.org/webreports/157/S157444.html   \n",
      "...                                                  ...   \n",
      "12494   http://www.nuforc.org/webreports/013/S13042.html   \n",
      "12495   http://www.nuforc.org/webreports/013/S13150.html   \n",
      "12496   http://www.nuforc.org/webreports/013/S13043.html   \n",
      "12497   http://www.nuforc.org/webreports/013/S13054.html   \n",
      "12498   http://www.nuforc.org/webreports/013/S13051.html   \n",
      "\n",
      "                                                    text               posted  \\\n",
      "0      Viewed some red lights in the sky appearing to...  2021-12-19T00:00:00   \n",
      "1      Look like 1 or 3 crafts from North traveling s...  2021-12-19T00:00:00   \n",
      "3      One red light moving switly west to east, beco...  2021-12-19T00:00:00   \n",
      "4      Bright, circular Fresnel-lens shaped light sev...  2021-12-19T00:00:00   \n",
      "5      I'm familiar with all the fakery and UFO sight...  2020-07-09T00:00:00   \n",
      "...                                                  ...                  ...   \n",
      "12494  star like stop and go satellite A white star-l...  2000-06-21T00:00:00   \n",
      "12495  Two Balls of Light sighted in Tulsa, Ok. At su...  2000-06-21T00:00:00   \n",
      "12496  Highley reflective silver oval/disk seen in sk...  2000-06-21T00:00:00   \n",
      "12497  On southwest corner of Tempe Lake heard and sa...  2000-06-21T00:00:00   \n",
      "12498  was sitting on my front porch with granddaught...  2000-06-21T00:00:00   \n",
      "\n",
      "       city_latitude  city_longitude  \n",
      "0          36.356650     -119.347937  \n",
      "1          39.174503      -84.481363  \n",
      "3          35.961561      -83.980115  \n",
      "4          38.798958      -77.095133  \n",
      "5          33.877422     -117.924978  \n",
      "...              ...             ...  \n",
      "12494      37.769992     -122.425394  \n",
      "12495      36.109456      -95.935245  \n",
      "12496      38.623825      -90.308528  \n",
      "12497      33.414036     -111.920920  \n",
      "12498      37.046000     -101.853100  \n",
      "\n",
      "[10000 rows x 12 columns]\n",
      "\n",
      "Данные после унитарного кодирования:\n",
      "       city_latitude  city_longitude  \\\n",
      "0          36.356650     -119.347937   \n",
      "1          39.174503      -84.481363   \n",
      "3          35.961561      -83.980115   \n",
      "4          38.798958      -77.095133   \n",
      "5          33.877422     -117.924978   \n",
      "...              ...             ...   \n",
      "12494      37.769992     -122.425394   \n",
      "12495      36.109456      -95.935245   \n",
      "12496      38.623825      -90.308528   \n",
      "12497      33.414036     -111.920920   \n",
      "12498      37.046000     -101.853100   \n",
      "\n",
      "       summary_ A couple stopped me and told me &quot;Am I crazy or those lights are moving?  ((Starlink satellites?))  \\\n",
      "0                                                  False                                                                 \n",
      "1                                                  False                                                                 \n",
      "3                                                  False                                                                 \n",
      "4                                                  False                                                                 \n",
      "5                                                  False                                                                 \n",
      "...                                                  ...                                                                 \n",
      "12494                                              False                                                                 \n",
      "12495                                              False                                                                 \n",
      "12496                                              False                                                                 \n",
      "12497                                              False                                                                 \n",
      "12498                                              False                                                                 \n",
      "\n",
      "       summary_ All total, I counted 15 aircraft flying in single file with no audible noise ((Starlink satellites?))  \\\n",
      "0                                                  False                                                                \n",
      "1                                                  False                                                                \n",
      "3                                                  False                                                                \n",
      "4                                                  False                                                                \n",
      "5                                                  False                                                                \n",
      "...                                                  ...                                                                \n",
      "12494                                              False                                                                \n",
      "12495                                              False                                                                \n",
      "12496                                              False                                                                \n",
      "12497                                              False                                                                \n",
      "12498                                              False                                                                \n",
      "\n",
      "       summary_ I and 3 others observed 4 lights in the sky.  \\\n",
      "0                                                  False       \n",
      "1                                                  False       \n",
      "3                                                  False       \n",
      "4                                                  False       \n",
      "5                                                  False       \n",
      "...                                                  ...       \n",
      "12494                                              False       \n",
      "12495                                              False       \n",
      "12496                                              False       \n",
      "12497                                              False       \n",
      "12498                                              False       \n",
      "\n",
      "       summary_ I kept watching and counted 10 objects that seemed to come from nowhere just appear out of the blackness.  ((Starlink satellites?))  \\\n",
      "0                                                  False                                                                                              \n",
      "1                                                  False                                                                                              \n",
      "3                                                  False                                                                                              \n",
      "4                                                  False                                                                                              \n",
      "5                                                  False                                                                                              \n",
      "...                                                  ...                                                                                              \n",
      "12494                                              False                                                                                              \n",
      "12495                                              False                                                                                              \n",
      "12496                                              False                                                                                              \n",
      "12497                                              False                                                                                              \n",
      "12498                                              False                                                                                              \n",
      "\n",
      "       summary_ I noticed the stars and another in a perfect line.    ((Starlink satellites))  \\\n",
      "0                                                  False                                        \n",
      "1                                                  False                                        \n",
      "3                                                  False                                        \n",
      "4                                                  False                                        \n",
      "5                                                  False                                        \n",
      "...                                                  ...                                        \n",
      "12494                                              False                                        \n",
      "12495                                              False                                        \n",
      "12496                                              False                                        \n",
      "12497                                              False                                        \n",
      "12498                                              False                                        \n",
      "\n",
      "       summary_ I saw what looked like a star moving then I seen more.  ((Starlink satellites?))  \\\n",
      "0                                                  False                                           \n",
      "1                                                  False                                           \n",
      "3                                                  False                                           \n",
      "4                                                  False                                           \n",
      "5                                                  False                                           \n",
      "...                                                  ...                                           \n",
      "12494                                              False                                           \n",
      "12495                                              False                                           \n",
      "12496                                              False                                           \n",
      "12497                                              False                                           \n",
      "12498                                              False                                           \n",
      "\n",
      "       summary_ I see 5 lights in a line, separated evenly, flying  at the same speed.  ((\"Starlink\" satellites??))((anonymous))  \\\n",
      "0                                                  False                                                                           \n",
      "1                                                  False                                                                           \n",
      "3                                                  False                                                                           \n",
      "4                                                  False                                                                           \n",
      "5                                                  False                                                                           \n",
      "...                                                  ...                                                                           \n",
      "12494                                              False                                                                           \n",
      "12495                                              False                                                                           \n",
      "12496                                              False                                                                           \n",
      "12497                                              False                                                                           \n",
      "12498                                              False                                                                           \n",
      "\n",
      "       summary_ We have witnessed multiple lights looking like satellites with varing degrees of brightness in orbit.   ((Starlink satellites?))  \\\n",
      "0                                                  False                                                                                           \n",
      "1                                                  False                                                                                           \n",
      "3                                                  False                                                                                           \n",
      "4                                                  False                                                                                           \n",
      "5                                                  False                                                                                           \n",
      "...                                                  ...                                                                                           \n",
      "12494                                              False                                                                                           \n",
      "12495                                              False                                                                                           \n",
      "12496                                              False                                                                                           \n",
      "12497                                              False                                                                                           \n",
      "12498                                              False                                                                                           \n",
      "\n",
      "       ...  posted_2020-07-03T00:00:00  posted_2020-07-09T00:00:00  \\\n",
      "0      ...                       False                       False   \n",
      "1      ...                       False                       False   \n",
      "3      ...                       False                       False   \n",
      "4      ...                       False                       False   \n",
      "5      ...                       False                        True   \n",
      "...    ...                         ...                         ...   \n",
      "12494  ...                       False                       False   \n",
      "12495  ...                       False                       False   \n",
      "12496  ...                       False                       False   \n",
      "12497  ...                       False                       False   \n",
      "12498  ...                       False                       False   \n",
      "\n",
      "       posted_2020-07-23T00:00:00  posted_2020-07-31T00:00:00  \\\n",
      "0                           False                       False   \n",
      "1                           False                       False   \n",
      "3                           False                       False   \n",
      "4                           False                       False   \n",
      "5                           False                       False   \n",
      "...                           ...                         ...   \n",
      "12494                       False                       False   \n",
      "12495                       False                       False   \n",
      "12496                       False                       False   \n",
      "12497                       False                       False   \n",
      "12498                       False                       False   \n",
      "\n",
      "       posted_2020-08-06T00:00:00  posted_2020-08-20T00:00:00  \\\n",
      "0                           False                       False   \n",
      "1                           False                       False   \n",
      "3                           False                       False   \n",
      "4                           False                       False   \n",
      "5                           False                       False   \n",
      "...                           ...                         ...   \n",
      "12494                       False                       False   \n",
      "12495                       False                       False   \n",
      "12496                       False                       False   \n",
      "12497                       False                       False   \n",
      "12498                       False                       False   \n",
      "\n",
      "       posted_2020-08-27T00:00:00  posted_2020-09-04T00:00:00  \\\n",
      "0                           False                       False   \n",
      "1                           False                       False   \n",
      "3                           False                       False   \n",
      "4                           False                       False   \n",
      "5                           False                       False   \n",
      "...                           ...                         ...   \n",
      "12494                       False                       False   \n",
      "12495                       False                       False   \n",
      "12496                       False                       False   \n",
      "12497                       False                       False   \n",
      "12498                       False                       False   \n",
      "\n",
      "       posted_2020-11-05T00:00:00  posted_2021-12-19T00:00:00  \n",
      "0                           False                        True  \n",
      "1                           False                        True  \n",
      "3                           False                        True  \n",
      "4                           False                        True  \n",
      "5                           False                       False  \n",
      "...                           ...                         ...  \n",
      "12494                       False                       False  \n",
      "12495                       False                       False  \n",
      "12496                       False                       False  \n",
      "12497                       False                       False  \n",
      "12498                       False                       False  \n",
      "\n",
      "[10000 rows x 53560 columns]\n"
     ]
    }
   ],
   "source": [
    "print(\"Данные до унитарного кодирования:\")\n",
    "print(df.head(10000))\n",
    "\n",
    "# Применение унитарного кодирования для категориальных признаков\n",
    "df_encoded = pd.get_dummies(df.head(10000), drop_first=True)\n",
    "\n",
    "print(\"\\nДанные после унитарного кодирования:\")\n",
    "print(df_encoded.head(10000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Дискретизация числовых признаков**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22.1938\n",
      "54.4333\n",
      "32.23950000000001\n"
     ]
    }
   ],
   "source": [
    "print(df['city_latitude'].min())\n",
    "print(df['city_latitude'].max())\n",
    "print(df['city_latitude'].max() - df['city_latitude'].min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Данные до дискретизации:\n",
      "[22, 36, 49, 63, 77, inf]\n",
      "\n",
      "Данные после дискретизации:\n",
      "    city_latitude price_bins\n",
      "0       36.356650      36-49\n",
      "1       39.174503      36-49\n",
      "3       35.961561      22-36\n",
      "4       38.798958      36-49\n",
      "5       33.877422      22-36\n",
      "6       36.141246      36-49\n",
      "8       40.294123      36-49\n",
      "10      40.698700      36-49\n",
      "12      44.072800      36-49\n",
      "13      42.312800      36-49\n"
     ]
    }
   ],
   "source": [
    "print(\"Данные до дискретизации:\")\n",
    "#print(df.head(10))\n",
    "\n",
    "\n",
    "# Определение интервалов и меток для дискретизации\n",
    "bins = [\n",
    "round(df['city_latitude'].min() + df['city_latitude'].max() * 0), \n",
    "round(df['city_latitude'].min() + df['city_latitude'].max() * 0.25), \n",
    "round(df['city_latitude'].min() + df['city_latitude'].max() * 0.50), \n",
    "round(df['city_latitude'].min() + df['city_latitude'].max() * 0.75),\n",
    "round(df['city_latitude'].min() + df['city_latitude'].max() * 1),\n",
    "float('inf')\n",
    "]\n",
    "print(bins)\n",
    "labels = [\n",
    "str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0)) + '-' + str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0.25)),\n",
    "str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0.25)) + '-' + str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0.5)),\n",
    "str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0.5)) + '-' + str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0.75)),\n",
    "str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0.75)) + '-' + str(round(df['city_latitude'].min() + df['city_latitude'].max() * 0.1)),\n",
    "str(round(df['city_latitude'].min() + df['city_latitude'].max() * 1)) + '+'\n",
    "]\n",
    "\n",
    "# Применение дискретизации\n",
    "df['price_bins'] = pd.cut(df['city_latitude'], bins=bins, labels=labels, right=False)\n",
    "\n",
    "print(\"\\nДанные после дискретизации:\")\n",
    "print(df[['city_latitude', 'price_bins']].head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**«Ручной» синтез признаков**\n",
    "\n",
    "Создание новых признаков на основе экспертных знаний и логики предметной области. К примеру, для данных о продаже домов можно создать признак цена за единицу товара."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Данные до синтеза признака:\n",
      "                                              summary         city state  \\\n",
      "0   Viewed some red lights in the sky appearing to...      Visalia    CA   \n",
      "1   Look like 1 or 3 crafts from North traveling s...   Cincinnati    OH   \n",
      "3   One red light moving switly west to east, beco...    Knoxville    TN   \n",
      "4   Bright, circular Fresnel-lens shaped light sev...   Alexandria    VA   \n",
      "5   I'm familiar with all the fakery and UFO sight...    Fullerton    CA   \n",
      "6   I was driving up lakes mead towards the lake a...    Las Vegas    NV   \n",
      "8   Wing shaped craft seen at night, no lights, no...         Orem    UT   \n",
      "10  Yellow light floating across my grass as it de...  Springfield    NJ   \n",
      "12  A trail of star like lights moving from the W....   Janesville    MN   \n",
      "13  Large bright ball not registering on any app a...       Bangor    MI   \n",
      "\n",
      "              date_time     shape       duration  \\\n",
      "0   2021-12-15T21:45:00     light      2 minutes   \n",
      "1   2021-12-16T09:45:00  triangle     14 seconds   \n",
      "3   2021-12-10T19:30:00  triangle  20-30 seconds   \n",
      "4   2021-12-07T08:00:00    circle            NaN   \n",
      "5   2020-07-07T23:00:00   unknown      2 minutes   \n",
      "6   2020-04-23T03:00:00      oval     10 minutes   \n",
      "8   2020-04-18T23:00:00     other     10 seconds   \n",
      "10  2020-05-13T03:37:00     light      7 seconds   \n",
      "12  2020-04-18T21:05:00     light     15 minutes   \n",
      "13  2020-04-18T22:30:00  triangle     45 minutes   \n",
      "\n",
      "                                                stats  \\\n",
      "0   Occurred : 12/15/2021 21:45  (Entered as : 12/...   \n",
      "1   Occurred : 12/16/2021 09:45  (Entered as : 12/...   \n",
      "3   Occurred : 12/10/2021 19:30  (Entered as : 12/...   \n",
      "4   Occurred : 12/7/2021 08:00  (Entered as : 12/0...   \n",
      "5   Occurred : 7/7/2020 23:00  (Entered as : 07/07...   \n",
      "6   Occurred : 4/23/2020 03:00  (Entered as : 4/23...   \n",
      "8   Occurred : 4/18/2020 23:00  (Entered as : 04/1...   \n",
      "10  Occurred : 5/13/2020 03:37  (Entered as : 05/1...   \n",
      "12  Occurred : 4/18/2020 21:05  (Entered as : 04/1...   \n",
      "13  Occurred : 4/18/2020 22:30  (Entered as : 04/1...   \n",
      "\n",
      "                                          report_link  \\\n",
      "0   http://www.nuforc.org/webreports/165/S165881.html   \n",
      "1   http://www.nuforc.org/webreports/165/S165888.html   \n",
      "3   http://www.nuforc.org/webreports/165/S165825.html   \n",
      "4   http://www.nuforc.org/webreports/165/S165754.html   \n",
      "5   http://www.nuforc.org/webreports/157/S157444.html   \n",
      "6   http://www.nuforc.org/webreports/155/S155608.html   \n",
      "8   http://www.nuforc.org/webreports/155/S155512.html   \n",
      "10  http://www.nuforc.org/webreports/155/S155647.html   \n",
      "12  http://www.nuforc.org/webreports/155/S155497.html   \n",
      "13  http://www.nuforc.org/webreports/155/S155495.html   \n",
      "\n",
      "                                                 text               posted  \\\n",
      "0   Viewed some red lights in the sky appearing to...  2021-12-19T00:00:00   \n",
      "1   Look like 1 or 3 crafts from North traveling s...  2021-12-19T00:00:00   \n",
      "3   One red light moving switly west to east, beco...  2021-12-19T00:00:00   \n",
      "4   Bright, circular Fresnel-lens shaped light sev...  2021-12-19T00:00:00   \n",
      "5   I'm familiar with all the fakery and UFO sight...  2020-07-09T00:00:00   \n",
      "6   I was driving up lakes mead towards the lake a...  2020-05-01T00:00:00   \n",
      "8   Wing shaped craft seen at night, no lights, no...  2020-05-01T00:00:00   \n",
      "10  Yellow light floating across my grass as it de...  2020-05-15T00:00:00   \n",
      "12  A trail of star like lights moving from the W....  2020-05-15T00:00:00   \n",
      "13  Large bright ball not registering on any app a...  2020-05-15T00:00:00   \n",
      "\n",
      "    city_latitude  city_longitude price_bins  \n",
      "0       36.356650     -119.347937      36-49  \n",
      "1       39.174503      -84.481363      36-49  \n",
      "3       35.961561      -83.980115      22-36  \n",
      "4       38.798958      -77.095133      36-49  \n",
      "5       33.877422     -117.924978      22-36  \n",
      "6       36.141246     -115.186592      36-49  \n",
      "8       40.294123     -111.701685      36-49  \n",
      "10      40.698700      -74.329600      36-49  \n",
      "12      44.072800      -93.728600      36-49  \n",
      "13      42.312800      -86.081300      36-49  \n",
      "\n",
      "Данные после синтеза признака 'relative_price':\n",
      "    city_latitude state  relative_appearing\n",
      "0       36.356650    CA            1.018610\n",
      "1       39.174503    OH            0.969926\n",
      "3       35.961561    TN            1.001705\n",
      "4       38.798958    VA            1.026087\n",
      "5       33.877422    CA            0.949149\n",
      "6       36.141246    NV            0.968071\n",
      "8       40.294123    UT            1.003457\n",
      "10      40.698700    NJ            1.008448\n",
      "12      44.072800    MN            0.970854\n",
      "13      42.312800    MI            0.984641\n"
     ]
    }
   ],
   "source": [
    "# Проверка первых строк данных\n",
    "print(\"Данные до синтеза признака:\")\n",
    "print(df.head(10))\n",
    "\n",
    "# Вычисление средней цены по категориям\n",
    "mean_price_by_category = df.groupby('state')['city_latitude'].transform('mean')\n",
    "\n",
    "# Создание нового признака 'relative_price' (относительная цена)\n",
    "df['relative_appearing'] = df['city_latitude'] / mean_price_by_category\n",
    "\n",
    "# Проверка первых строк данных после синтеза признака\n",
    "print(\"\\nДанные после синтеза признака 'relative_price':\")\n",
    "print(df[['city_latitude', 'state', 'relative_appearing']].head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Масштабирование признаков на основе нормировки и стандартизации**\n",
    "\n",
    "Масштабирование признаков - это процесс преобразования числовых признаков таким образом, чтобы они имели одинаковый масштаб. Это важно для многих алгоритмов машинного обучения, которые чувствительны к масштабу признаков, таких как линейная регрессия, метод опорных векторов (SVM) и нейронные сети."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Данные до масштабирования:\n",
      "   city_latitude  relative_city_latitude\n",
      "0      36.356650                1.018610\n",
      "1      39.174503                0.969926\n",
      "3      35.961561                1.001705\n",
      "4      38.798958                1.026087\n",
      "5      33.877422                0.949149\n",
      "\n",
      "Данные после нормировки:\n",
      "   city_latitude  relative_city_latitude\n",
      "0       0.439301                0.475114\n",
      "1       0.526705                0.349452\n",
      "3       0.427046                0.431477\n",
      "4       0.515056                0.494412\n",
      "5       0.362401                0.295823\n",
      "\n",
      "Данные после стандартизации:\n",
      "   city_latitude  relative_city_latitude\n",
      "0      -0.426560                0.533974\n",
      "1       0.097536               -0.862875\n",
      "3      -0.500043                0.048911\n",
      "4       0.027688                0.748491\n",
      "5      -0.887675               -1.459012\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "\n",
    "# Создание нового признака 'relative_city_latitude' (цена относительно средней цены в категории)\n",
    "mean_city_latitude_by_state = df.groupby('state')['city_latitude'].transform('mean')\n",
    "df['relative_city_latitude'] = df['city_latitude'] / mean_city_latitude_by_state\n",
    "\n",
    "# Проверка первых строк данных до масштабирования\n",
    "print(\"Данные до масштабирования:\")\n",
    "print(df[['city_latitude', 'relative_city_latitude']].head())\n",
    "\n",
    "# Масштабирование признаков на основе нормировки\n",
    "min_max_scaler = MinMaxScaler()\n",
    "df[['city_latitude', 'relative_city_latitude']] = min_max_scaler.fit_transform(df[['city_latitude', 'relative_city_latitude']])\n",
    "\n",
    "# Проверка первых строк данных после нормировки\n",
    "print(\"\\nДанные после нормировки:\")\n",
    "print(df[['city_latitude', 'relative_city_latitude']].head())\n",
    "\n",
    "# Стандартизация признаков\n",
    "standard_scaler = StandardScaler()\n",
    "df[['city_latitude', 'relative_city_latitude']] = standard_scaler.fit_transform(df[['city_latitude', 'relative_city_latitude']])\n",
    "\n",
    "# Проверка первых строк данных после стандартизации\n",
    "print(\"\\nДанные после стандартизации:\")\n",
    "print(df[['city_latitude', 'relative_city_latitude']].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Конструирование признаков с применением фреймворка Featuretools**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\type_sys\\utils.py:33: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  pd.to_datetime(\n",
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\featuretools\\synthesis\\deep_feature_synthesis.py:169: UserWarning: Only one dataframe in entityset, changing max_depth to 1 since deeper features cannot be created\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 23 features\n",
      "Elapsed: 00:14 | Progress:  95%|█████████▌"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\woodwork\\logical_types.py:841: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`\n",
      "  series = series.replace(ww.config.get_option(\"nan_values\"), np.nan)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed: 00:17 | Progress: 100%|██████████\n",
      "Новые признаки, созданные с помощью Featuretools:\n",
      "             city state     shape       duration  city_latitude  \\\n",
      "index                                                             \n",
      "0         Visalia    CA     light      2 minutes      -0.426560   \n",
      "1      Cincinnati    OH  triangle     14 seconds       0.097536   \n",
      "2       Knoxville    TN  triangle  20-30 seconds      -0.500043   \n",
      "3      Alexandria    VA    circle            NaN       0.027688   \n",
      "4       Fullerton    CA   unknown      2 minutes      -0.887675   \n",
      "\n",
      "       city_longitude price_bins  relative_appearing  relative_city_latitude  \\\n",
      "index                                                                          \n",
      "0         -119.347937      36-49            1.018610                0.775416   \n",
      "1          -84.481363      36-49            0.969926                0.301550   \n",
      "2          -83.980115      22-36            1.001705                0.977744   \n",
      "3          -77.095133      36-49            1.026087               -0.177743   \n",
      "4         -117.924978      22-36            0.949149                1.613646   \n",
      "\n",
      "      DAY(date_time)  ... NUM_CHARACTERS(stats) NUM_CHARACTERS(summary)  \\\n",
      "index                 ...                                                 \n",
      "0                 15  ...                   174                      91   \n",
      "1                 16  ...                   180                     114   \n",
      "2                 10  ...                   182                     108   \n",
      "3                  7  ...                   166                     127   \n",
      "4                  7  ...                   167                     134   \n",
      "\n",
      "      NUM_CHARACTERS(text)  NUM_WORDS(stats)  NUM_WORDS(summary)  \\\n",
      "index                                                              \n",
      "0                      811                22                  17   \n",
      "1                      302                22                  22   \n",
      "2                     1633                22                  20   \n",
      "3                     1813                21                  18   \n",
      "4                     1275                21                  28   \n",
      "\n",
      "       NUM_WORDS(text)  WEEKDAY(date_time)  WEEKDAY(posted)  YEAR(date_time)  \\\n",
      "index                                                                          \n",
      "0                  147                   2                6             2021   \n",
      "1                   59                   3                6             2021   \n",
      "2                  304                   4                6             2021   \n",
      "3                  322                   1                6             2021   \n",
      "4                  233                   1                3             2020   \n",
      "\n",
      "      YEAR(posted)  \n",
      "index               \n",
      "0             2021  \n",
      "1             2021  \n",
      "2             2021  \n",
      "3             2021  \n",
      "4             2020  \n",
      "\n",
      "[5 rows x 23 columns]\n"
     ]
    }
   ],
   "source": [
    "import featuretools as ft\n",
    "\n",
    "# Создание нового признака 'relative_city_latitude'\n",
    "mean_city_latitude_by_state = df.groupby('state')['city_latitude'].transform('mean')\n",
    "df['relative_city_latitude'] = df['city_latitude'] / mean_city_latitude_by_state\n",
    "\n",
    "# Создание EntitySet\n",
    "es = ft.EntitySet(id='jio_mart_items')\n",
    "\n",
    "# Добавление данных с явным указанием индексного столбца\n",
    "es = es.add_dataframe(dataframe_name='items_data', dataframe=df, index='index', make_index=True)\n",
    "\n",
    "# Конструирование признаков\n",
    "features, feature_defs = ft.dfs(entityset=es, target_dataframe_name='items_data', verbose=True)\n",
    "\n",
    "# Проверка первых строк новых признаков\n",
    "print(\"Новые признаки, созданные с помощью Featuretools:\")\n",
    "print(features.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Оценка качества**\n",
    "\n",
    "*Предсказательная способность Метрики:* RMSE, MAE, R²  \n",
    "\n",
    "*Методы:* Обучение модели на обучающей выборке и оценка на контрольной и тестовой выборках.  \n",
    "\n",
    "*Скорость вычисления Методы:* Измерение времени выполнения генерации признаков и обучения модели.  \n",
    "\n",
    "*Надежность Методы:* Кросс-валидация, анализ чувствительности модели к изменениям в данных.  \n",
    "\n",
    "*Корреляция Методы:* Анализ корреляционной матрицы признаков, удаление мультиколлинеарных признаков.  \n",
    "\n",
    "*Цельность Методы:* Проверка логической связи между признаками и целевой переменной, интерпретация результатов модели.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\sklearn\\metrics\\_regression.py:492: FutureWarning: 'squared' is deprecated in version 1.4 and will be removed in 1.6. To calculate the root mean squared error, use the function'root_mean_squared_error'.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.5489449749619157\n",
      "R²: 0.924423816721939\n",
      "MAE: 0.46826566221280963\n",
      "Training Time: 37.429771184921265 seconds\n",
      "Cross-validated RMSE: 1.535611497565107\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\tumvu\\AppData\\Local\\Temp\\ipykernel_54788\\399707436.py:70: FutureWarning: \n",
      "\n",
      "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect.\n",
      "\n",
      "  sns.barplot(x='Importance', y='Feature', data=importance_df_top, palette='viridis')\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 55984 characters]",
      "text/plain": [
       "<Figure size 1000x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 0.4358844786848851\n",
      "Train R²: 0.994185027626814\n",
      "Train MAE: 0.12184558416960284\n",
      "Корреляция: 0.96\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\tumvu\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\sklearn\\metrics\\_regression.py:492: FutureWarning: 'squared' is deprecated in version 1.4 and will be removed in 1.6. To calculate the root mean squared error, use the function'root_mean_squared_error'.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 66052 characters]",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error\n",
    "from sklearn.model_selection import cross_val_score\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import time\n",
    "import numpy as np\n",
    "\n",
    "# Загрузка данных\n",
    "df = pd.read_csv(\"../../datasets/nuforc_reports.csv\").head(2000)\n",
    "\n",
    "# Создание нового признака 'relative_city_latitude'\n",
    "mean_city_latitude_by_state = df.groupby('state')['city_latitude'].transform('mean')\n",
    "df['relative_city_latitude'] = df['city_latitude'] / mean_city_latitude_by_state\n",
    "\n",
    "# Предобработка данных\n",
    "# Преобразуем категориальные переменные в числовые\n",
    "df = pd.get_dummies(df, drop_first=True)\n",
    "\n",
    "# Разделение данных на признаки и целевую переменную\n",
    "X = df.drop('city_latitude', axis=1).dropna()\n",
    "y = df['city_latitude'].dropna()\n",
    "\n",
    "# Разделение данных на обучающую и тестовую выборки\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Выбор модели\n",
    "model = RandomForestRegressor(random_state=42)\n",
    "\n",
    "# Измерение времени обучения и предсказания\n",
    "start_time = time.time()\n",
    "\n",
    "# Обучение модели\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Предсказание и оценка\n",
    "y_pred = model.predict(X_test)\n",
    "\n",
    "end_time = time.time()\n",
    "training_time = end_time - start_time\n",
    "\n",
    "rmse = mean_squared_error(y_test, y_pred, squared=False)\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "mae = mean_absolute_error(y_test, y_pred)\n",
    "\n",
    "print(f\"RMSE: {rmse}\")\n",
    "print(f\"R²: {r2}\")\n",
    "print(f\"MAE: {mae}\")\n",
    "print(f\"Training Time: {training_time} seconds\")\n",
    "\n",
    "# Кросс-валидация\n",
    "scores = cross_val_score(model, X_train, y_train, cv=5, scoring='neg_mean_squared_error')\n",
    "rmse_cv = (-scores.mean())**0.5\n",
    "print(f\"Cross-validated RMSE: {rmse_cv}\")\n",
    "\n",
    "# Анализ важности признаков\n",
    "feature_importances = model.feature_importances_\n",
    "feature_names = X_train.columns\n",
    "\n",
    "importance_df = pd.DataFrame({'Feature': feature_names, 'Importance': feature_importances})\n",
    "importance_df = importance_df.sort_values(by='Importance', ascending=False)\n",
    "\n",
    "# Отобразим только топ-20 признаков\n",
    "top_n = 20\n",
    "importance_df_top = importance_df.head(top_n)\n",
    "\n",
    "plt.figure(figsize=(10, 8))\n",
    "sns.barplot(x='Importance', y='Feature', data=importance_df_top, palette='viridis')\n",
    "plt.title(f'Top {top_n} Feature Importance')\n",
    "plt.xlabel('Importance')\n",
    "plt.ylabel('Feature')\n",
    "plt.show()\n",
    "\n",
    "# Проверка на переобучение\n",
    "y_train_pred = model.predict(X_train)\n",
    "\n",
    "rmse_train = mean_squared_error(y_train, y_train_pred, squared=False)\n",
    "r2_train = r2_score(y_train, y_train_pred)\n",
    "mae_train = mean_absolute_error(y_train, y_train_pred)\n",
    "\n",
    "print(f\"Train RMSE: {rmse_train}\")\n",
    "print(f\"Train R²: {r2_train}\")\n",
    "print(f\"Train MAE: {mae_train}\")\n",
    "\n",
    "correlation = np.corrcoef(y_test, y_pred)[0, 1]\n",
    "print(f\"Корреляция: {correlation:.2f}\")\n",
    "\n",
    "# Визуализация результатов\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.scatter(y_test, y_pred, alpha=0.5)\n",
    "plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'k--', lw=2)\n",
    "plt.xlabel('Actual city_latitude')\n",
    "plt.ylabel('Predicted city_latitude')\n",
    "plt.title('Actual vs Predicted city_latitude')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Выводы и итог  \n",
    "\n",
    "**Время обучения:**\n",
    "\n",
    "Время обучения модели составляет 37 секунды, что является средним. Это указывает на то, что модель обучается быстро и может эффективно обрабатывать данные.\n",
    "\n",
    "**Предсказательная способность:**\n",
    "\n",
    "MAE (Mean Absolute Error): 0.12184558416960284 — это средняя абсолютная ошибка предсказаний модели. Значение MAE  невелико, что означает, что предсказанные значения в среднем отклоняются от реальных на 0.12184558416960284. Это может быть приемлемым уровнем ошибки.\n",
    "\n",
    "RMSE (Mean Squared Error): 0.4358844786848851 — это среднее значение квадратов ошибок.\n",
    "\n",
    "R² (коэффициент детерминации): 0.994185027626814 — это средний уровень, указывающий на то, что модель объясняет 99,4% вариации целевой переменной. Это свидетельствует о средней предсказательной способности модели.\n",
    "\n",
    "**Корреляция:**\n",
    "\n",
    "Корреляция (0.96) между предсказанными и реальными значениями говорит о том, что предсказания модели имеют  сильную линейную зависимость с реальными значениями. Это подтверждает, что модель хорошо обучена и делает точные прогнозы.\n",
    "\n",
    "**Надежность (кросс-валидация):**\n",
    "\n",
    "Среднее RMSE (кросс-валидация): 1.535611497565107 — это значительно ниже, чем обычное RMSE, что указывает на отсутствие проблем с переобучением - что и подтверждается тестом переобучением. \n",
    "\n",
    "Результаты визуализации важности признаков, полученные из линейной регрессии, помогают понять, какие из входных переменных наибольшим образом влияют на целевую переменную (city_latitude)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
