1133 lines
371 KiB
Plaintext
1133 lines
371 KiB
Plaintext
|
{
|
|||
|
|
"cells": [
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Лабораторная работа №2**"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 1,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd \n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import seaborn as sns\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"from imblearn.over_sampling import RandomOverSampler\n",
|
|||
|
|
"from imblearn.under_sampling import RandomUnderSampler\n",
|
|||
|
|
"from sklearn.preprocessing import LabelEncoder"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Диабет у индейцев Пима**"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 3,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Index(['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin',\n",
|
|||
|
|
" 'BMI', 'DiabetesPedigreeFunction', 'Age', 'Outcome'],\n",
|
|||
|
|
" dtype='object')\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd \n",
|
|||
|
|
"df = pd.read_csv(\"..//static//csv//diabetes.csv\")\n",
|
|||
|
|
"print(df.columns)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Проблемная область: медицина и эпидемиология\n",
|
|||
|
|
"\n",
|
|||
|
|
"Объект наблюдения: женщины индейского племени Пима, проживающие вблизи Финикса, штат Аризона, США\n",
|
|||
|
|
"\n",
|
|||
|
|
"Атрибуты: беременность, глюкоза, артериальное давление, толщина кожи, инсулин, индекс массы тела, родословная диабета, возраст, исход\n",
|
|||
|
|
"\n",
|
|||
|
|
"Пример бизнес-цели:\n",
|
|||
|
|
"1. Повышение эффективности скрининга диабета. Цель технического проекта: Разработать и обучить модель машинного обучения с точностью предсказания не менее 85% для автоматизированного скрининга диабета на основе данных датасета \"Диабет у индейцев Пима\".\n",
|
|||
|
|
"\n",
|
|||
|
|
"2. Снижение медицинских расходов. Цель технического проекта: Оптимизировать модель прогнозирования таким образом, чтобы минимизировать количество ложноотрицательных результатов (пациенты с диабетом, которые не были выявлены), что позволит снизить затраты на лечение осложнений.\n",
|
|||
|
|
"\n",
|
|||
|
|
"3. Повышение качества жизни пациентов. Цель технического проекта: Разработать интерфейс для модели, который будет предоставлять пациентам персонализированные рекомендации по профилактике и лечению диабета на основе их индивидуальных рисков, определенных моделью."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Проверяем на наличие выбросов"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 9,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество выбросов в столбце 'Glucose': 5\n",
|
|||
|
|
"Количество выбросов в столбце 'SkinThickness': 1\n",
|
|||
|
|
"Количество выбросов в столбце 'Insulin': 34\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABdIAAAFqCAYAAAAeIKisAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADIf0lEQVR4nOzdeXhU5d3G8e9M9n0lGwQIEJawCwhxQRRkVUGxlhYVLYpawCLWWlRQwRZFq1SLWqwCWtDWvoJClV1AZY+i7DsEErKTTPZtzvtHzNTImpDkTJL7c11zwZz1Ptmemd8853kshmEYiIiIiIiIiIiIiIjIeVnNDiAiIiIiIiIiIiIi4sxUSBcRERERERERERERuQgV0kVERERERERERERELkKFdBERERERERERERGRi1AhXURERERERERERETkIlRIFxERERERERERERG5CBXSRUREREREREREREQuQoV0EREREREREREREZGLUCFdREREREREREREROQiVEgXuQytW7fmvvvuMztGo/fyyy/Tpk0bXFxc6NGjR60ee8CAAQwYMKBWjykiIjWjdrV+1Ha7umHDBiwWC//5z38uut3ChQuxWCycOHHiis95ISdOnMBisfDKK69cctvnnnsOi8VS7XO0bt2aW265pSbxRESklum1Q/2oy/fkdcVisfDcc885ntfH6xBpulRIlyan8o/qzp07z7t+wIABdOnS5YrP8/nnn1f5Yy4Xt3r1av7whz9w7bXXsmDBAv785z9fcp+vvvqKu+66i+bNm+Pu7k5AQAB9+/Zl5syZpKam1kNqERFRu+qcqtuuLl++nBtuuIGwsDC8vb1p06YNd911FytXrqynxBVFEovFcsnHwoUL6y2TiIjUPr12cE7Vee1w33334evrW4/pRJyDq9kBRBqCgwcPYrVW73Onzz//nHnz5qnhvkzr16/HarXy7rvv4u7ufsntZ8yYwaxZs2jTpg333Xcfbdq0oaioiISEBP7yl7+waNEijh49Wg/JRUSkutSu1r3qtKuvvPIKTzzxBDfccAPTpk3D29ubI0eOsHbtWj766COGDh1arXPfc889jBkzBg8Pj2rtN3fuXPLy8hzPP//8cz788ENee+01QkNDHcuvueaaah33mWee4Y9//GO19hEREeei1w51r7rvyZ1VTV+HiFwOFdJFLkND/AOcn5+Pj4+P2TEuW1paGl5eXpfVYP/rX/9i1qxZ3HXXXXzwwQfn7PPaa6/x2muv1VVUERG5QmpX697ltqtlZWXMmjWLm2++mdWrV5/3ONXl4uKCi4tLtfcbNWpUlecpKSl8+OGHjBo1itatW1dZV53btV1dXXF11dseEZGGTK8d6l513pM7s5q+DhG5HBraReQy/Hw8ttLSUp5//nliY2Px9PQkJCSE6667jjVr1gAVtznNmzcPoMqtyJXy8/N5/PHHiY6OxsPDgw4dOvDKK69gGEaV8xYWFvLoo48SGhqKn58ft912G0lJSeeMAVY59ue+ffv49a9/TVBQENdddx0AP/zwg6PHtqenJxEREfzmN78hMzOzyrkqj3Ho0CHuvvtuAgICaNasGdOnT8cwDE6dOsXIkSPx9/cnIiKCv/zlL5f1tat8g962bVs8PDxo3bo1Tz31FMXFxY5tLBYLCxYsID8//7Ju254xYwahoaEX/KQ8ICDgkr0OLjRuWuX4rxs2bKiyfNu2bQwfPpygoCB8fHzo1q0bf/3rX6tss379eq6//np8fHwIDAxk5MiR7N+/v8o2ubm5TJkyhdatW+Ph4UFYWBg333wz33777TnnGzp0KAEBAXh7e3PDDTfwzTffXPSaREQaCrWrztOuZmRkYLPZuPbaa8+7Piws7KJ5iouLueWWWwgICGDz5s3A+dvYyvHGv/76a66++mo8PT1p06YN77///mVd98XMnz/f8fXo06cPO3bsqLL+QmOk//Of/+Tqq6/G29uboKAg+vfvf94PE35q0aJFuLq68sQTTwBVx2q/VA6AAwcOcOeddxIcHIynpye9e/fms88+q7LNpX4foOJDhvvvv58WLVrg4eFBZGQkI0eO1HiwItJo6bWD87x2uJDLbesvp5270Bxn99133zkfrv9cfb8OkaZFXTOkycrJySEjI+Oc5aWlpZfc97nnnmP27Nk88MADXH311dhsNnbu3Mm3337LzTffzEMPPURycjJr1qzhgw8+qLKvYRjcdtttfPnll4wfP54ePXqwatUqnnjiCZKSkqr0pL7vvvv497//zT333EO/fv3YuHEjI0aMuGCuX/ziF8TGxvLnP//Z8QJgzZo1HDt2jPvvv5+IiAj27t3L/Pnz2bt3L1u3bj3njeUvf/lLOnXqxIsvvsh///tfXnjhBYKDg/n73//OTTfdxEsvvcTixYv5/e9/T58+fejfv/9Fv1YPPPAAixYt4s477+Txxx9n27ZtzJ49m/3797N06VIAPvjgA+bPn8/27dv5xz/+AVz4tu1Dhw5x6NAhHnjggXobk23NmjXccsstREZG8rvf/Y6IiAj279/PihUr+N3vfgfA2rVrGTZsGG3atOG5556jsLCQN954g2uvvZZvv/3W0dg//PDD/Oc//2HSpEnExcWRmZnJ119/zf79+7nqqquAioL8sGHD6NWrF88++yxWq5UFCxZw00038dVXX3H11VfXy3WLiFSH2tWG2a6GhYXh5eXF8uXLmTx5MsHBwRc9/08VFhYycuRIdu7cydq1a+nTp89Ftz9y5Ah33nkn48ePZ9y4cbz33nvcd9999OrVi86dO1/2eX9qyZIl5Obm8tBDD2GxWJgzZw533HEHx44dw83N7YL7Pf/88zz33HNcc801zJw5E3d3d7Zt28b69esZPHjwefeZP38+Dz/8ME899RQvvPBCtXPs3buXa6+9lubNm/PHP/4RHx8f/v3vfzNq1Cj+7//+j9tvvx249O8DwOjRo9m7dy+TJ0+mdevWpKWlsWbNGhITEy9ZYBARcRZ67dAwXztczOW09ZfTztWFungdIk2QIdLELFiwwAAu+ujcuXOVfVq1amWMGzfO8bx79+7GiBEjLnqeiRMnGuf7FVu2bJkBGC+88EKV5XfeeadhsViMI0eOGIZhGAkJCQZgTJkypcp29913nwEYzz77rGPZs88+awDGr371q3POV1BQcM6yDz/80ACMTZs2nXOMCRMmOJaVlZUZLVq0MCwWi/Hiiy86lp89e9bw8vKq8jU5n127dhmA8cADD1RZ/vvf/94AjPXr1zuWjRs3zvDx8bno8QzDMD799FMDMObOnVtlud1uN9LT06s8SktLHetvuOEG44YbbnA8r/w5OH78eJXjfPnllwZgfPnll46vQUxMjNGqVSvj7Nmz55yzUo8ePYywsDAjMzPTsez77783rFarce+99zqWBQQEGBMnTrzg9dntdiM2NtYYMmRIleMXFBQYMTExxs0333zBfUVEzKB2tWG3q4ZhGDNmzDAAw8fHxxg2bJjxpz/9yUhISDhnu8o28uOPPzZyc3ONG264wQgNDTW+++67Ktudr41t1arVOV+jtLQ0w8PDw3j88cfPm+vll18+b1ttGIZx/PhxAzBCQkKMrKwsx/LK1wnLly93LKv8XlQ6fPiwYbVajdtvv90oLy+vctyftr2tWrVy/Fz+9a9/NSwWizFr1qwa5xg4cKDRtWtXo6ioqMr5rrnmGiM2Ntax7FK/D2fPnjUA4+WXX77gNiIizkyvHRr+a4fzbXu5bf3lfO9+/v79p+dt1apVlWU//17U1usQkfPR0C7SZM2bN481a9ac8+jWrdsl9w0MDGTv3r0cPny42uf9/PPPcXFx4dFHH62y/PHHH8cwDL744gsAVq5cCcBvf/vbKttNnjz5gsd++OGHz1nm5eXl+H9RUREZGRn069cP4JzhRKDi0+pKLi4u9O7dG8MwGD9+vGN5YGAgHTp04NixYxfMAhXXCjB16tQqyx9//HEA/vvf/150//Ox2WwA5/RGz8nJoVmzZlUeu3btqvbxf+67777j+PHjTJkyhcDAwCrrKnsOnDlzhl27dnHfffd
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x1000 with 3 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import seaborn as sns\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Загрузка датасета\n",
|
|||
|
|
"df = pd.read_csv(\"..//static//csv//diabetes.csv\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выбираем столбцы для анализа\n",
|
|||
|
|
"columns_to_check = ['Glucose', 'SkinThickness', 'Insulin']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для подсчета выбросов\n",
|
|||
|
|
"def count_outliers(df, columns):\n",
|
|||
|
|
" outliers_count = {}\n",
|
|||
|
|
" for col in columns:\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Считаем количество выбросов\n",
|
|||
|
|
" outliers = df[(df[col] < lower_bound) | (df[col] > upper_bound)]\n",
|
|||
|
|
" outliers_count[col] = len(outliers)\n",
|
|||
|
|
" \n",
|
|||
|
|
" return outliers_count\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Подсчитываем выбросы\n",
|
|||
|
|
"outliers_count = count_outliers(df, columns_to_check)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выводим количество выбросов для каждого столбца\n",
|
|||
|
|
"for col, count in outliers_count.items():\n",
|
|||
|
|
" print(f\"Количество выбросов в столбце '{col}': {count}\")\n",
|
|||
|
|
" \n",
|
|||
|
|
"# Создаем гистограммы\n",
|
|||
|
|
"plt.figure(figsize=(15, 10))\n",
|
|||
|
|
"for i, col in enumerate(columns_to_check, 1):\n",
|
|||
|
|
" plt.subplot(3, 3, i)\n",
|
|||
|
|
" sns.histplot(df[col], kde=True)\n",
|
|||
|
|
" plt.title(f'Histogram of {col}')\n",
|
|||
|
|
"plt.tight_layout()\n",
|
|||
|
|
"plt.show()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Можно увидеть, что количество выбросов маленькое. Можем сделать очистку для столбца 'Insulin'"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 12,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество удаленных строк: 34\n",
|
|||
|
|
"Количество выбросов в столбце 'Insulin': 7\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAwcAAAJOCAYAAADieHtfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABqw0lEQVR4nO3dd3wUdf7H8ffuZndTN5UkBJLQexUEY0VB6nkW7n52QTnbgQ3hPO48BfRE8c56KN7pgd6J7c6uKE3wVERBEaRJNZQUkpCebLbM749sVhdCC0k25fV8POaxu9/57sxndkjYd2a+MybDMAwBAAAAaPXMwS4AAAAAQNNAOAAAAAAgiXAAAAAAwIdwAAAAAEAS4QAAAACAD+EAAAAAgCTCAQAAAAAfwgEAAAAASYQDAAAAAD6EAwDNUocOHTRx4sRgl9HiPfroo+rUqZMsFosGDBgQ7HJOiMlk0syZM/2vFy5cKJPJpD179tTrevbu3avQ0FB9/vnndXr/nj17ZDKZtHDhwnqtK1hmzpwpk8nkf+1yuZSamqpnnnkmiFUBOFmEAwBBV/Plbe3atbXOHzZsmPr06XPK6/nwww8DvjTi2JYsWaLf/e53Ouuss7RgwQI99NBDR+07ceJERUZGNmJ1wTd79mwNHTpUZ5111hHzVq5cqcsuu0zJycmy2WxKTEzURRddpDfffDMIlQaH1WrV1KlT9ec//1mVlZXBLgfACSIcAGiWtm3bpn/84x8n9Z4PP/xQs2bNaqCKWp4VK1bIbDbrhRde0HXXXaexY8cGu6Q6ufbaa1VRUaH09PR6W+bBgwf14osv6pZbbjli3v3336/zzz9f33//vW6++WbNnz9f06dPV2lpqcaPH69FixbVWx1N3fXXX6+8vLxWtc1AcxcS7AIAoC7sdnuwSzhpZWVlioiICHYZJyw3N1dhYWGy2WzBLuWUWCwWWSyWel3mv//9b4WEhOiiiy4KaP/Pf/6j2bNn61e/+pUWLVokq9Xqnzd9+nR9/PHHcrlc9VpLUxYTE6ORI0dq4cKFuuGGG4JdDoATwJEDAM3S4WMOXC6XZs2apa5duyo0NFTx8fE6++yztXTpUknVp73MmzdPUvU56TVTjbKyMt19991KTU2V3W5X9+7d9Ze//EWGYQSst6KiQrfffrsSEhIUFRWlX/7yl9q/f/8R57nXnH+9efNmXXXVVYqNjdXZZ58tSdqwYYMmTpyoTp06KTQ0VMnJybrhhhuUn58fsK6aZfzwww+65pprFB0drTZt2uhPf/qTDMPQ3r17dfHFF8vhcCg5OVl//etfT+izc7vdeuCBB9S5c2fZ7XZ16NBBf/jDH+R0Ov19TCaTFixYoLKyMv9ndbLnxnfo0EG/+MUv9Nlnn2nIkCEKDQ1Vp06d9NJLLwX0O96+k6pPLRs2bNgR65g4caI6dOhwzDpqG3NworUdzdtvv62hQ4cecSrVn/70J8XFxemf//xnQDCoMWrUKP3iF7845rK3bt2qX/3qV4qLi1NoaKgGDx6sd999N6BPQUGBpk2bpr59+yoyMlIOh0NjxozRd999F9Bv5cqVMplMev311/XnP/9Z7du3V2hoqIYPH64dO3Ycse41a9Zo9OjRio6OVnh4uM4777xax1R89tlnOv300xUaGqrOnTvrueeeO+r2XHjhhfrss89UUFBwzO0G0DRw5ABAk1FUVKS8vLwj2k/kL60zZ87UnDlz9Jvf/EZDhgxRcXGx1q5dq2+++UYXXnihbr75Zh04cEBLly7Vv/71r4D3GoahX/7yl/rkk080adIkDRgwQB9//LGmT5+u/fv36/HHH/f3nThxol5//XVde+21OuOMM7Rq1SqNGzfuqHX9+te/VteuXfXQQw/5g8bSpUu1a9cuXX/99UpOTtamTZv097//XZs2bdKXX34ZEFok6fLLL1fPnj318MMP64MPPtCDDz6ouLg4Pffcc7rgggv0yCOP6OWXX9a0adN0+umn69xzzz3mZ/Wb3/xGL774on71q1/p7rvv1po1azRnzhxt2bJFb731liTpX//6l/7+97/rq6++0vPPPy9JOvPMM4+7Hw63Y8cO/epXv9KkSZM0YcIE/fOf/9TEiRM1aNAg9e7dW9Lx911DOZHaauNyufT111/r1ltvDWjfvn27tm7dqhtuuEFRUVF1qmnTpk0666yz1K5dO/3+979XRESEXn/9dV1yySX673//q0svvVSStGvXLr399tv69a9/rY4dOyonJ0fPPfeczjvvPG3evFkpKSkBy3344YdlNps1bdo0FRUVae7cubr66qu1Zs0af58VK1ZozJgxGjRokO6//36ZzWYtWLBAF1xwgf73v/9pyJAhkqSNGzdq5MiRatOmjWbOnCm32637779fSUlJtW7ToEGDZBiGvvjii+MGIwBNgAEAQbZgwQJD0jGn3r17B7wnPT3dmDBhgv91//79jXHjxh1zPZMnTzZq+7X39ttvG5KMBx98MKD9V7/6lWEymYwdO3YYhmEY69atMyQZd955Z0C/iRMnGpKM+++/3992//33G5KMK6+88oj1lZeXH9H2yiuvGJKMTz/99Ihl3HTTTf42t9tttG/f3jCZTMbDDz/sbz906JARFhYW8JnUZv369YYk4ze/+U1A+7Rp0wxJxooVK/xtEyZMMCIiIo65vGP1TU9PP2KbcnNzDbvdbtx9993+thPZd+edd55x3nnn1bre9PT0gLbD90XNv6/du3efdG212bFjhyHJePrppwPa33nnHUOS8fjjjx/z/TV2795tSDIWLFjgbxs+fLjRt29fo7Ky0t/m9XqNM8880+jatau/rbKy0vB4PEcsz263G7Nnz/a3ffLJJ4Yko2fPnobT6fS3P/nkk4YkY+PGjf51dO3a1Rg1apTh9Xr9/crLy42OHTsaF154ob/tkksuMUJDQ40ff/zR37Z582bDYrHU+vN14MABQ5LxyCOPnNDnAiC4OK0IQJMxb948LV269IipX79+x31vTEyMNm3apO3bt5/0ej/88ENZLBbdfvvtAe133323DMPQ4sWLJUkfffSRJOm3v/1tQL/bbrvtqMuubcBqWFiY/3llZaXy8vJ0xhlnSJK++eabI/r/5je/8T+3WCwaPHiwDMPQpEmT/O0xMTHq3r27du3addRapOptlaSpU6cGtN99992SpA8++OCY7z9ZvXr10jnnnON/3aZNmyPqPJV919C11abm9K/Y2NiA9uLiYkmq81GDgoICrVixQv/3f/+nkpIS5eXlKS8vT/n5+Ro1apS2b9+u/fv3S6oec2M2V/8X7vF4lJ+fr8jISHXv3r3Wf0PXX399wNiRmu2u2db169dr+/btuuqqq5Sfn+9fd1lZmYYPH65PP/1UXq9XHo9HH3/8sS655BKlpaX5l9ezZ0+NGjWq1u2q+ZxqOyoIoOnhtCIATcaQIUM0ePDgI9pjY2OP+8Vi9uzZuvjii9WtWzf16dNHo0eP1rXXXntCweLHH39USkrKEV/qevbs6Z9f82g2m9WxY8eAfl26dDnqsg/vK1V/CZw1a5ZeffVV5ebmBswrKio6ov/Pv4RJUnR0tEJDQ5WQkHBE++HjFg5Xsw2H15ycnKyYmBj/ttaXw2uXqvfnoUOH/K9PZd81dG3HYhw2HsXhcEiSSkpK6lTPjh07ZBiG/vSnP+lPf/pTrX1yc3PVrl07eb1ePfnkk3rmmWe0e/dueTwef5/4+Pgj3nf4ttZ8Ya/Z1ppgNmHChKPWV1RUJKfTqYqKCnXt2vWI+d27d/eHz5+r+ZwOP10OQNNEOADQIpx77rnauXOn3nnnHS1ZskTPP/+8Hn/8cc2fPz/gL++N7edHCWr83//9n7744gtNnz5dAwYMUGRkpLxer0aPHi2v13tE/9qutHO0q+8c/oX1aBrri9qJ1Hki+85kMtW6bT//UtwQtdWm5sv34SGiR48ekqrPya+Lmn0/bdq0o/4VvibUPfTQQ/rTn/6kG264QQ888IDi4uJkNpt15513nvC/Iemnba15z6OPPnrUm91FRkYGDFo/UTWf0+FhFkDTRDgA0GLExcX
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x600 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import seaborn as sns\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Загрузка датасета\n",
|
|||
|
|
"df = pd.read_csv(\"..//static//csv//diabetes.csv\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выбираем столбцы для очистки\n",
|
|||
|
|
"columns_to_clean = ['Insulin']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для удаления выбросов\n",
|
|||
|
|
"def remove_outliers(df, columns):\n",
|
|||
|
|
" for col in columns:\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Удаляем строки, содержащие выбросы\n",
|
|||
|
|
" df = df[(df[col] >= lower_bound) & (df[col] <= upper_bound)]\n",
|
|||
|
|
" \n",
|
|||
|
|
" return df\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Удаляем выбросы\n",
|
|||
|
|
"df_cleaned = remove_outliers(df, columns_to_clean)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выводим количество удаленных строк\n",
|
|||
|
|
"print(f\"Количество удаленных строк: {len(df) - len(df_cleaned)}\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"df = df_cleaned\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для подсчета выбросов\n",
|
|||
|
|
"def count_outliers(df, columns):\n",
|
|||
|
|
" outliers_count = {}\n",
|
|||
|
|
" for col in columns:\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Считаем количество выбросов\n",
|
|||
|
|
" outliers = df[(df[col] < lower_bound) | (df[col] > upper_bound)]\n",
|
|||
|
|
" outliers_count[col] = len(outliers)\n",
|
|||
|
|
" \n",
|
|||
|
|
" return outliers_count\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Подсчитываем выбросы\n",
|
|||
|
|
"outliers_count = count_outliers(df, columns_to_clean)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выводим количество выбросов для каждого столбца\n",
|
|||
|
|
"for col, count in outliers_count.items():\n",
|
|||
|
|
" print(f\"Количество выбросов в столбце '{col}': {count}\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Создаем гистограммы для очищенных данных\n",
|
|||
|
|
"plt.figure(figsize=(15, 6))\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Гистограмма для Insulin\n",
|
|||
|
|
"plt.subplot(1, 2, 1)\n",
|
|||
|
|
"sns.histplot(df_cleaned['Insulin'], kde=True)\n",
|
|||
|
|
"plt.title('Histogram of Insulin (Cleaned)')\n",
|
|||
|
|
"plt.xlabel('Insulin')\n",
|
|||
|
|
"plt.ylabel('Frequency')\n",
|
|||
|
|
"\n",
|
|||
|
|
"plt.tight_layout()\n",
|
|||
|
|
"plt.show()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Проверка на пропущенные значения"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 13,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество пропущенных значений в каждом столбце:\n",
|
|||
|
|
"Pregnancies 0\n",
|
|||
|
|
"Glucose 0\n",
|
|||
|
|
"BloodPressure 0\n",
|
|||
|
|
"SkinThickness 0\n",
|
|||
|
|
"Insulin 0\n",
|
|||
|
|
"BMI 0\n",
|
|||
|
|
"DiabetesPedigreeFunction 0\n",
|
|||
|
|
"Age 0\n",
|
|||
|
|
"Outcome 0\n",
|
|||
|
|
"dtype: int64\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка на пропущенные значения\n",
|
|||
|
|
"missing_values = df.isnull().sum()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Вывод результатов\n",
|
|||
|
|
"print(\"Количество пропущенных значений в каждом столбце:\")\n",
|
|||
|
|
"print(missing_values)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Пропущенных значений нет. Делаем разбиение на выборки"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 14,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Размер обучающей выборки: 440\n",
|
|||
|
|
"Размер контрольной выборки: 147\n",
|
|||
|
|
"Размер тестовой выборки: 147\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение на признаки (X) и целевую переменную (y)\n",
|
|||
|
|
"X = df.drop('Outcome', axis=1) # Признаки\n",
|
|||
|
|
"y = df['Outcome'] # Целевая переменная\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение на обучающую и оставшуюся часть (контрольная + тестовая)\n",
|
|||
|
|
"X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение оставшейся части на контрольную и тестовую выборки\n",
|
|||
|
|
"X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Вывод размеров выборок\n",
|
|||
|
|
"print(\"Размер обучающей выборки:\", X_train.shape[0])\n",
|
|||
|
|
"print(\"Размер контрольной выборки:\", X_val.shape[0])\n",
|
|||
|
|
"print(\"Размер тестовой выборки:\", X_test.shape[0])"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Разбивка на выборки сделана, проведем проверку на сбалансированность выборок"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 15,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Сбалансированность обучающей выборки:\n",
|
|||
|
|
"Outcome\n",
|
|||
|
|
"0 0.661364\n",
|
|||
|
|
"1 0.338636\n",
|
|||
|
|
"Name: proportion, dtype: float64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Сбалансированность контрольной выборки:\n",
|
|||
|
|
"Outcome\n",
|
|||
|
|
"0 0.659864\n",
|
|||
|
|
"1 0.340136\n",
|
|||
|
|
"Name: proportion, dtype: float64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Сбалансированность тестовой выборки:\n",
|
|||
|
|
"Outcome\n",
|
|||
|
|
"0 0.659864\n",
|
|||
|
|
"1 0.340136\n",
|
|||
|
|
"Name: proportion, dtype: float64\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение на признаки (X) и целевую переменную (y)\n",
|
|||
|
|
"X = df.drop('Outcome', axis=1) # Признаки\n",
|
|||
|
|
"y = df['Outcome'] # Целевая переменная\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение на обучающую и оставшуюся часть (контрольная + тестовая)\n",
|
|||
|
|
"X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42, stratify=y)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение оставшейся части на контрольную и тестовую выборки\n",
|
|||
|
|
"X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для проверки сбалансированности выборок\n",
|
|||
|
|
"def check_balance(y_train, y_val, y_test):\n",
|
|||
|
|
" print(\"Сбалансированность обучающей выборки:\")\n",
|
|||
|
|
" print(y_train.value_counts(normalize=True))\n",
|
|||
|
|
" \n",
|
|||
|
|
" print(\"\\nСбалансированность контрольной выборки:\")\n",
|
|||
|
|
" print(y_val.value_counts(normalize=True))\n",
|
|||
|
|
" \n",
|
|||
|
|
" print(\"\\nСбалансированность тестовой выборки:\")\n",
|
|||
|
|
" print(y_test.value_counts(normalize=True))\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка сбалансированности\n",
|
|||
|
|
"check_balance(y_train, y_val, y_test)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Выборки относительно сбалансированы, но не идеально. Сделаем приращение данных методом выборки с избытком (oversampling)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 16,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Сбалансированность обучающей выборки после SMOTE:\n",
|
|||
|
|
"Outcome\n",
|
|||
|
|
"0 0.5\n",
|
|||
|
|
"1 0.5\n",
|
|||
|
|
"Name: proportion, dtype: float64\n",
|
|||
|
|
"Сбалансированность обучающей выборки:\n",
|
|||
|
|
"Outcome\n",
|
|||
|
|
"0 0.5\n",
|
|||
|
|
"1 0.5\n",
|
|||
|
|
"Name: proportion, dtype: float64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Сбалансированность контрольной выборки:\n",
|
|||
|
|
"Outcome\n",
|
|||
|
|
"0 0.659864\n",
|
|||
|
|
"1 0.340136\n",
|
|||
|
|
"Name: proportion, dtype: float64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Сбалансированность тестовой выборки:\n",
|
|||
|
|
"Outcome\n",
|
|||
|
|
"0 0.659864\n",
|
|||
|
|
"1 0.340136\n",
|
|||
|
|
"Name: proportion, dtype: float64\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"from imblearn.over_sampling import SMOTE\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение на признаки (X) и целевую переменную (y)\n",
|
|||
|
|
"X = df.drop('Outcome', axis=1) # Признаки\n",
|
|||
|
|
"y = df['Outcome'] # Целевая переменная\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение на обучающую и оставшуюся часть (контрольная + тестовая)\n",
|
|||
|
|
"X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42, stratify=y)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение оставшейся части на контрольную и тестовую выборки\n",
|
|||
|
|
"X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Применение SMOTE для балансировки обучающей выборки\n",
|
|||
|
|
"smote = SMOTE(random_state=42)\n",
|
|||
|
|
"X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для проверки сбалансированности выборок\n",
|
|||
|
|
"def check_balance(y_train, y_val, y_test):\n",
|
|||
|
|
" print(\"Сбалансированность обучающей выборки:\")\n",
|
|||
|
|
" print(y_train.value_counts(normalize=True))\n",
|
|||
|
|
" \n",
|
|||
|
|
" print(\"\\nСбалансированность контрольной выборки:\")\n",
|
|||
|
|
" print(y_val.value_counts(normalize=True))\n",
|
|||
|
|
" \n",
|
|||
|
|
" print(\"\\nСбалансированность тестовой выборки:\")\n",
|
|||
|
|
" print(y_test.value_counts(normalize=True))\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка сбалансированности после SMOTE\n",
|
|||
|
|
"print(\"Сбалансированность обучающей выборки после SMOTE:\")\n",
|
|||
|
|
"print(y_train_resampled.value_counts(normalize=True))\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка сбалансированности контрольной и тестовой выборок\n",
|
|||
|
|
"check_balance(y_train_resampled, y_val, y_test)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Объекты вокруг земли**"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Проблемная область: космические объекты и их угроза для Земли\n",
|
|||
|
|
"\n",
|
|||
|
|
"Объект наблюдения: астероиды и другие малые тела Солнечной системы\n",
|
|||
|
|
"\n",
|
|||
|
|
"Атрибуты: имя объекта, минимальный и максимальный оценочные диаметры, относительная скорость, расстояние промаха, орбитальное тело, объекты программы \"Сентри\", абсолютная звездная величина, опасность\n",
|
|||
|
|
"\n",
|
|||
|
|
"Пример бизнес-цели:\n",
|
|||
|
|
"\n",
|
|||
|
|
"1. Разработка и продажа страховых продуктов для космических рисков. Цель технического проекта: разработка системы оценки рисков и ценообразования для страховых продуктов, защищающих от космических угроз.\n",
|
|||
|
|
"\n",
|
|||
|
|
"2. Разработка и продажа технологий для мониторинга и предотвращения космических угроз. Цель технического проекта: создание системы мониторинга и прогнозирования траекторий небесных тел для предотвращения космических угроз.\n",
|
|||
|
|
"\n",
|
|||
|
|
"3. Образовательные программы и сервисы. Цель технического проекта: разработка интерактивных образовательных материалов и сервисов, основанных на данных о небесных телах."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 22,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Index(['id', 'name', 'est_diameter_min', 'est_diameter_max',\n",
|
|||
|
|
" 'relative_velocity', 'miss_distance', 'orbiting_body', 'sentry_object',\n",
|
|||
|
|
" 'absolute_magnitude', 'hazardous'],\n",
|
|||
|
|
" dtype='object')\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import seaborn as sns\n",
|
|||
|
|
"\n",
|
|||
|
|
"# вывод всех столбцов\n",
|
|||
|
|
"df = pd.read_csv(\"..//static//csv//neo.csv\")\n",
|
|||
|
|
"print(df.columns)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Проверяем на выбросы"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 23,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество выбросов в столбце 'relative_velocity': 1574\n",
|
|||
|
|
"Количество выбросов в столбце 'miss_distance': 0\n",
|
|||
|
|
"Количество выбросов в столбце 'absolute_magnitude': 101\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABdEAAAISCAYAAAAjjoaeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdeXhU5fn/8ffMJDPZ92UIBAiLLAKiVAUXQEBQ0bqgdQfcsdAWteoXa1WgldaW0v7qvlSogooLWnEBlE1lUZFNkD0hJCH7vmdmzu+PISNDEgiQZLJ8Xtc1l+TMM+fcZzLmmbnnPvdjMgzDQERERERERERERERE6jD7OgARERERERERERERkdZKSXQRERERERERERERkQYoiS4iIiIiIiIiIiIi0gAl0UVEREREREREREREGqAkuoiIiIiIiIiIiIhIA5REFxERERERERERERFpgJLoIiIiIiIiIiIiIiINUBJdRERERERERERERKQBSqKLiIiIiIiIiIiIiDRASXRpU7p3787kyZN9HUa797e//Y0ePXpgsVgYPHhwsx6rOX6nkydPpnv37k26T1+bP38+JpOJlJSUZtn/U089hclkapZ9i4iciOb3ltFS83tzz1mnwmQy8dRTT3l+bo0xiog0Bc2pLaOp59TJkycTEhJy+oGdhJSUFEwmE/Pnz2/R43YULf0Z+9j3OtL0lEQXn6n98PL999/Xe//IkSMZMGDAaR/n008/1R+Sk7B8+XIeeeQRLrzwQl5//XWefvppX4dUr4yMDJ566im2bNni61DaraeffpoPP/zQ12GISBuj+b11aivze2tWXl7OU089xerVq30dioh0EJpTWyfNqZoTT5U+Y7dtSqJLm7J7925eeeWVk3rMp59+ysyZM5spovZn5cqVmM1mXnvtNSZOnMgVV1zh65DqlZGRwcyZM+tNor/yyivs3r275YNqwx5//HEqKiq8tmmCF5GWovm9+bXk/H777bdTUVFBt27dmu0Yp+tUYiwvL2fmzJlKGIhIq6Y5tfm1lc/MzUlz4onpM3b7oyS6tCk2mw1/f39fh3FSysrKfB3CScnOziYwMBCr1XpKjy8vL2/iiE6ev78/NpvN12G0KX5+fgQEBPg6DBHpoDS/N7/Tnd9PhsViISAgoFW3CWsLMYqInArNqc2vJedUabv0Gbv9URJd2pRj+7vV1NQwc+ZMevfuTUBAANHR0Vx00UWsWLECcPcVe+655wB3f6jaW62ysjIeeughEhMTsdls9OnTh7///e8YhuF13IqKCn77298SExNDaGgov/zlL0lPT6/Tc6q259XOnTu55ZZbiIyM5KKLLgJg27ZtTJ48mR49ehAQEIDdbufOO+8kLy/P61i1+9izZw+33XYb4eHhxMbG8sc//hHDMDh06BBXX301YWFh2O125s6d26jnzuFwMHv2bHr27InNZqN79+489thjVFVVecaYTCZef/11ysrKPM/V8fqj1V4+uGnTJoYPH05QUBCPPfYYAFVVVTz55JP06tULm81GYmIijzzyiNfx6pOfn8/vf/97Bg4cSEhICGFhYVx++eVs3brVM2b16tWce+65ANxxxx11Yj26J3pNTQ1RUVHccccddY5VXFxMQEAAv//97z3bTjXuo/3973/HZDJx8ODBOvfNmDEDq9VKQUGBZ9vGjRu57LLLCA8PJygoiBEjRvDNN9806ljPP/88Z555JjabjYSEBKZOnUphYWGdcRs3buSKK64gMjKS4OBgBg0axL/+9S/P/cf2azOZTJSVlbFgwQLP8zt58mRWrVqFyWRiyZIldY6xaNEiTCYT69evb1TsIiK1NL+3zvl927ZtjBgxgqCgIHr16sV7770HwJo1azj//PMJDAykT58+fPHFF16Pr6/f+Pfff8+4ceOIiYkhMDCQpKQk7rzzTq/Hvf322wwZMoTQ0FDCwsIYOHCg11zVGFVVVTzwwAPExsZ6fqdpaWl1xp1sjCkpKcTGxgIwc+ZMz/NY+zo52dfBvn37mDx5MhEREYSHh3PHHXfUW4jw5ptvct555xEUFERkZCTDhw9n+fLlXmM+++wzLr74YoKDgwkNDWX8+PHs2LHjpJ43EWk/NKe2rjn1q6++4oYbbqBr166ez5cPPPBAnQrlWgcOHGDcuHEEBweTkJDArFmz6jzXjZkvDxw4wA033EBUVBRBQUEMHTqUTz755ITPwciRIxk5cmSd7Ud/xj7RnAiwa9curr/+eqKioggICOAXv/gF//vf/054/GOZTCamTZvGu+++S//+/QkMDGTYsGFs374dgJdeeolevXoREBDAyJEj66x1cjLPf+0xAgICGDBgAEuWLKmz3lptH/m///3vvPzyy57Xyrnnnst3333ntb/GfsY+9vk93j6g8e91ANLT07nzzjuJj4/HZrNx5pln8p///Kehp1tOwM/XAYgUFRWRm5tbZ3tNTc0JH/vUU08xZ84c7r77bs477zyKi4v5/vvv+eGHH7j00ku57777yMjIYMWKFbzxxhtejzUMg1/+8pesWrWKu+66i8GDB7Ns2TIefvhh0tPTmTdvnmfs5MmTWbx4MbfffjtDhw5lzZo1jB8/vsG4brjhBnr37s3TTz/tmfBWrFjBgQMHuOOOO7Db7ezYsYOXX36ZHTt2sGHDhjp/GG+88Ub69evHX/7yFz755BP+9Kc/ERUVxUsvvcSoUaP461//ysKFC/n973/Pueeey/Dhw4/7XN19990sWLCA66+/noceeoiNGzcyZ84cfvrpJ09C9I033uDll1/m22+/5dVXXwXgggsuOO5+8/LyuPzyy7npppu47bbbiI+Px+Vy8ctf/pKvv/6ae++9l379+rF9+3bmzZvHnj17jnv50oEDB/jwww+54YYbSEpKIisri5deeokRI0awc+dOEhIS6NevH7NmzeKJJ57g3nvv5eKLL24wVn9/f6699lo++OADXnrpJa9qgQ8//JCqqipuuukmgNOK+2i/+tWveOSRR1i8eDEPP/yw132LFy9m7NixREZGAu5LAS+//HKGDBnCk08+idls5vXXX2fUqFF89dVXnHfeeQ0e56mnnmLmzJmMGTOG+++/n927d/PCCy/w3Xff8c0333gqUFasWMGVV15Jp06d+N3vfofdbuenn35i6dKl/O53v6t332+88Ybn/6t7770XgJ49ezJ06FASExNZuHAh1157rddjFi5cSM+ePRk2bFijnicRad80v7ft+b2goIArr7ySm266iRtuuIEXXniBm266iYULFzJ9+nSmTJnCLbfcwt/+9jeuv/56Dh06RGhoaL37ys7OZuzYscTGxvJ///d/REREkJKSwgcffOAZs2LFCm6++WZGjx7NX//6VwB++uknvvnmmwbnqoaejzfffJNbbrmFCy64gJUrVx73d9rYGGNjY3nhhRe4//77ufbaa7nuuusAGDRokCf+k3kd/OpXvyIpKYk5c+bwww8/8OqrrxIXF+c5d3AnJp566ikuuOACZs2ahdVqZePGjaxcuZKxY8cC7t/tpEmTGDduHH/9618pLy/nhRde4KKLLmLz5s3tbqF1kY5Kc2rbnVPfffddysvLuf/++4mOjubbb7/l3//+N2lpabz77rteY51OJ5dddhlDhw7lmWee4fPPP+fJJ5/E4XAwa9Ysz3N0ovkyKyuLCy64gPLycn77298SHR3NggUL+OUvf8l7771X53PcyTrRnLhjxw4uvPBCOnfuzP/93/8RHBzM4sWLueaaa3j//fdP+vhfffUV//vf/5g6dSoAc+bM4corr+SRRx7h+eef59e//jUFBQU888wz3HnnnaxcudLz2MY+/5988gk33ngjAwcOZM6cORQUFHDXXXfRuXPnemNatGgRJSUl3HfffZhMJp555hmuu+46Dhw40OCVIA19xj5ZjX2vk5WVxdChQz1fRMTGxvLZZ59x1113UVxczPTp00/62B2eIeIjr7/+ugEc93bmmWd6PaZbt27GpEmTPD+fddZZxvjx4497nKlTpxr1vdQ//PBDAzD+9Kc/eW2//vrrDZPJZOzbt88wDMP
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x1000 with 3 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import seaborn as sns\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выбираем столбцы для анализа\n",
|
|||
|
|
"columns_to_check = ['relative_velocity', 'miss_distance', 'absolute_magnitude']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для подсчета выбросов\n",
|
|||
|
|
"def count_outliers(df, columns):\n",
|
|||
|
|
" outliers_count = {}\n",
|
|||
|
|
" for col in columns:\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Считаем количество выбросов\n",
|
|||
|
|
" outliers = df[(df[col] < lower_bound) | (df[col] > upper_bound)]\n",
|
|||
|
|
" outliers_count[col] = len(outliers)\n",
|
|||
|
|
" \n",
|
|||
|
|
" return outliers_count\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Подсчитываем выбросы\n",
|
|||
|
|
"outliers_count = count_outliers(df, columns_to_check)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выводим количество выбросов для каждого столбца\n",
|
|||
|
|
"for col, count in outliers_count.items():\n",
|
|||
|
|
" print(f\"Количество выбросов в столбце '{col}': {count}\")\n",
|
|||
|
|
" \n",
|
|||
|
|
"# Создаем гистограммы\n",
|
|||
|
|
"plt.figure(figsize=(15, 10))\n",
|
|||
|
|
"for i, col in enumerate(columns_to_check, 1):\n",
|
|||
|
|
" plt.subplot(2, 3, i)\n",
|
|||
|
|
" sns.histplot(df[col], kde=True)\n",
|
|||
|
|
" plt.title(f'Histogram of {col}')\n",
|
|||
|
|
"plt.tight_layout()\n",
|
|||
|
|
"plt.show()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"В столбцах 'relative_velocity'и 'absolute_magnitude' присутствуют выбросы. Теперь можно очистить их от выбросов"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 24,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество выбросов в столбце 'relative_velocity': 1574\n",
|
|||
|
|
"Количество выбросов в столбце 'absolute_magnitude': 101\n",
|
|||
|
|
"Количество удаленных строк: 1678\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABc8AAAJOCAYAAABhpf9JAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3wUdf7H8dfupvdOEgghgPQmAQERREFQUSxYEAHxRERBD6xn+Smcnr2eJ3KeCp6CBbuAICigNIHQpEMgtJDee9n5/RGzR0gCAZJMQt7Px2Mfmpnvzr5nd0NmP/udz1gMwzAQEREREREREREREREHq9kBREREREREREREREQaGhXPRUREREREREREREROouK5iIiIiIiIiIiIiMhJVDwXERERERERERERETmJiuciIiIiIiIiIiIiIidR8VxERERERERERERE5CQqnouIiIiIiIiIiIiInETFcxERERERERERERGRk6h4LiIiIiIiIiIiIiJyEhXPRc4zrVq1Yvz48WbHOO+98sortG7dGpvNRo8ePer1sadPn47FYqnVba5YsQKLxcKKFStqdbtmqOvfAYvFwvTp0+ts+yey2+106dKFf/zjH2e9jfPp34Sq3qejRo3illtuMS+UiIjISc6nv70NWV0cj8+ZMweLxcLGjRtrZXs1pfdM7ajvzzSDBg1i0KBBtb7dq6++mrvvvvus719XucwQFxeHxWJhzpw5jmV/+9vf6NOnj3mhpMlR8VykATvdwdugQYPo0qXLOT/OokWL6q0YeD746aefePTRR+nfvz+zZ8/m+eefr3bs+PHjsVgsjpurqyvt2rXj6aefpqCgoB5Tl5k5c2aFAw8zPfDAA1gsFvbv31/tmCeffBKLxcK2bdvqMdmZWbNmDdOnTycjI6PWt/3pp59y5MgRpkyZUmldbGws99xzD61bt8bNzQ0fHx/69+/PW2+9RX5+fq1naagee+wxvvrqK7Zu3Wp2FBEROQ/peLxhOpPj8RPdcsstWCwWHnvssTpOWH/q6vh+0KBBWCwWLrjggirXL1261PEZ58svv6z1x68t8+bN48033zQ7Ro2tXr2an376qcr3aGJiIg8//DAdOnTAw8MDT09PoqOjee655+rks0hDNXXqVLZu3cr3339vdhRpIpzMDiAitWvPnj1YrWf2vdiiRYt45513dMBeQ7/88gtWq5UPPvgAFxeX0453dXXl/fffByAzM5PvvvuOZ599ltjYWObOnVvXcSuYOXMmQUFBlWa2DBw4kPz8/BrtT225/fbbefvtt5k3bx5PP/10lWM+/fRTunbtSrdu3eot1+nk5+fj5PS/P59r1qxhxowZjB8/Hj8/v1p9rFdeeYVRo0bh6+tbYfnChQu5+eabcXV1Zdy4cXTp0oWioiJWrVrFI488wo4dO3jvvfdqNUtDdeGFF9KrVy9ee+01/vvf/5odR0RERMfj9eBMj8cBsrKy+OGHH2jVqhWffvopL774Yq2fzWmG6o7va4Obmxv79+9n/fr1XHTRRRXWzZ07Fzc3N1MmBFWnqs808+bNY/v27UydOtW8YGfglVdeYfDgwbRt27bC8g0bNnD11VeTk5PDmDFjiI6OBmDjxo28+OKL/Prrr/z0009mRK53oaGhXHfddbz66quMGDHC7DjSBGjmuch5xtXVFWdnZ7NjnJHc3FyzI5yRpKQk3N3da3yg7uTkxJgxYxgzZgyTJ09myZIl9O3bl08//ZTExMQ6TlszVqsVNze3M/6gdy769OlD27Zt+fTTT6tcv3btWg4ePMjtt99eb5lqws3NrULxvK5s3ryZrVu3VmpJcvDgQUaNGkVkZCQ7d+7krbfe4u6772by5Ml8+umn7Ny5k86dO9d5vobklltu4euvvyYnJ8fsKCIiIjoerwdnejwO8NVXX1FaWsqHH37IkSNH+PXXX+sw4fmhTZs2tG/fvtLxekFBAd988w3Dhw83KVnVzPhMU5uSkpJYuHBhpeP/jIwMbrjhBmw2G5s3b+Y///kPkyZNYtKkSbz//vvExsYycOBAk1Kb45ZbbmHVqlUcOHDA7CjSBDTOf1FEpFon98srLi5mxowZXHDBBbi5uREYGMgll1zC0qVLgbK2Iu+88w5AhfYi5XJzc3nooYeIiIjA1dWV9u3b8+qrr2IYRoXHzc/P54EHHiAoKAhvb29GjBjBsWPHKvWHLu/XvXPnTkaPHo2/vz+XXHIJANu2bWP8+PGONhShoaH85S9/ITU1tcJjlW9j7969jBkzBl9fX4KDg/m///s/DMPgyJEjXHfddfj4+BAaGsprr71Wo+eupKSEZ599ljZt2uDq6kqrVq144oknKCwsdIyxWCzMnj2b3Nxcx3N1pqdJWiwWLrnkEgzDqPTH/scff2TAgAF4enri7e3N8OHD2bFjx2m3OXv2bC6//HJCQkJwdXWlU6dOvPvuuxXGtGrVih07drBy5UpH9vJeeCf3B5wyZQpeXl7k5eVVeqzbbruN0NBQSktLzzn37bffzu7du9m0aVOldfPmzcNisXDbbbcBUFhYyDPPPEPbtm1xdXUlIiKCRx99tMLrU50DBw5w8803ExAQgIeHB3379mXhwoWVxhUUFDB9+nTatWuHm5sbYWFh3HjjjcTGxjrGnPienj59Oo888ggAUVFRjuc1Li6OSy+9lO7du1eZp3379gwbNuyUmb/99ltcXFwqHQi//PLL5OTk8MEHHxAWFlbpfm3btuWvf/3rKbedkZHB1KlTHb/Xbdu25aWXXsJut1cY9+qrr3LxxRcTGBiIu7s70dHRVZ6Wa7FYmDJlCt9++y1dunTB1dWVzp07s3jx4kpjjx07xl/+8heaNWvmGPfhhx9WGnf06FGuv/56PD09CQkJYdq0adW+1ldccQW5ubmOf9dERETMpOPxhnk8PnfuXK644gouu+wyOnbseMozQPPy8rjnnnsIDAzEx8eHcePGkZ6eXmHMxo0bGTZsGEFBQbi7uxMVFcVf/vKXCmNq+tqdrLprHJW3EYqLiwNOfXwPNT/mO5XbbruNzz//vMJ9fvjhB/Ly8qq87syhQ4e47777aN++Pe7u7gQGBnLzzTc7Mp9o27ZtXHrppbi7u9OiRQuee+45Zs+eXWEfy/fzmmuuYdWqVVx00UW4ubnRunXrSmcdnvyZZtCgQSxcuJBDhw45np9WrVpV+VxWt41y7733Hm3atMHd3Z2LLrqI3377rcrn61w+syxcuJCSkhKGDBlSYfm///1vjh07xuuvv06HDh0q3a9Zs2Y89dRTp9x2TXPV5HMl1Pw1gZq/DzMyMhg/fjy+vr74+flxxx13VNuOpvw5+u6770653yK1QW1bRBqBzMxMUlJSKi0vLi4+7X2nT5/OCy+8wIQJE7jooovIyspi48aNbNq0iSuuuIJ77rmH+Ph4li5dyscff1zhvoZhMGLECJYvX85dd91Fjx49WLJkCY888gjHjh3jjTfecIwdP348X3zxBWPHjqVv376sXLnylDMRbr75Zi644AKef/55x8Hj0qVLOXDgAHfeeSehoaGO1hM7duxg3bp1lQ4gb731Vjp27MiLL77IwoULee655wgICODf//43l19+OS+99BJz587l4Ycfpnfv3qf9Nn7ChAl89NFH3HTTTTz00EP8/vvvvPDCC+zatYtvvvkGgI8//pj33nuP9evXO1qxXHzxxad9HU5WfpDm7+/vWPbxxx9zxx13MGzYMF566SXy8vJ49913ueSSS9i8ebPjQK8q7777Lp07d2bEiBE4OTnxww8/cN9992G325k8eTIAb775Jvfffz9eXl48+eSTQNmBVlVuvfVW3nnnHUd7kHJ5eXn88MMPjB8/HpvNds65b7/9dmbMmMG8efPo2bOnY3lpaSlffPEFAwYMoGXLltjtdkaMGMGqVauYOHEiHTt25I8//uCNN95g7969fPvtt9U+RmJiIhdffDF5eXk88MADBAYG8tFHHzFixAi+/PJLbrjhBsdjXnPNNfz888+MGjWKv/71r2RnZ7N06VK2b99OmzZtKm37xhtvZO/evXz66ae88cY
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1500x600 with 2 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import seaborn as sns\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выбираем столбцы для анализа\n",
|
|||
|
|
"columns_to_check = ['relative_velocity', 'absolute_magnitude']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для подсчета выбросов\n",
|
|||
|
|
"def count_outliers(df, columns):\n",
|
|||
|
|
" outliers_count = {}\n",
|
|||
|
|
" for col in columns:\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Считаем количество выбросов\n",
|
|||
|
|
" outliers = df[(df[col] < lower_bound) | (df[col] > upper_bound)]\n",
|
|||
|
|
" outliers_count[col] = len(outliers)\n",
|
|||
|
|
" \n",
|
|||
|
|
" return outliers_count\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Подсчитываем выбросы\n",
|
|||
|
|
"outliers_count = count_outliers(df, columns_to_check)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выводим количество выбросов для каждого столбца\n",
|
|||
|
|
"for col, count in outliers_count.items():\n",
|
|||
|
|
" print(f\"Количество выбросов в столбце '{col}': {count}\")\n",
|
|||
|
|
" \n",
|
|||
|
|
"# Выбираем столбцы для очистки\n",
|
|||
|
|
"columns_to_clean = ['relative_velocity', 'absolute_magnitude']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для удаления выбросов\n",
|
|||
|
|
"def remove_outliers(df, columns):\n",
|
|||
|
|
" for col in columns:\n",
|
|||
|
|
" Q1 = df[col].quantile(0.25)\n",
|
|||
|
|
" Q3 = df[col].quantile(0.75)\n",
|
|||
|
|
" IQR = Q3 - Q1\n",
|
|||
|
|
" lower_bound = Q1 - 1.5 * IQR\n",
|
|||
|
|
" upper_bound = Q3 + 1.5 * IQR\n",
|
|||
|
|
" \n",
|
|||
|
|
" # Удаляем строки, содержащие выбросы\n",
|
|||
|
|
" df = df[(df[col] >= lower_bound) & (df[col] <= upper_bound)]\n",
|
|||
|
|
" \n",
|
|||
|
|
" return df\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Удаляем выбросы\n",
|
|||
|
|
"df_cleaned = remove_outliers(df, columns_to_clean)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Выводим количество удаленных строк\n",
|
|||
|
|
"print(f\"Количество удаленных строк: {len(df) - len(df_cleaned)}\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Создаем гистограммы для очищенных данных\n",
|
|||
|
|
"plt.figure(figsize=(15, 6))\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Гистограмма для relative_velocity\n",
|
|||
|
|
"plt.subplot(1, 2, 1)\n",
|
|||
|
|
"sns.histplot(df_cleaned['relative_velocity'], kde=True)\n",
|
|||
|
|
"plt.title('Histogram of Relative Velocity (Cleaned)')\n",
|
|||
|
|
"plt.xlabel('Relative Velocity')\n",
|
|||
|
|
"plt.ylabel('Frequency')\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Гистограмма для absolute_magnitude\n",
|
|||
|
|
"plt.subplot(1, 2, 2)\n",
|
|||
|
|
"sns.histplot(df_cleaned['absolute_magnitude'], kde=True)\n",
|
|||
|
|
"plt.title('Histogram of Absolute Magnitude (Cleaned)')\n",
|
|||
|
|
"plt.xlabel('Absolute Magnitude')\n",
|
|||
|
|
"plt.ylabel('Frequency')\n",
|
|||
|
|
"\n",
|
|||
|
|
"plt.tight_layout()\n",
|
|||
|
|
"plt.show()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Сохраняем очищенный датасет\n",
|
|||
|
|
"df_cleaned.to_csv(\"..//static//csv//neo.csv\", index=False)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Можно заметить, что выбросов стало меньше"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 25,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Количество пропущенных значений в каждом столбце:\n",
|
|||
|
|
"id 0\n",
|
|||
|
|
"name 0\n",
|
|||
|
|
"est_diameter_min 0\n",
|
|||
|
|
"est_diameter_max 0\n",
|
|||
|
|
"relative_velocity 0\n",
|
|||
|
|
"miss_distance 0\n",
|
|||
|
|
"orbiting_body 0\n",
|
|||
|
|
"sentry_object 0\n",
|
|||
|
|
"absolute_magnitude 0\n",
|
|||
|
|
"hazardous 0\n",
|
|||
|
|
"dtype: int64\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Проверка на пропущенные значения\n",
|
|||
|
|
"missing_values = df.isnull().sum()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Вывод результатов\n",
|
|||
|
|
"print(\"Количество пропущенных значений в каждом столбце:\")\n",
|
|||
|
|
"print(missing_values)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Пропущенных значений в датасете нет. Можно перейти к созданию выборок"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 26,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Размер обучающей выборки: (54501, 9)\n",
|
|||
|
|
"Размер контрольной выборки: (18167, 9)\n",
|
|||
|
|
"Размер тестовой выборки: (18168, 9)\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение на признаки (X) и целевую переменную (y)\n",
|
|||
|
|
"# Предположим, что 'hazardous' - это целевая переменная\n",
|
|||
|
|
"X = df.drop('hazardous', axis=1)\n",
|
|||
|
|
"y = df['hazardous']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение на обучающую и остальную выборку (контрольную + тестовую)\n",
|
|||
|
|
"X_train, X_rem, y_train, y_rem = train_test_split(X, y, train_size=0.6, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение остатка на контрольную и тестовую выборки\n",
|
|||
|
|
"X_val, X_test, y_val, y_test = train_test_split(X_rem, y_rem, test_size=0.5, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Вывод размеров выборок\n",
|
|||
|
|
"print(\"Размер обучающей выборки:\", X_train.shape)\n",
|
|||
|
|
"print(\"Размер контрольной выборки:\", X_val.shape)\n",
|
|||
|
|
"print(\"Размер тестовой выборки:\", X_test.shape)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 27,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Распределение классов в обучающей выборке:\n",
|
|||
|
|
"hazardous\n",
|
|||
|
|
"False 0.902681\n",
|
|||
|
|
"True 0.097319\n",
|
|||
|
|
"Name: proportion, dtype: float64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение классов в контрольной выборке:\n",
|
|||
|
|
"hazardous\n",
|
|||
|
|
"False 0.902681\n",
|
|||
|
|
"True 0.097319\n",
|
|||
|
|
"Name: proportion, dtype: float64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение классов в тестовой выборке:\n",
|
|||
|
|
"hazardous\n",
|
|||
|
|
"False 0.902686\n",
|
|||
|
|
"True 0.097314\n",
|
|||
|
|
"Name: proportion, dtype: float64\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"from sklearn.model_selection import train_test_split\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение на признаки (X) и целевую переменную (y)\n",
|
|||
|
|
"X = df.drop('hazardous', axis=1)\n",
|
|||
|
|
"y = df['hazardous']\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение на обучающую и остальную выборку (контрольную + тестовую)\n",
|
|||
|
|
"X_train, X_rem, y_train, y_rem = train_test_split(X, y, train_size=0.6, random_state=42, stratify=y)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разбиение остатка на контрольную и тестовую выборки\n",
|
|||
|
|
"X_val, X_test, y_val, y_test = train_test_split(X_rem, y_rem, test_size=0.5, random_state=42, stratify=y_rem)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Функция для анализа сбалансированности\n",
|
|||
|
|
"def analyze_balance(y_train, y_val, y_test):\n",
|
|||
|
|
" print(\"Распределение классов в обучающей выборке:\")\n",
|
|||
|
|
" print(y_train.value_counts(normalize=True))\n",
|
|||
|
|
" \n",
|
|||
|
|
" print(\"\\nРаспределение классов в контрольной выборке:\")\n",
|
|||
|
|
" print(y_val.value_counts(normalize=True))\n",
|
|||
|
|
" \n",
|
|||
|
|
" print(\"\\nРаспределение классов в тестовой выборке:\")\n",
|
|||
|
|
" print(y_test.value_counts(normalize=True))\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Анализ сбалансированности\n",
|
|||
|
|
"analyze_balance(y_train, y_val, y_test)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Выборки хорошо сбалансированы и не нуждаются в корректировках"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"**Онлайн обучение**\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Проблемная область: Анализ гибкости и адаптации студентов к онлайн-обучению, включая их предпочтения, проблемы и успеваемость.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Объект наблюдения: Студенты, участвующие в онлайн-обучении, и их взаимодействие с образовательными платформами и ресурсами.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Атрибуты: возраст, пол, уровень образования, предпочтения в обучении, проблемы в обучении, успеваемость\n",
|
|||
|
|
"Возраст: Возраст студента.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Пример бизнес-цели: Разработка и внедрение платформы для онлайн-обучения, адаптированной к предпочтениям и потребностям студентов, с целью повышения их вовлеченности и успеваемости."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 38,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Index(['Education Level', 'Institution Type', 'Gender', 'Age', 'Device',\n",
|
|||
|
|
" 'IT Student', 'Location', 'Financial Condition', 'Internet Type',\n",
|
|||
|
|
" 'Network Type', 'Flexibility Level'],\n",
|
|||
|
|
" dtype='object')\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"import pandas as pd\n",
|
|||
|
|
"df = pd.read_csv(\"..//static//csv//students_adaptability_level_online_education.csv\")\n",
|
|||
|
|
"print(df.columns)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"При помощи ящика с усами и колонки возраста проверим набор на баланс."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": null,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAx8AAAIjCAYAAABia6bHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAodElEQVR4nO3deXSV1b344W/CEKiEcBUEooA4gbXiVMThoqKooHWoWMXrRJ1aBXGqVa9V0erSJd5qxaH1LjT2Sq1iUetVqqBoa2sdQByuFYHGEQQBCYggIPv3R3+kRsjEsAPkedbKanPO++7sczbvOfl4zntSkFJKAQAAsJ4VNvQEAACAxkF8AAAAWYgPAAAgC/EBAABkIT4AAIAsxAcAAJCF+AAAALIQHwAAQBbiAwAAyEJ8AGzi3nvvvSgoKIiysrKGngoAjZz4AKijsrKyKCgoqPK15ZZbRp8+fWLs2LHZ5/Pcc89VmUuzZs1i2223jVNPPTX+8Y9/rJOf8de//jWGDRsW8+fPXyfjNbQnn3wyCgoKorS0NFasWNHQ0wFodJo29AQANjbXXnttdO3aNVJKMWvWrCgrK4vDDz88Hn/88fje976XfT5Dhw6Nnj17xrJly2LSpElx9913xxNPPBFvvvlmlJaWrtXYf/3rX+Oaa66JQYMGRZs2bdbNhBvQqFGjYpttton33nsvnn322ejbt29DTwmgUfHKB0A99e/fP04++eQ45ZRT4ic/+Un8+c9/jmbNmsUDDzzQIPPp3bt3nHzyyfHDH/4wRowYETfffHPMmzcv7rvvvgaZz4Zq0aJF8dhjj8VFF10Uu+++e4waNaqhpwTQ6IgPgLXUpk2baNmyZTRtWvXF5EWLFsXFF18cnTp1iqKioujWrVvcfPPNkVKKiIjFixdH9+7do3v37rF48eLK/ebNmxcdO3aMfffdN7766qt6z+eggw6KiIjy8vIat3v22Wejd+/esdlmm0WbNm3i6KOPjr///e+V1w8bNiwuueSSiIjo2rVr5du73nvvvVrn8M23p638eu6551bZdtCgQavddtiwYVW2e/jhh+O73/1uFBcXV9nu5ptvrnU+ERGPPPJILF68OH7wgx/EwIEDY8yYMbFkyZJVtlu8eHEMHTo02rZtG8XFxXHUUUfFxx9/vNo5ffzxx3H66adH+/bto6ioKHbeeee455576jQfgMbI264A6qmioiLmzJkTKaWYPXt2jBgxIj7//PM4+eSTK7dJKcVRRx0VEyZMiDPOOCN22223eOqpp+KSSy6Jjz/+OG655ZZo2bJl3HfffbHffvvFFVdcEb/4xS8iImLw4MFRUVERZWVl0aRJk3rPb/r06RERscUWW1S7zfjx46N///6x7bbbxrBhw2Lx4sUxYsSI2G+//WLSpEmxzTbbxLHHHhvvvvtuPPDAA3HLLbdE27ZtIyKiXbt2dZrHIYccEqeeempERLzyyitx2223Vbtt27Zt45Zbbqn8/pRTTqly/YsvvhjHH3987LrrrnHjjTdGSUlJzJkzJy688MI6zSXin2+56tOnT3To0CEGDhwYl112WTz++OPxgx/8oMp2gwYNioceeihOOeWU2HvvveP555+PI444YpXxZs2aFXvvvXcUFBTEkCFDol27djF27Ng444wzYsGCBXHBBRfUeW4AjUYCoE7uvffeFBGrfBUVFaWysrIq2z766KMpItJ1111X5fLjjjsuFRQUpGnTplVedvnll6fCwsL0pz/9KY0ePTpFRLr11ltrnc+ECRNSRKR77rknffrpp2nGjBnpiSeeSNtss00qKChIr7zySkoppfLy8hQR6d57763cd7fddktbbrllmjt3buVlr7/+eiosLEynnnpq5WXDhw9PEZHKy8vrfD8tXbo0RUQaMmRI5WUrb9eECRNW2f6kk05KXbt2rXJZRKSrr7668vvLL788RUSaOXNm5WUrb9fw4cNrndOsWbNS06ZN03//939XXrbvvvumo48+usp2EydOTBGRLrjggiqXDxo0aJU5nXHGGaljx45pzpw5VbYdOHBgKikpSV988UWt8wJobLztCqCe7rjjjhg3blyMGzcu7r///ujTp0+ceeaZMWbMmMptnnzyyWjSpEkMHTq0yr4XX3xxpJSqfDrWsGHDYuedd47TTjstzj333DjggANW2a8mp59+erRr1y5KS0vjiCOOiEWLFsV9990X3/3ud1e7/cyZM2Py5MkxaNCg2HzzzSsv79GjRxxyyCHx5JNP1vlnr87KtzK1aNGiTtsvXbo0ioqKatxm4cKFUVhYuMYnvf/ud7+LwsLCGDBgQOVlJ554YowdOzY+++yzysv++Mc/RkTEueeeW2X/8847r8r3KaX4/e9/H0ceeWSklGLOnDmVX4cddlhUVFTEpEmT1miuAJsyb7sCqKe99tqryi/2J554Yuy+++4xZMiQ+N73vhfNmzeP999/P0pLS6O4uLjKvjvttFNERLz//vuVlzVv3jzuueee6NmzZ7Ro0SLuvffeKCgoqPN8rrrqqujdu3c0adIk2rZtGzvttNMq55983cqf3a1bt1Wu22mnneKpp56KRYsWxWabbVbnOXzdnDlzIiKipKSkTtvPnz8/WrVqVeM2++yzT9x+++1x/vnnx09/+tMoKSmpEg21uf/++2OvvfaKuXPnxty5cyMiYvfdd4+lS5fG6NGj4+yzz46If943hYWF0bVr1yr7b7/99lW+//TTT2P+/Plx9913x913373anzl79uw6zw+gsRAfAGupsLAw+vTpE7/85S9j6tSpsfPOO9d7jKeeeioi/vmqwdSpU1f55bcmu+yyywb1kbErT0jfZptt6rT9J598El26dKlxm4EDB8akSZNixIgR1f6yX52pU6fGK6+8EhERO+ywwyrXjxo1qjI+6mrl3wg5+eST47TTTlvtNj169KjXmACNgfgAWAeWL18eERGff/55RER06dIlxo8fHwsXLqzy6sc777xTef1Kb7zxRlx77bXxwx/+MCZPnhxnnnlmvPnmm3V+5aC+Vv7sKVOmrHLdO++8E23btq181aM+r8Cs9Oqrr0ZEVPu2r69btmxZTJs2Lfr161fjdoWFhXHzzTfHm2++GeXl5XHnnXfGrFmzqpzkX51Ro0ZFs2bN4n/+539WOYH/hRdeiNtuuy0++OCD6Ny5c3Tp0iVWrFgR5eXlVUJl2rRpVfZr165dFBcXx1dffbVBhR/Ahs45HwBradmyZfH0009H8+bNK99Wdfjhh8dXX30Vt99+e5Vtb7nlligoKIj+/ftX7jto0KAoLS2NX/7yl1FWVhazZs2q16c41VfHjh1jt912i/vuu6/KXy5/66234umnn47DDz+88rKVEVKfv3D+8MMPR7du3aJ79+61bvvYY4/F4sWLKz8euCYjRoyIZ599NkaNGhV9+/aN/fbbr07zGTVqVPTu3TtOOOGEOO6446p8rfwo4ZV/o+Wwww6LiIg777xzlZ/9dU2aNIkBAwbE73//+3jrrbdW+ZmffvppneYG0Nh45QOgnsaOHVv5Csbs2bPjt7/9bUydOjUuu+yyaN26dUREHHnkkdGnT5+44oor4r333otdd901nn766XjsscfiggsuiO222y4iIq677rqYPHlyPPPMM1FcXBw9evSIq666Kn72s5/FcccdVyUE1qXhw4dH//79Y5999okzzjij8qN2S0pKqvwtiz333DMiIq644ooYOHBgNGvWLI488sjVng/yj3/8I2666aZ4+eWX49hjj43777+/8rqVb3saN25cdO7cOTp06BBXX3113HnnnbHvvvvGoYceWuN8/+///i9++tOfxrBhw6Jnz551vp0vvfRSTJs2LYYMGbLa67faaqvYY489YtSoUXHppZfGnnvuGQMGDIhbb7015s6dW/lRu++++25EVH0l6MYbb4wJEyZEr1694qyzzopvf/vbMW/evJg0aVKMHz8+5s2bV+d5AjQaDfxpWwAbjdV91G6LFi3Sbrvtlu666660YsWKKtsvXLgwXXjhham0tDQ1a9Ys7bDDDmn48OGV202cODE1bdo0nXf
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1000x600 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Box plot для столбца 'Age'\n",
|
|||
|
|
"plt.figure(figsize=(10, 6))\n",
|
|||
|
|
"sns.boxplot(x=df['Age'])\n",
|
|||
|
|
"plt.title('Box Plot для Age')\n",
|
|||
|
|
"plt.xlabel('Age')\n",
|
|||
|
|
"plt.show()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Проверим на шум"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 40,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAIjCAYAAAAAxIqtAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKEUlEQVR4nO3dd3xUVf7/8fekhzQw1EAKTSIdERCQpiiwLEgRsEJAQBFFFleRdemyKFhYYZFFad8VG2wAFVmluksVKSKCSIko0iSQIiQkIef3B78MDGkz3CSTgdfz8cjjkdx75sxn5t5zk3fuvWdsxhgjAAAAAMB183J3AQAAAADg6QhWAAAAAGARwQoAAAAALCJYAQAAAIBFBCsAAAAAsIhgBQAAAAAWEawAAAAAwCKCFQAAAABYRLACAAAAAIsIVgBQCv3000+y2WxauHChu0tBCYqLi1NMTIy7y3BJ+/bt1b59++t6rM1m04QJE4q0npKS1xidMGGCbDabU49fuHChbDabfvrpp+IpEECJI1gBKFHfffedHnjgAUVHRysgIEBVq1bVvffeq5kzZxbbc77//vuaMWNGruXHjx/XhAkTtHv37mJ77mtt2LBBNpvN/uXr66saNWqof//+OnLkSJE8x+bNmzVhwgQlJSUVSX/u9vnnn8tmsykiIkLZ2dnuLseSmJgYh+1/9Vd6erq7yyvVUlJSNHHiRDVq1EjBwcEKDAxU/fr1NXr0aB0/ftzd5eXrb3/7m5YvX+7uMgCUAB93FwDg5rF582Z16NBBUVFRGjJkiCpXrqxffvlFW7du1d///nc988wzxfK877//vvbu3auRI0c6LD9+/LgmTpyomJgYNW7cuFieOz8jRoxQs2bNlJmZqZ07d2ru3LlauXKlvvvuO0VERFjqe/PmzZo4caLi4uJUtmzZoinYjRYvXqyYmBj99NNPWrdunTp27Ojukixp3LixnnvuuVzL/fz89M4773hcePzyyy+L/TmOHDmijh076ueff1afPn00dOhQ+fn5ac+ePZo3b56WLVumH3/8sdjrKMxf//pXvfjiiw7L/va3v+mBBx5Qjx49HJY/9thjevDBB+Xv71+CFQIoTgQrACVmypQpCgsL0/bt23P9wX/69Gn3FFUMzp8/r6CgoALbtGnTRg888IAkaeDAgbr11ls1YsQILVq0SGPGjCmJMj3C+fPntWLFCk2dOlULFizQ4sWLPT5YVa1aVY8++mie67y8PO9CEj8/v2LtPysrS7169dKpU6e0YcMG3XXXXQ7rp0yZoldffbVYa3CWj4+PfHyc+9PK29tb3t7exVwRgJLkeUdwAB7r8OHDqlevXp5nUSpWrJhr2XvvvafmzZurTJkyKleunNq2bevw3/EVK1aoa9euioiIkL+/v2rWrKnJkyfr0qVL9jbt27fXypUrdfToUfslVzExMdqwYYOaNWsm6XKwyVl39f0S27ZtU+fOnRUWFqYyZcqoXbt22rRpk0ONOfdU7Nu3Tw8//LDKlSuX6w8/Z9x9992SpISEhALbrVu3Tm3atFFQUJDKli2r+++/X/v373eo5/nnn5ckVa9e3f66nLmPI79L1DZs2JCrbVxcXJ5tr71fZunSpbrjjjsUEhLi0O61114rtB5JWrZsmdLS0tSnTx89+OCDio+Pz/OSubS0NI0YMULly5dXSEiIunfvrl9//TXPmn799VcNGjRIlSpVkr+/v+rVq6f58+c7VU9e/U2fPl02m+267zO62rX3WOXcx/Paa69p7ty5qlmzpvz9/dWsWTNt377d4bF79uxRXFycatSooYCAAFWuXFmDBg1SYmKiQ7ucffbQoUP2s5phYWEaOHCgLly4kKumwsbhtfdYZWRkaNy4cWratKnCwsIUFBSkNm3aaP369df1nvz73//Wt99+q5deeinPsRUaGqopU6Y4LFuyZImaNm2qwMBAlS9fXo8++qh+/fVXhzZxcXEKDg7Wr7/+qh49eig4OFgVKlTQn//8Z4djiCQlJSUpLi5OYWFhKlu2rAYMGJDnpbbX3mNls9l0/vx5LVq0yL7vx8XFScr/HqvZs2erXr168vf3V0REhIYPH57rudq3b6/69etr37596tChg8qUKaOqVatq2rRphbybAIoTZ6wAlJjo6Ght2bJFe/fuVf369QtsO3HiRE2YMEGtWrXSpEmT5Ofnp23btmndunW67777JF3+wyQ4OFijRo1ScHCw1q1bp3HjxiklJUXTp0+XJL300ktKTk7WsWPH9Oabb0qSgoODddttt2nSpEkaN26chg4dqjZt2kiSWrVqJelygOnSpYuaNm2q8ePHy8vLSwsWLNDdd9+t//3vf2revLlDvX369FHt2rX1t7/9TcYYl9+bw4cPS5LCw8PzbbNmzRp16dJFNWrU0IQJE5SWlqaZM2eqdevW2rlzp2JiYtSrVy/9+OOP+uCDD/Tmm2+qfPnykqQKFSo4Vce9996r/v37S5K2b9+ut956K9+25cuXt7+n0uVLm662ZcsW9e3bV40aNdIrr7yisLAwnTlzRn/605+cqkW6fBlghw4dVLlyZT344IN68cUX9emnn6pPnz4O7eLi4vTxxx/rscce05133qmvvvpKXbt2zdXfqVOndOedd8pms+npp59WhQoVtGrVKj3++ONKSUnJdbloYZKSkjR16lSXHpOZmakzZ844LCtTpozKlCmT72Pef/99paam6oknnpDNZtO0adPUq1cvHTlyRL6+vpKk1atX68iRIxo4cKAqV66s77//XnPnztX333+vrVu35ppUoW/fvqpevbqmTp2qnTt36t1331XFihUdzv44Mw6vlZKSonfffVcPPfSQhgwZotTUVM2bN0+dOnXS119/7fJlt5988omk3PtXfhYuXKiBAweqWbNmmjp1qk6dOqW///3v2rRpk3bt2uXwj51Lly6pU6dOatGihV577TWtWbNGr7/+umrWrKlhw4ZJkowxuv/++7Vx40Y9+eSTuu2227Rs2TINGDCg0Fr+9a9/afDgwWrevLmGDh0qSapZs2a+7SdMmKCJEyeqY8eOGjZsmA4cOKC3335b27dv16ZNm+zbWpLOnTunzp07q1evXurbt6+WLl2q0aNHq0GDBurSpYtT7xWAImYAoIR8+eWXxtvb23h7e5uWLVuaF154wXzxxRcmIyPDod3BgweNl5eX6dmzp7l06ZLDuuzsbPv3Fy5cyPUcTzzxhClTpoxJT0+3L+vatauJjo7O1Xb79u1GklmwYEGu56hdu7bp1KlTruerXr26uffee+3Lxo8fbySZhx56yKn3YP369UaSmT9/vvntt9/M8ePHzcqVK01MTIyx2Wxm+/btxhhjEhISctXWuHFjU7FiRZOYmGhf9u233xovLy/Tv39/+7Lp06cbSSYhIcGpmowxJiMjw0gyTz/9tH3ZkiVLjCSzfv36XO0feeQRU716dYdlksz48ePtP48ZM8ZIMidOnLAvy3ld06dPL7SmU6dOGR8fH/POO+/Yl7Vq1crcf//9Du127NhhJJmRI0c6LI+Li8tV0+OPP26qVKlizpw549D2wQcfNGFhYXnuUwW9xhdeeMFUrFjRNG3a1LRr167Q1xQdHW0k5frK6XPAgAEO+2rO+xUeHm7Onj1rX75ixQojyXz66af2ZXnV/sEHHxhJ5r///a99Wc4+O2jQIIe2PXv2NOHh4fafnR2H7dq1c3jtWVlZ5uLFiw7tz507ZypVqpTrOa99P/PSpEkTExYWVmCbHBkZGaZixYqmfv36Ji0tzb78s88+M5LMuHHj7MsGDBhgJJlJkybler6mTZvaf16+fLmRZKZNm+bwGtu0aZNrjOa8t1cLCgoyAwYMyFXrggULHMbp6dOnjZ+fn7nvvvsc3u9Zs2bZjxk52rVrZySZ//u//7Mvu3jxoqlcubLp3bt3Ie8SgOLCpYAASsy9996rLVu2qHv37vr22281bdo0derUSVWrVrX/V1qSli9fruzsbI0bNy7XPSdX/9c9MDDQ/n1qaqrOnDmjNm3a6MKFC/rhhx+uu87du3fr4MGDevjhh5WYmKg
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1000x600 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Scatter plot для столбцов 'Age' и 'Financial Condition'\n",
|
|||
|
|
"plt.figure(figsize=(10, 6))\n",
|
|||
|
|
"sns.scatterplot(x='Age', y='Financial Condition', data=df)\n",
|
|||
|
|
"plt.title('Scatter Plot для Age и Financial Condition')\n",
|
|||
|
|
"plt.xlabel('Age')\n",
|
|||
|
|
"plt.ylabel('Financial Condition')\n",
|
|||
|
|
"plt.show()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Разбиение набора данных на обучающую, контрольную и тестовую выборки"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 43,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Размер обучающей выборки: 723\n",
|
|||
|
|
"Размер контрольной выборки: 241\n",
|
|||
|
|
"Размер тестовой выборки: 241\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"#Удаление строк с пустыми значениями\n",
|
|||
|
|
"df_cleaned = df.dropna()\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение на обучающую и тестовую выборки\n",
|
|||
|
|
"train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение обучающей выборки на обучающую и контрольную\n",
|
|||
|
|
"train_df, val_df = train_test_split(train_df, test_size=0.25, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"print(\"Размер обучающей выборки:\", len(train_df))\n",
|
|||
|
|
"print(\"Размер контрольной выборки:\", len(val_df))\n",
|
|||
|
|
"print(\"Размер тестовой выборки:\", len(test_df))"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {},
|
|||
|
|
"source": [
|
|||
|
|
"Применение методов приращения данных (аугментации)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 44,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Распределение Gender в обучающей выборке после oversampling:\n",
|
|||
|
|
"Gender\n",
|
|||
|
|
"Male 397\n",
|
|||
|
|
"Female 397\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение Gender в контрольной выборке после oversampling:\n",
|
|||
|
|
"Gender\n",
|
|||
|
|
"Male 140\n",
|
|||
|
|
"Female 140\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение Gender в тестовой выборке после oversampling:\n",
|
|||
|
|
"Gender\n",
|
|||
|
|
"Female 126\n",
|
|||
|
|
"Male 126\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение Gender в обучающей выборке после undersampling:\n",
|
|||
|
|
"Gender\n",
|
|||
|
|
"Female 326\n",
|
|||
|
|
"Male 326\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение Gender в контрольной выборке после undersampling:\n",
|
|||
|
|
"Gender\n",
|
|||
|
|
"Female 101\n",
|
|||
|
|
"Male 101\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"\n",
|
|||
|
|
"Распределение Gender в тестовой выборке после undersampling:\n",
|
|||
|
|
"Gender\n",
|
|||
|
|
"Female 115\n",
|
|||
|
|
"Male 115\n",
|
|||
|
|
"Name: count, dtype: int64\n",
|
|||
|
|
"\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Разделение на обучающую и тестовую выборки\n",
|
|||
|
|
"train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"# Разделение обучающей выборки на обучающую и контрольную\n",
|
|||
|
|
"train_df, val_df = train_test_split(train_df, test_size=0.25, random_state=42)\n",
|
|||
|
|
"\n",
|
|||
|
|
"def check_balance(df, name):\n",
|
|||
|
|
" counts = df['Gender'].value_counts()\n",
|
|||
|
|
" print(f\"Распределение Gender в {name}:\")\n",
|
|||
|
|
" print(counts)\n",
|
|||
|
|
" print()\n",
|
|||
|
|
"\n",
|
|||
|
|
"def oversample(df):\n",
|
|||
|
|
" X = df.drop('Gender', axis=1)\n",
|
|||
|
|
" y = df['Gender']\n",
|
|||
|
|
" \n",
|
|||
|
|
" oversampler = RandomOverSampler(random_state=42)\n",
|
|||
|
|
" X_resampled, y_resampled = oversampler.fit_resample(X, y)\n",
|
|||
|
|
" \n",
|
|||
|
|
" resampled_df = pd.concat([X_resampled, y_resampled], axis=1)\n",
|
|||
|
|
" return resampled_df\n",
|
|||
|
|
"\n",
|
|||
|
|
"train_df_oversampled = oversample(train_df)\n",
|
|||
|
|
"val_df_oversampled = oversample(val_df)\n",
|
|||
|
|
"test_df_oversampled = oversample(test_df)\n",
|
|||
|
|
"\n",
|
|||
|
|
"check_balance(train_df_oversampled, \"обучающей выборке после oversampling\")\n",
|
|||
|
|
"check_balance(val_df_oversampled, \"контрольной выборке после oversampling\")\n",
|
|||
|
|
"check_balance(test_df_oversampled, \"тестовой выборке после oversampling\")\n",
|
|||
|
|
"\n",
|
|||
|
|
"def undersample(df):\n",
|
|||
|
|
" X = df.drop('Gender', axis=1)\n",
|
|||
|
|
" y = df['Gender']\n",
|
|||
|
|
" \n",
|
|||
|
|
" undersampler = RandomUnderSampler(random_state=42)\n",
|
|||
|
|
" X_resampled, y_resampled = undersampler.fit_resample(X, y)\n",
|
|||
|
|
" \n",
|
|||
|
|
" resampled_df = pd.concat([X_resampled, y_resampled], axis=1)\n",
|
|||
|
|
" return resampled_df\n",
|
|||
|
|
"\n",
|
|||
|
|
"train_df_undersampled = undersample(train_df)\n",
|
|||
|
|
"val_df_undersampled = undersample(val_df)\n",
|
|||
|
|
"test_df_undersampled = undersample(test_df)\n",
|
|||
|
|
"\n",
|
|||
|
|
"check_balance(train_df_undersampled, \"обучающей выборке после undersampling\")\n",
|
|||
|
|
"check_balance(val_df_undersampled, \"контрольной выборке после undersampling\")\n",
|
|||
|
|
"check_balance(test_df_undersampled, \"тестовой выборке после undersampling\")"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"metadata": {
|
|||
|
|
"kernelspec": {
|
|||
|
|
"display_name": "aimenv",
|
|||
|
|
"language": "python",
|
|||
|
|
"name": "python3"
|
|||
|
|
},
|
|||
|
|
"language_info": {
|
|||
|
|
"codemirror_mode": {
|
|||
|
|
"name": "ipython",
|
|||
|
|
"version": 3
|
|||
|
|
},
|
|||
|
|
"file_extension": ".py",
|
|||
|
|
"mimetype": "text/x-python",
|
|||
|
|
"name": "python",
|
|||
|
|
"nbconvert_exporter": "python",
|
|||
|
|
"pygments_lexer": "ipython3",
|
|||
|
|
"version": "3.12.5"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"nbformat": 4,
|
|||
|
|
"nbformat_minor": 2
|
|||
|
|
}
|