2024-10-19 00:46:39 +04:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Начало лабораторной работы"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Цены на кофе (12 варик)\n",
2024-10-20 01:50:42 +04:00
"2. Данные по инсультам (4 варик)\n",
"3. Онлайн обучение (20 варик)"
2024-10-19 00:46:39 +04:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Цены на кофе"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Index(['Date', 'Open', 'High', 'Low', 'Close', 'Adj Close', 'Volume'], dtype='object') \n",
"\n"
]
}
],
"source": [
"import numpy as np\n",
"import pandas as pd \n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"df = pd.read_csv(\"C:/Users/TIGR228/Desktop/МИИ/Lab1/AIM-PIbd-31-Afanasev-S-S/static/csv/Starbucks.csv\")\n",
"\n",
"print(df.columns, \"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Столбцы на русском"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. date: Дата\n",
"2. open: Цена открытия\n",
"3. high: Самая высокая цена дня\n",
"4. low: Самая низкая цена дня\n",
"5. Close: Цена закрытия\n",
"6. Adj Close: Скорректированная цена закрытия\n",
"7. Volume: Объем торгов\n",
"\n",
"Проблемная область: Прогнозирование динамики цен акций Starbucks на основе исторических данных о
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<bound method DataFrame.info of Date Open High Low Close Adj Close \\\n",
"0 1992-06-26 0.328125 0.347656 0.320313 0.335938 0.260703 \n",
"1 1992-06-29 0.339844 0.367188 0.332031 0.359375 0.278891 \n",
"2 1992-06-30 0.367188 0.371094 0.343750 0.347656 0.269797 \n",
"3 1992-07-01 0.351563 0.359375 0.339844 0.355469 0.275860 \n",
"4 1992-07-02 0.359375 0.359375 0.347656 0.355469 0.275860 \n",
"... ... ... ... ... ... ... \n",
"8031 2024-05-17 75.269997 78.000000 74.919998 77.849998 77.849998 \n",
"8032 2024-05-20 77.680000 78.320000 76.709999 77.540001 77.540001 \n",
"8033 2024-05-21 77.559998 78.220001 77.500000 77.720001 77.720001 \n",
"8034 2024-05-22 77.699997 81.019997 77.440002 80.720001 80.720001 \n",
"8035 2024-05-23 80.099998 80.699997 79.169998 79.260002 79.260002 \n",
"\n",
" Volume \n",
"0 224358400 \n",
"1 58732800 \n",
"2 34777600 \n",
"3 18316800 \n",
"4 13996800 \n",
"... ... \n",
"8031 14436500 \n",
"8032 11183800 \n",
"8033 8916600 \n",
"8034 22063400 \n",
"8035 4651418 \n",
"\n",
"[8036 rows x 7 columns]> \n",
"\n"
]
}
],
"source": [
"print(df.info, \"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Объектом наблюдения является - цена акций Starbucks <br>\n",
"Атрибуты — содержит набор информации о
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAy0AAAIjCAYAAAAObfTCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAD3A0lEQVR4nOzdd3xUVfr48c+5U9MbJCEQehVQ7CIIKFjAvjaUXbH8ZL+rrm3Xtq4NV11d1+7a17Ir+7W76tcCq65YEBUURekEEDCBENIz7d7z+2NKZpIJkJCZCeR5+7pk5t47Z84tifPMOec5SmutEUIIIYQQQoguykh1BYQQQgghhBBiRyRoEUIIIYQQQnRpErQIIYQQQgghujQJWoQQQgghhBBdmgQtQgghhBBCiC5NghYhhBBCCCFElyZBixBCCCGEEKJLk6BFCCGEEEII0aXZU10BIYQQYm/g8/moqqrCsixKSkpSXR0hhNirSEuLEEII0UFff/0155xzDj169MDlctGrVy9OO+20VFdLCCH2OtLSIvYYjz32GK+//jrffvstVVVVFBQUMGzYMC688EJ++ctfYhgSgwshkuff//43Z511FsOHD+f2229n0KBBABQWFqa4ZkIIsfdRWmud6koIsSvGjh1Lr169OOqoo8jOzqa6upovvviC//3f/+Wss87iX//6V6qrKIToJqqqqhg2bBiHH344L7/8Mk6nM9VVEkKIvZoELWKP4ff7cTgcrdb/9re/5eGHH6asrIz+/fsnv2JCiG7nr3/9K7fccgsbNmwgLy8v1dURQoi9nvSnEXuMeAELEAlUoruH/fvf/+b444+npKQEl8vFoEGDuO222zBNM+a1kyZNQikVWXr06MHxxx/P0qVLY/ZTSnHLLbfErPvLX/6CUopJkybFrPd4PNxyyy0MHToUt9tNr169+MUvfsGaNWsAWLduHUopnn322ZjXXXLJJSilOO+88yLrnn32WZRSOJ1Otm7dGrP/ggULIvX++uuvY7a9/PLLHHjggaSlpdGjRw9++ctfsmnTplbnbvny5Zx55pn07NmTtLQ0hg0bxg033ADALbfcEnNu4i3//e9/I+dx1KhRrcrfFW299p577kEpxbp162LWV1dXc8UVV1BaWorL5WLw4MHcddddWJYV2Sd8ju+5555W5Y4aNSrmmv33v/9FKcUrr7zSZh3PO++8XQqITz75ZPr374/b7aawsJCTTjqJ77//PmafZ555hqOOOorCwkJcLhf77LMPjz76aKuy+vfvH3MvAMyaNQu32x057+H9TjjhBObOncuYMWNwu93ss88+vPbaazGvDd9LLe+VaJMmTYqcm/B52dES/p0I3yvR6uvrKS4ujrlPwh599FFGjRpFenp6THk7ugZh33zzDVOnTiU7O5vMzEwmT57MF1980eo4d7S0/N1rae3atZxxxhnk5+eTnp7OYYcdxv/93//F7PPFF18wZswY7rjjjsi9OGTIEP785z/H3Iu5ubkx733ppZfu9Bi9Xi8333wzgwcPxuVyUVpayjXXXIPX643ZL1zm/fff36qM4cOHt3q/qqoqfv/73zN69GgyMzPJzs5m6tSpLFmyZKd12tk5jf6d2rJlCxdeeCFFRUW43W72228/nnvuuZjyon9H77vvPvr160daWhoTJ05s9ff3vPPOi/uegwcPjtnvb3/7GyNHjsTlclFSUsIll1xCdXV1zD67+jdfCNH1yJgWsceprq4mEAhQV1fHokWLuOeee5g+fTp9+/aN7PPss8+SmZnJVVddRWZmJh9++CE33XQTtbW1/OUvf4kpb/jw4dxwww1orVmzZg333nsv06ZNY8OGDTusw5133tlqvWmanHDCCXzwwQdMnz6dyy+/nLq6OubNm8fSpUsjfd5bWr16NU8++WSb72ez2fjnP//JlVdeGVn3zDPP4Ha78Xg8Mfs+++yznH/++Rx88MHceeedVFRU8MADD/DZZ5/xzTffkJubC8B3333HEUccgcPhYNasWfTv3581a9bw1ltvcfvtt/OLX/wi5kPBlVdeyYgRI5g1a1Zk3YgRI9qscyI0NjYyceJENm3axK9//Wv69u3L559/zvXXX8/PP/8c98Nbss2aNYvi4mI2b97Mww8/zJQpUygrKyM9PR0IfmAfOXIkJ510Ena7nbfeeouLL74Yy7K45JJL2iz35ptv5umnn+bFF19sFSivWrWKs846i//5n/9h5syZPPPMM5xxxhm89957HH300R06jhEjRvCPf/wj8vyJJ55g2bJl3HfffZF1++67b5uv/+tf/0pFRUWr9S+++CIXX3wxkyZN4re//S0ZGRksW7aMO+64Y6d1+uGHHzjiiCPIzs7mmmuuweFw8PjjjzNp0iQ+/vhjDj30UCZMmBBT79tvvx0gEowDHH744W2+R0VFBYcffjiNjY1cdtllFBQU8Nxzz3HSSSfxyiuvcOqppwKwbds2Pv30Uz799FMuuOACDjzwQD744AOuv/561q1bx2OPPRY5bz6fD4Bf/epXOz1Gy7I46aST+PTTT5k1axYjRozg+++/57777mPlypW88cYbMfu73W6eeeYZrrjiisi6zz//nPXr17cqe+3atbzxxhucccYZDBgwgIqKCh5//HEmTpzIjz/+uMOMZ9Hn9JNPPuGJJ57gvvvuo0ePHgAUFRUB0NTUxKRJk1i9ejWXXnopAwYM4OWXX+a8886jurqayy+/PKbc559/nrq6Oi655BI8Hg8PPPAARx11FN9//32kTACXy8VTTz0V89qsrKzI41tuuYVbb72VKVOm8Jvf/IYVK1bw6KOP8tVXX/HZZ5/FfOnVkb/5QoguQAuxhxk2bJgGIsu5556r/X5/zD6NjY2tXvfrX/9ap6ena4/HE1k3ceJEPXHixJj9/vCHP2hAb9myJbIO0DfffHPk+TXXXKMLCwv1gQceGPP6v//97xrQ9957b6v3tyxLa611WVmZBvQzzzwT2XbmmWfqUaNG6dLSUj1z5szI+meeeUYD+uyzz9ajR4+OrG9oaNDZ2dn6nHPO0YD+6quvtNZa+3w+XVhYqEeNGqWbmpoi+7/99tsa0DfddFNk3YQJE3RWVpZev3593Hq21K9fv5i6RZs4caIeOXJk3G0709Zr//KXv2hAl5WVRdbddtttOiMjQ69cuTJm3+uuu07bbDa9YcMGrXXzOf7LX/7SqtyRI0fGXLOPPvpIA/rll19us44zZ87U/fr1a9+Baa1feuklDeivv/46si7evXnsscfqgQMHxqyLPt+PP/64BvRDDz3U6rX9+vXTgH711Vcj62pqanSvXr30/vvvH1kXvpfC90o88X4fwnZ0Dm6++WYd/b+TLVu26KysLD116lQN6I8++iiy7eyzz9a5ubkx9+euXAOttT7llFO00+nUa9asiazbvHmzzsrK0hMmTGj3McVzxRVXaEB/8sknkXV1dXV6wIABun///to0zUi5gL7llltiXn/eeedpQH///fetygb0JZdcssP3/8c//qENw4h5f621fuyxxzSgP/vss5jyTj/9dG2322PusQsvvDDytyH6/TweT6T+YWVlZdrlcunZs2fvsF7RwvdS9O9m2P33368B/c9//jOyzufz6bFjx+rMzExdW1sbeV9Ap6Wl6Y0bN0b2XbhwoQb0lVdeGVk3c+ZMnZGR0WZ9tmzZop1Opz7mmGNiju/hhx/WgP773/8eWberf/OFEF2PdA8Te5xnnnmGefPm8cILL3DhhRfywgsvxHz7D5CWlhZ5XFdXR2VlJUcccQSNjY0sX748Zl+/309lZSVbt25lwYIFvP766+y7776RbxBb2rRpEw899BA33ngjmZmZMdteffVVevTowW9/+9tWr2vZfSZs0aJFvPzyy9x5551tZkD71a9+xfLlyyNde1599VVycnKYPHlyzH5ff/01W7Zs4eKLL8btdkfWH3/88QwfPjzSxWXr1q3Mnz+fCy64IKaFakf13BnTNKmsrKSysjLyzXJne/nllzniiCPIy8uLvFdlZSVTpkzBNE3mz58fs39jY2PMfpWVla26CIaF75O
"text/plain": [
"<Figure size 1000x600 with 2 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"plt.figure(figsize=(10, 6))\n",
"\n",
"plt.scatter(df['Volume'], df['Close'], c=df['Close'], alpha=0.6)\n",
"plt.colorbar(label='Close Price')\n",
"\n",
"plt.title(\"Зависимость цены закрытия от объема торгов\")\n",
"plt.ylabel(\"Цена закрытия\")\n",
"plt.xlabel(\"Объем торгов\")\n",
"plt.grid(visible=True)\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA+8AAAIjCAYAAAByJypeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACNQklEQVR4nOzdd3iT9f7G8TvpSPeik9mytyB7yFAUVATBecQjONDjD0XF48CjqByVgxPcG1w4UNwex2GJgOy9R8tuoUDTvZLn90dppLSFpk2btH2/rqtXmm+ePPkkDaV3vstkGIYhAAAAAADgsczuLgAAAAAAAJwd4R0AAAAAAA9HeAcAAAAAwMMR3gEAAAAA8HCEdwAAAAAAPBzhHQAAAAAAD0d4BwAAAADAwxHeAQAAAADwcIR3AAAAAAA8HOEdAADUaR9//LGSkpIc12fPnq1Dhw65ryAnJSUlyWQy6fnnn6/xxzaZTLrrrrtq/HEBAKUR3gHUaXv27NEdd9yh5s2by8/PTyEhIerXr59mzpypnJwcd5cHoAYsWbJEDz74oJKSkvTLL79owoQJMpsr9idQZmamHn/8cXXs2FGBgYFq0KCBunTponvuuUeHDx92HPfTTz/piSeeqKZnAACA5O3uAgCguvz444+65pprZLFYdNNNN6ljx47Kz8/XH3/8oQceeEBbtmzR22+/7e4yAVSz++67T4MGDVJCQoIkadKkSYqLizvn/QoKCjRgwABt375dY8eO1d13363MzExt2bJFc+bM0ahRo9SwYUNJReH9tddeI8ADAKoN4R1AnZSYmKjrr79ezZo104IFC0r8oT5hwgTt3r1bP/74oxsrBFBT2rZtqz179mjz5s2KjIxUixYtKnS/b775RuvWrdMnn3yiG264ocRtubm5ys/Pr45yHex2e7U/BgCg9mDYPIA66dlnn1VmZqbee++9MnvYWrZsqXvuucdxvXhe5yeffKI2bdrIz89P3bp10++//17qvocOHdItt9yimJgYWSwWdejQQe+//36ZdTzxxBMymUylvgYNGlTiuEGDBqljx46l7v/888/LZDKVmK9bWFiop556Sq1bt5bFYilx3tWrVzt9XFnGjRunoKCgUu1ffvmlTCaTFi1aVKI9Ly9Pjz/+uFq2bCmLxaImTZrowQcfVF5eXonjyps/O3z4cMXHx5d67n379lWDBg3k7++vbt266csvvzxr3cX+8Y9/qFWrVgoICFBERIQuvPBCLVmypMQx3377rS6//HI1bNhQFotFLVq00L///W/ZbLYSxw0aNKjUz+vpp5+W2WzWnDlzShzXsWNHrVmzRn379pW/v78SEhL05ptvlrjvokWLZDKZzvpcxo0b53g9iuc7n+1r3Lhxkormcp/5frHb7ercubNMJpNmz55d4nG+/PJLde/eXcHBwSXOd6651SdOnNA///lPderUSUFBQQoJCdGll16qDRs2lPlcT3+/HD58WPHx8erevbsyMzPP+ZoEBQU5nl+x+Pj4Um1z586VyWQq8T4qfu1mz56twMBA9erVSy1atNCECRNKvG7l2bNnjySpX79+pW4rnoYjFf28XnvtNUkq8ToWq+h7+fTfQx06dJDFYtHPP/9c4piXXnpJzZo1k7+/vwYOHKjNmzeXuL2s92txjWf+G7Pb7Zo5c6Y6deokPz8/RUVFadiwYef8/fDUU0/JbDbrlVdecbS98sor6tChgwICAhQeHq7u3buX+PdRluKf+7ne18X27t2ra665RhEREQoICFDv3r0r/CFseY/x1FNPOY45evSobr31VsXExMjPz0/nnXeePvjggzLPV/xv7cyvM1/jjRs3aty4cY6pW7Gxsbrlllt0/PjxEscV/18RHR2tgoKCErd9+umnjvOnpqZW6PkCqJvoeQdQJ33//fdq3ry5+vbtW+H7LF68WJ9//rkmTpwoi8Wi119/XcOGDdPKlSsdwTolJUW9e/d2/JEdFRWl//73v7r11luVnp6ue++9t8xzv/HGG44wPHny5Co9txdeeEGPPfaYRo0apYceekgWi0VLliwpNQWgosdVld1u14gRI/THH3/o9ttvV7t27bRp0ya99NJL2rlzp7755ptKnXfmzJkaMWKExowZo/z8fH322We65ppr9MMPP+jyyy8/633z8/N14403qnHjxjpx4oTeeustDRs2TNu2bVPTpk0lFf3xHRQUpEmTJikoKEgLFizQlClTlJ6erueee67cc8+aNUuPPvqoXnjhhVK9sSdPntRll12ma6+9Vn/729/0xRdf6M4775Svr69uueWWSr0OUVFR+uijjxzX582bp6+//rpE29l6kj/66CNt2rSpVPvy5ct17bXX6rzzztN//vMfhYaGKjU1Vffdd985a9q7d6+++eYbXXPNNUpISFBKSoreeustDRw4UFu3bnUMJT+T1WrVpZdeKh8fH/30009lfkBUGYWFhfrXv/5VoWN3796td955p0LHNmvWTJL04Ycf6tFHHy0RyE93xx136PDhw/rtt99K/FyKOfNeXrBggb744gvdddddioyMLBEGP/zwQ2VkZGjChAnKzc3VzJkzdeGFF2rTpk2KiYmp0HM63a233qrZs2fr0ksv1W233abCwkItWbJEf/75p7p3717mfR599FE988wzeuuttzR+/HhJ0jvvvKOJEyfq6quv1j333KPc3Fxt3LhRK1asKPVvpCwTJ05Ujx49SrTddtttJa6npKSob9++ys7O1sSJE9WgQQN98MEHGjFihL788kuNGjXqnI9z8cUX66abbirR1qVLF0lSTk6OBg0apN27d+uuu+5SQkKC5s6dq3HjxiktLa3Eh72nmzp1qmM6xgsvvKCTJ0+WuP23337T3r17dfPNNys2NtYxXWvLli36888/S72nMjIy9MMPP5R4PrNmzZKfn59yc3PP+RwB1HEGANQxVqvVkGSMHDmywveRZEgyVq9e7Wjbt2+f4efnZ4waNcrRduuttxpxcXFGampqiftff/31RmhoqJGdnV2i/ZFHHjEklTi+Q4cOxsCBA0scN3DgQKNDhw6l6nruuecMSUZiYqKjrU+fPka7du0Mu93uaJs1a5YhyVi1apXTx5Vl7NixRmBgYKn2uXPnGpKMhQsXOto++ugjw2w2G0uWLClx7JtvvmlIMpYuXepok2RMmDCh1Hkvv/xyo1mzZiXaznwt8/PzjY4dOxoXXnjhWWsvy8qVKw1Jxpdfflnu+Q3DMO644w4jICDAyM3NdbQNHDjQ8fP68ccfDW9vb+P+++8vdd+BAwcakowXXnjB0ZaXl2d06dLFiI6ONvLz8w3DMIyFCxcakoy5c+eWW+/YsWNLvR7FHn/8caO8/76Lf77F75fc3FyjadOmxqWXXmpIMmbNmuU4dvLkyYYk48iRI462xMREQ5Lx3HPPlVtb8XltNluJtsTERMNisRhTp051tBU/14ULFxq5ubnGoEGDjOjoaGP37t0l7nu21yQwMNAYO3ZsibZmzZqVaHv99dcNi8ViDB48uMTrVvx8Tn/e1157rdGxY0ejSZMmpc57puzsbKNNmzaGJKNZs2bGuHHjjPfee89ISUkpdeyECRPK/blU9L0syTCbzcaWLVtKtBc/D39/f+PgwYOO9hUrVhiSjPvuu8/Rdvr79XRnvqcWLFhgSDImTpxY6tjTf2ec/m/2/vvvN8xmszF79uwSx48cObLM31/n4szP/d577zUklfg9k5GRYSQkJBjx8fGl3o9nKu93T7EZM2YYkoyPP/7Y0Zafn2/06dPHCAoKMtLT00sc//bbb5f6P6Miv8cMwzA+/fRTQ5Lx+++/O9qK/13/7W9/M4YPH+5o37dvn2E2m42//e1vhiTj2LFjZ32eAOo2hs0DqHPS09MlScHBwU7dr0+fPurWrZvjetOmTTVy5Ej98ssvstlsMgxDX331la644goZhqHU1FTH19ChQ2W1WrV27doS5yzuKfHz8zvn49tsthLnTE1NVXZ2dqnjMjIyFB4eXm4voLPHVdXcuXPVrl07tW3btkTtF154oSRp4cKFJY7Pzc0t9TzPHCY
"text/plain": [
"<Figure size 1200x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"df['Year'] = pd.to_datetime(df['Date']).dt.year\n",
"\n",
"year_close = df.groupby('Year')['Close'].mean().reset_index()\n",
"\n",
"plt.figure(figsize=(12, 6))\n",
"\n",
"plt.plot(year_close['Year'], year_close['Close'], marker='.')\n",
"\n",
"plt.title(\"Средняя цена закрытия акций Starbucks по годам\")\n",
"plt.xlabel(\"Год\")\n",
"plt.ylabel(\"Средняя цена закрытия\")\n",
"\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Связь между объектами есть. Цена связана почти с о А
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Примеры бизнес-целей</h3>\n",
"\n",
"1. Прогнозирование динамики цен акций Starbucks на основе исторических данных о
"2. Наблюдение за изменениями цен акций Starbucks с
"\n",
"Эффект для бизнеса: Оценка и оптимизация цен, оценка и планирование затрат, выявление тенденций на рынке, стратегия планирования.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Цели технического проекта</h3>\n",
"<ul>Для первой цели:</ul>\n",
" <li>Вход: Исторические данные о
" <li>Целевой признак: Цена закрытия.</li>\n",
"<ul>Для второй цели:</ul>\n",
" <li>Вход: Исторические данные о
" <li>Целевой признак: Год</li>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Код ниже нужен для определения проблем данных</h3>"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Максимальные значения:\n",
" Date 2024-05-23\n",
"Open 126.080002\n",
"High 126.32\n",
"Low 124.809998\n",
"Close 126.059998\n",
"Adj Close 118.010414\n",
"Volume 585508800\n",
"Year 2024\n",
"dtype: object \n",
"\n",
"Столбцы с
" Index([], dtype='object') \n",
"\n",
"Признаки с
" Series([], dtype: float64) \n",
"\n",
"Годы:\n",
" 0 1992\n",
"1 1992\n",
"2 1992\n",
"3 1992\n",
"4 1992\n",
" ... \n",
"8031 2024\n",
"8032 2024\n",
"8033 2024\n",
"8034 2024\n",
"8035 2024\n",
"Name: Year, Length: 8036, dtype: int32\n"
]
}
],
"source": [
"import pandas as pd\n",
"\n",
"max_value = df.max(axis=0)\n",
"\n",
"columns_with_zero = df.columns[(df == 0).any()]\n",
"\n",
"numeric_data = df.select_dtypes(include='number')\n",
"shum = numeric_data.var()\n",
"low_dispers = 0.1\n",
"low_var_columns = shum[shum < low_dispers]\n",
"\n",
"df['Year'] = pd.to_datetime(df['Date']).dt.year\n",
"print(\"Максимальные значения:\\n\", max_value, \"\\n\")\n",
"print(\"Столбцы с
"print(\"Признаки с
"print(\"Годы:\\n\", df['Year'])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h4>Из полученных данных выяснилось:</h4></ul> <li>Столбцы с с с с с с с с о с
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<ol><h3>Примеры решения проблем для набора данных</h3></ol>\n",
" <li>Удаление выбросов на основе значения или Volume</li>\n",
" <li>Удаление или обновить устаревшие даты, так как наличие будущих дат может указывать на проблему с
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Оценка качества данных</h3>\n",
"1. Информативность. Н а б о р
"2. Степень покрытия. Н а б о р о
"3. Соответствие реальным данным. Данные вполне кажутся реальными, не считая некоторых редких выбросов.\n",
"4. Согласованность меток. Метки состояние и оценка вида, имеют четкие значения."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Разбиение данных на обучающую, контрольную и тестовую выборки</h3>"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Исходный размер строк: 8036 строк\n",
"Размер обучающей выборки: 5625 строк\n",
"Размер валидационной выборки: 1205 строк\n",
"Размер тестовой выборки: 1206 строк\n"
]
}
],
"source": [
"df_numeric = df.select_dtypes(include='number')\n",
"\n",
"x = df_numeric.drop(['Close'], axis=1)\n",
"y = df_numeric['Close']\n",
"\n",
"x_train, x_temp, y_train, y_temp = train_test_split(x, y, test_size=0.3, random_state=14)\n",
"\n",
"x_val, x_test, y_val, y_test = train_test_split(x_temp, y_temp, test_size=0.5, random_state=14)\n",
"\n",
"print(f\"Исходный размер строк: {df_numeric.shape[0]} строк\")\n",
"print(f\"Размер обучающей выборки: {x_train.shape[0]} строк\")\n",
"print(f\"Размер валидационной выборки: {x_val.shape[0]} строк\")\n",
"print(f\"Размер тестовой выборки: {x_test.shape[0]} строк\")\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAAIjCAYAAAAJLyrXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACwEklEQVR4nOzdd3hUVf4/8Ped3jKppABJKKFXxRYLIFXECurasay4Crrqrrr4VSm69hVdRSzroiugK4q68kNpChZAEaVHpIeWhCHJTCbT7sw9vz+GjIQkkAkTpr1fz8OjuXPvuWfuuTNzP/ec87mSEEKAiIiIiIiImk0V7QoQERERERHFGwZSREREREREYWIgRUREREREFCYGUkRERERERGFiIEVERERERBQmBlJERERERERhYiBFREREREQUJgZSREREREREYdJEuwJERERE0eLxeFBZWQmNRoPs7OxoV4ciqLa2FpWVlbBYLEhPT492dSgBsUeKiIiIksrSpUtx2WWXIS0tDUajEe3atcOf//znaFcrbjz11FNQFAUAoCgKnn766SjX6Hfz5s3D0KFDkZKSAovFgoKCAjz33HPRrhYlKAZSFHPeeecdSJIU+mcwGNC1a1dMnDgR5eXl0a4eUUyy2WyQJAlTpkyJdlWIYtprr72GkSNHwm634+WXX8aSJUuwZMkSTJs2LdpVixvvvvsuXnjhBezbtw//+Mc/8O6770a7SgCAv/3tb7jmmmuQkpKCt956C0uWLMHSpUtx9913R7tqlKA4tI9i1rRp09CxY0d4PB589913mDlzJhYuXIhNmzbBZDJFu3pERBRntm3bhgceeADjx4/Ha6+9BkmSol2luDRt2jTcfPPNePjhh6HX6zF79uxoVwkrVqzAs88+i6effhp/+9vfol0dShIMpChmjRo1CmeccQYA4I9//CMyMzPx4osv4rPPPsN1110X5doREVG8+ec//4nc3Fz885//ZBB1Ev7whz/gwgsvxPbt29GlSxe0adMm2lXCCy+8gHPPPZdBFJ1SHNpHcWPIkCEAgF27dgEAKisr8de//hV9+vSBxWKB1WrFqFGjsH79+gbbejweTJkyBV27doXBYEBeXh7GjBmDHTt2AAB2795dbzjhsf8GDx4cKmv58uWQJAn//e9/8cgjjyA3NxdmsxmXXXYZ9u7d22DfP/zwAy666CKkpqbCZDJh0KBB+P777xt9j4MHD250/40N15o9ezYGDBgAo9GIjIwMXHvttY3u/3jv7WiKouCll15Cr169YDAYkJOTgzvvvBNVVVX11uvQoQMuueSSBvuZOHFigzIbq/vzzz/f4JgCgNfrxeTJk1FUVAS9Xo/8/Hw89NBD8Hq9jR6rlr7P2tpa/OUvf0F+fj70ej26deuGF154AUKIBnVv7N+TTz4JAPD5fHj88ccxYMAApKamwmw244ILLsDXX3/daL1eeOEFTJ8+HYWFhTAajRg0aBA2bdpUb91bbrkFHTp0qLds7969MBqNkCQJu3fvDi13uVy49dZbYTab0bNnT6xduxYAIMsybr31VphMJvTr1w8//fRTvfLqzrErrriiwTG88847IUkSevfuXW953QVKZmYmjEYjBgwYgI8++qjxhjjG4MGDG5RXV+ax7wkAqqurcd9994Xap6ioCM8++2xoPgZQ/5geq3fv3o1+Xo9X38aOe2Muv/xydOjQAQaDAdnZ2bjsssuwcePGeuvMmjULQ4YMQXZ2NvR6PXr27ImZM2c2KKtDhw645ZZb6i0bP348DAYDli9fXm+9Sy65BIsXL0b//v1hMBjQs2dPzJ8/v962dUOij23vow0ePDh0bOqOy/H+1X12p0yZ0uBz5HQ6kZubC0mS6tUXAGbOnInevXvDZDLVK+9E50zdfur+paSk4KyzzsKnn3563O3q/PLLLxg1ahSsVissFguGDh2K1atX11tn9erVGDBgAO6++27k5ORAr9ejd+/eeOutt0LrCCHQoUMHXH755Q324fF4kJqaijvvvLPJYwM0bN/m/l7VndvvvPNOaNlvv/2GK6+8Eunp6TAajTjzzDMbHJOmznOLxdLgPGvsuxoAfv31V1x11VXIyMiAwWDAGWecgf/973/11qk7z3bv3o3s7OzQ90Lfvn0b1Lsxxw7dN5lM6NOnD/71r3/VW++WW26BxWI5blnH/r6sXr0avXv3xrXXXouMjIwmjxUAVFRU4Pbbb0dOTg4MBgP69evXYHjiyX53z549GyqVCs8880y95c05zhQ/2CNFcaMu6MnMzAQA7Ny5E59++imuvvpqdOzYEeXl5XjjjTcwaNAgbNmyBW3btgUABAIBXHLJJVi2bBmuvfZa/PnPf0ZNTQ2WLFmCTZs2oXPnzqF9XHfddbj44ovr7XfSpEmN1ufvf/87JEnCww8/jIqKCrz00ksYNmwY1q1bB6PRCAD46quvMGrUKAwYMACTJ0+GSqUKXWh9++23OOussxqU2759+9DEXafTibvuuqvRfT/22GO45ppr8Mc//hGHDh3CK6+8goEDB+KXX35BWlpag23Gjx+PCy64AAAwf/58fPLJJ/Vev/POO/HOO+/g1ltvxb333otdu3bh1VdfxS+//ILvv/8eWq220eMQjurq6kYnJSuKgssuuwzfffcdxo8fjx49emDjxo2YPn06fvvtt2ZfSAHHf59CCFx22WX4+uuvcfvtt6N///5YtGgRHnzwQezfvx/Tp0+vV9bw4cNx880311vWv39/AIDD4cC//vUvXHfddbjjjjtQU1ODt99+GyNHjsSPP/4YWq/Of/7zH9TU1GDChAnweDx4+eWXMWTIEGzcuBE5OTlNvp/HH38cHo+nwfL7778f7777LiZOnIj27duH5gC8+eabGDJkCJ588km8/PLLGDVqFHbu3ImUlJTQtgaDAf/v//0/VFRUhLKUud1u/Pe//4XBYGiwr5dffhmXXXYZbrjhBvh8PnzwwQe4+uqrsWDBAowePbrJuofL5XJh0KBB2L9/P+68804UFBRg5cqVmDRpEg4ePIiXXnopYvtqqfHjxyM3NxcHDhzAq6++imHDhmHXrl2h4cYzZ85Er169cNlll0Gj0eDzzz/H3XffDUVRMGHChCbLnTx5Mt5++23897//bXCTYdu2bfjDH/6AP/3pTxg3bhxmzZqFq6++Gl9++SWGDx/eovfRo0cPvPfee6G/33zzTZSUlNT7DPTt27fJ7f/xj380Omf1v//9L+6++24MHjwY99xzD8xmM0pKSvDUU081u2519bLZbHjttddw9dVXY9OmTejWrVuT22zevBkXXHABrFYrHnroIWi1WrzxxhsYPHgwVqxYgbPPPhsAcPjwYfz000/QaDSYMGECOnfujE8//RTjx4/H4cOH8be//Q2SJOHGG2/Ec889h8rKSmRkZIT28/nnn8PhcODGG29s9vsBmv97dazKykoMHDgQNTU1uPfee5Gbm4vZs2djzJgxmDNnTsRGZ2zevBnnnXce2rVrh7/97W8wm8348MMPccUVV+Djjz/GlVde2eS27733XoMbCicyffp0ZGVlweFw4N///jfuuOMOdOjQAcOGDWvxezh8+DDefPNNWCwW3HvvvWjTpk2jx8rtdmPw4MHYvn07Jk6ciI4dO2LevHm45ZZbUF1d3SDpSEu+uxcvXozbbrsNEydOrNdDdjLHmWKUIIoxs2bNEgDE0qVLxaFDh8TevXvFBx98IDIzM4XRaBT79u0TQgjh8XhEIBCot+2uXbuEXq8X06ZNCy3797//LQCIF198scG+FEUJbQdAPP/88w3W6dWrlxg0aFDo76+//loAEO3atRMOhyO0/MMPPxQAxMsvvxwqu0uXLmLkyJGh/QghhMvlEh07dhTDhw9vsK9zzz1X9O7dO/T3oUOHBAAxefLk0LLdu3cLtVot/v73v9fbduPGjUKj0TRYvm3bNgFAvPvuu6FlkydPFkd//L/99lsBQMyZM6fetl9++WWD5YWFhWL06NEN6j5hwgRx7FfKsXV/6KGHRHZ2thgwYEC9Y/ree+8JlUolvv3223rbv/766wKA+P7
"text/plain": [
"<Figure size 1000x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAAIjCAYAAAAJLyrXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACMzklEQVR4nOzdeXxU1f3/8fedJbNkJQSyQICwbwIVUXEBFxBxr1brVnFp9adi61JtaVUQtdYdq9bta1FbccEFW2sVpSquiCgoFZE9rIEEsk4y6/n9ETISEkIGJkyW1/Px4KG5M3PmM3PuLO+5555jGWOMAAAAAADNZkt0AQAAAADQ1hCkAAAAACBGBCkAAAAAiBFBCgAAAABiRJACAAAAgBgRpAAAAAAgRgQpAAAAAIgRQQoAAAAAYkSQAgAAAJpQUVGhtWvXqqqqKtGloBUhSAEAAOCAq6ys1IwZM6J/l5aW6tFHH01cQbswxujJJ5/U4YcfLq/Xq7S0NBUUFOgf//hHoktDK0KQQtw888wzsiwr+s/tdqt///6aPHmyioqKEl0e0CoVFxfLsixNmzYt0aUAwAHl8Xh088036/nnn9f69es1bdo0/etf/0p0WZKk888/X//v//0/DRo0SH//+9/17rvv6r333tOZZ56Z6NLQijgSXQDan+nTp6ugoEA1NTX6+OOP9dhjj+mtt97S0qVL5fV6E10eAABoBex2u2677TZddNFFikQiSktL07///e9El6XnnntOL730kv7xj3/o/PPPT3Q5aMUIUoi7iRMn6pBDDpEk/fKXv1Tnzp31wAMP6I033tB5552X4OoAAEBrccMNN+jnP/+51q9fr0GDBikjIyPRJenee+/VeeedR4jCXjG0Dy3uuOOOkyStWbNGkrR9+3b99re/1UEHHaSUlBSlpaVp4sSJWrJkSYPb1tTUaNq0aerfv7/cbrdyc3N15plnatWqVZKktWvX1htOuPu/Y445JtrWBx98IMuy9NJLL+kPf/iDcnJylJycrNNOO03r169vcN8LFizQiSeeqPT0dHm9Xo0dO1affPJJo4/xmGOOafT+Gxuu9Y9//EMjR46Ux+NRZmamzj333Ebvv6nHtqtIJKIZM2ZoyJAhcrvdys7O1hVXXKEdO3bUu16vXr10yimnNLifyZMnN2izsdrvvffeBs+pJPn9fk2dOlV9+/aVy+VSfn6+brrpJvn9/kafq319nFVVVbrhhhuUn58vl8ulAQMG6L777pMxpkHtjf274447JEmBQEC33nqrRo4cqfT0dCUnJ+voo4/W+++/32hd9913nx588EH17NlTHo9HY8eO1dKlS+td9+KLL1avXr3qbVu/fr08Ho8sy9LatWuj230+ny655BIlJydr8ODBWrRokSQpGAzqkksukdfr1fDhw/Xll1/Wa69uHzvjjDMaPIdXXHGFLMvS0KFD622/7777dMQRR6hz587yeDwaOXKkXnnllcY7YjfHHHNMg/bq2tz9MUm15zZce+210f7p27ev7r77bkUikeh1dn1Odzd06NBGX69N1dvY896Y008/Xb169ZLb7VbXrl112mmn6dtvv613nZkzZ+q4445T165d5XK5NHjwYD322GMN2urVq5cuvvjietsuv/xyud1uffDBB/Wud8opp2ju3LkaMWKE3G63Bg8erNdee63ebeuGRO/e37s65phjos9N3fPS1L+61+60adMavI4qKyuVk5Mjy7Lq1StJjz32mIYOHSqv11uvvb3tM3X3U/cvNTVVhx56qObMmdPk7WK57UcffaSzzz5bPXr0iL7PXHfddaqurm603T09N7vut7G8zzVWa2OfM7HUevHFFyslJaXB/bzyyisN+qeu33fvs5NPPrnRx/H1119r4sSJSktLU0pKio4//nh9/vnn9a6zp32vseHGdY+9uLi43nW//PJLWZalZ555pt72//73vzr66KOVnJysjIwMnX766Vq2bFm96+y6f3bv3l2jR4+Ww+HY4/65u+buO3t6L6tT975U9xiqqqq0dOlS5efn6+STT1ZaWpqSk5N1zDHH6KOPPmpw+9WrV+vss89WZmamvF6vDj/88AZH1WL5/rHr673OnXfeKZvNplmzZtXbHsv3FLQMjkihxdWFns6dO0uqfdOZM2eOzj77bBUUFKioqEhPPPGExo4dq++++055eXmSpHA4rFNOOUXz5s3Tueeeq9/85jeqqKjQu+++q6VLl6pPnz7R+zjvvPN00kkn1bvfKVOmNFrPnXfeKcuy9Lvf/U5bt27VjBkzNG7cOC1evFgej0dS7YfAxIkTNXLkSE2dOlU2my36Reujjz7SoYce2qDd7t2766677pJU+2XlyiuvbPS+b7nlFp1zzjn65S9/qW3btunhhx/WmDFj9PXXXzf6S9zll1+uo48+WpL02muv6fXXX693+RVXXKFnnnlGl1xyiX79619rzZo1euSRR/T111/rk08+kdPpbPR5iEVpaWn0se0qEonotNNO08cff6zLL79cgwYN0rfffqsHH3xQP/zwQ7O+SNVp6nEaY3Taaafp/fff12WXXaYRI0bonXfe0Y033qiNGzfqwQcfrNfW+PHjddFFF9XbNmLECElSeXm5/u///k/nnXeefvWrX6miokJPP/20JkyYoC+++CJ6vTrPPfecKioqdPXVV6umpkYPPfSQjjvuOH377bfKzs7e4+O59dZbVVNT02D7ddddp2effVaTJ09W9+7dddVVV0mSnnzySR133HG644479NBDD2nixIlavXq1UlNTo7d1u93697//ra1bt6pr166SpOrqar300ktyu90N7uuhhx7SaaedpgsuuECBQEAvvviizj77bL355ps6+eST91h7rHw+n8aOHauNGzfqiiuuUI8ePfTpp59qypQp2rx5c72TyRPl8ssvV05OjjZt2qRHHnlE48aN05o1a6LDjR977DENGTJEp512mhwOh/71r3/pqquuUiQS0dVXX73HdqdOnaqnn35aL730UoMvPytWrNDPf/5z/b//9/80adIkzZw5U2effbbefvttjR8/fp8eR935GnWefPJJLVu2rN5rYNiwYXu8/f3339/oOasvvfSSrrrqKh1zzDG65pprlJycrGXLlulPf/pTs2urq6u4uFh//etfdfbZZ2vp0qUaMGDAft929uzZ8vl8uvLKK9W5c2d98cUXevjhh7VhwwbNnj270TZ/+tOfRs9n+eijj/Tkk082WcOe3ud299hjj0UDUGOfM/tS676YP3++3nrrrQbb//e//+noo49WWlqabrrpJjmdTj3xxBM65phj9OGHH+qwww6LWw2Nee+99zRx4kT17t1b06ZNU3V1tR5++GEdeeSR+uqrr5r88WNP+2dT9me/a0xJSYkk6e6771ZOTo5uvPFGud1uPfXUUxo3bpzeffddjRkzRpJUVFSkI444Qj6fT7/+9a/VuXNnPfvsszrttNP0yiuv6Kc//Wm9tpvz/WN3M2fO1M0336z777+/3hGyffmeghZggDiZOXOmkWTee+89s23bNrN+/Xrz4osvms6dOxuPx2M2bNhgjDGmpqbGhMPherdds2aNcblcZvr06dFtf/vb34wk88ADDzS4r0gkEr2dJHPvvfc2uM6QIUPM2LFjo3+///77RpLp1q2bKS8vj25/+eWXjSTz0EMPRdvu16+fmTBhQvR+jDHG5/OZgoICM378+Ab3dcQRR5ihQ4dG/962bZuRZKZOnRrdtnbtWmO3282dd95Z77bffvutcTgcDbavWLHCSDLPPvtsdNvUqVPNri/bjz76yEgyzz//fL3bvv322w229+zZ05x88skNar/66qvN7m8Fu9d+0003ma5du5qRI0fWe07//ve/G5vNZj766KN6t3/88ceNJPPJJ580uL/dNedxzpkzx0gyd9xxR73b/uxnPzOWZZmVK1fWq/3qq6/e4/2FQiHj9/vrbduxY4fJzs42l156aXRb3b61675rjDELFiwwksx1110X3TZp0iTTs2fP6N9Lly41NpvNTJw40Ugya9asMcYYs2XLFpOUlGSmTJk
"text/plain": [
"<Figure size 1000x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAIjCAYAAADWYVDIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACOt0lEQVR4nOzdeXwTdf4/8NdM7qRNW1p6QS8olAICKyqLCHhwiCiyuioqK7Ku+lNwV93VXXZVEHUVT/wKnsvisYsHinisCooHHoiIgKAVudpytqRH0jTNNfP5/REaU9pCj5QcfT0fjz6g08kn72Qmk7wyn/l8JCGEABEREREREQEA5EgXQEREREREFE0YkoiIiIiIiEIwJBEREREREYVgSCIiIiIiIgrBkERERERERBSCIYmIiIiIiCgEQxIREREREVEIhiQiIiIiIqIQDElEREREFBVqa2uxc+dO+P3+SJdC3RxDEhEREVEc27dvH55//vng76Wlpfjvf/8buYJC+Hw+PPjggxg6dCgMBgNSUlLQr18/rFmzJtKlUTfHkERR6/nnn4ckScEfo9GI/v37Y/bs2aioqIh0eURRyWazQZIkzJs3L9KlEFGUkCQJs2bNwqpVq1BaWorbb78dn3/+eaTLgsfjwbhx43DnnXfizDPPxPLly/Hhhx/i448/xsiRIyNdHnVz2kgXQHQ88+fPR0FBAdxuN7744gs89dRTeO+997Bt2zaYzeZIl0dERBTVevXqhWuvvRbnnnsuACArKwuffvppZIsCsGDBAqxfvx6rVq3CmWeeGelyiJpgSKKoN2nSJJxyyikAgD/84Q9ITU3Fo48+irfeeguXX355hKsjIiKKfgsXLsRNN90Em82GwYMHw2KxRLQev9+PhQsX4s9//jMDEkUldrejmHP22WcDAPbs2QMAqK6uxl/+8hecdNJJSEhIgNVqxaRJk7Bly5Zmt3W73Zg3bx769+8Po9GIrKwsXHTRRdi1axeAQD/t0C5+R/+EHsg//fRTSJKEV199FX//+9+RmZkJi8WCKVOmYO/evc3ue/369Tj33HORlJQEs9mMsWPH4ssvv2zxMZ555pkt3n9LXaj+85//YPjw4TCZTOjRowemTZvW4v0f67GFUlUVCxcuxKBBg2A0GpGRkYHrr78eNTU1TdbLz8/H+eef3+x+Zs+e3azNlmp/6KGHmj2nQKD7xdy5c1FYWAiDwYCcnBzcfvvt8Hg8LT5XHX2c9fX1+POf/4ycnBwYDAYUFRXh4YcfhhCiWe0t/dx7770AAK/Xi7vuugvDhw9HUlISLBYLRo8ejU8++aTFuh5++GE89thjyMvLg8lkwtixY7Ft27Ym61599dXIz89vsmzv3r0wmUyQJAmlpaXB5S6XCzNnzoTFYsHAgQOxceNGAIF+/jNnzoTZbMbQoUPx7bffNmmvcR+bOnVqs+fw+uuvhyRJGDx4cJPlDz/8ME4//XSkpqbCZDJh+PDheP3111veEEc588wzm7XX2ObRjwkIXLx98803B7dPYWEhFixYAFVVg+uEPqdHGzx4cIuv12PV29Lz3pILL7wQ+fn5MBqNSE9Px5QpU7B169Ym6yxduhRnn3020tPTYTAYMHDgQDz11FPN2srPz8fVV1/dZNl1110Ho9HY5Jv+xtfb6tWrMWzYMBiNRgwcOBArVqxoctvGbspHb+9QZ555ZvC5aXxejvXT+NqdN29es9eR0+lEZmYmJElqdmbiqaeewuDBg2E2m5u0d7x9pvF+Gn8SExNx2mmnYeXKle26XUs/oTW29Zi8f/9+XHPNNcjOzobBYEBBQQFuuOEGeL3eZt3CW/oJvRbo448/xujRo2GxWJCcnIwLL7wQJSUlHX78y5cvDx7/09LSMH36dOzfv7/JOqH7dd++fTFixAhUV1e3eDxpydVXX92knpSUFJx55pnNuuu19p7QqHFfa9wG27dvR01NDRITEzF27FiYzWYkJSXh/PPPb3ZMBIBNmzZh0qRJsFqtSEhIwDnnnIOvv/66yTqN22Pt2rW4/vrrkZqaCqvViquuuqrF97C2vPYA4P333w9ut8TEREyePBk//PDDMZ83in08k0QxpzHQpKamAgB2796NlStX4pJLLkFBQQEqKirwzDPPYOzYsfjxxx+RnZ0NAFAUBeeffz7WrFmDadOm4U9/+hPq6urw4YcfYtu2bejbt2/wPi6//HKcd955Te53zpw5LdZz3333QZIk/PWvf0VlZSUWLlyIcePGYfPmzTCZTAACb4yTJk3C8OHDMXfuXMiyHPwQ9fnnn+O0005r1m7v3r1x//33Awh8ELnhhhtavO8777wTl156Kf7whz/g8OHDeOKJJzBmzBhs2rQJycnJzW5z3XXXYfTo0QCAFStW4M0332zy9+uvvx7PP/88Zs6ciT/+8Y/Ys2cPFi1ahE2bNuHLL7+ETqdr8Xloj9ra2uBjC6WqKqZMmYIvvvgC1113HYqLi7F161Y89thj+Pnnn4/7ISnUsR6nEAJTpkzBJ598gmuuuQbDhg3DqlWrcNttt2H//v147LHHmrQ1fvx4XHXVVU2WDRs2DADgcDjwr3/9C5dffjmuvfZa1NXVYcmSJZg4cSK++eab4HqNXnzxRdTV1WHWrFlwu914/PHHcfbZZ2Pr1q3IyMho9fHcddddcLvdzZbfcssteOGFFzB79mz07t0bN954IwDg2Wefxdlnn417770Xjz/+OCZNmoTdu3cjMTExeFuj0Yj//e9/qKysRHp6OgCgoaEBr776KoxGY7P7evzxxzFlyhRceeWV8Hq9eOWVV3DJJZfg3XffxeTJk1utvb1cLhfGjh2L/fv34/rrr0dubi6++uorzJkzBwcPHsTChQvDdl8ddd111yEzMxMHDhzAokWLMG7cOOzZsyfYBfipp57CoEGDMGXKFGi1Wrzzzju48cYboaoqZs2a1Wq7c+fOxZIlS/Dqq682+wJhx44duOyyy/D//t//w4wZM7B06VJccskl+OCDDzB+/PgOPY7i4mK89NJLwd+fffZZlJSUNHkNDBkypNXbP/LIIy1eI/rqq6/ixhtvxJlnnombbroJFosFJSUl+Oc//9nm2hrrstlsePLJJ3HJJZdg27ZtKCoqanH9iy66CIWFhcHfb7nlFhQXF+O6665r8niBth+TDxw4gNNOOw21tbW47rrrMGDAAOzfvx+vv/46XC4XxowZ0+T5u++++wAA//jHP4LLTj/9dADARx99hEmTJqFPnz6YN28eGhoa8MQTT2DUqFH47rvvmgX04z3+xuP0qaeeivvvvx8VFRV4/PHH8eWXX7Z6/G/U2vGkNWlpacF9Yt++fXj88cdx3nnnYe/evce8n2OpqqoCEHhv7devH+6++2643W4sXrwYo0aNwoYNG9C/f38AwA8//IDRo0fDarXi9ttvh06nwzPPPIMzzzwTn332GUaMGNGk7dmzZyM5ORnz5s3D9u3b8dRTT6GsrCwY1FrS2mvvpZdewowZMzBx4kQsWLAALpcLTz31FM444wxs2rSpTV+sUIwSRFFq6dKlAoD46KOPxOHDh8XevXvFK6+8IlJTU4XJZBL79u0TQgjhdruFoihNbrtnzx5hMBjE/Pnzg8v+/e9/CwDi0UcfbXZfqqoGbwdAPPTQQ83WGTRokBg7dmzw908++UQAEL169RIOhyO4/LXXXhMAxOOPPx5su1+/fmLixInB+xFCCJfLJQoKCsT48eOb3dfpp58uBg8eHPz98OHDAoCYO3ducFlpaanQaDTivvvua3LbrVu3Cq1W22z5jh07BADxwgsvBJfNnTtXhB4GPv/8cwFA/Pe//21y2w8++KDZ8ry8PDF58uRmtc+aNUscfWg5uvbbb79dpKeni+HDhzd5Tl966SUhy7L4/PPPm9z+6aefFgDEl19+2ez+jtaWx7ly5UoBQNx7771Nbvvb3/5WSJIkdu7c2aT2WbNmtXp/fr9feDyeJstqampERkaG+P3vfx9c1rhvhe67Qgixfv16AUDccsstwWUzZswQeXl5wd+3bdsmZFkWkyZNEgDEnj17hBBCHDp0SOj
"text/plain": [
"<Figure size 1000x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Статистические показатели для обучающей выборки:\n",
"Среднее значение: 2.53\n",
"Стандартное отклонение: 1.54\n",
"Минимальное значение: -1.06\n",
"Максимальное значение: 4.84\n",
"Количество наблюдений: 5625\n",
"\n",
"Статистические показатели для валидационной выборки:\n",
"Среднее значение: 2.44\n",
"Стандартное отклонение: 1.58\n",
"Минимальное значение: -1.02\n",
"Максимальное значение: 4.79\n",
"Количество наблюдений: 1205\n",
"\n",
"Статистические показатели для тестовой выборки:\n",
"Среднее значение: 2.49\n",
"Стандартное отклонение: 1.57\n",
"Минимальное значение: -1.09\n",
"Максимальное значение: 4.81\n",
"Количество наблюдений: 1206\n",
"\n"
]
}
],
"source": [
"import seaborn as sns\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"# Логарифмирование целевой переменной\n",
"df['Close_log'] = np.log(df['Close'])\n",
"\n",
"# Выбор признаков и целевой переменной\n",
"X = df.drop(['Close', 'Close_log'], axis=1)\n",
"y = df['Close_log']\n",
"\n",
"# Выбор только числовых признаков\n",
"X = X.select_dtypes(include='number')\n",
"\n",
"# Разделение данных на обучающую, валидационную и тестовую выборки\n",
"X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42)\n",
"X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)\n",
"\n",
"# Функция для построения гистограммы распределения целевого признака\n",
"def plot_distribution(data, title):\n",
" \"\"\"Построение гистограммы распределения целевого признака\"\"\"\n",
" plt.figure(figsize=(10, 6))\n",
" sns.histplot(data, kde=True, bins=30, color='skyblue')\n",
" plt.title(title)\n",
" plt.xlabel('Logarithm of Close Price')\n",
" plt.ylabel('Count')\n",
" plt.grid(True)\n",
" plt.show()\n",
"\n",
"# Построение гистограмм распределения целевого признака\n",
"plot_distribution(y_train, 'Распределение логарифма цены закрытия в обучающей выборке')\n",
"plot_distribution(y_val, 'Распределение логарифма цены закрытия в валидационной выборке')\n",
"plot_distribution(y_test, 'Распределение логарифма цены закрытия в тестовой выборке')\n",
"\n",
"# Функция для вывода статистических показателей\n",
"def get_statistics(df, name):\n",
" print(f\"Статистические показатели для {name} выборки:\")\n",
" print(f\"Среднее значение: {df.mean():.2f}\")\n",
" print(f\"Стандартное отклонение: {df.std():.2f}\")\n",
" print(f\"Минимальное значение: {df.min():.2f}\")\n",
" print(f\"Максимальное значение: {df.max():.2f}\")\n",
" print(f\"Количество наблюдений: {df.count()}\\n\")\n",
"\n",
"# Вывод статистических показателей для обучающей, валидационной и тестовой выборок\n",
"get_statistics(y_train, \"обучающей\")\n",
"get_statistics(y_val, \"валидационной\")\n",
"get_statistics(y_test, \"тестовой\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Oversampling и undersampling</h3>"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Распределение классов после SMOTE (oversampling):\n",
"Close_category\n",
"0 1157\n",
"1 1157\n",
"2 1157\n",
"3 1157\n",
"4 1157\n",
"Name: count, dtype: int64\n",
"Распределение классов после RandomUnderSampler (undersampling):\n",
"Close_category\n",
"0 1092\n",
"1 1092\n",
"2 1092\n",
"3 1092\n",
"4 1092\n",
"Name: count, dtype: int64\n"
]
}
],
"source": [
"from imblearn.over_sampling import SMOTE\n",
"from imblearn.under_sampling import RandomUnderSampler\n",
"import pandas as pd\n",
"import numpy as np\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"# Проверка наличия столбца 'Date' и создание столбца 'Year'\n",
"if 'Date' in df.columns:\n",
" df['Year'] = pd.to_datetime(df['Date'], errors='coerce').dt.year\n",
" df = df.drop(['Date'], axis=1)\n",
"\n",
"# Логарифмирование целевой переменной\n",
"df['Close_log'] = np.log(df['Close'])\n",
"\n",
"# Создание категорий для целевой переменной\n",
"df['Close_category'] = pd.qcut(df['Close_log'], q=5, labels=[0, 1, 2, 3, 4])\n",
"\n",
"# Выбор признаков и целевой переменной\n",
"X = df.drop(['Close', 'Close_log', 'Close_category'], axis=1)\n",
"y = df['Close_category']\n",
"\n",
"# Разделение данных на обучающую и тестовую выборки\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",
"\n",
"# Применение SMOTE для oversampling\n",
"smote = SMOTE(random_state=42)\n",
"X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)\n",
"\n",
"print(\"Распределение классов после SMOTE (oversampling):\")\n",
"print(pd.Series(y_train_smote).value_counts())\n",
"\n",
"# Применение RandomUnderSampler для undersampling\n",
"undersampler = RandomUnderSampler(random_state=42)\n",
"X_train_under, y_train_under = undersampler.fit_resample(X_train, y_train)\n",
"\n",
"print(\"Распределение классов после RandomUnderSampler (undersampling):\")\n",
"print(pd.Series(y_train_under).value_counts())\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Оценка сбалансированности выборок</h3>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Оценка необходимости аугментации данных"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Данные в выборке обучающей являются категориальными.\n",
"Проверка необходимости аугментации для валидационной выборки:\n",
"Среднее значение: 2.44, Стандартное отклонение: 1.58\n",
"25-й квантиль: 1.20\n",
"50-й квантиль (медиана): 2.53\n",
"75-й квантиль: 4.01\n",
"Выборка валидационной несбалансирована, рекомендуется аугментация.\n",
"\n",
"Данные в выборке тестовой являются категориальными.\n"
]
}
],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"def check_augmentation_need(data, name):\n",
" \"\"\"Проверка необходимости аугментации данных\"\"\"\n",
" # Проверка на наличие числовых значений\n",
" if isinstance(data.dtype, pd.CategoricalDtype):\n",
" print(f\"Данные в выборке {name} являются категориальными.\")\n",
" return\n",
" elif not np.issubdtype(data.dtype, np.number):\n",
" print(f\"Данные в выборке {name} не являются числовыми.\")\n",
" return\n",
"\n",
" # Проверка на наличие пустых значений\n",
" if data.isnull().any():\n",
" print(f\"Выборка {name} содержит пустые значения.\")\n",
" return\n",
"\n",
" quantiles = data.quantile([0.25, 0.5, 0.75])\n",
" mean = data.mean()\n",
" std = data.std()\n",
"\n",
" print(f\"Проверка необходимости аугментации для {name} выборки:\")\n",
" print(f\"Среднее значение: {mean:.2f}, Стандартное отклонение: {std:.2f}\")\n",
" print(f\"25-й квантиль: {quantiles[0.25]:.2f}\")\n",
" print(f\"50-й квантиль (медиана): {quantiles[0.5]:.2f}\")\n",
" print(f\"75-й квантиль: {quantiles[0.75]:.2f}\")\n",
"\n",
" if std > mean * 0.5:\n",
" print(f\"Выборка {name} несбалансирована, рекомендуется аугментация.\\n\")\n",
" else:\n",
" print(f\"Выборка {name} сбалансирована, аугментация не требуется.\\n\")\n",
"\n",
"# Пример использования функции\n",
"# y_train, y_val, y_test должны быть определены заранее\n",
"check_augmentation_need(y_train, \"обучающей\")\n",
"check_augmentation_need(y_val, \"валидационной\")\n",
"check_augmentation_need(y_test, \"тестовой\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Поскольку выборка валидационной несбалансирована и демонстрирует значительный разброс значений, что подтверждается квантилями и стандартным отклонением, применение методов аугментации рекомендуется для улучшения сбалансированности и качества модели."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Распределение 'Close_category' в обучающей выборке:\n",
" Close_category\n",
"2 1157\n",
"4 1134\n",
"1 1126\n",
"3 1116\n",
"0 1092\n",
"Name: count, dtype: int64\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_21436\\2926621768.py:29: FutureWarning: \n",
"\n",
"Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
"\n",
" sns.barplot(x=category_counts.index, y=category_counts.values, palette='viridis')\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAIjCAYAAAAN/63DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABfa0lEQVR4nO3dd3gU5d7G8Xs3pEAgdBKiNOkgRSkSUWmhiSCCIoJSDgJqQBEFzZEuiIAiHY6+ApZgAaTo4YSOKIQuSBcQRGmhh5a6z/sHJ3tYNkAS0tb5fq6LK+zMMzO/mWfLvbPPztqMMUYAAACARdizuwAAAAAgKxGAAQAAYCkEYAAAAFgKARgAAACWQgAGAACApRCAAQAAYCkEYAAAAFgKARgAAACWQgAGAOBv5sKFCzp48KASExOzuxRkIGOMzp07pwMHDmR3KR6PAAwAWeDIkSOy2WyaPXt2dpeCv6GEhASNHTtWNWrUkK+vrwoWLKjy5ctr5cqV2V2aR9i1a5cWLlzovL19+3b9+9//zr6CbnDp0iUNGjRIFStWlI+PjwoXLqwKFSpo//792V2aRyMA52CzZ8+WzWZz/vPz81OFChXUp08fnTp1KrvLAyxlzZo1stlsOnLkSIrz2rVrp6CgIPn4+KhYsWJq3bq1vvvuu6wvNIeZNm3a3zL0d+vWTQ0bNszuMiRJcXFxCg0N1eDBg9WwYUPNnTtXy5cv16pVqxQSEpLd5XmES5cuqXfv3tqwYYMOHDig1157TTt37szusnT27FmFhIRo0qRJevrpp7Vo0SItX75ca9asUenSpbO7PI+WK7sLwJ2NGDFCZcqUUWxsrH7++WdNnz5dS5Ys0a5du5QnT57sLg+wtKFDh2rEiBEqX768evfurVKlSuns2bNasmSJ2rdvr4iICHXq1Cm7y8w206ZNU5EiRdStW7fsLuVva8yYMdq4caOWLl2aY0K5pwkJCXH+k6QKFSqoZ8+e2VyVNGDAAJ04cUJRUVGqWrVqdpfzt0IA9gAtW7ZU7dq1JUkvvviiChcurPHjx2vRokV67rnnsrk6wLrmzZunESNG6Omnn9acOXPk7e3tnDdgwAAtXbpUCQkJ2VghUuvKlSvy9/fP7jLSLDExURMmTNAbb7xB+L1LCxcu1J49e3Tt2jVVq1ZNPj4+2VpPdHS0PvvsM82YMYPwmwkYAuGBGjduLEk6fPiwJOncuXN68803Va1aNeXNm1cBAQFq2bKlduzY4bZsbGyshg0bpgoVKsjPz0/FixdXu3btdOjQIUn/G6d4q383PsEmfyT8zTff6J///KeCgoLk7++vNm3a6M8//3Tb9saNG9WiRQvlz59fefLkUYMGDbRu3boU97Fhw4Ypbn/YsGFubb/88kvVqlVLuXPnVqFChdSxY8cUt3+7fbuRw+HQhAkTVLVqVfn5+SkwMFC9e/fW+fPnXdqVLl1aTzzxhNt2+vTp47bOlGofN26c2zGVrn+cOXToUJUrV06+vr4qUaKEBg4cqLi4uBSP1Y1uddyS/9348X1y/cuWLVPNmjXl5+enKlWquH1snzwU58ZlHQ6Hqlev7jamddiwYapSpYrzflivXj2XcXXJNd5///1utX/wwQdu21m0aJFatWql4OBg+fr6qmzZsnr33XeVlJTkts6bj+OoUaNkt9s1Z84c57SffvpJzzzzjEqWLOk8tq+//rquXbt2+wN7C4MHD1ahQoU0c+ZMl/CbrHnz5ineR260atUqPfroo/L391eBAgX05JNPau/evS5tLl26pH79+ql06dLy9fVVsWLF1LRpU23bts2lXVoeY7fjcDg0ceJEVatWTX5+fipatKhatGihLVu2ONvMmjVLjRs3VrFixeTr66sqVapo+vTpLuspXbq0du/erR9//DHF55ALFy6oX79+KlGihHx9fVWuXDmNGTNGDofDZT1nz57VCy+8oICAABUoUEBdu3bVjh07UhxTnZrjOWzYMNlsNu3Zs0edOnVSwYIF9cgjj2jWrFmy2Wz65Zdf3I7Je++9Jy8vLx07dixNx7J06dLOfbfb7QoKCtKzzz6ro0ePpmr5adOmqWrVqvL19VVwcLDCwsJ04cIF5/z9+/fr/Pnzypcvnxo0aKA8efIof/78euKJJ7Rr1y5nu9WrV8tms2nBggVu25gzZ45sNpuioqKcNd98xj75+X7NmjXOaal9PCUf7xtFREQ4n3cKFy6s5557zu2YdOvWTXnz5nWZNm/ePLc6JClv3rwpfsqQmteHG58/qlSpolq1ajnvXzfXnZKbn3eLFCmiVq1auRx/6frrQJ8+fW65npufazdv3iyHw6H4+HjVrl37tsdKStt9f9++ferQoYMCAgJUuHBhvfbaa4qNjXWr98bXrcTERD3++OMqVKiQ9uzZ49I2ta/DOQlngD1QclgtXLiwJOn333/XwoUL9cwzz6hMmTI6deqU/vWvf6lBgwbas2ePgoODJUlJSUl64okntHLlSnXs2FGvvfaaLl26pOXLl2vXrl0qW7ascxvPPfecHn/8cZfthoeHp1jPqFGjZLPZ9NZbbyk6OloTJkxQaGiotm/frty5c0u6/sBs2bKlatWqpaFDh8putztfQH/66SfVrVvXbb333nuvRo8eLUm6fPmyXn755RS3PXjwYHXo0EEvvviiTp8+rcmTJ+uxxx7TL7/8ogIFCrgt06tXLz366KOSpO+++87tBaF3796aPXu2unfvrldffVWHDx/WlClT9Msvv2jdunUpBp20unDhgnPfbuRwONSmTRv9/PPP6tWrlypXrqydO3fqo48+0m+//eYWJlNy43FLtmTJEn311VdubQ8cOKBnn31WL730krp27apZs2bpmWeeUWRkpJo2bXrLbXzxxRcpjo+7cuWKnnrqKZUuXVrXrl3T7Nmz1b59e0VFRaXYx3cye/Zs5c2bV/3791fevHm1atUqDRkyRDExMRo3btwtl5s1a5YGDRqkDz/80GX4wdy5c3X16lW9/PLLKly4sDZt2qTJkyfrr7/+0ty5c9NU24EDB7Rv3z794x//UL58+dK8b5K0YsUKtWzZUvfdd5+GDRuma9euafLkyapfv762bdvmHOP30ksvad68eerTp4+qVKmis2fP6ueff9bevXv14IMPSkrfY+xWevToodmzZ6tly5Z68cUXlZiYqJ9++kkbNmxwfho1ffp0Va1aVW3atFGuXLn0/fff65VXXpHD4VBYWJgkacKECerbt6/y5s2rd955R5IUGBgoSbp69aoaNGigY8eOqXfv3ipZsqTWr1+v8PBwnThxQhMmTJB0/THRunVrbdq0SS+//LIqVaqkRYsWqWvXruk+nsmeeeYZlS9fXu+9956MMXr66acVFhamiIgIPfDAAy5tIyIi1LBhQ91zzz2pPo7JHn30UfXq1UsOh0O7du3ShAkTdPz4cf3000+3XW7YsGEaPny4QkND9fLLL2v//v2aPn26Nm/e7HwuOnv2rKTrz8/ly5fX8OHDFRsbq6lTp6p+/fravHmzKlSooIYNG6pEiRKKiIjQU0895bZvZcuWTfN44fQ+nubMmaPnn39eNWrU0OjRo3X27FlNmjRJP//8s3755RcVKVIkTXXcSnpeH5K99dZbadpWpUqV9M4778gYo0OHDmn8+PF6/PHHU/1GJyXJfdunTx/VqlVL77//vk6fPp3isUrrfb9Dhw4qXbq0Ro8erQ0bNmjSpEk6f/68Pv/881vW8+KLL2rNmjVavny5qlSp4px+N8c5WxnkWLNmzTKSzIoVK8zp06fNn3/+ab7++mtTuHBhkzt3bvPXX38ZY4yJjY01SUlJLssePnzY+Pr6mhEjRjinzZw500gy48ePd9uWw+FwLifJjBs3zq1N1apVTYMGDZy3V69ebSSZe+65x8TExDinf/vtt0aSmThxonPd5cuXN82bN3duxxhjrl69asqUKWOaNm3qtq2HH37Y3H///c7bp0+fNpLM0KFDndOOHDlivLy8zKhRo1yW3blzp8mVK5fb9AMHDhhJ5rPPPnNOGzp0qLnxYfDTTz8ZSSYiIsJl2cjISLfppUq
"text/plain": [
"<Figure size 800x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Распределение 'Close_category' в валидационной выборке:\n",
" Close_category\n",
"0 263\n",
"1 242\n",
"3 238\n",
"4 238\n",
"2 224\n",
"Name: count, dtype: int64\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_21436\\2926621768.py:29: FutureWarning: \n",
"\n",
"Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
"\n",
" sns.barplot(x=category_counts.index, y=category_counts.values, palette='viridis')\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArcAAAIjCAYAAAAZajMiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYf0lEQVR4nO3dd3wUdf7H8fduSKOEniaQIC2AAtIEUZEaEFEOrHASOATUgIcoelGpoqgogjTPOwXOA1F6OQ1VQKoIoiBFQBCkF5NAgGST/f7+0OyPZRNIQsImw+v5eOwDdua7M5+Z7+7OO7PfnbUZY4wAAAAAC7B7uwAAAAAgrxBuAQAAYBmEWwAAAFgG4RYAAACWQbgFAACAZRBuAQAAYBmEWwAAAFgG4RYAAACWQbgFAACwEKfTqdOnT+uXX37xdileQbgFcNM6ePCgbDabpk6d6u1SABQCa9eu1apVq1z3V61apXXr1nmvoMscP35cAwYMUEREhPz8/FS+fHnVqlVLSUlJ3i7thiPc5tDUqVNls9lct4CAAFWvXl39+vXTiRMnvF0ecFNZtWqVbDabDh48mOm8zp07KzQ0VH5+fgoODlbHjh01d+7cG19oATNp0iRLBvoePXrovvvu83YZsLDDhw/r2Wef1fbt27V9+3Y9++yzOnz4sLfL0r59+9SoUSPNnDlTffv21eLFi7Vs2TKtWLFCxYoV83Z5N1wRbxdQWI0YMUKVK1fWpUuXtHbtWk2ePFlffvmlduzYoaJFi3q7POCmNnToUI0YMULVqlVT3759FRERoTNnzujLL79Uly5dNH36dHXt2tXbZXrNpEmTVK5cOfXo0cPbpQCFSufOnTV27FjVqVNHktS0aVN17tzZy1VJffv2lZ+fnzZu3KhbbrnF2+V4HeE2l9q3b6+GDRtKkp566imVLVtWY8aM0YIFC/TEE094uTrg5jV79myNGDFCDz/8sGbMmCFfX1/XvEGDBmnJkiVyOBxerBDZlZycfFOedULB5e/vr/Xr12vHjh2SpNtuu00+Pj5erWnLli1auXKlli5dSrD9E8MS8kjLli0lSQcOHJAknT17Vi+++KJuv/12FS9eXEFBQWrfvr1++OEHj8deunRJw4YNU/Xq1RUQEKCwsDB17txZ+/fvl/T/4wKzul3+MVzGx7Sff/65XnnlFYWGhqpYsWJ68MEHM/3oZNOmTWrXrp1KliypokWLqnnz5lmOH7rvvvsyXf+wYcM82v73v/9VgwYNFBgYqDJlyujxxx/PdP1X27bLOZ1OjR07VrVr11ZAQIBCQkLUt29f/f77727tIiMj9cADD3isp1+/fh7LzKz20aNHe+xTSUpJSdHQoUNVtWpV+fv7q2LFinrppZeUkpKS6b66XFb7LeN2+UfqGfUvXbpU9erVU0BAgGrVquXxUXrG8JjLH+t0OlWnTh2PMaTDhg1TrVq1XM/DJk2aaP78+R413nbbbR61v/vuux7rWbBggTp06KDw8HD5+/urSpUqev3115Wenu6xzCv34xtvvCG73a4ZM2a4pn3zzTd65JFHVKlSJde+ff7553Xx4sWr79gsDB48WGXKlNEnn3ziFmwzREdHZ/ocudzKlSt1zz33qFixYipVqpQeeugh7dq1y63NuXPnNGDAAEVGRsrf31/BwcFq06aNtm7d6tYuJ6+xq3E6nRo3bpxuv/12BQQEqHz58mrXrp2+++47V5spU6aoZcuWCg4Olr+/v2rVqqXJkye7LScyMlI//fSTVq9enel7SEJCggYMGKCKFSvK399fVatW1dtvvy2n0+m2nDNnzujJJ59UUFCQSpUqpZiYGP3www+ZjmHOzv4cNmyYbDabdu7cqa5du6p06dK6++67NWXKFNlsNn3//fce++TNN9+Uj4+Pjhw5kqN9GRkZ6dp2u92u0NBQPfbYYzp06FCePfbdd9/VXXfdpbJlyyowMFANGjTQ7NmzM13mlcPdMuuXnLzmM6v18tuVbbNbq81mU79+/TymP/DAA4qMjHSb1qNHD49phw8fVmBgYKZDiSZNmqTatWvL399f4eHhio2NVUJCglubnLxP5eRYkJaWptdff11VqlSRv7+/IiMj9corr3i8v0dGRqpHjx7y8fFR3bp1VbduXc2dO1c2m81jWzOTnedOxjHx3XffzXI5Ga+VDBs3blRAQID279/v2oehoaHq27evzp496/H4WbNmuY7P5cqV01//+leP11CPHj1UvHhx/fLLL4qOjlaxYsUUHh6uESNGyBjjUe/lz6lz586pQYMGqly5so4dO+aant3jeF7gzG0eyQiiZcuWlST98ssvmj9/vh555BFVrlxZJ06c0D//+U81b95cO3fuVHh4uCQpPT1dDzzwgFasWKHHH39cf//733Xu3DktW7ZMO3bsUJUqVVzreOKJJ3T//fe7rTcuLi7Tet544w3ZbDa9/PLLOnnypMaOHavWrVtr27ZtCgwMlPTHAad9+/Zq0KCBhg4dKrvd7jo4fvPNN2rcuLHHcitUqKBRo0ZJks6fP69nnnkm03UPHjxYjz76qJ566imdOnVK48eP17333qvvv/9epUqV8nhMnz59dM8990iS5s6dq3nz5rnN79u3r6ZOnaqePXvqueee04EDBzRhwgR9//33WrduXaYhJqcSEhJc23Y5p9OpBx98UGvXrlWfPn1Us2ZNbd++Xe+//75+/vlnj6CYmcv3W4Yvv/xSn332mUfbvXv36rHHHtPTTz+tmJgYTZkyRY888oji4+PVpk2bLNfx6aefavv27R7Tk5OT9Ze//EWRkZG6ePGipk6dqi5dumjDhg2Z9vG1TJ06VcWLF9fAgQNVvHhxrVy5UkOGDFFSUpJGjx6d5eOmTJmi1157Te+9957bkIBZs2bpwoULeuaZZ1S2bFl9++23Gj9+vH777TfNmjUrR7Xt3btXu3fv1t/+9jeVKFEix9smScuXL1f79u116623atiwYbp48aLGjx+vZs2aaevWra6D2NNPP63Zs2erX79+qlWrls6cOaO1a9dq165dql+/vqTcvcay0qtXL02dOlXt27fXU089pbS0NH3zzTfauHGj61OkyZMnq3bt2nrwwQdVpEgRLVq0SM8++6ycTqdiY2MlSWPHjlX//v1VvHhxvfrqq5KkkJAQSdKFCxfUvHlzHTlyRH379lWlSpW0fv16xcXF6dixYxo7dqykP14THTt21LfffqtnnnlGUVFRWrBggWJiYnK9PzM88sgjqlatmt58800ZY/Twww8rNjZW06dP1x133OHWdvr06brvvvtydbbqnnvuUZ8+feR0OrVjxw6NHTtWR48e1TfffJMnjx03bpwefPBBdevWTampqZo5c6YeeeQRLV68WB06dMh0ue+//77KlSsn6Y/30WvJ6jV/uXr16umFF16Q9MfJlyFDhni0yU2tuTFkyBBdunTJY/qwYcM0fPhwtW7dWs8884z27NmjyZMna/PmzXn2/n41Tz31lKZNm6aHH35YL7zwgjZt2qRRo0Zp165dHseiy6WlpbleQ9l1Pc+7rJw5c0aXLl3SM888o5YtW+rpp5/W/v37NXHiRG3atEmbNm2Sv7+/JLmOo40aNdKoUaN04sQJjRs3TuvWrfM4Pqenp6tdu3Zq0qSJ3nnnHcXHx2vo0KFKS0vTiBEjMq3F4XCoS5cuOnTokNatW6ewsDDXvBtxHHcxyJEpU6YYSWb58uXm1KlT5vDhw2bmzJmmbNmyJjAw0Pz222/GGGMuXbpk0tPT3R574MAB4+/vb0aMGOGa9sknnxhJZsyYMR7rcjqdrsdJMqNHj/ZoU7t2bdO8eXPX/a+//tpIMrfccotJSkpyTf/iiy+MJDNu3DjXsqtVq2aio6Nd6zHGmAsXLpjKlSubNm3aeKzrrrvuMrfddpvr/qlTp4wkM3ToUNe0gwcPGh8fH/PGG2+4PXb79u2mSJEiHtP37t1rJJlp06a5pg0dOtRc/tT85ptvjCQzffp0t8fGx8d7TI+IiDAdOnTwqD02NtZc+XS/svaXXnrJBAcHmwYNGrjt008//dTY7XbzzTffuD3
"text/plain": [
"<Figure size 800x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_21436\\2926621768.py:29: FutureWarning: \n",
"\n",
"Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
"\n",
" sns.barplot(x=category_counts.index, y=category_counts.values, palette='viridis')\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Распределение 'Close_category' в тестовой выборке:\n",
" Close_category\n",
"0 253\n",
"3 252\n",
"1 241\n",
"4 235\n",
"2 225\n",
"Name: count, dtype: int64\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArcAAAIjCAYAAAAZajMiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXOUlEQVR4nO3dd3wUdf7H8fduSKOEHpIIhF6lKEUDKr0EBBE8RDwpIiAGPEVBsdBEOUQFpehxPyWoYKGLh3QEpQlBFKQICKJUCSaBBJIl+/394WWPZQMkIWGzw+v5eOwDduY7M5+Z7+7sO7MzszZjjBEAAABgAXZvFwAAAADkFsItAAAALINwCwAAAMsg3AIAAMAyCLcAAACwDMItAAAALINwCwAAAMsg3AIAAMAyCLcAAAB55OLFizp16pSOHDni7VJuGoRbALiBDh8+LJvNptjYWG+XAvi0L7/8Ujt27HA9X7RokX766SfvFXSJ/fv3q3///goPD1dAQIDKlCmjqKgo8aOwNwbh1gfExsbKZrO5HkFBQapWrZoGDx6skydPers84Kby9ddfy2az6fDhw5mO69q1q8LCwhQQEKDQ0FB16tRJCxYsuPGF5jPTp0+3ZKDv06ePmjdv7u0ybko7d+7UP/7xD+3fv1+bN2/W448/rrNnz3q7LG3evFmNGzfWmjVr9Pzzz2v58uVauXKlFi1aJJvN5u3ybgoFvF0Asm7s2LGqWLGiLly4oG+//Vbvvvuuli5dql27dqlgwYLeLg+4qY0aNUpjx45V1apVNXDgQEVGRio+Pl5Lly5Vt27dNHv2bPXs2dPbZXrN9OnTVapUKfXp08fbpcAiHnvsMX3wwQeqVq2aJKlr16668847vVpTWlqa+vbtq2rVqmnFihUqWrSoV+u5WRFufUh0dLQaNmwo6a83dcmSJfXWW29p8eLFeuihh7xcHXDzmjdvnsaOHasHHnhAc+bMkb+/v2vcsGHDtHz5cjkcDi9WiKxKTk5WoUKFvF0GsqB06dLatWuX6wBPzZo1vV2SlixZon379mnv3r0EWy/itAQf1rJlS0nSoUOHJElnzpzRs88+qzp16qhw4cIKCQlRdHS0fvjhB49pL1y4oNGjR6tatWoKCgpSeHi4unbtqoMHD0r633mBV3pc+jVcxte0n332mV544QWFhYWpUKFC6ty5s3777TePZW/ZskXt27dX0aJFVbBgQTVr1kwbNmzIdB2bN2+e6fJHjx7t0fbjjz9WgwYNFBwcrBIlSqhHjx6ZLv9q63Ypp9OpyZMnq3bt2goKClKZMmU0cOBA/fnnn27tKlSooHvvvddjOYMHD/aYZ2a1T5w40WObSlJqaqpGjRqlKlWqKDAwUOXKldPw4cOVmpqa6ba61JW2W8bj0q/UM+pfsWKF6tevr6CgINWqVcvjq/SM02MundbpdKpu3boe55COHj1atWrVcr0O77zzTi1atMijxltvvdWj9jfeeMNjOYsXL1bHjh0VERGhwMBAVa5cWa+88orS09M95nn5dnz11Vdlt9s1Z84c17BvvvlGf/vb31S+fHnXtn366ad1/vz5q2/YK3j55ZdVokQJffDBB27BNkO7du0yfY1cas2aNbr77rtVqFAhFStWTPfdd5/27Nnj1ubs2bN66qmnVKFCBQUGBio0NFRt2rTR9u3b3dpl5z12NU6nU2+//bbq1KmjoKAglS5dWu3bt9e2bdtcbWbOnKmWLVsqNDRUgYGBqlWrlt599123+VSoUEE//fST1q1bl+k+JCEhQU899ZTKlSunwMBAValSRRMmTJDT6XSbT3x8vB555BGFhISoWLFi6t27t3744YdMz2HOyvYcPXq0bDabdu/erZ49e6p48eK66667NHPmTNlsNn3//fce2+S1116Tn5+fjh49mq1tWaFCBde62+12hYWF6cEHH7zmRUaXTpfZo0KFCq62Wd1nSdJXX32lZs2aqUiRIgoJCVGjRo1c75Fr7T8u3a9dvHhRr7zyiipXrqzAwEBVqFBBL7zwgsd+Kqvrn5ycrGeeecb1WqhevbreeOMNj3NVM/algYGBatCggWrWrHnFfWlmLl0XPz8/3XLLLRowYIASEhJcbTI+2+bNm3fF+fTp08etDzZv3qyKFStq/vz5qly5sgICAlS+fHkNHz480/3L9OnTVbt2bQUGBioiIkIxMTFuNUj/21fGxcWpSZMmCg4OVsWKFfXee++5tcuo9+uvv3YNO3bsmCpUqKCGDRvq3LlzruHX8/niCzhy68MygmjJkiUlSb/88osWLVqkv/3tb6pYsaJOnjypf/3rX2rWrJl2796tiIgISVJ6erruvfderV69Wj169NA//vEPnT17VitXrtSuXbtUuXJl1zIeeughdejQwW25I0aMyLSeV199VTabTc8995xOnTqlyZMnq3Xr1tqxY4eCg4Ml/fWBEx0drQYNGmjUqFGy2+2uD8dvvvlGjRs39phv2bJlNX78eEnSuXPnNGjQoEyX/fLLL6t79+567LHH9Mcff2jKlCm655579P3336tYsWIe0wwYMEB33323JGnBggVauHCh2/iBAwcqNjZWffv21ZNPPqlDhw5p6tSp+v7777Vhw4ZMQ0x2JSQkuNbtUk6nU507d9a3336rAQMGqGbNmtq5c6cmTZqkn3/+2SMoZubS7ZZh6dKl+uSTTzza7t+/Xw8++KAef/xx9e7dWzNnztTf/vY3LVu2TG3atLniMj766CPt3LnTY3hycrLuv/9+VahQQefPn1dsbKy6deumTZs2ZdrH1xIbG6vChQtr6NChKly4sNasWaORI0cqKSlJEydOvOJ0M2fO1EsvvaQ333zT7ZSAuXPnKiUlRYMGDVLJkiX13XffacqUKfr99981d+7cbNW2f/9+7d27V48++qiKFCmS7XWTpFWrVik6OlqVKlXS6NGjdf78eU2ZMkVNmzbV9u3bXR+ejz/+uObNm6fBgwerVq1aio+P17fffqs9e/bo9ttvl5Sz99iV9OvXT7GxsYqOjtZjjz2mixcv6ptvvtHmzZtd3yK9++67ql27tjp37qwCBQpoyZIleuKJJ+R0OhUTEyNJmjx5soYMGaLChQvrxRdflCSVKVNGkpSSkqJmzZrp6NGjGjhwoMqXL6+NGzdqxIgROn78uCZPnizpr/dEp06d9N1332nQoEGqUaOGFi9erN69e+d4e2b429/+pqpVq+q1116TMUYPPPCAYmJiNHv2bN12221ubWfPnq3mzZvrlltuyfJ2zHD33XdrwIABcjqd2rVrlyZPnqxjx47pm2++ueI0kydPdoWSPXv26LXXXtMLL7zgOkpZuHBhV9us7rNiY2P16KOPqnbt2hoxYoSKFSum77//XsuWLVPPnj314osv6rHHHpMknT59Wk8//bTb/vJSjz32mGbNmqUHHnhAzzzzjLZs2aLx48drz549HvvUa62/MUadO3fW2rVr1a9fP9WvX1/Lly/XsGHDdPToUU2aNOmK2+lK+9Kruf/++9W1a1ddvHhRmzZt0owZM3T+/Hl99NFH2ZrPpeLj4/XLL7/ohRdeUNeuXfXMM89o27Ztmjhxonbt2qX//Oc/rj8ORo8erTFjxqh169YaNGiQ9u3bp3fffVdbt271+Iz5888/1aFDB3Xv3l0PPfSQPv/8cw0aNEgBAQF69NFHM60lMTFR0dHR8vf319KlS12vldz4fMn3DPK9mTNnGklm1apV5o8//jC//fab+fTTT03JkiVNcHCw+f33340xxly4cMGkp6e7TXvo0CETGBhoxo4d6xr2wQcfGEnmrbfe8liW0+l0TSfJTJw40aNN7dq1TbNmzVzP165daySZW265xSQlJbmGf/7550aSefvtt13zrlq1qmnXrp1rOcYYk5KSYipWrGjatGnjsawmTZqYW2+91fX8jz/+MJLMqFGjXMMOHz5s/Pz8zKuvvuo27c6dO02BAgU8hu/fv99IMrNmzXINGzVqlLn07fDNN98YSWb27Nlu0y5btsxjeGRkpOnYsaNH7TExMebyt9jltQ8fPtyEhoaaBg0auG3Tjz76yNj
"text/plain": [
"<Figure size 800x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Проверка необходимости аугментации для признака 'Close_category' в обучающей выборке:\n",
"Минимальное количество наблюдений в классе: 1092\n",
"Максимальное количество наблюдений в классе: 1157\n",
"Выборка 'обучающей' сбалансирована, аугментация не требуется.\n",
"\n",
"Проверка необходимости аугментации для признака 'Close_category' в валидационной выборке:\n",
"Минимальное количество наблюдений в классе: 224\n",
"Максимальное количество наблюдений в классе: 263\n",
"Выборка 'валидационной' сбалансирована, аугментация не требуется.\n",
"\n",
"Проверка необходимости аугментации для признака 'Close_category' в тестовой выборке:\n",
"Минимальное количество наблюдений в классе: 225\n",
"Максимальное количество наблюдений в классе: 253\n",
"Выборка 'тестовой' сбалансирована, аугментация не требуется.\n",
"\n",
"Распределение классов после SMOTE (oversampling):\n",
"Close_category\n",
"0 1157\n",
"1 1157\n",
"2 1157\n",
"3 1157\n",
"4 1157\n",
"Name: count, dtype: int64\n",
"Распределение классов после RandomUnderSampler (undersampling):\n",
"Close_category\n",
"0 1092\n",
"1 1092\n",
"2 1092\n",
"3 1092\n",
"4 1092\n",
"Name: count, dtype: int64\n"
]
}
],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from imblearn.over_sampling import SMOTE\n",
"from imblearn.under_sampling import RandomUnderSampler\n",
"\n",
"# Логарифмирование целевой переменной\n",
"df['Close_log'] = np.log(df['Close'])\n",
"\n",
"# Создание категорий для целевой переменной\n",
"df['Close_category'] = pd.qcut(df['Close_log'], q=5, labels=[0, 1, 2, 3, 4])\n",
"\n",
"# Выбор признаков и целевой переменной\n",
"X = df.drop(['Close', 'Close_log', 'Close_category'], axis=1)\n",
"y = df['Close_category']\n",
"\n",
"# Разделение данных на обучающую, валидационную и тестовую выборки\n",
"X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42)\n",
"X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)\n",
"\n",
"def analyze_close_category_distribution(data, name):\n",
" \"\"\"Проверка и визуализация распределения признака 'Close_category'\"\"\"\n",
" category_counts = data.value_counts()\n",
" print(f\"Распределение 'Close_category' в {name} выборке:\\n\", category_counts)\n",
"\n",
" plt.figure(figsize=(8, 6))\n",
" sns.barplot(x=category_counts.index, y=category_counts.values, palette='viridis')\n",
" plt.title(f\"Распределение признака 'Close_category' в {name} выборке\")\n",
" plt.xlabel('Close Category')\n",
" plt.ylabel('Count')\n",
" plt.grid(True)\n",
" plt.show()\n",
"\n",
"analyze_close_category_distribution(y_train, 'обучающей')\n",
"analyze_close_category_distribution(y_val, 'валидационной')\n",
"analyze_close_category_distribution(y_test, 'тестовой')\n",
"\n",
"def check_close_category_augmentation(data, name):\n",
" print(f\"Проверка необходимости аугментации для признака 'Close_category' в {name} выборке:\")\n",
" min_count = data.value_counts().min()\n",
" max_count = data.value_counts().max()\n",
" print(f\"Минимальное количество наблюдений в классе: {min_count}\")\n",
" print(f\"Максимальное количество наблюдений в классе: {max_count}\")\n",
"\n",
" if max_count > min_count * 1.5:\n",
" print(f\"Выборка '{name}' несбалансирована, рекомендуется аугментация.\\n\")\n",
" else:\n",
" print(f\"Выборка '{name}' сбалансирована, аугментация не требуется.\\n\")\n",
"\n",
"check_close_category_augmentation(y_train, 'обучающей')\n",
"check_close_category_augmentation(y_val, 'валидационной')\n",
"check_close_category_augmentation(y_test, 'тестовой')\n",
"\n",
"# Применение SMOTE для oversampling\n",
"smote = SMOTE(random_state=42)\n",
"X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)\n",
"\n",
"print(\"Распределение классов после SMOTE (oversampling):\")\n",
"print(pd.Series(y_train_smote).value_counts())\n",
"\n",
"# Применение RandomUnderSampler для undersampling\n",
"undersampler = RandomUnderSampler(random_state=42)\n",
"X_train_under, y_train_under = undersampler.fit_resample(X_train, y_train)\n",
"\n",
"print(\"Распределение классов после RandomUnderSampler (undersampling):\")\n",
"print(pd.Series(y_train_under).value_counts())\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"В
]
2024-10-20 01:50:42 +04:00
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h2>Данные по инсультам</h2>"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" id gender age hypertension heart_disease ever_married \\\n",
"0 9046 Male 67.0 0 1 Yes \n",
"1 51676 Female 61.0 0 0 Yes \n",
"2 31112 Male 80.0 0 1 Yes \n",
"3 60182 Female 49.0 0 0 Yes \n",
"4 1665 Female 79.0 1 0 Yes \n",
"\n",
" work_type Residence_type avg_glucose_level bmi smoking_status \\\n",
"0 Private Urban 228.69 36.6 formerly smoked \n",
"1 Self-employed Rural 202.21 NaN never smoked \n",
"2 Private Rural 105.92 32.5 never smoked \n",
"3 Private Urban 171.23 34.4 smokes \n",
"4 Self-employed Rural 174.12 24.0 never smoked \n",
"\n",
" stroke \n",
"0 1 \n",
"1 1 \n",
"2 1 \n",
"3 1 \n",
"4 1 \n",
"\n",
"id\n",
"gender\n",
"age\n",
"hypertension\n",
"heart_disease\n",
"ever_married\n",
"work_type\n",
"Residence_type\n",
"avg_glucose_level\n",
"bmi\n",
"smoking_status\n",
"stroke\n"
]
}
],
"source": [
"import numpy as np\n",
"import pandas as pd \n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"df2 = pd.read_csv(\"C:/Users/TIGR228/Desktop/МИИ/Lab1/AIM-PIbd-31-Afanasev-S-S/static/csv/healthcare.csv\")\n",
"\n",
"print(df2.head(), \"\\n\")\n",
"print(*list(df2.columns), sep='\\n')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Объектом наблюдения являются пациенты и информация о
"\n",
"Атрибуты объекта:\n",
"\n",
"- id (Идентификатор)\n",
"- gender (Пол)\n",
"- age (Возраст)\n",
"- hypertension (Гипертония)\n",
"- heart_disease (Сердечное заболевание)\n",
"- ever_married (Был ли когда-либо в браке)\n",
"- work_type (Тип работы)\n",
"- Residence_type (Тип проживания)\n",
"- avg_glucose_level (Средний уровень глюкозы)\n",
"- bmi (Индекс массы тела)\n",
"- smoking_status (Статус курения)\n",
"- stroke (Инсульт)\n",
"\n",
"Связь между объектами:\n",
"Имеется связь между атрибутами, например между индексом массы тела (bmi) и риском инсульта (stroke), а
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\2664058835.py:4: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" df_clean['bmi_category'] = pd.cut(df_clean['bmi'], bins=range(0, 100, 10))\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAAJGCAYAAABRFrQ+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACwL0lEQVR4nOzdd1xT1/8/8FcS9haVVQVFxYGiVqvFQW3daK2jWi3OuirapXZYrYjVr912gZYOJ2q1tnVUrbOiLdrWhXuiVFkiMgRZyfn9wS/3QwhoAglJ5PV8PHiYe+87N+8ck5u8c889RyaEECAiIiIiIiKdyU2dABERERERkaVhIUVERERERKQnFlJERERERER6YiFFRERERESkJxZSREREREREemIhRUREREREpCcWUkRERERERHpiIUVERERERKQnFlJERERERER6YiFFRERERESkJxZSFmzlypWQyWTSn52dHQICAjBjxgykpaWZOj0iIiIiokeWlakToOpbuHAhGjdujIKCAhw+fBjLli3Djh07cObMGTg4OJg6PSIiIiKiRw4LqUdA//790bFjRwDApEmTULduXXz22WfYsmULRo0aZeLsiIiIiIgePeza9wh65plnAACJiYkAgMzMTMyePRtt2rSBk5MTXFxc0L9/f5w6dUrrvgUFBViwYAECAgJgZ2cHb29vDB06FFevXgUAXL9+XaM7Yfm/Hj16SPv6448/IJPJ8OOPP+Ldd9+Fl5cXHB0dMWjQIPz3339aj3306FH069cPrq6ucHBwwFNPPYU///yzwufYo0ePCh9/wYIFWrFr165Fhw4dYG9vD3d3d4wcObLCx3/QcytLpVLh888/R2BgIOzs7ODp6YmpU6fi7t27GnGNGjXCwIEDtR5nxowZWvusKPePP/5Yq00BoLCwEBEREWjatClsbW3RsGFDvPXWWygsLKywrco7evQoQkNDUadOHTg6OiIoKAhffPGFtH38+PFwcnLCtWvX0LdvXzg6OsLHxwcLFy6EEKJKbQE8uH2vX78uxam7rP77778a98/IyKiwnW7duoWXXnoJnp6esLW1RWBgIH744QeNGPVrsaK/w4cPS3EnTpxA//794eLiAicnJ/Ts2RNHjhx5YHsKIdCoUSM899xzWtsKCgrg6uqKqVOnauSh63ti06ZN0mu3Xr16GD16NG7duqURM378eI3nU6dOHfTo0QOHDh3SiNPn9Xjv3j3MmjUL/v7+sLa21th/RkbGA9ujsvdmr169NOLKd01+0DHkjz/+kNb9888/6N27N5ydneHo6Fjhc1Xvu+zr6uzZs6hTpw4GDhyIkpISaf21a9cwfPhwuLu7w8HBAU8++SR+++03jf0dPnwY3bp1Q7169WBnZwd/f3+8/fbbKCgo0HrM8q/b8m1T/v1ckQcdYxs1aqQRm5eXh1mzZqFhw4awtbVF8+bN8cknn2i9V3XNZ/HixZDL5Vi3bp207tChQxg+fDh8fX2lY84bb7yB+/fvSzHlX4cPe5/v3LkT3bt3h6OjI5ydnTFgwACcPXtWI5fK9tm0aVONuOjoaAQGBsLW1hY+Pj6YPn06srKytJ5r69atcezYMXTp0gX29vZo3Lgxli9frhFX0WsOAAYMGFDh8efAgQPo3r076tSpo5HjjBkzpJgFCxZAJpPBw8MDxcXFGvdfv369zu+t8ePHa/3///fff7C3t9dqX33e72Wf98M+/wD9jrll2zE5ORmNGjVCx44dce/ePWn9w753AMAnn3yCLl26oG7durC3t0eHDh3w008/PbC91NTHpMGDB2ttmzp1KmQyGVq3bq2xXp/HW7t2LTp16gQHBwfUqVMHISEh2L17t0bMzp078dRTT8HZ2RkuLi544oknNN5j6tdnZdSfnytXrqw0prJjatm/sve/cOECnn/+ebi7u8POzg4dO3bE1q1bK9x3Zcf1svvT5TjxKOIZqUeQ+uBTt25dAKVfFH799VcMHz4cjRs3RlpaGr755hs89dRTOHfuHHx8fAAASqUSAwcOxL59+zBy5Ei89tpryM3NxZ49e3DmzBk0adJEeoxRo0YhNDRU43HnzJlTYT6LFy+GTCbD22+/jfT0dHz++efo1asXTp48CXt7ewDA/v370b9/f3To0AERERGQy+VYsWIFnnnmGRw6dAidOnXS2m+DBg2wZMkSAKVf/KZNm1bhY7/33nsYMWIEJk2ahNu3b+Orr75CSEgITpw4ATc3N637TJkyBd27dwcA/Pzzz/jll180tk+dOhUrV67EhAkT8OqrryIxMRFff/01Tpw4gT///BPW1tYVtoM+srKypOdWlkqlwqBBg3D48GFMmTIFLVu2xOnTp7F06VJcunQJv/766wP3u2fPHgwcOBDe3t547bXX4OXlhfPnz2P79u147bXXpDilUol+/frhySefxEcffYRdu3YhIiICJSUlWLhwYbXaouxrZ8eOHVi/fn0VWwlIS0vDk08+KX1xqV+/Pnbu3ImJEyciJycHr7/+ukb8q6++iieeeEJjXfPmzQGUftHu3r07XFxc8NZbb8Ha2hrffPMNevTogYMHD6Jz584V5iCTyTB69Gh89NFHyMzMhLu7u7Rt27ZtyMnJwejRozXuo8t7Qt2uTzzxBJYsWYK0tDR88cUX+PPPP7Veu/Xq1cPSpUsBADdv3sQXX3yB0NBQ/PfffxW+xh/mzTffxPLlyzFx4kR07doV1tbWFb4XKlP2vanm7e1dYezSpUtRr149AKXt8iBXrlxBjx494ODggDfffBMODg749ttv0atXL+zZswchISEV3u+///5Dv3790KJFC2zcuBFWVqUffWlpaejSpQvy8/Px6quvom7duli1ahUGDRqEn376CUOGDAEA5ObmomXLlhgxYgQcHBwQHx+Pjz76CPn5+fjqq690ahN99e7dG2PHjtVY9+mnn2r8SCGEwKBBg3DgwAFMnDgR7dq1w++//44333wTt27dkl4TulqxYgXmzZuHTz/9FC+++KK0ftOmTcjPz8e0adNQt25d/P333/jqq69w8+ZNbNq0CUDpsaBssTxmzBgMGTIEQ4cOldbVr18fALBmzRqMGzcOffv2xYcffoj8/HwsW7YM3bp1w4kTJzSKBVtbW3z33XcaeTo7O0u3FyxYgMjISPTq1QvTpk3DxYsXsWzZMvzzzz9ax6C7d+8iNDQUI0aMwKhRo7Bx40ZMmzYNNjY2eOmllyptl7i4OOzYsUNrfWJiIgYMGABvb2/Mnz9fen5jxoypcD+5ubnYvn279LpSt7mdnZ1GUa6P+fPnV/m+FSl7jFy9ejX27NmjsV3fY65adnY2+vfvD2tra+zYsQNOTk4AdP/e8cUXX2DQoEEICwtDUVERNmzYgOHDh2P79u0YMGDAQ5+XnZ0dfvvtN6Snp8PDwwMAcP/+ffz444+ws7PTitf18SIjI7FgwQJ06dIFCxcuhI2NDY4ePYr9+/ejT58+AEqP5S+99BICAwMxZ84cuLm54cSJE9i1a5fG+6y6QkJCsGbNGmlZfTydO3eutK5Lly4ASj/vunbtisceewzvvPMOHB0dsXHjRgwePBibN2/WeI2qtWjRQtpXRkYG3njjDY3tuhwnHkmCLNaKFSsEALF3715x+/Zt8d9//4kNGzaIunXrCnt7e3Hz5k0hhBAFBQVCqVRq3DcxMVHY2tqKhQsXSut++OEHAUB89tlnWo+lUqmk+wEQH3/8sVZMYGCgeOqpp6TlAwcOCADiscceEzk5OdL6jRs3CgDiiy++kPbdrFkz0bdvX+lxhBAiPz9fNG7cWPTu3Vvrsbp06SJat24tLd++fVsAEBEREdK669evC4VCIRYvXqxx39OnTwsrKyut9ZcvXxYAxKpVq6R1ERERouzb5NChQwKAiI2N1bjvrl27tNb7+fmJAQMGaOU+ffp0Uf6tVz73t956S3h4eIgOHTpotOmaNWuEXC4Xhw4d0rj/8uXLBQDx559/aj2eWklJiWjcuLHw8/MTd+/e1dhWtt3HjRsnAIhXXnlFY/uAAQOEjY2NuH37tt5tIYQQly5dEgDEJ598Iq3
"text/plain": [
"<Figure size 1000x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import seaborn as sns\n",
"df2['bmi'] = pd.to_numeric(df2['bmi'], errors='coerce')\n",
"df_clean = df2.dropna(subset=['bmi', 'avg_glucose_level'])\n",
"df_clean['bmi_category'] = pd.cut(df_clean['bmi'], bins=range(0, 100, 10))\n",
"\n",
"plt.figure(figsize=(10, 6))\n",
"sns.boxplot(data=df_clean, x='bmi_category', y='avg_glucose_level')\n",
"plt.title('Распределение среднего уровня глюкозы по категориям индекса массы тела')\n",
"plt.xlabel('Категория индекса массы тела')\n",
"plt.ylabel('Средний уровень глюкозы')\n",
"plt.xticks(rotation=45)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"В с с
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Бизнес-цели</h3>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. **Бизнес-цель**: Определение наилучших регионов для внедрения программ профилактики инсультов.<br>\n",
" Эффект для бизнеса: Возможность выбора регионов с
"- **Цели технического проекта**:\n",
" - Построить модель для определения регионов с
" - **Входные признаки**: Индекс массы тела (BMI), Средний уровень глюкозы, Возраст, Пол.\n",
" - **Целевой признак**: Регион с
"2. **Бизнес-цель**: Оптимизация стратегий здравоохранения для снижения риска инсультов.\n",
" Эффект для бизнеса: Компании, предоставляющие медицинские услуги, могут использовать эти данные для выбора регионов, где их услуги будут наиболее востребованы.\n",
"- **Цели технического проекта**:\n",
" - Построить модель, определяющую регионы с
" - **Входные признаки**: Гипертония, Сердечное заболевание, Курение, Индекс массы тела (BMI).\n",
" - **Целевой признак**: Уровень риска инсультов.\n",
"3. **Бизнес-цель**: Определение экономического потенциала регионов на основе демографических данных и уровня здоровья населения.\n",
" Эффект для бизнеса: Компании могут определить регионы с
"- **Цели технического проекта**:\n",
" - Создать модель для ранжирования регионов по их экономическому потенциалу на основе демографических данных и уровня здоровья населения.\n",
" - **Входные признаки**: Индекс массы тела (BMI), Средний уровень глюкозы, Возраст, Пол, Гипертония, Сердечное заболевание.\n",
" - **Целевой признак**: Оценка экономического потенциала."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**<h3>Поиск проблем</h3>**"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Количество пропущенных ячеек: \n",
" id 0\n",
"gender 0\n",
"age 0\n",
"hypertension 0\n",
"heart_disease 0\n",
"ever_married 0\n",
"work_type 0\n",
"Residence_type 0\n",
"avg_glucose_level 0\n",
"bmi 201\n",
"smoking_status 0\n",
"stroke 0\n",
"dtype: int64\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA9oAAAIQCAYAAAB+ExYhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQWElEQVR4nO3deViU9f7/8dcAssiqKFvRIFYuJGlmRppYmWumZd+ThlvH9PctrMwWf5qhoUdOyylblE6ek1ZKpzqllSXlkthC7kaZkQtipaCSiJqAzNy/P7yYHyMoorcODM/Hdc0Vc9/v+573PQx3vuZePhbDMAwBAAAAAABTeLi6AQAAAAAA3AlBGwAAAAAAExG0AQAAAAAwEUEbAAAAAAATEbQBAAAAADARQRsAAAAAABMRtAEAAAAAMBFBGwAAAAAAExG0AQBuzTAM/fHHH9q+fburWwEAAI0EQRsA4HaOHDmiqVOnqk2bNvL29lZoaKiuvPJK5ebmuro1oN6bPn26LBaLq9sAgAaNoA0A9ciCBQtksVjO+Ljqqqtc3Wa9VlRUpISEBL388su666679NFHH2n58uVavXq1YmJiXN0eGrmYmBinv2dfX19dccUVevzxx/XHH3+4ur3zNnfuXC1YsMDVbQCAy3m5ugEAQHWpqalq1apVtel/+9vfXNBNw/L4449r3759ys7OVlxcnKvbAarp2LGjHn30UUlSaWmpNm7cqNmzZysrK0vr1q1zcXfnZ+7cuWrRooVGjx7t6lYAwKUI2gBQD/Xr10/XXnttten/+te/dPDgQRd01DDs379fb775pl577TVCNuqtSy65RMOHD3c8v++++xQQEKDnn39e27dv1xVXXOHC7gAAZuDUcQBo4CoqKjRjxgy1bt1aPj4+iomJ0ZQpU1RWVuZUFxMTU+0o0/vvvy+LxVLtlGq73a6XXnpJHTp0kK+vr1q2bKm+fftqw4YNjhqLxaLp06c79dG/f381b95cP/300xl77tmzp9Ppsy1atNCAAQP0448/OtVZLBaNHz/+tOupPNV+9+7dkqT169fLbrervLxc1157rXx9fRUaGqphw4Zpz549TsuOHj1aAQEB2rVrl/r06SN/f39FRUUpNTVVhmE41R47dkyPPvqooqOj5ePjozZt2uj555+vVidJCxcu1HXXXaemTZuqWbNm6tGjh7744gunmrlz5youLk4+Pj6KiopScnKyiouLz+k9Op21a9eqf//+atasmfz9/RUfH6+XXnrpjMvUdulC1d+3JG3evFn9+vVTUFCQAgICdMstt+i7776rcZ3e3t46cOCA07zs7GzHuqt+tqSTn83OnTvLz89PLVq00PDhw/X777871VT+Dqv673//K4vFotWrVztNDwgIqPb5r+3zJUm7d++WxWJxOh36l19+0R133KFmzZrJz89PXbp00ZIlS864ntpERERIkry8nI+BrFq1SjfeeKP8/f0VEhKiQYMGadu2bY758+fPl8Vi0RtvvOG03KxZs2SxWPTZZ585bcfzzz+vF198UVarVX5+fkpMTDyrz9TZ7GdiYmK0detWZWVlOX6vPXv2PNe3BAAaNI5oA0ADd9999+nNN9/UXXfdpUcffVRr165VWlqatm3bpsWLF592uYqKCj355JM1zhszZowWLFigfv366b777lNFRYW++uorfffddzUeaa/sY/Xq1Vq+fLnat29fa99t27bVk08+KcMwtHPnTr3wwgvq379/tUBcF0VFRZKk8ePHq3Pnzvr73/+uAwcO6OWXX9bXX3+tzZs3q0WLFo56m82mvn376vrrr9ezzz6rzMxMTZs2TRUVFUpNTZV08q7lt99+u7788kuNGTNGHTt21Oeff67HH39cv//+u1588UXH+p5++mlNnz5dN9xwg1JTU+Xt7a21a9dq1apV6t27t6STN5p6+umn1atXL91///3Kzc1Venq61q9fr2+++UZNmjQ57/do+fLluu222xQZGamHH35YERER2rZtm5YuXaqHH3641vfx1EsXjh49qvvvv9+pZuvWrbrxxhsVFBSkJ554Qk2aNNE///lP9ezZU1lZWeratatTvaenpxYuXKhHHnnEMW3+/Pny9fVVaWmpU+2CBQt07733qkuXLkpLS1NhYaFeeuklffPNN9q8ebNCQkJq3YYL5Y8//lCPHj105MgRPfTQQ4qIiNDChQt15513atGiRRo2bFit6zhx4oTjzJTS0lJt3rxZL7zwgnr06OH0vq9YsUL9+vVTbGyspk+fruPHj+uVV15Rt27dtGnTJsXExOjee+/Vhx9+qIkTJ+rWW29VdHS0fvjhBz399NMaM2aM+vfv7/Tab731lo4cOaLk5GSVlpbqpZde0s0336wffvhB4eHhp+35bPYzs2fP1oMPPqiAgADHvuVM6wQAt2YAAOqN+fPnG5KM9evX1zg/MTHRiIuLczzfsmWLIcm47777nOoee+wxQ5KxatUqxzSr1WqMGjXK8Xzu3LmGj4+PcdNNNxlWq9UxfdWqVYYk46GHHqr2+na73fGzJGPatGmGYRjG5MmTDU9PT2PJkiVntZ2JiYlGYmKi07QpU6YYkoz9+/c7vUZycvJp11P5fuXl5Tk9b9++vfHnn3866r788ktDkvHoo486po0aNcqQZDz44INO2zdgwADD29vbOHDggGEYhrFkyRJDkjFz5kyn177rrrsMi8Vi7NixwzAMw9i+fbvh4eFh3HHHHYbNZnOqrXzf9u/fb3h7exu9e/d2qnn11VcNScYbb7xR5/foVBUVFUarVq0Mq9VqHDp0qMY+Tud0n78DBw44/b4NwzAGDx5seHt7Gzt37nRM27t3rxEYGGj06NGj2jqHDRtmdOjQwTH92LFjRlBQkHHPPfc4vWZ5ebkRFhZmXHXVVcbx48cd9UuXLjUkGSkpKY5po0aNMvz9/Z16ff/99w1Jxpdffuk03d/f3+nzbxi1f74MwzDy8vIMScb8+fMNwzCMRx991JBkZGZmOmr+/PNPo127dkZERIRRXl5+xvVZrVZDUrVHt27djIMHDzrVduzY0QgLCzOKiooc077//nvDw8PDGDlypGPavn37jObNmxu33nqrUVZWZnTq1Mm47LLLjMOHD1fbDj8/P+O3335zTF+7dq0hyXjkkUcc06ZNm2ZU/SdiXfYzcXFx1T63ANAYceo4ADRglaeFTpw40Wl65Y2WPv300xqX+/PPP5Wamqrx48frsssuc5r3wQcfyGKxaNq0adWWq2nIn1dffVVpaWl6+eWXNWjQoLPuvfKo3oEDB5Sdna3FixcrPj7e6YizdPKI38GDB1VUVCS73X5W605OTpafn5/jec+ePdW5c+ca34+qpw5XnkpcXl6uFStWSDr5Hnt6euqhhx5yWu7RRx+VYRhatmyZJGnJkiWy2+1KSUmRh4fz/14r37cVK1aovLxcEyZMcKoZO3asgoKCqvV3tu9RVZs3b1ZeXp4mTJhQ7civWUM22Ww2ffHFFxo8eLBiY2Md0yMjI3XPPffo66+/VklJidMyI0aM0M8//+w4RfyDDz5QcHCwbrnlFqe6DRs2aP/+/XrggQfk6+vrmD5gwAC1bdv2tJ/pc3W2n6+jR4/q4MGD+uyzz9S+fXv16dPHMc/Pz08PPPCACgoKtGnTplpfs2vXrlq+fLmWL1+upUuX6m9/+5u2bt2q22+/XcePH5ck7du3T1u2bNHo0aPVvHlzx7Lx8fG69dZbHX/70snTzufMmaPly5frxhtv1JYtW/TGG28oKCio2msPHjxYl1xyieP5ddddp65duzqt71Tnup8BgMaMoA0ADVh+fr48PDx0+eWXO02PiIhQSEiI8vPza1zuhRdeUGlpqaZMmVJt3s6dOxUVFeX0j/vTWbZsmeNU5LoOTfTtt9+qZcuWCgsL0w033KCKigrHNeNV/fvf/1bLli3VokUL+fn5qUePHtWu561UuWzbtm2rzWvXrp3jWu5KHh4eTkFRkq688kpJctTm5+crKipKgYGB1dZXOV86+b55eHic8bT5yto2bdo4Tff29lZsbGy139fZvkdV7dy
"text/plain": [
"<Figure size 1200x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Количество выбросов (по Z-оценке): 118\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnMAAAIjCAYAAABoPp35AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADAhElEQVR4nOzdd3QU1dvA8e/spvdeCYSaEHondOkgCCggiNIU9aeIiihioVmwoGJ9VZCigqA0kSpViqGXUAMEQiAkIYX0vjvvH5ENazYYIJBs8nzOmXOys3dm7kx2Z5557r2ziqqqKkIIIYQQwixpyrsCQgghhBDizkkwJ4QQQghhxiSYE0IIIYQwYxLMCSGEEEKYMQnmhBBCCCHMmARzQgghhBBmTII5IYQQQggzJsGcEEIIIYQZk2BOCCGEEMKMSTAnhBBCCGHGJJgTooJauHAhiqJw8ODBYu/NnTsXRVEYOHAgOp2uHGonhBCiopBgTggzs2rVKv73v//RsWNHli5dilarLe8qCSGEKEcSzAlhRnbs2MHw4cMJCQnhjz/+wMbGpryrJIQQopxJMCeEmTh69CgDBgzA19eXTZs24ezsXKzMb7/9RosWLbC1tcXDw4PHH3+cmJgYozKjR4/GwcGBCxcu0KtXL+zt7fHz82PmzJmoqmooFxUVhaIozJ49m88++4waNWpga2tL586dOXHiRLFtnzlzhsGDB+Pm5oaNjQ0tW7ZkzZo1JvelS5cuKIpSbFq4cKFRuf/7v/+jYcOG2NnZGZVbvny50boaNmxYbBuzZ89GURSioqIM8240Xd88T6/X07hxY5Pb37ZtGx07dsTe3h4XFxcGDBjA6dOnjcpMnz4dRVFITEw0mn/w4MFi67xx7P9t+fLlKIrCjh07DPN27drFkCFDqF69OtbW1gQEBPDyyy+TnZ1tcvmWLVvi6OhodJxmz55drOzNbhwPKysrEhISjN4LCwszrOfmpv7S1Gv06NEm/783Tzf+B4GBgfTr148///yTpk2bYmNjQ0hICCtXrjRZ19L8727nOOfl5TF16lRatGiBs7Mz9vb2dOzYke3bt9/y2AlRkViUdwWEEP8tMjKS3r17Y21tzaZNm/D19S1WZuHChYwZM4ZWrVoxa9Ys4uPj+fzzz9mzZw9HjhzBxcXFUFan09G7d2/atm3LRx99xMaNG5k2bRoFBQXMnDnTaL0//vgj6enpPP/88+Tk5PD555/TtWtXjh8/jre3NwAnT56kffv2+Pv78/rrr2Nvb8+vv/7KwIEDWbFiBYMGDSpW3+DgYN58800AEhMTefnll43eX7ZsGc899xxdunThhRdewN7entOnT/P+++/f7eE08tNPP3H8+PFi87ds2UKfPn2oVasW06dPJzs7my+//JL27dtz+PBhAgMDy7Qe//bbb7+RlZXF//73P9zd3dm/fz9ffvklV65c4bfffjOUCwsLY+jQoTRp0oQPPvgAZ2dnk8fzVrRaLT///LPRMgsWLMDGxoacnJzbrtczzzxD9+7dDcs88cQTDBo0iIcfftgwz9PT0/D3uXPnePTRR3n22WcZNWoUCxYsYMiQIWzcuJEePXqUWO+S/ne3Iy0tjXnz5jF8+HDGjRtHeno6P/zwA7169WL//v00bdr0rtYvxH2hCiEqpAULFqiAunbtWrV27doqoPbs2dNk2by8PNXLy0tt2LChmp2dbZi/du1aFVCnTp1qmDdq1CgVUF944QXDPL1erz744IOqlZWVmpCQoKqqql68eFEFVFtbW/XKlSuGsvv27VMB9eWXXzbM69atm9qoUSM1JyfHaJ3t2rVT69atW6y+7du3Vx944AHD6xvbWrBggWHe8OHDVRcXF6P92b59uwqov/32m2Fe586d1QYNGhTbxscff6wC6sWLFw3zbhzTG/NycnLU6tWrq3369Cm2/aZNm6peXl5qUlKSYd6xY8dUjUajjhw50jBv2rRpKmA4bjccOHCg2DpHjRql2tvbF6vrb7/9pgLq9u3bDfOysrKKlZs1a5aqKIp66dIlw7wpU6aogBobG2uYd+N4fvzxx8XWcbMbx2P48OFqo0aNDPMzMzNVJycn9bHHHlMB9cCBA7ddr5sB6rRp00y+V6NGDRVQV6xYYZiXmpqq+vr6qs2aNStW19L8727nOBcUFKi5ublG5a5fv656e3urY8eONVlnISoaaWYVooIbPXo0ly9f5rHHHuPPP/80ysrccPDgQa5du8Zzzz1n1I/uwQcfJDg4mHXr1hVbZvz48Ya/FUVh/Pjx5OXlsWXLFqNyAwcOxN/f3/C6devWtGnThvXr1wOQnJzMtm3bGDp0KOnp6SQmJpKYmEhSUhK9evXi3LlzxZp68/LysLa2vuV+p6enY2dnd0/7BX799dckJSUxbdo0o/mxsbEcPXqU0aNH4+bmZpjfuHFjevToYdj3myUnJxv2PTExkdTU1BK3e3O5xMRE0tPTi5WxtbU1/J2ZmUliYiLt2rVDVVWOHDlieC89PR2NRmOUeb1dTzzxBGfOnDE0p65YsQJnZ2e6det2x/W6HX5+fkbZWycnJ0aOHMmRI0eIi4szuUxJ/7vbpdVqsbKyAgqbbZOTkykoKKBly5YcPnz4rtYtxP0iwZwQFVxycjI///wzixYtomnTprz44ovFAoVLly4BEBQUVGz54OBgw/s3aDQaatWqZTSvXr16AEZ9kgDq1q1bbJ316tUzlDt//jyqqvL222/j6elpNN240F67ds1o+ZSUFJN9mm4WGhrK1atXmT59OtHR0f8ZIN2u1NRU3n//fSZOnGhoLr7hVsezfv36JCYmkpmZaTQ/KCjIaN9vbma8WWZmZrHjNHbs2GLloqOjDcGkg4MDnp6edO7c2VD3G0JDQ9Hr9bz44otERkaSmJjI9evXb+tYeHp68uCDDzJ//nwA5s+fz6hRo9Boil8iSluv21GnTh0URTGaV9Ln8cZ2Svrf3YlFixbRuHFjbGxscHd3x9PTk3Xr1pXp502Ie0n6zAlRwX388ccMGTIEgO+//562bdsyZcoUvvnmm3KuWSG9Xg/ApEmT6NWrl8kyderUMXodFxdXYtkbXn75ZSIiInjnnXeYMWNG2VT2Jh9++CEajYZXX32VpKSku17fihUrcHJyMrw+e/Yszz//fLFyNjY2/PHHH0bzdu3aZdRXUafT0aNHD5KTk5k8eTLBwcHY29sTExPD6NGjDcccYNiwYRw+fJgvv/yS77///o7rP3bsWEaOHMkLL7zAzp07mTdvHrt27TIqczv1upfK8n/3888/M3r0aAYOHMirr76Kl5cXWq2WWbNmERkZWUY1FuLekmBOiAquU6dOhr9btWrF888/z9dff83IkSNp27YtADVq1AAgIiKCrl27Gi0fERFheP8GvV7PhQsXDNkPKAw+gGId+8+dO1esTmfPnjWUu5Hhs7S0LDEbdbMrV66Qnp5O/fr1b1nO1taWuXPncuTIEZydnZk2bRrHjh1j0qRJ/7mN/3L16lU+//xzZs2ahaOjY7GA4Obj+W9nzpzBw8MDe3t7o/mdOnXCw8PD8LqkZk+tVlvsOKWkpBi9Pn78OGfPnmXRokWMHDnSMH/z5s3F1qfRaJg9ezbHjx/n4sWLfPPNN8THx/P444+b3H5J+vTpg42NDcOGDaNDhw7Url27WDB3O/W6HTeyuzdn50r6PP7X/+52LV++nFq1arFy5Uqj7d9t860Q95M0swphZt577z18fX15+umnKSgoAKBly5Z4eXnx7bffkpubayi7YcMGTp8+zYMPPlhsPV999ZXhb1VV+eqrr7C0tCzWT2r16tVGfd7279/Pvn376NOnDwBeXl506dKF7777jtjY2GLb+fcjL5YuXQpQLOg0ZcqUKURHR/Pzzz/TvXt3WrRo8Z/LlMaMGTPw9vbm2WefNfm+r68vTZs2ZdGiRUaB1okTJ/jzzz/p27dvmdSjJDceBK3e9KgYVVX5/PPPTZb/8ssv2bZtG4sXL6Z79+60b9/+trdpYWHByJEjCQ8PN9nseyf1Kq2rV6+yatUqw+u0tDR+/PFHmjZtio+Pj1H
"text/plain": [
"<Figure size 640x480 with 2 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from scipy import stats\n",
"\n",
"missing_val = df2.isnull().sum()\n",
"print(\"Количество пропущенных ячеек: \\n\", missing_val)\n",
"\n",
"df2['bmi'] = pd.to_numeric(df2['bmi'], errors='coerce')\n",
"df2['avg_glucose_level'] = pd.to_numeric(df2['avg_glucose_level'], errors='coerce')\n",
"df2['age'] = pd.to_numeric(df2['age'], errors='coerce')\n",
"df2['hypertension'] = df2['hypertension'].astype(int)\n",
"df2['heart_disease'] = df2['heart_disease'].astype(int)\n",
"df2['smoking_status'] = df2['smoking_status'].astype('category').cat.codes\n",
"\n",
"# Удаление пропусков для корректного анализа\n",
"data = df2.dropna()\n",
"\n",
"# 1. Визуализация распределения данных (помогает выявить выбросы)\n",
"plt.figure(figsize=(12, 6))\n",
"sns.boxplot(data=data[['bmi', 'avg_glucose_level', 'age']])\n",
"plt.title('Поиск выбросов с
"plt.show()\n",
"\n",
"# Вычисление Z-оценки для выявления выбросов\n",
"z_score = np.abs(stats.zscore(data[['bmi', 'avg_glucose_level', 'age']]))\n",
"outliers = np.where(z_score > 3)\n",
"print(f\"Количество выбросов (по Z-оценке): {len(outliers[0])}\")\n",
"\n",
"# Построение корреляционной матрицы для поиска зашумленности\n",
"corr_matrix = data[['bmi', 'avg_glucose_level', 'age', 'hypertension', 'heart_disease', 'smoking_status']].corr()\n",
"sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')\n",
"plt.title('Корреляционная матрица')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Из матрицы корреляции можно сделать выводы относительно зависимости между атрибутами и выявить наиболее бесполезные из них. Признаки с
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Проведем анализа остаточных ошибок, чтобы убедиться в том, что выше упомянутые атрибуты действиельно бесполезны в данном контексе."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"По итогу вышло больше количество шумов"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA+0AAAIjCAYAAAB20vpjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACK0klEQVR4nOzdd3hUZf7+8XsmZdJ7rwQIJHQIRcSCgBSx4yqKihXWxe66ftV1rbv8XHftrrq6i7rqumKvqIggKr2X0EJ6n/Te5vz+iIzGAEJIMpPk/bquXDrnnDnPZ1JI7nmayTAMQwAAAAAAwOmYHV0AAAAAAAA4PEI7AAAAAABOitAOAAAAAICTIrQDAAAAAOCkCO0AAAAAADgpQjsAAAAAAE6K0A4AAAAAgJMitAMAAAAA4KQI7QAAALCz2WyyWq06ePCgo0sBAIjQDgAA0OcVFBTo1ltvVXx8vNzd3RUaGqohQ4aosrLS0aUBQJ/n6ugCAAAd98orr+jqq6+2P7ZYLIqLi9P06dN13333KTw83IHVAegJDhw4oDPOOENNTU26+eabNWbMGLm6usrT01Pe3t6OLg8A+jxCOwD0Ag899JASEhJUX1+v7777Ts8//7w+++wz7dy5U15eXo4uD4ATW7hwodzd3bV27VpFR0c7uhwAwC8Q2gGgF5g1a5bGjh0rSbruuusUHBysxx9/XB9++KEuvfRSB1cHwFlt2rRJK1as0JdffklgBwAnxZx2AOiFpkyZIklKT0+XJJWWlur3v/+9hg8fLh8fH/n5+WnWrFnatm1bu+fW19frgQce0KBBg+Th4aHIyEhdeOGFSktLkyRlZGTIZDId8WPy5Mn2e61cuVImk0n/+9//dM899ygiIkLe3t4699xzlZ2d3a7tdevWaebMmfL395eXl5dOP/10ff/994d9jZMnTz5s+w888EC7a19//XWlpKTI09NTQUFBmjt37mHbP9pr+zmbzaYnn3xSQ4cOlYeHh8LDw7Vw4UKVlZW1ua5fv346++yz27Vz4403trvn4Wp/7LHH2n1OJamhoUH333+/Bg4cKIvFotjYWP3hD39QQ0PDYT9XP/fLz1tISIhmz56tnTt3trmuublZDz/8sAYMGCCLxaJ+/frpnnvuadfGeeedp379+snDw0NhYWE699xztWPHjnav7cYbb9Qbb7yhwYMHy8PDQykpKfr222/bXJeZmanf/e53Gjx4sDw9PRUcHKzf/OY3ysjIaPc6ysvLddttt6lfv36yWCyKiYnRlVdeKavVav++O9rHoc/18bT5S8fzsyBJRUVFuvbaaxUeHi4PDw+NHDlSr7766q+2I7V+Lx26r9lsVkREhC655BJlZWW1q+dvf/vbEe/zwAMPtPneW7t2rTw8PJSWlqahQ4fKYrEoIiJCCxcuVGlpabvnL1261P6zFBISossvv1y5ubltrrnqqqvk4+OjgwcPasaMGfL29lZUVJQeeughGYbRrt5XXnnFfqyqqkopKSlKSEhQfn6+/fix/swBQG9ETzsA9EKHAnZwcLAk6eDBg/rggw/0m9/8RgkJCSosLNSLL76o008/Xbt371ZUVJQkqaWlRWeffba+/vprzZ07V7fccouqqqr01VdfaefOnRowYIC9jUsvvVRnnXVWm3bvvvvuw9bz5z//WSaTSXfddZeKior05JNPatq0adq6das8PT0lSStWrNCsWbOUkpKi+++/X2azWUuWLNGUKVO0evVqjR8/vt19Y2JitHjxYklSdXW1brjhhsO2fd999+niiy/Wddddp+LiYj3zzDM67bTTtGXLFgUEBLR7zoIFC3TqqadKkt577z29//77bc4vXLjQvp7AzTffrPT0dD377LPasmWLvv/+e7m5uR3283A8ysvL7a/t52w2m84991x99913WrBggZKTk7Vjxw498cQT2rdvnz744INfvXdSUpLuvfdeGYahtLQ0Pf744zrrrLPaBMDrrrtOr776qi666CLdcccdWrdunRYvXqzU1NR2n48FCxYoIiJCeXl5evbZZzVt2jSlp6e3mZqxatUq/e9//9PNN98si8Wif/zjH5o5c6bWr1+vYcOGSZI2bNigH374QXPnzlVMTIwyMjL0/PPPa/Lkydq9e7f9ftXV1Tr11FOVmpqqa665RmPGjJHVatVHH32knJwcJScn6z//+Y+97X/+859KTU3VE088YT82YsSI42rzaI7lZ6Gurk6TJ0/WgQMHdOONNyohIUFLly7VVVddpfLyct1yyy2/2s6pp56qBQsWyGazaefOnXryySeVl5en1atX/+pzj6SkpET19fW64YYbNGXKFP32t79VWlqannvuOa1bt07r1q2TxWKR9NMaGuPGjdPixYtVWFiop556St9//327n6WWlhbNnDlTJ510kv76179q2bJluv/++9Xc3KyHHnrosLU0NTVpzpw5ysrK0vfff6/IyEj7ue74mQMAp2UAAHqsJUuWGJKM5cuXG8XFxUZ2drbx1ltvGcHBwYanp6eRk5NjGIZh1NfXGy0tLW2em56eblgsFuOhhx6yH/v3v/9tSDIef/zxdm3ZbDb78yQZjz32WLtrhg4dapx++un2x998840hyYiOjjYqKyvtx99++21DkvHUU0/Z752YmGjMmDHD3o5hGEZtba2RkJBgnHnmme3aOvnkk41hw4bZHxcXFxuSjPvvv99+LCMjw3BxcTH+/Oc/t3nujh07DFdX13bH9+/fb0gyXn31Vfux+++/3/j5r8vVq1cbkow33nijzXOXLVvW7nh8fLwxe/bsdrUvWrTI+OWv4F/W/oc//MEICwszUlJS2nxO//Of/xhms9lYvXp1m+e/8MILhiTj+++/b9fez51++ult7mcYhnHPPfcYkoyioiLDMAxj69athiTjuuuua3Pd73//e0OSsWLFiiPe/9DXduPGjW1e2y+PZWZmGh4eHsYFF1xgP1ZbW9vufmvWrDEkGa+99pr92J/+9CdDkvHee++1u/7n3z+HzJ8/34iPjz9svcfa5uEcz8/Ck08+aUgyXn/9dfuxxsZGY+LEiYaPj0+bn4/DiY+PN+bPn9/m2GWXXWZ4eXkdUz2H/PL7+dDjqVOnGs3Nzfbjh/5teeaZZ+y1hoWFGcOGDTPq6urs133yySeGJONPf/qT/dj8+fMNScZNN91kP2az2YzZs2cb7u7uRnFxcZt6lyxZYthsNmPevHmGl5eXsW7dujY1H8/PHAD0RgyPB4BeYNq0aQoNDVVsbKzmzp0rHx8fvf/++/Y5qhaLRWZz6z/5LS0tKikpkY+PjwYPHqzNmzfb7/Puu+8qJCREN910U7s2fjmc+3hceeWV8vX1tT++6KKLFBkZqc8++0yStHXrVu3fv1+XXXaZSkpKZLVaZbVaVVNTo6lTp+rbb7+VzWZrc8/6+np5eHgctd333ntPNptNF198sf2eVqtVERERSkxM1DfffNPm+sbGRkmy9yweztKlS+Xv768zzzyzzT1TUlLk4+PT7p5NTU1trrNaraqvrz9q3bm5uXrmmWd03333ycfHp137ycnJSkpKanPPQ1Miftn+4Ryqqbi4WGvWrNH777+vESNGKCQkRJLsX5fbb7+9zfPuuOMOSdKnn37a5nhtba2sVqu2bt2ql156SeHh4Ro0aFCbayZOnKiUlBT747i4OJ133nn64osv1NLSIkn2UReHaiwpKdHAgQMVEBDQ7vt05MiRuuCCC9q9tuP9Pj3WNk/UZ599poiIiDZrTLi5uenmm29WdXW1Vq1a9av3aGhokNVqVVFRkb766iutWLFCU6dObXfdoa9HWVlZm+HoR3P77bfLxcXF/viKK65QeHi4/Wu9ceNGFRUV6Xe/+12bn7vZs2crKSmp3feE1DoN5JBDUyQaGxu1fPnydtfeeeedeuONN/T222+3G1VzvD9zANDbMDweAHqB5557ToMGDZKrq6vCw8M1ePBge0iXWodUP/XUU/rHP/6h9PR0e0iSfhpCL7UOqx88eLBcXTv310NiYmKbxyaTSQMHDrTPG96/f78kaf78+Ue8R0VFhQIDA+2PrVZ
"text/plain": [
"<Figure size 1200x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Среднее значение возраста: 42.865373803218574\n",
"Медиана возраста: 44.0\n"
]
}
],
"source": [
"plt.figure(figsize=(12, 6))\n",
"sns.histplot(data=data['age'], kde=True, bins=30)\n",
"plt.title('Распределение возраста по выборке')\n",
"plt.xlabel('Возраст')\n",
"plt.ylabel('Количество людей')\n",
"plt.show()\n",
"\n",
"mean_age = data['age'].mean()\n",
"median_age = data['age'].median()\n",
"print(f\"Среднее значение возраста: {mean_age}\")\n",
"print(f\"Медиана возраста: {median_age}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Определяем смещение. Если разница между средним и медианным значениями существенна, это может указывать на смещение. В
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Просачивания данных в столбце 'avg_glucose_level' не обнаружено.\n",
"Просачивания данных в столбце 'bmi' не обнаружено.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\3655327482.py:2: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" data['avg_glucose_level'] = pd.to_numeric(data['avg_glucose_level'].astype(str).str.replace(',', ''), errors='coerce')\n",
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\3655327482.py:3: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" data['bmi'] = pd.to_numeric(data['bmi'].astype(str).str.replace(',', ''), errors='coerce')\n"
]
}
],
"source": [
"# Приведение столбцов к числовым значениям\n",
"data['avg_glucose_level'] = pd.to_numeric(data['avg_glucose_level'].astype(str).str.replace(',', ''), errors='coerce')\n",
"data['bmi'] = pd.to_numeric(data['bmi'].astype(str).str.replace(',', ''), errors='coerce')\n",
"\n",
"# Проверка на аномалии в данных\n",
"invalid_avg_glucose = data[data['avg_glucose_level'] < 0] # Проверка на отрицательные значения\n",
"invalid_bmi = data[data['bmi'] < 0] # Проверка на отрицательные значения\n",
"\n",
"if not invalid_avg_glucose.empty:\n",
" print(\"Просачивание данных: Неверные значения среднего уровня глюкозы в следующих строках:\")\n",
" print(invalid_avg_glucose)\n",
"else:\n",
" print(\"Просачивания данных в столбце 'avg_glucose_level' не обнаружено.\")\n",
"\n",
"if not invalid_bmi.empty:\n",
" print(\"Просачивание данных: Неверные значения индекса массы тела в следующих строках:\")\n",
" print(invalid_bmi)\n",
"else:\n",
" print(\"Просачивания данных в столбце 'bmi' не обнаружено.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Также я хочу выявить наличие \"просачивания данных\" путем проверки аномалий. Для этого хочу сравнить уровень глюкозы и индекс массы тела с о В
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Размер данных после удаления выбросов: (4792, 12)\n",
"Обрабатываем столбец: avg_glucose_level, медианное значение: 91.68\n",
"Обрабатываем столбец: bmi, медианное значение: 28.1\n",
"Обрабатываем столбец: age, медианное значение: 44.0\n",
"Обрабатываем столбец: heart_disease, медианное значение: 0.0\n",
"Обрабатываем столбец: hypertension, медианное значение: 0.0\n",
"Выбросы заменены медианными значениями.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\1082538639.py:15: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" data[column] = np.where(col_z_score > 3, median_value, data[column])\n",
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\1082538639.py:15: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" data[column] = np.where(col_z_score > 3, median_value, data[column])\n",
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\1082538639.py:15: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" data[column] = np.where(col_z_score > 3, median_value, data[column])\n",
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\1082538639.py:15: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" data[column] = np.where(col_z_score > 3, median_value, data[column])\n",
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_22948\\1082538639.py:15: SettingWithCopyWarning: \n",
"A value is trying to be set on a copy of a slice from a DataFrame.\n",
"Try using .loc[row_indexer,col_indexer] = value instead\n",
"\n",
"See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
" data[column] = np.where(col_z_score > 3, median_value, data[column])\n"
]
}
],
"source": [
"data_no_outliers = data[(z_score < 3).all(axis=1)]\n",
"print(f\"Размер данных после удаления выбросов: {data_no_outliers.shape}\")\n",
"\n",
"# Столбцы для проверки и замены выбросов\n",
"columns_to_check = ['avg_glucose_level', 'bmi', 'age', 'heart_disease', 'hypertension']\n",
"\n",
"for column in columns_to_check:\n",
" col_z_score = np.abs(stats.zscore(data[column]))\n",
" \n",
" median_value = data[column].median()\n",
" \n",
" print(f\"Обрабатываем столбец: {column}, медианное значение: {median_value}\")\n",
" \n",
" # Замена выбросов медианными значениями\n",
" data[column] = np.where(col_z_score > 3, median_value, data[column])\n",
"\n",
"print(\"Выбросы заменены медианными значениями.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Решили проблему с
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Оценка качества набора данных:</h3>\n",
"\n",
"**Информативность:** Н а б о р о с
"\n",
"**Степень покрытия:** В
"\n",
"**Соответствие реальным данным:** Данные являются актуальными и могут быть основаны на реальных медицинских обследованиях и исследованиях, что делает их значимыми для клинической практики и научных исследований.\n",
"\n",
"**Согласованность меток:** Имена колонок в наборе данных четкие и интуитивно понятные, такие как \"age\" (возраст), \"avg_glucose_level\" (средний уровень глюкозы) и \"stroke\" (инсульт). Однако необходимо обратить внимание на формат данных, поскольку некоторые числовые значения могут быть представлены как строки, что требует предобработки для корректного анализа."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Устранение проблемы пропущенных данных</h3>"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Данные после заполнения средним значением: 0 пропущенных значений осталось.\n"
]
}
],
"source": [
"\n",
"data_filled_mean = data.copy()\n",
"data_filled_mean[\"avg_glucose_level\"] = pd.to_numeric(data_filled_mean[\"avg_glucose_level\"], errors='coerce')\n",
"\n",
"mean_value = data_filled_mean[\"avg_glucose_level\"].mean()\n",
"data_filled_mean[\"avg_glucose_level\"] = data_filled_mean[\"avg_glucose_level\"].fillna(mean_value)\n",
"\n",
"print(f\"Данные после заполнения средним значением: {data_filled_mean['avg_glucose_level'].isnull().sum()} пропущенных значений осталось.\")\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Обучающая выборка: 2945 строк\n",
"Валидационная выборка: 982 строк\n",
"Тестовая выборка: 982 строк\n"
]
}
],
"source": [
"columns_to_drop = [\"id\", \"stroke\"]\n",
"columns_to_drop = [col for col in columns_to_drop if col in data_filled_mean.columns]\n",
"\n",
"X = data_filled_mean.drop(columns=columns_to_drop)\n",
"y = data_filled_mean[\"stroke\"] \n",
"\n",
"X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)\n",
"X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)\n",
"\n",
"print(f\"Обучающая выборка: {X_train.shape[0]} строк\")\n",
"print(f\"Валидационная выборка: {X_val.shape[0]} строк\")\n",
"print(f\"Тестовая выборка: {X_test.shape[0]} строк\")"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Распределение в обучающей выборке (инсульт):\n",
" count 2945.000000\n",
"mean 0.039389\n",
"std 0.194551\n",
"min 0.000000\n",
"25% 0.000000\n",
"50% 0.000000\n",
"75% 0.000000\n",
"max 1.000000\n",
"Name: stroke, dtype: float64\n",
"\n",
"Распределение в валидационной выборке (инсульт):\n",
" count 982.000000\n",
"mean 0.046843\n",
"std 0.211411\n",
"min 0.000000\n",
"25% 0.000000\n",
"50% 0.000000\n",
"75% 0.000000\n",
"max 1.000000\n",
"Name: stroke, dtype: float64\n",
"\n",
"Распределение в тестовой выборке (инсульт):\n",
" count 982.000000\n",
"mean 0.047862\n",
"std 0.213582\n",
"min 0.000000\n",
"25% 0.000000\n",
"50% 0.000000\n",
"75% 0.000000\n",
"max 1.000000\n",
"Name: stroke, dtype: float64\n"
]
}
],
"source": [
"# Проверка распределения целевой переменной (инсульт) в обучающей, валидационной и тестовой выборках\n",
"print(\"Распределение в обучающей выборке (инсульт):\\n\", y_train.describe())\n",
"print(\"\\nР а с р е е е е
"print(\"\\nР а с р е е е е
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Дисбаланс данных:\n",
"\n",
"- В у
"\n",
"- Стандартное отклонение в обучающей выборке составляет 0.195, что свидетельствует о с
"\n",
"- В В с
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Методы приращения данных<h3>"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Размер обучающей выборки после Oversampling: 5658 строк\n",
"Распределение целевой переменной после Oversampling:\n",
"stroke\n",
"0 0.5\n",
"1 0.5\n",
"Name: proportion, dtype: float64\n",
"Размер обучающей выборки после Undersampling: 232 строк\n",
"Распределение целевой переменной после Undersampling:\n",
"stroke\n",
"0 0.5\n",
"1 0.5\n",
"Name: proportion, dtype: float64\n"
]
}
],
"source": [
"from imblearn.over_sampling import RandomOverSampler\n",
"from imblearn.under_sampling import RandomUnderSampler\n",
"\n",
"# Oversampling (увеличение выборки) для обучающих данных\n",
"ros = RandomOverSampler(random_state=42)\n",
"X_train_res, y_train_res = ros.fit_resample(X_train, y_train)\n",
"print(f\"Размер обучающей выборки после Oversampling: {X_train_res.shape[0]} строк\")\n",
"print(f\"Распределение целевой переменной после Oversampling:\\n{y_train_res.value_counts(normalize=True)}\")\n",
"\n",
"# Undersampling (уменьшение выборки) для обучающих данных\n",
"rus = RandomUnderSampler(random_state=42)\n",
"X_train_res_under, y_train_res_under = rus.fit_resample(X_train, y_train)\n",
"print(f\"Размер обучающей выборки после Undersampling: {X_train_res_under.shape[0]} строк\")\n",
"print(f\"Распределение целевой переменной после Undersampling:\\n{y_train_res_under.value_counts(normalize=True)}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h2>Онлайн обучение<h2>"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Index(['Education Level', 'Institution Type', 'Gender', 'Age', 'Device',\n",
" 'IT Student', 'Location', 'Financial Condition', 'Internet Type',\n",
" 'Network Type', 'Flexibility Level'],\n",
" dtype='object') \n",
"\n"
]
}
],
"source": [
"import numpy as np\n",
"import pandas as pd \n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"\n",
"df3 = pd.read_csv(\"C:/Users/TIGR228/Desktop/МИИ/Lab1/AIM-PIbd-31-Afanasev-S-S/static/csv/students_adaptability_level_online_education.csv\")\n",
"\n",
"print(df3.columns, \"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Основные столбцы:</br>\n",
"**Education Level** — Уровень образования</br>\n",
"**Institution Type** — Тип учебного заведения</br>\n",
"**Gender** — Пол</br>\n",
"**Age** — Возраст</br>\n",
"**Device** — Устройство</br>\n",
"**IT Student** — Студент IT</br>\n",
"**Location** — Местоположение</br>\n",
"**Financial Condition** — Финансовое состояние</br>\n",
"**Internet Type** — Тип интернета</br>\n",
"**Network Type** — Тип сети</br>\n",
"**Flexibility Level** — Уровень гибкости</br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h2>Проблемная область<h2>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Данный набор данных относится к анализу онлайн образования, е г о с с а
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3> Анализ содержимого<h3>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Объекты наблюдения:** Студенты, участвующие в онлайн обучении</br>\n",
"**Атрибуты объектов:** Уровень образования, тип учебного заведения, пол, возраст, устройство, является ли студент IT-специальности, местоположение, финансовое состояние, тип интернета, тип сети, уровень гибкости обучения</br>\n",
"**Связи между объектами:** Можно выявить связи между возрастом и использованием устройства для обучения, местоположением и типом интернет-соединения, а
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABFkAAAJOCAYAAABsntgnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd1gUV9sG8HtBehUBEcGCKIKCBSuo2MHYNbFGRaxRYw2xC2oUu2KNFewau8bE2MBCrIm9i1ii2EWa9Pn+8Nt5WXaBXVxYgft3XXvpTjnzzNmzc4Znz8xIBEEQQEREREREREREX0RL0wEQERERERERERUFTLIQEREREREREakBkyxERERERERERGrAJAsRERERERERkRowyUJEREREREREpAZMshARERERERERqQGTLEREREREREREasAkCxERERERERGRGjDJQkRERERERESkBkyyEBERfUVmz56NjIwMAEBGRgaCgoI0HBERERERKatIJ1kiIyMxZMgQODg4QF9fH6ampvD09ERwcDA+ffqk6fCIiIjkbNy4EQsWLMB///2HhQsXYuPGjRqLhf0oERERkWokgiAImg4iPxw+fBjfffcd9PT00LdvX1SvXh0pKSk4e/Ys9uzZA19fX6xZs0bTYRIREcnYuXMn+vbti5SUFOjp6WHLli349ttvCzwO9qNEREREqiuSSZaoqCi4ubnBzs4OJ0+eRJkyZWTmP3z4EIcPH8aoUaM0FCEREVH2Xr9+jYcPH6Jy5cqwsrIq8O2zHyUiIiLKmyJ5udC8efMQHx+P9evXy50YAoCjo6PMiaFEIhFf2traKFu2LAYPHoyYmBiZ9R49eoTvvvsOtra20NLSEtepXr26uMzZs2fRqFEjWFpaQl9fHw4ODhg/fjySkpLEZUJDQyGRSHD58mWV9026rq6uLt68eSMz79y5c2JMWcu+cOECfHx8YGZmBkNDQ3h5eSEiIkJmmYkTJ0JfX19menh4OCQSCcLDw8VpERER0NfXx8SJE+Xia9q0qUx9Sl+BgYHiMl5eXqhRo4bC/XNycoK3t3eOdVChQgWxXC0tLdjY2KB79+54+vSpzHIJCQkYN24c7O3toaenBycnJyxYsACZ84rv3r1DmzZtYGdnBz09PZQpUwa9e/fGkydPxGUeP34MiUSCBQsWYPHixShfvjwMDAzg5eWFmzdvymzz+vXr8PX1FYfW29jYwM/PD+/evZPbj+fPn2PAgAGwtbWFnp4eKlasiB9++AEpKSni55zTKzQ0VCzr5MmTaNy4MYyMjGBubo6OHTvizp07MtsLDAyERCKBtbU1UlNTZeZt375dLPft27c51r+vry8qVKggM+3Zs2cwMDCARCLB48eP1br+n3/+CS8vL5iYmMDU1BR169bFtm3bxPnZtTnpK2t5K1euRLVq1aCnpwdbW1sMHz5c7rsO/O9zV/TKusyCBQty3Oe9e/eiXr16sLCwgIGBAapWrYq5c+cia477ypUraNOmDUxNTWFsbIwWLVrg/PnzMstkbRuGhoZwdXXFunXrZJZTti1K20XWz/3y5cty7Swvn520XZqYmKBt27a4deuWzDK+vr6QSCSoWbOmXL0FBQVBIpHA2NhYbl5mAQEB0NHRkTsmAsDgwYNhbm4ucwzOqU3l1p4yf/4SiQQjRozA1q1b4eTkBH19fbi7u+P06dNycajy2T5+/BjW1tbw8PBAqVKl4ObmJvdZAMDdu3fRrVs3WFlZwcDAAE5OTpg8eTKA/32uOb2kx/Ws+2xpaYmmTZvK9aMhISFo3rw5rK2tUa1aNaxevRqrVq0S64L9aOHrR0ePHi03z9vbGxKJBO3atZOZnpycjICAADg6OkJPTw/29vb4+eefkZycLFfGli1bUK9ePRgaGqJkyZJo0qQJjh49KrPt7F6ZjzHK9OMAlC5PlTIB+eOt9NW0aVOZ5aTt2sLCAoaGhmjQoAEOHz4ss8wff/yBEiVKyPUXWT/bN2/ewMHBAV5eXkhJSVG5/qXHpazatWsnUxfS/ivrcWX48OGQSCTw9fWVmR4TE4PRo0eL9ebo6Ii5c+eK94/KXKZEIsH+/ftl1k9KSkLJkiWV6jMVfWcAoG3btnL1ld26Ob2k60uPk5nFx8fDxsZGbvv5ca4hLbNTp05y+zFkyBC5Y2N2KlSoIPd57dq1S67953TOUr16dbFdq1KHwOfzWT8/P5QuXRp6enqoVq0aNmzYIFO+tMydO3di0qRJsLGxgZGRETp06IBnz57J1Yui/V6wYEGezjWk5xmqfH6K5HRck1L2HFOZ/iRzfeT2GQDyxxIAmD9/vtwxS/pZ7N69W25bxsbGX/Tdz61tZd6+st/vsLAwNG7cWDx+SF+KjnNfkxKaDiA/HDp0CA4ODvDw8FB6nc6dO6NLly5IS0vDuXPnsGbNGnz69AmbN28GAKSnp6NDhw548uQJRo8ejSpVqkAikWDWrFky5cTFxcHZ2RndunWDoaEhzp07h3nz5iExMRHLli1T2z5qa2tjy5YtGDNmjDgtJCQE+vr6MieiwOc/wNu0aQN3d3cEBARAS0tLPFk+c+YM6tWrB+DzzRYfPHiAzp0748KFC6hYsaLcdqOiotCpUye0a9cOs2fPVhibnZ2deKPG+Ph4/PDDDzLz+/Tpg0GDBuHmzZsyB9FLly7h/v37mDJlSq7737hxYwwePBgZGRm4efMmlixZghcvXuDMmTMAAEEQ0KFDB4SFhWHAgAGoWbMm/vrrL/j7++P58+dYvHgxACAlJQUmJiYYNWoUSpUqhcjISCxbtgzXr1/HjRs3ZLa5adMmxMXFYfjw4UhKSkJwcDCaN2+OGzduoHTp0gCAY8eO4dGjR+jfvz9sbGxw69YtrFmzBrdu3cL58+fFzvzFixeoV68eYmJiMHjwYFStWhXPnz/H7t27kZiYiCZNmohtD4DYzqR/QAEQ2/fx48fRpk0bODg4IDAwEJ8+fcKyZcvg6emJf//9V+4EMy4uDr///js6d+4sTsuu7Shr2rRpeV43p/VDQ0Ph5+eHatWqYeLEiTA3N8eVK1dw5MgR9OrVS1wuc5uT+uOPP7B9+3aZaYGBgZg+fTpatmyJH374Affu3cOqVatw6dIlREREQEdHRy6GwYMHo3HjxgA+J0v27dun8v7Fxsaifv366NevH3R0dHDkyBFMmDABJUqUwLhx4wAAt27dQuPGjWFqaoqff/4ZOjo6WL16NZo2bYpTp06hfv36MmUuXrwYlpaWiI2NxYYNGzBo0CBUqFABLVu2BKB8W/xS2X12mzdvRr9+/eDt7Y25c+ciMTERq1atQqNGjXDlyhWZdlmiRAncunULV65cQa1atcTpoaGh0NfXzzWGPn36YMaMGdi5c6dMp5uSkoLdu3eja9euYjm5tanJkydj4MCBAIC3b99izJgxMm0gq1OnTmHnzp0YOXIk9PT0sHLlSvj4+ODixYvi8U3VzzZrPWY9FgGfk2iNGzeGjo4OBg8ejAoVKiAyMhKHDh3CrFmz0KVLFzg6OorLjxkzBs7Ozhg8eLA4zdnZWfx/1apVMXnyZAiCgMjISMyYMQPa2toy/eiqVatQrVo1dOjQASVKlMChQ4cwbNgw8USL/Wjh6kf19fWxdetWzJ8/Xzz2/ffffzhx4oTc9y4jIwMdOnTA2bNnMXjwYDg7O+PGjRtYvHgx7t+/L/MH9fTp0xEYGAgPDw/MmDEDurq6uHDhAk6ePInWrVtjyZIliI+PBwDcuXMHs2fPxqRJk8T2KE2qKtuPS7Vq1Qp9+/aVmbZw4UJ8+PBBfK9qmVLS4y0Aufb66tUreHh4IDExESNHjkSpUqWwceNGdOjQAbt37xb72m+++QaLFi3CmDFjUKVKFXTo0EFuO8nJyejUqRO0tbWxb98+6Orqqlz/X+Lhw4dYu3at3PTExER4eXnh+fPnGDJkCMqVK4e///4bEydORHR0NJYsWSKzvL6+PkJCQmSSB3v37v2i84TTp0/jjz/+yHU5Z2dnmfOnNWvW4M6dOzKfrZubW7brL1y4EK9evVI4Lz/ONfT19XH48GG8fv0a1tb
"text/plain": [
"<Figure size 1000x600 with 4 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"\n",
"plt.figure(figsize=(10, 6))\n",
"\n",
"# Связь между возрастом и использованием устройства для обучения\n",
"plt.subplot(2, 2, 1)\n",
"sns.scatterplot(data=df3, x='Age', y='Device')\n",
"plt.title('Связь между возрастом и использованием устройства')\n",
"plt.xlabel('Возраст')\n",
"plt.ylabel('Устройство')\n",
"\n",
"# Связь между местоположением и типом интернет-соединения\n",
"plt.subplot(2, 2, 2)\n",
"top_locations = df3['Location'].value_counts().index[:10]\n",
"sns.boxplot(data=df3[df3['Location'].isin(top_locations)], x='Location', y='Internet Type')\n",
"plt.title('Связь между местоположением и типом интернет-соединения')\n",
"plt.xticks(rotation=90)\n",
"plt.xlabel('Местоположение')\n",
"plt.ylabel('Тип интернета')\n",
"\n",
"# Связь между финансовым состоянием и гибкостью обучения\n",
"plt.subplot(2, 2, 3)\n",
"sns.boxplot(data=df3, x='Financial Condition', y='Flexibility Level')\n",
"plt.title('Связь между финансовым состоянием и гибкостью обучения')\n",
"plt.xticks(rotation=90)\n",
"plt.xlabel('Финансовое состояние')\n",
"plt.ylabel('Уровень гибкости')\n",
"\n",
"# Связь между типом учебного заведения и использованием устройства\n",
"plt.subplot(2, 2, 4)\n",
"top_institutions = df3['Institution Type'].value_counts().index[:10]\n",
"sns.boxplot(data=df3[df3['Institution Type'].isin(top_institutions)], x='Institution Type', y='Device')\n",
"plt.title('Связь между типом учебного заведения и устройством для обучения')\n",
"plt.xticks(rotation=90)\n",
"plt.xlabel('Тип учебного заведения')\n",
"plt.ylabel('Устройство')\n",
"\n",
"plt.tight_layout()\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Примеры бизнес-целей<h3>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Образовательные платформы могут использовать эти данные для анализа, какие устройства и типы интернет-соединений студенты используют чаще всего, чтобы оптимизировать свои курсы под предпочтительные форматы обучения.\n",
"2. Учебные заведения могут использовать эти данные для анализа возрастных моделей использования онлайн-обучения, что поможет лучше планировать программы и курсы для различных возрастных групп.\n",
"3. Компании в сфере EdTech могут использовать данные для выявления стран и регионов, где онлайн-обучение наиболее востребовано, и нацелить свои маркетинговые и инвестиционные усилия на эти рынки."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Цели технического проекта<h3>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Вход: Данные о б
"Целевой признак: Устройство, тип интернета.\n",
"\n",
"2. Вход: Данные о
"Целевой признак: Возраст, устройство.\n",
"\n",
"3. Вход: Данные о
"Целевой признак: Местоположение, тип интернета."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h3>Выявление и решение проблем<h3>"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAHWCAYAAACR5EiaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAn3ElEQVR4nO3deXSUZZ638W8CSSVsiZAQSEMSdlQ2RUDksDMsYgPKjIqAoKCI0Cg6PbbSAmI7TIPdKotInwmEHtRmaaRbEJrQLI0cQNlkGDAGCKDDkmFLwhJCyP3+oamXorJACFT4cX3OydFUPXXnfu48VblS9VQIcs45AQAAGBUc6AkAAADcTMQOAAAwjdgBAACmETsAAMA0YgcAAJhG7AAAANOIHQAAYBqxAwAATCN2gADIy8vTiRMndODAgUBPBQDMI3aAW+TYsWN6+eWXFR8fr9DQUEVHR+uee+5RZmZmoKcGAKaVD/QEgEBLSkrSM88843NZdHS07r33Xv3bv/2bevXqdcNfY9++fercubMuXbqkMWPG6P7771f58uUVHh6uihUr3vD4AIDCETvATyZNmqQ6derIOafjx48rKSlJDz/8sD7//HM98sgjNzT2iBEjFBoaqs2bN+tnP/tZKc0YAHAtiB3gJ7169dIDDzzg/XzYsGGKiYnRp59+ekOxs23bNq1Zs0arVq0idAAgADhnByhEZGSkwsPDVb78//+d4ODBgwoKCtK7775b6O0mTpyooKAg7+ebN29WWFiY9u/fr3vvvVcej0c1atTQiBEjdOrUKZ/bdurUSU2aNNG2bdv00EMPKTw8XHXq1NFHH33k93XS09O9QRYWFqbmzZtr3rx5ftvl5eXpgw8+UNOmTRUWFqbo6Gj17NlTW7du9W4TFBSk0aNHF7pPSUlJCgoK0sGDBwvdRpKGDh2qoKCgQj/WrVvns/2iRYvUsmVLhYeHKyoqSoMGDdL//u//Fvk1rpxP/keFChXUtGlT/ed//qfftmvWrFH79u1VsWJFRUZGqm/fvtq7d6/PNrNmzVLz5s0VERGhihUrqnnz5kpMTPTbt0qVKunAgQPq0aOHKlasqNjYWE2aNEnOOZ9t3333XT300EOqVq2awsPD1bJlSy1evLjAfZk/f75at26tChUq6K677lKHDh20atUqSVJCQkKR65mQkCDnnBISEtS3b1+/sbOzsxUREaERI0YUu6aAZTyzA/wkIyNDJ06ckHNO6enpmj59us6ePatBgwbd0LgnT55Udna2Ro4cqS5duuiFF17Q/v37NXPmTG3ZskVbtmyRx+Pxbn/69Gk9/PDDevzxxzVgwAAtXLhQI0eOVGhoqJ599llJ0oULF9SpUyft27dPo0ePVp06dbRo0SINHTpUZ86c0UsvveQdb9iwYUpKSlKvXr00fPhw5ebmasOGDdq8ebPPM1mlxePx+EXH119/rWnTpvlcln+uVKtWrTR58mQdP35cH3zwgTZu3KgdO3YoMjKy2K/13nvvKSoqSpmZmZozZ46ee+45JSQkqFu3bpKk1atXq1evXqpbt64mTpyoCxcuaPr06WrXrp22b9+uhIQESVJWVpa6d++uevXqyTmnhQsXavjw4YqMjFT//v29X+/y5cvq2bOnHnzwQU2ZMkUrV67UhAkTlJubq0mTJnm3++CDD9SnTx8NHDhQOTk5+tOf/qR/+Zd/0bJly9S7d2/vdm+99ZYmTpyohx56SJMmTVJoaKi2bNmiNWvWqHv37nr//fd19uxZSdLevXv17//+73rjjTd09913S5IqVaqkoKAgDRo0SFOmTNGpU6dUtWpV7/iff/65MjMzb/gYBm57DrjDzZ0710ny+/B4PC4pKcln27S0NCfJTZ06tdDxJkyY4K68a+V/3rVrV5ebm+v3dadPn+69rGPHjk6S+93vfue97OLFi65FixauevXqLicnxznn3Pvvv+8kufnz53u3y8nJcW3btnWVKlVymZmZzjnn1qxZ4yS5MWPG+M0zLy/P+/+S3KhRo4pdo7S0tEK3cc65IUOGuIoVK/pdvmjRIifJrV271jvX6tWruyZNmrgLFy54t1u2bJmT5MaPH1/k1yloPt99952T5KZMmeK9LH/dTp486b3sm2++ccHBwe7pp58udPzc3FxXpUoVN3r0aJ99k+R+8YtfeC/Ly8tzvXv3dqGhoe7//u//vJefP3/eZ7ycnBzXpEkT16VLF+9lqampLjg42D366KPu8uXLPttf+b3Jt3btWp81vFJKSoqT5GbNmuVzeZ8+fVxCQkKB4wF3El7GAn4yc+ZMJScnKzk5WfPnz1fnzp01fPhwLVmyxG/b8+fP68SJEzp9+rTfSxiFeeWVV1SuXDnv54MHD1ZMTIyWL1/us1358uV9XnYIDQ3ViBEjlJ6erm3btkmSvvjiC9WoUUMDBgzwbhcSEqIxY8bo7NmzWr9+vSTpz3/+s4KCgjRhwgS/+Vz5Upv040seJ06c0MmTJ5WXl3dN+1RSW7duVXp6ul588UWFhYV5L+/du7caN27styaFOX36tPfvFb333nsqV66cOnbsKEk6evSodu7cqaFDh/o829GsWTP90z/9k7744gufsS5fvqwTJ07o0KFDeu+995SZman27dv7fc0rX+7Lf/kvJydHq1ev9l4eHh7uM8eMjAy1b99e27dv916+dOlS5eXlafz48QoO9n0ovvp7U5yGDRuqTZs2+vjjj72XnTp1SitWrNDAgQOvezzAGmIH+Enr1q3VrVs3devWTQMHDtTy5ct1zz33eH+YXWnChAmKjo5W1apVVaFCBfXu3VupqakFjpv/g6Zx48Y+l5crV04NGjTwOw8mNjbW7+3oDRs2lCTvtocOHVKDBg38fkjmv7xx6NAhSdL+/fsVGxvr88O+MImJiYqOjlZUVJTCw8PVoUMHn/N6SlP+/Bo1auR3XePGjb3XF+f+++9XdHS06tWrpzlz5mjGjBlq3bp1sV/j7rvv1okTJ3Tu3DnvZampqYqOjlZCQoLGjRunDz/8UI8//rjP7YKDg1W3bl2fy67+3kjSsmXL9OCDDyosLExVq1ZVdHS0Zs2apYyMDO82+/fvV3BwsO65555r2tfiPP3009q4caN3vxctWqRLly5p8ODBpTI+cDsjdoBCBAcHq3Pnzjp69KhfyDz//PNKTk7WypUrNXnyZG3cuFH9+vUrcJwrf8svy/r27avk5GStWrVK06dP14EDB/TII48oKysr0FMr1Pz585WcnKzPP/9cL774okaNGqWkpKQSjRUXF6fk5GQtWbJEgwYN0tixY7Vs2bLrHmfDhg3q06ePwsLC9OGHH+qLL75QcnKynnrqqWt+FrAknnzySYWEhHif3Zk/f74eeOCBAmMPuNNwgjJQhNzcXEnyniSar0GDBt6TYHv06KHz589r3LhxOnz4sN8YderUkSSlpKT4PCuQl5en1NRU3XfffT7bHzlyROfOnfN5due7776TJO8JtfHx8dq1a5fy8vJ8nt359ttvvddLUr169fS3v/3N78TVgtSqVcu7T9KPJ78OHDhQO3bsKPJ2JZE/v5SUFHXp0sXnupSUFO/1xWnXrp13TR555BH9z//8jyZPnqyhQ4f6fI2rffvtt4qKivJZ4woVKnj3/9FHH9XBgwf19ttv+/zZgby8PB04cMD7bI7k/73585//rLCwMP3tb3/zOfF87ty5PnOoV6+e8vLytGfPHrVo0eKa9rcoVatWVe/evfXxxx9r4MCB2rhxo95///0bHhewgGd2gEJcunRJq1atUmhoqPflocLkn+Ny5Tk5+bp27SqPx6Np06b5nAvz8ccf6/jx435/wyc3N1ezZ8/2fp6Tk6PZs2crOjpaLVu2lCQ9/PDDOnbsmBYsWOBzu+nTp6tSpUre81b69+8v55zeeustv3kV9yxDUft0ox544AFVr15dH330kS5evOi9fMWKFdq7d6/PO5aux4ULF7zj1axZUy1atNC8efN05swZ7za7d+/WqlWr9PDDDxc6zuXLl3X69GmfueWbMWOG9/+dc5oxY4ZCQkLUtWtXST+uV1BQkC5fvuz
"text/plain": [
"<Figure size 700x500 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Размер данных до удаления выбросов: (1205, 11)\n",
"Размер данных после удаления выбросов: (1205, 11)\n"
]
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"import numpy as np\n",
"import pandas as pd \n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"df3 = pd.read_csv(\"C:/Users/TIGR228/Desktop/МИИ/Lab1/AIM-PIbd-31-Afanasev-S-S/static/csv/students_adaptability_level_online_education.csv\")\n",
"fig, axs = plt.subplots(1, 1, figsize=(7, 5))\n",
"\n",
"sns.boxplot(data=df3, x='Age', ax=axs)\n",
"axs.set_title(\"Выбросы по возрасту\")\n",
"\n",
"plt.show()\n",
"\n",
"print(\"Размер данных до удаления выбросов: \", df3.shape)\n",
"\n",
"# Функция для удаления выбросов с
"def remove_outliers(df, column):\n",
" Q1 = df[column].quantile(0.25)\n",
" Q3 = df[column].quantile(0.75)\n",
" IQR = Q3 - Q1\n",
" return df[~((df[column] < (Q1 - 1.5 * IQR)) | (df[column] > (Q3 + 1.5 * IQR)))]\n",
"\n",
"# Удаление выбросов по возрасту\n",
"df3_cleaned = remove_outliers(df3, 'Age')\n",
"\n",
"print(\"Размер данных после удаления выбросов: \", df3_cleaned.shape)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_15252\\1872101371.py:7: FutureWarning: \n",
"\n",
"Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect.\n",
"\n",
" sns.barplot(x=location_dist.values, y=location_dist.index, palette='coolwarm')\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1wAAAIjCAYAAAAX5hpkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLo0lEQVR4nO3deXwN9/7H8fdJRGS3RCRIiaV2tbdE7aqKUkoptRa3qK3W9mepLuhKUVq9aNFaSlHdLEVRNxRRmlRT4lJiF7GUkHx/f/SRcx0nIjkyIvF6Ph551PnOd77zmTmTNO/MzPfYjDFGAAAAAIBM55bVBQAAAABATkXgAgAAAACLELgAAAAAwCIELgAAAACwCIELAAAAACxC4AIAAAAAixC4AAAAAMAiBC4AAAAAsAiBCwAAAAAsQuACAAAAAIsQuID71Lx582Sz2exfefLk0YMPPqgBAwboxIkTWV0eAABAjpArqwsAkLUmTJigsLAwXblyRVu2bNHMmTP17bffat++ffL29s7q8gAAALI1Ahdwn2vevLlq1KghSXr++edVoEABvffee1q5cqU6deqUxdUBAABkb9xSCMBBo0aNJEmxsbGSpLNnz2rYsGGqVKmSfH195e/vr+bNm2vPnj1O6165ckXjx4/Xgw8+qDx58igkJERt27bVgQMHJEmHDh1yuI3x5q8GDRrYx9q4caNsNpsWL16sl19+WcHBwfLx8dGTTz6pI0eOOG07IiJCjz/+uAICAuTt7a369etr69atqe5jgwYNUt3++PHjnfouWLBA1atXl5eXl/Lnz6+OHTumuv209u1GycnJmjJliipUqKA8efKoUKFC6tu3r86dO+fQr3jx4mrZsqXTdgYMGOA0Zmq1v/32207HVJKuXr2qcePGqVSpUvL09FRoaKhGjBihq1evpnqsbhYREaEnnnhC+fLlk4+PjypXrqypU6dKkrp3757m+2uz2XTo0CF169ZNgYGBunbtmtP4jz32mMqUKePQlnIu3O7YStLRo0fVs2dPFSpUSJ6enqpQoYLmzJmT6nhffvml0/q+vr7q3r27JOfbblP7mjdvnn3dH3/8UY8++qh8fHyUN29etW7dWtHR0Q7jjx8/3mF9Pz8/1apVSytWrLjtsb953bRqSW89qbnxeEdGRjosO3r0qNzd3VM9fr///ruefvpp5c+fX3ny5FGNGjW0atUqp/Hj4+M1ZMgQFS9eXJ6enipatKi6du2q06dPp/lep/Z9unv3bjVv3lz+/v7y9fVV48aN9Z///Mdhe7d7H2/+3knPmClu9bPk5vdi6dKl9p8jgYGB6tKli44ePerQ59lnn1WBAgUUExPjVPuhQ4ccxnJzc9NHH32U4eOfMt4vv/zi0H769GmnY5Fyvt3o4sWLCg4Ols1m08aNG+3t3bt3V/HixR36TpkyRWXLlpWnp6eCg4PVt29fnT17NrXDCOR4XOEC4CAlHBUoUECSdPDgQa1YsULt27dXWFiYTpw4oY8++kj169dXVFSUChcuLElKSkpSy5YttX79enXs2FGDBg3ShQsXtHbtWu3bt08lS5a0b6NTp0564oknHLY7evToVOt54403ZLPZNHLkSJ08eVJTpkxRkyZNFBkZKS8vL0n//GLZvHlzVa9eXePGjZObm5vmzp2rRo0aafPmzapVq5bTuEWLFtXEiRMl/fNLxAsvvJDqtseMGaMOHTro+eef16lTpzRt2jTVq1dPu3fvVt68eZ3W6dOnjx599FFJ0vLly/XVV185LO/bt6/mzZunHj16aODAgYqNjdX06dO1e/dubd26VR4eHqkeh4yIj4+379uNkpOT9eSTT2rLli3q06ePypUrp7179+r999/XH3/8cdtf+teuXauWLVsqJCREgwYNUnBwsKKjo7V69WoNGjRIffv2VZMmTez9n3vuOT311FNq27atva1gwYJ67rnn9Nlnn+mHH35wCJXHjx/Xjz/+qHHjxqW6/YEDB6pmzZqSpM8++0xr1651WH7ixAk98sgjstlsGjBggAoWLKjvvvtOvXr1UkJCggYPHny7Q+egXr16mj9/vv31G2+8IUl65ZVX7G116tSRJK1bt07NmzdXiRIlNH78eP3999+aNm2awsPDtWvXLqdfRlPGPX36tD788EO1b99e+/btcwqbqZk5c6Z8fX3tr2NjYzV27FiHPhmtJzV58uTR3Llz7YFakj799FPlzp1bV65ccej722+/KTw8XEWKFNGoUaPk4+OjJUuWqE2bNlq2bJmeeuopSf98rz366KOKjo5Wz549Va1aNZ0+fVqrVq3SX3/9pXLlyjkc848//ljR0dF6//337W2VK1e2b/PRRx+Vv7+/RowYIQ8PD3300Udq0KCBNm3apIcfftihxpTbp1Ok9n2f0TElqWzZsvZz4vTp0xoyZIjD8pTv95o1a2rixIk6ceKEpk6dqq1btzr8HJkzZ44aNWqkFi1aKCIiQvny5XPa1vbt29WtWzcNGTJEffv2zfDxv1Pvvvtuup7xffPNN/XKK6+oXr166t+/v/3nXEREhCIiIuTp6Zkp9QDZhgFwX5o7d66RZNatW2dOnTpljhw5YhYtWmQKFChgvLy8zF9//WWMMebKlSsmKSnJYd3Y2Fjj6elpJkyYYG+bM2eOkWTee+89p20lJyfb15Nk3n77bac+FSpUMPXr17e/3rBhg5FkihQpYhISEuztS5YsMZLM1KlT7WOXLl3aNGvWzL4dY4y5fPmyCQsLM02bNnXaVp06dUzFihXtr0+dOmUkmXHjxtnbDh06ZNzd3c0bb7zhsO7evXtNrly5nNpjYmKMJPPpp5/a28aNG2du/DG7efNmI8ksXLjQYd3vv//eqb1YsWKmRYsWTrX379/f3Pyj++baR4wYYYKCgkz16tUdjun8+fONm5ub2bx5s8P6s2bNMpLM1q1bnbaX4vr16yYsLMwUK1bMnDt3zmHZjcc9rbpSJCUlmaJFi5pnnnnGof29994zNpvNHDx40KF9zZo1RpL58ssv7W2pHYdevXqZkJAQc/r0aYf2jh07moCAAHP58mVjzP/OraVLlzrV5uPjY7p165bq/tSvX9/heN6oSpUqJigoyJw5c8betmfPHuPm5ma6du1qb7v5nLhx/5YsWZLq2Deve+rUKYf2HTt2GElm7ty5Ga4nNSnHp1OnTqZAgQLm6tWr9mWlS5c2zz77rNPxa9y4salUqZK5cuWKvS05OdnUqVPHlC5d2t42duxYI8ksX77cabupnUfdunUzxYoVS7XONm3amNy5c5sDBw7Y244dO2b8/PxMvXr17G0pP+t27NjhsH5q3/fpHTNFeHi4adiwof11ys+4lPciMTHRBAUFmYoVK5q///7b3m/16tVGkhk7dqzDeCdOnDDFixc3DRs2NImJifbaY2NjzeHDh01wcLB58sknnX4mp/f4Z+RY3Hyunjx50vj5+ZnmzZsbSWbDhg32ZTe+T6dOnTJ58uQxdevWNdeuXbP3mTdvnpFkpk2b5nQcgZyOWwqB+1yTJk1UsGBBhYaGqmPHjvL19dVXX32lIkWKSJI8PT3l5vbPj4qkpCSdOXNGvr6+KlOmjHbt2mUfZ9myZQoMDNSLL77otI3Ubv1Kr65du8rPz8/++umnn1ZISIi+/fZbSVJkZKRiYmL07LPP6syZMzp9+rROnz6tS5cuqXHjxvrpp5+UnJzsMOaVK1eUJ0+eNLe7fPlyJScnq0OHDvYxT58+reDgYJUuXVobNmxw6J+YmChJaf7ldunSpQoICFDTpk0dxqxevbp8fX2dxrx27ZpDv9OnTztdWbjZ0aNHNW3aNI0ZM8bhKkjK9suVK6eyZcs6jJlyG+nN27/R7t27FRsbq8GDBztd2cvo++vm5qbOnTtr1apVunDhgr194cKFqlOnjsNVCEn2fU7rPTPGaNmyZWrVqpWMMQ7716xZM50/f97hfJWkCxcuOB1fV8TFxSkyMlLdu3dX/vz57e2VK1dW06ZN7efqjVK2Fx0drVmzZsnHx0ePPPKIS9vPjHpS06pVK9lsNvttaZs3b9Zff/2lZ555xqH
"text/plain": [
"<Figure size 1000x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_15252\\1872101371.py:16: FutureWarning: \n",
"\n",
"Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect.\n",
"\n",
" sns.barplot(x=institution_dist.values, y=institution_dist.index, palette='coolwarm')\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAAIjCAYAAAB2/jgmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABWD0lEQVR4nO3dd3gU5d7/8c8mIYVUIAlJIEAg9CIYFAEpShQQpCkIoiQgyvEBQVEpx0JRxIIKWLBwTjgKRxSVoqJ0pDw0hdCbEEQgGFoIoZPcvz/4ZR+WFLKwMSP7fl3XXld25p6Z7947u8knM3OPzRhjBAAAAAAodh7FXQAAAAAA4DICGgAAAABYBAENAAAAACyCgAYAAAAAFkFAAwAAAACLIKABAAAAgEUQ0AAAAADAIghoAAAAAGARBDQAAIB8GGN0/Phx7d69u7hLAeAmCGgAAABXOHXqlF588UVVr15d3t7eKlOmjKpVq6adO3cWd2koBlOnTtW+ffvsz6dMmaKDBw8WX0G46RHQAIuZMmWKbDab/eHr66tq1appwIAB+vPPP4u7PAC4qR07dkyNGzfWxIkT9eCDD2r27NlasGCBli5dqkqVKhV3eSgGy5cv15AhQ7Rv3z7NmzdP/fv3l4cHf0Kj6HgVdwEA8jZ69GjFxMTo3LlzWrFihSZNmqS5c+dqy5YtKlmyZHGXBwA3peeff16pqalatWqVateuXdzlwAKeeeYZtWzZUjExMZKkwYMHKzIyspirws2MgAZYVNu2bdWwYUNJUt++fVWmTBm98847mj17tnr06FHM1QHAzSctLU3/+c9/9NFHHxHOYFejRg3t2bNHW7ZsUWhoqKpUqVLcJeEmx/FZ4G/i7rvvliSlpKRIko4fP67nnntOdevWVUBAgIKCgtS2bVtt3Lgx17Lnzp3TyJEjVa1aNfn6+ioyMlJdunTRnj17JEn79u1zOK3y6kfLli3t61q6dKlsNpu+/PJL/fOf/1RERIT8/f3VoUMH/fHHH7m2vWbNGrVp00bBwcEqWbKkWrRooZUrV+b5Glu2bJnn9keOHJmr7dSpUxUXFyc/Pz+VLl1a3bt3z3P7Bb22K2VnZ2v8+PGqXbu2fH19VbZsWfXr108nTpxwaFepUiW1b98+13YGDBiQa5151f7WW2/l6lNJOn/+vEaMGKHY2Fj5+PgoOjpaQ4YM0fnz5/Psq6utWbNG9913n0qVKiV/f3/Vq1dPEyZMkCQlJiYW+P7abDbt27dPCQkJCg0N1cWLF3Ot/95771X16tUdpuXsC9fqW0k6ePCg+vTpo7Jly8rHx0e1a9fWv//97zzX9/XXX+daPiAgQImJiZJynwac12PKlCn2ZRcvXqxmzZrJ399fISEh6tixo7Zv3+6w/pEjRzosHxgYqNtvv12zZs26Zt9fvWxBtRS2nqsV1NdXf05y6rlSZmamIiIiZLPZtHTpUvv0/D5zV+4XOWbPnq127dopKipKPj4+qlKlil555RVlZWU5bCtnnZ06dcr1Ovr16yebzaY6deoU+HoLuy/m9MuVr0m6vM9ffTpiYT7j69atU3Z2ti5cuKCGDRvK19dXZcqUUY8ePbR///5c2wgICMhV39dff51nPxf0mnO+p3L2lbS0NIWFhally5Yyxtjb/fbbb/L399dDDz2U77okadKkSbrlllsUHBwsf39/3XLLLfrXv/7l0GbTpk1KTExU5cqV5evrq4iICPXp00fHjh1zaOfMZ6M4v+/z6+Nx48bl2pcrVapk/z7JMWPGDNlsNof95sr3xd/fX40aNVKVKlXUv39/2Wy2XOsAXIUjaMDfRE6YKlOmjCRp7969mjVrlrp27aqYmBj9+eef+vjjj9WiRQtt27ZNUVFRkqSsrCy1b99eixYtUvfu3TVo0CCdOnVKCxYs0JYtWxz+E9ijRw/dd999DtsdPnx4nvWMGTNGNptNQ4cOVVpamsaPH6/4+HglJyfLz89P0uU/RNu2bau4uDiNGDFCHh4eSkpK0t13363ly5fr9ttvz7Xe8uXLa+zYsZIu/1H55JNP5rntl156Sd26dVPfvn115MgRvffee2revLk2bNigkJCQXMs88cQTatasmSTp22+/1cyZMx3m9+vXT1OmTFHv3r01cOBApaSk6P3339eGDRu0cuVKlShRIs9+cEZ6err9tV0pOztbHTp00IoVK/TEE0+oZs2a2rx5s959913t2rXrmiFhwYIFat++vSIjIzVo0CBFRERo+/bt+v777zVo0CD169dP8fHx9vaPPvqoOnfurC5dutinhYWF6dFHH9Vnn32mefPmOYTQw4cPa/HixRoxYkSe2x84cKBuu+02SdJnn32mBQsWOMz/888/dccdd8hms2nAgAEKCwvTjz/+qMcee0wZGRl6+umnr9V1Dpo3b67PP//c/nzMmDGSpBdeeME+rUmTJpKkhQsXqm3btqpcubJGjhyps2fP6r333lPTpk21fv36XH/E56z36NGj+vDDD9W1a1dt2bIlVzjNy6RJkxz+YE9JSdHLL7/s0MbZenLUrFnT4TV/8skn2r59u9599137tHr16uVb29tvv53vNaxXfuZyzJ07V1988YXDtClTpiggIECDBw9WQECAFi9erJdfflkZGRl66623HNr6+vrqhx9+UFpamsLDwyVJZ8+e1ZdffilfX99868xxvftiQQrzGc8JJwMGDFBcXJxef/11HTlyRBMnTtSKFSu0YcMGhYaGOr1tZ4WHh2vSpEnq2rWr3nvvPQ0cOFDZ2dlKTExUYGCgPvzwwwKXP3XqlO69915VqVJFxhh99dVX6tu3r0JCQvTAAw9Iuvy9sXfvXvXu3VsRERHaunWrPvnkE23dulWrV6/OFfKv9dmw0ve9sy5duuTw/VGQ3377TZ9++ukNbxMokAFgKUlJSUaSWbhwoTly5Ij5448/zPTp002ZMmWMn5+fOXDggDHGmHPnzpmsrCyHZVNSUoyPj48ZPXq0fdq///1vI8m88847ubaVnZ1tX06Seeutt3K1qV27tmnRooX9+ZIlS4wkU65cOZORkWGf/tVXXxlJZsKECfZ1V61a1bRu3dq+HWOMOXPmjImJiTH33HNPrm01adLE1KlTx/78yJEjRpIZMWKEfdq+ffuMp6enGTNmjMOymzdvNl5eXrmm796920gy//nPf+zTRowYYa78+lu+fLmRZKZNm+aw7E8//ZRresWKFU27du1y1d6/f39z9Vfq1bUPGTLEhIeHm7i4OIc+/fzzz42Hh4dZvny5w/IfffSRkWRWrlyZa3s5Ll26ZGJiYkzFihXNiRMnHOZd2e8F1ZUjKyvLlC9f3jz00EMO09955x1js9nM3r17HabPnz/fSDJff/21fVpe/fDYY4+ZyMhIc/ToUYfp3bt3N8HBwebMmTPGmP/bt2bMmJGrNn9/f5OQkJDn62nRooVDf16pfv36Jjw83Bw7dsw+bePGjcbDw8P06tXLPu3qfeLK1/fVV1/lue6rlz1y5IjD9HXr1hlJJikpyel6riUhIcFUrFixwHpypKWlmcDAQNO2bVsjySxZssQ+r0WLFqZ27dq51vHWW28ZSSYlJcU+Led9ulK/fv1MyZIlzblz53Kts169embcuHH26Z9//rkpX768adasWZ7bvFJh98Wff/7ZSDKLFy92aHd1/xT2M57z/VurVi2H15uzbz777LMO2/D3989V+4wZMwrdzzlyvoOv3FeMMaZHjx6mZMmSZteuXfb3ZNasWfmuJz+XLl0yQUFBZsCAAfZpeb2fX3zxhZFkli1bZp9WmM+GFb7vndmXK1as6PB98uGHHxofHx9z1113Oew3eb0v3bp1M3Xq1DHR0dH5ficBN4pTHAGLio+PV1hYmKKjo9W9e3cFBARo5syZKleunCTJx8fHPopUVlaWjh07poCAAFWvXl3r16+3r+ebb75RaGionnrqqVzbyOtUtMLq1auXAgMD7c8ffPBBRUZGau7cuZKk5ORk7d69Ww8//LCOHTumo0eP6ujRozp9+rRatWqlZcuWKTs722Gd586du+Z/17/99ltlZ2erW7d
"text/plain": [
"<Figure size 1000x600 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"(Location\n",
" Town 77.593361\n",
" Rural 22.406639\n",
" Name: proportion, dtype: float64,\n",
" Institution Type\n",
" Private 68.298755\n",
" Public 31.701245\n",
" Name: proportion, dtype: float64)"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"\n",
"# Визуализация распределения студентов по местоположению\n",
"plt.figure(figsize=(10, 6))\n",
"location_dist = df3['Location'].value_counts()\n",
"sns.barplot(x=location_dist.values, y=location_dist.index, palette='coolwarm')\n",
"plt.title('Распределение студентов по местоположению')\n",
"plt.xlabel('Количество студентов')\n",
"plt.ylabel('Местоположение')\n",
"plt.show()\n",
"\n",
"# Визуализация распределения студентов по типу учебных заведений\n",
"plt.figure(figsize=(10, 6))\n",
"institution_dist = df3['Institution Type'].value_counts()\n",
"sns.barplot(x=institution_dist.values, y=institution_dist.index, palette='coolwarm')\n",
"plt.title('Распределение студентов по типам учебных заведений')\n",
"plt.xlabel('Количество студентов')\n",
"plt.ylabel('Тип учебного заведения')\n",
"plt.show()\n",
"\n",
"# Процентное распределение по местоположению и типам учебных заведений (для наглядности)\n",
"location_percentage = df3['Location'].value_counts(normalize=True) * 100\n",
"institution_percentage = df3['Institution Type'].value_counts(normalize=True) * 100\n",
"\n",
"location_percentage.head(10), institution_percentage.head(10)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Имеется смещение по местоположению или типам учебных заведений. Н о а
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(3, 2, 2, np.int64(980))"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 1. Проверим уникальность по ключевым столбцам\n",
"unique_devices = df3['Device'].nunique()\n",
"unique_locations = df3['Location'].nunique()\n",
"unique_institution_types = df3['Institution Type'].nunique()\n",
"\n",
"# 2. Проверка дубликатов\n",
"duplicates_count = df3.duplicated().sum() \n",
"\n",
"unique_devices, unique_locations, unique_institution_types, duplicates_count\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Таким образом, можно сказать, что наш набор данных демонстрирует высокую информативность, что позволяет проводить различные виды анализов."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_15252\\2241712000.py:7: FutureWarning: \n",
"\n",
"Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `y` variable to `hue` and set `legend=False` for the same effect.\n",
"\n",
" sns.countplot(data=df3, y='Institution Type', order=df3['Institution Type'].value_counts().index, palette='coolwarm')\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAAHWCAYAAAACSaoRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABR1UlEQVR4nO3de3zP9f//8ft7mx3YCTPbmBlzyqEDJeQ8IUJKKeXUQX2UYzlUcijpgFDooOYTSumASjkrfBxjzmkxEdOcZkaM7fn7w2/vr7cd7M179sr7dr1c3pfL3s/X8/V6Pd7Pvfbe7nu9Xs+3zRhjBAAAAAAodB6FXQAAAAAA4CICGgAAAABYBAENAAAAACyCgAYAAAAAFkFAAwAAAACLIKABAAAAgEUQ0AAAAADAIghoAAAAAGARBDQAAFBgLly4oOTkZO3fv7+wSwGAfwUCGgAAcKmEhAQ9+eSTCg8Pl7e3t0qXLq169erJGFPYpaEQvP7668rMzJQkZWZmasyYMYVcEWBtBDQALjV9+nTZbDb7w9fXV5UrV9azzz6rv//+u7DLA1DA1q5dqzvuuEPLli3TkCFDtHDhQi1evFhz586VzWYr7PJQCP773/9q7Nix+uuvvzRu3Dj997//LeySAEuzGf6dBcCFpk+frh49emjUqFGKjo7W2bNntWrVKs2YMUNRUVHavn27ihYtWthlAigA6enpuvnmmxUYGKhFixYpKCiosEuCBXzxxRfq2rWr0tPT5ePjo5kzZ+qBBx4o7LIAy/Iq7AIA3Jhat26tOnXqSJKeeOIJlSxZUuPHj9e8efP08MMPF3J1AArCd999p927d+u3334jnMHuoYceUtOmTfXHH3+oUqVKKlWqVGGXBFgalzgCuC6aNWsmSUpMTJQkHT9+XM8//7xq1qwpf39/BQYGqnXr1tqyZUu2dc+ePasRI0aocuXK8vX1VXh4uDp27Kg9e/ZIkvbt2+dwWeXljyZNmti3tWLFCtlsNn3xxRd68cUXFRYWpmLFiqldu3Y6cOBAtn2vW7dOrVq1UlBQkIoWLarGjRtr9erVOb7GJk2a5Lj/ESNGZOs7c+ZM1a5dW35+fipRooQ6d+6c4/7zem2XyszM1IQJE1S9enX5+vqqdOnS6tWrl06cOOHQr3z58mrbtm22/Tz77LPZtplT7W+//Xa2MZWkc+fOafjw4YqJiZGPj48iIyM1aNAgnTt3LsexulRu45b12Ldvn0P/KVOmqHr16vLx8VFERIR69+6tlJSUPPcxYsSIPPdhs9m0YsUKez2Xv74NGzbkOO55ba98+fIOfceOHav69eurZMmS8vPzU+3atfXVV19lqzVr/QkTJmRbVrVqVdlsNj377LO5vlZjjMqXL6/27dtnW3b27FkFBQWpV69eDuNyufLly6t79+4ObSkpKerXr58iIyPl4+OjmJgYvfnmm/Z7i6SLlzdGR0fr66+/VsWKFeXt7a1y5cpp0KBB+ueff7Ltw5ljMa/XnHVpddaxsmzZMnl4eOiVV15x6PfZZ5/JZrNp6tSpuW5Lkp5++mlVqlRJRYsWVYkSJdSsWTOtXLnSoc+8efPUpk0bRUREyMfHRxUrVtSrr76qjIwMh36XH98hISFq06aNtm/fnm2/+X1fyBqTvI7jLPn92cxtjNu2betwLGe9J02fPt2hX+/evWWz2RyOm0u/L6Ghofbjv1atWjluA8BFnEEDcF1khamSJUtKkvbu3au5c+eqU6dOio6O1t9//60PPvhAjRs31s6dOxURESFJysjIUNu2bbV06VJ17txZffv21alTp7R48WJt375dFStWtO/j4Ycf1j333OOw36FDh+ZYz+jRo2Wz2TR48GAlJydrwoQJio2NVXx8vPz8/CRd/COvdevWql27toYPHy4PDw/FxcXZ/1i74447sm23bNmy9hvg09LS9Mwzz+S472HDhunBBx/UE088oSNHjujdd99Vo0aNtHnzZgUHB2db56mnnlLDhg0lSd98842+/fZbh+W9evWyX17ap08fJSYm6r333tPmzZu1evVqFSlSJMdxcEZKSkqON/dnZmaqXbt2WrVqlZ566ilVq1ZN27Zt0zvvvKPff/9dc+fOveK2Lx23LAsWLNDnn3/u0DZixAiNHDlSsbGxeuaZZ7R7925NnTpVGzZsyPN1duzYUTExMfbn/fv3V7Vq1fTUU0/Z26pVq5ZrfYMHD851WYsWLdS1a1eHtnHjxmULxxMnTlS7du3UpUsXpaena/bs2erUqZO+//57tWnTxqGvr6+v4uLi1K9fP3vb//73P/3555+51pHFZrPp0Ucf1VtvvaXjx4+rRIkS9mXfffedUlNT9eijj15xO5c6c+aMGjdurIMHD6pXr14qV66c/ve//2no0KFKSkqyh8ljx45p7969evHFF9WxY0cNHDhQGzdu1Ntvv63t27frhx9+uC73oTVr1kz/+c9/NGbMGHXo0EG33XabkpKS9Nxzzyk2NlZPP/10nuunp6fr0UcfVdmyZXX8+HF98MEHatWqlXbt2qVy5cpJuhg+/P39NWDAAPn7+2vZsmV65ZVXlJqaqrffftthe1WrVtVLL70kY4z27Nmj8ePH65577nGY2fJq3hcuPfY2bNigSZMmOSx3xc9mfvzxxx/66KOP8tV3xowZ2rZtm0v2C9ywDAC4UFxcnJFklixZYo4cOWIOHDhgZs+ebUqWLGn8/PzMX3/9ZYwx5uzZsyYjI8Nh3cTEROPj42NGjRplb/vkk0+MJDN+/Phs+8rMzLSvJ8m8/fbb2fpUr17dNG7c2P58+fLlRpIpU6aMSU1Ntbd/+eWXRpKZOHGifduVKlUyLVu2tO/HGGPOnDljoqOjTYsWLbLtq379+qZGjRr250eOHDGSzPDhw+1t+/btM56enmb06NEO627bts14eXlla09ISDCSzH//+1972/Dhw82lb98rV640ksysWbMc1v3pp5+ytUdFRZk2bdpkq713797m8l8Jl9c+aNAgExoaamrXru0wpjNmzDAeHh5m5cqVDuu///77RpJZvXp1tv1dqnHjxqZ69erZ2t9++20jySQmJhpjjElOTjbe3t7m7rvvdjh23nvvPSPJfPLJJ3nu51JRUVGmW7duudZz6etbsGCBkWRatWqV4xj17t072zbatGljoqKiHNrOnDnj8Dw9Pd3UqFHDNGvWLNs2H3jgAePl5WU2btxob3/88cfNI488kus+L7V7924jyUydOtWhvV27dqZ8+fL2Y3rkyJFGksMxbkz28Xn11VdNsWLFzO+//+7Qb8iQIcbT09Ps37/fGGNMt27djCTTvXt3h35Zx+x3333nsA9njsW8XnPW+07WsWKMMadPnzYxMTGmevXq5uzZs6ZNmzYmMDDQ/Pnnn7luJzfr1683ksxXX31lb7v8+2mMMb169TJFixY1Z8+etbddfjwZY8yLL75oJJnk5GRjjPPvC+np6UaSefbZZ+1tc+bMMZLM8uXL7W3O/Gzm91jOer+Ni4uztz344IOmRo0aJjIy0uG4ufz7cvbsWVOuXDnTunXrbNsA8H+4xBFAgYiNjVWpUqUUGRmpzp07y9/fX99++63KlCkjSfLx8ZGHx8W3oIyMDB07dkz+/v6qUqWKNm3aZN/O119/rZCQED333HPZ9nEt/4nv2rWrAgIC7M8feOABhYeHa8GCBZKk+Ph4JSQk6JFHHtGxY8d09OhRHT16VKdPn1bz5s31yy+/OFzaJV28fMzX1zfP/X7zzTfKzMzUgw8+aN/m0aNHFRYWpkqVKmn58uUO/dPT0yVdHK/czJkzR0FBQWrRooXDNmvXri1/f/9s2zx//rxDv6NHj+rs2bN51n3w4EG9++67GjZsmPz9/bPtv1q1aqpatarDNrMua718/1dryZIlSk9PV79+/ezHjiQ9+eSTCgwM1A8//OCS/VzKGKOhQ4fq/vvvV926da9pW1lnZiXpxIkTOnnypBo2bOhwvGcpXbq02rRpo7i4OEkXz2B9+eWX6tGjR772VblyZdWtW1ezZs2ytx0/flw//vijunTpYv/ZCQ0NlST99dd
"text/plain": [
"<Figure size 1000x500 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Проверка на согласованность категорий\n",
"unique_devices = df3['Device'].unique() # Уникальные устройства\n",
"unique_institution_types = df3['Institution Type'].unique() # Уникальные типы учебных заведений\n",
"\n",
"# Пример для визуального анализа распределения по типам учебных заведений\n",
"plt.figure(figsize=(10, 5))\n",
"sns.countplot(data=df3, y='Institution Type', order=df3['Institution Type'].value_counts().index, palette='coolwarm')\n",
"plt.title('Распределение по типам учебных заведений')\n",
"plt.xlabel('Количество студентов')\n",
"plt.ylabel('Тип учебного заведения')\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Для оценки покрытия мы смотрим на то, насколько отличаются данные по типу учебного заведения и количеству студентов."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Устранение проблемы пропущенных данных"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\TIGR228\\AppData\\Local\\Temp\\ipykernel_15252\\2536815725.py:13: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
"The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
"\n",
"For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
"\n",
"\n",
" df_fillna_mean[column].fillna(df_fillna_mean[column].mean(), inplace=True)\n"
]
},
{
"data": {
"text/plain": [
"(Education Level 0\n",
" Institution Type 0\n",
" Gender 0\n",
" Age 0\n",
" Device 0\n",
" IT Student 0\n",
" Location 0\n",
" Financial Condition 0\n",
" Internet Type 0\n",
" Network Type 0\n",
" Flexibility Level 0\n",
" dtype: int64,\n",
" (1205, 11),\n",
" (1205, 11),\n",
" (1205, 11))"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"missing_values = df3.isnull().sum()\n",
"df_dropna = df3.dropna()\n",
"df_fillna_const = df3.fillna(0)\n",
"df_fillna_mean = df3.copy()\n",
"for column in df_fillna_mean.select_dtypes(include=['float64', 'int64']):\n",
" df_fillna_mean[column].fillna(df_fillna_mean[column].mean(), inplace=True)\n",
"missing_values, df_dropna.shape, df_fillna_const.shape, df_fillna_mean.shape\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Разбиение набора данных на обучающую, контрольную и тестовую выборки"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((723, 10), (241, 10), (241, 10))"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.model_selection import train_test_split\n",
"\n",
"# Разделим набор данных на признаки (X) и целевой признак (y)\n",
"X = df3.drop(columns=['Financial Condition'])\n",
"y = df3['Financial Condition']\n",
"\n",
"# Разделение на обучающую, контрольную и тестовую выборки\n",
"X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)\n",
"X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)\n",
"\n",
"# Проверка размера выборок\n",
"(X_train.shape, X_val.shape, X_test.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Оценка сбалансированности выборок"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(count 723\n",
" unique 3\n",
" top Mid\n",
" freq 527\n",
" Name: Financial Condition, dtype: object,\n",
" count 241\n",
" unique 3\n",
" top Mid\n",
" freq 182\n",
" Name: Financial Condition, dtype: object,\n",
" count 241\n",
" unique 3\n",
" top Mid\n",
" freq 169\n",
" Name: Financial Condition, dtype: object)"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Проверка распределения целевого признака по выборкам\n",
"train_dist = y_train.describe()\n",
"val_dist = y_val.describe()\n",
"test_dist = y_test.describe()\n",
"\n",
"train_dist, val_dist, test_dist"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Размеры после oversampling: (1581, 10) (1581,)\n",
"Размеры после undersampling: (156, 10) (156,)\n"
]
}
],
"source": [
"from imblearn.over_sampling import RandomOverSampler\n",
"from imblearn.under_sampling import RandomUnderSampler\n",
"\n",
"oversampler = RandomOverSampler(random_state=12)\n",
"X_train_over, y_train_over = oversampler.fit_resample(X_train, y_train)\n",
"\n",
"undersampler = RandomUnderSampler(random_state=12)\n",
"X_train_under, y_train_under = undersampler.fit_resample(X_train, y_train)\n",
"\n",
"print(\"Размеры после oversampling:\", X_train_over.shape, y_train_over.shape)\n",
"print(\"Размеры после undersampling:\", X_train_under.shape, y_train_under.shape)"
]
2024-10-19 00:46:39 +04:00
}
],
"metadata": {
"kernelspec": {
"display_name": "aimenv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
