AIM-PIbd-32-Katysheva-N-E/lab_2/lab2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Загрузка данных в DataFrame \"Список форбс\"</b>\n",
    "\n",
    "О рейтинге\n",
    "The World's Billionaires (\"Миллиардеры мира\") - ежегодный рейтинг самых богатых миллиардеров мира, составляемый и публикуемый в марте американским деловым журналом Forbes. Общее состояние каждого человека, включенного в список, оценивается в долларах США на основе его документально подтвержденных активов, а также с учетом долгов и других факторов. Этот рейтинг представляет собой список самых богатых людей, зарегистрированных по документам, за исключением тех, чье благосостояние не может быть полностью установлено.\n",
    "\n",
    "Методология Forbes\n",
    "Каждый год Forbes нанимает команду из более чем 50 репортеров из разных стран, чтобы отслеживать деятельность самых богатых людей мира, а иногда и групп или семей, которые делятся богатством. Предварительные опросы рассылаются тем, кто может попасть в список. \n",
    "\n",
    "По данным Forbes, они получили ответы трех типов: одни люди пытаются преувеличить свое богатство, другие сотрудничают, но не раскрывают деталей, а третьи отказываются отвечать на любые вопросы. \n",
    "Затем тщательно изучаются деловые сделки и оценивается стоимость ценных активов – земли, домов, транспортных средств, произведений искусства и т.д. – сделаны.\n",
    "\n",
    "Для проверки данных и уточнения оценки активов отдельных лиц проводятся собеседования. И, наконец, котировки акций, обращающихся на бирже, оцениваются по рыночным ценам примерно за месяц до публикации. \n",
    "Частные компании оцениваются в соответствии с преобладающим соотношением цены к продажам или цены к прибыли. \n",
    "Известные долги вычитаются из активов, чтобы получить окончательную оценку предполагаемого состояния человека в долларах США. \n",
    "Поскольку цены на акции быстро колеблются, истинное состояние человека и его рейтинг на момент публикации могут отличаться от того, в котором он находился на момент составления списка."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 2600 entries, 1 to 2578\n",
      "Data columns (total 6 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Name      2600 non-null   object \n",
      " 1   Networth  2600 non-null   float64\n",
      " 2   Age       2600 non-null   int64  \n",
      " 3   Country   2600 non-null   object \n",
      " 4   Source    2600 non-null   object \n",
      " 5   Industry  2600 non-null   object \n",
      "dtypes: float64(1), int64(1), object(4)\n",
      "memory usage: 142.2+ KB\n",
      "(2600, 6)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Networth</th>\n",
       "      <th>Age</th>\n",
       "      <th>Country</th>\n",
       "      <th>Source</th>\n",
       "      <th>Industry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rank</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Elon Musk</td>\n",
       "      <td>219.0</td>\n",
       "      <td>50</td>\n",
       "      <td>United States</td>\n",
       "      <td>Tesla, SpaceX</td>\n",
       "      <td>Automotive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jeff Bezos</td>\n",
       "      <td>171.0</td>\n",
       "      <td>58</td>\n",
       "      <td>United States</td>\n",
       "      <td>Amazon</td>\n",
       "      <td>Technology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bernard Arnault &amp; family</td>\n",
       "      <td>158.0</td>\n",
       "      <td>73</td>\n",
       "      <td>France</td>\n",
       "      <td>LVMH</td>\n",
       "      <td>Fashion &amp; Retail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bill Gates</td>\n",
       "      <td>129.0</td>\n",
       "      <td>66</td>\n",
       "      <td>United States</td>\n",
       "      <td>Microsoft</td>\n",
       "      <td>Technology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Warren Buffett</td>\n",
       "      <td>118.0</td>\n",
       "      <td>91</td>\n",
       "      <td>United States</td>\n",
       "      <td>Berkshire Hathaway</td>\n",
       "      <td>Finance &amp; Investments</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           Name  Networth  Age        Country  \\\n",
       "Rank                                                            \n",
       "1                    Elon Musk      219.0   50  United States   \n",
       "2                   Jeff Bezos      171.0   58  United States   \n",
       "3     Bernard Arnault & family      158.0   73         France   \n",
       "4                   Bill Gates      129.0   66  United States   \n",
       "5               Warren Buffett      118.0   91  United States   \n",
       "\n",
       "                  Source                Industry  \n",
       "Rank                                              \n",
       "1          Tesla, SpaceX             Automotive   \n",
       "2                 Amazon             Technology   \n",
       "3                   LVMH       Fashion & Retail   \n",
       "4              Microsoft             Technology   \n",
       "5     Berkshire Hathaway  Finance & Investments   "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"..//..//static//csv//Forbes Billionaires.csv\", index_col=\"Rank\")\n",
    "\n",
    "df.info()\n",
    "\n",
    "print(df.shape)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>2. Проблемная область</b>\n",
    "<br><br>\n",
    "Анализ данных из списка миллиардеров Forbes позволяет не только понять текущее состояние богатства в мире, но и выявить более глубокие тенденции и паттерны, которые могут помочь в принятии бизнес-решений, понимании экономических процессов и определении направлений для дальнейших исследований. Эти данные могут быть основой для многочисленных статей, отчетов и аналитических исследований, что делает их ценными для широкого круга специалистов в различных областях."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>3. Анализ содержимого</b>\n",
    "<br><br>\n",
    "\n",
    "1. Объектами наблюдения являются миллиардеры.\n",
    "2. В качестве атбирутов вредставлены: имя, величина богатства, возраст, страна, источник, индустрия\n",
    "3. Связей между объектами нет"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>4. Бизнес-цели</b>\n",
    "<br><br>\n",
    "1. Сравнив свой бизнес с другими успешными компаниями, основанными миллиардерами, можно извлечь ценные уроки о сильных сторонами и недостатках своей компании.\n",
    "2. Анализируя, в каких секторах работают миллиардеры и какие компании они развивают, можно выявить растущие рынки и индустрии, в которые стоит инвестировать.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Привести примеры целей технического проекта для каждой выделенной ранее\n",
    "бизнес-цели. Что поступает на вход, что является целевым признаком?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Получение сведений о пропущенных данных"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Типы пропущенных данных:\n",
    "- None - представление пустых данных в Python\n",
    "- NaN - представление пустых данных в Pandas\n",
    "- '' - пустая строка"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name        0\n",
      "Networth    0\n",
      "Age         0\n",
      "Country     0\n",
      "Source      0\n",
      "Industry    0\n",
      "dtype: int64\n",
      "\n",
      "Name        False\n",
      "Networth    False\n",
      "Age         False\n",
      "Country     False\n",
      "Source      False\n",
      "Industry    False\n",
      "dtype: bool\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Количество пустых значений признаков\n",
    "print(df.isnull().sum())\n",
    "\n",
    "print()\n",
    "\n",
    "# Есть ли пустые значения признаков\n",
    "print(df.isnull().any())\n",
    "\n",
    "print()\n",
    "\n",
    "# Процент пустых значений признаков\n",
    "for i in df.columns:\n",
    "    null_rate = df[i].isnull().sum() / len(df) * 100\n",
    "    if null_rate > 0:\n",
    "        print(f\"{i} процент пустых значений: %{null_rate:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заполнение пропущенных данных\n",
    "\n",
    "https://pythonmldaily.com/posts/pandas-dataframes-search-drop-empty-values\n",
    "\n",
    "https://scales.arabpsychology.com/stats/how-to-fill-nan-values-with-median-in-pandas/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(891, 11)\n",
      "Survived    False\n",
      "Pclass      False\n",
      "Name        False\n",
      "Sex         False\n",
      "Age         False\n",
      "SibSp       False\n",
      "Parch       False\n",
      "Ticket      False\n",
      "Fare        False\n",
      "Cabin       False\n",
      "Embarked    False\n",
      "dtype: bool\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>AgeFillNA</th>\n",
       "      <th>AgeFillMedian</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211536</td>\n",
       "      <td>13.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>27.0</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Graham, Miss. Margaret Edith</td>\n",
       "      <td>female</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112053</td>\n",
       "      <td>30.00</td>\n",
       "      <td>B42</td>\n",
       "      <td>S</td>\n",
       "      <td>19.0</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.45</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111369</td>\n",
       "      <td>30.00</td>\n",
       "      <td>C148</td>\n",
       "      <td>C</td>\n",
       "      <td>26.0</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>891</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Dooley, Mr. Patrick</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370376</td>\n",
       "      <td>7.75</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>32.0</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass                                      Name  \\\n",
       "PassengerId                                                               \n",
       "887                 0       2                     Montvila, Rev. Juozas   \n",
       "888                 1       1              Graham, Miss. Margaret Edith   \n",
       "889                 0       3  Johnston, Miss. Catherine Helen \"Carrie\"   \n",
       "890                 1       1                     Behr, Mr. Karl Howell   \n",
       "891                 0       3                       Dooley, Mr. Patrick   \n",
       "\n",
       "                Sex   Age  SibSp  Parch      Ticket   Fare Cabin Embarked  \\\n",
       "PassengerId                                                                 \n",
       "887            male  27.0      0      0      211536  13.00   NaN        S   \n",
       "888          female  19.0      0      0      112053  30.00   B42        S   \n",
       "889          female   NaN      1      2  W./C. 6607  23.45   NaN        S   \n",
       "890            male  26.0      0      0      111369  30.00  C148        C   \n",
       "891            male  32.0      0      0      370376   7.75   NaN        Q   \n",
       "\n",
       "             AgeFillNA  AgeFillMedian  \n",
       "PassengerId                            \n",
       "887               27.0           27.0  \n",
       "888               19.0           19.0  \n",
       "889                0.0           28.0  \n",
       "890               26.0           26.0  \n",
       "891               32.0           32.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fillna_df = df.fillna(0)\n",
    "\n",
    "print(fillna_df.shape)\n",
    "\n",
    "print(fillna_df.isnull().any())\n",
    "\n",
    "# Замена пустых данных на 0\n",
    "df[\"AgeFillNA\"] = df[\"Age\"].fillna(0)\n",
    "\n",
    "# Замена пустых данных на медиану\n",
    "df[\"AgeFillMedian\"] = df[\"Age\"].fillna(df[\"Age\"].median())\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>AgeFillNA</th>\n",
       "      <th>AgeFillMedian</th>\n",
       "      <th>AgeCopy</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211536</td>\n",
       "      <td>13.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>27.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Graham, Miss. Margaret Edith</td>\n",
       "      <td>female</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112053</td>\n",
       "      <td>30.00</td>\n",
       "      <td>B42</td>\n",
       "      <td>S</td>\n",
       "      <td>19.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.45</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111369</td>\n",
       "      <td>30.00</td>\n",
       "      <td>C148</td>\n",
       "      <td>C</td>\n",
       "      <td>26.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>26.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>891</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Dooley, Mr. Patrick</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370376</td>\n",
       "      <td>7.75</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>32.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass                                      Name  \\\n",
       "PassengerId                                                               \n",
       "887                 0       2                     Montvila, Rev. Juozas   \n",
       "888                 1       1              Graham, Miss. Margaret Edith   \n",
       "889                 0       3  Johnston, Miss. Catherine Helen \"Carrie\"   \n",
       "890                 1       1                     Behr, Mr. Karl Howell   \n",
       "891                 0       3                       Dooley, Mr. Patrick   \n",
       "\n",
       "                Sex   Age  SibSp  Parch      Ticket   Fare Cabin Embarked  \\\n",
       "PassengerId                                                                 \n",
       "887            male  27.0      0      0      211536  13.00   NaN        S   \n",
       "888          female  19.0      0      0      112053  30.00   B42        S   \n",
       "889          female   NaN      1      2  W./C. 6607  23.45   NaN        S   \n",
       "890            male  26.0      0      0      111369  30.00  C148        C   \n",
       "891            male  32.0      0      0      370376   7.75   NaN        Q   \n",
       "\n",
       "             AgeFillNA  AgeFillMedian  AgeCopy  \n",
       "PassengerId                                     \n",
       "887               27.0           27.0     27.0  \n",
       "888               19.0           19.0     19.0  \n",
       "889                0.0           28.0      0.0  \n",
       "890               26.0           26.0     26.0  \n",
       "891               32.0           32.0     32.0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"AgeCopy\"] = df[\"Age\"]\n",
    "\n",
    "# Замена данных сразу в DataFrame без копирования\n",
    "df.fillna({\"AgeCopy\": 0}, inplace=True)\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Удаление наблюдений с пропусками"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(183, 14)\n",
      "Survived    False\n",
      "Pclass      False\n",
      "Name        False\n",
      "Sex         False\n",
      "Age         False\n",
      "SibSp       False\n",
      "Parch       False\n",
      "Ticket      False\n",
      "Fare        False\n",
      "Cabin       False\n",
      "Embarked    False\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "dropna_df = df.dropna()\n",
    "\n",
    "print(dropna_df.shape)\n",
    "\n",
    "print(fillna_df.isnull().any())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создание выборок данных\n",
    "\n",
    "Библиотека scikit-learn\n",
    "\n",
    "https://scikit-learn.org/stable/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/lec2-split.png\" width=\"600\" style=\"background-color: white\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Функция для создания выборок\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "def split_stratified_into_train_val_test(\n",
    "    df_input,\n",
    "    stratify_colname=\"y\",\n",
    "    frac_train=0.6,\n",
    "    frac_val=0.15,\n",
    "    frac_test=0.25,\n",
    "    random_state=None,\n",
    "):\n",
    "    \"\"\"\n",
    "    Splits a Pandas dataframe into three subsets (train, val, and test)\n",
    "    following fractional ratios provided by the user, where each subset is\n",
    "    stratified by the values in a specific column (that is, each subset has\n",
    "    the same relative frequency of the values in the column). It performs this\n",
    "    splitting by running train_test_split() twice.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    df_input : Pandas dataframe\n",
    "        Input dataframe to be split.\n",
    "    stratify_colname : str\n",
    "        The name of the column that will be used for stratification. Usually\n",
    "        this column would be for the label.\n",
    "    frac_train : float\n",
    "    frac_val   : float\n",
    "    frac_test  : float\n",
    "        The ratios with which the dataframe will be split into train, val, and\n",
    "        test data. The values should be expressed as float fractions and should\n",
    "        sum to 1.0.\n",
    "    random_state : int, None, or RandomStateInstance\n",
    "        Value to be passed to train_test_split().\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    df_train, df_val, df_test :\n",
    "        Dataframes containing the three splits.\n",
    "    \"\"\"\n",
    "\n",
    "    if frac_train + frac_val + frac_test != 1.0:\n",
    "        raise ValueError(\n",
    "            \"fractions %f, %f, %f do not add up to 1.0\"\n",
    "            % (frac_train, frac_val, frac_test)\n",
    "        )\n",
    "\n",
    "    if stratify_colname not in df_input.columns:\n",
    "        raise ValueError(\"%s is not a column in the dataframe\" % (stratify_colname))\n",
    "\n",
    "    X = df_input  # Contains all columns.\n",
    "    y = df_input[\n",
    "        [stratify_colname]\n",
    "    ]  # Dataframe of just the column on which to stratify.\n",
    "\n",
    "    # Split original dataframe into train and temp dataframes.\n",
    "    df_train, df_temp, y_train, y_temp = train_test_split(\n",
    "        X, y, stratify=y, test_size=(1.0 - frac_train), random_state=random_state\n",
    "    )\n",
    "\n",
    "    # Split the temp dataframe into val and test dataframes.\n",
    "    relative_frac_test = frac_test / (frac_val + frac_test)\n",
    "    df_val, df_test, y_val, y_test = train_test_split(\n",
    "        df_temp,\n",
    "        y_temp,\n",
    "        stratify=y_temp,\n",
    "        test_size=relative_frac_test,\n",
    "        random_state=random_state,\n",
    "    )\n",
    "\n",
    "    assert len(df_input) == len(df_train) + len(df_val) + len(df_test)\n",
    "\n",
    "    return df_train, df_val, df_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pclass\n",
      "3    491\n",
      "1    216\n",
      "2    184\n",
      "Name: count, dtype: int64\n",
      "Обучающая выборка:  (534, 3)\n",
      "Pclass\n",
      "3    294\n",
      "1    130\n",
      "2    110\n",
      "Name: count, dtype: int64\n",
      "Контрольная выборка:  (178, 3)\n",
      "Pclass\n",
      "3    98\n",
      "1    43\n",
      "2    37\n",
      "Name: count, dtype: int64\n",
      "Тестовая выборка:  (179, 3)\n",
      "Pclass\n",
      "3    99\n",
      "1    43\n",
      "2    37\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Вывод распределения количества наблюдений по меткам (классам)\n",
    "print(df.Pclass.value_counts())\n",
    "\n",
    "data = df[[\"Pclass\", \"Survived\", \"AgeFillMedian\"]].copy()\n",
    "\n",
    "df_train, df_val, df_test = split_stratified_into_train_val_test(\n",
    "   data, stratify_colname=\"Pclass\", frac_train=0.60, frac_val=0.20, frac_test=0.20\n",
    ")\n",
    "\n",
    "print(\"Обучающая выборка: \", df_train.shape)\n",
    "print(df_train.Pclass.value_counts())\n",
    "\n",
    "print(\"Контрольная выборка: \", df_val.shape)\n",
    "print(df_val.Pclass.value_counts())\n",
    "\n",
    "print(\"Тестовая выборка: \", df_test.shape)\n",
    "print(df_test.Pclass.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Выборка с избытком (oversampling)\n",
    "\n",
    "https://www.blog.trainindata.com/oversampling-techniques-for-imbalanced-data/\n",
    "\n",
    "https://datacrayon.com/machine-learning/class-imbalance-and-oversampling/\n",
    "\n",
    "Выборка с недостатком (undersampling)\n",
    "\n",
    "https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/\n",
    "\n",
    "Библиотека imbalanced-learn\n",
    "\n",
    "https://imbalanced-learn.org/stable/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Обучающая выборка:  (534, 3)\n",
      "Pclass\n",
      "3    294\n",
      "1    130\n",
      "2    110\n",
      "Name: count, dtype: int64\n",
      "Обучающая выборка после oversampling:  (864, 3)\n",
      "Pclass\n",
      "3    294\n",
      "2    290\n",
      "1    280\n",
      "Name: count, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Survived</th>\n",
       "      <th>AgeFillMedian</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>28.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>32.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>28.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>45.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>7.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>859</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>26.887761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.890459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>861</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>17.481437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>17.078473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>863</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>17.220445</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>864 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Pclass  Survived  AgeFillMedian\n",
       "0         3         0      28.000000\n",
       "1         3         0      32.000000\n",
       "2         3         1      28.000000\n",
       "3         1         0      45.000000\n",
       "4         3         0       7.000000\n",
       "..      ...       ...            ...\n",
       "859       2         0      26.887761\n",
       "860       2         1       0.890459\n",
       "861       2         0      17.481437\n",
       "862       2         0      17.078473\n",
       "863       2         1      17.220445\n",
       "\n",
       "[864 rows x 3 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from imblearn.over_sampling import ADASYN\n",
    "\n",
    "ada = ADASYN()\n",
    "\n",
    "print(\"Обучающая выборка: \", df_train.shape)\n",
    "print(df_train.Pclass.value_counts())\n",
    "\n",
    "X_resampled, y_resampled = ada.fit_resample(df_train, df_train[\"Pclass\"])\n",
    "df_train_adasyn = pd.DataFrame(X_resampled)\n",
    "\n",
    "print(\"Обучающая выборка после oversampling: \", df_train_adasyn.shape)\n",
    "print(df_train_adasyn.Pclass.value_counts())\n",
    "\n",
    "df_train_adasyn"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}