In modern society, decisions are heavily based on data and information. Whether choosing a college based on placement records, fees, and faculty qualifications, or governments conducting a census to systematically record population details, data serves as the foundation for analysis and policy formulation.
"Data is not information, Information is not knowledge, Knowledge is not understanding, Understanding is not wisdom."
Socio-Technical Analysis & Core Systemic Impact: This quote by Gary Schubert highlights the cognitive hierarchy in data science. Raw data must undergo structured processing to become meaningful information, which is then contextualized into knowledge, synthesized into understanding, and ultimately applied as wisdom for decision-making.
Technical Profile: Data vs. Datum
Data processing reveals hidden patterns and possibilities that are otherwise invisible to humans. Key real-world applications include:
Data can be broadly classified into two categories based on its format and organization:
Data that is organized and recorded in a well-defined format is called structured data . It is typically stored in a tabular format consisting of rows and columns, where each column represents an attribute (parameter/characteristic) and each row represents an individual observation.
Table 5.1: Structured data about kitchen items in a shop
| ModelNo | ProductName | Unit Price | Discount(%) | Items_in_Inventory |
|---|---|---|---|---|
| ABC1 | Water bottle | 126 | 8 | 13 |
| ABC2 | Melamine Plates | 320 | 5 | 45 |
| ABC3 | Dinner Set | 4200 | 10 | 8 |
| GH67 | Jug | 80 | 0 | 10 |
| GH78 | Table Spoon | 120 | 5 | 14 |
| GH81 | Bucket | 190 | 12 | 6 |
| NK2 | Kitchen Towel | 25 | 0 | 32 |
Table 5.2: Attributes maintained for different activities
| Entity/Activities | Data Fields/Parameters/Attributes |
|---|---|
| Books at a shop | BookTitle , Author , Price , YearofPublication |
| Depositing fees in a school | StudentName , Class , RollNo , FeesAmount , DepositDate |
| Amount withdrawal from ATM | AccHolderName , AccountNo , TypeofAcc , DateofWithdrawal , AmountWithdrawn , ATMid , TimeOfWithdrawal |
Data that does not conform to a traditional row-and-column structure is called unstructured data . Examples include newspapers (variable layouts, mixed \text and images), emails, web pages, audio/video files, and social media posts.
Technical Profile: Metadata
Data collection involves identifying existing data or gathering new data from appropriate sources to prepare it for processing and storage.
Consider three common scenarios for collecting sales data in a grocery store:
Data storage is the process of saving processed or raw data on physical or digital storage devices for future retrieval and reuse. With the exponential growth of data, storage has become a critical challenge, mitigated by the falling cost of digital storage media.
Raw data (numbers, \text, images) cannot be directly used for decision-making. It must undergo a systematic transformation process to become meaningful information.
Technical Profile: The Data Process Cycle
| Problem Statement | Inputs Collected | Processing Steps | Generated Output |
|---|---|---|---|
| Online Student Registration & Admit Card Generation | Student details (name, address, qualification, marks, photo, signature, online fee payment details). | Verify correctness of data, check eligibility criteria, validate fee payment, verify photo/signature uploads, generate roll number, and append to the eligible applicant database. | Examination Admit Card specifying roll number, center address, date, and time of test. |
| ATM Cash Withdrawal | ATM PIN, account type, account number, card number, ATM location, date/time, and withdrawal amount. | Validate PIN, check account balance, deduct requested amount from the account, count physical currency notes, and initiate receipt printing. | Dispensed currency notes and a printed transaction slip. |
| Train Ticket Booking | Journey start/end stations, date, number of tickets, class of travel, berth preference, passenger names/ages, contact details, and payment details. | Verify login credentials, check berth availability, process payment, deduct booked seats from inventory, allocate PNR and berth numbers (or generate a waiting list number). | Confirmed train ticket with berth/coach details or a waitlisted ticket. |
Statistical techniques help summarize tabular data to reveal its underlying characteristics. These are broadly divided into measures of central tendency and measures of variability.
A measure of central tendency provides a single representative value that summarizes a dataset.
The Mean (or average) is the sum of all numeric values in an attribute divided by the total number of values.
Mathematical Definition:
Given $n$ values $x_1, x_2, x_3, \dots, x_n$, the mean $\bar{x}$ is computed as:
$$\bar{x} = \frac{\sum_{i=1}^{n} x_i}{n}$$
Example 5.1:
Height (in cm) of 9 students: [90, 102, 110, 115, 85, 90, 100, 110, 110]
$$\bar{x} = \frac{90 + 102 + 110 + 115 + 85 + 90 + 100 + 110 + 110}{9} = \frac{912}{9} = 101.33\text{ cm}$$
Outlier Warning: An outlier is an exceptionally large or small value compared to the rest of the dataset. The mean is highly sensitive to outliers, which can distort the average. Outliers should be identified and handled before calculating the mean.
The Median is the middle value of a dataset when the values are sorted in ascending or descending order. It divides the dataset into two equal halves.
Example 5.2:
Sorting the student heights from Example 5.1: [85, 90, 90, 100, 102, 110, 110, 110, 115]
Since $n = 9$ (odd), the median is the value at position 5, which is 102 cm .
The Mode is the value that appears most frequently in a dataset. It can be calculated for both numeric and non-numeric data.
Example 5.3:
In the student heights dataset, the value 110 occurs 3 \times (more than any other value). Thus, the mode is 110 cm .
Measures of variability (or dispersion) describe how spread out or diverse the data values are around the mean.
The Range is the difference between the maximum and minimum values in a dataset. It is highly sensitive to outliers.
$$\text{Range} = \text{Maximum} - \text{Minimum}$$
Example 5.4:
For the student heights dataset, the maximum height is 115 cm and the minimum is 85 cm.
$$\text{Range} = 115 - 85 = 30\text{ cm}$$
The Standard Deviation ($\sigma$) measures the spread of data by calculating the positive square root of the average of the squared differences of each value from the mean. Unlike range, it considers all data points.
Mathematical Formula:
$$\sigma = \sqrt{\frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n}}$$
Example 5.5 & Table 5.3: Standard deviation calculation for 9 students ($\bar{x} = 101.33$)
| Height ($x$) in cm | $x - \bar{x}$ | $(x - \bar{x})^2$ |
|---|---|---|
| 90 | -11.33 | 128.37 |
| 102 | 0.67 | 0.36 |
| 110 | 8.67 | 75.17 |
| 115 | 13.67 | 186.87 |
| 85 | -16.33 | 266.67 |
| 90 | -11.33 | 128.37 |
| 100 | -1.33 | 1.77 |
| 110 | 8.67 | 75.17 |
| 110 | 8.67 | 75.17 |
|
$n = 9$
$\bar{x} = 101.33$ |
$\sum(x - \bar{x}) = 0.03$ | $\sum(x - \bar{x})^2 = 938.00$ |
Calculation Steps:
| Problem Statement | Suitable Statistical Method |
|---|---|
| The management of a company wants to know about disparity in salaries of all employees. | Standard Deviation / Range |
| Teacher wants to know about the average performance of the whole class in a test. | Mean |
| Compare height of residents of two cities. | Mean (for average height) and Standard Deviation (for variation) |
| Find the dominant value from a set of values. | Mode |
| Compare income of residents of two cities. | Median (less sensitive to extreme income outliers) |
| Find the popular color for car after surveying the car owners of a small city. | Mode |