In modern society, decisions are heavily reliant on data and its systematic analysis. From choosing a college based on placement records, fees, and faculty qualifications, to governments formulating national policies using census data, data serves as the foundational bedrock for decision-making across all domains.
Technical Profile: Data vs. Information
Human beings and automated systems rely on data to uncover hidden traits, optimize operations, and make informed choices. Key examples include:
Data originates from diverse sources and can be broadly classified into two formats:
Structured data is organized and recorded in a well-defined format, typically stored in a tabular structure of rows and columns. Each column represents an attribute (or variable), and each row represents a single observation.
Table 7.1: Structured data about kitchen items in a shop
| ModelNo | ProductName | Unit Price | Discount(%) | Items_in_Inventory |
|---|---|---|---|---|
| ABC1 | Water bottle | 126 | 8 | 13 |
| ABC2 | Melamine Plates | 320 | 5 | 45 |
| ABC3 | Dinner Set | 4200 | 10 | 8 |
| GH67 | Jug | 80 | 0 | 10 |
| GH78 | Table Spoon | 120 | 5 | 14 |
| GH81 | Bucket | 190 | 12 | 6 |
| NK2 | Kitchen Towel | 25 | 0 | 32 |
Table 7.2: Attributes maintained for different activities
| Entity/Activities | Data Fields/Parameters/Attributes |
|---|---|
| Books at a shop | BookTitle, Author, Price, YearofPublication |
| Depositing fees in a school | StudentName, Class, RollNo, FeesAmount, DepositDate |
| Amount withdrawal from ATM | AccHolderName, AccountNo, TypeofAcc, DateofWithdrawal, AmountWithdrawn, ATMid, TimeOfWithdrawal |
Unstructured data refers to data that does not conform to a traditional row-and-column structure. Examples include emails, web pages, audio/video files, and social media posts.
Unstructured data is often described using metadata (data about data). For instance, an image file's metadata includes its size, resolution, and format (JPEG, PNG).
Data collection involves identifying and gathering data from appropriate sources to be stored for subsequent processing. Common scenarios for collecting sales data in a grocery store include:
Data storage is the process of saving data on physical or digital storage devices for future retrieval and use. With the exponential growth of data generation, storage has become a critical challenge, mitigated by the declining cost of digital storage media.
Technical Profile: Storage Media & Systems
Data processing is the sequence of operations performed on raw data to convert it into meaningful information. The standard workflow is represented by the Data Process Cycle :
Table 7.3: Examples of Data-Based Problem Statements
| Problem Statement | Inputs Collected | Processing Steps | Output Generated |
|---|---|---|---|
| Online Student Registration | Student details (name, address, marks, photo, signature, fee payment details) | Verify correctness, check eligibility, confirm fee payment, generate roll number | Examination Admit Card with roll number, center, and test schedule |
| ATM Cash Withdrawal | PIN, account number, card details, withdrawal amount | Validate PIN, check account balance, debit amount, count currency notes, initiate receipt printing | Currency notes and printed transaction slip |
| Train Ticket Booking | Journey details, class of travel, passenger details, payment details | Verify login, check berth availability, process payment, allocate PNR and berth/waiting list number | Train ticket with PNR, coach, and berth details |
Statistical techniques are applied to tabular data to summarize, comprehend, and reveal its underlying characteristics.
A single value that represents the center or typical value of a dataset. The three primary measures are:
The Mean (or average) is the sum of all numeric values divided by the total number of values.
Mathematical Definition:
$$\text{Mean} = \frac{\sum_{i=1}^{n} x_i}{n}$$Example 7.1: Calculate the mean height of students with heights (in cm): [90, 102, 110, 115, 85, 90, 100, 110, 110]
Socio-Technical Analysis & Core Systemic Impact:
$$\text{Mean} = \frac{90 + 102 + 110 + 115 + 85 + 90 + 100 + 110 + 110}{9} = \frac{912}{9} = 101.33\text{ cm}$$Note on Outliers: Mean is highly sensitive to outliers (extreme values), which can distort the average. Outliers should be identified and handled before calculating the mean.
The Median is the middle value of a dataset when the values are sorted in ascending or descending order. If the dataset has an odd number of values, the median is the middle value. If even, it is the average of the two middle values.
Example 7.2: Find the median of the height dataset: [90, 102, 110, 115, 85, 90, 100, 110, 110]
Socio-Technical Analysis & Core Systemic Impact:
- Sort the data in ascending order: [85, 90, 90, 100, 102, 110, 110, 110, 115]
- Since $n = 9$ (odd), the median is the value at position 5.
- Therefore, the Median is 102 cm .
The Mode is the value that appears most frequently in a dataset. A dataset can have no mode, a single mode, or multiple modes.
Example 7.3: Find the mode of the height dataset: [85, 90, 90, 100, 102, 110, 110, 110, 115]
Socio-Technical Analysis & Core Systemic Impact: The value 110 appears 3 \times, which is the highest frequency. Thus, the Mode is 110.
Measures of variability (or dispersion) describe the spread or variation of data values around the mean.
The Range is the difference between the maximum and minimum values in a dataset.
$$\text{Range} = \text{Maximum} - \text{Minimum}$$Example 7.4: Calculate the range of the height dataset.
Socio-Technical Analysis & Core Systemic Impact:
- Maximum value = 115 cm
- Minimum value = 85 cm
- $\text{Range} = 115 - 85 = 30\text{ cm}$
The Standard Deviation ($\sigma$) measures the spread of data by calculating the positive square root of the average of squared differences of each value from the mean.
Mathematical Formula:
$$\sigma = \sqrt{\frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n}}$$Example 7.5: Compute the standard deviation for the height of 9 students (Mean $\bar{x} = 101.33$ cm).
Socio-Technical Analysis & Core Systemic Impact:
Height ($x$) in cm $x - \bar{x}$ $(x - \bar{x})^2$ 90 -11.33 128.37 102 0.67 0.36 110 8.67 75.17 115 13.67 186.87 85 -16.33 266.67 90 -11.33 128.37 100 -1.33 1.77 110 8.67 75.17 110 8.67 75.17 Sum of squared differences: $\sum (x_i - \bar{x})^2 = 938.00$
Variance: $\frac{938.00}{9} = 104.22$
Standard Deviation: $\sigma = \sqrt{104.22} \approx 10.2\text{ cm}$
Exercise 1: Identify data required to be maintained to perform the following services: (a) Declare exam results and print e-certificates, (b) Register participants in an exhibition and issue biometric ID cards.
Socio-Technical Analysis & Core Systemic Impact:
- (a) Exam Results: Student ID, Student Name, Subject Codes, Marks Obtained, Maximum Marks, Grade, Date of Declaration.
- (b) Exhibition Registration: Participant Name, Organization, Contact Number, Email, Biometric Template (fingerprint/iris scan), Registration ID, Date of Issue.