---

Data Processing and Analysis

---

PART 1: DATA PROCESSING

---

1.1 Processing Operations

1. Editing (Scrutiny)

• Examination of collected data to detect and correct errors, omissions, and inconsistencies.
• Field editing: Done immediately after collection (e.g., checking illegible entries).
• Central editing: Done in the office on completed questionnaires/data sheets.
• Checks for: completeness, consistency, accuracy, uniformity of units.

2. Coding

• Assigning numeric or symbolic codes to responses to facilitate tabulation and analysis.
• Converts qualitative data into a quantifiable form.
• Example: Male = 1, Female = 2; Agree = 5, Disagree = 1.
• Should be exhaustive (covers all responses) and mutually exclusive (no overlap).

3. Classification

• Arranging data into groups or classes based on common characteristics.
• Two types:
  • Qualitative classification: Based on attributes (e.g., gender, religion).
  • Quantitative classification: Based on measurable values (e.g., age groups, income brackets).
• Basis of classification: Geographical, chronological, qualitative, or quantitative.

4. Tabulation

• Systematic arrangement of data in rows and columns (frequency tables, cross-tabs).
• Types:
  • Simple tabulation: One variable at a time.
  • Double/Cross tabulation: Two variables simultaneously.
  • Complex tabulation: Three or more variables.
• Facilitates comparison, trend identification, and further statistical analysis.

5. Transcription / Data Entry

• Transfer of data from questionnaires/instruments into a computer or database.
• Includes verification to avoid entry errors.

6. Data Cleaning

• Identifying and correcting corrupt, inaccurate, or irrelevant records.
• Handles: duplicate records, missing values, outliers, inconsistent formats.
• Methods: imputation (mean/median substitution), deletion, transformation.

---

1.2 Problems in Data Processing

---

PART 2: TYPES OF ANALYSIS

---

2.1 By Purpose / Objective

1. Descriptive Analysis

• Summarizes and describes the features of a dataset.
• Does NOT draw inferences or predictions.
• Tools: Mean, Median, Mode, Standard Deviation, Variance, Frequency distributions, Charts.
• Example: "The average score of students was 72."

2. Inferential Analysis (Inductive Analysis)

• Draws conclusions about a population based on a sample.
• Uses probability theory to make generalizations.
• Tools: Hypothesis testing (t-test, chi-square, ANOVA), Confidence intervals, Regression.
• Example: Testing whether a new drug is more effective than a placebo.

3. Exploratory Data Analysis (EDA)

• Explores data to discover patterns, anomalies, and relationships without prior hypotheses.
• Introduced by John Tukey (1977).
• Tools: Box plots, scatter plots, histograms, correlation matrices, heatmaps.
• Precedes confirmatory analysis.

4. Confirmatory Data Analysis (CDA)

• Tests specific hypotheses formulated before data collection.
• More formal and structured than EDA.
• Uses significance tests and p-values.

5. Predictive Analysis

• Uses historical data and statistical/ML models to predict future outcomes.
• Tools: Regression, Time Series, Neural Networks, Decision Trees.
• Example: Predicting next quarter's sales.

6. Prescriptive Analysis

• Recommends actions based on predictive models.
• Combines prediction with optimization.
• Common in operations research and business analytics.

7. Causal Analysis

• Determines cause-and-effect relationships.
• Requires controlled experiments or advanced methods (instrumental variables, regression discontinuity).
• Example: Does education level cause higher income?

8. Mechanistic Analysis

• Seeks to understand the exact mechanism by which changes in variables cause changes in other variables.
• Common in physical sciences and engineering.

---

2.2 By Data Type

---

2.3 By Dimensionality

---

2.4 Specialized Statistical Analysis Types

Content Analysis

• Systematic coding of communication content (text, media).
• Converts qualitative content into quantitative data.

Sentiment Analysis

• Classifies text as positive, negative, or neutral.
• Used in market research, social media analytics.

Survival Analysis

• Analyzes the time until an event of interest occurs.
• Example: Time to relapse after treatment.

Factor Analysis

• Reduces a large number of variables to fewer underlying factors.
• Used in psychometrics, questionnaire development.

Cluster Analysis

• Groups observations into clusters based on similarity.
• Unsupervised; no predefined categories.

Time Series Analysis

• Analyzes data collected at regular time intervals.
• Components: Trend, Seasonality, Cyclical, Irregular.

---

PART 3: SUMMARY TABLE

---

Key Exam Points to Remember

1. Editing vs. Coding: Editing checks for errors; coding converts responses to symbols/numbers.
2. Tabulation is the bridge between raw data and statistical analysis.
3. Descriptive analysis describes; inferential analysis infers - never confuse the two.
4. EDA is hypothesis-generating; CDA is hypothesis-testing.
5. Missing data can be handled by: listwise deletion, pairwise deletion, imputation, or multiple imputation.
6. Outliers can be detected by: z-scores (|z| > 3), IQR method (below Q1-1.5×IQR or above Q3+1.5×IQR), box plots.
7. Univariate → Bivariate → Multivariate is the natural progression of analysis complexity.