He Was Right... And Wrong | Inside Data Science Interview

Overview

The video presents an “inside” data science interview walkthrough (technical round + debrief) using the California Housing dataset in Google Colab. The interviewer and candidate discuss data understanding, exploratory analysis, statistical concepts, and then modeling—specifically framed as predicting latitude (or detecting it) from the available features.

1) Candidate background and interview structure

• The candidate (Karthik) highlights:
  • Training in data analytics/data science
  • Hands-on experience in generative AI (using models like Midjourney/DALL·E/GPT-4 for industry pattern generation)
  • Later building production-ready ML pipelines
• The interviewer outlines the interview flow:
  • First ~40 minutes: technical round (questions, reasoning, coding)
  • Later: review/back-and-forth about performance and approach

2) Data exploration on California Housing dataset (EDA)

The candidate:

• Loads the dataset and checks:
  • Data types (mostly floats/continuous numeric features)
  • Missing values (no nulls reported)
  • Summary statistics using describe() (mean, std, min/max, quartiles)
• Interprets key columns conceptually:
  • Latitude/longitude reflect geography; their ranges suggest multiple regions and spatial spread
  • House age, population, median income, and household composition variables are discussed

3) Outliers: detection and interpretation

• The candidate identifies likely outlier-heavy features using descriptive stats and box plots:
  • total rooms, bedrooms, population, households
• The interviewer probes interpretation:
  • Extremely large bedroom counts (thousands) seem unlikely for single homes.
  • This suggests the dataset may represent densely populated complexes (apartments/hotels/community housing) rather than typical family houses.
• The candidate explains a numerical outlier method using the IQR rule:
  • Outliers via Q1 − 1.5×IQR and Q3 + 1.5×IQR
  • They are challenged on why 1.5 is used; the interviewer hints it relates to behavior under a normal distribution (i.e., a less aggressive threshold than many alternatives).

4) Distributions, skewness, and kurtosis

• The interviewer asks about special properties of longitude/latitude distributions.
  • The candidate suggests bi-modal-like behavior (two peaks), implying two geographic clusters/regions.
• They discuss skewness:
  • Positive skew (right tail) vs negative skew (left tail)
  • How skewness relates to mean vs median
• The interviewer expands the discussion to kurtosis and implications:
  • Skewness can affect assumptions of normality.
  • It can impact linear regression and distance-based/sensitive models (e.g., SVM) by distorting distributions and making outliers more influential.
• Candidate’s model rationale:
  • Skew/outliers and distribution shape affect distance calculations and model sensitivity.
  • Variability/spread (e.g., standard deviation) can also matter.
• The interviewer pushes for more precise mathematical grounding:
  • Explanations are generally correct but sometimes not detailed enough mathematically.

5) Feature scaling and visualization for better outlier plots

To improve visual comparisons, the candidate moves from box plots to scaling:

• Scaling discussion includes:
  • StandardScaler (typical choice)
  • MinMaxScaler (generally less suitable here because it compresses to 0–1 and can overly emphasize extreme values/outliers)
  • Mentions RobustScaler and normalizer ideas
• After standard scaling, they revisit box plots and restate which features appear to have more outliers.

6) Modeling: predicting latitude from all features

• The interviewer shifts to modeling and asks about predicting latitude using all other features.
• Candidate proposes Random Forest Regressor (with XGBoost as an alternative), arguing:
  • Random forests handle nonlinear relationships
  • They are robust to outliers and require less tuning
  • XGBoost may improve accuracy by learning from errors sequentially, but needs more hyperparameter tuning and can be more sensitive to noise/outliers
• The interviewer’s technical “trap” question:
  • Would dropping latitude cause issues, or should the goal be specifically predicting latitude?
• Candidate’s initial reasoning:
  • Predicting latitude “should be hard” because latitude is not strongly dependent on non-geographic features.
  • Correct spatial/cluster structure is needed, and it isn’t captured if the right spatial signals are removed or ignored.
• The interviewer sets a pass/fail threshold:
  • Target accuracy ≥ 90% (interpreted as very high performance)
• Candidate builds a Random Forest regressor and expects poor results for accurate latitude prediction.
• The interviewer confirms the reasoning matches expectations:
  • Latitude prediction drops substantially.
  • Candidate suggests accuracy could be around ~50%, since location depends primarily on spatial relationships involving longitude/latitude and clustering.

7) Debrief: strengths and weaknesses

What the interviewer liked (positives)

• Strong coding/EDA fluency: histograms/figures without relying on external help
• Solid statistical instincts: why plots matter and how distributions influence analysis
• Practical model selection knowledge (Random Forest / XGBoost)

What the interviewer criticized (negatives)

• Advanced concept explanations (especially skewness/kurtosis) were less mathematically in-depth than expected
• For highly specific model-choice questions, the candidate sometimes responds with generic comparisons instead of directly addressing the question’s comparative requirements
• Biggest improvement area: business framing
  • Candidate adopts a “technical first” mindset
  • Interviewer expects reasoning from a requirements/business perspective, e.g.:
    • Why predict latitude at all
    • Consequences of doing so
    • How this differs from predicting income/house value
  • Interviewer emphasizes that latitude/longitude are highly related for location prediction, unlike more “income-like” variables

Presenters / Contributors

• Interviewer: Karthik
• Candidate: Karthik (also says “Hey B” at the start; the candidate’s name appears as Karthik in the transcript)