Core Concepts
Beginner Glossary
Plain-language guide to Featuresmith technical terminology
This glossary defines core technical concepts used throughout Featuresmith. Each entry explains What It Means, Why It Matters in Featuresmith, and a Simple Example.
DataFrame
- What It Means: A two-dimensional tabular data structure with named columns and rows.
- Why It Matters in Featuresmith: Featuresmith normalizes raw tabular inputs (CSV, Parquet, Excel) into DataFrames for fast statistical profiling.
- Simple Example:
A pandas or Polars table with columns 'age', 'income', and 'churn'.
Polars
- What It Means: An ultra-fast, multi-threaded DataFrame library written in Rust.
- Why It Matters in Featuresmith: Featuresmith uses Polars internally for high-performance vectorized profiling and dataset ingestion.
- Simple Example:
Processing 10 million rows in under a second using Polars primitives.
pandas
- What It Means: The standard Python data analysis library for working with tabular data.
- Why It Matters in Featuresmith: Featuresmith accepts pandas DataFrames natively via fs.load(df) or DataFrameConnector.
- Simple Example:
df = pd.read_csv('data.csv')
Schema
- What It Means: The structural blueprint of a dataset, defining column names and their expected data types.
- Why It Matters in Featuresmith: Featuresmith's SchemaHealthReviewer inspects column names and types for structural consistency.
- Simple Example:
{'age': Int64, 'name': String, 'income': Float64}
Dtype (Data Type)
- What It Means: The technical storage type of a column's values (e.g. Int64, Float64, Utf8, Datetime).
- Why It Matters in Featuresmith: Featuresmith inspects dtypes to detect type mismatches and recommend proper ML encodings.
- Simple Example:
'age' stored as Int64 vs 'price' stored as Float64.
Logical Type
- What It Means: The higher-level semantic type of a column (numeric, categorical, datetime, text, or identifier).
- Why It Matters in Featuresmith: Featuresmith infers logical types during profiling to apply specialized statistical rules.
- Simple Example:
'passenger_id' has numeric dtype but identifier logical type.
Profiling
- What It Means: Computing deterministic statistical descriptors (min, max, mean, quantiles, missingness, correlation) across a dataset.
- Why It Matters in Featuresmith: fs.profile() compiles a comprehensive ProfileResult without modifying raw data.
- Simple Example:
Calculating that 'income' has mean $65,000, max $250,000, and 2.5% missing values.
Missingness
- What It Means: The presence of null, NaN, or missing values in a dataset column.
- Why It Matters in Featuresmith: MissingValueReviewer flags columns exceeding configurable null thresholds (default 20%).
- Simple Example:
Column 'cabin' containing 77% null values in the Titanic dataset.
Cardinality
- What It Means: The number of unique distinct values in a categorical column.
- Why It Matters in Featuresmith: HighCardinalityReviewer flags categorical columns with excessive unique categories.
- Simple Example:
Column 'zip_code' containing 5,000 unique values across 6,000 rows.
Correlation (Pearson)
- What It Means: A statistical metric measuring linear relationship strength between two numeric columns (-1.0 to +1.0).
- Why It Matters in Featuresmith: Used by HighCorrelationRule and TargetCorrelationDetector to catch multicollinearity and leakage.
- Simple Example:
Correlation of +0.99 between 'total_bill' and 'tax_amount'.
Entropy
- What It Means: A statistical measure of randomness or unpredictability in a categorical feature.
- Why It Matters in Featuresmith: Featuresmith computes categorical entropy to measure value diversity.
- Simple Example:
High entropy in uniform category distributions vs zero entropy in constant columns.
Skewness
- What It Means: A measure of asymmetry in a numeric probability distribution around its mean.
- Why It Matters in Featuresmith: BasicStatisticsReviewer flags features with extreme skewness (>2.0) requiring log transformations.
- Simple Example:
Income distributions with a long right tail of high earners.
Kurtosis
- What It Means: A measure of the 'tailedness' and extreme outlier presence in a distribution.
- Why It Matters in Featuresmith: Featuresmith identifies heavy-tailed distributions with kurtosis >10.0.
- Simple Example:
Financial transaction amounts with sudden massive outlier spikes.
IQR (Interquartile Range)
- What It Means: The range between the 25th (Q1) and 75th (Q3) percentiles (IQR = Q3 - Q1).
- Why It Matters in Featuresmith: Used by OutlierDetectionRule to identify statistical outliers robustly.
- Simple Example:
Values beyond Q3 + 1.5*IQR flagged as outliers.
Target Column
- What It Means: The column representing the outcome variable or label being predicted in supervised machine learning.
- Why It Matters in Featuresmith: Declaring target_column='survived' enables target leakage detection across 6 pattern detectors.
- Simple Example:
'churn_label' in customer churn prediction.
Target Leakage
- What It Means: A severe bug where predictive features contain future outcome data unavailable at inference time.
- Why It Matters in Featuresmith: Featuresmith's LeakageReviewer detects correlation, timestamp, and outcome clones.
- Simple Example:
Including 'account_cancellation_date' in a churn model.
Deterministic Engine
- What It Means: Algorithms that always produce identical, repeatable outputs when given the same input.
- Why It Matters in Featuresmith: Featuresmith rule evaluations and scores are 100% deterministic and reproducible.
- Simple Example:
Running fs.review() on identical data always yields the exact same score.
Rule
- What It Means: An atomic quality assertion (e.g. quality.missing_value_threshold) evaluated against a dataset profile.
- Why It Matters in Featuresmith: Rules produce RuleFinding objects with assigned severities.
- Simple Example:
FullyEmptyColumnsRule checking for 100% null columns.
Reviewer
- What It Means: A domain-specific inspector inside the Review Engine that aggregates rule findings into a ReviewSection.
- Why It Matters in Featuresmith: 10 built-in reviewers evaluate dataset health deterministically.
- Simple Example:
LeakageReviewer evaluating target leakage risks.
Finding (RuleFinding)
- What It Means: A structured record representing a specific quality issue, warning, or passed check.
- Why It Matters in Featuresmith: Findings contain rule_id, title, description, column_name, severity, and evidence.
- Simple Example:
[CRITICAL] High missing values in column 'cabin'.
ML Readiness Score
- What It Means: An explainable 0–100 quality scorecard evaluating dataset health across 7 effective weighted dimensions.
- Why It Matters in Featuresmith: Provides a single auditable metric to gate pre-training data pipelines.
- Simple Example:
Score of 86.9/100 on Titanic dataset.
Dataset Diff
- What It Means: Comparing two dataset snapshot versions (old vs new) to identify quality drift and schema changes.
- Why It Matters in Featuresmith: fs.diff() yields an overall health verdict (unchanged, improved, regressed).
- Simple Example:
Detecting that a new daily dataset snapshot dropped column 'store_id'.