← All Case Studies
Architecture Case Study 05 · ML Pipeline
🧠 ML-Based Pricing Optimization
How machine-learning-toolkit's Bayesian elasticity models and min-price Airflow DAGs feed into dynamic_pricing runtime decisions — and why a silent DAG failure is a dangerous unmonitored dependency.
machine-learning-toolkitdynamic_pricingpricing-control-center-jtierAirflowBigQuerySageMaker
📖
Overview
machine-learning-toolkit runs on Google Cloud Composer (managed Airflow) and trains three key models: a Hierarchical Bayesian Price Elasticity model (STAN), a min-price pipeline, and a location embedding model. The critical architectural risk: all three feed directly into dynamic_pricing runtime pricing decisions via the Pricing Service API — and none have production monitoring for DAG failures.
⚙️
ML → Pricing Feed Pipeline — 7 Stages
Offline batch pipeline that feeds runtime pricing decisions. Failure is invisible to oncall.
1
Data Sources
Hive/EDW · Janus events · MDS · Teradata historical data
batch
→
2
Airflow DAGs
Cloud Composer · 11 DAGs · scheduled · min_price_pipeline + bayes_elasticity
scheduled
→
3
Bayesian STAN
model_pred_v12.stan · hierarchical prior · WMAPE <15% target
4–8h
→
4
Elasticity Table
median_bayesian_elasticity → BigQuery · per-deal β[d] coefficient
output
→
5
Pricing API Feed
POST /v1/retail/price → dynamic_pricing · min-price floor update
<30ms
→
6
dynamic_pricing
Enforces min-price floor · -1 sentinel = no discount allowed
runtime
→
7
PCC JTier feed
ILS pricing parameters · CustomILS flux model data · hourly sync
hourly
🔴
Zero alerting for DAG failures that affect the min-price feed — P0 observability gap
A missed min_price_pipeline run causes dynamic_pricing to serve stale price floors with no alert firing. Oncall only discovers the issue after a downstream SLA breach triggers manual investigation. Fix: add Airflow SLA miss callback → oncall@example.com for min_price_pipeline and bayes_elasticity_model DAGs.
🔗
Model Architecture & Risk
🧮
Bayesian elasticity: 3-level hierarchical prior captures category-level price sensitivity
The STAN model estimates deal-specific price elasticity β[d] using a 3-level hierarchy (deal → subcategory → category). Non-centered parameterization improves MCMC sampling efficiency. Output feeds the median_bayesian_elasticity BigQuery table, which dynamic_pricing reads for pricing optimization decisions.
⚠️
machine-learning-toolkit is an implicit upstream of dynamic_pricing — not visible in code dependency graphs
Standard dependency analysis (Maven/Gradle build graphs) shows no dependency between ML Toolkit and dynamic_pricing. The actual runtime dependency flows through Airflow DAG → Pricing Service API POST → dynamic_pricing MySQL tables. This makes ML failures invisible to the pricing team until prices are already wrong.
📊
WMAPE < 15% target: model accuracy directly determines pricing quality
The Bayesian elasticity model targets WMAPE (Weighted Mean Absolute Percentage Error) below 15%. Values above this threshold mean price elasticity estimates are too inaccurate for reliable pricing optimization. Bayesian hyperparameter optimization (via the bayesian_optimization module) tunes the model to maintain this threshold.
🔴
P0 observability gap: no alert for ML DAG failures affecting min-price feed
The highest-priority fix in the ML Toolkit: add Airflow SLA miss callback for min_price_pipeline and bayes_elasticity_model. A 5-line Python change eliminates the silent failure mode that makes price floor miscalculations invisible to oncall.
🧠
machine-learning-toolkit contributes to pricing optimization decisions across 3 downstream services
Outputs feed: (1) dynamic_pricing min-price floor, (2) PCC JTier ILS pricing params, (3) message-service deal-feed model for email personalisation. All three consume ML outputs without awareness that those outputs can silently go stale.
🎯
Key Takeaway
ML-based pricing optimization is powerful but fragile — three key models feed into runtime pricing decisions with zero production monitoring for pipeline failures. The fix is low-effort (add Airflow SLA callbacks) with critical impact. Secondary recommendation: add a data quality check DAG that validates median_bayesian_elasticity row counts before dynamic_pricing reads the new floor values.