Milan Shah - Tech Background

Summary

In January 2025, during Amazon’s Senior Leadership Yearly Review, Jeff Bezos highlighted a persistent customer pain point: customers frequently fell into “infinite browsing” when trying to find the right product while shopping because they feared missing out on a better option. In the following week, - Daniel Lloyd, VP of Personalization, asked me to take ownership of addressing this problem as part of operational planning. As an outcome, I led my Intent organization to build Help Me Decide (HMD), an AI-driven shopping feature designed to streamline purchase decisions by providing customers with the best product recommendation for their current mission, along with contextual explanations and trade-offs for premium or budget alternatives.

We decided to build HMD to collapse high-dimensional customer intent into a single, confidence-maximizing decision rather than another ranked list while preserving trust through explainability. This required fundamentally changing the existing recommendation paradigm: instead of optimizing primarily for relevance across a set of candidates, HMD needed to optimize for mission satisfaction and confidence in a single Top-1 decision, while incorporating temporal customer intent, product constraints, and trustworthy explanations.

Delivering HMD required orchestrating multiple engineering, applied science, and product teams within my organization, while aligning closely with senior leaders across partner orgs under significant technical ambiguity. The project spanned Amazon.com’s highest-traffic surfaces, serving over 250M monthly active users. As the single-threaded owner, I set the technical vision, reached an alignment within and beyond my organization, guided leaders within my org (managers and senior ICs) through architectural and modeling decisions, managed cross-org dependencies, and held my organization accountable for both customer experience and business outcomes. Nine months later, the launch exceeded expectations, in a three-week US marketplace A/B experiment, HMD drove $451M in Order Product Sales (OPS), the highest OPS recorded for a single weblab in 2025, and was called out in Amazon’s Q3 earnings as a flagship AI initiative by Andy Jassy.

This document provides a technical deep dive into the leadership, architecture, and trade-offs that enabled this outcome.

Success Criteria

I defined success across four dimensions:

• Customer: reduce effort and browsing required to reach a purchase decision while increasing customer confidence.

• Recommendation quality: improve Top-1 relevance and mission satisfaction, measured using Top-1 Accuracy, Precision@K, NDCG@K, and downstream add-to-cart/purchase behavior.

• Business: measure incremental Order Product Sales (OPS) and long-term customer value (GCCP).

• Experience and reliability: maintain strict latency and availability targets (99.9%), with guardrails for coverage, model stability, fairness, and explanation quality.

The primary business measure was incremental OPS, while ranking and engagement metrics served as leading indicators and diagnostics. This distinction was important because improvements in offline ranking metrics did not necessarily translate into better customer decisions.

Organizational Planning

I lead a 75+ person organization spanning customer understanding engine, recommendation systems, product intelligence, and customer-facing experiences. Delivering HMD required coordinated changes across all layers within my org. In February 2025, I convened my senior leadership team including Engineering Managers, Principal ICs, and applied scientists to assess readiness to deliver HMD. My goal was to evaluate current technical capabilities across existing AI-ML portfolio systems within my org, specifically Customer Understanding Engine (CUE), Generative Recommendations (GenRecs), Product Understanding (Ping), and Intent Experiences (MCX), to identify gaps, and allocate responsibilities. This process also guided headcount allocations and capability investments.

I empowered my leadership team to make detailed technical decisions within a clear framework of business goals, success criteria, and constraints. I deliberately separated system-level decisions, which I owned, from component-level decisions, which I delegated to senior engineering leaders closest to each problem.

By late February, I co-drafted a two-year vision document with my 2 PEs outlining cross-pillar architectural changes, headcount, milestones, and sub-goals, which cascaded to team-specific roadmaps aligned with HMD’s objectives.

Technical Strategy & Execution

CUE – Customer Understanding Engine

To support HMD, CUE needed holistic customer memory and intent planning capabilities.

Customer Memory:

• Captures both long-term preferences (stable tastes, budget, brand affinity) and short-term contextual needs (current mission).

• Implemented via a multi-agent LLM system:
  • Construction Agent: Builds structured long- and short-term memories.
  • Retrieval Agent: Fetches relevant memories for active missions.
  • Revise Agent: Updates memories in real-time.
  • Validation Agent: Ensures consistency and resolves conflicts.

Trade-off: Chose compact phrase-based representations over narrative descriptions to optimize update speed, embedding retrieval, and scalability.

Intent Planner:

• Elevates raw signals (searches, clicks, dwell time) into goal-level intents with structured metadata and natural-language explanations.

• Utilizes fine-tuned LLMs with RLHF and RLAI for reasoning, filtering, and confidence calibration.

• Example: Frequent browsing of dumbbells and yoga mats could be elevated into “create a compact home fitness space,” factoring in long-term behavioral patterns and budget signals.

Trade-offs: Prioritized precision and interpretability over model creativity, ensuring recommendations were actionable, explainable, and trustworthy.

GenRecs – Recommendation Engine

GenRecs required enhancements to handle short- and long-term intent, temporal behavior, and sequence interactions:

• Transformer-based sequence ranker: Captures sequential and contextual user-product interactions, replacing standard DLRM approach.

• Reasoning layer: Provides explanations for each recommendation, directly feeding into HMD’s messaging layer.

Architecture: Two-Tower candidate generation (User & Item Towers) → high-recall filter → Light Ranker → L2 Transformer Ranker. Candidates are filtered using Core Attributes to enforce must-have constraints.

Trade-offs: Transformer-based ranker increased compute requirements, debugging complexity, and GPU usage, but enabled contextual freshness and higher-quality recommendations aligned with customer intent.

Ping – Product Understanding

Two key APIs were built to support HMD:

• Core Attributes API: Identifies must-have product attributes per Concept Node (e.g., strollers must have weight capacity, brake system).

• Customer Review Hero Quote API: Selects a single, mission-aligned review using a two-tower transformer embedding approach.

These APIs ensured recommendations satisfied both behavioral relevance and mission-critical constraints. Trade-offs included balancing recall vs. precision and inference latency vs. coverage.

Intent Experiences, MCX – Front-End & Reasoning

Key components:

• Buy This Message (BTM): Personalized natural-language explanation of recommendations, highlighting trade-offs for Budget and Premium options. Generated in real time using a fine-tuned Mistral/ Sonnet LLM pipeline.

• Guided Product Service (GPS): Java-based microservice orchestrating content aggregation, recommendation sourcing, reasoning, and caching (Redis) for low-latency delivery.

• Cards Platform UI: Modular front-end rendering of HMD content, integrating GPS output for dynamic recommendations, BTM, and Hero Quote.

Trade-offs: Chose Redis for caching over disk-based storage to meet strict end-to-end latency budgets, balancing consistency with ephemeral mission-specific payloads. We accepted cache invalidation and eventual-consistency complexity in exchange for predictable low-latency serving and a better revisit experience.

Across the system, we optimized for three non-negotiables: customer confidence over coverage, interpretability over model expressiveness, and latency over freshness. All architectural decisions flowed from these constraints.

Actions I took Specifically:

• Single-threaded ownership & leadership
  • Single-threaded owner for Help Me Decide, accountable for the end-to-end technical strategy, architecture, execution, cross-org alignment, and business outcome.
  • Co-authored 2-year vision for HMD as an AI feature - to improve the customer experience by creating tools that make shopping easier and more enjoyable.
  • Set the system-level decision framework, defined architectural principles and success criteria, drove cross-pillar trade-offs, and delegated component-level ownership to managers and senior ICs so teams could execute autonomously at a high bar.
  • Made an explicit decision not to optimize for catalog-wide coverage in v1, prioritizing confidence over breadth because a low-confidence Top-1 recommendation could erode customer trust more than limited availability.

• Cross-org collaboration & program execution
  • When product, science, and engineering priorities conflicted — particularly around recommendation coverage, model quality, and launch timing — I established explicit decision criteria and made the system-level trade-off rather than allowing individual pillars to optimize locally. The most consequential example was prioritizing high-confidence recommendations over broad catalog coverage for v1.
  • Led bi-weekly pillar reviews (GenRecs, CUE, PING, Miss) and monthly VP updates to track progress, metrics, and risks.

• Key technical decisions I drove
  • Guided GenRecs team to use a Transformer-based sequence ranker over DLRM to support temporal modeling, Top-1 decisions, and explainable reasoning, accepting higher cost for materially better quality.
  • Guided MissionCX team to use Redis-based caching in Guided Product Service to reduce latency, stabilize recommendations, and improve revisit experience.
  • Guided CUE team to evaluate Customer Memory as a first-class primitive to enable durable personalization and explainability.
  • Chose Sonnet as the primary Buy This Message model after evaluating it against a fine-tuned Mistral model. Sonnet provided stronger general reasoning and contextual explanation quality, while Mistral provided greater control for compliance-sensitive and rewrite-heavy paths. Rather than standardize on a single model, I chose a workload-specific model strategy, accepting additional operational complexity in exchange for quality and control.
  • Positioned HMD as a decision system optimized for confidence, trust, and long-term extensibility.

• Operational readiness & launch
  • Co-authored Technical FAQs and led ORRs (Operational readiness reviews) across all pillars with strong dashboards, testing, canaries, and integration coverage.
  • Validated GenRecs reliability and GuidedProductService observability, SLOs, fallbacks, autoscaling, and rollback paths, gating residual risks behind P0/P1 actions.
  • Established ranking, engagement, and business metrics to measure success of HMD as a feature in production. Ranking quality was tracked with NDCG@K, Precision@K, and Top-1 Accuracy to ensure the primary recommendation was relevant and drove add-to-cart or purchase. Business impact was measured via OPS (immediate revenue) and GCCP (long-term value). Supporting metrics included CTR, reduced session duration, and engagement with explanations, with guardrails on latency, fairness, coverage, and model stability to balance customer experience and business outcomes.

• Launch governance & recognition
  • Secured legal approval and ensured delivery was recognized and celebrated across the org.

Outcome

After presenting the final demo to Doug Herrington, CEO of Worldwide Amazon Stores, on Oct. 23, 2025 Amazon announced Help Me Decide, a new AI-powered shopping feature that delivers clear product recommendations with the tap of a button, saving shoppers time and boosting confidence at checkout. I collaborated with PR team on content, legal team with approval, and put together Q&A guide for customers and internal partners - ensuring the launch announcement was going out smoothly.

• In a three-week U.S. A/B experiment, HMD generated $451M in OPS while meeting the experience and reliability guardrails established for launch. This represented the highest OPS recorded for a single Amazon weblab in 2025.

• Highlighted in Amazon Q3 earnings release as a strategic AI initiative.

• Featured in press coverage: Axios, Bloomberg, TechCrunch, Engadget, and more.