Your CEO wants the AI to "just figure it out" on leads, tickets, or search results. But the second that suggestion costs real money or trust, everyone remembers you're the PM who shipped it. The dirty secret? The smartest PMs treat algorithm choice like a product decision — not an engineering one.
LLMs vs. Classical ML vs. Rules
Pick the simplest one that works.
Rules are boring until the moment they save you. Deterministic, explainable, stupidly cheap. No drift.
Classical ML (gradient-boosted trees, logistic regression) finds patterns in structured data. Fast, cheap, debuggable.
LLMs eat unstructured text — emails, tickets, PDFs. They handle nuance but are non-deterministic, expensive, and will confidently lie about edge cases.
| Rules | Classical ML | LLM | |
|---|---|---|---|
| Data type | Structured, known | Structured + labeled | Unstructured text |
| Explainability | Perfect | High | Low |
| Cost at 1M/day | Pennies | Low | $$$ |
| Failure mode | Misses new patterns | Overfits to training | Hallucination |
| When to use | Obvious 70% | Learnable 25% | Messy 5% |
91% classical ML
94% LLM (but false flags)
Rina's fraud scorer. Rules caught 82%. Classical ML boosted to 91% — deterministic enough for auditors. GPT-4o on narrative notes hit 94%, but falsely flagged two high-value customers. "The model saw sarcasm in the merchant description." LLM kept for enrichment only; classical model owned the final score.