212: The Causal AI revolution and the boomerang effect in marketing decision science with Tobias Konitzer

Summary: Tobi challenged marketing’s fixation on prediction. He has built highly accurate LTV models, but accuracy alone does not move revenue. Marketing is intervention. Correlation shows patterns; causality tells you what happens when you pull a lever. That shift reshapes experimentation, explains why dynamic allocation can outperform static A B tests, and highlights how self learning systems can backfire or get stuck in local maxima. It also fuels his skepticism of unleashing agentic AI on historical data without a causal layer. If you want to change outcomes instead of forecast them, your systems need to understand levers and log decisions you can actually audit.

• (00:00) - Intro
• (01:22) - In This Episode
• (04:07) - Why Predictive Models Fail Without Causal Inference
• (09:49) - How to Validate Causal Impact on Customer Lifetime Value
• (13:04) - Reducing Uncertainty Around Causal Effects by Optimizing Levers, Not Labels
• (17:01) - Why Dynamic Allocation Works Better Than Fixed Horizon A B Testing
• (31:54) - The Boomerang Effect and Why Uninformed AI Sabotages Early Results
• (40:15) - Escaping Local Maxima and The Failure of Randomly Initialized Decisioning
• (44:04) - Why Agentic AI Trained on Data Warehouse Correlations Reinforces Bias
• (49:00) - The Power of Composable Decisioning
• (53:06) - How Machine Decisioning Transcends Marketing
• (01:01:41) - Why Clear Priority Hierarchies Improve Executive Decision Making

About Tobias

Tobias Konitzer, PhD is VP of AI at GrowthLoop, where he’s chasing closed-loop marketing powered by reinforcement learning, causality, and agentic systems. He’s spent the past decade focused on one core problem: moving beyond prediction to actually influencing outcomes.

Previously, Tobi was Chief Innovation Officer at Fenix Commerce, helping major eCommerce brands modernize checkout and delivery with machine learning. He also founded Ocurate, a venture-backed startup that predicted customer lifetime value to optimize ad bidding in real time, raising $5.5M and scaling to $500K+ ARR before its acquisition. Earlier, he co-founded PredictWise, building psychographic and behavioral targeting models that drove over $2M in revenue.

Tobi earned his PhD in Computational Social Science from Stanford and worked at Facebook Research on large-scale ML and bias correction. Originally from Germany and based in the Bay Area since 2013, he writes frequently about causal thinking, machine decisioning, and the future of marketing.

Why Predictive Models Fail Without Causal Inference

Prediction dominates most marketing roadmaps. Teams invest months refining churn models, tightening confidence intervals, and debating which threshold deserves a campaign. Tobi built an entire company on that logic. His team produced highly accurate lifetime value predictions using deep learning and granular event data. The forecasts were sharp. The lift curves were clean. Buyers were impressed.

Then lifecycle marketers asked a more uncomfortable question: what action should follow the score?

A predictive model encodes the current trajectory of a customer under existing policies. It describes what will likely happen if nothing changes. Marketing changes things constantly. The moment you intervene, you alter the system that generated the prediction. The forecast reflects yesterday’s conditions, not tomorrow’s strategy.

> “Prediction tells you the future if you do nothing. Causation tells you how to change it.”

Consider the Prediction Trap.

On the left, the status quo labels a person as high churn risk. The function is observation. The outcome is a description of what happens if you leave the system untouched. On the right, a lever gets pulled. The function is intervention. The outcome is directional change.

That shift in function changes how you work.

Prediction thinking centers on segmentation:

Who is likely to churn?
Who is likely to buy?
Who looks like high LTV?

Causal thinking centers on levers:

Which incentive reduces churn?
Which sequence increases repeat purchase?
Which offer raises lifetime value incrementally?

Tobi often uses an LTV example to expose the trap. Suppose high LTV customers frequently viewed a specific product early in their journey. A team might redesign the onboarding flow to feature that product more aggressively. The correlation looks persuasive. The causal effect remains unknown.

Several alternative explanations could drive the pattern:

The product may correlate with a specific acquisition channel.
The product may have been highlighted during a limited campaign.
The product view may signal prior brand familiarity.

Only an intervention test can estimate incremental impact. Correlation can guide hypothesis generation, but it cannot validate the lever itself.

Tobi also highlights a deeper issue. Acting on predictions introduces compounding uncertainty across multiple layers:

The predictive model carries statistical variance.
The translation from model features to campaign strategy introduces interpretation bias.
The experiment introduces sampling error.
Execution introduces operational noise.

Each layer adds variability. When teams treat prediction accuracy as the goal, they lose visibility into where uncertainty enters the system. When teams focus on intervention impact, they concentrate measurement on the lever that drives revenue.

Boardrooms already operate in causal language. Incremental ROI is causal. Budget allocation is causal. Executives care about what caused growth, not which segment looked promising in a dashboard. Prediction can inform prioritization. Causal inference determines what to scale.

If you want to move in that direction, adjust your operating model:

Start every initiative with a controllable lever.
Define the action before defining the segment.
Design experiments that isolate the incremental effect of that lever.
Randomized or adaptive allocation both estimate causal lift.
Report impact in revenue, retention, or contribution margin.
Tie every experiment to a business outcome.
Document assumptions and uncertainty.
Build institutional memory around what caused change.

Prediction remains useful. Intervention drives growth. Teams that understand that distinction build systems that learn through action instead of watching the future unfold from the sidelines.

Key takeaway: Anchor your marketing engine in causal experiments. For every predictive score, define the specific action it informs, test that action against a control, and quantify incremental lift tied directly to revenue or retention. Replace segment rankings with lever performance dashboards that show effect size, confidence, and business impact. When every campaign answers the question “What did this intervention cause?” your team shifts from observing trajectories to shaping them.

How to Validate Causal Impact on Customer Lifetime Value

Most teams treat high LTV segments as proof of where to spend. The model ranks customers. The top decile looks profitable. Budget flows upward. Tobi described asking the head of CRM at a billion dollar outdoor brand what he does when a model predicts someone will be high LTV. The answer came instantly: Spend more on them, no?

That instinct feels responsible. It also confuses observation with intervention. Introducing the high LTV Fallacy:

On the right side of the chart, you see a dense cluster labeled high LTV customers. Revenue increases with marketing spend. The correlation line slopes upward. It looks clean and convincing. They were going to buy anyway. That cluster may represent customers with higher income, stronger brand affinity, or deeper preexisting intent. Increasing spend toward them can inflate reported revenue while adding little incremental value.

Tobi shared a simple example that makes the risk concrete. Suppose most high LTV customers viewed a specific pair of jeans early in their journey. You decide to feature that product prominently in onboarding. You increase paid spend that drives traffic to that item. Revenue from that segment holds steady or even rises. The story writes itself. The jeans must drive lifetime value.

> “If you tune that lever, will you get a causal outcome? The answer is you don’t know.”

High LTV customers might have entered through a wealthier audience pool. They might have converted regardless of which product they saw. When you push spend toward people who were already going to buy, you shift budget without shifting behavior. The chart’s arrow pointing upward masks that reality. The revenue line moves. The counterfactual remains invisible.

If you want LTV to function as a decision tool instead of a narrative device, your workflow needs to focus on levers. That means disciplined experimentation:

Select one controllable intervention tied to the segment, such as product exposure, incentive size, or message framing.
Randomly assign comparable users to receive or not receive that intervention.
Measure incremental lifetime value against a holdout over a defined period.
Promote only the interventions that generate statistically credible lift.

You build a library of proven actions. You retire interventions that produce movement in dashboards without movement in behavior. Over time, your organization learns which levers actually create value and which simply correlate with it.

Boards increasingly ask causal questions framed as ROI. They want to know what changed because of your action. Segment labels answer who looks valuable. Controlled interventions answer what creates value.

Key takeaway: When your model flags a high LTV segment, treat it as a starting hypothesis. Design a single randomized test that isolates one lever for that group, measure incremental lifetime value against a holdout, and document the effect size. Scale only the interventions that produce measurable lift. Repeat this cycle consistently. You will replace budget reallocation based on correlation with a growing portfolio of verified revenue levers that compound over time.

Reducing Uncertainty Around Causal Effects by Optimizing Levers, Not Labels

Marketing teams often pour their energy into tightening prediction intervals around user labels. They want to say, with statistical confidence, that a specific user will churn or upgrade. They debate model accuracy and celebrate incremental gains in precision. The organization feels progress because the numbers look sharper.

Tobi redirects the spotlight to the lever. Marketing changes outcomes by acting. The central uncertainty sits in the effect of that action, not in the purity of the label.

> “What happens to customer X with that intervention, and what happens to customer X if you had not exposed them to that intervention? That is an unanswerable question at the end of the day.”

You observe one trajectory. The counterfactual remains hidden. That reality defines causal inference, and it defines marketing. A churn score forecasts the future under the current system. A treatment effect estimate quantifies how the future shifts when you pull a lever.

On the left side, uncertainty clusters around labels, who the user is. The old goal tightens prediction probability, for example becoming 90 percent confident that a user will churn. On the right side, uncertainty concentrates around levers, what you do. The new goal tightens the intervention outcome, for example estimating the impact of Message A versus Message B. The arrow pushes uncertainty toward the action.

Most organizations still operate with a layered pipeline:

Train a predictive model that scores users.
Extract correlated traits from high scoring segments.
Translate those correlations into campaign ideas.
Run experiments to validate the ideas.

Each layer introduces variance. The predictive model carries statistical uncertainty. The mapping from correlation to intervention introduces interpretation risk. The experiment introduces sampling noise. When these layers stack, uncertainty compounds across steps. The final decision rests on a chain of assumptions.

Tobi argues for a top down discipline. Start with the outcome metric that matters, such as incremental LTV over ninety days. Frame every model around estimating the causal lift of specific interventions on that metric. Treat predictions as intermediate signals. Treat estimated treatment effects as the primary object of optimization.

When you operate with that mindset, your questions change. You ask how much incremental margin comes from a discount for mid tier customers. You estimate how churn shifts when you adjust message timing. You quantify the distribution of effect sizes for each intervention. You update those estimates continuously as new data arrives. Your uncertainty lives in the spread of expected lift, not in the label assigned to a user.

This structure aligns with how executives evaluate marketing performance. Finance teams care about incremental revenue. Boards care about what changed because marketing acted. Effect estimation connects your modeling work directly to those conversations.

If you run lifecycle or growth today, you can implement this shift with discipline:

Define one primary outcome metric, such as incremental contribution margin per user.
Enumerate the interventions you can deploy over the next quarter.
Design experiments that estimate incremental lift per intervention.
Track confidence intervals around each effect size.
Reallocate traffic and budget toward interventions with the strongest expected lift while maintaining measured exploration.

This operating model feels more demanding at first because it forces you to connect action and outcome explicitly. Over time, it builds a coherent system where every experiment updates the same objective. Your uncertainty becomes structured and cumulative rather than scattered across disconnected models.

Prediction accuracy produces clean dashboards. Treatment effect estimation produces revenue movement.

Key takeaway: Shift your analytics from labeling users to estimating the causal impact of your actions. Define a single revenue outcome, list the levers you control, measure incremental lift for each lever, and allocate resources based on expected effect size and confidence intervals. Concentrating uncertainty around interventions creates a direct, measurable path from experiment design to financial results.

Why Dynamic Allocation Works Better Than Fixed Horizon A/B Testing

Dynamic allocation changes the economics of experimentation. In a fixed horizon A/B test, you split traffic 50 50, wait for significance, then scale the winner. The structure feels clean. You get a tidy chart. Variant A beat Variant B. Everyone nods.

Meanwhile, half your users keep seeing the weaker experience while you wait.

Tobi describes that as an efficiency problem hiding in plain sight. If your job is to maximize LTV, revenue per user, or contribution margin, every unnecessary exposure to the weaker variant has a cost. You feel it in the weekly numbers, even if the slide deck looks disciplined.

Dynamic allocation, often implemented through contextual multi armed bandits, behaves differently:

It starts with random assignment because you need unbiased signal.
It reallocates traffic toward the better performing variant as data accumulates.
It preserves controlled randomness so learning continues while optimization accelerates.

That system optimizes while it learns. It does not wait for a ceremonial end date.

“You can learn and you can optimize, but these things trade off each other.”

Academic work, including research by Garivier and Kaufmann presented at NeurIPS, demonstrates that fixed time experimentation followed by scaling underperforms dynamic allocation. The compounding effect of reallocating traffic early drives higher cumulative reward. For a lifecycle team chasing LTV, that difference is material.

But it’s not as simple to explain.

On the left, you see the old world. A bar chart. A beat B. Easy to explain. The narrative fits neatly into a CFO update.

On the right, you see flows bending over time. Traffic shifts. Variants rise and fall dynamically. Harder to narrate. Higher total reward.

That tradeoff is psychological.

Large, sophisticated companies understand the math. They still choose fixed horizon tests because the story is easier to defend in a boardroom. A temporally separated “test phase” and “rollout phase” maps to how humans reason about cause and effect. Dynamic allocation requires comfort with probability distributions that update continuously. Many executives prefer a clean headline over a shifting curve.

Tobi sees this as the real boundary of decisioning science. Your model can be optimal. Your organization still needs to adopt it. If a bandit reallocates traffic in real time but leadership shuts it down because it feels opaque, the revenue gain never materializes. Optimization in code must pair with explainability in narrative.

If you own experimentation, your mandate includes both.

Key takeaway: Dynamic allocation works because it reallocates traffic toward higher performing variants as evidence accumulates, which increases cumulative revenue instead of freezing half your audience in a weaker experience. When you shift the conversation from “Did A beat B?” to “Did we maximize total reward while learning?”, you align optimization with executive decision making and unlock compounding gains.

Agentic AI Risks
The Boomerang Effect and Why Uninformed AI Sabotages Early Results

Reinforcement learning systems in marketing often begin with random initialization. Teams define a small set of treatments, activate dynamic allocation, and expect performance to improve as data accumulates. Tobi focuses on the earlier step that most teams gloss over. He asks who defined the treatment universe and what causal evidence supports those choices.

In many organizations, that initial set comes from instinct and pattern recognition. A marketer proposes a handwritten birthday note for dog owners. Another suggests a grooming coupon. Someone else recommends a dog walker voucher because high value customers often engage in pet related activities. Each idea correlates with strong spenders. None of those correlations guarantee a positive causal effect.

When those treatments enter a reinforcement learning loop without causal priors, the system explores them at scale. Some ideas will backfire. A dog walker voucher can feel intrusive. A customer can interpret it as an assumption about their personal life. That customer may reduce spending or leave entirely. The algorithm detects the negative signal over time and shifts traffic away, but revenue absorbs the damage during the learning phase.

The pattern shown above reflects a common trajectory:

The system starts blind and distributes traffic across treatments without informed priors.
ROI drops below baseline as uninformed decisions generate negative lift.
The model reallocates traffic as feedback accumulates and performance recovers.

Tobi has observed this dynamic repeatedly in lifetime value programs. Early decisions depress LTV because the system lacks structured knowledge about which interventions are directionally sound. He describes this as the boomerang effect, where uninformed exploration produces a measurable dip before the model converges.

> “Your initial idea backfires causally. The system will learn, but in the meantime you have a lowering of LTV.”

The cold start problem compounds this risk. Reinforcement learning optimizes within the set of treatments you provide. If that set is narrow or poorly grounded, the system converges on the best option available inside that limited space. It cannot identify a superior intervention that was never proposed. Optimization amplifies the quality of your starting assumptions.

Tobi proposes a structural solution in the form of a Causal Customer Context Graph, something his colleague Anthony previously explored. This graph logs prior experiments with randomized assignment and explicit counterfactual outcomes. Each treatment is stored alongside the outcome of comparable customers who did not receive that intervention. Creative context, channel, timing, and other attributes are embedded so the system understands similarity across experiments. The graph encodes causal memory rather than raw correlation.

When you initialize reinforcement learning with that structured history, early allocation reflects informed priors instead of random guesses. Exploration still occurs. The system still balances learning and revenue. However, the starting point incorporates proven causal effects from related contexts. Revenue volatility decreases because the model no longer pays for ignorance with your top line.

Operationally, you can implement this in three concrete steps:

Log every experiment with randomized control and treated groups, and store the counterfactual outcome alongside the observed outcome.
Capture treatment metadata such as creative elements, offer type, channel, and timing so similarity between interventions is measurable.
Initialize new decisioning systems with weighted priors derived from prior causal lift rather than equal traffic splits across all arms.

These steps compress the depth and duration of the negative ROI dip. You still explore. You still converge. You do so with a memory layer that encodes what has previously moved the needle.

Reinforcement learning systems can generate durable gains in LTV. They can also degrade performance during early learning if they start without causal structure. The boomerang effect reflects design choices, not randomness.

Key takeaway: Random initialization in reinforcement learning introduces a predictable window of negative ROI because early decisions lack causal grounding. Before activating dynamic allocation, build a causal memory layer that stores randomized experiment results with counterfactual comparisons and rich treatment context. Initialize traffic using priors derived from demonstrated causal lift. When early exploration is informed by structured causal history, you reduce revenue drawdowns and accelerate convergence toward interventions that meaningfully increase lifetime value.

Escaping Local Maxima and The Failure of Randomly Initialized Decisioning

Reinforcement learning systems optimize exactly what you tell them to optimize. They begin with random initialization, distribute traffic across a fixed set of interventions, and then reallocate traffic toward the arm that shows the highest return. The mechanism works efficiently. The constraint sits in the boundaries you define.

> “It ultimately allocates the traffic to the intervention that maximizes returns, which is good, but by definition, that is a local optimum.”

The system evaluates a closed universe of treatments. It does not imagine new ones. If you define three arms such as a grooming discount, a handwritten birthday card, and a dog walker gift certificate, the algorithm will converge on the best performer among those three. It cannot evaluate a subscription bundle, a dynamic pricing model, or a loyalty tier unless you explicitly introduce them. Optimization always occurs inside the search space you construct.

We had a fantastic episode with the Chief Growth Officer at WealthSimple, Simon Lejeune, where he explains local maxima and says understanding it is one of the most important things for a growth marketer. 

The small hill represents a local maximum driven by clicks and opens. On the right, the larger peak represents a global maximum tied to revenue and lifetime value. A reinforcement learning system often converges on the smaller hill because shallow metrics move quickly and provide fast feedback. Revenue and lifetime value evolve more slowly and require longer measurement windows. When you optimize for speed of signal rather than depth of value, the algorithm stabilizes on the closest measurable lift.

For our lifecycle friends: let’s say a team launches a contextual bandit to optimize email creative. Early data shows that subject line A lifts open rate by four percent. Traffic shifts heavily toward that variant. The dashboard stabilizes and variance drops. Internal stakeholders see consistency and assume progress. Meanwhile, the business metric that matters, incremental revenue per user, moves marginally. The system converged efficiently on engagement rather than durable value because engagement was the defined reward.

The local maximum problem rests on three inputs that you control:

The candidate interventions you propose.
The reward metric you optimize.
The exploration budget you tolerate before concentrating traffic.

Each input narrows or expands the mountain range. A narrow intervention set produces shallow peaks. A vanity metric compresses the landscape into small hills. Limited exploration accelerates convergence before meaningful alternatives receive evaluation.

Tobi frames this as a structural trade off between exploration and exploitation. Every reinforcement learning system must decide how aggressively to scale a winning arm while still reserving traffic to test alternatives. Full exploration across all possible interventions is computationally and operationally infeasible. Full exploitation on the first positive signal locks the system into a constrained optimum. Marketing leaders must decide where to sit on that spectrum.

You can diagnose whether your system is stuck by running a disciplined review:

List every active arm in production and assess whether each intervention meaningfully impacts revenue or lifetime value.
Trace the reward function to confirm whether the algorithm optimizes incremental business outcomes rather than proxy engagement metrics.
Measure the percentage of traffic reserved for exploration and evaluate whether new interventions enter the system on a regular cadence.

These steps expose whether your model climbs the tallest mountain available or stabilizes on the nearest measurable bump.

Key takeaway: Reinforcement learning converges on local maxima when you restrict the intervention set and optimize shallow metrics such as clicks or opens. Define your reward as incremental revenue or lifetime value, introduce materially new interventions on a fixed schedule, and protect exploration traffic before concentrating on a winner. When you expand the search space and anchor the objective to business value, the algorithm can pursue higher peaks rather than settling for the closest hill.

Why Agentic AI Trained on Data Warehouse Correlations Reinforces Bias

Many teams now connect agentic AI directly to their data warehouse and expect intelligent decisions to follow. The reasoning feels efficient. The warehouse holds centralized data. The schemas look clean. The dashboards update in real time. The assumption is that an agent trained on this foundation will discover growth opportunities on its own.

Tobi views that assumption as risky because a warehouse reflects historical patterns, not verified causes. Your warehouse captures what happened under prior targeting rules, budget allocations, and creative strategies. It encodes associations that formed under those conditions. An agent trained on that dataset will internalize those associations and optimize around them.

The warehouse feeds the agent. The agent drives marketing actions. Those actions generate new data. That data flows back into the warehouse, shaped by the agent’s previous decisions. The loop reinforces whatever bias existed in the starting data.

Tobi often uses a simple LTV example to make the risk concrete. Imagine your data shows that customers who view “Style X” tend to have higher lifetime value. An agent trained purely on correlation will promote Style X more aggressively. It will allocate impressions, email placements, and homepage real estate toward that product. The system sees a pattern and increases exposure.

The model does not evaluate whether viewing Style X caused higher lifetime value. It sees co occurrence and treats it as signal. If high intent buyers simply happened to browse that product first, the agent will still amplify it.

> “Agentic can accelerate the things that are good. It will also accelerate the things that are bad. And it doesn’t distinguish between the two if you don’t.”

That statement carries operational weight. Acceleration multiplies both productive and destructive decisions. A flawed correlational belief becomes a scaled policy. Over time, your warehouse fills with new data shaped by those amplified actions. The signal becomes self reinforcing. Revenue may rise in narrow pockets while customer quality erodes elsewhere.

Auditability adds another layer of concern. Reinforcement learning systems provide traceable probability distributions over actions and observable reward updates. You can inspect why a specific allocation occurred at a specific time. A free roaming agent trained on warehouse correlations offers far less structured transparency. When performance shifts, the reasoning path is harder to reconstruct. Leadership will ask for a defensible explanation.

Tobi argues that you need a causal layer between the warehouse and the agent. That layer records interventions as interventions and links them to measured effects. It distinguishes between:
Observed behavioral patterns.
Experimentally validated treatment effects.
Contextual variables that modify those effects.

You can build this in stages.
Create a dedicated experiment ledger that logs treatment definitions, control definitions, and measured lift.
Store estimated treatment effects in a structured table accessible to your decisioning systems.
Connect your agent to this curated causal dataset before exposing it to the broader warehouse.
Update the ledger continuously as new experiments conclude.

That sequence grounds the agent in evidence rather than historical coincidence. The agent can still explore and adapt. It does so within boundaries shaped by causal validation instead of inherited bias.

Key takeaway: Build a causal experiment ledger before connecting agentic AI to your warehouse. Log every intervention, its control condition, and its measured effect in a structured, queryable table. Feed that curated causal dataset into your decisioning system first. Then expand to broader behavioral data. This ordering keeps your agent anchored to validated lift, preserves auditability, and reduces the risk of scaling self reinforcing bias across your revenue engine.

The Power of Composable Decisioning

Most decisioning systems still operate on a batch rhythm. Data leaves the warehouse. A model trains in a separate environment. Scores return on a schedule. Marketing teams execute journeys in parallel and hope the model still reflects current behavior. That architecture creates temporal drift and organizational friction.

Tobi argues for collapsing that separation. He frames journeys as causal language. Each step in a journey changes the probability of the next outcome. Learning therefore belongs inside the same environment where journeys execute. When learning and execution share infrastructure, the system updates in the same moment a customer moves.

A data warehouse forms the outer container. Inside it, a circular loop of Learning and Decisioning. 
No off prem training.
No separate model brain detached from the data.
Real time learning as customers experience the journey.

Below that sits the legacy pattern, where data exports to a model training environment and results flow back. Two separate systems coordinate through handoffs. Each handoff introduces latency and misalignment.

Composable decisioning embeds the policy directly inside warehouse infrastructure and existing orchestration tools. Training and traffic allocation run in parallel. Every exposure feeds immediate feedback into the model. Each decision becomes training data for the next decision. The system learns in motion.

Tobi’s perspective carries weight because he has built AI products in isolation before. He has raised capital, hired strong teams, and constructed standalone tools. He now favors retrofitting intelligence into products marketers already use. Marketers think in journeys and orchestration flows. They manage real customers under time pressure. Embedding reinforcement learning into those flows respects how work actually happens.

The architectural shift produces concrete effects:
The model updates as soon as outcomes register.
Decision logic remains co located with raw behavioral data.
Governance improves because data stays inside enterprise boundaries.
Context remains intact because features and actions share the same runtime.

That structure addresses the decoupled learning problem at its root. In a traditional setup, the learning pipeline optimizes on historical snapshots while the execution engine operates in the present. In a composable setup, the same loop governs both. The model observes, allocates, measures, and updates within a single environment.

> "Let's try to get to this parallelism of training and traffic allocation all happens seamlessly in the same breath. Essentially every person who is exposed to this decisioning system will lend intelligence to the system in real time."

Parallelism changes how you think about optimization. Instead of waiting for retraining cycles, you design decision points inside your journey builder. The reinforcement learning policy allocates traffic dynamically. The warehouse captures outcomes instantly. The policy updates continuously. Marketing becomes a living experiment rather than a sequence of static tests.

Teams running lifecycle or CRM programs can apply this logic immediately. Identify high leverage decision points inside your journeys. Instrument those decisions inside the warehouse. Embed a reinforcement learning policy that updates with each exposure. Measure outcomes in real time and feed them back into the same loop. Continuous learning replaces scheduled retraining.

Key takeaway: Keep training and decisioning in the same runtime as your data and journeys. Embed reinforcement learning inside your warehouse infrastructure so that every customer interaction updates the policy immediately. Co location of data, decisions, and feedback reduces drift, improves governance, and compounds performance over time.

How Machine Decisioning Transcends Marketing

Human decision making carries structural bias because cognition operates through shortcuts, emotion, and context. Tobias anchors this in behavioral economics and the work of Daniel Kahneman, who demonstrated that humans deviate from rational models in predictable ways. Mood influences judgment. Timing influences evaluation. Social cues influence perceived competence. A hiring manager in a positive emotional state may rate a marginal candidate higher. A judge under cognitive fatigue may assign harsher penalties. These patterns replicate across thousands of micro decisions and accumulate into systemic effects.

Tobias frames the core issue as auditability. Human reasoning leaves no durable ledger. When a decision is made, the brain does not record the weighting of variables, the confidence interval around the choice, or the contextual signals that shaped the conclusion. Memory stores a narrative. Narrative rarely captures variance. Organizations then inherit decisions without a clear record of how they were produced.

The visual metaphor Tobias uses captures this structural divide. 

One side of a scale holds the dark cloud of human intuition, optimizing for who. The other side holds machine decisioning, optimizing for what. The distinction maps to real behavior. Humans frequently optimize around identity, pedigree, similarity, and narrative coherence. Machines can be designed to optimize around defined outcomes and logged variables. When the objective function is explicit and the data inputs are stored, the reasoning process becomes inspectable.

> “You can go to these systems and say, what did you know at the point you made the decision, and why did you make it, with numbers.”

That capability changes governance. When a model recommends a treatment plan, you can retrieve the features it used and the probabilities it assigned. When a hiring algorithm ranks candidates, you can examine score distributions across demographic groups. When a sentencing model produces risk estimates, you can measure disparate impact across race and income. Bias becomes observable through data rather than inferred from anecdote.

High stakes domains amplify the importance of this structure:

In healthcare, treatment allocation models can log inputs, predicted risk, and outcome targets. That record supports periodic fairness audits.
In hiring, candidate scoring systems can store feature weights and evaluation thresholds. That record supports bias analysis and model recalibration.
In criminal justice, risk assessment tools can archive probability distributions and decision rules. That record supports external review and public accountability.

Delayed feedback complicates design. Recidivism rates and long term health outcomes unfold over years. Reinforcement learning in these environments must account for sparse and unstable reward signals. Tobias views this complexity as a reason to strengthen audit mechanisms rather than weaken them. Logged objectives and stored decision states create institutional memory that persists beyond individual actors.

You can operationalize this mindset in your own systems. Start with governance, not hype. Define explicit objectives. Log model state at inference time. Review outcomes on a fixed cadence.

A practical sequence looks like this:

Define the measurable outcome that guides decisions, such as retention, survival rate, promotion rate, or recidivism.
Log every automated decision with its input features, model parameters, and predicted probabilities at the moment of inference.
Conduct periodic bias audits across protected attributes and compare error rates and outcome distributions.
Adjust objective functions and constraints when disparities exceed defined thresholds.
Document changes so that governance becomes part of organizational infrastructure.

Machine decisioning introduces accountability through structure. Logged data, explicit objectives, and inspectable probabilities create a framework where bias can be measured and managed. Human intuition remains valuable in framing objectives and interpreting results, but system level decisions benefit from auditable logic.

Key takeaway: Define explicit outcome targets, log model inputs and probabilities at every decision point, and review performance across demographic groups on a scheduled cadence; auditable machine decisioning converts bias from an invisible cognitive habit into a measurable system variable that can be governed and improved over time.

Why Clear Priority Hierarchies Improve Executive Decision Making

Energy is a finite resource, and senior operators burn through it faster than they realize. Tobi spends his days building reinforcement learning systems, debating causal inference, and designing machine decisioning frameworks that optimize outcomes at scale. When the conversation turns to his personal system for deciding what deserves his energy, the answer is immediate.

> “My girls always deserve my attention. If there’s any question there, it always goes in favor of the girls.”

That statement functions as a hard constraint in his life. It sits above product roadmaps, board meetings, and AI research. It eliminates ambiguity before it appears. Many leaders talk about balance, yet they treat priorities as fluid. Tobi treats them as ordered. When two commitments collide, the hierarchy resolves the conflict without drama.

He frames the rest of his work through a pragmatic lens. He thinks deeply about causal models, marketing effectiveness, and civilizational progress. He also recognizes the scope of his domain. “We do marketing for the internet,” he says. That sentence reduces ego and sharpens perspective. Marketing can create value, fund teams, and improve customer experience. It carries bounded risk and bounded consequence. That framing regulates stress and clarifies trade offs.

His professional focus on bias and noise shows up at home in subtle ways. He studies how human decision making drifts from rationality. He designs systems that log decisions, audit assumptions, and reduce variance. Then he watches his children develop preferences, reactions, and early heuristics in real time. He sees how quickly patterns form. He sees how easily adults cement them. That observation reinforces his belief that humans carry structural bias. It also grounds him. Family becomes the context that keeps his ambition proportional.

If you examine his system, you can break it into practical components:

Establish a non negotiable priority at the top of your hierarchy.
Define a moral baseline for your work, such as do no harm and net positive contribution.
Assess the real stakes of your domain before amplifying stress.
Accept that optimization at work exists within a broader life context.

These rules create psychological margin. When your hierarchy is explicit, you stop renegotiating it every week. When your moral baseline is defined, you stop chasing every high status opportunity. When you calibrate stakes accurately, you reduce artificial urgency.

Many high achievers live in a state of constant escalation. Every message feels critical. Every missed opportunity feels irreversible. A clear hierarchy lowers that internal noise. It turns energy allocation into a structured decision rather than an emotional reaction.

Key takeaway: Write down your personal priority stack in explicit order and treat the top item as a fixed constraint. Define a simple moral filter for your work and evaluate projects against it before committing. Calibrate your stress to the true stakes of your domain instead of the perceived urgency around you. Clear hierarchies reduce decision fatigue, protect your energy, and keep ambition aligned with what matters most.

Episode Recap

Tobi came in swinging at something most of us have quietly accepted. Marketing teams obsess over prediction. Who will churn. Who will become high LTV. We celebrate model accuracy, then sit there unsure what to actually do with it. He has built those systems himself. They were precise. They were impressive. They still left marketers staring at a dashboard asking what now.

Marketing should be about intervention. You are not paid to forecast the status quo. You are paid to change it. Correlation tells you what high value customers tend to do. Causality tells you what happens when you push a lever. If your best customers all looked at a certain product, forcing that product on everyone else might tank performance. Patterns are not prescriptions.

That logic carried into experimentation. Fixed A/B tests feel safe and clean. You split traffic, wait, declare a winner. Dynamic allocation does something different. It shifts traffic toward what works in real time while still exploring. You learn and optimize at the same time. The math favors it. The politics often do not. Executives understand 50 50 splits. Fewer understand bandits.

He also warned about the boomerang effect. Self learning systems start blind. If your initial ideas are weak or intrusive, revenue drops before the model corrects itself. Reinforcement learning can also get stuck in local maxima, doubling down on something decent while a better option sits undiscovered. Exploration and optimization always fight each other.

That tension explains his skepticism about unleashing agentic AI on a warehouse full of correlations. An autonomous system will happily scale whatever looks strong in historical data, even if it is causally wrong. He wants a causal memory layer, a structured record of what interventions truly moved outcomes, so systems start informed instead of guessing.

Underneath all of it sits a bigger belief. Machine decisioning can be audited. Human decisioning cannot. If you can inspect the probabilities and reward estimates behind a choice, you can challenge it. You can improve it. Prediction describes the world. Causal systems try to move it. That is the line he keeps drawing.

✌️ 

—
Intro music by Wowa via Unminus
Cover art created with Midjourney

What is Humans of Martech?

Future-proofing the humans behind the tech. Follow Phil Gamache and Darrell Alfonso on their mission to help future-proof the humans behind the tech and have successful careers in the constantly expanding universe of martech.

[00:00:00] Phil: you've argued that predictive models, so aren't really useful for marketers because they only tell us what is likely to happen if we don't change anything like instead, marketers need to graduate to causal levers.

[00:00:11] Tobi: I mean, I've been obsessed with this problem for the last 10 years, I think marketing is ripe and open to counterfactual and causal thinking. and it does, lead a little bit to the realization that, you know, predictive models may be an input to something, right?

[00:00:24] But it's an intermediate step. if I say, this person has this kind of churn likelihood, What, what does it, what does it do for me, right?

[00:00:32] I need to understand what reduces the churn causally, right? So yeah, sure. I can take this prediction, segment it, and then run an experimentation program on top of it, And I think there is better answers now, and I think we'll get into that know, correlation does not address the jobs to be done by the marketer, which ultimately is all about a causal language.

[00:00:55] ​

[00:01:22] In This Episode
---

[00:01:22] Phil: What's up everyone? Today we have the pleasure of chatting with Toby Connier, VP of AI at Growth Loop. Toby holds a PhD from Stanford in computational social science and worked at Facebook Research. He also founded two startups, predict Wise and Later Accurate and Ad Tech startup that predicted LTV, that was acquired by Phoenix Commerce.

[00:01:40] In this episode, we cover why predictive models fail without causal inference, why dynamic allocation works better than fixed Horizon AB testing. The Boomerang effect and why uninformed AI sabotages early results, the power of composable decisioning, and how machine decisioning transcends marketing, all that, and a bunch more stuff after a quick word from two for awesome partners,

[00:02:02] ​

[00:03:40] Phil: Tobi, thank you so much for your time today. I'm really excited to chat.

[00:03:43] Tobi: Uh, Phil, so much to get into, uh, the disastrous effect of Agen ai. One thing I love to talk about, ~um, but you know what, uh, maybe we should start with an O to San Francisco. You'll start light and, uh, uh, it's, it's, it's a great place to be, although it gets a lot of unfair, uh, representation in the press. Uh, why don't we start there highlighting that part.~

[00:03:49] Phil: ~Yeah, yeah. You've been in the Bay Area for a while now, right? I, I was looking at your LinkedIn, like you studied at Duke and then you had internship in New York at Microsoft, but then it was Stanford and you've pretty much been in like Palo Alto Bay area for like over a decade now, right?~

[00:03:49] Tobi: ~Yeah, I stayed. Um, and I think there is something really cool about, you know, Palo Alto is a sleepy town, which you may know it was home of the first tech boom, ~

[00:03:49] ~but I think there is something to be said now about San Francisco actually being the epicenter of technology. And I think that matters. Um, and that makes, I think this area, particularly the urban area, particularly, um, cool and interesting to live in.~

[00:03:49] Phil: ~But originally from Europe and you said you were just traveling recently in, in France, you know, three different languages.~

[00:03:49] Tobi: ~Yeah, that's right. I'm, I'm European, um, which, uh, you know, depending on my day's mood, I, I think that's the answer to all my problems or, you know, the, the bane of my existence. Um, I'm originally from Germany. Um, and I came to grad school, which is initially supposed to be a year long thing, and I never really went back home.~

[00:03:49] ~Um, yeah, along the way. I lived in France for a year, um, and have been in the bay since 2013. So for a while~

[00:03:49] Phil: ~Very cool, very cool trajectory. Uh, for sure. I think a lot of folks end up staying in the Bay Area. I've had the pleasure of visiting, uh, multiple times. I have a cousin that has a startup down there. So, um, something that you did start doing a lot more recently also is writing on LinkedIn. So when we connected, I uh, discovered you through the multi-part series that you were writing on correlation versus causation after you joined Growth Loop.~

[00:03:49] ~And I'm excited to let you dig into that a little bit deeper today. Uh, ~the first thing I wanted to ask you about is the

[00:03:51] 1. Why Predictive Models Fail Without Causal Inference
---

[00:03:51] Phil: prediction trap, if you will. Um, you've argued that predictive models, so a lot of the propensity based models that, uh, companies are using today aren't really useful for marketers because they only tell us what is likely to happen if we don't change anything like the current status quo.

[00:04:07] So we're asking things like who is likely to buy? Convert who's likely to churn. The function is observation, and the outcome is describing the future if we do nothing else. But marketing is all about doing stuff, new channels, new campaigns. So you're saying that instead, marketers need to graduate to causal levers.

[00:04:26] So instead of asking who's likely to buy, we should be asking, how can I make them buy? And instead of saying who is likely to churn, we should be saying, how can I prevent someone from churning? So we're changing the function from observation to intervention, and the outcome is like instead of describing a future, if we do nothing, we're trying to describe how to change that future.

[00:04:47] Do I have that right? Like maybe explain the prediction trap as you see it and why A highly accurate observation of a customer's current trajectory is way different than having the causal levers to actually influence that [00:05:00] trajectory.

[00:05:00] Tobi: Yeah, I think first of all, Phil, that's very well put. I mean, I've been obsessed with this problem for the last 10 years, and it's actually part of my trajectory a little bit as well. You know, um, before joining Growth Loop, I was CEO of a prediction company that did, uh, customer lifetime value predictions.

[00:05:17] And it did it very accurately, and it did it, you know, with, with, uh, with deep learning and, uh, you know, taking, um, user level information, proprietary data, event stream data, all this kind of stuff to get prediction accuracy to the new level. Um, and I, you know, I was fairly new to the, to the field of marketing.

[00:05:36] Um, and, you know, we, we sold into D two C commerce marketers, right? Um, and you know, I, I remember that, you know, we had an easy time selling because we could prove that our LTV predictions were very accurate,

[00:05:49] right? But I remember sitting there with li particularly, um, the lifecycle side with marketers that asked really good questions like, you know, what do we actually do with LTV [00:06:00] predictions?

[00:06:00] Do we target, uh, you know, for, do we target high LTB customers more? Do we, do we give them more expensive products? Do we give them a discount? And, you know, we didn't have an answer to that because, you know, the, the, the nature of predictive models is, as you said, it represents the status quo, right? What will be this customer's trajectory if nothing changes,

[00:06:22] right?

[00:06:22] And quickly realize that obviously the world of marketing is a causal world, as you said, right? Like, that is the job of lifecycle marketers. Um, if I change X, what is can, what is the outcome? And y right? How can I maximize Y otherwise there is really no point for marketing if everything is status quo and forecasting on status quo.

[00:06:46] And that's the extent of it, right? There is no marketing role

[00:06:48] here. Um, and you know, I think we'll get there, but this is not, this is not limited to, to marketing. You know, I would say now where I'm now. Um, I don't understand, for example, [00:07:00] in grad school why I was not obsessed with outcomes-based thinking and reinforcement learning.

[00:07:04] Why did I even, you know, waste so much of my energy on, on prediction problems,

[00:07:08] right? It's, you know, I mean, um, I'll tell you that my, my, uh, my wife for example, works in diagnostics in, um, in healthcare or health tech, and it's the same problem there, right? You can predict that, um, you know, this is representative of a certain cancer, it can do early detection, but the real question is what intervention moves the needle,

[00:07:28] right? Um, and I actually think, you know, the interesting thing about marketing is that, that I think marketers, you know, first of all, they have a really hard job. Um, and second of all, I think they, they, I, I think marketing is ripe for this. I think marketers get it right. I think marketers understand that, um, correlation based, and we should get a couple of examples.

[00:07:50] But correlation based thinking really doesn't move the needle in a meaningful way. Um, I think the other thing that I've discovered is that boardrooms start to [00:08:00] care too. Boardrooms start to ask much more causal questions, right? Um, you know, framed in terms of ROI, but it's ultimately a causal question, which is, you know, what, what did the intervention cause, um, when it comes to the top line.

[00:08:13] So I think there is this, um, this trajectory change of marketing as a, as a field. I think marketing is ripe and open to counterfactual and causal thinking. Um, and it does, you know, coming back to your, to your entry point, um, it does, you know, lead a little bit to the realization that, you know, predictive models may be an input to something, right?

[00:08:36] But it's an intermediate, it's an intermediate step. Um, there is no point, you know, and I, again, let, let's put, uh, let's go to marketing. I'll give you one example and I'll turn it back over to you, right. Um, if I say, you know, this person has this kind of churn likelihood, right? What, what does it, what does it do for me, right?

[00:08:55] I need to understand what reduces the churn [00:09:00] causally, right? So yeah, sure. I can take this prediction, segment it, and then run an experimentation program on top of it, which ultimately, you know, back to my days at Accurate, which is the name of that company that I founded. That was our answer. And I think there is better answers now, and I think we'll get into that deal.

[00:09:16] Um, but, but that I think is what we mean by, by correlation or what I mean by, you know, correlation does not address the jobs to be done by the marketer, which ultimately is all about a causal language.

[00:09:32] Phil: Yeah. Okay. May,

[00:09:33] 2. How to Validate Causal Impact on Customer Lifetime Value
---

[00:09:33] Phil: maybe we can use LTV as an example, uh, to, to kind of unpack this. I feel like LTV is is really close to your heart and, and you were really deep in, in that space. It's, it's often like the gold standard when marketers are thinking of outcomes for experiments or like a research design. Um, maybe like, do you have an example with LTV where like correlative impact on LTV was dressed up as like causal returns?

[00:09:56] Like how, how can we think about that?

[00:09:58] Tobi: Yeah. [00:10:00] Um, I, I do think it's a good example and I'll give you two. So, so one, um, you know, it's this kind of interesting, you know, I said that there is this step change in thinking. Um, it's also true that, you know, six months ago I had conversation with, um, a head of CRM or Lifecycle Marketing of a billion dollar outdoor brand.

[00:10:19] Um, and, you know, essentially we, we asked, you know, or I asked, what, what, what do you do when you know that somebody is predicted to be high LTV? What do you do with that customer segment? And for him, the answer was obvious. He said, oh, you know, what do you mean we're gonna spend more money on that segment?

[00:10:33] That's obvious,

[00:10:34] right? And it was sort of that, that was still the old way of thinking, right? It was if you, if you wanna be technical, it's the wrong answer. So, but what does it mean? So, you know, let's, let's give a specific example. Let's say, um, all customers that are high LTV, um, have looked at a certain, um, product in the, you know, maybe as an entry point, you know, looked at genes, right?

[00:10:58] A certain style of genes, right? [00:11:00] So now you say, okay, well everybody who has ILTV. Uh, has looked at this product in the, the welcome flow. For example, let's kind of make this the mandatory option in the welcome flow, right? Will this causally increase LTV? You don't know, right? It, it is not a causal statement.

[00:11:18] It has there, there could be all kinds of reasons. You know, maybe it is true that, you know, um, the ad that you, that you, uh, uh, that you sent to Facebook advertise this. And you know, that segment was a wealthier segment for all you care about, right? Which we call spurious correlation, right? It's associative.

[00:11:39] Yes, there is a relationship there. But now the key cause of question is if you tune that lever, right, um, will you get an outcome? And the answer causal outcome? And the answer to that is you don't know.

[00:11:54] Phil: Yeah, it's, it's a really good example there. Um, like when I think back of my [00:12:00] earlier days in, in propensity modeling, like the first time I discovered this was, uh, my short stand at, at wordpress.com, we had big company full data team that was servicing and helping the marketing team. And they had built a homegrown ML propensity model that was, uh, attached to our CDP homegrown CDP.

[00:12:18] And like we could try to predict the likelihood that a certain segment of users would churn or would like convert or would graduate from a free plan to a paid plan. And I feel like a lot of. Teams within the company we're obsessing about, uh, uncertainty or like, how accurate is that prediction going to be if I wanna like do this after?

[00:12:39] And like you said, the focus wasn't on. Like what is it, what is, what is the thing that I want to do to like make that outcome change? And.

[00:12:48] 3. Reducing Uncertainty Around Causal Effects by Optimizing Levers, Not Labels
---

[00:12:48] Phil: Maybe you can chat about like the role of uncertainty here, because like, like I said, most teams are like tightening uncertainty around those predictions and labels.

[00:12:56] And if we're gonna rely on the model and this user is gonna churn, like we need to [00:13:00] be at least 90% or 95% confident in that prediction. Like we're all focused on guessing that future user state. But you read a lot about how the goal should be reducing uncertainty around the outcome of the intervention.

[00:13:12] Like we just talked about, not trying to guess that user's tape, because like what do we do with that information? So in other words, like instead of focusing on how confident we are that this user is gonna churn, we should be focusing on how confident we are that this intervention, this new campaign, this new email, this new message will prevent this user from churning.

[00:13:31] How does focusing on the estimated effect of our action, like changing a channel or a new offer, make marketing more predictable? Can you unpack that for us?

[00:13:39] Tobi: Yeah, sure. And I'll, I'll start, you know, I'll start by saying something that I said before, which is, marketing is not an enviable job, right? It is a, it's a hard job. Like what we're talking about is, you know, I think if you, if you wanna be a good marketer, you have to understand the fundamental problem of causal inference,

[00:13:54] which is ultimately what we're talking about here, right?

[00:13:57] What happens to customer X, uh, [00:14:00] uh, with that intervention, or exposed to that intervention, and what happens to customer X if you hadn't been exposed to that

[00:14:08] intervention, Right.

[00:14:09] That is an unanswerable question. Um, at the end of the day. Now to your question, what, you know, when it comes to uncertainty, um, I think this all relates to predictive models being maybe a, um, uh, an intermediate step that is, that is fine towards the end goal, right?

[00:14:30] So you could say something like, um, and, and we, you know, we did this many times. So you could say something like, yeah, I can, I can build, um, associated traits that are associated with higher LTV and I can build hypotheses from there, the cause or result of which then I need to test to a b testing, right?

[00:14:50] So that's fine. But now imagine, imagine kind of the role of uncertainty, right? Um, if you kind of think about uncertainty [00:15:00] for all these different steps. So now I have my predictive model that maybe generates these associative traits of LTV, right? Um, that has uncertainty with it like any. Right. Um, now you are essentially talking about, uh, you know, again, I think that's not a, that's maybe not a clean model, but how do you translate associations into testable hypothesis?

[00:15:21] Right? That is a, um, you know, I'd say that's a statistical model, right? It has uncertainty, um, uh, in it, right? Then you, the next question is, uh, what is the uncertainty around the tests that you're doing? The AB testing, right? Um, it has uncertainty to it all that uncertainty compounds, and that's my issue with kind of treating these, these things as end goals in themselves, right?

[00:15:47] We, we clearly said that they're not, they shouldn't be. They can't be. 'cause it's not what marketers are here to do. It's not the jobs to be done, the marketer, right? But the other issue is, if you think that way, then you have. A couple of [00:16:00] unrelated steps. Um, each of them come with uncertainty. That uncertainty compounds, right?

[00:16:05] As opposed to if you have an outcomes based thinking and you could say, yo, really all I care about, you know, I wanna move X, what happens to Y? Right? I don't care about anything else. That, that's all, that's all I care about as a marketer, right? Um, let me try to, to actually distill all my uncertainty into that outcomes based model.

[00:16:26] Um, that is easier said than done, but I think it maybe shows why, um, the sort of focus on predictive accuracy of this step one. And in general, the focus on predictive models, um, can lead to problems down the road because uncertainty will get out of hand, right?

[00:16:44] Phil: Yeah, so, so

[00:16:45] 4. Why Dynamic Allocation Works Better Than Fixed Horizon A B Testing
---

[00:16:45] Phil: you mentioned AB testing as a way to validate that. So it's not just correlative. We, we know for sure if this intervention is actually gonna affect the outcome in this cohort of people that we're predicting might churn or not. Um, you, you've talked about dynamic allocation when it comes to experimentation as a better way than just doing the standard AB test.

[00:17:06] Um, I, I, I was trying to like figure out like what are other terms to call, like dynamic allocation. I've seen some folks like call this like adaptive experimentation, but it's all like a bucket of reinforcement learning or contextual bas armed bandits. I've also seen folks like these are all terms like.

[00:17:23] Data folks are super familiar with. I think a lot of marketers right now are hearing you say, like, be bandits. And they're like, what the heck are you talking about, Toby? Um, can you walk us through like bandits and dynamic allocation? Um, these are like specific types of reinforcement learning, right?

[00:17:38] Tobi: Yes. Uh, that, that's right. I, and, and now we're could of at the heart of decision or decisioning science, which is really interesting conundrum. Well, you know, back in the day, um, what you did, and this is, this was also I think, the gold standard of the internet age, right?

[00:17:54] UAB tested, you found the winner, and you scaled the winner, right? [00:18:00] Um, and obviously internet companies, first of all, are admirable to that world because there is this immediate feedback loop of data,

[00:18:08] right? I mean, again, we're gonna talk about marketing here. One of the things that's so cool about marketing is that, which also makes a marketer's job even less enviable.

[00:18:16] Unfortunately, there is all the data at your fingertips, right? There is so much data and it's just not equivalent to other fields, right? In medicine, you can't, AB test doesn't work, right? Like you can't, uh, for ethical reasons, for other reasons, but in most fields, actually, it doesn't

[00:18:30] work, right? You can't just, um.

[00:18:33] Uh, you know, test one treatment against another on cancer patients and scale the winning treatment. Right? So it's sort of interesting. I mean, many fields by definition are more stuck in this correlational world because there is no way to get to that causal world. Marketing is

[00:18:48] not one of them. Um, right.

[00:18:49] But to your question, um, what is the difference between dynamic allocation and your right, that is ultimately reinforcement learning and traditional AB testing? Well, traditional [00:19:00] B testing essentially splits the traffic in our world, right? In a random way. So, you know, uh, ID ultimately, it'll lead to something like 50% will see that intervention.

[00:19:13] 50% will

[00:19:14] see that intervention. But the key thing is random assignment. So for every user who flows to the system, the machine flips a coin or you flip a coin, right? And there's all kinds of really great statistical properties that come from this approach, which is now, if you just look at the. The mean return of each condition, right?

[00:19:34] That's causal, that's causal difference, right? That's, you know, that's, um, probabilistically true, right? Um, so if I show, if I essentially randomize and I show you back to our old example, right? I show you the status quo, or I show you this pair of jeans that high LTV customers clicked on before their first purchase, right?

[00:19:53] And I measured against LTV. Now that mean difference is causal great, right? Um, but [00:20:00] you are wasting, you're wasting time or traffic, right? In the inferior condition will get 50% of traffic allocation. And that there is a real cost to it, right? Like, that's the whole, that's the whole trade off of experimentation.

[00:20:15] You as a marketer, you say, you know what? I found this great. I, I am convinced, you

[00:20:19] know, I figured out that, um, sending handwritten cards to, to dark owners, if you're like a pet food company or something, you know, like congratulates them for their birthday. Um, doc's birthday will increase LTV by 15%. Like I'm freaking convinced about this, right?

[00:20:35] Um, now you run the experiment, but if you're so convinced about it, remember you are withholding that treatment from the, the other 50%. You know, that's a real bummer. Um, now the whole point of dynamic allocation or reinforcement learning is that you could do both things in parallel. So as opposed to you run an experiment for a fixed timeframe and then you scale the winner, [00:21:00] right?

[00:21:01] Um, you basically say, okay, well you start random because all you got, right? But dynamically, as soon as there is some data coming back to the system, you actually, um, pipe more traffic to the condition that is winning, right? Which is, um, which at the end of the day is more efficient. Um, but there is also a real trade off here.

[00:21:23] You know, that that's worth noting. We, we call it exploration versus exploitation. Right. Um, these things are traded off of each other no matter what approach you choose. Now, the interesting thing, and that's why I started with decision signs, is, you know, if you wanna optimize in a metric, which again, is the marketer's job, right?

[00:21:42] And again, I, the way that I would describe the lifecycle marketing job is you have to optimize LTB. I mean, what else is there, right?

[00:21:49] Like, that's what you got. You start with a known customer profile in your CRM or CDP or whatever it is now. Um, and, and you max you, you max, you maximize the squeeze, right?

[00:21:58] That's your job, [00:22:00] right? So ultimately, I think marketing is about optimization. Um, and then there's this really neat paper out there, um, that is, um, uh, I think it's Gary Va Kaufman Etal, um, information processing systems that shows that if you do experimentation first and scale later, so you have a fixed timeframe in which you do experimentation and then you scale, you actually suboptimal.

[00:22:23] It's much better to do this dynamic allocation.

[00:22:25] Phil: Hmm.

[00:22:26] Tobi: So here is a question for you. Um, if that is the case, why do some of the most sophisticated companies in the US do this other approach of fixed timeframe experimentation and scale it up? You know, I mean, these guys, these companies like Netflix and, you know, these, these companies have a bazillion of PhDs that understand decisioning science much better than I do.

[00:22:48] Um, and they, they kind of center on this wrong approach.

[00:22:52] And that's an interesting question, isn't it? It's, I think it's one of the paradoxes of decisioning science is applied to, uh, to [00:23:00] marketing writ large.

[00:23:02] Phil: So, so why is it that, that, that happens? Like, is it, like I, I'm, I'm listening to you explain it and I, I get it. Like I want, maybe I can unpack that, maybe I wanna make sure I do get it. So, normal AB tests 50 50 split randomized split, and we're basically, the trade off is we're essentially wasting a lot of potential customers on the weaker option while we're waiting for enough data to reach stat zg, um, dynamic allocation is.

[00:23:31] Shifting more traffic towards whatever is the leading variant, like performing better. But we're still reserving some traffic and we're dynamically assigning that and we're still learning and avoiding being fooled by randomness. Like that's the part where it's like, I get it, but also how do we explain that?

[00:23:49] Like AB test, I feel like is just so standard. Everyone just gets it 50 50 random, that's it. But in the dynamic side, it's like, yeah, we're not randomly assigned it [00:24:00] anymore. We're doing it dynamically, but it's still random and we're still gonna hit stat sig. Like is that the trade off there? Like why are some folks still doing regular AB tests when a lot of people know that there is a better way to do it out there?

[00:24:13] Is it like explainability versus like what we know is actually better?

[00:24:17] Tobi: It is explainability, but I think in a different way. And by the way, you, you brought up a really good point that I sort of omitted. You know, even in dynamic allocation, you essentially say, Hey, you want to build some noise into the allocation process so you can explore right options that even initially are maybe suboptimal.

[00:24:36] And this is exactly this trade off between exploration, exploitation, or I can say it in a different language. Um, you can learn and you can optimize, but these things trade off each other. Right. So very simple example. Let's say you start this thing and you're like, oh, holy, holy cow. The, the handwritten note to the dock owners improves LTV by 20% biggest intervention I've ever come up with.

[00:24:59] You know, [00:25:00] now I'm gonna be CMO of the company very soon, right? So, you know, you put more and more traffic there, right? But the more traffic you put there, the less you learn. So these things do trade off of each other, right? But again, I would make the point that marketing for most instances, if you're a lifecycle marketer, you're not in the business of measuring results.

[00:25:21] You're in the business of optimizing. I think there is, there is edge cases to that. If you work with a software vendor, for example, you know, one of my last companies that was chief innovation officer of was, uh, um, did, uh, estimated delivery dates on the, on the, on the product pages, right? Based on supply chain data and all this good stuff.

[00:25:40] Real time ml, really cool stuff, right? If you, if you sell that. Um, into a, into a consumer facing company. Then the role of the consumer facing company, I think is to really just measure that, right? You can't optimize anything but lifecycle marketer writ large. I think the goal is to optimize, right? You want to, that's, I mean, [00:26:00] every, every CMO has this, or every, you know, director of CRM or whatever it is, VP Lifecycle Marketing has this written to their North Star, right?

[00:26:09] Increase L TB bags. We talked about it. That's

[00:26:10] not controversial, right? Um, so in that case, dynamic allocation is more efficient and there's a mathematical proof in that paper that I mentioned, right? Um, there is something about explainability here though, and that's kind of interesting. So if you look at companies like Zalando and Expedia, and I mentioned, you know, these guys tend to do, um, experimentation and scale.

[00:26:33] And the reason is in, that's kind of the psychological aspect of decisioning science is internal stakeholders, right? Dynamic bandits are harder to explain. So, you know, you, you, you, basically, your goal is to affect the change.

[00:26:49] Phil: Hmm.

[00:26:50] Tobi: say you work, you know, you, you let, again, I don't wanna call out specific companies, um, but let, let's say you've, you've discovered, you know, that, that really, there is a [00:27:00] big benefit in this LTV, the handwritten note example, right?

[00:27:03] Like, let's say that that test comes back positive, right? You're not done yet. You still need to convince the CMO or whoever it is to actually enact that intervention. It's a net no intervention, right? So you'll have to have explainability of the process, right? And it just turns out that in boardrooms and at the executive level, um, that more simple approach of fixed learning and then scaling, it's just easier explained.

[00:27:31] Phil: Hmm.

[00:27:32] Tobi: So it's, it's, it's almost like, you know, it's, it's, it's a human bias that leads to a suboptimal outcome, right? But. If you're the person enacting this, like if you're the PhD who leads decisioning science, right? Um, you're paid to do the change. So your job is to actually convince the stakeholder. And if you do dynamic allocation and you show, you show that, uh, all the traffic goes to one side, but the stakeholder shut you down because it doesn't understand the [00:28:00] principles of dynamic allocation or armed bandits.

[00:28:02] Yeah, I mean, you, you're nowhere, right? Like you won. But it's, but it's not, it did not turn into an actual win. And so, you know, that's one thing that I've been thinking about too that obviously relates to change management, change management and things like that. But it's, but decisioning science, I don't think just means, you know, how can we make the most optimal decision?

[00:28:22] It actually also incorporates, you know, how do we make sure that the decision gets implemented companywide. And that's a psychological aspect. And I think that is why, um, in many of these companies we're stuck with a slightly suboptimal, um, approach, which obviously still beats the status quo. Right. Oh, again, the handwritten example.

[00:28:44] If you, if you, you know, if you spare the cost of the control group because you do dynamic allocation and you find there to be a big winner, but then the change doesn't, is not implemented. Right. That's actually a worse outcome than you running the experiment and, um, and scaling the winner.

[00:28:59] [00:29:00] Right. And that the paper doesn't discuss this because this is, you know, beyond mathematics, it's psychology.

[00:29:04] But I've been obsessed about that too. How,

[00:29:06] you know, where are the limits of decisioning science? And it's not, it's not, you know, math on, on, on, on Global Optima only.

[00:29:15] Phil: Yeah, it is such a cool topic. I, I feel like there's a whole episode to do on like the, the change management side of experimentation and how ab tests are easy to explain. Everyone kind of grasps it 'cause it's been a concept forever. But when we introduce terms like bei bandits, like you start losing folks really quickly, there's probably a whole episode we can do around that.

[00:29:34] ​

[00:31:38] 5. The Boomerang Effect and Why Uninformed AI Sabotages Early Results
---

[00:31:38] Phil: did wanna let you talk about the boomerang effect, uh, related to this. Like there's, you know, a lot of self-learning systems that are plagued by the boomerang effect. You wrote a lot about this. Like they, they start by assigning random and uninformed treatments. How can these systems which lack causal priors, if you will, like, end up sabotaging big revenue [00:32:00] KPIs for weeks and months?

[00:32:02] During this like initial uninformed learning phase, um, before they actually start learning and, and actually start driving revenue. Like maybe walk us through that.

[00:32:10] Tobi: Yeah. Okay. Let, let's, let's do one segue, right? So in your world, now we're in this better world where we do dynamic allocation,

[00:32:17] right? So that's the world that you're describing. So, you know, and I, again, I think I, I think that's probably where marketing almost is, um, particularly at innovative companies, if you don't have to deal with the organizational width that we just described of, um, of, uh, behemoth companies, right?

[00:32:33] Um, so now we're saying great. We have, um, and again, maybe we bring it back to the example of LTV, just to make it a little bit more concrete, right? We have a bunch of different treatment conditions, like we can send the handwritten no to the dog's birthday. We send a, are you a dog owner?

[00:32:49] Phil: Yeah.

[00:32:50] Tobi: Okay. So tell me what other good things there are.

[00:32:52] So I'm, I'm like as creative as a fridge.

[00:32:54] Phil: Um, automated, like food deliveries or like treat [00:33:00] deliveries, uh, dog walker ads, I don't know.

[00:33:03] Tobi: love it like a dog walker gift certificate maybe. So you can go, you know, you can take your spouse on a vacation that has been suffering from

[00:33:10] this dog that you brought into the home. You know, the, the marriage is on the brink of collapse. Right. So, so, but there's, there's a bunch of really good ideas right now.

[00:33:20] You know, we kind of expose the ideas to, and I like your word of reinforcement learning. I mean, the, the industry now calls it AI decisioning, but that's ultimately what it's, right. It's the system essentially starts random, it listens to the results, you know, and then it allocates traffic dynamically more and more to the winning conditions.

[00:33:38] Right. Um, here's the problem though, and I'm gonna turn this back as a question to you. How did you, I mean, actually you chose it for me. So how did you choose the, the, the universe of treatments? How did you do that?

[00:33:51] Phil: Uh, well for this example, it's just off the cuff, but personal experience, like my own personal bias.

[00:33:58] Tobi: And that's probably how many marketers [00:34:00] do something like

[00:34:00] that, right? Um, um, and I don't wanna discard this either. I'm not, you know, intuition and institutional knowledge is a, is a thing, right? It's, it's not, it's, you know, I don't wanna come here and say there's a, um, but there is a danger here, right? Um, the danger is that five strategies that you just told me about actually, backfire.

[00:34:20] Okay? So let's make this more. So, you sent me this gift certificate for the dog walker, and I'm gonna say, what the hell? I mean, this guy thinks my marriage is on the brink of collapse because I brought this puppy, but my wife loves this puppy. You know, how, assuming is this, of this dog food company that's, you know, I'm never gonna buy there again, right?

[00:34:39] Problem, right? I mean, the system is gonna learn it,

[00:34:44] of course, right? You're gonna allocate less traffic to that condition. But in the meantime, you actually have a lowering of LTV,

[00:34:53] because I just went to dog food, company B. That leaves me alone. It doesn't devil in my private life. And I mean, these are [00:35:00] silly examples

[00:35:01] and you know,

[00:35:01] hopefully they're

[00:35:01] Phil: it makes

[00:35:02] Tobi: and, and maybe they're not funny, but, but I've seen this many, many times in my career.

[00:35:07] I've seen this with LTV right? Many, many times where you'll forget decisioning or not forget reinforcement learning or not, but your initial idea is backfire causally, right? Um, so, you know, that is one problem. Even if they don't backfire, let's say they're net neutral, right? You're still wasting space.

[00:35:28] Right? And even worse, and that's kind of an interesting question. You, you are stuck in the, in the treatment suggestions that you made to me, right? How do you explore new things, right?

[00:35:41] Um, these systems are not well set up to answer this. And this is. Where, you know, uh, an idea comes in that, that we've been building towards and I've been obsessed with for a long time, which is tying, I mean, I, we haven't talked about this.

[00:35:54] I, uh, we're gonna hopefully talk about agentic AI a little bit. Uh, oh, I did it. I, I said it in my [00:36:00] opener. You know, I, I am probably a skeptic. um, um, but one thing that I, that I have been really focused on is how to tie agentic AI to reinforcement learning and decisioning. Um, and so there is this thing, um, that we call CCCG, the causal customer context graph.

[00:36:25] Phil: Hmm.

[00:36:26] Tobi: Um, and it's an agentic, you know, it's an agentic context graph, right? But imagine you had this thing listening to all your experiments that you've ever done, right? It could build, um, as a semantic layer, it could build previous effects. Causal effects of similar interventions. Super hard to do, right? But essentially, let's break it down to our problem maybe, and we call it the cold start problem, right?

[00:36:53] You had to, I don't know if I can curse on this show, you had to invent all these different things. Can I curse on this [00:37:00] show?

[00:37:00] Phil: Yeah. Yeah,

[00:37:00] Tobi: You had to pull them out of your ass, right? That's the, that's the, um, but, but the, you know, now imagine that there is a system that says, look, you know, we don't know for sure, but we actually have previous causal statements encoded in this graph.

[00:37:17] And don't send this, this, uh, uh, dark walker thing because it's pretty proximate to something that you ran an experiment on three years ago. And our prior is that this is gonna backfire,

[00:37:29] right? So you can condense the space basically in something that is more prone to work. Now, you know, here is the problem.

[00:37:36] If you do this based on correlational data alone, like basically you say to the agent, look at all the warehouse and data. And initialize, you know, give me 10, 15 starting ideas, right? This thing is gonna basically be as bad as the whole idea of, of correlational decisioning. You know, it's just basically gonna say, okay.

[00:37:58] Um, [00:38:00] you, yeah. I mean, people that like walk their dog often spend more money on food. So we're gonna send them a doc, you know, it's a core. We are gonna send them a dog walker gift certificate, right? Still gonna backfire, but it's gonna base it off of correlational data. Um, you know, like a fun example if is, if you not the marketing, um, uh, core capacities of audience building, right?

[00:38:23] If you put a gen on audience building and you just ask this thing, give me an audience that causally minimizes churn, it'll give you an audience of highly engaged people that always buy. Why?

[00:38:34] Because correlation, they never churn, right? But causally it's, it's, it's nonsense.

[00:38:39] It's probably the worst audience that you can pick. Right. So here's this idea of the CCCG, um, this causal customer context path, right? Um, if, if you embed and log the causal context that you have on every one of their customers in the right way, you can avoid some of these [00:39:00] things, right? And here's I think where it gets really interesting. Um, you could do this across companies that we work with, for example, that you work with as a, as a MarTech vendor, right?

[00:39:13] The old data co-op model, for example, you say, look, um, you don't have to, but it's all Anonym anon, uh, anonymized data anyways, right? It's basically, um, telemetry data off of experimentation. That's really what we're talking about here,

[00:39:26] right? You put into this data co-op, that's what they used to, it was a big business model in Silicon Valley 20 years ago, 15 years ago.

[00:39:34] Um, and you benefit from essentially what everybody else has been testing on already.

[00:39:39] To avoid that boomerang effect. Right. Um, and then I think the related pathway, and I'll, I'll turn it back over to you, but the related pathway is, and kind of touched on this too, is you get stuck in what we will call local optima or local maxima in these systems as well.

[00:39:55] And you need a way to get out of these.

[00:39:58] 6. Escaping Local Maxima and The Failure of Randomly Initialized Decisioning
---

[00:39:58] Phil: Yeah, lo local [00:40:00] maximum is, or local maxima is a really interesting, um, problem, I guess. Like, or, or like one of the downsides of reinforcement learning too, like when I was like looking through how you're writing about the boomerang effect, like, it, it totally makes sense. Like we have this uninformed learning period as the system is basically starting blind and we have random initialization.

[00:40:22] Um, and I immediately thought of like the episode that we did with, uh, the chief growth officer at Wealthsimple last year, Simone Gen. He like called out local maxima as like one of the most important things that marketers need to understand. And I could see it cleanly in that boomerang effect because like, we have this in this random initialization period, like the system is basically blind and it's like, it, it could latch onto the first like, okay, result, like this looks promising.

[00:40:50] Let's just filter traffic dynamically towards that. Then in the long run, maybe that would've been like the local Maxima versus, you know, if we ran [00:41:00] that a bit longer, we could have found something that was a bit more global, like more LTV. Is that where you were kinda learning towards there? Like can you maybe explain like how that random initialization period could potentially latch onto the first tiny positive signals and, and what that could mean?

[00:41:16] Tobi: I mean, any initialization can latch onto the most positive signal, right? Like that's how these things are defined, and I think you'll have it right, is there is a set of fixed interventions. It ultimately allocates the traffic to the intervention that maximizes returns, which is good, but by definition, that is a local optimum, right?

[00:41:36] Or a local maximum. So the question for these systems is, what is your North Star? If, if it, in our example, it latches on to me, what, what was a good example that you, I, I like the, I like the dog walk. The, the, the dog food gift certificate, maybe. Is that one of the things that you brought?

[00:41:56] Phil: Yeah. Yeah. I was thinking too of like grooming, like pet grooming.

[00:41:59] Tobi: oh, I [00:42:00] love it.

[00:42:00] Let's use

[00:42:00] that one. Yeah. I love it. Love it. You know, dogs always, I mean, my in-laws have a dog. This thing always looks like, you know, 10 days of rain. You know, you get a, this really increases LTV, you look at your dog, you're much happier. You know, you, you, you're willing to causally increases the lever of, um, of LTV.

[00:42:18] 'cause now you buy food there all the time, right? Um, but maybe there is an intervention out there that is even better. You sent a doc on a cruise or something like that, I don't know. Um, but how is that system gonna know that it should, you know, first of all, how can it dynamically generate new candidates?

[00:42:36] I think this is a big open question. Um, and then also what is good enough,

[00:42:42] Phil: Hmm.

[00:42:42] Tobi: When it's an unknowable question, you can't know by definition. The only way to know that something is a global maximum is you've explored all possible different options, is not feasible,

[00:42:54] right? Um, so it's, it's, again, I think it's another example of trade offs, which [00:43:00] marketing is full of.

[00:43:01] And that's why it's such a fascinating field, right? There's, we've had many, I mean, there's trade off between learning and scaling or exploration exploitation, right? Um, there is trade off between optimizing on local versus always looking for global. Um, there is. There is the fundamental problem of causal inference, which ultimately is a trade off, right?

[00:43:18] So it's this world is full of trade-offs and that is that that and all the data is there, which makes it so fascinating from an intellectual perspective.

[00:43:28] Phil: Yeah, it's, uh, it, it's tricky. Like the, the downside too is like, you know, finding that global maximum might take forever and like, that's another trade off. There is the time. And so, yeah, it's a, yeah, it's, it's a really interesting space for sure. Um, you mentioned AG agentic a couple times there, so I, I do wanna let you get into that.

[00:43:47] Like,

[00:43:47] 7. Why Agentic AI Trained on Data Warehouse Correlations Reinforces Bias
---

[00:43:47] Phil: there's this growing trend. Uh, I've seen a lot of companies like, let's let agenda AI loose on our data warehouse. We spent all this time like building this source of truth. Finally, like we've removed silos. Let's just like slap agent AI on, on a couple of, uh, uh, on our data warehouse. Um, you've actually called this a terrible idea today and, and also in the series that you wrote on LinkedIn.

[00:44:10] Um, since a warehouse kind of only reflects the company's current status quo, which, uh, in, in some cases may have nothing to do with the causal relationship that you kinda chat out about already, like how does. Having AI train this way, risk accelerating potential negative dynamics, like aggressively promoting a product that correlates with high LTV but doesn't actually cause it like the dog walking example one there.

[00:44:35] Maybe chat about like that agentic trap, if you will.

[00:44:38] Tobi: Yeah, I mean, I, you know, I, I think what you outline is unfortunately a trend and I think it's lazy thinking.

[00:44:44] Um, my hope and I, my belief, oh. Look, I think we've all sold into marketing and have been frustrated at times with the position of the marketer.

[00:44:54] Um, but what I try to do here is to give marketers a lot of credit because it's, it's, [00:45:00] it's a very comp, it's, it's a very complicated business.

[00:45:03] My hope is that the market will see

[00:45:05] this as lazy thinking, and it's risky too. And that was your question, right? So imagine that, um, basically your data warehouse is full of correlations that causally backfire, right? So again, this example of, um, all the high LTV dog food customers go on a walk often.

[00:45:28] Right. Um, but sending the dog walker with their dogs, which maybe, you know, from a survey or something like that, um, but sending the, uh, the, the gift certificate of a dog walker actually causally backfires because now I think, you know, you're way interfering in my private life, right? Um, so this is what these systems would recommend, and there is a, there is a good paper out there out of Duke that makes this point in a, in a different, uh, um, in a slightly different way.

[00:45:56] Um, but it cannot distinguish if [00:46:00] you don't log it right. It cannot distinguish what is causal and what is correlational. Now, if you put the agentic machinery on top of it, right? We're not talking about this one example, getting into production and, you know, causally hurting your top line, right? We're talking about millions of examples like that, and that's, I think the danger with agentic.

[00:46:21] Agentic can accelerate the things that are good. But it will also accelerate the things that are bad to be really plainspoken. And it doesn't distinguish between the two

[00:46:31] if you don't. Right. This is why, you know, a more cleanly delineate delineated context graph that delineates between these are causal effects.

[00:46:41] This is correlational. Um, you know, uh, is I think the answer here, and that's complicated to build. But if you don't do something like that, the risk I think is disastrous. And I think there's other, or the possible outcome is disastrous. And I think there is other real issues here. Um, one issue for example [00:47:00] is the lack of auditability.

[00:47:02] Yeah. I mean, okay, you go to a land graph and you have an auditability suite, right? I mean, does it answer any of your questions? No. Uh, the nice thing about these decisioning systems about reinforcement learning. Is that I can audit them and we will, you know, that's a bigger point that I have in general.

[00:47:16] You know, no auditing is perfect, but I can say, Hey, at any given point, and you brought it up to, you know, Bayesian multi-arm bend with thumbs and sampling at any given point, my probability, distribution of rewards over arms of outcomes, over arms look like this. And I sampled this value and then I made this decision.

[00:47:34] Right? You can audit it. Um, what a agentic AI does, particularly if you let it go wild on decisioning, you can't audit in this way.

[00:47:42] And, you know, if anybody else claims anything else, it's a lie, right? So, you know, I do think there is this way for these two different trends to interact with each other, but I've described how I see this, the decisioning element is ultimately done by an explainable algorithm, such [00:48:00] as the reinforcement learning techniques that you mention.

[00:48:02] Um, you know. the the generation of the initialization, the cold star, and the generation of treatments that can be done by agent ai if it is done on the correct context. 'cause otherwise you're back in this world that we described, which is, you know, you're stuck in local, uh, local, local Optima. You might be stuck in local mini for all I care about.

[00:48:25] Right? Um, basically all your options backfire and you know you're screwed. Um, and we're back in that world. So, so that's, that's sort of my concern without this semantic layer that teaches the machine what is causal and what is not. It all make up what is and what is not, and you don't wanna be in that world.

[00:48:44] 8. The Power of Composable Decisioning
---

[00:48:44] Phil: So where does like composability fit into this whole like, solution here? Like you mentioned the context graph and how, you know, letting ag agentic loose on the warehouse could be really dangerous. Uh, but something else that you wrote was that like, [00:49:00] journeys are causal language, like a customer journey is causal language and AI has to be able to speak that language to be effective.

[00:49:10] Maybe chat about like the concept of composable decisioning where like the model learns in real time as a customer is moving through that journey without data ever leaving the warehouse. Uh, yeah. Chat about that, that, that problem of like decoupled learning and execution.

[00:49:26] Tobi: Yeah, it's, it's a good point. I think it's a. It's a slightly different point because this goes kind of deep into implementation. Um, and it goes to one thing that, that, you know, certainly I believe should be the, the standard for enterprise grade. And I think most enterprise companies do understand this as their standard, which is how much can we do without any of our data leaving the system?

[00:49:48] Right? Most decisioning systems are built something like, you know, you take the relevant data out, um, you train the model, um, and then you update the model on some sort of [00:50:00] cadence that you predefine,

[00:50:01] right? Um, when we build this thing, uh, we build it according to our, um, you know, I think philosophy on composability.

[00:50:12] Um, and there is a way, you know, uh, first of all. I'm not a big fan of building products from scratch. That's my

[00:50:20] bias in my career. Um, I've been there, done that, you know, I've raised close to $6 million, hired really smart people, went into a dark room and built a product. Um, and you know, I think if you're, particularly if you come from an academic background, that that is sort of the bias,

[00:50:37] but it, but it doesn't work and it shouldn't work, right?

[00:50:40] Because, because marketing is here and has workflows and jobs to be done. Um, and I think right now, this is the journey. I think marketers right now think about their work in terms of journeys and orchestration. Um, maybe audience building, but you know, orchestration. So there was two big questions. One, yeah.

[00:50:59] As [00:51:00] opposed to building a new, oh, Toby invented, you know, AI decisioning, suite tm, you know, please use it. And, you know, um, I, I much rather. Retrofit existing product that has usage because it addresses pain points of the customers with new capabilities. So that's a philosophical 0.1. And I think the philosophical point too is, you know, yeah, can we do it all in such a way that we still don't have to take any data out?

[00:51:26] And, and you mentioned the other point, if I have to take data out, train a model, you have this, um, this asymmetry, right? You there is the model learning step, which, which is a whole different pipeline. And then there is the decisioning at time of inference step, and they're disconnected, right? So we basically said, let's try to get to this.

[00:51:46] Parallelism of training and traffic allocation all happens seamlessly in the same breath. Essentially, every person who's exposed to this decisioning system will lend intelligence to the system in real [00:52:00] time. Right. Um, there is parallelism now. Uh, and also let's, let's use products that we know are there and use, because they address, um, they address customer pain points or buyer pain points.

[00:52:12] And that just, that's just my bias to building AI products because I've been on the other side of this. And, um, you know, if you ask my investors, the outcome for accurate was not, was not, um, a cursor or a, uh, you'll get it.

[00:52:28] Phil: Yeah, I, I love the self-reflection there, like going, getting funding and, and going in a dark room and, and building what you think people are just gonna love. And then just like asking people to, to, to end up using versus the flip side and thinking about it as journeys. I think, um, that's totally the right way to to, to go about it.

[00:52:47] Um, there's a couple more things I wanted to get into, Toby, but

[00:52:49] 9. How Machine Decisioning Transcends Marketing
---

[00:52:49] Phil: I do wanna make time, uh, for like the, the topic of machine decisioning and how it transcends marketing. Um, you, you wrote about that in, in, in some of your posts also, like looking at the bigger picture here, like you said, that machine decisioning is actually way bigger than just marketing.

[00:53:07] We just talked about the marketing use case here, specifically lifecycle marketers. How do you think shifting from like the human gut feel, our intuition to auditable machine decisioning help reduce systemic biases in, in critical areas like, like healthcare. You mentioned, um, like hiring.

[00:53:25] We also have like criminal justice. Um, yeah. Unpack that for us.

[00:53:29] Tobi: Yeah, I mean, I, you know, it's well documented that, um, uh, that human decisioning is biased, um, in, in foundational ways. I mean, this is, you know, the whole. Um, or, or suboptimal, even if it's not biased, you know, just in impacted by noise. Um, this is this whole, this the whole theory of behavioral economics and Don Kahneman, right?

[00:53:53] Um, who is a noble laureate. And I think the first eco, uh, the, the, the, the first sort of, um, noble [00:54:00] laureate and the behavioral economics, uh, uh, economics realm. Um, but it's this idea that humans actually don't behave as rational actors,

[00:54:08] Phil: Hmm.

[00:54:09] Tobi: They can be biased, but there can also be situational factors that influences you.

[00:54:13] And there is many examples, hiring, firing, right? You, you, you add your, you add a really good breakfast in the morning, and your kids, you know, you had really nice interaction with your kids, and you meet this candidate who doesn't meet your standards, but you know, you, you just have, you, you own a good day, right?

[00:54:30] This is not, this is, this is just how the human brain works. I mean, there, there is nothing about, there is nothing to be done about it. Right. And then there is bias decisioning as well. And I think their examples are bound too. I think if you look at, one of my favorite examples is, uh, traffic pullovers, right?

[00:54:46] You bet they're biased and they're biased by race, and it's provable, right? Um, at least the downstream effect that they're biased. Um, sentencing another example you brought up, right? Um, medical treatment allocation [00:55:00] is biased, right? Um, uh, so you have these two big factors that make human decisioning worse.

[00:55:07] And here's the biggest issue. You can't audit it, right? If you hired, I'm not gonna say you as in new Phil, but, but if you hired this guy because you had a really good breakfast, or you let this guy go because you had this awful fight with your wife and it, you know, it, I'm, I'm sorry. It just, it, this is reality today.

[00:55:25] It happens. Right. Um, how, how can you go and say, oh, these decisions are correct or not, or even forget about that. How can you go and say, here are the basis that Le led to this decision and let's revisit them. You can't because it's encoded in your brain and your brain doesn't remember, and your brain is not, you know, meant to be a system of record for these things that is auditable.

[00:55:49] Just doesn't work like

[00:55:49] Phil: Mm-hmm.

[00:55:50] Tobi: right? I mean, you know, my brain doesn't work very well. I've been sick for two weeks where my brain really didn't work, uh, well at all. But the human brains, you know, don't work very well. [00:56:00] The, I think the, the, the promise of machine decisioning is, in the very least, the decisioning process becomes auditable.

[00:56:06] Um, whether it's marketing, right? You can go to these systems and say, what did you know at the point you made the decision? And why the frick did you make the decision with numbers? Right? Um, um, and you can do it for all these different use cases as well. You can do it for sentencing, you can do it for. Um, uh, you know, for, for traffic stops.

[00:56:27] You know, I think there's, there's actually one interesting, uh, um, factor in here as well that, that some of these things have a really, really delayed feedback, uh, reward feedback system, right? Um, uh, like maybe essentially the, one of the benchmarks of how success or sentencing is, what is the, um, what is the, the rate of criminalization,

[00:56:48] Phil: Hmm.

[00:56:48] Tobi: right?

[00:56:49] That takes, I mean, how long does it take to measurement? Depends on your timeframe. It can take years, right? So these systems, and by the way, if that's the case, and we think about this in marketing too, it's a different subject. I'm not sure if you wanna come [00:57:00] back to that, but, um, then you're actually out of the world of multi-arm band.

[00:57:04] You're kind of in the world of these much more unstable systems, mark of decisioning processes, things like that, right? They're just less stable. It's much harder. It's a much harder problem, right? Um, but still, I think, I think that is the potential in in human decisioning, uh, in, sorry, in machine decisioning.

[00:57:21] Um, and I, it also gives me, um, you know, it, it gives me some hope because it is a way to tie agentic AI into something that improves outcomes in a meaningful

[00:57:35] way.

[00:57:35] Phil: Hmm.

[00:57:35] Tobi: And this is certainly one other, you know, there there's two, these two big streams that I'm really interested in marrying, right? Agentic AI and, and reinforcement learning.

[00:57:44] Um, and, you know, I, maybe I'll, I'll say two more words on, on, um, on, on why and, and where I think Agentic AI is a standalone, sort of lacks in atory ambition, [00:58:00] which is where

[00:58:00] we opened, um. You know, to me, and I made the point in, in a recent conference as well, if you look at the first internet era, so, you know, mid nineties in Silicon Valley, um, there was this feeling of, yo yeah, we're gonna make a lot of money on this, but we actually gonna better the world.

[00:58:18] If you talk to anybody who was there, you know, there was this, this, this civiliz atory ambition to make humankind better. And with Agen ai, you know, I haven't seen it at all. It's, it's,

[00:58:29] it's like a monetization play. You

[00:58:30] know, it's a bandwagon that, you know, you want to jump on or, um, or write something nonsensical, uh, on LinkedIn.

[00:58:38] You know, I was, I remember I was in panel discussion with the VC who said, oh, yo, we figured out agenda AI besides observability, and this guy has no clue what he's talking about. It sounds good. It's sound bites. you

[00:58:48] know, you just wanna be on the bw. But you don't think about, um, implications of this technology.

[00:58:54] Right? And there's obviously all these controversies that I'm, that I'm not, that, that I'm not gonna get into because I'm tired of them, [00:59:00] like the Claude Bot controversy that we've been following, right? It's funny, but it, but it, I think it shows there is a void of intellectual voices.

[00:59:08] Okay. Tying this to auditable decisioning in the way that we described here, I think gives agen AI a purpose, um, that actually can lead to civiliz atory improvement.

[00:59:22] Right? Um, and the last thing that I'll say on this is, you know, I am a skeptic. I mean, I, I think, you know, if you wanna take my blanket statement off, if you take Agentic AI as a, as a monolith, um, and you make that statement that it, it has no civiliz story potential or no potential to improve civilization, obviously that's too broad.

[00:59:41] Right? And I do, you know, what I miss is the, is intellectual voices, academic voices that accompany that. Um, there was a fantastic paper out there called The Agentic Economy out of Microsoft Research. That to me, you know, where, where it takes agentic AI as a monolith and it describes how agentic [01:00:00] economy can reduce income inequality and, you know, really lead to a civiliz story improvement.

[01:00:05] Um, I don't agree with the assumptions of that paper necessarily because, you know, I think, um, you know, it, it is sort of hinging on this idea of equals, which, you know, I think that's to be, to be, I'm skeptical on that.

[01:00:17] Um, but at least it is, it is an intellectual, um, contribution that otherwise I've been missing to me though, to kind of close the loop, tying those two things together and saying, you know, we have a auditable pathway to make sure that forget marketing for a second, that sentencing is more fair.

[01:00:35] Right. Let's do marketing that, that I can do better experiences for my customers, that also increase my top line, right? Win-win. Right? Um, uh, if tying that to AgTech AI to help with the, the, the issues that these systems have, the shortcomings these systems have, I think would give it, um, a place in, you know, [01:01:00] redefining what, uh, um, what, what can be done in terms of civiliz story improvement.

[01:01:05] Phil: It's such a cool answer. I I, I could listen to you talk about this for, for a full hour, Toby We'll, we'll only get to the, the paper that you mentioned there. It sounds super cool. Um, but yeah, I, I love your answer to, to ground us back into like, yeah, you know, we, we do marketing for a living, but there's, there's applications that are way bigger than this.

[01:01:23] And, uh, I appreciate that. Um,

[01:01:25] 10. Why Clear Priority Hierarchies Improve Executive Decision Making
---

[01:01:25] Phil: we got one last question for you. Uh, Toby, you're obviously VP of ai, you're an entrepreneur, uh, startup exec. You're a product innovator, but you're also a dad of two young girls. Uh, one question we ask everyone on the show is how do you decide what deserves your energy at any given moment, and what's your personal system for staying aligned with what actually makes you happy?

[01:01:45] Tobi: Oh, Phil, that's an easy answer. Um, my girls always deserve my attention, so if there's any, uh, um, if there's any, uh, question there that always goes in favor of the girls. Um, I also, you know, I, I think you can learn a lot from kids, [01:02:00] right? Even on the things that we're thinking about, sort of from a philosophical angle, um, human biases, human noise that leads to a suboptimal decision making.

[01:02:09] Like obviously we're lost causes, right? We are older, you know, we, but you see it sort of developing in, in kids and, and it's, it's interesting. Um, but then obviously from a, from a personal perspective, I mean, my girls and my family is, is, uh, is uh, is the center of my life. And I, you know, I, you know, I think about marketing and, and causal questions and, you know, civiliz obligatory improvements for sure.

[01:02:35] But at the end of the day, um, we do marketing for internet and I think that's okay. That's fine. Um, uh, you know, um, it's, I think it's not net negative and I think that's, that's probably a good, uh, compass to, to, to judge that what you wanna work on. And not, by the way, that's one of my compass, you know, I'm a pragmatist.

[01:02:54] I, I, um, you know, do no harm is a good place to start, right? But [01:03:00] at the end of the day, it's also marketing for the internet. So our, our tolerance for error is, is fairly high. And, you know, we, um, you know, we we're, we're, we're, we're, we're not solving, uh, the, the fam the, the most recent feminine in Central Africa.

[01:03:12] So,

[01:03:13] so, you know, it's an easy answer. It goes to my girls.

[01:03:17] Phil: I love it. Yeah. Great. Grounding answer there. Uh, I got a, a young, young boy and, uh, and a young girl, so, uh, totally relate to that. Um, To, has been super fun. I really appreciate your time and, and, and prep leading up to this. Uh, yeah. Tons of insights for, for folks in there. And, uh, yeah, well, like we said, we, we prepped, uh, using notebook here and, uh, we will share some of the, the graphs that, that came out of this.

[01:03:41] But yeah, really appreciate your time, Toby, this, this is super fun.

[01:03:44] Tobi: Uh, Phil, agreed. Uh, thank you also for your, for your prep and, and taking the correlation versus causation. Uh, seriously. Um, I am still thinking about how to communicate this, this, uh, in a way that is, that is digestible. Um, maybe this was a step towards it, maybe not, but I very much appreciate the [01:04:00] conversation.

[01:04:00] Phil: Yeah, hopefully. Hopefully a nice step towards that for sure.