Subscribe
Share
Share
Embed
METR sizes an AI's capability by the longest job it can finish, measured against how long a person would take, and through 2025 that length doubled roughly every seven months. A model counts as finishing when it comes out right half the time, so what you get is a coin flip you still have to check.
Show Notes
METR sizes an AI's capability by the longest job it can finish, measured against how long a person would take, and through 2025 that length doubled roughly every seven months. A model counts as finishing when it comes out right half the time, so what you get is a coin flip you still have to check.
Featuring: Margaret Atwood
Brought to you by:
Trends.vc: Discover new markets and opportunities.
HeadsUp: Know when your competitors move. Before your customers do.
Multi: Turn any content into a YouTube video.
Chapters:
(00:00) Atwood on waiting for perfection
(00:05) Choosing Multi's model off a public ranking
(00:18) METR measures how long AI jobs run
(00:40) A model that finishes half the time
(00:48) Launch now and swap in better models
Where to find Dru:
I write an essay daily. Get them free
Follow me on X
Watch my video essays on YouTube
Follow Dru's Notes on Apple Podcasts or Spotify
Subscribe to Dru's Notes for a new episode every day.
Creators and Guests
What is Dru's Notes?
One mental model each day in a few minutes. Habits, business, building products and thinking clearly. From Dru Riley, founder of Trends.vc.
"If I waited for perfection, I would never write a word."
Margaret Atwood
I chose the model that writes for Multi off a public ranking and it still gets things wrong.
Arena ranks frontier models on text, code, image, video and search. New entries keep landing near the top.
A research group called METR measures the longest job an AI can finish on software tasks. They size that job in how long the same work would take a person.
The length doubled roughly every 7 months.
A model counts as finishing when it comes out right half the time.
From 2019 to 2025 that adds up to about a thousandfold.
A tool that finishes your work half the time is a coin flip you still have to check. Each better model takes a bigger piece of the work off you.
Launch your product with the best model now and swap in better models as they arrive. Or keep waiting and better models arrive while you still have no product or users.
What could you launch now, then upgrade as tools improve?