Subscribe
Share
Share
Embed
The NeuralPod, by NeuralRec.ai
The NeuralPod is all about deep diving into technical machine learning topics and showing its real-world impact, bridging the gap between AI-first companies and adoption.
We chat with subject matter experts across reinforcement learning, recommender systems, deep learning and generative AI. As well as chatting with business leaders, VCs, operations and strategy experts.
Who are NeuralRec.ai? NeuralRec is a recruitment staffing agency. We build niche ML teams and represent some of the globe's best ML talent.
Chris: Hi Daniela.
Welcome to the podcast and the show.
Um, thank you joining.
Danijela: Thank you, Chris.
Chris: Great.
Well, um, yeah, let's,
let's just jump into it.
I think, um, for the show, it's gonna
be a mix of kind of your career as
usual, what you've done up to the BBC.
I think you've had a nice
mix of industries as well.
And, uh, it'd be great to go over
some of the, kind of the common themes
that you've seen in machine learning.
Um, I think had a, well you have
a PhD in mathematics and, um, I'm
sure that's provided a, a unique.
Approach to, to machine
learning for over the years.
And, um, you've done some
really interesting stuff around
DeepFakes and, and responsible ai.
Uh, so go through that today.
Um, so yeah, would you, you just like
to start Daniel, about talking people
through, um, your career and, and how you
got to the BBC and what led you there.
Danijela: Yeah.
Well, to be honest, my career has been,
uh, very random and for the longest
time actually, it's all that, that
is kind of what it is supposed to be.
Although very occasionally,
basically you would hear about
really extraordinary people who have.
Um, you know, dedicated themselves
to kind of one line of inquiry.
Like for example, Terrence Tao who
started, um, you know, doing math.
Obviously he was a math prodigy as well,
but he also started very early, or in
sports, it's like a common denominator
where like they all start very early
and they all kind of stick to it.
And, but like in, um.
In our world, in the world of the
information workers, it's very
common to think that, uh, some kind
of Brownian motion is, is okay and
is expected and almost kind of,
um, looked very positively upon.
And I think, you know, until I've,
uh, and I believe the same really,
and my journey has been like
super random, but, um, you know.
But since I've started reading the book
by Richard Humming, which is called
the Art of Science, engineering and
Engineering, he's basically there,
really laid out a mathematical proof why.
Uh, kind of basing your career on
the Brownian motion is like the
worst possible thing you can do.
So he's done like a little proof there
on like if you have, um, after has random
vogue, uh, your expected distance from the
origin after end steps is gonna be, uh,
like of theoral, magnitude of square uban.
Whereas like if you had a, a
tractor, which is like, um.
Like can be, um, basically compared
with some sort of vision or something
like that, you would, uh, your, your
approximate distance from the origin
is gonna be, oh, then, so like, it's.
You know, the half of
order of magnitude more.
And, you know, mathemat, even
mathematically it's so obvious that, you
know, jumping from one thing to another,
um, is like not the best strategy.
And, uh, you know, choosing, uh, uh,
being led by your interest and kind
of sticking with it for much longer,
uh, seems to kind of reap a lot of
bigger returns than, uh, than I think.
What people usually do.
So, you know, going back to, to your
original question, I did a, a PhD in
pure mats, then I've done two postdocs
in, um, kinda more applied side of mats.
And then I've switched to industry
to do some ML first, like, uh,
and then finance and the BBC.
Chris: Nice.
And, um, yeah, you, you kind of touched
on it, they're the industries that
you worked in, o obviously media, the
b, BBC finance, um, and then I think
synthetic biology during your postdoc.
Danijela: Yeah.
Chris: are there any common themes in
machine learning or mathematics that you
see running through that have helped you
lay, um, you know, saw this foundation?
'cause I think you've had
a great career so far.
So,
Danijela: Mm-hmm.
Chris: are the kind of things that
you've seen that have, that have.
Um, that may be unique to the industry
or, or, um, themes that are running free.
Danijela: Mm-hmm.
Mm-hmm.
Well, yes.
You know, I think I've been like super
fortunate to have studied maths, um, like.
At the time when I studied it, and,
uh, simply because like you see that,
uh, the number of scientific papers in
AI NML doubles every year, almost now.
And now even like the, the, the
progress of pace, the pace of
progress is like even faster.
So, which means.
Like for us in the field that, for
example, uh, what is going to be relevant
this year like, is like half of that.
We dunno.
And we have to kind of pick up and uh,
read and study and half of what's been
kind of relevant last year is gonna.
Probably become irrelevant.
And if you kind of, um, um, expand this
kind of time window, uh, let's say like
in 2010, I was just kind of checking.
Before this podcast, it was like
apparently per year in all the
fields, ai, computer science,
and uh, um, machine learning.
There were like around 4,000 papers.
And in 2024 there are
like, uh, 50,000 aura.
Upwards.
Like it's, it's incredible how
much, uh, the field has changed.
So the only basically currency you have
as a researcher is, uh, your ability
to solve problems and to learn quickly.
And to adapt quickly.
And this is something I think,
uh, you know, I've kind of
mastered doing, uh, maths.
And also on top of that, like, um.
I think what really helps is understanding
fundamentals well, because like a lot
of machine learning at today and a
lot of these publications are almost
kind of a, a consequences of, uh,
of this kind of main fundamental, a
couple of main fundamental principles.
So, so if you basically.
Do understand basics really, really,
really well, fundamentals, then
it's easier to build on top of that.
So like organizing your internal
knowledge, um, in a almost kind
of hierarchical way and allowing
yourself to integrate new knowledge
upon that kind of, uh, structural
we have is probably, uh, what's, uh.
What, what can be like, super beneficial.
And that's, that's what the
common teams are basically.
Uh, it's always difficult,
especially like in school.
You don't, um, you don't, or, or
at the university, no one teaches
you how to solve a problem,
especially not in a business setting.
Like they give you algorithms,
theorems, whatnot, whatnot.
Not, and then you are
kind of the, the, the.
The search space is much
narrower, uh, in these settings.
And then you are faced, you know,
you come to business setting and
then you're faced with all of this
kind of real world ambiguity, right?
So what do you do then?
And I think this is where like a
structured scientific thinking helps a
lot in a way, like your discipline, how
you approach the problem, how you, um.
Uh, frame it how you basically,
uh, set up your hypothesis.
A lot of people like just jump into the
problem, especially the junior ones,
and kind of throw a specific model at it
without actually kind of taking a step
back and saying, well, you know, looking
at my data and looking at this problem.
My hypothesis is that the data
is in that shape and that this
model is gonna work like that.
Um, and I expect these results and also
like a discipline of, you know, properly,
uh, baselining, benchmarking and so on.
Uh, so like there are a lot
of things that you can do.
With that kind of scientific mindset
that can help you along the way.
And the difference is like
difference can be like between
three months and 12 months.
And that's the difference probably between
junior and senior because senior knows how
to kind of set up a problem, knows how to
set up a hypothesis, is very keen And, uh.
Kind of zeroing in on ablation tests.
Like you need to have like a solid way
to measure, uh, the effect of any changes
you make to your original experimental
design, be it like, uh, addition of new
features or, you know, like conducting
all, all sorts of experiments until
you basically, um, uh, end up with the,
with the, with the proper solution.
Chris: Nice and super interesting.
I think coming from, um, you know, pure
mathematics background is, is obviously
highly beneficial for machine learning,
but I think other people might choose
computer science or, um, you know, one of
the other popular courses, um, university.
Do you think that, um, you know, having
that mathematics base, uh, coming into
more engineering focus problems, um.
coding, uh, was it, was there
anything you learned, um, you know,
was it a challenge picking up, uh,
programming languages, for example?
Or was it, did that come quite naturally?
Danijela: Well, well, uh, you know, I
was, uh, you know, I kind of graduated a
very long time ago, but like I had, uh,
specific, uh, programming courses ever
since I was in high, in high school.
So.
So it's like you don't kind graduate maths
without actually knowing how to code.
At that time when I was, uh, in school,
we were like looking at c plus plus.
It's on, but that is kind of actually,
uh, well we, we've even kind of studied
assembler, believe it or not, but
um, but definitely that's one aspect
that's, that is something that, uh,
tends to be challenging and that's
what the separation is between like
data scientists or machine learning.
Researchers in a way and kind of
properly formally trained engineers
who are like machine learning
engineers or software engineers.
So there is still that chasm and uh.
Yes, and the collaboration sometimes
can be, um, can have frictions, I guess.
Uh, but, but you know, I believe that as
long as the, uh, separation between the
two, two kind of tasks or the ownership
of the two different, uh, uh, people in
the team is clear, I don't think that.
So that could be an issue, but obviously,
obviously software engineering is a
huge part, especially nowadays when,
um, I think when the need for formally
training models is just coming to nothing.
Like there are just very few places who,
who do it, uh, uh, nowadays, and the,
the majority of focuses on AI engineering
that involves nothing, no training at all.
And it's all about like.
Uh, what's left from the traditional
machine learning is basically evaluation
and benchmarking and everything is pretty
much like, um, uh, engineering in a way,
Chris: Hmm.
Danijela: which is kind of sad.
Uh, and, and it's a little bit of,
you know, ironic that data scientists
and machine learning scientists
are basically, uh, the first one to
face this kind of effect, uh, of ai.
Uh, that's, I, I don't think that's
kind of talked about a lot because we
all hear about these people who kind
of end up with $100 million contract,
but there are, those are like oh 0.001
of the people who used to, uh, work
on machine learning tasks and now,
you know, everyone else has to kind
of shift their domain of operations
slightly and move more into engineering.
Chris: Yeah.
I, I, I think that's a really
interesting topic as well.
You've got, um, you know, really
intelligent PhD students coming
onto the market, but it's now
more competitive than ever.
I think probably slightly
different to when, um, you,
you and I came onto the market.
What advice could you give
to someone, um, you know, how
would you position against ai?
Um, I suppose, you know, it kind of
comes back to being able to link the
model to the business problem and.
Danijela: Mm-hmm.
Mm-hmm.
Chris: But how, how would you position
today if you was coming up the market?,
Danijela: I guess, , going
back to these fundamentals I
mentioned at the beginning, um.
I think, you know, a lot of people
compromise, um, on, on compromise
their interest for the sake of the,
let's say, financial benefit or, you
know, just finding a job and so on.
It's, it's all of ours reality.
But I think there is, uh,
a version, uh, of, uh.
A version of, of this world in
which everyone is kind of, uh, more
aligned with their inherent interest.
I'm, uh, you know, at the moment the
market is very dull if you want to,
like, this is how it feels to me.
Like there are very, very few places
that actually do interesting things.
A lot of enterprise use cases are
focused on, you know, plugging in
commercial models and building agents.
And although that is a.
You know, uh, that can potentially
be, uh, interesting to some people.
I think there is a, a huge group of
people that will find that kind of less,
less appealing, uh, because it's, um, I
think you lose a lot of agency and con
control in that kind of setting and you
kind of revert your work to engineering
rather than kind of this messy creative
process of training the models and, and.
Uh, doing more fun,
more fundamental things.
Chris: Uh, what, what, out of
interest, what, what do you think
the most interesting work in
machine learning is right now to
be done, um, in your perspective?
Danijela: Mm.
Well, it's always, you know, it's always
like at the frontier, that's where the
most interesting work is, obviously.
And if you are lucky enough to end
up in the big labs or even like
big research institutions, um,
you know, that's, that's amazing.
They're also like, I really like the
startup scene, uh, or scale up scene.
Like they are bold boldly
kind of taking on, um.
More kind of ambitious projects.
Uh, and I think this is, this is
where the new innovation or the
new breakthrough must come from.
Uh, you know, obviously the big
labs, they are all kind of going
in in one direction, uh, as we see
kind of scaling up and whatnot.
And we see like the emergence of the
new, uh, startup kind of cohort that is
looking into other ways of achieving, um.
Intelligence, if you will.
Right.
And also, you know, going back to
the agents somehow, I think like to
me it's very, very problematic that
like a large group of people today
think like you can kind of prompt the.
Model to do anything.
You know, even if it's, you know,
even if you assume that it's like
the most intelligent model, right?
Uh, like behind it is, uh, John Neuman,
right behind that model, it's John Neuman.
He can do anything extremely intel
intelligent, and you basically,
uh, give it a prompt, right?
And, uh.
I think this worldview where you think
everything is word bound and can be kind
of put into words, it's completely wrong.
Uh, you know, it's, it might, it might,
it's been held by, you know, the ancient
Greek, Socrates, Aristotle and Plato.
But like, what is then the
role of experience, right?
Like you can.
Theoretically, and that's like the
difference between theory and practice,
between you kind of, uh, hearing
something and then updating your weights.
So I think this element of like, even
if the model, like it's too much to
expect of a model to do things without
actually actively learning in a way as.
Human does.
Uh, so I think there's like
a huge disconnect there.
And maybe a market will correct itself
slightly right there because, um, you
know, at the moment everyone thinks now,
oh, we can do everything with agents.
Okay.
You know, up to an extent that
might be true, but you know, there's
like a huge element of what you do
as an organization or enterprise
that requires wait update in a way.
Um, especially if like.
If like, there are a lot of
very dys, Socratic things your
organization is dealing with,
which there are, there must be.
Right.
, I do hope that the market will,
um, uh, correct itself somewhat and
the people will realize to achieve
the maximum returns, uh, they
will, you know, have to kind of,
uh, shift their approach slightly.
But the question is like.
Yeah.
How, how much like of excellence
they will demand and, uh, how
good is good enough, I guess.
Chris: Yeah, cer certainly a
lot of hype in the market and
a lot of FOMO at the moment.
So like, like I say, I think, I think
there will be a market correction because,
Danijela: Mm-hmm.
Chris: speaking to a lot of companies,
they've, um, you know, invested
a lot into agents and maybe not
got their returns he wanted yet.
And having that human in the loop and
like a, an agent human team seems to
be, uh, definitely the best way to go.
Um.
In terms of AI innovation at BBC, then
I know you've built a great team there.
You've done some amazing work
around, um, LMS and, and DeepFakes
and, uh, yeah, really excited to
speak to you about that today.
Uh, do you wanna just introduce,
um, people to the, the structure
and vision of your, uh, research
teams and, and projects at BBC?
Danijela: So,
my team is around 25 people, and
it's a mix of, uh, computer vision
scientists, natural language processing
scientists, and uh, uh, engineers.
Uh, we have looked at.
Uh, a lot of modern innovations in
the field and have worked on the range
of projects from, um, you know, image
restoration, colorization to, um, neural
radiance fields to in computer vision
as well, to, uh, face anonymization.
And now, finally.
Kind of converged to the deep fake
detection and the work in that kind of,
uh, space of automatic fact verification.
And on the other side, on the LLM side,
we've been, um, on the NLP side, we've
been predominantly, uh, working with
the larger models and try to basically.
Um, uh, see or test their limitations
and see where they could be
useful for the BBC in particular.
And what are the, the maybe drawbacks of
using commercial models and how we can
kind of help the BBC solve these kind of
biggest issues with the commercial models.
So, so I know a lot of people
as, um, kind of voicing this
concern, but for the journalistic
organization, news organization,
I think it is very problematic
to rely on closed source models.
Uh, that are aligned, uh, in a
black box way, in a way which remove
a lot of agency from the user.
So basically it's increasingly, I
think, predominant that the internet
is seen through some sort of AI search,
which inherently will impose the
worldview of that model on the output.
And having people in the journalistic
organization who are kind of committed,
especially at the BBC, who are
committed to impartiality, committed
to, you know, no bias and committed
to the, uh, trustworthiness and truth
is, is very problematic, I believe.
And people, you know, in all
media organizations must be really
careful with the usage of this
closed, um, uh, closed tools.
Chris: Yeah, really, um,
really interesting point and
probably not enough work done.
Um, you know, it's all about how can
we, um, uh, commercialize the models
and, um, make as much money from
them as possible in, in some senses.
Um, but yeah, when I think of the
B-B-C-I-I, I don't necessarily always
think of, um, LLMs, what is, what
are some of the real world use cases,
um, that you are using them for?
And then the, the second question to
that is, you know, how, how do you
handle the bias in, in the models?
. Danijela: So one of the use cases is
the, the, the rewriting in the style
of the BBC, because like if you try
to do that with commercial models.
They would completely kind of go off track
and the style would be nothing like the
BBC style, despite probably the fact that
a lot of our training data is in these
models already in their training set.
Because we do know that, you know, in
common crawl, there are some of our,
um, some of our articles that have
appeared online since 2012 onwards.
So the training data.
Is de facto there, but they still
cannot, uh, pick up that style even
with, you know, if you should prompting
or kind of carefully prompting how to,
uh, how to rewrite a specific articles.
So when it comes to this use case that
we've worked on, it's like, um, um.
BBC has has a service which is called the
local Democracy Reporting Service, whereby
they kind of employ a lot of freelance
journalists that report on the local news.
And then these articles are then
given to the BBC journalists who
typically would just rewrite them
and publish them on our web pages.
Um, and this.
Because we have so many of these
journalists, the volume of this
article coming in is huge, and BBC
can rewrite only around like 10%.
And, uh, you know, we've recently
deployed that model and there has
been a, a press release on that one.
And, uh, you know, the model that we
fine, fine tuned is really much, much
better in terms of the style markers, um,
than, uh, than the out of the box model.
And.
Yes.
And in it's also, uh, relatively
cheaper to run because it's
a smaller model and yes.
Chris: Nice.
Okay, cool.
Well, um, yeah, it's super
interesting stuff and, um,
yeahm moving on to deep fakes.
Um, again, I wouldn't necessarily
associate, uh, BBC with, with DeepFakes
and there's certainly a lot coming
out now with some of the recent models
released by Google and how easy it is to
create a really high quality video with.
Um, you know, ne next to no
budget, and these are just the
things that we're aware of.
Um, you know, fir first of
all, of addressing the problem.
How, how are you kind of seeing, uh,
bad actors use DeepFakes and, what,
what do you see in the market right now?
Danijela: Well, I think, uh, there's
always been bad actors who would kind
of misinterpret or kind of present
events out of context or kind of,
um, put misinformation out there.
It's only that they are now like tool
set has this expanded enormously and
obviously these kind of sophisticated
actors will kind of continue doing
it and there is no automated way to.
So far to kind of capture some of it.
Um, that is, that does not
include the human in the loop.
However, you know, we see that the domain
operation increases significantly, right?
Um, even like one of the interesting
examples at the is that we see like a
huge increase of people kind of submitting
AI generated images to weather watchers.
Dunno whether you know, the service.
Yeah.
So, yeah, there's a service basically
where you, you as a contributor,
as a member of the audience, you
can upload like your own photo that
you took somewhere in England, uh,
with a specific weather conditions.
And if your one is chosen, you
get, you get a mention, right?
We see a huge increase of people just
kind of generating, uh, random, you know,
nature photos with specific weather.
And claiming it's, you know, somewhere
in England and so on and so on.
So, so that's one use case that's
like one obvious, uh, I would say
kind of less harmful, um, uh, way
or, uh, like less harmful, uh, use
case for this, um, uh, deep fakes.
But I guess, you know,
there are like many more.
Many other, more, more dire uh,
situations in which having a robust,
deep fake detector is, is necessary.
Chris: Yeah.
And I, I, I guess the actual
deepfake project that you led,
how, how can you spot a deepfake
Danijela: So we, uh, you know, I, uh,
our computer vision team, uh, led by ju
so, and, uh, I have a couple of super
talented, uh, uh, compu, uh, computer
vision scientists working on that problem.
Uh, need to mention them.
Les and Mark Goret.
Um.
Uh, they've, uh, built this
kind of classification model,
um, on the proprietary dataset.
So we kind of, first of all,
like to build any model.
You have to have a dataset,
uh, that reflects in a way
what you will see at test time.
Uh, so when we started,
such dataset hasn't existed.
So that was the first, uh, uh, first.
Step basically to create a data set.
And then we've experimented with
multitudes of different models.
Obviously when you are building
any classifier, you have to have
a feature extraction method and
then the classifier, uh, bit.
Uh, so we've experimented with many,
many different, uh, architectures.
On that data set that we've created
and, uh, come up with a solution
that is on par or beats most of the
commercial models on, on all of to date
available deepfake, um, generators.
Now for that model to be fit for
use in the newsroom, like you can't,
uh, just deploy it like that and
sorry, and give it to the journalist
and say, this image is open six.
Percent oh 0.6
with the probability of 60% fake or
this the, because like they can't
really deal in these uncertainties
and to make matters, um, averse.
Um.
The models, even if they are like
super accurate, um, on your test
set or validation set in the at test
and at, um, you know, in production
situation, you may end up with like,
uh, out of distribution examples.
That model typically mishandles
or gives like a super hype.
Confidence, like it's super
confident in the wrong predictions.
And I think a bulk of our work has been
in, um, in kind of developing methodology,
uh, to increase trust in these models.
And this kind of breaks down in two
things, which is one is uncertainty
quantification of these models, you know,
kind of building the models that know.
When they don't know in a way and saying,
no, I can't tell you this in a way.
And, um, and then the second bit
is about the interpretability.
Uh, like making sure that you dis explain,
uh, the decision making of the model, how
it came with a specific decision, and then
make that available to the journalist.
Now, you know.
I think that line of inquiry is kind
of becoming increasingly difficult
because, um, you know, unlike with the
traditional image classification, when,
where your, you know, features are, like,
you can see them, they are like visual
semantically available even to a human.
Uh, when it comes to deep fake detection,
usually the signals that lead to a
classification that something is fake,
something is, uh, real, are more.
Uh, are more subtle and they are not
available to you to see via the naked eye.
It can just be like a specific
noise pattern or so on.
So, so this is still a very much
an open problem on which we're
collaborating with, uh, Philip Tours,
uh, group at the University of Oxford.
And, uh, and it's also in the field
a very active, uh, area of research.
Um, but I think, you know.
The market is at the moment
underserved for this specific use case.
It's underserved, um, uh, in terms
of having, um, having uh, deep fake
detection models, uh, that journalists
or media organizations can trust.
And because it's like a high risk
use case, almost like medicine,
you know, I don't wanna kind of
put it out there, but it is in need
and you have to take great care.
To make sure like what you predict is
robust enough, whereas like compared to
the other fake detectors that maybe, uh,
X or Facebook or Instagram want to deploy,
which kind of is based only on the volume.
So like one relative percentage increase
in accuracy is gonna capture much more.
Uh, obviously deep fakes on, on
their kind of volume of millions.
Uh, then one percentage lower.
And we have like, our priorities are
slightly shifted, like, not like fully
to kind of saturate on accuracy at
the expense of maybe explainability
and uncertainty communication, because
like having the models you can trust
is more important than having like
the maximum, maximum, um, objective.
Chris: super interesting topic
that I could, uh, talk to you
about all day, to be honest.
And I guess, um, question I've got
as the detection models become more
advanced and obviously the, the
defects becoming more advanced,
are you, are you finding that the.
models can keep up with how advanced
some of the DeepFakes are becoming.
Um, and that's a naked eye.
It's, it's next to impossible now.
But, um, and from, from a
model's perspective, it, can
we keep up, do you think?
Danijela: Well, I think, you
know, as long as there's training
data, you can kind of keep up.
The question is like, whether the
models will at any point become
that advanced to kind of be good
at mimicking almost everything.
And, um, uh, you know,
it's an open question.
I dunno, you know, um.
Uh, whether that that
is gonna happen at all.
Uh, at this point it seems that, that
there's still like a wide margin, wide
differentiation between us, like, uh,
generated and what is, what is real.
Uh, but I'm not sure what's gonna
happen in the future, to be honest.
Right.
No, I think we'll have to come, come up
with some sort of, uh, watermarking or
like, BBC is also working on the C two
PA content, uh, provenance, um, project
together with Microsoft, Google, and
Adobe whereby like every piece of, uh.
Digital contents in terms of video,
audio, or imagery, uh, needs to have
like a specific metadata with the full
kind of lineage of how, where it's
been produced by what camera, and then
how it's been augmented and so on.
And then there is a world, I guess.
Uh, like there is the one
version of our future in which
like that becomes the standard.
Um, and you know, if you see
something online and if it doesn't
have that kind of C two PA, uh,
metadata, then you implicitly
should trust it less in a way I.
Chris: Nice.
Okay.
Well, um, yeah, great to know.
I think that segues, um, nicely
into governance and responsible ai.
It seems like a topic that you
are super passionate about.
I know at the BBC, uh, you chair
the, I hope we get this right,
the, the Gen AI Design Authority.
Um, you lead, uh, partnerships
with, Oxford around misinformation,
as you just mentioned.
Um, I think des uh, defining responsible
AI can be, uh, a challenge in itself
sometimes, especially across, you know,
the US and the UK work at different
companies, I think is different.
Um, what, what, what do you, um,
interpret as, you know, what do you
use as your framework and principles
to guide your responsible ai?
When you know you're developing
a model, how do you ensure that
it's, you know, AI for good?
Danijela: Oh wow.
Well, as I said at the beginning,
our hands are very much tied now,
you know, and I think in the future,
the hands of an, of the engineers
are gonna be even more tied.
The, even like at the, if you kind of,
if you do what we are doing, basically
augmentations of the model way, it's, but
like in the post training stage, like you
take an open source model and then you
kind of adapt it and change it and so on.
I dunno, like, there's been a, a recent
paper by Anthropic, I believe last
year where they, where they kind of
investigated how much the world model and
beliefs of a large language models can
be augmented in a post-training stage.
And it seems that it is
basically, um, limited in a way.
What they've done is like
they came up with a series of.
Documents that reflect
a different worldview.
So they take this base model and
they take the series of documents
that reflect a different worldview
and they continue pre-training and
then they do what they usually do.
And on the surface level, um, it
seemed that the model has changed
their beliefs and has different
biases and has different, uh.
Uh, worldview, but like if you kind
of are persistent and keep prompting
it, it seems that the demo just
learned to, to fake these things.
So I think, you know, when it comes to
response, responsible AI and um, uh.
You know, when it comes to model
build and the responsible AI should
be kind of transferred to the model
owners, uh, either in the open source
world or, um, in the commercial world,
and like more transparency should be
requested both in the, in terms of the
training data, but also in the, uh,
methodology used and the tests done
before that has been, uh, deployed
because, um, you know, there's really.
Very little you can do after you have
your base model, after someone has
spent, spent like tens of tens or
hundreds of millions of dollars on, on
training, which is not something like
any organization apart from big tech
can, can afford, uh, themselves to do.
Uh, so, so yes, you know,
obviously we do what we can.
Um, we make sure that, you know, we
evaluate very well on the use cases.
Uh.
Uh, that, that are interesting to us,
that are, that are relevant to the BBC.
And, you know, we try to do the
due diligence, but as I said, what
one can do, uh, as a third party
almost is, is there limited there?
Um, yeah.
Chris: Cool.
And, um, yeah, me media organization,
you kind of touched on it meta, you know,
if, if they improve, uh, their deep plate
detection by 1%, it's a huge, thing.
But I guess if just one gets through,
uh, the BBC can have huge ramifications.
Um, just with the, the spotlight, it
hasn't it, you know, how, how, how do
you balance that innovation of pushing?
Maybe it's.
State of the art model to production,
to, you know, balancing the needs of
an organization like the BBC and, um,
keeping trust with, with its user base.
Danijela: Yeah.
Um, I think the pressure to act quickly
has increased significantly in the past
two to three years for media and news
organizations, and many of our peers have
kind of jumped into this kind of, um.
Onto this bandwagon and have started,
um, experimenting, uh, with various
use cases and really kind of, um,
integrating them in their workflow
or the way how they work and so on.
So the BBC has a much.
You know, more risk averse position
and it has always had, and sometimes
we kind of pick up a lot of criticism
why it is that, that, that we're
so slow, we're not fast enough.
But I think that is with a good reason.
Um, and, um, I think.
Uh, the, as as you just pointed out,
the risk, uh, is, is magnificent there.
So we do try to give ourselves time
to test not only the models from the
technical viewpoint, but also what
effect they will have on the, on our
audiences, how our audiences feel,
how the creative industry feels about
us using specific models and so on.
So all of that has been done in the
background in the past couple of years,
and we now feel, and I think BBC now
feels they are now ready to, to take
on innovation and start kind of, um,
putting some of these use cases into
production and talking more about them.
Chris: Nice.
Okay.
Well, um, yeah, su super interesting.
Thanks.
Thanks for sharing that.
Umm, moving on to the, the kind of
final two parts of the, the podcast,
um, more leadership and culture focus.
You know, you've LA
managed teams of 20 plus.
Um, you, what, what's your philosophy
around leadership and, um, yeah, just to
take it back to the earlier part of the
podcast, do you think mathematics has
helped shaped your, um, leadership style?
Danijela: Well, I think first of all, like
people coming, leaders coming from like
STEM disciplines are almost diametrically
opposite in their leadership styles to, to
people coming from humanities and uh, uh,
you know, obviously that was one big part
of it, but you know, I think even more.
Mathematics is even more like fundamental
and abstract of all STEM disciplines.
So if we wanna go kind of back
to that, uh, uh, that view, so.
I think, you know, maths is pretty much
like the individual sport in a way.
You never see math, math papers
with like more than two authors.
Like three is plenty and
four is just like out there.
Like you instantly know
something is wrong.
There.
Some, someone is on that paper who
shouldn't have been because it's
like, it's impossible to tackle
such complex tasks with like big
groups of people because it like
requires intense, intense, uh.
Um, collab like thinking and
very close collaboration.
Whereas like in machine learning,
like you have groups and teams of,
like, we've seen now Gemini 2.5
has like 3,247 authors.
That's like wow.
Mind blowing, right?
Um, so.
Um, you know, obviously from that point
of view, uh, like there is a sense of
extreme ownership in maths community.
Uh, and that's what I've
got by Byman training.
And it's always like a, uh, a point
of, you know, giving credit where
the credit is you, but not giving
credit where no credit is you.
Um, but.
Yes.
Um, apart from that, you know, being
really having a really strong sense of
ownership and trying to kind of promote
that with my team and trying to, um,
uh, trying to basically instill in
them, uh, more, uh, more intellectual
curiosity, more independence, more
bias for action, I think it is.
Uh, very important, empowering people
on the ground to, to feel that they can,
when given a task, uh, explore freely.
Right?
Um, and yes, I don't think really there
is much, much more to it in a way.
Um.
Chris: Oh, Um, in, in terms of, um.
You know, BBBC is a, a
very complex organization.
There's lots of moving parts, um, you
know, lots of different stakeholders.
And then, you know, you throw
AI innovation into the mix.
Um, how, how do you, you balance
what you probably wanna do, which
is cutting it research with, um, you
know, taking maybe stakeholders on the
AI journey because there'd be lots of
leaders out there where, you know, the.
AI is potentially coming into the
organization now, uh, front and center.
And you, you've probably al already
been on that journey with the B, B, C,
um, you know, what, are there any, um,
tips you could pass on in terms of,
um, you know, ma stakeholder management
in terms of taking on that AI journey?
Danijela: In terms of
stakeholder management?
Yeah.
Um, so like at the BBC, I've been
like in the r and d department,
Chris: Hmm.
Danijela: which is like slightly
different by definition, our mandate
is to look slightly kind of, uh,
more in the future than you, your.
Typical product organization would do.
Uh, but I think, you know, um,
like one rule of thumb is like,
and we're fortunate now that,
that, you know, AI is so hyped up.
So everyone want, everyone's primed,
everyone wants to do something with ai.
And I think when.
Uh, and we are all given opportunity
to kind of pitch and give our views.
And I think when this can break down, um,
in kind product organizations is if you
come up, uh, if, if like your pitch or
your project is more, uh, geared towards,
uh, your interests and your kind of
scientific curiosity rather than actually
solving the real, uh, business problem.
Um.
And I think, you know, in general,
you can't go wrong with kind of
trying to understand business more,
uh, trying to understand what they
are, what they are coping with.
But I think it's almost kind of the
first time now, um, in a very long
time that there is an opportunity to
co-develop, like the ai, uh, staff should.
Definitely co-develop, uh, new feature
ideas, new product ideas with the
business and with the engineering.
Because a lot of, of what business
typically would come up with are
like the incremental features.
And although that is like super important,
it might not be like relevant in a, in,
um, in a year or so because like the field
is moving so quickly and you, you almost
must kind of take this kind of meta view.
Uh.
To, to either develop like a suite to,
to de develop and build for the future
rather than kind of building very
scattered, uh, uh, features, AI powered
features across the board to kind of
appease a set of different stakeholders.
So I think, you know, all of these
organizations should kind of focus more
on having a very solid AI strategy,
but that doesn't focus on now.
That, but it that it has like
a focus that is a little bit
further away in the future.
Chris: Yeah.
And I can imagine that can be, hard at
the moment because everyone wants a, an
AI agent to, to do things, um, for them.
And it'll be interesting to see,
um, where we are in six months
because no one could really predict
this stuff coming beyond, uh, yeah.
Danijela: is crazy.
I actually have to kind of jump in now
because the, the book, again, I'm gonna
say it and recommended to your readers
The Art of Doing Science and Engineering.
And it's been published in
1995, maybe by Richard Humming.
And in it he says, like he almost
has, has a thought exercise and says,
let's predict the world in 2020.
And I have to say he has been spot on and
he has been able to predict the future.
Or for example, I've, uh, recently watched
like Steve Jobs talking about Google Maps
in 1980 something like in his interview.
So like, there are people, and you know,
let's not forget all the investment
professionals whose daily job is
to predict the future, otherwise.
You know, they are not gonna
have their margins, uh, uh, met.
But there are people, there is a way how
to approach this kind of, uh, prediction
of the future because our, our world
is almost kind of rule-based and these
rules are not, uh, not as complicated
as like in machine learning, uh, models.
So there is a way to extrapolate
as long as you under properly
understand the fundamentals.
And I think, you know.
I think organizations in particular
should kind of employ their top most
AI talent to help them build that, uh,
worldview of the future and not rely
on, you know, people with various,
uh, I would say commercial interests
who would, you know, want to persuade
you of a specific worldview because,
because, you know, they are bottle nine
because of their valuation and whatnot.
But I think there is a, there is a
way how to reduce that uncertainty.
Chris: Interesting.
Yeah.
I think it comes back to, um, you
know, the role of the Chief AI officer.
I think that's becoming
more, uh, prominent.
Um, cool.
I think that segues onto the final part
in terms of, um, trends and, um, AI that
you are using it in your everyday life.
Um, you know, we, we've kind of touched
on a lot of this, uh, so we won't go
into too much detail, but is there
a particular, um, trend you're most
excited to see, um, moving forward?
Danijela: Uh, so yes, I am, I am like
super impressed by the work that, uh,
uh, people are doing in the space of
reinforcement learning, open-endedness.
Like there's been a lot of critiques
nowadays, uh, like recently with, um.
We on the, on the transformer
based architectures.
And, uh, there is a group of scientists
who are looking into, into kind of
building better representations for these
large language models, uh, that are.
Not, um, like stochastic gradient, descent
based, and there is like a really active,
active field of research, uh, there.
So I'm kind of quite, quite
excited about that and seeing where
that kind of goes in the future.
So it's like a, a
different way of machine.
Learning different way of learning
that is kind of more, um, that it
requires much less data and, uh, kind
of results in the representations
that are kind of human-like, uh,
as opposed to something within
traditional machine learning models.
Chris: Nice.
Yeah, it's certainly something I'm seeing.
Um, I think you go back like eight years
ago, it was just kind of deep mind working
on RL and multi agent, uh, theory and, uh,
it was very, um, focusing research labs.
But there's a lot of great startups in the
UK that are, um, moving away from even LMS
now focusing on, uh, a more RL approach.
So it's gonna be, um,
a really exciting time.
I think we're moving into kind of.
Uh, the next phase, which is,
you know, people focusing on,
um, reinforcement learning.
Um, do you, do you use
any app interest on here?
Do you use any AI tools in your, um,
everyday life to stay productive?
I know people are using college coding.
Um,
Danijela: Mm-hmm.
Chris: people I know use for text
and perplexity for searching.
Is there anything you use as, as a leader?
Danijela: Yeah, absolutely.
Absolutely.
You know, coding has definitely become,
you know, two orders of magnitude
easier nowadays, which is great.
You know, you can kind of quickly
prototype and quickly test out, uh,
uh, certain things, you know, uh, but.
I, I, you know, it was obviously
experimentation since the release
of the first models, how they
can be most beneficial to me.
And what I found is like that these models
are like people, people say, oh, it's
great, we can use them for brainstorming.
And when I hear like creatives
thinking that way, like my, I just
feel super sad because like these
models cannot be and should not
be used for brainstorming because
like they always have this kind of.
Almost kind of mean approach,
like this kind of, um, uh, a very
pedestrian way of doing things.
And it's not even a problem, you
know, that they have such a, such
a very generic view of things.
But the problem is if you as an individual
with your own worldview are exposed to
that, before you have the opportunity
to think on your own, your mind kind
of gets polluted and you cannot think
anymore about the given problem.
In a way that you would normally think.
So I think there is, there is this kind of
degradation of the personal creativity, so
I would never use them for brainstorming.
And the first thing you can do, and
I've been a, a witness of that on a
few kind of, uh, in a few situations,
if you are given like some kind of
open-ended problem or some kind of
task, you're supposed to kind of.
Let's say in interview settings or
whatnot, and you see people like
they all come up with the same,
uh, same presentations, uh, same
responses to the same question.
You know, this is just not, not okay.
You know, and I think people should
course correct a little bit on that one.
Um, so, so yeah, definitely.
But then also, you know, when it comes
to really frontier, um, frontier.
Uh, fields like topics.
For example, I was just looking
at something in uncertainty
quantification, and it's such
a messy field, I have to say.
Like it's, uh, it's very difficult
to kind of pick up heads and
tails and so on and so on and on
multiple occasions I was like, uh.
Trying to use deep research
show three, you know, like
just to help me a little bit.
Survey the field and every time it would
come up with a different worldview.
And I found it that, uh, myself, you know,
finding only two authoritative voices
could lead me to kind of open up like a
puzzle pieces, uh, like everything, like
my, um, yes, to open up the field much
better, much quicker than actually rely.
On a, on the completely, um.
You know, view of the language model
that is based on only completely
on the statistics and it's, uh, it
cannot recognize really quality.
But what is good in, like, if you wanna
go into the field that is very well
established, that is very well present
on the internet and that you don't know
much about, uh, then you just go and
really have a good conversation and can
kind of speed up your learning journey.
You know, tenfold 100 fold.
So that's, that's what's it very good in.
So, but I would be careful in
brainstorming and actually kind of,
um, inquiring about the, the fields
that are very much progressing
are like very much in flux.
Chris: I was, I was in London and met,
um, A CTO network and we had a, a quite
an interesting talk that, um, you know.
Will, will we start asking people, uh,
will company start asking people for
who are qualified pre pre chat GPT?
Because he kind of alluded to, um, you
know, everybody's coming up with the
same ideas because, um, the students
are just prompted, um, you know, a, a
chat bot building ideas out from that.
And he, he just said.
He is definitely seen it in people
that they're, um, they're interviewing.
So, um, be, be super interesting to
see how the market plays out there.
And I can definitely encourage
anyone looking for a job to, to
kind of, um, know, focus on the
critical thinking because, um.
I definitely agree.
So, um, that, that wraps it up.
Um, Daniel, thanks so
much for coming on today.
Really appreciate your time, sharing
your knowledge and, um, all the very
best you in your, your new job as
well in the next couple of months.
Danijela: Thank you.
Thank you, Chris.
It's been such a pleasure.
Chris: Thank you.