How AI Is Built | #32 Max Buckley on Improving Documentation Quality for RAG Systems | Search

Documentation quality is the silent killer of RAG systems. A single ambiguous sentence might corrupt an entire set of responses. But the hardest part isn't fixing errors - it's finding them.

Today we are talking to Max Buckley on how to find and fix these errors.

Max works at Google and has built a lot of interesting experiments with LLMs on using them to improve knowledge bases for generation.

We talk about identifying ambiguities, fixing errors, creating improvement loops in the documents and a lot more.

Some Insights:

• A single ambiguous sentence can systematically corrupt an entire knowledge base's responses. Fixing these "documentation poisons" often requires minimal changes but identifying them is challenging.
• Large organizations develop their own linguistic ecosystems that evolve over time. This creates unique challenges for both embedding models and retrieval systems that need to bridge external and internal vocabularies.
• Multiple feedback loops are crucial - expert testing, user feedback, and system monitoring each catch different types of issues.

Max Buckley: (All opinions are his own and not of Google)

• LinkedIn

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Understanding LLM Hallucinations 00:02 Challenges with Temporal Inconsistencies 00:43 Issues with Document Structure and Terminology 01:05 Introduction to Retrieval Augmented Generation (RAG) 01:49 Interview with Max Buckley 02:27 Anthropic's Approach to Document Chunking 02:55 Contextualizing Chunks for Better Retrieval 06:29 Challenges in Chunking and Search 07:35 LLMs in Internal Knowledge Management 08:45 Identifying and Fixing Documentation Errors 10:58 Using LLMs for Error Detection 15:35 Improving Documentation with User Feedback 24:42 Running Processes on Retrieved Context 25:19 Challenges of Terminology Consistency 26:07 Handling Definitions and Glossaries 30:10 Addressing Context Misinterpretation 31:13 Improving Documentation Quality 36:00 Future of AI and Search Technologies 42:29 Ensuring Documentation Readiness for AI

What is How AI Is Built ?

Real engineers. Real deployments. Zero hype. We interview the top engineers who actually put AI in production. Learn what the best engineers have figured out through years of experience. Hosted by Nicolay Gerold, CEO of Aisbach and CTO at Proxdeal and Multiply Content.

Nicolay Gerold: LLMs hallucinate.

And often we want to put the blame
on them, but on occasion, Or even

often, it's also our own fault, many
knowledge basis we use for retrieval

have temporal inconsistencies.

So they might have multiple
versions of the same document

from different time periods.

They might also have historical
information without timeline context.

So for when this information was relevant,
they might have a reference problems.

So for example, implicit references
to something like it, "this system"

without explaining what it means, but
also undefined aliases and terminology.

Especially in large organization
terminology evolves over time and

an organization can build an entire
terminology for internal communications.

And then you can also have like
structural issues in the document itself.

So I'm missing semantic structure.

And that is on top of plain
mistakes and human error.

Especially with the RAG is these issues
are more pronounced because you aren't

looking at a full document anymore,
where you can be sure there's a certain.

Contextual relevance, but you're
only looking at bits and pieces

and these bits and pieces or
parts of the documents, chunks how

they are called often created
through naive chunking.

So you're just chunking based on a length.

So for example, like 400 tokens.

And this means we have to be
extra careful when we are doing

retrieval, augmented generation.

And are feeding.

Facts.

Into our LLM.

And today on How AI Is Built
we are talking to Max Buckley.

Max works at Google and has done a lot
of interesting experiments with LLMs.

And especially on using them to improve
knowledge basis or documentations.

For retrieval augmented generation.

And we'll be talking about identifying
ambiguities, fixing errors, and

also creating improvement loops.

For your document base.

So welcome back to our series on search.

Let's do it.

Max Buckley: What Anthropic are
proposing is that you have a set of

documents that you want to chunk.

And then for each of these documents,
you break them into some arbitrary

set of chunks, let's say K chunks.

And then what you do is you get Claude
in their case, but an LLM to, look at the

entire document and then look at the chunk
with the kind of command to contextualize

this chunk, given the document.

So one of the examples is that you might
have a chunk that says something like the

revenue was 43 million and that chunk on
its own doesn't really tell you anything.

If someone searches for something
about revenue, they will find it, but

the revenue for what was 43 million.

But of course, given the document, the
contextualized version is something

given the sec filing for this company.

The revenue is 43 million, which
obviously has a much more meaning in it.

And they use this contextualized version,
both for embedding and semantic search,

but also for keywords and lexical search.

If you had something like SEC filing or
the company, then you could potentially

find this chunk and not just the revenue.

Obviously there's different
ways of thinking about chunking

and contextualized chunking.

One thing we were thinking about
was like, if documents contain

intrinsic structure, you can obviously
maybe make use of that, right?

If you have a hierarchy of headings,
obviously you get some information

from the file name, file path, but then
you have some hierarchy of headings,

so you have a H1, H2, H3, potentially
that also contains some information.

Like you can, Flatten or ignore maybe
the sort of intermediate paragraphs, but

knowing the hierarchy of headings also
gives you some more information that a

sentence or paragraph may lack, right?

And so including that can help, but the
anthropic approach particularly resonated

with me when I saw it, because as I said,
it uses the large context model, so to be

able to process a whole document plus the
chunk, you need tens of thousands, tokens,

hundreds of thousands of Spanish document
But it also uses context caching, right?

So effectively, you pay to cache the
document, and then each incremental

thing is just a small paragraph.

So they have some number cited
like 1 dollar, 2 cents or something

per million tokens processed
which is obviously very nice.

And most of the major providers now
offer this context caching feature.

Nicolay Gerold: And are they doing it in
fill in the middle basically how we see in

masked training, that you have the stuff
previous, empty stuff stuff after it,

and then you fill in the middle chunks as
well, or is it only the previous document?

Max Buckley: And no, it's, so
it's the entire document up top.

So the section that you're
contextualizing is effectively duplicated.

So like exactly, it's the whole document.

Nicolay Gerold: Yeah.

Max Buckley: but you're right.

You could obviously do it the other way.

You could think only include the
prefix that you've seen before.

Yeah, a lot of these things,
they're more art than science.

You can try things and see what works.

Another interesting area here, as well as.

Potentially like contextualizing
the chunk, is to actually have like

multiple embeddings for the same chunk.

So if you have a chunk to have the
raw chunk embedded potentially like

a summary of the chunk embedded
and hypothetical questions that the

chunk might answer, if your system is
questions and answer based, and what you

probably will see is a very long chunk.

The distance from your
user query could be.

Large, whereas the summary might be a
little bit smaller and then the question

might be significantly smaller because
they're both in that same space, right?

That kind of question space.

Nicolay Gerold: Yeah, and I think
it's different problem they're fixing.

The one where you embed the questions
is rather, That the question and the

document live in different spaces and
are formulated in a different way, use

different lingo, use different terms.

And the other one is more about
losing relationships and losing

information that's important
in the context of the document.

Max Buckley: Yeah, exactly.

Exactly that, right?

That the query content mismatch problem
is the reason for the hypothetical

questions or multiple embeddings.

But exactly the contextualized thing
also makes it more semantically

meaningful and more complete And I
think this is something, a problem

that comes with chunking, right?

That as you said earlier in
traditional search engines, you

think about documents and a document
contains a lot of information, but

a chunk can be quite small, right?

It could be a few hundred characters
or like a hundred characters, or

it could be like a sentence or some
words, and obviously on its own, it

doesn't necessarily mean all that
much, in the context of a particular

document, maybe it means something.

And so it's just a way of
getting around that problem.

Nicolay Gerold: Yeah.

And especially relative sizes because
chunking tends to be used like when I

have really long documents and traditional
search engines, like the majority

of applications is around e commerce
where your text sizes are very small.

What would you say are The main
issues we face, especially around

the internal documentation, but
also around the approaches to

chunking and search at the moment.

Max Buckley: So I can only speak
from my own experience, right?

But I think that there's a huge
opportunity that sort of LLMs provide

for internal knowledge management.

And specifically, I think rag
is the kind of key right now to

unlocking that if you're in a
company where you have an extensive.

Set of documentation and not even
strictly speaking documentation.

It could be questions and answers.

It could be historical transcripts or
whatever that can be a huge junked

and synthesized and used to answer
future questions and really expedite

people's knowledge discovery process.

But what we have seen in Google, for
example, is like all sorts of like quality

problems, data quality, document quality,
effective problems where, for example.

There may be two documented ways
that something works, or one.

Maybe they're both correct at
different points in history, right?

Someone documented it on this file
at one point, then a few years later,

they documented it on this file.

And now we have two files that say
the opposite as to what you should do.

And only one of them can be correct.

Like where it says, for example,
do the database update in parallel.

And here it says, do the
database update first.

Okay.

One is correct.

One is not correct.

And again, this is something where I think
LLMs can really help to solve the problem.

Obviously in the past, you could
ask some engineer to go and update

documentation without specifying exactly
what to update or where to start,

and there's a lot of documentation,
there's hundreds of thousands of files.

So it's not a very.

Satisfying task, whereas an LLM can,
especially with large contexts, can read

in a huge set of files and you can ask it
for errors, ambiguities, contradictions,

and it can give you back a list.

And many of them, you can go and
very quickly knockout, or you

can just go and change one line of
documentation and that issue is resolved.

Can also identify things
like areas that are under.

Documented or areas that are
mentioned, but not explained.

And of course, these could also be.

Action items for people to document.

And then of course this improvement then
will be reflected in the next set of like

answers where you read in the newest data.

But obviously this is not
perfect process, right?

Given today's LLMs, you will
get some hallucinations, right?

If you ask for, give me 10 errors,
contradictions, ambiguities, some

of them are really useful, really
punchy stuff, other ones are.

Maybe borderline stuff like capitalization
typos, small potatoes and other

stuff, maybe just hallucinated or
wrong, it says, this is this, and

especially if you ask it for file
references, it can give it to them too.

You can go and check.

So you don't necessarily waste
all that much time looking at

them, but sometimes they're just.

Not particularly saying something.

So I think having high quality
data is key and different.

Data sources maybe have a different
sort of quality bar or like different

expectation in terms of like maintenance.

So for example, formal documentation
might be something that people

strive to keep up to date.

Whereas if you have an internal questions
and answer repo, maybe the expectation

is that you answered a question
and that was good when you answered

it and it's not necessarily good.

Several years later, right?

So how you deal in your system with
timeliness of data is a question, right?

You could be very clever and try and
figure out this system didn't change much.

So maybe the older answers are still good.

Or maybe you just have some hard
cutoff and say, look, let's just

throw away things from a few
years ago as being out of date.

Nicolay Gerold: But in the end,
that's really about identifying the

places where there is information
and not really fixing it.

And I think that's an important
distinction because for finding

disambiguities, using something else
than LLMs will be really challenging.

Because You would need to come up with
an entire NLP pipeline to figure out

okay, this is conflicting information.

You could probably supplement it with
some NLP, where you use like entity

recognition to identify the major
entities, and then basically flag them

and maybe reduce the document size,
and insert all documents where you

have a certain entity, and see whether
there are conflicts around that entity.

What are your thoughts on that?

Like, how do you figure
this system working?

You're thrown in to a new company
which has documentation issues.

How do you approach it?

Max Buckley: What you've just described
is obviously possible, but sounds

to me like a lot of work, right?

Like you'd have to figure out a lot of
systems to orchestrate the various bits.

And with LLMs, you get
this magically, right?

That if you can just read like the
markdown files out of a GitHub repo,

you can write a Colab that, reads in
100 files, sticks them in the prompt to,

large context model like Gemini or Claude.

And you ask it the question, how
do I, and if you have context

caching even better, right?

Because then you could ask a
sequence of questions over the

same hundreds of markdown files.

And for each one, it gives you back
some kind of answer, which lets you

go and either action it yourself, or
maybe you file, requests for people

to other people to action them in
case that you're not the expert.

LLM thing is really a kind of, you Speed
boost here, I think compared to the

prior, because also a lot of these things
are sometimes linguistically subtle.

And what I mean by that is I've
seen instances where, after

loading a documentation base.

Into the prompt and asking some questions,
I could see that the answers I was getting

were not what I expected as someone who's
worked on the system for several years.

And what was interesting was I asked
the model for direct quotes as to

why certain things are the case,
and it gave me exact quotes, and

I could find the offending string.

And effectively, there was this
funny kind of phenomenon where.

A system was built on top of under
other underlying systems, and one

of the underlying systems had a
particular limitation and the higher

level system, provides an abstraction
that you get around that limitation.

And generally, when people are
talking about the top of system

X, if they're talking about
system wide, in the context of X.

But in this one paragraph, they were
talking about just why on its own.

And they made this statement that
they said that this system doesn't

support this programming language.

And then the model just understood
that to be a fact or something.

And every question about,
does this system work?

Does X work with programming language?

And they would say, no, because of this
one offensive or like offending sentence.

And literally changing the system,
the sentence to be just more specific

to say, while the underlying system.

Does not support this program image
system X provides an abstraction on top

of it that allows them to be compatible.

And this like simple fix instantly,
meant that all those kind of questions

went to the way you should be.

You expect, right?

That the ambiguity was gone, but the
funny thing is as a human, if you

didn't know, and you just read that
paragraph, you would rightly make the same

assumption that the LLM had just made.

If you were a beginner in the company,
you would absolutely read and go, okay, so

I'll remember not to use these together.

And so making it better for the LM
also made it better for the humans.

Nicolay Gerold: And It's how would you
actually go about figuring out that

you have disambiguities in the first
place because in most companies, the

documentation is way too large and you
have way too many documents that you

could just throw all of it into a long
context model, or you would have to

basically go into an iterative mode.

And iteratively pass 100, 000
tokens into the LLM and just tell

it, okay, are there disambiguities?

How would you actually approach
it in a more targeted way?

Maybe linking it back to the
search queries or where you

didn't have a hit or didn't.

Give the user satisfactory answer.

Max Buckley: so yeah, I suppose this for
the first part there, about how do you go

from, if you have like too big to fit in
a large context, and even if you have a

say, so certainly that's the problem, if
you have too big, but I don't think you

can necessarily tackle a whole company.

In one go, but potentially you can
do like smaller, like departmental or

functional or system based improvements.

But if you want to go bigger than
that, if you want to dynamically.

Yeah, gather these documentation.

Then that's when you're talking
about the kind of retrieval,

augmented generation, right?

When you have some sort of semantic
search and or it does not be semantic,

but let's just say semantic search
over the chunks and you return the

chunks and you generate the answers.

If you have one of those
systems, of course you can

gather like user feedback, right?

You can allow for thumbs up, thumbs down,
and potentially even like a qualitative,

like they can write a free text.

So what was wrong or right.

And even just from using the
system, maybe you can see that Your

chunking strategy is limited, right?

As a developer, you can find that
there are chunks that are maybe

useless or meaningless, but for
some reason are still being ranked

highly and still being retrieved.

As in the chunk itself might have like
nice headers, but then just say to do,

and the name of the person who's supposed
to actually write the documentation.

And of course, this is not
something you ever really need to

retrieve for any particular reason.

But your model retrieves it and then.

Suddenly it has fewer chunks, fewer
realized chunks than it would have

otherwise or the one that we commonly
see is with re ranking models that are

pointwise that score each particular
data point, given the query individually,

they often find a good chunk, but then
they find like pseudo close to duplicates

of it, given like we have a lot of
redundancy in documentation, especially

if you look across different teams.

So the re ranking model will often
give us like four of the same chunk

from different teams that have all
recommended the same best practice.

And you're like, okay we'd like
some diversity of choices there.

There are list wise re ranking models
that can get around that problem.

Sorry, I feel like I went on a tangent
and didn't fully answer your question.

That's.

Nicolay Gerold: during generation.

I think like we are at the moment
building an agent for coding.

And in that we feed in documentation,
but every time you make a generation,

we cross check, Hey, is this actually
consistent with the documentation?

Because of the priors of the LLMs, it
often makes a wrong choice in how to use

basically a certain programming language.

So this ambiguities are useful
in like multiple different ways.

And I think like, when you.

Can ask specific questions.

It's always better.

So you always have to think about
basically, I'm not looking for ambiguities

in general, but rather I'm looking for
specific ambiguities, which should be

informed from pre existing user queries,
which may be failed, or maybe it didn't.

return good results or where the
generation completely failed or where

the NLI model told you, Hey, this
answer is not factual consistent

with the input you gave the model.

Max Buckley: So we are, I haven't
really looked into these recitation

checks for like rag systems where
you can like annotate your data yet.

Only started looking at them recently,
but so that, if we're giving a large

number of chunks that we can say what part
of the response came from, or was like

heavily influenced by a particular source.

Which is obviously nice because now we
can Both visually, we can show it and say,

okay, this section came from number one,
or this came from number three, but of

course it also lets us see if there was
some chunks that we didn't use at all,

and that also lets us programmatically
get data as to which ones were actually

used by the model itself, right?

Without relying on the user,
thumbs up, thumbs downing it.

Because if we're retrieving a chunk
and never using it, then not such a

useful chunk for this set of queries.

Nicolay Gerold: Yeah.

And I think like the most
challenging part is errors.

So I would love to know, like, how
did you find this concrete error where

the fact that was in the documentation
was actually just plain wrong because

ambiguities, you have the same
information stated in different ways.

So this should be like easier to
identify, but clean errors in the facts

are really difficult in the end to find.

Max Buckley: Yeah.

In that particular case, I had
the benefit of being a domain

expert in that system, right?

So when I was playing with this large
context model, I was specifically

asking questions that I thought
like a user, a beginner would ask.

And I was like looking at the answers
knowing what I wanted or expected.

And when I got something that kind of
disagreed with that, I was able to fix it.

But what I also found interesting
was just asking for errors worked.

Okay.

I'm not going to say perfectly,
it doesn't find all the errors

necessarily, but it also worked, right?

I asked for errors and it just
sometimes gave me these funny

concrete examples or other ones
were ones who were like, I didn't

personally spot them, nor did the LLM.

But one team who I was working
with came back to me after a few

weeks and they said that they were
very happy with the LLM responses.

One problem was that the
model kept recommending people

come to their office hours.

And they don't have office hours.

And I was thinking pretty sure that
documentation says they have office hours.

And it did four times.

So in four different parts of the
documentation said if this doesn't solve

your problem, come to our office hours.

So they were saying it's a hallucination.

I was like, quickly removed it.

It hasn't happened since.

Nicolay Gerold: Go ahead.

Max Buckley: yeah, no.

I guess the line of what is an error is.

Difficult to quantify, right?

I mean, and even if it's independently
verifiable, if you, if the system,

like if the documentation says two
things that contradict each other

that there are two options and
somewhere else says there are three

options, then only one can be correct.

For something.

But.

Other cases, it's hard to, the model
couldn't have known that office hours

there are no office hours because it
wasn't even written somewhere that

this team do not have office hours
So if we had that situation, it

would obviously have been a explicit.

the obvious or something more
obvious and fixable, but it required

a kind of a user to point it out.

Nicolay Gerold: Yeah, I think it
only shows you like search systems.

You can never look at them in isolation.

You in the end need a complete team to
actually fix it, like subject matter

experts, but also you use a feedback to
actually point out the wrong information.

What I'm really curious about is to get
your thoughts on like misgenerations.

So if the model always adds information
or includes something, And that's a

bias from his, it's training data.

And I think like one concrete example is
always like when you ask models to write

something for like anything, social media,
it always falls into a certain style.

And I think this is also an interesting
direction that your documents you have

also feed into the context could bias
the model into a certain direction.

I would love to hear your thoughts of
like, how would you actually change

the documents you have to actually.

change the behavior of
the generation as well.

Max Buckley: Know what you mean in the
sense that if we're putting Markdown into

the, as a documentation, the model will
like just start spouting Markdown in its

response, which is actually quite useful,
but we don't have to say you responded

Markdown or whatever, it will just do it.

Bye.

Bye.

Generally, like my experiences so far
with using Gemini for generation has been

pretty good in terms of it not answering
when, or if something is explicitly

outside of the context so if we're asking
it some question, like, how many legs

does a horse have and, retrieval from
the internal documentation and questions.

It doesn't have any mention
of the anatomy of a horse.

It will usually say something
like, Oh, this question is not

answerable given this context.

So I haven't seen it necessarily do some
sort of like in domain hallucination too

much but I haven't got an exact number.

On that terms of the actual
sort of style that it picks up.

Difficult to say like I've definitely
seen it, like hallucinate when,

especially in like longer responses,
but again, not hugely badly.

I would say I think, rag in general
is good for reducing hallucinations,

versus just like prompting the
model and asking the question.

So that definitely helps.

As I said earlier, like when asking
for like contradictions or ambiguities,

probably if you ask for 10, two are.

Misunderstood,
misinterpreted, whatever else.

Yeah, there, there are some
there, but not a huge amount.

I would say it's, like what I was
doing with some of this was generating

a document with the kind of models
response with here's 10 ambiguities

in your documentation and sending
it to some other team and saying,

Hey take a look at these 10 things.

And people would come
back to me with fixed.

Not a real problem or maybe this
one doesn't, isn't really ours

cause it is owned by the other team.

So we're not going to do a
little bit and then fix fix.

And then maybe hallucination and okay.

I'll take it.

But in terms of like, the models
prior hard to, I haven't done any more

detailed analysis to say that it's.

There are not, one thing
is the case, right?

Is that like, in a company that's
been there for a long time, a larger

company, you're going to have a lot of
internal like evolution of language.

So a company that has hundreds of
thousands of employees has been

there for 25 plus years is going to.

Have its own internal vocabulary,
its own, way of naming things.

And what's interesting is if you use
a model like a model, but let's say

for example, Gemini, like a priori,
it doesn't understand the internal

language of a company like Google,
or it doesn't understand, how we use

certain words, given the like rag or
large context, it picks them up on

the fly, which is quite impressive.

The in context learning there is nice.

If actually we're asking a question that.

A bunch of terms that are like, I wouldn't
say meaningless, but they don't have the

same meaning in the inside, outside world.

And in some cases we're using an
internal or sorry, a externally

available embedding model, right?

So that embedding model doesn't
correctly embed the relationships

between some of the words, because
it's unknown kind of vocabulary,

limited their words, but that is

Nicolay Gerold: Yeah.

Do you think it, it would be easier to
run this only on the retrieved context?

So basically, on every query the user
gets, launch a background process,

which just asks, hey, are there
ambiguities considering this query

in this context and feeding like the
10, 20, 30 chunks we just retrieved?

And just run this continuously
because you have the contextualization

of the user query as well.

Max Buckley: You've certainly could
and write it out somewhere, like in the

background that sounds like a great idea.

Maybe I'll do that.

Thank you.

Absolutely.

What's interesting for us at least,
is that, I was giving the example

of like terminology It terminology
is difficult as you go broader, like

across the whole company, because
some of these systems could be, the

names could be used in different ways,
different kinds of different teams.

Being consistent is easier within
a particular team's code base

then across the whole company.

So just like.

Loading our use of the word and trying
to align that is a lot easier than

trying to align across teams or across
the whole world, the globe as it were.

But yes, you're right.

Especially when we retrieve like across
the entire company, we're going to get

lots of different chunks and maybe they'll
stay different things or wherever else.

It will be useful for us to keep
track of these, errors that kind

of, that happened in the background
and then actually fix them.

I didn't, hadn't thought of that.

So yeah.

Nicolay Gerold: Have fun
implementing like great weekend.

How do you actually
think about definitions?

Because I imagine you have quite
a lot of terms or lingo you use

internally for different systems.

So how do you actually think
about the use of definitions in?

And when to actually add the definitions
also into the context, or do you always

just run the rag and the definitions
are parts of the chunks as well?

Max Buckley: So this is actually
one that we have a bit of more

and more detailed thoughts on.

So actually that was something that
was some other people in my team found

in earlier for LLM efforts, right?

That the LLMs were just not aware of
Google internal semantics and vocabulary.

And so they actually found that
adding a kind of a quick glossary

of terms to the kind of beginning of
the prompt or like I suppose in the

system prompt really helped in terms
of the tasks they were trying to solve.

When we moved to large context approaches,
then a lot of these could be picked up

almost implicitly by the model, right?

Given like the model of dozens of
hundreds of files, it sees all these

terms being used in many different
contexts and seems to be able to

intuit what they actually mean, right?

Or some of them will be defined
at some point and other ones

would be maybe just implicit.

Though one funny thing, funny failure
mode again, that we saw with the large

context approach was many, as you said,
many systems have a lot of aliases, right?

Like different people refer
to it as different things.

They may be used the code name that
was used when it was being developed.

Maybe they use the external
cloud name for the thing.

They, as opposed to the internal
name, there's a lot of different

aliases for the same thing.

And when we were like
looking at some of the.

Questions, answer system data.

We noticed that there was a particular
subset of queries that were using

a particular name for a system and
not retrieving the kind of relevant.

Content and effectively they
were using the name for the back

end of the system instead of
what most people refer to it as.

And so the fix in this case was literally
adding one sentence to the documentation

that said X is a backend for Y and
every query that we've had since that

asked about X found the right thing.

But until that point, just that.

If we had the entire documentation of
the whole company or potentially we

would have got around that problem or
like semantic search or lexical search

potentially there, but this was when we
were just using large context and the.

Documentation for a particular system,
so that one sentence fixed those subset

of user questions, which was pretty cool.

But yeah, as I say that in general,
right now, a lot of what we're

looking at is semantic search, so
we're embedding the query and it

does seem to find the matching.

Terms.

If we don't like, even though we're
embedding it with a, maybe an X in

my current project, we're using an
externally available embedding model.

But it doesn't necessarily understand
the full semantics of that internal term,

but it still finds the things that are
similar and finds the correct thing.

Nicolay Gerold: Yeah.

And on a previous podcast with Nils
Reimers, we talked about like knowledge

editing in embeddings is actually
way easier than in language models.

Yeah.

So the source of failure is probably
rather the language model, because it

will, from its prior knowledge, mistake
the definition or the term for something

else, as opposed to the embedding
model, because in the embedding model,

if you fine tune it, you can basically
ingrain it as a different entity

based on the similarity you give it
with, through the continued training.

Max Buckley: And so we've
had good success here, right?

Some systems for example, have
are just a normal external word.

So if, if you ask about like guitar,
for example, it's a system at Google,

but also a real musical instrument,
depending on the approach that we've

tried, but if you do like rag, of course,
you return a bunch of documentation

about how do you set up the system
and very little to nothing about how

do you play the musical instrument.

So the LLM, It does a good job of
answering the question about the

system and not the musical instrument.

Probably if you asked it about
the musical instruments, you'd

have a different challenge.

It would probably understand
that you're just talking about

it and it would continue.

Like it would say, state something
like I said earlier, where it says

that there's no information here
about playing the musical instrument.

Nicolay Gerold: Yeah, I would actually
love to know like your thoughts on

like more context misinterpretation.

So when I have, for example, the sentence
like product A was replaced by product

B in 2020 and product B costs 100.

And then I have a user query
with this context, like how

much does product A cost?

Which information just isn't available,
but even if it's available, it

could be like screwed up by VLLM.

So have you spent any thoughts or
efforts of actually like minimizing

context, which could be misinterpreted?

By a reader or by an LLM

Max Buckley: So yes.

So I say I did do like myself, like
trying to for a particular system, more

using it as a kind of proof of concept.

Like where I found these kinds of chunks
that were ambiguous or unclear, or,

where the thing was citing something,
I was like, I found that it was quite

fruitful to like literally fix those
simple sentences if it could be done.

Now, of course, that doesn't scale to.

whole company.

But one thing I would like to do here, and
I haven't done this yet, is to potentially

use these things at pre submit time.

So to actually imagine that
someone is changing and this

obviously can be in Google or it
can be externally done as well.

If someone is changing a markdown
file, someone is like modifying or

updating the documentation in a company,
it will be great to have an LLM.

Check that change.

At the simplest thing, it could be
checking it just for like English

readability or like language of
your choice and readability, but

also like how clear is this, right?

Because sometimes people write something
like start a sentence with it and they

don't say what it is, maybe it's a new
paragraph, and of course, better to say

what it is, especially if it gets chunked
and, we don't rely on the header above it

or whatever else, but also just actually.

Does this sort of section make sense
in the context of this file or even

in the context of the whole directory?

And this way you could prevent some of
these issues going in the first place.

So you could improve the quality
of things that are getting

submitted, which of course in
turn would improve the quality of.

Things that are retrieved.

And I also think that if someone
is modifying the documentation,

they are, they have it loaded into
their personal context, right?

They're perfectly positioned to
read that, like automatically

generated comment on their change.

It says, prefer this or try, maybe
rewording this way, or, break the

sentence into two sentences, like it's
simple stuff, but it makes the thing more

readable to a human and also to an LLM.

Nicolay Gerold: Yeah And it
might even be interesting.

That's just more of a crazy thought
because it's another document The

similarity should work very well with the
already existing documentation You could

just do a naive chunking on the entire
existing documentation and run a retrieval

on it And feed all of the, like above a
threshold into a long context LLM again,

and see whether there's any contradictions
in there with the piece of documentation

you're currently writing in on, and then
you just highlight it to the writer.

Max Buckley: yeah, you certainly could.

And you certainly could, I
was like, that's a good point.

Like you could like, given what
they've written, look for that, find

the similar things and tell them
that it's similar to this or that.

In theory they can re, reuse those things,
they can embed or include those other

documentation in their documentation
rather than having to duplicate it.

They can potentially use it to
sanity check what they're writing,

especially if they're not the
team that owns the system, right?

If they're just a sort of third party
team, that's writing a kind of summary

of what the system is for their team.

Do they want to be correct?

But I really like your idea here
of even just for example, if

we have the rack system set up.

Setting like K to a large number,
like a hundred asking a question.

And then with that a hundred things
in the context, you could ask for

contradictions, errors, ambiguities, and
you would effectively get it across the

entire code base for a particular query.

Nicolay Gerold: Yeah, I think
like we talked about a lot.

I just want to have a list of
stuff to look out for, which

is like errors, ambiguities.

Contradictions out of data information
and consistent information, lost

relationships, missing definitions.

What else is out there, in your opinion,
that people should look out for in the

documentation when they're doing RAC
or in general when they're doing RAC?

Max Buckley: Say the contextual retrieval,
contextualizing the chunks, right?

The most naive approach to chunking
is to do say, a hundred character

sequences or something that this
obviously is bad for many reasons, right?

Because you're splitting
potentially words.

There's obviously more natural
things where you split on

new lines on paragraphs.

But of course.

Not all documents are well behaved.

Some people you'll find that when you
do your clever chunking strategy beyond

a hundred characters, that some people
have put everything in one sentence or

everything in one giant wall of text.

So you do need some kind
of fallback heuristics.

But even when you do these things,
you need to say, are these chunks

meaningful or like what extra information
can we capture with the chunks?

And if you're using.

Like any kind of structured data format,
then you can potentially have the chunk,

maybe the headers that are above it,
the file reference, maybe you can have

additional data generated by LLMs, right?

You could have a summary of the
document or some, you could have, like

I said, a contextualized embeddings
or questions, all of these things.

But of course, all of these things
rely again on the quality of the.

Source data.

Nicolay Gerold: Yeah, and I think
the two different types of contextual

chances, like basically position, but
also information, which is just missing.

So basically, if I have like this
without a reference to it, these are

like, I think the two major types.

Especially I think in anthropic
case, they rather take out the.

Second one, so basically the missing
information and not the position

of one in the position of one, you
can get into a lot of areas, like

depending on the type of information
you have, I see, I've seen html rack.

I'm not sure whether you've seen
that paper who are basically trying

to do something similar for html,
as you mentioned, markdown but

only take semantically meaningful
elements in the html tree, which

is also really interesting.

Nice.

And.

I think the LLM space, it's like
such a hyped area at the moment.

I always like to ask people like
underappreciated and overhyped.

What do you think?

Like in the data and AI space, what's
an underappreciated technology and

what's like an overhyped technologY?

Max Buckley: I mean, I
feel LMS are, clearly not

underappreciated or under hyped.

Fear we're at the cusp of an AI winter.

I like in the sense of, not that the
things aren't great, but just that

with the scaling, slowing down some
of the momentum will will be lost.

I think that's not a problem in the sense
that I think that RAG is hugely valuable.

I think that for a lot of companies,
for a lot of use cases, this.

Can solve the problem in a way
better way than we had before.

Like a lot of like customer
service type use cases.

This is amazingly useful.

Like you could have a small team
of elite customer service people

supporting a huge army of bots.

And again, just every.

Correct answer that they give becomes
like gold for future iterations.

And that's what is happening
with the documentation here.

If you can document it correctly once,
then it can be found and synthesized for

hundreds of user questions going forward.

And that's super valuable to
me because I don't want to

answer the same question twice.

Think that like right now we're
obviously very focused on LLMs.

Obviously traditional supervised
learning and all that is still

there, but not getting as much hype,
but I think that's the, useful.

My feeling is that things like AlphaFold
will be more significant historically

than LLMs, but that's just my intuition.

I wouldn't say it's not hyped.

It's obviously getting its own traction,
but it's not as accessible to normal

people, I think that as ChatGPT are.

Nicolay Gerold: I think in the future
we will see more and more aI systems

where you actually combine multiple types
of AI to actually get a good output.

And I think RAG is already
an example for that.

You have a retrieval system
and the LLM for generation.

And usually your retrieval consists of
multiple components as well, which often

include like more of the traditional
AI in the form of classifiers.

. . Max Buckley: I think that one thing
that I think will happen out of this,

like huge hardware buildup for LLMs, it
looks like LMS are seemingly plateauing

according to some of the news I've
been reading in terms of scaling.

But all of this hardware build out will
make it much easier and cheaper to do

all sorts of data science going forward.

You won't have as much of a struggle to
get access to the GPUs if someone went

and built millions of them for lms.

All sorts of other sort of supervised
learning use cases and unsupervised

learning can be potentially done.

I think it would be a lot easier
to found like a deep tech or a

deep RL startup today than it
would have been several years ago.

Especially if my suspicion
about the AI winter comes true.

Nicolay Gerold: Yeah.

And also with the trend.

That they are doing specialized
hardware for LLMs and then LLMs are

here to stay, like they will probably
move on to specialized hardware

and then you have an abundance of
GPUs for other types of models.

Max Buckley: One thing I really liked
watching your podcast, the other

episodes was like, one thing that's
interesting is, search is not a new space.

But I feel like with rag, it feels
new to a lot of people, or it's like

a discovery to people, like people,
like there's all these fields that

are partially overlapping and they're
converging in this kind of fun way.

So it's very interesting to see, even
within Google, like different people

have approached rag very differently.

Some people are really just focusing on
query, like actual prompt optimization,

trying to get the prompt right to
make the model work a certain way.

And then other people thinking
much more deeply about the actual

data like myself or maybe some more
people focus on the infrastructure.

And it's interesting because
people are just coming at it

from different perspectives
with different kinds of priors.

So I think there's going to
be even more fusion of these

communities over the years to come.

Like the, I think we're
democratizing some of search.

Like I think that rag type use cases make
search useful in all sorts of domains and,

multimodality music, even be more domains.

Nicolay Gerold: Thing is like search
is having a renaissance, but we first

basically had to rediscover a lot of this
stuff, which was already figured out.

And I think that's always been like,
there is a new technology hype cycle.

You combine it you figure out new ways of
using it, and then you're building on old

technologies without really knowing or
thinking about the old technology, which

would save you most of the work, probably.

Max Buckley: yeah it's fascinating.

Like for me, at least I took some
Coursera's on back in like 2014,

maybe on like how search engines
work, the kind of old TF-IDF, BM25

things and built some simple things.

And I hadn't really thought about search.

I obviously use search, but I
hadn't really thought about it or

built search systems until kind
of end of 22, beginning of, 23.

So it was interesting to go back and see,
okay, there's been a lot of progress.

And again, some of these things work
really nicely just now they're so easy

to use, like it's so easy to prototype
with some of these LM applications.

Like this one thing I love with
large context that you have some

documentation of your company.

You can just slap that in there,
ask a question if it can answer it.

You're like, all right.

It works, it's a business
value in a couple of hours,

Nicolay Gerold: yeah, and what is
something you're excited about, or

what is something you would love to
see built in your field or around it?

Max Buckley: Very excited
about all things AI.

I think it's a very exciting field.

I have this like lucky coincidence that
I got into ML in 2012, end of 2012.

I just discovered the
term machine learning.

So I've been like chasing this
for a while and then it suddenly

became the coolest thing ever and.

22.

So it's been very interesting to
watch while I'm new to search,

I'm not new to machine learning.

It's really interesting to see
how this whole thing plays out.

It's I wouldn't have thought, I don't
know that we would have had touchy with he

publicly in the end of 22 and I was taken
back by how good it was and like have

since been just impressed by the progress.

I'm happy to see where it goes next.

Riding in those self driving
Waymo's is also pretty cool.

I can attest to that fact.

Nicolay Gerold: I think
are they in Zurich?

No, right?

Max Buckley: No.

In San

Nicolay Gerold: I think in Europe
you will have to wait a few

decades before they come here.

Max Buckley: yeah I'm a little bit
worried about Europe's approach to AI,

to be honest like it feels when you're
last place in a race, it's not a great

idea to tie your shoelaces together

Nicolay Gerold: Yeah, US
innovates, we regulate.

Yeah, that's the fun stuff.

Nice.

And if people want to follow along
with you, where can they do that?

Max Buckley: LinkedIn is
probably the best place.

I post stuff reasonably regularly.

Okay.

Nicolay Gerold: Yeah.

And I will put that in show notes
and Max is a really good follow

especially when you want to stay
up to date on a lot of research as

well in like the retrieval space.

I think I've been abusing Max's post
a lot to not do the work of actually

finding the interesting papers.

So what can we take away?

I think.

The key idea.

Is to make your
documentation ready for AI.

And I think this is a theme
we will see more and more.

Across the different types of documents
and across the labor we are working

with, so for example, also, In code.

There are a lot of work will be
spent on making code basis or

documentation or anything else.

Ready for AI.

So actually we can use LLMs, optimally.

We can send agents in there to fix
bugs or to write some new features

at some point in the future.

And I think first of all, we have to fix
up what they are in the end, working with.

Because we might have like inconsistent,
documentations, some wrong facts in there.

or ambiguities all of
which has to be removed.

So AI doesn't make mistakes.

We try in the end then
cost by factual errors.

And for search specifically, I
think it's very valuable to make

like the different document quality
issues you might encounter explicit.

And just to be on the lookout for them.

And to do spot checks.

Or just free feed, like a large
portion of it where you might.

Expect failure cases or which
is like a particularly important

part of your documentation, feed
it into a long context window.

LLM.

Like Gemini and just ask it whether
it can find any inconsistencies,

any ambiguities or whatever.

I think in general, the more
sensible approach would be to

actually do that on the fly.

So actually when you're running a RAG
system, Um, you probably already ranking

anyway, so you can do one additional.

Model call without.

Immense cost.

It depends on your load in the end.

But you can, if your load is too high,
that you don't really want to spend,

like, um, the model cost of feeding
all the chunks into an extra LLM and

asking, Hey, are there any ambiguities?

Um, If then you could do spot checks.

So on 10% of the, of the generations.

Just take all the chunks.

Throw it into a long context alum.

Ask it.

Whether there are any ambiguities, any
factual errors, and when it classes

classify some ass factually inconsistent
or whatever, just throwing error.

Maybe read, retrieve.

And with something else, for
example, fall back to BM 20.

And actually throw the error or a warning.

So your search people are notified and
can actually look at the specific chunks.

And correct them themselves
or with an expert.

And I think this already is.

A good feedback loop.

To be consistently improving the
materials you're working with.

And.

I think this can get you out of that, that
you have like a lot of silent killers.

In your search database.

Because a single ambiguous sentence.

Might.

Corrupt.

And entire set of responses and
fixing those often, it's like

not enough work, but identifying
them is really challenging.

And especially if you have a really
large search database, Finding

this is completely infeasible,
so you have to do it consistently

and over time and continuously.

Eradicating the mistakes.

We will be continuing our series on.

RAG explicity next week.

I would love to know what
you think of this episode.

, let me know in the comments.

Also like, and subscribe.

I will talk to you soon.

Catch you next week.