Subscribe
Share
Share
Embed
Fine Tune Llama 3 For Sql Query Generation Tutorial
Dispatches from a 13-site AI empire — what actually works in production, what fails, and what nobody tells you about building with AI.
# Transcript
**Generated:** 2026-06-15 13:15 UTC
**Source:** deepgram
**Niche:** ai
**Episode:** ep_187_
---
Nick. This is Signal Notes. I'm Nick. And today, we're going to fine tune Llama 3 for SQL generation
in an afternoon for under $50.
And I'm gonna show you why your generic AI chatbot will get you fired. Nick, here's the scene.
You're building an analytics dashboard for your company. You integrate ChatGPT or Claude.
You give it your schema. You ask for quarterly sales. It returns a query. It looks right.
You run it. It's wrong. It invented a column called fiscal quarter that doesn't exist. Your VP sees
a number that's off by 40%.
Now you're in a meeting explaining why AI generated SQL isn't ready for production. That happened to
a friend of mine last month. He doesn't want me to use his name, but he's running 13 sites. He
trusted a generic model.
It cost him a morning of credibility. Here's the thing. Everyone is selling you on AISQL tools. Text
to SQL, natural language queries.
The promise is seductive. Type a question, get a database answer. But the reality is these tools are
brittle. They hallucinate schemas.
They expose sensitive table names and prompts. They can't learn your company's specific business
logic. Your sales table might have a column called gross revenue that everyone internally calls top
line. A generic model won't know that.
So what's the operator's move? You don't wait for OpenAI to release a better SQL model. You don't
pay for a closed source tool that locks you into their API. You fine tune your own model.
You own the weights. You own the data. You own the deployment. And the barrier to doing that has
crashed.
2 years ago, fine-tuning a large language model required a cluster of a 1 hundreds and a PhD
student. Student. Today, we're using llama38b, open weights, available on Hugging Face. We're
running it on a single RTX 4,090 from RunPod at 79¢ an hour.
We're using Unsloth, a library that cuts training time in half and memory usage by 50%. The total
GPU cost for our run? 62¢. 62¢ to produce a model that generates SQL specific to our database, our
conventions, our business logic.
This isn't research. This is a deployable artifact, like a Docker container for your brain. Nick,
everyone starts with the wrong question. They ask, how do I give my model my entire include foreign
keys, indexes, column descriptions.
The prompt becomes a novel. The model gets confused. The cost per query skyrockets because you're
paying for all those tokens. We found the opposite works.
fine-tuning doesn't need your schema. It needs your pattern. It needs examples of how your company
actually writes SQL, not the theoretical structure of your database, the practical queries your team
runs every day. Here's the difference.
A generic prompt approach looks like this. Here are 50 tables. Here are their relationships. Now
write me a query for Q3 sales.
The model has to reason through all that context. It's slow. It's expensive. It's error prone.
A fine-tuning example looks like this. ASAS instruction, get total sales for Q3 20 24. Output,
select some amount. From sales, where sale, date between 20 24 7 0 1 and 20 24 0 9 30.
That's it. That's 1 training example. 200 to 500 of these, and your model learns the dialect. It
learns that your company calls revenue amount.
It learns that you use between for date ranges, not Marsan, and orte. It learns that your sales
table is called sales, not transactions or orders. The key insight is this, you're not teaching at
your database, you're teaching at your company's dialect of SQL. Every company has 1.
Your team uses left join more than inner join. You use coalesce for null handling. You name your
aggregates consistently. These patterns are in your query logs.
They're sitting there waiting to be extracted. So how do you build this dataset? You start with your
database logs. Most databases log every query.
PostgreSQL has PG stat statements. MySQL has the general query log. Even if you don't have full
logging, your application code has the queries it runs. Export your top 100 most run select queries.
These are your goal. Now, you need to annotate them. Each query needs a plain English instruction.
This is the manual part, but it's not that manual.
For a 100 queries, you're looking at maybe 2 hours and you can use cheap AI to help. We used GPT 4 0
Mini at $3 per million input tokens to clean up and format our query logs. We fed it the raw query
and said, write a short instruction that describes what this query does. Use business language, not
SQL language.
It did a solid job for pennies. The format is JSON, 1 JSON object per line. Each object has 2
fields, instruction and output. The instruction is the natural language request.
The output is the SQL query. That's it. No schema, no table descriptions, no foreign key
documentation, just pairs. Why does this work?
Because the model learns to map natural language patterns to SQL patterns. It doesn't need to
understand your database structure. It needs to understand the mapping between Q3 sales and sale
date between 20 24 0 7 z u 1 and 20 24 z u 9 30 o. That mapping is in your logs.
It's specific to your company. It's your moat. Nick. Let's get practical.
Here's the exact tool chain we used. I'm gonna walk through it step by step. No magic, no
gatekeeping, just the commands and the costs. Step 1, the environment.
We started in Google Colab for prototyping. Free GPU, but limited to T4s and limited runtime. Good
for testing your dataset format. Bad for actual training.
So we moved to run pod. We spun up a pod with an RTX 4,090. Cost, 79¢ per hour. We ran for 47
minutes.
Total cost, 62¢. Step 2, the library. UnSloth.
This is the key. UnSloth is a drop in replacement for Hugging Face's transformers library that's
optimized for fine-tuning. It gives you a 2 times speed up and 50% less memory usage.
It supports llama 3 out of the box. You install it with pip. Then you import it, load the model, and
configure the training. Here's the code in plain English.
You import Unsloth. You load llama 3 8 b with 4 bit quantization. You set the LoRa rank to 16. LoRa
is low rank adaptation.
It's the technique that makes fine-tuning cheap. Instead of updating all 7,000,000,000 parameters,
you update a small subset. The rest stay frozen. This is why you can do it on a single GPU.
You load your JSONL file. You format it into the chat template that llama 3 expects. You set the
training parameters, learning rate, batch size, number of epochs. For 500 examples, we used 3
epochs.
That's 1,500 training steps. Each step takes about 2 seconds. Total training time, 47 minutes. Step
3, the conversion.
Once training is done, you have a set of adapter weights. Those are the LoRa weights. They're not a
standalone model. You need to merge them with the base model and export to gguf format.
Gguf is the format used by LAMAD CPP, which lets you run the model on a CPU or a modest GPU. No
cloud dependency, no API call. You run it locally. The conversion takes about 10 minutes.
The output is a single file, about 4 gigabytes. That's your deployable artifact. You can copy it to
any machine. You can run it on a laptop.
You can serve it with a simple API using llama. Cpp's built in server. The total cost breakdown. Run
pod GPU time.
0.62, dataset preparation using GPT 4 mini, 0.03, Colab prototype, free, total 0.6 5 under $1. For a
custom SQL generation model that knows your company's dialect, That's the shift. That's what's
happening right now. The cost of custom AI has dropped below the cost of a coffee.
Nick, if you wanna skip the setup and go straight to training, we've got you covered. We packaged
our exact Colab notebook, the dataset format template, and the inference script. It's all pre
written. Head to signalnotes.aillama SQL to download the kit.
No email required, just the tools that worked for us. Nick. fine-tuning is step 1. Step 2 is what
separates a demo from a deployment.
And the biggest mistake I see is skipping validation. Your fine tuned model will still make errors.
It's not perfect. It might generate a syntax error.
It might generate a query that works, but returns the wrong data. It might generate a query that
tries to drop a table. You need a validation pipeline. Here's our pipeline.
3 stages. Stage 1, syntax check. After the model generates SQL, we run it through a lightweight
parser. We use SQL parse for Python.
It catches missing parentheses, wrong keyword order, unclosed strings. This takes milliseconds. If
it fails, we return an error to the user and log the failure for retraining. Stage 2, explain plan.
If syntax passes, we run explain on the query. This tells us the query plan without actually
executing the query. It catches table name errors, column name errors, ambiguous references. It also
tells us if the query will be expensive.
If the plan shows a full table scan on 1000000000 row table, we flag it. We don't run it. Stage 3,
read only transaction with time out. If explain passes, we execute the query inside a read only
transaction.
This prevents any write operations. We set a time out of 30 seconds. If the query takes longer, we
kill it. This protects the database from runaway queries.
The read only transaction is critical. You don't want a model generated query to accidentally delete
data. Use set transaction, read only in PostgreSQL. Use start transaction, read only in MySQL.
This pipeline catches about 95% of errors before they reach the database. The remaining 5% we catch
through monitoring. Every query that returns an error gets logged. We periodically review these logs
and add them to our training dataset.
The model gets better over time. Now, let's talk cost. Running your own model is dramatically
cheaper than API calls. Our model runs on a single RTX 4,090.
We can serve about 50 queries per 2nd. The cost per query is approximately 70 micro cents. That's 0
to 1007 per generated SQL query. Compare that to OpenAI's API.
GPT 4 costs about 2 tenths of a cent per query, depending on prompt length. That's 0 TAUs per query.
Our model is 28 times cheaper and we own the infrastructure. No rate limits, no API outages, no data
privacy audits.
This is the contrarian take. Everyone says just use the assistance API, just use the ChatGPT plugin.
But when you own the model, you control the entire stack. You're not integrating a vendor risk.
You're shipping a feature. Your security team doesn't need to audit OpenAI's data handling policies.
Your compliance team doesn't need to worry about where your SQL schema is being sent. The model runs
on your hardware.
The data never leaves your network. And the savings add up. We're handling about 80% of our internal
analytics request with this model. That's roughly 40 queries per day.
At OpenAI pricing, that would be about 80¢ per day. On our own infra, it's about 3¢ per day. Over a
year, that's a savings of about $280. Not life changing?
But the real savings is time. Our data team saves about 4 hours per week of ad hoc query writing.
That's 200 hours per year. At a fully loaded cost of $100 per hour, that's $20,000 in saved labor.
Nick, here's what I want you to do today. Don't fine tune. Not yet. Just audit your database logs
for 1 hour.
Export your top 100 most run select queries. Look at them. See the patterns. Notice how your team
writes SQL.
Notice the column names. Notice the join patterns. Notice the date formatting. Then take 1 query,
just 1.
Write a plain English instruction for it. Format it as a JSON object. Instruction, your instruction,
output, your SQL. That's your 1st training example.
You're already 50% of the way there. Do this for a week, collect 200 examples, then come back to
this episode, download our kit, run the notebook, you'll have a working model in under 2 hours.
Nick, if you're building production AI systems, you need to know what breaks and how to fix it. Our
sister show, In Production, covers exactly that.
Real system alerts, real post mortems, real deployment stories from people who run AI in production.
Not theory, not speculation, just the alerts and the fixes. Find it wherever you get podcasts. Nick,
that's the episode.
Here's the short version. Your generic SQL chatbot will get you fired. fine-tuning is cheap and
fast. Your query logs are your dataset.
The pipeline you wrap around your model is what makes it production grade. This is Nick. Ship it,
monitor it, and we'll talk about scaling it next time.