{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Machine Learning Tech Brief By HackerNoon","title":"Let's Build Our Own LLM (Part 1): Tokenization and Data Prep","html":"","width":"100%","height":180,"duration":1175,"description":"\n This story was originally published on HackerNoon at: https://hackernoon.com/lets-build-our-own-llm-part-1-tokenization-and-data-prep.\nHow LLMs turn text into numbers: BPE tokenization explained step by step, why your choice of tokenizer shapes model quality, and building a data pipeline.\nCheck more stories related to machine-learning at: https://hackernoon.com/c/machine-learning.\n You can also check exclusive content about #ai-engineering, #llms, #tokenization, #byte-pair-encoding-(bpe), #ai-data-pipeline, #bpe-algorithm, #huggingface-tokenizers, #hackernoon-top-story, and more.\nThis story was written by: @jayrajch. Learn more about this writer by checking @jayrajch's about page,\n and for more stories, please visit hackernoon.com.\nLLMs don't see words, they see tokens, chunks of text learned by an algorithm called Byte Pair Encoding that repeatedly glues the most frequent character pairs together. A tokenizer trained on Reddit will shred \"myocardial\" into meaningless fragments; one trained on medical text keeps it whole. That choice ripples through everything. This article walks through BPE merge-by-merge with a toy corpus, compares how GPT-4, LLaMA-2 and BERT tokenize clinical text, then covers the data pipeline, deduplication, quality filtering, and the token-count math you should do before spending a dollar on GPUs. Working Python code for all of it.","thumbnail_url":"https://img.transistorcdn.com/KyA01h2FD2insgk-wX_xzV6vbJnTNl2BvPYVL-XaI9A/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9zaG93/LzQxMjcyLzE2ODM1/ODI0ODgtYXJ0d29y/ay5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}