This story was originally published on HackerNoon at:
https://hackernoon.com/i-built-a-tiny-gpt-that-speaks-sanskrit-in-a-weekend-heres-what-broke.
The off-the-shelf models are bad at Sanskrit largely because of tokenization and data scarcity
Check more stories related to machine-learning at:
https://hackernoon.com/c/machine-learning.
You can also check exclusive content about
#transformers,
#sanskrit,
#from-scratch-transformer,
#gpt,
#build-your-own-gpt,
#tokenization,
#llms,
#hackernoon-top-story, and more.
This story was written by:
@amitshukla. Learn more about this writer by checking
@amitshukla's about page,
and for more stories, please visit
hackernoon.com.
I gave myself a weekend and a constraint: build a GPT small enough to understand completely, but on a language that would actually fight back — Sanskrit. I have a pile of Devanagari text and an NVIDIA DGX Spark sitting on my desk, so why not?