Just Do It | Linh B. Ngo

A trip to Barnes & Noble with my children inspired me to pick up another new technical book (I got my non-technical fix from Kindle Unlimited), Build a Large Language Model (From Scratch) by Sebastian Raschka. Getting into bed at 9:30PM, I went through the book carefully (by my estimate) and finished the first two chapters in an hour. The book was fascinating, as the author went through the in-depth details of the why and how of fundamental LLM tasks such as tokenization and embedding. I loved it then, and in the morning, I had a disappointing feeling that I had not learned much about anything at all.

Recalling my lesson from Learning How to Learn, the only Coursera course that I completed, I realized that I ran into an illusion of learning the previous evening. My background allowed me to understand and appreciate the beautiful details and simplicities in Raschka’s book. At the same time, these have remained Raschka’s knowledge and not yet mine. I need to do something to make it mine.

In the remainder of this essay, I will record a sample of my learning process for the concept of embedding in Chapter 2 of the book. This helps me to reflect and optimize on my own learning activities prior to moving on to more complex chapters later. It should be noted that both this blog post and the coding activities are/will be done completely on LazyVim as part of my other training process.

Just Do It (Something, Anything)

Raschka maintains a GitHub repository accompanying the book. The repository is frequently updated (latest commit was less than a month prior to the date of this blog). I forked this repository to keep a copy, but decided against simple cloning and running the codes as is since it still felt like demonstrative surface learning. Instead, I created a new learning repository in which each directory will be dedicated to all the coding activities created from scratch for each technical book going forward.

I decided to go with just the pyproject.toml setup file instead of maintaining an additional requirements.txt file. The toml file is roughly similar to the original, with some additional packages added as I plan to further explore beyond the book’s provided code examples. An internal README.md is maintained to describe specific preparation steps.

Embedding

When our research group worked on setting up a RAG, one of the things I learned is that we need to run the raw text through an embed model (nomic-embed-text) to make it ready for the orchestrator/generator agents of the framework. The students managed to make this initial data processing stage worked, but I didn’t fully grasp the underlying concepts. The beginning of Chapter 2 of the book discusses this concept of embedding.

Embedding is the process of converting raw text into continuous-valued vectors. The chapter did not include mathematical discussion but provided illustrative figures of the concepts. I want to see how eagle, duck, goose, and squirrel can be clustered together as per Figure 2.3. A preliminary inquiry with Gemini generated a sample code, which was not very successful. The printouts below showed that the default generated options for Word2Vec training did not work, as a squirrel is closer to a duck than a goose does.

 (llms_from_scratch_env) lngo@DESKTOP-A639ILL:~/workspace/linhbngo/my-learning/llms-from-scratch/ch02$ python word2vec_demo1.py 
=== RAW WORD EMBEDDINGS (4 Dimensions) ===
Eagle vector:    [0.6833 0.3334 0.694  0.175 ]
Goose vector:    [ 0.5307  0.1152  0.8012 -0.0018]
Duck vector:     [0.6185 0.2096 0.78   0.3212]
Squirrel vector: [0.5538 0.3936 0.7921 0.3226]

========================================

=== COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.9460  (Very High - both are water birds)
Eagle vs Goose:    0.9466  (Moderate/High - both are flying birds)
Eagle vs Duck:     0.9786  (Moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.9839 (Low - completely different context)
Squirrel vs Goose: 0.9206 (Low - completely different context)
Squirrel vs Eagle: 0.9783 (Low - completely different context)
 

Some further discussion with Gemini helped me to:

Truly appreciate data size. As you will see later, even a 100ish MB text corpus is still nowhere near enough.
Better understand the various options of Word2Vec, or of the embedding process in general.

More specifically, I modified the generated python file to read in text data from the Aristo MINI Corpus. With more than 1.1 millions science-related sentences, this seems to be a large and generic enough training dataset. Yet, the results still looked bad, even with stopwords being taken off.

 # with stopwords
=== COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.6160  (Very High - both are water birds)
Eagle vs Goose:    0.4722  (Moderate/High - both are flying birds)
Eagle vs Duck:     0.4270  (Moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.6647 (Low - completely different context)
Squirrel vs Goose: 0.6885 (Low - completely different context)
Squirrel vs Eagle: 0.6810 (Low - completely different context)

# without stopwords
=== COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.5542  (Very High - both are water birds)
Eagle vs Goose:    0.5729  (Moderate/High - both are flying birds)
Eagle vs Duck:     0.5057  (Moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.5693 (Low - completely different context)
Squirrel vs Goose: 0.5692 (Low - completely different context)
Squirrel vs Eagle: 0.6871 (Low - completely different context)
 

Then I realized that perhaps Aristo Corpus is not about animals. Switching to the Wikipedia text via HuggingFace/Saleforces definitely makes more sense. I started out with using export ... to manually set the access token for HuggingFace and ended up deciding to use hf auth login for convenient sake.

The third version of the code has substantially more edits, many of which made to help me clarifying concepts and streamlining the experimentation process.

Data downloaded from HF is now cached in data (gitignored) for repeated usage.
More explanations are added to clarify the purposes of various options in Word2Vec training call.
Modifications are made to the notes at the end of the print statement strings to clarify that these are not conclusions but expectations, which could be mismatched with the actual scores.
- model = Word2Vec(sentences, vector_size=50, window=5, min_count=5, seed=42, sample=1e-3, workers=8, epochs=10)
Printouts on unique word counts are added to test my hypothesis that unbalanced appearances can be a contributor to the unexpected scores.

The outcomes are making better sense, so to speak. Within the birds, Goose and Duck are now noticeably closer to each other than to Eagle. However, Squirrel is still uncomfortably close to all three bird words! The unique counts showed that Goose and Squirrel have similar low counts, as compared to Eagle and Duck. I wonder if this explains why, within the four-way comparison results, Squirrel is closer to Goose than to Duck and Eagle. In other words, the embedding process in this run with the current settings has not been able to completely isolate the semantic meaning of the words from their non-semantic, pure statistical existential attributes. The names for these types of issues include frequency bias and data sparsity. I think with the four-sentence training repository and even the Aristo Corpus, I was facing the issue of data sparsity. Initially, the four-sentence training dataset was absurdly small. Later, the Aristo Corpus focuses more on scientific facts and less on animals! I probably don’t have the data sparsity issue with the Wikitext data, but now, due to the imbalance between the counts of the birds and the squirrel, I have a frequncy bias issue.

 === COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.6013  (Very High - both are water birds)
Eagle vs Goose:    0.4644  (Moderate/High - both are flying birds)
Eagle vs Duck:     0.3376  (Moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.5701 (Low - completely different context)
Squirrel vs Goose: 0.7013 (Low - completely different context)
Squirrel vs Eagle: 0.6302 (Low - completely different context)
Total processed lines/sentences: 1162179
Example line: ['valkyria', 'chronicles', 'iii']

=== NEW ANIMAL COUNTS ===
Goose: 308 times
Duck: 1131 times
Squirrel: 317 times
Eagle: 2838 times
 

I decided to rerun the program with the following set of options.

 model = Word2Vec(sentences, vector_size=100, window=10, min_count=10, seed=42, sample=1e-5, workers=8, epochs=100)
 

vector_size is increased so that each word is represented by a larger vector.
window is longer in hope of capturing longer sentences/contexts.
min_count is increased to tell Word2Vec to be more aggressive in removing low-frequency words
epochs is increased so that more training happens (there is a risk of over-fitting here!).

The outcomes below are interesting so to speak. All similarity scores are reduced. Within the birds, Goose and Duck remain more similar to each other compared to Eagle. Squirrel is still closer to the birds than the birds to one another.

 === COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.4405  (Expect very High - both are water birds)
Eagle vs Goose:    0.3396  (Expect moderate/High - both are flying birds)
Eagle vs Duck:     0.3212  (Expect moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.4260 (Expect low - completely different context)
Squirrel vs Goose: 0.3794 (Expect low - completely different context)
Squirrel vs Eagle: 0.4536 (Expect low - completely different context)
 

Feeding this outcome into Gemini taught me a few new things as well. Bigger is not always better. It turns out that window=10 mean look at 10 words to the left and 10 words to the right, for a total of 20 words span. I incorrectly assumed the concept of sliding windows in typical programming here. I also did not really appreciate the impact of epochs=100 until I see the numbers putting into a coherence sentence: Running 100 epochs on words that only appear ~300 times forces a 100-dimensional vector to perfectly memorize the specific quirks of those few sentences (quote from Gemini).

Of course, before I learned the above paragraph, I already jumped the gun and made the numbers bigger. More specifically, I wondered if a bigger decription (larger embedding via vector_size=256) can better describe the words. Once again, reasoning based on raw numbers say no: with only 317 appearances, it is going to be difficulty to represent those instances (the words and their surrounding context words) in as many as 256 elements. I am creating my own data sparsity problem here.

 model = Word2Vec(sentences, vector_size=256, window=10, min_count=10, seed=42, sample=1e-5, workers=8, epochs=100)
 

The outcomes of course performed badly as predicted based on Gemini’s feedback. While Goose is quite similar to Duck as compared to Eagle, Squirrel seems to be flying high in the sky with all the birds!

 === COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.3635  (Expect very High - both are water birds)
Eagle vs Goose:    0.2265  (Expect moderate/High - both are flying birds)
Eagle vs Duck:     0.1891  (Expect moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.2188 (Expect low - completely different context)
Squirrel vs Goose: 0.1841 (Expect low - completely different context)
Squirrel vs Eagle: 0.3091 (Expect low - completely different context)
 

Reducing to window=5 and epochs=15 helped making the similarity between Goose and Duck more prominent, but it did not help the Squirrel versus birds situation.

 === COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.5315  (Expect very High - both are water birds)
Eagle vs Goose:    0.3510  (Expect moderate/High - both are flying birds)
Eagle vs Duck:     0.3248  (Expect moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.4501 (Expect low - completely different context)
Squirrel vs Goose: 0.5222 (Expect low - completely different context)
Squirrel vs Eagle: 0.4521 (Expect low - completely different context)
 

I attempted to rerun and also print out the top-15 neighboring words with highest frequencies. It turns out that while Goose and Duck share two high-ranking words (quail and aythya), Squirrel also has quail in the neighboring list. In fact, two of the top neighboring words are woodpecker and woodpeckers, which would imply birdiness! On the side of the Eagle, it seems that a lot of the related words came from military context (aldertag: Eagle Day) or US monetary context (coin, mintmark). In other words, the dataset I am using does not have dedicated natural contexts for these words as I thought.

 === COSINE SIMILARITY SCORES === 
Goose vs Duck: 0.5619 (Expect very High - both are water birds) 
Eagle vs Goose: 0.3475 (Expect moderate/High - both are flying birds) 
Eagle vs Duck: 0.3322 (Expect moderate/High - both are flying birds) 
---------------------------------------- 
Squirrel vs Duck: 0.4530 (Expect low - completely different context) 
Squirrel vs Goose: 0.5122 (Expect low - completely different context) 
Squirrel vs Eagle: 0.4556 (Expect low - completely different context) 
Total processed lines/sentences: 1162179 
Example line: ['valkyria', 'chronicles', 'iii'] 
=== NEW ANIMAL COUNTS === 
Goose: 308 times 
Goose: ['greylag', 'quail', 'swans', 'quails', 'pochard', 'geese', 'partridges', 'wigeon', 'blackbirds', 'kingfishers', 'coot', 'anser', 'pipit', 'aythya', 'loon'] 
Duck: 1131 times 
Duck: ['daffy', 'lame', 'mucky', 'pekin', 'goose', 'fulvous', 'quail', 'anas', 'angler', 'bat', 'eared', 'aythya', 'aviculture', 'pheasant', 'ruddy'] 
Squirrel: 317 times 
Squirrel: ['squirrels', 'woodpecker', 'woodpeckers', 'eared', 'porcupines', 'sciurus', 'cottontail', 'jackrabbit', 'accipiter', 'weasels', 'titi', 'wagtails', 'lanius', 'bellied', 'quail'] 
Eagle: 2838 times 
Eagle: ['bald', 'owl', 'obverse', 'bellied', 'haliaeetus', 'adlertag', 'leucocephalus', 'coin', 'eagles', 'gobrecht', 'mintmark', 'wreath', 'bison', 'wildcat', 'lion']
 

Further exploration with Gemini led me to switch to a simple wikipedia dataset (Tralalabs/simple-english-wikipedia). This dataset is sourced from Simple Wikipedia, which is written with a severely restricted vocabulary and basic grammar. The Saleforce/wikitext dataset is from the general Wikipedia, which contains more complex languages, topics, and proses. Keeping all the options the same and only replacing the data set (customized code to account for data structure differences) give us the following outcomes.

 === COSINE SIMILARITY SCORES ===
Goose vs Duck:     0.5688  (Expect very High - both are water birds)
Eagle vs Goose:    0.4368  (Expect moderate/High - both are flying birds)
Eagle vs Duck:     0.2698  (Expect moderate/High - both are flying birds)
----------------------------------------
Squirrel vs Duck:  0.4850 (Expect low - completely different context)
Squirrel vs Goose: 0.5291 (Expect low - completely different context)
Squirrel vs Eagle: 0.3502 (Expect low - completely different context)
Total processed lines/sentences: 3345393
Example line: ['monththisyear', 'april', 'apr', 'is', 'the', 'fourth', 'month', 'of', 'the', 'year', 'in', 'the', 'julian', 'calendar', 'julian', 'and', 'gregorian', 'calendar', 's', 'and', 
'comes', 'between', 'march', 'and', 'may']

=== NEW ANIMAL COUNTS ===
Goose: 712 times
Goose: ['geese', 'greylag', 'branta', 'amusingplanet', 'chough', 'anser', 'duck', 'ducklings', 'muskrat', 'cock', 'swans', 'eider', 'quail', 'gummi', 'whistling']
Duck: 2935 times
Duck: ['daffy', 'darkwing', 'quacky', 'aflac', 'goofy', 'mallard', 'quack', 'lame', 'talespin', 'goose', 'quacks', 'ducky', 'beanstalk', 'vdka', 'fbg']
Squirrel: 614 times
Squirrel: ['nutkin', 'hairy', 'squirrels', 'whistling', 'hooded', 'skunk', 'raccoon', 'screwy', 'aardvark', 'possum', 'nosed', 'bigfoot', 'hamster', 'scaredy', 'ladybug']
Eagle: 4643 times
Eagle: ['harpy', 'leucocephalus', 'chrysaetos', 'haliaeetus', 'desert', 'tailed', 'bird', 'kwong', 'sky', 'huntress', 'vib', 'falcon', 'purple', 'bald', 'blue']
(
 

The results are far from semantically perfect, but it definitely improved. Goose is close to Duck, and Eagle is closer to Goose (flying?) than Duck (can fly, but not culturally known for that!). Squirrel is close to Duck, but the neighboring words gave a clear explanation: cultural artifacts through famous cartoon characters such as Daffy Duck and Darkwing Duck and Screwy Squirrel and Nutkin Squirrel. Squirrel and Goose are close though one another through their shared closeness with Duck, and through the fact that they both have low unique counts comparing to Duck and Eagle. On that note, Eagle is definitely distinguished from the other words now, and I am happy to see that Eagle is more similar to Goose than Squirrel! We are getting there semantically.

I stopped at this point to move on to the remainder of Chapter 02, but I am entertaining some testing idea in my head: Perhaps I can do a preliminary clean up on the training data and remove all culturally related entries. Perhaps that will boost the score accuracy further.

What Else Did I Learn?

The biggest lesson here is that data quality determines everything. I suspect that complex algorithms and techniques can account for data quality, but it is likely that we are paying for it through computational costs. Working with these relatively large datasets (hundred of MBs) and long run time (tens of minutes) also seems to adjust my expectation/anticipation in working with ML/AI. It will take time.

I am aware that Word2Vec is one of the earlier techniques that cannot be compared to the modern embedding models. At the same time, it is rudimentary enough to let me observe the intricacies of generating numerical representations of words in such a way that the surrounding context of that word inside the training data is preserved. Whether that training data carries the intended semantic is an entirely different question.

I’d like to think my LazyVim skills improved quite a bit. I am now more comfortable with the Visual mode and how to select/yank (select/copy) using keyboards. I also learned that I am not supposed to wq when opening up files from an explorer tree and that bd is the correct option if I want to just close a file.

Conclusion

Learning by doing. That sentence is absolutely correct. Also, AI-supported learning is wonderful. At the same time, I have to acknowledge the fact that a lot of my background knowledge has helped me in framing my questions to Gemini. If I was to start from scratch, I will not know the appropriate question and follows-up to consider. I am stopping this entry here as it is getting quite lengthy, but this has been a wonderful learning process. I was not able to produce a result that would illustrate the idea in Figure 2.3, and yet I feel strongly that through all the failed attempts, I have owned part of that knowledge now. And the best part is that I am not even getting into the deeper technical part of the Chapter yet!

Just Do It (Something, Anything)

Embedding

What Else Did I Learn?

Conclusion

Enjoy Reading This Article?