Learn How Keyword Search Works and How Language Models can Improve it!

Search is everywhere! We all use search engines like Google, Bing, or You.com (Si apre in una nuova finestra) every day. Moreover, we search for information in apps like YouTube or X. It’s hard to imagine our everyday lives without searching. Companies also use search systems to search their internal documents. Keyword Search is the most commonly used technique to build search systems. 

We’ll discuss the following points:

• How does the keyword search work?

• How do you use a keyword search system in Python?

• How can language models improve Search Systems?

We have no time to waste. Let’s get started.

How does the keyword search work?

First, we want to introduce the high-level concept behind keyword search. The following figure shows a simple example.

Example: Keyword Search (Inspired by [1])

Let’s say we have the query “What does the abbreviation EPS stand for in the financial industry?”. We are searching a very small archive that contains four sentences. We can compare how many words match between the query and the archive. The first sentence contains two words that are common to the query. The second sentence has the word “The”, which also contains the query. In the third sentence, no word is equal. The last sentence contains four words from the query. In the figure, you can see the number of words for each sentence on the right side. In addition, you can see that the fourth sentence has the most words in common with the query. A keyword search might retrieve that as the answer.