Search result scoring

Explains how the results provided in ISearchResults<T> are ordered, and how LIFTI calculates its scores.

LIFTI uses a version of the Okapi BM25 algorithm to score search results. At the simplest level this means that search results will come back ordered by relevance. Fuzzy matching affects the scores for search results depending on the distance between the target word and the search term.

Once nice feature of LIFTI is that in you also get each field scored independently. The overall score for a document is just a sum of these, but you could easily just re-order the results by one field over another should you so wish.

ISearchResults<T>.OrderByField returns a new instance of ISearchResults<T> with the results re-ordered considering only the scores for a single field:

var index = new FullTextIndexBuilder<int>()
    .WithObjectTokenization<Customer>(o => o
        .WithKey(c => c.Id)
        .WithField("Name", c => c.Name)
        .WithField("Profile", c => c.ProfileHtml, textExtractor: new XmlTextExtractor())
    )
    .Build();

await index.AddAsync(new Customer { Id = 1, Name = "Joe Bloggs", ProfileHtml = "<a>Something else something</a>" });
await index.AddAsync(new Customer { Id = 2, Name = "Joe Something", ProfileHtml = "<a>Something else</a>" });

// Searching for "Something" will result in ID 2 being ordered before ID 1.
// "Something" appears twice in each document overall, however document 2 has fewer words, therefore the matches
// are more statistically significant.
var results = index.Search("something");
PrintSearchResults(results); 

// Output
// 2
// 1

// But if you only consider the "Profile" field, then the Something only appears once in document 2,
// therefore document 1 will come first.
results = results.OrderByField("Profile");
PrintSearchResults(results);

// Output
// 1
// 2

Score boosting

You can boost scores in a number of ways:

  • Boosting search terms as part of the query: thanks^1.5.
  • Boosting fields associated to an object: .WithField("Name", c => c.Name, scoreBoost: 1.5D).
  • Boosting objects based on a freshness date, e.g. the date it was last updated.
  • Boosting objects based on a magnitude value, e.g. a star rating.
Last modified January 16, 2024: V6.0.0 (#107) (125ae87)