Index

One key feature of Qdrant is the effective combination of vector index and traditional index. This is crucial because merely having a vector index is not sufficient to enable efficient vector search under filtering conditions. In simple terms, the vector index accelerates vector search, while the payload index accelerates filtering.

The index within a paragraph exists independently, but the parameters of the index itself are configured for the entire set.

Not all paragraphs automatically have indexes. This depends on the requirements of the optimization settings and usually depends on the number of stored points.

Payload Index

The payload index in Qdrant is similar to the index in a traditional document-oriented database. This index is built for specific fields and types, used to quickly retrieve points based on corresponding filter conditions.

It is also used to accurately estimate the cardinality of filtering conditions, which helps the query planner choose search strategies.

Creating an index requires additional computational resources and memory, so it is crucial to carefully choose the fields to be indexed. Qdrant does not make this choice for you, but rather delegates it to the user.

To mark a field as indexable, the following method can be used:

PUT /collections/{collection_name}/index

{
    "field_name": "name of the field to be indexed",
    "field_schema": "keyword"
}

Available field types are:

  • keyword - for keyword payload, affecting matching filter conditions.
  • integer - for integer payload, affecting matching and range filter conditions.
  • float - for float payload, affecting range filter conditions.
  • bool - for boolean payload, affecting matching filter conditions (available from version 1.4.0 onwards).
  • geo - for geographical payload, affecting geographical bounding box and geographical radius filter conditions.
  • text - a special index type suitable for keyword/string payload, affecting full-text search filter conditions.

Payload indexes may consume additional memory, so it is recommended to only index fields used in filtering conditions. If you need to filter based on many fields, and memory constraints do not allow all of them to be indexed, it is advisable to choose the fields that limit the search results the most. Generally, the more unique values a payload value has, the more effective the index usage will be.

Full-text Index

Available from version 0.10.0

Qdrant supports full-text search for string payloads. Full-text index allows you to filter points based on the presence of words or phrases in the payload field.

Configuring a full-text index is slightly complex because you can specify tokenization parameters. Tokenization is the process of breaking a string into tokens and then indexing these tokens into an inverted index.

To create a full-text index, you can use the following method:

PUT /collections/{collection_name}/index

{
    "field_name": "field name to index",
    "field_schema": {
        "type": "text",
        "tokenizer": "word",
        "min_token_len": 2,
        "max_token_len": 20,
        "lowercase": true
    }
}

The available tokenization methods are:

  • word - Separates the string based on spaces, punctuation, and special characters.
  • whitespace - Separates the string based on spaces.
  • prefix - Separates the string based on spaces, punctuation, and special characters, then creates a prefix index for each word. For example, hello will be indexed as h, he, hel, hell, hello.
  • multilingual - A special type of tokenization based on the charabia package. It allows correct tokenization and lemmatization of multiple languages, including languages with non-Latin alphabets and non-space separators. Refer to the charabia documentation for a complete list of supported languages and normalization options. In the default build configuration, qdrant does not include support for all languages as it would increase the size of the binary files. Chinese, Japanese, and Korean are not enabled by default, but can be enabled by building qdrant from the source code using the --features multiling-chinese, multiling-japanese, multiling-korean flags.

Please refer to the full-text matching examples for an example of using a full-text index for queries.

Vector Indexing

Vector indexing is a data structure based on vectors, constructed through specific mathematical models. With vector indexing, we can efficiently query multiple vectors similar to the target vector.

Currently, Qdrant only uses HNSW as the vector index.

HNSW (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. According to certain rules, it constructs a multi-layer navigation structure for the graph. In this structure, the upper layers are sparser, with greater distances between nodes. The lower layers are denser, with smaller distances between nodes. The search starts from the top layer, finding the nearest node to the target in that layer, and then entering the next layer for another search. After multiple iterations, it can quickly approach the target position.

For performance improvement, HNSW limits the maximum degree of nodes on each layer of the graph to m. Additionally, you can specify the search range using ef_construct (during index construction) or ef (while searching for the target).

These parameters can be configured in the configuration file:

storage:
  hnsw_index:
    m: 16
    ef_construct: 100
    full_scan_threshold: 10000

During the collection creation process, the ef parameter can be configured and is equal to ef_construct by default.

HNSW was chosen for several reasons. Firstly, HNSW is highly compatible with allowing Qdrant to modify filters during the search process. Secondly, according to public benchmark tests, it is one of the most accurate and fastest algorithms.

Available from v1.1.1

HNSW parameters can also be fine-tuned for collections and named vectors to optimize search performance by setting hnsw_config.

Filterable Index

Separate payload indexes and vector indexes alone cannot fully address the use of filters for searching.

In cases where the filters are weak, HNSW index can be directly used. In cases where the filters are strict, payload index can be used and completely rescored. However, in intermediate cases, this approach is ineffective.

On one hand, we cannot afford to perform a full scan on too many vectors. On the other hand, when using overly strict filters, the HNSW graph begins to break down.

Additional edges allow you to efficiently use the HNSW index to search for neighboring vectors and apply filters while searching in the graph.

This approach minimizes the overhead of condition checks, as you only need to compute conditions for a small portion of points participating in the search.