Collections

The concept of collections in the Qdrant vector database can be analogous to the table structure in MYSQL, used to uniformly store the same type of vector data. Each piece of data stored in a collection is referred to as a point in Qdrant. Here, a point is similar to the mathematical geometric space concept, representing the representation of a vector in geometric space (just consider it as a piece of data).

In the same collection, each point's vector must have the same dimensions and be compared using a single metric. Named vectors can be used to include multiple vectors in a single point, with each vector having its own dimensions and metric requirements.

Distance metrics are used to measure the similarity between vectors. The choice of metric depends on how the vectors are obtained, especially the method used for training neural network encoders.

Qdrant supports the following popular types of metrics:

  • Dot product: Dot
  • Cosine similarity: Cosine
  • Euclidean distance: Euclid

To improve search efficiency, cosine similarity is achieved by taking the dot product of normalized vectors. Upon upload, vectors are automatically normalized. In addition to metrics and vector sizes, each collection also uses its own set of parameters to control collection optimization, index building, and cleaning. These settings can be changed at any time through the corresponding requests.

Setting Up Multi-Tenancy

How many collections should be created? In most cases, you only need to use a single collection with payload-based partitioning. This approach is known as multi-tenancy. For most users, this is efficient but requires additional configuration. Learn how to set up multi-tenancy.

When should multiple collections be created? When you have a limited number of users and require isolation. This approach is more flexible but may be more expensive, as creating a large number of collections may lead to resource overhead. Additionally, you need to ensure that they do not interfere with each other in any way, including in terms of performance.

Creating a collection

PUT /collections/{collection_name}

{
    "vectors": {
      "size": 300,
      "distance": "Cosine"
    }
}

In addition to the required options, custom values can also be specified for the following collection options:

  • hnsw_config - For index details, refer to the index section.
  • wal_config - Configuration related to write-ahead logging. For more detailed information on WAL, please refer to it.
  • optimizers_config - For optimizer details, refer to the optimizer section.
  • shard_number - Defines how many shards the collection should have. For more information, please refer to the distributed deployment section.
  • on_disk_payload - Defines the location for storing payload data. If set to true, it will only store the payload on disk. This can be very useful in limiting RAM usage when dealing with large payloads.
  • quantization_config - For detailed information on quantization, refer to quantization.

The default parameters for optional collection parameters are defined in the configuration file.

For more information on collection and vector parameters, please see schema definition and configuration file.

Available from v1.2.0

Vectors are stored in RAM to achieve very fast access. The on_disk parameter can be set in the vector configuration. If set to true, all vectors will be stored on disk. This will enable the use of memory mapping, suitable for importing large amounts of data.

Creating a collection from another collection

Available from v1.0.0

A collection can be initialized from another existing collection.

This can be useful for quickly trying different configurations of the same dataset.

When setting the vector configuration in the new collection, make sure the vectors have the same size and distance function as in the original collection.

PUT /collections/{collection_name}

{
    "vectors": {
      "size": 300,
      "distance": "Cosine"
    },
    "init_from": {
       "collection": {from_collection_name}
    }
}

Multi-vector collection

Available from v0.10.0

Each record can have multiple vectors. This feature allows multiple vectors to be stored in a collection. To distinguish the vectors within a record, define their unique names when creating the collection. Each named vector under this schema has its own distance and size:

PUT /collections/{collection_name}

{
    "vectors": {
        "image": {
            "size": 4,
            "distance": "Dot"
        },
        "text": {
            "size": 8,
            "distance": "Cosine"
        }
    }
}

For a few special cases, a collection without any vector storage can also be created.

Available from v1.1.1

For each named vector, you can optionally specify hnsw_config or quantization_config to deviate from the collection configuration. This can be used to optimize search performance at the vector level.

Available from v1.2.0

Vectors are stored in memory for quick access. For each vector, on_disk can be set to true to always store all vectors on disk. This will enable memory mapping, suitable for ingesting large amounts of data.

Deleting a collection

DELETE /collections/{collection_name}

Update collection parameters

Dynamic parameter updates can be helpful, such as more efficient initial loading of vectors. For example, you can disable indexing during upload and enable it immediately after upload completion. This way, you will not waste extra computational resources to rebuild the index.

The following command enables indexing for a segment containing vectors larger than 10000 kB:

PATCH /collections/{collection_name}

{
    "optimizers_config": {
        "indexing_threshold": 10000
    }
}

The following parameters can be updated:

  • optimizers_config - See detailed descriptions of optimizers.
  • hnsw_config - See detailed descriptions of the index.
  • quantization_config - See detailed descriptions of quantization.
  • vectors - Configuration for specific vectors, including their respective hnsw_config, quantization_config, and on_disk settings.
  • params - Other collection parameters, including write_consistency_factor and on_disk_payload.

The complete API specification is located in the schema definitions.

Available since v1.4.0

Qdrant 1.4 adds support for updating more collection parameters at runtime. HNSW index, quantization, and disk configuration can now be changed without recreating the collection. Segments (with index and quantization data) will automatically rebuild in the background to match the updated parameters.

In the following example, the entire collection and the HNSW index and quantization parameters for my_vector are updated:

PATCH /collections/{collection_name}

{
    "vectors": {
        "my_vector": {
            "hnsw_config": {
                "m": 32,
                "ef_construct": 123
            },
            "quantization_config": {
                "product": {
                    "compression": "x32",
                    "always_ram": true
                }
            },
            "on_disk": true
        }
    },
    "hnsw_config": {
        "ef_construct": 123
    },
    "quantization_config": {
        "scalar": {
            "type": "int8",
            "quantile": 0.8,
            "always_ram": false
        }
    }
}

Note: To update vector parameters in a collection without named vectors, an empty ("") name can be used.

Calls to this endpoint may block as it waits for existing optimizers to complete. We do not recommend using this feature in a production database as rebuilding the index may introduce significant overhead.

Collection Information

Qdrant allows for the determination of configuration parameters for an existing collection to better understand the distribution of points and the indexing situation.

GET /collections/{collection_name}

{
    "result": {
        "status": "green",
        "optimizer_status": "ok",
        "vectors_count": 1068786,
        "indexed_vectors_count": 1024232,
        "points_count": 1068786,
        "segments_count": 31,
        "config": {
            "params": {
                "vectors": {
                    "size": 384,
                    "distance": "Cosine"
                },
                "shard_number": 1,
                "replication_factor": 1,
                "write_consistency_factor": 1,
                "on_disk_payload": false
            },
            "hnsw_config": {
                "m": 16,
                "ef_construct": 100,
                "full_scan_threshold": 10000,
                "max_indexing_threads": 0
            },
            "optimizer_config": {
                "deleted_threshold": 0.2,
                "vacuum_min_vector_number": 1000,
                "default_segment_number": 0,
                "max_segment_size": null,
                "memmap_threshold": null,
                "indexing_threshold": 20000,
                "flush_interval_sec": 5,
                "max_optimization_threads": 1
            },
            "wal_config": {
                "wal_capacity_mb": 32,
                "wal_segments_ahead": 0
            }
        },
        "payload_schema": {}
    },
    "status": "ok",
    "time": 0.00010143
}

If vectors are inserted into the collection, the status field may change to "yellow" during optimization and will turn to "green" once all points have been processed successfully.

The following color states may be encountered:

  • ? green: Collection is ready
  • ? yellow: Collection is undergoing optimization
  • ? red: Engine has encountered an unrecoverable error

Some other properties of interest include:

  • points_count - Total number of objects (vectors and their payloads) stored in the collection
  • vectors_count - Total number of vectors in the collection. If each object has multiple vectors, this may not equal points_count.
  • indexed_vectors_count - Total number of vectors stored in the HNSW index. Qdrant does not store all vectors in the index, only those that may create index segments based on the given configuration.

Indexing Vectors in HNSW

In some cases, you may find that the value of indexed_vectors_count is less than vectors_count. This is an intentional behavior depending on the optimizer's configuration. A new index segment will be built if the size of non-indexed vectors exceeds the indexing_threshold (in KB). If your collection is very small or the vector dimension is low, HNSW segments may not be created, and indexed_vectors_count may equal 0.

You can reduce the value of indexing_threshold in the collection parameters to create a new index segment in the existing collection.

Collection Aliases

In a production environment, there may be a need to seamlessly switch between different versions of vectors, such as upgrading to a new version of neural networks.

In these cases, it may not be feasible to stop the service and rebuild the collection using the new vectors. An alias is an additional name for an existing collection. Querying can be fully executed using the alias instead of the collection name.

Therefore, a second collection can be built in the background, and then the alias can be switched from the old collection to the new collection. As alias changes are atomic, concurrent requests are not affected during the switching process.

Create Alias

POST /collections/aliases

{
    "actions": [
        {
            "create_alias": {
                "alias_name": "production_collection",
                "collection_name": "example_collection"
            }
        }
    ]
}

Delete Alias

POST /collections/aliases

{
    "actions": [
        {
            "delete_alias": {
                "alias_name": "production_collection"
            }
        }
    ]
}

Switch Collection

Multiple alias operations can be executed atomically. For example, you can use the following command to switch the underlying collection:

POST /collections/aliases

{
    "actions": [
        {
            "delete_alias": {
                "alias_name": "production_collection"
            }
        },
        {
            "create_alias": {
                "alias_name": "production_collection",
                "collection_name": "new_collection"
            }
        }
    ]
}

List Collection Aliases

GET /collections/{collection_name}/aliases

List All Aliases

GET /aliases

List All Collections

GET /collections