Python Milvus Similarity Search

Performing Vector Similarity Search

This topic introduces how to use Milvus to search for entities.

In Milvus, vector similarity search calculates the distance between the query vector and the vectors in the collection (using a specified similarity measure) and returns the most similar results. You can execute mixed searching by specifying a boolean expression to filter scalar or primary key fields.

The following example demonstrates how to perform vector similarity search on a dataset containing 2000 rows of data, including book ID (primary key), word count (scalar field), and book synopsis (vector field), to simulate searching for specific books based on their vectorized descriptions. Milvus will return the most similar results based on the query vector and search parameters you define.

Loading the Collection

All search and query operations within Milvus are executed in memory. Before performing vector similarity search, load the collection into memory.

from pymilvus import Collection
collection = Collection("book")      # Get the existing collection.
collection.load()

Preparing the Search Parameters

Prepare the corresponding parameters for your search scenario. The following example defines the parameters that the search will use to compute the distance using Euclidean distance and retrieve vectors from the ten nearest clusters built by the IVF_FLAT index.

search_params = {
    "metric_type": "L2",
    "offset": 5,
    "ignore_growing": False,
    "params": {"nprobe": 10}
}

Performing Vector Search

Use Milvus for vector search. To search within a specific partition, specify a list of partition names.

Milvus supports setting a consistency level for searches. The example in this topic sets the consistency level to "Strong". You can also set the consistency level to "Bounded", "Session", or "Eventually". For more information on the four consistency levels in Milvus, see Consistency.

results = collection.search(
    data=[[0.1, 0.2]], 
    anns_field="book_intro", 
    param=search_params,
    limit=10,
    expr=None,
    output_fields=['title'],
    consistency_level="Strong"
)

results[0].ids

results[0].distances

hit = results[0][0]
hit.entity.get('title')

Check the primary key values and distances of the most similar vectors.

results[0].ids
results[0].distances

Once the search is complete, release the loaded collection in Milvus to reduce memory consumption.

collection.release()

Limitations

Performing Mixed Search

Mixed search is essentially a vector search with attribute filtering. By specifying a boolean expression used to filter scalar fields or primary key fields, searches can be limited under specific conditions.

The following example demonstrates how to perform mixed search based on the regular vector search. Let's say you want to search for certain books based on vectorized summaries, but you only want to search within a specific word count range. Then, you can specify a boolean expression in the search parameters to filter the word_count field. Milvus will only search for similar vectors within the entities that match the expression.

By specifying a boolean expression, you can filter scalar fields of entities during vector search. The following example limits the search scope to vectors within the specified word_count value range.

You can also use dynamic fields in the filtering expression and use output fields in the search request. For example, refer to the dynamic pattern.

search_param = {
  "data": [[0.1, 0.2]],
  "anns_field": "book_intro",
  "param": {"metric_type": "L2", "params": {"nprobe": 10}, "offset": 0},
  "limit": 10,
  "expr": "word_count <= 11000",
}
res = collection.search(**search_param)

Check the returned results.

assert len(res) == 1
hits = res[0]
assert len(hits) == 2
print(f"- Total hits: {len(hits)}, hits ids: {hits.ids} ")
print(f"- Top1 hit id: {hits[0].id}, distance: {hits[0].distance}, score: {hits[0].score} ")

Performing Range Search

Range search is a method of filtering search results based on the distance between the query vector and the vector field values. Different distance metrics can be used to measure the distance.

When performing range search, Milvus first conducts vector similarity search. Then, it executes vector filtering based on the specified distance conditions and returns the vectors whose distance falls within a specific range.

The following example demonstrates how to perform range search based on regular vector search.

Loading the Collection

All search and query operations in Milvus are executed in memory. Before performing vector similarity search, the collection needs to be loaded into memory.

from pymilvus import Collection
collection = Collection("book")      # Get an existing collection
collection.load()

Configuring Vector Filtering Ranges

In comparison to regular vector search, range search in Milvus is controlled by introducing two new parameters, radius and range_filter, to obtain the desired search results within a specific search range.

The radius specifies the minimum angle at which the vectors are considered similar. An optional range_filter can be utilized in conjunction with the radius to filter the vector field values based on the similarity to the query vector within a specific range. Both the radius and range_filter parameters have a data type of FLOAT. Setting these two parameters effectively balances the accuracy and efficiency of the search.

Typically, similarity is measured by the distance between the vector field value and the query vector. The selection of different distance metrics will have a significant impact on the configuration of radius and range_filter.

For instance, in the case of L2 distance, the search results must be filtered based on the vector field values with a distance less than radius. This is because in L2 distance, smaller distances indicate greater similarity. With this knowledge, if you want to filter out some of the most similar vectors, you can specify an effective range_filter value that is less than radius.

search_params = {
    "metric_type": "L2",
    "params": {
        "radius": 10.0,
        "range_filter" : 5.0
    }
}

In the case of IP distance, the scenario is different. With IP distance, a larger distance signifies greater similarity. Therefore, in contrast to L2 distance, the values of radius and range_filter in IP distance are opposite. In other words, when using range_filter to filter out some of the most similar vectors based on IP distance, the effective range_filter value must be greater than radius, and the distance of resulting vectors should be greater than radius but less than or equal to range_filter.

search_params = {
    "metric_type": "IP",
    "params": {
        "radius": 0.8,
        "range_filter" : 1.0
    }
}

Performing Range Search

By specifying radius and range_filter based on the distance measurement type, you can define the range of result vectors to be returned.

Perform range search on similarity within the range of 5.0 and 10.0 based on L2 distance:

search_param = {
  "data": [[0.1, 0.2]], # Query vector
  "anns_field": "book_intro", # Field to be searched
  "param": { "metric_type": "L2", "params": { "nprobe": 10, "radius": 10.0, "range_filter" : 5.0 }, "offset": 0 },
  "limit": 2,
  "output_fields": ["int64", "float"]  # Fields to be returned
}

res = collection.search(**search_param)

Perform range search on similarity within the range of 1.0 and 0.8 based on IP distance:

search_param = {
  "data": [[0.1, 0.2]], # Query vector
  "anns_field": "book_intro", # Field to be searched
  "param": {"metric_type": "IP", "params": { "nprobe": 10, "radius": 0.8, "range_filter" : 1.0 }, "offset": 0 },
  "limit": 2,
  "output_fields": ["int64", "float"]  # Fields to be returned
}

res = collection.search(**search_param)

Summary

In Milvus, range search can return vector results that are similar within the specified distance range. This functionality is enabled by specifying radius and range_filter in the search parameters. The table below summarizes the configuration of these two parameters for different distance measurement types.