4. 🔍 Implementing video search functionality
✨ With the Tenyks SDK, you can perform similarity search based on vector embeddings.
- 🔮 By default, Tenyks magically generates the embeddings for your dataset, but you can bring your own embeddings (see custom embeddings for more details).
- Learn more about embeddings
- Learn more about multimodal search
4.1 Search (Text to Image)
Text to Image search: You can search for images that match the provided text.
Example: Based on the video clip you see above, we'll look for "religious group" or "nuns"
search_result = dataset.search_images(
sort_by="vector_text(religious group)",
)
display_images(search_result, n_images_to_show=3)
4.2 Search (Image to Image)
Image to Image search: Given an image as an input, you can search for similar images to the provided input. 🔍🖼️
Example: Given an image of a railroad, let's search for similar images. 🚂🛤️
To accomplish this, let's first take advantage of text search:
- We'll query the video for the word "railroad" using a text-based search. 🔤🎥
- The first two results of the above query will be used as the input to retrieve similar images. 📄➡️🖼️
4.2.1 Using text search to find "railroads" images
search_result_railroad = dataset.search_images(
sort_by="vector_text(railroad)",
)
display_images(search_result_railroad[0:2], n_images_to_show=2)
4.2.2 Searching for similar "railroads" images
For Image to Image search, we are expected to provide the image key
of the inputs. Let's find them.
img_keys = []
for img in search_result_railroad[0:2]:
print(img.key)
img_keys.append(img.key)
img_to_img_search_result_railroad = dataset.search_images(
sort_by=f"vector_image({', '.join(img_keys)})",
)
display_images(img_to_img_search_result_railroad, n_images_to_show=12)
Voilà! 🎉✨
4.3 Search (Object Similarity)
Object search: Given an object (e.g., bounding box) as an input, you can search for similar objects to the provided input.
Our current example does not have neither annotation nor predictions for objects, thus in pur case we are unable to execute
search_result_train_car = dataset.search_images(
sort_by="vector_text(train car)",
)
display_images(search_result_train_car[0:2], n_images_to_show=2, draw_bboxes=False)
img_anns = []
for img in search_result_train_car[0:2]:
img_anns.append(img.annotations)
annotations_ids = []
for ann in img_anns[0:2]:
print(ann[0].id)
annotations_ids.append(ann[0].id)
""" Output
2e9e9ad32f5db586bca778b8d2c042223f3b7b8b27ce0b972a0ea029
1f413b62fad56776ad67321c0643771df8a9d6cf60a85bf415dfc85e
"""
object_search_result = dataset.search_images(
sort_by=f"vector_object({', '.join(annotations_ids)})",
)
display_images(object_search_result, n_images_to_show=3, draw_bboxes=True)
Updated about 1 month ago