6. 🎯 Object-level search

Object search: Given an object (e.g., bounding box) as an input, you can search for similar objects to the provided input.

Let's find police vehicles! 👮 🚔

search_result_police_vehicle = dataset.search_images(  
    sort_by="vector_text(police vehicle)",  
)

display_images(search_result_police_vehicle, n_images_to_show=9, draw_bboxes=False)

The bottom right image contains a police vehicle

search_result_police_vehicle[11]

""" Output
Image(dataset_key='nuImages', key='n008-2018-05-14-14-13-44-0400__CAM_BACK__1526322452537570_jpg.rf.88188c6f1a1766e1a9d3453c5e3be83b', filename='n008-2018-05-14-14-13-44-0400__CAM_BACK__1526322452537570_jpg.rf.88188c6f1a1766e1a9d3453c5e3be83b.jpg', type='.jpg', width=1600, height=900, raw_image_url=Url('https://tenyks-prod-storage.s3.amazonaws.com/tenyks/nuImages/images/n008-2018-05-14-14-13-44-0400__CAM_BACK__1526322452537570_jpg.rf.88188c6f1a1766e1a9d3453c5e3be83b.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAQ2MYPUCQBAJS26MP%2F20240929%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20240929T020021Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=5bf502c8e0d1464516c87c0f8a071f41a1001ddc5ca24e88a072cfa5045cc950'), tags=[], annotations=[Annotation(coordinates=[521.0, 375.0, 448.0, 380.0], category=Category(name='vehicle.car', color='#6455B8', id=16), id='2e4478a94f32c4e53edf5f60ee8c347321fd8e4eea5fc60b3aeab959', segmentation=[], tags=[]), Annotation(coordinates=[906.0, 391.0, 154.0, 95.0], category=Category(name='vehicle.car', color='#6455B8', id=16), id='99f0e25ab9dea9f7e4f49afa27ccc36f786c8369e96a8f8980ba28b4', segmentation=[], tags=[]), Annotation(coordinates=[391.0, 381.0, 46.0, 107.0], category=Category(name='human.pedestrian.adult', color='#FF7F0E', id=1), id='01bf09884acf83a65264bbe3d0cc2e23abaf6dc69162c2048336be4b', segmentation=[], tags=[]), Annotation(coordinates=[942.0, 392.0, 483.0, 276.0], category=Category(name='vehicle.emergency.police', color='#24EBB9', id=19), id='3fb90fb42d4b2cca4f612a94bb11fb5f4e6ebd011919ff084f7bd3b8', segmentation=[], tags=[])], predictions=[])
"""

We obtain the ids and the categories of the annotations

for ann in search_result_police_vehicle[11].annotations:  
  print(ann.id, ann.category.name)

""" Output
2e4478a94f32c4e53edf5f60ee8c347321fd8e4eea5fc60b3aeab959 vehicle.car
99f0e25ab9dea9f7e4f49afa27ccc36f786c8369e96a8f8980ba28b4 vehicle.car
01bf09884acf83a65264bbe3d0cc2e23abaf6dc69162c2048336be4b human.pedestrian.adult
3fb90fb42d4b2cca4f612a94bb11fb5f4e6ebd011919ff084f7bd3b8 vehicle.emergency.police
"""

We search for the specific object id with category "police"

object_search_result = dataset.search_images(  
    sort_by=f"vector_object(3fb90fb42d4b2cca4f612a94bb11fb5f4e6ebd011919ff084f7bd3b8)",  
)

display_images(object_search_result, n_images_to_show=3, draw_bboxes=True)

The top result returns an image with a police vehicle!