Hybrid Image Search at Scale: SigLIP, SPLADE, YOLOv8 and Gemini Captions

Keyword matching on tags only goes so far with images. Teams building e-commerce catalogs, content management tools or digital asset libraries want search that grasps what a picture shows, its style and its emotional tone, and not only the words someone typed into a tag field. The design below describes a hybrid image search system that retrieves from a repository of around a million images using both visual content and text queries, built mostly from existing AI services so development effort stays small.

The architecture combines several specialized components. They're covered below in the order data flows through them: embeddings, image analysis, description generation, query handling and operations.

Two embedding models

The system rests on two separate embedding pipelines, each capturing a different side of the content.

Images use SigLIP. After evaluating CLIP and ImageBind as well, SigLIP won out. It's a contrastive image-text model trained with a sigmoid loss, and reported results put its zero-shot top-1 ImageNet accuracy around 76%, a couple of points above comparable CLIP models. It's also lighter on compute, roughly 40% less in the comparison cited here, and it generalizes to new domains without fine-tuning. Each image becomes a dense vector in a space where nearby points look alike.

Text uses SPLADE v2. SPLADE produces sparse vectors tied directly to a vocabulary, which brings three benefits:

• Sparse output stays interpretable, since every dimension maps to a real term.
• It expands queries lexically on its own, easing the vocabulary mismatch that plagues retrieval ("sofa" versus "couch").
• Because dimensions are terms, a match can be traced to the exact words that caused it, which supports explaining results.

Understanding each image

Raw pixels get turned into layers of queryable information in a multi-stage pipeline.

Object detection with YOLOv8. YOLO looks at the whole image in one network pass, so it's much faster than two-stage detectors such as Faster R-CNN. In this setup the large variant averages about 50 ms per image, enough for efficient batching. It reaches 53.9% mAP on COCO across 80 common classes, and the model spreads easily across a GPU cluster.

From generic classes to named entities. Detection yields labels like person, car and building. Rich search needs specific identities, so a web entity service converts them:

Example Transformation:

YOLO: "person" (coordinates: [120, 45, 220, 320])
Web Extraction: "Elon Musk" (confidence: 0.96)

YOLO: "building" (coordinates: [50, 100, 400, 600])
Web Extraction: "Eiffel Tower" (confidence: 0.92)

That service recognizes more than five million entities in groups such as public figures, landmarks, products and brands, and events. The step bridges visual detection and meaning. A query for a particular singer then works, instead of a generic one for a person.

Labels from a vision API. General concepts, activities and themes come from label detection, which in testing recognized more than 10,000 distinct concepts across the repository. It helps with three things: activities such as hiking or cooking, scenes such as beach or forest and objects too specific for YOLO's classes, like a cappuccino where YOLO only says cup.

Safe search. Content gets flagged along five dimensions: adult, violence, medical, spoof (memes and edited images) and racy. Every dimension carries a confidence score, so filtering can be graduated to fit organizational policy. Automating this layer replaced a large amount of manual review.

Gemini for metadata and descriptions

Gemini produces structured metadata. Careful prompt engineering led to a template that reliably extracts 25 attributes, each with a confidence between 0 and 1, for example sensitivity level and emotion:

System: You are an expert image analyzer. Extract the following attributes from the image, with a confidence score (0-1):

1. sensitivity (none, low, medium, high)
2. emotion (neutral, joy, sadness, surprise, etc.)
3. emotion_triggered (yes/no)
4. text_overlay (yes/no)
5. has_frames (yes/no)
6. has_religious_symbols (yes/no)
7. has_scattered_objects (yes/no)
8. style (photographic, illustrated, cartoon, abstract, etc.)
9. has_crowd (yes/no)
...
[full list of 25 attributes]

Response format: JSON object with attributes as keys and values as described above.

The combined outputs of detection, entities, attributes and metadata then feed a second Gemini prompt that writes a full description. Prompt design matters a lot here, because the descriptions have to be:

1. Rich in keywords, which helps text search.
2. Consistent in structure, so they parse reliably.
3. Meaningful in context, capturing why detected entities matter.
4. Aware of attributes such as emotion and style.
5. Written with likely search terms in mind.

The template looks like this:

Generate a comprehensive, search-optimized description for this image based on the following data:

[Entity data from object detection and recognition]
[Label data from Vision API]
[Structured metadata from previous Gemini analysis]

Your description should:
1. Begin with the most significant entities and their actions/relationships
2. Include key visual attributes (colors, style, composition)
3. Mention emotional tone and aesthetic qualities
4. Incorporate likely search terms
5. Be 3-5 sentences in length

Take raw detection data first:

Objects: person (0.98), guitar (0.95), microphone (0.92)
Entities: Taylor Swift (0.97)
Labels: concert, performance, stage, entertainment
Metadata: emotion=joy, has_crowd=yes, style=photographic, dominant_color=purple

After enrichment, it becomes a searchable sentence:

Taylor Swift performs energetically on stage with an acoustic guitar during a concert, singing into a microphone with passionate expression. The image captures the excitement of a live performance with purple stage lighting creating a vibrant atmosphere. This high-quality photograph conveys feelings of joy and excitement, with Swift's iconic performance style clearly visible. The composition includes partial views of an enthusiastic crowd in the foreground, making this suitable for music, entertainment, and celebrity content.

Descriptions of that kind carry plenty of searchable content, stay meaningful in context, include visual and emotional attributes and mirror what users are likely to type.

Answering a query

Queries pass through six steps:

1. Analysis. Intent classification decides whether the user wants specific entities, styles, emotions or other attributes.
2. Entity extraction. Named entities in the query are matched against the entity database so they get the right weight.
3. Embedding. The query is embedded with SPLADE v2 and SigLIP, matching how images were represented.
4. Vector search. The query embedding retrieves the nearest image embeddings from the index.
5. Policy filtering. Region-specific and organization-level rules apply: safety thresholds suited to the user's region, brand safety, age-appropriate filtering and licensing rights.
6. Reranking. Several factors adjust the first-pass order:

• Diversity, using a greedy algorithm that keeps a minimum cosine distance between chosen images.
• Recency, boosting newer items with a time decay: score_boost = base_score * (1 + alpha * exp(-beta * days_old)).
• Popularity, using logarithmic dampening so heavily used items don't dominate.
• Business rules for domain-specific adjustments.

Two hard problems

Latency. Chaining several services stacks delays, and first measurements exceeded four seconds per image end to end. Fixes included asynchronous parallel calls, batching for non-interactive pipelines, placing services close together to cut network time and tiering work by image complexity. Average time fell to 850 ms per image, with 95% finishing in under 1.2 seconds.

Integration complexity. Each service has its own API, error behavior and performance profile. What helped was a unified abstraction layer with consistent error handling, circuit breakers and fallbacks per service, health metrics for each dependency and staged degradation so the system keeps working through partial outages.

Results and next steps

Building on managed AI services paid off. Development took four months against an estimate above twelve for a custom build, search-result satisfaction rose 40% and manual tagging and categorization dropped by 80%.

Several directions remain open: personalization from user behavior, interactive result refinement, queries that mix text and images, feedback loops that learn from clicks and fine-tuning embedding models for particular industries.