A Hybrid CPU–GPU Execution Engine for Similarity Search with Cost-Based Operator Placement in E-Commerce Product Retrieval

Abstract

Modern e-commerce product retrieval increasingly relies on high-dimensional similarity search over learned embeddings, while still requiring structured filtering, personalization, and business constraints. These workloads combine compute-intensive vector operations with latency-sensitive control flow and memory-heavy metadata access, making end-to-end performance dependent on how operators are distributed across heterogeneous hardware. This paper presents a hybrid CPU--GPU execution engine for similarity search that performs cost-based operator placement across devices, targeting low tail latency and high throughput under realistic retrieval pipelines. The engine represents retrieval as a directed acyclic operator graph that includes approximate nearest neighbor candidate generation, exact distance refinement, attribute predicates, and ranking feature computations. A device-aware optimizer places each operator on CPU or GPU using a calibrated cost model that accounts for compute, memory bandwidth, kernel launch overheads, interconnect transfer, and pipeline backpressure. The execution runtime supports asynchronous scheduling with micro-batching, explicit data residency, and overlapping of transfers with compute, enabling mixed placement plans that avoid GPU underutilization on control-heavy operators while accelerating dense linear algebra and top-$k$ selection. The proposed approach is evaluated through an e-commerce retrieval setting with large catalogs and mixed query patterns, demonstrating that cost-based placement can adapt to hardware configurations and workload shifts without manual tuning. Observed outcomes include improved stability of tail latency and better resource balance compared with fixed-placement baselines.