Information Retrieval
1. Introduction For decades, computer science education and practice have followed certain established patterns when it comes to selecting data structures. We learn that linked lists offer $O(1)$ insertion and deletion, that skip lists provide $O(\log n)$ search with simpler implementation than balanced trees, and that hash tables
Information Retrieval
Abstract Range queries like value:[0 TO 1000000] present significant performance challenges in search engines. This paper provides a comprehensive mathematical framework for understanding and optimizing range query performance. We present rigorous models for tiered indexing, range decomposition, information-theoretic tier selection, block-skipping probability, and performance optimization in distributed environments. Each
Database
Abstract This essay extends the unified query algebra framework by incorporating two critical capabilities missing from the original formulation: general aggregation operations and hierarchical data structures. We demonstrate that while posting lists provide a powerful abstraction for many scenarios, they impose restrictions that prevent the framework from handling certain important
Database
Abstract This supplementary essay extends the theoretical framework of unified query algebras across heterogeneous data paradigms by incorporating join operations and multiple field handling. We present a rigorous mathematical formulation that generalizes the posting list abstraction to accommodate joined document tuples while preserving the algebraic properties of the original framework.
Database
Abstract This research essay presents a formal algebraic framework that unifies operations across transaction processing, text retrieval, and vector search paradigms within a single mathematical structure. By establishing posting lists as a universal abstraction with well-defined algebraic properties, we develop a comprehensive theoretical foundation that preserves the expressivity of each
Database
Introduction Modern applications increasingly demand database systems that seamlessly integrate traditional transaction processing with advanced search capabilities. This essay explores architectural innovations that enable efficient faceted search, hybrid vector-text querying with full boolean expressivity, and unified query optimization across heterogeneous paradigms. By examining both theoretical foundations and practical implementation strategies,
Thoughts
1. Introduction In the domain of similarity search for high-dimensional vectors, approximate nearest neighbor (ANN) algorithms have become indispensable for applications ranging from recommendation systems to image retrieval. Modern vector databases commonly employ sophisticated indexing methods, with HNSW (Hierarchical Navigable Small World) combined with IVF (Inverted File) and PQ (Product
Thoughts
1. Introduction In our previous explorations of probabilistic frameworks for information retrieval, we examined how transformations like softmax and sigmoid convert raw similarity scores into probabilities, enabling principled fusion of heterogeneous retrieval systems. While these transformations provide elegant mathematical foundations for ranking, they introduce a critical challenge when handling conjunctive
Thoughts
1. Introduction The transformation of BM25 similarity scores into probability estimates represents a critical challenge in information retrieval systems. This process is essential for creating interpretable search results and enabling integration with probabilistic frameworks. While supervised learning approaches using query-document relevance pairs typically yield optimal results, practical implementations often face
Thoughts
Introduction In our previous exploration of probability transformations in vector search, we examined how softmax enables the normalization of disparate scoring systems into comparable probabilistic frameworks. This follow-up article delves deeper into the mathematical theory underpinning these transformations, with a specific focus on Bayesian probabilistic frameworks and their application to
Database
Introduction Modern search systems often combine vector search (dense embedding retrieval) with traditional text search (BM25/TF-IDF) to improve result relevance. This is known as hybrid search, which leverages both semantic similarity and exact keyword matching. In practice, hybrid approaches have shown significant improvements in search quality across various domains