Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement ST_norm_from_LUT for AdditiveQuantizer search #3882

Open
mdouze opened this issue Sep 24, 2024 · 1 comment
Open

Implement ST_norm_from_LUT for AdditiveQuantizer search #3882

mdouze opened this issue Sep 24, 2024 · 1 comment
Assignees

Comments

@mdouze
Copy link
Contributor

mdouze commented Sep 24, 2024

Summary

The additive quantizers need to carry around the norms of the vectors to compute L2 distances, or they can be computed.
The storage options work but the ST_norm_from_LUT is not implemented (it is described in eq (16) of the Faiss paper).
the symptom is a crash because the interpreter does not propagate the exception correctly.

import faiss
from faiss.contrib.datasets import SyntheticDataset

ds = SyntheticDataset(20, 1000, 100, 100)
index = faiss.index_factory(ds.d, "RQ2x4")
index.rq.search_type = faiss.AdditiveQuantizer.ST_norm_from_LUT
index.train(ds.get_train())
index.add(ds.get_database())
faiss.omp_set_num_threads(1) # otherwise crashes the interpreter
index.search(ds.get_queries(), 10)

result:

RuntimeError: Error in virtual void faiss::IndexAdditiveQuantizer::search(faiss::idx_t, const float*, faiss::idx_t, float*, faiss::idx_t*, const faiss::SearchParameters*) const at [/home/circleci/miniconda/conda-bld/faiss-pkg_1709244517602/work/faiss/IndexAdditiveQuantizer.cpp:284](http://localhost:8921/home/circleci/miniconda/conda-bld/faiss-pkg_1709244517602/work/faiss/IndexAdditiveQuantizer.cpp#line=283): search type 2 not supported
@mdouze
Copy link
Contributor Author

mdouze commented Oct 1, 2024

This is more a note for myself to implement it.

@mnorris11 mnorris11 removed their assignment Oct 1, 2024
@mdouze mdouze self-assigned this Oct 1, 2024
mdouze added a commit to mdouze/faiss that referenced this issue Oct 7, 2024
Summary:
The norm computation ST_norm_from_LUT was not implemented in Faiss. See issue

facebookresearch#3882

This diff adds an implementation for it. It is probably not very quick. A few precomputed tables for AdditiveQuantizer were moved form ResidualQuantizer.

Differential Revision: D63975689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants