Use more samples [skip ci]

CHANGELOG.md

@@ -1,3 +1,8 @@
+## 0.4.2 (unreleased)
+
+- Added notice when index created with little data
+- Fixed installation error with Postgres 12.0-12.2
+
 ## 0.4.1 (2023-03-21)
 
 - Improved performance of cosine distance

README.md

@@ -2,7 +2,11 @@
 
 Open-source vector similarity search for Postgres
 
-Supports exact and approximate nearest neighbor search for L2 distance, inner product, and cosine distance
+Supports
+
+- exact and approximate nearest neighbor search
+- L2 distance, inner product, and cosine distance
+- any [language](#languages) with a Postgres client
 
 [![Build Status](https://github.com/pgvector/pgvector/workflows/build/badge.svg?branch=master)](https://github.com/pgvector/pgvector/actions)
 
@@ -36,7 +40,7 @@ Create a vector column with 3 dimensions
 CREATE TABLE items (id bigserial PRIMARY KEY, embedding vector(3));
 ```
 
-Insert values
+Insert vectors
 
 ```sql
 INSERT INTO items (embedding) VALUES ('[1,2,3]'), ('[4,5,6]');
@@ -124,7 +128,7 @@ SELECT embedding <-> '[3,1,2]' AS distance FROM items;
 For inner product, multiply by -1 (since `<#>` returns the negative inner product)
 
 ```sql
-SELECT -1 * (embedding <#> '[3,1,2]') AS inner_product FROM items;
+SELECT (embedding <#> '[3,1,2]') * -1 AS inner_product FROM items;
 ```
 
 For cosine similarity, use 1 - cosine distance
@@ -133,7 +137,7 @@ For cosine similarity, use 1 - cosine distance
 SELECT 1 - (embedding <=> '[3,1,2]') AS cosine_similarity FROM items;
 ```
 
-#### Averaging
+#### Aggregates
 
 Average vectors
 
@@ -153,15 +157,11 @@ By default, pgvector performs exact nearest neighbor search, which provides perf
 
 You can add an index to use approximate nearest neighbor search, which trades some recall for performance. Unlike typical indexes, you will see different results for queries after adding an approximate index.
 
-Two keys to achieving good recall are:
+Three keys to achieving good recall are:
 
 1. Create the index *after* the table has some data
-2. Choose an appropriate number of lists (lower is better for recall, higher is better for speed)
-
-A good place to start is:
-
-- `rows / 1000` for up to 1M rows
-- `sqrt(rows)` for over 1M rows
+2. Choose an appropriate number of lists - a good place to start is `rows / 1000` for up to 1M rows and `sqrt(rows)` for over 1M rows
+3. When querying, specify an appropriate number of [probes](#query-options) (higher is better for recall, lower is better for speed) - a good place to start is `lists / 10` for up to 1M rows and `sqrt(lists)` for over 1M rows
 
 Add an index for each distance function you want to use.
 
@@ -190,7 +190,7 @@ Vectors with up to 2,000 dimensions can be indexed.
 Specify the number of probes (1 by default)
 
 ```sql
-SET ivfflat.probes = 1;
+SET ivfflat.probes = 10;
 ```
 
 A higher value provides better recall at the cost of speed, and it can be set to the number of lists for exact nearest neighbor search (at which point the planner won’t use the index)
@@ -199,7 +199,7 @@ Use `SET LOCAL` inside a transaction to set it for a single query
 
 ```sql
 BEGIN;
-SET LOCAL ivfflat.probes = 1;
+SET LOCAL ivfflat.probes = 10;
 SELECT ...
 COMMIT;
 ```
@@ -279,8 +279,10 @@ Language | Libraries / Examples
 --- | ---
 C++ | [pgvector-cpp](https://github.com/pgvector/pgvector-cpp)
 C# | [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet)
+Crystal | [pgvector-crystal](https://github.com/pgvector/pgvector-crystal)
 Elixir | [pgvector-elixir](https://github.com/pgvector/pgvector-elixir)
 Go | [pgvector-go](https://github.com/pgvector/pgvector-go)
+Haskell | [pgvector-haskell](https://github.com/pgvector/pgvector-haskell)
 Java, Scala | [pgvector-java](https://github.com/pgvector/pgvector-java)
 Julia | [pgvector-julia](https://github.com/pgvector/pgvector-julia)
 Lua | [pgvector-lua](https://github.com/pgvector/pgvector-lua)
@@ -291,6 +293,7 @@ Python | [pgvector-python](https://github.com/pgvector/pgvector-python)
 R | [pgvector-r](https://github.com/pgvector/pgvector-r)
 Ruby | [pgvector-ruby](https://github.com/pgvector/pgvector-ruby), [Neighbor](https://github.com/ankane/neighbor)
 Rust | [pgvector-rust](https://github.com/pgvector/pgvector-rust)
+Swift | [pgvector-swift](https://github.com/pgvector/pgvector-swift)
 
 ## Frequently Asked Questions
 
@@ -445,7 +448,7 @@ To request a new extension on other providers:
 - Google Cloud SQL - vote or comment on [this page](https://issuetracker.google.com/issues/265172065)
 - Azure Database - vote or comment on [this page](https://feedback.azure.com/d365community/idea/7b423322-6189-ed11-a81b-000d3ae49307)
 - DigitalOcean Managed Databases - vote or comment on [this page](https://ideas.digitalocean.com/app-framework-services/p/pgvector-extension-for-postgresql)
-- Render - vote or comment on [this page](https://feedback.render.com/features/p/add-pgvector-extension-to-postgresql)
+- Heroku Postgres - vote or comment on [this page](https://github.com/heroku/roadmap/issues/156)
 
 ## Upgrading
 

src/ivfbuild.c

@@ -419,7 +419,7 @@ ComputeCenters(IvfflatBuildState * buildstate)
 
 	/* Target 50 samples per list, with at least 10000 samples */
 	/* The number of samples has a large effect on index build time */
-	numSamples = buildstate->lists * 50;
+	numSamples = buildstate->lists * 200;
 	if (numSamples < 10000)
 		numSamples = 10000;
 
@@ -431,8 +431,18 @@ ComputeCenters(IvfflatBuildState * buildstate)
 	/* TODO Ensure within maintenance_work_mem */
 	buildstate->samples = VectorArrayInit(numSamples, buildstate->dimensions);
 	if (buildstate->heap != NULL)
+	{
 		SampleRows(buildstate);
 
+		if (buildstate->samples->length < buildstate->lists)
+		{
+			ereport(NOTICE,
+					(errmsg("ivfflat index created with little data"),
+					 errdetail("this will cause poor recall"),
+					 errhint("drop the index until the table has more data")));
+		}
+	}
+
 	/* Calculate centers */
 	IvfflatBench("k-means", IvfflatKmeans(buildstate->index, buildstate->samples, buildstate->centers));
 

src/ivfflat.c

@@ -13,7 +13,6 @@
 #endif
 
 int			ivfflat_probes;
-int			ivfflat_bound;
 static relopt_kind ivfflat_relopt_kind;
 
 /*
@@ -33,10 +32,6 @@ _PG_init(void)
 	DefineCustomIntVariable("ivfflat.probes", "Sets the number of probes",
 							"Valid range is 1..lists.", &ivfflat_probes,
 							1, 1, IVFFLAT_MAX_LISTS, PGC_USERSET, 0, NULL, NULL, NULL);
-
-	DefineCustomIntVariable("ivfflat.bound", "Sets the max results from index (experimental)",
-							NULL, &ivfflat_bound,
-							0, 0, INT_MAX, PGC_USERSET, 0, NULL, NULL, NULL);
 }
 
 /*

src/ivfflat.h

@@ -78,7 +78,6 @@
 
 /* Variables */
 extern int	ivfflat_probes;
-extern int	ivfflat_bound;
 
 /* Exported functions */
 PGDLLEXPORT void _PG_init(void);

src/ivfscan.c

@@ -126,10 +126,6 @@ GetScanItems(IndexScanDesc scan, Datum value)
 	 */
 	BufferAccessStrategy bas = GetAccessStrategy(BAS_BULKREAD);
 
-	/* Set the max number of results */
-	if (ivfflat_bound > 0)
-		tuplesort_set_bound(so->sortstate, ivfflat_bound);
-
 	/* Search closest probes lists */
 	while (!pairingheap_is_empty(so->listQueue))
 	{

src/vector.c

@@ -100,7 +100,7 @@ CheckStateArray(ArrayType *statearray, const char *caller)
 	return (float8 *) ARR_DATA_PTR(statearray);
 }
 
-#if PG_VERSION_NUM < 120000
+#if PG_VERSION_NUM < 120003
 static pg_noinline void
 float_overflow_error(void)
 {