Fixed logic [skip ci]

Increase ef_search if needed to improve recall
Test result count [skip ci]
2026-07-22 12:07:34 +08:00 · 2024-09-23 13:52:50 -07:00 · 2023-11-18 15:44:38 -08:00 · 2023-11-18 13:58:05 -08:00 · 2023-11-10 14:03:13 -08:00 · 2023-11-10 13:28:48 -08:00
18 changed files with 988 additions and 1455 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,11 +1,6 @@
-## 0.5.2 (unreleased)
+## 0.6.0 (unreleased)

- Improved performance of HNSW
- Added support for on-disk parallel index builds for HNSW
- Reduced memory usage for HNSW index builds
- Reduced WAL generation for HNSW index builds
- Fixed error with logical replication
- Fixed `invalid memory alloc request size` error with HNSW index build
+- Added support for inline filtering with HNSW

 ## 0.5.1 (2023-10-10)

--- a/README.md
+++ b/README.md
@@ -10,7 +10,7 @@ Store your vectors with the rest of your data. Supports:

 Plus [ACID](https://en.wikipedia.org/wiki/ACID) compliance, point-in-time recovery, JOINs, and all of the other [great features](https://www.postgresql.org/about/) of Postgres

-[![Build Status](https://github.com/pgvector/pgvector/actions/workflows/build.yml/badge.svg)](https://github.com/pgvector/pgvector/actions)
+[![Build Status](https://github.com/pgvector/pgvector/workflows/build/badge.svg?branch=master)](https://github.com/pgvector/pgvector/actions)

 ## Installation

@@ -161,97 +161,8 @@ You can add an index to use approximate nearest neighbor search, which trades so

 Supported index types are:

- [HNSW](#hnsw) - added in 0.5.0
 - [IVFFlat](#ivfflat)
-
-## HNSW
-
-An HNSW index creates a multilayer graph. It has better query performance than IVFFlat (in terms of speed-recall tradeoff), but has slower build times and uses more memory. Also, an index can be created without any data in the table since there isn’t a training step like IVFFlat.
-
-Add an index for each distance function you want to use.
-
-L2 distance
-
-```sql
-CREATE INDEX ON items USING hnsw (embedding vector_l2_ops);
-```
-
-Inner product
-
-```sql
-CREATE INDEX ON items USING hnsw (embedding vector_ip_ops);
-```
-
-Cosine distance
-
-```sql
-CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);
-```
-
-Vectors with up to 2,000 dimensions can be indexed.
-
-### Index Options
-
-Specify HNSW parameters
-
- `m` - the max number of connections per layer (16 by default)
- `ef_construction` - the size of the dynamic candidate list for constructing the graph (64 by default)
-
-```sql
-CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WITH (m = 16, ef_construction = 64);
-```
-
-A higher value of `ef_construction` provides better recall at the cost of index build time / insert speed.
-
-### Query Options
-
-Specify the size of the dynamic candidate list for search (40 by default)
-
-```sql
-SET hnsw.ef_search = 100;
-```
-
-A higher value provides better recall at the cost of speed.
-
-Use `SET LOCAL` inside a transaction to set it for a single query
-
-```sql
-BEGIN;
-SET LOCAL hnsw.ef_search = 100;
-SELECT ...
-COMMIT;
-```
-
-### Index Build Time
-
-Indexes build significantly faster when the graph fits into `maintenance_work_mem`
-
-```sql
-SET maintenance_work_mem = '8GB';
-```
-
-A notice is shown when the graph no longer fits
-
-```text
-NOTICE:  hnsw graph no longer fits into maintenance_work_mem after 100000 tuples
-DETAIL:  Building will take significantly more time.
-HINT:  Increase maintenance_work_mem to speed up builds.
-```
-
-Note: Do not set `maintenance_work_mem` so high that it exhausts the memory on the server
-
-### Indexing Progress
-
-Check [indexing progress](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING) with Postgres 12+
-
-```sql
-SELECT phase, round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS "%" FROM pg_stat_progress_create_index;
-```
-
-The phases for HNSW are:
-
-1. `initializing`
-2. `loading tuples`
+- [HNSW](#hnsw) - added in 0.5.0

 ## IVFFlat

@@ -304,16 +215,6 @@ SELECT ...
 COMMIT;
 ```

-### Index Build Time
-
-Speed up index creation on large tables by increasing the number of parallel workers (2 by default)
-
-```sql
-SET max_parallel_maintenance_workers = 7; -- plus leader
-```
-
-For a large number of workers, you may also need to increase `max_parallel_workers` (8 by default)
-
 ### Indexing Progress

 Check [indexing progress](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING) with Postgres 12+
@@ -331,6 +232,75 @@ The phases for IVFFlat are:

 Note: `%` is only populated during the `loading tuples` phase

+## HNSW
+
+An HNSW index creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance (in terms of speed-recall tradeoff). There’s no training step like IVFFlat, so the index can be created without any data in the table.
+
+Add an index for each distance function you want to use.
+
+L2 distance
+
+```sql
+CREATE INDEX ON items USING hnsw (embedding vector_l2_ops);
+```
+
+Inner product
+
+```sql
+CREATE INDEX ON items USING hnsw (embedding vector_ip_ops);
+```
+
+Cosine distance
+
+```sql
+CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);
+```
+
+Vectors with up to 2,000 dimensions can be indexed.
+
+### Index Options
+
+Specify HNSW parameters
+
+- `m` - the max number of connections per layer (16 by default)
+- `ef_construction` - the size of the dynamic candidate list for constructing the graph (64 by default)
+
+```sql
+CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WITH (m = 16, ef_construction = 64);
+```
+
+### Query Options
+
+Specify the size of the dynamic candidate list for search (40 by default)
+
+```sql
+SET hnsw.ef_search = 100;
+```
+
+A higher value provides better recall at the cost of speed.
+
+Use `SET LOCAL` inside a transaction to set it for a single query
+
+```sql
+BEGIN;
+SET LOCAL hnsw.ef_search = 100;
+SELECT ...
+COMMIT;
+```
+
+### Indexing Progress
+
+Check [indexing progress](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING) with Postgres 12+
+
+```sql
+SELECT phase, round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS "%" FROM pg_stat_progress_create_index;
+```
+
+The phases for HNSW are:
+
+1. `initializing`
+2. `loading tuples`
+
 ## Filtering

 There are a few ways to index nearest neighbor queries with a `WHERE` clause
@@ -345,10 +315,17 @@ Create an index on one [or more](https://www.postgresql.org/docs/current/indexes
 CREATE INDEX ON items (category_id);
 ```

+Or a composite HNSW index for approximate search (added in 0.6.0)
+
+```sql
+CREATE INDEX ON items USING hnsw (embedding vector_l2_ops, category_id);
+```
+
 Or a [partial index](https://www.postgresql.org/docs/current/indexes-partial.html) on the vector column for approximate search

 ```sql
-CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WHERE (category_id = 123);
+CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100)
+    WHERE (category_id = 123);
 ```

 Use [partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html) for approximate search on many different values of the `WHERE` columns
@@ -406,7 +383,7 @@ Language | Libraries / Examples
 --- | ---
 C | [pgvector-c](https://github.com/pgvector/pgvector-c)
 C++ | [pgvector-cpp](https://github.com/pgvector/pgvector-cpp)
-C#, F#, Visual Basic | [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet)
+C# | [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet)
 Crystal | [pgvector-crystal](https://github.com/pgvector/pgvector-crystal)
 Dart | [pgvector-dart](https://github.com/pgvector/pgvector-dart)
 Elixir | [pgvector-elixir](https://github.com/pgvector/pgvector-elixir)
@@ -415,10 +392,8 @@ Haskell | [pgvector-haskell](https://github.com/pgvector/pgvector-haskell)
 Java, Kotlin, Groovy, Scala | [pgvector-java](https://github.com/pgvector/pgvector-java)
 JavaScript, TypeScript | [pgvector-node](https://github.com/pgvector/pgvector-node)
 Julia | [pgvector-julia](https://github.com/pgvector/pgvector-julia)
-Lisp | [pgvector-lisp](https://github.com/pgvector/pgvector-lisp)
 Lua | [pgvector-lua](https://github.com/pgvector/pgvector-lua)
 Nim | [pgvector-nim](https://github.com/pgvector/pgvector-nim)
-OCaml | [pgvector-ocaml](https://github.com/pgvector/pgvector-ocaml)
 Perl | [pgvector-perl](https://github.com/pgvector/pgvector-perl)
 PHP | [pgvector-php](https://github.com/pgvector/pgvector-php)
 Python | [pgvector-python](https://github.com/pgvector/pgvector-python)
@@ -491,14 +466,6 @@ and query with:
 SELECT * FROM items ORDER BY embedding::vector(3) <-> '[3,1,2]' LIMIT 5;
 ```

-#### Do indexes need to fit into memory?
-
-No, but like other index types, you’ll likely see better performance if they do. You can get the size of an index with:
-
-```sql
-SELECT pg_size_pretty(pg_relation_size('index_name'));
-```
-
 ## Troubleshooting

 #### Why isn’t a query using an index?
@@ -582,7 +549,7 @@ sum(vector) → vector | sum | 0.5.0
 If your machine has multiple Postgres installations, specify the path to [pg_config](https://www.postgresql.org/docs/current/app-pgconfig.html) with:

 ```sh
-export PG_CONFIG=/Library/PostgreSQL/16/bin/pg_config
+export PG_CONFIG=/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config
 ```

 Then re-run the installation instructions (run `make clean` before `make` if needed). If `sudo` is needed for `make install`, use:
@@ -591,14 +558,6 @@ Then re-run the installation instructions (run `make clean` before `make` if nee
 sudo --preserve-env=PG_CONFIG make install
 ```

-A few common paths on Mac are:
-
- EDB installer - `/Library/PostgreSQL/16/bin/pg_config`
- Homebrew (arm64) - `/opt/homebrew/opt/postgresql@16/bin/pg_config`
- Homebrew (x86-64) - `/usr/local/opt/postgresql@16/bin/pg_config`
-
-Note: Replace `16` with your Postgres server version
-
 ### Missing Header

 If compilation fails with `fatal error: postgres.h: No such file or directory`, make sure Postgres development files are installed on the server.
@@ -606,14 +565,10 @@ If compilation fails with `fatal error: postgres.h: No such file or directory`,
 For Ubuntu and Debian, use:

 ```sh
-sudo apt install postgresql-server-dev-16
+sudo apt install postgresql-server-dev-15
 ```

-Note: Replace `16` with your Postgres server version
-
-### Missing SDK
-
-If compilation fails and the output includes `warning: no such sysroot directory` on Mac, reinstall Xcode Command Line Tools.
+Note: Replace `15` with your Postgres server version

 ### Windows

@@ -628,7 +583,7 @@ Note: The exact path will vary depending on your Visual Studio version and editi
 Then use `nmake` to build:

 ```cmd
-set "PGROOT=C:\Program Files\PostgreSQL\16"
+set "PGROOT=C:\Program Files\PostgreSQL\15"
 git clone --branch v0.5.1 https://github.com/pgvector/pgvector.git
 cd pgvector
 nmake /F Makefile.win
@@ -678,22 +633,22 @@ pgxn install vector
 Debian and Ubuntu packages are available from the [PostgreSQL APT Repository](https://wiki.postgresql.org/wiki/Apt). Follow the [setup instructions](https://wiki.postgresql.org/wiki/Apt#Quickstart) and run:

 ```sh
-sudo apt install postgresql-16-pgvector
+sudo apt install postgresql-15-pgvector
 ```

-Note: Replace `16` with your Postgres server version
+Note: Replace `15` with your Postgres server version

 ### Yum

 RPM packages are available from the [PostgreSQL Yum Repository](https://yum.postgresql.org/). Follow the [setup instructions](https://www.postgresql.org/download/linux/redhat/) for your distribution and run:

 ```sh
-sudo yum install pgvector_16
+sudo yum install pgvector_15
 # or
-sudo dnf install pgvector_16
+sudo dnf install pgvector_15
 ```

-Note: Replace `16` with your Postgres server version
+Note: Replace `15` with your Postgres server version

 ### conda-forge

@@ -763,6 +718,7 @@ Thanks to:
 - [k-means++: The Advantage of Careful Seeding](https://theory.stanford.edu/~sergei/papers/kMeansPP-soda.pdf)
 - [Concept Decompositions for Large Sparse Text Data using Clustering](https://www.cs.utexas.edu/users/inderjit/public_papers/concept_mlj.pdf)
 - [Efficient and Robust Approximate Nearest Neighbor Search using Hierarchical Navigable Small World Graphs](https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf)
+- [HQANN: Efficient and Robust Similarity Search for Hybrid Queries with Structured and Unstructured Constraints](https://arxiv.org/pdf/2207.07940.pdf)

 ## History

@@ -803,25 +759,7 @@ make prove_installcheck PROVE_TESTS=test/t/001_wal.pl  # TAP test
 To enable benchmarking:

 ```sh
-make clean && PG_CFLAGS="-DIVFFLAT_BENCH" make && make install
-```
-
-To show memory usage:
-
-```sh
-make clean && PG_CFLAGS="-DHNSW_MEMORY -DIVFFLAT_MEMORY" make && make install
-```
-
-To enable assertions:
-
-```sh
-make clean && PG_CFLAGS="-DUSE_ASSERT_CHECKING" make && make install
-```
-
-To get k-means metrics:
-
-```sh
-make clean && PG_CFLAGS="-DIVFFLAT_KMEANS_DEBUG" make && make install
+make clean && PG_CFLAGS=-DIVFFLAT_BENCH make && make install
 ```

 Resources for contributors
--- a/sql/vector--0.5.1--0.6.0.sql
+++ b/sql/vector--0.5.1--0.6.0.sql
@@ -0,0 +1,10 @@
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION vector UPDATE TO '0.6.0'" to load this file. \quit
+
+CREATE FUNCTION hnsw_attribute_distance(integer, integer) RETURNS float8
+	AS 'MODULE_PATHNAME', 'hnsw_int4_attribute_distance' LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE;
+
+CREATE OPERATOR CLASS vector_integer_ops
+	DEFAULT FOR TYPE integer USING hnsw AS
+	OPERATOR 2 = (integer, integer),
+	FUNCTION 3 hnsw_attribute_distance(integer, integer);
--- a/sql/vector.sql
+++ b/sql/vector.sql
@@ -290,3 +290,13 @@ CREATE OPERATOR CLASS vector_cosine_ops
 	OPERATOR 1 <=> (vector, vector) FOR ORDER BY float_ops,
 	FUNCTION 1 vector_negative_inner_product(vector, vector),
 	FUNCTION 2 vector_norm(vector);
+
+-- hnsw attributes
+
+CREATE FUNCTION hnsw_attribute_distance(integer, integer) RETURNS float8
+	AS 'MODULE_PATHNAME', 'hnsw_int4_attribute_distance' LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE;
+
+CREATE OPERATOR CLASS vector_integer_ops
+	DEFAULT FOR TYPE integer USING hnsw AS
+	OPERATOR 2 = (integer, integer),
+	FUNCTION 3 hnsw_attribute_distance(integer, integer);
--- a/src/hnsw.c
+++ b/src/hnsw.c
@@ -14,7 +14,6 @@
 #endif

 int			hnsw_ef_search;
-bool		hnsw_enable_parallel_build;
 static relopt_kind hnsw_relopt_kind;

 /*
@@ -40,11 +39,6 @@ HnswInit(void)
 	DefineCustomIntVariable("hnsw.ef_search", "Sets the size of the dynamic candidate list for search",
 							"Valid range is 1..1000.", &hnsw_ef_search,
 							HNSW_DEFAULT_EF_SEARCH, HNSW_MIN_EF_SEARCH, HNSW_MAX_EF_SEARCH, PGC_USERSET, 0, NULL, NULL, NULL);
-
-	/* Behind a variable for now since can be slower than building in memory */
-	DefineCustomBoolVariable("hnsw.enable_parallel_build", "Enables or disables building indexes in parallel",
-							 NULL, &hnsw_enable_parallel_build,
-							 false, PGC_USERSET, 0, NULL, NULL, NULL);
 }

 /*
@@ -94,34 +88,6 @@ hnswcostestimate(PlannerInfo *root, IndexPath *path, double loop_count,
 		return;
 	}

-	/*
-	 * Do not use index if large % of tuples will be filtered unless
-	 * enable_seqscan = off
-	 */
-	if (list_length(path->indexinfo->indrestrictinfo) > 0)
-	{
-		double		selectivity = 1;
-		ListCell   *lc;
-
-		foreach(lc, path->indexinfo->indrestrictinfo)
-		{
-			RestrictInfo *rinfo = lfirst(lc);
-
-			if (rinfo->norm_selec >= 0 && rinfo->norm_selec <= 1)
-				selectivity *= rinfo->norm_selec;
-		}
-
-		if (selectivity < 0.1)
-		{
-			*indexStartupCost = 1.0e10 - 1;
-			*indexTotalCost = 1.0e10 - 1;
-			*indexSelectivity = 0;
-			*indexCorrelation = 0;
-			*indexPages = 0;
-			return;
-		}
-	}
-
 	MemSet(&costs, 0, sizeof(costs));

 	index = index_open(path->indexinfo->indexoid, NoLock);
@@ -201,7 +167,7 @@ hnswhandler(PG_FUNCTION_ARGS)
 	IndexAmRoutine *amroutine = makeNode(IndexAmRoutine);

 	amroutine->amstrategies = 0;
-	amroutine->amsupport = 2;
+	amroutine->amsupport = 3;
 #if PG_VERSION_NUM >= 130000
 	amroutine->amoptsprocnum = 0;
 #endif
@@ -209,7 +175,7 @@ hnswhandler(PG_FUNCTION_ARGS)
 	amroutine->amcanorderbyop = true;
 	amroutine->amcanbackward = false;	/* can change direction mid-scan */
 	amroutine->amcanunique = false;
-	amroutine->amcanmulticol = false;
+	amroutine->amcanmulticol = true;
 	amroutine->amoptionalkey = true;
 	amroutine->amsearcharray = false;
 	amroutine->amsearchnulls = false;
@@ -256,3 +222,17 @@ hnswhandler(PG_FUNCTION_ARGS)

 	PG_RETURN_POINTER(amroutine);
 }
+
+/*
+ * Get the distance between two int4 attributes
+ */
+PGDLLEXPORT PG_FUNCTION_INFO_V1(hnsw_int4_attribute_distance);
+Datum
+hnsw_int4_attribute_distance(PG_FUNCTION_ARGS)
+{
+	int32		a = PG_GETARG_INT32(0);
+	int32		b = PG_GETARG_INT32(1);
+	double		distance = ((double) a) - ((double) b);
+
+	PG_RETURN_FLOAT8(distance);
+}
--- a/src/hnsw.h
+++ b/src/hnsw.h
@@ -4,9 +4,7 @@
 #include "postgres.h"

 #include "access/generic_xlog.h"
-#include "access/parallel.h"
 #include "access/reloptions.h"
-#include "lib/ilist.h"
 #include "nodes/execnodes.h"
 #include "port.h"				/* for random() */
 #include "utils/sampling.h"
@@ -16,15 +14,12 @@
 #error "Requires PostgreSQL 11+"
 #endif

-#if PG_VERSION_NUM < 120000
-#include "access/relscan.h"
-#endif
-
 #define HNSW_MAX_DIM 2000

 /* Support functions */
 #define HNSW_DISTANCE_PROC 1
 #define HNSW_NORM_PROC 2
+#define HNSW_ATTRIBUTE_DISTANCE_PROC 3

 #define HNSW_VERSION	1
 #define HNSW_MAGIC_NUMBER 0xA953A953
@@ -73,10 +68,8 @@

 #if PG_VERSION_NUM >= 150000
 #define RandomDouble() pg_prng_double(&pg_global_prng_state)
-#define SeedRandom(seed) pg_prng_seed(&pg_global_prng_state, seed)
 #else
 #define RandomDouble() (((double) random()) / MAX_RANDOM_VALUE)
-#define SeedRandom(seed) srandom(seed)
 #endif

 #if PG_VERSION_NUM < 130000
@@ -94,28 +87,25 @@
 #define HnswGetMl(m) (1 / log(m))

 /* Ensure fits on page and in uint8 */
-#define HnswGetMaxLevel(m) Min(((BLCKSZ - MAXALIGN(SizeOfPageHeaderData) - MAXALIGN(sizeof(HnswPageOpaqueData)) - offsetof(HnswNeighborTupleData, indextids) - sizeof(ItemIdData)) / (sizeof(ItemPointerData)) / (m)) - 2, 255)
-
-#define HnswGetNeighbors(element, lc) (AssertMacro((element)->level >= (lc)), &(element)->neighbors[lc])
+#define HnswGetMaxLevel(m) Min(((BLCKSZ - MAXALIGN(SizeOfPageHeaderData) - MAXALIGN(sizeof(HnswPageOpaqueData)) - offsetof(HnswNeighborTupleData, indextids) - sizeof(ItemIdData)) / (sizeof(ItemPointerData)) / m) - 2, 255)

 /* Variables */
 extern int	hnsw_ef_search;
-extern bool hnsw_enable_parallel_build;
+
+typedef struct HnswNeighborArray HnswNeighborArray;

 typedef struct HnswElementData
 {
-	slist_node	next;
-	ItemPointerData heaptids[HNSW_HEAPTIDS];
-	uint8		heaptidsLength;
+	List	   *heaptids;
 	uint8		level;
 	uint8		deleted;
-	uint32		hash;
-	struct HnswNeighborArray *neighbors;
+	HnswNeighborArray *neighbors;
 	BlockNumber blkno;
 	OffsetNumber offno;
 	OffsetNumber neighborOffno;
 	BlockNumber neighborPage;
 	Datum		value;
+	IndexTuple	itup;
 }			HnswElementData;

 typedef HnswElementData * HnswElement;
@@ -124,6 +114,7 @@ typedef struct HnswCandidate
 {
 	HnswElement element;
 	float		distance;
+	bool		matches;
 	bool		closer;
 }			HnswCandidate;

@@ -148,59 +139,6 @@ typedef struct HnswOptions
 	int			efConstruction; /* size of dynamic candidate list */
 }			HnswOptions;

-typedef struct HnswGraph
-{
-	slist_head	elements;
-	HnswElement entryPoint;
-	long		memoryUsed;
-	long		memoryTotal;
-	bool		flushed;
-	double		indtuples;
-}			HnswGraph;
-
-typedef struct HnswSpool
-{
-	Relation	heap;
-	Relation	index;
-}			HnswSpool;
-
-typedef struct HnswShared
-{
-	/* Immutable state */
-	Oid			heaprelid;
-	Oid			indexrelid;
-	bool		isconcurrent;
-	int			scantuplesortstates;
-
-	/* Worker progress */
-	ConditionVariable workersdonecv;
-
-	/* Mutex for mutable state */
-	slock_t		mutex;
-
-	/* Mutable state */
-	int			nparticipantsdone;
-	double		reltuples;
-	HnswGraph	graphData;
-
-#if PG_VERSION_NUM < 120000
-	ParallelHeapScanDescData heapdesc;	/* must come last */
-#endif
-}			HnswShared;
-
-#if PG_VERSION_NUM >= 120000
-#define ParallelTableScanFromHnswShared(shared) \
-	(ParallelTableScanDesc) ((char *) (shared) + BUFFERALIGN(sizeof(HnswShared)))
-#endif
-
-typedef struct HnswLeader
-{
-	ParallelContext *pcxt;
-	int			nparticipanttuplesorts;
-	HnswShared *hnswshared;
-	Snapshot	snapshot;
-}			HnswLeader;
-
 typedef struct HnswBuildState
 {
 	/* Info */
@@ -219,24 +157,22 @@ typedef struct HnswBuildState
 	double		reltuples;

 	/* Support functions */
-	FmgrInfo   *procinfo;
+	FmgrInfo  **procinfos;
 	FmgrInfo   *normprocinfo;
-	Oid			collation;
+	Oid		   *collations;

 	/* Variables */
-	HnswGraph	graphData;
-	HnswGraph  *graph;
+	List	   *elements;
+	HnswElement entryPoint;
 	double		ml;
 	int			maxLevel;
+	long		memoryLeft;
+	bool		flushed;
+	bool		useIndexTuple;
 	Vector	   *normvec;

 	/* Memory */
-	MemoryContext graphCtx;
 	MemoryContext tmpCtx;
-
-	/* Parallel builds */
-	HnswLeader *hnswleader;
-	HnswShared *hnswshared;
 }			HnswBuildState;

 typedef struct HnswMetaPageData
@@ -294,9 +230,9 @@ typedef struct HnswScanOpaqueData
 	MemoryContext tmpCtx;

 	/* Support functions */
-	FmgrInfo   *procinfo;
+	FmgrInfo  **procinfos;
 	FmgrInfo   *normprocinfo;
-	Oid			collation;
+	Oid		   *collations;
 }			HnswScanOpaqueData;

 typedef HnswScanOpaqueData * HnswScanOpaque;
@@ -314,11 +250,11 @@ typedef struct HnswVacuumState
 	int			efConstruction;

 	/* Support functions */
-	FmgrInfo   *procinfo;
-	Oid			collation;
+	FmgrInfo  **procinfos;
+	Oid		   *collations;

 	/* Variables */
-	struct tidhash_hash *deleted;
+	HTAB	   *deleted;
 	BufferAccessStrategy bas;
 	HnswNeighborTuple ntup;
 	HnswElementData highestPoint;
@@ -332,28 +268,32 @@ int			HnswGetM(Relation index);
 int			HnswGetEfConstruction(Relation index);
 FmgrInfo   *HnswOptionalProcInfo(Relation index, uint16 procnum);
 bool		HnswNormValue(FmgrInfo *procinfo, Oid collation, Datum *value, Vector * result);
+void		HnswCommitBuffer(Buffer buf, GenericXLogState *state);
 Buffer		HnswNewBuffer(Relation index, ForkNumber forkNum);
 void		HnswInitPage(Buffer buf, Page page);
+void		HnswInitRegisterPage(Relation index, Buffer *buf, Page *page, GenericXLogState **state);
 void		HnswInit(void);
-List	   *HnswSearchLayer(Datum q, List *ep, int ef, int lc, Relation index, FmgrInfo *procinfo, Oid collation, int m, bool inserting, HnswElement skipElement);
+List	   *HnswSearchLayer(Datum q, IndexTuple qtup, ScanKeyData *keyData, List *ep, int ef, int lc, Relation index, FmgrInfo **procinfos, Oid *collations, int m, bool loadVec, HnswElement skipElement, bool inMemory);
 HnswElement HnswGetEntryPoint(Relation index);
 void		HnswGetMetaPageInfo(Relation index, int *m, HnswElement * entryPoint);
 HnswElement HnswInitElement(ItemPointer tid, int m, double ml, int maxLevel);
+void		HnswFreeElement(HnswElement element);
 HnswElement HnswInitElementFromBlock(BlockNumber blkno, OffsetNumber offno);
-void		HnswInsertElement(HnswElement element, HnswElement entryPoint, Relation index, FmgrInfo *procinfo, Oid collation, int m, int efConstruction, bool existing);
-HnswCandidate *HnswEntryCandidate(HnswElement em, Datum q, Relation rel, FmgrInfo *procinfo, Oid collation, bool loadVec);
-void		HnswUpdateMetaPage(Relation index, int updateEntry, HnswElement entryPoint, BlockNumber insertPage, ForkNumber forkNum, bool building);
+void		HnswInsertElement(HnswElement element, HnswElement entryPoint, Relation index, FmgrInfo **procinfos, Oid *collations, int m, int efConstruction, bool existing, bool inMemory);
+HnswElement HnswFindDuplicate(HnswElement e, Relation index);
+HnswCandidate *HnswEntryCandidate(HnswElement em, Datum q, IndexTuple qtup, ScanKeyData *keyData, Relation rel, FmgrInfo **procinfos, Oid *collations, bool loadVec, bool inMemory);
+void		HnswUpdateMetaPage(Relation index, int updateEntry, HnswElement entryPoint, BlockNumber insertPage, ForkNumber forkNum);
 void		HnswSetNeighborTuple(HnswNeighborTuple ntup, HnswElement e, int m);
 void		HnswAddHeapTid(HnswElement element, ItemPointer heaptid);
 void		HnswInitNeighbors(HnswElement element, int m);
-bool		HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_tid, Relation heapRel, bool building);
-void		HnswUpdateNeighborPages(Relation index, FmgrInfo *procinfo, Oid collation, HnswElement e, int m, bool checkExisting, bool building);
-void		HnswLoadElementFromTuple(HnswElement element, HnswElementTuple etup, bool loadHeaptids, bool loadVec);
-void		HnswLoadElement(HnswElement element, float *distance, Datum *q, Relation index, FmgrInfo *procinfo, Oid collation, bool loadVec);
-void		HnswSetElementTuple(HnswElementTuple etup, HnswElement element);
-void		HnswUpdateConnection(HnswElement element, HnswCandidate * hc, int m, int lc, int *updateIdx, Relation index, FmgrInfo *procinfo, Oid collation);
+bool		HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_tid, Relation heapRel);
+void		HnswUpdateNeighborPages(Relation index, FmgrInfo **procinfos, Oid *collations, HnswElement e, int m, bool checkExisting);
+void		HnswLoadElementFromTuple(HnswElement element, HnswElementTuple etup, bool loadHeaptids, bool loadVec, Relation index);
+void		HnswLoadElement(HnswElement element, float *distance, bool *matches, Datum *q, IndexTuple qtup, ScanKeyData *keyData, Relation index, FmgrInfo **procinfos, Oid *collations, bool loadVec);
+void		HnswSetElementTuple(HnswElementTuple etup, HnswElement element, bool useIndexTuple);
+void		HnswUpdateConnection(HnswElement element, HnswCandidate * hc, int m, int lc, int *updateIdx, Relation index, FmgrInfo **procinfos, Oid *collations, bool inMemory);
 void		HnswLoadNeighbors(HnswElement element, Relation index, int m);
-PGDLLEXPORT void HnswParallelBuildMain(dsm_segment *seg, shm_toc *toc);
+void		HnswElementSetData(HnswElement element, Relation index, Datum value, Datum *values, bool *isnull);

 /* Index access methods */
 IndexBuildResult *hnswbuild(Relation heap, Relation index, IndexInfo *indexInfo);
@@ -370,32 +310,6 @@ IndexScanDesc hnswbeginscan(Relation index, int nkeys, int norderbys);
 void		hnswrescan(IndexScanDesc scan, ScanKey keys, int nkeys, ScanKey orderbys, int norderbys);
 bool		hnswgettuple(IndexScanDesc scan, ScanDirection dir);
 void		hnswendscan(IndexScanDesc scan);
-
-/* Hash tables */
-typedef struct TidHashEntry
-{
-	ItemPointerData tid;
-	char		status;
-}			TidHashEntry;
-
-#define SH_PREFIX tidhash
-#define SH_ELEMENT_TYPE TidHashEntry
-#define SH_KEY_TYPE ItemPointerData
-#define SH_SCOPE extern
-#define SH_DECLARE
-#include "lib/simplehash.h"
-
-typedef struct PointerHashEntry
-{
-	uintptr_t	ptr;
-	char		status;
-}			PointerHashEntry;
-
-#define SH_PREFIX pointerhash
-#define SH_ELEMENT_TYPE PointerHashEntry
-#define SH_KEY_TYPE uintptr_t
-#define SH_SCOPE extern
-#define SH_DECLARE
-#include "lib/simplehash.h"
+FmgrInfo  **HnswInitProcinfos(Relation index);

 #endif
--- a/src/hnswbuild.c
+++ b/src/hnswbuild.c
--- a/src/hnswinsert.c
+++ b/src/hnswinsert.c
@@ -5,7 +5,6 @@
 #include "hnsw.h"
 #include "storage/bufmgr.h"
 #include "storage/lmgr.h"
-#include "utils/datum.h"
 #include "utils/memutils.h"

 /*
@@ -93,7 +92,7 @@ HnswFreeOffset(Relation index, Buffer buf, Page page, HnswElement element, Size
 * Add a new page
 */
 static void
-HnswInsertAppendPage(Relation index, Buffer *nbuf, Page *npage, GenericXLogState *state, Page page, bool building)
+HnswInsertAppendPage(Relation index, Buffer *nbuf, Page *npage, GenericXLogState *state, Page page)
 {
 	/* Add a new page */
 	LockRelationForExtension(index, ExclusiveLock);
@@ -101,11 +100,7 @@ HnswInsertAppendPage(Relation index, Buffer *nbuf, Page *npage, GenericXLogState
 	UnlockRelationForExtension(index, ExclusiveLock);

 	/* Init new page */
-	if (building)
-		*npage = BufferGetPage(*nbuf);
-	else
-		*npage = GenericXLogRegisterBuffer(state, *nbuf, GENERIC_XLOG_FULL_IMAGE);
-
+	*npage = GenericXLogRegisterBuffer(state, *nbuf, GENERIC_XLOG_FULL_IMAGE);
 	HnswInitPage(*nbuf, *npage);

 	/* Update previous buffer */
@@ -116,7 +111,7 @@ HnswInsertAppendPage(Relation index, Buffer *nbuf, Page *npage, GenericXLogState
 * Add to element and neighbor pages
 */
 static void
-WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPage, BlockNumber *updatedInsertPage, bool building)
+WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPage, BlockNumber *updatedInsertPage)
 {
 	Buffer		buf;
 	Page		page;
@@ -134,9 +129,10 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag
 	OffsetNumber freeOffno = InvalidOffsetNumber;
 	OffsetNumber freeNeighborOffno = InvalidOffsetNumber;
 	BlockNumber newInsertPage = InvalidBlockNumber;
+	bool		useIndexTuple = IndexRelationGetNumberOfAttributes(index) > 1;

 	/* Calculate sizes */
-	etupSize = HNSW_ELEMENT_TUPLE_SIZE(VARSIZE_ANY(DatumGetPointer(e->value)));
+	etupSize = HNSW_ELEMENT_TUPLE_SIZE(useIndexTuple ? IndexTupleSize(e->itup) : VARSIZE_ANY(DatumGetPointer(e->value)));
 	ntupSize = HNSW_NEIGHBOR_TUPLE_SIZE(e->level, m);
 	combinedSize = etupSize + ntupSize + sizeof(ItemIdData);
 	maxSize = HNSW_MAX_SIZE;
@@ -144,7 +140,7 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag

 	/* Prepare element tuple */
 	etup = palloc0(etupSize);
-	HnswSetElementTuple(etup, e);
+	HnswSetElementTuple(etup, e, useIndexTuple);

 	/* Prepare neighbor tuple */
 	ntup = palloc0(ntupSize);
@@ -156,16 +152,8 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag
 		buf = ReadBuffer(index, currentPage);
 		LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);

-		if (building)
-		{
-			state = NULL;
-			page = BufferGetPage(buf);
-		}
-		else
-		{
-			state = GenericXLogStart(index);
-			page = GenericXLogRegisterBuffer(state, buf, 0);
-		}
+		state = GenericXLogStart(index);
+		page = GenericXLogRegisterBuffer(state, buf, 0);

 		/* Keep track of first page where element at level 0 can fit */
 		if (!BlockNumberIsValid(newInsertPage) && PageGetFreeSpace(page) >= minCombinedSize)
@@ -185,12 +173,7 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag
 		if (HnswFreeOffset(index, buf, page, e, ntupSize, &nbuf, &npage, &freeOffno, &freeNeighborOffno, &newInsertPage))
 		{
 			if (nbuf != buf)
-			{
-				if (building)
-					npage = BufferGetPage(nbuf);
-				else
-					npage = GenericXLogRegisterBuffer(state, nbuf, 0);
-			}
+				npage = GenericXLogRegisterBuffer(state, nbuf, 0);

 			break;
 		}
@@ -199,7 +182,7 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag
 		/* Skip if both tuples can fit on the same page */
 		if (combinedSize > maxSize && PageGetFreeSpace(page) >= etupSize && !BlockNumberIsValid(HnswPageGetOpaque(page)->nextblkno))
 		{
-			HnswInsertAppendPage(index, &nbuf, &npage, state, page, building);
+			HnswInsertAppendPage(index, &nbuf, &npage, state, page);
 			break;
 		}

@@ -208,8 +191,7 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag
 		if (BlockNumberIsValid(currentPage))
 		{
 			/* Move to next page */
-			if (!building)
-				GenericXLogAbort(state);
+			GenericXLogAbort(state);
 			UnlockReleaseBuffer(buf);
 		}
 		else
@@ -217,33 +199,22 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag
 			Buffer		newbuf;
 			Page		newpage;

-			HnswInsertAppendPage(index, &newbuf, &newpage, state, page, building);
+			HnswInsertAppendPage(index, &newbuf, &newpage, state, page);

 			/* Commit */
-			if (building)
-				MarkBufferDirty(buf);
-			else
-				GenericXLogFinish(state);
+			GenericXLogFinish(state);

 			/* Unlock previous buffer */
 			UnlockReleaseBuffer(buf);

 			/* Prepare new buffer */
+			state = GenericXLogStart(index);
 			buf = newbuf;
-			if (building)
-			{
-				state = NULL;
-				page = BufferGetPage(buf);
-			}
-			else
-			{
-				state = GenericXLogStart(index);
-				page = GenericXLogRegisterBuffer(state, buf, 0);
-			}
+			page = GenericXLogRegisterBuffer(state, buf, 0);

 			/* Create new page for neighbors if needed */
 			if (PageGetFreeSpace(page) < combinedSize)
-				HnswInsertAppendPage(index, &nbuf, &npage, state, page, building);
+				HnswInsertAppendPage(index, &nbuf, &npage, state, page);
 			else
 			{
 				nbuf = buf;
@@ -297,14 +268,7 @@ WriteNewElementPages(Relation index, HnswElement e, int m, BlockNumber insertPag
 	}

 	/* Commit */
-	if (building)
-	{
-		MarkBufferDirty(buf);
-		if (nbuf != buf)
-			MarkBufferDirty(nbuf);
-	}
-	else
-		GenericXLogFinish(state);
+	GenericXLogFinish(state);
 	UnlockReleaseBuffer(buf);
 	if (nbuf != buf)
 		UnlockReleaseBuffer(nbuf);
@@ -338,12 +302,12 @@ ConnectionExists(HnswElement e, HnswNeighborTuple ntup, int startIdx, int lm)
 * Update neighbors
 */
 void
-HnswUpdateNeighborPages(Relation index, FmgrInfo *procinfo, Oid collation, HnswElement e, int m, bool checkExisting, bool building)
+HnswUpdateNeighborPages(Relation index, FmgrInfo **procinfos, Oid *collations, HnswElement e, int m, bool checkExisting)
 {
 	for (int lc = e->level; lc >= 0; lc--)
 	{
 		int			lm = HnswGetLayerM(m, lc);
-		HnswNeighborArray *neighbors = HnswGetNeighbors(e, lc);
+		HnswNeighborArray *neighbors = &e->neighbors[lc];

 		for (int i = 0; i < neighbors->length; i++)
 		{
@@ -370,7 +334,7 @@ HnswUpdateNeighborPages(Relation index, FmgrInfo *procinfo, Oid collation, HnswE
 			 */

 			/* Select neighbors */
-			HnswUpdateConnection(e, hc, lm, lc, &idx, index, procinfo, collation);
+			HnswUpdateConnection(e, hc, lm, lc, &idx, index, procinfos, collations, false);

 			/* New element was not selected as a neighbor */
 			if (idx == -1)
@@ -379,16 +343,8 @@ HnswUpdateNeighborPages(Relation index, FmgrInfo *procinfo, Oid collation, HnswE
 			/* Register page */
 			buf = ReadBuffer(index, hc->element->neighborPage);
 			LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
-			if (building)
-			{
-				state = NULL;
-				page = BufferGetPage(buf);
-			}
-			else
-			{
-				state = GenericXLogStart(index);
-				page = GenericXLogRegisterBuffer(state, buf, 0);
-			}
+			state = GenericXLogStart(index);
+			page = GenericXLogRegisterBuffer(state, buf, 0);

 			/* Get tuple */
 			itemid = PageGetItemId(page, offno);
@@ -430,12 +386,9 @@ HnswUpdateNeighborPages(Relation index, FmgrInfo *procinfo, Oid collation, HnswE
 					elog(ERROR, "failed to add index item to \"%s\"", RelationGetRelationName(index));

 				/* Commit */
-				if (building)
-					MarkBufferDirty(buf);
-				else
-					GenericXLogFinish(state);
+				GenericXLogFinish(state);
 			}
-			else if (!building)
+			else
 				GenericXLogAbort(state);

 			UnlockReleaseBuffer(buf);
@@ -447,7 +400,7 @@ HnswUpdateNeighborPages(Relation index, FmgrInfo *procinfo, Oid collation, HnswE
 * Add a heap TID to an existing element
 */
 static bool
-HnswAddDuplicate(Relation index, HnswElement element, HnswElement dup, bool building)
+HnswAddDuplicate(Relation index, HnswElement element, HnswElement dup)
 {
 	Buffer		buf;
 	Page		page;
@@ -460,16 +413,8 @@ HnswAddDuplicate(Relation index, HnswElement element, HnswElement dup, bool buil
 	/* Read page */
 	buf = ReadBuffer(index, dup->blkno);
 	LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
-	if (building)
-	{
-		state = NULL;
-		page = BufferGetPage(buf);
-	}
-	else
-	{
-		state = GenericXLogStart(index);
-		page = GenericXLogRegisterBuffer(state, buf, 0);
-	}
+	state = GenericXLogStart(index);
+	page = GenericXLogRegisterBuffer(state, buf, 0);

 	/* Find space */
 	itemid = PageGetItemId(page, dup->offno);
@@ -484,84 +429,60 @@ HnswAddDuplicate(Relation index, HnswElement element, HnswElement dup, bool buil
 	/* Either being deleted or we lost our chance to another backend */
 	if (i == 0 || i == HNSW_HEAPTIDS)
 	{
-		if (!building)
-			GenericXLogAbort(state);
+		GenericXLogAbort(state);
 		UnlockReleaseBuffer(buf);
 		return false;
 	}

 	/* Add heap TID */
-	etup->heaptids[i] = element->heaptids[0];
+	etup->heaptids[i] = *((ItemPointer) linitial(element->heaptids));

 	/* Overwrite tuple */
 	if (!PageIndexTupleOverwrite(page, dup->offno, (Item) etup, etupSize))
 		elog(ERROR, "failed to add index item to \"%s\"", RelationGetRelationName(index));

 	/* Commit */
-	if (building)
-		MarkBufferDirty(buf);
-	else
-		GenericXLogFinish(state);
+	GenericXLogFinish(state);
 	UnlockReleaseBuffer(buf);

 	return true;
 }

-/*
- * Find duplicate element
- */
-static bool
-HnswFindDuplicate(Relation index, HnswElement element, bool building)
-{
-	HnswNeighborArray *neighbors = HnswGetNeighbors(element, 0);
-
-	for (int i = 0; i < neighbors->length; i++)
-	{
-		HnswCandidate *neighbor = &neighbors->items[i];
-
-		/* Exit early since ordered by distance */
-		if (!datumIsEqual(element->value, neighbor->element->value, false, -1))
-			return false;
-
-		if (HnswAddDuplicate(index, element, neighbor->element, building))
-			return true;
-	}
-
-	return false;
-}
-
 /*
 * Write changes to disk
 */
 static void
-WriteElement(Relation index, FmgrInfo *procinfo, Oid collation, HnswElement element, int m, int efConstruction, HnswElement entryPoint, bool building)
+WriteElement(Relation index, FmgrInfo **procinfos, Oid *collations, HnswElement element, int m, int efConstruction, HnswElement dup, HnswElement entryPoint)
 {
 	BlockNumber newInsertPage = InvalidBlockNumber;

-	/* Look for duplicate */
-	if (HnswFindDuplicate(index, element, building))
-		return;
+	/* Try to add to existing page */
+	if (dup != NULL)
+	{
+		if (HnswAddDuplicate(index, element, dup))
+			return;
+	}

 	/* Write element and neighbor tuples */
-	WriteNewElementPages(index, element, m, GetInsertPage(index), &newInsertPage, building);
+	WriteNewElementPages(index, element, m, GetInsertPage(index), &newInsertPage);

 	/* Update insert page if needed */
 	if (BlockNumberIsValid(newInsertPage))
-		HnswUpdateMetaPage(index, 0, NULL, newInsertPage, MAIN_FORKNUM, building);
+		HnswUpdateMetaPage(index, 0, NULL, newInsertPage, MAIN_FORKNUM);

 	/* Update neighbors */
-	HnswUpdateNeighborPages(index, procinfo, collation, element, m, false, building);
+	HnswUpdateNeighborPages(index, procinfos, collations, element, m, false);

-	/* Update entry point if needed */
+	/* Update metapage if needed */
 	if (entryPoint == NULL || element->level > entryPoint->level)
-		HnswUpdateMetaPage(index, HNSW_UPDATE_ENTRY_GREATER, element, InvalidBlockNumber, MAIN_FORKNUM, building);
+		HnswUpdateMetaPage(index, HNSW_UPDATE_ENTRY_GREATER, element, InvalidBlockNumber, MAIN_FORKNUM);
 }

 /*
 * Insert a tuple into the index
 */
 bool
-HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_tid, Relation heapRel, bool building)
+HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_tid, Relation heapRel)
 {
 	Datum		value;
 	FmgrInfo   *normprocinfo;
@@ -569,8 +490,9 @@ HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_ti
 	HnswElement element;
 	int			m;
 	int			efConstruction = HnswGetEfConstruction(index);
-	FmgrInfo   *procinfo = index_getprocinfo(index, 1, HNSW_DISTANCE_PROC);
-	Oid			collation = index->rd_indcollation[0];
+	FmgrInfo  **procinfos = HnswInitProcinfos(index);
+	Oid		   *collations = index->rd_indcollation;
+	HnswElement dup;
 	LOCKMODE	lockmode = ShareLock;

 	/* Detoast once for all calls */
@@ -580,7 +502,7 @@ HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_ti
 	normprocinfo = HnswOptionalProcInfo(index, HNSW_NORM_PROC);
 	if (normprocinfo != NULL)
 	{
-		if (!HnswNormValue(normprocinfo, collation, &value, NULL))
+		if (!HnswNormValue(normprocinfo, collations[0], &value, NULL))
 			return false;
 	}

@@ -596,7 +518,7 @@ HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_ti

 	/* Create an element */
 	element = HnswInitElement(heap_tid, m, HnswGetMl(m), HnswGetMaxLevel(m));
-	element->value = value;
+	HnswElementSetData(element, index, value, values, isnull);

 	/* Prevent concurrent inserts when likely updating entry point */
 	if (entryPoint == NULL || element->level > entryPoint->level)
@@ -613,10 +535,13 @@ HnswInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer heap_ti
 	}

 	/* Insert element in graph */
-	HnswInsertElement(element, entryPoint, index, procinfo, collation, m, efConstruction, false);
+	HnswInsertElement(element, entryPoint, index, procinfos, collations, m, efConstruction, false, false);
+
+	/* Look for duplicate */
+	dup = HnswFindDuplicate(element, index);

 	/* Write to disk */
-	WriteElement(index, procinfo, collation, element, m, efConstruction, entryPoint, building);
+	WriteElement(index, procinfos, collations, element, m, efConstruction, dup, entryPoint);

 	/* Release lock */
 	UnlockPage(index, HNSW_UPDATE_LOCK, lockmode);
@@ -650,7 +575,7 @@ hnswinsert(Relation index, Datum *values, bool *isnull, ItemPointer heap_tid,
 	oldCtx = MemoryContextSwitchTo(insertCtx);

 	/* Insert tuple */
-	HnswInsertTuple(index, values, isnull, heap_tid, heap, false);
+	HnswInsertTuple(index, values, isnull, heap_tid, heap);

 	/* Delete memory context */
 	MemoryContextSwitchTo(oldCtx);
--- a/src/hnswscan.c
+++ b/src/hnswscan.c
@@ -15,12 +15,13 @@ GetScanItems(IndexScanDesc scan, Datum q)
 {
 	HnswScanOpaque so = (HnswScanOpaque) scan->opaque;
 	Relation	index = scan->indexRelation;
-	FmgrInfo   *procinfo = so->procinfo;
-	Oid			collation = so->collation;
+	FmgrInfo  **procinfos = so->procinfos;
+	Oid		   *collations = so->collations;
 	List	   *ep;
 	List	   *w;
 	int			m;
 	HnswElement entryPoint;
+	ScanKeyData *keyData = scan->keyData;

 	/* Get m and entry point */
 	HnswGetMetaPageInfo(index, &m, &entryPoint);
@@ -28,15 +29,15 @@ GetScanItems(IndexScanDesc scan, Datum q)
 	if (entryPoint == NULL)
 		return NIL;

-	ep = list_make1(HnswEntryCandidate(entryPoint, q, index, procinfo, collation, false));
+	ep = list_make1(HnswEntryCandidate(entryPoint, q, NULL, keyData, index, procinfos, collations, false, false));

 	for (int lc = entryPoint->level; lc >= 1; lc--)
 	{
-		w = HnswSearchLayer(q, ep, 1, lc, index, procinfo, collation, m, false, NULL);
+		w = HnswSearchLayer(q, NULL, keyData, ep, 1, lc, index, procinfos, collations, m, false, NULL, false);
 		ep = w;
 	}

-	return HnswSearchLayer(q, ep, hnsw_ef_search, 0, index, procinfo, collation, m, false, NULL);
+	return HnswSearchLayer(q, NULL, keyData, ep, hnsw_ef_search, 0, index, procinfos, collations, m, false, NULL, false);
 }

 /*
@@ -83,7 +84,7 @@ GetScanValue(IndexScanDesc scan)

 		/* Fine if normalization fails */
 		if (so->normprocinfo != NULL)
-			HnswNormValue(so->normprocinfo, so->collation, &value, NULL);
+			HnswNormValue(so->normprocinfo, so->collations[0], &value, NULL);
 	}

 	return value;
@@ -107,9 +108,9 @@ hnswbeginscan(Relation index, int nkeys, int norderbys)
 									   ALLOCSET_DEFAULT_SIZES);

 	/* Set support functions */
-	so->procinfo = index_getprocinfo(index, 1, HNSW_DISTANCE_PROC);
+	so->procinfos = HnswInitProcinfos(index);
 	so->normprocinfo = HnswOptionalProcInfo(index, HNSW_NORM_PROC);
-	so->collation = index->rd_indcollation[0];
+	so->collations = index->rd_indcollation;

 	scan->opaque = so;

@@ -188,13 +189,15 @@ hnswgettuple(IndexScanDesc scan, ScanDirection dir)
 		ItemPointer heaptid;

 		/* Move to next element if no valid heap TIDs */
-		if (hc->element->heaptidsLength == 0)
+		if (list_length(hc->element->heaptids) == 0)
 		{
 			so->w = list_delete_last(so->w);
 			continue;
 		}

-		heaptid = &hc->element->heaptids[--hc->element->heaptidsLength];
+		heaptid = llast(hc->element->heaptids);
+
+		hc->element->heaptids = list_delete_last(hc->element->heaptids);

 		MemoryContextSwitchTo(oldCtx);

@@ -204,6 +207,9 @@ hnswgettuple(IndexScanDesc scan, ScanDirection dir)
 		scan->xs_ctup.t_self = *heaptid;
 #endif

+		/* TODO Check during scan */
+		scan->xs_recheck = scan->numberOfKeys > 0;
+
 		scan->xs_recheckorderby = false;
 		return true;
 	}
@@ -220,6 +226,7 @@ hnswendscan(IndexScanDesc scan)
 {
 	HnswScanOpaque so = (HnswScanOpaque) scan->opaque;

+	pfree(so->procinfos);
 	MemoryContextDelete(so->tmpCtx);

 	pfree(so);
--- a/src/hnswutils.c
+++ b/src/hnswutils.c
@@ -7,89 +7,10 @@
 #include "utils/datum.h"
 #include "vector.h"

-#if PG_VERSION_NUM >= 130000
-#include "common/hashfn.h"
-#else
-#include "utils/hashutils.h"
+#if PG_VERSION_NUM < 130000
+#define TYPSTORAGE_PLAIN 'p'
 #endif

-#if PG_VERSION_NUM < 170000
-static inline uint64
-murmurhash64(uint64 data)
-{
-	uint64		h = data;
-
-	h ^= h >> 33;
-	h *= 0xff51afd7ed558ccd;
-	h ^= h >> 33;
-	h *= 0xc4ceb9fe1a85ec53;
-	h ^= h >> 33;
-
-	return h;
-}
-#endif
-
-/* TID hash table */
-static uint32
-hash_tid(ItemPointerData tid)
-{
-	union
-	{
-		uint64		i;
-		ItemPointerData tid;
-	}			x;
-
-	/* Initialize unused bytes */
-	x.i = 0;
-	x.tid = tid;
-
-	return murmurhash64(x.i);
-}
-
-#define SH_PREFIX		tidhash
-#define SH_ELEMENT_TYPE	TidHashEntry
-#define SH_KEY_TYPE		ItemPointerData
-#define	SH_KEY			tid
-#define SH_HASH_KEY(tb, key)	hash_tid(key)
-#define SH_EQUAL(tb, a, b)		ItemPointerEquals(&a, &b)
-#define	SH_SCOPE		extern
-#define SH_DEFINE
-#include "lib/simplehash.h"
-
-/* Needed to include simplehash.h twice */
-#if PG_VERSION_NUM < 120000
-#undef SH_EQUAL
-#define sh_log2 pointerhash_sh_log2
-#define sh_pow2 pointerhash_sh_pow2
-#endif
-
-/* Pointer hash table */
-static uint32
-hash_pointer(uintptr_t ptr)
-{
-#if SIZEOF_VOID_P == 8
-	return murmurhash64((uint64) ptr);
-#else
-	return murmurhash32((uint32) ptr);
-#endif
-}
-
-#define SH_PREFIX		pointerhash
-#define SH_ELEMENT_TYPE	PointerHashEntry
-#define SH_KEY_TYPE		uintptr_t
-#define	SH_KEY			ptr
-#define SH_HASH_KEY(tb, key)	hash_pointer(key)
-#define SH_EQUAL(tb, a, b)		(a == b)
-#define	SH_SCOPE		extern
-#define SH_DEFINE
-#include "lib/simplehash.h"
-
-typedef union
-{
-	pointerhash_hash *pointers;
-	tidhash_hash *tids;
-}			visited_hash;
-
 /*
 * Get the max number of connections in an upper layer for each element in the index
 */
@@ -130,6 +51,22 @@ HnswOptionalProcInfo(Relation index, uint16 procnum)
 	return index_getprocinfo(index, 1, procnum);
 }

+/*
+ * Init procs
+ */
+FmgrInfo  **
+HnswInitProcinfos(Relation index)
+{
+	int			keyAttributes = IndexRelationGetNumberOfKeyAttributes(index);
+	FmgrInfo  **procinfos = palloc(keyAttributes * sizeof(FmgrInfo *));
+
+	procinfos[0] = index_getprocinfo(index, 1, HNSW_DISTANCE_PROC);
+	for (int i = 1; i < keyAttributes; i++)
+		procinfos[i] = index_getprocinfo(index, i + 1, HNSW_ATTRIBUTE_DISTANCE_PROC);
+
+	return procinfos;
+}
+
 /*
 * Divide by the norm
 *
@@ -184,6 +121,27 @@ HnswInitPage(Buffer buf, Page page)
 	HnswPageGetOpaque(page)->page_id = HNSW_PAGE_ID;
 }

+/*
+ * Init and register page
+ */
+void
+HnswInitRegisterPage(Relation index, Buffer *buf, Page *page, GenericXLogState **state)
+{
+	*state = GenericXLogStart(index);
+	*page = GenericXLogRegisterBuffer(*state, *buf, GENERIC_XLOG_FULL_IMAGE);
+	HnswInitPage(*buf, *page);
+}
+
+/*
+ * Commit buffer
+ */
+void
+HnswCommitBuffer(Buffer buf, GenericXLogState *state)
+{
+	GenericXLogFinish(state);
+	UnlockReleaseBuffer(buf);
+}
+
 /*
 * Allocate neighbors
 */
@@ -206,6 +164,17 @@ HnswInitNeighbors(HnswElement element, int m)
 	}
 }

+/*
+ * Free neighbors
+ */
+static void
+HnswFreeNeighbors(HnswElement element)
+{
+	for (int lc = 0; lc <= element->level; lc++)
+		pfree(element->neighbors[lc].items);
+	pfree(element->neighbors);
+}
+
 /*
 * Allocate an element
 */
@@ -220,26 +189,41 @@ HnswInitElement(ItemPointer heaptid, int m, double ml, int maxLevel)
 	if (level > maxLevel)
 		level = maxLevel;

-	element->heaptidsLength = 0;
+	element->heaptids = NIL;
 	HnswAddHeapTid(element, heaptid);

 	element->level = level;
 	element->deleted = 0;
+	element->itup = NULL;

 	HnswInitNeighbors(element, m);

-	element->value = PointerGetDatum(NULL);
-
 	return element;
 }

+/*
+ * Free an element
+ */
+void
+HnswFreeElement(HnswElement element)
+{
+	HnswFreeNeighbors(element);
+	list_free_deep(element->heaptids);
+	if (element->itup)
+		pfree(element->itup);
+	pfree(element);
+}
+
 /*
 * Add a heap TID to an element
 */
 void
 HnswAddHeapTid(HnswElement element, ItemPointer heaptid)
 {
-	element->heaptids[element->heaptidsLength++] = *heaptid;
+	ItemPointer copy = palloc(sizeof(ItemPointerData));
+
+	ItemPointerCopy(heaptid, copy);
+	element->heaptids = lappend(element->heaptids, copy);
 }

 /*
@@ -254,6 +238,7 @@ HnswInitElementFromBlock(BlockNumber blkno, OffsetNumber offno)
 	element->offno = offno;
 	element->neighbors = NULL;
 	element->value = PointerGetDatum(NULL);
+	element->itup = NULL;
 	return element;
 }

@@ -331,7 +316,7 @@ HnswUpdateMetaPageInfo(Page page, int updateEntry, HnswElement entryPoint, Block
 * Update the metapage
 */
 void
-HnswUpdateMetaPage(Relation index, int updateEntry, HnswElement entryPoint, BlockNumber insertPage, ForkNumber forkNum, bool building)
+HnswUpdateMetaPage(Relation index, int updateEntry, HnswElement entryPoint, BlockNumber insertPage, ForkNumber forkNum)
 {
 	Buffer		buf;
 	Page		page;
@@ -339,43 +324,35 @@ HnswUpdateMetaPage(Relation index, int updateEntry, HnswElement entryPoint, Bloc

 	buf = ReadBufferExtended(index, forkNum, HNSW_METAPAGE_BLKNO, RBM_NORMAL, NULL);
 	LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
-	if (building)
-	{
-		state = NULL;
-		page = BufferGetPage(buf);
-	}
-	else
-	{
-		state = GenericXLogStart(index);
-		page = GenericXLogRegisterBuffer(state, buf, 0);
-	}
+	state = GenericXLogStart(index);
+	page = GenericXLogRegisterBuffer(state, buf, 0);

 	HnswUpdateMetaPageInfo(page, updateEntry, entryPoint, insertPage);

-	if (building)
-		MarkBufferDirty(buf);
-	else
-		GenericXLogFinish(state);
-	UnlockReleaseBuffer(buf);
+	HnswCommitBuffer(buf, state);
 }

 /*
 * Set element tuple, except for neighbor info
 */
 void
-HnswSetElementTuple(HnswElementTuple etup, HnswElement element)
+HnswSetElementTuple(HnswElementTuple etup, HnswElement element, bool useIndexTuple)
 {
 	etup->type = HNSW_ELEMENT_TUPLE_TYPE;
 	etup->level = element->level;
 	etup->deleted = 0;
 	for (int i = 0; i < HNSW_HEAPTIDS; i++)
 	{
-		if (i < element->heaptidsLength)
-			etup->heaptids[i] = element->heaptids[i];
+		if (i < list_length(element->heaptids))
+			etup->heaptids[i] = *((ItemPointer) list_nth(element->heaptids, i));
 		else
 			ItemPointerSetInvalid(&etup->heaptids[i]);
 	}
-	memcpy(&etup->data, DatumGetPointer(element->value), VARSIZE_ANY(DatumGetPointer(element->value)));
+
+	if (useIndexTuple)
+		memcpy(&etup->data, element->itup, IndexTupleSize(element->itup));
+	else
+		memcpy(&etup->data, DatumGetPointer(element->value), VARSIZE_ANY(DatumGetPointer(element->value)));
 }

 /*
@@ -390,7 +367,7 @@ HnswSetNeighborTuple(HnswNeighborTuple ntup, HnswElement e, int m)

 	for (int lc = e->level; lc >= 0; lc--)
 	{
-		HnswNeighborArray *neighbors = HnswGetNeighbors(e, lc);
+		HnswNeighborArray *neighbors = &e->neighbors[lc];
 		int			lm = HnswGetLayerM(m, lc);

 		for (int i = 0; i < lm; i++)
@@ -448,7 +425,7 @@ LoadNeighborsFromPage(HnswElement element, Relation index, Page page, int m)
 		if (level < 0)
 			level = 0;

-		neighbors = HnswGetNeighbors(element, level);
+		neighbors = &element->neighbors[level];
 		hc = &neighbors->items[neighbors->length++];
 		hc->element = e;
 	}
@@ -476,13 +453,13 @@ HnswLoadNeighbors(HnswElement element, Relation index, int m)
 * Load an element from a tuple
 */
 void
-HnswLoadElementFromTuple(HnswElement element, HnswElementTuple etup, bool loadHeaptids, bool loadVec)
+HnswLoadElementFromTuple(HnswElement element, HnswElementTuple etup, bool loadHeaptids, bool loadVec, Relation index)
 {
 	element->level = etup->level;
 	element->deleted = etup->deleted;
 	element->neighborPage = ItemPointerGetBlockNumber(&etup->neighbortid);
 	element->neighborOffno = ItemPointerGetOffsetNumber(&etup->neighbortid);
-	element->heaptidsLength = 0;
+	element->heaptids = NIL;

 	if (loadHeaptids)
 	{
@@ -497,14 +474,158 @@ HnswLoadElementFromTuple(HnswElement element, HnswElementTuple etup, bool loadHe
 	}

 	if (loadVec)
-		element->value = datumCopy(PointerGetDatum(&etup->data), false, -1);
+	{
+		if (IndexRelationGetNumberOfAttributes(index) > 1)
+		{
+			TupleDesc	tupdesc = RelationGetDescr(index);
+			bool		unused;
+
+			element->itup = CopyIndexTuple((IndexTuple) &etup->data);
+			element->value = index_getattr(element->itup, 1, tupdesc, &unused);
+		}
+		else
+		{
+			Vector	   *vec = palloc(VARSIZE_ANY(&etup->data));
+
+			memcpy(vec, &etup->data, VARSIZE_ANY(&etup->data));
+			element->value = PointerGetDatum(vec);
+		}
+	}
+}
+
+/*
+ * Get the tuple descriptor
+ */
+static TupleDesc
+HnswTupleDesc(Relation index)
+{
+	TupleDesc	tupdesc = CreateTupleDescCopyConstr(RelationGetDescr(index));
+
+	/* Prevent compression */
+	TupleDescAttr(tupdesc, 0)->attstorage = TYPSTORAGE_PLAIN;
+
+	return tupdesc;
+}
+
+/*
+ * Set element data
+ */
+void
+HnswElementSetData(HnswElement element, Relation index, Datum value, Datum *values, bool *isnull)
+{
+	/* TODO Create once per index build */
+	TupleDesc	tupdesc = HnswTupleDesc(index);
+	bool		unused;
+	Datum		tmp;
+
+	tmp = values[0];
+	values[0] = value;
+	element->itup = index_form_tuple(tupdesc, values, isnull);
+	values[0] = tmp;
+
+	element->value = index_getattr(element->itup, 1, tupdesc, &unused);
+
+	FreeTupleDesc(tupdesc);
+}
+
+/*
+ * Get the attribute distance
+ */
+static inline double
+AttributeDistance(double e)
+{
+	/* TODO Better bias */
+	/* must be >> max(w * g) + 1 / log10(2) */
+	double		bias = 4.32;
+
+	return e > 0 ? bias - 1.0 / log10(e + 1) : 0;
+}
+
+/*
+ * Get the distance
+ */
+static double
+GetDistance(IndexTuple itup, Datum vec, Datum q, IndexTuple qtup, ScanKeyData *keyData, Relation index, FmgrInfo **procinfos, Oid *collations, bool *matches)
+{
+	double		g = DatumGetFloat8(FunctionCall2Coll(procinfos[0], collations[0], q, vec));
+
+	Assert(PointerIsValid(matches));
+	*matches = true;
+
+	if (IndexRelationGetNumberOfKeyAttributes(index) > 1)
+	{
+		double		w = 0.25;
+		double		e = 0.0;
+		TupleDesc	tupdesc = RelationGetDescr(index);
+
+		if (keyData)
+		{
+			/* TODO need to pass length of key data */
+			int			keyCount = 1;
+
+			for (int i = 0; i < keyCount; i++)
+			{
+				ScanKey		key = &keyData[i];
+				bool		isnull;
+				Datum		value = index_getattr(itup, key->sk_attno, tupdesc, &isnull);
+				bool		attnull = key->sk_flags & SK_ISNULL;
+
+				if (isnull || attnull)
+				{
+					if (isnull != attnull)
+					{
+						e += 1000;
+						*matches = false;
+					}
+				}
+				else if (!DatumGetBool(FunctionCall2Coll(&key->sk_func, key->sk_collation, value, key->sk_argument)))
+				{
+					double		ei = fabs(DatumGetFloat8(FunctionCall2Coll(procinfos[key->sk_attno - 1], collations[key->sk_attno - 1], value, key->sk_argument)));
+
+					if (ei > 0)
+						e += ei;
+					else
+						/* Distance is zero for inequality */
+						e += 1000;
+
+					*matches = false;
+				}
+			}
+
+			return w * g + AttributeDistance(e);
+		}
+		else if (qtup)
+		{
+			int			keyCount = IndexRelationGetNumberOfKeyAttributes(index) - 1;
+
+			for (int i = 0; i < keyCount; i++)
+			{
+				bool		isnull;
+				bool		attnull;
+				Datum		value = index_getattr(itup, i + 2, tupdesc, &isnull);
+				Datum		value2 = index_getattr(qtup, i + 2, tupdesc, &attnull);
+
+				if (isnull || attnull)
+				{
+					if (isnull != attnull)
+						e += 1000;
+				}
+				else
+					e += fabs(DatumGetFloat8(FunctionCall2Coll(procinfos[i + 1], collations[i + 1], value, value2)));
+			}
+
+			return w * g + AttributeDistance(e);
+		}
+	}
+
+	return g;
 }

 /*
 * Load an element and optionally get its distance from q
 */
 void
-HnswLoadElement(HnswElement element, float *distance, Datum *q, Relation index, FmgrInfo *procinfo, Oid collation, bool loadVec)
+HnswLoadElement(HnswElement element, float *distance, bool *matches, Datum *q, IndexTuple qtup, ScanKeyData *keyData, Relation index, FmgrInfo **procinfos, Oid *collations, bool loadVec)
 {
 	Buffer		buf;
 	Page		page;
@@ -520,11 +641,27 @@ HnswLoadElement(HnswElement element, float *distance, Datum *q, Relation index,
 	Assert(HnswIsElementTuple(etup));

 	/* Load element */
-	HnswLoadElementFromTuple(element, etup, true, loadVec);
+	HnswLoadElementFromTuple(element, etup, true, loadVec, index);

 	/* Calculate distance */
 	if (distance != NULL)
-		*distance = (float) DatumGetFloat8(FunctionCall2Coll(procinfo, collation, *q, PointerGetDatum(&etup->data)));
+	{
+		IndexTuple	itup = NULL;
+		Datum		value;
+
+		if (IndexRelationGetNumberOfAttributes(index) > 1)
+		{
+			TupleDesc	tupdesc = RelationGetDescr(index);
+			bool		unused;
+
+			itup = (IndexTuple) &etup->data;
+			value = index_getattr(itup, 1, tupdesc, &unused);
+		}
+		else
+			value = PointerGetDatum(&etup->data);
+
+		*distance = GetDistance(itup, value, *q, qtup, keyData, index, procinfos, collations, matches);
+	}

 	UnlockReleaseBuffer(buf);
 }
@@ -533,24 +670,24 @@ HnswLoadElement(HnswElement element, float *distance, Datum *q, Relation index,
 * Get the distance for a candidate
 */
 static float
-GetCandidateDistance(HnswCandidate * hc, Datum q, FmgrInfo *procinfo, Oid collation)
+GetCandidateDistance(HnswCandidate * hc, Datum q, IndexTuple qtup, ScanKeyData *keyData, Relation index, FmgrInfo **procinfos, Oid *collations)
 {
-	return DatumGetFloat8(FunctionCall2Coll(procinfo, collation, q, hc->element->value));
+	return GetDistance(hc->element->itup, hc->element->value, q, qtup, keyData, index, procinfos, collations, &hc->matches);
 }

 /*
 * Create a candidate for the entry point
 */
 HnswCandidate *
-HnswEntryCandidate(HnswElement entryPoint, Datum q, Relation index, FmgrInfo *procinfo, Oid collation, bool loadVec)
+HnswEntryCandidate(HnswElement entryPoint, Datum q, IndexTuple qtup, ScanKeyData *keyData, Relation index, FmgrInfo **procinfos, Oid *collations, bool loadVec, bool inMemory)
 {
 	HnswCandidate *hc = palloc(sizeof(HnswCandidate));

 	hc->element = entryPoint;
-	if (index == NULL)
-		hc->distance = GetCandidateDistance(hc, q, procinfo, collation);
+	if (inMemory)
+		hc->distance = GetCandidateDistance(hc, q, qtup, keyData, index, procinfos, collations);
 	else
-		HnswLoadElement(hc->element, &hc->distance, &q, index, procinfo, collation, loadVec);
+		HnswLoadElement(hc->element, &hc->distance, &hc->matches, &q, qtup, keyData, index, procinfos, collations, loadVec);
 	return hc;
 }

@@ -600,22 +737,16 @@ CreatePairingHeapNode(HnswCandidate * c)
 * Add to visited
 */
 static inline void
-AddToVisited(visited_hash v, HnswCandidate * hc, Relation index, bool *found)
+AddToVisited(HTAB *v, HnswCandidate * hc, bool inMemory, bool *found)
 {
-	if (index == NULL)
-	{
-#if PG_VERSION_NUM >= 130000
-		pointerhash_insert_hash(v.pointers, (uintptr_t) hc->element, hc->element->hash, found);
-#else
-		pointerhash_insert(v.pointers, (uintptr_t) hc->element, found);
-#endif
-	}
+	if (inMemory)
+		hash_search(v, &hc->element, HASH_ENTER, found);
 	else
 	{
 		ItemPointerData indextid;

 		ItemPointerSet(&indextid, hc->element->blkno, hc->element->offno);
-		tidhash_insert(v.tids, indextid, found);
+		hash_search(v, &indextid, HASH_ENTER, found);
 	}
 }

@@ -623,38 +754,57 @@ AddToVisited(visited_hash v, HnswCandidate * hc, Relation index, bool *found)
 * Algorithm 2 from paper
 */
 List *
-HnswSearchLayer(Datum q, List *ep, int ef, int lc, Relation index, FmgrInfo *procinfo, Oid collation, int m, bool inserting, HnswElement skipElement)
+HnswSearchLayer(Datum q, IndexTuple qtup, ScanKeyData *keyData, List *ep, int ef, int lc, Relation index, FmgrInfo **procinfos, Oid *collations, int m, bool loadVec, HnswElement skipElement, bool inMemory)
 {
+	ListCell   *lc2;
+
 	List	   *w = NIL;
 	pairingheap *C = pairingheap_allocate(CompareNearestCandidates, NULL);
 	pairingheap *W = pairingheap_allocate(CompareFurthestCandidates, NULL);
 	int			wlen = 0;
-	visited_hash v;
-	ListCell   *lc2;
+	uint64		additional = 0;
+	uint64		maxAdditional = keyData ? 4 * ef : 0;
+	HASHCTL		hash_ctl;
+	HTAB	   *v;

 	/* Create hash table */
-	if (index == NULL)
-		v.pointers = pointerhash_create(CurrentMemoryContext, ef * m * 2, NULL);
+	if (inMemory)
+	{
+		hash_ctl.keysize = sizeof(HnswElement *);
+		hash_ctl.entrysize = sizeof(HnswElement *);
+	}
 	else
-		v.tids = tidhash_create(CurrentMemoryContext, ef * m * 2, NULL);
+	{
+		hash_ctl.keysize = sizeof(ItemPointerData);
+		hash_ctl.entrysize = sizeof(ItemPointerData);
+	}
+
+	hash_ctl.hcxt = CurrentMemoryContext;
+	v = hash_create("hnsw visited", 256, &hash_ctl, HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);

 	/* Add entry points to v, C, and W */
 	foreach(lc2, ep)
 	{
 		HnswCandidate *hc = (HnswCandidate *) lfirst(lc2);
-		bool		found;

-		AddToVisited(v, hc, index, &found);
+		AddToVisited(v, hc, inMemory, NULL);

 		pairingheap_add(C, &(CreatePairingHeapNode(hc)->ph_node));
 		pairingheap_add(W, &(CreatePairingHeapNode(hc)->ph_node));

+		/* Do not count elements that do not match filter towards ef */
+		if (!hc->matches)
+		{
+			if ((++additional) <= maxAdditional)
+				continue;
+		}
+
 		/*
 		 * Do not count elements being deleted towards ef when vacuuming. It
 		 * would be ideal to do this for inserts as well, but this could
 		 * affect insert performance.
 		 */
-		if (skipElement == NULL || hc->element->heaptidsLength != 0)
+		if (skipElement == NULL || list_length(hc->element->heaptids) != 0)
 			wlen++;
 	}

@@ -671,14 +821,14 @@ HnswSearchLayer(Datum q, List *ep, int ef, int lc, Relation index, FmgrInfo *pro
 			HnswLoadNeighbors(c->element, index, m);

 		/* Get the neighborhood at layer lc */
-		neighborhood = HnswGetNeighbors(c->element, lc);
+		neighborhood = &c->element->neighbors[lc];

 		for (int i = 0; i < neighborhood->length; i++)
 		{
 			HnswCandidate *e = &neighborhood->items[i];
 			bool		visited;

-			AddToVisited(v, e, index, &visited);
+			AddToVisited(v, e, inMemory, &visited);

 			if (!visited)
 			{
@@ -686,10 +836,10 @@ HnswSearchLayer(Datum q, List *ep, int ef, int lc, Relation index, FmgrInfo *pro

 				f = ((HnswPairingHeapNode *) pairingheap_first(W))->inner;

-				if (index == NULL)
-					eDistance = GetCandidateDistance(e, q, procinfo, collation);
+				if (inMemory)
+					eDistance = GetCandidateDistance(e, q, qtup, keyData, index, procinfos, collations);
 				else
-					HnswLoadElement(e->element, &eDistance, &q, index, procinfo, collation, inserting);
+					HnswLoadElement(e->element, &eDistance, &e->matches, &q, qtup, keyData, index, procinfos, collations, loadVec);

 				Assert(!e->element->deleted);

@@ -713,8 +863,18 @@ HnswSearchLayer(Datum q, List *ep, int ef, int lc, Relation index, FmgrInfo *pro
 					 * vacuuming. It would be ideal to do this for inserts as
 					 * well, but this could affect insert performance.
 					 */
-					if (skipElement == NULL || e->element->heaptidsLength != 0)
+					if (skipElement == NULL || list_length(e->element->heaptids) != 0)
 					{
+						/*
+						 * Do not count elements that do not match filter
+						 * towards ef
+						 */
+						if (!e->matches)
+						{
+							if ((++additional) <= maxAdditional)
+								continue;
+						}
+
 						wlen++;

 						/* No need to decrement wlen */
@@ -769,46 +929,48 @@ CompareCandidateDistances(const void *a, const void *b)
 * Calculate the distance between elements
 */
 static float
-HnswGetDistance(HnswElement a, HnswElement b, int lc, FmgrInfo *procinfo, Oid collation)
+HnswGetCachedDistance(HnswElement a, HnswElement b, int lc, Relation index, FmgrInfo **procinfos, Oid *collations)
 {
+	bool		matches;
+
 	/* Look for cached distance */
 	if (a->neighbors != NULL)
 	{
-		HnswNeighborArray *neighbors = HnswGetNeighbors(a, lc);
+		Assert(a->level >= lc);

-		for (int i = 0; i < neighbors->length; i++)
+		for (int i = 0; i < a->neighbors[lc].length; i++)
 		{
-			if (neighbors->items[i].element == b)
-				return neighbors->items[i].distance;
+			if (a->neighbors[lc].items[i].element == b)
+				return a->neighbors[lc].items[i].distance;
 		}
 	}

 	if (b->neighbors != NULL)
 	{
-		HnswNeighborArray *neighbors = HnswGetNeighbors(b, lc);
+		Assert(b->level >= lc);

-		for (int i = 0; i < neighbors->length; i++)
+		for (int i = 0; i < b->neighbors[lc].length; i++)
 		{
-			if (neighbors->items[i].element == a)
-				return neighbors->items[i].distance;
+			if (b->neighbors[lc].items[i].element == a)
+				return b->neighbors[lc].items[i].distance;
 		}
 	}

-	return DatumGetFloat8(FunctionCall2Coll(procinfo, collation, a->value, b->value));
+	return GetDistance(a->itup, a->value, b->value, b->itup, NULL, index, procinfos, collations, &matches);
 }

 /*
 * Check if an element is closer to q than any element from R
 */
 static bool
-CheckElementCloser(HnswCandidate * e, List *r, int lc, FmgrInfo *procinfo, Oid collation)
+CheckElementCloser(HnswCandidate * e, List *r, int lc, Relation index, FmgrInfo **procinfos, Oid *collations)
 {
 	ListCell   *lc2;

 	foreach(lc2, r)
 	{
 		HnswCandidate *ri = lfirst(lc2);
-		float		distance = HnswGetDistance(e->element, ri->element, lc, procinfo, collation);
+		float		distance = HnswGetCachedDistance(e->element, ri->element, lc, index, procinfos, collations);

 		if (distance <= e->distance)
 			return false;
@@ -821,13 +983,12 @@ CheckElementCloser(HnswCandidate * e, List *r, int lc, FmgrInfo *procinfo, Oid c
 * Algorithm 4 from paper
 */
 static List *
-SelectNeighbors(List *c, int m, int lc, FmgrInfo *procinfo, Oid collation, HnswElement e2, HnswCandidate * newCandidate, HnswCandidate * *pruned, bool sortCandidates)
+SelectNeighbors(List *c, int m, int lc, Relation index, FmgrInfo **procinfos, Oid *collations, HnswElement e2, HnswCandidate * newCandidate, HnswCandidate * *pruned, bool sortCandidates)
 {
 	List	   *r = NIL;
 	List	   *w = list_copy(c);
 	pairingheap *wd;
-	HnswNeighborArray *neighbors = HnswGetNeighbors(e2, lc);
-	bool		mustCalculate = !neighbors->closerSet;
+	bool		mustCalculate = !e2->neighbors[lc].closerSet;
 	List	   *added = NIL;
 	bool		removedAny = false;

@@ -849,7 +1010,7 @@ SelectNeighbors(List *c, int m, int lc, FmgrInfo *procinfo, Oid collation, HnswE

 		/* Use previous state of r and wd to skip work when possible */
 		if (mustCalculate)
-			e->closer = CheckElementCloser(e, r, lc, procinfo, collation);
+			e->closer = CheckElementCloser(e, r, lc, index, procinfos, collations);
 		else if (list_length(added) > 0)
 		{
 			/*
@@ -858,7 +1019,7 @@ SelectNeighbors(List *c, int m, int lc, FmgrInfo *procinfo, Oid collation, HnswE
 			 */
 			if (e->closer)
 			{
-				e->closer = CheckElementCloser(e, added, lc, procinfo, collation);
+				e->closer = CheckElementCloser(e, added, lc, index, procinfos, collations);

 				if (!e->closer)
 					removedAny = true;
@@ -871,7 +1032,7 @@ SelectNeighbors(List *c, int m, int lc, FmgrInfo *procinfo, Oid collation, HnswE
 				 */
 				if (removedAny)
 				{
-					e->closer = CheckElementCloser(e, r, lc, procinfo, collation);
+					e->closer = CheckElementCloser(e, r, lc, index, procinfos, collations);
 					if (e->closer)
 						added = lappend(added, e);
 				}
@@ -879,7 +1040,7 @@ SelectNeighbors(List *c, int m, int lc, FmgrInfo *procinfo, Oid collation, HnswE
 		}
 		else if (e == newCandidate)
 		{
-			e->closer = CheckElementCloser(e, r, lc, procinfo, collation);
+			e->closer = CheckElementCloser(e, r, lc, index, procinfos, collations);
 			if (e->closer)
 				added = lappend(added, e);
 		}
@@ -891,7 +1052,7 @@ SelectNeighbors(List *c, int m, int lc, FmgrInfo *procinfo, Oid collation, HnswE
 	}

 	/* Cached value can only be used in future if sorted deterministically */
-	neighbors->closerSet = sortCandidates;
+	e2->neighbors[lc].closerSet = sortCandidates;

 	/* Keep pruned connections */
 	while (!pairingheap_is_empty(wd) && list_length(r) < m)
@@ -909,6 +1070,34 @@ SelectNeighbors(List *c, int m, int lc, FmgrInfo *procinfo, Oid collation, HnswE
 	return r;
 }

+/*
+ * Find duplicate element
+ */
+HnswElement
+HnswFindDuplicate(HnswElement e, Relation index)
+{
+	HnswNeighborArray *neighbors = &e->neighbors[0];
+
+	/* TODO Implement */
+	if (IndexRelationGetNumberOfAttributes(index) > 1)
+		return NULL;
+
+	for (int i = 0; i < neighbors->length; i++)
+	{
+		HnswCandidate *neighbor = &neighbors->items[i];
+
+		/* Exit early since ordered by distance */
+		if (!datumIsEqual(e->value, neighbor->element->value, false, -1))
+			break;
+
+		/* Check for space */
+		if (list_length(neighbor->element->heaptids) < HNSW_HEAPTIDS)
+			return neighbor->element;
+	}
+
+	return NULL;
+}
+
 /*
 * Add connections
 */
@@ -916,7 +1105,7 @@ static void
 AddConnections(HnswElement element, List *neighbors, int m, int lc)
 {
 	ListCell   *lc2;
-	HnswNeighborArray *a = HnswGetNeighbors(element, lc);
+	HnswNeighborArray *a = &element->neighbors[lc];

 	foreach(lc2, neighbors)
 		a->items[a->length++] = *((HnswCandidate *) lfirst(lc2));
@@ -926,9 +1115,9 @@ AddConnections(HnswElement element, List *neighbors, int m, int lc)
 * Update connections
 */
 void
-HnswUpdateConnection(HnswElement element, HnswCandidate * hc, int m, int lc, int *updateIdx, Relation index, FmgrInfo *procinfo, Oid collation)
+HnswUpdateConnection(HnswElement element, HnswCandidate * hc, int m, int lc, int *updateIdx, Relation index, FmgrInfo **procinfos, Oid *collations, bool inMemory)
 {
-	HnswNeighborArray *currentNeighbors = HnswGetNeighbors(hc->element, lc);
+	HnswNeighborArray *currentNeighbors = &hc->element->neighbors[lc];

 	HnswCandidate hc2;

@@ -949,21 +1138,23 @@ HnswUpdateConnection(HnswElement element, HnswCandidate * hc, int m, int lc, int
 		HnswCandidate *pruned = NULL;

 		/* Load elements on insert */
-		if (index != NULL)
+		if (!inMemory)
 		{
 			Datum		q = hc->element->value;
+			IndexTuple	qtup = hc->element->itup;
+			ScanKeyData *keyData = NULL;

 			for (int i = 0; i < currentNeighbors->length; i++)
 			{
 				HnswCandidate *hc3 = &currentNeighbors->items[i];

 				if (DatumGetPointer(hc3->element->value) == NULL)
-					HnswLoadElement(hc3->element, &hc3->distance, &q, index, procinfo, collation, true);
+					HnswLoadElement(hc3->element, &hc3->distance, &hc3->matches, &q, qtup, keyData, index, procinfos, collations, true);
 				else
-					hc3->distance = GetCandidateDistance(hc3, q, procinfo, collation);
+					hc3->distance = GetCandidateDistance(hc3, q, qtup, keyData, index, procinfos, collations);

 				/* Prune element if being deleted */
-				if (hc3->element->heaptidsLength == 0)
+				if (list_length(hc3->element->heaptids) == 0)
 				{
 					pruned = &currentNeighbors->items[i];
 					break;
@@ -980,7 +1171,7 @@ HnswUpdateConnection(HnswElement element, HnswCandidate * hc, int m, int lc, int
 				c = lappend(c, &currentNeighbors->items[i]);
 			c = lappend(c, &hc2);

-			SelectNeighbors(c, m, lc, procinfo, collation, hc->element, &hc2, &pruned, true);
+			SelectNeighbors(c, m, lc, index, procinfos, collations, hc->element, &hc2, &pruned, true);

 			/* Should not happen */
 			if (pruned == NULL)
@@ -1021,7 +1212,7 @@ RemoveElements(List *w, HnswElement skipElement)
 		if (skipElement != NULL && hc->element->blkno == skipElement->blkno && hc->element->offno == skipElement->offno)
 			continue;

-		if (hc->element->heaptidsLength != 0)
+		if (list_length(hc->element->heaptids) != 0)
 			w2 = lappend(w2, hc);
 	}

@@ -1032,33 +1223,29 @@ RemoveElements(List *w, HnswElement skipElement)
 * Algorithm 1 from paper
 */
 void
-HnswInsertElement(HnswElement element, HnswElement entryPoint, Relation index, FmgrInfo *procinfo, Oid collation, int m, int efConstruction, bool existing)
+HnswInsertElement(HnswElement element, HnswElement entryPoint, Relation index, FmgrInfo **procinfos, Oid *collations, int m, int efConstruction, bool existing, bool inMemory)
 {
 	List	   *ep;
 	List	   *w;
 	int			level = element->level;
 	int			entryLevel;
 	Datum		q = element->value;
+	IndexTuple	qtup = element->itup;
+	ScanKeyData *keyData = NULL;
 	HnswElement skipElement = existing ? element : NULL;

-#if PG_VERSION_NUM >= 130000
-	/* Precompute hash */
-	if (index == NULL)
-		element->hash = hash_pointer((uintptr_t) element);
-#endif
-
 	/* No neighbors if no entry point */
 	if (entryPoint == NULL)
 		return;

 	/* Get entry point and level */
-	ep = list_make1(HnswEntryCandidate(entryPoint, q, index, procinfo, collation, true));
+	ep = list_make1(HnswEntryCandidate(entryPoint, q, qtup, keyData, index, procinfos, collations, true, inMemory));
 	entryLevel = entryPoint->level;

 	/* 1st phase: greedy search to insert level */
 	for (int lc = entryLevel; lc >= level + 1; lc--)
 	{
-		w = HnswSearchLayer(q, ep, 1, lc, index, procinfo, collation, m, true, skipElement);
+		w = HnswSearchLayer(q, qtup, keyData, ep, 1, lc, index, procinfos, collations, m, true, skipElement, inMemory);
 		ep = w;
 	}

@@ -1076,11 +1263,11 @@ HnswInsertElement(HnswElement element, HnswElement entryPoint, Relation index, F
 		List	   *neighbors;
 		List	   *lw;

-		w = HnswSearchLayer(q, ep, efConstruction, lc, index, procinfo, collation, m, true, skipElement);
+		w = HnswSearchLayer(q, qtup, keyData, ep, efConstruction, lc, index, procinfos, collations, m, true, skipElement, inMemory);

 		/* Elements being deleted or skipped can help with search */
 		/* but should be removed before selecting neighbors */
-		if (index != NULL)
+		if (!inMemory)
 			lw = RemoveElements(w, skipElement);
 		else
 			lw = w;
@@ -1090,7 +1277,7 @@ HnswInsertElement(HnswElement element, HnswElement entryPoint, Relation index, F
 		 * sortCandidates to true for in-memory builds to enable closer
 		 * caching, but there does not seem to be a difference in performance.
 		 */
-		neighbors = SelectNeighbors(lw, lm, lc, procinfo, collation, element, NULL, NULL, false);
+		neighbors = SelectNeighbors(lw, lm, lc, index, procinfos, collations, element, NULL, NULL, false);

 		AddConnections(element, neighbors, lm, lc);

--- a/src/hnswvacuum.c
+++ b/src/hnswvacuum.c
@@ -12,9 +12,12 @@
 * Check if deleted list contains an index TID
 */
 static bool
-DeletedContains(tidhash_hash * deleted, ItemPointer indextid)
+DeletedContains(HTAB *deleted, ItemPointer indextid)
 {
-	return tidhash_lookup(deleted, *indextid) != NULL;
+	bool		found;
+
+	hash_search(deleted, indextid, HASH_FIND, &found);
+	return found;
 }

 /*
@@ -107,13 +110,11 @@ RemoveHeapTids(HnswVacuumState * vacuumstate)
 			if (!ItemPointerIsValid(&etup->heaptids[0]))
 			{
 				ItemPointerData ip;
-				bool		found;

 				/* Add to deleted list */
 				ItemPointerSet(&ip, blkno, offno);

-				tidhash_insert(vacuumstate->deleted, ip, &found);
-				Assert(!found);
+				(void) hash_search(vacuumstate->deleted, &ip, HASH_ENTER, NULL);
 			}
 			else if (etup->level > highestLevel && !(entryPoint != NULL && blkno == entryPoint->blkno && offno == entryPoint->offno))
 			{
@@ -194,8 +195,8 @@ RepairGraphElement(HnswVacuumState * vacuumstate, HnswElement element, HnswEleme
 	GenericXLogState *state;
 	int			m = vacuumstate->m;
 	int			efConstruction = vacuumstate->efConstruction;
-	FmgrInfo   *procinfo = vacuumstate->procinfo;
-	Oid			collation = vacuumstate->collation;
+	FmgrInfo  **procinfos = vacuumstate->procinfos;
+	Oid		   *collations = vacuumstate->collations;
 	BufferAccessStrategy bas = vacuumstate->bas;
 	HnswNeighborTuple ntup = vacuumstate->ntup;
 	Size		ntupSize = HNSW_NEIGHBOR_TUPLE_SIZE(element->level, m);
@@ -206,10 +207,10 @@ RepairGraphElement(HnswVacuumState * vacuumstate, HnswElement element, HnswEleme

 	/* Init fields */
 	HnswInitNeighbors(element, m);
-	element->heaptidsLength = 0;
+	element->heaptids = NIL;

 	/* Add element to graph, skipping itself */
-	HnswInsertElement(element, entryPoint, index, procinfo, collation, m, efConstruction, true);
+	HnswInsertElement(element, entryPoint, index, procinfos, collations, m, efConstruction, true, false);

 	/* Update neighbor tuple */
 	/* Do this before getting page to minimize locking */
@@ -230,7 +231,7 @@ RepairGraphElement(HnswVacuumState * vacuumstate, HnswElement element, HnswEleme
 	UnlockReleaseBuffer(buf);

 	/* Update neighbors */
-	HnswUpdateNeighborPages(index, procinfo, collation, element, m, true, false);
+	HnswUpdateNeighborPages(index, procinfos, collations, element, m, true);
 }

 /*
@@ -257,7 +258,7 @@ RepairGraphEntryPoint(HnswVacuumState * vacuumstate)
 		LockPage(index, HNSW_UPDATE_LOCK, ShareLock);

 		/* Load element */
-		HnswLoadElement(highestPoint, NULL, NULL, index, vacuumstate->procinfo, vacuumstate->collation, true);
+		HnswLoadElement(highestPoint, NULL, NULL, NULL, NULL, NULL, index, vacuumstate->procinfos, vacuumstate->collations, true);

 		/* Repair if needed */
 		if (NeedsUpdated(vacuumstate, highestPoint))
@@ -286,7 +287,7 @@ RepairGraphEntryPoint(HnswVacuumState * vacuumstate)
 			 * point is outdated and empty, the entry point will be empty
 			 * until an element is repaired.
 			 */
-			HnswUpdateMetaPage(index, HNSW_UPDATE_ENTRY_ALWAYS, highestPoint, InvalidBlockNumber, MAIN_FORKNUM, false);
+			HnswUpdateMetaPage(index, HNSW_UPDATE_ENTRY_ALWAYS, highestPoint, InvalidBlockNumber, MAIN_FORKNUM);
 		}
 		else
 		{
@@ -295,7 +296,7 @@ RepairGraphEntryPoint(HnswVacuumState * vacuumstate)
 			 * is outdated, this can remove connections at higher levels in
 			 * the graph until they are repaired, but this should be fine.
 			 */
-			HnswLoadElement(entryPoint, NULL, NULL, index, vacuumstate->procinfo, vacuumstate->collation, true);
+			HnswLoadElement(entryPoint, NULL, NULL, NULL, NULL, NULL, index, vacuumstate->procinfos, vacuumstate->collations, true);

 			if (NeedsUpdated(vacuumstate, entryPoint))
 			{
@@ -371,7 +372,7 @@ RepairGraph(HnswVacuumState * vacuumstate)

 			/* Create an element */
 			element = HnswInitElementFromBlock(blkno, offno);
-			HnswLoadElementFromTuple(element, etup, false, true);
+			HnswLoadElementFromTuple(element, etup, false, true, index);

 			elements = lappend(elements, element);
 		}
@@ -419,7 +420,7 @@ RepairGraph(HnswVacuumState * vacuumstate)
 			 * was replaced and highest point was outdated.
 			 */
 			if (entryPoint == NULL || element->level > entryPoint->level)
-				HnswUpdateMetaPage(index, HNSW_UPDATE_ENTRY_GREATER, element, InvalidBlockNumber, MAIN_FORKNUM, false);
+				HnswUpdateMetaPage(index, HNSW_UPDATE_ENTRY_GREATER, element, InvalidBlockNumber, MAIN_FORKNUM);

 			/* Release lock */
 			UnlockPage(index, HNSW_UPDATE_LOCK, lockmode);
@@ -441,6 +442,7 @@ MarkDeleted(HnswVacuumState * vacuumstate)
 	BlockNumber insertPage = InvalidBlockNumber;
 	Relation	index = vacuumstate->index;
 	BufferAccessStrategy bas = vacuumstate->bas;
+	bool		useIndexTuple = IndexRelationGetNumberOfAttributes(index) > 1;

 	/*
 	 * Wait for index scans to complete. Scans before this point may contain
@@ -529,7 +531,18 @@ MarkDeleted(HnswVacuumState * vacuumstate)

 			/* Overwrite element */
 			etup->deleted = 1;
-			MemSet(&etup->data, 0, VARSIZE_ANY(&etup->data));
+			if (useIndexTuple)
+			{
+				IndexTuple	itup = (IndexTuple) &etup->data;
+
+				MemSet(itup, 0, IndexTupleSize(itup));
+			}
+			else
+			{
+				Vector	   *vec = (Vector *) (&etup->data);
+
+				MemSet(vec, 0, VARSIZE_ANY(vec));
+			}

 			/* Overwrite neighbors */
 			for (int i = 0; i < ntup->count; i++)
@@ -564,7 +577,7 @@ MarkDeleted(HnswVacuumState * vacuumstate)
 	}

 	/* Update insert page last, after everything has been marked as deleted */
-	HnswUpdateMetaPage(index, 0, NULL, insertPage, MAIN_FORKNUM, false);
+	HnswUpdateMetaPage(index, 0, NULL, insertPage, MAIN_FORKNUM);
 }

 /*
@@ -574,6 +587,7 @@ static void
 InitVacuumState(HnswVacuumState * vacuumstate, IndexVacuumInfo *info, IndexBulkDeleteResult *stats, IndexBulkDeleteCallback callback, void *callback_state)
 {
 	Relation	index = info->index;
+	HASHCTL		hash_ctl;

 	if (stats == NULL)
 		stats = (IndexBulkDeleteResult *) palloc0(sizeof(IndexBulkDeleteResult));
@@ -584,8 +598,8 @@ InitVacuumState(HnswVacuumState * vacuumstate, IndexVacuumInfo *info, IndexBulkD
 	vacuumstate->callback_state = callback_state;
 	vacuumstate->efConstruction = HnswGetEfConstruction(index);
 	vacuumstate->bas = GetAccessStrategy(BAS_BULKREAD);
-	vacuumstate->procinfo = index_getprocinfo(index, 1, HNSW_DISTANCE_PROC);
-	vacuumstate->collation = index->rd_indcollation[0];
+	vacuumstate->procinfos = HnswInitProcinfos(index);
+	vacuumstate->collations = index->rd_indcollation;
 	vacuumstate->ntup = palloc0(BLCKSZ);
 	vacuumstate->tmpCtx = AllocSetContextCreate(CurrentMemoryContext,
 												"Hnsw vacuum temporary context",
@@ -595,7 +609,10 @@ InitVacuumState(HnswVacuumState * vacuumstate, IndexVacuumInfo *info, IndexBulkD
 	HnswGetMetaPageInfo(index, &vacuumstate->m, NULL);

 	/* Create hash table */
-	vacuumstate->deleted = tidhash_create(CurrentMemoryContext, 256, NULL);
+	hash_ctl.keysize = sizeof(ItemPointerData);
+	hash_ctl.entrysize = sizeof(ItemPointerData);
+	hash_ctl.hcxt = CurrentMemoryContext;
+	vacuumstate->deleted = hash_create("hnswbulkdelete indextids", 256, &hash_ctl, HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
 }

 /*
@@ -604,8 +621,9 @@ InitVacuumState(HnswVacuumState * vacuumstate, IndexVacuumInfo *info, IndexBulkD
 static void
 FreeVacuumState(HnswVacuumState * vacuumstate)
 {
-	tidhash_destroy(vacuumstate->deleted);
+	hash_destroy(vacuumstate->deleted);
 	FreeAccessStrategy(vacuumstate->bas);
+	pfree(vacuumstate->procinfos);
 	pfree(vacuumstate->ntup);
 	MemoryContextDelete(vacuumstate->tmpCtx);
 }
--- a/src/ivfbuild.c
+++ b/src/ivfbuild.c
@@ -543,10 +543,10 @@ CreateListPages(Relation index, VectorArray centers, int dimensions,
 	pfree(list);
 }

-#ifdef IVFFLAT_KMEANS_DEBUG
 /*
 * Print k-means metrics
 */
+#ifdef IVFFLAT_KMEANS_DEBUG
 static void
 PrintKmeansMetrics(IvfflatBuildState * buildstate)
 {
--- a/src/ivfkmeans.c
+++ b/src/ivfkmeans.c
@@ -6,10 +6,6 @@
 #include "ivfflat.h"
 #include "miscadmin.h"

-#ifdef IVFFLAT_MEMORY
-#include "utils/memutils.h"
-#endif
-
 /*
 * Initialize with kmeans++
 *
@@ -155,23 +151,6 @@ QuickCenters(Relation index, VectorArray samples, VectorArray centers)
 	}
 }

-#ifdef IVFFLAT_MEMORY
-/*
- * Show memory usage
- */
-static void
-ShowMemoryUsage(Size estimatedSize)
-{
-#if PG_VERSION_NUM >= 130000
-	elog(INFO, "total memory: %zu MB",
-		 MemoryContextMemAllocated(CurrentMemoryContext, true) / (1024 * 1024));
-#else
-	MemoryContextStats(CurrentMemoryContext);
-#endif
-	elog(INFO, "estimated memory: %zu MB", estimatedSize / (1024 * 1024));
-}
-#endif
-
 /*
 * Use Elkan for performance. This requires distance function to satisfy triangle inequality.
 *
@@ -252,10 +231,6 @@ ElkanKmeans(Relation index, VectorArray samples, VectorArray centers)
 		vec->dim = dimensions;
 	}

-#ifdef IVFFLAT_MEMORY
-	ShowMemoryUsage(totalSize);
-#endif
-
 	/* Pick initial centers */
 	InitCenters(index, samples, centers, lowerBound);

--- a/src/vector.c
+++ b/src/vector.c
@@ -177,15 +177,14 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_in);
 Datum
 vector_in(PG_FUNCTION_ARGS)
 {
-	char	   *lit = PG_GETARG_CSTRING(0);
+	char	   *str = PG_GETARG_CSTRING(0);
 	int32		typmod = PG_GETARG_INT32(2);
 	float		x[VECTOR_MAX_DIM];
 	int			dim = 0;
 	char	   *pt;
 	char	   *stringEnd;
 	Vector	   *result;
-	char	   *litcopy = pstrdup(lit);
-	char	   *str = litcopy;
+	char	   *lit = pstrdup(str);

 	while (vector_isspace(*str))
 		str++;
@@ -269,7 +268,7 @@ vector_in(PG_FUNCTION_ARGS)
 				(errcode(ERRCODE_DATA_EXCEPTION),
 				 errmsg("vector must have at least 1 dimension")));

-	pfree(litcopy);
+	pfree(lit);

 	CheckExpectedDim(typmod, dim);

@@ -725,7 +724,7 @@ vector_spherical_distance(PG_FUNCTION_ARGS)
 }

 /*
- * Get the L1 distance between two vectors
+ * Get the L1 distance between vectors
 */
 PGDLLEXPORT PG_FUNCTION_INFO_V1(l1_distance);
 Datum
@@ -904,8 +903,8 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_lt);
 Datum
 vector_lt(PG_FUNCTION_ARGS)
 {
-	Vector	   *a = PG_GETARG_VECTOR_P(0);
-	Vector	   *b = PG_GETARG_VECTOR_P(1);
+	Vector	   *a = (Vector *) PG_GETARG_VECTOR_P(0);
+	Vector	   *b = (Vector *) PG_GETARG_VECTOR_P(1);

 	PG_RETURN_BOOL(vector_cmp_internal(a, b) < 0);
 }
@@ -917,8 +916,8 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_le);
 Datum
 vector_le(PG_FUNCTION_ARGS)
 {
-	Vector	   *a = PG_GETARG_VECTOR_P(0);
-	Vector	   *b = PG_GETARG_VECTOR_P(1);
+	Vector	   *a = (Vector *) PG_GETARG_VECTOR_P(0);
+	Vector	   *b = (Vector *) PG_GETARG_VECTOR_P(1);

 	PG_RETURN_BOOL(vector_cmp_internal(a, b) <= 0);
 }
@@ -930,8 +929,8 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_eq);
 Datum
 vector_eq(PG_FUNCTION_ARGS)
 {
-	Vector	   *a = PG_GETARG_VECTOR_P(0);
-	Vector	   *b = PG_GETARG_VECTOR_P(1);
+	Vector	   *a = (Vector *) PG_GETARG_VECTOR_P(0);
+	Vector	   *b = (Vector *) PG_GETARG_VECTOR_P(1);

 	PG_RETURN_BOOL(vector_cmp_internal(a, b) == 0);
 }
@@ -943,8 +942,8 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_ne);
 Datum
 vector_ne(PG_FUNCTION_ARGS)
 {
-	Vector	   *a = PG_GETARG_VECTOR_P(0);
-	Vector	   *b = PG_GETARG_VECTOR_P(1);
+	Vector	   *a = (Vector *) PG_GETARG_VECTOR_P(0);
+	Vector	   *b = (Vector *) PG_GETARG_VECTOR_P(1);

 	PG_RETURN_BOOL(vector_cmp_internal(a, b) != 0);
 }
@@ -956,8 +955,8 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_ge);
 Datum
 vector_ge(PG_FUNCTION_ARGS)
 {
-	Vector	   *a = PG_GETARG_VECTOR_P(0);
-	Vector	   *b = PG_GETARG_VECTOR_P(1);
+	Vector	   *a = (Vector *) PG_GETARG_VECTOR_P(0);
+	Vector	   *b = (Vector *) PG_GETARG_VECTOR_P(1);

 	PG_RETURN_BOOL(vector_cmp_internal(a, b) >= 0);
 }
@@ -969,8 +968,8 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_gt);
 Datum
 vector_gt(PG_FUNCTION_ARGS)
 {
-	Vector	   *a = PG_GETARG_VECTOR_P(0);
-	Vector	   *b = PG_GETARG_VECTOR_P(1);
+	Vector	   *a = (Vector *) PG_GETARG_VECTOR_P(0);
+	Vector	   *b = (Vector *) PG_GETARG_VECTOR_P(1);

 	PG_RETURN_BOOL(vector_cmp_internal(a, b) > 0);
 }
@@ -982,8 +981,8 @@ PGDLLEXPORT PG_FUNCTION_INFO_V1(vector_cmp);
 Datum
 vector_cmp(PG_FUNCTION_ARGS)
 {
-	Vector	   *a = PG_GETARG_VECTOR_P(0);
-	Vector	   *b = PG_GETARG_VECTOR_P(1);
+	Vector	   *a = (Vector *) PG_GETARG_VECTOR_P(0);
+	Vector	   *b = (Vector *) PG_GETARG_VECTOR_P(1);

 	PG_RETURN_INT32(vector_cmp_internal(a, b));
 }
--- a/test/t/012_hnsw_build_recall.pl
+++ b/test/t/012_hnsw_build_recall.pl
@@ -83,32 +83,11 @@ for my $i (0 .. $#operators)
 		push(@expected, $res);
 	}

-	# Build index serially
-	$node->safe_psql("postgres", qq(
-		SET max_parallel_maintenance_workers = 0;
-		CREATE INDEX idx ON tst USING hnsw (v $opclass);
-	));
+	# Add index
+	$node->safe_psql("postgres", "CREATE INDEX ON tst USING hnsw (v $opclass);");

-	# Test approximate results
 	my $min = $operator eq "<#>" ? 0.80 : 0.99;
 	test_recall($min, $operator);
-
-	$node->safe_psql("postgres", "DROP INDEX idx;");
-
-	# Build index in parallel
-	my ($ret, $stdout, $stderr) = $node->psql("postgres", qq(
-		SET client_min_messages = DEBUG;
-		SET min_parallel_table_scan_size = 1;
-		SET hnsw.enable_parallel_build = on;
-		CREATE INDEX idx ON tst USING hnsw (v $opclass);
-	));
-	is($ret, 0, $stderr);
-	like($stderr, qr/using \d+ parallel workers/);
-
-	# Test approximate results
-	test_recall($min, $operator);
-
-	$node->safe_psql("postgres", "DROP INDEX idx;");
 }

 done_testing();
--- a/test/t/018_hnsw_filtering.pl
+++ b/test/t/018_hnsw_filtering.pl
@@ -1,79 +0,0 @@
-use strict;
-use warnings;
-use PostgresNode;
-use TestLib;
-use Test::More;
-
-my $dim = 3;
-my $nc = 50;
-my $limit = 20;
-
-my $array_sql = join(",", ('random()') x $dim);
-
-# Initialize node
-my $node = get_new_node('node');
-$node->init;
-$node->start;
-
-# Create table and index
-$node->safe_psql("postgres", "CREATE EXTENSION vector;");
-$node->safe_psql("postgres", "CREATE TABLE tst (i int4, v vector($dim), c int4);");
-$node->safe_psql("postgres",
-	"INSERT INTO tst SELECT i, ARRAY[$array_sql], i % $nc FROM generate_series(1, 10000) i;"
-);
-$node->safe_psql("postgres", "CREATE INDEX idx ON tst USING hnsw (v vector_l2_ops);");
-
-# Generate query
-my @r = ();
-for (1 .. $dim)
-{
-	push(@r, rand());
-}
-my $query = "[" . join(",", @r) . "]";
-my $c = int(rand() * $nc);
-
-# Test attribute filtering
-my $explain = $node->safe_psql("postgres", qq(
-	EXPLAIN ANALYZE SELECT i FROM tst WHERE c = $c ORDER BY v <-> '$query' LIMIT $limit;
-));
-like($explain, qr/Seq Scan/);
-
-# Test attribute filtering with few rows removed
-$explain = $node->safe_psql("postgres", qq(
-	EXPLAIN ANALYZE SELECT i FROM tst WHERE c != $c ORDER BY v <-> '$query' LIMIT $limit;
-));
-like($explain, qr/Index Scan using idx/);
-
-# Test distance filtering
-$explain = $node->safe_psql("postgres", qq(
-	EXPLAIN ANALYZE SELECT i FROM tst WHERE v <-> '$query' < 1 ORDER BY v <-> '$query' LIMIT $limit;
-));
-like($explain, qr/Index Scan using idx/);
-
-# Test distance filtering without order
-$explain = $node->safe_psql("postgres", qq(
-	EXPLAIN ANALYZE SELECT i FROM tst WHERE v <-> '$query' < 1;
-));
-like($explain, qr/Seq Scan/);
-
-# Test distance filtering without limit
-$explain = $node->safe_psql("postgres", qq(
-	EXPLAIN ANALYZE SELECT i FROM tst WHERE v <-> '$query' < 1 ORDER BY v <-> '$query';
-));
-like($explain, qr/Seq Scan/);
-
-# Test attribute index
-$node->safe_psql("postgres", "CREATE INDEX attribute_idx ON tst (c);");
-$explain = $node->safe_psql("postgres", qq(
-	EXPLAIN ANALYZE SELECT i FROM tst WHERE c = $c ORDER BY v <-> '$query' LIMIT $limit;
-));
-like($explain, qr/Bitmap Index Scan on attribute_idx/);
-
-# Test partial index
-$node->safe_psql("postgres", "CREATE INDEX partial_idx ON tst USING hnsw (v vector_l2_ops) WHERE (c = $c);");
-$explain = $node->safe_psql("postgres", qq(
-	EXPLAIN ANALYZE SELECT i FROM tst WHERE c = $c ORDER BY v <-> '$query' LIMIT $limit;
-));
-like($explain, qr/Index Scan using partial_idx/);
-
-done_testing();
--- a/test/t/018_ivfflat_deletes.pl
+++ b/test/t/018_ivfflat_deletes.pl
@@ -0,0 +1,43 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $dim = 3;
+
+my $array_sql = join(",", ('random()') x $dim);
+
+# Initialize node
+my $node = get_new_node('node');
+$node->init;
+$node->start;
+
+# Create table and index
+$node->safe_psql("postgres", "CREATE EXTENSION vector;");
+$node->safe_psql("postgres", "CREATE TABLE tst (i serial, v vector($dim));");
+$node->safe_psql("postgres",
+	"INSERT INTO tst (v) SELECT ARRAY[$array_sql] FROM generate_series(1, 10000) i;"
+);
+$node->safe_psql("postgres", "CREATE INDEX ON tst USING ivfflat (v vector_l2_ops);");
+
+# Delete data
+$node->safe_psql("postgres", "DELETE FROM tst WHERE i % 100 != 0;");
+
+my $exp = $node->safe_psql("postgres", qq(
+	SET enable_indexscan = off;
+	SELECT i FROM tst ORDER BY v <-> '[0,0,0]';
+));
+
+# Run twice to make sure correct tuples marked as dead
+for (1 .. 2)
+{
+	my $res = $node->safe_psql("postgres", qq(
+		SET enable_seqscan = off;
+		SET ivfflat.probes = 100;
+		SELECT i FROM tst ORDER BY v <-> '[0,0,0]';
+	));
+	is($res, $exp);
+}
+
+done_testing();
--- a/test/t/019_hnsw_filtering.pl
+++ b/test/t/019_hnsw_filtering.pl
@@ -0,0 +1,109 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+my $node;
+my @queries = ();
+my @cs = ();
+my @expected;
+my $limit = 20;
+my $dim = 3;
+my $array_sql = join(",", ('random()') x $dim);
+my $nc = 50;
+
+sub test_recall
+{
+	my ($min, $operator) = @_;
+	my $correct = 0;
+	my $total = 0;
+
+	my $explain = $node->safe_psql("postgres", qq(
+		SET enable_seqscan = off;
+		EXPLAIN ANALYZE SELECT i FROM tst WHERE c = $cs[0] ORDER BY v $operator '$queries[0]' LIMIT $limit;
+	));
+	like($explain, qr/Index Cond/);
+
+	for my $i (0 .. $#queries)
+	{
+		my $actual = $node->safe_psql("postgres", qq(
+			SET enable_seqscan = off;
+			SELECT i FROM tst WHERE c = $cs[$i] ORDER BY v $operator '$queries[$i]' LIMIT $limit;
+		));
+		my @actual_ids = split("\n", $actual);
+		my %actual_set = map { $_ => 1 } @actual_ids;
+
+		is(scalar(@actual_ids), $limit);
+
+		my @expected_ids = split("\n", $expected[$i]);
+
+		foreach (@expected_ids)
+		{
+			if (exists($actual_set{$_}))
+			{
+				$correct++;
+			}
+			$total++;
+		}
+	}
+
+	cmp_ok($correct / $total, ">=", $min, $operator);
+}
+
+# Initialize node
+$node = get_new_node('node');
+$node->init;
+$node->start;
+
+# Create table
+$node->safe_psql("postgres", "CREATE EXTENSION vector;");
+$node->safe_psql("postgres", "CREATE TABLE tst (i int4, v vector($dim), c int4);");
+$node->safe_psql("postgres",
+	"INSERT INTO tst SELECT i, ARRAY[$array_sql], i % $nc FROM generate_series(1, 20000) i;"
+);
+
+# Generate queries
+for (1 .. 20)
+{
+	my @r = ();
+	for (1 .. $dim)
+	{
+		push(@r, rand());
+	}
+	push(@queries, "[" . join(",", @r) . "]");
+	push(@cs, int(rand() * $nc));
+}
+
+# Get exact results
+@expected = ();
+for my $i (0 .. $#queries)
+{
+	my $res = $node->safe_psql("postgres", "SELECT i FROM tst WHERE c = $cs[$i] ORDER BY v <-> '$queries[$i]' LIMIT $limit;");
+	push(@expected, $res);
+}
+
+# Add index
+$node->safe_psql("postgres", "CREATE INDEX ON tst USING hnsw (v vector_l2_ops, c);");
+
+# Test recall
+test_recall(0.99, '<->');
+
+# Test vacuum
+$node->safe_psql("postgres", "DELETE FROM tst WHERE c > 5;");
+$node->safe_psql("postgres", "VACUUM tst;");
+
+# Test columns
+my ($ret, $stdout, $stderr) = $node->psql("postgres", "CREATE INDEX ON tst USING hnsw (c);");
+like($stderr, qr/first column must be a vector/);
+
+($ret, $stdout, $stderr) = $node->psql("postgres", "CREATE INDEX ON tst USING hnsw (c, v vector_l2_ops);");
+like($stderr, qr/first column must be a vector/);
+
+($ret, $stdout, $stderr) = $node->psql("postgres", "CREATE INDEX ON tst USING hnsw (v vector_l2_ops, c, c);");
+like($stderr, qr/index cannot have more than two columns/);
+
+($ret, $stdout, $stderr) = $node->psql("postgres", "CREATE INDEX ON tst USING hnsw (v vector_l2_ops, v vector_l2_ops);");
+like($stderr, qr/column 2 cannot be a vector/);
+
+done_testing();
Author	SHA1	Message	Date
Andrew Kane	9f944e5c54	Fixed logic [skip ci]	2024-09-23 13:52:50 -07:00
Andrew Kane	e70e582d2f	Increase ef_search if needed to improve recall	2023-11-18 15:44:38 -08:00
Andrew Kane	bacd99b37a	Test result count [skip ci]	2023-11-18 13:58:05 -08:00
Andrew Kane	08bd246529	Fixed vacuum test [skip ci]	2023-11-10 14:03:13 -08:00
Andrew Kane	f57f2b6821	Added support for inline filtering with HNSW	2023-11-10 13:28:48 -08:00