Commit Graph

64 Commits

Author SHA1 Message Date
Andrew Kane
3ecb9a3cb2 Renamed HnswInsertElement to HnswFindElementNeighbors [skip ci] 2024-01-22 16:59:08 -08:00
Andrew Kane
16d7de79f6 Improved function names [skip ci] 2024-01-22 16:43:50 -08:00
Heikki Linnakangas
ca3b4cd029 Remove HnswSpool
It was just used to pass heap/index relations to
HnswParallelScanAndInsert. I think it was copied from nbtsort.c, which
is more complicated. I don't think we need a struct like this.

(That said, I actually think that we should have a state object that
would hold fields like 'heap', 'index', 'procinfo', 'collation'
etc. Passing that object around would simplify the signatures of many
functions. But that's a different story).
2024-01-19 00:26:47 -08:00
Heikki Linnakangas
d96e486274 Remove unused 'scantuplesortstates' field 2024-01-19 00:26:47 -08:00
Andrew Kane
d801a843f4 Removed HnswPtrSetNull to avoid setting relptr_off directly 2024-01-16 17:08:13 -08:00
Andrew Kane
cad48d9203 Improved locking 2024-01-16 13:34:55 -08:00
Heikki Linnakangas
719b4b7436 Use LWLocks instead of SpinLocks (#410)
Spinlocks should be held only for a few instructions, for multiple
reasons:

- You have to be very careful not to elog() out while holding a
  spinlock, because there is no mechanism to release the spinlock on
  error.

- Waiters can waste a lot of cycles spinning if the lock is
  contended. I you wait on a spinlock for too long, the PostgreSQL
  implementation will actually PANIC, see s_lock_stuck().

The flushLock is particularly problematic. It is held in exclusive
mode, which means it holds a spinlock, over the call to
FlushPages(). FlushPages() performs lots of I/O so it can take a very
long time (>= minutes), and can also easily error out for various
reasons.

allocatorLock would perhaps be OK as a spinlocks, but even that feels
a bit heavy, so I converted that to an LWLock, too.

entryLock is usually held for a very short time, in shared mode, so
that would be fine as a spinlock. However, in the rare case that the
entry point is updated, it's held for a very long time. An LWLock used
in shared mode is about as fast a spinlock, that path is pretty
heavily optimized.

I think we have some problems with the per-element spinlocks too. In
HnswUpdateNeighborPagesInMemory(), it's held over a call to
HnswUpdateConnection(), but HnswUpdateConnection() can error out at
least in case of an out-of-memory error (it uses lappend(), which
calls palloc()). It also calls the distance function, and I don't
think they are guaranteed to be ereport-free either. However, I didn't
address that in this PR, it needs a bit more thinking.
2024-01-16 13:25:03 -08:00
Andrew Kane
1612b84069 Fixed error on Windows [skip ci] 2024-01-15 19:33:16 -08:00
Andrew Kane
9d3e4e74df Added support for in-memory parallel index builds for HNSW 2024-01-15 15:07:31 -08:00
Andrew Kane
c7d60346d8 Improved macro [skip ci] 2024-01-13 20:02:41 -08:00
Andrew Kane
597bfdc76b Added HnswGetNeighbors macro 2024-01-13 20:00:34 -08:00
Andrew Kane
cbf3eb4fa5 Improved HNSW build and insert code 2024-01-13 10:07:42 -08:00
Andrew Kane
1881b857f9 Simplified code 2024-01-09 18:53:31 -08:00
Andrew Kane
9e680884bd Moved indtuples to HnswGraph 2024-01-07 22:23:49 -08:00
Andrew Kane
19a0e1b341 Moved graph to separate struct 2024-01-07 20:15:30 -08:00
Andrew Kane
cb4c770df2 Switched to slist for elements to reduce allocations and remove limit 2024-01-07 18:26:19 -08:00
Andrew Kane
8ee37b60a0 Improved memory estimate for HNSW index builds 2024-01-03 13:47:50 -05:00
Andrew Kane
9b73b3d1a6 Reduced memory and allocations for heap TIDs - closes #385 2024-01-03 13:41:34 -05:00
Andrew Kane
e8c3bf0cef Improved memory tracking for HNSW index builds - #384 2023-12-22 13:35:43 -05:00
Andrew Kane
dcbe0b6f0d Reduced memory usage for HNSW index builds - #384
Co-authored-by: Heikki Linnakangas <heikki.linnakangas@iki.fi>
2023-12-22 12:41:47 -05:00
Andrew Kane
6738fa0bd7 Added HNSW_MEMORY flag to show memory usage - #384 [skip ci] 2023-12-20 16:49:16 -05:00
Andrew Kane
e88a425c9b Reduced WAL generation for HNSW index builds - thanks @hlinnaka 2023-12-19 20:37:32 -05:00
Andrew Kane
921427ee03 Replace dynahash hash table in HNSW with simplehash for speed - #378
Co-authored-by: Heikki Linnakangas <heikki.linnakangas@iki.fi>
2023-12-17 11:24:13 -05:00
Andrew Kane
dfee5d4045 Added support for on-disk parallel index builds for HNSW 2023-11-11 19:29:45 -08:00
Andrew Kane
94f7304ccd Keep vector for now to be overly cautious about packing [skip ci] 2023-11-09 18:43:55 -08:00
Andrew Kane
d078db3d25 Switched HnswElementTuple to generic data and zero full section 2023-11-09 18:28:25 -08:00
Andrew Kane
3cf6f62900 Switched to datum for HnswElement 2023-11-09 17:35:39 -08:00
Andrew Kane
84e073888c Removed vector-specific code from HNSW_ELEMENT_TUPLE_SIZE [skip ci] 2023-11-09 16:57:01 -08:00
Andrew Kane
3f3463bde5 Improved memory calculation for HNSW and removed vector-specific code 2023-11-09 16:21:26 -08:00
Andrew Kane
a713e2acaa Improved performance of HNSW index builds - closes #292
Co-authored-by: Heikki Linnakangas <heikki.linnakangas@iki.fi>
2023-10-05 13:21:26 -07:00
Andrew Kane
6e1312ddbe DRY max size 2023-10-04 21:43:34 -07:00
Andrew Kane
bca50a03fa Use consistent variable name 2023-09-12 19:24:31 -07:00
Andrew Kane
b164833933 Removed pinning for HNSW index scan 2023-09-11 12:12:28 -07:00
Andrew Kane
d05d6ee83d Get m from metapage 2023-09-03 01:35:21 -07:00
Jonathan S. Katz
e50a79108f Set default HNSW "ef_construction" to 64 (#230) 2023-08-26 16:05:08 -07:00
Andrew Kane
e406b7f5ea Added comments [skip ci] 2023-08-23 21:03:07 -07:00
Andrew Kane
6d88a9e1d2 Updated HNSW_PAGE_ID [skip ci] 2023-08-21 22:59:53 -07:00
Andrew Kane
1e851c12c0 Updated comment [skip ci] 2023-08-21 22:55:24 -07:00
Andrew Kane
8ed3cc5f0b Improved macro [skip ci] 2023-08-21 22:52:58 -07:00
Andrew Kane
bace0891bd Updated comments [skip ci] 2023-08-21 22:51:24 -07:00
Andrew Kane
0e1de45463 Improved locking code [skip ci] 2023-08-21 15:42:59 -07:00
Andrew Kane
3f3b3ca8e3 Made function static [skip ci] 2023-08-21 03:27:42 -07:00
Andrew Kane
02f4e0ec8b Revert "Added version to reduce stale reads and writes and prepare for optimistic locking"
This reverts commit ef1209eaf4.
2023-08-21 02:47:27 -07:00
Andrew Kane
ef1209eaf4 Added version to reduce stale reads and writes and prepare for optimistic locking 2023-08-20 17:08:20 -07:00
Andrew Kane
687263ccd4 DRY HNSW vacuum code 2023-08-20 14:52:31 -07:00
Andrew Kane
a03f6ae4bc Added prefix to function name [skip ci] 2023-08-18 00:54:09 -07:00
Andrew Kane
b72a22b3c0 Fixed duplicate connections when repairing graph 2023-08-16 17:07:19 -07:00
Andrew Kane
c3bafc76e8 Only update entry point on inserts if level is greater 2023-08-15 12:31:06 -07:00
Andrew Kane
e1d6654063 Revert "Improved HNSW vacuum performance"
This reverts commit c5b2f3ac8b.
2023-08-13 17:19:22 -07:00
Andrew Kane
c5b2f3ac8b Improved HNSW vacuum performance 2023-08-13 16:50:36 -07:00