Andrew Kane
2d0f162bd7
Added support for in-memory parallel index builds for HNSW
...
Co-authored-by: Heikki Linnakangas <heikki.linnakangas@iki.fi >
2024-01-22 23:19:10 -08:00
Heikki Linnakangas
4c6928bd3c
Remove HnswSpool
...
It was just used to pass heap/index relations to
HnswParallelScanAndInsert. I think it was copied from nbtsort.c, which
is more complicated. I don't think we need a struct like this.
(That said, I actually think that we should have a state object that
would hold fields like 'heap', 'index', 'procinfo', 'collation'
etc. Passing that object around would simplify the signatures of many
functions. But that's a different story).
2024-01-22 23:11:25 -08:00
Heikki Linnakangas
6fd05dd6f6
Remove unused 'scantuplesortstates' field
2024-01-22 23:08:20 -08:00
Andrew Kane
597bfdc76b
Added HnswGetNeighbors macro
2024-01-13 20:00:34 -08:00
Andrew Kane
cbf3eb4fa5
Improved HNSW build and insert code
2024-01-13 10:07:42 -08:00
Andrew Kane
65d060ac86
Reverted FlushPages pattern for parallel builds
2024-01-08 10:45:31 -08:00
Andrew Kane
62ee33bb92
Improved locking code
2024-01-08 09:05:12 -08:00
Andrew Kane
520e274dde
Improved locking code
2024-01-07 22:34:41 -08:00
Andrew Kane
9e680884bd
Moved indtuples to HnswGraph
2024-01-07 22:23:49 -08:00
Andrew Kane
19a0e1b341
Moved graph to separate struct
2024-01-07 20:15:30 -08:00
Andrew Kane
c7fe1571ee
Improved code
2024-01-07 18:30:51 -08:00
Andrew Kane
cb4c770df2
Switched to slist for elements to reduce allocations and remove limit
2024-01-07 18:26:19 -08:00
Andrew Kane
85fdecd79b
Moved FlushPages before HnswEndParallel
2024-01-07 17:50:46 -08:00
Andrew Kane
6132428914
Improved number of parallel workers for HNSW index builds - closes #397
2024-01-05 19:46:08 -08:00
Andrew Kane
81d13bd40f
Improved code [skip ci]
2024-01-03 13:53:23 -05:00
Andrew Kane
8ee37b60a0
Improved memory estimate for HNSW index builds
2024-01-03 13:47:50 -05:00
Andrew Kane
cae630784b
Improved BuildCallback [skip ci]
2023-12-30 20:55:29 -05:00
Andrew Kane
d87bcd2deb
Added comments [skip ci]
2023-12-30 18:29:01 -05:00
Andrew Kane
736576220a
Improved BuildCallback
2023-12-30 18:24:03 -05:00
Andrew Kane
1e422cd62b
Improved readability [skip ci]
2023-12-22 16:39:13 -05:00
Andrew Kane
569c69580a
Improved InsertTuple code - #384
...
Co-authored-by: Heikki Linnakangas <heikki.linnakangas@iki.fi >
2023-12-22 15:08:28 -05:00
Andrew Kane
59509c3a17
Added extra 5% to memory estimate
2023-12-22 14:04:05 -05:00
Andrew Kane
61738846af
Updated comment [skip ci]
2023-12-22 14:03:33 -05:00
Andrew Kane
e8c3bf0cef
Improved memory tracking for HNSW index builds - #384
2023-12-22 13:35:43 -05:00
Andrew Kane
50d1aed3d8
Improved memory usage logging [skip ci]
2023-12-22 13:09:11 -05:00
Andrew Kane
66e14d2434
Updated indentation [skip ci]
2023-12-22 12:59:50 -05:00
Andrew Kane
42cd4c6833
Fixed call to GenerationContextCreate for Postgres < 15
2023-12-22 12:49:07 -05:00
Andrew Kane
dcbe0b6f0d
Reduced memory usage for HNSW index builds - #384
...
Co-authored-by: Heikki Linnakangas <heikki.linnakangas@iki.fi >
2023-12-22 12:41:47 -05:00
Andrew Kane
f61d4087b5
Slightly improved memory estimation [skip ci]
2023-12-21 10:31:36 -05:00
Andrew Kane
57554e5b46
Added todo [skip ci]
2023-12-20 17:52:31 -05:00
Andrew Kane
6738fa0bd7
Added HNSW_MEMORY flag to show memory usage - #384 [skip ci]
2023-12-20 16:49:16 -05:00
Andrew Kane
43e0b3d9d4
Mark buffer contents as dirty when not logging
2023-12-20 16:16:25 -05:00
Andrew Kane
2bff7ccaa2
Improved memory estimation - #384 [skip ci]
2023-12-20 11:10:27 -05:00
Andrew Kane
e88a425c9b
Reduced WAL generation for HNSW index builds - thanks @hlinnaka
2023-12-19 20:37:32 -05:00
Andrew Kane
a59aa02dd9
Only show message if flushed due to memory [skip ci]
2023-12-04 15:29:23 -08:00
Andrew Kane
2fef497b7e
Fixed check
2023-12-04 15:22:12 -08:00
Andrew Kane
a156f6c7ae
Fixed invalid memory alloc request size error with HNSW - fixes #43
2023-12-04 15:00:32 -08:00
Andrew Kane
dfee5d4045
Added support for on-disk parallel index builds for HNSW
2023-11-11 19:29:45 -08:00
Andrew Kane
c5e8c46b80
Switched from VECTOR_SIZE to VARSIZE_ANY [skip ci]
2023-11-09 19:41:38 -08:00
Andrew Kane
fbb904ae2f
Use pointer for VARSIZE_ANY
2023-11-09 17:50:28 -08:00
Andrew Kane
3cf6f62900
Switched to datum for HnswElement
2023-11-09 17:35:39 -08:00
Andrew Kane
84e073888c
Removed vector-specific code from HNSW_ELEMENT_TUPLE_SIZE [skip ci]
2023-11-09 16:57:01 -08:00
Andrew Kane
3f3463bde5
Improved memory calculation for HNSW and removed vector-specific code
2023-11-09 16:21:26 -08:00
Andrew Kane
6e1312ddbe
DRY max size
2023-10-04 21:43:34 -07:00
Andrew Kane
4ef5bca275
Use BLCKSZ for consistency
2023-10-04 21:37:55 -07:00
Xiaoran Wang
1ecf6ada76
Include ItemIdData when computing the maxSize for the data in a page ( #274 )
...
As the data is aligned, for hnsw, the combined size won't be in the range
(8156 (maxSize exlucding `ItemIdData`), 8160]. So even if the
ItemIdData is not included in the maxSize, it works well now, but I
think it's better to make it correct.
2023-10-04 21:26:46 -07:00
jeff-davis
b247b688a8
No need to MarkBufferDirty(); GenericXLogFinish() does that. ( #265 )
2023-09-15 13:14:10 -07:00
Andrew Kane
c3bafc76e8
Only update entry point on inserts if level is greater
2023-08-15 12:31:06 -07:00
Andrew Kane
86a062e504
Set entry level in CreateMetaPage [skip ci]
2023-08-15 11:47:05 -07:00
Andrew Kane
451e36cee7
Added check for ef_construction [skip ci]
2023-08-10 21:09:38 -07:00
