Use LWLocks instead of SpinLocks (#410)

Spinlocks should be held only for a few instructions, for multiple
reasons:

- You have to be very careful not to elog() out while holding a
  spinlock, because there is no mechanism to release the spinlock on
  error.

- Waiters can waste a lot of cycles spinning if the lock is
  contended. I you wait on a spinlock for too long, the PostgreSQL
  implementation will actually PANIC, see s_lock_stuck().

The flushLock is particularly problematic. It is held in exclusive
mode, which means it holds a spinlock, over the call to
FlushPages(). FlushPages() performs lots of I/O so it can take a very
long time (>= minutes), and can also easily error out for various
reasons.

allocatorLock would perhaps be OK as a spinlocks, but even that feels
a bit heavy, so I converted that to an LWLock, too.

entryLock is usually held for a very short time, in shared mode, so
that would be fine as a spinlock. However, in the rare case that the
entry point is updated, it's held for a very long time. An LWLock used
in shared mode is about as fast a spinlock, that path is pretty
heavily optimized.

I think we have some problems with the per-element spinlocks too. In
HnswUpdateNeighborPagesInMemory(), it's held over a call to
HnswUpdateConnection(), but HnswUpdateConnection() can error out at
least in case of an out-of-memory error (it uses lappend(), which
calls palloc()). It also calls the distance function, and I don't
think they are guaranteed to be ereport-free either. However, I didn't
address that in this PR, it needs a bit more thinking.

src/hnsw.h

@@ -116,6 +116,11 @@
 /* Variables */
 extern int	hnsw_ef_search;
 
+/* These are initialized when the module is loaded */
+extern int	entryLockTrancheId;
+extern int	allocatorLockTrancheId;
+extern int	flushLockTrancheId;
+
 typedef struct HnswElementData HnswElementData;
 typedef struct HnswNeighborArray HnswNeighborArray;
 
@@ -177,24 +182,6 @@ typedef struct HnswOptions
 	int			efConstruction; /* size of dynamic candidate list */
 }			HnswOptions;
 
-typedef enum HnswLWLockMode
-{
-	RW_EXCLUSIVE,
-	RW_SHARED
-}			HnswLWLockMode;
-
-/*
- * Readers-writers with weak priority to the readers
- *
- * https://doi.org/10.1007/978-3-642-32027-9
- */
-typedef struct HnswRWLock
-{
-	volatile int readers;
-	slock_t		readersMutex;
-	slock_t		globalMutex;
-}			HnswRWLock;
-
 typedef struct HnswGraph
 {
 	/* Graph state */
@@ -203,16 +190,16 @@ typedef struct HnswGraph
 	double		indtuples;
 
 	/* Entry state */
-	slock_t		entryLock;
+	LWLock		entryLock;
 	HnswElementPtr entryPoint;
 
 	/* Allocations state */
-	slock_t		allocatorLock;
+	LWLock		allocatorLock;
 	long		memoryUsed;
 	long		memoryTotal;
 
 	/* Flushed state */
-	HnswRWLock	flushLock;
+	LWLock		flushLock;
 	bool		flushed;
 }			HnswGraph;