From: kettenis <kettenis@openbsd.org>
Date: Mon, 2 Aug 2021 19:07:29 +0000 (+0000)
Subject: Add memory barrier (data FENCE) before making the SBI call to issue a
X-Git-Url: http://artulab.com/gitweb/?a=commitdiff_plain;h=06f70a83ac3b09596c4bed7ec25771f3c1b05238;p=openbsd

Add memory barrier (data FENCE) before making the SBI call to issue a
FENCE.I instruction on the remote harts.  According to the RISC-V
ISA documentation this is necessary to make remote harts observe earlier
stores.  Also issue the local FENCE.I first; this is what both FreeBSD and
Linux does.  Seems to fix at least some of the issue we've seen running
GENERIC.MP on MP machines.

ok jca@
---

diff --git a/sys/arch/riscv64/riscv64/pmap.c b/sys/arch/riscv64/riscv64/pmap.c
index 0835a1a4036..e6fe8338ea9 100644
--- a/sys/arch/riscv64/riscv64/pmap.c
+++ b/sys/arch/riscv64/riscv64/pmap.c
@@ -1,4 +1,4 @@
-/*	$OpenBSD: pmap.c,v 1.18 2021/07/30 13:17:33 kettenis Exp $	*/
+/*	$OpenBSD: pmap.c,v 1.19 2021/08/02 19:07:29 kettenis Exp $	*/
 
 /*
  * Copyright (c) 2019-2020 Brian Bamsch <bbamsch@google.com>
@@ -105,18 +105,29 @@ icache_flush(void)
 	CPU_INFO_ITERATOR cii;
 	struct cpu_info *ci;
 	unsigned long hart_mask = 0;
+#endif
+
+	fence_i();
 
+#ifdef MULTIPROCESSOR
 	CPU_INFO_FOREACH(cii, ci) {
 		if (ci == curcpu())
 			continue;
 		hart_mask |= (1UL << ci->ci_hartid);
 	}
 
-	if (hart_mask != 0)
+	/*
+	 * From the RISC-V ISA:
+	 *
+	 * To make a store to instruction memory visible to all RISC-V
+	 * harts, the writing hart has to execute a data FENCE before
+	 * requesting that all remote RISC-V harts execute a FENCE.I.
+	 */
+	if (hart_mask != 0) {
+		membar_sync();
 		sbi_remote_fence_i(&hart_mask);
+	}
 #endif
-
-	fence_i();
 }
 
 struct pmap kernel_pmap_;