I just finished an initial foray back into plan9 kernel code. The Harvey fork has been modified so that the kernel now links at 0xffff_8000_0000_0000.

https://github.com/Harvey-OS/harvey/pull/1043/commits/71f450f801d006924e707971f4d28b1760fcf1ec

The bootstrap and secondary CPU startup code was, basically, completely rewritten and I made numerous little cleanups across the kernel.

Why do this?

Plan 9 traditionally used a segmented kernel architecture where the kernel was linked at -256MiB (0xF000_0000 on 32-bit systems; 0xFFFF_FFFF_F000_0000 on 64-bit). The symbol `KSEG0` referred to this address.

More virtual memory in the kernel portion of the address space was available at `KSEG2`; on Harvey, this was -1TiB. There was also a section of virtual address space at VMAP for mapping things like MMIO regions and other stuff.

But why do this? Observe that the physical address space is much smaller than the virtual address space (at least on current machines); it's much easier to maintain a direct map, by simply starting at the top of the kernel-portion of the canonical address space (x86_64 has a gap)

So I did that. The VMAP is gone; the semi-recursive PDMAP is gone; a small virtual address space allocator in asm.c is gone. Now, to map, say, a PCI BAR, one simply maps it at KZERO + its physical address; much simpler.

Another few improvements were made: kernel text is now mapped read-only; read-only data similarly. Anything that is not kernel text is mapped no-execute. Large extents of physical memory are mapped into the kernel using 2MiB or 1GiB pages.

Also, every process now has its own page table (though the kernel is shared across processes). Previously, each physical CPU had its own page table, and a process swap would clear the user-portion of the PML4 from pointers to pages for the outgoing process...

...and copy in pointers to the incoming process's pages.

Oh, and flush the TLB, of course.

Now, to switch address spaces between processes, we simply reload the root page table pointer (which, on x86_64, implicitly flushes the TLB). Kernel text is a global page, since it's immutable on boot.

(Aside: at this point, the astute reader may notice that much of the description of the old system was clearly influenced by the MIPS architecture; from the names of segments to the system for changing address spaces clearing coming from a software-TLB history.)

Anyway. The new system simplifies the architecture and makes the kernel arguably more robust. Ironically, failures have increased because it has uncovered more basic errors in the kernel. We'll have to fix some of that.