Core: Handle various edge cases related to executable permissions. (#3660)

* Fix flag handling on Windows

Fixes a weird homebrew kalaposfos made

* Fix backing protects

Windows requires that protections on areas committed through MapViewOfFile functions are less than the original mapping.
The best way to make sure everything works is to VirtualProtect the code area with the requested protection instead of applying prot directly.

* Fix error code for sceKernelMapDirectMemory2

Real hardware returns EINVAL instead of EACCES here

* Fix prot setting in ProtectBytes

* Handle some extra protection-related edge cases.

Real hardware treats read and write as separate perms, but appends read if you call with write-only (this is visible in VirtualQuery calls)

Additionally, execute permissions are ignored when protecting dmem mappings.

* Properly handle exec permission behavior for memory pools

Calling sceKernelMemoryPoolCommit with executable permissions returns EINVAL, mprotect on pooled mappings ignores the exec protection.

* Clang

* Allow execution protection for direct memory

Further hardware tests show that the dmem area is actually executable, this permission is just hidden from the end user.

* Clang

* More descriptive assert message

* Align address and size in mmap

Like most POSIX functions, mmap aligns address down to the nearest page boundary, and aligns address up to the nearest page boundary.
Since mmap is the only memory mapping function that doesn't error early on misaligned length or size, handle the alignment in the libkernel code.

* Clang

* Fix valid flags

After changing the value, games that specify just CpuWrite would hit the error return.

* Fix prot conversion functions

The True(bool) function returns true whenever value is greater than 0. While this rarely manifested before because of our wrongly defined CpuReadWrite prot, it's now causing trouble with the corrected values.
Technically this could've also caused trouble with games mapping GpuRead permissions, but that seems to be a rare enough use case that I guess it never happened?

I've also added a warning for the case where `write & !read`, since we don't properly handle write-only permissions, and I'm not entirely sure what it would take to deal with that.

* Fix some lingering dmem issues

ReleaseDirectMemory was always unmapping with the size parameter, which could cause it to unmap too much. Since multiple mappings can reference the same dmem area, I've calculated how much of each VMA we're supposed to unmap.
Additionally, I've adjusted the logic for carving out the free dmem area to properly work if ReleaseDirectMemory is called over multiple dmem areas.

Finally, I've patched a bug with my code in UnmapMemory.

src/core/memory.h

@@ -30,7 +30,8 @@ namespace Core {
 enum class MemoryProt : u32 {
     NoAccess = 0,
     CpuRead = 1,
-    CpuReadWrite = 2,
+    CpuWrite = 2,
+    CpuReadWrite = 3,
     CpuExec = 4,
     GpuRead = 16,
     GpuWrite = 32,
@@ -239,7 +240,7 @@ public:
 
     s32 Protect(VAddr addr, u64 size, MemoryProt prot);
 
-    s64 ProtectBytes(VAddr addr, VirtualMemoryArea vma_base, u64 size, MemoryProt prot);
+    s64 ProtectBytes(VAddr addr, VirtualMemoryArea& vma_base, u64 size, MemoryProt prot);
 
     s32 VirtualQuery(VAddr addr, s32 flags, ::Libraries::Kernel::OrbisVirtualQueryInfo* info);