Repository navigation
xen/dom0: refactor dom0 memory growth to avoid splitting based on max - #59
alexandermerritt wants to merge 7 commits into
Conversation
This reverts commit 0a2ac17. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
This reverts commit 8788269. We no longer want to split the unpopulated boot-time space between the dom0 balloon and grant and foreign mappings. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
This reverts commit 7498978. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
Consolidate existing duplicated code. No functional change. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
The boot-time unbacked address space reaches the unpopulated allocator (page_list) one 4 KiB page at a time, with a lock round trip and a list push for each page. On a large dom0 that is millions of round trips at boot. page_list also is unaware of page contiguity, limiting dom0 balloon growth to 4 KiB updates. Use gen_pool to manage this boot-time address space, with pfns for addresses. gen_pool performs coalescing for us. Grants and foreign mappings still take pages from page_list first. Carved ZONE_DEVICE sections stay on page_list, and a page is returned to where it came from by its zone. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
With no pages parked on the balloon, dom0 grows by hotplugging a memory section at the first free physical address. That address space can belong to a PCI device that has not claimed it yet, and Xen does not stop dom0 from growing when dom0_mem sets no max:. dom0's balloon needs room to grow into, and grows faster when Xen populates a whole 2 MiB extent at a time. The unpopulated allocator keeps that room, the boot-time unbacked address space. Grant and foreign mappings and the balloon have to share it. Use the gen_pool to feed the dom0 balloon when it is instructed to grow. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
Add `info/room_kb` to the sysfs of the xen balloon driver, so user tools can discover the amount of memory dom0 can grow by. Reported are only complete 2MiB blocks in the unpopulated allocator. Signed-off-by: Alexander M. Merritt <alexander@edera.dev>
alexandermerritt
changed the title
max
| * they rarely need fill_list() to carve a region out of address space that | ||
| * may belong to a device. | ||
| */ | ||
| #define MAPPING_FLOOR_PAGES PAGES_PER_SECTION |
There was a problem hiding this comment.
IIUC the mapping space is shared with other stuff besides the balloon.
If we run out of mapping space, what happens? This implies fill_list() is used, but couldn't that be PCI space in some cases and wouldn't that be Bad?
There was a problem hiding this comment.
This change set is not strictly splitting the memory, rather it is separating it into two pools: 2mb blocks (gen-alloc) and single pages (page_list). Balloon allocates only from the 2mb blocks, and the grant/foreign mappings take first from the single pages, then from remaining 2mb blocks. So the latter have access to all the remaining memory areas after the balloon grows (if at all).
fill_list is part of the existing code and carries a risk of overlap, yes.
Are you wondering if the balloon using this memory is increasing the risk that fill_list gets invoked?
| static struct page *take_page(int node) | ||
| { | ||
| struct page *pg = page_list[node]; | ||
| unsigned long pfn = 0; | ||
|
|
||
| if (pg) { | ||
| page_list[node] = pg->zone_device_data; | ||
| list_count[node]--; | ||
| return pg; | ||
| } | ||
|
|
||
| if (boot_pool[node]) | ||
| pfn = gen_pool_alloc(boot_pool[node], 1); | ||
| return pfn ? pfn_to_page(pfn) : NULL; | ||
| } |
There was a problem hiding this comment.
this is a considerably more expensive scan under lock than before, correct?
There was a problem hiding this comment.
yea you're right, and since all the boot mem goes to the gen-alloc first, every mapping alloc gets hit.
I'll see if we can spill some blocks to the page_list when it becomes empty, to try to amortize it.
| int node; | ||
|
|
||
| mutex_lock(&list_lock); | ||
| for (node = 0; node < MAX_NUMNODES && nr < nr_blocks; node++) { |
There was a problem hiding this comment.
AFAIK this fills all of NUMA node 0 before it touches node 1, so I don't think (could be wrong) that this plays particularly nicely with NUMA?
There was a problem hiding this comment.
For my own understanding -
There's an arch/arm/xen/enlighten.c - shouldn't it get the same/equiv change?
Description
Part of COR-141
Prior work in ENG-99 split unbacked boot-time memory using the
max:parameter indom0_memfrom Xen between the dom0 balloon driver and grant / foreign mappings. This makes it hard to remain flexible across varied deployments.Remove the implied function of
max:from splitting this memory, and add ranges to an existing range map allocator (gen-alloc) alongside the existing page list. Balloon and mappings share this unified memory range: the balloon consumes super pages found in the gen-alloc instance, and mappings take frompage_list.Testing
Boot test under QEMU (TCG, PV dom0, edera/4.22 Xen,
dom0_mem=512M, nomax:).room_kbandfree -mchange accordingly.room_kbwas 0 after, and nothing was left parked on the balloon.Not tested: PVH dom0, grant and foreign mappings under load, more than one NUMA node, arm64 dom0.