Build Modes

Invocation of the HVM build mode

flowchart LR

xenguest_main("
    <tt>xenguest
    --mode hvm_build
    /
    --mode pvh_build
    /
    --mode pv_build
</tt>+<tt>
domid
mem_max_mib
mem_start_mib
image
store_port
store_domid
console_port
console_domid")
    -->  do_hvm_build("<tt>do_hvm_build()</tt> for HVM
    ") & do_pvh_build("<tt>do_pvh_build()</tt> for PVH")
         --> stub_xc_hvm_build("<tt>stub_xc_hvm_build()")

xenguest_main --> do_pv_build(<tt>do_pvh_build</tt> for PV) -->
    stub_xc_pv_build("<tt>stub_xc_pv_build()")

click do_pv_build
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L575-L594" _blank
click do_hvm_build
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L596-L615" _blank
click do_pvh_build
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L617-L640" _blank
click stub_xc_hvm_build
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L2329-L2436" _blank

Walk-through of the HVM build mode

The domain build functions stub_xc_hvm_build() and stub_xc_pv_build() call these functions:

  • get_flags() to get the platform data from the Xenstore for filling out the fields of struct flags and struct xc_dom_image.
  • configure_vcpus() which uses the platform data from the Xenstore:
    • When platform/vcpu/<vcpu-num>/affinity is set: set the vCPU affinity. By default, this sets the domain’s node_affinity mask (NUMA nodes) as well. This configures get_free_buddy() to prefer memory allocations from this NUMA node_affinity mask.
    • If platform/vcpu/weight is set, the domain’s scheduling weight
    • If platform/vcpu/cap is set, the domain’s scheduling cap (%cpu time)
  • xc_dom_boot_mem_init() to call <domain_type>_build_setup_mem(),

Call graph of do_hvm_build() with emphasis on information flow:

flowchart TD

do_hvm_build("<tt>do_hvm_build()</tt> for HVM")
    --> stub_xc_hvm_build("<tt>stub_xc_hvm_build()</tt>")

get_flags("<tt>get_flags()</tt>") --"VM platform_data from XenStore"
    --> stub_xc_hvm_build

stub_xc_hvm_build
    --> configure_vcpus("configure_vcpus()")

configure_vcpus --"When<br><tt>platform/
                   vcpu/%d/affinity</tt><br>is set"
    --> xc_vcpu_setaffinity

configure_vcpus --"When<br><tt>platform/
                   vcpu/cap</tt><br>or<tt>
                   vcpu/weight</tt><br>is set"
    --> xc_sched_credit_domain_set

stub_xc_hvm_build
    --"struct xc_dom_image, mem_start_mib, mem_max_mib"
        --> hvm_build_setup_mem("hvm_build_setup_mem()")
            -- "struct xc_dom_image
                with
                optional vmemranges"
                    -->  xc_dom_boot_mem_init

subgraph libxenguest
    xc_dom_boot_mem_init("xc_dom_boot_mem_init()")
        -- "struct xc_dom_image
            with
            optional vmemranges" -->
                meminit_hvm("meminit_hvm()")
                    -- page_size(1GB,2M,4k, memflags: e.g. exact) -->
                        xc_domain_populate_physmap("xc_domain_populate_physmap()")
end

subgraph direct xenguest hypercalls
    xc_vcpu_setaffinity("xc_vcpu_setaffinity()")
    --> vcpu_set_affinity("vcpu_set_affinity()")
        --> domain_update_node_aff("domain_update_node_aff()")
            -- "if auto_node_affinity
                is on (default)"--> auto_node_affinity(Update dom->node_affinity)

    xc_sched_credit_domain_set("xc_sched_credit_domain_set()")
end

click do_hvm_build
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L596-L615" _blank
click xc_vcpu_setaffinity "../../../../../lib/xenctrl/xc_vcpu_setaffinity/index.html" _blank
click vcpu_set_affinity
"https://github.com/xen-project/xen/blob/e16acd806/xen/common/sched/core.c#L1353-L1393" _blank
click domain_update_node_aff
"https://github.com/xen-project/xen/blob/e16acd806/xen/common/sched/core.c#L1809-L1876" _blank
click stub_xc_hvm_build
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L2329-L2436" _blank
click hvm_build_setup_mem
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L2002-L2219" _blank
click get_flags
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L1164-L1288" _blank
click configure_vcpus
"https://github.com/xenserver/xen.pg/blob/65c0438b/patches/xenguest.patch#L1297" _blank
click xc_dom_boot_mem_init
"https://github.com/xen-project/xen/blob/e16acd806/tools/libs/guest/xg_dom_boot.c#L110-L125"
click meminit_hvm
"https://github.com/xen-project/xen/blob/e16acd806/tools/libs/guest/xg_dom_x86.c#L1348-L1648"
click xc_domain_populate_physmap
"../../../../../lib/xenctrl/xc_domain_populate_physmap/index.html" _blank
click auto_node_affinity
"../../../../../lib/xenctrl/xc_domain_node_setaffinity/index.html#flowchart-in-relation-to-xc_set_vcpu_affinity" _blank

The function hvm_build_setup_mem()

For HVM domains, hvm_build_setup_mem() is responsible for deriving the memory layout of the new domain, allocating the required memory and populating for the new domain. It must:

  1. Derive the e820 memory layout of the system memory of the domain including memory holes depending on PCI passthrough and vGPU flags.
  2. Load the BIOS/UEFI firmware images
  3. Store the final MMIO hole parameters in the Xenstore
  4. Call the libxenguest function xc_dom_boot_mem_init() (see below)
  5. Call construct_cpuid_policy() to apply the CPUID featureset policy

It starts this by:

  • Getting struct xc_dom_image, max_mem_mib, and max_start_mib.
  • Calculating start and size of lower ranges of the domain’s memory maps
    • taking memory holes for I/O into account, e.g. mmio_size and mmio_start.
  • Calculating lowmem_end and highmem_end.

It then calls xc_dom_boot_mem_init():

The function xc_dom_boot_mem_init()

hvm_build_setup_mem() calls xc_dom_boot_mem_init() to allocate and populate the domain’s system memory:

flowchart LR
subgraph xenguest
hvm_build_setup_mem[hvm_build_setup_mem#40;#41;]
end
subgraph libxenguest
hvm_build_setup_mem --vmemranges--> xc_dom_boot_mem_init[xc_dom_boot_mem_init#40;#41;]
xc_dom_boot_mem_init -->|vmemranges| meminit_hvm[meninit_hvm#40;#41;]
click xc_dom_boot_mem_init "https://github.com/xen-project/xen/blob/39c45c/tools/libs/guest/xg_dom_boot.c#L110-L126" _blank
click meminit_hvm "https://github.com/xen-project/xen/blob/39c45c/tools/libs/guest/xg_dom_x86.c#L1348-L1648" _blank
end

Except error handling and tracing, it only is a wrapper to call the architecture-specific meminit() hook for the domain type:

rc = dom->arch_hooks->meminit(dom);

For HVM domains, it calls meminit_hvm() to loop over the vmemranges of the domain for mapping the system RAM of the guest from the Xen hypervisor heap. Its goals are:

  • Attempt to allocate 1GB superpages when possible
  • Fall back to 2MB pages when 1GB allocation failed
  • Fall back to 4k pages when both failed

It uses xc_domain_populate_physmap() to perform memory allocation and to map the allocated memory to the system RAM ranges of the domain.

For more details on the VM build step involving xenguest and Xen side see: https://wiki.xenproject.org/wiki/Walkthrough:_VM_build_using_xenguest