CMA (Contiguous Memory Allocator) in Linux is a memory management mechanism designed to provide large contiguous blocks of physical memory for specific use cases, such as DMA (Direct Memory Access) operations or device drivers that require continuous memory regions. When discussing CMA (Contiguous Memory Allocator), it's crucial to focus on physical memory*rather than virtual memory due to the specific requirements of devices and hardware components that rely on direct access to memory. physical memory is referenced instead of virtual memory in the context of CMA.
Purpose:
Some hardware, like certain device drivers or subsystems (e.g., graphics or networking devices), need large chunks of physically contiguous memory. However, as Linux uses a virtual memory system, physical memory can become fragmented over time. CMA ensures that these devices get the required memory, even if the system is fragmented.
How it Works:
- CMA reserves a portion of memory at boot time, which can later be allocated in contiguous blocks when requested.
- During normal system operation, this reserved memory isn't locked away—it can be used for general purpose allocations. However, when a contiguous allocation request comes, this memory is freed and given to the device or driver that requested it.
Device and Hardware Requirements:
- Certain hardware components (like GPUs, network cards, or other peripherals) require contiguous blocks of physical memory for DMA (Direct Memory Access) operations or other high-performance activities.
- DMA is a process where devices communicate directly with the physical memory without the CPU's intervention. For DMA to work efficiently, the hardware needs physically contiguous memory, meaning that the memory addresses are adjacent in physical memory.
- Virtual memory is designed to make efficient use of available memory for software processes, but virtual memory can be fragmented and non-contiguous in physical memory. This is because virtual memory maps logical addresses to scattered physical memory locations.
CMA Guarantees Physical Contiguity:
The key feature of CMA is that it reserves a contiguous region of physical memory that can be allocated on demand. Virtual memory, on the other hand, is not necessarily contiguous in physical terms.
- Even though processes use virtual addresses (which are convenient for applications), the underlying devices or drivers that require DMA or large contiguous blocks of memory need physical addresses, and CMA ensures that this need is met.
Physical vs. Virtual Memory:
- Physical Memory: Refers to the actual RAM installed in the system. It's where data is physically stored, and it must be contiguous for hardware operations.
- Virtual Memory: Is an abstraction provided by the operating system that allows applications to use more memory than physically available. It is divided into pages, which can be scattered across different locations in physical memory.
For example, a 4 GB virtual address space could be mapped to non-contiguous physical memory chunks. However, if a device needs to access a block of memory directly through DMA, the physical memory it accesses must be contiguous.
Virtual Memory Can’t Be Used Directly for DMA or Hardware Access:
- Virtual memory is designed for software abstraction and can be fragmented across the physical memory. This is fine for applications but unsuitable for devices that require access to memory in a sequential physical block.
- When devices perform DMA, they must work with real physical addresses. Therefore, allocating memory in virtual space doesn’t meet the requirement unless the physical memory behind those virtual addresses is also contiguous, which is why CMA allocates from physical memory directly.
How CMA Works with Physical Memory:
- CMA reserves a chunk of physical memory at boot time that can later be allocated in contiguous blocks when requested by the device drivers. It does so to ensure that even when the system’s physical memory becomes fragmented, there will still be a large contiguous block of physical memory available for hardware components.
- Although this memory is allocated from the physical address space, it can be used by virtual memory applications when it's not being actively used by a device.
CMA deals with physical memory because certain devices and hardware require contiguous blocks of physical RAM for tasks like DMA. Virtual memory, which can be fragmented across different physical locations, doesn't meet the needs of these operations. Physical memory contiguity ensures that devices can perform high-speed data transfers efficiently, whereas virtual memory, though beneficial for applications, cannot guarantee this contiguity.
CMA Allocation and Range:
- CMA typically reserves a contiguous memory range at boot time based on system configuration or kernel parameters. The location and size of the CMA region are either:
- Automatically determined by the kernel based on memory requirements.
- Specified manually using kernel boot parameters.
- CMA memory is reserved in the physical memory address space and is set aside as a separate region from the rest of the memory.
Kernel Boot Parameters:
- The size and location of the CMA region can be controlled via kernel boot parameters, such as:
- `cma=size[M/G]`: Specifies the size of the CMA region. For example, `cma=512M` would reserve 512 MB for CMA.
- `cma_start=address`: Specifies the starting address of the CMA region in the physical memory.
- `cma_end=address`: Specifies the ending address of the CMA region.
Example:
cma=256M cma_start=0x20000000 cma_end=0x30000000
This reserves 256 MB for CMA starting at the physical address `0x20000000`.
Where is CMA Allocated?:
- CMA is allocated during boot time and usually resides in the lower end of the physical memory to ensure that DMA or other hardware requests can access it easily.
- CMA allocations can be made in any part of the memory, but it usually starts from a specific predefined region (if not defined manually).
6. Checking CMA Information in Linux:
You can get details about CMA configuration by looking at certain files in the `/proc` or `/sys` filesystems:
/proc/meminfo: This file contains general memory information, including CMA. You can find the CMA reserved region under the entry `CmaTotal` and the currently used CMA memory under `CmaFree`.
Example:
CmaTotal: 262144 kB
CmaFree: 131072 kB
/sys/kernel/debug/cma: If CMA debugging is enabled, this directory will provide detailed information about the CMA memory allocations.
Example to View CMA Memory: cat /proc/meminfo | grep Cma
Output might look like this
CmaTotal: 524288 kB
CmaFree: 512000 kB
This tells you the total reserved CMA memory (`CmaTotal`) and the currently available CMA memory (`CmaFree`).
Summary:
- CMA is allocated at boot and reserves contiguous memory blocks for devices needing such memory.
- It can be specified using boot parameters (size, start, and end address).
- The reserved memory is used by the system when no contiguous memory requests are made and freed when needed for such operations.
- You can check the allocation and usage through system files like `/proc/meminfo`.
========================FADUMP=========================================
Firmware assisted dump (fadump) is a dump capturing mechanism provided as a reliable alternative to kdump on IBM POWER systems. The fadump utility captures the vmcore file from a fully-reset system with PCI and I/O devices. This mechanism uses firmware to preserve memory regions during a crash and then reuses the kdump userspace scripts to save the vmcore file. The memory regions consist of all system memory contents, except the boot memory, system registers, and hardware Page Table Entries (PTEs).
The fadump mechanism offers improved reliability over the traditional dump type, by rebooting the partition and using a new kernel to dump the data from the previous kernel crash.
README: /usr/share/doc/kexec-tools/fadump-howto.txt
In the Secure Boot environment, the GRUB2 boot loader allocates a boot memory region, known as the Real Mode Area (RMA). The RMA has a size of 512 MB, which is divided among the boot components and, if a component exceeds its size allocation, GRUB2 fails with an out-of-memory (OOM) error.
Options for Using fadump:
fadump=on: This is the default setting for enabling fadump. It reserves memory from a special area called **CMA (Contiguous Memory Allocator)**. Think of this as a memory-saving technique that allows some of this reserved memory to still be used by other parts of the system during normal operation. The idea is to avoid wasting memory that would otherwise sit idle.
fadump=nocma: This option tells the system not to use the special CMA-backed memory for fadump. Instead, it reserves a portion of memory separately and completely, which might be useful if you want to capture more detailed information, like user-level data, during a crash. By not using CMA, this memory is reserved exclusively for fadump and isn't used for other tasks while the system is running.
fadump=on: Imagine you have a spare room (memory) in your house. Normally, you leave it empty just in case you need to store something later (for fadump). But with this setting, you let guests use the room for sleeping when you don’t need it. When something goes wrong (system crash), you ask them to leave so you can use it to store important things (dump data).
fadump=nocma: Now, if you set the option to nocma, it's like keeping that spare room off-limits to guests at all times, so it's always ready for storing important stuff whenever you need it.
fadump=on (default): Allows the reserved memory to be used for other tasks when the system is working normally, saving memory.
fadump=nocma : Keeps the reserved memory off-limits to other tasks, ensuring that it's available for storing more detailed data during a crash.
The system with SLES distro will automatically choose whether to use `fadump=nocma` or `fadump=on`, depending on the KDUMP_DUMPLEVEL setting. On RHEL based systems , you can set fadump=on /fadump=nocma using grubby command followd by reboot. Or else you can add "/etc/default/grub" file to add these options and run "grub2-mkconfig -o /boot/grub2/grub.cfg"
KDUMP_DUMPLEVEL determines how much information is captured in a system crash. If it’s set to exclude user pages, the system will automatically use `fadump=on` (the default behavior). But, if user pages are **included** in the dump, it will switch to `fadump=nocma`.
