Introduction
As server architectures continue to grow in scale, modern systems can contain hundreds of processor cores and terabytes of memory. Managing memory efficiently becomes critical to achieving optimal performance. This is where NUMA (Non-Uniform Memory Access) architecture comes into play.
In a NUMA-based system, processors and memory are organised into groups called NUMA nodes. Each NUMA node consists of a set of CPU cores and memory that are physically closer to each other. Accessing local memory within the same NUMA node is significantly faster than accessing memory located in another node. This design helps improve scalability and reduces memory access latency in large enterprise servers such as IBM Power systems.
Understanding NUMA nodes and NUMA topology is essential for system administrators, performance engineers, virtualization architects, and Linux developers, especially when working with large configurations, virtualization technologies, and memory-intensive workloads.
NUMA Node: A group of CPUs and memory that are physically close to each other.
NUMA Topology: The layout/map showing how all NUMA nodes are connected inside a server.
For your PowerVM work, think of NUMA as "keeping the CPU and its memory in the same neighbourhood so they can talk faster."
NUMA Node = CPU neighbourhood + its local memory.
Best performance occurs when a workload's CPUs and memory stay in the same neighbourhood. When CPUs must fetch data from another neighbourhood, performance drops due to remote memory access.
Step 1: Understanding the Diagram
The diagram shows:
- 256 CPU cores
- 1 TB Memory
- 16 NUMA Nodes
- Each NUMA node contains:
- 16 CPU cores
- 64 GB Memory
NUMA Node Layout
|
NUMA Node |
CPU Cores |
Memory |
|
Node 0 |
0-15 |
64 GB |
|
Node 1 |
16-31 |
64 GB |
|
Node 2 |
32-47 |
64 GB |
|
Node 3 |
48-63 |
64 GB |
|
Node 4 |
64-79 |
64 GB |
|
Node 5 |
80-95 |
64 GB |
|
Node 6 |
96-111 |
64 GB |
|
Node 7 |
112-127 |
64 GB |
|
Node 8 |
128-143 |
64 GB |
|
Node 9 |
144-159 |
64 GB |
|
Node 10 |
160-175 |
64 GB |
|
Node 11 |
176-191 |
64 GB |
|
Node 12 |
192-207 |
64 GB |
|
Node 13 |
208-223 |
64 GB |
|
Node 14 |
224-239 |
64 GB |
|
Node 15 |
240-255 |
64 GB |
Step 2: Physical Interpretation
Imagine the server has 4 processor sockets.
Socket 0
├── NUMA 0
├── NUMA 1
├── NUMA 2
└── NUMA 3
Socket 1
├── NUMA 4
├── NUMA 5
├── NUMA 6
└── NUMA 7
Socket 2
├── NUMA 8
├── NUMA 9
├── NUMA 10
└── NUMA 11
Socket 3
├── NUMA 12
├── NUMA 13
├── NUMA 14
└── NUMA 15
Each NUMA node has its own local memory.
Step 3: How Workloads Get Mapped
Example 1: Database Workload
Suppose Oracle or PostgreSQL starts.
Linux scheduler may place:
Database Process
|
+--> CPUs 0-15
|
+--> Memory from Node 0
Result:
CPU ---> Local Memory
Fastest possible access ✅
Example 2: Java Application
Suppose JVM requires:
- 32 cores
- 128 GB memory
Linux may place it across:
Node 4
Node 5
CPUs 64-95
Memory 128 GB
This is still good because nodes 4 and 5 are neighbours.
Example 3: SAP HANA
Large memory workload:
64 Cores
256 GB Memory
Linux may distribute across:
Node 8
Node 9
Node 10
Node 11
Socket 2
Everything remains inside one socket.
Performance is excellent.
Step 4: Bad Mapping Example
Suppose workload threads run here:
CPU Node 0
but memory gets allocated here:
Memory Node 15
Diagrammatically:
CPU Core 2
|
|
+---------------------------+
|
V
Memory Node 15
Now every memory access crosses multiple interconnects.
Effects:
- Higher latency
- Lower throughput
- More cache misses
- Poor benchmark numbers
Step 5: LPM
Before migration:
LPAR
CPUs -> Node 0
Memory -> Node 0
Everything local.
CPU 0
|
+--> Memory 0
After migration (bad case):
CPU 0
|
+-------------------> Memory Node 12
Remote memory access.
Application slows down.
Step 6: Example MaxConfig Workload Placement
Workload A (Kernel Build)
16 Cores
32 GB Memory
Placement:
NUMA Node 0
Workload B (Stress-ng)
32 Cores
64 GB Memory
Placement:
NUMA Node 1
NUMA Node 2
Workload C (Database)
64 Cores
256 GB Memory
Placement:
NUMA Node 4
NUMA Node 5
NUMA Node 6
NUMA Node 7
Workload D (AI/ML)
128 Cores
512 GB Memory
Placement:
NUMA Node 8-15
How Linux Decides
You can view actual placement using:
numactl --hardware
Show NUMA layout.
numastat -p <pid>
Show memory used by each NUMA node.
numactl --show
Show current NUMA policy.
lscpu | grep NUMA
Show NUMA configuration.
In simple terms: NUMA is all about keeping processors and their data in the same neighbourhood so they can communicate faster and work more efficiently
Conclusion
NUMA architecture plays a crucial role in the performance and scalability of modern multi-socket servers. By grouping processors and memory into NUMA nodes, systems can provide faster local memory access and reduce communication overhead across the server. The NUMA topology serves as a roadmap that describes how these nodes are interconnected and how data flows between them.
For large Power systems and MaxConfig environments, understanding NUMA behaviour is essential for workload placement, performance tuning, virtualization, and Live Partition Mobility (LPM). Keeping CPU threads and memory within the same NUMA node whenever possible helps minimise latency and maximise application performance. As server configurations continue to grow in size and complexity, NUMA awareness becomes a key factor in achieving efficient and predictable system performance.
