Comprehensive Guide to Linux Kernel Selftests

The Linux kernel includes a powerful suite of self-tests found under the `tools/testing/selftests/` directory. These self-tests provide a framework for developers to verify specific code paths within the kernel, ensuring the robustness and reliability of individual components. This guide covers how to set up, run, and customize these tests to suit different testing needs.

Overview of Kernel Self-Tests

Kernel self-tests serve as essential tools for validating new code additions or changes to the Linux kernel. They can be run post-kernel build, installation, and boot to validate that code changes do not introduce regressions or unexpected behavior. One critical feature within these self-tests is the hot-plug testing, which allows developers to simulate and verify the dynamic plugging and unplugging of CPUs and memory.

Hot-Plug Tests: Limited and Full Modes

Hot-plug testing, by nature, can be complex due to dependencies on hardware readiness for CPU and memory offlining. To mitigate potential system hangs during these tests, the kernel provides:

Limited Mode: CPU hot-plug is tested on a single CPU, and memory hot-plug is restricted to 2% of available hot-plug-capable memory.

Full Mode: This extensive test mode performs hot-plug actions across all hot-plug capable CPUs and 10% of the memory. This mode can be useful for systems where comprehensive validation of hot-plug readiness is required.

Building and Running Self-Tests

$ make -C tools/testing/selftests

$ make -C tools/testing/selftests run_tests

Alternatively, to both build and run in a single command, use:

$ make kselftest

Note: Some tests, particularly those requiring hardware access, may need root privileges.

Running Specific Self-Tests

Kernel self-tests offer flexibility for targeting specific components or subsystems. To run only the tests related to a specific subsystem, use the `TARGETS` variable:

For example, to run only `ptrace` tests:

$ make -C tools/testing/selftests TARGETS=ptrace run_tests

$ make TARGETS="size timers" kselftest

Refer to the `Makefile` located at `tools/testing/selftests/Makefile` for a complete list of possible test targets.

Running Full Range Hot-Plug Self-Tests

For systems requiring full-scale hot-plug testing, the following commands build and execute the tests across all hot-plug-capable resources.

$ make -C tools/testing/selftests hotplug

$ make -C tools/testing/selftests run_hotplug

 Installing Kernel Self-Tests

To simplify running self-tests on other systems, the `kselftest_install.sh` script allows installation to a default or user-specified location.

Default Installation

$ cd tools/testing/selftests

$ ./kselftest_install.sh

Custom Installation Location

Specify an installation directory for easier management and accessibility:

$ cd tools/testing/selftests

$ ./kselftest_install.sh <install_dir>

Running Installed Self-Tests

After installing, run the installed tests using the provided `run_kselftest.sh` script, which executes all self-tests in the specified installation directory.

$ cd kselftest

$ ./run_kselftest.sh

---------------------------------------

Testing BPF (Berkeley Packet Filter) and fprobe features within Linux kernel self-tests involves several additional steps because these tests require specific kernel configurations and privileges to ensure proper operation. Here’s a breakdown of the steps required:

Enable Required Kernel Configurations

To run BPF and fprobe self-tests, ensure the kernel has the necessary configurations enabled. These settings are often found in the kernel's `.config` file before building the kernel.

CASE 1 : For BPF Self-Tests

Enable the following configurations:

- `CONFIG_BPF`: Enable BPF support

- `CONFIG_BPF_SYSCALL`: Enable BPF syscall interface

- `CONFIG_BPF_JIT`: Enable BPF Just-In-Time (JIT) compiler

- `CONFIG_HAVE_EBPF_JIT`: Enable extended BPF JIT support

- `CONFIG_DEBUG_INFO_BTF`: Enable BPF Type Format (BTF), which is often required for BPF self-tests

- `CONFIG_NET`: Enable networking stack if testing network-related BPF functionality

If running BPF tracing , make sure the following options are also enabled:

- `CONFIG_TRACING`: Enable kernel tracing

- `CONFIG_FUNCTION_TRACER`: Enable function tracing (for eBPF tracing features)

- `CONFIG_BPF_EVENTS`: Enable BPF events for tracing

CASE2: For fprobe Self-Tests

fprobe relies on certain configurations, particularly in the tracing subsystem:

- `CONFIG_KPROBES`: Enable kernel probes

- `CONFIG_FPROBE`: Enable fprobe functionality, which is typically under kernel debugging and tracing options

- `CONFIG_KPROBE_EVENTS`: Enable kprobe events (needed for fprobe usage and testing)

Use `make menuconfig` or `make nconfig` in your kernel source directory to enable these options interactively.

--------------------------------------------

Recompile and Boot into the Custom Kernel

After setting the configurations, recompile the kernel to include the BPF and fprobe functionality, then reboot into this kernel.

$ make -j$(nproc) && make modules_install && make install

$ reboot

Install Required User-Space Utilities

Some BPF and fprobe tests may require user-space tools for BPF program management or tracing. Ensure these tools are installed:

1. bpftool: Used to inspect and manage BPF programs and maps.
2. iproute: Provides tools like `tc` for attaching BPF programs to network traffic.

 

 Run BPF and fprobe Self-Tests : Navigate to the `tools/testing/selftests/` directory in your kernel source and specify the `TARGETS` for BPF and fprobe:

Running BPF Tests  

$ make -C tools/testing/selftests TARGETS=bpf run_tests

BPF self-tests are designed to cover various aspects, such as BPF JIT, network socket filters, and tracing. Some BPF tests may require root privileges, so consider running with `sudo` if you encounter permission errors.

Running fprobe Tests

$ make -C tools/testing/selftests TARGETS=fprobe run_tests

The fprobe tests are generally under the tracing or ftrace section. Since fprobe works closely with kprobes and tracing functionalities, you may see additional output related to probe events.

Troubleshooting Common Issues

Missing Kernel Configurations: If tests fail with messages about unsupported configurations, recheck that all necessary kernel options are enabled.

Permission Issues: Some BPF programs and fprobes require privileged access. Run the tests with `sudo` if needed.

BTF (BPF Type Format) Requirements: Some BPF tests require BTF data, which may need to be enabled explicitly in `.config` (`CONFIG_DEBUG_INFO_BTF`).

By following these steps, you should be able to run BPF and fprobe tests as part of the Linux kernel self-tests, verifying their functionality and stability within your custom-built kernel.

Reference:

https://www.kernel.org/doc/Documentation/kselftest.txt

Overview of the Linux Test Project (LTP) and Submitting Patches to the Mailing List

The Linux Test Project (LTP) is a collaborative initiative aimed at enhancing the reliability, robustness, and stability of the Linux kernel. This project was initiated by SGI, OSDL, and Bull, and is currently developed and maintained by a consortium of major technology companies, including SUSE, Red Hat, Fujitsu, IBM, Cisco, and Oracle. The primary objective of LTP is to provide a comprehensive suite of tests that serve the open-source community.

Goals and Objectives

LTP's primary objective is to provide a collection of automated tools to rigorously test the Linux kernel and associated system libraries. The test suites focus on various kernel components, including file systems, networking, memory management, and system calls. By using LTP, developers can gain valuable insights into potential bugs, stability issues, or areas for performance optimization.

Key Components of LTP:

1. Test Suites: LTP offers several specialized test suites, including:
2.  Syscalls: Verifies the implementation and response of different system calls.
3.  File Systems: Tests various file systems for integrity, performance, and stability.
4.  Networking: Assesses network protocol handling, performance, and error management.
5.  Security: Evaluates the kernel’s security features, enforcing secure data handling and user permissions.

The main goals of LTP include:

1. Testing Automation: Automating the testing process to improve the quality of the Linux kernel and its associated system libraries.
2. Reliability Validation: Ensuring that the Linux kernel can handle various workloads without failure.
3. Robustness Assessment: Testing the kernel's ability to recover from errors and handle unexpected situations gracefully.

Testing Suites : LTP consists of a collection of tools designed specifically for testing the Linux kernel and related features. These testing suites cover a wide range of scenarios to validate different aspects of kernel performance. 

 Key Features:

1. Comprehensive Coverage: The tests are designed to cover various functionalities within the kernel.
2. Stress Testing: Some tests are intended to stress specific components of the system, which can help identify potential issues before they affect production environments.

How LTP Works: The LTP testing workflow follows these primary steps:

1. Initialization: Set up the test environment and prerequisites, configuring the necessary resources.

2. Test Selection: Choose relevant tests based on the kernel area under evaluation—such as file systems, network protocols, or memory management.

3. Test Execution: Run the selected tests on the system to validate the kernel's behavior and performance.

4. Result Collection: Gather data from each test run, noting any failures, errors, or performance metrics.

5. Reporting: Summarize test results with detailed logs and reports for analysis.

Important Warning

It is crucial to note that LTP tests should not be run on production systems. Certain tests like "growfiles, doio, and iogen" are particularly aggressive as they stress the I/O capabilities of the system. These tests are designed to discover or even induce problems, making them unsuitable for environments where stability is critical.

Contribution and Community

LTP is an open-source project that encourages contributions from developers around the world. Users can report issues, submit pull requests, and participate in discussions regarding improvements and new features. The project maintains an active GitHub repository where users can access the latest code, documentation, and testing results.

How to Subscribe: Visit the Mailing List Subscription Page: Go to the LTP Mailing List Info Page to find details about the mailing list. Fill Out Subscription Form: You will need to provide your email address and follow the instructions to complete your subscription.

https://lists.linux.it/listinfo/ltp

This is a mailing list for the Linux Test Project. The list is used for patches, bug reports as well as general discussion about LTP and Linux testing.

Sending Patches: Prepare Your Patch. Make sure your patch is ready for submission. It's advisable to test your patch using GitHub Actions to ensure it compiles cleanly across various architectures and distributions. Use git send-email: The preferred method for sending patches is using git send-email. This will format your patch correctly for the mailing list. Once prepared, send your patch to the mailing list (ltp@lists.linux.it).

To post a message to all the list members, send email to ltp@lists.linux.it

LWN.net is a subscriber-supported publication that focuses on providing in-depth coverage of the Linux and free software development communities. LWN relies on subscriptions to fund its operations, as traditional advertising models have proven ineffective for their needs. Subscribing helps ensure the continuity of their publication. https://lwn.net/Articles/708182/#t

Test case tutorial:

This is a step-by-step tutorial on writing a simple C LTP test, where topics of the LTP and Linux kernel testing will be introduced gradually using a concrete example. Please refer this link

https://linux-test-project.readthedocs.io/en/latest/developers/test_case_tutorial.html

The list of syscalls which are tested under testcases/kernel/syscalls:

• kernel syscalls: 365
• tested syscalls: 341

https://linux-test-project.readthedocs.io/en/latest/users/stats.html

source

NOTE:

Cross-Compilation Process: 

1. Source Code Compilation: process begins with compiling source files (e.g., .c or .cpp) into object files (.o). This is done using a cross-compiler that targets the architecture of the device for which the code is intended
2. Linking: After generating object files, a linker combines them into a single executable file, such as  .ELF (or main.axf). This step may also involve using a linker script that defines memory regions and other configurations specific to the target device1.
3. Conversion: Once the ELF(or .axf) file is created, it can be converted into other formats like .bin, which are suitable for uploading to the target hardware.

For the ppc64le (PowerPC 64-bit Little Endian) architecture ( the equivalent of the .axf** format used in ARM cross-compilation) is typically the ELF (Executable and Linkable Format). The ELF format is widely used for executable files, object code, shared libraries, and core dumps across various architectures, including ppc64le. When cross-compiling for ppc64le, the output binary files are usually in ELF format, which can be identified by the .elf extension or simply no extension at all.

Toolchain Setup: To cross-compile for ppc64le, you need to install a suitable toolchain. A toolchain is a set of programming tools that work together to facilitate the development, building, and deployment of software applications. This typically includes compilers, linkers, debuggers, and libraries that are executed in a sequence where the output of one tool serves as the input for the next. This architecture, developed by the AIM alliance (Apple, IBM, and Motorola), is based on the RISC (Reduced Instruction Set Computing) principles and supports both 32-bit and 64-bit processing.

 Example : install gcc-powerpc64le-linux-gnu

Compiling and Linking: Use the cross-compiler to compile source files:

 powerpc64le-linux-gnu-gcc -o output_binary source.c

This command generates an ELF executable named "output_binary".

 file output_binary

The output should indicate that it is an ELF file for PowerPC architecture and the resulting ELF binaries can be transferred to a ppc64le system for execution. This format allows developers to create executables that are compatible with PowerPC systems, facilitating software development in embedded and server environments.

Tests setup:

The internal LTP library provides a set of features that permits to customize tests behavior by setting environment variables and using specific tests arguments.

Library environment variables:

Following environment variables are expected to be set by LTP users. Therefore, with some exceptions, they have LTP_ prefix. Environment variables with TST_ prefix are used inside LTP shell API and should not be set by users.

https://linux-test-project.readthedocs.io/en/latest/users/setup_tests.html

Installation and tests execution:  Basics requirements to build LTP are the following:

• git
• autoconf
• automake
• make
• gcc
• m4
• pkgconf / pkg-config
• libc headers
• linux headers
• git-email

git clone --recurse-submodules https://github.com/linux-test-project/ltp.git

cd ltp

make autotools

./configure

Running single tests: 

cd testcases/kernel/syscalls/foo

make

PATH=$PATH:$PWD ./foo01

To compile all tests :

make

# install LTP inside /opt/ltp by default

make install

Running tests: To run all the test suites

cd /opt/ltp

# run syscalls testing suite

./kirk -f ltp -r syscalls

Note: Many test cases have to be executed as root.

Test suites (e.g. syscalls) are defined in the runtest directory. Each file contains a list of test cases in a simple format.

Each test case has its own executable or script that can directly executed:

testcases/bin/abort01

# some tests have arguments

testcases/bin/mesgq_nstest -m none

# vast majority of tests have a help

testcases/bin/ioctl01 -h

# Many require certain environment variables to be set

LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh

Most commonly, the PATH variable needs to be set and also LTPROOT, but there are a number of other variables which usually kirk sets for you.

How to send a patch:

Here are the detailed steps for cloning the Linux Test Project (LTP), adding tests, building, running test cases, and sending patches to the community

Pre-requisite : subscribe to mailing list as mentioned in above section.

1. Check Your Email ID and Username:

   - Verify your Git configuration by running:  

     cat .gitconfig

2. Navigate to the Patch Directory:

   - Change to the directory where your patch is located    

     cd LTP_vxyz_patch/

3. Clone the LTP Repository:

   - Clone the LTP repository from GitHub:

     git clone git@github.com:linux-test-project/ltp.git

4. Change Directory to LTP:

   - Move into the cloned LTP directory    

     cd ltp

5. Build the Project:

   - Execute the build script to compile the project:   

     ./build.sh

6. Navigate to the Test Case Directory:

   - Go to the specific test case directory you want to modify or add tests  

     cd testcases/kernel/mem/hugetlb/hugeshmget

7. Modify/Create a Test Case:

   - Open the test case file in a text editor (e.g., `vi`) and write or modify your test case:

     vi hugeshmget06.c

8. Compile and Check the Test Case:

   - Build and check your test case for any errors:

     make && make check

9. Run the Test Case:

   - Execute your test case to verify its functionality:

     ./hugeshmget06

10. Run with Input Argument:

    - Optionally, run your test case with an input argument for loop of count=5  (e.g., `-i 5`):

      ./hugeshmget06 -i 5

11. Check Git Status:

    - Check which files have been modified or added in your Git repository    

      git status

12. Stage Your Changes:

    - Add your modified files to the staging area for commit:

      git add ../../../../../runtest/hugetlb hugeshmget06.c ../../.gitignore

13. Check Git Status Again:

    - Verify that your changes are staged correctly:   

      git status

14. Commit Your Changes:

    - Commit your changes with a sign-off message:   

      git commit -s

15. Create a Patch File:

    - Generate a patch file for your commit:    

      git format-patch -1

16. Edit the Patch File Before Sending:

    - Open the generated patch file and edit it as necessary (e.g., update version, email IDs):

      vi 0001-Migrating-the-libhugetlbfs-testcases-shm-gettest.c-t.patch

17. Send the Patch via Email:

    - Use `git send-email` to send your patch to the community mailing list and reviewers:

      git send-email --to=ltp@lists.linux.it --cc=reviewer1@xyz.co --cc=reviewer2@abc.co ./0001-Migrating-the-libhugetlbfs-testcases-shm-gettest.c-t.patch

    - Ensure that you fill in the subject line and other email details as prompted.

18. Respond to Review Comments via Email:

    - After receiving feedback, reply to the email addressing any review comments and mention any new versions of patches based on those comments.

By following these steps, you can effectively contribute to the Linux Test Project by adding new tests, verifying their functionality, and submitting patches for review by the community.

Search your Patch in mailing list :

To search details on patch with search key  for example - "libhugetlbfs" or "shmget-test" or by username "sachinpb"

Example 1 : https://lore.kernel.org/ltp/?q=libhugetlbfs

Example 2 : https://lore.kernel.org/ltp/?q=sachinpb

Example 3: https://lore.kernel.org/ltp/ZvqQxP9KVW6PqFOo@yuki.lan/

[LTP] [ANNOUNCE] The Linux Test Project has been released for SEPTEMBER 2024

Let us test the newly added testcase "hugeshmget06 A hugepage shm test"

Step 1 : Clone the LTP repo from https://github.com/linux-test-project/ltp

Step 2 : Confirm the testcase merged with master branch

Step 3 : Check the new testcase "hugeshmget06.c" existing in "/root/ltp/testcases/kernel/mem/hugetlb/hugeshmget"

Step 4 : Build test cases by running script ./build.sh as shown below:

Step 5 : Do make install  and check the binaries at "/root/ltp-install/"

Step 6 : Execute  the testcase ./hugeshmget06 as shown below 

Step 7: Execute the above testcase in loop of any count . For example count=3 

Conclusion

The Linux Test Project plays a vital role in ensuring that the Linux kernel remains reliable and robust for users across various industries. By providing a structured approach to testing, LTP helps developers identify potential issues early in the development process, ultimately contributing to a more stable operating environment for all Linux users.

For more information or to get involved with LTP, you can visit their [GitHub repository](https://github.com/linux-test-project/ltp).

In this blog, we've explored the Linux Test Project (LTP), covering how to add tests and submit patches to the mailing list. I hope you found this information useful and that it enhances your understanding and contributions to LTP. Best of luck on your journey with LTP! 

Reference :

1) https://linux-test-project.readthedocs.io/en/latest/
2) https://linux-test-project.readthedocs.io/en/latest/index.html
3) https://jitwxs.cn/51dc9e04

Git Unlocked:: A Beginner's Guide to Version Control

Git is source code management systems. Its distributed version control system (DVCS) which facilitates multiple developers and teams to work in parallel.  Git’s primary emphasis is on providing speed while maintaining data integrity. Git also provides ability to perform almost all the operations offline, when network is not available. All the changes can be pushed to the server on network’s availability.

source: GIT workflow structure

Git is designed as a distributed version control system, which means:

Data Structure:

Git uses a data structure called a Merkle tree, where each commit points to its parent commit, forming a chain of versions. Each commit has a unique SHA-1 hash that identifies it and its content.

Objects:

There are four main types of objects in Git:

1. Blobs: Store file data.
2. Trees: Represent directories and contain blobs or other trees.
3. Commits: Point to trees and contain metadata such as author, date, and commit message.
4. Tags: Reference specific commits for marking release points24.

Branches:

Branches are pointers to commits, allowing multiple lines of development within the same repository. The default branch is usually called master or main.

Staging Area:

Before committing changes, files are added to the staging area using git add. This allows users to review changes before finalizing them in a commit.

Remote Repositories:

Git allows synchronization between local repositories and remote ones (like GitHub), enabling collaboration among multiple users.

Some of the commonly used terms are listed below :

1. Repository - is directory that contains all the project files.
2. Clone - it creates a working copy of local repository.
3. Branch - is created to encapsulate the development of new features or to fix bugs.
4. HEAD - points to the last commit.
5. Commit - commits changes to HEAD and not to remote repository.
6. Pull - Gets the changes from the remote repository to the local repository.
7. Push - Commits the local repository changes to the remote repository.

---------------------------------------------------------------------------------------------------------

To understand when to use "git fetch" instead of "git pull", let's break it down into simpler terms with an example.

git fetch: Think of this as checking for updates. It retrieves the latest changes from a remote repository (like GitHub) but does not apply those changes to your current working files. It’s like downloading new episodes of your favorite show but not watching them yet.

git pull: This command does two things at once: it fetches the latest changes and  immediately merges them into your current files. It’s like downloading those episodes and then watching them right away.

When to Use "git fetch"

1. You Want to Review Changes First: If you're working on a project and you want to see what others have done before you integrate their changes, `git fetch` allows you to check for updates without making any changes to your own work right away.

2. Avoiding Conflicts : If you have uncommitted changes in your local files, using `git pull` could create conflicts because it tries to merge changes automatically. By using `git fetch`, you can first see what has changed and then decide how to handle it.

3. Working Offline or Later: If you're in a situation where you can't deal with merging right away (like being on a beach without Wi-Fi), you can fetch the updates now and merge them later when you're ready.

Example : Imagine you're working on a team project.

1. You’re writing code on your laptop.

2. Your teammate pushes some updates to the shared repository while you’re still working.

3. You want to see what they changed without disrupting your current work.

  git fetch origin

  This command checks for any updates from the remote repository named "origin" and downloads the latest information about those changes, but it doesn’t change any of your files yet.

4. After fetching, you can review what your teammate did:

   git log origin/main

   This shows you the commit history of the remote branch so you can see what’s new.

5. Once you're ready and have made sure there are no conflicts, you can merge those changes:

   git merge origin/main

   Now your local files are updated with your teammate’s changes.

This way , use "git fetch" when you want to keep your local repository updated without immediately affecting your working files. This gives you more control over how and when to integrate changes from others, especially useful in collaborative environments or when you're not ready to merge yet.

--------------------------------------------------------------------

The recursive clone of a Git repository:

This refers to the process of cloning a repository along with all its submodules. Submodules are essentially repositories nested inside another repository, allowing you to include external libraries or components as part of your project.

How It Works : When you perform a standard clone using the command:

git clone [repository-url]

It only clones the main repository. If that repository contains submodules, those submodule directories will be created but will remain empty until you initialize and update them.

To clone a repository along with its submodules, you use the `--recurse-submodules` option:

git clone --recurse-submodules [repository-url]

This command does the following:

1. Clones the Main Repository: It creates a local copy of the main repository.

2. Initializes Submodules: It automatically initializes any submodules defined in the ".gitmodules" file.

3. Updates Submodules: It fetches the content of those submodules, ensuring they are populated with the correct data.

Example: Imagine you have a project that relies on a library hosted in another Git repository. If you want to include this library as a submodule in your project, your main repository might look like this:

- Main Repository: "MyProject"

  - Submodule: "ExternalLibrary"

If you were to clone MyProject without the recursive option, you'd end up with:

MyProject/

├── .git/

├── ExternalLibrary/  (empty)

└── other files...

with that option 

git clone --recurse-submodules [repository-url]

You would get:

MyProject/

├── .git/

├── ExternalLibrary/  (with content)

└── other files...

Why use Recursive Clone?

1. Convenience: It saves time by automatically fetching and setting up all necessary components for your project.
2. Consistency: Ensures that all required libraries or modules are at the correct version specified in the main repository.

Using a recursive clone is essential when working with projects that depend on multiple repositories to ensure everything is correctly set up from the start.

$ git clone --recursive git@github.xyz.com:smpi/xyz-tests.git

Cloning into 'xyz-tests'...

remote: Enumerating objects: 24, done.

remote: Counting objects: 100% (24/24), done.

remote: Compressing objects: 100% (21/21), done.

remote: Total 18759 (delta 13), reused 7 (delta 3), pack-reused 18735

Receiving objects: 100% (18759/18759), 125.23 MiB | 33.33 MiB/s, done.

Resolving deltas: 100% (12772/12772), done.

Updating files: 100% (5233/5233), done.

Submodule 'smpi-ci/mtt' (https://github.com/open-mpi/mtt-legacy.git) registered for path 'smpi-ci/mtt'

Submodule 'tests/hpc-smpi-fvt' (git@github.xyz.com:smpi/hpc-smpi-fvt.git) registered for path 'tests/hpc-smpi-fvt'

Submodule 'tests/ompi-tests' (git@github.xyz.com:smpi/ompi-tests.git) registered for path 'tests/ompi-tests'

Cloning into '/data/nfs_smpi_ci/xyz-tests/smpi-ci/mtt'...

remote: Enumerating objects: 1abc, done.        

remote: Counting objects: 100% (21/21), done.        

remote: Compressing objects: 100% (18/18), done.        

remote: Total 13184 (delta 7), reused 10 (delta 3), pack-reused 13163        

Receiving objects: 100% (13184/13184), 3.94 MiB | 20.67 MiB/s, done.

Resolving deltas: 100% (8686/8686), done.

Cloning into '/data/nfs_smpi_ci/xyz-tests/tests/hpc-smpi-fvt'...

remote: Enumerating objects: 15511, done.        

remote: Total 15511 (delta 0), reused 0 (delta 0), pack-reused 15511        

Receiving objects: 100% (15511/15511), 81.99 MiB | 28.47 MiB/s, done.

Resolving deltas: 100% (10318/10318), done.

Cloning into '/data/nfs_smpi_ci/xyz-tests/tests/ompi-tests'...

remote: Enumerating objects: 36104, done.        

remote: Total 36104 (delta 0), reused 0 (delta 0), pack-reused 36104

Receiving objects: 100% (36104/36104), 103.97 MiB | 26.81 MiB/s, done.

Resolving deltas: 100% (23820/23820), done.

Submodule path 'smpi-ci/mtt': checked out 'mnccd42d37c232883d3a600ac4151868a3327b7'

Submodule path 'tests/hpc-smpi-fvt': checked out 'abc5ab4afacfacccc04dacb2c55a41477d2c02'

Submodule path 'tests/ompi-tests': checked out 'xyx7f6e194df5bbc16e836f8d63556be363a94ca5'

$ 

------------

NOTE: With version 2.13 of Git and later, --recurse-submodules can be used instead of --recursive

With  older Git , you can use:

git clone --recursive git://github.com/foo/bar.git

For already cloned repos, or older Git versions, use:

git clone git://github.com/foo/bar.git

cd bar

git submodule update --init --recursive

------------

The "git merge" :

The "git merge" command is a fundamental feature in Git that allows developers to combine changes from different branches into a single branch. This process is essential for integrating various lines of development, such as feature branches or bug fixes, into the main codebase.

What git merge Does?

1. Combines Changes: The primary function of "git merge" is to integrate the histories of two branches. When you execute "git merge <branch-name>", Git takes the changes from the specified branch and merges them into your current branch, which is often referred to as the "receiving branch".

2. Creates a Merge Commit: If the branches have diverged (i.e., both have new commits since they last shared a common ancestor), Git creates a new commit known as a merge commit. This commit has two parent commits: one from each branch being merged. This allows Git to maintain a complete history of changes.

3. Finds the Common Base: Before merging, Git identifies the common ancestor (or merge base) of the two branches. It then computes the differences (diffs) between this common ancestor and each of the branches to apply those changes simultaneously.

4. Handling Conflicts: If there are conflicting changes in the same part of a file from both branches, Git will pause the merge process and prompt you to resolve these conflicts manually. Once resolved, you can finalize the merge with a commit.

Types of Merging

1) Fast-Forward Merge: If the current branch is directly behind the branch being merged (meaning it has no new commits since their last common ancestor), Git can simply move the pointer of the current branch forward to the latest commit of the other branch without creating a new commit 

2) Three-Way Merge : When both branches have diverged, a three-way merge occurs. Here, Git uses three points (the two branch tips and their common ancestor) to create a new merge commit that reconciles changes from both branches.

Example: To illustrate how "git merge" works, let's go through a practical example involving two branches: main and feature.

1. Initial Setup: You start with a repository that has a `main` branch and you create a new branch called `feature` for development.

   git checkout -b feature main

2. Making Changes in the Feature Branch:

   You make some changes in the `feature` branch and commit them.

   # Edit some files

   git add <file>

   git commit -m "Add new feature"

3. Switching Back to Main:

   After completing your work on the "feature" branch, you switch back to the "main" branch.

   git checkout main

4. Making Additional Changes in Main: While you were working on the "feature", someone else made changes to the "main" branch. You can update it before merging:

   git pull origin main

5. Merging the Feature Branch into Main: Now, you can merge the changes from the "feature" branch into the "main" branch.

  

   git merge feature

If there are conflicting changes (e.g., both branches modified the same line in a file), Git will pause and prompt you to resolve these conflicts manually before completing the merge:

# Resolve conflicts in your editor, then stage the resolved files

git add <resolved-file>

# Complete the merge after resolving conflicts

git commit -m "Merge feature into main with conflict resolution"

Using "git merge", developers can effectively integrate changes from multiple branches, making it easier to collaborate on projects. Understanding how to handle different types of merges and resolve conflicts is crucial for maintaining a clean project history and ensuring smooth collaboration among team members.

-----------

The "git rebase" :

Git rebase is a powerful command in Git that allows you to integrate changes from one branch into another by moving the base of your current branch to a different commit. This process effectively rewrites the commit history, making it appear as though you started your work from a different point in the project history.

What Does Git Rebase Do?

1. Changing the Base: When you perform a rebase, you change the base of your current branch to a specified commit from another branch. This means that all the commits in your current branch will be reapplied on top of the new base commit, creating new commits in the process.

2. Linear History: One of the main advantages of rebasing is that it helps maintain a clean and linear project history. This is particularly useful for simplifying the commit log and making it easier to follow changes over time. Instead of having a branching structure, rebasing makes it look like all changes were made sequentially.

3. Interactive Rebasing: Git rebase can be performed in two modes: standard and interactive. The interactive mode allows you to modify commits during the rebase process, such as editing commit messages, squashing multiple commits into one, or even removing commits altogether.

To perform a standard rebase, you would typically use:

git rebase <base-branch>

For example, if you want to rebase your feature branch onto the main branch:

git checkout feature

git rebase main

This command will take all commits from "feature" and apply them on top of "main", effectively updating "feature" with the latest changes from `main`.

 Example: 

1. Initial Setup

   - You have two branches: main and feature

   - You make several commits on feature while others continue to update main.

2. Rebasing:

   - To incorporate the latest changes from main into feature, you would execute:

   git checkout feature

   git rebase main

3. Resolving Conflicts:

   - If there are conflicts during the rebase, Git will pause and allow you to resolve them. After resolving conflicts in the affected files, you would continue the rebase with:

   git add <resolved-file>

   git rebase --continue

Advantages of Git Rebase

1. Cleaner Commit History: By avoiding unnecessary merge commits, rebasing results in a more straightforward project history that is easier to read and understand.
2. Easier Debugging: A linear history simplifies tracking down bugs and understanding how features were developed over time.
3. Better Collaboration: Rebasing helps keep feature branches up-to-date with ongoing changes in the main branch, which can reduce conflicts when it's time to merge back into main.

What Happens During Rebase?

- Updating Your Work: When you run this command, Git looks at the latest commits in the `main` branch and applies your feature branch's commits as if you started working on them after the latest changes in main. This effectively updates your feature branch with any new changes that have been made in main.

- Cleaner History: Unlike merging, which creates a new commit that combines both branches (often making the history look complicated), rebasing keeps your project history neat and linear. It looks like all your work was done after the latest changes from main, even if it wasn't.

 Example : Imagine you are working on a project:

1. You create a branch called feature to add a new feature.

2. While you're working, someone else adds new updates to the main branch.

3. To ensure your feature includes those updates, you use git rebase main.

4. After running this command, your commits from feature will be placed on top of the latest commits from main, making it look like you developed your feature with all the latest updates in mind

NOTE:

While rebasing can be very useful, it is important to note that rewriting history can lead to complications if not handled carefully—especially if you've already pushed your changes to a shared repository. It’s generally advised not to rebase public branches that others may be using. 

Git rebase is a powerful tool for managing project history and integrating changes across branches while maintaining clarity and organization in your commit log.

Git rebase and the combination of `git pull` followed by `git merge` are related but not equivalent operations. Here’s a breakdown of their differences and how they function:

Git Pull vs. Git Rebase

1. Git Pull:

   - The `git pull` command is essentially a combination of two commands: git fetch followed by git merge. When you run git pull, Git fetches the latest changes from the remote branch and then merges those changes into your current branch, creating a new merge commit if necessary. This can lead to a more complex commit history with multiple merge commits, especially in collaborative environments where many changes are being made concurrently.

   git pull origin main

2. Git Rebase:

   - The git rebase command, on the other hand, takes the commits from your current branch and re-applies them on top of another branch (often the updated main branch). This effectively rewrites the commit history, creating a linear progression of commits without additional merge commits. This can make the project history cleaner and easier to follow

   git rebase main

Key Differences

History Structure: 

  - Merge: Results in a branching history with merge commits that reflect the integration points between branches.

  - Rebase: Produces a linear history that appears as if all changes were made sequentially on top of the base branch.

Conflict Resolution:

  - During a merge, if conflicts arise, you resolve them and then create a new merge commit.

  - During a rebase, conflicts must be resolved at each commit being reapplied, and you continue the rebase process after resolving each conflict

Use Cases:

  - Use git pull when you want to quickly integrate remote changes into your local branch, accepting the additional merge commit.

  - Use git rebase when you want to keep a clean history, especially in feature branches that have not yet been shared with others 

That way,  use git fetch when you want to check for updates without altering your current work, git pull for quickly updating your branch with remote changes (but be cautious of conflicts), and git merge when you want to combine different branches of work. 

Git Fork : A fork is a complete copy of a repository that allows you to make changes independently without affecting the original repository. It is often used in collaborative projects, especially on platforms like GitHub.

Purpose: Forking is typically used when you want to contribute to someone else's project. You can create your own version of the repository to experiment with or develop features without needing direct access to the original repository.

Isolation: A fork creates an entirely separate repository, meaning that changes made in your fork do not impact the original repository unless you explicitly submit a pull request.

Fprobe: Efficient Kernel Function Tracing in Linux

Fprobe is a function entry/exit probe mechanism in the Linux kernel, built on top of the ftrace framework. It allows developers to attach callbacks to function entry and exit points, similar to kprobes and kretprobes, but with improved performance for multiple functions through a single handler. This makes fprobe particularly useful for tracing and debugging kernel functions without the overhead associated with full ftrace features.

Key Features of Fprobe:

Performance: Provides faster instrumentation for multiple functions compared to traditional kprobes and kretprobes.

Structure: The fprobe structure includes fields for entry and exit handlers, allowing customized behavior when functions are entered or exited.

Registration: Fprobes can be registered using various methods, including function-name filters or direct address registration.

The usage of fprobe :

The fprobe is a wrapper of ftrace (+ kretprobe-like return callback) to attach callbacks to multiple function entry and exit. User needs to set up the struct fprobe and pass it to register_fprobe(). Typically, fprobe data structure is initialized with the entry_handler and/or exit_handler as below.

NOTE: callbacks refer to functions that are executed in response to specific events, particularly when a function is entered or exited

A typical fprobe setup might look like this:

struct fprobe fp = {
    .entry_handler = my_entry_callback,
    .exit_handler = my_exit_callback,
};
register_fprobe(&fp, "func*", "func2");

-----------------------------------------------------

Simple C program that demonstrates how to use fprobe in the Linux kernel. This program sets up an fprobe to monitor the entry and exit of a specific kernel function. For this example, we'll assume that you want to probe a function named "target_function".

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/fprobe.h>
#include <linux/init.h>

// Function prototypes
int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);
void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);
void target_function(void); // Declare target_function prototype

// Define the fprobe structure
static struct fprobe fp = {
    .entry_handler = my_entry_callback,
    .exit_handler = my_exit_callback,
};

// The function we want to probe
void target_function(void) {
    printk(KERN_INFO "Inside target_function\n");
}

// Entry callback
int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {
    printk(KERN_INFO "Entered target_function\n");
    return 0; // Return an int as expected
}

// Exit callback
void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {
    printk(KERN_INFO "Exited target_function\n");
}

// Module initialization
static int __init my_module_init(void) {
    // Register the fprobe for the target function
    if (register_fprobe(&fp, "target_function", NULL)) {
        printk(KERN_ERR "Failed to register fprobe\n");
        return -1;
    }

    printk(KERN_INFO "Fprobe registered successfully\n");

    // Call the target function to see the callbacks in action
    target_function();

    return 0;
}

// Module cleanup
static void __exit my_module_exit(void) {
    // Unregister the fprobe
    unregister_fprobe(&fp);
    printk(KERN_INFO "Fprobe unregistered successfully\n");
}

module_init(my_module_init);
module_exit(my_module_exit);

MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Fprobe Example Module");
MODULE_AUTHOR("SACHIN P BAPPALIGE");

--------------------------------------------------------------------------

Compilation and Loading:

Step 1 : Save the code in a file named fprobe_example.c.

Step 2 : Create a Makefile as shown below

obj-m += fprobe_example.o

all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

--------------------------------------

# make
make -C /lib/modules/6.12.0-rc2+/build M=/root/fprobe modules
make[1]: Entering directory '/root/linux'
  CC [M]  /root/fprobe/fprobe_example.o
  MODPOST /root/fprobe/Module.symvers
  CC [M]  /root/fprobe/fprobe_example.mod.o
  CC [M]  /root/fprobe/.module-common.o
  LD [M]  /root/fprobe/fprobe_example.ko
  BTF [M] /root/fprobe/fprobe_example.ko
make[1]: Leaving directory '/root/linux'

# ls

Makefile  Module.symvers  fprobe_example.c  fprobe_example.ko  fprobe_example.mod  fprobe_example.mod.c  fprobe_example.mod.o  fprobe_example.o  modules.order

--------------------------------------------------

Step 3 : Insert module-->  insmod fprobe_example.ko

Step 4: Check kernel logs with dmesg to see output from the callbacks.

Step 5: Remove the module using  -->  rmmod fprobe_example

Here’s a C program that demonstrates how to use fprobe to monitor the do_fork function in the Linux kernel. This example sets up entry and exit callbacks to log when the do_fork function is called and when it returns.

-------------------------------------

#include <linux/module.h>

#include <linux/kernel.h>

#include <linux/fprobe.h>

#include <linux/init.h>

#include <linux/sched.h>

// Function prototypes for the entry and exit callbacks

int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);

void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);

// Define the fprobe structure

static struct fprobe fp = {

    .entry_handler = my_entry_callback,

    .exit_handler = my_exit_callback,

};

// Entry callback for do_fork

int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {

    printk(KERN_INFO "do_fork called: PID = %d\n", current->pid);

    return 0; // Return an int as required

}

// Exit callback for do_fork

void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {

    printk(KERN_INFO "do_fork returned: PID = %d\n", current->pid);

}

// Module initialization function

static int __init my_module_init(void) {

    // Register the fprobe for the do_fork function

    if (register_fprobe(&fp, "do_fork", NULL)) {

        printk(KERN_ERR "Failed to register fprobe for do_fork\n");

        return -1;

    }

    printk(KERN_INFO "Fprobe registered successfully for do_fork\n");

    return 0;

}

// Module cleanup function

static void __exit my_module_exit(void) {

    // Unregister the fprobe

    unregister_fprobe(&fp);

    printk(KERN_INFO "Fprobe unregistered successfully for do_fork\n");

}

module_init(my_module_init);

module_exit(my_module_exit);

MODULE_LICENSE("GPL");

MODULE_DESCRIPTION("Fprobe Example for do_fork");

MODULE_AUTHOR("SACHIN P BAPPALIGE");

------------------------------------------------------------------------------

NOTE: Fprobe and kprobe are both mechanisms used in the Linux kernel for tracing and debugging, but they have distinct characteristics and use cases.

Kprobe :

Purpose: Kprobes allow developers to insert probes at almost any kernel function entry or exit point. They enable dynamic instrumentation by setting breakpoints on specified functions.

Structure: Each kprobe consists of a handler that is invoked when the probe is hit, allowing for custom actions or logging.

Performance: Kprobes can introduce overhead, especially when multiple probes are registered, as they rely on software breakpoints (SWBP) which can be slower compared to other methods.

Recursion Handling: Kprobes maintain a per-CPU variable (`current_kprobe`) to manage recursion safely, which prevents re-entry into the same probe handler.

Developers commonly probe various Linux kernel functions to gather insights about system performance, debug issues, and monitor behavior. Here are some typical functions that are often probed:

Commonly Probed Kernel Functions : 

1)  do_fork: Used for process creation. Probing this function can help track process spawning and resource allocation.

2)  sys_read / sys_write: These functions handle system calls for reading from and writing to files. Probing them allows developers to monitor file I/O operations.

3)  schedule: Responsible for context switching between processes. Probing this function can provide insights into scheduling behavior and CPU usage.

4)  kmalloc / kfree: Functions for memory allocation and deallocation in the kernel. Probing these can help identify memory usage patterns and potential leaks.

5)  vfs_read / vfs_write:Virtual filesystem layer functions that manage file operations. Probing these functions helps in understanding file access patterns across different filesystems.

7)  tcp_sendmsg / tcp_recvmsg:Used for sending and receiving TCP messages. Probing these can assist in analyzing network performance and behavior.

8)  netif_receive_skb: This function processes incoming packets in the network stack. Probing it can help monitor network traffic handling.

9)  exit_notify: Handles the cleanup of a process upon exit. Probing this function can track process termination and resource cleanup.

10) __wake_up: Used to wake up processes waiting on a condition variable. Probing this can provide insights into synchronization and concurrency issues.

11) File System Operations: Functions like lookup, create, and unlink are also commonly probed to monitor filesystem interactions

In summary, while both kprobe and fprobe serve the purpose of tracing kernel functions, fprobe is tailored for efficiency and ease of use in scenarios requiring multiple function tracing. Kprobes offer broader capabilities but at the cost of performance overhead and complexity.

Reference :

1) https://docs.kernel.org/trace/fprobe.html