Tuesday, April 4, 2023

Linux Test Harness : avocado and op-test framework

A Test Harness, also known as a testing framework or testing tool, is a software tool or library that provides a set of functions, APIs, or interfaces for writing, organizing, and executing tests. Test harnesses provide a structured way to write tests and automate the testing process. 

Linux avocado test framework and Linux op-test framework are both open-source testing frameworks designed for testing and validating Linux-based systems. Both frameworks are widely used in the Linux community and have a strong user base. The choice between the two depends on the specific testing needs and requirements of the user.

The Linux avocado test framework is a modular and extensible testing framework that allows users to write and run tests for different levels of the Linux stack, including the kernel, user space, and applications. It provides a wide range of plugins and tools for testing, including functional, performance, and integration testing. The framework is easy to install and use and supports multiple test runners and reporting formats.

On the other hand, the Linux op-test framework is a set of Python libraries and utilities that automate the testing of hardware and firmware components in Linux-based systems. It provides a high-level Python API for interacting with hardware and firmware interfaces, as well as a set of pre-built tests for validating various hardware components such as CPU, memory, and storage. The framework is highly flexible and customizable, allowing users to create their own tests and integrate with other testing tools and frameworks.

While both frameworks are designed for testing Linux-based systems, the Linux avocado test framework provides a broad range of testing capabilities across different levels of the Linux stack, while the Linux op-test framework focuses specifically on automating hardware and firmware testing. The choice between the two depends on the specific testing needs and requirements of the user.

The Linux avocado test framework provides a plugin called "avocado-vt" which can be used to run tests that require a reboot between different test stages. This plugin enables the framework to run destructive tests, like kernel crash dump (kdump) testing, that require the system to be rebooted multiple times.

Similarly, the Linux op-test framework also provides support for testing scenarios that require system reboot. The framework includes a "reboot" library that allows users to reboot the system under test and wait for it to come back up before continuing with the test. This library can be used to test scenarios like kdump and fadump that require system reboot.

The community maintained avocado tests repository:

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.

This repository contains a collection of miscellaneous tests and plugins for the Linux Avocado test framework that cover a wide range of functional, performance, and integration testing scenarios. The tests are designed to be modular and easy to use, and can be integrated with the Avocado test framework to extend its capabilities.

https://github.com/avocado-framework-tests/avocado-misc-tests

How to run avocado misc tests :

To run the Avocado Misc Tests, you first need to install the Linux Avocado test framework on your system. Once you have installed the framework, you can clone the Avocado Misc Tests repository from GitHub by running the following command in a terminal:

git clone https://github.com/avocado-framework-tests/avocado-misc-tests.git 

git clone git@github.com:avocado-framework-tests/avocado-misc-tests.git

# git clone git@github.com:avocado-framework-tests/avocado-misc-tests.git
Cloning into 'avocado-misc-tests'...
remote: Enumerating objects: 18087, done.
remote: Counting objects: 100% (451/451), done.
remote: Compressing objects: 100% (239/239), done.
remote: Total 18087 (delta 242), reused 368 (delta 208), pack-reused 17636
Receiving objects: 100% (18087/18087), 6.15 MiB | 16.67 MiB/s, done.
Resolving deltas: 100% (11833/11833), done.
#

This repository is dedicated to host any tests written using the Avocado API. It is being initially populated with tests ported from autotest client tests repository, but it's not limited by that.

After cloning the repository, you can navigate to the avocado-misc-tests directory and run the tests using the avocado run command. For example, to run all the tests in the network category, you can run the following command:

cd avocado-misc-tests
avocado run network/

This will run all the tests in the network category. You can also run individual tests by specifying the path to the test file, like this:

avocado run network/test_network_ping.py

This will run the test_network_ping.py test in the network category.

Before running the tests, you may need to configure the Avocado framework to use the appropriate test runner, test environment, and plugins for your system. You can find more information on how to configure and use the Avocado framework in the official documentation: 

https://avocado-framework.readthedocs.io/en/latest/

$ avocado run  avocado-misc-tests/generic/stress.py
JOB ID     : 0018adbc07c5d90d242dd6b341c87972b8f77a0b
JOB LOG    : $HOME/avocado/job-results/job-2016-01-18T15.32-0018adb/job.log
TESTS      : 1
 (1/1) avocado-misc-tests/generic/stress.py:Stress.test: PASS (62.67 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
JOB HTML   : $HOME/avocado/job-results/job-2016-01-18T15.32-0018adb/html/results.html
TIME       : 62.67 s

There are a few more interesting things about the Avocado test framework and its usability and use cases:

  1. Flexible test design: The Avocado test framework is designed to be flexible and adaptable to a wide range of testing scenarios. It supports various test types, including functional, integration, performance, and stress tests, and can be used to test software at different levels of abstraction, from system-level to individual components. Avocado also provides a wide range of plugins and interfaces for integrating with other tools and frameworks, making it easy to customize and extend its capabilities.
  2. Easy to use: Avocado is designed to be easy to use, even for users who are new to testing or have limited programming experience. It uses a simple YAML-based syntax for defining tests and test plans, and provides a user-friendly command-line interface for running tests and viewing results. Avocado also includes detailed documentation and tutorials to help users get started quickly.
  3. Scalability and distributed testing: Avocado supports distributed testing across multiple systems, making it easy to scale up testing to handle large workloads. It includes a built-in job scheduler for managing test execution across multiple systems, and can be integrated with various cloud-based services for running tests in the cloud.
  4. Community support: Avocado is an open-source project maintained by a vibrant community of developers and testers. The community provides regular updates and bug fixes, and is actively involved in improving the usability and functionality of the framework. The Avocado community also provides support through various channels, including GitHub, mailing lists, and IRC.
  5. Use cases: Avocado is used by various organizations and companies for testing different types of software, including operating systems, virtualization platforms, container platforms, and cloud services. It is particularly well-suited for testing complex, distributed systems that require a high degree of automation and scalability. Some of the organizations that use Avocado include Red Hat, IBM, Intel, and Huawei.

License

Except where otherwise indicated in a given source file, all original contributions to Avocado are licensed under the GNU General Public License version 2 (GPLv2) or any later version. By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the Avocado project, present and future, pursuant to the license of the project.

==============================================================

               OpenPower Test Framework

===============================================================

The community maintained, op-tests repository.

https://github.com/open-power/op-test

git clone git@github.com:open-power/op-test.git

# git clone git@github.com:open-power/op-test.git
Cloning into 'op-test'...
remote: Enumerating objects: 8716, done.
remote: Counting objects: 100% (623/623), done.
remote: Compressing objects: 100% (275/275), done.
remote: Total 8716 (delta 416), reused 480 (delta 347), pack-reused 8093
Receiving objects: 100% (8716/8716), 23.89 MiB | 23.39 MiB/s, done.
Resolving deltas: 100% (6488/6488), done.
#

Pre-requisites for op-tests: [Please do not forget to do it on remote host]

1) yum install sshpass 
2) pip3 install pexpect
3) echo "set enable-bracketed-paste off" > .inputrc ; export INPUTRC=$PWD/.inputrc
    bind 'set enable-bracketed-paste off'


How to run testcase:

CASE 1:

                 ./op-test -c machine.conf --run testcases.RunHostTest --host-cmd ls

                Testcase: https://github.com/open-power/op-test/blob/master/testcases/RunHostTest.py

                where machine.conf :

                [op-test]
                bmc_type=OpenBMC /EBMC_PHYP/FSP_PHYP
                bmc_username=abc
                bmc_ip=w39
                bmc_username=root
                bmc_password=0penBmc
                hmc_ip=a.b.c.d
                hmc_username=hmcuser
                hmc_password=hmcpasswd123
                host_ip=x.y.x.k
                host_user=hostuser
                host_password=hostpasswd123
                system_name=power10
                lpar_name=lpar_name_1
                lpar_prof=default_profile

                CASE2:

                # cat ebmc-type.conf
                [op-test]
                bmc_type=EBMC_PHYP
                bmc_username=service
                bmc_password=0penBmc!
                bmc_ip=A.B.C.D
                hmc_ip=myhmc.com
                hmc_username=hmcuser
                hmc_password=hmcpassword
                system_name=myhost
                lpar_name=myhost-lpar7-sachinpb
                lpar_prof=default_profile
                lpar_gateway=A.B.C.1
                lpar_subnet=255.255.255.0
                lpar_hostname=myhost-lpar7.com
                lpar_mac=CD:EF:GH:IJ:KL
                host_ip=X.Y.Z.U
                host_user=user
                host_password=userpassword
                dump_server_ip=
                dump_server_pw=
                dump_path=/mnt
                linux_src_dir=
                kernel_image=
                initrd_image=
                num_of_iterations=

                ./op-test -c ebmc-type.conf  --run testcases.RunHostTest --host-cmd-file cmd.conf

                where : 

                # cat cmd-script.sh
                echo "welcome SACHIN P B"
                hostname
                uptime
                date
                #

                OUTPUT:  should have run the above script as shown below:

                #  ./op-test -c ebmc-type.conf  --run testcases.RunHostTest --host-cmd-file cmd-script.sh
                Logs in: /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648
                2023-08-29 09:56:48,758:op-test:setUpLoggerFile:INFO:Preparing to set location of Log File to /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758035.main.log
                2023-08-29 09:56:48,758:op-test:setUpLoggerFile:INFO:Log file: /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758035.main.log
                2023-08-29 09:56:48,758:op-test:setUpLoggerDebugFile:INFO:Preparing to set location of Debug Log File to /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758291.debug.log
                [console-expect]#which whoami && whoami
                /usr/bin/whoami
                root
                [console-expect]#echo $?
                echo $?
                0
                [console-expect]#echo "welcome SACHIN P B"
                echo "welcome SACHIN P B"
                welcome SACHIN P B
                [console-expect]#echo $?
                echo $?
                0
                [console-expect]#hostname
                hostname
                myhost.com
                [console-expect]#echo $?
                echo $?
                0
                [console-expect]#uptime
                uptime
                 09:58:15 up  7:50,  2 users,  load average: 0.08, 0.02, 0.01
                [console-expect]#echo $?
                echo $?
                0
                [console-expect]#date
                date
                Tue Aug 29 09:58:15 CDT 2023
                [console-expect]#echo $?
                echo $?
                0
                ok
                Ran 1 test in 7.510s
                OK
                2023-08-29 09:58:17,787:op-test:<module>:INFO:Exit with Result errors="0" and failures="0"

                ------------------------------------------------------------------------------------------------------------------

                Example 2: 

                python3 op-test -c machine.conf --run testcases.PowerNVDump.KernelCrash_disable_radix

                python3 op-test -c machine.conf --run testcases.PowerNVDump.KernelCrash_XIVE_off

                python3 op-test --run-suite osdump-suite -c CR-machine.conf

                python3 op-test --run testcases.RunHostTest -c CR-Machine.conf --host-cmd-file CR-Machine_command.conf --host-cmd-timeout 286400


                Example 3: [Testcase file : PowerNVDump.py]

                1) How to execute ONLY kdump tests :

                         python3 op-test --run-suite osdumpkdumpsuite -c machine.conf 

                2) How to execute ONLY Fadump tests :

                         python3 op-test --run-suite osdumpfadumpsuite -c machine.conf

                3) How to run sanity tests includes basic kdump and fadump

                          python3 op-test --run-suite osdumpsanitysuite -c machine.conf

                Example 4: [ Testcase file : OpTestKexec.py]

                How to run the kexec test in op-tests framework

                ./op-test -c machine.conf --run testcases.OpTestKexec.OpTestKexec.test_load_unload

                ./op-test -c machine.conf --run testcases.OpTestKexec.OpTestKexec.test_load_and_exec

                ./op-test -c machine.conf --run testcases.OpTestKexec.OpTestKexec.test_syscall_load_and_exec

                ./op-test -c machine.conf --run testcases.OpTestKexec.OpTestKexec.test_kexec_unsigned_kernel

                ./op-test -c machine.conf  --run testcases.OpTestKexec.OpTestKexec.test_kexec_in_loop  

                where machine.conf :

                [op-test]
                bmc_type=FSP_PHYP
                bmc_username=bmcadmin
                bmc_password=**********
                bmc_ip=ABC-fsp.america.com
                hmc_ip=HMC1.america.com
                hmc_username=adminhmc
                hmc_password=********
                system_name=System123
                lpar_name=system123-lp4_SACHINPB
                lpar_prof=default_profile
                lpar_gateway=9.x.y.1
                lpar_subnet=255.255.255.0
                lpar_hostname=System123-lp4.com
                lpar_mac=A:B:C:D
                host_ip=9.X.y.z
                host_user=root
                host_password=**********
                dump_server_ip=9.m.n.c
                dump_server_pw=**********
                dump_path=/mnt
                linux_src_dir=
                kernel_image=
                initrd_image=
                num_of_iterations=100

                ===============================================================
                How to analyze op-test output :

                Traverse this directory path: op-test/test-reports/test-run-$date
                There are are  3 log files to investigate the test failure or life-cycle of testsuite
                # pwd
                /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-$DATE
                #
                # ls -1
                $DATE.log
                $DATE.main.log
                $DATE.debug.log
                #

                1 )  $DATE.log  ====> You could see console related commands and outputs 

                For example :
                lssyscfg    -m    Serverx0  -r lpar --filter lpar_names=Serverx0-lp6 -F state
                lsrefcode  -m    Serverx0  -r lpar --filter lpar_names=Serverx0-lp6 -F refcode
                chsysstate -m   Serverx0   -r lpar -n Server-lp6 -o shutdown --immed

                2)  $DATE.main.log
                If you add any statements in log.info , that will be logged in this file 

                log.info("=============== Testing kdump/fadump over ssh ===============")

                3) $DATE.debug.log
                If you add any comments with log.debug, that will be logged in this file 

                log.debug("SACHIN_DEBUG: In loop1")

                ============================================================
                Listed are some interesting things about the op-test framework and its use cases:
                1. Testing hardware systems: The op-test framework is designed for testing hardware systems, particularly servers, using the OpenPOWER architecture. It includes a wide range of tests that cover different aspects of hardware functionality, such as power management, CPU, memory, and I/O.
                2. Integration with OpenBMC: The op-test framework integrates with the OpenBMC project, an open-source implementation of the Baseboard Management Controller (BMC) firmware that provides out-of-band management capabilities for servers. This integration allows users to control and monitor server hardware using the OpenBMC interface, and to run tests on the hardware using the op-test framework.
                3. UEFI and firmware testing: The op-test framework includes support for testing UEFI firmware and other low-level system components, such as the Hostboot bootloader. This allows users to test the system firmware and ensure that it is functioning correctly.
                4. Easy to use: The op-test framework is designed to be easy to use, even for users who are not familiar with hardware testing. It uses a simple command-line interface and provides detailed documentation and tutorials to help users get started quickly.
                5. Scalability: The op-test framework is designed to be scalable and can be used to test multiple systems in parallel. This makes it suitable for testing large server farms and data centers.
                6. Community support: The op-test framework is an open-source project with an active community of developers and testers. The community provides regular updates and bug fixes, and is actively involved in improving the usability and functionality of the framework. The op-test community also provides support through various channels, including GitHub, mailing lists, and IRC.
                7. Use cases: The op-test framework is used by various organizations and companies for testing hardware systems, including server manufacturers, data center operators, and cloud service providers. Some of the organizations that use the op-test framework include IBM, Google, and Rackspace.
                How to contribute to op-test framework open source community :

                1) mkdir kdump_xive_off_check

                2) cd kdump_xive_off_check

                3) git clone git@github.com:SACHIN-PB/op-test.git

                    Fork the repository from master : https://github.com/open-power/op-test

                     NOTE: In Git, forking a repository means creating a copy of the original repository into your own GitHub account. 
                   This is typically done when you want to contribute to an open-source project or collaborate with other developers.

                4) git config user.email

                5) git config user.name

                NOTE: To get proper username and email . Please do the following setup at /root directory
                # cat .gitconfig
                [user]
                        email = sachin@linux.XYZ.com
                        name = Sachin P B
                #

                6) git branch

                7) git remote  -v
                    origin  git@github.com:SACHIN-PB/op-test.git (fetch)
                    origin  git@github.com:SACHIN-PB/op-test.git (push)

                8) git remote add upstream git@github.com:open-power/op-test.git

                9) git remote  -v
                      origin  git@github.com:SACHIN-PB/op-test.git (fetch)
                      origin  git@github.com:SACHIN-PB/op-test.git (push)
                      upstream        git@github.com:open-power/op-test.git (fetch)
                      upstream        git@github.com:open-power/op-test.git (push)

                10) git checkout -b "kdump_xive_off_check"

                11) git branch

                12) vi testcases/PowerNVDump.py

                13) git diff

                14) git status

                15) git add testcases/PowerNVDump.py

                16) git status

                17) git commit -s

                18) git branch

                19) git push origin kdump_xive_off_check
                Enumerating objects: 7, done.
                Counting objects: 100% (7/7), done.
                Delta compression using up to 16 threads
                Compressing objects: 100% (4/4), done.
                Writing objects: 100% (4/4), 880 bytes | 880.00 KiB/s, done.
                Total 4 (delta 3), reused 0 (delta 0), pack-reused 0
                remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
                remote:
                remote: Create a pull request for 'kdump_xive_off_check' on GitHub by visiting:
                remote:      https://github.com/SACHIN-PB/op-test/pull/new/kdump_xive_off_check
                remote:
                To github.com:SACHIN-PB/op-test.git
                 * [new branch]      kdump_xive_off_check -> kdump_xive_off_check
                #

                20) Create PR using the link created at step 19 and request for the review
                Example https://github.com/open-power/op-test/pull/7XYZ4:

                21) You can update your PR by running these commands 
                git commit --amend
                git push -f origin kdump_xive_off_check
                ======================

                Reference:
                1)  https://github.com/open-power/op-test/blob/master/testcases/RunHostTest.py
                2)  https://github.com/avocado-framework-tests/avocado-misc-tests
                3)  https://avocado-framework.readthedocs.io/en/latest/

                No comments:

                Post a Comment