Twister: additional scripts

[majunkier]

Add the execution of external scripts at precise moments. These scripts can be strategically deployed in three distinct phases: pre-script, post-flash-script and post-script. This functionality could help configuring the environment optimally before testing.
Script phases are already implemented solution, this PR serves as an extension to broaden their application and utility.

Usage is similar to quarantine file :

twister -p <BOARD> -T <TEST_SCENARIO> --scripting-list <PATH_TO_SCRIPTING_YAML>

An example of entries in a scripting list yaml:

    - scenarios:
    - sample.basic.helloworld
    platforms:
    - frdm_k64f
    pre_script:
        <PATH_TO_PRE_SCRIPT>
    post_flash_script:
        <PATH_TO_POST_FLASH_SCRIPT>
    post_script:
        <PATH_TO_POST_SCRIPT>
    comment:
        Testing extra scripts feature

Usage of all (pre_script, post_flash_script, post_script) is not mandatory, it could be only one parameter.

One of the primary benefits of this approach is the ability to execute scripts tailored to individual scenarios.

[evgeniy-paltsev]

Are we planning to use it for tests in Zephyr tree?

[majunkier]

Solution is designed with flexibility in mind, making it suitable for use in a variety of contexts, including the Zephyr tree.

[hakehuang]

why no just put to the map.yaml file? along with --device-testing?

[majunkier]

why no just put to the map.yaml file? along with --device-testing?

Currently, we have the capability to run scripts based on specified runner parameters; however, this approach limits us to the execution of pre scripts only. An alternative method would involve enhancing the map file to accommodate script execution, but this leads to redundancy when dealing with multiple identical boards, as we would need to replicate the script type, scenario, and path information for each entry.
By this approach its easy to execute scripts tailored to individual scenarios.

[hakehuang]

Currently, we have the capability to run scripts based on specified runner parameters; however, this approach limits us to the execution of pre scripts only. An alternative method would involve enhancing the map file to accommodate script execution, but this leads to redundancy when dealing with multiple identical boards, as we would need to replicate the script type, scenario, and path information for each entry.
By this approach its easy to execute scripts tailored to individual scenarios.

This could be easily out of control as no one knows what you do in the rep/post, why we need customer for each case? why not just use pytest framework to do so?

[majunkier]

This could be easily out of control as no one knows what you do in the rep/post, why we need customer for each case? why not just use pytest framework to do so?

I concur that allowing unrestricted modifications to scripts by everyone could pose risks. However, the proposed solution does not necessitate storing the scripts or the scripting-list.yaml file within the Zephyr repository, similar to how the quarantine-list file is managed. You will have the flexibility to manage scripts within your own environment. This file is not mandatory, it serves to enhance the functionality of Twister for use in personalized settings.

[hakehuang]

You will have the flexibility to manage scripts within your own environment. This file is not mandatory, it serves to enhance the functionality of Twister for use in personalized settings.

this thing is if go this way, the twister will be diverged in its behavour, quaratine is to skip test cases. but adding pre/post staff will impact the way test runs. it is better in a controlable way. unless you have strong user case to supporting doing so.

[majunkier]

You will have the flexibility to manage scripts within your own environment. This file is not mandatory, it serves to enhance the functionality of Twister for use in personalized settings.

this thing is if go this way, the twister will be diverged in its behavour, quaratine is to skip test cases. but adding pre/post staff will impact the way test runs. it is better in a controlable way. unless you have strong user case to supporting doing so.

There are many ways of usage, i could take an example power management tests, where we can set up the necessary power conditions, calibrate or reset power measurement tools, perform initial checks to validate that the device is in the right state for power measurements and gather power usage data