Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
home:Ledest:erlang
erlang
3871-ct-add-internal_doc-and-ct_notes.md.patch
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File 3871-ct-add-internal_doc-and-ct_notes.md.patch of Package erlang
From 87291144f9c78e9390c5fdf2905454829368276c Mon Sep 17 00:00:00 2001 From: Jakub Witczak <kuba@erlang.org> Date: Tue, 20 Jun 2023 18:32:11 +0200 Subject: [PATCH] ct: add internal_doc and ct_notes.md --- lib/common_test/internal_doc/ct_notes.md | 161 +++++++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 lib/common_test/internal_doc/ct_notes.md diff --git a/lib/common_test/internal_doc/ct_notes.md b/lib/common_test/internal_doc/ct_notes.md new file mode 100644 index 0000000000..84c70b97ef --- /dev/null +++ b/lib/common_test/internal_doc/ct_notes.md @@ -0,0 +1,161 @@ +# CT test_server + +## Problem (GH-7119, OTP-11894, OTP-14480) +I think the most confusing thing is that today OTP behavior and design seems to be a mix of Configuration and Testcase centric attributes: +1. (Configuration centric) CT hook callback looks as designed to wrap around CT Configuration functions (i.e. you have *pre* and *post* to wrapp around init_per_testcase or end_per_testcase) + - Furthermore if you consider hook callback function names, there are no hooks wrapping around Testcase function at all! +2. (Testcase centric) AND at the same the hook execution order is determined by relation to CT Testcase callback +### Next step ideas +1. improve existing documentation for hooks (actually it was planned for many years but down prioritized) +2. add mermaid diagrams to docs when that is possible +3. introduce a CT option for Configuration centric hook execution order (maybe named ct_hooks_order := [testcase(default) | configuration]) +### CT hooks priorities (documentation sketch) +Let's assume: +1. cth_A and cth_B being CT hook modules to be installed +2. A, B are CT hook priorities where A is more important than B (higher priority) +3. hook priorities affect order in which hook callbacks of the same type are executed +4. Testcase functions - are CT callback functions defined in suite which will have test verdict assigned (e.g. ok, fail, skip) +5. Configuration functions - are CT callback functions defining initialization and cleanup routines for testcases (e.g. init_per_testcase, end_per_testcase) + +```mermaid +--- +title: Diagram legend +--- +flowchart LR + dc((("double circle"))) -.- i>item determining hook execution order] + pgr[/"parallelogram"/] -.- j>item not relevant for hook execution order] +``` + +#### Testcase centric (default) +1. execution order is related to position in relation to Testcase +2. all *init* hook callbacks(positioned before Testcase) are executed in some order +3. all *end* hook callbacks(positioned after Testcase) are executed in reversed order +4. *pre* and *post* hook callback type is not affecting execution order +```mermaid +--- +title: Testcase centric CT hook execution order (default) +--- +flowchart TD + subgraph hooks + pre_init_pt_A["(A) pre_init_per_testcase"] --> pre_init_pt_B + end + subgraph suite + pre_init_pt_B["(B) pre_init_per_testcase"] --> init_pt[/"init_per_testcase"/] + end + init_pt --> post_init_pt_A + subgraph hooks + post_init_pt_A["(A) post_init_per_testcase"] --> post_init_pt_B + end + subgraph suite + post_init_pt_B["(B) post_init_per_testcase"] --> testcase + testcase((("Testcase"))) + end + subgraph hooks + testcase --> pre_end_pt_B + pre_end_pt_B["(B) pre_end_per_testcase"] --> pre_end_pt_A + end + subgraph suite + pre_end_pt_A["(A) pre_end_per_testcase"] --> end_per_test_case + end + subgraph hooks + end_per_test_case[/"end_per_testcase"/] --> post_end_pt_B + post_end_pt_B["(B) post_end_per_testcase"] --> post_end_pt_A["(A) post_end_per_testcase"] + end +``` +#### Configuration centric (option candidate) +- all *pre* hooks will be executed in some order +- all *post* hooks will be executed in reversed order +- relation to Testcase is not relevant for hook execution order +- Note: I think, it might be considered a bit inconsistent because there are no hook callbacks wrapping around Testcase (e.g. pre_testcase, post_testcase) +- Above could be ugly workarounded by using pre_init_per_testcase with post_end_per_testcase +- Adding pre_testcase, post_testcase might not be simple but maybe it is not needed + +```mermaid +--- +title: Configuration centric CT hook execution order (option) +--- +flowchart TD + subgraph hooks + pre_init_pt_A["(A) pre_init_per_testcase"] --> pre_init_pt_B + end + subgraph suite + pre_init_pt_B["(B) pre_init_per_testcase"] --> init_pt((("init_per_testcase"))) + end + init_pt --> post_init_pt_B + subgraph hooks + post_init_pt_B["(B) post_init_per_testcase"] --> post_init_pt_A + end + subgraph suite + post_init_pt_A["(A) post_init_per_testcase"] --> testcase + testcase[/"Testcase"/] + end + subgraph hooks + testcase --> pre_end_pt_A + pre_end_pt_A["(A) pre_end_per_testcase"] --> pre_end_pt_B + end + subgraph suite + pre_end_pt_B["(B) pre_end_per_testcase"] --> end_per_test_case + end + subgraph hooks + end_per_test_case((("end_per_testcase"))) --> post_end_pt_B + post_end_pt_B["(B) post_end_per_testcase"] --> post_end_pt_A["(A) post_end_per_testcase"] + end +``` + +## processes +```mermaid +flowchart LR + tc["`tc + every CT callback function`"] + testcase["`testcase + the user Testcase`"] + testcase --> tc +``` + +## test_server code + +```mermaid +mindmap + code + run_test_case_apply/1 + run_test_case_apply/6 + ))run_test_case_eval_fun/9(( + run_test_case_eval/9 + do_init_tc_call/4 + run_test_case_eval1/6 + do_end_tc_call/4 + ct_util mark_process/0 + run_test_case_msgloop/1 + [receive] + run_test_case_eval1/6 + ts_tc/3 +``` +## processes +```mermaid +flowchart LR + job["`(job process GL) + captures output`"] + case["`(case process) + runs test suite`"] + + case --news--> job +``` +## concept +```mermaid +mindmap + root)test_server( + interfaces + sup + get_loc/1 + set_ct_state/1 + ctrl + run_test_case_apply/1 + init_target_info/0 + init_memory_checker/0 + suite + init + target_info + memory_checker + questions + why call_end_conf not called when init/end_per_testcase not defined? +``` -- 2.35.3
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor
