Changeset 96564 in vbox for trunk/src/VBox/ValidationKit/docs/AutomaticTestingRevamp.txt
- Timestamp:
- Sep 1, 2022 9:06:13 AM (3 years ago)
- svn:sync-xref-src-repo-rev:
- 153434
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/src/VBox/ValidationKit/docs/AutomaticTestingRevamp.txt
r82972 r96564 242 242 other words, cygwin will not be used on Windows! 243 243 244 The testbox will provide the unixy util ties that ships with kBuild and possibly244 The testbox will provide the unixy utilities that ships with kBuild and possibly 245 245 some additional ones from tools/*.*/bin in the VirtualBox tree (wget, unzip, 246 zip, and so on). The test drivers will avoid invoking any of these utilit es246 zip, and so on). The test drivers will avoid invoking any of these utilities 247 247 directly and instead rely on generic utility methods in the test driver 248 248 framework. That way we can more easily reimplement the functionality of the 249 core utilit es and drop the dependency on them. It also allows us to quickly249 core utilities and drop the dependency on them. It also allows us to quickly 250 250 work around platform specific oddities and bugs. 251 251 … … 257 257 run under the testbox script, they can be executed in the VirtualBox development 258 258 environment. This is important for bug analysis and for simplifying local 259 testing by the developers before commit ing changes. It also means the test259 testing by the developers before committing changes. It also means the test 260 260 drivers can be developed locally in the VirtualBox development environment. 261 261 … … 335 335 336 336 - Two level testing - L1 quick smoke tests and L2 longer tests performed on 337 builds passing L1. (Klaus (IIRC) me nt this could be realized using337 builds passing L1. (Klaus (IIRC) meant this could be realized using 338 338 test dependency.) 339 339 - Black listing builds (by revision or similar) known to be bad. … … 344 344 - Useful to be able to partition testboxes (run specific builds on some 345 345 boxes, let an engineer have a few boxes for a while). 346 - Intera tion with ILOM/...: reset systems.347 - Be able to suspend testing on selected testboxes when doing maint ance346 - Interaction with ILOM/...: reset systems. 347 - Be able to suspend testing on selected testboxes when doing maintenance 348 348 (where automatically resuming testing on reboot is undesired) or similar 349 349 activity. 350 - Abort testing on selec ed testboxes.350 - Abort testing on selected testboxes. 351 351 - Scheduling of tests requiring more than one testbox. 352 352 - Scheduling of tests that cannot be executing concurrently on several … … 357 357 - Configure tests with variable configuration to get better coverage. Two modes: 358 358 - TM generates the permutations based on one or more sets of test script arguments. 359 - Each configuration permu ation is specified manually.359 - Each configuration permutation is specified manually. 360 360 - Test specification needs to be flexible (select tests, disable test, test 361 361 scheduling (run certain tests nightly), ... ). … … 367 367 primary log associated with it. 368 368 - Test sets stores further result as a recursive collection of: 369 - hiera chical subtest name (slash sep)369 - hierarchical subtest name (slash sep) 370 370 - test parameters / config 371 371 - bool fail/succ … … 448 448 for a testcase. It will schedule on a testcase level, using the combined 449 449 priority of the testcase in the test group and the test group with the testbox 450 partition, trying to spread the test case argument var ation out accordingly450 partition, trying to spread the test case argument variation out accordingly 451 451 over the whole scheduilng queue. Which argument variation to start with, is 452 452 not undefined (random would be best). … … 486 486 the end (exclusive). 487 487 488 Tuple-versioning has a shortcom ming wrt to keys, both primary and foreign.488 Tuple-versioning has a shortcoming wrt to keys, both primary and foreign. 489 489 The primary key of a table employing tuple-versioning is really 490 490 'id' + 'valid_period', where the latter is expressed using two fields … … 494 494 495 495 Foreign key references to a table using tuple-versioning is running into 496 trouble because of the time ax sis and that to our knowledge foreign keys496 trouble because of the time axis and that to our knowledge foreign keys 497 497 must reference exactly one row in the other table. When time is involved 498 498 what we wish to tell the database is that at any given time, there actually … … 634 634 635 635 3. If the testbox was marked as disabled, respond with an IDLE command to the 636 testbox [done]. (Note! Must do this after TestBoxStatuses maint ainance from636 testbox [done]. (Note! Must do this after TestBoxStatuses maintenance from 637 637 point 2, or abandoned tests won't be cleaned up after a testbox is disabled.) 638 638 … … 740 740 741 741 Also, we need some way of dealing with 'abort' and 'reboot' commands being 742 issued while waiting. The easy way out is preten tit's a time out.742 issued while waiting. The easy way out is pretend it's a time out. 743 743 744 744 When changing the status to 'gang-timeout' we have to be careful. First of all, … … 789 789 This is a subfunction of scenario #1 and #2. The actions taken are the same in 790 790 both situations. The precondition for taking this path is that the row in the 791 testboxstatus table is refer ing to a testset (i.e. testset_id is not NULL).791 testboxstatus table is referring to a testset (i.e. testset_id is not NULL). 792 792 793 793 … … 795 795 796 796 1. If the testset is incomplete, we need to completed: 797 a) Add a message to the root TestResults row, creating one if neces ary,797 a) Add a message to the root TestResults row, creating one if necessary, 798 798 that explains that the test was abandoned. This is done 799 799 by inserting/finding the string into/in TestResultStrTab and adding … … 939 939 ------------ 940 940 941 The goal for this milestone is configuration and converting current test scases,941 The goal for this milestone is configuration and converting current testcases, 942 942 the result will be the a minimal test deployment (4-5 new testboxes). 943 943 … … 983 983 984 984 Once deployed, a golden rule will be that all new features shall have test 985 coverage. Prefer rably, implemented by someone else and prior to the feature985 coverage. Preferably, implemented by someone else and prior to the feature 986 986 implementation. 987 987 … … 1060 1060 :Status: $Id$ 1061 1061 :Copyright: Copyright (C) 2010-2020 Oracle Corporation. 1062
Note:
See TracChangeset
for help on using the changeset viewer.