软件测试参考文献

软件测试

Generating Functional DesignVeri cation Tests

WITH

THEGROWINGCOMPLEXITY

and interaction of vehicle functions, a reliableSUSANA STOICAfunctional test methodology has become aFord Motor Company

major concern of the automobile industry. In1995, Ford Motor Company started a concen-trated effort to de ne for itself and its suppliersan effective test strategy for functionality veri- cation. Ford successfully applied the strate-gy in a highly cost-conscious environment thatdemands high quality and fast test turnaround.The robust test method (RTM) presented hereis an important component of this strategy.RTM was born out of Japanese industry’sneed to establish its competitive capabilityafter World War II by providing high-qualityproducts. Genichi Taguchi, through his re-search in the 1950s and early 1960s, devel-oped and validated a Design of Experimentstechnique that uses orthogonal arrays.1Madhav Phadke,2,3while working at AT&Tin the 1980s, further developed the tech-nique by de ning a methodology for apply-ing orthogonal arrays to software testing. InThe Robust Test Method,

1994, he introduced the methodology toa functional test

Ford Motor Company. RTM’s introductiongeneration methodology,

into the test process translated into a quan-translated into atum jump in test quality, allowing the de-quantum jump in testtection of previously undetected faults.quality at Ford. AppliedRobust testing was rst applied to subsystem-to subsystem-levellevel functional design veri cation. It led todesign veri cation, itthe detection of more errors than moduledetected more errorsand system functional tests combined, whilethan module and systemreducing test turnaround time by 30%.

tests combined, whilereducing test turnaround

RTM advantages

time by 30%.

RTM is a functional test generationmethodology based on orthogonal arrays.4

JULY–SEPTEMBER 19990740-7475/99/$10.00 ©1999 IEEEBy functional Design Verification we under-stand the totality of tests used to verify that agiven device under test (DUT) delivers thefunctionality de ned in the correspondingengineering speci cation.5,6

RTM uses the same set of orthogonal ar-rays7used originally by Taguchi, butTaguchi’s design-of-experiments methoddeals with process parameter variations,while RTM focuses on functional input val-ues to the device under test.

RTM’s major advantages are that

sit is applicable at all integration stages(including software)5

sit enables easy correlation of test resultsfrom different stages

sit offers a structured approach to tests

it is applicable to both input-dependentand state-dependent output values, aslong as the relationships are wellde ned

s

its black-box approach allows test setde nition in parallel with design, thusenabling a shorter project turnaroundsit is independent of a speci c productimplementation

s

when speci cations change during de-velopment, it enables very fast changesof the test set

sit saves development time

s

being independent of the test engineer’sprior experience, RTM typically expandsthe palette of tested combinations

s

it enables considerable product im-provement in the product definitionphase

53

软件测试

DESIGN VERIFICATION

The design and test of one of the rst modules to use RTM il-lustrates the advantages of this method. For the original mod-ule, the designers had spent three weeks to produce atraditional, intuitive, functional test set. Using RTM to design anew generation of the same module, the group needed onlytwo days to produce a new test set. At the same time, they ex-panded the number of combinations tested. Moreover, the orig-inal design had 18 different ways in which the software codecould enter a given state. By applying RTM, the designers re-designed the code ow so that the new design had only threeaccess options to the same piece of code.

RTM’s main distinction is that it tests several input changesat the same time, rather than one at a time. The orthogonalarray methodology ensures that the test space coverage is bal-anced (that is, all inputs are tested an equal number of times).As a result of this property and the fact that the test set de n-ition is implementation independent, RTM usually de nestest cases that the engineers never thought of. It also detects

end cases (combinationsand/or sequences of inputsignals that occur very rarely)that could cause unexpectedresults during normal func-tional mode. In the above-mentioned case, RTM-basedtesting found an activationsequence that would havedisabled the module’s func-

tionality. Following this nd-ing, the engineers modi ed

the design to eliminate theproblem.

In comparison with other test methodologies (see Table 1),RTM offers a de nite advantage in coverage and time spentdesigning the test set. RTM also gives considerably better cov-erage than one-factor-at-a-time testing, which is comparablein test set size and test generation time. (One-factor-at-a-timetests change one input parameter value at a time.)

The typical approach to functional design veri cation isa combination of one-factor-at-a-time and random/intuitivetesting. The random/intuitive method mainly tests for faultspreviously found in similar products. This is an insuf cientmethod, especially for products with completely new func-tions and thus no previously known failure modes.

RTM’s only disadvantage is that it cannot give a measureof the percentage of fault detection. Producing test traceabilitytables, which list all the functional requirements to be metand the tests covering them, compensates for this disadvan-tage. In our experience, once we have tested a product withRTM, combined with detailed timing veri cations using one-factor-at-a-time tests, we ndno further functional errors at

the higher levels of productintegration or in the eld.

RTM effects on productdevelopment

Figure 1 shows the typicalproduct developmentprocess model. Followingthis model, one developstest speci cations only afterthe design phase is completeand performs tests only afterthe implementation phase.With RTM, the processchanges somewhat, asshown in Figure 2, re ectingthe capability of designingthe RTM-based test set as

54IEEE DESIGN & TEST OF COMPUTERS

软件测试

soon as the rst cut of the design requirements speci cationbecomes available, even before the speci cation is released.Having the test set produced very early in the design cycleby an independent person allows a more-objective look atthe completeness and clarity of the software/hardware def-inition. The result is a more-reliable design.

If a speci cation remains the same during the design cycle,the RTM-generated test sequence does not change even if theparticular implementation changes. Hence our up-front workis not lost. Sometimes the design speci cation changes late inthe design phase due to changing hardware requirements ornew customer speci cations. Even in these cases, once RTMis used for test set generation, reapplying it takes much lesstime than what traditional test generation methods require.

3.If the phone is Off, and the TV is switched On, every-thing else is Off.

4.If both the phone and the TV are Off, and the CD play-er is switched On, the radio and the tape deck are Off.5.If the phone, the TV, and the CD player are Off, and thetape is switched On, the radio is Off.Once we understand these interdependencies, we areready to set up the test set. We know that we have six fac-tors (inputs), each with two levels. As in any other testmethod, we must start from an initial level, called the over-all mean,for all the factors. The overall mean will representthe rst test. We will need additional tests to cover all theother factor levels not covered in the original test. Thus, wewill need at least one more level for each factor—that is,each factor has one more degree of freedom. Hence, theminimum number of tests, or the total number of degrees offreedom,is Nfr= 1 + 6 ×(2 1) = 7.

Another parameter to consider in calculating the mini-mum number of tests needed is the product of the numberof levels of the two highest-level count factors, or Ph. In ourexample, all the factors have two levels; hence, Ph= 2 ×2 =4. The minimum number of tests is Tmin= MAX (Nfr, Ph), whichin our example is 7.

As mentioned before, RTM uses orthogonal arrays to de- ne the optimum factor combinations to achieve a balancedcoverage of the test space. One can find these arrays inbooks dealing with the design of experiments (Ross1andPhadke,2for example). The arrays are matched to the prob-lem according to the number of tests and the number of fac-tors needed in the orthogonal array, as well as the number

The methodology

To understand RTM, consider a very simple, hypotheticalaudio system. The system consists of a number of differentaudio components: a radio, a phone, a tape deck, a CD play-er, and a TV set. These components can be in On or Offstates. All the devices can function only when the ignitionkey is On, so we consider the ignition key’s state anothervariable, or factor. Table 2 presents a synopsis of the inputfactors and their levels(the values taken by the factors), sort-ed according to priorities defined in the system require-ments. We refer to this as version A of our system de nition.The different factors have certain functional priorities:1.If the ignition key is in the Off position, everything elseis Off.

2.If the phone is On, everything else is Off.

JULY–SEPTEMBER 199955

软件测试

DESIGN VERIFICATION

of levels in each factor. For example, an orthogonal arrayof type L36(211×312) contains 36 tests for a total of 23 factors,of which 11 have two levels and 12 have three levels.

For our case, we require an orthogonal array with a min-imum of seven tests and designed for a minimum of six fac-tors, each with two levels. The smallest orthogonal arraysatisfying these conditions is the L8(27) shown in Table 3,which has eight tests (> 7) and can accommodate up to sev-en (> 6) two-level factors.2

This array has one more factor than we need, which meansthat we can discard one of the columns. Is any column bet-ter to discard than the others? To answer this question, we rst must see why this array is considered orthogonal. Wenotice that each level (1 and 2) appears an equal number oftimes in each column under “Factors.” Also, we notice thateach level of a certain factor combines an equal number oftimes with the levels of another factor. That is, each factorhas four level 1s and four level 2s. If we follow the combina-tions of factors, we can see two interactions of type 1-1, twoof type 1-2, two of type 2-1, and two of type 2-2. This meansthat all the possible interactions will be tested an equal num-ber of times. Hence, if we omit any column, we will still pre-serve the same interactions. In our case, we will omit column7, as we have only six factors. To de ne the tests, we need toassign each factor to a given column and then substitute thelevel numbers with level values, as shown in Table 4.

The next step is to verify that the ve requirements previ-ously speci ed for the system are covered by the tests in theorthogonal array. Analyzing Table 4, we nd that we havemore than enough tests to verify that all the audio systemcomponents are Off while the ignition key is in the Off po-sition (requirement 1), as well as for the case when thephone is On (requirement 2). However, we do not haveenough tests to cover requirements 3, 4, and 5. The solutionis to generate an orthogonal array that assumes the ignitionkey is On for all tests and changes some of the factor-level as-signments to obtain different factor interactions (Table 5.)

56

Array manipulation techniques

The number of orthogonal arrays one can choose from islimited. Because the factors usually do not t the availabletables exactly, we can either modify the existing tables ormodify the test requirements de nition. There are three tablemodification techniques: merging columns, adding dummylevels, and compounding factors.

To explain these techniques, we modify the audio systemde nition given in Table 2 by adding levels to several fac-tors, obtaining version B shown in Table 6. For example, the

IEEE DESIGN & TEST OF COMPUTERS

软件测试

ignition key now has three levels: two levels at which the au-dio system is functional, Accessory (Acc.) and Run, and theOff level. The phone has one additional level, Hold, whichis equivalent to Off from the point of view of the rest of thesystem. The tape deck has three levels, and the radio has atotal of four levels.

In this case the individual factors have different numbersof levels, which means that we cannot choose a table thatexactly fits our requirements. First we calculate the mini-mum number of tests:Tmin= MAX(Nfr, Ph) = 12

where Nfr= 1 + 3 ×(3 1) + 2 ×(2 1) + 1 ×(4 1) = 12 andPh= 4 ×3 = 12.

Searching the orthogonal array list, we nd array L12. Thisarray cannot be used because all the factors have only twolevels and no column manipulation is allowed. Next is theL′16(45), which has ve 4-level factors, one less factor thanwe need. Considering that we have two 2-level factors, wecan combine them into one 4-level factor, thus reducing thefactor count:

sLevel 1 = TV and CD are both Off.sLevel 2 = TV is Off; CD is On.sLevel 3 = TV is On; CD is Off.

s

Level 4 = TV and CD are both On.

Because we have four levels available for certain para-meters that need only three, we must apply one more tech-nique to adapt the original L′16table to our needs: usingdummy levels to ll in for missing ones. We choose the dum-my levels so as to have more of the critical levels for a cer-tain variable or to allow a better test of the rest of the factors.For example, there is no need to test any state more than theothers for the tape deck, but by choosing the Off state as thefourth, dummy level, we will allow more visibility for the ra-dio states.

JULY–SEPTEMBER 1999Combining the TV and CD factors reduced the factor countby one, hence, we have enough columns to cover all the fac-tors. Table 7 represents the resulting orthogonal array.

Another option is to choose the L18(21×37) orthogonalarray, which has dimensions much closer to our needs. Toadapt this array, we must merge two columns, one withtwo and another with three levels, to accommodate a min-imum four-level variable. We chose to merge columns 1and 2 into a six-level column and assigned each of the oth-er factors to a three-level column. (Well-defined rules gov-ern which columns can be merged and what influence themerging has on the remaining columns. Usually, certaincolumns must remain open when two others merge. Formore information on this topic, see Phadke2andRaghavarao.7) Table 8 shows the result (column 8, notshown, was left empty).

For this example, we chose to ll the dummy level for TVand CD with Off states (marked Off′) to allow better visibil-ity of the Tape and Radio states. For the Radio dummy lev-els, we chose FM2′and AM′. We could have chosen any twoof the three radio bands.

57

软件测试

DESIGN VERIFICATION

Simplifying the test set

The methodologies for adjusting the test set are using slid-ing levels and reducing (removing from the table) nonessen-

tial factor levels. Their purpose is to ful ll the test needs witha smaller set of tests—that is, a smaller orthogonal array.Let us consider an even more complex set of factors,which includes, in addition to the ones mentioned for ver-sion B, three levels of each radio band: AM, FM1, and FM2.Hence, for version C, the Radio factor has 10 levels: threeAMs, three FM1s, three FM2s, and Off. Table 9 shows the fac-tor table with the associated values and their levels.

Now the minimum number of tests is Tmin= MAX(Nfr, Ph)= 30, where Nfr= 1 + 2 ×(2 1)+3 ×(3 1)+1 ×(10 1) =18 and Ph= 10 ×3 = 30.

To reduce the number of tests below 30, we use the slid-ing-level technique. With this technique, we cover the Radiofactor using two factors each with fewer levels. One factordetermines the band (am, fm1, fm2, or Off), and the otherdetermines the wavelengths inside the band. For example,to combine Radio = FM1 and Radio level = 2 in the orthog-onal table, we select FM1 for the test and 95.5 for the wave-length. Table 10 (next page) represents the modified Cversion of the audio factor set.

For the modi ed factors de nition, Tmin= MAX(Nfr, Ph) =14, where Nfr= 1 + 4 ×(3 1) + 2 ×(2 1) + 1 ×(4 1) = 14and Ph= 3 ×4 = 12.

Hence, we must look for a table that can accommodatetwo 2-level factors, four 3-level factors, and one 4-level fac-tor. L18meets this condition perfectly, in spite of the increasein state count. Table 11 contains the resulting test set. In thiscase, we use column 8 for the wavelength definition.Column 2 could be used instead of column 8 with the sameeffectiveness. For a better understanding of the transforma-tion, we added a column to the right of the table to showthe previously existing generic values in column 8.

Another way of reducing the test set is by analyzing theimportance of testing all the different state combinations. Ifnot all of them need testing, we can directly reduce the testset. If all are important, we can split the tests into severalsmaller sets.

For example, we may decide that testing two wavelength

58IEEE DESIGN & TEST OF COMPUTERS

软件测试

values for each of the AM, FM1, and FM2 bands is suf cient.We decide to alternate which wavelength value is tested andmerge the TV and CD factors. Thus, we obtain version Dshown in Table 12. Having five factors, each with four orfewer levels, means that we can t the tests in the L16′(45)table, as seen in Table 13. For better coverage of the Radiostates, we replaced the TV+CD state “On/Off,” which doesnot yield much information, with “Off/Off.”

If the coverage of the different radio stations is insuf cientin Tables 11 and 13, rst we run the tests to make sure thatthe interactions of system components are correct. Then werun a separate set of tests to verify that the radio stations arecorrect. The maximum interaction that the latter tests mustverify is that the radio remains on the same wavelength whenother audio system components are switched On and Off.

Setting up state-dependent tests

Let’s consider an even more complex situation, in whichthe next state of some factors

depends on a previous state.In this case, we have only aphone and a radio in the car(version E). This system hasthe following requirements:1.The ignition key hasfour positions: Off, Acc.,Run, and Crank.

2.If the ignition key is in

the Off position, every-thing else is Off.3.If the ignition key is inCrank position, thephone and radio aredisabled.4.If the phone is On, theradio cannot be heard,

even if it is in the On

position.

5.If the phone is on Hold

or Off and the radio is

On, the radio can be

heard.

6.The radio can be on the

AM, FM1, or FM2 band.

7.The radio also has a

Seek key that cansearch for the next radiostation in increasing(Seek_U) or decreasing(Seek_D) order.8.When the radio is per-forming a Seek opera-

JULY–SEPTEMBER 199959

软件测试

DESIGN VERIFICATION

60IEEE DESIGN & TEST OF COMPUTERS

软件测试

JULY–SEPTEMBER 199961

软件测试

DESIGN VERIFICATION

array that meets our needs. It can accommodate one 2-levelfactor and nine 4-level factors. For our test set, we combinecolumns 1 and 2, generating an eight-level factor, and wehave eight columns left. Table 16 contains the full orthogo-nal array de nition. (It is important in a real test environmentto specify the exact order in which the different inputs mustbe activated because they sometimes lead to different out-put state sequences.)

Completing the test set de nition

To complete the test set de nition, we must de ne the ex-pected outputs and produce the requirements for the testtraceability table.

As an example of how to de ne the expected results, seeTable 16. It preserves the states for the Radio band’s previ-ous values from the last change. In the table, the valuesmarked * do not make sense, hence they were skipped.Although the orthogonal array columns could have beenmoved without losing any test content, we left them in their

original locations for an easier understanding of how the ex-ample speci c values were lled in.

To produce the traceability table, we must list all the re-quirements and then all the tests covering that requirement,making sure that the coverage is suf cient. Table 17 is thetraceability table for the example in Table 16.

Table 18 compares the number of tests we would gener-ate using exhaustive testing with the number of tests we gen-erate using RTM. The table shows a considerable reductionin the number of tests—the more complex the problem, thegreater the reduction. All the examples, as well as the ex-perience of those who have used the technique, show thatRTM is a useful tool for the test engineer. Since one cannotafford to run all the tests that would be generated with theexhaustive method, RTM offers a valid alternative.

RTM is not a panacea. For example, one might decidethat the design verification results provide enough confi-dence in the nal product’s quality that testing factor inter-actions during manufacturing is unnecessary. If one reliesmainly on process control, there is no reason to apply themethodology. Also, for detailed timing veri cation, one-fac-tor-at-a-time testing offers a more practical approach. RTM’smain use is interaction testing—testing systems with multi-ple components and/or multiple inputs that can change atthe same time.

Furthermore, there is more than one way to apply thetechniques presented here to a given problem. The qualityof the resulting test set depends on certain factors:

ss

the availability of a well-de ned requirements speci -cation for the system under test

the test engineer’s having enough time to thoroughlyunderstand the speci cation

62IEEE DESIGN & TEST OF COMPUTERS

软件测试

s

the knowledge and experience of the person setting upthe data and choosing the orthogonal array

FINALLY,if we apply RTM only after the product is built, wedon’t take full advantage of the methodology. We obtain thefull bene t by using RTM as a tool for nalizing the design

quirements speci cation phase.

5.S. Stoica, “System Design Verification Tests—An Overview,”Lecture Series, Proc. Int’l Test Conf., IEEE Computer SocietyPress, Los Alamitos, Calif., 1999.

6.S. Stoica, “A Lifecycle Approach to Design Validation—Is It Nec-essary? Is It Feasible?,” Proc. Int’l Test Conf., IEEE CS Press, 1998.7.D. Raghavarao, Construction of Combinatorial Problems in De-sign Experiments, John Wiley & Sons, New York, 1971.

Acknowledgments

I thank Euge Greenstein and Deepak Goel, who encouraged meto write about RTM, Madhav Phadke, who introduced me to themethodology, and Marion Mahoney, who de ned the orthogonal-array format used in the article. I also owe thanks to the many RTMtrainees who gave me a better understanding of the problems testengineers face in different domains of electronics design.

is a senior technical special-software testing. Stoica has an MSc and a PhD from the Polytech-nical Institute in Bucharest, Romania.

References

1.P.J. Ross, Taguchi Methods for Quality Engineering, McGraw-Hill, New York, 1988.

2.M.S. Phadke, Quality Engineering Using Robust Design, Pren-tice-Hall, Englewood Cliffs, N.J., 1989.

3.M.S. Phadke, “Planning Ef cient Software Tests,”

Crosstalk: J.of Defense Software Engineering, Oct. 1997.

4.G. Taguchi and S. Konishi, Orthogonal Arrays and LinearGraphs, ASI Press, Dearborn, Mich., 1987.

Send questions and comments about this article to SusanaStoica, Ford Motor Co., Research and Vehicle Development, MD5015 – 1D053, 20,000 Rotunda Drive, Dearborn, MI 48121-2053; ssto-ica@

JULY–SEPTEMBER 199963

本文来源：https://www.bwwdw.com/article/nrz1.html