Autoscale testcase-provided timeout to compensate for slooow machines jv
authorJan Vrany <jan.vrany@fit.cvut.cz>
Mon, 14 Nov 2016 23:43:14 +0000
branchjv
changeset 3229ec2abb1218e
parent 318 b7f0437a6d18
child 323 8ec01ca33d6e
Autoscale testcase-provided timeout to compensate for slooow machines

Each test case has a timeout to guard against runaway tests. However
on really slow machines the timeout us not big enough. To compensate for this,
asses the "speed" of machine running tests and scale default timeout
if machine is slower than some (arbitrary) norm.

The speed assesment is done by measuring time to run (arbitrary) benchmark
code. This has the advantage to reflect actual machine load, not only
hardvare spec.

However, we may need to play with these magic numbers to make it working.
Generally a workaround.
reports/Builder__TestReport.st
     1.1 --- a/reports/Builder__TestReport.st	Sat May 28 10:40:05 2016 +0100
     1.2 +++ b/reports/Builder__TestReport.st	Mon Nov 14 23:43:14 2016 +0000
     1.3 @@ -11,7 +11,7 @@
     1.4  
     1.5  TestResult subclass:#Result
     1.6  	instanceVariableNames:'format time npassed nfailed nerror nskipped collector'
     1.7 -	classVariableNames:''
     1.8 +	classVariableNames:'TimeoutScale'
     1.9  	poolDictionaries:''
    1.10  	privateIn:TestReport
    1.11  !
    1.12 @@ -380,9 +380,16 @@
    1.13  initialize
    1.14      super initialize.
    1.15  
    1.16 -    npassed := nfailed := nerror := nskipped := 0
    1.17 +    npassed := nfailed := nerror := nskipped := 0.
    1.18 +    TimeoutScale isNil ifTrue:[ 
    1.19 +        | bench |
    1.20 +
    1.21 +        bench := Time millisecondsToRun:[3000 timesRepeat: [ 2000 factorial ]].
    1.22 +        TimeoutScale := (bench / 2500"mean value of an i5 32bit") max: 1.
    1.23 +    ].
    1.24  
    1.25      "Created: / 31-01-2013 / 13:52:33 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.26 +    "Modified: / 14-11-2016 / 23:37:40 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.27  ! !
    1.28  
    1.29  !TestReport::Result methodsFor:'running'!
    1.30 @@ -481,11 +488,12 @@
    1.31  
    1.32  !TestReport::Result methodsFor:'running-private'!
    1.33  
    1.34 -lightForkCase:aTestCase debugged: debugged timeout: timeout
    1.35 -    | thread sema stime etime timeouted error stack log logPos |
    1.36 +lightForkCase:aTestCase debugged: debugged timeout: timeoutBase
    1.37 +    | timeout thread sema stime etime timeouted error stack log logPos |
    1.38  
    1.39      Logger trace: 'Running %1>>%2' with: aTestCase nameForHDTestReport with: aTestCase selectorForHDTestReport.
    1.40      Transcript show:'F'.
    1.41 +    timeout := (timeoutBase * TimeoutScale) rounded. 
    1.42      sema := Semaphore new.
    1.43      stime := OperatingSystem getMillisecondTime.
    1.44      log := false.
    1.45 @@ -507,7 +515,7 @@
    1.46  
    1.47      etime := OperatingSystem getMillisecondTime.
    1.48      timeouted ifTrue:[
    1.49 -        error := TimeoutError new messageText: 'Timed out'.
    1.50 +        error := TimeoutError new messageText: ('Timed out (effective %1ms, base %2ms, scale %3)' bindWith: timeout with: timeoutBase with:TimeoutScale) .
    1.51          outcome result: TestResult stateError.
    1.52          format 
    1.53              writeTestCase: aTestCase outcome: outcome 
    1.54 @@ -522,13 +530,14 @@
    1.55      Transcript cr.
    1.56  
    1.57      "Created: / 12-01-2012 / 17:42:12 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.58 -    "Modified: / 29-10-2014 / 13:59:54 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.59 +    "Modified: / 16-11-2016 / 23:26:07 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.60  !
    1.61  
    1.62 -unixForkCase:aTestCase debugged: debugged timeout: timeout
    1.63 -    | pid status sema stime etime error stack suiteFailuresBefore suiteErrorsBefore log logPos |
    1.64 +unixForkCase:aTestCase debugged: debugged timeout: timeoutBase
    1.65 +    | timeout pid status sema stime etime error stack suiteFailuresBefore suiteErrorsBefore log logPos |
    1.66  
    1.67      Transcript show:'forking...'.
    1.68 +    timeout := (timeoutBase * TimeoutScale) rounded.   
    1.69      sema := Semaphore new.
    1.70      stime := OperatingSystem getMillisecondTime.
    1.71      log := false.
    1.72 @@ -578,7 +587,7 @@
    1.73                      [ log := true.
    1.74                      status isNil 
    1.75                          ifTrue:
    1.76 -                            [ error := TimeoutError new parameter:timeout.
    1.77 +                            [ error := TimeoutError new messageText: ('Timed out (effective %1ms, base %2ms, scale %3)' bindWith: timeout with: timeoutBase with:TimeoutScale) .
    1.78                              stack := 'Oops, timed out!! (timeout was ' , timeout printString , ' sec)'. ]
    1.79                          ifFalse:
    1.80                              [ status status == #signal 
    1.81 @@ -615,7 +624,7 @@
    1.82      Transcript cr.
    1.83  
    1.84      "Created: / 12-01-2012 / 17:43:07 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.85 -    "Modified: / 06-06-2014 / 00:54:14 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.86 +    "Modified: / 16-11-2016 / 23:26:36 / Jan Vrany <jan.vrany@fit.cvut.cz>"
    1.87  ! !
    1.88  
    1.89  !TestReport class methodsFor:'documentation'!