Opened 11 years ago

Closed 11 years ago

#458 closed enhancement (fixed)

Test engine for tickets

Reported by: Dimitar Misev Owned by: Dimitar Misev
Priority: major Milestone: 9.0
Component: systemtest Version: development
Keywords: Cc: Jinsongdi Yu, Piero Campalani, abeccati
Complexity: Medium

Description (last modified by Dimitar Misev)

Add test engine for testing open bugs reported in tickets, in source:systemtest/testcases_open/test_tickets

The source:systemtest/testcases_open/test_tickets/README summarizes how the test engine works:

====

This test suite includes test cases for bugs reported in tickets. The following 
directories are important for adding tests here:

 - queries  - files containing queries to be tested. Each file should contain 
              one query. The files should be specifically named, as the test 
              execution and reporting depends on this:
              
                NNN-TTT_test-case-name.EXT
              
              where NNN is a test case sequence number, TTT is ticket number.
              EXT indicates the type of the query, and right now can be one of
              
               * rasql
               * wcps (abstract syntax)
               * wcs (KVP URLs)
               * secore (KVP URLs)
               
              Additional bash script can accompany each test case, for any
              extra steps that need to be done before/after running the test
              case, like importing/removing testdata, or for custom check of
              results. The bash scripts should be named:
              
               * NNN-TTT_test-case-name.EXT.pre.sh (run before the test)
               * NNN-TTT_test-case-name.EXT.post.sh (run after the test)
               * NNN-TTT_test-case-name.EXT.check.sh (custom output check)
                
 - oracle   - expected data for testcases, name should be same as the name of
              the corresponding test case, with .oracle appended:
              
                NNN-TTT_test-case-name.EXT.oracle
                
 - testdata - any data used by the tests.
              Testdata from rasql tests (test_grey, test_grey2, test_rgb2)
              and petascope tests (mr, rgb, eobstest) is automatically
              inserted and available in the tests.
 
To run the tests execute test.sh. The test script always returns 0 (successfull
execution), while the summary says that a test is either PASSED if the result
is as expected, or SKIPPED if it failed (ticket still not fixed).

Once a ticket has been fixed and a tests passes, the query in queries should be
renamed to

  NNN-TTT_test-case-name.EXT.fixed

When a query is marked as fixed in this way, the test suite behaves as regular
test suite, i.e. when the test fails the test script exits with an error.

Change History (6)

comment:1 by Dimitar Misev, 11 years ago

Description: modified (diff)
Status: newaccepted

Patch submitted

comment:2 by Dimitar Misev, 11 years ago

Resolution: fixed
Status: acceptedclosed

comment:3 by abeccati, 11 years ago

Resolution: fixed
Status: closedreopened

comment:4 by abeccati, 11 years ago

Status: reopenedassigned

Changeset c75cf3eb487edaefdc1946054796e35d61cc52ae removed loading of the mean_summer_airtemp dataset for testing WMS. WMS tests now fail with empty result.
Coverage insertion has to be re-introduced for WMS tests to work.

comment:5 by Dimitar Misev, 11 years ago

Patch submitted

comment:6 by Dimitar Misev, 11 years ago

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.