Enhancement of petascope regression test suite

[abeccati]

As part of the proposed projects we have regression test refactoring

Wrt regression testing we need improvement especially with petascope tests (documented here)

Most compelling is the inclusion of an oracle or similar structure for evaluating query output and enable also queries that purposefully generate errors to be part of successful test cases.

Currently the test system is under improvement and the reference test.sh script is the one into test_wcps directory. The development direction is to make that script flexible enough to be directly linked from the other directories (test_wms, test_wcs) and run on the local data on these. This approach is already used in the test_wms part and should be adopted also for the test_wcs as soon as it is proven stable.

[Dimitar Misev]

Note that I have submitted some systemtest refactoring patch, you should do your work after applying that on.

[Dimitar Misev]

Note I've updated a bit the test page at wiki:RasdamanTestSuites#Petascopetests

The configuration file for petascope and the whole systemtest now is source:systemtest/conf/test.cfg

[Piero Campalani]

As soon as the oracle-based testing system is set up, as a next step we should add (many) error-prone queries and verify they actually return an error/exception (this will help us reviewing error messages and identifying cumbersome ones, e.g. see #394).

[Dimitar Misev]

That should go in another ticket Piero, I don't think Nikolce has the background to add such tests.

[Piero Campalani]

Yes, it was general next-step, not Nikolce's next step
It was a note, then we can either create a new ticket or go on with this one, whatever.

[abeccati]

I strongly second Piero's proposal and created #396 for that purpose (we should keep tasks/ticket as "atomic" tasks as much as we can to ease management)
Let's finalize infrastructure first with this ticket.

@nkolev: Please "accept" the ticket (accept option at bottom of modify ticket) so we can track active assignment.

[Dimitar Misev]

patch submitted

[Piero Campalani]

Missing ref from comment:8 : changeset:a6625cd.

Now WCS tests are left.

[abeccati]

Also WMS tests are missing, basic ones should be included. Meanwhile patch submitted to master (2013-07-23 14:02:17) for WMS inclusion: links directly to the test.sh engine script.

[abeccati]

Included first test passing getCapabilities and loading WMS test data "mean_summer_airtemp" as laid out by Piero at ​http://rasdaman.org/wiki/WmsImportTools
Please note that the WMS engine is the same as the wcps one (to be used as refernce) and the patch (2013-07-23 19:31:01) extends it to detect service name according to the base directory where it is linked from (e.g. test_wms). This can be reused for all services to ease maintenance.

[abeccati]

Changing to assigned since some changes have been committed meantime and no active progress seen in some time. nkolev please re-accept when working on it again and read the updated doc at RasdamanTestSuites to get aligned.
Adding also Jin (systemtests component owner) in cc for overall test systems coordination.

[Dimitar Misev]

Alan it seems like your patch has broken the WCPS test, please check. E.g.

src/systemtest/testcases_petascope/test_wcps$ ./test.sh 
SCRIPT_DIR is set to: systemtest/testcases_petascope/test_wcps
Old logfile found, copying it to systemtest/testcases_petascope/test_wcps/log.save
test.sh: starting test at Sat Aug  3 19:20:43 CEST 2013
test.sh: 
Service name is: wms
Service endpoint is set to: http://localhost:8080/petascope/wms
test.sh: rgb already imported.
test.sh: mr already imported.
test.sh: eobstest already imported.
test.sh: mean_summer_airtemp already imported.

--------------------------------------------------------
running test: 01-just_trim.test

for c in ( rgb ) return encode( c[ x(0:100), y(0:100) ], "png" )

KVP request: http://localhost:8080/petascope/wms?for c in ( rgb ) return encode( c[ x(0:100), y(0:100) ], "png" )


real	0m0.698s
user	0m0.004s
sys	0m0.008s
test.sh: byte comparison
/home/dimitar/projects/rasdaman/community/src/systemtest/testcases_petascope/test_wcps/oracle/01-just_trim.oracle.data /home/dimitar/projects/rasdaman/community/src/systemtest/testcases_petascope/test_wcps/output/01-just_trim.test.out differ: byte 1, line 1
test.sh:  ->  QUERY FAILED

Also not clear why is SCRIPT_DIR printed.

[Dimitar Misev]

test.sh:60 has

SVC_NAME=$(echo test_wms | cut -d "_" -f 2)

[Dimitar Misev]

Ok I submitted patch that fixes it.

[Dimitar Misev]

Actually closing this; for further issues please open new ticket.