Opened 5 years ago
Closed 5 years ago
#2258 closed defect (fixed)
improve netcdf import (analysis) speed
Reported by: | Dimitar Misev | Owned by: | Dimitar Misev |
---|---|---|---|
Priority: | major | Milestone: | 10.0 |
Component: | wcst_import | Version: | 9.8 |
Keywords: | Cc: | Bang Pham Huu | |
Complexity: | Medium |
Description
Analysing netcdf files can be very slow if expressions are used in the ingredients file, e.g.:
Analyzing file (1/20): processed:2017090100.m01.nc ... Elapsed time: 13.337 s.
it is much slower (13.3s) than the rasdaman import (1.14s)
[2020-03-14 14:11:35] 1/20 - file 'processed:2017090100.m01.nc' - grid domains [0,0,0:24,0:5,0:420,0:460] of size 71.12 MB; Total time to ingest file 1.14 s @ 62.39 MB/s. Progress: [##----------------------------] 1/20 5.00%
The problem is that numpy array is converted into python list and then operations like min/max are done on the list. This is slow, such operations should be performed directly on numpy arrays.
Note:
See TracTickets
for help on using tickets.