cast loses precision in petascope

[Vlad Merticariu]

in CrsComputer, in the method getNumericPixelIndicesForIrregularAxes, line 235 + 236, BigDecimal objects are created from Double numbers. This happens with loss of precision.

To avoid, create the BigDecimals from the string representation of the number:

new BigDecimal(numericSubset.getLowerLimit()) → new BigDecimal(numericSubset.getLowerLimit().toString())

[Bang Pham Huu]

this ticket when fixed as Vlad suggested will have problem when ingesting coverage irr_cube_2 first slice, the SQL query is

SELECT MIN(ps_vector_coefficients.coefficient_order), MAX(ps_vector_coefficients.coefficient_order), COUNT(ps_vector_coefficients.coefficient_order) FROM ps_vector_coefficients, ps_grid_axis, ps_coverage WHERE ps_vector_coefficients.grid_axis_id = ps_grid_axis.id AND ps_grid_axis.gridded_coverage_id = ps_coverage.id AND ps_grid_axis.rasdaman_order=2 AND ps_coverage.name='test_irr_cube_2'

AND ps_vector_coefficients.coefficient >= -2.57615745067596435546875E-12 AND ps_vector_coefficients.coefficient <= -2.57615745067596435546875E-12

and it will return 0 coeffecient and exception

before the patch, the SQL query is

SELECT MIN(ps_vector_coefficients.coefficient_order), MAX(ps_vector_coefficients.coefficient_order), COUNT(ps_vector_coefficients.coefficient_order) FROM ps_vector_coefficients, ps_grid_axis, ps_coverage WHERE ps_vector_coefficients.grid_axis_id = ps_grid_axis.id AND ps_grid_axis.gridded_coverage_id = ps_coverage.id AND ps_grid_axis.rasdaman_order=2 AND ps_coverage.name='test_irr_cube_2' 

AND ps_vector_coefficients.coefficient >= 0E-35 AND ps_vector_coefficients.coefficient <= 0E-35

so it can return 1 coeffcient (0)

So I think it should be fixed from all other places which used Double as well.

[Vlad Merticariu]

You should find out which are the real coefficients and which is the correct sql query.

[Bang Pham Huu]

Replying to vmerticariu:

You should find out which are the real coefficients and which is the correct sql query.

the second query as the coeffcient is 0.

[Vlad Merticariu]

Why does the first one produce a different value?

[Bang Pham Huu]

Replying to vmerticariu:

Why does the first one produce a different value?

What I can see here is because BigDecimal treat different between Double and String, but need time to understand correctly.

[Bang Pham Huu]

the problem is, for example BigDecimal a = new BigDecimal(double) as the result, a will have random fraction numbers after the real value of double.
e.g: double = 0.2356
then a will have value: 0.235653485834584395929090423904902349023904290349023904

so must use BigDecimal(double.toString()) wherever it is used.

[Bang Pham Huu]

The patch was applied, close ticket.