Opened 7 months ago

Closed 7 months ago

#1460 closed defect (fixed)

cast loses precision in petascope

Reported by: vmerticariu Owned by: bphamhuu
Priority: major Milestone: 9.3
Component: petascope Version: development
Keywords: Cc:
Complexity: Medium

Description

in CrsComputer?, in the method getNumericPixelIndicesForIrregularAxes, line 235 + 236, BigDecimal? objects are created from Double numbers. This happens with loss of precision.

To avoid, create the BigDecimals? from the string representation of the number:

new BigDecimal?(numericSubset.getLowerLimit()) -> new BigDecimal?(numericSubset.getLowerLimit().toString())

Change History (7)

comment:1 Changed 7 months ago by bphamhuu

this ticket when fixed as Vlad suggested will have problem when ingesting coverage irr_cube_2 first slice, the SQL query is

SELECT MIN(ps_vector_coefficients.coefficient_order), MAX(ps_vector_coefficients.coefficient_order), COUNT(ps_vector_coefficients.coefficient_order) FROM ps_vector_coefficients, ps_grid_axis, ps_coverage WHERE ps_vector_coefficients.grid_axis_id = ps_grid_axis.id AND ps_grid_axis.gridded_coverage_id = ps_coverage.id AND ps_grid_axis.rasdaman_order=2 AND ps_coverage.name='test_irr_cube_2'

AND ps_vector_coefficients.coefficient >= -2.57615745067596435546875E-12 AND ps_vector_coefficients.coefficient <= -2.57615745067596435546875E-12

and it will return 0 coeffecient and exception

before the patch, the SQL query is

SELECT MIN(ps_vector_coefficients.coefficient_order), MAX(ps_vector_coefficients.coefficient_order), COUNT(ps_vector_coefficients.coefficient_order) FROM ps_vector_coefficients, ps_grid_axis, ps_coverage WHERE ps_vector_coefficients.grid_axis_id = ps_grid_axis.id AND ps_grid_axis.gridded_coverage_id = ps_coverage.id AND ps_grid_axis.rasdaman_order=2 AND ps_coverage.name='test_irr_cube_2' 

AND ps_vector_coefficients.coefficient >= 0E-35 AND ps_vector_coefficients.coefficient <= 0E-35

so it can return 1 coeffcient (0)

So I think it should be fixed from all other places which used Double as well.

comment:2 follow-up: Changed 7 months ago by vmerticariu

You should find out which are the real coefficients and which is the correct sql query.

comment:3 in reply to: ↑ 2 Changed 7 months ago by bphamhuu

Replying to vmerticariu:

You should find out which are the real coefficients and which is the correct sql query.

the second query as the coeffcient is 0.

comment:4 follow-up: Changed 7 months ago by vmerticariu

Why does the first one produce a different value?

comment:5 in reply to: ↑ 4 Changed 7 months ago by bphamhuu

Replying to vmerticariu:

Why does the first one produce a different value?

What I can see here is because BigDecimal? treat different between Double and String, but need time to understand correctly.

comment:6 Changed 7 months ago by bphamhuu

the problem is, for example BigDecimal? a = new BigDecimal?(double) as the result, a will have random fraction numbers after the real value of double.
e.g: double = 0.2356
then a will have value: 0.235653485834584395929090423904902349023904290349023904

so must use BigDecimal?(double.toString()) wherever it is used.

comment:7 Changed 7 months ago by bphamhuu

  • Milestone set to 9.3
  • Resolution set to fixed
  • Status changed from new to closed

The patch was applied, close ticket.

Note: See TracTickets for help on using tickets.