Opened 3 months ago

Closed 3 months ago

#1558 closed defect (fixed)

encode to csv and json outputs rounded values for float and double, but not always

Reported by: bbell Owned by: bbell
Priority: major Milestone: 9.4
Component: conversion Version: development
Keywords: Cc: dmisev
Complexity: Medium

Description

creating a structure of type {float, double, octet, double, short}, and ingesting a file with 00000001 for every byte (such a file is attached) yields rounded-off values for floats and doubles when using encode with the "csv" or "json" parameter, but inconsistently (the first value of the first cell is not rounded).

suppose you have inserted the data into a collection named "test_struct"

rasql -q 'select encode(c, "json") from test_struct as c' --out string

has an output of the form:

$ rasql -q 'select encode(c,"json") from test_struct as c' --out string
rasql: rasdaman query tool v1.0, rasdaman 9.4.0.
opening database RASBASE at localhost:7001...ok
Executing retrieval query...ok
Query result collection has 1 element(s):
  Result object 1: [[["2.36943e-38 0.000000 1 0.000000 257"],["0.000000 0.000000 1 0.000000 257"],["0.000000 0.000000 1 0.000000 257"],["0.000000 0.000000 1 0.000000 257"],

(repeats with 2.36943e-38 never appearing again...)

It is worth remarking that the double value for the following 8 bytes:

00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001

is 7.7486e-304

Most likely, this problem has to do with precision handling in csv/json output.

Attachments (1)

file23k (22.5 KB) - added by bbell 3 months ago.
binary file with every byte 00000001

Download all attachments as: .zip

Change History (8)

Changed 3 months ago by bbell

binary file with every byte 00000001

comment:1 Changed 3 months ago by bbell

The culprit here is the use of "fixed" on line csv.cc:208

    case r_Type::DOUBLE:
        f << fixed << static_cast<double>(ptr->get_double(val));

Dropping that stream operation yields the expected outputs; however, is this a feature rather than a flaw?

fixed is coming from ios_base.h:

  // [27.4.5.4] floatfield manipulators
  /// Calls base.setf(ios_base::fixed, ios_base::floatfield).
  inline ios_base&
  fixed(ios_base& __base)
  {
    __base.setf(ios_base::fixed, ios_base::floatfield);
    return __base;
  }

comment:2 Changed 3 months ago by dmisev

Perhaps check with git whatchanged first if there's some particular patch that put fixed there.

In any case, I'm also not sure whether it makes more sense with fixed or without.

comment:3 Changed 3 months ago by dmisev

But removing it will certainly be a breaking change.

comment:4 Changed 3 months ago by bbell

Agreed that it would not be a minor change. One other thing I noticed is that an r_Double is a typedef for double, and we are also throwing a static_cast<double> there where it (should?) not be needed.

What I can do for now is try to find out why the first element of the first struct is treated a bit differently (I suspect it's viewed as a float until the struct's first double is encountered and supersedes, because "fixed" is only being used on doubles, not on floats). I'll look into it a bit more and update again here.

In any case, should the behaviours of float and double both be the same for json/csv output?

comment:5 Changed 3 months ago by bbell

After discussion at yesterday's meeting, we will implement the change to be consistent for both float and double, and in particular, the output will reflect the previous behaviour of floats (i.e. no use of 'fixed' via ios_base.h)

comment:6 follow-up: Changed 3 months ago by dmisev

  • Component changed from undecided to conversion
  • Milestone set to 9.4

Should it be closed?

comment:7 in reply to: ↑ 6 Changed 3 months ago by bbell

  • Resolution set to fixed
  • Status changed from new to closed

Replying to dmisev:

Should it be closed?

I think so.

Note: See TracTickets for help on using tickets.