Opened 6 months ago

Closed 4 months ago

#1506 closed defect (fixed)

data ingestion with directql has a segfault

Reported by: bbell Owned by: bbell
Priority: major Milestone: Future
Component: undecided Version: development
Keywords: directql, ingest, segfault Cc: dmisev
Complexity: Medium

Description

In directql.cc, in the doStuff() function, there seems to be some issues with the way in which the char* pointer named fileContents is handled.

Ideally, the use of smart pointers (e.g. shared_ptr and unique_ptr) could be implemented in this section of code. A simpler, albeit more rudimentary, fix could be to copy the way that memory allocation is handled in rasql.cc's doStuff() function; however, it is important to keep in mind that these functions work differently!

trace for the problem:

rasql -q 'create collection test_tmp GreySet?' --user <> --passwd <>
valgrind directql -q 'insert into test_tmp values decode($1)' -f mr_1.png --user <> --passwd <>

you should get 5-6 errors here, but resolving the problem in directql.cc should fix all but 1-2 of them.

Change History (7)

comment:1 Changed 5 months ago by bbell

Revealed a bit more information about this problem today:

lines 1040-1045 of directql.cc experience the following undefined behaviour:

the data processed in server->insertTile is considered transient, so is not inserted into the collection

the data pointer processed in server->executeInsert is the null pointer.

Most of this behaviour arises in servercomm2.cc, in particular in the insertTile method there (not to be confused with other insertTile methods in the source code).

One way forward might be to break this part of directql.cc up into two cases. One case could handle raw, initial data insertion, while the other case could handle all other behaviour (updates + transient data operations).

Last edited 5 months ago by bbell (previous) (diff)

comment:2 Changed 4 months ago by bbell

commenting out line 1129 of directql.cc allows one to insert data successfully with directql. Of course, doing so leads to some memory leaks since that line is where the pointer to the file data, after it has been read into memory, is freed.

comment:3 follow-up: Changed 4 months ago by dmisev

So what exactly is the memory error that happens?

If we try to free the pointer, there's an error. But if we don't, there's a leak. Something is not right.

comment:4 in reply to: ↑ 3 Changed 4 months ago by bbell

Replying to dmisev:

So what exactly is the memory error that happens?

If we try to free the pointer, there's an error. But if we don't, there's a leak. Something is not right.

we get a double free or corruption error

comment:5 Changed 4 months ago by bbell

  • Resolution set to fixed
  • Status changed from new to closed

comment:6 Changed 4 months ago by dmisev

  • Resolution fixed deleted
  • Status changed from closed to reopened

We still don't have directql ingestion test.

comment:7 Changed 4 months ago by bbell

  • Resolution set to fixed
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.