data ingestion with directql has a segfault

[bbell]

In directql.cc, in the doStuff() function, there seems to be some issues with the way in which the char* pointer named fileContents is handled.

Ideally, the use of smart pointers (e.g. shared_ptr and unique_ptr) could be implemented in this section of code. A simpler, albeit more rudimentary, fix could be to copy the way that memory allocation is handled in rasql.cc's doStuff() function; however, it is important to keep in mind that these functions work differently!

trace for the problem:

rasql -q 'create collection test_tmp GreySet' —user ≠ —passwd ≠
valgrind directql -q 'insert into test_tmp values decode($1)' -f mr_1.png —user ≠ —passwd ≠

you should get 5-6 errors here, but resolving the problem in directql.cc should fix all but 1-2 of them.

[bbell]

Revealed a bit more information about this problem today:

lines 1040-1045 of directql.cc experience the following undefined behaviour:

the data processed in server→insertTile is considered transient, so is not inserted into the collection

the data pointer processed in server→executeInsert is the null pointer.

Most of this behaviour arises in servercomm2.cc, in particular in the insertTile method there (not to be confused with other insertTile methods in the source code).

One way forward might be to break this part of directql.cc up into two cases. One case could handle raw, initial data insertion, while the other case could handle all other behaviour (updates + transient data operations).

[bbell]

commenting out line 1129 of directql.cc allows one to insert data successfully with directql. Of course, doing so leads to some memory leaks since that line is where the pointer to the file data, after it has been read into memory, is freed.

[Dimitar Misev]

So what exactly is the memory error that happens?

If we try to free the pointer, there's an error. But if we don't, there's a leak. Something is not right.

[bbell]

Replying to dmisev:

So what exactly is the memory error that happens?

If we try to free the pointer, there's an error. But if we don't, there's a leak. Something is not right.

we get a double free or corruption error

[Dimitar Misev]

We still don't have directql ingestion test.