rasnet: query fails when executed right after start_rasdaman.sh

[Dimitar Misev]

I'm writing a script which automatically restarts rasdaman, and it took me awhile to figure out the issue. See below on how to reproduce.

$ start_rasdaman.sh && rasql -q 'select c from RAS_COLLECTIONNAMES AS c' --out string -d RASBASE

----------------Checking starting Rasdaman Servers----------------

start_rasdaman.sh: starting rasdaman server complex...
nohup: appending output to ‘nohup.out’
start_rasdaman.sh: starting all rasdaman servers...


start_rasdaman.sh: done.

rasql: rasdaman query tool v1.0, rasdaman v9.1.0-g1c8a741 -- generated on 04.11.2015 16:50:03.
opening database RASBASE at localhost:7001...terminate called after throwing an instance of 'std::runtime_error'
  what():  There is no available server for the client.
Aborted (core dumped)

[Dimitar Misev]

Adding

sleep 0.1

after start_rasdaman.sh fixes the issue, but it would be better to fix it within rasdaman - once start_rasdaman.sh is executed, rasdaman should be ready.

[Alex Toader]

rasmgr is ready, but the rasserver processes have not yet completed the startup sequence.

The more important question is: What should happen in rasmgr when a request for a new server is received but there are no available servers? Should the client wait until a server becomes available? Should the rasmgr attempt to find an available server a fixed number of times and if no server is found return an error?

[Dimitar Misev]

Perhaps rasmgr should wait for all rasservers to finish starting before it finishes its own startup?

[Dimitar Misev]

Also notice the Aborted (core dumped) at the end, not quite sure whether it happens within rasnet.

[Peter Baumann]

"core dump" is critical. The startup sequence is indeed depending on executing the config commands first. We might think about a specific command for enabling/disabling queries (not server control), question is whether it is worth the effort. Service ops just need to wait for 1 sec at most before sending queries.

Wrt Alex's question: currently we have a synchronous paradigm. Once we go async we could think about functionality as described.

[Dimitar Misev]

Replying to atoader:

rasmgr is ready, but the rasserver processes have not yet completed the startup sequence.

Yes I understand this, but the script is start_rasdaman.sh and obviously rasdaman is not started after calling it. I think it would be best if rasmgr simply waits for the servers before it returns.

The more important question is: What should happen in rasmgr when a request for a new server is received but there are no available servers? Should the client wait until a server becomes available? Should the rasmgr attempt to find an available server a fixed number of times and if no server is found return an error?

This is a separate concern and should go into another ticket I'd say.

[Peter Baumann]

indeed, please file a separate ticket for the dump.

[Dimitar Misev]

Now, when you start a rasserver process, because there is no function to determine when the process has started, rasmgr waits for a registration message from the server announcing that it is online. This incurs a delay of several milliseconds. Also, this process is fully controlled by server manager inside rasmgr that has a thread that checks, at a set interval, the list of registered servers and starts or shutsdown servers.

Because the client request is instant and there is no waiting involved, there is no time for the process described above to take place.

Ok so in rasmgr's main() we just wait until a registration message has arrived before declaring that rasmgr is ready to receive messages from clients? Maybe I'm missing something though.

Alternatively if option 2. in #1061 is supported then that would work as well fixing this ticket.

[Alex Toader]

When you start rasmgr from the script, you do not wait for any particular signal from rasmgr before you proceed. So even if I wait in rasmgr main for at least a server to register, it wouldn't change anything.

It looks like option 2 is a cleaner way to fix this.

[Dimitar Misev]

Ah yes you're right, we start it in the background. Agreed about option 2.