Opened 4 years ago

Closed 3 years ago

#442 closed defect (fixed)

Occassional error on rasimport

Reported by: jpass Owned by: aherzig
Priority: major Milestone: 9.0.x
Component: rasgeo Version: 8.4
Keywords: rasimport Cc: dmisev, mase
Complexity: Medium

Description (last modified by jpass)

When running a batch import using python I get occasional errors like:

ERROR - RasdamanHelper2::updateImage, l. 685: image update failed: RasManager Error: Write transaction in progress, please retry again later.

ERROR - rimport::main, l. 1371: RasManager Error: Write transaction in progress, please retry again later.

The batch process is using python subprocess like...

p3 = subprocess.Popen(rasimport,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p3.wait()

Where rasimport is something like:

"time rasimport -d " + str(tile) + "/" +  str(subtile) + " -coll pga_cir -t RGBImage:RGBSet -s jpg -oid 6145"

It appears that rasimport is reporting it is finished, when in fact the processing is not complete.

Change History (10)

comment:1 Changed 4 years ago by jpass

  • Description modified (diff)

comment:2 Changed 4 years ago by dmisev

  • Cc dmisev added
  • Milestone changed from 8.5 to 9.0
  • Owner changed from dmisev to aherzig
  • Status changed from new to assigned

It could as well be that some error during the import happened which didn't clear the transaction properly. You could try restarting the server and retrying the update? Ultimately for best stability and control I'd use rasql rather than rasimport.

comment:3 Changed 4 years ago by jpass

The problem with retrying the update is that this batch import ran for about 5 days and ingested over 1Tb of data.

comment:4 Changed 4 years ago by dmisev

Some error happened during the import. Did you check the logs of rasdaman and return code of rasimport?

comment:5 Changed 4 years ago by jpass

I tried to check the logs (they are very big), but didn't spot anything unusual though it is possible I missed something; what string should I search for?

comment:6 follow-up: Changed 4 years ago by dmisev

"Error", but most likely there was some segfault or error in postgres which is not even logged, so it will be hard to figure out. I assume rasimport is not prepared for handling such unexpected issues and then ends up with inconsistent transaction state.

comment:7 in reply to: ↑ 6 Changed 4 years ago by jpass

Replying to dmisev:

"Error", but most likely there was some segfault or error in postgres which is not even logged, so it will be hard to figure out. I assume rasimport is not prepared for handling such unexpected issues and then ends up with inconsistent transaction state.

I tried "Error" but didn't find anything

comment:8 Changed 3 years ago by mase

  • Cc mase added
  • Component changed from undecided to rasgeo

comment:9 Changed 3 years ago by mase

Tested with v9.0.0 and v9.0.1 and same problem remains although intermittent and not repeatable when running same set of loading commands.

comment:10 Changed 3 years ago by mase

  • Resolution set to fixed
  • Status changed from assigned to closed

I've been running imports with v9.0.3 and v9.0.4 for a while now and although the intermittent nature of the problem makes it difficult to be certain I think it has been long enough to decide that it doesn't seem to be repeating. Will close for now and re-open if it re-occurs.

Note: See TracTickets for help on using tickets.