Changes between Version 2 and Version 6 of Ticket #1073


Ignore:
Timestamp:
Nov 18, 2015, 5:40:30 PM (8 years ago)
Author:
Dimitar Misev
Comment:

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #1073 – Description

    v2 v6  
    3030Between 10,000 and 100,000 files per directory seems like a good number well supported across filesystems. If we take 100,000 on ext3 that gives us a lower limit of 3 billion tiles.
    3131
    32 Based on this my proposal is to distribute tiles in 100,000 per directory, so that we have this organization:
     32= Single-level nesting =
     33
     34Distributing tiles in 100,000 per directory we have this organization:
    3335
    3436{{{
    … …  
    5355The subdirectory index in TILES is dir_index = tile_index / 100,000. The 100,000 number can be a compile time constant that can be adjusted as necessary. By default it is maybe better if it is `2^16` or `2^17` so that the dir_index can be computed with a fast bit shift.
    5456
    55 I would like to stay away from creating complicated tree-like schemes nesting multiple subdirectories. It's the job of the filesystem to handle this load, if we ever reach some limits with this scheme on a particular filesystem it seems very unlikely that we'll be able to work around it ourselves, without actually adapting the filesystem underneat.
     57With 30,000 subdirs this gives us a "lower" limit of ~12 PB (with 4MB tile size).
     58
     59= Two-level nesting =
     60
     61{{{
     62$RASDATA
     63 |_ RASBASE
     64 |_ TILES
     65      |  dir1_index
     66      |_ 0
     67      |  |  dir2_index
     68      |  |_ 0
     69      |  |  |_ 1
     70      |  |  |_ 2
     71      |  |  |_ 3
     72      |  |  |_ ..
     73      |  | 
     74      |  |_ 1
     75      |  |  |_ 100,000
     76      |  |  |_ 100,002
     77      |  |  |_ ..
     78      |  | 
     79      |  |_ 2
     80      |  |_ ...
     81      |  |_ 32,767
     82      |   
     83      |_ 1
     84      |  |_ 32,768
     85      |  |_ 32,769
     86      |  |_ ...
     87      | 
     88      |_ ...
     89}}}
     90
     91The subdirectory index in TILES is:
     92 - dir2_index = tile_index / max_files (100,000)
     93 - dir1_index = dir2_index / max_dirs (32,768)
     94 
     95This sets a "lower" limit of ~400 EB (with 4MB tiles).
     96
     97= Backwards compatibility =
    5698
    5799Rasdaman could support both structures (old and new) with a simple check at startup; in v10.0 we can enforce this structure. update_db.sh can be executed to migrate to the new directory structure.