next up previous
Next: RFI/radars Up: E-ALFA Hardware Previous: SDFITS notes

Options for transport/storage of datasets by E-ALFA users

The size of E-ALFA data sets raises issues concerning the mode of data delivery to systems beyond the raw data archive storage location. While NAIC may assume responsibility for delivery of the level I data products, it is also likely that the observing team will need and desire access to the raw data. Data rates may be of order 0.5 MB/sec and data sets therefore, large. These volumes may be large to many of us, but they are not extraordinary by today's general standards. (The LSST will record 18 TB/night.)

Three principle options exist for the transport of raw data: (a) network transfer, (b) magnetic tape and (c) hard disk. We note several points:
  1. The current internet connection to Arecibo can provide sustained data transfer of order 340 KB/sec (according to Arun).

  2. If the raw data are stored initially in an archive maintained at NAIC but physically located at a mainland site such as CTC or NCSA, then the access rate for Internet2 sites should be higher than 0.5 MB/sec, thus keeping up with the anticipated E-ALFA data rate.

  3. The current Mammoth drives at Arecibo have a write rate of 1-5 MB/sec, max. capacity of 60 GB/tape. The technology appears not to be moving forward (according to Arun).

  4. Numerous astronomical facilities use DLT tapes. SuperDLT drives write at 16 MB/sec up to 160 GB/tape. Tape drives cost about $3.3K and a tape costs about $80 (varies some).

  5. The price of bare hard disk is currently about $0.8/GB (for USB ~$2-3/GB) and continues to fall.

  6. While tape shipment has been the designated mode for large data volume initiatives such as VLBI and SDSS, those programs are moving toward disk-based recording systems.

  7. Current setups which allow rapid access to hard disks (>~ 20 MB/sec, i.e. Firewire) are worth about $300-$500, an order of magnitude cheaper than tape drives.

  8. IDE drives appear to be surpassing SCSI ones. 2 TB of hard disk storage can now be constructed from 8 X 250 GB IDE drives in a single enclosure. Some initial investment ($5K) is needed to provide a compatible controller (e.g. with 8 IDE channels) for such a tower of spinning disk. Compatibility at both ends is required.

  1. NAIC should archive the raw data, possibly in collaboration with partners, and provide access to these data by members of the E-ALFA science team via network transfer from a mainland location.

  2. A basic, inexpensive and easy-to-maintain Linux-based hard disk storage/retrieval system which serves typical researcher needs (e.g., budgets, resources) should be specified and developed as a collaborative effort led by NAIC.

next up previous
Next: RFI/radars Up: E-ALFA Hardware Previous: SDFITS notes
This page created and maintained by Martha Haynes.

Last modified: Wed Jun 18 08:51:10 EDT 2003