Large files and memory

Last time: Accessing and Controlling UNIX Devices

Prof. Brian D. Davison

Computer Science & Engineering, Lehigh University

Announcements

• Reminder: project 6, due on Wednesday
• Today
  • Short Quiz 9: Filesystems, Devices, and Perl
  • Return Exam #2: Range 50-100; Mean 87
  • Large files and memory

Large Files

• Unfortunately, in typical systems that permit large files, open(2) will return an EOVERFLOW error if a file larger than 2 GB is encountered.
• This is not because the file system is incapable of managing the storage of large files (typically they can).
• Associated with every open file descriptor is a file pointer (an offset) -- the location within the file where the next read() or write() will occur.
• For many years, that offet was a 32-bit integer, limiting file offsets to (2^31)-1.
   
  
• Consider the implementation of tail(1)...

Large Files

• The solution is obvious -- need a larger representation of offsets!
• and off64_t is the replacement for off_t.
• But... changing a standard type can be dangerous to programs that are unaware.
• As a result, we have three classes of programs:
  • Large-file unaware (may break when encountering large files)
  • Large-file safe (may quit safely with an error when given large files)
  • Large-file aware (know and can use large files)

Support for Large Files

• First, the best way to be certain of safety is to check the return code of all kernel function calls.
  • Old programs, if they check the return code properly (e.g., are large-file safe), will see an error value when they attempt to work with a large file.
• OSes that support large files have added transitional APIs with 64-bit functions and types, that can access both small and large files.
  • New functions are named xxx64() for every relevant function xxx()
  • New data types are xxx64_t for every relevant data type xxx_t
  • E.g., open64() and off64_t

Building Programs with Large File Support

• Default compilation continues to use old, small-file support.
  • (Sufficient to build large-file safe programs.)
• Transitional compilation environment -- both 32-bit and 64-bit functions available
  • Enables both xxx() and xxx64()
  • Set _LARGEFILE64_SOURCE to 1 before including any system headers
• Large file compilation environment
  • All xxx() source interfaces are remapped to xxx64() calls, and relevant data types are automatically converted.
  • Set _FILE_OFFSET_BITS to 64 before including any system headers
• See Sun's Large File White Paper for more details.

Useful man pages

• lf64(5) - transitional interfaces for 64-bit file offsets
  Describes 32 vs 64 bit functions and types.
• lfcompile64(5) - transitional compilation environment

32/64-bit memory models

• This semester we have been building 32 bit programs, even though our OS supports 64 bits
• You might want to convert applications with the following characteristics:
  • Can benefit from more than 4 gigabytes of virtual address space
  • Are restricted by 32-bit interface limitations
  • Can benefit from full 64-bit registers to do efficient 64-bit arithmetic
  • Can benefit from the performance improvement that the 64–bit instruction set provides

32/64-bit memory models

• In C, a long is the same size as the memory model (either 32 bits or 64 bits)
• A pointer is either 32 bits or 64 bits
• While the OS installed on the Suns is 64 bits, the default memory model for our compiler is only 32 bits.
  • If you calculate sizeof() for long or for int *, both will be 4 bytes in the 32 bit model.
  • Under the 64 bit model, they are each 8 bytes.
• You can explicitly choose one at compile time.
  • gcc -m64 ... chooses the 64 bit model
  • gcc -m32 ... chooses the 32 bit model
  • cc -xarch=generic64 ... chooses the 64 bit model for the Sun compiler
• Helpful hints are available:
  • Porting programs with gcc to 64 bit environments
  • See also the Solaris 64-bit Developer's Guide and a whitepaper on Migrating to Solaris 64-bit: 32-bit Applications and Data Model.

Memory Mapped Files

• Many operating systems permit a file to be mapped into (virtual) memory.
  • A range of memory addresses are now associated with the file.
  • Reads and writes to those memory locations are treated as reads and writes to the file.
• Useful when you need to create a very large data structure but don't have enough real memory for it.
• You can create a file of appropriate size, mmap() it, and access your data structure using pointer arithmetic instead of file operators.
• Usage:

  fd = open(filename, O_RDWR);
  char *array = mmap(0, MEMSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  ...
  munmap(array, MEMSIZE);
  close(fd); 
  

• Often simpler than many random lseek() calls on a file.

Memory Mapped Files

• Memory mapped files can also be used for inter-process communication.
• The same memory mapped file can be mapped by multiple processes.
• The mapping includes options to MAP_SHARED or MAP_PRIVATE, to determine what to do on writes.
• Effectively operates much like shared memory operations, except that changes to the memory are backed-up on disk.
   
  
• For performance reasons, can also mlock() the region of memory so that it always resides in RAM.