Large files and memory

Last time: More Pipes

Prof. Brian D. Davison

Computer Science & Engineering, Lehigh University

Additional Credits

Announcements

• Reminder: Homework #8 is due today.
• Reminder: Project 10 is due Wednesday.
• Today
  • Short Quiz #9
  • Large files and memory
  • Course Evaluations

Large Files

• Unfortunately, in typical systems that permit large files, open(2) will return an EOVERFLOW error if a file larger than 2 GB is encountered.
• This is not because the file system is incapable of managing the storage of large files (typically they can).
• Associated with every open file descriptor is a file pointer (an offset) -- the location within the file where the next read() or write() will occur.
• For many years, that offset was a 32-bit integer, limiting file offsets to (2^31)-1.
   
  
• Consider the implementation of tail(1)...

Large Files

• The solution is obvious -- need a larger representation of offsets!
• and off64_t is the replacement for off_t.
• But... changing a standard type can be dangerous to programs that are unaware.
• As a result, we have three classes of programs:
  • Large-file unaware (may break when encountering large files)
  • Large-file safe (may quit safely with an error when given large files)
  • Large-file aware (know and can use large files)

Support for Large Files

• First, the best way to be certain of safety is to check the return code of all kernel function calls.
  • Old programs, if they check the return code properly (e.g., are large-file safe), will see an error value when they attempt to work with a large file.
• OSes that support large files have typically added transitional APIs with 64-bit functions and types, that can access both small and large files.
  • New functions are named xxx64() for every relevant function xxx()
  • New data types are xxx64_t for every relevant data type xxx_t
  • E.g., open64() and off64_t

Building Programs with Large File Support

• Default compilation continues to use old, small-file support.
  • (Sufficient to build large-file safe programs.)
• Transitional compilation environment -- both 32-bit and 64-bit functions available
  • Enables both xxx() and xxx64()
  • Set _LARGEFILE64_SOURCE to 1 before including any system headers
• Large file compilation environment
  • All xxx() source interfaces are remapped to xxx64() calls, and relevant data types are automatically converted.
  • Set _FILE_OFFSET_BITS to 64 before including any system headers
• See Sun's Large File White Paper for more details.

Useful man pages

• lf64(5) - transitional interfaces for 64-bit file offsets
  Describes 32 vs 64 bit functions and types.
• lfcompile64(5) - transitional compilation environment

32/64-bit memory models

• This semester we have been building 32 bit programs, even though our OS supports 64 bits
• You might want to convert applications with the following characteristics:
  • Can benefit from more than 4 gigabytes of virtual address space
  • Are restricted by 32-bit interface limitations
  • Can benefit from full 64-bit registers to do efficient 64-bit arithmetic
  • Can benefit from the performance improvement that the 64-bit instruction set provides

32/64-bit memory models

• In C, a long is the same size as the memory model (either 32 bits or 64 bits)
• A pointer can vary: either 32 bits or 64 bits
• While the OS installed on the Suns is 64 bits, the default memory model for our compiler is only 32 bits.
  • If you calculate sizeof() for long or for int *, both will be 4 bytes in the 32 bit model.
  • Under the 64 bit model, they are each 8 bytes.
• You can explicitly choose one at compile time.
  • gcc -m64 ... chooses the 64 bit model (-m32 for 32 bit)
• Helpful hints are available:
  • Porting programs with gcc to 64 bit environments
  • See also the Solaris 64-bit Developer's Guide.

Memory Mapped Files

• Many operating systems permit a file to be mapped into (virtual) memory.
  • A range of memory addresses are now associated with the file.
  • Reads and writes to those memory locations are treated as reads and writes to the file.
• Useful when you need to create a very large data structure but don't have enough real memory for it.
• You can create a file of appropriate size, mmap() it, and access your data structure using pointer arithmetic instead of file operators.

Memory Mapped Files

• Usage:

  #include <sys/mman.h>
  
  fd = open(filename, O_RDWR);
  char *array = mmap(0, MEMSIZE, PROT_READ | PROT_WRITE, 
                     MAP_SHARED, fd, 0);
  ...
  munmap(array, MEMSIZE);
  close(fd); 
  

• Often simpler than many random lseek()+read() calls on a file.

Memory Mapped Files

• Memory mapped files can also be used for inter-process communication.
• The same memory mapped file can be mapped by multiple processes.
• The mapping includes options to MAP_SHARED or MAP_PRIVATE, to determine what to do on writes.
• Effectively operates much like shared memory operations, except that changes to the memory are backed-up on disk.
   
  
• For performance reasons, one can also mlock() the region of memory so that it always resides in RAM.

Course Evaluation

• We value your honesty!
• These evaluations are seen
  • by the department chair and dean to evaluate and compare faculty,
  • by the dean, to compare departments,
  • and perhaps by the provost, to compare colleges.
• I won't see them until sometime next semester. But I'd also like your feedback on a few specific questions...

Course Evaluation

• Additional questions:
1. How prepared are you for future work requiring programming in C?
2. How much have you learned about using and programming in UNIX?
3. What did you like best about the course?
4. What should be changed for the next time the course is taught?