Large files and memory

Last time: Accessing and Controlling UNIX Devices

Unfortunately, in typical systems that permit large files, open(2) will return an EOVERFLOW error if a file larger than 2 GB is encountered.
This is not because the file system is incapable of managing the storage of large files (typically they can).
Associated with every open file descriptor is a file pointer (an offset) -- the location within the file where the next read() or write() will occur.
For many years, that offet was a 32-bit integer, limiting file offsets to (2^31)-1.
Consider the implementation of tail(1)...

The solution is obvious -- need a larger representation of offsets!
and off64_t is the replacement for off_t.
But... changing a standard type can be dangerous to programs that are unaware.
As a result, we have three classes of programs:
- Large-file unaware (may break when encountering large files)
- Large-file safe (may quit safely with an error when given large files)
- Large-file aware (know and can use large files)

First, the best way to be certain of safety is to check the return code of all kernel function calls.
- Old programs, if they check the return code properly (e.g., are large-file safe), will see an error value when they attempt to work with a large file.
OSes that support large files have added transitional APIs with 64-bit functions and types, that can access both small and large files.
- New functions are named xxx64() for every relevant function xxx()
- New data types are xxx64_t for every relevant data type xxx_t
- E.g., open64() and off64_t

Default compilation continues to use old, small-file support.
- (Sufficient to build large-file safe programs.)
Transitional compilation environment -- both 32-bit and 64-bit functions available
- Enables both xxx() and xxx64()
- Set _LARGEFILE64_SOURCE to 1 before including any system headers
Large file compilation environment
- All xxx() source interfaces are remapped to xxx64() calls, and relevant data types are automatically converted.
- Set _FILE_OFFSET_BITS to 64 before including any system headers
See Sun's Large File White Paper for more details.

This semester we have been building 32 bit programs, even though our OS supports 64 bits
You might want to convert applications with the following characteristics:
- Can benefit from more than 4 gigabytes of virtual address space
- Are restricted by 32-bit interface limitations
- Can benefit from full 64-bit registers to do efficient 64-bit arithmetic
- Can benefit from the performance improvement that the 64–bit instruction set provides

In C, a long is the same size as the memory model (either 32 bits or 64 bits)
A pointer is either 32 bits or 64 bits
While the OS installed on the Suns is 64 bits, the default memory model for our compiler is only 32 bits.
- If you calculate sizeof() for long or for int *, both will be 4 bytes in the 32 bit model.
- Under the 64 bit model, they are each 8 bytes.
You can explicitly choose one at compile time.
- gcc -m64 ... chooses the 64 bit model (-m32 for 32 bit)
Helpful hints are available:
- Porting programs with gcc to 64 bit environments
- See also the Solaris 64-bit Developer's Guide and a whitepaper on Migrating to Solaris 64-bit: 32-bit Applications and Data Model.

Many operating systems permit a file to be mapped into (virtual) memory.
- A range of memory addresses are now associated with the file.
- Reads and writes to those memory locations are treated as reads and writes to the file.
Useful when you need to create a very large data structure but don't have enough real memory for it.
You can create a file of appropriate size, mmap() it, and access your data structure using pointer arithmetic instead of file operators.

Usage:

fd = open(filename, O_RDWR);
char *array = mmap(0, MEMSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
...
munmap(array, MEMSIZE);
close(fd);

Many OSes use mmap() to run programs. Try truss ls.

Memory mapped files can also be used for inter-process communication.
The same memory mapped file can be mapped by multiple processes.
The mapping includes options to MAP_SHARED or MAP_PRIVATE, to determine what to do on writes.
Effectively operates much like shared memory operations, except that changes to the memory are backed-up on disk.
For performance reasons, can also mlock() the region of memory so that it always resides in RAM.