Help on using the FileMapping unit
==================================

File mapping is an easy way to share data between applications, to work with large data structures, and to be sloppy about saving not too important files which are repetitively accessed (for instance, log files). It is supported by the OS (Win9x and WinNT/2000/XP), which handles the allocation of space on disk and in memory, saving data as required.

File mapping can be done using a physical file on disk, or by using the system page file. In either case, the OS maintains a link between the disk contents and its representation in memory, such that its contents appears to be in memory to the application.


Basic usage
===========
There are essentially two steps in using a mapped file object: First, the object needs to be allocated. This is done by creating an instance of TFileMapping. This instructs the OS to set up and manage either (part of) a file, or an area of the page file. Secondly, the application creates an instance of TFileView, by calling TFileMapping.MakeView. The TFileView instance has methods for reading from memory and writing to memory, as well as direct access to the memory through its BaseAddress property. Since this memory is managed by the OS, it can not be modified in any way (ReallocMem, FreeMem, etc). It can only be written to, and read from.

If you want to share the view to the mapped object between applications, it is necessary to name the TFileMapping instance and to supply a named TMutex instance to the TFileMapping.MakeView method. The OS maintains a list of all named objects (so the map and mutex names need to be different), and if two applications request access to an object by the same name, they actually refer to the same object (provided that the access rights are compatible). In short, all applications accessing the same file or page file location need to use the exact same names for the TFileMapping instance and the TMutex instance.

There can be multiple views (TFileView instances) on a single TFileMapping instance. In accessing a large file this allows you, for example, to simultaneously read an index at the beginning of the file and some blob of data further on in the file. Moving a view around on a mapped file is not possible; in that case you should destroy the old view and create a new view elsewhere on the mapped file. Physical memory is not allocated until it is needed, the OS takes care of this. The allocation does not count towards the memory usage of the application, so this is a potential workaround for the limited process working set size (not thoroughly tested).


TMutex
======

  TMutex = class
  private
    FHandle   : THandle;
    FTimeOut  : cardinal;
  public
    constructor Create(const name: string);
    destructor  Destroy; override;

    function    Get: boolean;
    function    Release: boolean;
    property    TimeOut: cardinal read FTimeOut write FTimeOut;
  end;

TMutex is a very simple wrapper for the OS mutexes. A mutex works like a traffic light. When it's red you wait, when it's green you go. When you share a mapped file between applications it is important that the applications all look at the same traffic light, or accidents will happen. Hence the name passed to the mutex. The first application to instantiate a TMutex with a certain name actually creates the mutex, and becomes the owner. Subsequent instantiations of TMutex using the same name actually only obtain a handle to an already existing object.

Since mutex management and use is completely contained inside the TFileView class I will not elaborate too much here. The one property of interest is the TimeOut value. It defaults to 1000 milliseconds, but you might want to tweak that for your particular situation.

The the TFileView object assumes ownership of the TMutex object, so do not try to destroy the mutex. Play it safe and call something like:

MyView := MyFileMap.MakeView(TMutex.Create('MyMutex_name'), 0);


TFileMapping
============

  TFileMapping = class
  private
    FFileHandle : THandle;
    FMapHandle  : THandle;
    FSize       : cardinal;
    FHiSize     : cardinal;
    FPageSize   : cardinal;
    FProtect    : cardinal;
    FShared     : boolean;
  public
    constructor Create(const fn, name: string; sec: PSecurityAttributes;
                       protect: cardinal; size: cardinal; hisize: cardinal = 0);
    destructor  Destroy; override;

    function    MakeView(mtx: TMutex; start: cardinal; starthi: cardinal = 0;
                         size: cardinal = 0; access: cardinal = 0): TFileView;
  end;

TFileMapping creates the actual mapping object. If a file name (fn) is supplied to the constructor, the file in question is mapped. The file is opened with an access mode compatible with the protect parameter. The file is also marked for exclusive access, so the constructor will fail if the file is already opened elsewhere. The sec parameter is applied to the actual mapping object, as well as to the disk file if one is to be opened; pass nil for default security parameters. If size is set to 0, the entire file is mapped, otherwise size indicates the number of bytes to be mapped from the beginning of the file. hisize gives the high 32-bit part of the total size for files over 4Gb in size.

If no file name is specified, the page file will be used. In this case, size indicates the amount of space to use and must be greater than 0. It is advisable to use multiples of 4096, as this is the size of a single page. hisize will be ignored in this case.

If a name is given, the file map can be shared between applications. Otherwise, the file map is available only to the application that created the object. When accessing an existing file map the size argument must be accurate for the disk file size or page file area, or the creation of views on the map will fail. (There is no interface in the Windows API to query the existing file map, so the second application must "know" the size of the file map.)

The protect parameter is one of the following constants:

PAGE_READONLY   read-only access
PAGE_READWRITE  read/write access
PAGE_WRITECOPY  copy-on-write access

In addition, there are so-called section specifiers that must be bitwise OR-ed into the protect parameter. These are not really relevant to the intended use of the code, so they are not supported by it. If you specify them they will pass through to the OS, though. Check the Windows SDK help for details.

The MakeView method does exactly what it's name suggests: it creates a TFileView instance, which is a view onto the mapped file. A mutex must be supplied if the file map object is named, otherwise nil can be passed. The start/starthi parameters form an offset in bytes, which indicates where in the mapped object the view is supposed to start, relative to the start of the mapped object. Size indicates the size in bytes of the view. When size is 0 the entire mapped file is taken into view.

The access parameter specifies the desired access to the view. This must be compatible with the access mode of the mapped file. If 0 is specified (the default) the access right of the mapped file will be inherited, which is achieved through the fantastic instruction view_access := 8 div (map_access and $E). Kudos to Micro$oft. The plain constants are:

FILE_MAP_COPY   copy-on-write access
FILE_MAP_WRITE  write, no, sorry, read/write access
FILE_MAP_READ   read-only access


TFileView
=========

  TFileView = class
  private
    FFileMap  : TFileMapping;
    FMemory   : pointer;
    FCurrent  : pointer;
    FPosition : cardinal;
    FMutex    : TMutex;
    FSize     : cardinal;
  private
    procedure   SetPosition(pos: cardinal);
  public
    constructor Create(map: TFileMapping; mtx: TMutex;
                       access, size, start: cardinal; starthi: cardinal = 0);
    destructor  Destroy; override;

    function    Read(buffer: pointer; count: cardinal): cardinal;
    function    Write(buffer: pointer; count: cardinal): cardinal;
    function    WriteSafe(buffer: pointer; count: cardinal): cardinal;
    function    Flush: boolean;

    function    Lock: boolean;
    procedure   Unlock;

    property    BaseAddress: pointer read FMemory;
    property    Position: cardinal read FPosition write SetPosition;
  end;

The TFileView class does the actual work of reading from and writing to the mapped object. The constructor is already discussed in the previous section, under the MakeView method description. It is discouraged to call the constructor directly, because of the dependencies between the file map and the view. The application is responsible for disposing of the TFileView instance when done.

The Read method reads count bytes into the buffer from the current position in the mapped file. The return value is the number of bytes actually read into the buffer. The only reason why this would be less than count is when the read would extend past the end of the view. Specifying a huge value for count therefore effectively reads up to the end of the view.

The Write method writes count bytes to the view starting at the current position. If the view access mode is FILE_MAP_READ an exception will be raised. The return value is the number of bytes actually written to the view. This will be less than count if the write would go beyond the end of the view. After the write operation the affected memory pages are marked by the OS as dirty and they will evntually be written to the physical disk file. The Flush method can be used to write all dirty pages to physical storage. The WriteSafe method combines both a write and a flush of the pages affected in that particular write operation. DO NOT USE LOCK/UNLOCK WITH THE READ/WRITE/WRITESAFE METHODS.

After a read or a write the FPosition and FCurrent members are updated. Read and writes always take place at the current position. Use the Position property to navigate to a specific position in the view.

Possibly the most interesting use of the TFileView object is to use the BaseAddress property to map variables in the local application to addresses in the view. Consider the follow code:

Appl1:                                  | Appl2:
                                        |
var struct1: ^TComplexStruct;           | var struct2: ^TComplexStruct;
....                                    | ....
struct1 := localViewAppl1.BaseAddress;  | struct2 := loaclViewAppl2.BaseAddress;
....                                    | ....

The two applications both have a view at the same position to a single mapped file, in this case most likely the page file. Both point a variable of some pointer type to the BaseAddress (or anywhere else in the view) of their respective views. Writing to struct1 in Appl1 makes that data immediately available through struct2 in Appl2. Note that the BaseAddress of the respective views is likely to be different, it refers to the local virtual address space, but the physical RAM is shared. There is no need to allocate memory space for struct1/2, as it uses the memory allocated for the view. Also note that such a use does not update the FPosition and FCurrent properties. Even more important, this use is not automatically protected by the mutex of the view, as in using Read and Write. The calling application must use Lock and Unlock to protect the access. Always make sure to pair these methods, or the view will effectively become useless.


Examples
========

Creating the file mapping object and a view to it from a file for private use:

map := TFileMapping.Create('c:\autoexec.bat', '', nil, PAGE_READWRITE, 0);
view := map.MakeView(nil, 0);
....
view.Free;
map.Free;

Create a shared area on the page file, with 2 structs on the view:

map := TFileMapping.Create('', 'unique_name', nil, PAGE_READWRITE, 4096);
view := map.MakeView(TMutex.Create('unique_mutex_name'), 0);
myStruct1 := view.BaseAddress;
myStruct2 := pointer(cardinal(myStruct1) + SizeOf(myStruct1));
if view.Lock then begin  // Lock might time out
  myStruct1^.Count := 65;
  myStruct2^.Name  := 'John Doe';
  view.Unlock;
end;
....
view.Free;
map.Free;

Please note that long strings cannot be used in such structs, use short strings such as string[8]. 


Important issues
================

Creating a map of a disk file causes the file to grow if the specified size exceeds the file size. If you specify 0 for the size of the map, the entire file is mapped. Views on the map are limited in size by the largest block of contiguous virtual address space, which on NT is 2Gb minus what's already being used and 1Gb on W9x minus what's already in use.

A single file map can support multiple views. That can be used to optimize access to huge files.

Views on a shared mapped object provides so-called "coherent" data. What one view shows, is shown on every view positioned at the same location in a mapped object. Mapping files is not compatible with standard read/write operations, though. Therefore, the file is opened with exclusive access.

DO NOT SHARE FILES ON MULTIPLE MACHINES OVER A NETWORK. Files to be mapped can reside on a network, but the mutex is not network-aware, nor is the view memory shared between multiple views on different machines. This implies that writing to a networked file should not be done by multiple applications residing on different machines; it is in fact impossible. The only practical use of this code with networked files would be in managing large files by (multiple) applications on a single computer. I suspect the overhead will be such that this is not very practical. A better approach might very well be to write a custom socket app, with a server side managing the huge file, and a client side setting up required data locally on a pagefile shared object for local applications.


=========================
Patrick van Laake

pvanlaake@users.sourceforge.net