RFC45:GDAL数据集和栅格带作为虚拟内存映射
作者：连鲁奥
联系人：spatialys.com上的even dot rouault
状态：通过、实施
总结
本文档建议对GDAL进行添加，以便GDAL数据集和栅格带的图像数据可以被视为虚拟内存映射，希望使用起来更简单。
理论基础
当需要从GDAL数据集或栅格带中读取或写入图像数据时，必须对读取或写入的感兴趣区域使用RasterIO（）接口。对于小图像，最方便的解决方案通常是在单个请求中读取/写入整个图像，其中感兴趣的区域是完整的栅格范围。对于较大的图像，尤其是当它们不完全适合RAM时，这是不可能的，如果要对整个图像进行操作，则必须使用窗口策略来避免内存问题：通常通过按扫描线或按平铺图像的块进行扫描线（或扫描线组）。当算法需要访问每个感兴趣像素周围的像素邻域时，这会使算法的编写更加复杂，因为必须考虑这个额外窗口的大小，从而导致感兴趣区域重叠。没有什么是不能解决的，但这需要一些额外的思考，从以下主要目的转移注意力。
此RFC的建议添加是使图像数据显示为使用指针访问的单个数组，而不受与数据集大小相关的RAM大小的限制（CPU体系结构和操作系统施加的限制除外）
技术方案
低级机械：cpl_virtualem.h
支持这种新功能的低级机器是一个cplvirtualem对象，它表示一个虚拟内存区域（在Linux上，是mmap（）函数分配的一个虚拟内存区域）。这个虚拟内存区域最初只是按虚拟内存空间保留，但在物理内存中没有实际的分配。此保留的虚拟内存空间受访问权限保护，该权限会导致任何访问该空间的尝试都会导致异常-页面错误，即在POSIX系统上触发SIGSEGV信号（分段错误）。幸运的是，分割错误可以由软件用信号处理器捕获。当这种分段错误发生时，我们的专用信号处理程序将检查它是否发生在由它负责的虚拟内存区域中，如果发生，它将继续用合理的值（感谢用户提供的回调）填充已访问的虚拟内存区域的部分（一个“页面”）。然后，在再次尝试触发分段错误的指令之前，它将对页面设置适当的权限（只读或读写）。从访问内存映射的用户代码的角度来看，这是完全透明的，这相当于从一开始就填充了整个虚拟内存区域。
对于大于RAM的非常大的映射，这仍然会导致在某个点上发生磁盘交换。为了避免这种情况，一旦达到在创建cplvirtualem对象时定义的阈值，分段错误处理程序将逐出最近使用最少的页面。
对于写支持，可以传递另一个回调。它将在页被逐出之前被调用，以便用户代码有机会将其内容刷新到更持久的存储中。
我们还提供了另一种创建cplvirtualem对象的方法，即使用内存文件映射机制。这可以由“原始”数据集（例如EHdr驱动程序）使用，其中磁盘上的数据组织与内存中数组的组织直接匹配。
高水平使用
介绍了四种新的API（在下一节中详细介绍）：
gdaldatasetgetvirtualem（）：采用与GDALDatasetRasterIO（）几乎相同的参数，但pData缓冲区除外。它返回一个cplvirtualem * 对象，从中可以使用CPLVirtualMemGetAddr（）获取虚拟内存映射的基址。
gdalStartBandGetVirtualMem（）：相当于在栅格带对象而不是数据集对象上操作的GDALDatasetGetVirtualMem（）。
gdaldatasetgettiledvirtualem（）：这是一个相当原始的API。映射不显示图像数据的二维视图（即按行组织的视图），而是将其公开为平铺数组，这在数据集本身平铺时更适合于性能方面。
当它们是多个波段时，可能有3种不同的波段组件组织。据我们所知，没有一种标准的方式来调用这些组织，因此下面的模式将最好地说明这一点：
按像素交错的尖端/平铺
由平铺交错的位/带
BSQ/波段序列组织
gdalrasterbandgetiledvirtualem（）：相当于在栅格带对象而不是数据集对象上操作的gdaldatasetgettiledvirtualem（）。
GDALGetVirtualMemAuto(): simplified version of GDALRasterBandGetVirtualMem() where the user only specifies the access mode. The pixel spacing and line spacing are returned by the function. This is implemented as a virtual method at the GDALRasterBand level, so that drivers have a chance of overriding the base implementation. The base implementation just uses GDALRasterBandGetVirtualMem(). Overridden implementation may use the memory file mapping mechanism instead. Such implementations will be done in the RawRasterBand object and in the GeoTIFF driver.
新API的详细信息
由cpl_virtualem.cpp实现
/**
* \file cpl_virtualmem.h
*
* Virtual memory management.
*
* This file provides mechanism to define virtual memory mappings, whose content
* is allocated transparently and filled on-the-fly. Those virtual memory mappings
* can be much larger than the available RAM, but only parts of the virtual
* memory mapping, in the limit of the allowed the cache size, will actually be
* physically allocated.
*
* This exploits low-level mechanisms of the operating system (virtual memory
* allocation, page protection and handler of virtual memory exceptions).
*
* It is also possible to create a virtual memory mapping from a file or part
* of a file.
*
* The current implementation is Linux only.
*/
/** Opaque type that represents a virtual memory mapping. */
typedef struct CPLVirtualMem CPLVirtualMem;
/** Callback triggered when a still unmapped page of virtual memory is accessed.
* The callback has the responsibility of filling the page with relevant values
*
* @param ctxt virtual memory handle.
* @param nOffset offset of the page in the memory mapping.
* @param pPageToFill address of the page to fill. Note that the address might
* be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
* @param nToFill number of bytes of the page.
* @param pUserData user data that was passed to CPLVirtualMemNew().
*/
typedef void (*CPLVirtualMemCachePageCbk)(CPLVirtualMem* ctxt,
size_t nOffset,
void* pPageToFill,
size_t nToFill,
void* pUserData);
/** Callback triggered when a dirty mapped page is going to be freed.
* (saturation of cache, or termination of the virtual memory mapping).
*
* @param ctxt virtual memory handle.
* @param nOffset offset of the page in the memory mapping.
* @param pPageToBeEvicted address of the page that will be flushed. Note that the address might
* be a temporary location, and not at CPLVirtualMemGetAddr() + nOffset.
* @param nToBeEvicted number of bytes of the page.
* @param pUserData user data that was passed to CPLVirtualMemNew().
*/
typedef void (*CPLVirtualMemUnCachePageCbk)(CPLVirtualMem* ctxt,
size_t nOffset,
const void* pPageToBeEvicted,
size_t nToBeEvicted,
void* pUserData);
/** Callback triggered when a virtual memory mapping is destroyed.
* @param pUserData user data that was passed to CPLVirtualMemNew().
*/
typedef void (*CPLVirtualMemFreeUserData)(void* pUserData);
/** Access mode of a virtual memory mapping. */
typedef enum
{
/*! The mapping is meant at being read-only, but writes will not be prevented.
Note that any content written will be lost. */
VIRTUALMEM_READONLY,
/*! The mapping is meant at being read-only, and this will be enforced
through the operating system page protection mechanism. */
VIRTUALMEM_READONLY_ENFORCED,
/*! The mapping is meant at being read-write, and modified pages can be saved
thanks to the pfnUnCachePage callback */
VIRTUALMEM_READWRITE
} CPLVirtualMemAccessMode;
/** Return the size of a page of virtual memory.
*
* @return the page size.
*
* @since GDAL 1.11
*/
size_t CPL_DLL CPLGetPageSize(void);
/** Create a new virtual memory mapping.
*
* This will reserve an area of virtual memory of size nSize, whose size
* might be potentially much larger than the physical memory available. Initially,
* no physical memory will be allocated. As soon as memory pages will be accessed,
* they will be allocated transparently and filled with the pfnCachePage callback.
* When the allowed cache size is reached, the least recently used pages will
* be unallocated.
*
* On Linux AMD64 platforms, the maximum value for nSize is 128 TB.
* On Linux x86 platforms, the maximum value for nSize is 2 GB.
*
* Only supported on Linux for now.
*
* Note that on Linux, this function will install a SIGSEGV handler. The
* original handler will be restored by CPLVirtualMemManagerTerminate().
*
* @param nSize size in bytes of the virtual memory mapping.
* @param nCacheSize size in bytes of the maximum memory that will be really
* allocated (must ideally fit into RAM).
* @param nPageSizeHint hint for the page size. Must be a multiple of the
* system page size, returned by CPLGetPageSize().
* Minimum value is generally 4096. Might be set to 0 to
* let the function determine a default page size.
* @param bSingleThreadUsage set to TRUE if there will be no concurrent threads
* that will access the virtual memory mapping. This can
* optimize performance a bit.
* @param eAccessMode permission to use for the virtual memory mapping.
* @param pfnCachePage callback triggered when a still unmapped page of virtual
* memory is accessed. The callback has the responsibility
* of filling the page with relevant values.
* @param pfnUnCachePage callback triggered when a dirty mapped page is going to
* be freed (saturation of cache, or termination of the
* virtual memory mapping). Might be NULL.
* @param pfnFreeUserData callback that can be used to free pCbkUserData. Might be
* NULL
* @param pCbkUserData user data passed to pfnCachePage and pfnUnCachePage.
*
* @return a virtual memory object that must be freed by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem CPL_DLL *CPLVirtualMemNew(size_t nSize,
size_t nCacheSize,
size_t nPageSizeHint,
int bSingleThreadUsage,
CPLVirtualMemAccessMode eAccessMode,
CPLVirtualMemCachePageCbk pfnCachePage,
CPLVirtualMemUnCachePageCbk pfnUnCachePage,
CPLVirtualMemFreeUserData pfnFreeUserData,
void *pCbkUserData);
/** Return if virtual memory mapping of a file is available.
*
* @return TRUE if virtual memory mapping of a file is available.
* @since GDAL 1.11
*/
int CPL_DLL CPLIsVirtualMemFileMapAvailable(void);
/** Create a new virtual memory mapping from a file.
*
* The file must be a "real" file recognized by the operating system, and not
* a VSI extended virtual file.
*
* In VIRTUALMEM_READWRITE mode, updates to the memory mapping will be written
* in the file.
*
* On Linux AMD64 platforms, the maximum value for nLength is 128 TB.
* On Linux x86 platforms, the maximum value for nLength is 2 GB.
*
* Only supported on Linux for now.
*
* @param fp Virtual file handle.
* @param nOffset Offset in the file to start the mapping from.
* @param nLength Length of the portion of the file to map into memory.
* @param eAccessMode Permission to use for the virtual memory mapping. This must
* be consistent with how the file has been opened.
* @param pfnFreeUserData callback that is called when the object is destroyed.
* @param pCbkUserData user data passed to pfnFreeUserData.
* @return a virtual memory object that must be freed by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem CPL_DLL *CPLVirtualMemFileMapNew( VSILFILE* fp,
vsi_l_offset nOffset,
vsi_l_offset nLength,
CPLVirtualMemAccessMode eAccessMode,
CPLVirtualMemFreeUserData pfnFreeUserData,
void *pCbkUserData );
/** Create a new virtual memory mapping derived from an other virtual memory
* mapping.
*
* This may be useful in case of creating mapping for pixel interleaved data.
*
* The new mapping takes a reference on the base mapping.
*
* @param pVMemBase Base virtual memory mapping
* @param nOffset Offset in the base virtual memory mapping from which to start
* the new mapping.
* @param nSize Size of the base virtual memory mapping to expose in the
* the new mapping.
* @param pfnFreeUserData callback that is called when the object is destroyed.
* @param pCbkUserData user data passed to pfnFreeUserData.
* @return a virtual memory object that must be freed by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem CPL_DLL *CPLVirtualMemDerivedNew(CPLVirtualMem* pVMemBase,
vsi_l_offset nOffset,
vsi_l_offset nSize,
CPLVirtualMemFreeUserData pfnFreeUserData,
void *pCbkUserData);
/** Free a virtual memory mapping.
*
* The pointer returned by CPLVirtualMemGetAddr() will no longer be valid.
* If the virtual memory mapping was created with read/write permissions and that
* they are dirty (i.e. modified) pages, they will be flushed through the
* pfnUnCachePage callback before being freed.
*
* @param ctxt context returned by CPLVirtualMemNew().
*
* @since GDAL 1.11
*/
void CPL_DLL CPLVirtualMemFree(CPLVirtualMem* ctxt);
/** Return the pointer to the start of a virtual memory mapping.
*
* The bytes in the range [p:p+CPLVirtualMemGetSize()-1] where p is the pointer
* returned by this function will be valid, until CPLVirtualMemFree() is called.
*
* Note that if a range of bytes used as an argument of a system call
* (such as read() or write()) contains pages that have not been "realized", the
* system call will fail with EFAULT. CPLVirtualMemPin() can be used to work
* around this issue.
*
* @param ctxt context returned by CPLVirtualMemNew().
* @return the pointer to the start of a virtual memory mapping.
*
* @since GDAL 1.11
*/
void CPL_DLL *CPLVirtualMemGetAddr(CPLVirtualMem* ctxt);
/** Return the size of the virtual memory mapping.
*
* @param ctxt context returned by CPLVirtualMemNew().
* @return the size of the virtual memory mapping.
*
* @since GDAL 1.11
*/
size_t CPL_DLL CPLVirtualMemGetSize(CPLVirtualMem* ctxt);
/** Return if the virtual memory mapping is a direct file mapping.
*
* @param ctxt context returned by CPLVirtualMemNew().
* @return TRUE if the virtual memory mapping is a direct file mapping.
*
* @since GDAL 1.11
*/
int CPL_DLL CPLVirtualMemIsFileMapping(CPLVirtualMem* ctxt);
/** Return the access mode of the virtual memory mapping.
*
* @param ctxt context returned by CPLVirtualMemNew().
* @return the access mode of the virtual memory mapping.
*
* @since GDAL 1.11
*/
CPLVirtualMemAccessMode CPL_DLL CPLVirtualMemGetAccessMode(CPLVirtualMem* ctxt);
/** Return the page size associated to a virtual memory mapping.
*
* The value returned will be at least CPLGetPageSize(), but potentially
* larger.
*
* @param ctxt context returned by CPLVirtualMemNew().
* @return the page size
*
* @since GDAL 1.11
*/
size_t CPL_DLL CPLVirtualMemGetPageSize(CPLVirtualMem* ctxt);
/** Return TRUE if this memory mapping can be accessed safely from concurrent
* threads.
*
* The situation that can cause problems is when several threads try to access
* a page of the mapping that is not yet mapped.
*
* The return value of this function depends on whether bSingleThreadUsage has
* been set of not in CPLVirtualMemNew() and/or the implementation.
*
* On Linux, this will always return TRUE if bSingleThreadUsage = FALSE.
*
* @param ctxt context returned by CPLVirtualMemNew().
* @return TRUE if this memory mapping can be accessed safely from concurrent
* threads.
*
* @since GDAL 1.11
*/
int CPL_DLL CPLVirtualMemIsAccessThreadSafe(CPLVirtualMem* ctxt);
/** Declare that a thread will access a virtual memory mapping.
*
* This function must be called by a thread that wants to access the
* content of a virtual memory mapping, except if the virtual memory mapping has
* been created with bSingleThreadUsage = TRUE.
*
* This function must be paired with CPLVirtualMemUnDeclareThread().
*
* @param ctxt context returned by CPLVirtualMemNew().
*
* @since GDAL 1.11
*/
void CPL_DLL CPLVirtualMemDeclareThread(CPLVirtualMem* ctxt);
/** Declare that a thread will stop accessing a virtual memory mapping.
*
* This function must be called by a thread that will no longer access the
* content of a virtual memory mapping, except if the virtual memory mapping has
* been created with bSingleThreadUsage = TRUE.
*
* This function must be paired with CPLVirtualMemDeclareThread().
*
* @param ctxt context returned by CPLVirtualMemNew().
*
* @since GDAL 1.11
*/
void CPL_DLL CPLVirtualMemUnDeclareThread(CPLVirtualMem* ctxt);
/** Make sure that a region of virtual memory will be realized.
*
* Calling this function is not required, but might be useful when debugging
* a process with tools like gdb or valgrind that do not naturally like
* segmentation fault signals.
*
* It is also needed when wanting to provide part of virtual memory mapping
* to a system call such as read() or write(). If read() or write() is called
* on a memory region not yet realized, the call will fail with EFAULT.
*
* @param ctxt context returned by CPLVirtualMemNew().
* @param pAddr the memory region to pin.
* @param nSize the size of the memory region.
* @param bWriteOp set to TRUE if the memory are will be accessed in write mode.
*
* @since GDAL 1.11
*/
void CPL_DLL CPLVirtualMemPin(CPLVirtualMem* ctxt,
void* pAddr, size_t nSize, int bWriteOp);
/** Cleanup any resource and handlers related to virtual memory.
*
* This function must be called after the last CPLVirtualMem object has
* been freed.
*
* @since GDAL 1.11
*/
void CPL_DLL CPLVirtualMemManagerTerminate(void);
由gdalvirtualem.cpp实现
/** Create a CPLVirtualMem object from a GDAL dataset object.
*
* Only supported on Linux for now.
*
* This method allows creating a virtual memory object for a region of one
* or more GDALRasterBands from this dataset. The content of the virtual
* memory object is automatically filled from dataset content when a virtual
* memory page is first accessed, and it is released (or flushed in case of a
* "dirty" page) when the cache size limit has been reached.
*
* The pointer to access the virtual memory object is obtained with
* CPLVirtualMemGetAddr(). It remains valid until CPLVirtualMemFree() is called.
* CPLVirtualMemFree() must be called before the dataset object is destroyed.
*
* If p is such a pointer and base_type the C type matching eBufType, for default
* values of spacing parameters, the element of image coordinates (x, y)
* (relative to xOff, yOff) for band b can be accessed with
* ((base_type*)p)[x + y * nBufXSize + (b-1)*nBufXSize*nBufYSize].
*
* Note that the mechanism used to transparently fill memory pages when they are
* accessed is the same (but in a controlled way) than what occurs when a memory
* error occurs in a program. Debugging software will generally interrupt program
* execution when that happens. If needed, CPLVirtualMemPin() can be used to avoid
* that by ensuring memory pages are allocated before being accessed.
*
* The size of the region that can be mapped as a virtual memory object depends
* on hardware and operating system limitations.
* On Linux AMD64 platforms, the maximum value is 128 TB.
* On Linux x86 platforms, the maximum value is 2 GB.
*
* Data type translation is automatically done if the data type
* (eBufType) of the buffer is different than
* that of the GDALRasterBand.
*
* Image decimation / replication is currently not supported, i.e. if the
* size of the region being accessed (nXSize x nYSize) is different from the
* buffer size (nBufXSize x nBufYSize).
*
* The nPixelSpace, nLineSpace and nBandSpace parameters allow reading into or
* writing from various organization of buffers. Arbitrary values for the spacing
* parameters are not supported. Those values must be multiple of the size of the
* buffer data type, and must be either band sequential organization (typically
* nPixelSpace = GDALGetDataTypeSize(eBufType) / 8, nLineSpace = nPixelSpace * nBufXSize,
* nBandSpace = nLineSpace * nBufYSize), or pixel-interleaved organization
* (typically nPixelSpace = nBandSpace * nBandCount, nLineSpace = nPixelSpace * nBufXSize,
* nBandSpace = GDALGetDataTypeSize(eBufType) / 8)
*
* @param hDS Dataset object
*
* @param eRWFlag Either GF_Read to read a region of data, or GF_Write to
* write a region of data.
*
* @param nXOff The pixel offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the left side.
*
* @param nYOff The line offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the top.
*
* @param nXSize The width of the region of the band to be accessed in pixels.
*
* @param nYSize The height of the region of the band to be accessed in lines.
*
* @param nBufXSize the width of the buffer image into which the desired region
* is to be read, or from which it is to be written.
*
* @param nBufYSize the height of the buffer image into which the desired
* region is to be read, or from which it is to be written.
*
* @param eBufType the type of the pixel values in the data buffer. The
* pixel values will automatically be translated to/from the GDALRasterBand
* data type as needed.
*
* @param nBandCount the number of bands being read or written.
*
* @param panBandMap the list of nBandCount band numbers being read/written.
* Note band numbers are 1 based. This may be NULL to select the first
* nBandCount bands.
*
* @param nPixelSpace The byte offset from the start of one pixel value in
* the buffer to the start of the next pixel value within a scanline. If defaulted
* (0) the size of the datatype eBufType is used.
*
* @param nLineSpace The byte offset from the start of one scanline in
* the buffer to the start of the next. If defaulted (0) the size of the datatype
* eBufType * nBufXSize is used.
*
* @param nBandSpace the byte offset from the start of one bands data to the
* start of the next. If defaulted (0) the value will be
* nLineSpace * nBufYSize implying band sequential organization
* of the data buffer.
*
* @param nCacheSize size in bytes of the maximum memory that will be really
* allocated (must ideally fit into RAM)
*
* @param nPageSizeHint hint for the page size. Must be a multiple of the
* system page size, returned by CPLGetPageSize().
* Minimum value is generally 4096. Might be set to 0 to
* let the function determine a default page size.
*
* @param bSingleThreadUsage set to TRUE if there will be no concurrent threads
* that will access the virtual memory mapping. This can
* optimize performance a bit. If set to FALSE,
* CPLVirtualMemDeclareThread() must be called.
*
* @param papszOptions NULL terminated list of options. Unused for now.
*
* @return a virtual memory object that must be freed by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem CPL_DLL* GDALDatasetGetVirtualMem( GDALDatasetH hDS,
GDALRWFlag eRWFlag,
int nXOff, int nYOff,
int nXSize, int nYSize,
int nBufXSize, int nBufYSize,
GDALDataType eBufType,
int nBandCount, int* panBandMap,
int nPixelSpace,
GIntBig nLineSpace,
GIntBig nBandSpace,
size_t nCacheSize,
size_t nPageSizeHint,
int bSingleThreadUsage,
char **papszOptions );
** Create a CPLVirtualMem object from a GDAL raster band object.
*
* Only supported on Linux for now.
*
* This method allows creating a virtual memory object for a region of a
* GDALRasterBand. The content of the virtual
* memory object is automatically filled from dataset content when a virtual
* memory page is first accessed, and it is released (or flushed in case of a
* "dirty" page) when the cache size limit has been reached.
*
* The pointer to access the virtual memory object is obtained with
* CPLVirtualMemGetAddr(). It remains valid until CPLVirtualMemFree() is called.
* CPLVirtualMemFree() must be called before the raster band object is destroyed.
*
* If p is such a pointer and base_type the C type matching eBufType, for default
* values of spacing parameters, the element of image coordinates (x, y)
* (relative to xOff, yOff) can be accessed with
* ((base_type*)p)[x + y * nBufXSize].
*
* Note that the mechanism used to transparently fill memory pages when they are
* accessed is the same (but in a controlled way) than what occurs when a memory
* error occurs in a program. Debugging software will generally interrupt program
* execution when that happens. If needed, CPLVirtualMemPin() can be used to avoid
* that by ensuring memory pages are allocated before being accessed.
*
* The size of the region that can be mapped as a virtual memory object depends
* on hardware and operating system limitations.
* On Linux AMD64 platforms, the maximum value is 128 TB.
* On Linux x86 platforms, the maximum value is 2 GB.
*
* Data type translation is automatically done if the data type
* (eBufType) of the buffer is different than
* that of the GDALRasterBand.
*
* Image decimation / replication is currently not supported, i.e. if the
* size of the region being accessed (nXSize x nYSize) is different from the
* buffer size (nBufXSize x nBufYSize).
*
* The nPixelSpace and nLineSpace parameters allow reading into or
* writing from various organization of buffers. Arbitrary values for the spacing
* parameters are not supported. Those values must be multiple of the size of the
* buffer data type and must be such that nLineSpace >= nPixelSpace * nBufXSize.
*
* @param hBand Rasterband object
*
* @param eRWFlag Either GF_Read to read a region of data, or GF_Write to
* write a region of data.
*
* @param nXOff The pixel offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the left side.
*
* @param nYOff The line offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the top.
*
* @param nXSize The width of the region of the band to be accessed in pixels.
*
* @param nYSize The height of the region of the band to be accessed in lines.
*
* @param nBufXSize the width of the buffer image into which the desired region
* is to be read, or from which it is to be written.
*
* @param nBufYSize the height of the buffer image into which the desired
* region is to be read, or from which it is to be written.
*
* @param eBufType the type of the pixel values in the data buffer. The
* pixel values will automatically be translated to/from the GDALRasterBand
* data type as needed.
*
* @param nPixelSpace The byte offset from the start of one pixel value in
* the buffer to the start of the next pixel value within a scanline. If defaulted
* (0) the size of the datatype eBufType is used.
*
* @param nLineSpace The byte offset from the start of one scanline in
* the buffer to the start of the next. If defaulted (0) the size of the datatype
* eBufType * nBufXSize is used.
*
* @param nCacheSize size in bytes of the maximum memory that will be really
* allocated (must ideally fit into RAM)
*
* @param nPageSizeHint hint for the page size. Must be a multiple of the
* system page size, returned by CPLGetPageSize().
* Minimum value is generally 4096. Might be set to 0 to
* let the function determine a default page size.
*
* @param bSingleThreadUsage set to TRUE if there will be no concurrent threads
* that will access the virtual memory mapping. This can
* optimize performance a bit. If set to FALSE,
* CPLVirtualMemDeclareThread() must be called.
*
* @param papszOptions NULL terminated list of options. Unused for now.
*
* @return a virtual memory object that must be freed by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem CPL_DLL* GDALRasterBandGetVirtualMem( GDALRasterBandH hBand,
GDALRWFlag eRWFlag,
int nXOff, int nYOff,
int nXSize, int nYSize,
int nBufXSize, int nBufYSize,
GDALDataType eBufType,
int nPixelSpace,
GIntBig nLineSpace,
size_t nCacheSize,
size_t nPageSizeHint,
int bSingleThreadUsage,
char **papszOptions );
typedef enum
{
/*! Tile Interleaved by Pixel: tile (0,0) with internal band interleaved
by pixel organization, tile (1, 0), ... */
GTO_TIP,
/*! Band Interleaved by Tile : tile (0,0) of first band, tile (0,0) of second
band, ... tile (1,0) of first band, tile (1,0) of second band, ... */
GTO_BIT,
/*! Band SeQuential : all the tiles of first band, all the tiles of following band... */
GTO_BSQ
} GDALTileOrganization;
/** Create a CPLVirtualMem object from a GDAL dataset object, with tiling
* organization
*
* Only supported on Linux for now.
*
* This method allows creating a virtual memory object for a region of one
* or more GDALRasterBands from this dataset. The content of the virtual
* memory object is automatically filled from dataset content when a virtual
* memory page is first accessed, and it is released (or flushed in case of a
* "dirty" page) when the cache size limit has been reached.
*
* Contrary to GDALDatasetGetVirtualMem(), pixels will be organized by tiles
* instead of scanlines. Different ways of organizing pixel within/across tiles
* can be selected with the eTileOrganization parameter.
*
* If nXSize is not a multiple of nTileXSize or nYSize is not a multiple of
* nTileYSize, partial tiles will exists at the right and/or bottom of the region
* of interest. Those partial tiles will also have nTileXSize * nTileYSize dimension,
* with padding pixels.
*
* The pointer to access the virtual memory object is obtained with
* CPLVirtualMemGetAddr(). It remains valid until CPLVirtualMemFree() is called.
* CPLVirtualMemFree() must be called before the dataset object is destroyed.
*
* If p is such a pointer and base_type the C type matching eBufType, for default
* values of spacing parameters, the element of image coordinates (x, y)
* (relative to xOff, yOff) for band b can be accessed with :
* - for eTileOrganization = GTO_TIP, ((base_type*)p)[tile_number(x,y)*nBandCount*tile_size + offset_in_tile(x,y)*nBandCount + (b-1)].
* - for eTileOrganization = GTO_BIT, ((base_type*)p)[(tile_number(x,y)*nBandCount + (b-1)) * tile_size + offset_in_tile(x,y)].
* - for eTileOrganization = GTO_BSQ, ((base_type*)p)[(tile_number(x,y) + (b-1)*nTilesCount) * tile_size + offset_in_tile(x,y)].
*
* where nTilesPerRow = ceil(nXSize / nTileXSize)
* nTilesPerCol = ceil(nYSize / nTileYSize)
* nTilesCount = nTilesPerRow * nTilesPerCol
* tile_number(x,y) = (y / nTileYSize) * nTilesPerRow + (x / nTileXSize)
* offset_in_tile(x,y) = (y % nTileYSize) * nTileXSize + (x % nTileXSize)
* tile_size = nTileXSize * nTileYSize
*
* Note that for a single band request, all tile organizations are equivalent.
*
* Note that the mechanism used to transparently fill memory pages when they are
* accessed is the same (but in a controlled way) than what occurs when a memory
* error occurs in a program. Debugging software will generally interrupt program
* execution when that happens. If needed, CPLVirtualMemPin() can be used to avoid
* that by ensuring memory pages are allocated before being accessed.
*
* The size of the region that can be mapped as a virtual memory object depends
* on hardware and operating system limitations.
* On Linux AMD64 platforms, the maximum value is 128 TB.
* On Linux x86 platforms, the maximum value is 2 GB.
*
* Data type translation is automatically done if the data type
* (eBufType) of the buffer is different than
* that of the GDALRasterBand.
*
* @param hDS Dataset object
*
* @param eRWFlag Either GF_Read to read a region of data, or GF_Write to
* write a region of data.
*
* @param nXOff The pixel offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the left side.
*
* @param nYOff The line offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the top.
*
* @param nXSize The width of the region of the band to be accessed in pixels.
*
* @param nYSize The height of the region of the band to be accessed in lines.
*
* @param nTileXSize the width of the tiles.
*
* @param nTileYSize the height of the tiles.
*
* @param eBufType the type of the pixel values in the data buffer. The
* pixel values will automatically be translated to/from the GDALRasterBand
* data type as needed.
*
* @param nBandCount the number of bands being read or written.
*
* @param panBandMap the list of nBandCount band numbers being read/written.
* Note band numbers are 1 based. This may be NULL to select the first
* nBandCount bands.
*
* @param eTileOrganization tile organization.
*
* @param nCacheSize size in bytes of the maximum memory that will be really
* allocated (must ideally fit into RAM)
*
* @param bSingleThreadUsage set to TRUE if there will be no concurrent threads
* that will access the virtual memory mapping. This can
* optimize performance a bit. If set to FALSE,
* CPLVirtualMemDeclareThread() must be called.
*
* @param papszOptions NULL terminated list of options. Unused for now.
*
* @return a virtual memory object that must be freed by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem CPL_DLL* GDALDatasetGetTiledVirtualMem( GDALDatasetH hDS,
GDALRWFlag eRWFlag,
int nXOff, int nYOff,
int nXSize, int nYSize,
int nTileXSize, int nTileYSize,
GDALDataType eBufType,
int nBandCount, int* panBandMap,
GDALTileOrganization eTileOrganization,
size_t nCacheSize,
int bSingleThreadUsage,
char **papszOptions );
/** Create a CPLVirtualMem object from a GDAL rasterband object, with tiling
* organization
*
* Only supported on Linux for now.
*
* This method allows creating a virtual memory object for a region of one
* GDALRasterBand. The content of the virtual
* memory object is automatically filled from dataset content when a virtual
* memory page is first accessed, and it is released (or flushed in case of a
* "dirty" page) when the cache size limit has been reached.
*
* Contrary to GDALDatasetGetVirtualMem(), pixels will be organized by tiles
* instead of scanlines.
*
* If nXSize is not a multiple of nTileXSize or nYSize is not a multiple of
* nTileYSize, partial tiles will exists at the right and/or bottom of the region
* of interest. Those partial tiles will also have nTileXSize * nTileYSize dimension,
* with padding pixels.
*
* The pointer to access the virtual memory object is obtained with
* CPLVirtualMemGetAddr(). It remains valid until CPLVirtualMemFree() is called.
* CPLVirtualMemFree() must be called before the raster band object is destroyed.
*
* If p is such a pointer and base_type the C type matching eBufType, for default
* values of spacing parameters, the element of image coordinates (x, y)
* (relative to xOff, yOff) can be accessed with :
* ((base_type*)p)[tile_number(x,y)*tile_size + offset_in_tile(x,y)].
*
* where nTilesPerRow = ceil(nXSize / nTileXSize)
* nTilesCount = nTilesPerRow * nTilesPerCol
* tile_number(x,y) = (y / nTileYSize) * nTilesPerRow + (x / nTileXSize)
* offset_in_tile(x,y) = (y % nTileYSize) * nTileXSize + (x % nTileXSize)
* tile_size = nTileXSize * nTileYSize
*
* Note that the mechanism used to transparently fill memory pages when they are
* accessed is the same (but in a controlled way) than what occurs when a memory
* error occurs in a program. Debugging software will generally interrupt program
* execution when that happens. If needed, CPLVirtualMemPin() can be used to avoid
* that by ensuring memory pages are allocated before being accessed.
*
* The size of the region that can be mapped as a virtual memory object depends
* on hardware and operating system limitations.
* On Linux AMD64 platforms, the maximum value is 128 TB.
* On Linux x86 platforms, the maximum value is 2 GB.
*
* Data type translation is automatically done if the data type
* (eBufType) of the buffer is different than
* that of the GDALRasterBand.
*
* @param hBand Rasterband object
*
* @param eRWFlag Either GF_Read to read a region of data, or GF_Write to
* write a region of data.
*
* @param nXOff The pixel offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the left side.
*
* @param nYOff The line offset to the top left corner of the region
* of the band to be accessed. This would be zero to start from the top.
*
* @param nXSize The width of the region of the band to be accessed in pixels.
*
* @param nYSize The height of the region of the band to be accessed in lines.
*
* @param nTileXSize the width of the tiles.
*
* @param nTileYSize the height of the tiles.
*
* @param eBufType the type of the pixel values in the data buffer. The
* pixel values will automatically be translated to/from the GDALRasterBand
* data type as needed.
*
* @param nCacheSize size in bytes of the maximum memory that will be really
* allocated (must ideally fit into RAM)
*
* @param bSingleThreadUsage set to TRUE if there will be no concurrent threads
* that will access the virtual memory mapping. This can
* optimize performance a bit. If set to FALSE,
* CPLVirtualMemDeclareThread() must be called.
*
* @param papszOptions NULL terminated list of options. Unused for now.
*
* @return a virtual memory object that must be freed by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem CPL_DLL* GDALRasterBandGetTiledVirtualMem( GDALRasterBandH hBand,
GDALRWFlag eRWFlag,
int nXOff, int nYOff,
int nXSize, int nYSize,
int nTileXSize, int nTileYSize,
GDALDataType eBufType,
size_t nCacheSize,
int bSingleThreadUsage,
char **papszOptions );
由gdalrasterband.cpp实现
/** \brief Create a CPLVirtualMem object from a GDAL raster band object.
*
* Only supported on Linux for now.
*
* This method allows creating a virtual memory object for a GDALRasterBand,
* that exposes the whole image data as a virtual array.
*
* The default implementation relies on GDALRasterBandGetVirtualMem(), but specialized
* implementation, such as for raw files, may also directly use mechanisms of the
* operating system to create a view of the underlying file into virtual memory
* ( CPLVirtualMemFileMapNew() )
*
* At the time of writing, the GeoTIFF driver and "raw" drivers (EHdr, ...) offer
* a specialized implementation with direct file mapping, provided that some
* requirements are met :
* - for all drivers, the dataset must be backed by a "real" file in the file
* system, and the byte ordering of multi-byte datatypes (Int16, etc.)
* must match the native ordering of the CPU.
* - in addition, for the GeoTIFF driver, the GeoTIFF file must be uncompressed, scanline
* oriented (i.e. not tiled). Strips must be organized in the file in sequential
* order, and be equally spaced (which is generally the case). Only power-of-two
* bit depths are supported (8 for GDT_Bye, 16 for GDT_Int16/GDT_UInt16,
* 32 for GDT_Float32 and 64 for GDT_Float64)
*
* The pointer returned remains valid until CPLVirtualMemFree() is called.
* CPLVirtualMemFree() must be called before the raster band object is destroyed.
*
* If p is such a pointer and base_type the type matching GDALGetRasterDataType(),
* the element of image coordinates (x, y) can be accessed with
* *(base_type*) ((GByte*)p + x * *pnPixelSpace + y * *pnLineSpace)
*
* This method is the same as the C GDALGetVirtualMemAuto() function.
*
* @param eRWFlag Either GF_Read to read the band, or GF_Write to
* read/write the band.
*
* @param pnPixelSpace Output parameter giving the byte offset from the start of one pixel value in
* the buffer to the start of the next pixel value within a scanline.
*
* @param pnLineSpace Output parameter giving the byte offset from the start of one scanline in
* the buffer to the start of the next.
*
* @param papszOptions NULL terminated list of options.
* If a specialized implementation exists, defining USE_DEFAULT_IMPLEMENTATION=YES
* will cause the default implementation to be used.
* When requiring or falling back to the default implementation, the following
* options are available : CACHE_SIZE (in bytes, defaults to 40 MB),
* PAGE_SIZE_HINT (in bytes),
* SINGLE_THREAD ("FALSE" / "TRUE", defaults to FALSE)
*
* @return a virtual memory object that must be unreferenced by CPLVirtualMemFree(),
* or NULL in case of failure.
*
* @since GDAL 1.11
*/
CPLVirtualMem *GDALRasterBand::GetVirtualMemAuto( GDALRWFlag eRWFlag,
int *pnPixelSpace,
GIntBig *pnLineSpace,
char **papszOptions ):
CPLVirtualMem CPL_DLL* GDALGetVirtualMemAuto( GDALRasterBandH hBand,
GDALRWFlag eRWFlag,
int *pnPixelSpace,
GIntBig *pnLineSpace,
char **papszOptions );
便携性
cplvirtualem低级机器现在只在Linux上实现。它假设从SIGSEGV处理程序返回是可能的，这明显违反了POSIX，但实际上，大多数POSIX（和非POSIX，如Windows）系统似乎应该能够在分段错误之后恢复执行。
移植到其他POSIX操作系统（如MacOSX）应该是可以通过适度的努力实现的。Windows的API提供了与POSIX API类似的功能，包括VirtualAlloc（）、VirtualProtect（）和SetUnhandledExceptionFilter（），尽管移植无疑需要更多的工作。
存在 libsigsegv 运行在各种操作系统上的这证明了它可以移植到其他平台上。
最棘手的部分是确保当两个并发线程试图访问相同的初始未映射页时，事情能够可靠地工作。如果不特别小心，一个线程可以在另一个线程完全填充页面之前访问它所填充的页面。在Linux上，使用mremap（）调用可以轻松避免这种情况。当一个页面被填充时，我们实际上不会将目标页面传递给用户回调，而是一个临时页面。当回调完成其任务时，此临时页将被mremap（）'ed到其目标位置，这是一个原子操作。已经测试了不具有此mremap（）调用的POSIX系统的另一个实现：任何可以访问内存映射的已声明线程都会在临时页被memcpy'ed到其目标位置之前暂停，然后再继续。这要求线程首先声明它们对cplvirtualmedmeclarethread（）内存映射的“兴趣”。暂停线程是一个有趣的不明显的问题：解决方法是发送一个SIGUSR1信号并让它在信号处理程序中等待这个SIGUSR1信号。。。尚未调查是否/如何在Windows上执行此操作。为此，引入了cplvirtualmisaccessthreadsafe（）。
就cplvirtualemfilemapnew（）而言，POSIX系统上使用mmap（）的内存文件映射应该是可移植的。Windows有CreateFileMapping（）和MapViewOfFile（）API，它们的功能与mmap（）类似。
性能
与谨慎使用GDALRasterIO（）的代码相比，这种新功能不会带来奇迹般的性能提升。处理分段错误是有代价的（操作系统捕获一个硬件异常，然后调用用户程序分段错误处理程序，该程序执行正常的GDAL I/O操作，并播放使某些CPU缓存失效的页映射和权限等）。然而，当一个页面被实现时，访问它的速度应该非常快，因此使用适当的访问模式和缓存大小，应该会有良好的性能。
还应注意的是，在当前的实现中，页面的实现是以序列化的方式完成的，也就是说，如果使用两个不同内存映射的两个线程同时导致分段错误，它们将不会被两个不同的线程处理，而是一个接一个地处理。
在使用内存文件映射时，getVirtualMemoto（）返回的虚拟内存对象的开销应小于手动管理页面错误的开销。但是，GDAL无法控制操作系统用来缓存页面的策略。
局限性
虚拟内存空间的最大大小（以及虚拟内存映射）取决于CPU体系结构和操作系统限制：
在Linux AMD64上，128 TB。
在Linux x86上，2 GB。
在Windows AMD64上（当前实现不支持），8 TB。
在Windows x86上（当前实现不支持），2 GB。
显然，这个新功能的主要兴趣是针对AMD64平台。
在具有4 GB RAM的Linux AMD64计算机上，已成功使用gdaldatasetgettiledvirtualem（）的Python绑定访问 Europe 3'' DEM dataset ，这是一个20 GB的压缩GeoTIFF（和288000 * 180000 * 4=193 GB未压缩）
开放性问题
由于它目前只在Linux上工作，我们现在是否应该将API标记为实验性的呢？
向后兼容性问题
C/C++ API ->兼容（新API）。C ABI-->兼容（新API）。C++ AABI——>不兼容，因为GDRARASTEP带有一种新的虚拟方法。
更新的驱动程序
RawRasterBand对象和GeoTIFF驱动程序将被更新，以实现getvirtualmauto（）并在可能的情况下提供内存文件映射（请参阅上面关于何时可能的文档限制）。
在以后的步骤中，其他驱动程序（例如VRT驱动程序（用于VRTRawRasterBand））也可以提供getvirtualmauto（）的专门实现。
SWIG绑定
Python绑定中提供了高级API（数据集和栅格带）API。
gdaldatasetgetvirtualem（）被映射为Dataset.GetVirtualArray（），它返回一个NumPy数组。
def GetVirtualMemArray(self, eAccess = gdalconst.GF_Read, xoff=0, yoff=0,
xsize=None, ysize=None, bufxsize=None, bufysize=None,
datatype = None, band_list = None, band_sequential = True,
cache_size = 10 * 1024 * 1024, page_size_hint = 0, options = None):
"""Return a NumPy array for the dataset, seen as a virtual memory mapping.
If there are several bands and band_sequential = True, an element is
accessed with array[band][y][x].
If there are several bands and band_sequential = False, an element is
accessed with array[y][x][band].
If there is only one band, an element is accessed with array[y][x].
Any reference to the array must be dropped before the last reference to the
related dataset is also dropped.
"""
与gdaldatasetgettiledvirtualem（）类似：
def GetTiledVirtualMemArray(self, eAccess = gdalconst.GF_Read, xoff=0, yoff=0,
xsize=None, ysize=None, tilexsize=256, tileysize=256,
datatype = None, band_list = None, tile_organization = gdalconst.GTO_BSQ,
cache_size = 10 * 1024 * 1024, options = None):
"""Return a NumPy array for the dataset, seen as a virtual memory mapping with
a tile organization.
If there are several bands and tile_organization = gdal.GTO_BIP, an element is
accessed with array[tiley][tilex][y][x][band].
If there are several bands and tile_organization = gdal.GTO_BTI, an element is
accessed with array[tiley][tilex][band][y][x].
If there are several bands and tile_organization = gdal.GTO_BSQ, an element is
accessed with array[band][tiley][tilex][y][x].
If there is only one band, an element is accessed with array[tiley][tilex][y][x].
Any reference to the array must be dropped before the last reference to the
related dataset is also dropped.
"""
Band对象有以下3种方法：
def GetVirtualMemArray(self, eAccess = gdalconst.GF_Read, xoff=0, yoff=0,
xsize=None, ysize=None, bufxsize=None, bufysize=None,
datatype = None,
cache_size = 10 * 1024 * 1024, page_size_hint = 0, options = None):
"""Return a NumPy array for the band, seen as a virtual memory mapping.
An element is accessed with array[y][x].
Any reference to the array must be dropped before the last reference to the
related dataset is also dropped.
"""
def GetVirtualMemAutoArray(self, eAccess = gdalconst.GF_Read, options = None):
"""Return a NumPy array for the band, seen as a virtual memory mapping.
An element is accessed with array[y][x].
def GetTiledVirtualMemArray(self, eAccess = gdalconst.GF_Read, xoff=0, yoff=0,
xsize=None, ysize=None, tilexsize=256, tileysize=256,
datatype = None,
cache_size = 10 * 1024 * 1024, options = None):
"""Return a NumPy array for the band, seen as a virtual memory mapping with
a tile organization.
An element is accessed with array[tiley][tilex][y][x].
Any reference to the array must be dropped before the last reference to the
related dataset is also dropped.
"""
注意：dataset/Band.getvirtualem（）/gettiledvirtualem（）方法也可用。它们返回一个virtualem python对象，该对象有一个GetAddr（）方法，该方法返回一个python memoryview对象（需要python 2.7或更高版本）。但是，对于非字节数据类型，使用此类对象似乎并不实际。
测试套件
autotest套件将被扩展以测试这个RFC的Python API。它还将测试RawRasterBand中getvirtualmauto（）的专用实现和GeoTIFF驱动程序。在autotest/cpp中，test_virtualem.cpp文件测试两个线程对同一页的并发访问。
实施
甚至ruault也将在GDAL/OGR主干中实现。建议的实施作为 patch .
投票历史
+1名来自Evner、FrankW、DanielM和JukkaR