A Storage Representation For Efficient Access To Large, Multidimensional Arrays
by Quam, Lynn H.
Technical Note 220
Institution: AI Center, SRI International
Address: 333 Ravenswood Ave., Menlo Park, CA 94025
This paper addresses problems associated with accessing elements of large multidimensional arrays when the order of access is either unpredictable or is orthogonal to the conventional order of array storage. Large arrays are defined as arrays that are larger than the physical memory immediately available to store them. Such arrays must be accessed either by the virtual memory system of the computer and operating system, or by direct input and output of blocks of the array to a file system. In either case, the direct result of an inappropriate order of reference to the elements of the array is the very time-consuming movement of data between levels in the memory hierarchy, often costing factors of three orders of magnitude in algorithm performance. The access to elements of large arrays is decomposed into three steps: transforming the subscript values of an n-dimensional array into the element number in a one-dimensional virtual array, mapping the virtual array position to physical memory position, and accessing the array element in physical memory. The virtual-to-physical mapping step is unnecessary on computer systems with sufficiently large virtual address spaces. This paper is primarily concerned with the first step. A subscript transformation is proposed that solves many of the order-of-access problems associated with conventional array storage. This transformation is based on an additive decomposition of the calculation of element number in the array into the sum of a set of integer functions applied to the set of subscripts as follows: