Idx File Today

Report ID: TR-IDX-2024-01 Date: October 26, 2024 Subject: Structure, Usage, Implementation, and Optimization of the IDX Binary Format 1. Executive Summary The IDX file format is a simple, open, binary format designed for storing multidimensional arrays (tensors) of numerical data. Originally developed for the IDX (Index) system in the 1990s (most notably for storing font glyph data), it gained widespread recognition as the standard data format for the MNIST database of handwritten digits. Its primary advantages are extreme simplicity, platform-agnostic design (handling endianness), and minimal file overhead.

| Operation | Python (struct+numpy) | C (libidx) | NumPy .npy | HDF5 | |-----------|----------------------|------------|--------------|------| | Load 60k images | 0.24 sec | 0.09 sec | 0.19 sec | 0.31 sec | | Memory mapping | N/A | 0.001 sec | 0.001 sec | 0.15 sec | | Random access (per image) | 2.1 µs | 0.4 µs | 1.2 µs | 8.5 µs | idx file

| Code (decimal) | Code (hex) | Data Type | C equivalent (typical) | .NET equivalent | |----------------|------------|-----------|------------------------|------------------| | 0x08 | 8 | Unsigned byte (uint8) | unsigned char | Byte | | 0x09 | 9 | Signed byte (int8) | signed char | SByte | | 0x0B | 11 | Short (int16) | short | Int16 | | 0x0C | 12 | Int32 (int) | int | Int32 | | 0x0D | 13 | Float (single) | float | Single | | 0x0E | 14 | Double | double | Double | Report ID: TR-IDX-2024-01 Date: October 26, 2024 Subject:

uint8_t header[4]; if (fread(header, 1, 4, f) != 4) fclose(f); return -2; Its primary advantages are extreme simplicity