Spectral reflectance database from hyperspectral images

Description

To evaluate the image quality deterioration caused by metamerism for digital cameras' color correction, I collected 43,714,197 spectral reflectance data from several public databases and preprocessed them so that finally 7,326,497 reliable reflectance data were obtained. Since these spectral reflectances practically exist in the real scenes, they provide useful information for the researches of color science and digital image processing, including but not limited to the estimation of metamers, the training sample selection for color adaptation/color correction, the statistical analysis for auto white balance, the calculation of color gamut, etc.

Sources

The total reflectance data were collected from two types of online databases: individual spectral reflectance databases and hyperspectral image databases. Some sources of data are listed as follows (some were downloaded long ago so I can't remember where I found them).

Spectral reflectance data:

Hyperspectral image (HSI) data:

Data processing

First, I preprocessed (downsampling or interpolating) all reflectance data or HSIs so that each spectral reflectances has 31 dimensions: from 300nm to 700nm at 10nm interval. Next, spectral reflectances with too low energy ($\sum{}\mathbf{r} < 0.2$), which might suffer from high noise in the HSI, were discarded. Since the spectral reflectance data extracted from one hyperspectral image usually have high similarity, I then clustered those with "correlation coefficient" higher than 0.99 and remove most of them, only kept those on the vertices of the cluster in the 6-dimension hyperspace, which is the dimensionality reduction result of original 31-dimension hyperspace performed by PCA, on account that the Qhull algorithm is time-consuming when dimensionality is high. More detail about the spectral reflectance data processing will be described in the paper (coming soon).

The correlation coefficient used here is calculated as: $$\mathrm{corr}(\mathbf{r}_1, \mathbf{r}_2) = \frac{(\mathbf{r}_1 - \bar{\mathbf{r}_1})^\mathsf{T}(\mathbf{r}_2 - \bar{\mathbf{r}_2})}{\sqrt{(\mathbf{r}_1 - \bar{\mathbf{r}_1})^\mathsf{T}(\mathbf{r}_1 - \bar{\mathbf{r}_1})}\cdot\sqrt{(\mathbf{r}_2 - \bar{\mathbf{r}_2})^\mathsf{T}(\mathbf{r}_2 - \bar{\mathbf{r}_2})}}\,,$$ where $\bar{\mathbf{r}_i} = \frac{1}{n}\sum^n_j{}\mathrm{r}_{ij}$ and $n = 31$ in this case.

The distribution of chromaticities of all reflectance spectra (under CIE D65) before and after data processing are illustrated in Fig.1. It can be noticed that large amount of spectral reflectances on the periphery of xy diagram were removed, bacause these data usually correspond to the dark "pixel" in the HSI and their colors are biased by the high spectral noise.


Figure 1. Distribution of reflectance spectra before (left) and after (right) processing

For the sake of storage size, the data were converted from double to single precision in MATLAB.

Download

You may use this database for research purposes on condition that you acknowledge the source.

The distribution of chromaticities of this compact version (under CIE D65) is shown in Fig.2.


Figure 2. Distribution of individual reflectance spectra


Feel free to contact jqx1991(at)gmail(dot)com with any suggestions/corrections/comments.