Fusion of multispectral and pancromatic images with sparse representations

Image fusion is a technique used to combine two or more images into a single image that represents better visual perception and provides better understanding of target scene. Image fusion is a genereal name for combination process of images. In remote sensing field it is called "pansharpening" when high-resolution pancromatic image and low-resolution multispectral image are combined to obtaion single image. While pancromatic image have higher resolution spatial details without color information, multispectral image has color informtion with lower spatial resolution. Thus pansharpening process aims to combine these images to obtain one single image that is high resolution multispectral image. The goal is to preserve spectral characteristics while improving spatial information. Satellites in the remote sensing field provide high-resolution pancromatic image and low-resolution multispectral images. Technological restrictions in sensors, data transmission rates, limited storage capacity make getting high-resolution multispectral images difficult. Such reasons causes to store images seperately. However, it can be overcome by using image fusion methods. Studies for years, many image fusion methods have been developed and introduced their advantages and disadvantages. Among these, it is known that à trous wavelet transform based fusion methods give best results both in spectral and in spatial resolution. The most basic image fusion technique which is known as color transformation based is Intensity Hue Saturation (IHS) method. In this method MS image in Red Green Blue (RGB) color space is transformed to the IHS color space. Intensity component of transformed image is replaced histogtam matched image which is obtained between pancromatic and intensity component. And fusion process is completed with inverse IHS transform. As the IHS transform is for three bands multispectral images, Generalized IHS (GIHS) method has been developed for more than three bands multispectral images. Color based transform methods are easy to use and adapt with less complexity. On the other hand wavelet based transforms are most widely used methods because they give better results. Traditional wavelet trasforms, (DWT) is a traditional wavelet transform used in pansharpening is orthogonal transform and also shift-invariant that results from decimation step of wavelet transform. Even though decimation step is useful in other applications, in image fusion it may cause artifacts in fused image. À trous wavelet transform overcome this problem. À trous wavelet transform uses high pass filter to obtain detail image (wavelet plane) as DWT. However, it solves shift-invariance problem since decimation is not applied. There are several well-know methods uses à trous wavelet as base transform. These are Additive Wavelet L Component (AWL), ATWT, Proportional AWL (AWLP), ATWT-Context Based Decision (ATWT-CBD). In AWL method, histogram matching is performed a between luminance (L) band of MS image and PAN image. Histogram matched image is decomposed by à trous wavelet transform to obtain wavelet planes and these planes are added to the L band. New values are transformed into RGB space. Disadvantage of AWL method is that it is only for three bands MS images. Therefore, AWLP method is developed for more than three bands. Wavelet planes are added to MS proportionally in AWLP while directly in ATWT. Thus, AWLP improves spectral information better than ATWT. Unlike these methods, in ATWT-CBD the details are weighted before adding to MS image. Weight coefficients are calculated on sliding windows separately. Besides traditional image fusion methods, image fusion has gained different aspects with the use of sparse representation. Sparsity approach has been implemented in many topics, such as denoising, feature exctraction, image decomposition, inpainting; likewise it has been performed plenty of studies based on sparsity in image fusion. A signal or an image can be represented as a linear combination of elementary signals (so-called atoms) and coefficents, most of them zero. Dictionary can be both fixed and trained. Fixed dictionaries can be made up of set of functions or set of image patches or signal examples generated randomly from raw data itself. Trained dictionaries can be generated by using learning algorthim such as K-SVD (K-Singular Value Decomposition). K-SVD is an algoritm developed for obtaining better dictionaries and works iteratively. The algorithm basically consists of two steps. The first step is called sparse coding in which the sparse coefficients are found by using a pursuit algorithm. The second step is dictionary update in which one dictionary atom is updated at a time. The dictionary is updated until converges. Sparse Representations with Detail Injection Pansharpening (SRDIP) is one of the recent method that published in pansharpening field. In SRDIP method, high and low frequency wavelet planes are extracted by using à trous wavelet transform. Training data set for each wavelet planes are constructed by getting image patches from planes itself. Trained dictionaries are generated from these data sets adaptively using K-SVD dictionary training. In proposed method, high frequency wavelet planes which are extracted by à trous wavelet transform are injected to the multispectral images proportionally as in AWLP method. The amount of wavelet planes are controlled λ sparsity contribution parameter because it is thought that wavelet planes to be injected to multispectral images have much information than needed. Therefore, proposed method's aim is to find the proportion of the wavelet plane needs to be added to MS image. An iterative algorithm has been applied to solve this. In this study, color transformation based GIHS, à trous wavelet transform based ATWT, AWLP, ATWT-CBD methods, a new method SRDIP which combination of à trous wavelet transform and sparsity and also proposed method are compared. Different images from Quickbird satellite are used to interpret the results. Performances of above methods and proposed method are compared in terms of visual and quantitative. Sharpness, blurness , distortions, missing spatial details are important determinants in visual assessment. Quantitative assessment can be done in two way as synthetic test and real test. In synthetic test, MS and PAN images are degraded. Thus, original MS image is used as reference to assess fused image. However in real test, there is no reference image to compare visually because fusion image and original MS image have different resolutions. To quantitatively evaluate synthetic test, Correlation Coeefficient (CC), Q4, ERGAS (Erreur Relative Globale Adimensionnelle de Synthèse), Relative Average Spectral Error (RASE), Spectral Angular Mapping (SAM), Spaital ERGAS (SERGAS) are most common performance metrics. As to real test, Quality with No Reference (QNR), spectral distortion and spatial distortion are the metrics that is used to assess results. GIHS method has worst results in terms of quantitative assessment. Although it has sharper results, it causes color distortion in green areas. ATWT and AWLP methods have over-enhancement problem along the edges. Even though some of the spectral metrics show that ATWT-CBD improves spectrally, it suffers from missing spatial details locally. Proposed method shows that it improves the results in spectral metrics compared to ATWT and AWLP methods and also reduces artifacts especially tree areas. Whereas proposed method has better metric results than SRDIP in real tests, SRDIP improves the metrics better than all methods examined in terms of synthetic tests. Compairing in terms of fusion processing time, GIHS, ATWT and AWLP methods complete the fusion process less than 1 second whereas ATWT-CBD takes approximately 35 second because it is calculated on square windows. SRDIP method is slightly slower than proposed method with trained dictionary. However, proposed method with non-trained dictionary is approximately six times faster than SRDIP.