Each image has its own color content that greatly influences the
perception of human observer. Being able to transfer the color
content of an image into another image, while preserving other
features, (like texture), opens a new horizon in
human-perception-based image processing. In this paper, after a
brief review on the few efficient works performed in the field, a
novel fuzzy principle component analysis (PCA) based color
transfer method is proposed. The proposed method accomplishes the
transformation based on a set of corresponding user-selected
regions in images along with a blending ratio parameter set by the
user. Results show more robustness and higher speed when comparing
our proposed method with other available approaches.
Image Recoloring, Principle Component Analysis (PCA).
Altering color appearance of an image, due to the information
extracted from another image has been under investigation in
recent years. Reinhard et al. [1] discuss
that removing a dominant and undesirable colorcast (such as the
yellow in the photos taken under incandescent illumination) will
be a handy tool. Chang et al. [2] state that as
each artist has its own style like the "yellow that Van
Gogh likes to use", it will be fascinating to be able to
transfer the feeling of a painting to a photograph, what they call
"giving the viewers the impression similar to that given by
the painting". Greenfield et al.'s
work [3] is similar to Chang et al.'s work
in spite for the fact that the two images in their method are both
colored-paintings. They state their method as "liquefy the
color of a painting in such a way that it could be poured to
another".
The notion of color transfer is not widespread in the
literature. Perhaps the first work in this field is the method
developed by Reinhard et al. [1]. they
designed a method for color transfer by choosing a suitable color
space and applying the reference image's color appearance to the
source image by means of some statistical parameters. The paper
emphasizes that the proper choice of a suitable color space is of
great importance [1]. Because when the images of the
nature are represented in many typical color spaces, there is a
high correlation between channels, making single channel
alteration a difficult task. They choose lab color
space by Ruderman et al. [4]. Reinhard
et al. express that the lab color space has never
been applied otherwise or compared to the other color spaces.
Authors are not aware of any correspondence to the color space
since then, rather
than [1,2,3,4,5,6].
Using the lab color space, Reinhard et al.
state their color transfer method as mapping the color vectors
using first and second order statistics. They express that, as
that method tries to transfer one image's appearance to another
one, it is possible to select such source and reference images
that do not work well. To overcome this shortcoming, they proposed
to use the concept of swatches. In the new method, user
must select some different swatches in the two images. For example
the grass, the sky, and so on. Then a classification task takes
place and each pixel will be altered according to the statistics
of the corresponding swatches in the two images that it belongs.
One of the main contributions of that work is the idea of
computing the altered color vector in each of the classes and
blending them inversely proportional to the distances from the
initial point to each of the clusters.
Chang et al. [2] used an early work by
Berlin et al. [7] that examined 98 languages
from several families, and reported that there are regularities in
the number of basic colors and their spread on the color space.
They stated that in developed languages, there are eleven color
terms. In English, they are black, white,
red, green, yellow, blue,
brown, pink, orange, purple, and
gray. Chang et al.'s work [2] that is
implemented entirely in the CIE-La*b* color space, uses later
works that defined the spread of these categories. Their method
begins with making the 11 loci of the points in source image
that belong to any of the 11 clusters. Then, they generate the
convex hull that encloses all of the pixels within each of the
categories. The same task is performed in the reference image. For
the given vector [c\vec]1 in the i-th category, the recolored
vector is computed using a linear mapping between the convex
hulls. It easy to see that Chang et al. [2] is
using the same mapping idea of Reinhard et al.
's [1] swatches, except that the swatches are
preselected in his method; although resulting in less needed user
supervision, but also less adaptivity when dealing with special
problems. Also, the method leaves few room for user intervention.
Greenfield et al. [3] organized the images
into pyramids to produce a palette for each image by successively
down-sampling it. The color transfer is then performed based on
the two palettes, computed in the source and the reference images.
Although, that work seems to use a different method compared with
the Reinhard et al.'s method [1], but when
they tend to transfer the actual color, they apply lab color space and alter the a and b content of
points, leaving l unchanged, like what Reinhard et
al. [1] have proposed.
(a)
(b)
Figure 1: Typical result of the proposed color transform method.
(a) Original Image. (b) Recolored Image.
When working with multi-spectral images, data dimension is an
important problem, (showing itself as a factor increasing the
processing time massively). There are a few works on using
dimension reduction methods in color images (e.g., see
[8]). Principal Component Analysis (PCA) is a fast
linear dimension reduction method [9]. The basic idea
behind the PCA is to find the linear transform giving the maximum
amount of variance out of the set of given vectors. The axis of
i-th maximum variance is denoted as [v\vec]i. In practice,
the computation of [v\vec]i is accomplished by using the
covariance matrix C = E{([x\vec]-[`[x\vec]])([x\vec]-[`[x\vec]])T}, where [v\vec]i is the eigenvector of C corresponding
to the i-th largest eigenvalue [9].
In [10] the authors proposed a novel PCA-based
dimension reduction method for natural color images, reducing the
3-D color space to a shifted version of a 1-D vector space.
The idea is developed further to define the linear partial
reconstruction error (LPRE) in [11]. The LPRE
likelihood measure, fuzzyfication scheme, and homogeneity decision
are proved to outperform the conventional Euclidean and
Mahalonobis distances, dominantly [11].
In this paper, firstly, a new color transfer method is proposed
that transforms the color information from the reference image to
the source image, (according to one set of corresponding regions
in the images). The general color transfer method uses the fusion
of the set of single region recolored results, using an
LPRE-based fuzzyfication technique. User contribution in our
proposed method is limited to select a few corresponding regions
in the reference and source image (like the swatch idea of
Reinhard et. al. [1]), along with tuning a
one-parameter membership function.
Some authors call the color transfer method as recoloring,
but in this paper, these two terms are used interchangeable. We
call the image to be recolorized and the image from which the
color information is extracted as the source and the
reference images, respectively. In all formulas, variables
indexed as x1 and x2 belong to the source and the reference
images, respectively. Also, variables denoted as x1prime are
relating to the source image after colorizing task has taken
place. All vectors in this paper are assumed to be column vectors.
The rest of this paper is organized as follows:
Section II-A introduces the single region recoloring
method, while Section II-B states the
fuzzification scheme. Section II-C describes the
proposed fusion technique for rendering the resulting image.
Section III states the experimental results and
discussions and the conclusion are given in
Section IV.
Assume that the image I1 is to be recolorized using the
information in image I2. Also, assume that the region r1 in
I1 should pretend the region r2 in I2. Let the vector
[(h)\vec]r1 denote the expectation vector of r1 and the
3×3 matrix Vr1 contain the eigenvectors of the
covariance matrix of r1 as its columns sorted by corresponding
eigenvalue in a descending fashion. The vector [(h)\vec]r2
and the matrix Vr2 are defined in a similar way for the
region r2 in the reference image (I2). By modelling the
color information in r1, as an ellipsoid spread around
[(h)\vec]r1, the proposed recoloring method for single
selected regions in the source image and the reference image is
defined as:
®
c
¢ 1
=Vr2Vr1-1(
®
c
1
-
®
h
r1
)+
®
h
r2
.
(1)
The linear transformation described in (1), first
subtracts the center of the ellipsoid from all points to move it
to the center. Then using Vr1 which is the PCA matrix of
r1, the pixels of the source image are converted to the PCA
coordinates. Using Vr2 and [(h)\vec]r2 the
transformation goes in the inverse direction.
In [11], the authors proposed to use the error made by
neglecting the two less important principal components as a
likelihood measure. In this method, the LPRE likelihood of the
vector [c\vec] to the cluster r is defined as:
er(
®
c
)=||
®
v
T
(
®
c
-
®
h
)
®
v
-(
®
c
-
®
h
)||
(2)
where [v\vec] shows the direction of the first principal
component and ||[x\vec]|| denotes the normalized L1 norm.
Investigating (2) makes clear that, to make er([c\vec])
comparable over different clusters, a normalization scheme is
crucial. In [11] the authors proposed to use the
following stochastic margin as the normalization factor:
||f||r,p=arge
æ è
P[x\vec] Î r{f(
®
x
) £ e} ³ p
ö ø
(3)
where p is the inclusion percentage. Equation (3) leads
to the definition of the normalized likelihood function:
~
e
r,p
(
®
c
)=
er(
®
c
)
||er||r,p
.
(4)
Also, the ||er||r,p is a proper homogeneity
criterion [11].
Note that er,p([c\vec]) is giving lower values for the color
vectors similar to those that exist in r. Thus, a fuzzy
membership function is needed to map
[-1,1]®[1,1-e] and
[-¥,-1]È[1,¥]®[1-e,0].
In [11], the authors proposed a manipulated form of the
well-known low-pass Butterworth filter for the sake of
tunability and simplicity, as:
Ba, b(x)=
æ è
1+
æ è
x
ta,b
ö ø
2Na, b
ö ø
-1/2
(5)
where Na, b and ta, b are defined
as:
Na, b=
é ë
log2
æ è
a
Ö
1-b2
b
Ö
1- a2
ö ø
ù û
(6)
ta, b = a[1/(Na, b)]
æ è
1-a2
ö ø
-[1/(2Na, b)]
(7)
and [x] denotes the nearest integer value to x. The function
is designed in the way that satisfies Ba,b(1)=a
and Ba,b(2)=b. Note that selecting a large
member of ]0,1[ as the a value and a small member of
]0,a[ as the b value, leads to a desired
fuzzyfication. Also, note that the above definition of membership
functions is in contrast with the general selection of the
Gaussian functions.
Regarding the definition of the normalized reconstruction error
and the re-formulated Butterworth function, the source image is
fuzzyficated with respect to the query region r as:
hr,a,b=Ba,b
æ è
~
e
r,p
(
®
c
)
ö ø
.
(8)
Now, the points in r and the points similar to them, in the
color sense, are mostly giving membership values in the range of
[1,a]; while color information that are not like r are
ranked with poor values. Thus, we set a = 0.99 and p=1/2. Note that by tuning the b parameter, one can easily
control the spread of the membership function. Now, the
fuzzyfication scheme is defined as,
Assume that the user has selected n corresponding regions in the
source image and the reference image, respectively (call them
r11¼r1n and r21¼r2n ). As discussed in
Section II-A, using each set of corresponding regions,
[r1i,r2i], equation (1) gives the recolored image.
We propose to blend the results using the fuzzyfication scheme
proposed in Section II-B. Hence, the proposed
color transfer method is formulated as:
All algorithms are developed in MATLAB 6.5, on a 1100
MHz Pentium III personal computer with 256MB of
RAM. Some of the sample images used in this paper are shown
in Figure 2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
Figure 2: Some typical sample images. (a), (b), (c), (d), (i), and
(k) Source images. (e), (f), (g), (h), (j), and (l) Reference
images. (a), (b), (e), (f), (i) and (j) Adopted
from [1]. (c), (g) Adopted from [3].
(d), (h) Adopted from [2]. (k) "Mc. Cormic Creck
State Park, Indiana " by Mike Briner,
mbphoto@spraynet.com, www.mikebrinerphoto.com
(adopted with permission of the author). (l) "Hanging
Lake" by Brent Reed, brent@reedservices.com
(adopted with permission of the author).
The proposed color transform method is performed on the sample
images (illustrated in Figure 2). The results of our
method along with results of other methods are shown in
Figure 3 . The b value and description of the
selected regions in images are expressed in the caption of
Figure 3. It is worth mentioning that transferring color
information of a 512×512 image into a 512×512
image according to the four selected medium-sized regions only
takes about 4 seconds.
(a)
(b)
(c)
(d)
(e)
(f)
Figure 3: Results of the proposed color transform method applied on
sample images shown in Figure 2. Source images are
Figures 2-a, 2-b, 2-c,
2-d, 2-i, and 2-k,
respectively. reference images are Figures 2-e,
2-f, 2-g, 2-h,
2-j, and 2-l respectively. Parameters are
set to values of (a) b = 0.5, r={Sky,Water}, (c)
b = 0.1, r={Leaves, Sky}, (c) b = 0.8, r={Skulls,Background}, (d) b = 0.95, r={Leaves,Sky,Earth}, (e)
b = 0.95, r={Sky,Building,Pavement}, (f) b = 0.2, r={Leaves,Bushes,Bark}.
(a)
(b)
(c)
(d)
(e)
Figure 4: Results of other available methods applied on sample
images shown in Figure 2. Source images are Figures
2-a, 2-b, 2-c,
2-d, and 2-i, respectively. Reference
images are Figures 2-e, 2-f,
2-g, 2-h, and 2-j
respectively. (a),(b),(e) Reinhard et al.'s
method [1]. (c) Greenfield et al.'s
method [3], (d) Chang et al.'s
method [2].
Comparing the results in Figure 3 (the results of our
proposed method), and Figure 4 (the results of other
methods) shows the performance of our proposed method. While there
is a discontinuity in the sky in Figure 4-e, the sky in
Figure 3-e simulates a real night sky, having in mind
that the photograph is taken at daytime. This fact is more visible
when one confirms that the recolorized Figure 4-e does
not pretend to be a night scene while Figure 3-e does
so. This is also visible in Figure 4-b where the blue
sky has nothing to do with the entirely white sky of
Figure 2-f. The sky in Figure 3-b is more
whitened. Although, Greenfield et al.'s
method [3] has tried to transfer the cold colors
of Figure 2-g to Figure 2-c, but
Figure 4-c still contains the warm violet-blue
color on the topmost skull. In Figure 3-c the colors are
mostly cold. Figure 4-d made by Chang et
al. [2], seems acceptable in spite of the fact that
the method that has produced it, is a massively expensive
operation.
It also should be noted that using a reference image of entirely
different scene compared with the source image does not force the
process to fail, but the results of such operation must be
deliberated carefully. The same event occurs when giving regions
of the source image and the reference image in a scattered
fashion; for example trying to transfer color information of
leaves to sand.
No exact time measurement is reported in other works. Considering
the 4 seconds record of our method while other methods use
sophisticated jobs of segmentation and convex hull computation,
the outstanding performance our proposed method is clear.
It must be emphasized that in the proposed method, using an image
as the source image and the reference image at the same time
(recoloring an image with itself) when working on almost the same
regions in the reference image and the source image, gives an
image that cannot be recognized from the original image. In
addition when using single-region version of the method, the
process is completely reversible.
Although in all samples discussed here the reference image was
unique, there is no limitation that prevents the user from using
two or more images as the reference image. This option may be
useful when trying to recolor a source image due to the sky in
first reference image and the leaves in second reference image.
A new fuzzy principle component analysis based color transform is
proposed and tested on different images, including 5 images
adopted from previous work for the sake of performance comparison.
The images synthesized by our method are more pretending the
reference images when compared with other available approaches.
Also, while other references did not report the time needed for
their proposed methods, our proposed method's computation cost of
a few seconds for 512×512 images is promising.
Acknowledgements
The first author wishes to thank Ms. Azadeh Yadollahi for
her encouragement and invaluable ideas.
E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, "Color transfer between
images," IEEE Computer Graphics and Applications, vol.
September/October, 2001.
Y. Chang, S. Saito, and M. Nakajima, "A framework for transfer colors based on
the basic color categories," in Proceedings of the Computer Graphics
International (CGI'03), IEEE, 2003.
G. R. Greenfield and D. H. House, "Image recoloring induced by palette color
associations," Journal of WSCG'03, vol. 11(1), pp. February 3-7,
2003, 2003.
D. Ruderman, T. Cronin, and C. Chiao, "Statistics of cone response to natural
images: Implementation for visual coding," Optical Doc. Of America,
vol. 15(8), pp. 2036-2045, 1998.
W.-Q. Yan and M. S. Kankanhalli, "Colorizing infrared home videos," in
Proceedings of IEEE International Conference on Multimedia and Expo
(ICME 2003), Baltimore, July 2003.
T. Welsh, M. Ashikhmin, and K. Mueller, "Transferring color to grayscale
images," in proceedings of ACM SIGGRAPH 2002, San Antonio, July 2002,
pp. 277-280.
S.-C. Cheng and S.-C. Hsia, "Fast algorithm's for color image processing by
principal component analysis," Journal of Visual Communication and
Image Representation, vol. 14, pp. 184-203, 2003.
A. Abadpour and S. Kasaei, "A new parametric linear adaptive color space and
its pca-based implementation," in The 9th Annual CSI Computer
Conference, CSICC, Tehran, Iran, Feb. 2004, pp. 125-132.
