Nowadays with things like SAM (Segment Anything Model), product segmentation has become much easier, but 10 years ago it was a really tough problem. When I first tried SAM, I spent over 20 minutes testing random things while going "this actually works??" That's how impressive it was.
A Shift in Thinking: Finding the Answer Outside the Image
The Flipkart team's ultimate conclusion was "this isn't a problem you can solve with image processing alone."
Here's the key idea. Flipkart's catalog system already had color attributes stored for every product. When you register a shirt, color information like "red+blue" gets entered. The insight was to combine this metadata with image analysis.
However, you can't just use the color attribute values directly for the palette. "Red" doesn't mean the same red every time. Wine red is red, cherry red is red, and coral could arguably be red too. You need to find that specific shade of red that's actually in the product photo for it to be meaningful.
The Actual Process
Here's roughly how it worked.
First, apply color quantization to the product image to reduce it to about 16 colors. Then run a histogram to extract the RGB values of the top 8-10 most frequent colors. Simultaneously, fetch the product's color attributes from the catalog API (e.g., grey, pink). Convert these attributes to RGB (grey → #808080, pink → #FFC0CB), calculate the "distance" between the top colors and the attribute colors, and select the closest match. Finally, combine the selected colors to create the palette image.
Histogram extraction with ImageMagick looks like this:
convert quantized.jpeg -colors 16 -depth 8 -format "%c" histogram:info: | sort -nr | head -n8
They reportedly combined im4java for image processing with the Catalano framework (a scientific computing library for Java) for color distance calculations.
Why You Shouldn't Compare Colors in RGB
I need to get into some color theory here, because understanding this part is what makes the use of CIELAB make sense.
Intuitively, it seems like you could measure how similar two colors are using Euclidean distance in RGB. Square the differences in R, G, B, add them up, take the square root, and you're done. But the problem is that mathematical distance in RGB space doesn't match how the human eye perceives color differences.
Even if the RGB distance is exactly 10, in some color combinations the difference is immediately noticeable while in others the colors look almost identical. The human eye is quite sensitive to subtle differences in greens but relatively insensitive to blues. Perceptual sensitivity itself varies by color region.
To solve this, the CIE (International Commission on Illumination) created the CIELAB color space. L* is lightness, a* is the red-green axis, and b* is the yellow-blue axis. Euclidean distance measured in this space corresponds much more closely to human color perception differences. This is called Delta E (ΔE). A ΔE value of about 2.3 is reportedly the level at which people start to notice "hmm, that looks a bit different."
Honestly, CIELAB isn't perfect either. It was later discovered that perceptual uniformity breaks down in highly saturated regions, which is why improved formulas like CIE94 and CIEDE2000 came out sequentially. CIEDE2000 is the current industry standard, but the calculation is much more complex. For an e-commerce palette use case in 2014 though, CIE76 level accuracy was probably sufficient.
Results: Surprisingly Usable
There were several advantages.
Speed was on a completely different level. Comparing it to manual work was meaningless. Even with 500,000 new products per day, processing happened instantly with no queue, unaffected by seasonal fluctuations.
File size reduction. Manually cropped palettes included color gradients and were larger (averaging 529 bytes), while auto-generated palettes were solid colors with better compression efficiency, around 301 bytes. That's roughly a 43% reduction. When you're dealing with millions of images, this difference is not negligible.
Guaranteed 1:1 aspect ratio. Manual crops varied in proportion depending on who did them, but auto-generated ones were always perfect squares, maintaining consistency across the entire site.
Results were reportedly especially good for products with fine patterns spread throughout or multi-color products. Making a palette for a striped polo shirt manually requires precisely targeting the intersection point where colors overlap, which is harder than you'd think. For a polo shirt where the top third is white, the palette could look completely different depending on where you crop.
Limitations: Honestly, There Were Weak Spots
The biggest weakness of this approach is that it's completely dependent on catalog data quality. If the color attributes stored in the backend are wrong, the palette will obviously be wrong too. If a seller registered something as "navy" but it's actually closer to black? The palette will pick the wrong color.
The color name → RGB mapping is also a surprisingly tedious task. Colors like peacock, navy, and skin need to be manually defined, and attributes like "multicolor" can't be converted to RGB at all, so they need exception handling. It's supposedly a one-time thing, but I imagine there were quite a few edge cases.
Also, this is just my thought, but you know those seller-taken photos shot under yellowish lighting? In those cases, colors will inevitably be distorted during the quantization stage, so no matter how accurately you calculate the CIELAB distance, the results will be off. This must have been an issue in practice.
Oh, and this came to mind while writing: if you look at Coupang these days, instead of color palette squares they just show small thumbnails of the product in that color. Network speeds have gotten faster and CDN costs aren't as burdensome as they used to be, so it feels like the concept of palette images itself is gradually disappearing. Sometimes when the times change, the problem itself ceases to exist.
Why I Wrote Up a 10-Year-Old Post
You'd never use this exact technology today. These days you'd segment the product area with a segmentation model and extract the dominant color, which is far more accurate.
But the reason I like this case is that it shows you don't need hard tech to solve hard problems. Isolating products from images was nearly impossible in 2014, but a single shift in perspective — "the catalog already has color data, doesn't it?" — made the problem easy.
I've also had experiences where I got fixated on a technically cool approach and wasted time for nothing... I once spent 3 hours fine-tuning a model only to end up solving it in 5 minutes with a rule-based approach. The feeling of futility was something else. Reading posts like this brings back that memory and keeps me humble.
If you're studying AI image generation or computer vision, don't just chase the latest papers — read practical engineering posts like this from time to time too. Transformers and diffusion models are important, but the ability to redefine the problem is ultimately more powerful. That's what I think.
