Optical image/data encryption techniques are mostly based on the manipulation of spatial distributions of light’s amplitude, phase, and polarization. Information encoding with phase involves complex interferometric set-up and polarization encoding requires Stoke’s parameter measurement. Hence, they create difficulties in optical implementation. Considering the practical limitations, in this study, we demonstrate a method of single-shot intensity recording-based color image encryption by encoding the information in binary polarization states. The proposed method does not require Stoke parameter calculation. As a proof-of-concept, we demonstrated the technique with coherent and partially coherent light sources. Use of partially coherent light overcomes the speckle problem and makes the system cost-effective, useful for practical applications.