Secure Image Display through Visual Cryptography:Exploiting Temporal Responsibilities of the Human Eye

Jong-Uk Hou1, Dongkyu Kim2, Hyun-Ji Song3, and Heung-Kyu Lee4∗

School of Computing, KAIST, Daejeon, South Korea{juheo1, dkim2, hjsong3, hklee4}@mmc.kaist.ac.kr

ABSTRACTWe propose a new protection scheme for displaying a staticbinary image on a screen. The protection is achieved by avisual cryptography algorithm that divides the target im-ages into several divisions. The visual difference betweenthe text and the background is induced by exploiting thetemporal responsibilities of the human eye. With the resultsof our user study, we demonstrate that encrypted visual in-formation was mentally recovered by the human visual sys-tem. Moreover, the images captured from our scheme do notprovide any meaningful information to the human eye, sothat our method provides a strong security measure againstscreenshot piracy.

KeywordsVisual cryptography, screenshot, copyright protection, dis-play algorithms

1. INTRODUCTIONWith the remarkable development of the Internet, a num-

ber of people have shared and collected various digital datavia the Internet. The demand for multimedia content such aspictures and videos have rapidly increased, and the value ofthe content has also grown. To collect the content displayedon a computer monitor or mobile device, taking screenshot isone of the method used most. Using the print-screen key andbuilt-in capture function, users can easily take a screenshot.There are also various screen grabber tools that capture abitmap of the content displayed on a monitor. While thescreenshot offers convenience, it causes copyright infringe-ment when a screenshot of copyrighted content is distributedwithout permission. Therefore, secure image displays againstscreenshot is essential in the multimedia industry.

There are several methods that protect images againstscreen capturing. Generally, they operate on an operating

∗Corresponding author

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.

IH&MMSec 2016, June 20-23, 2016, Vigo, Spainc© 2016 ACM. ISBN 978-1-4503-4290-2/16/06. . . $15.00

DOI: http://dx.doi.org/10.1145/2909827.2930805

system or on hardware [11, 13]. Snapchat, a mobile mes-saging application, does not protect screenshots from beingtaken but notifies the sender if the recipient takes a screen-shot [1]. In addition, digital watermarking and fingerprintingcan be used to create an active protection system that al-lows to control the distribution of content [2]. Further, adigital forensic technique such as that proposed in [9] canhelp analyze the source of the captured image.

Yamamoto et al. [17] proposed a display technique thatensures security of visual information based on visual cryp-tography. The encoded image on the display decoded withthe mask, and its visual data is recognizable on the limitedviewing zone. In addition, a screenshot protection techniquewith an image processing approach was proposed [18]. Theydistorted visual data displayed on the monitor to make thescreenshot meaningless. They exploited the human visualsystem to empower viewers to automatically and mentallyrecover the distorted content into a meaningful form. How-ever, in spite of their contribution, the screenshot of theprotected image can be still identified.

In this paper, we encrypt a static binary image (e.g., textdata) shown on a screen based on secret image sharing [12,14]. Instead of image recovery with spatial processing, eachshared image is sequentially displayed on the screen andrecovered by the human eye. To make this possible, in ourscheme the encrypted images are generated as a specific formexploiting the temporal responsibilities of the human visualsystem. In a user study, encrypted images were successfullyrecovered by the human visual system, and the capturedimages from our scheme were not able to be recognized bythe human eye.

The rest of this paper is organized as follows. In Section2, we briefly introduce the background knowledge, and inSection 3 we explain the details of the proposed method.To demonstrate the performance of the proposed scheme,the quantitative evaluation and user study are presented inSection 4, and in Section 5, we present our conclusions.

2. TEMPORAL RESPONSIBILITYInformation on images is sampled by our eyes and pro-

jected onto the retina in a periodic manner [8]. Then, thecollected information is integrated so that objects appearto be stable and move smoothly. Because there is a finiteamount of time required to process visual information, thereare limitations on the responsiveness of our visual system torates of change. When periodic visual stimuli called flickerare presented to the human eye, the stimuli are perceived asseparate if the rate is below a certain value. If the rate of

169

(a) (b)

Figure 1: (a) Plot for Bloch’s Law (data plotted as logL andlog t), and (b) apparent brightness of flashes with variousluminances (Broca and Sulzer data from [5])

presentation is above a certain critical rate, the flicker van-ishes from the human eye. To perceive flashes of light oneafter the other, an appropriate integration time is required.

Temporal integration time is related to temporal summa-tion, which refers to the eye’s ability to sum the effects ofeach flicker over time. However, temporal summation onlyoccurs within a certain period of time, called the criticalduration [7]. According to Bloch’s Law of temporal summa-tion, the product of luminance L and stimulus duration tequals the constant value k.

L · tp = k, (1)

where the constant value k is determined by the total lumi-nous energy, and p describes whether temporal summationis complete (=1) or partial (0<p<1). No temporal summa-tion occurs when p=0 (See Fig. 1(b)). Temporal summationis also affected by other test variables such as backgroundluminance and the size of the stimulus. Critical duration islonger for brighter background luminance and a smaller areaof the stimulus.

To readily investigate the visual system’s response to flicker,a temporal contrast sensitivity function was plotted by DeLange [3]. A temporal contrast sensitivity function is a plotof how flicker varies with contrast level. The eye appearsto be most sensitive to a frequency of 15-20 Hz at high lu-minance (above 100 trolands). To detect flicker at a highfrequency, the maximum contrast level is required. In addi-tion, the Broca-Sulzer effect describes the apparent transientincrease in brightness of a flash of short duration. Subjec-tive flash brightness occurs with a flash duration of 50 to100 milliseconds. Detailed information can be found in [7].

3. PROPOSED METHODIn this study, we encrypted a static binary image (e.g., text

data) shown on a screen based on secret image sharing [12,14]. As shown in Fig. 2, each shared image was sequentiallydisplayed on the screen and recovered by the human eye. Inour method, pixels from the image are categorized by twotypes: type 1 for background pixels, and type 2 for maincontent. Each type of pixel is encoded to the form of pseudo-random noise, so that the content from each of the distortedimages cannot be recognized. We induced a visual differenceof each pixel type by exploiting the temporal responsibilitiesof the human eye.

Figure 2: Overall process of the proposed scheme

Fig. 3 shows the main feature of the temporal resolutionexploited in our scheme. A flickering signal over a criticalduration made itself conspicuous to the human eye. In con-trast, a flickering below the critical duration was perceivedas a plain gray pixels. In addition, following features wereused to maximize the perceptibility of the recovered con-tent: 1) sharing number n is controlled by the refresh rateand the maximum luminance of the screen (by Bloch’s Law),2) brightness of the shared image is varied by the luminanceof the flickering (by the Broca-Sulzer effect).

The proposed method is divided into three steps: 1) deter-mining the critical duration, 2) encrypted image generation,and 3) content recovery. We now describe each step in detail.

3.1 Determining a critical durationTo determine the critical duration for our scheme, the

maximum luminance lm(cd/m2) and frequency fm(Hz) aremeasured from the display. Then, critical duration tc for thegiven environment is determined by Bloch’s Law and theBroca-Sulzer effect [5, 7]. For instance, tc is around 90 mil-liseconds for the pixels of the display with lm = 350cd/m2.

The upper bound of the number of frames n that inducesthe temporal summation is calculated as follows:

n = d tc · fm2e, (2)

Flickering over n frames made itself conspicuous to humaneye. In contrast, flickering below n frames was perceived asplain gray pixels (See Fig. 3). We use the value n as a thresh-old to set content pixels apart from background pixels.

3.2 Encrypted image generationWe denote a static display of the screen as image I and

denote the intensity at the (x, y) coordinate of the screenas I(x, y). Next, each pixel from I are categorized by two

Figure 3: Apparent brightness of pixels from the proposedalgorithm

170

types:

{type 1: background pixels,

type 2: pixels for main contents.(3)

A threshold value of 0.5 is used for separating image pixelsinto two categories.

To induce visual difference of each pixel type, I is sharedto Si for i = 0, 1, ..., N . The random noise image as theinitial share S0 is generated as follows:

S0(x, y) = rand({0,1}), (4)

where the function rand({0,1}) generates a random number∈ {0, 1}. Next, shared images Si for i = 1, ..., N are recur-sively generated as follows:

Si(x, y) =

{if I(x, y) ∈ type 1 : not(Si−1(x, y)),

otherwise : S∗i (x, y),

(5)

S∗i (x, y) =

{if i is divisible by n : rand({0,1}),otherwise : Si−1(x, y),

(6)

where n denotes the maximum length of the frames thatinduces the temporal summation, and not() is a functionthat changes a value 0 and 1 into 1 and 0, respectively. Asa result of the generation, pixels for background (type 1)appeared as a flickering signal for every frame. Pixels forcontent (type 2) also appeared as a flickering signal, buttheir durations were at least n times longer than those oftype 1. Fig. 4(b) presents the shared image S1 for Fig.4(a). As we can see in the figure, the alphabet ‘A’ from theoriginal image is never recognized in the shared image.

Meanwhile, with random noise based generation there is ahigh probability of having a locally biased area in the synthe-sized image. A mass of random noise from the backgroundpixels can cause visual disturbance, so that varying the vi-sual difference of each pixel type is more difficult to achieve.Therefore, to evenly distribute the random noise from thelocal area, values from Si are varied using the basis matricesM0 and M1 defined as follows:

M0 =

[1 0

0 1

], M1 =

[0 1

1 0

]. (7)

If Si(x, y) is 0, the pixel is replaced by M0. The same replace-ment is conducted for a value 1 and M1 (See Fig. 4(c)). Dueto the pixel expansion through the replacement, one pixelfrom the original image is expanded into four pixels.

3.3 Content recoveryThe individual shares give no clue that a specific pixel is

‘0’ or ‘1’ regardless of the amount of computational power.The only way to reconstruct the original image on the ma-chine requires that the shared images Si be collected. Incontrast to conventional schemes such as those in [10, 14],shared images from our scheme can be recovered not onlyon the machine but also on the human visual system.

To achieve this, the generated S is sequentially displayedon the screen as a videos. Fig. 4(d) shows the expected viewof the recovered image. By temporal summation, pixels for

(a) (b)

(c) (d)

Figure 4: Sample of results: (a) input image I, (b) S1, (c)encoded S1 with M0 and M1, and (d) expected view of re-covered image.

the background (type 1) are perceived as gray pixels when1/fm is larger than tc. In contrast, pixels for content (type2) appear as a flickering signal because their temporal du-ration is at least n frames, which is not able to induce thetemporal summation. As a result, the human eye can distin-guish between the contents pixels and the background pixels,and can read (or recognize) the protected content.

4. EXPERIMENTAL EVALUATIONIn this section, we demonstrate the effectiveness of our

method in terms of content protection through deep ex-perimental analyses, which include objective and subjectivetests. Similar to the study in [18], we generated 400 × 400images as seen in Fig.5(a). The pixel intensities of charac-ters and background were set to 0.25 and 0.75, respectively.All the characters were randomly drawn from capital lettersand numbers.

The experiments were carried out under the following en-vironments. We used a desktop computer equipped with In-tel(R) i7-4790 CPU and NVIDIA GeForce GTX750. It wasconnected to Qnix QX2414 LED 144 (144Hz, 350cd/m2)and Samsung SyncMaster 226BW (60Hz, 300cd/m2). Thenumbers in parentheses indicate the refresh rate and max-imum brightness of the monitors. For the mobile environ-ment, an Apple iPhone 6 was employed and its display had a60Hz refresh rate and 500cd/m2 maximum luminance. Thecritical durations tc, and n for each display were determinedas follows: Qnix (tc ≈ 100ms, n = 6), Samsung (tc ≈ 90ms,n = 3), and iPhone (tc ≈ 70ms, n = 3).

We compared our method with three different methods asfollows: the visual cryptography of Naor and Shamir [10],Ito et al. [6], and the video based method of Chia et al. [18].

171

(a) (b) (c) (d) (e)

Figure 5: Sample of screenshot images: (a) original image, (b) Chia et al. [18], (c) Naor and Shamir [10], (d) Ito et al. [6], and(e) our method.

Table 1: Comparison table of the related algorithms

Algorithms [6,10] [18] oursRecovery by machine ◦ ◦ ◦

Recovery by human eye × (or ◦) ◦ ◦Concealment of visual info. ◦ × ◦

We used (2, 2) threshold scheme for the visual cryptographyalgorithms [6,10]. For the method of Chia et al., the same nvalues corresponding our method were used.

Table 1 presents a summary of comparisons with our methodfor recovery by machine, recovery by the human eye, andconcealment of visual information. The detailed results ofcomparison will be handled in the following subsections.

4.1 Quantitative EvaluationTo check the information distortion capability of our method,

we conducted a quantitative evaluation by measuring theaverage root mean square (RMS) distances between the dis-torted images and their corresponding original images. Thedistorted images indicated frames composing a video whichwere generated to protect an image by each method. Thiswas repeated over 1,000 different test images with adjust-ment of the number of whole frames n. Fig. 6 shows the meanaverage-RMS distances according to variations of value n.The distances in our method and visual cryptography werealmost constant at 0.56 and thus invariant to n values. Mean-while, the method of Chia et al. was affected by n and its dis-tance was gradually converged into 0.32, which meant that

Figure 6: The mean average-RMS distances according tovariations of value n.

each distorted image could reveal information of the originalto some extent. Note that each frame in our method con-sisted of the same number of black and white pixels whichwere randomly selected as explained in Section 3.2, and thusdid not unveil any information.

4.2 User StudyFor a qualitative evaluation, we performed a user study

which consisted of visual information recoveries under twodifferent circumstances, normal and abnormal. Ten subjectsparticipated in this study, who had normal or corrected-to-normal vision. The mean age was 28.5, and the youngestand eldest were 25 and 33, respectively. All the subjects wereguided to look at the screens approximately 50cm away and30cm for settled monitors and mobile phones, respectively.

4.2.1 Distortion of Visual InformationIn the first user study, we looked for how much informa-

tion the distorted images would reveal visually. From the ndistorted images, a randomly selected image was presentedto the subjects. Then they were guided to read characters inorder and we calculated recognition rates, or accuracy. Therecognition rate was defined as the percentage of charactersidentified correctly.

Table 2 shows the average accuracy of distorted imagesgenerated by each method. It is not surprising that ourmethod and visual cryptography methods had a 0% recogni-tion rate because they performed pixel-wise encryption, andthus characters and background were encoded with binarynumbers, 0 and 1. In contrast, the method of Chia et al. gen-erated distorted images with distorting planes in which eachpixel value was a real number between -1 and 1. It led to theleakage of visual information to some extent and a high av-erage recognition rate. Note that the image resolution of thistest was higher than that of Chia’s test (100 × 100), whichmade visual information prominent owing to the increase ofabsolute amount of pixels.

Table 2: The average character recognition rates of distortedimages generated by compared methods

Display type [18] [10] [6] OursQnix LED (144Hz) 100 0 0 0

Samsung (60Hz) 99.6 0 0 0iPhone6 (60Hz) 100 0 0 0

Average 99.87 0 0 0

172

Figure 7: The average character recognition rates after show-ing streams of images generated by compared methods

4.2.2 Normal Recovery of Visual InformationWe laid our whole distorted images in correct order when

generating a stream or video and confirmed whether hu-mans could recover visual content normally from it. As inSec. 4.2.1, we measured the average recognition rates aftershowing streams of images to subjects. Each stream playedduring 30 seconds which were assumed to be enough timefor the subjects to write 25 characters on the images.

Fig. 2 shows the results of user study. Ours and the methodof Chia et al. demonstrated high average recognition ratesregardless of monitor’s refresh rate. In both methods, dueto the composite effects of low spatial resolution and insuffi-cient time, some subjects could not discriminate between ‘S’and ‘5’, and between ‘I’ and ‘1’, which resulted in not 100%recognition rates.

Meanwhile, image streams of the visual cryptography meth-ods also created recognizable artifacts that provided feasi-ble information to subjects. Due to the generation processof the visual cryptography methods, visual differences be-tween content region and background region also presentedfrom the encrypted streams. The larger refresh rates is, themore distinct differences arised. It is natural that increasingthe refresh rate decreases flickering and consequently thetemporal summation of consecutive streams is the same asbit-OR operation in visual cryptography. The visual differ-ences were not so clear as those of our methods, so thatthe methods [6, 10] demonstrated relatively low recognitionrates, especially in the case of the iPhone 6 with small dis-play.

4.2.3 Abnormal Recovery of Visual InformationIn this section, we assumed a possible scenario in which an

attacker would capture several distorted images from a gen-erated stream and try to recover its information by stackingthe captured images. To measure robustness against thatkind of attack, we randomly picked k distorted images fromthe total set. As an attack, we averaged the selected imagesand presented the resulting image to the subjects.

Fig. 8 shows the averaged result using 5 screenshot im-ages. Our method and visual cryptography output illegiblenoisy images by averaging the pixel-wise encrypted imagesand consequently produced a low recognition rate. Note thatsome case of our method results distinguishable artifact (Seecircled region in Fig. 8(c)), so that an observant eye mayperceive the abnormally recovered content. Meanwhile, Chiaet al. distorted every image with the same amount of noiseand without order. In addition, their algorithm was designedto result in zero noise when summing the distorted images,

(a) (b) (c)

Figure 8: Sample of averaged images using 5 screenshots:(a) Chia et al. [18], (b) Naor and Shamir [10], and (c) ourmethod.

and thus recognition rates would clearly ascend with an in-crease in k.

5. DISCUSSION AND CONCLUSIONIn this paper, we proposed a new protection scheme for a

static binary image shown on a screen exploiting the tem-poral responsibilities of the human visual system. With theresults of our user study, we demonstrated that encryptedvisual information was mentally recovered by the human vi-sual system. Moreover, captured images from our scheme didnot provide any meaningful information to either humans orcomputer programs; thus, our method provides strong secu-rity against screenshot piracy. To best our knowledge, thereare no visual cryptography schemes that consider the tem-poral aspect, so existing methods ineffectively perform theway described in this paper.

On the other hand, the proposed methodology has a fol-lowing limitation. Our method is based on the simple noisepattern generated by recursion, so that it is obvious thata malicious user reveals visual information through differ-ence between subsequent frames. Therefore, we have a fu-ture work to complement the weakness. For instance, wewould depart from the recursion based algorithm and devisea statistical noise pattern which can make temporal respon-sibility different. In addition, we will exploit more featuresrelated to temporal responsibilities to our scheme.

We found various potential applications for the proposedmethod. First of all, the proposed scheme can enhance thesecurity of a mobile authentication system, such as a one-time-password system [4] or personal identification num-ber [15]. The proposed method can be used for the type ofchallenge-response test (e.g., CAPTCHA [16]) used in com-puting to determine whether or not the user is a human. Inaddition, we would like to apply our method in combinationwith existing security methods to enhance security. Regard-ing such applications, we believe that the proposed methodcan be used in a variety of fields.

AcknowledgmentsThis research project was supported by Ministry of Cul-ture, Sports and Tourism(MCST) and from Korea Copy-right Commission in 2015. The work of Jong-Uk Hou wassupported by a Global PH.D Fellowship Program throughthe National Research Foundation of Korea(NRF) fundedby the Ministry of Education (2015H1A2A1030715).

173

6. REFERENCES[1] J. Charteris, S. Gregory, and Y. Masters. Snapchat

‘selfies’: The case of disappearing data’. Rhetoric andReality: Critical perspectives on educational technology,pages 389–393, 2014.

[2] I. Cox, M. Miller, J. Bloom, J. Fridrich, andT. Kalker. Digital Watermarking and Steganography.Morgan Kaufmann Publishers Inc., San Francisco,CA, USA, 2 edition, 2008.

[3] H. De Lange Dzn. Research into the dynamic natureof the human fovea-cortex systems with intermittentand modulated light. i. attenuation characteristicswith white and colored light. The Journal of theOptical Society of America, 48(11):777–783, 1958.

[4] N. Haller, C. Metz, P. Nesser, and M. Straw. Aone-time password system. Technical report, 1998.

[5] W. J. Hart. The temporal responsiveness of vision.Adler’s physiology of the eye, Clinical application,1987.

[6] R. Ito, H. Kuwakado, and H. Tanaka. Image sizeinvariant visual cryptography. IEICE transactions onfundamentals of electronics, communications andcomputer sciences, 82(10):2172–2177, 1999.

[7] M. Kalloniatis and C. Luu. Temporal resolution.Webvision: The Organization of the Retina and VisualSystem, 2005.

[8] H. Kolb, E. Fernandez, and R. Nelson. Webvision: Theorganization of the retina and visual system. 1995.

[9] J.-W. Lee, M.-J. Lee, H.-Y. Lee, and H.-K. Lee.Screenshot identification by analysis of directionalinequality of interlaced video. EURASIP Journal onImage and Video Processing, 2012(1):1–15, 2012.

[10] M. Naor and A. Shamir. Advances in Cryptology —

EUROCRYPT’94: Workshop on the Theory andApplication of Cryptographic Techniques Perugia,Italy, May 9–12, 1994 Proceedings, chapter Visualcryptography, pages 1–12. Springer Berlin Heidelberg,Berlin, Heidelberg, 1995.

[11] H. Okhravi and D. M. Nicol. Trustgraph: Trustedgraphics subsystem for high assurance systems. InComputer Security Applications Conference, 2009.ACSAC’09. Annual, pages 254–265. IEEE, 2009.

[12] A. Shamir. How to share a secret. Communications ofthe ACM, 22(11):612–613, 1979.

[13] M. Stamp. Digital rights management: The technologybehind the hype. J. Electron. Commerce Res.,4(3):102–112, 2003.

[14] C.-C. Thien and J.-C. Lin. Secret image sharing.Computers & Graphics, 26(5):765–770, 2002.

[15] G. J. Tomko and A. Stoianov. Method and apparatusfor securely handling a personal identification numberor cryptographic key using biometric techniques,Jan. 27 1998. US Patent 5,712,912.

[16] L. Von Ahn, M. Blum, N. J. Hopper, and J. Langford.Captcha: Using hard ai problems for security. InAdvances in Cryptology—EUROCRYPT 2003, pages294–311. Springer, 2003.

[17] H. Yamamoto, Y. Hayasaki, and N. Nishida. Secureinformation display with limited viewing zone by useof multi-color visual cryptography. Optics express,12(7):1258–1270, 2004.

[18] A. Yong-Sang Chia, U. Bandara, X. Wang, and

H. Hirano. Protecting against screenshots: An imageprocessing approach. In Proceedings of the IEEEConference on Computer Vision and PatternRecognition, pages 1437–1445, 2015.

174