2402 IEEE COMMUNICATIONS LETTERS, VOL. 23, NO. 12, DECEMBER 2019

Spectrum Sensing Across Multiple Service Providers: A DiscountedThompson Sampling Method

Min Zhou, Tianyu Wang , Member, IEEE, and Shaowei Wang , Senior Member, IEEE

Abstract— Dynamic spectrum access is a potential solution tothe long-standing radio spectrum scarcity and usage inefficiencyissue, for which spectrum sensing is one of the key challenges.In this letter, we investigate the spectrum sensing order problemin the scenario that the secondary user equipped with limitedsensing capability can get access to the idle spectrum acrossmultiple network service providers opportunistically by spectrumsensing, where the probability of the spectrum being idle variesat temporal scale and is not available for the users. We employan online learning method, referred to as discounted Thompsonsampling, to address the formulated optimization task, which cantrack the changes of the probability of the spectrum availabilityand yields more spectrum access opportunities compared to othermethods.

Index Terms— Online learning, spectrum sensing, Thompsonsampling.

I. INTRODUCTION

DURING the past decades, the rapid growth of mobiledata traffic incurs radio spectrum scarcity issue facing

mobile service providers (SPs). However, investigations havealso shown that the spectrum licensed on different agencies isunderutilized. That is, part of the licensed spectrum, whichis generally divided into a number of channels in mobilecommunication systems, is idle at temporal and spatial scales,leading to spectrum usage inefficiency. Dynamic spectrumaccess is deemed as a potential solution to spectrum scarcityand usage inefficiency problem since the secondary users(SUs) are expected to exploit and get access to the licensedspectrum opportunistically provided that they do not throw toomuch interference to the licensed primary users [1], [2]. It hasattracted much attention in the past two decades.

Spectrum sensing is the prerequisite of dynamic spectrumaccess. However, spectrum sensing is generally expensiveand time-consuming since it is difficult for an SU to sense

Manuscript received July 2, 2019; revised August 4, 2019; acceptedSeptember 3, 2019. Date of publication September 17, 2019; date of currentversion December 10, 2019. This work was partially supported by the NationalNatural Science Foundation of China (61671233, 61801208, 61931023), theJiangsu Science Foundation (BK20170650), the Postdoctoral Science Foun-dation of China (BX201700118, 2017M621712), the Jiangsu PostdoctoralScience Foundation (1701118B), and the the open research fund of NationalMobile Communications Research Laboratory (2019D02). The associate edi-tor coordinating the review of this letter and approving it for publication wasO. Popescu. (Min Zhou and Tianyu Wang are co-first authors.) (Correspondingauthor: Shaowei Wang.)

M. Zhou and S. Wang are with the School of Electronic Scienceand Engineering, Nanjing University, Nanjing 210023, China (e-mail:MF1823074@smail.nju.edu.cn; wangsw@nju.edu.cn).

T. Wang is with the School of Electronic Science and Engineering, NanjingUniversity, Nanjing 210023, China, and also with the National MobileCommunications Research Laboratory, Southeast University, Nanjing 210096,China (e-mail: tianyu.alex.wang@nju.edu.cn).

Digital Object Identifier 10.1109/LCOMM.2019.2941717

a wide range of spectrum due to the hardware limitation [3].As a result, spectrum sensing order is important for the SU indynamic spectrum access systems. Ideally, an SU should sensethe channel with the highest probability of being idle. In [4], adescending channel sensing order is proposed given the prioridle probability of channels. If the activities of primary usersare unknown, an SU can sense the channels based on theachievable rates as discussed in [5]. The case of the unknownidle probability of channels is investigated in [6], where aQ-learning method is introduced to improve system perfor-mance. A two-stage learning algorithm is proposed in [7],where the SU performs channel selection with reinforcementlearning to minimize the time spent on channel sensing.

However, the aforementioned methods cannot cope withthe following common scenarios: First, the probability ofa given portion of spectrum being idle, e.g., the spectrumlicensed to an agency, is usually unavailable; second, eventhough we could estimate the probability of spectrum beingidle by observing it for quite a long time, it could changeaperiodically. In other words, we may use outdated statisti-cal information to make decision in this case, resulting indeteriorating system performance. These facts motivate usto set up a reasonable spectrum sensing model and developefficient algorithm to track the changes of the idle probabilityof spectrum for dynamic spectrum access systems.

In this letter, we investigate the spectrum sensing problemacross multiple SPs, where the probability of spectrum beingidle is different for each SP. Moreover, the statistical law ofthe probability of idle spectrum also varies at temporal scale.We formulate an online learning framework to track thespectrum idle probability and introduce an efficient algorithm,the discounted Thompson sampling (DTS), to address theformulated optimization task. Numerical results show that ourproposed method yields spectrum access opportunities almostas many as the best fixed decision in hindsight. The latter isimpossible to implement since it needs a prophet. Moreover,our proposed algorithm can track the changes of the idleprobability of spectrum more quickly than other methods asdiscussed in [8]–[11].

The rest of this letter is organized as follows. In Section II,problem formulation is demonstrated. We give the DTS-basedspectrum sensing approach in details in Section III. Section IVpresents simulation results and discussions. Conclusions aregiven in Section V.

II. PROBLEM FORMULATION

Consider spectrum sensing order, also referred to as chan-nels selection across multiple SPs. In practical commercial

1558-2558 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

ZHOU et al.: SPECTRUM SENSING ACROSS MULTIPLE SPs: DTS METHOD 2403

Fig. 1. An illustration of the channels of 3 SPs spanning in frequency andtime slots.

mobile communication systems, different mobile SPs areassigned different frequency bands for data transmission byradio regulatory agencies. Specifically, during the same timeperiod, the number of active users served by each SP isdifferent in the same region, so the probability of the licensedchannels being idle is different for each SP. Moreover, theidle probability of the channels licensed to each SP will varyafter a period of time due to the mobility of users. The keychallenge for the SU equipped with limited sensing capabilityis to decide which portion of the spectrum should be sensedto produce more channels free of primary user activities totransmit data by the SU.

Consider that there are N = 3 SPs and the spectrumlicensed to SP i consists of C non-overlapping channels. Dueto the limited spectrum sensing capability of hardware, an SUcannot sense a large range of spectrum. Specifically, at giventime slot t, the SU can only keep sensing the channels licensedto one SP which fall into the same frequency band range.We investigate the total data throughput of the SU who utilizesthe idle channels by spectrum sensing in T time slots. Theprobability of spectrum being idle for SP i at time slot t isdenoted by Pi, where i ∈ {I, II, III}. In practical scenarios,each channel of an SP could have own probability of beingidle. Assume that the reward of each channel c of SP i obtainedat time slot t, which is denoted as Hi,c, is Bernoulli withprobability Pi, which can be written as Hi,c ∼ Bernoulli(Pi)for c ∈ {1, 2, . . . , C}. If channel c is available for the SU,the reward of channel c equals 1; otherwise, it equals 0. Thesystem model is shown in Fig. 1. Recall that Pi cannot beknown in advance and would change at temporal scale sothat it makes no sense to estimate Pi by observing it for along time. It is necessary to fulfill spectrum sensing task in anonline manner and develop efficient algorithm to keep trackof the changes of Pi quickly.

At time slot t ∈ {1, 2, . . . , T}, an SU chooses SP i ∈ {I, II,III} and senses over the corresponding spectrum. The numberof licensed channels being idle obtained by the SU is denoted

as Ri(t)(t) at time slot t where i(t) is the index of the SPchosen at time slot t. According to the definition of Hi,c,we have Ri(t)(t) =

∑Cc=1 Hi(t),c, which is binomial(C, Pi).

The SU should dynamically choose the SPs so as to maximizethe expected number of idle channels obtained during T timeslots. Therefore, our optimization problem can be formulatedas follows:

maxi(t)∈{I, II, III}

E[T∑

t=1

Ri(t)(t)]. (1)

The optimization task illustrated in (1) requires that theSU should select a sequence of SPs in advance so as toobtain maximum cumulative number of idle channels. Thenumber of idle channels obtained at each time slot is related tothe unknown time-varying probability of spectrum being idle,which should be learned in an online manner. The optimizationproblem defined by (1) can be considered as a multi-armedbandit (MAB) problem [12], [13], where the SP to be selectedcan be regarded as an arm in the MAB problem. The numberof idle channels obtained of selecting SP i at time slot tfollows a certain distribution with unknown parameter Pi. Theobjective is to choose the optimal SP (pull the optimal arm)to sense over its assigned spectrum at time slot t to maximizethe expected number of idle channels (reward in the MABproblem) so that the SU has more opportunities to transmitdata during T time slots.

III. DISCOUNTED THOMPSON SAMPLING ALGORITHM

In the MAB problem, the decision maker needs to selectone of the optional arms to pull at each time slot to maximizethe expected value of reward. Thompson sampling (TS) is anefficient algorithm to deal with the MAB problem. The TShas access to the history of pulled arms and the reward inthe previous time slots and employs this history to select thepromising arm at current time slot [14], [15]. Comprehensivetheoretical analysis of regret upper bound and performanceguarantee of TS can be found in [16]. Since the general TScannot address our optimization problem with non-stationaryarms directly, we introduce a discounted Thompson samplingmethod.

Binary reward of 0 or 1 is defined for each arm in theclassical MAB problem while the reward in our case, i.e.,the number of idle channels obtained from SP i at time slott, is an integer distributed in a bounded interval randomly.We can scale the reward and do a Bernoulli trial with successprobability of the scaled reward [14], where ri is a binaryvariable of the observed output of Bernoulli trial. However,we cannot know the corresponding parameter of Bernoullidistribution related to Pi. Let ni represent the total time slotsthat SP i has been selected by now. Si denotes the total timeslots of ri = 1 and Fi is the total time slots of ri = 0. Thatis, Si + Fi = ni. Let Hi represent a vector consisting of allri’s observed so far when SP i is selected. The parametriclikelihood function of Hi is given by

pi(Hi|Pi) = PSi

i (1 − Pi)Fi , (2)

which corresponds to Binomial distribution. Assume that theprior probability of Pi obeys Beta distribution, the posterior

2404 IEEE COMMUNICATIONS LETTERS, VOL. 23, NO. 12, DECEMBER 2019

TABLE I

DISCOUNTED THOMPSON SAMPLING ALGORITHM

probability of Pi, denoted as pi(Pi|Hi), can be updated withpi(Hi|Pi), which is deduced in the appendix.

The SU selects an SP and performs spectrum sensing on thecorresponding channels at each time slot using Algorithm 1,where the hyperparameter γ controls the confidence level ofSi (or Fi) of each SP, to avoid being trapped into alwaysselecting an SP if Si of the SP is too large. Compared withthe Dynamic TS proposed in [11], our DTS method updatesSi (or Fi) of all SPs at each time slot, while the Dynamic TSonly updates the parameters of the selected SP. In addition, ourmethod applies the discount factor during all time slots, whilethe Dynamic TS uses it only if the condition Si + Fi > J issatisfied for the selected SP, where J is a threshold decidingwhether to introduce a discount factor or not.

IV. SIMULATION RESULTS

Consider 3 SPs licensed different radio spectrum whichconsists of C = 20 channels. Pi of each SP i follows standarduniform distribution, that is, Pi is sampled from uniformdistribution bounded in (0, 1). Each channel of SP i yieldsreward according to Bernoulli distribution with parameter Pi.To show the performance gain of our proposed algorithm,we compare our DTS algorithm with other representativeones: discounted ε-greedy, discounted upper confidence bound(Discounted UCB) [9] and dynamic TS [11]. For the UCBalgorithm, the SU chooses each SP once at first. Then atany time slot t > 3, the SU selects the SP that maximizesthe defined upper confidence index which is expressed asa simple function of the empirical distribution of reward.For the ε-greedy algorithm, the SU chooses an SP randomlywith probability ε, or chooses the SP that maximizes theaverage reward with probability 1 − ε. For our consideredscenarios, ε = 0.1. Compared with standard UCB and ε-greedyalgorithms, their discounted versions (Discounted UCB andDiscounted ε-greedy) introduce a discount factor to averagepast rewards in order to reduce the effect of the past observa-tions. For the Discounted ε-greedy algorithm, we employ thesame discounting method to handle the average reward as theDiscounted UCB [9].

Fig. 2. Normalized regret as a function of the number of time slots fordifferent settings of Pi.

Figure 2 illustrates the normalized regret as a function oftime slots for different settings of Pi. Pi remain unchangedin T = 500 time slots and the hyperparameter γ equals 1.The normalized regret is the ratio of the cumulative rewarddifference between the oracle decision in hindsight and thementioned algorithms over the cumulative reward of the oracledecision. All results are averaged by 100 Monte Carlo simula-tions. In setting 1, 2, 3, the prior Pi of each SP i ∈ {I, II, III}is {0.10, 0.28, 0.37}, {0.27, 0.24, 0.20} and {0.10, 0.80, 0.25},respectively. Always choosing the SP with the largest Pi forspectrum sensing all through 500 time slots is the oracledecision if the SU know Pi in advance, which always yieldsregret of 0. It can be seen from Fig. 2 that the normalized regretof DTS can reach a small value for all three different settingsof Pi, indicating that the total number of available channelsobtained by DTS is only slightly less than that of the oracledecision even if the difference in Pi among three SPs is small.Recall that the DTS dose not require the prior information ofPi, facilitating its applications in practical dynamic spectrumsystems.

Figure 3 shows the normalized throughput as a function oftime slots with fixed Pi for setting 1, where the normalizedthroughput is defined as the ratio of the total number of idlechannels obtained by the mentioned algorithms to that of theoracle decision in hindsight, that is, the SU could always selectSP III and the normalized throughput of the oracle decisionin hindsight is always 1. In Fig. 3, we can find that the totalnumber of idle channels produced by the four algorithms isclose to the oracle decision in hindsight provided that timeslots are large enough. Compared with the Discounted UCBand the Discounted ε-greedy methods, the DTS algorithmyields higher throughput since it can tend to select the SP withthe largest Pi quickly, bringing the SU more opportunities toget idle channels for data transmission.

Considering the time-varying Pi of SP i, we investigate thetotal number of idle channels as a function of time slots whenPi changes every 1000 time slots. The hyperparameter γ is setto 0.99 based on a series of numerical experiments. The prior

ZHOU et al.: SPECTRUM SENSING ACROSS MULTIPLE SPs: DTS METHOD 2405

Fig. 3. Normalized throughput as a function of the number of time slotswith fixed Pi.

TABLE II

TIME-VARYING Pi

Fig. 4. Total number of obtained idle channels as a function of the numberof time slots with time-varying Pi.

Pi is shown in Table II, where k represents the kth change ofPi. As can be seen in Fig. 4, the gap between the DTS andany scheme of always choosing a given SP becomes largerand larger as the increasing of time slots. The total number ofidle channels obtained by our proposed DTS algorithm is atleast 52% more than others, indicating that the DTS algorithmcan track the changes of Pi fairly quickly.

Fig. 5. Normalized throughput as a function of the number of time slotswith time-varying Pi.

Fig. 6. Normalized regret as a function of the number of time slots withtime-varying Pi.

Figure 5 shows the normalized throughput as a function oftime slots with time-varying Pi, which further indicates thatour proposed DTS can track the changes of Pi. Moreover,the total number of idle channels produced by the DTS iscloser to that of the oracle decision in hindsight comparedwith others. The reason is that the proposed DTS can find theSP with the largest Pi faster than the Discounted UCB, theDiscounted ε-greedy and the Dynamic TS without knowingthe prior knowledge about Pi.

Figure 6 shows the normalized regret as a function of timeslots with time-varying Pi. The normalized regret demon-strates the performance gap between the oracle decision inhindsight and the mentioned four algorithms, and can beregarded as the tracking error of different algorithms. In Fig. 6,it can be seen that the normalized regret of the four algorithmsreach a small and almost constant value provided that the timeslots are large enough even though Pi changes at temporalscale. Compared with the Discounted UCB, the Discounted

2406 IEEE COMMUNICATIONS LETTERS, VOL. 23, NO. 12, DECEMBER 2019

ε-greedy and the Dynamic TS, our proposed DTS algorithmproduces a lower and stabler average regret.

V. CONCLUSION

We investigated the channels selection problem across mul-tiple commercial mobile communication service providerswithout prior knowledge about the probability of chan-nels being idle. The optimization task is formulated as anonline learning problem and we developed an efficient dis-counted Thompson sampling algorithm to solve it. Numericalresults show that our proposed method can generate spectrumaccess opportunities almost as many as the oracle decisionin hindsight. The idle channels obtained by our proposedmethod are much more than the discounted upper confidencebound, the discounted ε-greedy and the dynamic Thompsonsampling methods. Moreover, our proposed algorithm canadapt to the changes of the probability of channels beingidle.

APPENDIX

Since Beta distribution is conjugate prior of Binomial dis-tribution and Pi obeys a prior Beta distribution, the posteriorprobability of Pi can be written as follows based on Bayesrule [17]:

pi(Pi|Hi) =pi(Hi|Pi)

Γ(αi + βi)Γ(αi)Γ(βi)

Pαi−1i (1 − Pi)βi−1

pi(Hi), (3)

where

Γ(αi) =∫ ∞

0

xαi−1e−x dx, (4)

and αi and βi are two parameters which determines the meanand variance of Beta distribution. Since no prior knowledgeabout Pi is available at the beginning, we initialize αi = βi =1 and Beta(1, 1) is actually uniform distribution. Substituting(2) in (3), pi(Pi|Hi) can be rewritten as

pi(Pi|Hi)=Γ(αi+βi)

Γ(αi)Γ(βi)pi(Hi)PSi+αi−1

i (1−Pi)Fi+βi−1. (5)

Let M = Γ(αi + βi)/[Γ(αi)Γ(βi)pi(Hi)], we have

pi(Pi|Hi) = MPSi+αi−1i (1 − Pi)Fi+βi−1. (6)

Notice that∫

pi(Pi|Hi) dPi = 1 and∫xαi−1(1 − x)βi−1 dx = Γ(αi)Γ(βi)/Γ(αi + βi), we have

pi(Pi|Hi)=Γ(αi+βi + ni)

Γ(αi+Si)Γ(βi+Fi)PSi+αi−1

i (1 − Pi)Fi+βi−1,

(7)

which also corresponds to Beta distribution with parametersof Si + αi and Fi + βi. Then the posterior probability of Pi

can be updated as follows:

pi(Pi|Hi) = Beta(Si + αi, Fi + βi). (8)

ACKNOWLEDGMENT

The authors would like to thank the editors and the anony-mous reviewers whose invaluable comments helped substan-tially improve the presentation of this letter.

REFERENCES

[1] S. Wang, Z.-H. Zhou, M. Ge, and C. Wang, “Resource allocationfor heterogeneous cognitive radio networks with imperfect spectrumsensing,” IEEE J. Sel. Areas Commun., vol. 31, no. 3, pp. 464–475,Mar. 2013.

[2] Y. Zhang and S. Wang, “Resource allocation for cognitive radio-enabledfemtocell networks with imperfect spectrum sensing and channel uncer-tainty,” IEEE Trans. Veh. Technol., vol. 65, no. 9, pp. 7719–7728,Sep. 2016.

[3] A. Ghasemi and E. S. Sousa, “Spectrum sensing in cognitive radio net-works: Requirements, challenges and design trade-offs,” IEEE Commun.Mag., vol. 46, no. 4, pp. 32–39, Apr. 2008.

[4] H. Jiang, L. Lai, R. Fan, and H. V. Poor, “Optimal selection of channelsensing order in cognitive radio,” IEEE Trans. Wireless Commun., vol. 8,no. 1, pp. 297–307, Jan. 2009.

[5] H. T. Cheng and W. Zhuang, “Simple channel sensing order in cog-nitive radio networks,” IEEE J. Sel. Areas Commun., vol. 29, no. 4,pp. 676–688, Apr. 2011.

[6] H. Li, “Multi-agent Q-learning of channel selection in multi-usercognitive radio systems: A two by two case,” in Proc. IEEE SMC,San Antonio, TX, USA, Oct. 2009, pp. 1893–1898.

[7] V. Raj, I. Dias, T. Tholeti, and S. Kalyani, “Spectrum access in cognitiveradio using a two-stage reinforcement learning approach,” IEEE J. Sel.Topics Signal Process., vol. 12, no. 1, pp. 20–34, Feb. 2018.

[8] R. Vishnu and S. Kalyani, “Taming non-stationary bandits:A Bayesian approach,” Jul. 2017, arXiv:1707.09727. [Online].Available: https://arxiv.org/abs/1707.09727

[9] A. Garivier and E. Moulines, “On upper-confidence bound policiesfor switching bandit problems,” in Proc. Int. Conf. Algorithmic Learn.Theory, Oct. 2011, pp. 174–188.

[10] O. Besbes, Y. Gur, and A. Zeevi, “Stochastic multi-armed-bandit prob-lem with non-stationary rewards,” in Proc. Adv. Neural Inf. Process.Syst., Dec. 2014, pp. 199–207.

[11] N. Gupta, O.-C. Granmo, and A. Agrawala, “Thompson samplingfor dynamic multi-armed bandits,” in Proc. ICMLA, Dec. 2011,pp. 484–489.

[12] S. Bubeck and N. Cesa-Bianchi, “Regret analysis of stochastic and non-stochastic multi-armed bandit problems,” Found. Trends Mach. Learn.,vol. 5, no. 1, pp. 1–122, Dec. 2012.

[13] Y. Lin, T. Wang, and S. Wang, “UAV-assisted emergency communi-cations: An extended multi-armed bandit perspective,” IEEE Commun.Lett., vol. 23, no. 5, pp. 938–941, May 2019.

[14] S. Agrawal and N. Goyal, “Analysis of Thompson sampling for themulti-armed bandit problem,” in Proc. COLT, Jun. 2012, pp. 1–26.

[15] O. Chapelle and L. Li, “An empirical evaluation of Thompson sampling,”in Proc. Adv. Neural Inf. Process. Syst., Dec. 2011, pp. 2249–2257.

[16] S. Agrawal and N. Goyal, “Further optimal regret bounds for Thompsonsampling,” in Proc. AISTATS, May 2013, pp. 99–107.

[17] D. J. Russo, B. Van Roy, A. Kazerouni, I. Osband, and Z. Wen,“A tutorial on Thompson sampling,” Found. Trends Mach. Learn.,vol. 11, no. 1, pp. 1–96, Jul. 2018.