Low Voltage Low Power

Neuromorphic Circuits Design

By

Syed Muffassir M.S.Ali4th Sem, M.Tech (II nd Part),

Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, India.

email:muffassir@gmail.com phone: +91-963728663

Guided by

Dr. Suhas. S. Gajre Asst.Prof,

Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, India.

email:ssgajre@sggs.ac.in

Abstract

This report outlines the need, requirements and the method of adopting the Low Voltage Low Power circuits for the Neuromorphs. Basic building structures of the future neuro-computer that will emulate the behavior and structure of the human brain are simulated in 180nm gpdk technology. A complete basic neuron circuit is designed by selecting the appropriate multiplier and activation function circuit for the synapse and activation stages respectively at very low voltage and low power. Another most important block i.e. Novel Winner take all circuit (WTA) is simulated and optimized for low voltage and low power. All these circuits had been designed using subthreshold mode of the MOS and using the translinear principle. These circuits work in current mode and when compared qualitatively, are better than that found in the literature.

1. Introduction Neuromorphic engineering is an interdisciplinary approach to the design of information processing electronic systems

that are inspired by function, structural organization, and physical foundations of biological neural systems. Neuromorphic is a term coined by Carver Mead [1]. It is also called as Bio-Mimetic Engineering. Neuromorphic engineering aims at systems that attain intelligent behaviour through adaptation and learning in their interaction with the environment, and that are more robust and orders of magnitude more energy efficient than conventional approaches using digital electronics and user-programmed intelligence.

Neuron is the basic structure in the brain and according to one estimate there are about 100 billion neurons in the human brain [2]. Neural networks in the brain are highly parallel and consume very less power. Biological information systems are compact, energy efficient, and excel at sensory perception and motor control-areas in which modern digital computers falter. For instant, Honeybee which has around 950000 neurons requires10 J/op, atleast 10 more efficient than digital silicon (20watts vs. 10 watts). What we need, is to exploit the physics of silicon to reproduce the bio-physics of neural systems. In analog VLSI implementation of the neural networks we identify the following goals [3]:

Low power consumption Low voltage operation High accuracy through overall massive parallelism Small size to accommodate huge number of neurons and interconnections

To achieve the above goals we have designed the basic building blocks of the neuromorphs using the translinear principle [4]: versatile & efficient; current mode of operation: robust to technological parameters, wide dynamic range and easy implementation of sums [5]; weak inversion region: low power and low voltage operation [5] and differentially coding the information: high noise and interference immunity.

In this project we have designed and simulated the circuit of neuron & novel winner take all; and optimized them for the low voltage and low power operation and utilized minimum number of transistors in order to occupy minimum area so as to make it feasible to design systems with thousands of such neurons a reality. For the neuron circuit design we require a four quadrant current multiplier and activation function circuit. Though many different approaches can be envisaged for implementation of the four quadrant multipliers [6], we have selected the Gravati multiplier [7] for the achieving the goals mentioned above. This circuit has been simulated in 180푛푚 for 0.7 V to operate in weak inversion. The circuit for the winner take all is implemented in strong inversion requiring high voltage and power consumption making it unsuitable for the Neuromorphs [8]. We have optimized and simulated it in weak inversion for low voltage and low power operation making it a novel implementation of winner take all circuit.

This report has been organized as follows. In section 2 a single neuron has been designed utilizing low voltage and low power. Four quadrant multiplier and activation function circuits are simulated and presented. In section 3, novel implementation in weak inversion of the winner take all circuit is presented and simulated. It has been compared with the analogous current mode WTA circuits found in literature. Finally in section 4, conclusions have been drawn.

2. Neuron Circuit Design

Neuron circuit block diagram, which is shown in Fig.1, is obtained by the combination of the building blocks. Briefly, input x is multiplied by its weight w and the sum of these products are applied to an activation function and the output y is obtained. The multiplication, which is mentioned in Fig.1a, is obtained in Fig.1b by using a multiplier with two inputs x and w. The summation and the activation function (Fig.1a) are realized by using sigmoidal circuit as shown in Fig.1b.

Fig.1: (a) Neuron Model (b) Neuron Circuit block diagram

Thus when implementing artificial neural networks in analog hardware, in particular synaptic circuits design is a very challenging task because large synaptic arrays are needed. Usually massive parallel systems are to be integrated on the same silicon die, and then the synaptic circuit power consumption strongly determines the overall chip power consumption: if N is the number of the network inputs and/or neurons, the number of synapses is roughly proportional to N2. Same considerations can be done for the size (e.g. silicon area). In particular, in the feedforward (i.e. recall) phase, the synaptic circuit is basically a four-quadrant analog multiplier. The sigmoidal circuit i.e. activation function circuit can be realized using the differential transconductance amplifier.

2.1. Synapse Circuit Design

Taking into consideration the design guidelines mentioned beforehand in section.1, we design the synapse circuit in the subthreshold mode of MOS by using the translinear principle. Expressing the channel current in MOS transistor in weak inversion [1] as

퐼 = 퐼 푒 [1 − 푒 ] (1)

Where 퐼 is specific current term, 푉 is the threshold voltage,휂 is the weak inversion slope factor. The synapse design implemented uses the translinear principle of operation [4]. Alternate topology of the translinear

loop that implements the translinear principle which is useful in our design of four quadrant multiplier is shown in the Fig.2. This circuit differentially encodes the information and uses current mode of operation.

Fig.2. Alternate Topology of Translinear Loop

After some mathematical calculations for the subthreshold MOS translinear loop [9] one can obtain for the Fig.2 as

1. 3 2. 4DS DS DS DSI I I I (2) .

Fig.3. Circuit Topology for the implementation of Synapse. Such four alternating translinear loops are required for the four quadrant multiplier implementation as shown in the

Fig.3. circuit topology for the synapse implementation. In the following we have considered the inputs ( and X WI I ) and output ( 퐼 ) signals as differential and balanced current mode signals (see Fig.3) as

(1 )2B

XII x (3)

(1 )2B

XII x (4)

(1 )2B

wII w (5)

(1 )2B

wII w (6)

Where and x w are the input information carrying variables ( 1 1, 1 1x w );

BI is the bias reference current;

& X XI I are the positive and negative input current components;

& W WI I are the positive and negative weight current components; The transistors M1, M2, M3, M4 when in weak inversion and saturation region from the translinear loop whose drain

currents are BI , wI , 1oI , XI respectively and can be written as

1. .X W o BI I I I

1X w

oB

I III

(7)

The transistors M7, M8, M11, M12 forms the translinear loop whose drain currents are BI , wI , XI , 1oI respectively and can be written as

2. .X W o BI I I I

2X w

oB

I III

(8)

The currents from 1 2 and o oI I (from (7) and (8)) are summed at node 2n ; the result is the positive single ended

term of the output current OUTI as

1 2OUT o oI I I

X W X WOUT

B

I I I III

(8)

In a similar way , the current term 3oI ( the result of the operation of the translinear loop made by transistors M1, M2,

M5, M6) is summed at node 1n to the current term 4oI (the result of the operation of the translinear loop made by

transistors M7, M8, M9, M10). The result is the negative single ended term of the output current OUTI as

3 4OUT o oI I I

X W X WOUT

B

I I I III

(9)

It has been verified through experimental measurements that the translinear loops are rather insensitive to the values of POLV to the larger extent. The final output current is given from (8) and (9) by 퐼 as

OUT OUT OUT BI I I xwI (10) The complete circuit is shown in the Fig. 4.

2.1.1. Simulation of Synapse Design The circuit shown in Fig. 4 had been simulated in 180nm gpdk technology in CADENCE. This circuit is supplied

with the 0.7V voltage. The transistor sizes were calculated by using the inversion coefficient IC=0.1. We refer the interested reader [10] on how to calculate them. The transistor sizes are reported in Table.1.

Table.1. Transistor sizes of MOS in Fig. 4.

Transistor

1 12M M 1 2p pM M

Size W( m ) 32.4 m 1 m Size L ( m ) 0.18 m 12 m

In the following measurement results, BI was set to 250nA, OUTI and OUTI vary in the range of [0nA to 250nA],

while OUTI varies in the range of [-250nA to 250nA]. Fig. 5a shows the DC measured characteristics of the multiplier. The x input is on the x-axis, and the w input is used as the parameter. Fig. 5b shows the DC measured transfer

characteristics of the multiplier in the case: w as input which is on the x-axis, and the x is used as the swept parameter. One can note that the multiplier exhibits linear behavior with respect to both the inputs.

Fig. 4. Overall circuit implementation of the four quadrant multiplier for the synapse circuit design.

Fig. 5 Measured DC Transfer Characteristics of the synapse circuit (a) when w input is used a parameter. (b) When x input is

used a parameter. In Fig.6 Transient analysis for the sinusoidal inputs for the designed four quadrant multiplier is done. The w input is

set to high frequency of 4KHz sinusoidal waveform with the peak value of160nA ; the x input was set to low sinusoidal frequency of 100Hz with the peak value of 40nA . The resulting modulated waveform is shown in the Fig.6 (b). The THD i.e. Total Harmonic Distortion is calculated for the 41ms transient analysis using the calculator thd function in the CADENCE. The calculated thd is found out to be 3.5% . The Power consumed is 0.58nW , which is very less.

Fig. 6 (a) x and w input waveforms for the transient analysis. (b) Output current waveforms when the two different sinusoids are

applied to the multiplier. Table.2 Compares this implementation with the subthreshold region circuits from paper [11], [12], [13], [7].It shows

that this is the best in terms of reduced voltage and power consumption.

Table.2. Comparison with other subthreshold multiplier circuits in terms of supply voltage and power.

[11] [12] [13] [7] This implementation

Process 0.35 m 2 m 0.35 m 0.35 m 180 nm Supply Voltage

3V 1.5V 1.5V 2V 0.7V

Power NA 3.1 W 6.7 W 5.5 W 0.58 nW

2.2. Activation Function Circuit Design

Activation function is the output stage of a neuron which chooses a value of its output interval according to its input and transmits it as an input for the synapses of other layer neurons. This activation function can be designed using the differential transconductance amplifier circuit as shown in Fig. 7.

Fig.7. MOS Differential Transconductance Amplifier for the implementation of Activation Function (Sigmoid Circuit)

The two output currents in the differential pair circuit can be subtracted from one another to form a single bidirectional output

current. The subtraction is performed by connecting a current mirror of the complementary transistor type to the differential pair, as shown in Fig. 7. As long as all MOSFETs stay in saturation and the differential pair is operated below threshold, the output current is given by

1 2

1 2

/ /

1 2 1 2/ / tanh ( )2

n T n T

n T n T

V U V Un

out b bV U V UT

e eI I I I I V Ve e U

(11)

The simulation result for the designed activation function is shown in the below figure Fig.8 for the supply voltage of 0.7 V in 180nm gpdk technology.

Fig.8. DC Voltage Characteristics of the activation function.

3. Design of Winner Take All Circuit

Winner-take-all is a computational principle applied in computational models of neural networks by which neurons in a layer compete with each others for activation (It selects the highest intensity signal among the multiple inputs). They are commonly used in computational models of the brain, particularly for distributed decision-making in the cortex. Important examples include hierarchical models of vision [14], and models of selective attention and recognition. They are also common in artificial neural networks and neuromorphic analog VLSI circuits. It has been formally proven that the winner-take-all operation is computationally powerful compared to other nonlinear operations, such as thresholding [15].

3.1. Current Mode WTA Circuits

Fig. 9. Current Mode Winner Take All Neural Network

In current mode approach to WTA neural network shown in Fig. 9 the thk output current winner selection is based on

criterion of maximum activation among all m neurons participating in a competition. Weights of the winning neuron with the largest OUTki are adjusted, while the weights of the others remain unaffected. As shown above the CM WTA implements the max function.

1,2.... 1max

n

OUTk ij INji m ji w i

(12)

3.2. Novel Implementation of WTA Circuit in Subthreshold MOS

The circuit proposed by D.Moro-Frias et.al [8] has been implemented here in the subthreshold region of the MOS transistors. The complete implemented circuit diagram has been shown in Fig. 10. It consists of n identical cells ( n =3 in Fig. 10), each with three transistors: 1iM , 2iM , 3iM and a DC bias current ( 1, 2.... )BiI i n . The cells are

connected together at the low-impedance common node cV to a DC sink current source named cI .The additional

transistor 3iM reduces the resistance seen at node cV through negative feedback. The winning output encodes the logarithm of the associated input.

In order to understand the operation principle, consider first the case in which all the current inputs are equal :

1 2 3in in inI I I I . In this case 2iM transistors sink bI each one, whereas 3iM transistors sink the same current

( 3 ) / 3c bI I .

When the input condition changes to 1inI I I and 2 3in inI I I sinks an extra current equal to I ,

incrementing the voltage at node 1V and therefore incrementing the voltage at the common node cV . Now 21M and 31M

must also sink I I but 2inI and 3inI are just I , so the drain voltages of these transistors decrease in order to

compensate for the increase in cV . For large values of I , 21M and 31M must leave saturation, driving 2V and 3V to

approximately 0V . As desired, the output associated with the smaller input diminishes. Now cI flows only through the

winner cell, so a current c bI I flows through 13M .

In order to get a copy of the winning current outM is connected to node cV . In this way, the gate to source voltage of

outM is set to the same gate to source voltage as the 1iM transistors and drains a current equal to the winning one.

Fig.10. Circuit Diagram of the CM WTA Circuit in the Subthreshold Mode of MOS transistors.

3.3. Simulation Results

The WTA circuit shown in the Fig.10 is designed in 180nm process technology. For any large scale system, resolution, supply and power consumption are the parameters used for characterization [16]. The power measured is 52.3

W . The dimensions and other parameters are listed in Table. 3. Table.3

Parameter Value

ddV 0.8V

biI 20nA

cI 80nA

L 2 m

1 2,Mi MiW W 625nm

3MiW 1.25 m

3.3.1. Transient Response Fig.11 shows the Transient analysis for the sinusoidal input currents of 20nA peak-peak at the frequencies of

1 ,2 and 5MHz MHz MHz for 1 2 3, and in in inI I I respectively. Since the circuit is Winner Take All, as expected the output current follows the envelope of the input currents.

Fig. 11.Transient Response of the WTA where outI follows the envelope of inputs.

3.3.2. Resolution Measurements For resolution measurements, the input currents for the first and third cell were 1 10inI nA and 2 1inI nA . The

input current for the third cell, 3inI , was incremented from 0 40nA .The DC response (Fig.12) explains the operation

of WTA. The resolution was measured at 20% of the 1V s final value and was found to be 600pA.

Fig.12. DC Response

3.3.3. Comparison The novel implemented WTA is compared with the similar WTA topologies found in literature. Performance wise

this implementation is superior to others as shown in Table. 4

Table.4. Performance comparison Parameter [17] [18] [19] [8] This

Implementation Input I I I I I

Output I I V I I Vdd 2.5V 1.2V 3.3V 2.5V 0.8V

Trans/Cell 3 3 ~12 3 3 Resolution 1.06 A 3.9 A - 1.55 A 0.6 nA

Power 203.3 W 133.9 W 87.5 W 281.7 W 52.3 W Technology 0.13 m 0.13 m 0.35 m 0.13 m 180 nm

4. Conclusion In this work we have successfully implemented the building blocks of the neuromorphs at low voltage and low

power. The novel neuron circuit was designed by selecting the multiplier and activation function circuits that work at 0.7V and low power consumed is in nW . Another important block Winner Take All is implemented in subthreshold MOS at 0.8V and power consumed is 52.3 W . Performance wise it is superior to others.

5. References

[1] C. Mead, Analog VLSI and Neural Systems. Reading, MA: Addison-Wesley Publishing Company, 1989. [2] DARPA Neural Network Study. Fairfax, VA: AFCEA Press, 1988.

[3] E. A. Vittoz, “Analog VLSI Signal Processing: Why, Where, and How?” Journal of VLSI Signal Processing, vol. 8, pp. 27–44, 1994. [4] B.Gilbert, “Translinear Circuits: A Proposed Classification,” Electronics Letters, vol. 11, no. 1, pp. 14–16, 1975. [5] Andreou, A.G.; Boahen, K.A.; Pouliquen, P.O.; Pavasovic, A.; Jenkins, R.E.; Strohbehn, K.; , "Current-mode subthreshold MOS circuits for analog VLSI neural systems," Neural Networks, IEEE Transactions on , vol.2, no.2, pp.205-213, Mar 1991. [6] G. Han and E. Sanchez-Sinencio, “CMOS Transconductance Multpliers: a Tutorial,” IEEE Trans. on Circuits and Systems II: Analog and Digital Signal Processing, vol. 45, no. 12, pp. 1550–1563, 1998. [7] M. Gravati, M. Valle, G. Ferri, N. Guerrini, L. Reyes , “A Novel Current-Mode Very Low Power Analog CMOS Four Quadrant Multiplier”, Proc. ESSCIRC,Grenoble,France,2005. [8] D.Moro-Frias, M.T.Sanz-Pascual, and c.A.de la Cruz Blas, “A Novel Current- Mode Winner-Take-All Topology,” European Conference on Circuit Theory and Design(ECCTD), 2011, no. 20, pp. 134–137. [9] A. G. Andreou, K. A. Boahen, A. Pavasovic, P. O. Pouliquen, R. E. Jenkins, and K. Strohbehn, “Current-Mode Subthreshold MOS Circuits for Analog VLSI Neural Systems,” IEEE Trans. Neural Netw, vol. 2, no. 2, pp. 205–213, 1991. [10] D. M.Binkley, Tradeoffs and Optimization in Analog CMOS Design. The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England: John Wiley & Sons Ltd, 2008. [11] Cyril Prasanna Raj P, S.L. Pinjare, “Design and Analog VLSI Implementation of Neural Network” EJSR.Vol.27 No.2 (2009), p.199-216.

[12] S. Liu, C. Chang, “CMOS subthreshold four quadrant multiplier based on unbalanced source coupled pairs” Int .J. Electronics, vol.78,No.2, pp 327-332,Feb.1995. [13] W. Liu, S. Liu, “Design of a CMOS low-power and low-voltage four-quadrant analog multiplier” AICSP, pp.307-312, Sept. 2009. [14] M. Riesenhuber and T. Poggio, “Hierarchical Models of Object Recognition in Cortex,” Nature Neuroscience, pp. 2–11, 1999. [15] W.Maass, “On the Computational Power of Winner-Take-All,” Neural Computation, 2000. [16] Z. S. Gnay and E. S. Sinencio, “CMOS Winner-Take-All Circtuis: A Detail Comparison,” Proceedings of 1997 IEEE international Symposium on Circuits and Systems ISCAS ’97, 1997,vol. 1, pp. 41–44. [17] B. Sekerkiran and U. Cilingiroglu, “Improving the Resolution of Lazzaro Winner-Take-All Circuit,” International Conference on Neural Networks, , 1997, vol. 2, pp. 1005–1008. [18] S. Hemati and A. H. Banihashemi, “A Current Mode Maximum Winner-Take- All Circuit with Low Voltage Requirement for Min-Sum Analog Iterative Decoders,” Proceedings of the 2003 10th IEEE International Conference on Electronics, Circuits and Systems, 2003. ICECS 2003, vol. 1, pp. 4–7. [19] A. Fish, V. Milrud, and O. Yadid-Pechit, “High Speed and High Precision Current Winner-Take All Circuit,” IEEE transactions on Circuits and Systems-II, vol. 52, no. 3, pp. 131–135, 2005.