Generalising to German Plural Noun Classes,from the Perspective of a Recurrent Neural Network

Verna Dankers? , Anna Langedijk? , Kate McCurdy ,Adina Williams , Dieuwke Hupkes

ILCC, University of Edinburgh ILLC, University of Amsterdam Facebook AI Research{vernadankers, annalangedijk}@gmail.com, kate.mccurdy@ed.ac.uk,

{adinawilliams, dieuwkehupkes}@fb.com

Abstract

Inflectional morphology has since long beena useful testing ground for broader questionsabout generalisation in language and the via-bility of neural network models as cognitivemodels of language. Here, in line with thattradition, we explore how recurrent neural net-works acquire the complex German plural sys-tem and reflect upon how their strategy com-pares to human generalisation and rule-basedmodels of this system. We perform analysesincluding behavioural experiments, diagnos-tic classification, representation analysis andcausal interventions, suggesting that the mod-els rely on features that are also key predictorsin rule-based models of German plurals. How-ever, the models also display shortcut learning,which is crucial to overcome in search of morecognitively plausible generalisation behaviour.

1 Introduction

Language is a complex and mysterious system,which requires that speakers systematically gener-alise but also that they admit exceptions (Jackend-off and Audring, 2018; Bybee and Hopper, 2001;Pinker, 1998). A clear illustration of this is the do-main of morphology, where suffixes and affixes canbe productively used to express a particular gram-matical property, but where there are also severalwords that follow irregular patterns for the samegrammatical function. For example, while the pasttense for most English verbs is formed by affixing-ed (walk → walked), the past tense of break isnot breaked but broke. If an English speaker en-counters an unknown form such as treak, they mustdecide whether to attach -ed, or instead go withthe irregular form troke. Precisely because of suchintricacies, the computational task of acquiring a

?Equal contribution.

<f> k a t z e </s>k a t z e n (-n)

<f> m u t t i </s>

<n> k l e i d </s>

<m> w a g e n </s>w a g e n (-∅)

k l e i d e r (-e r)

m u t t i s (-s)

<n> p f e r d </s>p f e r d e (-e)

encoder decoder

feminine

neuter

short lengthvowel

Figure 1: Illustration of how our models predict pluralnouns. Line thickness indicates performance per plu-ral class; colour gradients show how that performanceincreases while the encoder processes the word.

morphological system capable of generalisation hasa strong historical connection to cognitive science(e.g. Seidenberg and Plaut, 2014).

Making progress on how human minds processthis interesting subdomain of language, however, isa challenging enterprise: probing the internal repre-sentations of human minds is difficult and, in somecases, potentially unethical. Several researchershave therefore opted to instead investigate neuralmodels that show generalisation behaviour that issimilar to humans in key aspects and use them tolearn more about human language processing (fora prime example of this cycle, see Lakretz et al.,2019, 2021; Baroni, 2021). Using neural modelsin such a fashion starts with an accurate under-standing of how they approach the phenomena ofinterest: without that, we are constrained in ourattempts to use them to devise hypotheses abouthuman language processing and compare them toexisting theories (Hupkes, 2020; Baroni, 2021).

In that vein, we present a detailed examinationof how recurrent neural networks (RNNs) processthe complex German plural system which – con-trary to the classical example of English past tense– features generalisation of multiple classes (Mar-

cus et al., 1995; McCurdy et al., 2020; Belth et al.,2021). We ask what kind of representations suchmodels learn, whether they support human-like gen-eralisation, and we make a start with comparingtheir learnt solutions with existing models of Ger-man plural inflection. We train RNNs to predict theform of a German plural noun (or its plural class)from its singular form and grammatical gender andpresent an elaborate investigation of the resultingmodels. We perform behavioural analyses (§3)of the models’ predictions as well as structuralanalyses of their internal representations, aimed atidentifying the features that the models have learntto associate with each of the classes (§4). For thelatter, we use diagnostic classifiers (Hupkes et al.,2018) that we afterwards use to intervene in themodel to establish causal connections between theinternal encodings and the predictions the modelgenerates from these encodings (§5).

We find that our networks show a mixture ofcognitively plausible generalisation and relianceupon ‘shortcuts’ or heuristics (McCoy et al., 2019;Geirhos et al., 2020) (see Figure 1). On the neg-ative side, the model’s ability to cope with nounsin low-frequency plural classes is very brittle. Ourbehavioural analyses reveal that the models over-rely on length as a heuristic to predict the rare class-s. However, we also find that our models correctlylearn that key predictors of plural class includegrammatical gender and the last few letters of aword, which are the same features considered bythe recent decision-tree-based cognitive model ofBelth et al. (2021). Our diagnostic classifiers addi-tionally show how these predictors are encoded inthe model’s recurrent hidden representations. Thisinteresting overlap between neural and rule-basedmodels raises questions as to what neural modelsmight teach us about the cognitive implementationof rule-based domains.1

2 Methods

German plural inflection comprises multiple pluralclasses with different frequencies (Clahsen et al.,1992; Marcus et al., 1995; Clahsen, 1999; McCurdyet al., 2020; Zaretsky and Lange, 2016). Most plu-ral nouns are inflected with one of five suffixes(Clahsen et al., 1992): -(e)n, -e, -ø, -er and -s (Ta-ble 1 shows some examples). The suffixes -e, -er,

1The data and implementation are available here.

Class Example Frequency LengthSingular Plural

-(e)n Frau Frauen 44.7% 11.7-e Hund Hunde 26.3% 11.0-ø Wagen Wagen 16.9% 11.3-er Wald Wälder 3.5% 10.4-s Auto Autos 5.4% 8.0-? Cello Celli 3.2% 10.6

Table 1: The German plural system as represented inthe Wiktionary dataset with examples, along with in-flection class frequency and average (singular form)word length.

and -ø sometimes combine with umlaut.2 The lit-erature on German plural generalisation has mea-sured success rates based on correct suffixation (asopposed to a combination of the suffix and umlaut),and for simplicity, we keep this focus in the currentstudy (McCurdy et al., 2020; Belth et al., 2021).

2.1 Dataset

In our experiments, we use the Wiktionary dataset,3

which contains orthographic representations ofpairs of singular and plural forms of German nounsin the nominative case.4 Given a control token indi-cating grammatical gender and a singular form (asequence of discrete characters followed by a stoptoken, e.g. <f> f r a u </s>), the model is trainedto predict the plural form (f r a u e n). Nouns thatdid not have a gender listed were excluded. Thetraining, validation, and test splits consist of 46k,6.5k, and 6.6k instances, respectively. Masculine,feminine and neuter nouns appear 23k, 25k and 11ktimes, respectively. Table 1 indicates the frequencyof each of the plural classes, with the average lengthof the inputs per class. Notice that the nouns thattake the -s class are, on average, 8 characters long,while the overall average length is 11 characters.

To label plural forms predicted by the model,we consider whether the plural form has one ofthe five acceptable suffixes (-(e)n, -e, -ø, -er or -s)and whether the singular form appears in the pluralform (after removing umlauts). Predictions thatbelong to one of these five classes are consideredwell-formed. Otherwise, the input belongs to theunknown inflection class -?.

2How exactly this works is a theoretically open domain(Alexiadou and Müller, 2008; Müller, 2015; Trommer, 2020).

3The dataset is available here.4In German, the mapping from orthographic representa-

tions to phonological representations is comparatively straight-forward (Neef, 2004, 2011); which is why we use this ortho-graphic form for our investigations.

2.2 ModelWe study a recurrent encoder-decoder model im-plemented with the OpenNMT library (Klein et al.,2017). Modern RNNs trained to perform sequence-to-sequence tasks, including morphological inflec-tion, typically have a bidirectional encoder and anattention mechanism (Corkery et al., 2019; Mc-Curdy et al., 2020). While effective in terms oftask accuracy, that setup is further away from howhumans process the task at hand (incrementallyand in one go) than unidirectional models (see alsoHupkes, 2020; Christiansen and Chater, 2016; Ba-roni, 2021). Furthermore, in models with attention,there is no bottleneck between the encoder and de-coder that forces the model to create one localisedrepresentation of the input and its potential pluralclass. We, therefore, study unidirectional modelswithout attention. The encoder and decoder con-sist of two-layer unidirectional LSTMs, a hiddendimensionality of 128, character embeddings ofsize 128 and a dropout of 0.1 between layers. TheAdadelta (Zeiler, 2012) optimiser is used, with abatch size of 64. During evaluation, we apply beamsearch with a beam of size five. We train five mod-els with different random initialisations. All resultspresented are averaged over those models.

Rule-based comparison Belth et al. (2021) pro-pose a cognitive model of morphological learningwhich uses recursive application of the frequencythreshold defined by the Tolerance Principle (Yang,2016) to identify productive rules, resulting in a de-cision tree. The model checks at each node whetherto keep traversing the tree, apply a learnt rule, ormatch the input form to a stored exception.For Ger-man plural inflection, their model relies upon gram-matical gender and the last few characters of theinput word as features in the decision tree. We traintheir model on our dataset for comparison.5

3 Behavioural results

In Table 2, we summarise the models’ performanceafter 25 epochs. With 92.3%, our suffix accuraciesare competitive, despite the unidirectionality andremoval of attention. The RNNs outperform therule-based model of Belth et al. (2021) on unseendata.6 Figure 2a shows the RNNs’ training curveper plural class, with all classes undergoing rapid

5We make examples of these decision trees available here.A part of the model is visualised in Appendix E.

6The Belth et al. model does not handle stem changes suchas umlaut, which negatively impacts its full noun accuracy.

Model Measure Train Validation Test

Bidirec. & att. noun 97.4±.3 92.9±.2 93.0±.2noun[-1] 97.8±.3 93.9±.2 94.0±.1

Unidirectional noun 95.8±.5 87.8±.6 87.9±.5noun[-1] 97.4±.2 92.2±.2 92.3±.3

Belth et al. noun 99.9±0 78.8±0 78.2±0noun[-1] 99.9±0 89.2±0 89.0±0

Table 2: Accuracy (noun) and final letter accuracy(noun[-1]), with standard deviations over seeds.

10 20epochs

50

60

70

80

90

100

accu

racy

(%)

-(e)n-e-ø-er-s

(a) All classes

10 20epochs

50

60

70

80

90

100

accu

racy

(%)

-s, consonant-s, vowel

(b) -s

Figure 2: Training accuracy across epochs for the fiveplural classes, and for the -s nouns with stems endingin vowels and consonants, separately.

increases in performance during the first 5 epochsand less rapid but still substantial increases betweenepochs 5 and 10. Particularly notable is the curvefor samples from the -s class, which is learnt moreslowly than the other classes. Figure 2b detailsthe training curve for the -s class by separatinginputs ending in a consonant, and those ending in avowel, which suggests that mostly the inputs fromthe former class are learnt later during training.

Overgeneralisation Throughout training, sam-ples can be assigned suffixes different from their tar-get suffix. This rarely happens for majority classes(-(e)n, -e, -ø) (with the exception of -e, which, dur-ing one epoch, is predicted as -(e)n in 5% of thecases), but is more frequent for the rarer, minor-ity classes -er and -s. The models tend to gener-alise the suffixes of majority classes to the minorityclasses, a phenomenon traditionally referred to asovergeneralisation (Feldman, 2005). Maximumovergeneralisation typically occurs early on duringtraining, as shown in Figure 3 (c.f. Korrel et al.,2019; Hupkes et al., 2020; Dankers et al., 2021).

Wug testing Next, we apply the trained modelsto 24 nonce word stimuli from Marcus et al. (1995).Of these stimuli, 12 are rhymes – i.e. phonologi-cally familiar words rhyming with an existing word– and 12 non-rhymes – i.e. phonotactically atypi-cal words. We feed them to the network following

5 10 15 20 25epochs

0

20

40

60

80

100%

of s

ampl

es-?

-s

-e-ø

-(e)n

(a) -s

5 10 15 20 25epochs

0

20

40

60

80

100

% o

f sam

ples

-?

-er

-e

(b) -er

Figure 3: Cumulative distribution of the predicted suf-fix for training samples with -s and -er as plural class.

<m>,

r.

<f>,

r.

<n>,

r.

<m>,

n.r.

<f>,

n.r.

<n>,

n.r.

0

25

50

75

100

% o

f sam

ples

(a) Nonce predictions

regu

lar

see

haup

t

liebl

ings

0

25

50

75

100

% o

f sam

ples

-(e)n-e-ø-er-s-?

(b) -s compound predictions

Figure 4: Distribution of plural classes, (a) for noncewords with three gender tags (‘r.’ marks rhymes, and‘n.r.’ non-rhymes), or (b) for Wiktionary validation datafor -s, in the ‘regular’ format and as compounds withthe words indicated prepended to the singular noun.

each of the gender tokens; the distribution over plu-ral classes for the predictions of converged modelsis shown in Figure 4a. Similar to previous work(McCurdy et al., 2020, which only considered theneuter tag) non-rhymes fall more often into the un-known class (/?). Different from McCurdy et al.(2020) though, -s predictions are more frequentthan -er, and are also more frequent for non-rhymesthan for rhymes.7 Figure 4a also illustrates thatgender impacts the models’ predictions: the femi-nine tag seems related to -(e)n predictions, and theneuter tag to -er predictions.

Enforcing gender In wug testing, changing thegender changes the model’s predictions. Is this thecase for Wiktionary data as well? To find out, wecompare the model’s predictions for samples fromthe validation set with its predictions for the samesamples force-fed with new gender control tokens.Figure 5 visualises the corresponding results. In-troducing the feminine control token has the mostprominent effect: the vast majority of model predic-tions changes to -(e)n. Providing the masculine or

7We trained 95 further model instances with the samesetup, to approach the sample size of speakers tested by Mc-Curdy et al. (2020), and found that the pattern of increased-s production on non-rhymes is not statistically reliable.

-(e)n -e -ø -er -s0

20

40

60

% o

f sam

ples

pred.<f> pred.

-(e)n -e -ø -er -s

pred.<m> pred.

-(e)n -e -ø -er -s

pred.<n> pred.

Figure 5: Predicted plural classes change when newgender control tokens (<f>, <m>, <n>) are enforcedfor nouns from the Wiktionary validation data that nor-mally would not have that grammatical gender.

neuter one reduces the amount of -(e)n predictions,increasing -e, -ø and -s predictions. The pluralclass -er only appears associated with the neutergrammatical gender. Taken together, the impactof gender on both wug data and Wiktionary datasuggests the model has learnt to strongly rely onthe gender markers.

Enforcing length The high frequency of -s pre-dictions for nonce words is remarkable, given therelative rarity of the -s plural. We observe, however,that the nonce words overall are rather short (4.6characters), and that the nouns from the -s classare the shortest in Wiktionary (see Table 1). Toinvestigate whether the model has learnt a causalconnection between input length and emitting -s,we sample an equal number of nouns from theWiktionary validation set of each gender that arebalanced for whether their singular form ends in avowel or a consonant. We then lengthen them byprepending nouns of three lengths (“See”, “Haupt”or “Lieblings”) to form compounds, which simu-lates a length increase without altering the target’splural class (which is generally determined by thesecond noun in a compound in German). Our re-sults confirm that the models emit -s less often forlonger inputs (see Figure 4b), suggesting that theyrely on length as a shortcut for predicting -s.8

4 Diagnostic classification

We now look into how and where the plural classesare encoded by the model. To do so, we use di-agnostic, or probing classifiers (DCs, Adi et al.,2017; Belinkov et al., 2017; Hupkes et al., 2018;Conneau et al., 2018), commonly used to estimatethe extent to which hidden representations of a

8Since copying the stem of longer words might be difficult,we focus on suffix accuracy only in Figure 4b. Appendix Adiscusses a related experiment for nonce words.

Class Gender n final letters Bothn = 1 n = 2 n = 1 n = 2

-(e)n 90.3 76.1 87.2 93.4 95.6-e 59.8 55.8 74.9 73.7 87.3-ø 0.0 67.4 88.2 79.0 92.5-er 0.0 0.0 41.1 55.6 78.6-s 0.0 39.0 49.0 41.2 55.8

Macro F1 30.0 47.7 68.1 68.6 82.0

Table 3: Performance (F1) of the plural class of themodels’ outputs for the validation set, for several ma-jority baselines, conditioned on 1) gender tag, 2) finalletter(s) of the singular form, or 3) both.

neural model reflect a specific linguistic property.DCs are simple classifiers that are trained to predictthat property from the representation. The DC’sperformance on new data is assumed indicative ofwhether the linguistic property was, in fact, en-coded.9 Our experiments with DCs target: i) howwell different hidden representations encode pluralclasses; ii) how the DC performance evolves overtime while processing an input; iii) what is specialabout the hidden representations’ neurons that aremost salient to the DC.

4.1 Setup

At the end of a training epoch, we extract themodel’s representations for a subset of the trainingdata and the validation data. The training data sub-set contains an equal number of samples for eachplural class. We record the hidden state ~hl,et , mem-ory cell state ~c l,e

t and the activations for the input,forget and output gates ~i l,et , ~f l,e

t and ~o l,et , from

the encoder layers l ∈ {1, 2}, with 1 ≤ t ≤ m; mbeing the length of the input. The hidden state andmemory cell state from the ultimate time step ofthe encoder form the initialisation of the decodingLSTM. In the absence of an attention mechanism,these representations form the information bottle-neck between the encoder and the decoder.

We train DCs to predict the plural class a modelwill assign to an input from intermediate time steps.If the to be predicted suffix can accurately be in-ferred by the DCs, and this generalises to unseen

9This argument has been problematised (e.g. Voita andTitov, 2020; Pimentel et al., 2020b; Sinha et al., 2021): the DCmay learn the task instead of extracting information. To assessthe extent of this problem for our case, we ran linguisticallymeaningless control experiments (Hewitt and Liang, 2019)and show that our DCs can reach up to 41% macro-averagedF1-scores there. The setup and results are listed in Appendix B.Furthermore, our experiments with interventions (§5) causallylink our DC experiments to the model’s behaviour.

examples, that strengthens the hypothesis that thesuffixes are distinctly encoded in the hidden repre-sentations of the model. The targets used to trainthe DCs are the plural class of a model’s predic-tion, rather than the true target class. We only trainand evaluate DCs on well-formed model predic-tions, which means the amount of samples availablefor training and evaluation changes across epochs.Training lasts for 50 epochs, with a batch size of16, a learning rate of .00025 and Adam as opti-miser. We train five DCs per model and evaluatethe DCs using the Wiktionary validation data, withF1-scores per plural class and the macro averageacross classes.

We compare with rule-based baselines, wherethe plural class is estimated from the grammaticalgender or the final characters of singular nouns.The F1-scores are provided in Table 3. Less well-informed baselines predicting one class only, orpredicting at random according to the frequenciesof the different classes, obtain macro-averaged F1-scores of 12.5 and 19.6, respectively.

4.2 DC results

We first consider the difference between differentmodel components (i.e. hidden states, gates) andprocessing steps. For every input token, we con-sider the first and the last three time steps. At timestep 1, the model processes the gender tag, fol-lowed by the first and second character of the nounin time steps 2 and 3. Time steps -3 and -2 corre-spond to the last two characters of the noun; timestep -1 is the end-of-sequence (EOS) token. Wetrain separate DCs for every time step, using repre-sentations from the 25th (i.e., final) epoch. Figure 6reports the F1-scores of the DCs, per plural class;the macro-averaged F1-score is shown in black.

In Figure 6a, we visualise results for DCs trainedon the concatenated hidden and memory cell states,for multiple time steps in the encoder. Figure 6bsummarises the performance in the last time stepfor the remaining model components (full figuresare in Appendix B). The graphs show that DCstrained on hidden and memory cell states consis-tently outperform DCs trained on gates, and that thememory cell state alone captures nearly the sameamount of information as the concatenated hiddenand memory cell states. For all components, per-formance increases when the model has processedmore characters. A remarkable exception to thisis the -(e)n class, for which performance immedi-

1 2 3 -3 -2 -1time steps

0

20

40

60

80

100(m

acro

) F1

-(e)n-e-ø-er-sall

(a) [~h1,et ;~h2,e

t ;~c 1,et ;~c 2,e

t ]

[h1, e

t; h

2, et

]

[c1, et

; c2, et

]

[i1, et

; i2, et

]

[f1, et

; f2, et

]

[o1, et

; o2, et

]

model components

0

20

40

60

80

100

(mac

ro) F

1

(b) Remaining components

Figure 6: Performance for DCs trained and evaluatedwith data per time step, separately. Negative time stepsare relative to the position of the EOS token (position-1). In (a), performance is shown for the concatenationof the hidden and memory cell states. (b) shows thefinal time step for the remaining model components.

1 2 3 -3 -2 -1test time step

12

3-3

-2-1

train

tim

e st

ep

(a) All classes

1 2 3 -3 -2 -1test time step

12

3-3

-2-1

train

tim

e st

ep

0

20

40

60

80

100

F 1

(b) -(e)n

Figure 7: DC F1-score when training on representa-tions from one time step, and testing on representationsfrom another, (a) averaged over all plural classes, and(b) shown for -(e)n only.

ately peaks at the first time step. Considering thebehavioural analyses indicating a large impact ofthe feminine gender tag on this class, the DC mayhave learnt that this is a strong predictor for thatcategory. The gender tag is fed during the first timestep, and may be encoded still towards the end.

Focusing on the DC performance for the concate-nations of the hidden and memory cell states, theseF1-scores are either similar to the highest baselineperformance in Table 3 (for -(e)n, -ø, -e) or evensub-par compared to those scores (for -s and -er).Taken together with the impact of gender observedfor -(e)n, and the fact that the scores increase as theword is being processed, this suggests that the lastfew letters and the grammatical gender are essentialfeatures in predicting the plural class.

Generalisability across time steps FollowingGiulianelli et al. (2018), we now test how wellDCs generalise across time steps to get an indica-tion of when consistent representations of the pluralclasses are formed. Considering again epoch 25,

we test our DCs trained on the concatenation of thehidden and memory cell states from one time stepand evaluate on another, for time steps 1, 2, 3, -3,-2 and -1. We show the results in Figure 7, wherethe diagonal corresponds to results in Figure 6a,and the off-diagonal entries represent generalisa-tion across time steps. Classifiers trained on earlytime steps do not generalise to representations closeto the end. This is unsurprising, considering thatfeatures learnt by early DCs cannot be based on thenoun, since it has not been processed yet. Giventhe average input length of 11, time steps 3 and -3will typically be far apart, which is why time step3 need not generalise to time step -3 for the major-ity of the inputs. Yet, generalisation is not goodeven among early or late time steps, with the ex-ception of the -(e)n class (see Figure 7b), for whichthere are blocks visible in the upper left and bottomright corners, suggesting that the DC relies on thesame feature in multiple time steps. The absence ofblocks in the lower left and upper right corner im-plies that that feature is differently encoded at thebeginning than at end of processing – e.g. becausethe hidden representations store more informationlater on. For the remaining classes, even the lasttwo time steps do not generalise perfectly to oneanother, which either means that the plural classis not decided until the end or that the decision isencoded in multiple ways, with the DCs in differenttime steps picking up on different features.

Performance over epochs Figure 8 shows theF1-scores for DCs trained on the hidden andmemory cell states for different training epochs.Because the number of well-formed predictionschanges over the course of training, the size of thedataset available for training DCs increases overtime. Nonetheless, the DCs’ performance on evalu-ation data remains stable, or even slightly decreasesover time. A potential cause could be that inputs forwhich the model learns to emit a class after the ini-tial epochs are atypical nouns for which the modelmemorises a suffix to emit, but whose features donot generalise towards new inputs.

4.3 Dissecting the representation space

To better understand the features that the DCs relyon, we train sparse DCs by applying L0 regular-isation to the DCs’ parameters, that reduces thenumber of non-zero weights in the classifier. TheseDCs are trained on representations from epoch five,when performance of the DC peaked in Figure 8.

3 4 5 10 15 20 25epochs

40

60

80

100

(mac

ro) F

1

-(e)n-e-ø-er-sall

Figure 8: DCs trained on the concatenated hidden andmemory cell states for seven training epochs.

The L0-norm is not differentiable and cannot sim-ply be added to the training objective as a regulari-sation term, but is incorporated through a mask thatis multiplied with the weights of the linear layer,where a collection of non-negative stochastic gatesdetermine which weights are set to zero. We referthe reader to Louizos et al. (2018) for a detailedexplanation of this regularisation technique. Thesparse DCs are trained for 50 epochs, with a learn-ing rate of .001; the L0 component in the loss isweighted by a hyperparameter λ = .005.

We use sparse DCs with, approximately, 95%of the hidden dimensions excluded, to visualisethe non-zero weighted dimensions per output class,using t-SNE projections. We only include dimen-sions that are not pruned by five DCs trained withdifferent random seeds.10 We then visually inspectthe representations by considering features such asthe RNN’s predicted plural class, the class the DCpredicts, the grammatical gender, the singular nounlength, and the last few letters of a word. We havefour main observations: (1) The gender tags aregrouped for -(e)n and -er – e.g. see Figure 9b for-er. This corresponds to the fact that in Figure 5,masculine and neuter proved predictive of -(e)nand -er, respectively. Furthermore, the femininetag is grouped for -e and -ø, but the three tags arescattered for -s, as shown in Figure 9a. (2) For allclasses, the length is an important organisationalfeauture in the representation space, but for allclasses except -s, the DC still predicts that class fornearly all input lengths – e.g. compare Figures 9cand 9d. (3) The final letter of the singular noun isa prominent organisational feature too; there areclusters of ‘e’, ‘t’ and ‘r’, in particular, that areamong the top five most frequent final letters ofnouns in Wiktionary (see, for example, Figure 9e).Vowels other than ‘e’ are typically scattered acrossthe representation space, except for -s, for which

10We inspect the visualisations separately for five models,and include findings that hold for multiple models.

40 20 0 20 40

40

20

0

20

40<m><n><f>

(a) -s, gender

50 25 0 25 5040

20

0

20

40 <n><f><m>

(b) -er, gender

40 20 0 20 40

40

20

0

20

4048121620

(c) -s, length

50 0 50

40

20

0

20

40

36912151821

(d) -(e)n, length

50 25 0 25 50

40

20

0

20

40 egtrn

(e) -ø, last letter

40 20 0 20 4040

20

0

20

40

aeoiyu

(f) -s, vowels

Figure 9: T-SNE visualisations of hidden and memorycell states. The dimensions t-SNE uses vary per figureand are those most relevant to the plural class in thecaption. Colour schemes show (a, b) gender tags, (c,d) lengths of singular nouns, (e) the most frequent lastletters of singular nouns, (f) vowels occurring as thelast letter. Grey approximately marks the area in whichthe DC predictions match the plural class in the caption.

they cluster (Figure 9f). (4) Lastly, while all pre-dicted classes cluster together when we select thedimensions from the sparse DC for that class, -ecannot easily be localised, potentially due to thefact that two other suffixes can involve adding an‘e’ to the singular noun (i.e. -(e)n and -er).

5 Interventions

Until now, we have only been able to relate DCfeatures to models’ behaviour by making adapta-tions in the inputs fed to the model. To strengthenthese results, we now ask: can we also change themodels’ behaviour without changing the input, bychanging the input’s hidden representation? To doso, we use interventions (Giulianelli et al., 2018)that halt the model while it processes inputs andchange its representations using the DC. We moni-

-(e)n -e -ø -er -s ?prediction after intervention

-(e)n

-e

-ø

-er

-s

targ

et43 0.83 0 0.230.23 13

0.32 21 0.63 0 0.53 5.8

0.32 4.9 18 0.54 1.7 12

0 0.660.49 25 0.82 12

1.8 2.2 0.620.26 8.7 150

10

20

30

40

%

(a) Changed predictions

-(e)n -e -ø -er -sunchanged predictions

-(e)n

-e

-ø

-er

-s

targ

et

0 32 4.8 0 5.9

47 0 12 4.6 8.2

14 30 0 1.1 17

0.41 60 0 0 0.98

22 38 10 0.93 00

20

40

%

(b) Unchanged predictions

Figure 10: The results of interventions, showing targetclass distribution for (a) interventions that changed thesuffix of the prediction, and (b) those that did not.

tor the effect to establish a causal link between theDC’s results and the model’s predictions.

Setup Following Giulianelli et al. (2018), the DCfindings are linked to models’ behaviour by adapt-ing the hidden representations through the signalprovided by DCs, while monitoring the impact onthe models’ predictions. We perform interventionson the hidden and memory cell state from the finalencoder time step, by running the RNN, halting itafter the encoder processed the input and interven-ing on the decoder’s initialisation before it predictsan output. Assuming~h is the hidden representation,we use the DC as follows: ~h← ~h− α∇~h

LDC(~h).We perform interventions with respect to the trueplural classes of samples from the validation set forwhich the prediction is well-formed but not correct,with α empirically set to 2.0.11

Results Figure 10a summarises the impact ofinterventions during the fifth epoch. For well-formed model predictions that have been assignedthe wrong plural class, we can change the model’sprediction to the right class in up to 43% of thesamples with target -(e)n confirming that the infor-mation detected by DCs is partially also used bythe model. For -e, -ø and -er, a smaller, yet stillsubstantial percentage can be corrected (18-25%).However, this comes at a cost; some previouslywell-formed predictions are no longer well-formed(-? in the figure). In most of these cases, though,it is not the plural class that was corrupted, butthe rest of the noun – i.e. the intervention some-times negatively impacts the decoder’s ability torecover all of the noun’s characters. That the DChas picked up on class-specific features can be de-duced from the fact that the interventions eitherchange the class to the correct one, or make thepredictions less well-formed, but hardly ever cause

11The success of interventions depends on α. Figure 14a inAppendix C illustrates how.

the model to emit a different incorrect plural class.In many cases, intervening does not lead to any

changes in the models’ predictions, as shown in Fig-ure 10b. Predictions belonging to the plural class -eare typically immune to interventions, which may,again, be due to the fact that two other classes (i.e.-(e)n and -er) contain ‘e’ as part of their suffix.

6 Related work

Our analysis draws upon current research inves-tigating neural model representations. We applythese techniques to German plural generalisation,a challenging domain with an extensive cognitiveand linguistic literature.

Morphological inflection in neural networksRecently, others have explored the potential lin-guistic and cognitive implications of morphologicalgeneralisation in neural networks. Malouf (2017)visualised the representation space learnt by RNNsto draw connections with more traditional linguis-tic categories. King et al. (2020) and Gorman et al.(2019) grouped sequence-to-sequence model errorsinto linguistically meaningful categories. Neuralmodels have been used to estimate the informa-tion theoretic contribution of meaning to gender(Williams et al., 2019) and of meaning and formto gender and declension class (Williams et al.,2020). McCarthy et al. (2020) used grammaticalgender classes to track phylogenetic relationshipsbetween related languages, while others used themto model morphological learnability (Elsner et al.,2019; Cotterell et al., 2019; Forster et al., 2021).

Probing has been used to investigate how lin-guistic information is encoded in neural modelrepresentations (Alain and Bengio, 2017; Hupkeset al., 2018; Goodwin et al., 2020; Ravichanderet al., 2020), including morphological structure(Torroba Hennigen et al., 2020). Much recentdebate has focused on appropriate methods forprobing (Belinkov, 2021; Hewitt and Liang, 2019;Hall Maudslay et al., 2020; Pimentel et al., 2020a;Ravichander et al., 2021; White et al., 2021). Ourwork applies probing to German plural inflectionand bolsters it using causal interventions.

German plurals and the past tense debate Ina wider context, our work fits within the (in)famouspast tense debate, one of the longest and mostvigorous conflicts in cognitive science (e.g. Sei-denberg and Plaut, 2014), which contrasted neuralnetwork models of English past tense inflection

(Rumelhart and McClelland, 1986) against theo-ries of generalisation which emphasised a need forsymbolic rules (Pinker and Prince, 1988).

German plurals have been an important phe-nomenon for this debate. Dual-route theorists ar-gued that German speakers show rule-based gener-alisation for one plural class – the numerically rare-s class – and analogical generalisation of the otherclasses (Clahsen et al., 1992; Marcus et al., 1995;Clahsen, 1999). This account has been contestedby schema theories of German plural generalisa-tion (Köpcke, 1988; Bybee, 1995). Later exper-iments also cast doubt on the dual-route accountof speaker preference for -s (Hahn and Nakisa,2000; Zaretsky and Lange, 2016; McCurdy et al.,2020), and recent rule-based models of Germanplural inflection model all plural classes with aunified approach (Yang, 2016; Belth et al., 2021).The speaker preference for -s on unusual inputs,such as the non-rhyme words developed by Marcuset al. (1995), has been claimed as a key signature ofhuman-like generalisation in contrast to neural net-work models (Clahsen, 1999). Similar to Goebeland Indefrey (2000), our neural model shows thisbehaviour, although one should be careful in inter-preting this given the length shortcut observed for-s. The question of how rules might be representedneurally is still open to debate and investigation.Our work continues to weaken the original empir-ical objections to connectionist models (see alsoKirov and Cotterell 2018).

7 Discussion & Conclusion

For more than 30 years, the field of inflectionalmorphology has been a testing ground for broaderquestions about generalisation in language, cen-tred around the extent to which explicit rules arerequired. In this discussion, neural networks aretraditionally considered as an alternative to the ex-plicit representation of rules. However, recent stud-ies have shown that such models show interestinggeneralisation patterns – sometimes comparableto behaviour observed in humans (Corkery et al.,2019; Kirov and Cotterell, 2018). This raises thequestion of what kind of solution is implemented byneural networks to process language in seeminglyrule-governed domains, how these solutions relateto rule-based models, and what it teaches us abouthuman processing of inflectional morphology. Ourstudy takes a step in this direction by exploringhow an RNN encodes generalisation behaviour.

We find that an RNN shows a mixture of human-like generalisation and reliance upon ‘shortcuts’.The models correctly learn that key predictors ofplural class include grammatical gender and thelast few letters of a word, which are the same fea-tures used by the recent rule-based cognitive modelof Belth et al. (2021). Our DCs show how thesepredictors are largely encoded in the hidden rep-resentations of the encoder. Variation in the clas-sifiers’ performance may reflect that some pluralclasses are encoded more consistently than others;for instance, feminine gender is highly predictiveof the -(e)n class. Alternatively, the decoder maydecide the plural class for some inputs.

On the other hand, the models’ ability to copewith nouns in low-frequency plural classes is verybrittle. The DCs perform worst for the minorityclasses, it proved hard to change the model’s pre-dictions to -s in the interventions, and behaviouralanalyses suggested the model overly relies onlength as a shortcut to predict this class. By con-trast, we see model bias for the frequent class -e inovergeneralisation behaviour (§3), in representa-tion space dispersion (§4), and in resistance to in-terventions (§5). We speculate that the character-based RNN may conflate the -e class with the ‘e’character that appears in the -(e)n and -er classes.

In summary, we contribute a detailed analysis ofhow an RNN processes the complex task of pluralinflection in German. Interestingly, we find cogni-tively plausible generalisation behaviour throughlearnt representations which echo recent rule-basedmodels. Future work could address the broaderquestions raised by these findings, such as whatconstitutes a rule given overlap in strategy betweenneural and rule-based models, and how a mechanis-tic understanding of how neural networks approachseemingly rule-governed domains might contributeto understanding how such generalisation is instan-tiated in the human brain.

Acknowledgements

We thank Elia Bruni and Ryan Cotterell for theirfeedback on this paper. KM is supported by the EP-SRC Centre for Doctoral Training in Data Science,funded by the UK Engineering and Physical Sci-ences Research Council (grant EP/L016427/1) andthe University of Edinburgh. VD is supported bythe UKRI Centre for Doctoral Training in NaturalLanguage Processing, funded by the UKRI (grantEP/S022481/1) and the University of Edinburgh.

ReferencesYossi Adi, Einat Kermany, Yonatan Belinkov, Ofer

Lavi, and Y. Goldberg. 2017. Fine-grained analysisof sentence embeddings using auxiliary predictiontasks. In Proceedings of the 5th International Con-ference on Learning Representations (ICLR).

Guillaume Alain and Yoshua Bengio. 2017. Under-standing intermediate layers using linear classifierprobes. In ICLR 2017, Workshop Track Proceed-ings.

Artemis Alexiadou and Gereon Müller. 2008. Classfeatures as probes. In Asaf Bachrach and AndrewNevins, editors, Inflectional Identity, volume 18of Oxford Studies in Theoretical Linguistics, pages101–155. Oxford University Press, Oxford.

Marco Baroni. 2021. On the gap between theo-retical and computational linguistics. Keynote atEACL2021.

Yonatan Belinkov. 2021. Probing classifiers:Promises, shortcomings, and alternatives. CoRR,abs/2102.12452.

Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Has-san Sajjad, and James Glass. 2017. What do neu-ral machine translation models learn about morphol-ogy? In Proceedings of the 55th Annual Meeting ofthe Association for Computational Linguistics (ACL)(Volume 1: Long Papers), pages 861–872.

Caleb Belth, Sarah Payne, Deniz Beser, Jordan Kodner,and Charles Yang. 2021. The Greedy and RecursiveSearch for Morphological Productivity. In CogSci.

Joan Bybee. 1995. Regular morphology and the lexi-con. Language and Cognitive Processes, 10(5):425–455.

Joan L Bybee and Paul J Hopper. 2001. Frequencyand the emergence of linguistic structure, volume 45.John Benjamins Publishing.

Morten H Christiansen and Nick Chater. 2016. Thenow-or-never bottleneck: A fundamental constrainton language. Behavioral and brain sciences, 39.

Harald Clahsen. 1999. Lexical entries and rules oflanguage: A multidisciplinary study of German in-flection. Behavioral and Brain Sciences, 22(6):991–1013.

Harald Clahsen, Monika Rothweiler, Andreas Woest,and Gary F. Marcus. 1992. Regular and irregularinflection in the acquisition of German noun plurals.Cognition, 45(3):225–255.

Alexis Conneau, Germán Kruszewski, Guillaume Lam-ple, Loïc Barrault, and Marco Baroni. 2018. Whatyou can cram into a single $&!#* vector: Probingsentence embeddings for linguistic properties. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (ACL) (Vol-ume 1: Long Papers), pages 2126–2136.

Maria Corkery, Yevgen Matusevych, and Sharon Gold-water. 2019. Are we there yet? Encoder-decoderneural networks as cognitive models of English pasttense inflection. In Proceedings of the 57th Confer-ence of the Association for Computational Linguis-tics, pages 3868–3877, Florence, Italy. Associationfor Computational Linguistics.

Ryan Cotterell, Christo Kirov, Mans Hulden, and Ja-son Eisner. 2019. On the complexity and typol-ogy of inflectional morphological systems. Transac-tions of the Association for Computational Linguis-tics (TACL), 7:327–342.

Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2021.The paradox of the compositionality of natural lan-guage: a neural machine translation case study.CoRR, abs/2108.05885.

Micha Elsner, Andrea D. Sims, Alexander Erd-mann, Antonio Hernandez, Evan Jaffe, Lifeng Jin,Martha Booker Johnson, Shuan Karim, David L.King, Luana Lamberti Nunes, Byung-Doh Oh,Nathan Rasmussen, Cory Shain, Stephanie An-tetomaso, Kendra V. Dickinson, Noah Diewald,Michelle McKenzie, and Symon Stevens-Guille.2019. Modeling morphological learning, typol-ogy, and change: What can the neural sequence-to-sequence framework contribute? Journal of Lan-guage Modelling, 7(1):53.

Naomi Feldman. 2005. Learning and overgeneraliza-tion patterns in a connectionist model of the Germanplural. Master’s thesis, University of Vienna.

Martina Forster, Clara Meister, and Ryan Cotterell.2021. Searching for search errors in neural mor-phological inflection. In Proceedings of the 16thConference of the European Chapter of the Associ-ation for Computational Linguistics (EACL): MainVolume, pages 1388–1394, Online. Association forComputational Linguistics.

Robert Geirhos, Jörn-Henrik Jacobsen, ClaudioMichaelis, Richard Zemel, Wieland Brendel,Matthias Bethge, and Felix A Wichmann. 2020.Shortcut learning in deep neural networks. NatureMachine Intelligence, 2(11):665–673.

Mario Giulianelli, Jack Harding, Florian Mohnert,Dieuwke Hupkes, and Willem Zuidema. 2018. Un-der the hood: Using diagnostic classifiers to in-vestigate and improve how language models trackagreement information. In Proceedings of the 2018EMNLP Workshop BlackboxNLP: Analyzing and In-terpreting Neural Networks for NLP, pages 240–248.

Rainer Goebel and Peter Indefrey. 2000. A recurrentnetwork with short-term memory capacity learningthe German-s plural. Models of language acqui-sition: Inductive and deductive approaches, pages177–200.

Emily Goodwin, Koustuv Sinha, and Timothy J.O’Donnell. 2020. Probing linguistic systematicity.In Proceedings of the 58th Annual Meeting of theAssociation for Computational Linguistics (ACL),pages 1958–1969.

Kyle Gorman, Arya D McCarthy, Ryan Cotterell, Eka-terina Vylomova, Miikka Silfverberg, and Mag-dalena Markowska. 2019. Weird inflects but ok:Making sense of morphological generation errors.In Proceedings of the 23rd Conference on Computa-tional Natural Language Learning (CoNLL), pages140–151.

Ulrike Hahn and Ramin Charles Nakisa. 2000. GermanInflection: Single Route or Dual Route? CognitivePsychology, 41(4):313–360.

Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel,Adina Williams, and Ryan Cotterell. 2020. A tale ofa probe and a parser. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics, pages 7389–7395.

John Hewitt and Percy Liang. 2019. Designing andinterpreting probes with control tasks. In Proceed-ings of the 2019 Conference on Empirical Methodsin Natural Language Processing and the 9th Inter-national Joint Conference on Natural Language Pro-cessing (EMNLP-IJCNLP), pages 2733–2743.

Dieuwke Hupkes. 2020. Hierarchy and interpretabil-ity in neural models of language processing. Ph.D.thesis, University of Amsterdam.

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, andElia Bruni. 2020. Compositionality decomposed:how do neural networks generalise? Journal of Arti-ficial Intelligence Research, 67:757–795.

Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema.2018. Visualisation and ‘diagnostic classifiers’ re-veal how recurrent and recursive neural networksprocess hierarchical structure. Journal of ArtificialIntelligence Research, 61:907–926.

Ray Jackendoff and Jenny Audring. 2018. Morphologyand memory: toward an integrated theory. Topics incognitive science.

David King, Andrea Sims, and Micha Elsner. 2020. In-terpreting sequence-to-sequence models for Russianinflectional morphology. In Proceedings of the Soci-ety for Computation in Linguistics 2020, pages 481–490.

Christo Kirov and Ryan Cotterell. 2018. RecurrentNeural Networks in Linguistic Theory: RevisitingPinker and Prince (1988) and the Past Tense Debate.Transactions of the Association for ComputationalLinguistics, 6:651–665.

Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senel-lart, and Alexander M Rush. 2017. OpenNMT:open-source toolkit for neural machine translation.In ACL (System Demonstrations).

Kris Korrel, Dieuwke Hupkes, Verna Dankers, andElia Bruni. 2019. Transcoding compositionally: Us-ing attention to find more generalizable solutions.In Proceedings of the 2019 ACL Workshop Black-boxNLP: Analyzing and Interpreting Neural Net-works for NLP, pages 1–11.

Klaus-Michael Köpcke. 1988. Schemas in German plu-ral formation. Lingua, 74(4):303–335.

Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito,Marco Marelli, Marco Baroni, and Stanislas De-haene. 2021. Mechanisms for handling nested de-pendencies in neural-network language models andhumans. Cognition, page 104699.

Yair Lakretz, German Kruszewski, Theo Desbordes,Dieuwke Hupkes, Stanislas Dehaene, and Marco Ba-roni. 2019. The emergence of number and syntaxunits in LSTM language models. In Proceedings ofthe 2019 Conference of the North American Chap-ter of the Association for Computational Linguistics:Human Language Technologies (NAACL), Volume 1(Long and Short Papers), pages 11–20.

Christos Louizos, Max Welling, and Diederik PKingma. 2018. Learning sparse neural networksthrough L0 regularization. In Proceedings of the6th International Conference on Learning Represen-tations (ICLR).

Robert Malouf. 2017. Abstractive morphological learn-ing with a recurrent neural network. Morphology,27(4):431–458.

Gary F Marcus, Ursula Brinkmann, Harald Clahsen,Richard Wiese, and Steven Pinker. 1995. Germaninflection: The exception that proves the rule. Cog-nitive psychology, 29(3):189–256.

Arya D. McCarthy, Adina Williams, Shijia Liu, DavidYarowsky, and Ryan Cotterell. 2020. Measuring thesimilarity of grammatical gender systems by com-paring partitions. In Proceedings of the 2020 Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 5664–5675.

Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.Right for the wrong reasons: Diagnosing syntacticheuristics in natural language inference. In Proceed-ings of the 57th Annual Meeting of the Associationfor Computational Linguistics (ACL), pages 3428–3448.

Kate McCurdy, Sharon Goldwater, and Adam Lopez.2020. Inflecting when there’s no majority: Limita-tions of encoder-decoder neural networks as cogni-tive models for German plurals. In Proceedings ofthe 58th Annual Meeting of the Association for Com-putational Linguistics (ACL), pages 1745–1756.

Gereon Müller. 2015. Remarks on nominal inflectionin German. Akademie Verlag.

Martin Neef. 2004. The relation of vowel letters tophonological syllables in English and German. Writ-ten Language & Literacy, 7(2):205–234.

Martin Neef. 2011. Die Graphematik des Deutschen,volume 500. Walter de Gruyter.

Tiago Pimentel, Naomi Saphra, Adina Williams, andRyan Cotterell. 2020a. Pareto probing: Trading offaccuracy for complexity. In Proceedings of the 2020Conference on Empirical Methods in Natural Lan-guage Processing (EMNLP), pages 3138–3153.

Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay,Ran Zmigrod, Adina Williams, and Ryan Cotterell.2020b. Information-theoretic probing for linguisticstructure. In Proceedings of the 58th Annual Meet-ing of the Association for Computational Linguistics(ACL), pages 4609–4622.

Steven Pinker. 1998. Words and rules. Lingua, 106(1-4):219–242.

Steven Pinker and Alan Prince. 1988. On language andconnectionism: Analysis of a parallel distributedprocessing model of language acquisition. Cogni-tion, 28(1-2):73–193.

Abhilasha Ravichander, Yonatan Belinkov, and EduardHovy. 2021. Probing the probing paradigm: Doesprobing accuracy entail task relevance? In Proceed-ings of the 16th Conference of the European Chap-ter of the Association for Computational Linguistics(EACL): Main Volume, pages 3363–3377, Online.Association for Computational Linguistics.

Abhilasha Ravichander, Eduard Hovy, Kaheer Sule-man, Adam Trischler, and Jackie Chi Kit Cheung.2020. On the systematicity of probing contextual-ized word representations: The case of hypernymyin BERT. In Proceedings of the Ninth Joint Con-ference on Lexical and Computational Semantics,pages 88–102.

D E Rumelhart and J McClelland. 1986. On Learningthe Past Tenses of English Verbs. In Parallel dis-tributed processing: Explorations in the microstruc-ture of cognition, pages 216–271. MIT Press, Cam-bridge, MA.

Mark S. Seidenberg and David C. Plaut. 2014.Quasiregularity and Its Discontents: The Legacyof the Past Tense Debate. Cognitive Science,38(6):1190–1228.

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, JoellePineau, Adina Williams, and Douwe Kiela. 2021.Masked language modeling and the distributionalhypothesis: Order word matters pre-training for lit-tle. CoRR, abs/2104.06644.

Lucas Torroba Hennigen, Adina Williams, and RyanCotterell. 2020. Intrinsic probing through dimen-sion selection. In Proceedings of the 2020 Confer-ence on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 197–216.

Jochen Trommer. 2020. The subsegmental structureof German plural allomorphy. Natural Language &Linguistic Theory.

Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length.In Proceedings of the 2020 Conference on EmpiricalMethods in Natural Language Processing (EMNLP),pages 183–196.

Jennifer C. White, Tiago Pimentel, Naomi Saphra, andRyan Cotterell. 2021. A non-linear structural probe.In Proceedings of the 2021 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies(NAACL-HLT), pages 132–138, Online. Associationfor Computational Linguistics.

Adina Williams, Damian Blasi, Lawrence Wolf-Sonkin, Hanna Wallach, and Ryan Cotterell. 2019.Quantifying the semantic core of gender systems. InProceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 5734–5739.

Adina Williams, Tiago Pimentel, Hagen Blix, Arya D.McCarthy, Eleanor Chodroff, and Ryan Cotterell.2020. Predicting declension class from form andmeaning. In Proceedings of the 58th Annual Meet-ing of the Association for Computational Linguistics(ACL), pages 6682–6695.

Charles D. Yang. 2016. The price of linguistic produc-tivity: how children learn to break the rules of lan-guage. The MIT Press, Cambridge, Massachusetts.

Eugen Zaretsky and Benjamin P Lange. 2016. No mat-ter how hard we try: Still no default plural marker innonce nouns in Modern High German. In A blend ofMaLT: selected contributions from the Methods andLinguistic Theories Symposium 2015, number Band15 in Bamberger Beiträge zur Linguistik, pages 153–178. University of Bamberg Press, Bamberg.

Matthew D. Zeiler. 2012. ADADELTA: an adaptivelearning rate method. CoRR, abs/1212.5701.

A Additional behavioural analyses

Here, we present two additional analyses for thenonce word stimuli from Marcus et al. (1995).Firstly, we present them to the model as com-pounds, to investigate whether these longer inputschange the model predictions too (see §3). We forma novel noun-noun compound with the nonce wordin the second position while keeping the neuter tag(e.g. presenting <n> t i e r b r a l </s> tothe model instead of <n> b r a l </s>), usingthree nouns of different genders (“der Zahn”, “dieHand”, “das Tier”). Generally, the plural class isdetermined by the second noun in a compound inGerman, but it is possible that our models might bebiased by the first noun of the compound to emit adifferent plural class. There is only a small impactof the specific noun used to form a compound (Fig-ure 11). A pattern that is more pronounced is that,overall, there are fewer -s predictions, and manymore -er predictions. Considering that by creatinga compound we increased the length of the nonceword (from 4.6 to 8.6), this suggests a correlationbetween input length and plural class emitted, ashas been previously observed in the main paper.

Secondly, we investigate how models’ predic-tions for nonce words change during training, asshown in Figure 12. Small fluctuations aside, thenonce predictions do not appear to change substan-tially after the point of overgeneralisation shownin Figure 3, even though the model’s training ac-curacy increased until the end of training. Thispattern is consistent in the remaining productionsfor which Figure 4 only showed the final epoch’spredictions for brevity. It seems the predicted suffixclasses have been decided for the majority of theinputs early on during training. Considering thatGerman plural noun prediction suffers from a lackof generalisation for minority classes, this initialphase during training might be a crucial period toremedy this.

zahn

, r.

hand

, r.

tier,

r.

zahn

, n.r.

hand

, n.r.

tier,

n.r.

0

25

50

75

100

% o

f sam

ples

-(e)n-e-ø-er-s-?

Figure 11: Predicted plural classes for nonce wordswhen presented as a compound with the neuter gender.

10 20epochs

0

20

40

60

80

100

% o

f non

ce p

redi

ctio

ns -?

-e

-er

-s

(a) Rhymes

10 20epochs

0

20

40

60

80

100

% o

f non

ce p

redi

ctio

ns -?

-e

-er-s

(b) Non-Rhymes

Figure 12: Predictions for the nonce nouns from Mar-cus et al. (1995), presented with the neuter gender tag.

1 2 3 -3 -2 -1time steps

0

20

40

60

80

100

(mac

ro) F

1

-(e)n-e-ø-er-sall

(a) [~h1,et ;~h2,e

t ;~c 1,et ;~c 2,e

t ]

1 2 3 -3 -2 -1time steps

0

20

40

60

80

100

(mac

ro) F

1

(b) [~h1,et ;~h2,e

t ]

1 2 3 -3 -2 -1time steps

0

20

40

60

80

100

(mac

ro) F

1

(c) [~c 1,et ;~c 2,e

t ]

1 2 3 -3 -2 -1time steps

0

20

40

60

80

100

(mac

ro) F

1

(d) [~i1,et ;~i2,et ]

1 2 3 -3 -2 -1time steps

0

20

40

60

80

100

(mac

ro) F

1

(e) [~f1,et ; ~f2,e

t ]

1 2 3 -3 -2 -1time steps

0

20

40

60

80

100

(mac

ro) F

1

(f) [~o 1,et ;~o 2,e

t ]

Figure 13: DCs trained over various model componentsextracted from the model’s encoder. The DC is trainedand evaluated with data per time step, separately. Neg-ative time steps are relative to the position of the EOStoken in position -1.

B Additional DC analyses

In Figure 13a, we visualise results for DCs trainedon the concatenated hidden and memory cell states,for multiple time steps in the encoder. The remain-ing graphs present the same performance measuresfor DCs trained on (13b) the hidden states, (13c)memory cell states, and the (13d) input, (13e) for-get and (13f) output gates.

C Additional analyses interventions

§5 presented causal interventions with α = 2.0.That hyperparameter controls the size of the updateof the hidden representation. A large α yields moresuccessful interventions, but also more interven-tions that are not well-formed. As α increases, sodoes the frequency of these errors. We summarisethis trend for each of the plural classes in Figure 14.

0.5 1 1.5 2 2.5 3.0 3.50

20

40

60

% o

f pre

dict

ions

per

cla

ss

(a) Successful interventions

0.5 1 1.5 2 2.5 3.0 3.50

20

40

60

% o

f pre

dict

ions

per

cla

ss

-(e)n-e-ø-er-s

(b) % not well-formed

Figure 14: Visualisation of the impact of the step sizeα on the percentage of changed predictions per targetclass, with correct predictions per class and predictionsthat are no longer well-formed.

3 5 10 15 20 25epochs

0

20

40

60

80

100

(mac

ro) F

1

-(e)n-e-ø-er-sall

Figure 15: Performance (Macro F1) over epochs forDCs trained on the control task.

D Control tasks

In order to assess the ability of our probes to learna linguistically meaningless control mapping, wetrain a DC in a control setting (Hewitt and Liang,2019), by randomly reassigning labels to each in-put. Note that we cannot use word identity, as isused by Hewitt and Liang (2019), as a basis for ourlabel shuffling in our morphological task: words donot reappear. Instead, we randomly assign classesbased on two features of the input word: gender tagand final two letters. For an example, see Table 4.Control classes are sampled based on their actualfrequency as described in Table 1. We again trainfive DCs with different random initialisations, in-cluding different control labelings, and report theiraverages. The control DC can reach a 41% macro-averaged F1-score. Figure 15 shows an overviewof the accuracy of the control DCs per epoch.

We further use the control DCs to perform causalinterventions. As described in §5, we only performinterventions on those predictions that are well-formed, but mapped to the incorrect class. The

Gender & Features Control classSingular form

<m> Strauch m, ch -(e)n<m> Tisch m, ch -(e)n<m> Wagen m, en -e

Table 4: An excerpt from a possible control label shuf-fling. The correct plural forms emitted by the recurrentmodel are Sträucher (-er), Tische (-e) and Wagen (-ø)respectively. The control mapping instead picks a ran-dom class based on two features: gender and final twoletters. “Strauch” and “Tisch” are assigned to the samerandomly chosen class -(e)n, such that the mapping isdeterministic, but linguistically meaningless.

gradient produced by the control DC, with whichwe update the activations is, unlike the originalsetup, not informed by the actual class outputtedby the recurrent model. This “class” in our controltask is instead a random set of words, that do notnecessarily correspond to the plural form emitted.Successful interventions are therefore coinciden-tal. The results for all interventions are listed inFigure 16. The impact of the hyperparameter α onthe control DC is visualised in Figure 17. Someoutputs can be corrected using an effectively mean-ingless hidden state update: the maximum percent-age of corrected predictions never reaches above20% for any particular plural class, as can be seenin Figure 17a (compare Figure 14a).

-(e)n -e -ø -er -s ?prediction after intervention

-(e)n

-e

-ø

-er

-s

targ

et

2.10.078 0 0 1.8 1.6

0.58 8.4 0 0 0.072 7

2.5 2 3.1 0.760.51 15

0 0.95 0 13 0.29 17

0.48 4.3 0.530.33 2.9 110

5

10

15

%

(a) Changed predictions

-(e)n -e -ø -er -sunchanged predictions

-(e)n

-e

-ø

-er

-s

targ

et

0 80 5.5 0 9.4

55 0 9.6 12 6.7

19 35 0 1.3 21

0.37 68 0 0 0.66

24 51 3.9 1.3 00

20

40

60

%

(b) Unchanged predictions

Figure 16: The results of performing interventions(α = 2.0) using the DC trained on the control task. (a)Target class distribution for interventions that changedthe prediction’s suffix. (b) Target class distribution forinterventions that did not change the suffix.

0.5 1 1.5 2 2.5 3.0 3.50

20

40

60

% o

f pre

dict

ions

per

cla

ss

(a) Successful interventions

0.5 1 1.5 2 2.5 3.0 3.50

20

40

60

% o

f pre

dict

ions

per

cla

ss

-(e)n-e-ø-er-s

(b) % not well-formed

Figure 17: The impact of α on the changed predictionsper target class for DCs trained on the control task.

E Rule-based model

We train rule-based models in the manner suggested by Belth et al. (2021). A part of the model can beseen in Figure 18. We make visualisations of the full models available here.

¬[f|v|ß|ag|ai|an|bs|dt|eb|eg|eh|em|ex|ez|hs|ig|ih|il|im|ix|ks|kt|lg|ll|lm|ls|mm|od|ol|ps|rh|rk|rp|rs|ss|tz|ug|uh|un|uz|xt|yd|yl|ym|zt|aar|aht|alz|ang|ank|ans|app|arg|arm|arn|arz|aub|aum|bak

¬[ds|en|er|ln|tl|nel|sus|vel|xus|agel|akel|atus|ckel|ddel|ebel|edel|efel|ekel|emel|esel|ferl|gsel|hsel|igel|isel|itel|itus|mpel|ndel|ngel|nkel|ntel|obel|odel|ogel|pfel|rbel|rfel|rkel|rtel|scal|tsel|ttel|tzel|ubel|upel|utel|watt|ammel|bugel|essel|hadel|hugel|iegel|immel|ippel|kabel|kegel|lugel|nabel|oppel|pegel|rudel|segel|uffel|ummel|uppel]#

-∅

[ds|en|er|ln|tl|nel

¬[ab|ad|as|ch|ed|hn|ld|lz|mt|nn|pt|rn|att|aul|aus|aut|eib|eid|ett|gut|ind|olk|ort|fand|gott|gtum|htum|land|nest|stum|drand|kband|mband|nrand|tband]#

-er

[ab|ad|as|ch|ed|hn|ld|lz|mt|nn|pt|rn|att|aul|aus|aut|eib|eid|ett|gut|ind|olk|ort|fand|gott|gtum|htum|land|nest|stum|drand|kband|mband|nrand|tband]#

¬bau#

-ten

bau#

¬[ds|en|er|ln|tl|nel|olt|sus|vel|xus|agel|akel|atus|ckel|ddel|ebel|edel|efel|ekel|emel|esel|ferl|gsel|hsel|igel|isel|itel|itus|mpel|ndel|ngel|nkel|ntel|obel|odel|ogel|pfel|rbel|rfel|rkel|rtel|scal|tsel|ttel|tzel|ubel|upel|utel|watt|ammel|bugel|essel|hadel|hugel|iegel|immel|ippel|kabel|kegel|lugel|nabel|oppel|pegel|rudel|segel|uffel|ummel|uppel]#

-∅

[ds|en|er|ln|tl|nel|olt|sus|vel|xus|agel|akel|atus|ckel|ddel|ebel|edel|efel|ekel|emel|esel|ferl|gsel|hsel|igel|isel|itel|itus|mpel|ndel|ngel|nkel|ntel|obel|odel|ogel|pfel|rbel|rfel|rkel|rtel|scal|tsel|ttel|tzel|ubel|upel|utel|watt|ammel|bugel|essel|hadel|hugel|iegel|immel|ippel|ka

¬[rb|zip|rial]#

-ien

[rb|zip|rial]#

[ds|en|er|ln|tl|bel|fel|gel|kel|mel|nel|pel|sel|sus|tel|vel|xus|zel|adel|atus|ddel|edel|ferl|itus|ndel|odel|scal|udel|watt]#

dort|echt|eind|etar|fest|fett|gast|grad|hain|hand|heer|heid|hied|hron|ieck|koch|kret|lach|laus|leck|lent|lett|lick|llit|lort|ltar|maus|meer|mied|nast|neck|nett|nort|ntal|ntar|nton|ogon|omen|pann|phon|pier|porn|port|pott|rach|rack|ranz|reck|rest|rort|rott|rtal|rton|ruch|runk|sack|sort|such|tand|teck|tett|tier|tort|tton|ucht|vier|wand|wanz|wart|weck|werb|wind|wirt|ylen|zial|zier|ahorn|ament|arett|atron|brand|dicht|eback|gment|iment|oment|onier|ostum|rband|richt|rnier|ument|wicht]#

¬m

-∅

m

-n ¬n

n

¬tel#

-s

tel#

failed

¬[fel|kel|sel]#

-∅

[fel|kel|sel]#

Figure 18: A part of a rule-based cognitive model (Belth et al., 2021), trained using the Wiktionary training set.