What a reader can predict from the sentence so far.

The one-paragraph version

Closed inventories are steep and open inventories are flat. Triggers, dependency roles, phrase skeletons, argument frames, and part-of-speech tags each have a handful of values that cover most of the text, and each predicts the next one well. Modifiers, adjuncts, and words do not. A reader who tracks the role of the last token and the skeleton of the open phrase narrows the next tag from eleven effective choices to five. Getting below five needs the word itself: the verb's argument frame after a verb, and the morphology of the incoming word everywhere else.

• Knowing nothing
  11.3
• Previous tag
  7.0
• Previous word
  6.1
• Previous tag and its role
  5.2
• Tag, role, and skeletons of the two open phrases
  4.9
• Structure plus the incoming word's suffix and shape, unknown content words
  2.3

Effective choices for the next tag, held-out. Each bar is a different question, answered in a later section.

1The milestone 1 hypothesis: partially held

The hypothesis: the top 20 triggers cover at least 70% of phrase openings, and for each such trigger the top 5 patterns cover at least 60% of its occurrences. A trigger is the leftmost token of a phrase's subtree. A pattern is the ordered sequence of dependency labels of the head's direct dependents, with the head marked.

The coverage clause holds only because half of all phrases have no trigger. 49.4% are bare pronouns, adjectives, adverbs, nouns, or verbs whose leftmost token is the head itself, and those six SELF cases are the ones that pass. Functional triggers — determiners, prepositions, complementisers, relative pronouns, particles, and conjunctions — open 34.6% of phrases. Lexical modifiers open the remaining 16.0%.

The trigger distribution itself is steep. Thirteen functional triggers cover 70% of functionally-triggered phrases and 38 cover 90%. What is flat is the pattern that follows. the has 910 distinct patterns and needs 11 to reach 60%. that as a complementiser needs 87.

Triggers predict the phrase's role far better than its shape. of opens a noun modifier 94% of the time, relative that a relative clause 92%, and a conjunct 97%, infinitival to an open complement 58%.

The clause layer is flat for the same reason. After a subject opens a clause, 1,959 distinct clause patterns follow and 75 are needed to reach 60%.

2Stopping the pattern at the head

First refinement: keep only the dependents that precede the head. Functional triggers passing the 60% test go from 1 to 8 of 20, the median number of distinct patterns per trigger from 292 to 94, and the median normalised entropy from 0.74 to 0.64. Infinitival to becomes 89% mark HEAD:VERB. After a subject, three clause patterns cover 60% instead of 75.

The remaining spread for prepositions is two binary choices: whether a determiner is present, and whether the head is a noun, proper noun, or pronoun. Half of functionally-triggered phrases carry post-head material that this view ignores.

3Pattern families

Second refinement: treat modifiers and auxiliaries as optional slots. A family is the pattern with those labels removed, displayed with the slots it absorbs in brackets.

Five families cover 83% of functional phrases:

Every preposition collapses to a single family at 90% to 98%. The clausal triggers keep a real split between subject-first and subjectless clauses. The coordinators stay hard because a conjunct can be any phrase type.

4How narrow is the next tag

• Structure without words beats the word. Sixteen tags and fifty roles, about 300 combinations, outperform 15,000 word types, and the structural figures barely move between train and held-out while the word figures collapse.
• Longer history hurts. Tag 3-grams are the best pure-sequence model at 6.4; 7-grams are worse than nothing. Stacking more than two open phrases also hurts. The information is local and structural.
• The pattern-so-far adds nothing once the role and phrase type are known.
• The role works by saying whether the phrase is still open. Pronoun as subject: auxiliary or verb next, 92%. Noun inside a compound: noun next, 93%. Adjective as modifier: noun next, 76%. Pronoun as object or adjective as predicate: wide open, the phrase just closed.
• The verb's role predicts nothing about what follows it. A preposition's role adds nothing because case is 95% of prepositions.

By trigger word, there are three bands. Determiners, possessives, subject pronouns, and if leave two to three choices. Prepositions leave five to seven, mostly the determiner question. and, or, that, as leave eight to ten, barely better than nothing. The narrowing decays fast: two tokens after the, uncertainty is back to nine choices.

For the ambiguous words, knowing the reading halves the uncertainty. to goes from 2.16 to 1.25 bits once infinitive marker and preposition are separated, that from 2.98 to 1.87, which from 2.54 to 1.55, is from 3.11 to 1.92. The caveat is that the reading is often decided by the very next word, so this is an upper bound on what a reader can use.

5Words are an open set

Next-word predictability measured on this corpus is a corpus artefact. A third of tokens follow words seen fewer than 50 times, whose next-word lists are memorised. The honest measure is the Good-Turing chance that the next word is one never seen in that position: 19% after the, 25% after a, 6% after if.

Word-level expectation is meaningful only for closed-class slots. After if, ten words cover 78% of continuations. After the, ten words cover 9%.

6Verb complementation

Whether the verb's frame is predictable depends entirely on whether adjuncts count. This conclusion reversed twice during the work, and each reversal came from whether adjuncts were in the measurement.

Four core frames cover 88% of verb tokens before the verb is known: nothing 36.5%, object 35.8%, open complement 8.9%, clausal complement 7.2%. The lemma then halves what is left, stably across frequency buckets, so it is learned per verb but it transfers. For frequent verbs the dominant frame holds 60% to 90% of the time. want takes an infinitive 63%, think a clause 73%, let object plus infinitive 94%, take an object 75%.

A second test kept obl in the frame and grouped verbs into classes by dominant frame. Five classes cover 92% of verb tokens, but within-class hit rates are only 50% to 59%. The difference is obl: Universal Dependencies labels a required prepositional complement and an optional adverbial the same way, so keeping it reimports adjunct noise. Verbs like go, look, and work need a "verb plus prepositional phrase" class that this corpus cannot confirm.

Adding the governing verb's lemma to the structural state helps only at post-verb positions, which are 19% of the corpus. Effective choices there fall from 8.9 to 7.6 and the best guess improves from 31% to 36%. The gain is modest because the frame is not the tag. An object begins with a determiner, pronoun, noun, or proper noun; a clausal complement with a complementiser, pronoun, or bare verb. The verb fixes the argument type and the argument's own trigger reopens the uncertainty.

After a verb the next word is usually a function word, but which one is a set, not a tag. Nine of the 22 most frequent verbs put at least three quarters of their next-tag mass on closed classes. Only let (pronoun 97%) and want (particle 73%) narrow it to one. Across all positions, a reader can be 90% sure a function word is next only 4% of the time, and 90% sure a content word is next 28% of the time.

7Morphology

For an unknown content word, morphology is the whole lexicon, and it beats structural expectation 68% to 51%. The two cues are nearly independent, so multiplied they reach 79% and 2.3 effective choices.

The decisive suffixes are the familiar ones. -tion and -ment mark nouns above 93%, -lly adverbs at 98%, -ed verbs at 85%, -al adjectives at 82%. -ing is the weak one at 70% verb and 22% noun, which is where the structural prior earns its place. About half of longer content words end in a suffix that is decisive by itself. What remains confused after both cues is proper noun versus common noun, and adjective versus noun, which English marks neither morphologically nor structurally.

8What the trainer needs

Three functions, with the weights the data gives them.

1. Structural state tracking. The last token's role plus the family skeleton of the open phrase and its parent. This is the expectation generator: eleven choices down to five, in about 300 rules that transfer to unseen text.
2. Tag identification of the incoming word. Structure proposes, the word disposes. Morphology and shape are the main tag-fixer for content words and the only one for unknown words. Structure and morphology are complementary and should be combined, not ranked.
3. The verb's core frame. The one lexical fact worth holding in the state. Four frames as the universe, then the dominant frame for the fifty most frequent verbs, one fact each. Everything else after a verb is an adjunct.

For the curriculum: teach the readings of ambiguous triggers as separate triggers (to infinitive versus to preposition, that complementiser versus relative). Teach families as skeleton plus optional slots, without modifier order. Teach that determiners and subject pronouns give near-certain tag expectations while prepositions give only "a noun phrase is coming". Treat coordinators as not teachable as triggers.

9Caveats that apply throughout

• EWT is web text: reviews, email, blogs, forum posts. Not simple text.
• All structural contexts assume the reader has parsed the already-read tokens correctly. The numbers are upper bounds on what an actual reader has.
• 225,000 tokens is small. Every rich context overfits; only the held-out columns should be quoted.
• Universal Dependencies attaches conjunctions to the second conjunct, so the coordination results understate what a reader who has the first conjunct's type could do.
• Full dependency subtypes were kept (acl:relcl differs from acl). The coarse option exists and was not used.
• The optional-slot set for families is a judgement call and was chosen generously.
• None of the definitions were tuned to improve the numbers. Each refinement was run once, as a separate step, and reported against the previous one.

Scripts, in the order the questions were asked: mine_triggers, next_token_analysis, context_ladder, verb_frames, pattern_families, whole_context, verb_in_state, morphology_role, verb_frame_classes. Each writes a report of the same name into the project's out directory.

← Back to the Reading Trainer · All projects