Can a small model find the stat changes in a LitRPG book? I tested four ways

This is a WriteIDE Lab note, not a release.

WriteIDE tracks stats, items, skills and quests. The obvious feature is to import a finished book and have the app list every change: HP down, a level up, a new skill. I wanted to know whether that can work before I build it, so I measured it on two of my own books.

What I mean by a model, and what the tool does

A model is a program that has learned patterns from a huge pile of text. You hand it some words and it hands back a guess, such as "this line looks like a stat change". It does not know your book. It knows what lines like this one usually look like. The big models you have heard of run on someone else's servers. The small ones I tried run on a laptop.

A rule is the opposite: a plain instruction I wrote myself, such as "a short line with a number and a plus sign is probably a change". A rule does exactly what it says and nothing more, so it stays quiet when it is unsure.

Here is the job. WriteIDE reads a finished book and offers a list: "this looks like a change". You confirm each one, and nothing is written without you. Every offer is either right or wrong, and I count both. The test is which way of making that list works best.

The test

An AI assistant read both books and labelled every change it was sure of: 260 in the first book, 123 in the second. It marked a further 70 and 47 as ambiguous and kept them out of the score. One reader labelled both books, so there is no second opinion on the labels.

The bar was set before I started. The tool had to find at least 60 percent of the real changes, make at most one wrong offer per chapter, and take under three minutes of my time to set up. I scored only on half of each book that the tool had not been shown.

Way 1: sentence rules

My first design was a quiet note beside the sentence you just wrote, powered by plain rules. On both books it made no offers at all. The first book is full of boxes and notices, and only 3 of its 8,981 sentences name both a character and an item I had recorded. The second book is in first person, and the rule needs a named person starting the sentence. Zero found, zero wrong. In the next build I am hiding that feature while I decide what to do with it.

Way 2: read the boxes

Many LitRPG changes sit in a game box, so I tried rules that read box lines. Once I showed it one small box as an example, it found 4.6 percent of the first book's changes and 50.4 percent of the second's. Wrong offers stayed low, from 0.00 to 0.30 per chapter, because the rules say nothing when they are unsure.

The two books split by form. The second book writes 98 percent of its changes in boxes. The first writes 68 percent in boxes and 31 percent as narration. Narrated changes, a hit or a lost sword in a paragraph, were found 0 times in either book, in any test.

Way 3: small models

I tried two small models, both run on one laptop. Used as a filter on the word-list results, with nobody clicking anything, they found about 41 to 58 percent. They also made between 4.7 and 5.9 wrong offers per chapter, against my limit of one. A model that tried to read the prose directly found 48 percent in the first book at 12 wrong offers per chapter. It took 8.6 minutes per 50,000 words of that book, and 8.6 percent of its offers matched my labels.

Two caveats. Nobody read those wrong offers, so some may be real changes my labels missed. And the same model, prompt and settings gave different answers on the Mac's graphics chip and its processor: different categories on 7 of 10 invented lines. The same book could get different offers on two machines.

Way 4: my own clicks

The only help that earned anything was me. I answered yes or no to about 40 of its offers. A very small program, a classifier, learned from those answers which kinds of lines I accept, and then hid offers that looked like the ones I rejected. It can only remove offers, never add them. On the second book that reached 58.5 percent found at 0.83 wrong per chapter, just under the 60 percent line. On the first book it reached 29.7 percent at 3.81 wrong.

The clicks were simulated from the labels, and the three minutes are my estimate (about 160 seconds), not a timed run. Held to the one-wrong-per-chapter limit, a version with no model, only letter patterns, was about six points behind the embedding model on the second book and level with it on the first. Comparing the helpers fairly, clicks added four to eight points, and no helper cleared the bar on both books.

Why the layout is per book

Teaching matches word for word. Of 40 box-line shapes in the first book's unseen half, 9 appeared in the half I had shown it. In the second book it was 6 of 20. The same notice is often worded differently by one word or a plus sign. What decided the score was not how LitRPG a book is. It was whether the book writes its changes in one stable form. A survival story with a few repeated meters was easy. A book with varied notices was hard.

What I am doing

I am shipping no generative model. Box reading is the only part with any reach, and it is safe because it stays quiet when unsure. I need a third book, from an author who is not me, before I build the click classifier. It would still not find the first book's narrated changes. Those would need reading prose, and the model that tried was far over the wrong-offer limit.

Two books by one author prove little about everyone else's.