New: opt-in AI analysis on your documents (and more editor improvements)

I'm so freaking excited about this update and I'm working on a full blog post explaining all the new features, but I wanted to give all the wonderful forumsfolk a heads up beforehand about what's new. I've been working on this feature for almost 6 months so it's super exciting to see it finally come to fruition.

Edit: Here's the full blog post! :)

Anyway, there's a ton of stuff packed in the update that went out last night. The two big AI features currently require a lot of processing power on the server (and one literally runs on a supercomputer at IBM) so they're currently only available to Premium users, but I'll be monitoring costs and figuring out ways to get them into more peoples' hands.

So here we go:

• The document editor is now waaaaay prettier and runs snappier. The width of the document is now properly sized according to recommended online guidelines and the typography on headers, etc is now more reasonably styled. The editor should also work a bit better on phones and tablets, also.

• You can now add your documents to a universe. You can also filter your document list by universe (like you can any other kind of page), and you'll find documents automatically hide themselves whenever you're using the "universe focus" feature at the top of the sidebar (that only shows pages from a singular universe). You can now also manage your document privacy settings per-universe (like you can other pages) – so if you add a document to a public universe, that document also automatically becomes public; marking that universe private will also mark all the pages within it as private, and so on.

• You can now analyze your documents with AI to measure writing style and content. I'll post some screenshots in a second comment because I think this feature is too cool and want to show it off, but you can see a ton of useful information about your writing style, the characters in your story, their emotions, and how you stack up against some famous authors/pieces. Right now it's mostly a comparison to Hemingway and/or typical writing, but I'd love to use this analysis tech to analyze more authors so you can choose your favorite authors to compare your own writing to. :) This feature is entirely opt-in and no AI will run on your documents unless you request an analysis. This feature is only available to Premium users.

• You can now add quick reference tabs to the document editor for any of your pages. These tabs appear on the right side of the screen and will open a sidebar containing all of that page's information whenever you click one. This lets you reference any answer to any field on any page while writing without having to leave the editor. Quick reference tabs are automatically created whenever you @mention a page, whenever the AI detects you're referring to one of your existing pages (during an analysis), or whenever you manually add a quick reference. This feature is only available to Premium users right now.

• Adding a quick reference page to a document also creates a "Documents" tab on that content page. For example, if you add "Alice" to a couple documents she appears in, she'll also get a tab on her page that lists those documents.

Screenshots below, but – as always – if anyone has any feedback or suggestions on how to improve the document editor, the new document analysis, or the new quick reference feature, I'd love to hear them! :) This is a huge update with tons and tons of new code behind the scenes, so if anyone sees anything wonky anywhere, please do let me know!

Happy worldbuilding!

Estimated reading times:

Readability measurements:

Character emotional ranges:

Style measurements:

Here's what the quick reference sidebar looks like in the editor when you click a tab:

Edit: @numbinside You're too fast! :P

@Riorlyne There's actually a bunch of different writing gauges out there that measure different things behind the scenes, and they're all good for measuring different things (and therefore not as good at measuring other things).

For example, the long names in fantasy novels might be disproportionately marked as harder to read by e.g. the Coleman-Liau index (which measures letters per word as a big metric) or the Flesch-Kincaid reading ease (which measures syllables-per-word). On the other hand, measurements like FORCAST only care about the number of one-syllable words compared to two-or-more-syllable words, so there's no difference between naming a character "Billy" and naming him "Biltaskothaster".

The analysis takes the five most popular of these readability metrics and combines them into a singular number. Since documents can belong to universes now and universes have a "genre" field, I should be able to improve things by adjusting weights for e.g. FORCAST to be weighted higher when the document is set in a fantasy or sci-fi world. Right now all the weights are (mostly) equal, but that's something that'll improve over time as more analyses happen.

A bit of a dream of mine is to also come up with my own readability formula (and be able to do things like excluding names altogether), but that's a bit far off. ;)

Haha no, these are great questions. The blog post is up now (here!) but these specifics aren't in it. I'm going to try to add more of this kind of information to the analysis itself (for example, explaining what each reading metric measures and what is important to raise/lower if you want to affect that score).

The emotions are from a set of neural networks and classifiers run on IBM Watson. After running entity recognition on the writing (to pull out the character names and references), a large neural network looks at emotions and tone at a lexicon/word-level and relates them to the extracted entities with a lot of crazy statistics. Obviously, "emotion" is a super-broad field and hard to pick up even as a human, so the models Watson uses might not always be 100% accurate, but they are always improving little by little. I've had a few weird readings where a character's dominant emotion wasn't anything close to what I expected, but in one case it was interesting to read through with that context and see what positive words were used around that character, rather than by/toward that character itself. I changed the positive words to ones that matched the tone I intended for that character a bit better and I think it turned out better in the end. That might not always work out though. :)

I'll take a look at those explanation links on Safari/iPad; thanks for the heads up!

Ha, that's a complicated answer. But first let me preface that those reference links are probably the most experimental of all the analysis features.

In the most basic sense, it works primarily by looking at word choice and compares the words used to words used in labeled training data. For example, a training document about cars would probably talk about speed, color, doors, hoods, and so on. Obviously, those words could all apply to things other than cars on their own, so the neural network doing the classification looks at all the potential tags of all the words and makes its best guess by cross referencing all the topics that every known word could relate to and measuring how likely that topic is based on how many words (potentially) reference it. I have a confidence threshold that I discard guesses under (so if the AI isn't at least e.g. 70% certain a topic applies, I ignore it). Finding the proper confidence level that minimizes incorrect predictions without excluding correct ones is what I can (easily) adjust here, so I'm watching for feedback on these features to see if I need to adjust that confidence threshold up (if guesses are wrong) or down (if no guesses or not enough guesses).

I'm not really familiar with blood stain analysis myself (ha), but I'd guess something in the word choices in your story is similar enough to the word choices used in blood stain pattern analysis. Neural networks are great for classification problems like this, but unfortunately they're not good at all at describing why they classified something. It's our job as humans to make the call on whether those classifications are right or not; sometimes they are and sometimes they get just a little confused. :)

@ninja_violinist There isn't a way to analyze a first person narrator yet. The most exciting part of this tech (in my opinion) is that it's literally bleeding edge work in a new frontier. That's a double edge sword though: it usually means no one has done X or Y before, which means I get to do it myself – which also means it's incredibly hard and will probably take a significant amount of time. I appreciate the patience and shared excitement though. :D

I'm researching the best way to analyze and merge recognized entities now. What you can do if you do have a Tim page is to click "Link page" on each of the different nicknames and select your Tim page. That should link them all together and have them all treated as a singular person – Tim.

@"Eris the Melancholy" If I could find a nice way to list which words most/least contribute to that melancholy feel, would that be useful? I'm trying to walk the fine line between offering insights into how people are writing versus just telling them what to do. Everyone's voice is unique and I want to embrace that, rather than conform everyone into a single style. :)