Fake scene skipping + Sprite layer logging = Semi-automated visual novel testing!

Share this post:

The last few days, I've been focusing on eliminating all the bugs I've had waiting in my to-do list while I focused on implementing new episode 2 content.

I was aware (and worried about/trying not to go crazy thinking about) how every time I fix a bug, I might be introducing other bugs. I'm doing things like going through scenes and making sure all the characters and props are layered and positioned correctly, a lot of which is automated by the engine, so there's a very real risk that I change the engine code and some scene in episode 1 that I haven't touched in ages is suddenly graphically broken just as an episode 2 scene is fixed. In the back of my head I was dissatisfied with the lack of a reliable solution--other than manually testing each previous episode before releasing any updates. (Because, the more episodes I release, each one would add 30ish minutes of manual testing time, EVERY time I want to push a patch.)

There are 2 seemingly unrelated features I've added to the engine recently. On a walk this morning, I had a eureka moment that I could combine them to make a very robust, almost fully automated way to test scene visuals.

1. "Fake" scene skipping

Very early on, I decided that a "scene" in the engine would be a data structure containing the background, character positions, prop positions, time of day, light source data, etc. of any given location in an episode. So I would define a scene like "Finn's bedroom - night" which the engine would reuse every time I cut to Finn's bedroom at nighttime throughout the episode. If Finn appears, moves, opens a book (which I handle as a "prop" in the scene), etc., those things are recorded in the scene even if the episode goes to another scene before coming back later.

The implication of this, is that if an episode proceeds in a timeline like so:

1. Show Finn's bedroom at night, Finn has a phonecall so a phone prop needs to be added
2. Show Finn the next day at school
3. Show Finn's bedroom at night again, after school has ended.

When we go back to the nighttime bedroom scene for #3, the phone prop WILL STILL BE THERE when it's not supposed to be. That's a bug. But, for a sequence like this:

1. George is at home and makes a phonecall to Ernest
2. Ernest answers the phone at band practice
3-infinity. Cut between George's scene and Ernest's scene as they alternate speaking lines

This is an ideal system because I WANT the props, characters, etc. to be preserved whenever I cut between the two scenes.

So to fix the first, problematic case where Finn's phone prop isn't supposed to be there later, I have two choices:

a) remove the phone prop manually before switching to the school
b) create a second scene data structure, i.e. "Finn's Bedroom - night 2" and recreate the backdrop and everything else from the first bedroom night scene there. This is harder, but prevents a host of bugs I could introduce by changing the phonecall scene and forgetting to remove things before I reuse it.

I hope this isn't hopelessly confusing, because I can't really use more dev time to proofread and explain it better Tongue

Here's where fake scene skipping comes in.

Imagine I want the player to be able to skip from #1 and #3 in the episode timeline. I can't just use a goto statement to skip over every script instruction in #1 and #2, because then we'll arrive at #3 with props and characters missing which might need to be there. My first solution to this problem was to declare certain instantaneous instructions in my scripting language as "unskippable", i.e. "ADDCHARACTER FINN LEFT FACINGRIGHT" in #1 would always be called even when skipping to #3.

I ran into problems with that approach when I realized I needed to implement certain script instructions that have unskippable side-effects, but necessarily take multiple frames to complete. For example, tweening a character between 2 positions, and expecting that they will stay at the destination position any time the scene is reused later.

For these, if I declared them unskippable, the player would click "Go to scene 3" and still have to watch certain animations play.

So for a long time I was planning to keep the skip feature out of release builds, and only use it myself when testing, acknowledging that tweens would just be ignored if I skipped them.

Eventually I came up with the solution that, instead of declaring certain script instructions "unskippable", I should require EVERY script instruction to implement 2 behaviors: a normal behavior, and a "skipping" behavior.

So an instruction that tweens a character's position, would have a logic like this:

if skipping
    just set the character's position to the destination in the Scene object
else
    animate the character moving between the positions, THEN set the character's position to the destination in the Scene object

Then what really happens under the hood when the player skips a scene, is that EVERY INSTRUCTION in the script STILL GETS CALLED, but its side-effects are carried out instantaneously, and any multi-frame processes are skipped.

This means that I can launch my build, skip to the end credits of an episode, and pretty much every piece of logic in the script will still have been carried out step by step.

2. Sprite layer logging

After enough times manually inserting random print statements to figure out what went wrong when I encountered a visual bug (like a prop appearing behind another prop when I wanted it in the foreground), I had devised a reusable tool for that. It's a function in my engine that iterates through every sprite layer and camera, outputting something like this:

Logging Sprites
###############
Camera #0 (0, 0, 1280x720)
bgColor: -16777216
--------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
---------------
Camera #1 (0, 0, 1280x720)
bgColor: -16777216
--------------
|   0. {assets/images/daySkyEdited.png at (174.117647058824,0) with origin (x: 640 | y: 360), angle 0, scale (x: 1 | y: 1), size 1280x720, alpha 1, frame 0}
|   1. {assets/images/paperLandscape.jpg at (174.117647058824,0) with origin (x: 1100 | y: 850), angle 0, scale (x: 0.424 | y: 0.424), size 931.764705882353x720, alpha 1, frame 0}
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   2. {assets/images/flyman.png at (773.833333333333,166) with origin (x: 249.5 | y: 187), angle 0, scale (x: 1 | y: 1), size 499x374, alpha 1, frame 0}
|   ---------------
|   ---------------
|   ---------------
---------------
Camera #2 (0, 0, 1280x720)
bgColor: 0
--------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
0. {null at (0,0) with origin (x: 640 | y: 360), angle 0, scale (x: 1 | y: 1), size 1280x720, alpha 0, frame 0}
1. {null at (0,0) with origin (x: 640 | y: 360), angle 0, scale (x: 1 | y: 1), size 1280x720, alpha 1, frame 0}
---------------
Camera #3 (0, 0, 1280x720)
bgColor: 0
--------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
|   ---------------
---------------

I would use this to figure out the bug and fix it.

Combining the two

So my wild idea, was to add a script instruction, ASSERTLOGSPRITES, which I could add to a script in any scene where I've fixed a bug. It generates the sprite layer log from the correctly displayed scene, and writes it to a file which I add to version control. Then, in subsequent builds, it regenerates the sprite log, and COMPARES it with the known correct log for that scene.

With ASSERTLOGSPRITES calls placed throughout an episode script, I can test an entire episode for bug regressions simply by running it and skipping past the final scene, BECAUSE nothing is really skipped, so all the state changes and assertions happen instantaneously

To make it even easier to fix regression bugs, I made ASSERTLOGSPRITES output a fully formatted diff between the expected and actual states of a scene:

(filename containing spoiler redacted)