Oof yes you read that right. Suborbital Salvage had a launch day crash.

The kicker however was that it was only happening for some users, which was rather perplexing since the Playdate isn't like a desktop or laptop where there's hundreds of potential hardware configurations. Every Playdate user should have the same hardware. Well, technically one of two revisions of the hardware, but I have both revisions and I promise I tested on both of them before launch!

Researching the problem

At first I only saw one report, and thought maybe it was the rare Heisenbug that other devs had been reporting. However once I had a couple more reports, I realized I had a genuine problem on my hand.

I started by sanity checking myself and retested the build I uploaded to Catalog. No dice, game loaded fine.

While noodling on some potential ideas, scizzorz reached out after they also encountered the same crash. I'm super grateful for this! Being able to collaborate with another Playdate dev made it super easy to troubleshoot the problem.

We landed pretty quickly on the issue being the game taking too long to load, and being killed by the OS via the 10 second watchdog timer. However once again I was perplexed on why exactly this wasn't happening on either of my devices.

I was starting to wonder if this was some sort of bug with the Playdate OS, but then Scenic Route shared that Squid God's video on Playdate performance talks about how devices with lots of games take longer to load games (the relevant part of the video starts at 16:10).

Turns out that there's a quirk with flash storage: the fuller it is, the worse disk performance gets!

At this point I took a break to check notifications and realized Panic had gotten back to me with a promo code. Catalog games aren't automatically added to the creator's account, so the current work around is to give yourself a promo code.

After downloading the Catalog version of the game I was surprised to find that I was able to reproduce the crash. I immediately diffed the Catalog build and the build I uploaded, but only found some minor changes to the pdxinfo and an additional 28 bytes added to main.pdz. Seemed unlikely that these changes would cause a problem...

Then scizzorz shared another helpful thought: maybe the checksum validation (which is related to the DRM in Catalog games) was being counted against the startup time.

I did some digging and found this dev forum post. I'm not 100% certain if this is specifically regarding Catalog games DRM, but seems to line up with the observed behavior. Relevant excerpt:

Part of it is Lua runtime, which uses a random hashing of tables so the performance of each launch can vary up to 10%. This is done for security purposes to try to prevent people hacking games, as table contents will be in different places each launch.

Implementing a solution

The solution is probably the least interesting part of this story.

A very long time ago I had the inkling that Suborbital's startup time might cause problems, so I had backlogged a task for myself to to spread it out over multiple frames. I never implemented it because of the aforementioned not seeing problems on my devices. Lesson learned: don't be a lazy game developer - make a proper loader!

Implementing it was very straight forward since Playdate supports Lua's coroutines. I had a small bit of startup code just sitting at the top of a few files that I had to refactor into functions, but then all I had to do was add a new preload state to my game, and call said functions with coroutine.yield() between them.

Snippet from my preloader state:

redrawLoaderMessage("loading music")
coroutine.yield()

musicManager.load()

redrawLoaderMessage("loading textures")
coroutine.yield()

loader.load()

redrawLoaderMessage("initializing systems")
coroutine.yield()

player.preload()
playerData.load()
dustGenerator.load()
warningManager.init()
collisionManager.addGroupCollision("player", "obstacle")

redrawLoaderMessage("loading entities")
coroutine.yield()

entityManager.init(entityDefinitions)

I also had to dig a bit deeper into the chunk loading and static batching functions so that I could yield after processing batches of chunks. Yielding after every chunk/iteration actually made boot time slower (tho it didn't crash anymore) so I opted to do a batch of 10 each frame.

local chunkFiles = playdate.file.listFiles("chunks")
local startTime = playdate.getCurrentTimeMilliseconds()
local batchCount = 0
for i = 1, #chunkFiles do
  local path = "chunks/"..chunkFiles[i]
  local file = playdate.file.open(path)
  local size = playdate.file.getSize(path)
  local contents = file:read(size)

  local chunk = json.decode(contents)
  chunk.name = chunkFiles[i]
  module.chunksByName[chunk.name] = chunk

  file:close()

  if i % 10 == 0 then -- every 10 batches, yield a frame
    coroutine.yield()
    local endTime = playdate.getCurrentTimeMilliseconds()
    local duration = endTime - startTime
    print("batch "..batchCount.." took "..duration.."ms")
    batchCount = batchCount + 1
    startTime = endTime
  end
end

I also started a thread in the Playdate dev forums to get this information into the developer docs for better visibility.