IguaRPG 2 Devlog

[hubol]

This is the official thread for my IguaRPG 2 Devlog.

RULES

• Have a vibe that is good

[hubol]

Update #1: Intro to smooch

Towards the end of development of IguaRPG, my code-change-to-reload time was slowing down due to the number of simultaneous HTTP requests needed to get all of the sprites and audio files! To be fair, this was probably only ~5 seconds, but it was still annoying compared to the 2 seconds I had enjoyed earlier!

So, I have been working on a new automated tool called smooch that will handle a couple things for me:

• Pack textures into texture atlases
• Convert sound effect files into lossy, browser-digestible formats and pack them into zip files
• Generate source code to conveniently access these assets from game code

Previously, I had some tools for items 2 and 3, but they were a bit naive. Here were some problems I identified:

• The source code generated by the tools was not configurable... This made it so that I could not alter the generated code without creating a new "release" of the npm packages I created... I don't know npm that well, so I hadn't automated the release process or anything, which made this very inconvenient. (See a sample of the old generated code here here)
• The tools did not retain their state after exiting. This meant that when I restarted my computer and ran my developer tools for IguaRPG, the numerous sound and music files for the game would be converted to lossy formats before I could playtest.

If I implement the items above, I should be able to render more sprites in future games, since packing textures into texture atlases allows the GPU to do fewer context switches! And I should hopefully enjoy speedy development reload times even as the project grows.

I have a nice (WIP) readme on the feature/v1 branch of the GitHub repository for smooch.

Because I have gone full corporate weirdo (not really, but it sometimes feels that way), I have a Trello board tracking the remaining tasks that I foresee needing to address in this niche tool.

[SquareWheel]

You’re almost there! The board you are trying to access requires a Trello account.

Well heck you too, Trello.

smooch sounds cool.  You got really rather good at all this stuff, Hubs.

[hubol]

Oops I just fixed the permission issue (-:

And thanks! I try ^_^

[dcco]

Towards the end of development of IguaRPG, my code-change-to-reload time was slowing down due to the number of simultaneous HTTP requests needed to get all of the sprites and audio files!

I've had this problem in game dev too only to a worse degree because scala compiler is notoriously slow (30s rebuild times pretty common). I've been theorizing a language that lets you change game code while the program is running (for certain classes of changes)..... Probably will be a long time before I actually do it, but definitely relate to the struggle lol. I think a resource management cool is a much more practical middle ground for that kind of thing lol. I can even imagine it doing something like excluding resources / dummying them out when you only need to test subsets of the game (Music has always taken me the longest to load so often not loading most of the game music helps me).

Also I've heard of Trello before, have you used it a lot already? I've been thinking of using one of those corporate task organizer things but I'm bad at adapting new workflows lol.

Overall excited to follow this devlog while I'm unable to do game dev of my own lol.

[hubol]

I've had this problem in game dev too only to a worse degree because scala compiler is notoriously slow (30s rebuild times pretty common). I've been theorizing a language that lets you change game code while the program is running (for certain classes of changes)..... Probably will be a long time before I actually do it, but definitely relate to the struggle lol.

The compiler problem is interesting. Some languages(?) have a "hot swap" feature. I know that I was able to use this when I was working with Java in Eclipse and maybe JetBrains. Basically you run the program with some special configuration and, while the program is running, certain changes to Java source files are picked up. This stackoverflow discussion from 2018 indicates that might not exist for Scala: https://stackoverflow.com/questions/50696440/can-hot-swap-be-used-with-scala

I think this problem is probably why some game engines embed scripting engines for the game logic. That way the big C++ (or whatever) engine gets compiled less frequently and your scripts are reloaded and reinterpreted by the engine.

I can even imagine it doing something like excluding resources / dummying them out when you only need to test subsets of the game (Music has always taken me the longest to load so often not loading most of the game music helps me).

Dummying them out sounds interesting, but also sounds a bit overkill for what I've done in my awesome game development career. I know that for certain (all?) formats for sound, you can stream the file to the audio device. This could be useful for music, since you wouldn't have to wait for the entire file to load, just enough to fill the buffer to send to the audio device.

In IguaRPG, I didn't do this because I didn't learn how. Maybe someday... But what I did there was load the music asynchronously and as-needed. When you enter the desert town, for example, the music files for the field, bar, inn, oracle, and witch's house all start to load (I called this "warming"). This is done with the following code written in the start-up of each "level" (roughly equivalent to the "Room Start" event of a Game Maker "room"):

Code: [Select]

jukebox.play(Country).warm(NewDesertOracle, Bluehouse, Shop, DesertField, Witch);
From: https://github.com/hubol/igua-rpg/blob/518103cf89ff2e0921af74631d750d42befaff6f/src/levels/desertTown.ts#L39

When .play is called on the jukebox object, the jukebox asynchronously waits for the "warming" process to finish (or starts one if one was not already started), then begins to play the song! The asynchronous nature here means that the game doesn't freeze during this time.

I found the warming approach to be satisfactory. And when the music file is not "warmed" (e.g. when the player loads their save file), the load time for the music file is pretty fast on desktop. Probably less than a second! This is probably from a combination of:

• Using the .ogg format
• Having short music files ^_^
• SSDs are fast

(It is quite a bit worse in the browser since it has to make an HTTP request and that can take a while!)

Also I've heard of Trello before, have you used it a lot already? I've been thinking of using one of those corporate task organizer things but I'm bad at adapting new workflows lol.

I have now used it for SBW2 and IguaRPG! I like it because I choose not to empty out the "Done" column, so I end up with a big huge pile at the end of the project, which I think is cute! You can see the one I had for Igua, here, I think: https://trello.com/b/Zmb2cdPl/iguana I've got some silly column names.

[dcco]

The compiler problem is interesting. Some languages(?) have a "hot swap" feature. I know that I was able to use this when I was working with Java in Eclipse and maybe JetBrains. Basically you run the program with some special configuration and, while the program is running, certain changes to Java source files are picked up. This stackoverflow discussion from 2018 indicates that might not exist for Scala: https://stackoverflow.com/questions/50696440/can-hot-swap-be-used-with-scala

Yeah, it's definitely harder to figure out how to do stuff in Scala rather than Java. They do have a build tool that does rolling re-compilation (saves information about previously-compiled code in between recompiles) which makes things more bearable - but still kind of an annoying language for some aspects of game programming I find.

I think this problem is probably why some game engines embed scripting engines for the game logic. That way the big C++ (or whatever) engine gets compiled less frequently and your scripts are reloaded and reinterpreted by the engine.

Yeah I guess it would basically be a custom scripting language, but then the main language is also a custom language. Probably overkill, and not sure if it's worth bc I'll have to make all my own tools too, but I've been wanting to make my own language for game programming anyway.

Dummying them out sounds interesting, but also sounds a bit overkill for what I've done in my awesome game development career. I know that for certain (all?) formats for sound, you can stream the file to the audio device. This could be useful for music, since you wouldn't have to wait for the entire file to load, just enough to fill the buffer to send to the audio device.

In IguaRPG, I didn't do this because I didn't learn how. Maybe someday... But what I did there was load the music asynchronously and as-needed. When you enter the desert town, for example, the music files for the field, bar, inn, oracle, and witch's house all start to load (I called this "warming"). This is done with the following code written in the start-up of each "level" (roughly equivalent to the "Room Start" event of a Game Maker "room"):
...

Oh actually for browser, loading things asynchronously sounds like a better solution, since that's how browsers are supposed to work anyway. Sensible behavior for when the files aren't warmed probably is the trickiest part - but past that it's probably good enough to get most of the media loaded seamlessly. For my desktop game I front-loaded all media files at the start, but when I think about it doing something similar probably wouldn't be a huge issue, and would reduce the initial start-up time by a lot.

[hubol]

I've been wanting to make my own language for game programming anyway.

I want to see it!!!!!!!!!!! Do it!!!!!!

[hubol]

Making some progress on smooch...

Have had to invent a loooooot of really kooky stuff. It's fun. Maybe I'll talk about it more sometime.

[SquareWheel]

I would read all about the kooky stuff.

[suizinshu]

i'd also love to hear about the kooky stuff... !!!

[hubol]

Update #2: smooch adventures?!

Since I am writing adequate commit messages, I will use my git history to see what wacky stuff I've been up to!

W1. Normalized paths in console output
I thought it would be a cool idea to rewrite paths printed to the console so that they are relative to the current working directory. Since paths are different on Windows and Unix machines (Windows uses \ instead of /), my logging abstraction also normalizes this. These changes mean that my naive automated tests can check that particular paths are being correctly output! I also think it improves the readability of the logs, which is somewhat of a priority for this project.

Here's a sample:

Code: [Select]

            NativeDependencies Checking that native dependencies are installed to .smooch/native-deps
            NativeDependencies Native dependencies appear to be installed.
                        Ffmpeg Set Ffmpeg path to .smooch/native-deps/node_modules/@ffmpeg-installer/win32-x64/ffmpeg.exe
             SmoochJsonWatcher Started
                     FsWatcher Started with subscriptions: audiCnv
                     FsWatcher Ignored Message #0 (Catch-up) <no events>
W2. Converting audio files
The previous incarnation of smooch used sox to convert audio files. This required the user to install a native dependency before running the tool. You also had to make sure that you installed the liblame thing for mp3 or whatever. It got annoying!

Apparently there are some packages on npm that seemingly abuse the "optionalDependencies" property in package.json. Examples: https://github.com/kribblo/node-ffmpeg-installer/blob/master/package.json , https://github.com/parcel-bundler/watcher/blob/69f3099f009330d53abdb0e7d53e9f9204af04b3/scripts/build-npm.js (this code generates a package.json!)

They use this property to install binary packages that are installable on the user's architecture. I searched for a package like this for sox, but I couldn't find one. So I used the @ffmpeg-installer/ffmpeg library. So, smooch now uses ffmpeg to convert audio files. I implemented this and it seems to work fine! Cool!

W3. Type hints in template programs
A core component of smooch is the ability to get metadata about transformed assets and apply the metadata to a template program to generate a code file. Special context is exposed to the code file, and it's not always intuitive how to use that context without types. VS Code supports loading .d.ts (TypeScript declaration) files from within JavaScript files (our template programs are plain JavaScript files) using JSDoc. I love using TypeScript in clever ways, and don't like doing lots of manual work. So, I found a tool to generate a single .d.ts file from some types (for some reason this feature is not built into TypeScript)!! The result is some example template programs with types embedded using JSDoc!: https://github.com/hubol/smooch/blob/973ade74ca20f731e7a44a1e9da2492b8704ee97/dist/templates/audio-convert.js

W4. Zip archiving of audio files
I got to implement this and it's looking good! At first I was exploring the possibility of keeping a cached zip file and only adding newly converted audio files to it. But on my machine it takes only ~200ms to zip the entirety of IguaRPG's sound files to a new zip file.

W5. Library bugs, version conflicts, native dependencies...
I ran into an issue with one of the libraries I'm using, @parcel/watcher. I observed that when upgrading to version 2.3.0, it was possible for one of the functions exposed to me to hang infinitely. I did some digging and found this commit: https://github.com/parcel-bundler/watcher/commit/9b7c657df8ad1c0d020541eae0dd38b921c94fb8 ...And if I upgrade to that particular version, I also experience the hanging. I tried to create a small repro so that I could report an issue, but to no avail. I am guessing they introduced a deadlock, but I have no idea how to read and/or debug C++.

@parcel/watcher is supposed to provide performant file system events. The library is even used by VS Code! I would like to use it in this project. But, I don't think it's a good idea for smooch to require a specific version of the library...

Imagine we have this package.json in our IguaRPG 2 project:

Code: [Select]

{
   ...
   "devDependencies": {
      "smooch": "^1.0.0",
      "parcel": "^9.0.0",
   }
}
Both packages depend on @parcel/watcher. smooch requires a specific version 2.2.0 due to the aforementioned bug, but parcel might need something higher than that. npm will be unable to resolve this conflict!

This would be no issue if @parcel/watcher had no native dependencies. I could simply pack all of its JavaScript code into my smooch bundle, and there are now no worries about conflicting dependencies.

But because of the native dependencies, I have two choices:
1. Download specific versions of their binaries for all supported platforms, and include them in my repository and require them from my bundle
2. Install the native dependencies using npm at runtime, and place them in a special folder that only smooch is aware of

I went with 2, even though I'm pretty sure it's unorthodox AF. It actually required surprisingly little code: https://github.com/hubol/smooch/blob/f663e32a20f9d0e7486e1150051b449f4001a462/lib/common/native/native-dependency.ts

The NativeDependencies class exposes isInstalled, install, and require methods. And I think they look pretty nice! I also did some crazy shit so that the correct API of the package is returned from the require method (-:

But yeah, now these dependencies are installed to a separate package at runtime, ensuring that the desired version is installed with the correct native dependency! I'm glad that I was able to accomplish this with very little code and by leveraging shit that already exists!

[SquareWheel]

That's awesome, thanks for writing all that out.

Windows paths using \ is the worst ever.  It's one of those 30 years ago decisions to allow a certain flag syntax: "program.exe /flag", but now lives with us forever.  Between that, case sensitivity, and \r\n, cross-platform development has lots of interesting pitfalls.

I think it's not terribly uncommon for people to pin specific versions of deps, but I think you're right that it is error-prone.  It's easy to come back a year later and be confused why updating your deps has exploded everything.  Tough situation when there's no upstream fix though.

What's zipping the audio files for?  I figure it can't be adding any more compression, so just for packaging?

I've avoided playing with TS mostly because I haven't needed to, but I secretly hold the idea that eventually JS will get optional typing and TS will become more of a niche utility for its extra features at that time.  In the same way that LESS and SASS have fallen away as CSS has become more powerful, we seem to move towards fewer steps and less bundling where possible.

[dcco]

If JS gets some kind of optional typing that would be dope bc I agree it's one of those things that should just be in the language. I feel like Java was kind of that way wrt Scala where after I adopted Scala, Java slowly began getting more and more features (first-order functions, less type annotations) that would maybe bring me back to it. BUT TS is still kinda nutty in terms of what it can do imo (although I'm really not a fan of how the 'any' type works) so idk JS has its work cut out for it imo.

[dcco]

I want to see it!!!!!!!!!!! Do it!!!!!!

I actually did start working on a language for game stuff - doesn't have any of the stuff I was talking about, but it *does* have an actual type system which I'm playing around with a lot. Main features are ADTs and also a function overload system that distinguishes functions of the same name by the argument of the first type. (There will also eventually be polymorphism but I haven't needed it yet). Basically I'm messing around with various type design decisions to see what works and what doesn't.

(syntax highlighter scuffed bc it's hard to keep up with it while actively making the language :U)