I did implement one of them, Diving, a month later. But the other one, Penetration, lingered in its unimplemented state for another 5 month. Until about two weeks ago, when the urge to finally have Geisha done became too great for me to handle. Which then lead to me actually working on the Gob engine again, and since last Saturday, Geisha is officially supported by the daily build of ScummVM (and will be in the coming ScummVM 1.5.0 release).

There was just one tiny thing still bothering me: The missing AdLib music, in MDY/TBR format (.mdy files containing the music score, .tbr files containing the instrument parameters). I’m a sucker for a good AdLib (and SID, MOD, …) tune.

Now, like Fascination, Geisha contains a PCM-based alternative, but it’s far from sounding as awesome as proper AdLib music. We did have some code intending to play those MDY files in the repository, but it was hopelessly broken. Moreover, this code and the ADL (a later Coktel Vision AdLib music file format) player’s was in a very shoddy and undocumented.

Therefore, I decided to just rewrite it completely, which I then did over the weekend. The result is a new working MDY player, a fixed glitch in the ADL player, and, maybe most importantly, clean and documented code for the Gob AdLib stuff.

And what is also very interesting is that I found out that the MDY/TBR format is in actuality the MUS/SND format — apparently created by Ad Lib as a more simple alternative to the ROL format. This format has been used in other, non-Coktel-Vision, games as well, so it might just be that it can be reused by potential future ScummVM engines.

Just shows how what you think is a straight-forward task can still surprise you.

As promised in my last post, I will now talk about eos.

“What is eos?”, I hear you ask. Well, in short, it’s an open source project I started to portably reimplement BioWare‘s 3D engines, starting with Neverwinter Night‘s Aurora engine. Quite a daring task, and one I can’t begin to hope to finish on my own, especially since I’m not really that versed in all that 3D stuff. But working on it is fun and scratches an itch, and maybe more people will join me in that particular quest some day.

Here’s a quick rundown on the current state of eos:

General

Working:

• “Engine” framework
• Global configuration subsystem
• Debug console with readline-like abilities
• Reading classes for a shockful of BioWare file formats, including archives, images, GFF formats
• Resource manager
• Basic manager classes for 3D graphics and sound
• Playing of Bink videos (decoder courtesy of the FFmpeg project)
• XMV demuxer (which we then reworked to fit into FFmpeg)

Missing:

• Animation subsystem
• 3D positional audio
• WMV p-frames

Neverwinter Nights

Working:

• Partially working main menu
• Loading of modules and areas, including placeables and creatures
• Moving around in areas
• Interacting with doors
• Talking to NPCs
• Script execution

Missing:

• Area lighting
• Most script functions
• Proper “segmented” models (each body part a separate model)
• PC, walking, combat, inventory, journal
• Script triggers

Knights of the Old Republic (1 + 2)

Working:

• Very stubby main menu
• Loading of areas, including placeables and creatures
• Moving around in areas
• Unswizzling Xbox textures

Missing:

• Interacting with objects, dialogs
• Scripts
• PC, walking, combat, inventory, journal
• Area lighting
• Proper materials, including metallic reflections

Jade Empire

Menu loading works partially, everything else is still missing.

Neverwinter Nights 2

Model loading works partially, everything else is still missing.

The Witcher

Model loading works partially, everything else is still missing.

Dragon Age: Origins

Texture loading works, everything else is still missing.

Sonic Chronicles: The Dark Brotherhood

HERF archive reading and texture loading works, everything else is missing.

Dragon Age 2

A potential target, but made difficult due to the changed archive format which doesn’t include file names anymore, only their hashes.

As can be seen, the current focus is mostly on Neverwinter Nights and Knights of the Old Republic, but even there, a lot is still missing and the games are of course far from being actually playable. I am mostly working alone on that project, with help from clone2727; chances of eos ever becoming usable are next to none this way.

If anyone out there is interested in helping, this would increase said chances and we’d be overjoyed. I’m especially looking for someone with actual OpenGL experience who could probably completely rewrite my graphics code to be far more effective. Of course, there’s a lot of other places that need capable hands. A quick and dirty TODO I’ve written a while ago can be found here. The project itself is located on github, here.

So, without further ado, let’s dive right into ScummVM with…

Urban Runner

Since my last post on Urban Runner, a lot has happened: The hotel lock-up has been fixed (thanks to SylvianTV), saving has been implemented and a handful of glitches has been stomped. As of ScummVM 1.3.0 (May 28, 2011), Urban Runner is fully working and supported.

Another problem that showed later was that saving and loading didn’t work on the PSP. After investigating the issue, we found the cause: the number of file handles on the PSP is severely limited. Since I’m opening all manually open all archive files on startup — the CD detection script doesn’t work when all data files are in one directory and the NO_CD mode requires all data files to be present in CD1.ITK — this already maxed out the available file handles. Luckily, I was able to resolve that by not manually opening CD1.ITK; the scripts already open it, but I, wrongly, feared they might try to close and reopen it.

Adibou

After Urban Runner was working, I started investigating the script opcodes needed for the Adi and Adibou (Addy and Addy Junior in German) educational games. Among them are opcodes to read string and integer values out of INI and xBase database files, for which I wrote quick loader classes. After a few days, I managed to get the environment CDs (which ship with each game to serve as a hub to start the individual lessons and provide some quick games to play in between) partially playable. With a lot of glitches, of course.

Geisha

The next Coktel Vision game I’ve turned to is Geisha, an erotic puzzle adventure (or rather, a lot of puzzles embedded into a thin layer of adventuring). Under the hood, it’s very similar to Fascination, so it seemed like a low hanging fruit to finish. In fact, it is now already completable in ScummVM. The only thing missing is two hardcoded minigames, Diving and Penetration. I also already implemented about half of Diving; Penetration, however, is still completely unimplemented.

Still missing as well is a player for the AdLib music format MDY/TBR (the former containing the track, the latter the instrument definitions). Like in Fascination, there is exactly one MDY track in Geisha, the title music. And like in Fascination, there is a PCM-based alternative, which is normally played when there is no AdLib compatible sound card, and on the Atari ST. Right now, ScummVM always plays that alternative in both games, but it would be nice to have the AdLib music working too. Strangerke already started writing an MDY player, but it’s currently still not working properly.

Being an erotic adventure, it does feature full nudity. I’d like to keep this blog somewhat family-friendly, so I’m not embedding any videos or screenshots.

Dark Seed II

One game I did not make any real progress on is Dark Seed II. The scripts seem to work somehow differently to how I thought they would. Right now, they lock up in at least two places early on: when talking to the fortune teller and when entering the mirror labyrinth (both located on the carnival grounds). Moreover, I can’t seem to understand the numerical parameters accompanying a lot of the opcodes, and I’m not sure about the evaluation order of the different functions.

Likewise, the walking code is still wrong, as I can’t find the correct ways to evaluate the walkmap scaling parameter, and the character scaling factor. The algorithm I use is also probably not the optimal one for the provided walkmaps.

Nearly all the code (except the archive decompression) I have written is not based on reverse engineering, but educated guesses, and in those cases, I apparently guessed wrong and am not “educated” enough to guess correctly. To fix this, I will have to actually reverse the binary.

As an interesting side note, we found out that there is a Japanese Sega Saturn version of Dark Seed II. The archives are organized a bit differently, and the movies are Cinepak instead of MSVideo1 and Indeo3, but adding support for the former was easy and clone2727 already had a decoder for the latter, so the state of the version in ScummVM is similar to the DOS versions. The only missing features are proper line wrapping according to Japanese grammar rules (instead of just allowing wrapping after each character), the different dialog interface (the Japanese version plays more like a Japanese graphical novel) and the Saturn-specific music format. If anyone has any information on that music format, I’d be very much interested.

Update: Thanks to peres for informing me that the last character in each line was wrong (I can’t read Japanese, so I hadn’t noticed). It turned out to be a bug in my line trimming functions and is now fixed. The screenshot above has been changed to reflex the fix. For comparison, here’s the broken version.

Conclusion

2011 was a bit of a slow year for me when it comes to ScummVM. I fixed a few bugs, started a few things, but didn’t really finish anything (except for Urban Runner, but the lion’s share there was done in 2010).
I’ll talk more about what I’ve done in the other big project I was involved in in the next post.

After ScummVM got 16bit support for GSoC 2009, I chose to wait a bit before adapting the gob engine, since a lot of other code was entangled into the problem, including the CoktelVideo classes and the Indeo3 decoder.
But then clone2727 came along and completely revamped the VideoDecoder interface, making it 16bit-ready. So I rewrote the IMD/VMD classes to fit that interface accordingly. Next, I rewrote the gob engine drawing functions to support 16bit surfaces as well, and plugged everything together to get Urban Runner’s Indeo3 VMDs running in 16bit colour mode.

Unfortunately, I then hit a road block. One big thing still missing for a full 16bit Urban Runner experience was still high-color non-Indeo3′d VMD videos. I tried several approaches, but after the CoktelDecoder rewrite, they either didn’t work anymore or were dead-ugly. Becoming frustrated quickly, I started several other projects, handled some RL problems, etc., basically stopping working on it completely for several months.

Until yesterday. I found a way to implement the missing pieces, fixed a few smaller issue, and can now lay all that all to rest. So, I officially declare my dithering “series” dead .

Now, what’s next?
First of all, the one big problem in Urban Runner, the hotel lock-up, is still present. However, a guy by the handle of SylvainTV took a good look and gave me some valuable hints: The videos are supposed to play “in the background”, while the script execution continues. According to him, he hacked in a small proof of concept, and with that, the lock-up is gone. Of course, I need to find the proper way of hooking that into the engine, otherwise I invite all hell of strange bugs . I already have a few suspicious I need to check out.

Another glitch that’s bugging me is how the sprites aren’t completely correct yet. All sprite data in Urban Runner is in YUV (or YUV840, I guess you’d call it, since each 4×4 block of Y has one U and V value) and I apparently haven’t figured the exact format out yet. As you can see in the following screenshot, it looks like one colour channel is a bit offset to the lower right. The original does not feature that glitch, so it’s obviously my fault.

Urban Runner menu, 3x magnified

Anyone who might me interested in looking over my code and fiddling a bit with it, please contact me.

UPDATE: Thanks, again, to SylvainTV, the YUV glitch is fixed.

Urban Runner menu, 3x magnified, fixed

More seriously, the good news came very recently: in less than two weeks, with the impressive help of SylvainTV, all the versions of Fascination became playable and completable, or at least all the ones I possess.

The main thing missing currently, only used by PC versions of the game, is the AdLib player (and IMHO, the SoundBlaster sounds a lot better). There’s a wrong delay (too short) between rooms in several places of most versions. This seems to be caused by the delay required at the time to load from the floppies, and it’s easily patchable.

Last detail, the (rare) CD version still requires a tool for the STK extraction. joostp is cleaning it up, and will commit it when it’ll be in a good state.

Maybe it’s a good time now to grab your originals and send us some feedback on the game, specially if you find some of those too short delays!

Here are some screenshots of the CD version.

Stay tuned for further news! I promise I’ll try to give some before this summer

The problem was, of course, my own damn fault; I misunderstood the nature of cursor resources. They’re not, as I assumed, plain cursors, one by one each taking up a resource entry. Instead, there are cursor groups, each capable of possessing a variable amount of cursor images. That isn’t even a quirk of NE files, it’s like that in PE files as we well, so my complaining about the executables being in that old format doesn’t apply either. Mea culpa.

In Dark Seed II, the cursors are embedded into the resource part of the Windows 3.x (NE, “New Executable”) executable dark0001.exe. Specs are available on the Microsoft website, but they don’t quite match the executable, which claims that each cursor is only 32 byte big.

I did manage to extract the cursors in the end, with the help of the great tool “Multi Ripper” (a project that sadly appears to be dead). But when I then began to look for the resulting cursors in the Dark Seed executable, I couldn’t find them there, at least not directly: The headers are, only partly though, in the resource table, where you would expect them. However, the cursors’ data are at a completely different place, a place that I couldn’t construct with any of the offsets I read from the various tables. So I am completely at a loss.

Running Multi Ripper through a disassembler didn’t really help either, since it’s either deliberately obfuscated, or just completely unreadable due to it being written mostly in Clipper (yeah o_O). The link to a wine resource compiler Fingolfin send me didn’t help either (thanks again for looking, though); that compiler agrees with neither the specs, nor with the executable.

Since I was able to at least extract the cursors, I did what I had to do: hard-code the cursor data. So right now, the Dark Seed II engine comes with about 10kb of static cursor data. Not much of a problem, but certainly not as “nice” and clean as I’d like.

Well, if any of you happens to know a thing or two about resource tables in New Executables, please, pipe in and tell me what I did horribly wrong.