xoreos Not-Thanksgiving 2016

xoreos is a FLOSS project aiming to reimplement BioWare’s Aurora engine (and derivatives), covering their games starting with Neverwinter Nights and potentially up to Dragon Age II. This post gives a short update on the current progress.

Note: This is a cross-post of a news item on the xoreos website.

And again a year is nearing its end. Like last year and the year before, I’d like to turn my gaze inwards.

A lot of things happened with xoreos this past year, albeit most of them hidden and “under the hood”:

  • I wrote about disassembling NWScript bytecode. The tasks I mentioned there are still open, too. If anybody wants to take them up, I’d be happy to explain them in more detail :).
  • We released xoreos 0.0.4, nicknamed “Chodo”. That was the only release of xoreos in 2016. xoreos 0.0.4 included some minor fixes and features for Neverwinter Nights, and the xoreos-tools package included the new NWScript disassembler.
  • In April, I reached a streak of a full year of daily xoreos commits. Due to some real life things, I had to take a break there, though. I’m now again at three months of daily commits, but there is a three-month “hole” between April and August.
GitHub contribution graph in April

GitHub contribution graph in April

 

GitHub contribution graph in November

GitHub contribution graph in November

  • Farmboy0 fleshed out the Jade Empire engine a bit, mostly in the scripts department.
  • Supermanu implemented a huge chunk of the character generator for Neverwinter Nights.
  • Farmboy0 fixed a glitch in the Neverwinter Nights animation system that has plagued xoreos for quite some time: the animation scaling in various creature models was off. This lead to, for example, the head and arms of elves detaching from the body during the yawn animation.
  • I then implemented a few more animation script functions, too, which is especially noticeable in the intro animation for Hordes of the Underdark. I also fixed a mistake in the keyframe interpolation. This takes care of another glitch in Neverwinter Nights: model nodes rotating the wrong way around.
  • smbas added support for Lua scripts in The Witcher. A lot of the initialization code that sets up the classes and functions The Witcher expects to find is still missing, so nothing obvious is visible as of yet.
  • Farmboy0 moved the window handling from the GraphicsManager into a new WindowManager class, making the code more readable.
  • I fundamentally restructured our build system, or at least the autotools part of it (xoreos can be built using either autotools or CMake). Previously, we used a recursive autotools setup, where make recurses into each subdirectory. This is, unfortunately, pretty slow, among other drawbacks. I changed it to be non-recursive now, with the top-level Makefile instead being created using (recursive) includes.
  • I then introduced various smart pointer templates into the codebase, making it easier to read and easier to keep track of memory allocations.
  • berenm added AppVeyor integration. Like Travis CI (which we already use as well), AppVeyor is a continuous integration service. This means that every single commit to the public xoreos repository will now be built on Microsoft Windows, using Microsoft Visual Studio 2015, in addition to gcc and clang on GNU/Linux (via Travis CI). This ensures that any compilation breakage on these systems is immediately visible and can be fixed at once.
  • GitHub added a new feature, “Projects”, that provide Kanban-like boards of tasks. I took the time to fill the xoreos Projects page with boards for tasks from our TODO list.
  • There were of course also various clean-ups, minor fixes and expanded code documentation.
Animation with glitch

Animation with glitch

Animation without glitch

Animation without glitch

Animations in the HotU intro

Animations in the HotU intro

Additionally, there are several tasks currently being worked on, among them:

  • Supermanu is looking into pathfinding.
  • mirv is still working on rewriting the OpenGL renderer.
  • I am currently writing unit tests for the xoreos codebase, using Google Test. I already found multiple issues, bugs, and corner cases while adding them.

From my side of things, my current plan is to make my unit tests branch public some time in December. I’ll write a small announcement here about it then. A new release of xoreos, 0.0.5, should follow early next year.

As always, this all wouldn’t have been possible without a lot of people. For them I am thankful.

  • Farmboy0, for various fixes, implementations and file format spelunking.
  • Supermanu, for his character generator work and pathfinding research.
  • mirv, for continuing to work on the OpenGL rewrite.
  • smbas, for his work on Lua and The Witcher.
  • berenm, for the AppVeyor integration and CMake knowledge.
  • TC01, for writing a Fedora specfile for the xoreos projects.
  • CromFr, for taking a stab at the walkmesh structure in NWN2’s TRN files.
  • clone2727, for invaluable ideas and corrections.
  • The folks at GamingOnLinux, who continue to be a great resource for all things related to Games on Linux.

I am also thankful for all the people who take the time to explain things to others, people who write interesting, useful or needed articles, and people who provide mentoring and help. Relatedly: a week ago, Stephanie Hurlburt published an article with engineers who are willing to mentor or answer programming/engineering questions. I for one think that’s a really great idea. Please take a look at that article.

And now, let’s see what the next year has in store for us. If you, however, found all this terribly interesting and would like to help with our little project, then please, feel free to contact us! 🙂

xoreos Not-Thanksgiving 2015

xoreos is a FLOSS project aiming to reimplement BioWare’s Aurora engine (and derivatives), covering their games starting with Neverwinter Nights and potentially up to Dragon Age II. This post gives a short update on the current progress.

Note: This is a cross-post of a news item on the xoreos website.

The end of the year is approaching fast, and just like last year, I want to use this time for some retrospection.

First of all, what happened in the last year?

  • berenm added support for building xoreos with CMake, by the way of parsing the automake files used for the autotools build system. This way, xoreos can now be built with either CMake or autotools. I was skeptical at first, especially since I harbour no love for CMake, but it is working reasonably well and I am quite happy with it. In hindsight, I was wrong to reject this pull request for so long.
  • I focused on supporting all the different model formats used in the Aurora games, and then I made all the games display their in-game areas with objects.
  • xoreos adopted the Contributor Covenant as its Code of Conduct, in the hopes that it helps foster a friendly and welcoming community.
  • The big one: our first official release, xoreos 0.0.2, nicknamed “Aribeth”.
  • I overhauled the script system, making it more generic. This way, I was able to apply it to all targeted games, except Sonic Chronicles: The Dark Brotherhood (which doesn’t seem to use any scripts at all). This included figuring out and implementing four new script bytecode opcodes: two for array access in Dragon Age: Origins, and two for reference creation in Dragon Age II.
  • I implemented reflective environment mapping for Neverwinter Nights and the two Knights of the Old Republic games.
  • I added a new tool to the xoreos-tools package: xml2tlk, which can recreate TLK talk table files out of XML files created by tlk2xml.
  • With these changes, I decided to push out xoreos 0.0.3, nicknamed “Bastila”.

This is all old news, more or less already discussed in previous blog posts. However, since then, I added yet another new tool to the xoreos-tools package: ncsdis. It’s a disassembler for NCS files, the stack-based compiled bytecode of the C-like NWScript, BioWare’s scripting language used throughout their Aurora-based games.

It basically replaces the disassembler within the old OpenKnightsN WScript compiler, with various added benefits. I’ll write a bit more about this tool in the near future, so for now I’ll just leave you with an example assembly listing it can produce, as well as a control flow graph it can create (with the help of Graphviz). As you can see, it already groups the instruction by blocks and subroutines. It performs a static analysis of the stack (to figure out subroutine parameters and return types) and it also analyzes the control flow to detect assorted control structures (loops, if/else). I plan to grow it into a full-fledged NWScript decompiler.

Additionally, I also added support for BioWare’s Neverwinter Nights premium modules, like Kingmaker, to xoreos.

On the documentation side of things,

  • I added comments and documentation to various files in the xoreos sources, hopefully making them more understandable and useful for potential new contributors and otherwise interested people. Considering how awful my memory is at, this is also a kind of future-proofing.
  • Farmboy0 added “research” subpages for various games on our wiki, filling them with information about their workings.
  • I extended our TODO list considerably.
  • I added an example configuration file, and extended the documentation on the wiki on how to compile and run xoreos.
  • I wrote man pages for each tool in xoreos and for xoreos itself. I also added the former to the wiki.

Phew! This is again a bigger list than I had anticipated. This wouldn’t have been possible without these people, for whom I am thankful:

  • I am thankful to berenm for providing the CMake bindings, despite my grumbling about it.
  • I am thankful to Supermanu, for continuing on chipping away on the Neverwinter Nights character generator.
  • I am thankful to Farmboy0, for working on xoreos’ Jade Empire engine and researching game internals.
  • I am thankful to mirv, for continuing with the huge task of rewriting my naive OpenGL code.
  • I am thankful to Coraline Ada Ehmke for creating the Contributor Covenant.
  • I am thankful to all the people in the different BioWare modding communities, for having figured out many different things already. Skywing for example, who had emailed me a few years ago about certain NWScript issues, issues I recently stumbled over again.
  • I am thankful to fuzzie, for giving me pointers on the NCS disassembler/decompiler.
  • I am thankful to the GamingOnLinux people, who do a lot of work reporting on all sorts of Linux-related gaming news, and who so graciously mirror my xoreos blog posts.
  • I am thankful to kevL, for notifying me of issues with xoreos’ build system on configurations I hadn’t thought about.
  • I am thankful to clone2727, for putting up with rants and ravings.
  • I am thankful to all the people who told me when I was wrong, for example when I wrongheadedly silenced clang static analyzer warnings, without understanding what I was doing.
  • I am thankful to everybody else who gave me hints and tips, taught me tricks and procedure, showed me new things, old things, forgotten things, broken things.
  • I am thankful to all the people who are not angry with me for forgetting them, because they are aware that this is not meant as a personal slight ;).

Now that I have these mushy feelings out of my system, here’s hoping for another great year! 🙂

And like always, if you want to join our effort, please don’t hesitate to contact us!

Interview for the Neverwinter Nights Podcast

(Cross-posted from the xoreos website)

This past Saturday, I was being interviewed for the Neverwinter Nights Podcast about the xoreos project. I talked a bit how this project got started, what I’m focusing on and about the current situation.

Not being used to being interviewed, or speaking English for that matter, my nervousness is quite audible. But I hope I still got my points across. 🙂

So, if you’re interested, give it a listen.

xoreos needs your help

TL;DR: xoreos needs contributors badly. Especially OpenGL wizards. I might give you cake.

(If you haven’t heard about xoreos yet, please read the short introduction on its website, or my previous blog post about it before continuing with this post. Thanks.)

The xoreos logo: the word "xoreos" rendered in neon green

People who are following xoreos a bit will already have noticed this: development has slowed considerably.

This has several causes:
1) I’m a bit busy-ish with real life
2) I’ve got (too many) other projects
3) Motivation is a bit low

Continue reading

Yearly Update, Part II: eos

[Minor Update: Because of name clashes, eos has been renamed to xoreos.]

[Major Update: xoreos needs your help!]

As promised in my last post, I will now talk about eos.

“What is eos?”, I hear you ask. Well, in short, it’s an open source project I started to portably reimplement BioWare‘s 3D engines, starting with Neverwinter Night‘s Aurora engine. Quite a daring task, and one I can’t begin to hope to finish on my own, especially since I’m not really that versed in all that 3D stuff. But working on it is fun and scratches an itch, and maybe more people will join me in that particular quest some day.

Continue reading