Yearly Update, Part II: eos

[Minor Update: Because of name clashes, eos has been renamed to xoreos.]

[Major Update: xoreos needs your help!]

As promised in my last post, I will now talk about eos.

“What is eos?”, I hear you ask. Well, in short, it’s an open source project I started to portably reimplement BioWare‘s 3D engines, starting with Neverwinter Night‘s Aurora engine. Quite a daring task, and one I can’t begin to hope to finish on my own, especially since I’m not really that versed in all that 3D stuff. But working on it is fun and scratches an itch, and maybe more people will join me in that particular quest some day.

Here’s a quick rundown on the current state of eos:

General

Working:

  • “Engine” framework
  • Global configuration subsystem
  • Debug console with readline-like abilities
  • Reading classes for a shockful of BioWare file formats, including archives, images, GFF formats
  • Resource manager
  • Basic manager classes for 3D graphics and sound
  • Playing of Bink videos (decoder courtesy of the FFmpeg project)
  • XMV demuxer (which we then reworked to fit into FFmpeg)

Missing:

  • Animation subsystem
  • 3D positional audio
  • WMV p-frames

Neverwinter Nights

Working:

  • Partially working main menu
  • Loading of modules and areas, including placeables and creatures
  • Moving around in areas
  • Interacting with doors
  • Talking to NPCs
  • Script execution

Missing:

  • Area lighting
  • Most script functions
  • Proper “segmented” models (each body part a separate model)
  • PC, walking, combat, inventory, journal
  • Script triggers

Knights of the Old Republic (1 + 2)

Working:

  • Very stubby main menu
  • Loading of areas, including placeables and creatures
  • Moving around in areas
  • Unswizzling Xbox textures

Missing:

  • Interacting with objects, dialogs
  • Scripts
  • PC, walking, combat, inventory, journal
  • Area lighting
  • Proper materials, including metallic reflections

Jade Empire

Menu loading works partially, everything else is still missing.

Neverwinter Nights 2

Model loading works partially, everything else is still missing.

The Witcher

Model loading works partially, everything else is still missing.

Dragon Age: Origins

Texture loading works, everything else is still missing.

Sonic Chronicles: The Dark Brotherhood

HERF archive reading and texture loading works, everything else is missing.

Dragon Age 2

A potential target, but made difficult due to the changed archive format which doesn’t include file names anymore, only their hashes.

 

As can be seen, the current focus is mostly on Neverwinter Nights and Knights of the Old Republic, but even there, a lot is still missing and the games are of course far from being actually playable. I am mostly working alone on that project, with help from clone2727; chances of eos ever becoming usable are next to none this way.

If anyone out there is interested in helping, this would increase said chances and we’d be overjoyed. I’m especially looking for someone with actual OpenGL experience who could probably completely rewrite my graphics code to be far more effective. Of course, there’s a lot of other places that need capable hands. A quick and dirty TODO I’ve written a while ago can be found here. The project itself is located on github, here.

flattr this!

  1. eyebex says:

    Wow, great stuff, looks like a real fun project! :-) A quick look at the source code suggests that you’re only targeting Linux. Is that correct, or are you targeting Windows, too? Granted, targeting Linux makes more sense as the original games run on Windows anyway …

    • DrMcCoy says:

      Well, my machines are Linux-only, yes. But eos is supposed to be portable, and compile and run on at least Mac OS X and Windows too.
      In fact, my partner in crime clone2727 is a Mac-guy and eos works just fine there. As for Windows, he also had a Windows compile box in the past, and eos was running fine there as well. Granted, there has been quite some changes in the code base since then, but I don’t expect any major problems.

      With a few changes mainly in filesystem related parts, eos should also work on other systems, provided the library dependencies are met (SDL, OpenGL, OpenAL, Boost, and various libraries for decoding audio formats).

      • clone2727 says:

        Yes, it used to work on Windows. I need to go fix my Windows build environment though because I (accidentally) trashed it when trying to update autoconf when DrMcCoy required a newer version at one point or something :P As DrMcCoy said, it works on Mac OS X (at least Snow Leopard).

  2. NSpacer says:

    Love it! Very excited about the potential here. I also had the itch after firing up Gemrb and wondering where the Aurora version would pop up. Would love to help but my hardcore programming skills are an untapped potential. If there is any grunt work that you want to farm out, I would love to contribute in any way I can.

  3. murlock says:

    I’m very impressed by your work !

  4. WallAR says:

    Hey! IT IS AWESOME! I’ve been looking for NWN opensource engine for years!!! I’m looking forward to future ;-) I mean OpenMW and now eos. It’s just awesome. I can imagine that people that today still working on nwn mods could gain such powerfull tool to make changes in game ;-) I

  5. Richard P says:

    Holy sheckles! I am gobsmacked! I hadn’t heard of this project till today and I’m so glad that I did. This is amazing work! I’ve always loved playing NWN on Linux and have played it for over 3 years. DM’d for a while and created and scripted on modules. Glad to see it getting a second lease on life.

    will download and try to contribute where I can. I’m a seasoned developer on practically all procedural and OO languages, but haven’t done much 3D and audio. That’s about to change though I have to admit this is probably going to be a steep learning curve. Hopefully there’ll be documentation and architectural designs and tech specs.

  6. Richard P says:

    Err, where can the source be downloaded from?

  7. Ryein says:

    Whats the point of this? You have done a good job, but why do this?

    • DrMcCoy says:

      Because it “scratches an itch”.
      Because it’s fun.
      Because I like crawling through disassembly and hex dumps.
      Because I like solving puzzles.
      Because I can learn from it.
      Because I love the games.
      Because I’d to play them on my Linux box without resorting to wine.
      Because I want them to be able to run on platforms they normally wouldn’t.
      Because I want to preserve them for future generations.

      That enough for you? :)

  8. peachykeen says:

    Hey, just saw this project. It looks awesome, and you’ve certainly gotten further than anyone before. It’s some pretty impressive work.

    I did notice you have scripting and lighting, among other things, on the todo list. I don’t know if you’ve seen any of it, but there’s a ton of info the devs released that’s helped a few people take apart the original engines. Among the results are a new script compiler and runtime (by Skywing), and I did quite a bit of research into the graphics for NWShader. I’m sure everyone would be more than happy to provide any info we can to further your work. If you shoot me an email or drop in to the chatspike #nwn2cr IRC channel, there are a few people who would be *very* interested.

    I also am curious about how you’re doing the graphics (I assume OpenGL). NWN1′s method was a mess, so anything would be better. :P

    Awesome work so far, and best of luck.
    Cheers, peachykeen

    • DrMcCoy says:

      Well, I do already have a rudimentary script interpreter for NWScript, what’s missing is most of the engine functions. And it’s only working for NWN right now, the extensions added in the other games are not implemented yet, but I do know most of what’s needed for that.

      I also started implementing lighting, by checking for every tile, which 8 lights are closest:
      http://drmccoy.de/eos/nwn/20110610T230251.png
      While this seems to work in most indoor areas, it breaks down when there’s too many lights around (like 9 lights for each tile in a neat grid in default desert areas), you see the tile boundaries in the lighting differences of each tile. I need info on how the original actually does lighting.

      As for the graphics, yes, it’s of course OpenGL. I’ve written a rather complicated rendering pipeline using an abstract Renderable class, and that seems to work, but I fear it’s not the most performant way to do it (and could maybe be described as a bit of a mess too, I’m afraid). I also am not that knowlegable in OpenGL in the first place, I mostly muddle my way through; someone with proper OpenGL experience should probably completely redo my code. :P

      • peachykeen says:

        The lighting in the original is kind of a pain. As best as i was able to figure out, it’s based mostly on the nearest 8 lights. Boundaries will be visible, there isn’t much you can do about that without upgrading to per-pixel.

        Unfortunately, it’s not just the fixed-function lighting; implementing that precisely doesn’t match their lighting solution. I’ll have to dig back through the logs, but I want to say they brighten things more than usual. In addition, some things like vertex colors and texture alpha are used to control lighting or which texture is used. It wasn’t a pleasant system to take apart, that’s for sure. I’ll see if I can get you more details.

        On the renderer, they did originally use an abstracted-out renderer. It’s not terribly complicated, though. The main source of slowdown was passing a large number of vertexes across every frame and not caching them; if you can do any vertex buffering speed should improve greatly. Only a few types of objects (skinmeshes and jiggle, mostly) need to pass geometry. However, using proper shaders for both of those will fix the second slowest part of the engine.

        One major hangup, and the reason NWN2 runs incredibly slow, is the interface for the render module was designed for immediate-mode and later they mashed buffers into it, not to mention the OpenGL to Direct3D shift. Definitely something to watch for when putting things together; you won’t be able to recreate each game’s render style precisely if you want it to run at all. :P NWN2 was set up much better, sort of.

        Anyway, if I can dig out anything good, I’ll definitely send it over, or if you have any specific questions, lemme know. :)

  9. doctoren says:

    A couple of years ago I was looking into creating a resource manager for OGRE3D which would convert NWN MDL to an OGRE supported format in memory. I was actually thinking about giving it another go but could no longer find a digital copy of Torlack’s work. I did however still have a hardcopy of the pages. Nevertheless I was happy to see you’ve included Torlack’s documentation. On another note I’m really impressed at was already have been accomplished. I’ll definitely find some time to work on the code but I’m afraid that I won’t be able to contribute on the short term. I hope I might be at a later stage as this is perhaps the most awesome project I’ve come across in ages. I do own all the games and the thought of being able to play them natively on Linux is just amazing. Even though NWN has a Linux client its still not as straightforward to get up and running as I would like to.

  10. Trev says:

    Wow, looks very promising. I have a few questions though:

    1. Is this the client implementation or does it include / do you plan a server implementation too?

    2. Will this be multi-threaded to run on multiple processors / cores?

    3. Will it compile as 64bit (more than 4GB memory)?

    Just asking because I know some people running persistent world NWN servers and the 4GB / 1 Core limit really caps out scalability. If this project addresses those issues I think there might be several people interested in helping out.

    • DrMcCoy says:

      1. I do plan to also implement a server. Networking in general is quite low on the TODO for now, though
      2. It’s multi-threaded, yes. At the moment probably not really optimally though
      3. Yes, it does compile as a 64-bit binary

  11. Tess says:

    I realize that this is still early development and such, but is there a chance this project can be ported to Android, the way GemRB has been? Or are there technical constraints that would keep a 3D engine like this from ever running decently on mobile phones?

    Either way, this is really cool, and I think it’s great that you’re doing this! :)

    • DrMcCoy says:

      I think there might be a chance that it could be ported to mobile devices.
      I guess the “only” things we need to heed is that all libraries we depend on need to work (*should* be no problem, I hope), and that we’ve got OpenGL ES there, which has some limitations I understand.

      That said, I personally have no such device and I have no real clue about all the limitations on these devices, so take this with a grain of salt.

  12. Astromen Pete says:

    Kudos for doing this. I, for one, never found a graphical MUD quite like NWN that my friends and I could spend Sundays on.

  13. Antonio says:

    Ohhh! This is quite impressive! I’ll be looking this interesting project, thanks!

  14. Man0waR says:

    Hi DrMcoy.

    Just found your blog and project when i was looking for game engines. I’m a current player / DM / creator of a persistant world of neverwinter nights. Every day i take a look by the vault and the custom content challenge forum in nwn bioware social network.

    I must congratulate your awesome work and also have few question about your already explained goals on it.

    -is this project a sandbox to strip the mechanism of the engine aurora only for knowledge or could be a way to implement a better graphic engine attached to the core of the game? So we could have a new graphic engine but the same game. I know all of this is hardcoded but.. Could it be possible?

    - As I saw in your blog you are already put your hands in newest bioware projects like Dragon Age, could it be possible to extract the graphic engine and animations from the game in order to customize a new one? (I have no idea about this but im curious).

    All this questions are focused on a dream that many of the nwn users have. A Renewal version of the best RPG multiplayer game that can be found. Just an utopic example Dragon Age 1 graphics engine attached to the nwn 1 coreset. A forbidden dream?

    Thanks for your efforts and hope your work will grow in the nwn community.

    • DrMcCoy says:

      The main goals of the project are two-fold: Creating a portable reimplementation that allows you to play the games on other systems, mainly on Mac OS X and Linux; and for me to scratch an itch because I’m so inclined.

      This actually includes getting the games to run looking and feeling like they did orginally, and not with upgraded graphics. But while such thing would be very, very low priority for me, I would not object someone adding enhanced graphics as an optional improvement (apart from the usual objections for bad code design or utter disregard of the code formating style). Just keep in mind that I probably won’t use or develop them.

      Moreover, this will not be something you can do on a spare afternoon. It’s a major undertaking involving programmers, artists, designers, whathaveyou.
      Also, at this early stage of xoreos’ development, it’s complete lunacy to even think about doing this.

      “you are already put your hands in newest bioware projects like Dragon Age”

      Well, xoreos can read the Dragon Age resource archives and containing textures, that’s pretty much it. I did do a lot of reading up and fiddling with the other resources, though and have a broad overview on what’s different from older engine incarnations.

      I can therefore say with good confidence that there’s nearly no feasable way of getting the Dragon Age graphical subsystem to work together with NWN. Dragon Age just thinks in a lot of different concepts than NWN. You’re asking for bolting car doors onto a motorcycle.
      It will not be possible to change NWN into a radically different area organization and design paradigm. Or change how weapon and NPC model construction works. Or how walkmaps are only 2D. The most you can do is painting the bikeshed in a different color: create higher resolution models and textures, better animations, throw in a few advanced shaders and the like. And even then, you have to think hard about how to then deal with already existing community created art.

      And again, I personally would not like such a change. I like how NWN looks and am not a particular fan of Dragon Age’s overuse of the blur shader. :P

      • Man0waR says:

        Thank you for your quick response.

        I was curious on how the game works, i’m developer and i know the cost of your work in time and efforts.

        Anyways I’ll keep my dream of an enhanced modern engine running the NWN game. Good luck with eos :)

  1. [...] eos – otwartoźródłowy projekt, którego zadaniem jest ponowna implementacja silnika gry BioWare i zrobienie go przenośnym. Na początek autor zaczął pracę nad silnikiem Aurora od gry Neverwinter Nights. Aktualnie autor posiada framework "Engine", globalny podsystem konfiguracji, konsolę debugowania z możliwością wpisywania komend, działające klasy do obsługi formatu plików BioWare, w tym obrazków, archiwów, formaty GFF. Prócz tego menedżer zasobów, podstawowy menedżer klas dla grafiki 3D i dźwięku, działające filmy w grze (dekoder oparty o projekt FFmpeg), demuxer XMV. [...]