Next Generation Emulation banner

1 - 6 of 6 Posts

·
Premium Member
Joined
·
423 Posts
Discussion Starter #1
You may not realize this, but the during the last two weeks I've been working on XBE-loading almost every night (even during my holiday, no less!). In this period, I've stumbled over lots of little issues. Below a summary, for anyone interested;

First, let me tell you that the actual loading of the XBE sections turned out to be very easy; Using my XBE Explorer tool, I found there's no relocation information to apply, so I didn't have to rebase any code at all. So, after loading the XBE, I started our symbol-detection and it produced the same report as before - indicating this part "just works".

Before I reached this, I had to overcome a serious setback, however: It so happens that when Windows launches an executable, a page gets allocated at address $10000 (amongst a few others). I haven't looked into what this page actually contains, but in itself this prevents me from allocating this address myself. This fact effectively rendered my "single-executable" idea impossible right from the start.

To overcome this, I started on an alternative : I made a launcher executable, with an ImageBase of $10000 and an uninitialized data-section of 128 MB, with the sole purpose to reserve this as an Xbox-compatible virtual-memory range. When starting this executable, the Windows loader reserves the given virtual-memory range for me (while loading all other resources at higher addresses). Then, once loaded, this launcher does just 1 thing : transfer control over to my DLL containing the actual emulator-code.

This appears to be almost the same approach as before, but with a twist : Instead of creating a new executable for every XBE, I now have only one (17 KB small) 'placeholder' executable, which I can overwrite with any XBE that I can get my hands on!

I moved all GUI & configuration code over to the Dxbx kernel DLL, so this now runs in the same process (& thread) as the launcher. I simply removed all EXE-generation code and replaced it with the XBE-loading code I mentioned above. And lo and behold : at this point the GUI shows, I can trigger the XBE loading, do some controller configuration, dump XBE contents, etc.

So at this point, everything seemed all good and well, but once I actually tried launching a game, it all tumbled down again; I got non-responding code and exceptions all over the place.

After a few days of experimenting, I first discovered that the Windows API CreateThread crashes right after the DOSHeader and NTHeader structures are overwritten with XBE contents. I really want to have as much of the XBE contents in the same virtual-memory locations as on the actual Xbox1, so I started hacking on the DOSHeader/NTHeader structures.

After much trial-and-error I discovered that there are two little things that should be present : The DOSHeader.e_magic field (which should contain the 2-letter signature 'MZ'), and the DOSHeader. _lfanew field (which should point to a ImageBase-relative NTHeader structure). Luckily, the XBE isn't damaged much with these two fixups (and when needed, the original XBE contents could be swapped in temporarily, much like EmuSwapFS).

So I allocated a copy of the NTHeader structure, and let the _lfanew point to that, instead of the original location. This was enough to make CreateThread work again, but you can guess it already : that was not the end of my ordeal.

The next problem I encountered, was a call to CreateWindow that didn't return (the registered message-proc started handling messages alright) - weird!

After consulting with a colleague of mine, I realized that the NTHeader contains pointers to a few pieces of data that weren't there anymore. Sure, there's the launchers' original sections, but other things are relevant too, like TLS, resource blocks, BaseOfData, etc. So I had to fixup all that too - copying data and patching up the pointers.

Alas, this didn't help with the hanging CreateWindow API. So I tried something different : The CreateWindow call takes place in a new thread, but once I moved that back to the main thread, the call returned!
I still don't know why CreateWindow was hanging, and in the previous approach (the XBE>EXE conversion, like Cxbx does) this same code did work without any problem.... but at least I got one step further. (It might have something to do with the fact that the GUI is now running in the same process, but I don't see why that should be a problem).

Anyway, after that, initialization works completely, so I finally saw emulation starting again. The log gets filled with all the usual debug-messages (PSCreateSystemThreadEx, XapiThreadStartup, XapiIniProcess, RtlCreateHeap, RtlInitializeCriticalSection, etc) up until IDirect3D8_CreateDevice gets called... which failes with an "Invalid Call" message.

This is where I'm stuck right now, as I'm not to well-versed with Direct3D.

I suspect the API failures (CreateThread, CreateWindow, CreateDevice) are somehow all related to each other, and are either caused by the fact I now have the GUI in-process, or because some contents of the launcher EXE are still not relocated correctly.



Well, that's it for now, I hope you've enjoyed reading this. And if anyone has an idea or suggestion to help me out - please tell!

Cheers
- PatrickvL
 

·
Shadow of nothing...
Joined
·
6,071 Posts
I see that you're using the same techniques _SF_ used for Xeon. I've been reading this thread looking for _SF_'s old pixel shader conversion code when I was looking for a method to add pixel shader support for Cxbx. That particular post caught my interest as it was a clue/hint to how _SF_ got .xbe files to load/run without converting to a .exe like in Cxbx. Notice how he said that the purpose of Xeon.exe was to reserve the memory address 0x10000 - 0x4000000 (which is 64MB and more could be used to emulate a debug Xbox's memory, or worse yet, Sega Chihiro which can be up to 1GB from what I've seen). This is what I plan to do for Xenoborg now that I know what _SF_ did in general (I intentionally stopped working on it for a while until I could figure this out). Me and @ruantec were trying to figure out how to do this last night, but it appears that you've already done most of the hard work. I was going to try changing the image base to that address, but I forgot.

I do have a suggestion, but it may not be feasible as I haven't looked too far into it yet. I'm quite sure the DOS header is a fixed size, but what about the NT header? If it is, would it be possible you could lower the address to fit in the DOS and NT headers so they don't get corrupted? (example: Base Address = 0x10000 - sizeof( DOSHEADER ) - sizeof( NTHEADER )) I'd try that, but once again, I'm not to sure if that would work okay. Either way, this was an interesting read. Thanks for your efforts!

Shogun.
 

·
Premium Member
Joined
·
423 Posts
Discussion Starter #3
Well, tonight I finished the XBE Loading and committed it to our SVN.

After many failing attempts to fix the CreateDevice call, I reconsidered my options and went for a different approach, which looks a bit more like the original configuration;

Instead of having everything (GUI and emulation) in the DLL, I reverted the code to it's former state, so the Dxbx (GUI) executable and the DxbxKrnl (emulation) DLL where separate again.
To facilitate XBE Loading, I made a small Launcher application, who'se sole purpose it was to reserve the XBox1 Virtual Memory range (128 MB starting from $00010000) and transfer control to the DLL, much like before.
The DLL on it's turn is now capable of loading the complete XBE into memory, overwriting the Launcher application in the process. (As I wrote before, there's no real harm done once the proper NT Header fixups are in place.)

Compared to the 'conversion' approach, this is better because now we get almost full control over the loading of XBE's; Delayed loading of XBE sections can be supported, for example. Also, the theoretical case of having multiple of those delay-loaded sections in an XBE, all configured to load to the same Virtual Address on demand, can be made to work (not that I've done this already, but it's sure great that it's possible now).

With this approach, the problems I was having with CreateThread, CreateWindow and CreateDevice 'magically' disappeard. I still don't know they where failing previously, and now that I'm back to the (working!) "launch-the-XBE-in-another-process" approach, I lost interest in explaining the failures, to be honest.


While finetuning all this, I suddenly realised that the Launcher application doesn't have to be a separate executable; Instead, the Dxbx executable can just launch itself, passing on a few arguments via the command-line to boostrap the XBE loading, logging and screen-drawing code.
For this to work, the Dxbx executable itself should load to ImageBase $00010000 and reserve 128 Mb of virtual memory, just like the separate Launcher did. Also, the re-launched Dxbx executable should not start the GUI again, but transfer control to the DLL too. Luckily, this was all quite simple, I just had to merge a few lines of code back into Dxbx.exe.

With this last step, Dxbx became it's own XBE Loader, which I think is rather cool ! :cool:


Some might wonder why I went through all this trouble, so let me explain it one last time :

The main development of Dxbx takes place in the DxbxKrnl DLL. Every little change in this DLL results in a slightly different memory layout. Normally this won't be a problem, but because the XBE>EXE conversion code also has to fill in the Kernel Thunk Table, the converted EXE references fixed locations in the DLL; locations that won't be valid anymore after even the slightest change in the DLL code.
The new XBE Loading code no longer generates an executable, but 'just' loads it into memory. The offending Kernel Thunk Table also gets processed right then and there, so we no longer suffer from Kernel API location-mismatches.
As a bonus, the XBE Loading is much more controlled now, so I think it was worth it spending my time on.


Now this is working, I'll probably do some translation work for a change - one of these days I want Dxbx to show me some screenies, damn'it! ;)


PS: In the process I also started experimenting with Git, which is a real relief to work with compared to Subversion. But don't worry: I'll keep pushing milestones to SVN too!
 

·
Registered
Joined
·
10 Posts
I would like to point out one thing about your pre-allocated memory space:

Halo 1 and Halo 2 require specific memory addresses to be allocated for their game data. That is, their game data is specifically cached for a fixed base address in memory (which is above the typical 0x7FFEFFF user-space ceiling on NT machines). Both allocate a fixed buffer at 0x8001000 (or something to that affect).

Now, this isn't really much of an issue as you can change the amount of user-space memory to allow 0x8* and higher memory addresses (something I've done for projects dealing with Halo related stuff). However, I'm not sure how this will fit in with your pre-allocated memory. Maybe Xeon had a hack-around for this (since it was basically for Halo1)?

I'm not aware of any other games which used fixed address allocations (using VirtualAlloc)
 

·
Premium Member
Joined
·
423 Posts
Discussion Starter #5
@ kornman00 - Thanks for this, I haven't looked into Halo yet. But if I may ask : where did you gather this information?
 

·
Registered
Joined
·
10 Posts
I've wrote PC side programs to interface with their .map (where the data is cached in a memory savvy layout) files. At first I used a rather sneaky template class for all pointers used in structures loaded from these cache files. Basically, I would just use VirtualAlloc at an address half of 0x8001000 (0x400...) and the template class would take care of fixing the pointer value in a single bit operation.

However, I then looked into increasing the user-space memory so that I could allocate addresses higher than 0x7FFEFFF. For Vista, there is a /3GB switch you can use. XP has a method too. I forget if Windows 2000 has one, but take note that in doing this, you're taking away from the kernel memory (so it now only has access to 1GB). What I ended up doing was actually giving user-space memory 2.5 and IIRC, that was enough to do my works related to Halo.

They allocate at a MUCH higher address on the 360 tho (for Halo 3 and beyond)...at 0xA0000000. And it's a pretty big allocation (more memory after all). But hey, that's the 360 :)
 
1 - 6 of 6 Posts
Top