VRAM usage, performance questions.

Users who are viewing this thread

xenoargh

Grandmaster Knight
I, like most of the modders here, have hit all sorts of frustrating walls with Warband, where all of a sudden the mod causes crashes for end-users, yet hardly anything has changed- maybe a new suit of armor, a slightly different troop loadout.  I've had to deal with lots of complaints about this topic, I've done a lot of experiments and I've done what I could to help people out. 

But I've hit some limits here that are mysterious, to say the least, and I would like some answers to help all of us making mods understand why we're having problems and hopefully how to avoid them.

I've done some tests, and have found some crazy, crazy stuff is going on.  Amongst other things, I finally caught the "crashes or freezes suddenly when troops load" error, and I'm almost entirely certain what it is- VRAM ran out.

Here's what I think I know, at this point:

1.  If you turn off 'load textures on demand', it doesn't load the textures into RAM for use when a scene is built and actors are placed.  Instead, it's putting all of the textures into VRAM.  This causes a crash upon game load if the end-user doesn't have enough room, throwing a fairly mysterious-sounding error message, instead of gracefully switching off the option and loading normally.

Why can't it simply calculate how much space it needs for the textures, place them into RAM, and then move them to VRAM as needed?  Why is the choice between loading from the hard drive to VRAM or loading directly to VRAM?

2.  On that note... my mod + Native is less than 1GB in size, in terms of textures.  It's around 750MB, and the textures are DDS, so they're supposed to stay compressed on the card.  What's using up the rest of the VRAM?  The vertex buffers?  How does that work, in brief?

3.  On the general subject of memory use... if it's loading textures into VRAM, then what is using 1GB of RAM for, while running?  All of the data of the text files, combined, is less than 100MB, and most of it isn't being used for logical purposes.  The Troops don't need Agent simulations unless we're in a battle, otherwise they're abstracts and should be really small pieces of data.  What is using all that memory, if it's not content data?

4.  In game_variables.txt, there is the following:

# Initial Buffer_sizes
# Modders may adjust these large enough so that the game will not need to restore
# buffers during the game. However keeping them excessively large will reduce performance in most systems.
#Buffer_sizes (if smaller than 65535, this also represents the maximum vertex count for a corresponding mesh)
new_buffer_size_dx7_regular_ffp_static  = 262144
new_buffer_size_dx7_regular_ffp_dynamic = 262144

new_buffer_size_regular_ffp_static =  32768
new_buffer_size_regular_ffp_dynamic    =  32768
new_buffer_size_regular_static          = 131072
new_buffer_size_regular_dynamic        =  65536
new_buffer_size_skinning_static        = 131072
new_buffer_size_skinning_dynamic        =  65536
new_buffer_size_normal_map_static      = 131072
new_buffer_size_normal_map_dynamic      =  65536
new_buffer_size_normal_map_skinning_static  = 131072
new_buffer_size_normal_map_skinning_dynamic =  65536
What do these numbers mean?  Are they stored on the GPU, or are they stored in RAM?  And why does changing them cause end-users to report massive stability problems?

5.  Lastly, a more general question:  what can we do to ensure that end-users aren't crashing constantly?  What are the performance walls here?

After nearly a year of fighting with crashes, at least I finally know what is causing them.  What I need to know is a bit more about how it works, so that I know what I need to change.  Please give us further information about the way it's allocating memory; in particular:

A.  Do we gain any benefits by cutting down item variety, in terms of the sizes of the vertex buffers?
B.  Why doesn't the engine soft-fail texture loading, if it's going to exceed the currently-allocated VRAM, so that the geometry can still load and render with a default texture without resulting in a corrupted state?  In short, why is it running outside of the limits in the first place?  Aren't drivers reporting that number accurately?
C.  Are there any tools or console commands that will allow us to get a record of the allocations and track performance?
D.  When the engine leaves a combat scene, unloading the GPU also causes crashes or stability problems.  What is happening there?
 
OK, I've done some further investigations, and here are some conclusions:

1.  Lowering texture quality may let all the textures get loaded onto a card, but performance is glitchier.  Moreover, what is that doing, exactly?  ATi cards pretty much require power-of-two skin sizes, or shader failures may result.  So unless it's going to half-sizes or is removing mips, it's hard to say how this is supposed to be working at all without causing glitches.

2.  There is a definitely correlation between sound and general levels of glitchiness.  If I have sound variety and music on, then I am much more likely to see a sudden halt to the game state, regardless of other factors.  Given that the mod's using custom sounds, but they're OGG format just like Warband's native sounds, what is happening here?

If I run the game without sounds at all, I am much, much, much more likely to get through a big battle without a failure either making the game unplayable until I hit ALT-ENTER or making it crash.  But that's pretty much like saying that the game doesn't work, when pretty much every battle, I have to ALT-ENTER to keep playing... and that may fail at any time.

Lastly, what is happening, at a technical level, when ALT-ENTER is being invoked?  This seems like the only way to fix the game most of the time, and it's probably a big key in terms of what's breaking here.
 
Have you checked the rgl_log to see if it's spitting out anything useful? One thing you might want to check is to see if the crash that appears to be caused by the sound is giving a different error to the other, since then it could indicate two separate issues.

Gut instinct on this otherwise would be that it's dumping sound/graphics work off on the CPU if the sound and graphics systems can't handle it, and that's where the problem is occurring. You might want to try monitoring the CPU use and see if it starts acting erratically. On a similar note if you have a multi-core CPU see if using the force single threading option makes a difference.

From what I can tell the options set for the things like buffers over-ride the configuration it creates from polling the hardware, hence why you'd get stability issues; in effect you can set them to something that the player's card won't support and it will still try to force them.
 
The only thing the RGL log ever gives me back is HERR_BUFFER_LOCK errors when it goes off the rails.  If I use ALT-ENTER, that apparently forces the buffer to get rebuilt, but it only staves off the errors, and sometimes that leads to a lock.

In terms of other basic fixes I've tried:

1.  Force single-threading is on. 
2.  I've set affinity to one of the unused cores (I'm using XP Pro, so most of them are idle except for the odd MT application I own), and that helps very slightly. 
3.  I've set Warband as a trusted application so that my anti-virus doesn't go off.
4.  I tried the Warband Tamer application- it helps slightly.
5.  I've tried the /3GB switch; it made things a bit less stable.
6.  I've tried lowering the texture quality and loading all of the textures; it's unstable at that point.

The sound issue is the one area I'm not quite sure how to poke at more.  Warband doesn't seem to offer a way to limit the number of polyphony or otherwise control the sound system, otherwise I'd code a patch with MS and see if I could fix it through deliberately cutting down polyphony.  It doesn't even have an option to change the sounds in any way other than volume and shutting them off, so I presume they're using a standard call to DirectSound.

But the fact that most of the time, the buffer locks seems to be directly associated with sounds (for example, I've had it crash when loading the tavern scenes, simply because I turned the music back on) makes me think that that is probably part of the problem.  Like a lot of people these days, I didn't buy a sound card; the Realtek chipset on my mobo is more than sufficient for most games. 

I keep thinking that the problem there may be a problem with the way things are getting queued;  for example, if the engine's about to spawn 50 guys, it should  know that it's about to do that, and only attempt to play 1 sound, not 50.  But it may just be CPU lock in general at that that point; from what I've seen, the troops are spawned in the T position, they're then being given their equipment, then they're animating, and the sounds playing at the same time may tie up the CPU for a quarter of a second.  This may be too long for the graphics engine to deal with, so it goes off the rails, instead of waiting for the next update. 

I'd much rather that it froze the sim state and waited for the entire process to finish rather than suddenly not being able to play or having to hit ALT-ENTER to reset the buffer and continue at a high risk of having the game lock up my GPU at some point (I've seen hard-lock on the GPU, too, which is ugly, and something I've never had happen on any other game I've played on this box).
 
OK, did another test.

If I shut off sounds entirely... even if I'm watching 50 guys spawn at once, the CPU logjam when troops spawn is a lot smoother, and I didn't have a lock.  I tested various scenarios, and I'm pretty sure that this is where things are going off the rails.

The problem then is that playing without sound is, well, playing without sound.  It makes the game a lot less fun.

But at least that gives me something to report.
 
I may have found a solution!!!

I just tried converting all of the OGG files to 16-bit 22khz mono WAV.  Guess what? No more errors.  Small lag when new troops spawn, but no lockups after three battles in a row, including one that would normally have been a show-stopper (120 vs. 400).

I'll have to keep testing this, and some of my sound files were corrupted by the conversion (there are a few Warband sounds that are 8-bit, and some custom OGGs don't seem to have survived) but this looks like it might be the fix!
 
More testing; thus far, not a single lockup.  I'll keep testing this after converting the music tracks as well.  Who knows, maybe I can finally have full sound variety and music, too  :grin:
 
Still not a single lockup.  150+ battlesize works for the first time since I bought the game, in DirectX 9.  Music and sound variety on doesn't seem to matter.

I can't believe that this was such a magic bullet; there must be a catch or something.  I'll keep testing it.

Oh, and don't try this at home unless you're familiar with the Module System; you absolutely must change all of the sound references, you can't just change the sound files.
 
Sounds like it's a sound memory issue. Which with an IC would be system memory. Probably it's either building a sound buffer too small for what's needed or it's just too slow. I assume you already checked you have the latest driver for the Realtek IC?
 
Yeah, I have checked that out.  Realtek's driver releases are pretty rare, there's nothing new.

I haven't had a crash since I rebuilt the sound files.  It's been pretty amazing, actually playing the game without worrying about having to hit ALT-ENTER every couple of minutes :smile:
 
Man I love you, I'll convert the sounds and music right now. :smile:

Edit: I have converted them, but I noticed that the files are oversized. The music folder now has 1.21 (or something like that) GB's and the sound one got 90 MB. Is that normal ? Can you tell me which program you are using for converting the files ?

Never mind, used your program to convert in the proper way.
 
On the subject of VRAM.  I just recently had a graphics crash in a mod.  RGL ERROR too many vertices or some such.  So I go look at the rgl_log and noticed this:


Starting new log file.
Version:  1.134

-- OS: Microsoft Windows Vista  Service Pack 2 (build 6002), 64-bit
-- RAM: Available physical RAM: 4054MB/4096MB
-- CPU: Intel(R) Core(TM) i7 CPU        920  @ 2.67GHz (GenuineIntel)
- L2 cache size: 256K
- CPU Features: FPU MMX SSE SSE2 SSE3 SSSE3 EST HTT
- Number of CPUs: 8 ( Speed: ~2672MHz / ~2672MHz / ~2672MHz / ~2672MHz / ~2672MHz / ~2672MHz / ~2672MHz / ~2672MHz )

-- GPU:
- D3D9 Adapter: 0 / Driver: aticfx32.dll / Description: ATI Radeon HD 4870 X2
- Texture Memory: 744 ( Available Texture Memory: 723 )

Now if the game is not going to use the other GPU on the board then that should make texture memory at least 1024 with 990 available. Should it?
 
Now if the game is not going to use the other GPU on the board then that should make texture memory at least 1024 with 990 available. Should it?
That is different.  Both of those boards are 512s, or 1024s?  And I didn't know you couldn't run the game with Crossfire... that's a bummer.

@Swadian Man at Arms:  Don't convert the music.  OGG is a streaming format, it needs to remain OGG or will probably cause memory issues for some people.  And the size after conversion will be a little larger per-file, yeah; OGG is compressed, like MP3.

 
xenoargh said:
Now if the game is not going to use the other GPU on the board then that should make texture memory at least 1024 with 990 available. Should it?
That is different.  Both of those boards are 512s, or 1024s?  And I didn't know you couldn't run the game with Crossfire... that's a bummer.
Its not exactly two cards in crossfire.  Its one card with two GPU's on it. Both are 1024's I assume since the log only shows 700 or so for memory that it is only using 1 GPU.

If this game uses linked GPU's I would be very surprised.
 
Hmm.  IDK what to tell you then.  If it's a dual-GPU board, it seems like it would probably be treated like Crossfire, and the memory may not be uniformly available to a single GPU.  I'm too ignorant to comment further on that subject  :lol:
 
xenoargh said:
Now if the game is not going to use the other GPU on the board then that should make texture memory at least 1024 with 990 available. Should it?
That is different.  Both of those boards are 512s, or 1024s?  And I didn't know you couldn't run the game with Crossfire... that's a bummer.

@Swadian Man at Arms:  Don't convert the music.  OGG is a streaming format, it needs to remain OGG or will probably cause memory issues for some people.  And the size after conversion will be a little larger per-file, yeah; OGG is compressed, like MP3.

Will it affect my performance ? Unfortunately, I cannot convert them back to OGG for some reason. :mrgreen:

(Anyone got the Music OGG files ?) :mrgreen:
 
I got 2 GB RAM. And it would be great of anyone can upload the original Music and Sound files. The problem is that I have to edit sounds.txt everytime I dowanlod a mod, so I want to use my converted .wav files just for specific mods, not for all instead.
 
I just did the conversion.  And now almost none of the sounds are playing.  I know I did the conversion right, because the sounds play in other media players and are recognized as .wavs (and are much bigger). I'm thinking it's a bug related to Wine, which means that I'm one of 5 people on this forum for which this fix won't work.  Since I really don't want to mess around with Wine, and since I didn't really have the problem all that badly, I undid the conversion.
(EDIT: The problem was that I put the converted sounds into the module's sound folder, and didn't set scan_module_sounds in the module's module.ini file.  Whoops.  Thanks to Caba for this fix.)

Whatever the case, I decided to peek around the Warband folder.  I noticed that it uses the fmod sound system, and specifically a version of it that is very out of date, 4.28.7.  So I replaced it with the latest version in its branch, 4.28.26.  I'm going to see if this makes it work better with the oggs.  Of course, with me being a Linux user, my results may not reflect that of the average windows user.  So, somebody else who actually has Windows may want to try this if they have these problems, to see if it magically makes it better.
 
Back
Top Bottom