PackBSP progress: File search order

Still rewriting various bits and pieces of PackBSP. Today I want to talk about how Source games handle finding files, as far as I can tell from guesswork, testing, and code comments. I'd welcome any corrections on this stuff.

If the game need a file, like  "sounds/weapon/pistol_shot.wav", where exactly does it get it from? For most Source-engine games, this turns out the ordering is a bit more complex than I had originally expected.

The very first place the game looks is for items packed within the current map (BSP) file. (Inside the Pakfile lump.) After that, it will look in locations defined within the gameinfo.txt file, an example portion of which is below.

FileSystem
{
    SteamAppId	1234	// A made-up ID for demonstration purposes of the "london" GCF
    SearchPaths
    {
        Game	|gameinfo_path|.
        Game	|all_source_engine_paths|foo
        Game	|all_source_engine_paths|bar
        Game	|all_source_engine_paths|baz
    }
}

So what, you might wonder, does all this actually mean? Well, the SteamAppId determines what GCF (or NCF, etc.) files get loaded. These are special archives that are used for delivering the games on Steam, and contain the bulk of the content. We'll get back to those in just a second.

First, the |gameinfo_path| placeholder refers to the folder which contains the gameinfo.txt file that you're looking at. Pretty straightforward, although there's a wrinkle regarding languages and pak-files we'll examine later.

Now about that |all_source_engine_paths| placeholder? Let's skip back and look at that ID number. For this example, let's assume our game ID of 1234 is--inside Steam's content-distribution settings--marked as depending on two more IDs: 2345 and 3456. Now each of those IDs corresponds to one GCF file:

• 1234 = london.gcf
• 2345 = paris.gcf
• 3456 = berlin.gcf

Again, most of this is ID-based stuff is internal to Steam, and so you're not expected to know it. These GCFs are what is meant when the |all_source_engine_paths| placeholders are seen.

So far, this is what we expect the search order for "my/example.file" to look like, assuming that gameinfo.txt exists at c:\program files\steam\steamapps\steam_logon_here\examplegame\eg\gameinfo.txt

mapfile : /my/example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\my\example.file
c:\program files\Steam\steamapps\london.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /foo/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\london.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /bar/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\london.gcf : /baz/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /baz/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /baz/my/example.file

Looks like a lot of potential places, doesn't it? It gets even more interesting. There are two additional wrinkles we need to take into account. First, any path to a "real" folder will automatically include any PAK (and presumably VPK) files that are within that folder root. I'm not sure of the exact matching mechanism, but our "where it looks" list then becomes:

mapfile : /my/example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\my\example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\pak01.pak     : /my/example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\pak01_dir.vpk : /my/example.file
c:\program files\Steam\steamapps\london.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /foo/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\london.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /bar/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\london.gcf : /baz/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /baz/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /baz/my/example.file

Oh yeah, the second wrinkle. Internationalization. What about people in other countries? They may have different dialogue and text to match their language. The following language types seem to be defined:

• danish
• dutch
• english
• finnish
• french
• german
• italian
• japanese
• korean
• norwegian
• polish
• portuguese
• russian
• schinese
• spanish
• swedish
• tchinese

Chinese is present twice, with "t-" and "s-" variations, which refer to the "traditional" versus "simplified" character set. Whenever a "Game" path is defined in gameinfo.txt, an additional path is first made to correspond to the language the user has set. So if our current user has his language set to "spanish", the list now becomes:

mapfile : /my/example.file

c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg_spanish\my\example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg_spanish\pak01.pak     : /my/example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg_spanish\pak01_dir.vpk : /my/example.file

c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\my\example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\pak01.pak     : /my/example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\pak01_dir.vpk : /my/example.file

c:\program files\Steam\steamapps\london.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /foo/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\london.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /bar/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\london.gcf : /baz/my/example.file
c:\program files\Steam\steamapps\paris.gcf  : /baz/my/example.file
c:\program files\Steam\steamapps\berlin.gcf : /baz/my/example.file

Unfortunately, it doesn't end there. Steam has it's own method for managing languages with GCFs, so depending on how all those mysteries are configured, Steam may download another set of GCFs and--purely as one of many possibilities--you might get this:

mapfile : /my/example.file

c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg_spanish\my\example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg_spanish\pak01.pak     : /my/example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg_spanish\pak01_dir.vpk : /my/example.file

c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\my\example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\pak01.pak     : /my/example.file
c:\program files\Steam\steamapps\steam_logon_here\examplegame\eg\pak01_dir.vpk : /my/example.file

c:\program files\Steam\steamapps\london spanish.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\london.gcf         : /foo/my/example.file
c:\program files\Steam\steamapps\paris spanish.gcf  : /foo/my/example.file
c:\program files\Steam\steamapps\paris.gcf          : /foo/my/example.file
c:\program files\Steam\steamapps\berlin spanish.gcf : /foo/my/example.file
c:\program files\Steam\steamapps\berlin.gcf         : /foo/my/example.file

c:\program files\Steam\steamapps\london spanish.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\london.gcf         : /bar/my/example.file
c:\program files\Steam\steamapps\paris spanish.gcf  : /bar/my/example.file
c:\program files\Steam\steamapps\paris.gcf          : /bar/my/example.file
c:\program files\Steam\steamapps\berlin spanish.gcf : /bar/my/example.file
c:\program files\Steam\steamapps\berlin.gcf         : /bar/my/example.file

c:\program files\Steam\steamapps\london spanish.gcf : /baz/my/example.file
c:\program files\Steam\steamapps\london.gcf         : /baz/my/example.file
c:\program files\Steam\steamapps\paris spanish.gcf  : /baz/my/example.file
c:\program files\Steam\steamapps\paris.gcf          : /baz/my/example.file
c:\program files\Steam\steamapps\berlin spanish.gcf : /baz/my/example.file
c:\program files\Steam\steamapps\berlin.gcf         : /baz/my/example.file

Holy gazpacho, Señor Batman! That's a lot of ground to cover. Thankfully, a lot of the GCF and base-folder combinations don't exist, and they're probably screened out early-on and never checked again.

So in a way that's what I have to reverse-engineer into PackBSP, although I can probably ignore the language-variations. At some point I should cut out the middleman and revisit the learning experience of messing with Valve's SDK code, but for now I'm going with what I know mostly-works.