Chronika: recording your world's history [feedback needed]

[int19h]

Ever wondered why some kingdoms expand while others do not, or how prices change over time? Or what happens if you buy all the food, or kill all the lords? I did... so I came up with an idea:

A mod that periodically snapshots the state of the world, and records those snapshots in the save (in such a way that doesn't require the mod to be present to load it). Then, whenever you feel like it, you can then dump all that recorded information to XML, and then process it to your heart's content: graph it, query it, upload it into a relational database and do OLAP etc. So you could e.g. compare prices on wheat in Rovalt against the same in other towns over time, and how it is affected by villages being looted, or bandit activity. Stuff like that.

Here's a screenshot demonstrating a prototype I whipped up:

Now, what is the "state of the world", exactly? That's where I need feedback! In theory, the dump can contain everything; but, obviously, some things are more useful than others - and all of them take up space! The prototype that I have right now dumps a lot of stats for all the settlements. Here's what it looks like once exported to XML:

https://gist.github.com/int19h/1a8474c7734ff6e1d36a258639bc87e2

Inside the save, data is stored much more efficiently than XML would allow, utilizing string tables and variable-length encoding for numbers. Still, it's a lot - as you can see from the screenshot, with this prototype, a single snapshot containing all the data you can see in the gist takes up around 100 kilobytes. So, about 8.5 megabytes of data for each year of the game, if snapshots are recorded daily. It'll be a lot more once I add kingdoms, clans, parties etc - especially parties. For this reason, I'll probably make some of it configurable - but I still need to implement the logic to save whatever needs to be saved.

So, if you are interested in something like this - what kind of data would you want in the snapshot?

Also, if you have any feedback on the structure of output XML, that would also be helpful. I'm definitely sticking to XML over JSON here because of the wealth of mature tools in that ecosystem: schemas, XSLT, XQuery etc; and it can be trivially converted to any other format if necessary. But the specific elements etc are open to amendments. As it is, it mostly follows the internal structure of the game.

 

[Apocal]

Economy:
Average prosperity of towns
Most prosperous town
Average hearth count in villages
Number of raided villages
Average and lowest price of grain
Average and lowest price of a warhorse (might be tricky)

Factions:
Number of active armies
Number of independent parties
Average party speed
Average tier in mobile parties
Average garrison size
Number of rebellions
Number of clans joined or defected
Total faction wealth

 

[Terco_Viejo]

Very much agree with Apocal's suggestions. I add, if you see the possibility to implement a political map and make it functional, you crown yourself with the mod.

VC Political Map OSP

 

[int19h]

Keep in mind that any kind of sum/average/mean/min/max metric can be calculated after exporting, aggregated from raw data in the snapshots. Aggregating it when recording would take less space, but in general, I want to allow drilling down into stuff as much as possible - e.g. for village hearth count, a plotted distribution is more useful than a simple average, and sometimes you might want to see which villages are the outliers that skew the average.

With that in mind:

Average prosperity of towns
Most prosperous town

Prosperity of each town is already recorded, so this can be calculated.

Average hearth count in villages

Hearth count of each village is already recorded, so this can be calculated.

Number of raided villages

This can be derived from village state (normal / being raided / looted), which is also recorded separately for each.

Average and lowest price of grain
Average and lowest price of a warhorse (might be tricky)

I intend to record the complete inventories of all settlements and all parties, and market data for markets. So you would be able to determine averages and minimums for all goods - either globally, or in some geographic area, or for a specific faction, etc.

Why would warhorses be more tricky? Is there something special about their prices?

 

[int19h]

if you see the possibility to implement a political map and make it functional, you crown yourself with the mod.

It would certainly be possible to render something like that from exported data, and even make it animated, showing territorial gains/losses over time. But that would be separate from the mod - the mod is just about gathering and exporting, how to process data afterwards is a separate thing. I'll probably have to write a bunch of sample scripts showcasing how to do that kind of stuff, and this is a good idea for one of them.

But, that wouldn't be an in-game map showing the current state of affairs. Since the mod is really all about recording historical information, something like that isn't in scope for it. It would make for an interesting mod in its own right, though. The tricky part is determining where to draw the boundaries, since the game doesn't really have such a notion - but presumably this can be done in the same manner as VC Political Map does it. I'll need to take a look at how exactly that works.

 

[Apocal]

Keep in mind that any kind of sum/average/mean/min/max metric can be calculated after exporting, aggregated from raw data in the snapshots. Aggregating it when recording would take less space, but in general, I want to allow drilling down into stuff as much as possible - e.g. for village hearth count, a plotted distribution is more useful than a simple average, and sometimes you might want to see which villages are the outliers that skew the average.

Oh, cool. I wasn't sure how deep you wanted to go.

Why would warhorses be more tricky? Is there something special about their prices?

Because there are multiple types of warhorses and players (I'm assuming they are like me) buy multiple types when looking at upgrading troops (where type doesn't matter).

 

[froggyluv]

I would love soemthing like this presented in a storied narrative. I pretty much want the same for all Debug notices in all sandbox games

 

[int19h]

I would love soemthing like this presented in a storied narrative.

The name is Greek for "chronicle", so there's a certain hint

That said, I think this will have to wait until the underlying functionality is polished, and some in-game UI could be added on top. I'm also not sure how to reconcile the sheer volume of data collected with anything that could be presented as a storied narrative - historical chroniclers would not, as a rule, record the price of bread in all towns across the empire every day. I suppose it could be stretched somewhat. For now, I'll keep it simple, and present it solely as an out-of-character tool for those who just want to have a better insight into the sandbox.

 

[int19h]

Because there are multiple types of warhorses

Ah, I see what you mean now! Yeah, tracking all items individually will take care of that also - you can then aggregate that data on whatever criteria you want, including all warhorses, or only specific types etc.