Savegame UI / Gadget

[Argh]

OK, that bug with the "spaces in path" issue has been resolved by Auswaschbar. Yay!

So pretty much I just need to finish the UI (which will be basically just something that looks reasonable and works, mind you).

I still want a "real" UI, but I guess that will have to wait. I'd rather get people testing stuff and figure out what's missing or could be improved.

[very_bad_soldier]

Can you give me a theoretical use-case where it might be a major problem, if I preserve heightmap changes incompletely?

How about this (probably even applicable for all mods): There is a narrow passage with a slope. Its just steep enough so units can still pass it. In the game some projectiles hit the slope making the slope a bit more steep, just so that units can't pass it anymore.
It would be a minor heightmap change but would lead to major change in gameplay.

[Tribulex]

true ultimate because limits do exist in the game, the heightmap really needs perfect reconstruction i think. kind of annoying but imo its the only way.

you can of course save time on 100% hard maps, but thats it.

one other idea for optimization.

Divide the map into sectors, and unless a unit enters a certain sector, dont store heightmap info for that sector, since a sector that had no units in it is much less likely to have important heightmap information. you can use this same system to compress the savegames if there is no terraform in a sector dont store the data for that sector.

Does this make sense? Divide and conquer

[CarRepairer]

Another thing to eventually consider for terrain LoS. If the Spring terrain view bug were to be fixed, you'd want to go through the process of terrain not just once but once for each team.

Save:
1. For each team:
1a. CallAsTeam( GetGroundHeight(all the map) )
1b. Save this terrain data for the team.
2. GetGroundHeight(all the map) to save the real terrain.

Load:
1. For each team:
1a. SetGroundHeight(all the map) using team's terrain data.
1b. Give brief vision to the whole map to the team.
1c. Reset map height.
2. SetGroundHeight(all the map) with the real terrain data.
3. Load the units now.

#3 must be last so as not to give brief view of all units in step 1b.

Is this overkill? Yes, but it should one day be doable.

[Tribulex]

good idea

[Argh]

@Car: when the engine supports per-player terrain (i.e., what you're seeing is not necessarily the heightmap) it should not be necessary to do all that- you'd see the un-altered heightmap state, or the state as set via synced Lua... unless you have LOS. Well, unless that feature isn't written correctly. I would suspect that we'll see these features as the "real" heightmap and the one used for visuals continues to diverge.

[R-TEAM]

Hi,

@Car: when the engine supports per-player terrain (i.e., what you're seeing is not necessarily the heightmap) it should not be necessary to do all that- you'd see the un-altered heightmap state, or the state as set via synced Lua... unless you have LOS. Well, unless that feature isn't written correctly. I would suspect that we'll see these features as the "real" heightmap and the one used for visuals continues to diverge.

oehhmmm ... wrong - sorry.
IF the feature is implemented correct then it creat more than 2 alteration of the hightmap.
Not only the "fresh" state at start and the actual state .....
If a terrain altered and you spot it - but altered again ofter you have no LOS more to it, then you only know the NOT Actual hightmap in this area - but this is not more the "fresh" hightmap ...
So it is real a per-team-based hightmap ...

(you can implement redundant checks if more than one team share the same data and so dont save the same data more than one time ..)

Regards
R-TEAM

[Argh]

I have some pretty sobering news.

If I save the heightmap at 1:1 resolution, even if I just store areas where it differs, it's still hundreds of thousands of operations in a typical situation, which takes about 5 minutes to save about 2 minutes to restore.

That's not happening, basically. That's an unacceptable lag for a savegame, by any stretch of the imagination. I'll have to use a threshold, and maybe lower the resolution if that doesn't cure the problem.

IF the feature is implemented correct then it creat more than 2 alteration of the hightmap.

The engine doesn't support that at this time- there is one heightmap, period. There is no difference between what you can observe and what's really there, there are no controls for what Player X's observable heightmap looks like vs. Player Y, etc..

Sorry, but there isn't any good fix for that, Lua-side, at this time. That's an engine-side change that will have to be made, basically. I am fairly certain that there's been at least one Feature Request about this, and I am quite sure that the Devs know that this is a problem.



Also, I seem to be hitting a weird error where if the savegame state > some number of lines... Lua gives me an error saying that a "control structure is too long". That's a new one. Hrmm, looks like it's due to too long of a statement within an IF or WHILE loop... and tens of thousands of lines heightmap alterations will surely do that, lol. I'll either have to break that up into multiple IFs or stuff all of that into a function at the start of load.

Good news, though, is that I've resolved most of the speed issues with heightmaps and am pretty close to a 1:1 state. Again, sorry if you think it should either be perfect or it doesn't work, but we're going to have to choose here, and I'd rather catch 99%+ of heightmap changes than catch none or wait a long time for saves and loads to occur. That, and if people want to fork that portion of the code for a particular game, they're welcome to do so ofc.

[Argh]

OK, got the problem of the "control structure too long" error under control. Heightmap changes are now recorded as faithfully as I think is reasonable. It takes a considerable length of time to save at this point (around a minute over here), but loading a save doesn't take very long.

Note to "I want it to be even better than it already is" folks: moving from what I'm doing (16X16 sample size, > 8 change, absolute) is probably going to push times up by a power of two, and saves are already 6+MB and take a minute to save, on a fairly smallish test map that has a lot of height changes to record (my "floating world" piece)... and I'm fairly certain that I'm getting very near the stack size limit for tables, due to the way the save is currently structured- a problem that I can work around, if needbe, but... I'd really rather not, because structuring the logic will get pretty cumbersome, fast, and this is already complicated in terms of flow- I'm not looking forward to documenting this code at all.

So... GG states.

Anybody have any bright ideas? I wrote a brute-force algorithm, but I'm not happy about it, and it's not factored into the time statement above- that's 'just' recording Units, positions, Commands, economy, and the state of the heightmap.

To make this really feature-complete, GG states need to be saved- that would give AIs a common place to dump if asked to dump their states, as well as saving game data for games that don't use OTA economies, Lua controls over certain states of play, etc.

That's a given, it's a must-have. But I'd really like a fast, clean way to make sure that we're getting everything and that it will be written back to the gamestate correctly.

[zwzsg]

My mod "Balladium Annihilating" edits the heightmap of whole map. It takes a couple seconds. Make sure to use SetHeightMapFunc. Make sure the inner loops code is optimised.

So... GG states.

How are you going to handle the case where a GG table store unitID? Because then they'd need to be translated to the new unit IDS, and there's no way a generic savegame code might be able to guess if a table is used to store unitID or something else.

[Argh]

The problem's not on load- the delay's really pretty reasonable there, and I think that's about as short as it's going to get, plus or minus a second or three (which I know is a lot of room for optimization, but in terms of the end-user's experience, not so much perceived difference, if that makes any sense, so I haven't been worrying about it a lot yet).

If you have any ideas about how to solve for saving the states of the heightmap and playing them back, when you don't know whether you're going to store any given sector ahead of time, please let me know. I basically just iterate through the whole map, get whether or not I'm going to store it, and if yes, then the changes get put into a Spring.SetHeightMapFunc(). I guess I could write it to a table instead, and run Spring.SetHeightMapFunc() only once- that'd probably cut a second or two off of load, but not major.

It's save, and the problem's structural.

Due to not having io in Gadgets (for the billionth time- WHY?!?), the flaming hoops I've had to traverse to write a really complete save-state have been pretty asinine, because there are certain walls that appear when you aren't writing 100 instructions, but 100,000.

To put it really simply, if io was functional for Gadgets, the only part the Widget would have to do is to provide a UI, and save would operate several multiples faster than it currently can.

How are you going to handle the case where a GG table store unitID? Because then they'd need to be translated to the new unit IDS, and there's no way a generic savegame code might be able to guess if a table is used to store unitID or something else.

You're absolutely right, as of right now. There's also the problem that code that changes a Unit's menu (like, say, P.U.R.E.'s upgrade system) or other stuff like that (say, code that trades resources for activating a COB state, but no global stuff is ever referenced) will not be restored.

I don't think there are any easy ways around these things- we built our stuff without ever having to deal with this, now we're just going to have to deal with it. This would have been true for any engine-side implementation that wasn't the entire gamestate at the moment it was saved, due to all of the places we can and do store data.

There aren't any cute solutions to this, I don't think, and for now, it's fair to say that a lot of things that rely on Lua will be borked in a savegame, until we get our collective houses in order.

[CarRepairer]

Another thing to eventually consider for terrain LoS. If the Spring terrain view bug were to be fixed, you'd want to go through the process of terrain not just once but once for each team.

Save:
1. For each team:
1a. CallAsTeam( GetGroundHeight(all the map) )
1b. Save this terrain data for the team.
2. GetGroundHeight(all the map) to save the real terrain.

Load:
1. For each team:
1a. SetGroundHeight(all the map) using team's terrain data.
1b. Give brief vision to the whole map to the team.
1c. Reset map height.
2. SetGroundHeight(all the map) with the real terrain data.
3. Load the units now.

#3 must be last so as not to give brief view of all units in step 1b.

Is this overkill? Yes, but it should one day be doable.

@Car: when the engine supports per-player terrain (i.e., what you're seeing is not necessarily the heightmap) it should not be necessary to do all that- you'd see the un-altered heightmap state, or the state as set via synced Lua... unless you have LOS.

Not quite. If Team 1 gets a view of an altered heightmap in Point A and then loses LoS in that point, that altered heightmap should stay "ghosted" for Team 1. Then point 1 might get altered and Team 2 gets a view of it briefly, then loses it, so they have a ghosted view that's different from Team 1's. And then a nuke falls on it and the true heightmap is different from both those views. So there's a true heightmap, a Team 1 ghosted heightmap, and a Team 2 ghosted heightmap. So you'd need to save a heightmap for each team plus one true heightmap. (For onlookers, I'm using the term ghosted here but it's a poor choice. It stems from ghosting the structures that you get a brief vision of, but in the case of heightmaps there is nothing translucent really. I should think of a better way to say it).

I have some pretty sobering news.

If I save the heightmap at 1:1 resolution, even if I just store areas where it differs, it's still hundreds of thousands of operations in a typical situation, which takes about 5 minutes to save about 2 minutes to restore.

I'd rather catch 99%+ of heightmap changes than catch none or wait a long time for saves and loads to occur.

Hey it's always a process. If adding a threshold and getting 99% of the heightmap reduces save time from 5 minutes to 30 seconds, so be it. 99% is way better than 0%.

[lurker]

I basically just iterate through the whole map, get whether or not I'm going to store it, and if yes, then the changes get put into a Spring.SetHeightMapFunc(). I guess I could write it to a table instead, and run Spring.SetHeightMapFunc() only once- that'd probably cut a second or two off of load, but not major.

Chunks how big? If large, okay, if small, I don't think you realize just how much spring does after setting the height value itself, and how very faster doing it all at once is.

io in Gadgets

You have to be doing this the wrong way. unsynced gadget->widget is a matter of a single line of code to move a string, and for loading you can use the exact same code in a widget and an unsynced gadget.



And gadgets should already largely support being reloaded (should not do) via dumping a bit of data into *params.

[Argh]

Chunks how big? If large, okay, if small, I don't think you realize just how much spring does after setting the height value itself, and how very faster doing it all at once is.

Good point, but there are so many commands that I hit the "too many constants" wall, if I try to put them all in the same loop. Meh, loading is not an issue in terms of speed, vs. anything you see commercially (my savegames are loading about 10X faster than DoW's, for example). Saving... was.

You have to be doing this the wrong way. unsynced gadget->widget is a matter of a single line of code to move a string, and for loading you can use the exact same code in a widget and an unsynced gadget.

No... moving those strings across the interface, then wasting code making sure that they're the messages we want to catch, gsub'ing them to remove the key-phrase, etc., is a massive waste of CPU. Gadgets should just have access to io.

That said... I've figured out what was causing problems with string transfer, and sped things up about 15X. Saves are in the very comfortable "a few seconds" category. The first time I tested the fix, I was sure it was borked, it ran so fast.

And gadgets should already largely support being reloaded (should not do) via dumping a bit of data into *params.

I'm not sure what you mean there. I put exactly one line into modOptions. There's a Gadget that detects whether we're loading a savegame, and it in turn uses VFS to load the actual savegame state, which is pure Lua.


Anyhow, I guess it's time to work on the UI now, and then we can all have something to actually test. GG is working, although it may need to be rigged to go down one more level to be fairly safe for most games, IDK yet, we can talk about that when people have tested it out.

[R-TEAM]

Hi,

@Car
I said the same thing
(from multiple hightmaps)

@Argh
Dont get me wrong..i am VERY happy any doing real a step forward with Save/load !!!!
But i see a potential "open can of worms" if the hightmap dont get saved accurate at 100%.
Maybe a little bit paranoic - but the time has taught us, that the unexpectet will happen and so it will give problems/bugs with the non accurate restoring.IF it works i am happy
And as Car and I said, IF (remember the "IF" - i know ATM the engine dont support mutiple hightmaps) the engine suport multiple hightmaps
then musst all saved.
I dont say ATM it musst implemented.But the code should writen, with the possibility to implement it later, in mind.
So it is easyer IF this happen ...
You see the poblems with save/load engine side who come from no one have thinked at the beginning from spring over this ...
(o.k. - this is a different story - spring was primary build as a TA-record player, so save/load would be sille here ...)

Regards
R-TEAM

[lurker]

You have to be doing this the wrong way. unsynced gadget->widget is a matter of a single line of code to move a string, and for loading you can use the exact same code in a widget and an unsynced gadget.

No... moving those strings across the interface, then wasting code making sure that they're the messages we want to catch, gsub'ing them to remove the key-phrase, etc., is a massive waste of CPU. Gadgets should just have access to io.

That said... I've figured out what was causing problems with string transfer, and sped things up about 15X. Saves are in the very comfortable "a few seconds" category. The first time I tested the fix, I was sure it was borked, it ran so fast.

I'm going to try again. Sending a string to a widget doesn't require splitting it up. Sending a string to synced gadgets does, BUT you have to do that whether you have unsynced gadget OR widget reading it. (And it can be done in 10 lines sending code, 10 lines receiving, not sure what your major complaint is.) If you want SYNCED gadgets to have I/O, then you need to say that explicitly, because that is a fundamental change to the sync-safety system however it gets implemented.

I'm not sure what you mean there.

Gadgets should support /luaui reload and equivalent. Most don't. If they did, you could make a save system that preserves their data pretty well.

[Forboding Angel]

Pardon me for butting in where I don't belong, but reading over this a "seemingly simple" solution struck me.

Couldn't you just have a widget that acts as a go-between for I/O? That being it's only function. It strikes me that it should be simple enough to pass parameters to the widget and then make the widget do it's thing.

Maybe this has already been said before, in which case ignore it, but just a thought.

[lurker]

That's the obvious solution, correct. I don't know what argh did. Also, he seems to be under the impression that UI has to be widget.

[Forboding Angel]

That's the obvious solution, correct. I don't know what argh did. Also, he seems to be under the impression that UI has to be widget.

Well from what I've been lead to believe, implementing nice looking graphical stuff in a gadget is a serious pain in the ass compared to doing it with a widget.

[lurker]

I'm pretty sure gadgets have the same UI calls.