Luarules <-> AI, part 1 of 0: resources

[SpliFF]

Which is exactly why this should be done right rather than done quickly. Why invent another legacy system if the plan is to replace it with something more generic? Then it has to be supported for years because a few mods and AIs used it.

I think the trap is engine people trying to decide what variables Lua/Java/Whatever will want to read. A better approach would be to store pretty much any engine gameplay variable in an exportable form unless there is a clear performance impact. I suspect there is a lot of stuff in C variables that are only inaccessable because of programming convenience. What would be ideal I think is for many key variables to be written back to a global array that has a set of access functions in all hosted languages. So in C the code would be something like

Code: Select all

#include globalvars
int game_time = 0;
// var name, access flags, description
createGlobalVarInt("game_time", AI|GAIA|SYNCED, "Game time in ticks");

// ... then in later code ...

int game_time = getGlobalVarInt("game_time");

// ... do C stuff with game_time ...

setGlobalVarInt("game_time", game_time);

Then in Lua ..

Code: Select all

local gvars = Spring.getGlobalVars()

print "Elapsed Time: " .. gvars.game_time

Same kind of thing in Java/Python etc.

If used consistently it would provide a framework for what I was talking about earlier. Obviously it needs more thought but I think the principle of the concept is sound.

[hoijui]

thats... exactly a i though of it.
in addition, your example just made me think that an other standard function should be included, like this:

Code: Select all

int current_value = getGlobalVarInt("game_time");

int new_value = current_value / 2;

setIfGlobalVarInt("game_time", new_value, current_value);

even if we do not yet implement it with perfect internal acurracy (i dont know how hard this is in C), it could be usefull.

an other question is...
where would we have these functions?
eg: we could have them inside each property intensive class inside the engine:

• Unit
• UnitDef
• Feature
• FeatureDef
• Team
• WeaponDef

and probably some more

or we could have a single set of these functions... maybe?
i can not see that as realistic, because of the indexing:

Code: Select all

int ammo = global_getGlobalVarInt("ammo", UNIT, unit_id, weapon_id);
int name = global_getGlobalVarInt("name", UNIT_DEF, unitDef_id);

the speed problem could be addressed, as said, in this way:

Code: Select all

// in addition to this possibility:
float cur_health = unit.GetGlobalVarFloat("health");
// ... we could do:
const int I_UNIT_HEALTH = Unit::GetGlobalVarIndex("health");
for (int u=0; u < units_size; ++u) {
    float cur_health = units[u].GetGlobalVarFloatByIndex(I_UNIT_HEALTH);
    // do stuff here ...
}

which (as already said) makes looking up a var an O(1) operation, and the 1 would be a function call (could be inlined) plus an addition for calculating the array position of the return.

So far, i heard no real critisism on the idea itsself.
There has to be, right?

maybe we could start a new branch on the spring main repo.. or better somewhere else maybe, where you, spliff, have write access too.

... which brings me to the idea: should we have a second official spring repo on github, where much more then the current 7 ppl have write access, for experimental branches, and possibly with a branch that is only meant for ppl to submit patches more easily?
eg: they submit to this second repo, and anyone of the 7 can then transfer the commit to the master. makes sense?

back on topic:
then we could implement this idea for a small class, that generally sees little changing, FeatureDef, maybe?

[imbaczek]

... which brings me to the idea: should we have a second official spring repo on github, where much more then the current 7 ppl have write access, for experimental branches, and possibly with a branch that is only meant for ppl to submit patches more easily?
eg: they submit to this second repo, and anyone of the 7 can then transfer the commit to the master. makes sense?

makes sense, but somebody has to maintain and test all this, and we're always short on manpower... for now, usual github forks + forum/#sy spamming must suffice.

oh and I'm +1 on the general idea. I could even live with a script which parses unitdefs, etc and creates proper AI/Lua export code during the build process.

[zwzsg]

Are you still trying to design a system where every AI can play every mod? I though it had became clear now that generic C++ AI fails unless the mod is a TA reskin.

[Tobi]

Some small notes: please don't use getFloatVarByIndex(...) stuff throughout the engine. For vars used by the engine, make e.g. a bunch of constants initialized one time to the index of the variable, and then write a bunch of getters in e.g. the CUnit class.

Or even better would be to store engine vars at fixed indices so we can just have a constant enum with all property indices.

e.g.

Code: Select all

enum {
  //...
  PROP_IDX_HEALTH = 13,
  //...
  PROP_LAST_ENGINE_PROPERTY
};

class CUnit : CWorldObjectWithProperties {
  //...
  float getHealth() const { return getFloatProperty(PROP_IDX_HEALTH); }
  float setHealth(float x) { setFloatProperty(PROP_IDX_HEALTH, x); }
  //...
};

Please keep in mind that exporting ALL variables may sound nice, but it also means that no changes can be made to the engine without breaking backward compatibility.

There are also a variety of variables that need special handling when being exported. (E.g. categories, pointers to model, cob, commandAI etc.)

Another small note, many of this is already possible using CREG (e.g. reading arbitrary variables based on their name.)

[hoijui]

I could even live with a script which parses unitdefs, etc and creates proper AI/Lua export code during the build process.

mm.. i was thinking on this, but this would require these classes, eg. UnitDef, to follow a strict format convention, including comments and special comments, used like annotations...
i dont think this is possible. it is used in the C AI Interface, as this is edited by fewer ppl (so far, kloot and me), it works there.
SSkirmishAICallback.h is parsed by an AWK script of the Java interface and the C# interface (and most likely most others to come).
but well... it is not really nice, as it is quite complex and i am pretty sure that maintaining the AWK scripts would be a horror for anyone except me (not that its nice for me ).

@tobi: yeah, sounds reasonable.
what about this:

Code: Select all

class CUnit : CWorldObjectWithProperties {
  //...
  static const int PROP_IDX_HEALTH = CUnit::GetFloatVarIndex("health");
  float getHealth() const { return getFloatProperty(PROP_IDX_HEALTH); }
  float setHealth(float x) { setFloatProperty(PROP_IDX_HEALTH, x); }
  //...
};

the pro is, it is all in one place like this, the contra.. it is kind of not nice, right? :/
it is an implementation detail though.

@zwzsg
the goals is not to write one AI that works for all mods.
the goal is, to make life of devs easier, engine, lua, AI, mod, ...
for me in particular, i see for example this pro:
i am an AI dev, that wrote an AI for BA. it is workign more or less stable now, and i want to expand it to be bale to play CA aswell, or write a second AI that plays CA.
with the new system, i can just start up my AI with CA and let it print all the property names plus descriptions into a log file, and i am ready to go. no need to ask the CA devs for all the special tags they use (which maybe noone really has a compleet list of) ...
you get it.

[imbaczek]

re generation, maybe go the other way: don't parse C++, make a simpler definition file and generate C++ (and lua, etc) from that. methods other than get/sets (which i'm not sure are a good idea, due to the monstrous patch size they'll inevitably make) could be done via a custom include.

[hoijui]

you mean, we would have engine side properties of UnitDef in an XML file, and use it to generate the C++ UnitDef setters and getters, plus the XML file could be parsed by other things.. eg the AI interface?
(sorry, not able to think right now )

[imbaczek]

almost, and i'm not sure if xml is the way to go :> but yeah, the idea is pretty much that - we have one defintions file which then gets transformed into e.g. UnitDef class, its Lua bindings, its AI bindings, etc.

i'd do it strictly in a scripting language, like e.g. this pseudo-python:

Code: Select all

class UnitDef:
    name = str
    maxdamage = int
    ...

and then use Python's reflection to emit proper C++ for the class header, Lua bindings, etc. also, it wouldn't have to be done at build time; if somebody wanted to add a unitdef tag, he'd add it to this python template, run the generator and commit the results.

complex types like structs, etc, would take some coding in the generator, of course - but as long as unitdefs are kept relatively simple, generation should be "simple", too.

(inspiration comes from django models and similar projects.)

[hoijui]

mmm.. i dont like this approach much.
and even if.. i still think XML would be better
it can be read by anything, one does not need to know an other language (eg phyton).
also, if the mod would have his definitions for unitDef too, these definitions had to be merged, and mod devs would need to code phyton or XML or whatever it would be.

[imbaczek]

the reason I suggest a scripting language is that you don't have to read in anything, you've got the data right there, it's being read by the interpreter. when using xml, there's a lot of useless boilerplate code for reading the damn thing, and in the best case you end up with a structure similar to the one described anyway.

i don't quite get why mods would want to add custom unitdef parameters; unitdefs are used mainly in the engine, after all. if you mean custom params, there should be a distinction IMHO, as is now (there's a separate custom params mapping AFAIK, at least in Lua.)

[zwzsg]

Can please please pretty please keep XML out of Spring? I can't stand XML and its hype, as XML achieve about the opposite of what it promises.

[imbaczek]

we're talking about stuff you shouldn't need to touch if you don't change engine sources.

[jK]

... I totally dislike this conversation:

1. saving unit params in a map, just for easier maintenance (something what isn't even true in my book). With the cost of a the massive slowdown of any access to those values (just count how often those params are read! such a change would cause a massive slowdown of the whole engine!)

2. most of the here requested stuff is already doable via Unit/Team/GameRules (new ressources, more advance ammo system, ...) and yeah those values are readable by the AIs already!

[hoijui]

jk, you mean even when we access the vars through an index?
i mena.. it is one addition more per access then, it can not be THAT massive of a slowdown, is it?

of course, most of the current vars would not profit much from the conversion, it is just maintenance that would be easier.
it definatly would make maintanance a lot easier, and it would unify things -> all props are in one place, accessible through the smae mechanism. its not like some are accessible as members of the class, and some as props in a collection, as it is now.

@imbaczek:

I could even live with a script which parses unitdefs, etc and creates proper AI/Lua export code during the build process.

this sounds like.. if the other thing is too much work, this would be ok for you too, but the other thing is better.
would you prefer the script thing?

the other (, initial) idea, as outlined by tobis code example...
as he showed it, implemented in a calss called WorldObjectWithProperties, it could be introduced virusly, and be tested (also performance wise) with easy, no?
if someone did it :D

[Tobi]

Accessing the variables through an index means (almost) no slowdown.

I'm pretty sure the compiler generates 100% identical code for e.g. these two pieces of code:

Code: Select all

struct UnitDef {
  float a, b, c;
};

UnitDef ud;
float temp = ud.c;

and

Code: Select all

struct UnitDef {
  float params[3];
};

UnitDef ud;
float temp = ud.params[2];

Of course if the (maximum) number of variables isn't fixed then one extra indirection is needed.

I think I tend to prefer a code generation approach too though, in particular if the generated code is as clear as human written code.

And I agree that XML would be a bad choice, way too verbose (near human unreadable), requires huge (extra) libs to parse, etc. Better go for Lua or Python.

EDIT:

Then again, do we really need technical solution for organisatorial problem?

Couldn't we just try 1) put big "greppable" comment on every place that needs a tweak when a variable is added/changed, 2) publicly scrutinize every dev who forgets to honor this comment?

[imbaczek]

Then again, do we really need technical solution for organisatorial problem?

Couldn't we just try 1) put big "greppable" comment on every place that needs a tweak when a variable is added/changed, 2) publicly scrutinize every dev who forgets to honor this comment?

I thought the current state of affairs proved that this approach doesn't quite work?

[Tobi]

I don't see any comments in UnitDef.h even hinting at the fact you need to edit more places to add/change a field.

[imbaczek]

ah yes, comments are a part of documentation and as such don't exist, true.

[hoijui]

:D

yeahh.. fi we add such comments...
eg in UnitDef.. it will not work!
they would be in the end or in hte beginning of the .h file, which is huge, and if i as a dev would want to add a bool property to UnitDef, i would not read the header of the class first, end even if i knew there are other things to edit in other classes, when i add something here... maybe i just do not care about AIs or Lua enough, to go through all the trouble and a function in four other files, if i can just add my "bool foo;" to UnitDef and what i want will work wiht that.

i mean... we have computers to do work for us that is automatable, casue this way there will be less erorrs and it is guaranteed to be done as defined by a standard, and it is done fast and for free...

i thinki i do not yet really get how it would work with generated code.
this would be used for engine defined props only? would id define member vars in aclass for each property?
maybe give sample pseudo code if possible, please :D