Encoding of string data in Spring?

[Anarchid]

So, i've been thinking of having some i18n being of some use to Spring in general, and to things i want to do with it, specifically.

Obviously, some level of that 'can already be done with lua' (even if in ugly form of translation tables and unitdef post-processing), so i'm not going to lobby for an engine inclusion of gettext and a lua interface to its calls - even though that would be so very nice.

However, a question buggeth me: how much does Spring understand in terms of Unicode support? Can i, for instance, have Cyrillic (or just maybe Chinese) characters in unit names or script messages?

If i can't, then why? Is this just the issue of lacking fonts, which i can override (possibly mod-side, possibly by deploying with engine in a package?), or something buried deeper, like the engine not even getting the idea of multibyte characters?

[CarRepairer]

I believe you're out of luck. See ZK. We have unit translations but the engine doesn't even recognize many european characters, such as Polish letters.

[smoth]

which is odd because Swedish does have special characters

[Anarchid]

Any ideas on how terribly hard would it be to upgrade human-readable parts of the text to unicode? Even if going to such ugly depths as encoding it with escapes as in html?

[knorke]

I guess one could make its own bitmap font drawing thing with chinese symbols.

[smoth]

there are a GREAT deal of them...

[Tobi]

One could say that, at the moment, Spring is guaranteed unicode unaware.

Lua strings are just arrays of bytes, not arrays of characters, and internally pretty much all strings (user visible or not) are std::strings, which are also just arrays of bytes, not arrays of characters.

The fact that these types can store any blob of bytes might help though - with a font renderer that decodes the bytes into characters using UTF-8, suddenly many parts of the engine may "support" Unicode. Will hit bugs though in code that actually manipulates / indexes / sorts / compares those strings though, and it won't be easy to find and fix all those bugs, I think.

Also, a font renderer with Unicode support has its own issues (too many characters to eagerly draw onto a texture atlas...).

[Anarchid]

Hmm. I polled a couple of guys on those scary std:string thingies and opengl - they seem somewhat confident this could be arsed just with content, so long as noone mixes encodings.

Looks like there is one surefire way to find out - by staging tests. (this evening, probs)

Not holding any high hopes though, seeing as most of what i know about cpp and guys who write it, is accrued from reading the FQA

[gajop]

if you're going to allow spring to display anything other than ASCII, please don't do any hacks and just use unicode (there's a reason it's a standard)

[hoijui]

gajop +1!
do not introduce more hacks, cause hacks are bad.

[Anarchid]

do not introduce more hacks, cause hacks are bad.

This sentiment seems to imply lobbying for (and-or implementing) Engine support of Unicode (and possibly gettext) as a better alternative.

Still, if this can be implemented solely or initially as a content-side solution, wouldn't that be better for the time being?

[gajop]

do not introduce more hacks, cause hacks are bad.

This sentiment seems to imply lobbying for (and-or implementing) Engine support of Unicode (and possibly gettext) as a better alternative.

Still, if this can be implemented solely or initially as a content-side solution, wouldn't that be better for the time being?

I've no idea what's going in the engine (is std::string used?), but you can attempt to replace it with http://site.icu-project.org/ or in case of char[], you can replace it with http://en.wikipedia.org/wiki/C%2B%2B11# ... g_literals (if we're using the c++1x compiler?).

The reason it would be bad to implement a 'temporary' solution is that it will in all likelyhood become a permanent one, and when someone finally decides to migrate the engine to unicode, it will be a pain to integrate it with the proposed custom encoded user side.

So, take your time, we are in no rush, but do it properly.

[CarRepairer]

If you're going in that direction, in zk the plan is to just replace letters. Like ö with oe (hope I got that right). The unitdef files have the proper translations and within the game characters will be replaced. I haven't implemented that part yet however if you've looked at zk the units have a bunch of translations such as polish, finnish, italian. Press esc and click on the flag to see them.

[Anarchid]

If you're going in that direction, in zk the plan is to just replace letters.

That pretty much rules out any languages not using a predominantly-latin alphabet - because at that point, you'd be transliterating not only umlauts or special characters, but the whole text - and would be better without having such a translation at all.

So, take your time, we are in no rush, but do it properly.

Depends on who is to take that time. As i mentioned earlier, me, i'm not really a cpp guy, and thus my ability to push things into engine by writing them is, at best, questionable and limited.

And the actual engine devs could really focus their time better on really important stuff like the path-follower.

Though, if it comes that the content-wise solution won't work anyway, i guess either of these (learn cpp, hack engine | troll dev, lobby modification) ways is the one to pick.

[CarRepairer]

Of course russian and chinese are out. But all of our translations are latin alphabet based. So it's better to at least fix those up than have dollar signs and question marks everywhere.

[Anarchid]

So it's better to at least fix those up than have dollar signs and question marks everywhere.

I guess so.

Also, i've tried what i could with the content, and it seems that without reworking the engine string handling, there won't be no utf-joy for anyone :\

[Pako]

I did hacky hacks for translations, mostly seems to work perfectly.

example:

Code: Select all

Chili.Label.SetCaption = function(self, newcaption)
	newcaption = gettext(newcaption)
	if (self.caption == newcaption) then return end 
	self.caption = newcaption
	self:UpdateLayout()
	self:Invalidate()
      end

[zerver]

IMO multi-language support is not a priority at the moment.

I have done some unicode development under visual studio, which has excellent helper macros and stuff to simplify things, but code cleanliness still suffers a lot.

[AF]

IMO as a member of an english speaking forum multi-language support (also known as non-english support) is not a priority at the moment (for the english speaking engine developers).

I have done some unicode development under visual studio, which has excellent helper macros and stuff to simplify things, but code cleanliness still suffers a lot.

I think it's laughable that despite numerous exposes, the people of this forum are still utterly ignorant of the large chinese and european sub communities that have shunned this forum, and the works they've accomplished.

Multi-language support is a priority, and people have put a lot fo effort into trying to get around the issue. Go look at the Chinese community and their attempts to make the english parts of our UIs playable.

It's rather easy to sit back in your fort eating pork ribs and say there's no food issue when there are 9 billion people locked outside of your 50 year food store starving to death

[zerver]

Likewise is it easy to sit back and say that it is a priority without being willing to code it. It is possibly the most unrewarding/uninteresting engine rewrite you can make. Trust me, it is not a priority unless a developer comes and says so.