A guide for UTF8 Language Translations

[juaxix]

This is a guide to provide all the steps to get your app translated in an elegant, efficient and simple way.

First of all, forget about the number of languages you want to introduce in your app, just think in a general process with strings and hash tables

I suppose you have some HUD's with objects, ok,...before use their names in any part of your AI code, take your time to set up a common names philosophy and rename with the following pattern.
**********************************
STEP 1: Name your hud components
**********************************
* Start your hud component names as:

lbl : for labels
btn : for buttons
chk : for checks
edt : for edits

Now is time for create a list of all these hud component names and its values.
We are going to use XML files to store the language strings.
Let's start with en.xml (see codes), we have to use a text editor with UTF8 encoding, like UltraEdit32 or Notepad++ or other freesource. New -> File -> Encode UTF8.

**********************************
STEP 2: Create the XML File for each lang
**********************************
+ Create a template XML file with english:

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?>
<language name="en" title="English">
<!-- HUD Translations here -->
<key name="hud_name.hud_component_name">*translate text here*</key>
<!-- GAME Translations here -->

<!-- Other Translations here -->
</language>


Between <language></language> we are going to add all the HUD strings, to store this data create in your MainAI one hashtable and a XML:

* languageVars: hashtable to store the strings
* languageXML: XML to parse the xml language files

and a boolean:

* isTraslated : flag to know when to call the parser function

**********************************
STEP 3: Create the Code
**********************************
In your MainAI -> main State -> onEnter ,add:

Code: Select all

if ( system.getOSLanguage ( ) == system.kLanguageSpanish ) then
       application.setCurrentUserEnvironmentVariable  (  "language", "es" )
      elseif ( system.getOSLanguage ( ) == system.kLanguageJapanese ) then
       application.setCurrentUserEnvironmentVariable  (  "language", "ja" )
      else --default language:
       application.setCurrentUserEnvironmentVariable  (  "language", "en" )
      end


you could use anothe hashtable here with codes and so on...

With the current language in an environment variable, we can call the receive event of the XML, add the code after:

Code: Select all

    xml.empty ( this.languageXML ( ) )
    -- load language
    xml.receive (
      this.languageXML ( ), "file://"..application.getPackDirectory ( ).."/"..
      application.getCurrentUserEnvironmentVariable ( "language" )..".xml"
    )
    this.isTraslated ( false )


this receive instruction will read en.xml from the project directory file, now all we have to do is wait for the parsing process to end in MainAI -> onLoop, add:

Code: Select all

if ( not this.isTraslated ( ) ) then
     if (xml.getReceiveStatus ( this.languageXML ( )) == 1 ) then
        user.postEvent ( application.getCurrentUser ( ), 0,"MainAI","onLoadLanguageXML","mainHud" )
        this.isTraslated ( true )
     end
    end


This onLoadLanguageXML is our parser,

Code: Select all

function main.onLoadLanguageXML ( hudname )
--------------------------------------------------------------------------------
    local hRoot      = xml.getRootElement ( this.languageXML ( ) )
    local me         = application.getCurrentUser ( )
    local n          = xml.getElementChildCount ( hRoot  ) - 1
    local vars       = this.languageVars ( )
    local hAttribute = xml.getElementAttributeAt ( hRoot, 0 )
    hashtable.empty ( vars )
    for i=0, n  do
     local child = xml.getElementChildAt ( hRoot, i )
     local s = xml.getElementName (child) 
     if (  s == "key" )  then
        -- add
        hashtable.add (
          vars,
          xml.getAttributeValue (xml.getElementAttributeAt ( child,0 )).."",
          xml.getElementValue ( child ) ..""
        )
     end
   end
    -- clean memory ;)
    xml.empty ( this.languageXML ( ) )
    if ( hudname ~= "" ) then
     user.postEvent ( me, 0, "MainAI", "onApplyLanguageToHUD" , hudname )
    end
--------------------------------------------------------------------------------
end


well, now, if our game starts with a hud, we must first load this hud before call the "translate" function, with:

Code: Select all

hud.newTemplateInstance ( application.getCurrentUser ( ), "mainHud", "mainHud" )


(or we can add a check in the translate function, your choice).
In the example I call the onLoadLanguageXML function with "mainHud", and when the parser finish the whole movement it calls to onApplyLanguageToHUD, which is the following function:

Code: Select all

function main.onApplyLanguageToHUD ( hud_name )
--------------------------------------------------------------------------------
    local me   = application.getCurrentUser ( )
    local vars = this.languageVars ( )
    local i,n,m  = 0, hashtable.getSize ( vars ), string.getLength ( hud_name )
    for i=0,n-1 do
     local key = hashtable.getKeyAt ( vars, i )
     local hdn = string.getSubString ( key, 0, m)
     if (  hdn == hud_name ) then
      -- include this hud in the translation
      local c = hud.getComponent ( me, key )
      if (  c == nil ) then
       -- nothing to do here XD
       log.warning ( "Key: ",key,". wrong" )
      else
       local obj = string.getSubString ( string.getSubString (
                   key, m+1,
                   string.getLength ( key ) - m - 1 )
                   , 0,3)

       if (  obj == "lbl" ) then
        -- label
        hud.setLabelText ( c,  hashtable.get ( vars, key ) )
       elseif (obj=="btn") then
        -- button
        hud.setButtonText( c, hashtable.get ( vars, key ) )
       elseif (obj=="chk") then
        --check
        hud.setCheckText( c, hashtable.get ( vars, key ) )
       elseif (obj=="edt") then
        --edit
        hud.setEditText ( c, hashtable.get ( vars, key ) )
       else
        --
        -- no more translations for the HUD, exit.
        -- This is an efficient issue
        --       
        return nil
       end

      end --check component
     
     end --check hud

    end --for loop
--------------------------------------------------------------------------------
end



Add in your code the calls to translate the hud's just after its creation with:

Code: Select all

user.postEvent ( me, 0, "MainAI", "onApplyLanguageToHUD" , hudname )


To change the language to Japanese:

Code: Select all

     -- save the selected language in the current user environment variable
     application.setCurrentUserEnvironmentVariable ( "language", "ja" )
     application.saveCurrentUserEnvironmentVariable( "language" )
     -- load the language from XML
     xml.empty  ( this.languageXML ( ) )
     this.isTraslated ( false )
     xml.receive (
         this.languageXML ( ),
         "file://"..
         application.getPackDirectory ( ).."/ja.xml"
     )


Remember to add a True Font Type importing it as dynamic and set this font in all the hud components with UTF8, if you want to add text in UTF8 directly to the HUD in the Shiva Editor, just encode the UTF8 string with ANSI (with the notepad++ or another) and then copy paste into the text label.

_______________________
EXAMPLE: JAPANESE
_______________________

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?>
<language name="ja" title="Japanese">
<!-- HUD Translations -->
<key name="mainHud.btnPlay">今すぐプレイ!</key>
</language>


the first thing to do is just set language environment variable to "ja", then empty the xml, set false the isTraslated boolean var.
In the end of the parser function we have our function onApplyLanguageToHud with "mainHud" as parameter.
If you have a button named "btnPlay" with an english text in the HUD named "mainHud" it will be translated with "今すぐプレイ!" .
We can also have this value with

Code: Select all

hashtable.get( this.languageVars(), "mainHud.btnPlay" )


and do the translations ourselves

If you are using XCode Project, go to Project and target Info configuration and in Localizations add all the languages, then to let the AppleStore know those languages create,in xcode, a new file -> resource -> Strings, call it Localizable.strings, and select this new file, it will appear in the right bar of the Xcode Ide (4.0+), the File Inspector, where you would see "Localization", there you have to add the languages also, doing this the binary will write with all those information in it header.

Hope you like the guide.

[NiCoX]

Great tutorial/guide, thanks for sharing

An additional idea: if you set up such a system a bit late (ie. not at the beginning of your project), you can create a script that will parse every HUD components and generate the template XML file.

[rusty]

Hi, nice tutorial...


Concerning...

True Font Type importing it as dynamic and set this font in all the hud components with UTF8

When you say 'dynamic', do you mean to overwrite / override the one being used, at run time?

For instance I have used Create -> Resource -> Font and chosen a TrueType one (DejaVu), but it obviously will not handle Japanese, but will, say, Spanish fine. If I detect that I need Japanese, will I need to rewrite the font file used by the HUD's? Else how?



Example Spanish xml file...

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?>
<language name="es" title="Spanish">
<!-- HUD Translations here -->
<key name="mainHud.btnPlay">Desempeñar!</key>
<key name="mainHud.lblTitle">Español</key>
<!-- GAME Translations here -->

<!-- Other Translations here -->
</language>


BTW, I think that 'Jugar' is a better translation for 'Play', but, hey, I ain't got a clue at the moment!

[juaxix]

Hi, nice tutorial...
...
When you say 'dynamic', do you mean to overwrite / override the one being used, at run time?
...

BTW, I think that 'Jugar' is a better translation for 'Play', but, hey, I ain't got a clue at the moment!

Dynamic is the font type, if your font does not come with the language set you want to use you have to change the components fonts in running time or choose a font that includes all languages encodings, i think this last option is the best one...

And yes ,'Jugar' is the actual translation of 'Play'
See here:
http://www.youtube.com/watch?v=rNwJ5bJz2yo

[rusty]

Cheers... there's a few more opciones there too!

Well, i've not found any dynamic fonts yet and the test Japanese ones i've played with just seem to provide letter for letter translations, so come through just as 'blocks' as well... but will look deeper later...


Here's how I quickly set it up to change the font.

Code: Select all

--------------------------------------------------------------------------------
function Lang_MainAI.onApplyLanguageToHUD ( hud_name )
--------------------------------------------------------------------------------
   
    local l = application.getCurrentUserEnvironmentVariable ( "language" )
    local font = "Arial"
    if(l == "es") then
        font = "Arial"
    elseif(l == "ja") then
        font = "Japanese"
    end

    local me   = application.getCurrentUser ( )
    local vars = this.languageVars ( )
    local i,n,m  = 0, hashtable.getSize ( vars ), string.getLength ( hud_name )
    for i=0,n-1 do
        local key = hashtable.getKeyAt ( vars, i )
        local hdn = string.getSubString ( key, 0, m)
        if (  hdn == hud_name ) then
            -- include this hud in the translation
            local c = hud.getComponent ( me, key )
            if (  c == nil ) then
                -- nothing to do here XD
                log.warning ( "Key: ",key,". wrong" )
            else
                local obj = string.getSubString ( string.getSubString (
                    key, m+1,
                    string.getLength(key) - m - 1)
                    , 0,3)
               
                if (  obj == "lbl" ) then
                    -- label
                    hud.setLabelText(c,  hashtable.get(vars, key))
                    hud.setLabelFont(c, font)
                elseif (obj=="btn") then
                    -- button
                    hud.setButtonText(c, hashtable.get(vars, key))
                    hud.setButtonFont(c, font)
                elseif (obj=="chk") then
                    --check
                    hud.setCheckText(c, hashtable.get(vars, key))
                    hud.setCheckFont(c, font)
                elseif (obj=="edt") then
                    --edit
                    hud.setEditText(c, hashtable.get(vars, key))
                    hud.setEditFont(c, font)
                else
                    --
                    -- no more translations for the HUD, exit.
                    -- This is an efficient issue
                    --
                    return nil
                end
           
            end --check component
       
        end --check hud
       
    end --for loop
   
--------------------------------------------------------------------------------
end
--------------------------------------------------------------------------------


It's probably inefficient, so later i'll set it up to use the default font and just change that globally:

Code: Select all

sFont = hud.getDefaultFontName ( hUser )
hud.setDefaultFont ( hUser, sFont )


[flippout]

Great tutorial/guide, thanks for sharing

An additional idea: if you set up such a system a bit late (ie. not at the beginning of your project), you can create a script that will parse every HUD components and generate the template XML file.

Hey NicoX, any suggestions on how to start with such an endeavor?

I thought perhaps I could make use of hud.getComponentCount() and hud.getcomponentAt() - but it looks like this will simply return the components for the huds that are active right now. My app has a variety of huds that all exist at different times during the app's lifecycle.

Is there some other way to parse this info?

[rusty]

... choose a font that includes all languages encodings, i think this last option is the best one...

I don't know if it's me but all the dynamic fonts i'm finding, once imported, well become very un-dynamic, static even!


Have you got an example font and if need be, any special instructions for importing into ShiVa.


Many thanks!


EDIT:

It's Japanese i'm really interested in!

[broozar]

nice! i totally overlooked this helpful thread.



http://www.stonetrip.com/developer/wiki/index.php?title=A_guide_for_UTF8_Language_Translations