Phantasy Star Generation Translation tool

Discuss/post fan stuff (images, fictions, games...)

Phantasy Star Generation Translation tool

Postby Tryphon » Sun Sep 23, 2012 9:35 pm

What is it ?

algoring is a tool that can extract all the dialogs from PSG 1 or 2, and replaces them with (almost) whatever.

It is still in very alpha development and needs lots of debugging, but it should be worth a try.

It is command line based. I planned to make a complete editor but I realised that it was more comfortable to use my prefered editor (Notepad++ for me) which has always more features than I could code.

It is named after those rings you can find in these two games, that let you understand what people say. I thought about naming it Mogic Hat

How do I use it ?

algoring has two mode : extract mode and rebuild (the default, because you usually use extract once and rebuild some thousands times).

Here's a exemple :

1) I make a copy of the ISO, to work with.

2) Using my favourite ISO editor (personnally, I like UltraIso a lot ; moreover, the trial version is free and lets you use < 300 Mb Isos, which is largely sufficient for PSG1 & 2), I extract the file EVENT.DAT somewhere (let's say to d:/somewhere).

3) I launch algoring in extract mode (Note : I use / instread of \ for delimiting directories, though I work on Windows ; both should work) :

Code: Select all
algoring extract d:/somewhere/EVENT.DAT


It will extract all the sub-files contained in EVENT.DAT (using the descompressador tool, from OrakioRob's team ; glory to them for this marvellous tool).
Those sub-files will be named xx-yyyyyyyy.DAT, where xx is the number of the subfile in the EVENT.DAT, and yyyyyyyy the adress the sub-file starts.
It will the parses all these subfiles and extract strings, convert them in UTF8 encoding and decode the special codes. The results will be saved in files xx.txt (for example, file 03-00003800.DAT will be decoded in 03.txt).

4) I can now edit txt files. For example, let's open 03.txt, go to line 109, where we can find :

僕はその頃,毎晩のように悪夢にうなされていた…。[wait][end]
---
ひとりの少女が
巨大で悪魔のような怪物と戦っている夢だ。[wait][end]
---
僕は,それを近くで見ているのだが
動くことも出来ず,声も出せず,ただ少女が
魔物に痛めつけられるのを見ていることしか出来ないのだ。[wait]
[clear]そして…
少女が力尽きようとしているところで
決まって僕は目が覚める。[wait][end]


These are the first lines of text in the introduction (no, I don't speak japanese, I'll show later how I found them easily). Note the --- that separates each dialog. YOU MUSTN'T DELETE THEM, else the tool won't be able to know which dialog must be replaced by which one, resulting in all dialogs shifted, or an error.

Let's replace them by the well known introducy :

I am haunted by nightmares every night.[wait][end]
---
A young girl is battling a giant demon.[wait][end]
---
I am close by but can not move or speak!
All I can do is watch while the demon keeps striking at the girl.
Just as she is fighting for her life, I awake![wait]
[clear]I awake in my room, dimly lit by the early dawn.[wait][end]


and save the result (as 03.txt)

5) Launch algoring, this time in rebuild mode :
Code: Select all
algoring -t 03 d:/somewhere/EVENT.DAT


The -t 03 flag means : treat only file 03.txt, leave the others unchanged. It speeds up the process. If you don't use this flag, algoring will use the others txt files. Since they contains the dialogs of the game, the result would be (almost) the same, but slower.

You get a new EVENT.DAT file (in the same directory than the algoring program, unless you precised another location with -o flag)

6) With your ISO editor, remove EVENT.DAT from your PSG iso and replace it with this new EVENT.DAT.

7) Let's run PCSX2 and try :

03.gif
03.gif (102.77 KiB) Viewed 19677 times


Seems to work, but...

02.gif
02.gif (62.88 KiB) Viewed 19677 times


You can note that some letters are missing. DON'T PANIC! The conceptors of the games didn't include all latin letters in their font, but it's possible to change it (the procedure is described somewhere in PSG1 translation thread, but I think I'll make a tool to automatize the process).

Usage :

Complete (for now) list of options :

Current version is 0.1 (23/09/2012)

extract : actives extract mode (default is rebuild). Create a 'temp' directory whete it decompresses all subfiles from the EVENT.DAT (with holy descompressador tool) and a 'text' directory (if doesn't exist) where it writes all dialogs in separate txt files.

-cols n : format text so as to get at most n characters per row (default : 27, should be 32 for PSG1)

-h : shows help (and stops)

-i id1, ...,idn : ignore files with id given (in hexadecimal). DON'T USE SPACES!

-o path : precise working dir (default : current)

-rows n : format text so as to get at most n rows of characters (default : 3, should be 4 for PSG1)

-s : when extracting, adds to each string infos about the file they belong to and their adress. When rebuilding, add these info too. Be careful, if you both extract and ebuild with this flag, infos will show twice. It permits to localize dialogs (that's how I found the intro text in my example)

-t id1, ..., idn : treats only files with numeros given (in hexadecimal). DON'T USE SPACES ! It's the opposite of -i and overrides it.


Complete list of tags in strings :

[value] : replaces some meseta values in particular in shops' dialogs
[wait] : waits for the circle button to be pressed. Normally, you can ignore it since algoring should be able to automatically insert it (excepted at special occasions). Though, this part has not been heavily tested (so report bugs please).
[newline] : go to next line. Normally, automatically inserted (see above remark). More, simple carriage return has the same effect
[clear] : clear the window and go to top. Same remark as before : should be automatic, so don't use it.
[select] : pops up a Yes/No menu
[end] : end of string, should be automatic (but isn't, for now, I'll fix that)
[blue], [red], [purple], [green], [cyan], [yellow], [white] : change color of text (doh)
[cross], [triangle], [square], [circle] : print some characters that are to find on most keyboards

Oh, and while I'm at it, the character ~ can be used to denote an insecable space (not very useful in English I guess).

Known issues :

- the analysis of the options you give in the command line is very poor. You can enter misformed options and the program doesn't detect it. Thus you can get stupid errors or results searching a lot for what you did wrong.

- it seems that there's an upper limit to the size of a text subfile (I mean, the DAT subfile), around 56 Kb. Only one file is near this limit (file 03 from PSG2, the one I took in the example). Try not to be too verbose when translating it.

- For the moment, that's all, but I fear this list will grow fast.

What can I do ?

Please test it ! It's very alpha and likely full of bugs (most of them likely easy to fix, but I suspect the algorithm to make a mistake somewhere). Take a modified EVENT.DAT, for example the following, which contains original dialogs with extra infos added by the -s flag (except file 03 where I deleted some text to avoid too big a text file ; now introduction and Governor's talks are much faster), and play, speak to everyone, and report all problems.

Download testing EVENT.DAT

Especially, the game can crash. If it happens, please give all the details (which file, which line, when did it occur, if possible upload your memory card) and I'll see what I can do.

Future improvements

First, let's check the game doesn't crash (the worst problem for me), then fix all the bugs in text decoding and formatting.

Then it'll be time for improvements.

I plan (in no particular order, and without garanty I'll do it) :
- automatic Google Translation of dialogs when extracting
- a true recompression (although I don't think it'll be very effective on text, and I don't expect it to permit to overpass size limitation at 56 Kb)

If you have idea, they will be taken in consideration.

Download :

Executable : download me!

Python source :
algoring.zip
(17.49 KiB) Downloaded 868 times
Tryphon
Scorpius
Scorpius
 
Posts: 357
Joined: Wed Mar 28, 2012 11:29 pm

Re: Phantasy Star Generation Translation tool

Postby Edge » Mon Sep 24, 2012 3:13 pm

Nice work, should help the world in the whole translation process.
I've traveled back in time... 300 years... Just to call you a "Cracker"!
User avatar
Edge
Scorpius
Scorpius
 
Posts: 391
Joined: Sat Feb 20, 2010 1:59 am
Location: Quincy, IL

Re: Phantasy Star Generation Translation tool

Postby Missagh » Mon Sep 24, 2012 3:27 pm

This is mind blowing! Thank you very much for making this Tryphon, I knew you were working on something but I had no idea that you actually made it work to this extend! I will post about this in the FB page and also add it to the PS Generation 2 English Translation page eventually. Keep us updated as you progress. Hope that this will make the translation process a lot easier!
-- Missagh Alami, webmaster of Phantasy Star Cave.
User avatar
Missagh
The Administrator
The Administrator
 
Posts: 1189
Joined: Mon Jan 02, 2006 3:16 am
Location: Sweden and UK

Re: Phantasy Star Generation Translation tool

Postby Kyence » Mon Sep 24, 2012 11:03 pm

You rock!
User avatar
Kyence
Blastoid
Blastoid
 
Posts: 141
Joined: Mon Apr 04, 2011 6:24 pm

Re: Phantasy Star Generation Translation tool

Postby Missagh » Fri Sep 28, 2012 6:18 pm

-- Missagh Alami, webmaster of Phantasy Star Cave.
User avatar
Missagh
The Administrator
The Administrator
 
Posts: 1189
Joined: Mon Jan 02, 2006 3:16 am
Location: Sweden and UK

Re: Phantasy Star Generation Translation tool

Postby Tryphon » Fri Sep 28, 2012 8:56 pm

Fame !
Compliments from girl !
At last ! I can die now :)
Tryphon
Scorpius
Scorpius
 
Posts: 357
Joined: Wed Mar 28, 2012 11:29 pm

Re: Phantasy Star Generation Translation tool

Postby Paul Jensen » Sat Sep 29, 2012 7:27 am

Tryphon, thanks for releasing the source code for your translation tool. The other day I converted Orakio Rob et al's descompressador into Visual Basic, and I used it to extract the files contained in PSG1's EVENT.DAT. I couldn't make sense of some of the codes in the files until I read your source code.

Could you explain to me how the pointers are stored? I can't figure it out.

Also, have you thought about making your program compatible with Cartographer and Atlas? They're basically the standard tools for script extraction and insertion.

One more thing: Please don't use Google translate! It's just so ... crappy.
Paul Jensen
Locusta
Locusta
 
Posts: 31
Joined: Sat May 23, 2009 5:32 am

Re: Phantasy Star Generation Translation tool

Postby BenoitRen » Sat Sep 29, 2012 2:41 pm

Paul Jensen wrote:The other day I converted Orakio Rob et al's descompressador into Visual Basic

Why would you do such a horrible thing?!
Get Xenoblade Chronicles!
User avatar
BenoitRen
Moderator
Moderator
 
Posts: 3122
Joined: Fri Jul 27, 2007 5:19 pm
Location: Belgium

Re: Phantasy Star Generation Translation tool

Postby Tryphon » Sat Sep 29, 2012 4:15 pm

Paul Jensen wrote:Tryphon, thanks for releasing the source code for your translation tool. The other day I converted Orakio Rob et al's descompressador into Visual Basic, and I used it to extract the files contained in PSG1's EVENT.DAT. I couldn't make sense of some of the codes in the files until I read your source code.

Could you explain to me how the pointers are stored? I can't figure it out.


The DAT files seem to be a mix of strings and instructions (?). Some instructions do have a pointer as params. Others don't.

My first attempt to make this tool was trying to find out what were the lengths of each instruction, and which, among them, do have pointers as params. I think I succeded in that, concerning PSG1.

But when I switched to PSG2, I realized that 'instructions' codes were different. And I had difficulties figuring out the lengths of each code, which resulted in missing some pointers, or to the contrary modifying some codes that weren't pointers.

I modified my tool to have a list of probable pointers and compare with what my script found. Then I realized that this list of probable pointers was likely the true list, and that my first approach was not necessary anymore.

Pointers are now characterized by some properties, such as :
- they start at an adress multiple of 4
- they point to an adress multiple of 4, or to a string
- they are precedeed by the code I've already identified as having pointers as params
- they do not point before the first string (because those pointers, if they are pointers, won't have to be recomputed anyway)

I'm still not sure that this approach is perfect. I'm sure it's possible to build a valid DAT file where my tool will find 'false' pointers. But for now it didn't happen, and if it does, I'll try to find a better characterization, or keep a particular list of false pointers.

Also, have you thought about making your program compatible with Cartographer and Atlas? They're basically the standard tools for script extraction and insertion.

One more thing: Please don't use Google translate! It's just so ... crappy.


I'm not a ROM hacker. I don't even know these tools. I keep the idea (but I've some work finding some interesting things in SLPM file).
Tryphon
Scorpius
Scorpius
 
Posts: 357
Joined: Wed Mar 28, 2012 11:29 pm

Re: Phantasy Star Generation Translation tool

Postby Paul Jensen » Sun Sep 30, 2012 2:48 am

BenoitRen wrote:
Paul Jensen wrote:The other day I converted Orakio Rob et al's descompressador into Visual Basic

Why would you do such a horrible thing?!


Heh. :)

I did it because I couldn't get the original source to compile correctly, so I converted it over to a language I know well. Also, I wanted a GUI.

It turned out to be a great exercise in translation. I got to learn a little bit of C++ and Portuguese in the process. :)
Paul Jensen
Locusta
Locusta
 
Posts: 31
Joined: Sat May 23, 2009 5:32 am

Re: Phantasy Star Generation Translation tool

Postby CUE » Sun Oct 07, 2012 4:39 pm

Tryphon, do you have all opcodes for PSG2?
User avatar
CUE
Pinchant
Pinchant
 
Posts: 5
Joined: Sun Oct 07, 2012 4:35 pm

Re: Phantasy Star Generation Translation tool

Postby Tryphon » Sun Oct 07, 2012 5:22 pm

I hope. I think I have most (if not all) those which are followed by pointers. Do you want them ?

Since you're on the board, I thank you for your work on the compression, and I'd be grateful if you had some hints on a problem that bothers me on PSG2, about which I made a thread on romhacking.
Tryphon
Scorpius
Scorpius
 
Posts: 357
Joined: Wed Mar 28, 2012 11:29 pm

Re: Phantasy Star Generation Translation tool

Postby CUE » Sun Oct 07, 2012 7:36 pm

The font is a complete headache. I tried add a VWF font, but I'm not a PS2 expert, and I left the game.

Do you need the source code? (compression, graphic extractor)
User avatar
CUE
Pinchant
Pinchant
 
Posts: 5
Joined: Sun Oct 07, 2012 4:35 pm

Re: Phantasy Star Generation Translation tool

Postby Tryphon » Sun Oct 07, 2012 10:57 pm

I think I've found where the code for displaying text is located. I will now make experiments to change the 16b comparizons by 8b, but I'm afraid I'll have to add some andi $00ff to mask bits 9 - 16 (because test-and-branching instructions seem to operate on 32b), and I'm afraid that, doing that, the code gets longer, messing everything. We'll see...

How did you try for the VWF ? Did you find a way to specify individual width for each character ? Did you find interesting values ?

For the source code, I think it could help. I'd like to add compression for algoring, even if I'm not sure LZ compression will be really effective on text. I coded a graphic extractor on my side too, but can have missed things, so I'd gladly look at your sources :)
Tryphon
Scorpius
Scorpius
 
Posts: 357
Joined: Wed Mar 28, 2012 11:29 pm

Re: Phantasy Star Generation Translation tool

Postby Tryphon » Thu Oct 11, 2012 3:15 pm

I think I've found how to change the encoding. I hope i have identified all routines for displaying text (it seems there are 3 of these : one for dialogs, one for menus, and one for location captions). I will now make a test font and change some dialogs to check.

The main advantage in having a 1 byte encoding (rather than 2 bytes for japanese) is that we have twice as much room for english text (and thus, 56 kb size limitation for DAT files is not so much a problem anymore).

Edit : after some testing, as far as I can tell, IT WORKS. It was true and tough hacking, I wouldn't have thought I was able to do that. Now I'll look if it's possible to use a Variable Width Font (yes xrick, I now think it's possible :) )

@ CUE : Kyence sent me your binaries, but not the source. I'm still interested by these. There seems to be a little problem with the packer : I've used it on a test font file (at pos $1e000 in MAPDATA.DAT), but only the first plane of the font showed. I tried the same font file with no compression, all 3 planes showed. Since I didn't try on others graphics files, I don't know if the problem is general. I believe Kyence did use the packer and she didn't report problems.

I've found some interesting things, such as items' price, inclusing weapons (but not the stats alteration they cause).
Tryphon
Scorpius
Scorpius
 
Posts: 357
Joined: Wed Mar 28, 2012 11:29 pm

Next

Return to Fan stuff

Who is online

Users browsing this forum: No registered users and 15 guests

cron