I've been working on a plugin called "po", that adds support for multi-lingual wikis, translated with gettext, using [po4a](http://po4a.alioth.debian.org/). More information: * It can be found in my "po" branch: `git clone git://gaffer.ptitcanardnoir.org/ikiwiki.git` * It is self-contained, *i.e.* it does not modify ikiwiki core at all. * It is documented (including TODO and plans for next work steps) in `doc/plugins/po.mdwn`, which can be found in the same branch. * No public demo site is available so far, I'm working on this. My plan is to get this plugin clean enough to be included in ikiwiki. The current version is a proof-of-concept, mature enough for me to dare submitting it here, but I'm prepared to hear various helpful remarks, and to rewrite parts of it as needed. Any thoughts on this? > Well, I think it's pretty stunning what you've done here. Seems very > complete and well thought out. I have not read the code in great detail > yet. > > Just using po files is an approach I've never seen tried with a wiki. I > suspect it will work better for some wikis than others. For wikis that > just want translations that match the master language as closely as > possible and don't wander off and diverge, it seems perfect. (But what happens > if someone edits the Discussion page of a translated page?) > > Please keep me posted, when you get closer to having all issues solved > and ready for merging I can do a review and hopefully help with the > security items you listed. --[[Joey]] >> Thanks a lot for your quick review, it's reassuring to hear such nice words >> from you. I did not want to design and write a full translation system, when >> tools such as gettext/po4a already have all the needed functionality, for cases >> where the master/slave languages paradigm fits. >> Integrating these tools into ikiwiki plugin system was a pleasure. >> >> I'll tell you when I'm ready for merging, but in the meantime, >> I'd like you to review the changes I did to the core (3 added hooks). >> Can you please do this? If not, I'll go on and hope I'm not going to far in >> the wrong direction. >> >>> Sure.. I'm not completly happy with any of the hooks since they're very >>> special purpose, and also since `run_hooks` is not the best interface >>> for a hook that modifies a variable, where only the last hook run will >>> actually do anything. It might be better to just wrap >>> `targetpage`, `bestlink`, and `beautify_urlpath`. But, I noticed >>> the other day that such wrappers around exported functions are only visible by >>> plugins loaded after the plugin that defines them. >>> >>> Update: Take a look at the new "Function overriding" section of >>> [[plugins/write]]. I think you can just inject wrappers about a few ikiwiki >>> functions, rather than adding hooks. The `inject` function is pretty >>> insane^Wlow level, but seems to work great. --[[Joey]] >>> >>>> Thanks a lot, it seems to be a nice interface for what I was trying to achieve. >>>> I may be forced to wait two long weeks before I have a chance to confirm >>>> this. Stay tuned. --[[intrigeri]] >>>> >>>>> I've updated the plugin to use `inject`. It is now fully self-contained, >>>>> and does not modify the core anymore. --[[intrigeri]] >> >> The Discussion pages issue is something I am not sure about yet. But I will >> probably decide that "slave" pages, being only translations, don't deserve >> a discussion page: the discussion should happen in the language in which the >> pages are written for real, which is the "master" one. --[[intrigeri]] >> >> I think that's a good decision, you don't want to translate discussion, >> and if the discussion page turns out multilingual, well, se la vi. ;-) >> >> Relatedly, what happens if a translated page has a broken link, and you >> click on it to edit it? Seems you'd first have to create a master page >> and could only then translate it, right? I wonder if this will be clear >> though to the user. >> >>> Right: a broken link points to the URL that allows to create >>> a page that can either be a new master page or a non-translatable >>> page, depending on `po_translatable_pages` value. The best >>> solution I can thing of is to use [[plugins/edittemplate]] to >>> insert something like "Warning: this is a master page, that must >>> be written in $MASTER_LANGUAGE" into newly created master pages, >>> and maybe another warning message on newly created >>> non-translatable pages. It seems quite doable to me, but in order >>> to avoid breaking existing functionality, it implies to hack a bit >>> [[plugins/edittemplate]] so that multiple templates can be >>> inserted at page creation time. [[--intrigeri]] >>> >>>> I implemented such a warning using the formbuilder_setup hook. >>>> --[[intrigeri]] >> >> And also, is there any way to start a translation of a page into a new >> lanauge using the web interface? >> >>> When a new language is added to `po_slave_languages`, a rebuild is >>> triggered, and all missing PO files are created and checked into >>> VCS. An unpriviledged wiki user can not add a new language to >>> `po_slave_languages`, though. One could think of adding the needed >>> interface to translate a page into a yet-unsupported slave >>> language, and this would automagically add this new language to >>> `po_slave_languages`. It would probably be useful in some >>> usecases, but I'm not comfortable with letting unpriviledged wiki >>> users change the wiki configuration as a side effect of their >>> actions; if this were to be implemented, special care would be >>> needed. [[--intrigeri]] >>> >>>> Actually I meant into any of the currently supported languages. >>>> I guess that if the template modification is made, it will list those >>>> languages on the page, and if a translation to a language is missing, >>>> the link will allow creating it? >>>> >>>>> Any translation page always exist for every supported slave >>>>> language, even if no string at all have been translated yet. >>>>> This implies the po plugin is especially friendly to people who >>>>> prefer reading in their native language if available, but don't >>>>> mind reading in English else. >>>>> >>>>> While I'm at it, there is a remaining issue that needs to be >>>>> sorted out: how painful it could be for non-English speakers >>>>> (assuming the master language is English) to be perfectly able >>>>> to navigate between translation pages supposed to be written in >>>>> their own language, when their translation level is most >>>>> often low. >>>>> >>>>> (It is currently easy to display this status on the translation >>>>> page itself, but then it's too late, and how frustrating to load >>>>> a page just to realize it's actually not translated enough for >>>>> you. The "other languages" loop also allows displaying this >>>>> information, but it is generally not the primary >>>>> navigation tool.) >>>>> >>>>> IMHO, this is actually a social problem (i.e. it's no use adding >>>>> a language to the supported slave ones if you don't have the >>>>> manpower to actually do the translations), that can't be fully >>>>> solved by technical solutions, but I can think of some hacks >>>>> that would limit the negative impact: a given translation's >>>>> status (currently = percent translated) could be displayed next >>>>> to the link that leads to it; a color code could as well be used >>>>> ("just" a matter of adding a CSS id or class to the links, >>>>> depending on this variable). As there is already work to be done >>>>> to have the links text generation more customizable through >>>>> plugins, I could do both at the same time if we consider this >>>>> matter to be important enough. --[[intrigeri]] >>>>> >>>>>> The translation status in links is now implemented in my >>>>>> `po`branch. It requires my `meta` branch changes to >>>>>> work, though. I consider the latter to be mature enough to >>>>>> be merged. --[[intrigeri]] >> FWIW, I'm tracking your po branch in ikiwiki master git in the po >> branch. One thing I'd like to try in there is setting up a translated >> basewiki, which seems like it should be pretty easy to do, and would be >> a great demo! --[[Joey]] >> >>> I've merged your changes into my own branch, and made great >>> progress on the various todo items. Please note my repository >>> location has changed a few days ago, my user page was updated >>> accordingly, but I forgot to update this page at the same time. >>> Hoping it's not too complicated to relocated an existing remote... >>> (never done that, I'm a Git beginner as well as a Perl >>> newbie) --[[intrigeri]] >>>> >>>> Just a matter of editing .git/config, thanks for the heads up. >>>>> >>>>> Joey, please have a look at my branch, your help would be really >>>>> welcome for the security research, as I'm almost done with what >>>>> I am able to do myself in this area. --[[intrigeri]] >>>>>> >>>>>> I came up with a patch for the WrapI18N issue --[[Joey]] I've set this plugin development aside for a while. I will be back and finish it at some point in the first quarter of 2009. --[[intrigeri]] > Abstract: Joey, please have a look at my po and meta branches. > > Detailed progress report: > > * it seems the po branch in your repository has not been tracking my > own po branch for two months. any config issue? > * all the plugin's todo items have been completed, robustness tests > done > * I've finished the detailed security audit, and the fix for po4a > bugs has entered upstream CVS last week > * I've merged your new `checkcontent` hook with the `cansave` hook > I previously introduced in my own branch; blogspam plugin updated > accordingly > * the rename hook changes we discussed elsewhere are also part of my > branch > * I've introduced two new hooks (`canremove` and `canrename`), not > a big deal; IMHO, they extend quite logically the plugin interface > * as highlighted on [[bugs/pagetitle_function_does_not_respect_meta_titles]], > my `meta` branch contains a new feature that is really useful in a > translatable wiki > > As a conclusion, I'm feeling that my branches are ready to be > merged; only thing missing, I guess, are a bit of discussion and > subsequent adjustments. > > --[[intrigeri]] > I've looked it over and updated my branch with some (untested) > changes. > >> I've merged your changes into my branch. Only one was buggy. > > Sorry, I'd forgotten about your cansave hook.. sorry for the duplicate > work there. > > Reviewing the changes, mostly outside of `po.pm`, I have > the following issues. > > * renamepage to renamelink change would break the ikiwiki > 3.x API, which I've promised not to do, so needs to be avoided > somehow. (Sorry, I guess I dropped the ball on not getting this > API change in before cutting 3.0..) >> >> Fixed, see [[todo/need_global_renamepage_hook]]. >> > * I don't understand the parentlinks code change and need to figure it > out. Can you explain what is going on there? >> >> I'm calling `bestlink` there so that po's injected `bestlink` is >> run. This way, the parent links of a page link to the parent page >> version in the proper language, depending on the >> `po_link_to=current` and `po_link_to=negotiated` settings. >> Moreover, when using my meta branch enhancements plus meta title to >> make pages titles translatable, this small patch is needed to get >> the translated titles into parentlinks. >> > * canrename's mix of positional and named parameters is way too > ugly to get into an ikiwiki API. Use named parameters > entirely. Also probably should just use named parameters > for canremove. > * `skeleton.pm.example`'s canrename needs fixing to use either > the current or my suggested parameters. >> >> Done. >> > * I don't like the exporting of `%backlinks` and `$backlinks_calculated` > (the latter is exported but not used). >> >> The commit message for 85f865b5d98e0122934d11e3f3eb6703e4f4c620 >> contains the rationale for this change. I guess I don't understand >> the subtleties of `our` use, and perldoc does not help me a lot. >> IIRC, I actually did not use `our` to "export" these variables, but >> rather to have them shared between `Render.pm` uses. >> >>> My wording was unclear, I meant exposing. --[[Joey]] >>> > * What is this `IkiWiki::nicepagetitle` and why are you > injecting it into that namespace when only your module uses it? > Actually, I can't even find a caller of it in your module. >> >> I guess you should have a look to my `meta` branch and to >> [[bugs/pagetitle_function_does_not_respect_meta_titles]] in order >> to understand this :) >> >>> It would probably be good if I could merge this branch without >>> having to worry about also immediatly merging that one. --[[Joey]] >> > * I'm very fearful of the `add_depends` in `postscan`. > Does this make every page depend on every page that links > to it? Won't this absurdly bloat the dependency pagespecs > and slow everything down? And since nicepagetitle is given > as the reason for doing it, and nicepagetitle isn't used, > why do it? >> >> As explained in the 85f865b5d98e0122934d11e3f3eb6703e4f4c620 log: >> this feature hits performance a bit. Its cost was quite small in my >> real-world use-cases (a few percents bigger refresh time), but >> could be bigger in worst cases. When using the po plugin with my >> meta branch changes (i.e. the `nicepagetitle` thing), and having >> enabled the option to display translation status in links, this >> maintains the translation status up-to-date in backlinks. Same when >> using meta title to make the pages titles translatable. It does >> help having a nice and consistent translated wiki, but as it can >> also involve problems, I just turned it into an option. >> > * The po4a Suggests should be versioned to the first version > that can be used safely, and that version documented in > `plugins/po.mdwn`. >> >> Done. >> >> --[[intrigeri]] > > --[[Joey]]