git: Fix handling of utf-8 filenames in recentchanges.

Seems that the problem is that once the \nnn coming from git is converted
to a single character, decode_utf8 decides that this is a standalone
character, and not part of a multibyte utf-8 sequence, and so does nothing.
I tried playing with the utf-8 flag, but that didn't work. Instead, use
decode("utf8"), which doesn't have the same qualms, and successfully
decodes the octets into a utf-8 character.

Rant:

Think for a minute about fact that any and every program that parses git-log,
or git-show, etc output to figure out what files were in a commit needs to
contain this snippet of code, to convert from git-log's wacky output to a
regular character set:

if ($file =~ m/^"(.*)"$/) {
	($file=$1) =~ s/\\([0-7]{1,3})/chr(oct($1))/eg;
}

(And it's only that "simple" if you don't care about filenames with
embedded \n or \t or other control characters.)

Does that strike anyone else as putting the parsing and conversion in the
wrong place (ie, in gitweb, ikiwiki, etc, etc)? Doesn't anyone who actually
uses git with utf-8 filenames get a bit pissed off at seeing \xxx\xxx
instead of the utf-8 in git-commit and other output?

diff --git a/IkiWiki/Plugin/git.pm b/IkiWiki/Plugin/git.pm
index 08ee4cb63c07a5cc572e33f9f1fac262f4d58cd9..14b0ab2851dc03b561e4c86e9e0c5d660a8ca3a2 100644 (file)
--- a/IkiWiki/Plugin/git.pm
+++ b/IkiWiki/Plugin/git.pm
@@ -308,13 +308,16 @@ sub parse_diff_tree ($@) { #{{{
                        my $sha1_to = shift(@tmp);
                        my $status = shift(@tmp);
 
+                       # git does not output utf-8 filenames, but instead
+                       # double-quotes them with the utf-8 characters
+                       # escaped as \nnn\nnn.
                        if ($file =~ m/^"(.*)"$/) {
                                ($file=$1) =~ s/\\([0-7]{1,3})/chr(oct($1))/eg;
                        }
                        $file =~ s/^\Q$prefix\E//;
                        if (length $file) {
                                push @{ $ci{'details'} }, {
-                                       'file'      => decode_utf8($file),
+                                       'file'      => decode("utf8", $file),
                                        'sha1_from' => $sha1_from[0],
                                        'sha1_to'   => $sha1_to,
                                };

diff --git a/debian/changelog b/debian/changelog
index 002591083a5d19b6301c1c7a9c8facc9879cb6ef..db80dcf837ca3087ba26f38e2e46aac626be6965 100644 (file)
--- a/debian/changelog
+++ b/debian/changelog
@@ -5,6 +5,7 @@ ikiwiki (2.66) UNRELEASED; urgency=low
     ctime.
   * attachment: Add admin() pagespec to test if the uploading user is a wiki
     admin.
+  * git: Fix handling of utf-8 filenames in recentchanges.
 
  -- Joey Hess <joeyh@debian.org>  Thu, 25 Sep 2008 13:45:55 -0400
 

diff --git a/doc/bugs/unicode_encoded_urls_and_recentchanges.mdwn b/doc/bugs/unicode_encoded_urls_and_recentchanges.mdwn
index 9184a6b84ead0d193f15ad4f52dabd6283f04629..2568aef387638dbb2f2cc1e7d7a11dec11fe603c 100644 (file)
--- a/doc/bugs/unicode_encoded_urls_and_recentchanges.mdwn
+++ b/doc/bugs/unicode_encoded_urls_and_recentchanges.mdwn
@@ -1,9 +1,13 @@
 it appears that unicode characters in the title that are unicode letters are spared the __ filename encoding but instead saved in their utf8 encoding. (correct me if i'm wrong; didn't find the code that does this.) -- see below for examples.
 
+> Filenames can have any alphanumerics in them without the __ escaping.
+> Your locale determines whether various unicode characters are considered
+> alphanumeric. In other words, it just looks at the [[:alpha:]] character
+> class, whatever your locale defines it to be. --[[Joey]]
+
 this is not a problem per se, but (at least with git backend) the recent changes missinterpret the file name character set (it seems to read the filenames as latin1) and both display wrong titles and create broken links.
 
 the problem can be shown with an auto-setup'd ikiwiki without cgi when manually creating utf8 encoded filenames and running ikiwiki with LANG=en_GB.UTF-8 .
 
-----
+> Encoding issue, I figured out a fix. [[done]] --[[Joey]]
 
-[[right→link]], [[germanöumlaut]], [[smalµmu]], [[cyrillicԀletter]], [[tibetan༆something]], [[devanagariऔletter]], [[subscript₉nine]], [[subscriptₐa]]