Fixes for behavior changes in perl 5.10's CGI

Something has changed in CGI.pm in perl 5.10. It used to not care
if STDIN was opened using :utf8, but now it'll mis-encode utf-8 values
when used that way by ikiwiki. Now I have to binmode(STDIN) before
instantiating the CGI object.

In 57bba4dac132a06729eeec809f5e1a5adf829806, I changed from decoding
CGI::Formbuilder fields to utf-8, to decoding cgi parameters before setting
up the form object. As of perl 5.10, that approach no longer has any effect
(reason unknown). To get correctly encoded values in FormBuilder forms,
they must once again be decoded after the form is set up.

As noted in 57bba4da, this can cause one set of problems for
formbuilder_setup hooks if decode_form_utf8 is called before the hooks, and
a different set if it's called after. To avoid both sets of problems, call
it both before and after. (Only remaining problem is the sheer ugliness and
inefficiency of that..)

I think that these changes will also work with older perl versions, but I
haven't checked.

Also, in the case of the poll plugin, the cgi parameter needs to be
explcitly decoded before it is used to handle utf-8 values. (This may have
always been broken, not sure if it's related to perl 5.10 or not.)

diff --git a/IkiWiki/CGI.pm b/IkiWiki/CGI.pm
index 781974c13d5c8dde65e772bc0c4c3f952fc0f958..5fccfb4741d4375760777a530916fa535e0273c2 100644 (file)
--- a/IkiWiki/CGI.pm
+++ b/IkiWiki/CGI.pm
@@ -77,10 +77,13 @@ sub check_canedit ($$$;$) { #{{{
        return $canedit;
 } #}}}
 
-sub decode_cgi_utf8 ($) { #{{{
-       my $cgi = shift;
-       foreach my $f ($cgi->param) {
-               $cgi->param($f, map { decode_utf8 $_ } $cgi->param($f));
+sub decode_form_utf8 ($) { #{{{
+       my $form = shift;
+       foreach my $f ($form->field) {
+               $form->field(name  => $f,
+                            value => decode_utf8($form->field($f)),
+                            force => 1,
+               );
        }
 } #}}}
 
@@ -103,7 +106,6 @@ sub cgi_signin ($$) { #{{{
        my $q=shift;
        my $session=shift;
 
-       decode_cgi_utf8($q);
        eval q{use CGI::FormBuilder};
        error($@) if $@;
        my $form = CGI::FormBuilder->new(
@@ -127,10 +129,12 @@ sub cgi_signin ($$) { #{{{
        $form->field(name => "do", type => "hidden", value => "signin",
                force => 1);
        
+       decode_form_utf8($form);
        run_hooks(formbuilder_setup => sub {
                shift->(form => $form, cgi => $q, session => $session,
                        buttons => $buttons);
        });
+       decode_form_utf8($form);
 
        if ($form->submitted) {
                $form->validate;
@@ -161,7 +165,6 @@ sub cgi_prefs ($$) { #{{{
        my $session=shift;
 
        needsignin($q, $session);
-       decode_cgi_utf8($q);
        
        # The session id is stored on the form and checked to
        # guard against CSRF.
@@ -197,11 +200,13 @@ sub cgi_prefs ($$) { #{{{
                ],
        );
        my $buttons=["Save Preferences", "Logout", "Cancel"];
-
+       
+       decode_form_utf8($form);
        run_hooks(formbuilder_setup => sub {
                shift->(form => $form, cgi => $q, session => $session,
                        buttons => $buttons);
        });
+       decode_form_utf8($form);
        
        $form->field(name => "do", type => "hidden", value => "prefs",
                force => 1);
@@ -255,8 +260,6 @@ sub cgi_editpage ($$) { #{{{
        my $q=shift;
        my $session=shift;
        
-       decode_cgi_utf8($q);
-       
        my @fields=qw(do rcsinfo subpage from page type editcontent comments);
        my @buttons=("Save Page", "Preview", "Cancel");
        eval q{use CGI::FormBuilder};
@@ -276,10 +279,12 @@ sub cgi_editpage ($$) { #{{{
                wikiname => $config{wikiname},
        );
        
+       decode_form_utf8($form);
        run_hooks(formbuilder_setup => sub {
                shift->(form => $form, cgi => $q, session => $session,
                        buttons => \@buttons);
        });
+       decode_form_utf8($form);
        
        # This untaint is safe because titlepage removes any problematic
        # characters.
@@ -366,6 +371,7 @@ sub cgi_editpage ($$) { #{{{
                }
 
                my $content=$form->field('editcontent');
+
                run_hooks(editcontent => sub {
                        $content=shift->(
                                content => $content,
@@ -379,7 +385,7 @@ sub cgi_editpage ($$) { #{{{
                        linkify($page, $page,
                        preprocess($page, $page,
                        filter($page, $page, $content), 0, 1))));
-               
+       
                if ($new) {
                        delete $pagesources{$page};
                }
@@ -642,7 +648,9 @@ sub cgi (;$$) { #{{{
                eval q{use CGI};
                error($@) if $@;
        
+               binmode(STDIN);
                $q=CGI->new;
+               binmode(STDIN, ":utf8");
        
                run_hooks(cgi => sub { shift->($q) });
        }

diff --git a/IkiWiki/Plugin/poll.pm b/IkiWiki/Plugin/poll.pm
index 82e862c59b92b8162e1684cd6ff4badbd8d37486..6edf233b4f07e3b7b926a765970bdaed538b75b8 100644 (file)
--- a/IkiWiki/Plugin/poll.pm
+++ b/IkiWiki/Plugin/poll.pm
@@ -4,6 +4,7 @@ package IkiWiki::Plugin::poll;
 use warnings;
 use strict;
 use IkiWiki 2.00;
+use Encode;
 
 sub import { #{{{
        hook(type => "preprocess", id => "poll", call => \&preprocess);
@@ -78,7 +79,7 @@ sub sessioncgi ($$) { #{{{
        my $cgi=shift;
        my $session=shift;
        if (defined $cgi->param('do') && $cgi->param('do') eq "poll") {
-               my $choice=$cgi->param('choice');
+               my $choice=decode_utf8($cgi->param('choice'));
                if (! defined $choice) {
                        error("no choice specified");
                }

diff --git a/debian/changelog b/debian/changelog
index 118a0347d64cb6204dbae9a13b2b5e07c68a56fe..080de759bbe6600abc12ba94be673d93dd54ef3a 100644 (file)
--- a/debian/changelog
+++ b/debian/changelog
@@ -9,6 +9,8 @@ ikiwiki (2.46) UNRELEASED; urgency=low
     wikis that automatically ping one another to stay up to date.
   * Optimised file statting code when scanning for modified pages;
     cut the number of system calls in half. (Still room for improvement.)
+  * Fixes for behavior changes in perl 5.10's CGI that broke utf-8 support
+    in several interesting ways.
 
  -- Joey Hess <joeyh@debian.org>  Mon, 05 May 2008 19:34:51 -0400
 

diff --git a/doc/bugs/poll_plugin:_can__39__t_vote_for_non-ascii_options.mdwn b/doc/bugs/poll_plugin:_can__39__t_vote_for_non-ascii_options.mdwn
index e09e6c29deead12def1aaa8d2cc7d12b81b491b3..0f045c25406413da6bf3d30f9e7af8db2353ed30 100644 (file)
--- a/doc/bugs/poll_plugin:_can__39__t_vote_for_non-ascii_options.mdwn
+++ b/doc/bugs/poll_plugin:_can__39__t_vote_for_non-ascii_options.mdwn
@@ -3,7 +3,5 @@ I don't seem to be able to vote for options that have non-ascii names, using the
 As an example, see http://test.liw.fi/testpoll/index.html: the "red", "green", and "blue" options work fine, but the "ehkä" one does not.
 --[liw](http://liw.fi/)
 
-> It's not just the poll plugin, editing a page with utf-8 will corrupt it
-> now! I wonder if this is a perl 5.10 change; it was working just fine as
-> recently as May 8th; I upgraded perl on the 9th and it's broken for me
-> now. --[[Joey]]
+> Ok, 4.5 hours of beating my head against a brick wall, and I've fixed this.
+> [[done]] --[[Joey]]