Re: coding problem in article display with gnus 5.13 incl. in emacs cvs

[Patrick Drechsler]

Reiner Steib <address@hidden> writes:

> On Tue, Apr 08 2008, Patrick Drechsler wrote:
>
>> after updating GNU/Emacs CVS a few days ago I noticed that the
>> article buffer sometimes does not display umlaut characters correctly
>> anymore.
>
> I'm not aware of any changes in Gnus, thus most probably a change in
> Emacs is the reason.  (Cc-ing emacs-devel, keeping full citation.)
>
>> Symptoms: Either the article buffer displays (A) the character's
>> control sequence (ie "ä" -> "\344") or (B) it skips the character it
>> can not display. This did not happen in previous versions of Gnus
>> v5.13 shipped with Emacs CVS.
>>
>> To reproduce (A):
[...snipped...]

> Does the problem only appear in Gcc-ed messages or in usual mail/news
> messages as well?  If the later, could you send such a message to
> address@hidden (gmane.test).

If I send a test message to gmane.test[1] using Thunderbird, and then
view the message with Gnus, the characters are displayed correctly when
visiting the group. On the other hand Gnus displays the message
incorrectly in my "Sent" folder:
,----
| User-Agent: Thunderbird 1.5.0.14ubu (X11/20080306)
| MIME-Version: 1.0
| Newsgroups: gmane.test
| Subject: test
| Content-Type: text/plain; charset=UTF-8; format=flowed
| Content-Transfer-Encoding: 8bit
| [...]
|         character: ä (4194276, #o17777744, #x3fffe4)
| preferred charset: eight-bit (Raw bytes 0-255)
|        code point: 0xE4
|            syntax: w  which means: word
|       buffer code: #xE4
|         file code: not encodable by coding system windows-1252-unix
|           display: no font available
`----

Also other "normal" Emails are displayed incorrectly. Example:
,----
| User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
| MIME-Version: 1.0
| Content-Type: text/plain; charset=UTF-8; format=flowed
| Content-Transfer-Encoding: 8bit
| [...]
|         character: ä (4194276, #o17777744, #x3fffe4)
| preferred charset: eight-bit (Raw bytes 0-255)
|        code point: 0xE4
|            syntax: w  which means: word
|       buffer code: #xE4
|         file code: not encodable by coding system windows-1252-unix
|           display: no font available
`----

>> Placing the cursor on "\344" and `M-x describe-char' results in:
>>
>> ,----
>> |         character: \344 (4194276, #o17777744, #x3fffe4)
>> | preferred charset: eight-bit (Raw bytes 0-255)
>> |        code point: 0xE4
>> |            syntax: w       which means: word
>> |       buffer code: #xE4
>> |         file code: not encodable by coding system undecided-unix
>> |           display: no font available
>> | 
>> | Character code properties are not shown: customize what to show
>> | 
>> | There are text properties here:
>> |   auto-composed        t
>> `---- 
>>
>> Here is the output of `M-x describe-char' from the ä character in
>> second sentence:
>>
>> ,----
>> |         character: ä (228, #o344, #xe4)
>> | preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
>> |        code point: 0xE4
>> |            syntax: w       which means: word
>> |          category: j:Japanese l:Latin
>> |       buffer code: #xC3 #xA4
>> |         file code: #xC3 #xA4 (encoded by coding system utf-8-emacs-unix)
>> |           display: by this font (glyph code)
>> |      bitstream vera sans 
>> mono:pixelsize=13:foundry=bitstream:weight=regular:slant=roman:width=normal 
>> (#x6C)
>> `----
>>
>> (Don't ask me why the category indicates "j:Japanese")
>>
>> Concerning (B):
>>
>> On other occations I noticed that displayed umlauts are deleted or
>> replaced by a space.
>>
>> Here is an example:
>>
>> ,----
>> | List-Id: Linux <linux.listserv.heise.de>
>> | To: address@hidden
>> | From: Linux <address@hidden>
>> | Date: Mon, 07 Apr 2008 04:02:02 +0200
>> | Content-Type: multipart/alternative; boundary="-----b007da1ee42"
>> | Content-Transfer-Encoding: 8bit
>> | MIME-Version: 1.0
>> | Subject: heise online News Linux 07.04.2008
>> | Message-Id: <address@hidden>
>> | [...] 
>> |
>> | *  Nachrichten berblick zum Thema Linux der vergangenen sieben Tage        
>>   *
>> | *
>> |    Benchmark-Suite f r Linux
>> | 
>> |    Phoronix hat eine umfassende Benchmark-Suite f r Linux ver ffentlicht.
>> | [...] 
>> `----
>>
>> Can other people using Gnus (not "No Gnus") with current Gnu Emacs
>> CVS confirm this behaviour?
>
> FTR: Gnus 5.13 is the same as "No Gnus", modulo delays in
> synchronization.

OK, thanks for correcting me.

>> Going through the Gnus settings below, the only thing that catches my
>> eye is "mm-charset-synonym-alist". Searching the Gnus manual for the
>> terms "charset" results in section 3.20 Charsets. But I have trouble
>> understanding it.
>
> `mm-charset-synonym-alist' should be irrelevant.

OK.

>> Can anybody give me a pointer which variable(s) need to be adopted?
>>
>> Previous versions of Gnus v5.13 included with Emacs CVS displayed the
>> article buffer correctly.
>
> It would be very good if you could identify the change causing the
> problem using bisection (specifying the date to cvs).

I have never done this, but I'll give a try this weekend. How do I check
out older cvs versions by date? This does not work:

--8<---------------cut here---------------start------------->8---
cvs -z3 \
    -D "2008-02-01 00:00:00" \
    -d:pserver:address@hidden:/cvsroot/emacs \
    co emacs
--8<---------------cut here---------------end--------------->8---

Thanks

Patrick

>> ,----[ Emacs version and settings: ]
>> | In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
>> |  of 2008-04-04 on golem
>> | Windowing system distributor `The X.Org Foundation', version 11.0.70000000
>> | configured using `configure  '--prefix=/home/patrick/prg/stow/emacs-devel' 
>> '--with-xft' '--with-x-toolkit=gtk''
>> | 
>> | Important settings:
>> |   value of $LC_ALL: nil
>> |   value of $LC_COLLATE: nil
>> |   value of $LC_CTYPE: nil
>> |   value of $LC_MESSAGES: nil
>> |   value of $LC_MONETARY: nil
>> |   value of $LC_NUMERIC: nil
>> |   value of $LC_TIME: nil
>> |   value of $LANG: de_DE.UTF-8
>> |   value of $XMODIFIERS: nil
>> |   locale-coding-system: utf-8-unix
>> |   default-enable-multibyte-characters: t
>> | 
>> | Major mode: Message
>> | 
>> | Minor modes in effect:
>> |   gnus-message-citation-mode: t
>> |   mml-mode: t
>> |   delete-selection-mode: t
>> |   show-paren-mode: t
>> |   savehist-mode: t
>> |   pc-selection-mode: t
>> |   iswitchb-mode: t
>> |   display-time-mode: t
>> |   which-function-mode: t
>> |   shell-dirtrack-mode: t
>> |   tooltip-mode: t
>> |   mouse-wheel-mode: t
>> |   menu-bar-mode: t
>> |   file-name-shadow-mode: t
>> |   global-font-lock-mode: t
>> |   font-lock-mode: t
>> |   blink-cursor-mode: t
>> |   global-auto-composition-mode: t
>> |   auto-composition-mode: t
>> |   auto-compression-mode: t
>> |   column-number-mode: t
>> |   line-number-mode: t
>> |   transient-mark-mode: t
>> |   abbrev-mode: t
>> `----
>>
>> ,----[ Gnus settings ]
>> | (setq gnus-default-nntp-server "")
>> | (setq gnus-select-method
>> |       '(nntp "news.myserver.foo"))
>> | (setq gnus-message-archive-group
>> |       '((if
>> |        (message-news-p)
>> |        "nnimap+mail.mymailserver.bla:INBOX.sent.news" 
>> "nnimap+mail.mymailserver.bla:INBOX.Sent")))
>> | (setq gnus-secondary-select-methods
>> |       '((nnml "private")
>> |    (nntp "news.gmane.org")
>> |    (nnimap "mail.mymailserver.bla"
>> |            (nnimap-list-pattern
>> |             ("*")))))
>> | (setq gnus-group-change-level-function 'gnus-topic-change-level)
>> | (setq gnus-summary-expunge-below -999)
>> | (setq gnus-summary-mode-hook
>> |       '(gnus-agent-mode))
>> | (setq gnus-exit-group-hook
>> |       '(gnus-uu-delete-work-dir gnus-uu-clean-up))
>> | (setq gnus-group-line-format "%M%S%p%P%5y:%B%(%G%)%l %O\n")
>> | (setq gnus-group-catchup-group-hook
>> |       '(gnus-topic-update-topic))
>> | (setq gnus-exit-gnus-hook
>> |       '(mm-destroy-postponed-undisplay-list))
>> | (setq gnus-visible-headers 
>> "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:\\|^X-Newsreader:\\|^X-Mailer:\\|^User-Agent:\\|^X-User-Agent:\\|^X-Injected-Via-Gmane:\\|^Message-ID:")
>> | (setq gnus-saved-headers 
>> "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^Followup-To:\\|^Reply-To:\\|^Organization:\\|^Summary:\\|^Keywords:\\|^To:\\|^[BGF]?Cc:\\|^Posted-To:\\|^Mail-Copies-To:\\|^Mail-Followup-To:\\|^Apparently-To:\\|^Gnus-Warning:\\|^Resent-From:\\|^X-Sent:")
>> | (setq gnus-article-prepare-hook
>> |       '(bbdb/gnus-pop-up-bbdb-buffer))
>> | (setq gnus-init-file "~/.emacs.d/.gnus")
>> | (setq gnus-startup-hook
>> |       '(bbdb-insinuate-gnus))
>> | (setq gnus-setup-news-hook
>> |       '(gnus-agent-queue-setup 
>> gnus-fixup-nnimap-unread-after-getting-new-news))
>> | (setq gnus-save-newsrc-hook
>> |       '(bbdb-offer-save))
>> | (setq gnus-group-mode-hook
>> |       '(gnus-agent-mode gnus-topic-mode))
>> | (setq gnus-posting-styles
>> |       '((".*"
>> |     (organization "none")
>> |     (Face
>> |      '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))
>> |    (".*gmane.comp.lang.r.general"
>> |     (signature-file "~/.emacs.d/mailsignature/sig-workcam-r")
>> |     (Face
>> |      '(gnus-convert-png-to-face "~/.emacs.d/faces/ant.png")))))
>> | (setq gnus-home-score-file "~/.emacs.d/total-score-file.SCORE")
>> | (setq gnus-server-mode-hook
>> |       '(gnus-agent-mode))
>> | (setq mm-charset-synonym-alist
>> |       '((ibm866 . cp866)
>> |    (unicode . utf-16-le)
>> |    (ks_c_5601-1987 . cp949)
>> |    (windows-31j . cp932)
>> |    (iso8859-1 . iso-8859-1)))
>> | (setq message-send-mail-function 'smtpmail-send-it)
>> | (setq message-post-method
>> |       '(nntp "news"))
>> | (setq message-setup-hook
>> |       '(bbdb-define-all-aliases bbdb-insinuate-message))
>> | (setq message-mode-hook
>> |       '(#[nil "\302\030\303        !)\207"
>> |          [gnus-article-copy gnus-setup-message-group nil 
>> gnus-configure-posting-styles]
>> |          2]
>> |    #[nil "\302 \211\020\211\021\207"
>> |          [message-mailer message-newsreader gnus-extended-version]
>> |          2]))
>> | (setq message-header-setup-hook
>> |       '(gnus-inews-insert-archive-gcc gnus-inews-insert-gcc))
>> | (setq message-signature 'randomsig-signature)
>> `----


Footnotes: 
[1] Archived-At: <http://permalink.gmane.org/gmane.test/4646>

-- 
For animals, the entire universe has been neatly divided into things
to (a) mate with, (b) eat, (c) run away from, and (d) rocks.
        (Terry Pratchett, Equal Rites)