Re: [GNUe] RE: gnue-common bug: mysqldbdrv.py with MySQLdb 1.2.2 always decodes query result using latin1

[Oleg Noga]

Hello Johannes,

Thank You for support.

This is how actual (not patched) gnue-common works:

my connections.conf:

[gnue]
comment = GNUe Test Database
provider = mysql
host = 10.17.13.40
port = 3308
dbname   = gnue
username = gnue
password = gnue
ownerpwd = gnue
encoding = cp1251

So mysqldbdrv.py makes "SET NAMES cp1251".

Query statements encoded fine and sent to server with correct encoding
cp1251 because i have also set cp1251 with sys.setdefaultencoding().

Problem with decoding of query result:
Actual mysqldbdrv.py not passes charset to MySQLdb.Connection constructor.
Then MySQLdb.Connection constructor calls Connection.set_character_set
function with internal-default latin1 client charset.
Please see the code:

site-packages\MySQLdb\connections.py

class Connection:
....
    def set_character_set(self, charset):
        """Set the connection character set to charset. The character
        set can only be changed in MySQL-4.1 and newer. If you try
        to change the character set from the current value in an
        older version, NotSupportedError will be raised."""
        if self.character_set_name() != charset:
            try:
                super(Connection, self).set_character_set(charset)
            except AttributeError:
                if self._server_version < (4, 1):
                    raise NotSupportedError, "server is too old to set charset"
                self.query('SET NAMES %s' % charset)
                self.store_result()
        self.string_decoder.charset = charset
        self.unicode_literal.charset = charset

So the variable "self.string_decoder.charset" actually set to latin1. Then
"self.string_decoder" called to decode strings from query result. This strings
are "cp1251" encoded, type str, (because of "SET NAMES cp1251"). The
result is unicode strings with special latin chars instead cyrillic
chars. 

As you see, this function can work with mysql >= 4.1. So we can also
allow encoding with MySQL 4.1.

My tests:
Patched gnue-common with MySQLdb 1.2.2    on mysql-5.0.27: OK
Patched gnue-common with MySQLdb 1.2.1_p2 on mysql-5.0.27: OK
Patched gnue-common with MySQLdb 1.2.2    on mysql-4.1.15: OK

Patched gnue-common with MySQLdb 1.2.2    on mysql-4.0.21: ERROR
gnue-schema: (1193, "Unknown system variable 'NAMES'")

It is not a gnue assertion because i have not set version in
connection.conf. 

My testing environment:

win32
mysql-5.0.27 and mysql-4.1.15
Python 2.4.4
MySQL-python 1.2.1-p2 and MySQL-python 1.2.2
gnue-common 0.6.8 + mysqldbdrv.py patch
gnue-forms 0.6.0

-- 
Best regards,
 Oleg                            mailto:address@hidden



    ----- Original Message -----
    From: Johannes Vetter <address@hidden>
    To: address@hidden
    Date: Thursday, July 12, 2007, 11:22:13 AM
    Subject: [GNUe] RE: gnue-common bug: mysqldbdrv.py with MySQLdb 1.2.2 
always decodes query result using latin1

JV> Hi Oleg,

JV> I'm lacking a MySQLdb 1.2.2 driver as well as a MySQL 5 Server here, so
JV> I cannot reproduce your problem or verify your patch.

JV> But the value for "self._encoding" is taken from your connections.conf.
JV> If there is no such value provided, it will default to "utf-8".  This
JV> value will be used for all encoding and decoding of the unicode values
JV> in use.

JV> So if you appear to have everything encoded/decoded in latin-1 you're
JV> likely having an "encoding=latin-1 (or iso8859-1)" in your
JV> connections.conf (for that connection).

JV> Aside from that, your patch suggests setting the charset via keyword
JV> instead of performing an explicit "SET NAMES" when using a MySQL 5+.  As
JV> you've mentioned this works fine with the MySQLdb 1.2.2 driver.  Can you
JV> check this with a MySQLdb 1.2.1 driver ?  If it works there too, we can
JV> integrate your patch.


JV> Thanks for your suggestions,

JV> Have a nice day

JV> Johannes