[Home]

Summary:ASTERISK-15724: [patch] Problem inserting CDR records when certain characters are used
Reporter:Maciej Krajewski (jamicque)Labels:
Date Opened:2010-03-01 15:41:31.000-0600Date Closed:2010-09-22 10:18:50
Priority:MajorRegression?No
Status:Closed/CompleteComponents:CDR/cdr_pgsql
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) 20100827__issue16940.diff.txt
( 1) 20100921__issue16940__1.6.2.diff.txt
Description:Hi, 1 on every 20 CDRs ends like lob below:

[Mar  1 19:35:00] VERBOSE[25629] cdr_pgsql.c:        > [INSERT INTO cdr ("calldate","clid","src","dst","dcontext","channel","dstchannel","lastapp","lastdata","duration","billsec
","disposition","amaflags","accountcode","uniqueid","userfield") VALUES ('2010-03-01 19:34:15','"+48514549083" <514549083>','514549083','metalzbyt','CALLEX','SIP/metalzbyt-00000
843','Local/slawomir_plata@CALLEX-f2ea;1','AGI','HANGUP',45,0,'NO ANSWER',3,'metalzbyt','1267468455.4476','A:+48514549083;SI:SIP/FreeconNet;B:+48586863734;DI:CALLEX/Ko?cierzyna;
')]
[Mar  1 19:35:00] DEBUG[25631] pbx.c: Function result is 'A:+48514549083;SI:SIP/FreeconNet;B:sabina_armatowska|sabina_armatowska|sabina_armatowska;DI:CALLEX/sabina_armatowska;'
[Mar  1 19:35:00] DEBUG[25629] cdr_pgsql.c: inserting a CDR record.
[Mar  1 19:35:00] DEBUG[19732] devicestate.c: No provider found, checking channel drivers for Local - sabina_armatowska@CALLEX
[Mar  1 19:35:00] ERROR[25629] cdr_pgsql.c: Failed to insert call detail record into database!
[Mar  1 19:35:00] ERROR[25629] cdr_pgsql.c: Reason: ERROR:  invalid byte sequence for encoding "UTF8": 0xb6
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".

[Mar  1 19:35:00] DEBUG[19732] chan_local.c: Checking if extension sabina_armatowska@CALLEX exists (devicestate)
[Mar  1 19:35:00] ERROR[25629] cdr_pgsql.c: Connection may have been lost... attempting to reconnect.
[Mar  1 19:35:00] DEBUG[19732] devicestate.c: Changing state for Local/sabina_armatowska@CALLEX - state 1 (Not in use)
[Mar  1 19:35:00] DEBUG[19732] devicestate.c: device 'Local/sabina_armatowska@CALLEX' state '1'
[Mar  1 19:35:00] DEBUG[19740] app_queue.c: Device 'Local/sabina_armatowska@CALLEX' changed to state '1' (Not in use) but we don't care because they're not a member of any queue
.
[Mar  1 19:35:00] ERROR[25629] cdr_pgsql.c: Connection reestablished.
[Mar  1 19:35:00] ERROR[25629] cdr_pgsql.c: HARD ERROR!  Attempted reconnection failed.  DROPPING CALL RECORD!
[Mar  1 19:35:00] ERROR[25629] cdr_pgsql.c: Reason: ERROR:  invalid byte sequence for encoding "UTF8": 0xb6
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".


I've tried to insert CDR manually and it worked fine. I guess that there can be something wrong with cdr_pgsql.

My database is in utf-8.
Comments:By: Maciej Krajewski (jamicque) 2010-03-01 15:44:41.000-0600

there is smthing wrong with encoding above. The "Ko?cierzyna" in CDR has a special polish sign s with '.

it shoould look like this: http://pastebin.com/Ja1XWNMa


By: Maciej Krajewski (jamicque) 2010-03-01 16:40:41.000-0600

Here is a similar example:
[Mar  1 16:11:42] VERBOSE[15243] cdr_pgsql.c:        > [INSERT INTO cdr ("calldate","clid","src","dst","dcontext","channel","dstchannel","lastapp","lastdata","duration","billsec
","disposition","amaflags","accountcode","uniqueid","userfield") VALUES ('2010-03-01 16:11:37','"krystyna_wisniewska" <*>','*','100','CALLEX','SIP/krystyna_wisniewska-00000736',
'Local/beata_damaszke@CALLEX-14dc;1','Dial','Local/lukasz_sasien@CALLEX&Local/maria_amrozewska@CALLEX&Local/michal_jablonski',5,1,'ANSWERED',3,'krystyna_wisniewska','1267456297.
3814','A:+48587638137;SI:CALLEX/krystyna_wisniewska;CRG:1;B:-100;DI:CALLEX/Gda?sk ')]
[Mar  1 16:11:42] DEBUG[15243] cdr_pgsql.c: inserting a CDR record.
[Mar  1 16:11:42] ERROR[15243] cdr_pgsql.c: Failed to insert call detail record into database!
[Mar  1 16:11:42] ERROR[15243] cdr_pgsql.c: Reason: ERROR:  invalid byte sequence for encoding "UTF8": 0xf1736b20
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".

[Mar  1 16:11:42] ERROR[15243] cdr_pgsql.c: Connection may have been lost... attempting to reconnect.
[Mar  1 16:11:42] ERROR[15243] cdr_pgsql.c: Connection reestablished.
[Mar  1 16:11:42] ERROR[15243] cdr_pgsql.c: HARD ERROR!  Attempted reconnection failed.  DROPPING CALL RECORD!
[Mar  1 16:11:42] ERROR[15243] cdr_pgsql.c: Reason: ERROR:  invalid byte sequence for encoding "UTF8": 0xf1736b20
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
By: Maciej Krajewski (jamicque) 2010-03-02 06:40:58.000-0600

the problem is concerning polish special signs.
By: Tilghman Lesher (tilghman) 2010-08-27 16:19:51

Note that this just adds another setting to the config file.  The default is LATIN9, which should be correct for your purposes.  See http://www.postgresql.org/docs/8.1/static/multibyte.html for a list of all possible settings.
By: Maciej Krajewski (jamicque) 2010-09-13 03:58:58

UTF-8 suprot polish letters, the database coding is UTF-8, so i guess it should work?
By: Tilghman Lesher (tilghman) 2010-09-17 15:15:37

No, the error already has indicated that your letters are NOT UTF-8, so that's right out.  Have you tested the patch?
By: Maciej Krajewski (jamicque) 2010-09-20 06:08:54

I've tried to patch it againts 1.6.2.13
Hunk #1 succeeded at 54 with fuzz 2 (offset -7 lines).
Hunk #2 succeeded at 112 with fuzz 1 (offset 27 lines).
Hunk #3 FAILED at 235.
Hunk #4 FAILED at 299.
Hunk ASTERISK-1 FAILED at 319.
By: Tilghman Lesher (tilghman) 2010-09-21 16:02:56

First patch was against 1.4.  We always try to patch against the earliest version with a bug, then merge forward.

I have uploaded a second patch, containing the changes, specifically for 1.6.2.
By: Maciej Krajewski (jamicque) 2010-09-22 07:14:32

I confirm, the patch fixed the problem, the crash no longer occurs.
By: Digium Subversion (svnbot) 2010-09-22 09:48:05

Repository: asterisk
Revision: 288265

U   branches/1.4/cdr/cdr_pgsql.c
U   branches/1.4/configs/cdr_pgsql.conf.sample

------------------------------------------------------------------------
r288265 | tilghman | 2010-09-22 09:48:04 -0500 (Wed, 22 Sep 2010) | 9 lines

Allow the encoding to be set, in case local charset does not agree with database.

(closes issue ASTERISK-15724)
Reported by: jamicque
Patches:
      20100827__issue16940.diff.txt uploaded by tilghman (license 14)
      20100921__issue16940__1.6.2.diff.txt uploaded by tilghman (license 14)
Tested by: jamicque

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=288265
By: Digium Subversion (svnbot) 2010-09-22 10:04:52

Repository: asterisk
Revision: 288266

U   branches/1.4/UPGRADE.txt

------------------------------------------------------------------------
r288266 | tilghman | 2010-09-22 10:04:52 -0500 (Wed, 22 Sep 2010) | 5 lines

Document addition of encoding parameter.

(issue ASTERISK-15724)
Reported by: jamicque

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=288266
By: Digium Subversion (svnbot) 2010-09-22 10:11:10

Repository: asterisk
Revision: 288267

_U  branches/1.6.2/
U   branches/1.6.2/UPGRADE.txt
U   branches/1.6.2/cdr/cdr_pgsql.c
U   branches/1.6.2/configs/cdr_pgsql.conf.sample

------------------------------------------------------------------------
r288267 | tilghman | 2010-09-22 10:11:10 -0500 (Wed, 22 Sep 2010) | 23 lines

Merged revisions 288265-288266 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4

........
 r288265 | tilghman | 2010-09-22 09:48:04 -0500 (Wed, 22 Sep 2010) | 9 lines
 
 Allow the encoding to be set, in case local charset does not agree with database.
 
 (closes issue ASTERISK-15724)
  Reported by: jamicque
  Patches:
        20100827__issue16940.diff.txt uploaded by tilghman (license 14)
        20100921__issue16940__1.6.2.diff.txt uploaded by tilghman (license 14)
  Tested by: jamicque
........
 r288266 | tilghman | 2010-09-22 10:04:52 -0500 (Wed, 22 Sep 2010) | 5 lines
 
 Document addition of encoding parameter.
 
 (issue ASTERISK-15724)
 Reported by: jamicque
........

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=288267
By: Digium Subversion (svnbot) 2010-09-22 10:14:02

Repository: asterisk
Revision: 288268

_U  branches/1.8/
U   branches/1.8/UPGRADE.txt
U   branches/1.8/cdr/cdr_pgsql.c
U   branches/1.8/configs/cdr_pgsql.conf.sample

------------------------------------------------------------------------
r288268 | tilghman | 2010-09-22 10:14:02 -0500 (Wed, 22 Sep 2010) | 30 lines

Merged revisions 288267 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.6.2

................
 r288267 | tilghman | 2010-09-22 10:11:09 -0500 (Wed, 22 Sep 2010) | 23 lines
 
 Merged revisions 288265-288266 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.4
 
 ........
   r288265 | tilghman | 2010-09-22 09:48:04 -0500 (Wed, 22 Sep 2010) | 9 lines
   
   Allow the encoding to be set, in case local charset does not agree with database.
   
   (closes issue ASTERISK-15724)
    Reported by: jamicque
    Patches:
          20100827__issue16940.diff.txt uploaded by tilghman (license 14)
          20100921__issue16940__1.6.2.diff.txt uploaded by tilghman (license 14)
    Tested by: jamicque
 ........
   r288266 | tilghman | 2010-09-22 10:04:52 -0500 (Wed, 22 Sep 2010) | 5 lines
   
   Document addition of encoding parameter.
   
   (issue ASTERISK-15724)
   Reported by: jamicque
 ........
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=288268
By: Digium Subversion (svnbot) 2010-09-22 10:18:49

Repository: asterisk
Revision: 288278

_U  trunk/
U   trunk/cdr/cdr_pgsql.c
U   trunk/configs/cdr_pgsql.conf.sample

------------------------------------------------------------------------
r288278 | tilghman | 2010-09-22 10:18:49 -0500 (Wed, 22 Sep 2010) | 37 lines

Merged revisions 288268 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.8

................
 r288268 | tilghman | 2010-09-22 10:14:02 -0500 (Wed, 22 Sep 2010) | 30 lines
 
 Merged revisions 288267 via svnmerge from
 https://origsvn.digium.com/svn/asterisk/branches/1.6.2
 
 ................
   r288267 | tilghman | 2010-09-22 10:11:09 -0500 (Wed, 22 Sep 2010) | 23 lines
   
   Merged revisions 288265-288266 via svnmerge from
   https://origsvn.digium.com/svn/asterisk/branches/1.4
   
   ........
     r288265 | tilghman | 2010-09-22 09:48:04 -0500 (Wed, 22 Sep 2010) | 9 lines
     
     Allow the encoding to be set, in case local charset does not agree with database.
     
     (closes issue ASTERISK-15724)
      Reported by: jamicque
      Patches:
            20100827__issue16940.diff.txt uploaded by tilghman (license 14)
            20100921__issue16940__1.6.2.diff.txt uploaded by tilghman (license 14)
      Tested by: jamicque
   ........
     r288266 | tilghman | 2010-09-22 10:04:52 -0500 (Wed, 22 Sep 2010) | 5 lines
     
     Document addition of encoding parameter.
     
     (issue ASTERISK-15724)
     Reported by: jamicque
   ........
 ................
................

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=288278