[FIXED in 2.1.20100409.0] Character Code Conversion Error

[GordS]

I was creating my second AutoCode file maintenance program. The only difference between this program and my first is that I did not select all the fields to display in the initial grid.

When I load the program I get no records in the initial grid. What is causing this problem?

[sean.lanktree]

What response are you getting (from the back-end program) when you initially load the program?

[GordS]

Sean

Here is the Response I get per Firebug.

{"totalCount":10.00000,"autogrid":[{"CMCUSTOMER":"DEMOSALES","CMNAME":"Sales Demo","CMSTATUS":"A","CMLIBRARY":"DEMOSALES","CMOBSERVER":"","CMSCNDYCNT":0,"CMDOCUMENT":"Y"},{"CMCUSTOMER":"DEMOSPONS","CMNAME":"Sponsorship Demo","CMSTATUS":"A","CMLIBRARY":"DEMOSPONS","CMOBSERVER":"","CMSCNDYCNT":0,"CMDOCUMENT":"N"},{"CMCUSTOMER":"PROSPECTS","CMNAME":"Sponsorship Management Software","CMSTATUS":"A","CMLIBRARY":"PROSPECTS","CMOBSERVER":"","CMSCNDYCNT":1,"CMDOCUMENT":"Y"},{"CMCUSTOMER":"RBC","CMNAME":"RBC","CMSTATUS":"A","CMLIBRARY":"RBC","CMOBSERVER":"","CMSCNDYCNT":0,"CMDOCUMENT":"N"},{"CMCUSTOMER":"SCC","CMNAME":"Skills/Compéz:šBªâB¡CsâB~äËCsâ=CuC»äËCsäÍäB]CuC»äË…CuCýCuâ)äÍäB]CuC»BeäÍäC¨äÍCs‰CuCýCuâ)äÍäB]âÁCuCýCuäByCuCýäËBiäÍäC¨äÍCs‰CuCýCuâ)CsCaäÍäC¨äÍCu â`äÍäC¨CuC»âÑCuCýCuäByCuCýäËBiäÍäC¨äÍCs‰äËä/CuCýCuäByCuCýäÍCs-CuCýCuäByäÍäB]CsCjäÍäC¨äÍCuâ`äÍäC¨CuC»âÑCuCýCuäByCuCýäËBiCuC»Cuä ÍäC¨äÍCuâ`äÍäC¨CuCýäËäÍäC¨äÍCuâ`CuCýCuâ)äËä¦CuCýCu äByCuCýäÍCs-CuCýCuäByäÍäB]CsCjäÍäC¨äÍCuâ`äÍäC¨CuC»âÑäÍäB]äÍCuCýCuäByCuCýäÍCs-CuCýCuäByäÍäC¨CuC» CuCýCuäByCuCýäÍCs-äÍäC¨äÍCs‰CuC»CuBwäÍäC¨äÍCuâ`äÍäC¨CuCýäËäÍäC¨äÍCuâ `CuCýCuâ)äËä¦CuCýCuäByCuCýäÍCs-CuCýCuäByäÍäB]CsCjCuCýCuâ)CuCý"äÍäC¨äÍCuâ`äÍäC¨CuCýäËäÍäC¨äÍCuâ`CuCýCuäByäÍäB]�äÍäC¨äÍCuâ`äÍäC¨CuCýäËCuCýCuäByCuCýäËBiäÍäB]äÍâÏCuCýCuäByCuCýäÍCs-CuCýCuäByäÍäC¨CuC» CuCýCuäByCuCýäÍCs-äÍäC¨äÍCs‰CuC»CuBwäÍäC¨äÍCuâ`äÍäC¨CuCýäËäÍäC¨äÍCuâ `CuCýCuâ)äËä¦äÍäC¨äÍCs‰äÍäC¨‚CuCýCuäByCuCýäÍCs-CuCýCu

I see where it goes crazy. The customer name is Skills/Compétences Canada. The e with the weird character on top is causing the problem. Do I need to change the data. The word is French. The rest of the data is in English.

[sean.lanktree]

Is this same customer being returned in your other program?

[GordS]

Sean

That program does not use the customer file. This is the first program that accesses the customer master. Our current version of our system accesses and displays this field okay. But it is not using SQL to do this. Could it be how SQL is accessing the data or a CCSID issue?

[sean.lanktree]

It sounds like it. Can you logout and then log back in and select French as your language? Let's see if your response is any different.

[robert.swanson]

One other suggestion - try putting a CHAR() around the offending field. i.e., something like this:
Select cmcustomer, CHAR(cmname), cmstatus ... from .....

[GordS]

Sean We cannot select a language at the moment. I know it was there before. I must have changed something so it does not show.

Robert I changed the SQL select statement to Char(CMNAME) and I get the same response.

[robert.swanson]

You probably turned off multilingual in the "Settings" applications (in the Administration folder). Check the multilingual box and then stop/restart your Valence instance.

[GordS]

Robert

I changed the language to French and I get the same result.

[sean.lanktree]

How was this file created? Is there a particular CCSID attached to the CMNAME field?

[GordS]

Sean

It was created with CRTPF. Their is no specific CCSID attached except the default of the O/S. I entered the customer's name into the system by copying and pasting it into the field in our current web application. I obviously do not have the the French e on my keyboard. The system seemed to except and display it okay. Our current web application uses Bob Cozzi's CGILIB.

[robert.swanson]

Could you try pasting that same customer name into one of the customer records in the DEMOCMAST physical file and then see if it renders properly in the File Maintenance example app?

[GordS]

Robert

I copied and pasted the name into a record in DEMOCAST. I closed the tab and reloaded the grid. I get the same result as my application gets.

I copied the customer's name in earlier reply. You should be able to copy and paste in into your test system to check and see what results you get. Here it is again for your convenience. Skills/Compétences Canada

[robert.swanson]

Hmmmm. I just pasted that text in on our end and it works fine. I also connected to a customer's box in Quebec just to see, works there too.

Must be something unusual with your CCSID setup somewhere... What's your system's CCSID set to? (dspsysval qccsid)

[GordS]

Robert - The CCSID is 37.

[richard.milone]

GordS, could you please do a DSPFFD on the file and paste the full results here.

[GordS]

Richard - Here is the information you requested.

Display File Field Description
Input parameters
File . . . . . . . . . . . . . . . . . . . : APCUSTLA
Library . . . . . . . . . . . . . . . . . : *LIBL
File Information
File . . . . . . . . . . . . . . . . . . . : APCUSTLA
Library . . . . . . . . . . . . . . . . . : AMPMFILES
File location . . . . . . . . . . . . . . . : *LCL
Externally described . . . . . . . . . . . : Yes
Number of record formats . . . . . . . . . : 1
Type of file . . . . . . . . . . . . . . . : Logical
File creation date . . . . . . . . . . . . : 11/18/06
Text 'description'. . . . . . . . . . . . . : Customer Master by Code
Record Format Information
Record format . . . . . . . . . . . . . . . : RAPCUST
Format level identifier . . . . . . . . . . : 4EE0E338E3035
Number of fields . . . . . . . . . . . . . : 24
Record length . . . . . . . . . . . . . . . : 366
Format text . . . . . . . . . . . . . . . . : Customer Master
Field Level Information
Data Field Buffer Buffer Field Column
Field Type Length Length Position Usage Heading
CMCUSTOMER CHAR 10 10 1 Both Code
Field text . . . . . . . . . . . . . . . : Code
Coded Character Set Identifier . . . . . : 37
CMNAME CHAR 40 40 11 Both Name
Field text . . . . . . . . . . . . . . . : Name
Coded Character Set Identifier . . . . . : 37
CMADDRESS1 CHAR 40 40 51 Both Address 1
Field text . . . . . . . . . . . . . . . : Address 1
Coded Character Set Identifier . . . . . : 37
CMADDRESS2 CHAR 40 40 91 Both Address 2
Field text . . . . . . . . . . . . . . . : Address 2
Coded Character Set Identifier . . . . . : 37
CMADDRESS3 CHAR 40 40 131 Both Address 3
Field text . . . . . . . . . . . . . . . : Address 3
Coded Character Set Identifier . . . . . : 37
CMPOSTZIP CHAR 10 10 171 Both Postal/Zip Code
Field text . . . . . . . . . . . . . . . : Postal/Zip Code
Coded Character Set Identifier . . . . . : 37
CMCOUNTRY CHAR 20 20 181 Both Country
Field text . . . . . . . . . . . . . . . : Country
Coded Character Set Identifier . . . . . : 37
CMINVEMAIL CHAR 60 60 201 Both Invoice e-Mail
Field text . . . . . . . . . . . . . . . : Invoice e-Mail
Coded Character Set Identifier . . . . . : 37
CMPHONE CHAR 20 20 261 Both Phone
Field text . . . . . . . . . . . . . . . : Phone
Coded Character Set Identifier . . . . . : 37
CMFAX CHAR 20 20 281 Both Fax
Field text . . . . . . . . . . . . . . . : Fax
Coded Character Set Identifier . . . . . : 37
CMLOGONAME CHAR 40 40 301 Both Logo
Field text . . . . . . . . . . . . . . . : Logo
Coded Character Set Identifier . . . . . : 37
CMSTATUS CHAR 1 1 341 Both Status
Field text . . . . . . . . . . . . . . . : Status
Coded Character Set Identifier . . . . . : 37
CMLIBRARY CHAR 10 10 342 Both File Library
Field text . . . . . . . . . . . . . . . : File Library
Coded Character Set Identifier . . . . . : 37
CMBILLABLE CHAR 1 1 352 Both Billable
Field text . . . . . . . . . . . . . . . : Billable
Coded Character Set Identifier . . . . . : 37
CMCURRENCY CHAR 3 3 353 Both Currency
Field text . . . . . . . . . . . . . . . : Currency
Referenced information
Referenced file . . . . . . . . . . . . : BMCURR
Library . . . . . . . . . . . . . . . : AMPMFILES
Referenced record format . . . . . . . : RBMCURR
Referenced field . . . . . . . . . . . : BCCURRENCY
Attributes changed . . . . . . . . . . : Other
Coded Character Set Identifier . . . . . : 37
Data Field Buffer Buffer Field Column
Field Type Length Length Position Usage Heading
CMOBSERVER CHAR 2 2 356 Both Observer
Field text . . . . . . . . . . . . . . . : Observer
Referenced information
Referenced file . . . . . . . . . . . . : APOBSV
Library . . . . . . . . . . . . . . . : AMPMFILES
Referenced record format . . . . . . . : RAPOBSV
Referenced field . . . . . . . . . . . : OGCODE
Attributes changed . . . . . . . . . . : Other
Coded Character Set Identifier . . . . . : 37
Data Field Buffer Buffer Field Column
Field Type Length Length Position Usage Heading
CMSCNDYCNT PACKED 1 0 1 358 Both # of
Secondary Contacts
Field text . . . . . . . . . . . . . . . : # of Secondary Contacts
CMTIMEOUT PACKED 3 0 2 359 Both Timeout Setting
Field text . . . . . . . . . . . . . . . : Timeout Setting
CMINDCLASS CHAR 1 1 361 Both Industry Class.
Field text . . . . . . . . . . . . . . . : Industry Class.
Coded Character Set Identifier . . . . . : 37
CMRELSECT CHAR 1 1 362 Both Relevant Sector
Field text . . . . . . . . . . . . . . . : Relevant Sector
Coded Character Set Identifier . . . . . : 37
CMEVENT CHAR 1 1 363 Both Event
Field text . . . . . . . . . . . . . . . : Event
Coded Character Set Identifier . . . . . : 37
CMPROJACTY CHAR 1 1 364 Both Project Activity
Field text . . . . . . . . . . . . . . . : Project Activity
Coded Character Set Identifier . . . . . : 37
CMCOLOUR CHAR 1 1 365 Both Colour
Field text . . . . . . . . . . . . . . . : Colour
Coded Character Set Identifier . . . . . : 37
CMDOCUMENT CHAR 1 1 366 Both Document
Storage
Field text . . . . . . . . . . . . . . . : Document Storage
Coded Character Set Identifier . . . . . : 37

[GordS]

The character in question has a hex value of 51. I changed it to a normal "e" and the program works as expected.

I changed it back to a hex 51 and debugged the program so I could figure where the error was occurring. It seems the procedure vvUtility_convertCCSID is causing the problem. From what I can tell the data is okay until it runs thru iConv procedure. I am not sure what this is.

To get us working we will change it to an English e.

Do we need some PTF's?

[sean.lanktree]

did you happen to notice what the from CCSID was when vvUtility_convertCCSID was executing?

[GordS]

Sean - The from CCSID was 37. I checked for any PTF's related to iConv. We have all of the PTF's that I found.

[richard.milone]

We have identified the bug in the Valence core that's causing this on some systems and will be fixed on the next update.

[richard.milone]

This is now fixed as of Valence 2.1.20100409.0.