[ast-developers] Fix parsing of iso8859 characters

[Siteshwar Vashisht]

Previous discussion at [1]. But patch suggested in [1] might cause regressions due to removal of code.

Reproducer scripts are provided in iso_and_utf_sh.tar

Reproducer steps :
1. bash iso.sh > iso_vars.sh
2. ksh -x iso_vars.sh 2>&1 | tail -3
+ VAR8595=$'/a/\xe7foo/ab\xe3c'
+ VAR8596=$'/a/\xe7foo/ab\xe3c\nVAR8597=/a/\xe7foo/ab\xe3c'
iso_vars.sh: line 8596: ": invalid variable name

Expected output :
"invalid variable name" error should not appear.

glibc sets MB_LEN_MAX to 16 which is higher than most of the multibyte character sets require. For most of the multibyte character sets, 6 should be a sufficient maximum length. Compiling ksh using '-DCUSTOM_MB_LEN_MAX=6' with attached patch fixes this issue.

[1] https://www.mail-archive.com/ast-***@lists.research.att.com/msg01091.html

[Siteshwar Vashisht]

----- Original Message -----

Sent: Tuesday, April 25, 2017 12:29:30 AM
Subject: [ast-developers] Fix parsing of iso8859 characters
Previous discussion at [1]. But patch suggested in [1] might cause
regressions due to removal of code.
Reproducer scripts are provided in iso_and_utf_sh.tar
1. bash iso.sh > iso_vars.sh
2. ksh -x iso_vars.sh 2>&1 | tail -3
+ VAR8595=$'/a/\xe7foo/ab\xe3c'
+ VAR8596=$'/a/\xe7foo/ab\xe3c\nVAR8597=/a/\xe7foo/ab\xe3c'
iso_vars.sh: line 8596: ": invalid variable name
"invalid variable name" error should not appear.
glibc sets MB_LEN_MAX to 16 which is higher than most of the multibyte
character sets require. For most of the multibyte character sets, 6 should
be a sufficient maximum length. Compiling ksh using '-DCUSTOM_MB_LEN_MAX=6'
with attached patch fixes this issue.

This patch breaks if we increase size of iso_vars.sh by increasing length of loop in iso.sh. So this patch does not seem like a sufficient solution for the problem.

[1]
--
--
Siteshwar Vashisht
_______________________________________________
ast-developers mailing list
http://lists.research.att.com/mailman/listinfo/ast-developers