[ast-developers] Array values freed but referenced later on

[Dr. Werner Fink]

Hi,

found by using valgrind and _AST_std_malloc=0

Two examples from valgrind output

==634== 7 errors in context 36 of 56:
==634== Invalid read of size 2
==634== at 0x481047: nv_type (nvtype.c:1362)
==634== by 0x437756: nv_name (name.c:3874)
==634== by 0x44392C: nv_diropen (nvtree.c:198)
==634== by 0x443F2D: walk_tree (nvtree.c:1209)
==634== by 0x444675: put_tree (nvtree.c:1357)
==634== by 0x439685: nv_putval (name.c:1713)
==634== by 0x48EE6E: array_putval (array.c:670)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x43D9E8: sh_unscope (name.c:2564)
==634== by 0x459953: sh_funscope_20120720 (xec.c:4096)
==634== by 0x45A17D: sh_funct (xec.c:3416)
==634== by 0x45C4BB: sh_exec (xec.c:1587)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)
==634== Address 0x64214a8 is 24 bytes inside a block of size 66 free'd
==634== at 0x4C2701C: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==634== by 0x43A544: nv_delete (name.c:1385)
==634== by 0x48F28C: array_putval (array.c:695)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x43D9E8: sh_unscope (name.c:2564)
==634== by 0x459953: sh_funscope_20120720 (xec.c:4096)
==634== by 0x45A17D: sh_funct (xec.c:3416)
==634== by 0x45C4BB: sh_exec (xec.c:1587)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)

or

==634== 6 errors in context 34 of 56:
==634== Invalid read of size 8
==634== at 0x43770B: nv_name (name.c:3872)
==634== by 0x44392C: nv_diropen (nvtree.c:198)
==634== by 0x443F2D: walk_tree (nvtree.c:1209)
==634== by 0x438047: nv_getval (name.c:2904)
==634== by 0x432B74: varsub (macro.c:1461)
==634== by 0x434C60: copyto (macro.c:634)
==634== by 0x436745: sh_macexpand (macro.c:245)
==634== by 0x436936: sh_macpat (macro.c:423)
==634== by 0x45B2AE: sh_exec (xec.c:2924)
==634== by 0x45D8CA: sh_exec (xec.c:2286)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)
==634== Address 0x58e4250 is 16 bytes inside a block of size 66 free'd
==634== at 0x4C2701C: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==634== by 0x43A544: nv_delete (name.c:1385)
==634== by 0x48DE3F: nv_associative (array.c:1789)
==634== by 0x48EF95: array_putval (array.c:685)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x4423FC: outval (nvtree.c:888)
==634== by 0x442B59: genvalue (nvtree.c:1117)
==634== by 0x44432F: walk_tree (nvtree.c:1284)
==634== by 0x444675: put_tree (nvtree.c:1357)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x478C3F: unall (typeset.c:1285)
==634== by 0x460664: sh_exec (xec.c:1382)
==634== by 0x45D8CA: sh_exec (xec.c:2286)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)

the messages belong to nv_name() at

Namval_t *nq= shp->last_table, *mp= (Namval_t*)np->nvenv;
if(mp && mp->nvname==0 || *mp->nvname==0)
mp = 0;

or below in nv_type() at

Namfun_t *fp;
if(nv_isattr(np,NV_BLTIN|BLT_DCL)==(NV_BLTIN|BLT_DCL))


at it is caused at array_putval() due

if(mp!=np)
{
array_clrbit(aq->bits,aq->cur,ARRAY_CHILD);
aq->val[aq->cur].cp = 0;
if(!xfree)
nv_delete(mp,ap->table,0);
}


or at nv_associative() due

case NV_ADELETE:
if(ap->cur)
{
if(!ap->header.scope || (Dt_t*)ap->header.scope==ap->header.table || !nv_search(ap->cur->nvname,(Dt_t*)ap->header.scope,0))
ap->header.nelem--;
_nv_unset(ap->cur,NV_RDONLY);
nv_delete(ap->cur,ap->header.table,0);
ap->cur = 0;
}


But if I replace the `0' in the nv_delete() calls with NV_NOFREE then I
see memory leaks. That is that the code in array_putval() and nv_associative()
is missing some checks if a NV_NOFREE that is if the Namval_t type
is still referenced ... or code which removes the references as well.


Werner
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140410/dccca458/attachment.sig>

[Dr. Werner Fink]

Post by Dr. Werner Fink
Hi,
found by using valgrind and _AST_std_malloc=0
Two examples from valgrind output
==634== Invalid read of size 2
==634== at 0x481047: nv_type (nvtype.c:1362)
==634== by 0x437756: nv_name (name.c:3874)
==634== by 0x44392C: nv_diropen (nvtree.c:198)
==634== by 0x443F2D: walk_tree (nvtree.c:1209)
==634== by 0x444675: put_tree (nvtree.c:1357)
==634== by 0x439685: nv_putval (name.c:1713)
==634== by 0x48EE6E: array_putval (array.c:670)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x43D9E8: sh_unscope (name.c:2564)
==634== by 0x459953: sh_funscope_20120720 (xec.c:4096)
==634== by 0x45A17D: sh_funct (xec.c:3416)
==634== by 0x45C4BB: sh_exec (xec.c:1587)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)
==634== Address 0x64214a8 is 24 bytes inside a block of size 66 free'd
==634== at 0x4C2701C: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==634== by 0x43A544: nv_delete (name.c:1385)
==634== by 0x48F28C: array_putval (array.c:695)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x43D9E8: sh_unscope (name.c:2564)
==634== by 0x459953: sh_funscope_20120720 (xec.c:4096)
==634== by 0x45A17D: sh_funct (xec.c:3416)
==634== by 0x45C4BB: sh_exec (xec.c:1587)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)
or
==634== Invalid read of size 8
==634== at 0x43770B: nv_name (name.c:3872)
==634== by 0x44392C: nv_diropen (nvtree.c:198)
==634== by 0x443F2D: walk_tree (nvtree.c:1209)
==634== by 0x438047: nv_getval (name.c:2904)
==634== by 0x432B74: varsub (macro.c:1461)
==634== by 0x434C60: copyto (macro.c:634)
==634== by 0x436745: sh_macexpand (macro.c:245)
==634== by 0x436936: sh_macpat (macro.c:423)
==634== by 0x45B2AE: sh_exec (xec.c:2924)
==634== by 0x45D8CA: sh_exec (xec.c:2286)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)
==634== Address 0x58e4250 is 16 bytes inside a block of size 66 free'd
==634== at 0x4C2701C: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==634== by 0x43A544: nv_delete (name.c:1385)
==634== by 0x48DE3F: nv_associative (array.c:1789)
==634== by 0x48EF95: array_putval (array.c:685)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x4423FC: outval (nvtree.c:888)
==634== by 0x442B59: genvalue (nvtree.c:1117)
==634== by 0x44432F: walk_tree (nvtree.c:1284)
==634== by 0x444675: put_tree (nvtree.c:1357)
==634== by 0x43BB3C: _nv_unset (name.c:2655)
==634== by 0x478C3F: unall (typeset.c:1285)
==634== by 0x460664: sh_exec (xec.c:1382)
==634== by 0x45D8CA: sh_exec (xec.c:2286)
==634== by 0x40F5D6: exfile (main.c:627)
==634== by 0x40FF59: sh_main (main.c:399)
==634== by 0x54CDC35: (below main) (in /lib64/libc-2.11.3.so)
the messages belong to nv_name() at
Namval_t *nq= shp->last_table, *mp= (Namval_t*)np->nvenv;
if(mp && mp->nvname==0 || *mp->nvname==0)
mp = 0;
or below in nv_type() at
Namfun_t *fp;
if(nv_isattr(np,NV_BLTIN|BLT_DCL)==(NV_BLTIN|BLT_DCL))
at it is caused at array_putval() due
if(mp!=np)
{
array_clrbit(aq->bits,aq->cur,ARRAY_CHILD);
aq->val[aq->cur].cp = 0;
if(!xfree)
nv_delete(mp,ap->table,0);
}
or at nv_associative() due
if(ap->cur)
{
if(!ap->header.scope || (Dt_t*)ap->header.scope==ap->header.table || !nv_search(ap->cur->nvname,(Dt_t*)ap->header.scope,0))
ap->header.nelem--;
_nv_unset(ap->cur,NV_RDONLY);
nv_delete(ap->cur,ap->header.table,0);
ap->cur = 0;
}
But if I replace the `0' in the nv_delete() calls with NV_NOFREE then I
see memory leaks. That is that the code in array_putval() and nv_associative()
is missing some checks if a NV_NOFREE that is if the Namval_t type
is still referenced ... or code which removes the references as well.

Indeed ... after running the debugger I see

Breakpoint 1, nv_delete (np=0x82b910, root=0x82cc40, flags=0) at /usr/src/packages/BUILD/ksh93/src/cmd/ksh93/sh/name.c:1387
1387 free((void*)np);
(gdb) print *np
$39 = {nvlink = {rh = {__rght = 0x82f220, __ptbl = 0x82f220}, lh = {__left = 0x0, __hash = 0}}, nvname = 0x82b950 "e", nvflag = 0, pad1 = 0, nvsize = 2,
nvfun = 0x0, nvalue = {cp = 0x0, ip = 0x0, c = 0 '\000', i = 0, u = 0, lp = 0x0, idp = 0x0, llp = 0x0, s = 0, sp = 0x0, dp = 0x0, ldp = 0x0, f = 0,
fp = 0x0, array = 0x0, np = 0x0, up = 0x0, rp = 0x0, funp = 0x0, nrp = 0x0, bfp = 0}, nvshell = 0x7cca20, nvenv = 0x0}

with searching in the heap I see

(gdb) find /g 0x007d3000, 0x00896000, 0x82b910
0x82c5e8
0x82cb98
0x82cba8
warning: Unable to access target memory at 0x8924b0, halting search.
3 patterns found.
(gdb) next
nv_associative (np=0x82cae0, sp=0x0, mode=<optimized out>) at /usr/src/packages/BUILD/ksh93/src/cmd/ksh93/sh/array.c:1790
1790 ap->cur = 0;
(gdb) next
1913 }
(gdb) find /g 0x007d3000, 0x00896000, 0x82b910
0x82c5e8
0x82cb98
warning: Unable to access target memory at 0x8924a0, halting search.
2 patterns found.
(gdb) next
array_putval (np=0x82cae0, string=0x0, flags=1, dp=<optimized out>) at /usr/src/packages/BUILD/ksh93/src/cmd/ksh93/sh/array.c:686
686 np->nvalue.cp = 0;
(gdb) next
716 if(array_elem(ap)==0 && (ap->flags&ARRAY_SCAN))
(gdb) next
724 if(!mp || mp!=np || is_associative(ap))
(gdb) next
765 while(!string && nv_nextsub(np));
(gdb) next
653 int xfree = (ap->fixed||is_associative(ap))?0:array_isbit(aq->bits,aq->cur,ARRAY_NOFREE);
(gdb) find /g 0x007d3000, 0x00896000, 0x82b910
0x82c5e8
warning: Unable to access target memory at 0x895d70, halting search.
1 pattern found.
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000437723 in nv_name (np=0x82c5b0) at /usr/src/packages/BUILD/ksh93/src/cmd/ksh93/sh/name.c:3874
3874 if(mp && (mp->nvname==0 || *mp->nvname==0))

and mp is (Namval_t*)np->nvenv here

(gdb) print (Namval_t*)np->nvenv
$42 = (Namval_t *) 0x82b910

... is there a NV_REV flag missed?
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140411/9fd32135/attachment.sig>

[Dr. Werner Fink]

Post by Dr. Werner Fink
with searching in the heap I see
(gdb) find /g 0x007d3000, 0x00896000, 0x82b910
0x82c5e8
0x82cb98
0x82cba8
warning: Unable to access target memory at 0x8924b0, halting search.
3 patterns found.
(gdb) next
nv_associative (np=0x82cae0, sp=0x0, mode=<optimized out>) at /usr/src/packages/BUILD/ksh93/src/cmd/ksh93/sh/array.c:1790
1790 ap->cur = 0;
(gdb) next
1913 }
(gdb) find /g 0x007d3000, 0x00896000, 0x82b910
0x82c5e8
0x82cb98
warning: Unable to access target memory at 0x8924a0, halting search.
2 patterns found.
(gdb) next
array_putval (np=0x82cae0, string=0x0, flags=1, dp=<optimized out>) at /usr/src/packages/BUILD/ksh93/src/cmd/ksh93/sh/array.c:686
686 np->nvalue.cp = 0;
(gdb) next
716 if(array_elem(ap)==0 && (ap->flags&ARRAY_SCAN))
(gdb) next
724 if(!mp || mp!=np || is_associative(ap))
(gdb) next
765 while(!string && nv_nextsub(np));
(gdb) next
653 int xfree = (ap->fixed||is_associative(ap))?0:array_isbit(aq->bits,aq->cur,ARRAY_NOFREE);
(gdb) find /g 0x007d3000, 0x00896000, 0x82b910
0x82c5e8
warning: Unable to access target memory at 0x895d70, halting search.
1 pattern found.
(gdb) cont
Continuing.
Program received signal SIGSEGV, Segmentation fault.
0x0000000000437723 in nv_name (np=0x82c5b0) at /usr/src/packages/BUILD/ksh93/src/cmd/ksh93/sh/name.c:3874
3874 if(mp && (mp->nvname==0 || *mp->nvname==0))
and mp is (Namval_t*)np->nvenv here
(gdb) print (Namval_t*)np->nvenv
$42 = (Namval_t *) 0x82b910
... is there a NV_REV flag missed?

OK the attached patch does avoid this trouble as well as some of already
reported one.
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ksh93-crashes.dif
Type: text/x-patch
Size: 2813 bytes
Desc: ksh93-crashes.dif
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140411/40bf55cf/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140411/40bf55cf/attachment.sig>