Pike-devel August 2008

pike-devel@lists.lysator.liu.se

8 participants
34 discussions

Split?
by Martin Stjernholm, Roxen IS ＠ Pike developers forum 01 Sep '08

01 Sep '08

>From my point of view now would be a good time to split. Or, at least, it doesn't look likely to get much better. The recent yellow dots is due to a test case I added for a type check problem. I can disable that test again if it make things look bad.. So, opinions about splitting?

13 58

master()->add_predefine()
by Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) ＠ Pike (-) developers forum 31 Aug '08

31 Aug '08

It was discovered that master()->add_predefine() does not work in 7.6 compat mode in 7.8. The predefine functions has been present since 7.4, so they should be added to the Pike_7_4_master class. Is it ok to commit this fix (yes, I have tested it)? Index: master.pike.in =================================================================== RCS file: /pike/data/cvsroot/Pike/7.8/lib/master.pike.in,v retrieving revision 1.451 diff -u -r1.451 master.pike.in --- master.pike.in 14 Aug 2008 13:14:10 -0000 1.451 +++ master.pike.in 30 Aug 2008 21:39:12 -0000 @@ -643,6 +643,9 @@ if ((major == 7) && (minor > 0)) return Pike_7_2_master::this; return Pike_7_2_master::get_compat_master(major, minor); } + void add_predefine (string name, string value); + void remove_predefine (string name); + mapping get_predefines(); } //! Pike 7.6 master compatibility interface.

3 3

7.8 alpha 2
by Mirar ＠ Pike developers forum 29 Aug '08

29 Aug '08

That doesn't break hilfe either. Only that exact order breaks: | > string s = Stdio.read_bytes("sugar.jpg"); | > object i = Image.JPEG.decode(s); | >> i; | (1) Result: Image.Image( 300 x 300 /* 263.7Kb */) | > Image.JPEG.encode(i); | *** glibc detected *** /usr/local/bin/pike: double free or corruption (out): 0x0000000000b52860 *** | Error while mapping shared library sections: | ¹=ãÈ¬àx¬<5`Ó 3Ú9ÿ: No such file or directory. | ======= Backtrace: ========= | /lib/libc.so.6[0x2b5197a4f08a] | /lib/libc.so.6(cfree+0x8c)[0x2b5197a52c1c] | ... Skipping the "i;" doesn't lead to a fault. Replacing the "i;" with werror("%O\n",i); stops it from breaking as well. This bug goes away if you look at it the wrong way. :P

6 23

RFC int2string optimisation
by Stephen R. van den Berg 28 Aug '08

28 Aug '08

It seemed like a good idea. commit beeba499793328cc118992884d8e175caffa6433 Author: Stephen R. van den Berg <srb(a)cuci.nl> Date: Thu Aug 21 13:17:21 2008 +0200 Optimise optimisation and move inline int2string from roxen.c to operators.c diff --git a/src/modules/_Roxen/roxen.c b/src/modules/_Roxen/roxen.c index 253f2b3..dcf6086 100644 --- a/src/modules/_Roxen/roxen.c +++ b/src/modules/_Roxen/roxen.c @@ -456,33 +456,6 @@ static void f_html_encode_string( INT32 args ) void o_cast_to_string(); case PIKE_T_INT: - /* Optimization, this is basically a inlined cast_int_to_string */ - { - char buf[21], *b = buf+19; - int neg, i, j=0; - i = Pike_sp[-1].u.integer; - pop_stack(); - if( i < 0 ) - { - neg = 1; - i = -i; - } - else - neg = 0; - - buf[20] = 0; - - while( i >= 10 ) - { - b[ -j++ ] = '0'+(i%10); - i /= 10; - } - b[ -j++ ] = '0'+(i%10); - if( neg ) b[ -j++ ] = '-'; - push_text( b-j+1 ); - } - return; - case PIKE_T_FLOAT: /* Optimization, no need to check the resultstring for * unsafe characters. diff --git a/src/operators.c b/src/operators.c index 3d12cbd..d1eaf75 100644 --- a/src/operators.c +++ b/src/operators.c @@ -343,11 +343,10 @@ PMOD_EXPORT void o_cast_to_int(void) /* Special case for casting to string. */ PMOD_EXPORT void o_cast_to_string(void) { - char buf[200]; switch(sp[-1].type) { case PIKE_T_STRING: - return; + break; case T_OBJECT: if(!sp[-1].u.object->prog) { @@ -387,10 +386,40 @@ PMOD_EXPORT void o_cast_to_string(void) get_name_of_type(sp[-1].type)); } } - return; + break; case T_INT: +#if 0 sprintf(buf, "%"PRINTPIKEINT"d", sp[-1].u.integer); +#else + { +#define sizeNUM(v) (8*sizeof(v)*4/10+1+1) + INT32 org; + char buf[sizeNUM(org)]; + register char*b = buf+sizeof buf-1; + register unsigned i; + org = sp[-1].u.integer; + pop_stack(); + *b-- = '\0'; + i = org; + + if( org < 0 ) + i = -i; + + goto jin; + do { + i /= 10; +jin: *b-- = '0'+(i%10); + } + while( i >= 10 ); + + if( org<0 ) + *b = '-'; + else + b++; + push_text( b ); + } +#endif break; case T_ARRAY: @@ -462,18 +491,18 @@ PMOD_EXPORT void o_cast_to_string(void) pop_stack(); push_string(s); } - return; + break; - case T_FLOAT: + case T_FLOAT: { + char buf[sizeNUM(double)*2]; sprintf(buf, "%f", (double)sp[-1].u.float_number); + push_text(buf); break; + } default: Pike_error("Cannot cast %s to string.\n", get_name_of_type(sp[-1].type)); } - - sp[-1].type = PIKE_T_STRING; - sp[-1].u.string = make_shared_string(buf); } PMOD_EXPORT void o_cast(struct pike_type *type, INT32 run_time_type) -- Sincerely, Stephen R. van den Berg. "I was standing in the park wondering why frisbees got bigger as they get closer. Then it hit me."

5 19

7.8 build fix for native Java methods
by Jonas Walld�n ＠ Pike developers forum 28 Aug '08

28 Aug '08

I'd like to enable native Java methods on OS X x86_64. Unfortunately the current configuration code fails to find ffi headers in Apple's standard location (/usr/include/ffi/ffi.h) so I've got a small patch for that: ceylon:Java $ cvs diff -u configure.in jvm.c Index: configure.in =================================================================== RCS file: /pike/data/cvsroot/Pike/7.8/src/modules/Java/configure.in,v retrieving revision 1.71 diff -u -r1.71 configure.in --- configure.in 18 Jul 2008 20:23:03 -0000 1.71 +++ configure.in 28 Aug 2008 13:36:22 -0000 @@ -313,9 +313,19 @@ PIKE_PKG_CONFIG(libffi) - AC_CHECK_HEADERS(jni.h winbase.h setjmp.h ffi.h) + # FFI on OS X (10.5 or newer) expects MACOSX defined + DO_IF_OS(Darwin, + [ + AC_SYS_COMPILER_FLAG(-DMACOSX, define_macosx, CPPFLAGS) + ]) + + AC_CHECK_HEADERS(jni.h winbase.h setjmp.h ffi.h ffi/ffi.h) AC_CHECK_TYPES([ffi_arg, ffi_sarg],,,[#ifdef HAVE_FFI_H #include <ffi.h> +#else +#ifdef HAVE_FFI_FFI_H +#include <ffi/ffi.h> +#endif #endif]) pike_cv_java=no @@ -413,7 +423,7 @@ pike_java_ffi=no AC_CHECK_LIB(ffi, ffi_prep_closure, [ - if test "$ac_cv_header_ffi_h" = yes; then + if test "$ac_cv_header_ffi_h" = yes -o "$ac_cv_header_ffi_ffi_h" = yes; then LIBS="-lffi $LIBS" JAVA_LIBS="-lffi ${JAVA_LIBS}" AC_DEFINE(HAVE_FFI) Index: jvm.c =================================================================== RCS file: /pike/data/cvsroot/Pike/7.8/src/modules/Java/jvm.c,v retrieving revision 1.91 diff -u -r1.91 jvm.c --- jvm.c 13 May 2008 16:00:07 -0000 1.91 +++ jvm.c 28 Aug 2008 13:36:22 -0000 @@ -50,6 +50,10 @@ #ifdef HAVE_FFI_H #include <ffi.h> +#else +#ifdef HAVE_FFI_FFI_H +#include <ffi/ffi.h> +#endif /* HAVE_FFI_FFI_H */ #endif /* HAVE_FFI_H */ #ifdef _REENTRANT Roxen will need it for 7.8 so ok to commit now?

3 5

pelix disk full?
by Stephen R. van den Berg 28 Aug '08

28 Aug '08

Can't commit. -- Sincerely, Stephen R. van den Berg. "Papers in string theory are published at a rate above the speed of light. This is no problem since no information is being transmitted." -- H. Kleinert

7 9

NULL results in SQL (Re: sql_null in Roxen 5.0)
by Stephen R. van den Berg 26 Aug '08

26 Aug '08

Martin Stjernholm wrote: >"Stephen R. van den Berg" <srb(a)cuci.nl> wrote: >> Pike 7.8 returns Pike UNDEFINED objects in Sql.Sql queries for columns >> that return NULL values (as opposed to integer zero); Pike <7.8 returned >> integer zeros. >On the pike level the interface isn't ambiguous as it was on the RXML >level, since all other values are returned as strings. It's therefore >not a problem returning either a normal zero or UNDEFINED for an sql >NULL. >Therefore I don't really see the point with the change in 7.8. Note >also that a mapping cannot carry UNDEFINED as a value for natural >reasons, so in Sql.query() those UNDEFINED's still becomes normal >zeroes. Aaargh. Didn't know that. >The fact that all non-NULL values are returned as strings is another >problem in the Sql module, but it does avoid this particular problem. >If a better API is added to it, where integers are returned as ints >etc, then there should be an SqlNull object there too for sql NULLs. >Using UNDEFINED for it would be abuse (see below). (Further >discussions about this should go to the pike dev list.) Well, actually, a cursory check amongst database backends in the current (and older) Pike versions shows: Returns - Mysql string/float/int - Oracle string/float/int - pgsql string/float/int - Odbc string/wide-string - Postgres string So the current backend database API actually desires native types. I.e. using UNDEFINED for NULL there is desirable (because queries to Mysql/Oracle/pgsql cannot distinguish NULL and integer-zero values now) and works. >Those changes shouldn't be necessary since all non-null values are >strings. If your sql glue does something else then it doesn't adhere >to the Sql module API. Well, actually it does, since I copied the behaviour from the Mysql low-level driver. >Btw, I wouldn't call the UNDEFINED value in pike an object. It's not, >and it'd be a bit misleading to think of it as one. Well, in light of learning that mappings cannot contain UNDEFINEDs, I propose to introduce the following SqlNULL() type which should be returned by any mappingvalues from Sql.Sql()->query() which are actually an SQL NULL type. Any thoughts? If there are no objections I'd like to get it into 7.8. The SqlNULL type is carefully tuned to be transparent for most normal uses (casts, comparisons, printing and truth-values). commit b68690a72d6855ecc187a4be35037cb080db69d4 Author: Stephen R. van den Berg <srb(a)cuci.nl> Date: Fri Aug 22 13:01:00 2008 +0200 Sql: Use SqlNULL to represent NULL values returned from Sql.query() diff --git a/CHANGES b/CHANGES index f1c8f6f..060b5db 100644 --- a/CHANGES +++ b/CHANGES @@ -691,6 +691,8 @@ o Regexp.PCRE.Widestring o Sql + - Return an SqlNULL() object for NULL values in Sql.Sql()->query() results + - Bugfixes in listing Postgres fields. - If ENABLE_SPAWN_RSQLD is defined, rsqld will be spawned when diff --git a/lib/modules/Sql.pmod/Sql.pike b/lib/modules/Sql.pmod/Sql.pike index 3ec85d5..98a7dac 100644 --- a/lib/modules/Sql.pmod/Sql.pike +++ b/lib/modules/Sql.pmod/Sql.pike @@ -339,6 +339,16 @@ protected string _sprintf(int type, mapping|void flags) return sprintf("Sql.%O", master_sql); +// Actually the following object is a constant and needs only +// one instance per Pike runtime +.sql_util.SqlNULL SqlNULL = .sql_util.SqlNULL(); + +final protected void replace_SqlNULL(array(mixed) row) { + foreach(row;int i;mixed value) + if(zero_type(value)) + row[i] = SqlNULL; +} + protected array(mapping(string:mixed)) res_obj_to_array(object res_obj) { if (res_obj) @@ -366,11 +376,15 @@ protected array(mapping(string:mixed)) res_obj_to_array(object res_obj) fieldnames = map(fieldnames, lower_case); if(has_table) - while (row = res_obj->fetch_row()) + while (row = res_obj->fetch_row()) { + replace_SqlNULL(row); res += ({ mkmapping(fieldnames, row + row) }); + } else - while (row = res_obj->fetch_row()) + while (row = res_obj->fetch_row()) { + replace_SqlNULL(row); res += ({ mkmapping(fieldnames, row) }); + } return res; diff --git a/lib/modules/Sql.pmod/sql_util.pmod b/lib/modules/Sql.pmod/sql_util.pmod index 9a0bb92..482538d 100644 --- a/lib/modules/Sql.pmod/sql_util.pmod +++ b/lib/modules/Sql.pmod/sql_util.pmod @@ -207,3 +207,61 @@ class MySQLBrokenUnicodeWrapper #endif + +class SqlNULL +{ + constant is_SqlNULL = 1; + + // Treat these objects as indistinguishable from each other. We + // ought to ensure that there's only one in the pike process + // instead, but that's tricky to solve in the PCode codec. + protected int `== (mixed other) { + return zero_type(other) + || objectp(other) && other->is_SqlNULL; + } + + protected int `! () { + return 1; + } + + protected int(0..1) _is_type(mixed type) { + return 1; + } + + protected mixed cast(string to) { + mixed ret=UNDEFINED; + switch(to) { + case "string": + ret=""; + break; + case "float": + ret=Math.nan; + break; + case "mixed":case "object": + ret=this; + } + return ret; + } + + protected int __hash() { + return 17; + } + + protected string _sprintf (int flag) { + string ret=UNDEFINED; + switch(flag) { + case 'O':ret="SqlNULL()"; + break; + case 's':case 'd':case 'f':case 'g':ret=""; + break; + } + return ret; + } + + int _encode() { + return 0; + } + + void _decode (int dummy) { + } +} -- Sincerely, Stephen R. van den Berg. "If you try to fail and succeed, which have you done?"

4 10

catch in c
by Arne Goedeke 24 Aug '08

24 Aug '08

I would like to know how to do a catch in a cmod. I happen to use f_string_to_utf8 now and it throws on non unicode chars, so I have to free some mem. arne

5 7

RFC optimisation of file.c read size
by Stephen R. van den Berg 23 Aug '08

23 Aug '08

In the course of tracking down the system-call-overhead of the pgsql driver, I encountered the dual read-buffer strategy in file.c. Although it was/is functional as is, it could be polished a bit. Any thoughts? I tried to cater for both large reads from files as well as network sized reads from arriving packets. commit e44d08a94ebc9e38cd107177abd1fdfdae8a8e73 Author: Stephen R. van den Berg <srb(a)cuci.nl> Date: Tue Aug 19 10:53:08 2008 +0200 file.c optimise buffer management, minimise read size/systemcalls diff --git a/src/modules/files/file.c b/src/modules/files/file.c index e69043a..7d4ef16 100644 --- a/src/modules/files/file.c +++ b/src/modules/files/file.c @@ -121,8 +121,6 @@ #define FD (THIS->box.fd) #define ERRNO (THIS->my_errno) -#define READ_BUFFER 8192 - /* Don't try to use socketpair() on AmigaOS, socketpair_ultra works better */ #ifdef __amigaos__ #undef HAVE_SOCKETPAIR @@ -130,6 +128,10 @@ /* #define SOCKETPAIR_DEBUG */ +#define READ_BUFFER 8192 +#define DIRECT_BUFSIZE (64*1024) +#define SMALL_NETBUF 2048 + struct program *file_program; struct program *file_ref_program; @@ -472,13 +474,16 @@ static struct pike_string *do_read(int fd, { ONERROR ebuf; ptrdiff_t bytes_read, i; + INT32 try_read; bytes_read=0; *err=0; - if(r <= 65536) + if(r <= DIRECT_BUFSIZE + || all && (r<<1) > r && (((r-1)|r)+1)!=(r<<1)) /* r<<1 != power of two */ { struct pike_string *str; + i=r; str=begin_shared_string(r); @@ -488,7 +493,9 @@ static struct pike_string *do_read(int fd, int fd=FD; int e; THREADS_ALLOW(); - i = fd_read(fd, str->str+bytes_read, r); + + try_read = i; + i = fd_read(fd, str->str+bytes_read, i); e=errno; THREADS_DISALLOW(); @@ -515,6 +522,24 @@ static struct pike_string *do_read(int fd, } break; } + + /* + * Large reads cause the kernel to validate more memory before + * starting the actual read, and hence cause slowdowns. + * Ideally you read as much as you're going to receive. + * This buffer calculation algorithm tries to optimise the + * read length depending on past read chunksizes. + * For network reads this allows it to follow the packetsizes. + * The initial read will attempt the full buffer. + */ + if( i < try_read ) + i += SMALL_NETBUF; + else + i += (r-i)>>1; + if (i < SMALL_NETBUF) + i = SMALL_NETBUF; + if(i > r-SMALL_NETBUF) + i = r; }while(r); UNSET_ONERROR(ebuf); @@ -533,18 +558,22 @@ static struct pike_string *do_read(int fd, /* For some reason, 8k seems to work faster than 64k. * (4k seems to be about 2% faster than 8k when using linux though) * /Hubbe (Per pointed it out to me..) + * + * The slowdowns most likely are because of memory validation + * done by the kernel for buffer space which is later unused + * (short read) */ -#define CHUNK ( 1024 * 8 ) - INT32 try_read; dynamic_buffer b; b.s.str=0; initialize_buf(&b); SET_ONERROR(ebuf, free_dynamic_buffer, &b); + i = all && (r<<1)>r ? DIRECT_BUFSIZE : READ_BUFFER; do{ int e; char *buf; - try_read=MINIMUM(CHUNK,r); + + try_read = i; buf = low_make_buf_space(try_read, &b); @@ -555,23 +584,13 @@ static struct pike_string *do_read(int fd, check_threads_etc(); - if(i==try_read) - { - r-=i; - bytes_read+=i; - if(!all) break; - } - else if(i>0) + if(i>=0) { bytes_read+=i; r-=i; - low_make_buf_space(i - try_read, &b); - if(!all) break; - } - else if(i==0) - { - low_make_buf_space(-try_read, &b); - break; + if(try_read > i) + low_make_buf_space(i - try_read, &b); + if(!all || !i) break; } else { @@ -588,6 +607,16 @@ static struct pike_string *do_read(int fd, break; } } + + /* See explanation in previous loop above */ + if( i < try_read ) + i += SMALL_NETBUF; + else + i += (DIRECT_BUFSIZE-i)>>1; + if (i < SMALL_NETBUF) + i = SMALL_NETBUF; + if(i > r-SMALL_NETBUF) + i = r; }while(r); UNSET_ONERROR(ebuf); @@ -596,7 +625,6 @@ static struct pike_string *do_read(int fd, ADD_FD_EVENTS (THIS, PIKE_BIT_FD_READ); return low_free_buf(&b); -#undef CHUNK } } @@ -2457,7 +2485,7 @@ static void file_listxattr(INT32 args) if( res<0 && errno==ERANGE ) { /* Too little space in buffer.*/ - int blen = 65536; + int blen = DIRECT_BUFSIZE; do_free = 1; ptr = xalloc( 1 ); do { @@ -2527,7 +2555,7 @@ static void file_getxattr(INT32 args) if( res<0 && errno==ERANGE ) { /* Too little space in buffer.*/ - int blen = 65536; + int blen = DIRECT_BUFSIZE; do_free = 1; ptr = xalloc( 1 ); do { -- Sincerely, Stephen R. van den Berg. E-mails should be like a lady's skirt: Long enough to cover the subject, and short enough to be interesting.

3 2

string(0..255)
by Martin Stjernholm, Roxen IS ＠ Pike developers forum 23 Aug '08

23 Aug '08

#pragma strict_types int main (int argc, array(string) argv) { string s = argv[0]; Stdio.File ("test", "cw")->write (s); } produces these warnings for the write call: foo.pike:5: Warning: Type mismatch in argument 1 to write. foo.pike:5: Warning: Expected: array(string(0..255)) | string(0..255) | __attribute__("sprintf_format", string(0..255)). foo.pike:5: Warning: Got : string. This is strictly speaking correct and in the long run it's probably a good idea to restrict I/O functions to octet strings. But in practice this warning is very cumbersome and will probably cause widespread proliferation of the fairly clumsy type "string(0..255)" if one tries to fix it. My view: o There should be a better type name than "string(0..255)" for octet strings. (That type cannot be a typedef since typedef'ed types still don't work very well.) o If we're to separate the types for octet and char strings, I think it should be done also when char strings happen to contain only narrow 8-bit chars. I.e. the string width is not really the important characteristic, rather it's whether the string contains characters or binary octets. o Fixing the types for the I/O functions requires discussion (see e.g. points above). It ought to be done consistently over all I/O subsystems. Compat and utility functions might be required. This is not something to embark on at this time, I think. To sum up, for 7.8 I think it's better to not introduce string(0..255) in write() etc. Opinions?

8 23

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel August 2008