Pike-devel

pike-devel@lists.lysator.liu.se

3041 discussions

pelix disk full?
by Stephen R. van den Berg 28 Aug '08

28 Aug '08

Can't commit. -- Sincerely, Stephen R. van den Berg. "Papers in string theory are published at a rate above the speed of light. This is no problem since no information is being transmitted." -- H. Kleinert

7 9

NULL results in SQL (Re: sql_null in Roxen 5.0)
by Stephen R. van den Berg 26 Aug '08

26 Aug '08

Martin Stjernholm wrote: >"Stephen R. van den Berg" <srb(a)cuci.nl> wrote: >> Pike 7.8 returns Pike UNDEFINED objects in Sql.Sql queries for columns >> that return NULL values (as opposed to integer zero); Pike <7.8 returned >> integer zeros. >On the pike level the interface isn't ambiguous as it was on the RXML >level, since all other values are returned as strings. It's therefore >not a problem returning either a normal zero or UNDEFINED for an sql >NULL. >Therefore I don't really see the point with the change in 7.8. Note >also that a mapping cannot carry UNDEFINED as a value for natural >reasons, so in Sql.query() those UNDEFINED's still becomes normal >zeroes. Aaargh. Didn't know that. >The fact that all non-NULL values are returned as strings is another >problem in the Sql module, but it does avoid this particular problem. >If a better API is added to it, where integers are returned as ints >etc, then there should be an SqlNull object there too for sql NULLs. >Using UNDEFINED for it would be abuse (see below). (Further >discussions about this should go to the pike dev list.) Well, actually, a cursory check amongst database backends in the current (and older) Pike versions shows: Returns - Mysql string/float/int - Oracle string/float/int - pgsql string/float/int - Odbc string/wide-string - Postgres string So the current backend database API actually desires native types. I.e. using UNDEFINED for NULL there is desirable (because queries to Mysql/Oracle/pgsql cannot distinguish NULL and integer-zero values now) and works. >Those changes shouldn't be necessary since all non-null values are >strings. If your sql glue does something else then it doesn't adhere >to the Sql module API. Well, actually it does, since I copied the behaviour from the Mysql low-level driver. >Btw, I wouldn't call the UNDEFINED value in pike an object. It's not, >and it'd be a bit misleading to think of it as one. Well, in light of learning that mappings cannot contain UNDEFINEDs, I propose to introduce the following SqlNULL() type which should be returned by any mappingvalues from Sql.Sql()->query() which are actually an SQL NULL type. Any thoughts? If there are no objections I'd like to get it into 7.8. The SqlNULL type is carefully tuned to be transparent for most normal uses (casts, comparisons, printing and truth-values). commit b68690a72d6855ecc187a4be35037cb080db69d4 Author: Stephen R. van den Berg <srb(a)cuci.nl> Date: Fri Aug 22 13:01:00 2008 +0200 Sql: Use SqlNULL to represent NULL values returned from Sql.query() diff --git a/CHANGES b/CHANGES index f1c8f6f..060b5db 100644 --- a/CHANGES +++ b/CHANGES @@ -691,6 +691,8 @@ o Regexp.PCRE.Widestring o Sql + - Return an SqlNULL() object for NULL values in Sql.Sql()->query() results + - Bugfixes in listing Postgres fields. - If ENABLE_SPAWN_RSQLD is defined, rsqld will be spawned when diff --git a/lib/modules/Sql.pmod/Sql.pike b/lib/modules/Sql.pmod/Sql.pike index 3ec85d5..98a7dac 100644 --- a/lib/modules/Sql.pmod/Sql.pike +++ b/lib/modules/Sql.pmod/Sql.pike @@ -339,6 +339,16 @@ protected string _sprintf(int type, mapping|void flags) return sprintf("Sql.%O", master_sql); +// Actually the following object is a constant and needs only +// one instance per Pike runtime +.sql_util.SqlNULL SqlNULL = .sql_util.SqlNULL(); + +final protected void replace_SqlNULL(array(mixed) row) { + foreach(row;int i;mixed value) + if(zero_type(value)) + row[i] = SqlNULL; +} + protected array(mapping(string:mixed)) res_obj_to_array(object res_obj) { if (res_obj) @@ -366,11 +376,15 @@ protected array(mapping(string:mixed)) res_obj_to_array(object res_obj) fieldnames = map(fieldnames, lower_case); if(has_table) - while (row = res_obj->fetch_row()) + while (row = res_obj->fetch_row()) { + replace_SqlNULL(row); res += ({ mkmapping(fieldnames, row + row) }); + } else - while (row = res_obj->fetch_row()) + while (row = res_obj->fetch_row()) { + replace_SqlNULL(row); res += ({ mkmapping(fieldnames, row) }); + } return res; diff --git a/lib/modules/Sql.pmod/sql_util.pmod b/lib/modules/Sql.pmod/sql_util.pmod index 9a0bb92..482538d 100644 --- a/lib/modules/Sql.pmod/sql_util.pmod +++ b/lib/modules/Sql.pmod/sql_util.pmod @@ -207,3 +207,61 @@ class MySQLBrokenUnicodeWrapper #endif + +class SqlNULL +{ + constant is_SqlNULL = 1; + + // Treat these objects as indistinguishable from each other. We + // ought to ensure that there's only one in the pike process + // instead, but that's tricky to solve in the PCode codec. + protected int `== (mixed other) { + return zero_type(other) + || objectp(other) && other->is_SqlNULL; + } + + protected int `! () { + return 1; + } + + protected int(0..1) _is_type(mixed type) { + return 1; + } + + protected mixed cast(string to) { + mixed ret=UNDEFINED; + switch(to) { + case "string": + ret=""; + break; + case "float": + ret=Math.nan; + break; + case "mixed":case "object": + ret=this; + } + return ret; + } + + protected int __hash() { + return 17; + } + + protected string _sprintf (int flag) { + string ret=UNDEFINED; + switch(flag) { + case 'O':ret="SqlNULL()"; + break; + case 's':case 'd':case 'f':case 'g':ret=""; + break; + } + return ret; + } + + int _encode() { + return 0; + } + + void _decode (int dummy) { + } +} -- Sincerely, Stephen R. van den Berg. "If you try to fail and succeed, which have you done?"

4 10

catch in c
by Arne Goedeke 24 Aug '08

24 Aug '08

I would like to know how to do a catch in a cmod. I happen to use f_string_to_utf8 now and it throws on non unicode chars, so I have to free some mem. arne

5 7

RFC optimisation of file.c read size
by Stephen R. van den Berg 23 Aug '08

23 Aug '08

In the course of tracking down the system-call-overhead of the pgsql driver, I encountered the dual read-buffer strategy in file.c. Although it was/is functional as is, it could be polished a bit. Any thoughts? I tried to cater for both large reads from files as well as network sized reads from arriving packets. commit e44d08a94ebc9e38cd107177abd1fdfdae8a8e73 Author: Stephen R. van den Berg <srb(a)cuci.nl> Date: Tue Aug 19 10:53:08 2008 +0200 file.c optimise buffer management, minimise read size/systemcalls diff --git a/src/modules/files/file.c b/src/modules/files/file.c index e69043a..7d4ef16 100644 --- a/src/modules/files/file.c +++ b/src/modules/files/file.c @@ -121,8 +121,6 @@ #define FD (THIS->box.fd) #define ERRNO (THIS->my_errno) -#define READ_BUFFER 8192 - /* Don't try to use socketpair() on AmigaOS, socketpair_ultra works better */ #ifdef __amigaos__ #undef HAVE_SOCKETPAIR @@ -130,6 +128,10 @@ /* #define SOCKETPAIR_DEBUG */ +#define READ_BUFFER 8192 +#define DIRECT_BUFSIZE (64*1024) +#define SMALL_NETBUF 2048 + struct program *file_program; struct program *file_ref_program; @@ -472,13 +474,16 @@ static struct pike_string *do_read(int fd, { ONERROR ebuf; ptrdiff_t bytes_read, i; + INT32 try_read; bytes_read=0; *err=0; - if(r <= 65536) + if(r <= DIRECT_BUFSIZE + || all && (r<<1) > r && (((r-1)|r)+1)!=(r<<1)) /* r<<1 != power of two */ { struct pike_string *str; + i=r; str=begin_shared_string(r); @@ -488,7 +493,9 @@ static struct pike_string *do_read(int fd, int fd=FD; int e; THREADS_ALLOW(); - i = fd_read(fd, str->str+bytes_read, r); + + try_read = i; + i = fd_read(fd, str->str+bytes_read, i); e=errno; THREADS_DISALLOW(); @@ -515,6 +522,24 @@ static struct pike_string *do_read(int fd, } break; } + + /* + * Large reads cause the kernel to validate more memory before + * starting the actual read, and hence cause slowdowns. + * Ideally you read as much as you're going to receive. + * This buffer calculation algorithm tries to optimise the + * read length depending on past read chunksizes. + * For network reads this allows it to follow the packetsizes. + * The initial read will attempt the full buffer. + */ + if( i < try_read ) + i += SMALL_NETBUF; + else + i += (r-i)>>1; + if (i < SMALL_NETBUF) + i = SMALL_NETBUF; + if(i > r-SMALL_NETBUF) + i = r; }while(r); UNSET_ONERROR(ebuf); @@ -533,18 +558,22 @@ static struct pike_string *do_read(int fd, /* For some reason, 8k seems to work faster than 64k. * (4k seems to be about 2% faster than 8k when using linux though) * /Hubbe (Per pointed it out to me..) + * + * The slowdowns most likely are because of memory validation + * done by the kernel for buffer space which is later unused + * (short read) */ -#define CHUNK ( 1024 * 8 ) - INT32 try_read; dynamic_buffer b; b.s.str=0; initialize_buf(&b); SET_ONERROR(ebuf, free_dynamic_buffer, &b); + i = all && (r<<1)>r ? DIRECT_BUFSIZE : READ_BUFFER; do{ int e; char *buf; - try_read=MINIMUM(CHUNK,r); + + try_read = i; buf = low_make_buf_space(try_read, &b); @@ -555,23 +584,13 @@ static struct pike_string *do_read(int fd, check_threads_etc(); - if(i==try_read) - { - r-=i; - bytes_read+=i; - if(!all) break; - } - else if(i>0) + if(i>=0) { bytes_read+=i; r-=i; - low_make_buf_space(i - try_read, &b); - if(!all) break; - } - else if(i==0) - { - low_make_buf_space(-try_read, &b); - break; + if(try_read > i) + low_make_buf_space(i - try_read, &b); + if(!all || !i) break; } else { @@ -588,6 +607,16 @@ static struct pike_string *do_read(int fd, break; } } + + /* See explanation in previous loop above */ + if( i < try_read ) + i += SMALL_NETBUF; + else + i += (DIRECT_BUFSIZE-i)>>1; + if (i < SMALL_NETBUF) + i = SMALL_NETBUF; + if(i > r-SMALL_NETBUF) + i = r; }while(r); UNSET_ONERROR(ebuf); @@ -596,7 +625,6 @@ static struct pike_string *do_read(int fd, ADD_FD_EVENTS (THIS, PIKE_BIT_FD_READ); return low_free_buf(&b); -#undef CHUNK } } @@ -2457,7 +2485,7 @@ static void file_listxattr(INT32 args) if( res<0 && errno==ERANGE ) { /* Too little space in buffer.*/ - int blen = 65536; + int blen = DIRECT_BUFSIZE; do_free = 1; ptr = xalloc( 1 ); do { @@ -2527,7 +2555,7 @@ static void file_getxattr(INT32 args) if( res<0 && errno==ERANGE ) { /* Too little space in buffer.*/ - int blen = 65536; + int blen = DIRECT_BUFSIZE; do_free = 1; ptr = xalloc( 1 ); do { -- Sincerely, Stephen R. van den Berg. E-mails should be like a lady's skirt: Long enough to cover the subject, and short enough to be interesting.

3 2

string(0..255)
by Martin Stjernholm, Roxen IS ＠ Pike developers forum 23 Aug '08

23 Aug '08

#pragma strict_types int main (int argc, array(string) argv) { string s = argv[0]; Stdio.File ("test", "cw")->write (s); } produces these warnings for the write call: foo.pike:5: Warning: Type mismatch in argument 1 to write. foo.pike:5: Warning: Expected: array(string(0..255)) | string(0..255) | __attribute__("sprintf_format", string(0..255)). foo.pike:5: Warning: Got : string. This is strictly speaking correct and in the long run it's probably a good idea to restrict I/O functions to octet strings. But in practice this warning is very cumbersome and will probably cause widespread proliferation of the fairly clumsy type "string(0..255)" if one tries to fix it. My view: o There should be a better type name than "string(0..255)" for octet strings. (That type cannot be a typedef since typedef'ed types still don't work very well.) o If we're to separate the types for octet and char strings, I think it should be done also when char strings happen to contain only narrow 8-bit chars. I.e. the string width is not really the important characteristic, rather it's whether the string contains characters or binary octets. o Fixing the types for the I/O functions requires discussion (see e.g. points above). It ought to be done consistently over all I/O subsystems. Compat and utility functions might be required. This is not something to embark on at this time, I think. To sum up, for 7.8 I think it's better to not introduce string(0..255) in write() etc. Opinions?

8 23

Re: RFC int2string optimisation
by Mirar ＠ Pike developers forum 22 Aug '08

22 Aug '08

Maybe add this? Index: src/testsuite.in =================================================================== RCS file: /pike/data/cvsroot/Pike/7.8/src/testsuite.in,v retrieving revision 1.847 diff -u -r1.847 testsuite.in --- src/testsuite.in 17 Aug 2008 16:23:11 -0000 1.847 +++ src/testsuite.in 22 Aug 2008 06:10:06 -0000 @@ -7701,6 +7701,10 @@ test_eq([[(float)-1]],[[-1.0]]) test_eq([[(string)1]],[["1"]]) test_eq([[(string)-1]],[["-1"]]) +test_do(return (string)1e308) +test_do(return (string)1e200) +test_do(return (string)1e-200) +test_do(return (string)1e-307) test_any([[ array foo=({ 1 }); return (string)sizeof(foo);

2 2

Re: 7.8 alpha 2
by bertrand.lupart＠linkeo.com 20 Aug '08

20 Aug '08

Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > Not MacOSX/PPC as far as I can see. I've readded my 10.4/PPC to xenofarm, but i guess this bug requires 10.5/PPC -- Bertrand LUPART http://bertrand.gotpike.org/

1 0

7.8 alpha 2
by Peter Bortas ＠ Pike developers forum 20 Aug '08

20 Aug '08

7.8 alpha 2 is avilable as http://pike.ida.liu.se/pub/pike/alpha/7.8.8/Pike-v7.8.8.tar.gz It's not a beta because the changelog isn't complete. Other than that I know of no release blockers.

10 63

automatic casts
by Arne Goedeke 18 Aug '08

18 Aug '08

I am wondering how hard it would be to implement automatic casts from objects to basic types. You probably know what I mean... small example: void fun(int a) { ... } object i = Int(32); // Class offering a cast method fun(i); // cast to int automatically I admit that it introduces new traps and some complexity. Any opinions? arne

5 7

Webbased git browser (gitweb) available
by Stephen R. van den Berg 18 Aug '08

18 Aug '08

Currently still at port 8080, soon at port 80: http://pike-git.lysator.liu.se:8080/ Not quite the same as using your local repository with gitk, but nice for people lacking git, that want a (snap)shot or to have a peek. -- Sincerely, Stephen R. van den Berg. "The difficult we do today; the impossible takes a little longer."

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel