Pike-devel May 2018

pike-devel@lists.lysator.liu.se

7 participants
20 discussions

Pike 8.0.622 beta/release candidate
by Peter Bortas 24 May '18

24 May '18

Pike 8.0.622 beta/release candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.622/Pike-v8.0.622.tar.gz Other builds: https://pike.lysator.liu.se/pub/pike/beta/8.0.622/Pike-v8.0.622-Darwin-15.4… https://pike.lysator.liu.se/pub/pike/beta/8.0.622/Pike-v8.0.622-win32-oldli… Changes since 8.0 rel 12: - Eliminates race condition in read_cstring - Fixed bug in normalize_path on win32 (This bug only exists in rel 11) Regards, -- Peter Bortas

3 2

pike build msql error with 8.0 git
by Lance Dillon 23 May '18

23 May '18

So trying to build pike 8.0 from git, and got an error about Msql: === configuring in Msql (/home/riffraff/src/pike/8.0/pike/build/linux-4.16.6-302.fc28.x86_64-x86_64/modules/Msql) configure: running /bin/bash -norc /home/riffraff/src/pike/8.0/pike/src/modules/Msql/configure --disable-option-checking '--prefix=/usr/local' --cache-file=/dev/null --srcdir=/home/riffraff/src/pike/8.0/pike/src/modules/Msql configure: error: cannot find sources (msqlmod.cmod) in /home/riffraff/src/pike/8.0/pike/src/modules/Msql configure: error: /home/riffraff/src/pike/8.0/pike/src/modules/Msql/configure failed for Msql configure: error: /home/riffraff/src/pike/8.0/pike/src/modules/configure failed for modules ll Msql/ total 52 -rw-rw-r--. 1 riffraff riffraff 403 May 10 20:22 acconfig.h -rw-rw-r--. 1 riffraff riffraff 2948 May 10 20:22 configure.in -rw-rw-r--. 1 riffraff riffraff 187 May 10 20:22 Makefile.in -rw-rw-r--. 1 riffraff riffraff 29556 May 10 20:22 msqlmod.c -rwxrwxr-x. 1 riffraff riffraff 824 May 10 20:22 test_msqlmod.pike -rw-rw-r--. 1 riffraff riffraff 0 May 10 20:22 testsuite.in -rw-rw-r--. 1 riffraff riffraff 334 May 10 20:22 version.h Before build attempt: $ pwd /home/riffraff/src/pike/8.0/pike/src/modules/Msql ? ~/src/pike/8.0/pike/src/modules/Msql [8.0|?19] 20:26 $ grep cmod * After (make distclean && make): ll total 220 -rw-rw-r--. 1 riffraff riffraff 403 May 10 20:26 acconfig.h -rwxrwxr-x. 1 riffraff riffraff 160015 May 10 20:27 configure -rw-rw-r--. 1 riffraff riffraff 2948 May 10 20:26 configure.in -rw-rw-r--. 1 riffraff riffraff 0 May 10 20:27 dependencies -rw-rw-r--. 1 riffraff riffraff 187 May 10 20:26 Makefile.in -rw-rw-r--. 1 riffraff riffraff 2236 May 10 20:27 msql_config.h.in -rw-rw-r--. 1 riffraff riffraff 29556 May 10 20:26 msqlmod.c -rw-rw-r--. 1 riffraff riffraff 4 May 10 20:27 stamp-h.in -rwxrwxr-x. 1 riffraff riffraff 824 May 10 20:26 test_msqlmod.pike -rw-rw-r--. 1 riffraff riffraff 0 May 10 20:26 testsuite.in -rw-rw-r--. 1 riffraff riffraff 334 May 10 20:26 version.h ? ~/src/pike/8.0/pike/src/modules/Msql [8.0|?19] 20:28 $ grep cmod * configure: PMOD_TARGETS=`echo $srcdir/*.cmod | sed -e "s/\.cmod/\.c/g" | sed -e "s|$srcdir/|\\$(SRCDIR)/|g"` So something in the build is creating configure so that it looks for *.cmod, and fails. However, I just now encountered that after I removed Msql, I had the same problem with Mysql Hmm, maybe it is me? Is there something I'm doing wrong? I've done this before in the past and haven't had this problem before.

5 12

interesting terminfo error
by Lance Dillon 23 May '18

23 May '18

I encountered this error on a system that had it's term type set to xterm-256color: $ TERM=xterm-256color pike Terminfo: unparsable terminfo file "/usr/share/terminfo/x/xterm-256color" /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Terminfo.pmod:406: Stdio.Terminfo.Terminfo()->create("/usr/share/terminfo/x/xterm-256color") /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Terminfo.pmod:746: Stdio.Terminfo.TerminfoDB("/usr/share/terminfo/")->load("xterm-256color") /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Terminfo.pmod:760: Stdio.Terminfo.TerminfoDB("/usr/share/terminfo/")->`[]("xterm-256color") /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Terminfo.pmod:814: Stdio.Terminfo.MetaTerminfoDB()->`[]("xterm-256color") /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Terminfo.pmod:862: Stdio.Terminfo.getTerminfo("xterm-256color") /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Terminfo.pmod:894: Stdio.Terminfo.getTerm("xterm-256color") /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Terminfo.pmod:886: Stdio.Terminfo.getTerm(UNDEFINED) /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Readline.pike:429: Stdio.Readline()->OutputController()->create(UNDEFINED,UNDEFINED) /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Readline.pike:6: Stdio.Readline()->OutputController() /usr/local/pike/8.0.498/lib/modules/Stdio.pmod/Readline.pike:1722: Stdio.Readline()->create(UNDEFINED,UNDEFINED,UNDEFINED,UNDEFINED) /usr/local/pike/8.0.498/lib/modules/Tools.pmod/Hilfe.pmod:2531: Tools.Hilfe.StdinHilfe()->create(UNDEFINED) /usr/local/pike/8.0.498/lib/modules/Tools.pmod/Hilfe.pmod:2476: Tools.Hilfe.StdinHilfe() $ file /usr/share/terminfo/x/xterm /usr/share/terminfo/x/xterm-256color /usr/share/terminfo/x/xterm: Compiled terminfo entry "xterm" /usr/share/terminfo/x/xterm-256color: Compiled 32-bit terminfo entry "xterm-256color" This occurs on every term type that says "Compiled 32-bit terminfo entry" I might dig through the code and see if I can make changes to let Terminfo.pmod recognize and use 32-bit terminfo files.

2 2

Pike 8.0.608 beta/release candidate
by Peter Bortas 21 May '18

21 May '18

Pike 8.0.608 beta/release candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.608/Pike-v8.0.608.tar.gz Other builds: https://pike.lysator.liu.se/pub/pike/beta/8.0.608/Pike-v8.0.608-Darwin-15.4… https://pike.lysator.liu.se/pub/pike/beta/8.0.608/Pike-v8.0.608-win32-oldli… Real Mac binary from Bill soon hopefully Changes since Pike 8.0.498 (release 11) ---------------------------------------------------------------------- New Features ------------ o Crypto.Hash.SCRAM New class to support SCRAM authentication. o Concurrent - Add map_with() as an alias to flat_map(). - Add a convenience variant to zip(). - depend() can be used to collect multiple dependencies before finalising the Promise. o Debug.find_all_clones() New convenience function to find all objects that are clones of a program. o MIME Added a setter for the message boundary prefix. o Sql.pgsql - Added support for PostgreSQL 10's stronger password authentication (SCRAM-SHA-256). - Slightly reduced overhead. - Support IEEE binary float database-wireformat. - Support NUMERIC database-wireformat. - Prevent new queries from starting in the local_backend. - Make sure huge queries are not fully loaded into memory. o Sql.Sql Make big_typed_query() gracefully fallback to big_query(). o SSL.File Support query_fd() and set_buffer_mode() methods. o Stdio.File [NT] A major clean-up and consistency improvment of the I/O layer on NT. - All filesystem paths are now encoded in UTF-8 from Pike-code's point of view. Previously this was a mix of Latin-1 and UTF-16, with some stuff affected by the filesystem default character set. This makes the entire filesystem accessible from Pike-code. This new behavior can be detected by the presence of the constant Stdio.__HAVE_UTF8_FS__. - Added new low-level fd handling functions. This should fix several potential race-conditions in the I/O layer on NT. o Stdio.Terminfo Support the new terminfo format from ncurses 6. o Thread.Farm Added Thread.Farm()->set_thread_name_cb() to help applications that monitor thread creation and termination for the purpose of tracking thread names. o Thread.ResourceCount New module to allow threadsafe race-condition-free counting of resources. o Tools.Standalone.check_http - Allow redirect responses. - Support --expect option. o Web.Sass This is a module that interfaces to the SASS compilation library "LibSass" (https://github.com/sass/libsass). Bug fixes --------- o ADT.CritBit - Due to an off-by-one error the last limb of bignums was never compared. This resulted (sometimes) in unequal bignums comparing equal inside of the tree. o Compiler - Improved variant robustness. - Fixed type leak for getters/setters implemented with variant functions. o Calendar - Updated tzdata to 2018d. - Calendar.TimeRanges.NullTimeRange now has a working __hash(). The expression ([ Calendar.TimeRanges.nulltimerange : 17 ]) no longer gives a backtrace. o Concurrent - Less filling, deadlock-free, faster, available without running backend. - Handle an empty argument array properly in results(). Previously code such as: Concurrent.Future f2 = Concurrent.results(({})); f2->on_success(lambda(array(string) a) { werror("success: %O.\n", a); }) ->on_failure(lambda(mixed err) { werror (describe_backtrace(err)); }); would lead to a backtrace because the future was destructed prematurely. - Readable/understandable documentation. o Crypto.CCM Fixed initialization of the iv. o Crypto.Hash Censor the password argument from backtraces in crypt_hash(). o Filesystem.Monitor - Fixed some backend_check rescheduling isusues. adjust_monitor() didn't reschedule the backend_check() call_out if the adjusted monitor already was at the head of monitor_queue. This caused polls to be delayed in some common cases. register_path() didn't reschedule the backend_check() call_out. If the newly registered monitor ends up at the head of the monitor_queue, the backend_check() call_out likely needs to be rescheduled. - Fixed issue with next_poll being set to zero causing polling every second. - Fixed indexing the NULL value error when InotifyMonitor was used in combination with a filter_file() that doesn't accept everything. - Fixed incorrect creation of new monitors. Monitors were created using monitor() on the top level rather than using Monitor::monitor(), which prevented the symlinks' overridden DefaultMonitor from setting the correct state on newly created monitors. We now force a check on the directory monitor instead, which will pick up the new file instantly and create the sub monitor. - Fixed race condition on deletion. This fixes a race that could occur on rapid exists => delete => exists transitions, where the directory monitor never noticed the change but the sub monitor was removed from the monitor lookup mapping. The directory monitor is now notified on sub-monitor release. - Added set_stable_time() function. - Removed obsolete special case for co_id == 1. o HTTPLoop: Fixed some potential race conditions. o Image.Image - Improved argument checking in apply_curve(). Non-integer array values lead to use of uninitialized curve values. o Image.PVR Fixed size check for twiddled images. o MIME Support trailing newline in MIME part epilogues. o Protocols.HTTP.Query - Fixed incoming chunked transfers (timed async works now, chunk options are properly 'parsed'). Trailer headers are now correctly parsed. - Improved handling of multiple headers of the same kind. - Fixed a bug where base64 encoded HTTP basic auth tokens were encoded in a pretty-printed format including line breaks, which lead to malformed HTTP requests on long username and password combinations. - Do not wait for SSL.File()->close() to complete when called from the destruct() callback. o Protocols.HTTP.Server - The header parser exception mode now doesn't immediately throw an exception. - Attempt to ensure that data is sent before terminating an HTTPS connection. o Protocols.LDAP Fixed race condition on initializing LDAPS. o Search.Database.MySQL Fixed issue where blobs didn't get updated after the initial blobs got full. o Search.Utils Updated debug string format. o Sql.pgsql - Suppress spurious 'File not open' blurts on stderr when the connection is still pending and we inspect the connection object. - Signal EOF early before closing the SQL result portal and thoroughly cleanup bufcon/stashcount to prevent occasional deadlocks on asynchronous object destructions. - Suppress rare "access in destructed object" messages. - Simplified fundamentally flawed reconnect logic, made failures consistent. - Repair SSL connect logic. - Update documentation. - Repair fetch_row_array(). - Ensure that all database errors result in exceptions (eventually). - affected_rows() and status_command_complete() now wait until they have a meaningful result to report. - Closing the database will wait until all running queries have been committed. - Report correct number of affected_rows() for INSERTs. - Eliminate a rare deadlock when running many simultaneous queries on a single filedescriptor. o mktime/System.TM - Make timezone management consistent (especially UTC handling). - Fix memory leak in strftime(). o Standards.IIM Disabled debug output for unknown segment markers. o Stdio - Restore the FD_SETSIZE on NT to 65536. Due to #include file restructuring, the FD_SETSIZE value on NT had fallen back to the default (64). This issue has probably affected all releases of Pike since 7.9.2. - mv() [NT] Fixed single byte buffer overrun in mv() on NT. A NUL-terminator was written outside a malloced buffer. This caused intermittent crashes after unusual use of mv(). Fixes the crash in [PIKE-90]. - send_file() Improved support for TLS/SSL; wait for TLS handshake to complete before calling the sendfile done callback when sending an empty string. This fixes the issue where the connection got closed before TLS handshaking was completed, confusing the other side. o Image Creating images with mode "grey" now works as documented, e.g. Image.Image(10,10,"grey",123); Optimizations ------------- o mktime/System.TM Considerable speedup and a reduction in codesize. o Unicode Avoid scanning the string in NFC mode if the string is 8-bit, as NFC normalization of 8-bit strings is the identity function. Building & Tools ---------------- o Hilfe Fix commands having priority over code. [LysLysKOM 22552693]. o Image.SVG Fall back to using the unprefixed pkg-config if the arch-prefixed doesn't exist. o Mysql Support MariaDB 10.2 headerfiles. o export - Fixed fix_configure. This was broken in multiple ways: * Did not consider timestamp on aclocal.m4 * Did not consider post_modules * Did not check configure scripts directly under modules or post_modules * When fixing modules, it looked for the configure scripts in the build directory rather than the source directory * Did not use the run_autoconfig wrapper This has caused xenofarm to export snapshots with out of date configure scripts. o precompile Support precompilation with Pike 8.1 again. o run_autoconfig Fix localdir when relative and not "./" o smartlink Fixed multiple issues on Darwin & NetBSD. o Whitefish Use the standard VPATH. o Yp Support using libtirpc. Glibc 2.26 has removed the Sun RPC implementation... Regards, -- Peter Bortas

4 8

Found and fixed a race condition in Stdio.Buffer->read_cstring()
by Stephen R. van den Berg 17 May '18

17 May '18

Committed fixes to both 8.1 and 8.0. The symptoms I had was that during a read_cstring(), the range_error() at times was called with a negative number. At that point, the cstring was already partially consumed and gone from the buffer (below offset = 0). I then scrutinised the Stdio.Buffer code to look for race conditions, and the only thing I could come up with is that because of the excursion to range_error(), somehow the buffer content changes and/or the locked_move lock is accidentally removed, and read_cstring() subsequently doesn't understand what happened. The fix avoids the use of io_rewind() after range_error() calls and is modelled after that what sscanf() does, i.e. call range_error(0) to indicate the amount we need is unknown. -- Stephen.

1 0

RFC Debug.Rapidlog() new module
by Stephen R. van den Berg 16 May '18

16 May '18

There is a new module I'd like to introduce called Debug.Rapidlog. When you inherit it, it overrides werror(), and allows for logging that interferes minimally with timing (it logs into a buffer, which then is emptied from a different thread at idle times). The idea behind this module is that it allows logging timing sensitive code. I find that when I add logging to (e.g. pgsql) to chase rare bugs, the logging itself often causes the nature of the bug to change (or disappear). Any objections/suggestions? I committed it to 8.1 for now, but I need the module in 8.0 as well eventually, since I have to run it in production systems at times. So please also evaluate fitness for inclusion in 8.0 (no rush though, it does not need to be in the latest release). -- Stephen.

2 2

Pike 8.0.610 beta/release candidate
by Peter Bortas 14 May '18

14 May '18

Pike 8.0.610 beta/release candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.610/Pike-v8.0.610.tar.gz Other builds: https://pike.lysator.liu.se/pub/pike/beta/8.0.610/Pike-v8.0.610-Darwin-15.4… https://pike.lysator.liu.se/pub/pike/beta/8.0.610/Pike-v8.0.610-win32-oldli… Changes since Fridays build: Solaris build fix. If you have time you can rebuild for this Bill, but I'm actually quite OK with using the Friday binary builds for Mac and RH with this release if needed. Regards, -- Peter Bortas

1 0

Compile problem on illumos for 8.0.608
by H. William Welliver III 13 May '18

13 May '18

There seems to be one, possibly two compile problems on illumos (ex-opensolaris). I just verified that 8.0.498 compiles without error, however a fresh download of 8.0.608 fails with this error: #### Making dynamic: post_modules/Shuffler There's no master to handle the error. Dumping it raw: ({ /* 2 elements */ "Failed to fetch current path.\n", ({ /* 4 elements */ object(home/zino/hack/pike8-rel/pike/src/builtin.cmod:2442), object(home/zino/hack/pike8-rel/pike/src/builtin.cmod:2442), object(home/zino/hack/pike8-rel/pike/src/builtin.cmod:2442), object(home/zino/hack/pike8-rel/pike/src/builtin.cmod:2442) }) }) Makefile:89: recipe for target 'override' failed make[4]: *** [override] Error 10 Makefile:100: recipe for target 'Shuffler' failed make[3]: *** [Shuffler] Error 1 I have output of the pike invocation with -t saved here: http://bill.welliver.org/dist/pike/8.0.608-shuffler-trace.txt I haven’t tried to find the source of the problem yet; just wanted to signal that there might be an issue. Bill

2 1

Re: Pike 8.0.608 beta/release candidate
by H. William Welliver III 13 May '18

13 May '18

I uploaded RPMs for x86_64 to the latest beta download directory. > On May 13, 2018, at 10:47 AM, Martin Bähr <mbaehr(a)realss.com> wrote: > > Excerpts from H. William Welliver III's message of 2018-05-13 10:09:53 -0400: >> I also have RPMs for EL/Centos 6/7 as well if anyone wants to give those a try. Just let me know and I’ll put them somewhere. > > yes, please. > i am running my servers on centos 7... > > greetings, martin.

1 0

Pike 8.0.528 release candidate
by Peter Bortas 11 May '18

11 May '18

As is tradition a release has been prepared at the new years party with Champagne in one hand and keyboard in the other waiting for the clock to hit 24:00. Pike 8.0.528 release candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.528/Pike-v8.0.528.tar.gz Other builds: https://pike.lysator.liu.se/pub/pike/beta/8.0.528/Pike-v8.0.528-Darwin-15.4… https://pike.lysator.liu.se/pub/pike/beta/8.0.528/Pike-v8.0.528-win32-oldli… Real Mac build comes when Bill has time to have a look at that. Changes since Pike 8.0.498 (release 11) ---------------------------------------------------------------------- New Features ------------ o Sql.pgsql - Added support for PostgreSQL 10's stronger password authentication (SCRAM-SHA-256). - Slightly reduced overhead. - Support IEEE binary float database-wireformat. - Prevent new queries from starting in the local_backend. o Crypto.Hash.SCRAM New class to support SCRAM authentication. o Thread.ResourceCount New module to allow threadsafe race-condition-free counting of resources. o SSL.File Support query_fd() and set_buffer_mode() methods. o Toole.Standalone.check_http - Allow redirect responses. - Support --expect option. Bug fixes --------- o Compiler Improved variant robustness. o Calendar Updated tzdata to 2017c. o Crypto.CCM Fixed initialization of the iv. o Filesystem.Monitor - Fixed issue with next_poll being set to zero causing polling every second. - Fixed indexing the NULL value error when InotifyMonitor was used in combination with a filter_file() that doesn't accept everything. o HTTPLoop: Fixed some potential race conditions. o Image.Image - Improved argument checking in apply_curve(). Non-integer array values lead to use of uninitialized curve values. o Image.PVR Fixed size check for twiddled images. o MIME Support trailing newline in MIME part epilogues. o Protocols.HTTP Fixed incoming chunked transfers (timed async works now, chunk options are properly 'parsed'). o Protocols.LDAP Fixed race condition on initializing LDAPS. o Search.Utils Updated debug string format. o Sql.pgsql - Suppress spurious 'File not open' blurts on stderr when the connection is still pending and we inspect the connection object. - Signal EOF early before closing the SQL result portal and thoroughly cleanup bufcon/stashcount to prevent occasional deadlocks on asynchronous object destructions. - Suppress rare "access in destructed object" messages. - Rip out fundamentally flawed reconnect logic, made failures consistent. - Repair SSL connect logic. - Update documentation. - Repair fetch_row_array(). o mktime/System.TM Make timezone management consistent (especially UTC handling). o Standards.IIM Disabled debug output for unknown segment markers. o Stdio.send_file Improved support for TLS/SSL; wait for TLS handshake to complete before calling the sendfile done callback when sending an empty string. This fixes the issue where the connection got closed before TLS handshaking was completed, confusing the other side. o Image Creating images with mode "grey" now works as documented, e.g. Image.Image(10,10,"grey",123); Optimizations ------------- o mktime/System.TM Considerable speedup and a reduction in codesize. Building & Tools ---------------- o Image.SVG Fall back to using the unprefixed pkg-config if the arch-prefixed doesn't exist. o Mysql Support MariaDB 10.2 headerfiles. o precompile Support precompilation with Pike 8.1 again. o smartlink Fixed multiple issues on Darwin & NetBSD. o Whitefish Use the standard VPATH. o Yp Support using libtirpc. Glibc 2.26 has removed the Sun RPC implementation... Regards, -- Peter Bortas

5 12

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel May 2018