Pike-devel April 2016

pike-devel@lists.lysator.liu.se

8 participants
10 discussions

Optimised map
by Martin Karlgren 09 Jan '17

09 Jan '17

Hi, Having realised the benefits of functional programming, I’ve been quite annoyed by the rumour of how expensive function calls are in Pike. I decided to look into f_map and could see how much seemingly unnecessary work it does when it calls apply_svalue once for each entry in the array – it should be possible to reuse the pike_frame if it’s about to be thrown away after the function call (if it has other refs on the other hand, it can’t be reused – it’s probably used as a scope in some other frame). I’ve pushed my optimised variant in marty/optimised_map – it seems to work quite well and provides a major speedup. In fact, it’s a bit faster than the corresponding foreach variant. I haven’t verified correctness in various corner cases, and some input on whether it’s correct to do the things init_frame_reuse_context does only once before multiple function calls would be nice too. The *_reuse_context stuff in interpret.c should be applicable wherever the same svalue is applied repeatedly with the same number of arguments (I haven’t looked for it outside of f_map really). What do you all think? Good idea or did I overlook something? Without optimisation: map: 1.660797 array index: 1.335115 array append: 1.17917 With optimisation: map: 0.877659 array index: 1.351158 array append: 1.189812 Test program: int main() { array base = allocate(10000000, 1); float gmap = gauge { array res = map (base, lambda(int i) { return i + 2; }); }; float garrayindex = gauge { array res = allocate(sizeof(base)); foreach (base; int idx; int i) { res[i] = i + 2; } }; float garrayappend = gauge { array res = ({}); foreach (base, int i) { res += ({ i + 2 }); } }; werror ("map: %O\n", gmap); werror ("array index: %O\n", garrayindex); werror ("array append: %O\n", garrayappend); } /Marty

4 12

Re: Setting socket options: so many, uhh, options!
by Chris Angelico 15 Sep '16

15 Sep '16

On Tue, Mar 10, 2015 at 10:05 PM, Mirar @ Pike developers forum <10353(a)lyskom.lysator.liu.se> wrote: > Wait, I said REUSEPORT? What's the difference to REUSEADDR? Mysteries > of TCP sockets... Here's a decent explanation, I think: http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and… ChrisA

3 6

Revamping the Pike site
by Pontus Östlund 10 Jun '16

10 Jun '16

Hi guys! As awesome a programming language Pike is, the official Pike site does not really reflect that - to say the least ;) The site today does not exacly shout that the language it self is constantly developed and rapidly progressing. I thought I might do something about that. And so I have. Most things are starting to come together, so in a few weeks I feel we can go live with the new site. But before that happens I would love to have your thoughts and feedback on it. At the moment I have the site on my laptop which is not accessible from outsdie, but I’ve put toghether a page with some screenshots and some explainations which you can checkout. I have also revamped the module refdoc to better align with the new site. And since the module refdoc is 100% static HTML pages I’ve put a sample of it online on Github. Page with screenshots and som explainations of the new site: http://poppa.github.io/pike-dumps/ <http://poppa.github.io/pike-dumps/> Module refdoc: http://poppa.github.io/pikedoc/ <http://poppa.github.io/pikedoc/> So feel free to comment and give feedback. Oh, and since there are some really, really, really, really, really, really old content on the site we need to clean up some old stuff that’s no longer viable. I’ve already done some of that on my local copy, but I’ll put together a list of stuff we need to look into. But lets take that later... Regards ----------------------------- Pontus Östlund Developer • Roxen AB +46 70-662 81 69 www.roxen.com <http://www.roxen.com/> | twitter.com/roxen <https://twitter.com/roxen>

8 34

Re: Stdio.File()->pipe() and PROP_IPC
by Chris Angelico 27 Apr '16

27 Apr '16

On Thu, Apr 28, 2016 at 1:40 AM, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: >>Consider this usage: >> >>object stdout = Stdio.File(); >>Process.create_process(cmd, (["stdout": stdout->pipe()]); >> >>Is this at risk of failing, where stdout->pipe(Stdio.PROP_IPC) would succeed? > > Not currently, as the default type of pipe generated by pipe() is > fd_pipe() (which maps to CreatePipe on WIN32 and is IPC capable). > >>Does that imply that PROP_IPC is effectively always implied when >>creating a pipe? > > Not on WIN32. All currently-supported platforms appear to have IPC capabilities in the default pipe. Would it be worth changing the default from PROP_NONBLOCK|PROP_BIDIRECTIONAL to PROP_NONBLOCK|PROP_BIDIRECTIONAL|PROP_IPC? That wouldn't break any current code, and would mean that simple calls to stdout->pipe() would be guaranteed to work (instead of just happening to work). ChrisA

2 1

Stdio.File()->pipe() and PROP_IPC
by Chris Angelico 27 Apr '16

27 Apr '16

What exactly does PROP_IPC (fd_INTERPROCESSABLE in the source) permit? Consider this usage: object stdout = Stdio.File(); Process.create_process(cmd, (["stdout": stdout->pipe()]); Is this at risk of failing, where stdout->pipe(Stdio.PROP_IPC) would succeed? I can't find anything that indicates exactly what "IPC" means, and whether using one end of a pipe for a process's standard stream counts as IPC. It seems like it ought to, but the Pike standard library isn't consistent about requesting it (see Sql.pmod/rsql.pike:36, Git.pmod/Export.pike:46, and githelper.pike:60 from the companion repo). In _Stdio/file.c, file_pipe(), checks are made for a pipe (if PIPE_CAPABILITIES is defined), a socketpair, or a socketpair_ultra. All three of these have fd_INTERPROCESSABLE set, so far as I can tell. Does that imply that PROP_IPC is effectively always implied when creating a pipe? ChrisA

2 1

Re: Pike 8.0.222 release candidate
by Chris Angelico 25 Apr '16

25 Apr '16

On Wed, Apr 6, 2016 at 10:25 AM, Peter Bortas @ Pike developers forum <10353(a)lyskom.lysator.liu.se> wrote: > Well, the build machine is an XP. I don't mind dropping anything > before 7. But that means a whole new build environment. Not happening > this week. Fair enough. I'd guess that GTK 2.24.30 should run on XP, at least. I'm just picking this up again after a flat-out week of musical theatre, and am finding this: http://gtk-win.sourceforge.net/home/index.php/Main/Downloads The other option would be compiling GTK itself from source: http://ftp.gnome.org/pub/gnome/sources/gtk+/2.24/ ISTR the last time I tried to build Pike on Windows was a bit of a disaster. Sadly, there is nothing simple about cross-compilation. It'd be really awesome if the Windows 10 Linux subsystem makes that easier. ChrisA

2 3

Fwd: [Pike-automation] New Defects reported by Coverity Scan for Pike-experiment
by Peter Bortas 17 Apr '16

17 Apr '16

Since Arne has been active on his faster_calls branch I switched the experiment stream on Coverity Scan to have a look at it. These are the new issues if found when going from marty/optimised_map to arne/faster_calls. Without looking closer I suspect the four in Image isn't related to the optimizations code, but the two in interpreter probably are. If you want to have a look at them follow the link at the bottom. Gives a better overview than the condensed ASCII in this mail. -- Peter Bortas ---------- Forwarded message ---------- From: <scan-admin(a)coverity.com> Date: Sun, Apr 17, 2016 at 6:48 PM Subject: [Pike-automation] New Defects reported by Coverity Scan for Pike-experiment To: pike-automation(a)lists.lysator.liu.se Hi, Please find the latest report on new defect(s) introduced to Pike-experiment found with Coverity Scan. 6 new defect(s) introduced to Pike-experiment found with Coverity Scan. 1 defect(s), reported by Coverity Scan earlier, were marked fixed in the recent build analyzed by Coverity Scan. New defect(s) Reported-by: Coverity Scan Showing 6 of 6 defect(s) ** CID 1358595: Incorrect expression (UNUSED_VALUE) /home/covscan/pike/pike-git/src/interpret.c: 2202 in unlink_previous_frame() ________________________________________________________________________________________________________ *** CID 1358595: Incorrect expression (UNUSED_VALUE) /home/covscan/pike/pike-git/src/interpret.c: 2202 in unlink_previous_frame() 2196 */ 2197 current->save_sp=prev->save_sp; 2198 current->save_mark_sp=prev->save_mark_sp; 2199 current->flags = prev->flags; 2200 2201 /* Unlink the top frame temporarily. */ >>> CID 1358595: Incorrect expression (UNUSED_VALUE) >>> Assigning value from "prev" to "Pike_interpreter_pointer->frame_pointer" here, but that stored value is overwritten before it can be used. 2202 Pike_interpreter.frame_pointer=prev; 2203 2204 #ifdef PROFILING 2205 { 2206 /* We must update the profiling info of the previous frame 2207 * to account for that the current frame has gone away. ** CID 1358594: Control flow issues (MISSING_BREAK) /home/covscan/pike/pike-git/src/interpret.c: 1956 in do_trace_frame_call() ________________________________________________________________________________________________________ *** CID 1358594: Control flow issues (MISSING_BREAK) /home/covscan/pike/pike-git/src/interpret.c: 1956 in do_trace_frame_call() 1950 safe_describe_svalue(&obj_sval, 0, NULL); 1951 obj_name = complex_free_buf(&save_buf); 1952 PIKE_FN_START(function->name->size_shift == 0 ? 1953 function->name->str : "[widestring fn name]", 1954 obj_name.str); 1955 } >>> CID 1358594: Control flow issues (MISSING_BREAK) >>> The above case falls through to this one. 1956 case FRAME_CLONE: 1957 case FRAME_PARENT_CLONE: 1958 if (UNLIKELY(Pike_interpreter.trace_level)) { 1959 struct svalue tmp; 1960 struct program *p = frame->ptr; 1961 dynamic_buffer save_buf; ** CID 1358593: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match_phase() ________________________________________________________________________________________________________ *** CID 1358593: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match_phase() 196 DOUBLE_LOOP(0,1,1, *1, *1, *1) 197 else if (type==2) 198 DOUBLE_LOOP(0, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 199 else if (type==3) 200 DOUBLE_LOOP(AVOID_IS_TOO_BIG,1,1, *1, *1, *1) 201 else if (type==4) >>> CID 1358593: Null pointer dereferences (FORWARD_NULL) >>> Dereferencing null pointer "haystack_certi". 202 DOUBLE_LOOP(AVOID_IS_TOO_BIG, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 203 204 #undef NORMCODE 205 #undef AVOID_IS_TOO_BIG 206 #undef PIXEL_VALUE_DISTANCE 207 #undef DOUBLE_LOOP ** CID 1358592: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match_norm_corr() ________________________________________________________________________________________________________ *** CID 1358592: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match_norm_corr() 196 DOUBLE_LOOP(0,1,1, *1, *1, *1) 197 else if (type==2) 198 DOUBLE_LOOP(0, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 199 else if (type==3) 200 DOUBLE_LOOP(AVOID_IS_TOO_BIG,1,1, *1, *1, *1) 201 else if (type==4) >>> CID 1358592: Null pointer dereferences (FORWARD_NULL) >>> Dereferencing null pointer "haystack_certi". 202 DOUBLE_LOOP(AVOID_IS_TOO_BIG, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 203 204 #undef NORMCODE 205 #undef AVOID_IS_TOO_BIG 206 #undef PIXEL_VALUE_DISTANCE 207 #undef DOUBLE_LOOP ** CID 1358591: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match_norm() ________________________________________________________________________________________________________ *** CID 1358591: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match_norm() 196 DOUBLE_LOOP(0,1,1, *1, *1, *1) 197 else if (type==2) 198 DOUBLE_LOOP(0, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 199 else if (type==3) 200 DOUBLE_LOOP(AVOID_IS_TOO_BIG,1,1, *1, *1, *1) 201 else if (type==4) >>> CID 1358591: Null pointer dereferences (FORWARD_NULL) >>> Dereferencing null pointer "haystack_certi". 202 DOUBLE_LOOP(AVOID_IS_TOO_BIG, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 203 204 #undef NORMCODE 205 #undef AVOID_IS_TOO_BIG 206 #undef PIXEL_VALUE_DISTANCE 207 #undef DOUBLE_LOOP ** CID 1358590: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match() ________________________________________________________________________________________________________ *** CID 1358590: Null pointer dereferences (FORWARD_NULL) /home/covscan/pike/pike-git/src/modules/Image/match.h: 202 in image_match() 196 DOUBLE_LOOP(0,1,1, *1, *1, *1) 197 else if (type==2) 198 DOUBLE_LOOP(0, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 199 else if (type==3) 200 DOUBLE_LOOP(AVOID_IS_TOO_BIG,1,1, *1, *1, *1) 201 else if (type==4) >>> CID 1358590: Null pointer dereferences (FORWARD_NULL) >>> Dereferencing null pointer "haystack_certi". 202 DOUBLE_LOOP(AVOID_IS_TOO_BIG, haystack_certi[j], needle_certi[ny*nxs+nx],.r,.g,.b) 203 204 #undef NORMCODE 205 #undef AVOID_IS_TOO_BIG 206 #undef PIXEL_VALUE_DISTANCE 207 #undef DOUBLE_LOOP ________________________________________________________________________________________________________ To view the defects in Coverity Scan visit, https://scan.coverity.com/projects/pike-experiment?tab=overview

1 0

pike -x benchmark
by Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) ＠ Pike (-) developers forum 15 Apr '16

15 Apr '16

I noticed that starting the benchmark allocates an enormous amount of memory. The reason seems to be Foreach{,2,3,4} benchmarks. Each of these has the line array const_array = enumerate(10000000); on the top level. This means that there is an object occupying 160M (10M * sizeof(svalue)) in each of these _programs_, so it is always present when the programs are loaded, not just when the benchmarks are run. So in short, loading these four benchmarks (pike -x benchmark loads all the benchmarks) allocates 640M ram. That's a bit steep for my Raspberry Pi which only has 1G in total... (If I change the size of the arrays to 3M elements then the benchmarks complete normally.) Could perhaps each of these arrays be allocated before the respective benchmark is run, and freed afterwards, instead of being allocated when the benchmark is loaded and freed never?

1 0

Re: Pike 8.0.222 release candidate
by Chris Angelico 06 Apr '16

06 Apr '16

On Wed, Apr 6, 2016 at 9:50 AM, Peter Bortas @ Pike developers forum <10353(a)lyskom.lysator.liu.se> wrote: >>I can't recommend this to my users while it ships GTK 2.12.11 :( Is >>there anything I can do to help out there? > > Well. GTK+ used to be available as an all-in-one package for Windows > with a readme file saying > > "This is a bundle containing the GTK+ stack and its dependencies for > Windows. As such it is nothing more than a combination of the > individual packages it consists of. For a list, see > share/doc/gtk+-bundle-2.12.11-20080707/components.lst. Both run-time > and developer packages are included." > > I _think_ that is discontinued. If you can figure out a place to get a > working build environment for compiling with Visual Studio that would > help a bit. If you can find a place to get something that works with > VC9 it would help even more. According to this page [1], the current versions require Vista or later. Not sure whether that means GTK3 or GTK2 though. What Windows versions is Pike aiming to support? ChrisA [1] http://www.gtk.org/download/windows.php

2 1

Pike 8.0.222 release candidate
by Peter Bortas 05 Apr '16

05 Apr '16

No final release today. The disk in my virtual Windows machine broke. Fixing that took up time that should have been used for testing, so you will only get some completely untested builds. Pike 8.0.222 release candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.222/Pike-v8.0.222.tar.gz Windows build: https://pike.lysator.liu.se/pub/pike/beta/8.0.222/Pike-v8.0.222-win32-oldli… Changes since Pike 8.0.182 (release 2) ---------------------------------------------------------------------- New features ------------ o Roxen.http_decode_string() now decodes UTF-16 surrogate pairs. o Protocols.HTTP.Query now always sends a content-length header. o Report feature flags for System.FSEvents and System.Inotify. Bug fixes --------- o The machine code generator on ia32 now aligns the stack on a 16 byte boundary. This makes it possible to call code that uses SSE2 instructions. o Exceptions thrown in GTK signal handlers no longer cause segfaults. o The SSL client code now interprets the RFCs more leniently. o Multiple fixes for the handling of accelerated Filesystem.Monitor monitors, which could sometimes lose notifications. o Improved handling of response content-length in Protocols.HTTP.Query. o Fixed case where unfinished programs could lose their parents. o Fixed "Invalid service"-error from connect() on Fedora Core 4. o Fixed next_result() in Sql.sql_array_result. o Fixed sizeof() in Array.Iterator, Multiset.Iterator and String.Iterator. Building and installing ----------------------- o Multiple fixes for building on Solaris. o Improved scanning for Odbc header files. o Fixed some issues with --with-abi flags not propagating correctly to the modules. -- Peter Bortas

3 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel April 2016