Pike-devel August 2017

pike-devel@lists.lysator.liu.se

4 participants
10 discussions

GTK/GDK on OSX
by H William Wellliver III 08 Jan '18

08 Jan '18

I’m running into some build problems on OSX. It seems that the GTK module assumes that if a system is NT that gdkwin.h is used and otherwise gdkx.h is used. Unfortunately, it seems that on some builds of GTK/GDK for OS X, neither of those is correct and that gdkquartz.h should be used. I’m guessing that some configure tests are needed to determine which of the 3 files is actually present. However a quick look at configure.in leaves me unsure how best to include that. Anyone with more expertise in the GTK module’s inner workings care to weigh in? Bill

4 9

Conditional inclusion of inotify_config.h?
by Chris Angelico 22 Nov '17

22 Nov '17

Branch: rosuav/inotify-config Currently, inotify.cmod won't go looking for inotify_config.h unless HAVE_INOTIFY_CONFIG_H is defined. This doesn't seem to match the way other autoheader files get included, and it results in System.Inotify not being available even on systems that support inotify. How does this system work? Is the config file absent in some situations? Not merging this change into trunk yet in case it breaks stuff. ChrisA

2 1

more on smartlink on darwin
by H. William Welliver III 22 Nov '17

22 Nov '17

I’ve looked a little more into the linking problems I’ve been having and noticed that on Darwin, smartlink.c is compiled to use LD_LIBRARY_PATH/LD_RUN_PATH. That’s problematic, because Darwin uses DYLD_LIBRARY_PATH and doesn’t have an equivalent of LD_RUN_PATH, so it doesn’t seem like smart link is doing much of anything for us on Darwin. I considered switching it to use -Wl,-rpath, however the darwin linker -rpath flag works differently from all of the other variants in that it only accepts one path per instance of the flag. smartlink.c seems to be set up to use a colon delimited list, which won’t work on Darwin (well, it isn’t rejected but won’t work as required). Before I start cluttering up smartlink.c with the code necessary to properly support the Darwin linker, does anyone have any thoughts? Bill

2 1

The Crypto module and mere mortals
by Pontus Östlund 22 Nov '17

22 Nov '17

Everytime I need to use the Crypto module, a great wave of anxiety sweeps over me. For most common programmers the case is usually that you need to MD5 or SHA256 som some data to send over HTTP to some REST API or something like that, or compare something to a hash stored in a database or something like that. I had to make HMAC-SHA256 hash yesterday and it took me like 1,5 hours to figure that stuff out: String.string2hex(Crypto.HMAC(Crypto.SHA256)(secret)(data)); Not the most intuitive stuff if you ask me. That led me to the question: Why aren't there any helper methods directly in Crypto, for the more common use cases? So I've been fiddling around with it some and though that maybe something like below could be nice to have. This is just a mockup so feel free to point out if there are better ways to write the code, and if this is something that would fit at all. //! Typedef for a generic hashing function that takes a string as argument //! and returnes a hashed and @[String.string2hex]'ed string. typedef function(string(8bit):string(8bit)) hash_func_t; //! Typedef for a generic HMAC hash function that takes data and a secret as //! arguments and returns a hashed and @[String.string2hex]'ed string typedef function(string(8bit),string(8bit):string(8bit)) hash_hmac_func_t; protected hash_func_t make_hash_func(Crypto.Hash hash, void|bool raw) { return lambda (string in) { string s = hash->hash(in); return raw ? s : String.string2hex(s); }; } protected hash_hmac_func_t make_hash_hmac_func(Crypto.Hash hash, void|bool raw) { return lambda (string data, string secret) { string s = Crypto.HMAC(hash)(secret)(data); return raw ? s : String.string2hex(s); }; } //! Create a HMAC hash of @[data] and @[secret] using the Crypto module @[hash]. //! //! @param hash //! @param data //! @param secret string hash_hmac(Crypto.Hash hash, string data, string secret) { return make_hash_hmac_func(hash)(data, secret); } //! Creates a HMAC hashing function using the Crypto module @[hash]. This can //! later be called with the data and secret to make a hash of. //! //! @returns //! @tt{function(string, string : string)@} variant hash_hmac_func_t hash_hmac(Crypto.Hash hash) { return make_hash_hmac_func(hash); } //! Creates a hash of @[data] using the hashing algorithm in @[hash]. //! If @[raw] is given the hash will not pass through @[String.string2hex]. string hash(Crypto.Hash hash, string data, void|bool raw) { return make_hash_func(hash, raw)(data); } //! Creates a hashing function using the Crypto module @[hash]. This can later //! be called with the data to make a hash of. //! If @[raw] is given the hash will not pass through @[String.string2hex]. //! //! @returns //! @tt{function(string : string)@} variant hash_func_t hash(Crypto.Hash hash, void|bool raw) { return make_hash_func(hash, raw); } hash_hmac_func_t hmac_sha256 = hash_hmac(Crypto.SHA256); hash_hmac_func_t hmac_md5 = hash_hmac(Crypto.MD5); hash_func_t md5 = hash(Crypto.MD5); hash_func_t sha256 = hash(Crypto.SHA256); int main(int argc, array(string) argv) { string data = "my data"; string secret = "abc123"; write("%-20s %s\n", "hash_hmac(MD5):", hash_hmac(Crypto.MD5, data, secret)); write("%-20s %s\n", "hash_hmac_md5:", hmac_md5(data, secret)); write("%-20s %s\n", "hash_hmac_sha256:", hmac_sha256(data, secret)); write("%-20s %s\n", "hash(md5):", hash(Crypto.MD5, data)); write("%-20s %s\n", "hash(md5:raw):", hash(Crypto.MD5, data, true)); write("%-20s %s\n", "md5:", md5(data)); write("%-20s %s\n", "sha256:", sha256(data)); return 0; } Regards ----------------------------- Pontus Östlund Developer • Roxen AB +46 70-662 81 69 www.roxen.com <http://www.roxen.com/> | twitter.com/roxen <https://twitter.com/roxen>

3 2

autoconf and ac_tool_prefix
by H. William Welliver III 22 Nov '17

22 Nov '17

I’ve noticed that there are a number of checks like this in various configure.ins: MY_AC_PATH_PROG(FT_CONFIG, ${ac_tool_prefix}freetype-config, no) The pkgsrc framework automatically sets the --host option (among other things) on configure, which causes ac_tool_prefix to be set to something like x86_64-sun-solaris2.11. The problem is that in this case (and others), there is no x86_64-sun-solaris2.11-freetype-config, only freetype-config. That causes a lot of dependencies to be left unfound, which isn’t ideal. I notice that the check for gcc falls back to gcc, so it’s possible there’s a bug in autoconf itself. Would anyone care to offer an opinion as to the proper solution to this problem? I could create a bunch of symlinks, but that doesn’t seem like the best way to solve this (though it might be the quickest). Bill

2 1

Interactive debugging and live data inspection
by Stephen R. van den Berg 22 Nov '17

22 Nov '17

I just checked in a debugging aid I created to be able to do some live data inspection in running Pike programs (I used it to find the cause of a deadlock in pgsql). I'd appreciate feedback on the implementation and usability of Debug.Peek (in Pike 8.1). To get a rough impression of what it does, here's an excerpt out of the documentation: ------------------------------------------------ Class Debug.Peek Description Allows for interactive debugging and live data structure inspection in both single- and multi-threaded programs. Creates an independent background thread that every pollinterval will show a list of running threads. Optionally, a triggersignal can be specified which allows the dump to be triggered by signal. Example: In the program you'd like to inspect, insert the following one-liner: Debug.Peek("/tmp/test.pike"); Then start the program and keep it running. Next you create a /tmp/test.pike with the following content: void create() { werror("Only once per modification of test.pike\n"); } int main() { werror("This will run every iteration\n"); werror("By returning 1 here, we disable the stacktrace dumps\n"); return 0; } void destroy() { werror("destroy() runs just as often as create()\n"); } Whenever you edit /tmp/test.pike, it will automatically reload the file. ------------------------------------------------ -- Stephen.

3 4

Proposal: Create Stdio.Port from systemd-provided socket
by Chris Angelico 22 Nov '17

22 Nov '17

Branch: rosuav/systemd-sockets Currently, you can create a Stdio.Port("stdin") as a means of accepting a socket handed to you as FD 0. However, there's no way to accept a socket given as any other FD, making it impossible to use this for systemd's sockets. Example: $ cat /etc/systemd/system/listendemo.service [Unit] Description=Listen Demo Requires=listendemo.socket [Service] User=rosuav ExecStart=/usr/local/bin/pike /home/rosuav/listendemo.pike WorkingDirectory=/home/rosuav $ cat /etc/systemd/system/listendemo.socket [Socket] ListenStream=123 $ cat listendemo.pike object mainsock = Stdio.Port("systemd", acceptloop); void acceptloop() { while (object sock=mainsock->accept()) { write("Accepted sock: %O\n", sock); sock->write("Stub\n"); sock->close(); } } int main() { write("Main sock: %O\n", mainsock); return -1; } Note that the "User=rosuav" directive means that Pike is run as a non-root user. This spares the application the hassle of managing privileges securely, as it never gets root privileges (assuming, of course, that the only reason it needed root was to bind to a privileged port, which is true for a lot of programs). As with "stdin" mode, the second parameter is optional (blocking vs nonblocking). This won't break on non-systemd systems for two reasons: firstly, all it ever does is query environment variables, and secondly, it does this only if the application requests it explicitly, so you have to actually ask for it. Is this something that would be useful? Feel free to bikeshed the API - it's not in 8.1 yet, so anything can change. ChrisA

3 6

Roadtrip through Scandinavia, spontaneous meetup(s)?
by Stephen R. van den Berg 03 Aug '17

03 Aug '17

As it happens, I'm doing a roadtrip (with my wife, and 2 kids (19 and 7)) through Scandinavia right now. Our itinerary is a little bit flexible, we're planning and booking one day ahead, however, it looks like we are going to be in and around the following places in the next few days (actually, these are the places we plan on spending the nights on that day, so we're going to be in that place the following morning until lunch (at least) as well): Fri Aug 4th: Copenhagen Sat Aug 5th: Goteborg Sun Aug 6th: Linkoping Mon Aug 7th: Stockholm Tue Aug 8th: Stockholm Wed Aug 9th: Oslo If any Pikers think it'd be nice to meet up casually for lunch, dinner or just a beer, in and around those dates/cities, drop me a note. Incidentally, I can only read email replies, since the mailinglist gateway is still down (I presume). -- Stephen. <srb(a)cuci.nl>

2 2

Pike 8.0.466 beta/release candidate
by Peter Bortas 03 Aug '17

03 Aug '17

Pike 8.0.466 beta/release candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.466/Pike-v8.0.466.tar.gz Other builds: https://pike.lysator.liu.se/pub/pike/beta/8.0.466/Pike-v8.0.466-Darwin-15.4… https://pike.lysator.liu.se/pub/pike/beta/8.0.466/Pike-v8.0.466-win32-oldli… Last months release candidate never got promoted to release, so the changes is very similar. Changes since Pike 8.0.438 (release 9) ---------------------------------------------------------------------- New Features ------------ o Concurrent The Concurrent module simplifies asynchronous code by synchronizing events in different ways. As an example the connect() function shown below will respond with a Concurrent.Future object that at some point will represent either a connected socket or a failure. Concurrent.Future connect(string host, int port) { Stdio.File con = Stdio.File(); Concurrent.Promise p = Concurrent.Promise(); if( !con->async_connect(host, port, lambda(int success) { if(success) p->success(con); else p->failure("Failed to connect to "+host+":"+port+"\n"); })) { p->failure("Failed to open socket.\n"); } return p->future(); } The returned future can then be used in various ways. // On success, call make_request(con, query). On failure call // werror(msg). connect(host, port) ->on_failure(werror) ->on_success(make_request, query); // On success, call make_request(con, query1) followed by // make_request(resp, query2), where resp is the return value from // make_reqest. connect(host, port) ->then(make_request, werror, query1) ->then(make_request, werror, query2); // Call bridge_ports(con1, con2) when both connections are // established. Concurrent.all(connect(host1, port1), connect(host2, port2)) ->then(bridge_ports, failure); // Call make_request(con) once either of the connections are // established. Concurrent.race(connect(host1, port1), connect(host2, port2)) ->then(make_requet, query); o Debug.Inspect Allows for interactive debugging and live data structure inspection in both single- and multi-threaded programs. Example: In the program you'd like to inspect, insert the following one-liner: Debug.Inspect("/tmp/test.pike"); Then start the program and keep it running. Next you create a /tmp/test.pike with the following content: void create() { werror("Only once per modification of test.pike\n"); } int main() { werror("This will run every iteration\n"); werror("By returning 1 here, we disable the stacktrace dumps\n"); return 0; } void destroy() { werror("destroy() runs just as often as create()\n"); } Whenever you edit /tmp/test.pike, it will automatically reload the file. Bug fixes --------- o ADT.Heap Fixed heap corruption when the same object is pushed more than once. It will now be considered as calling adjust(). o FSEvents EventStreamMonitor now works with other backends. o Filesystem.Monitor - Reduced initialization latency. - Changed the polling heuristic. This is intended to reduce poll (and notification) delays in some circumstances. - Improved robostness of adjust_monitor(). - Removed some dead and obsolete code. - Scan accelerated monitors too. Both the Inotify and FSEvents APIs claim that they support notifications on eg network file systems, and while they will succeed in notifying on changes performed by the local host on such file systems, they will not on changes performed by other hosts. To avoid missing such changes these monitors need to be actively scanned too. - Fixed lost acceleration with InotifyMonitor Recover from race-condition where when a file was repeatedly created and deleted, acceleration was lost. - Fixed symlink issue with InotifyMonitor. - Added support for more fine-grained tracing of monitors. o Inotify Addressed an issue where the backend might be stuck in pending indefinitely. o mappings Fixed an off by one error in random(mapping) that randomly caused values of type PIKE_T_FREE from the freelist to be exposed to Pike. o Nettle.Hash Fixed NULL-dereference in Nettle.Hash()->crypt_hash(). o Parser.HTML Fixed a condition where Pike would run out of stack space for large documents. o Search Fixed a race condition when updating the database. o Sql.pgsql - Sped up BEGIN/COMMIT statements. - Preserve the initial error message in case of multiple error messages during the same transaction. - Flush out unseen error messages upon connection close to stderr. - Closed a prepared-statement-cache race when the same statement is offered multiple times before it finalises the cache entry. - Fix sync errors with the database in case of multiple running statements on a single connection that generate multiple errors. - Eliminate spurious sync errors on pipe-lined transactions (multiple transactions in flight on a single file descriptor). - CancelRequests (to cancel running queries) caused sync errors with the database when multiple queries were in flight. - Eliminate transient error messages on database restarts. o Sql.SQLite When using query bindings, strings are now always stored as strings; previously, 8-bit strings were stored as binary objects (BLOBs) and wide strings were stored as unicode strings, which caused 8-bit strings to be unequal (and thus not returned) when used in a standard string query. Binary values may be stored as BLOBs using bindings by wrapping the binary string in a multiset. For example, to store the value "myBinary" as a BLOB value using a binding, use: (<"myBinary">). o Standards.BSON - Fixed incorrect encoding/decoding of Binary data. - Bugfixes for the decoding of Binary values of subtype 2, Binary values of subtypes above 0x7f and Datetime values with negative values. o Standards.JSON Fixed integer overflow in decode() on 64-bit platforms. o Stdio.Buffer - Fixed a crash when attempting to create a rewind key on a buffer returned by read_buffer(). - Fixed integer overflow in read_json(). Building -------- o Configure Attempt to detect the tool prefix depending on ABI. Improves detection of pkg-config. o Java Detect system installation of JavaVM framework on MacOS X. o Stdio Improved support for NetBSD. Regards, -- Peter Bortas

2 1

problems with configure and smartlink
by H William Wellliver III 01 Aug '17

01 Aug '17

I've been working on getting pike packages set up using pkgsrc, and I've run into some strange behavior on (at least) Linux: The configure scripts for modules seem to be using the smartlink binary and the generated Makefiles use the smartlink script at bin/smartlink. Is that intentional? Also, I think the configure test that controls how the smartlink binary works is broken (or at least different than the script), and causes configure and compilation to work incorrectly when using the pkgsrc cwrappers: The smartlink script converts -R options to -Wl,-rpath,..., and that the smartlink binary at least on the current system I'm using, seems to just drop them in favor of environment variables. For example (I modified the script to print the command it would run): [root@pkgsrc-pbulk Pike-v8.0.462]# SMARTLINK_DEBUG=true ./smartlink gcc -R/tmp -L/tmp SMARTLINK: gcc -L/tmp -L/tmp ... [root@pkgsrc-pbulk Pike-v8.0.462]# SMARTLINK_DEBUG=true bin/smartlink gcc -R/tmp -L/tmp SMARTLINK: gcc -L/tmp -Wl,-rpath,/tmp ... It seems to be because of the results of this test: checking for ld.so run path method... use LD_LIBRARY_PATH/LD_RUN_PATH It would seem like using the environment variables would be a use case of last resort. Perhaps -Wl should be used for more than just Linux 2.x? bill

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel August 2017