Pike-devel July 2017

pike-devel@lists.lysator.liu.se

4 participants
9 discussions

GTK/GDK on OSX
by H William Wellliver III 08 Jan '18

08 Jan '18

I’m running into some build problems on OSX. It seems that the GTK module assumes that if a system is NT that gdkwin.h is used and otherwise gdkx.h is used. Unfortunately, it seems that on some builds of GTK/GDK for OS X, neither of those is correct and that gdkquartz.h should be used. I’m guessing that some configure tests are needed to determine which of the 3 files is actually present. However a quick look at configure.in leaves me unsure how best to include that. Anyone with more expertise in the GTK module’s inner workings care to weigh in? Bill

4 9

The Crypto module and mere mortals
by Pontus Östlund 22 Nov '17

22 Nov '17

Everytime I need to use the Crypto module, a great wave of anxiety sweeps over me. For most common programmers the case is usually that you need to MD5 or SHA256 som some data to send over HTTP to some REST API or something like that, or compare something to a hash stored in a database or something like that. I had to make HMAC-SHA256 hash yesterday and it took me like 1,5 hours to figure that stuff out: String.string2hex(Crypto.HMAC(Crypto.SHA256)(secret)(data)); Not the most intuitive stuff if you ask me. That led me to the question: Why aren't there any helper methods directly in Crypto, for the more common use cases? So I've been fiddling around with it some and though that maybe something like below could be nice to have. This is just a mockup so feel free to point out if there are better ways to write the code, and if this is something that would fit at all. //! Typedef for a generic hashing function that takes a string as argument //! and returnes a hashed and @[String.string2hex]'ed string. typedef function(string(8bit):string(8bit)) hash_func_t; //! Typedef for a generic HMAC hash function that takes data and a secret as //! arguments and returns a hashed and @[String.string2hex]'ed string typedef function(string(8bit),string(8bit):string(8bit)) hash_hmac_func_t; protected hash_func_t make_hash_func(Crypto.Hash hash, void|bool raw) { return lambda (string in) { string s = hash->hash(in); return raw ? s : String.string2hex(s); }; } protected hash_hmac_func_t make_hash_hmac_func(Crypto.Hash hash, void|bool raw) { return lambda (string data, string secret) { string s = Crypto.HMAC(hash)(secret)(data); return raw ? s : String.string2hex(s); }; } //! Create a HMAC hash of @[data] and @[secret] using the Crypto module @[hash]. //! //! @param hash //! @param data //! @param secret string hash_hmac(Crypto.Hash hash, string data, string secret) { return make_hash_hmac_func(hash)(data, secret); } //! Creates a HMAC hashing function using the Crypto module @[hash]. This can //! later be called with the data and secret to make a hash of. //! //! @returns //! @tt{function(string, string : string)@} variant hash_hmac_func_t hash_hmac(Crypto.Hash hash) { return make_hash_hmac_func(hash); } //! Creates a hash of @[data] using the hashing algorithm in @[hash]. //! If @[raw] is given the hash will not pass through @[String.string2hex]. string hash(Crypto.Hash hash, string data, void|bool raw) { return make_hash_func(hash, raw)(data); } //! Creates a hashing function using the Crypto module @[hash]. This can later //! be called with the data to make a hash of. //! If @[raw] is given the hash will not pass through @[String.string2hex]. //! //! @returns //! @tt{function(string : string)@} variant hash_func_t hash(Crypto.Hash hash, void|bool raw) { return make_hash_func(hash, raw); } hash_hmac_func_t hmac_sha256 = hash_hmac(Crypto.SHA256); hash_hmac_func_t hmac_md5 = hash_hmac(Crypto.MD5); hash_func_t md5 = hash(Crypto.MD5); hash_func_t sha256 = hash(Crypto.SHA256); int main(int argc, array(string) argv) { string data = "my data"; string secret = "abc123"; write("%-20s %s\n", "hash_hmac(MD5):", hash_hmac(Crypto.MD5, data, secret)); write("%-20s %s\n", "hash_hmac_md5:", hmac_md5(data, secret)); write("%-20s %s\n", "hash_hmac_sha256:", hmac_sha256(data, secret)); write("%-20s %s\n", "hash(md5):", hash(Crypto.MD5, data)); write("%-20s %s\n", "hash(md5:raw):", hash(Crypto.MD5, data, true)); write("%-20s %s\n", "md5:", md5(data)); write("%-20s %s\n", "sha256:", sha256(data)); return 0; } Regards ----------------------------- Pontus Östlund Developer • Roxen AB +46 70-662 81 69 www.roxen.com <http://www.roxen.com/> | twitter.com/roxen <https://twitter.com/roxen>

3 2

autoconf and ac_tool_prefix
by H. William Welliver III 22 Nov '17

22 Nov '17

I’ve noticed that there are a number of checks like this in various configure.ins: MY_AC_PATH_PROG(FT_CONFIG, ${ac_tool_prefix}freetype-config, no) The pkgsrc framework automatically sets the --host option (among other things) on configure, which causes ac_tool_prefix to be set to something like x86_64-sun-solaris2.11. The problem is that in this case (and others), there is no x86_64-sun-solaris2.11-freetype-config, only freetype-config. That causes a lot of dependencies to be left unfound, which isn’t ideal. I notice that the check for gcc falls back to gcc, so it’s possible there’s a bug in autoconf itself. Would anyone care to offer an opinion as to the proper solution to this problem? I could create a bunch of symlinks, but that doesn’t seem like the best way to solve this (though it might be the quickest). Bill

2 1

Interactive debugging and live data inspection
by Stephen R. van den Berg 22 Nov '17

22 Nov '17

I just checked in a debugging aid I created to be able to do some live data inspection in running Pike programs (I used it to find the cause of a deadlock in pgsql). I'd appreciate feedback on the implementation and usability of Debug.Peek (in Pike 8.1). To get a rough impression of what it does, here's an excerpt out of the documentation: ------------------------------------------------ Class Debug.Peek Description Allows for interactive debugging and live data structure inspection in both single- and multi-threaded programs. Creates an independent background thread that every pollinterval will show a list of running threads. Optionally, a triggersignal can be specified which allows the dump to be triggered by signal. Example: In the program you'd like to inspect, insert the following one-liner: Debug.Peek("/tmp/test.pike"); Then start the program and keep it running. Next you create a /tmp/test.pike with the following content: void create() { werror("Only once per modification of test.pike\n"); } int main() { werror("This will run every iteration\n"); werror("By returning 1 here, we disable the stacktrace dumps\n"); return 0; } void destroy() { werror("destroy() runs just as often as create()\n"); } Whenever you edit /tmp/test.pike, it will automatically reload the file. ------------------------------------------------ -- Stephen.

3 4

Proposal: Create Stdio.Port from systemd-provided socket
by Chris Angelico 22 Nov '17

22 Nov '17

Branch: rosuav/systemd-sockets Currently, you can create a Stdio.Port("stdin") as a means of accepting a socket handed to you as FD 0. However, there's no way to accept a socket given as any other FD, making it impossible to use this for systemd's sockets. Example: $ cat /etc/systemd/system/listendemo.service [Unit] Description=Listen Demo Requires=listendemo.socket [Service] User=rosuav ExecStart=/usr/local/bin/pike /home/rosuav/listendemo.pike WorkingDirectory=/home/rosuav $ cat /etc/systemd/system/listendemo.socket [Socket] ListenStream=123 $ cat listendemo.pike object mainsock = Stdio.Port("systemd", acceptloop); void acceptloop() { while (object sock=mainsock->accept()) { write("Accepted sock: %O\n", sock); sock->write("Stub\n"); sock->close(); } } int main() { write("Main sock: %O\n", mainsock); return -1; } Note that the "User=rosuav" directive means that Pike is run as a non-root user. This spares the application the hassle of managing privileges securely, as it never gets root privileges (assuming, of course, that the only reason it needed root was to bind to a privileged port, which is true for a lot of programs). As with "stdin" mode, the second parameter is optional (blocking vs nonblocking). This won't break on non-systemd systems for two reasons: firstly, all it ever does is query environment variables, and secondly, it does this only if the application requests it explicitly, so you have to actually ask for it. Is this something that would be useful? Feel free to bikeshed the API - it's not in 8.1 yet, so anything can change. ChrisA

3 6

Pike 8.0.462 beta/release candidate
by Peter Bortas 14 Jul '17

14 Jul '17

Pike 8.0.462 beta/release candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.462/Pike-v8.0.462.tar.gz Other builds: https://pike.lysator.liu.se/pub/pike/beta/8.0.462/Pike-v8.0.462-Darwin-15.4… https://pike.lysator.liu.se/pub/pike/beta/8.0.462/Pike-v8.0.462-win32-oldli… Hopefully Bill has time to update it with better mac builds before we release it. Changes since last beta: * Debug.Peek renamed Debug.Inspect -- Peter Bortas

1 0

Pike 8.0.460 beta
by Peter Bortas 12 Jul '17

12 Jul '17

Pike 8.0.460 beta candidate: https://pike.lysator.liu.se/pub/pike/beta/8.0.460/Pike-v8.0.460.tar.gz Other builds: https://pike.lysator.liu.se/pub/pike/beta/8.0.460/Pike-v8.0.460-Darwin-15.4… https://pike.lysator.liu.se/pub/pike/beta/8.0.460/Pike-v8.0.460-win32-oldli… This is not a release candidate. There are some changes without CHANGES and I'm not quite comfortable with the namespace the Debug.Peek functionality is living in. Changes since Pike 8.0.438 (release 9) ---------------------------------------------------------------------- New Features ------------ o Concurrent The Concurrent module simplifies asynchronous code by synchronizing events in different ways. As an example the connect() function shown below will respond with a Concurrent.Future object that at some point will represent either a connected socket or a failure. Concurrent.Future connect(string host, int port) { Stdio.File con = Stdio.File(); Concurrent.Promise p = Concurrent.Promise(); if( !con->async_connect(host, port, lambda(int success) { if(success) p->success(con); else p->failure("Failed to connect to "+host+":"+port+"\n"); })) { p->failure("Failed to open socket.\n"); } return p->future(); } The returned future can then be used in various ways. // On success, call make_request(con, query). On failure call // werror(msg). connect(host, port) ->on_failure(werror) ->on_success(make_request, query); // On success, call make_request(con, query1) followed by // make_request(resp, query2), where resp is the return value from // make_reqest. connect(host, port) ->then(make_request, werror, query1) ->then(make_request, werror, query2); // Call bridge_ports(con1, con2) when both connections are // established. Concurrent.all(connect(host1, port1), connect(host2, port2)) ->then(bridge_ports, failure); // Call make_request(con) once either of the connections are // established. Concurrent.race(connect(host1, port1), connect(host2, port2)) ->then(make_requet, query); o Filesystem.Monitor - Changed the polling heuristic. This is intended to reduce poll (and notification) delays in some circumstances. - Improved robostness of adjust_monitor(). - Removed some dead and obsolete code. - Scan accelerated monitors too. Both the Inotify and FSEvents APIs claim that they support notifications on eg network file systems, and while they will succeed in notifying on changes performed by the local host on such file systems, they will not on changes performed by other hosts. To avoid missing such changes these monitors need to be actively scanned too. o Sql.pgsql Sped up BEGIN/COMMIT statements. Preserve the initial error message in case of multiple error messages during the same transaction. Flush out unseen error messages upon connection close to stderr. o Debug.Peek Allows for interactive debugging and live data structure inspection in both single- and multi-threaded programs. Example: In the program you'd like to inspect, insert the following one-liner: Debug.Peek("/tmp/test.pike"); Then start the program and keep it running. Next you create a /tmp/test.pike with the following content: void create() { werror("Only once per modification of test.pike\n"); } int main() { werror("This will run every iteration\n"); werror("By returning 1 here, we disable the stacktrace dumps\n"); return 0; } void destroy() { werror("destroy() runs just as often as create()\n"); } Whenever you edit /tmp/test.pike, it will automatically reload the file. Bug fixes --------- o Search Fixed a race condition when updating the database. o Parser.HTML Fixed a condition where Pike would run out of stack space for large documents. o ADT.Heap Fixed heap corruption when the same object is pushed more than once. It will now be considered as calling adjust(). o Inotify Addressed an issue where the backend might be stuck in pending indefinitely. o FSEvents EventStreamMonitor now works with other backends. o Standards.BSON Fixed incorrect encoding/decoding of Binary data. o Stdio.Buffer Fixed a crash when attempting to create a rewind key on a buffer returned by read_buffer(). o mappings Fixed an off by one error in random(mapping) that randomly caused values of type PIKE_T_FREE from the freelist to be exposed to Pike. o Sql.pgsql Closed a prepared-statement-cache race when the same statement is offered multiple times before it finalises the cache entry. Fix sync errors with the database in case of multiple running statements on a single connection that generate multiple errors. -- Peter Bortas

2 6

module dumping when using buildroot
by H William Wellliver III 06 Jul '17

06 Jul '17

I've noticed that the module dumping portion of the installer fails to work properly when using a buildroot (for packaging purposes). It seems that the call to pike -x dump clears the module path, which causes pike to try to use a module path reflective of the final installed location, which doesn't exist yet. This causes compilation of dump to fail because (at least) Getopt can't be found. I'm not sure why -DFAKEROOT and friends don't avoid this problem. I didn't really dig too far into that aspect of the problem. I can confirm that setting a PIKE_MODULE_PATH containing the built module directories from the pike dist as part of the create_process call that starts dump causes the initial compilation problem to go away. I'm not sure whether dump's module path should be those from the build directory, or from the fakeroot-installed directory. The problem has been replicated on linux and solaris by doing "make install buildroot=/tmp". Any thoughts? Bill

1 0

executable bit set on non-executable files in lib
by H. William Welliver III 05 Jul '17

05 Jul '17

I’ve been working on a pike configuration for pkgsrc, and in the process, pkgsrc has noticed that there are a number of files in lib that are set executable. I know that there are a few files that can be run standalone, but these don’t seem to be in that category. Any objection to me updating those files so that they’re not executable? Examples: Web.pmod/Api.pmod/Twitter.pmod Parser.pmod/LR.pmod/GrammarParser.pmod Any thoughts? Bill

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel July 2017