Pike-devel May 2019

pike-devel@lists.lysator.liu.se

4 participants
17 discussions

Proposal: named threads
by H. William Welliver III 29 May '19

29 May '19

I’ve been thinking about adding support for named threads and would like to get some feedback on my proposed solution. Named threads can be useful when, for example, debugging or printing a thread dump. There are varying levels of OS support for named threads, ranging from none at all (older versions of illumos, and Solaris before 11.3) to being able to name any thread at any time (glibc, some of the BSDs, etc). I’d like to shoot for a level of functionality that should be possible for all cases: - Ability to set a thread’s name at creation time - Ability to get a thread’s name any time - Add thread name to thread dump, if set A brief survey of the API provided by other languages suggests that most seem to accept a thread name following the function to be run. Since the existing Pike thread API accepts varargs after the function, there’s no way to determine when a thread name has been provided. With this in mind, I suggest the opposite: variant protected void create(string thread_name, function f, mixed … args); public string name(); Assuming `()() can have variants, a similar variant would be created in Thread.Thread as well. On systems that don’t support naming threads, a parameter would be added either to the pike thread object itself or to the underlying OS thread’s thread specific data. In this situation, the thread name would be available through pike, but not in gdb, etc (but it wouldn’t have been anyway, so no big loss there). Some questions: Each OS has its own limit on the length of the thread name. What should the behavior be? Fail if the length is too long, truncate it silently, or set the maximum length but store the full length somewhere so Pike code can see the full length? There seems to be some difference in behavior, with Solaris silently truncating and the others indicating errors. Being able to set thread names after creation seems like a possibly useful feature. Darwin permits setting the thread name only on the current thread. So, if there were a desire to be able to set the thread name outside thread creation, this would be a cross-platform limitation if we wanted to maintain consistency. Is that a desirable behavior, and is such a limitation warranted? Or, do we emulate the capability by insert code somewhere that indicates the name has been changed and when the current thread is about to run, change the name at that time? What else have I missed? As always, comments and suggestions are welcome! Bill

3 2

Revamping googleapis bindings; overloading `[] ?
by Stephen R. van den Berg 29 May '19

29 May '19

After some tinkering with the current Google API integration code, I can conclude the following: a. The currently implemented interface in Pike is skimpy at best (it covers less than 1% of the total available interfaces). b. The current implementation uses outdated interface versions (oauth v1 instead of v2, and token interface v1 instead of v4). c. The current implementation still defaults to Google Plus (which has been closed down). So, I was trying my hand at some Google API integration code that makes use of the JSON-discovery documents Google makes available. This way the interface will cover 100% of all available Google API calls, and will always be current (it will then even support alpha and beta interfaces). In doing so, I tried to get rid of one level of indirection, by overloading the `-> and `[] operators. But this still seems to result in something that does not play well with classes we inherit from. Any suggestions here? Or should I simply stick in a member called "resrc" to make the jump to the dynamically loaded resources and methods? P.S. The reason I'm implementing this is because I'm in the process of creating a webcrawler that crawls like Googlebot (which includes Javascript parsing, but with Googlebot rules). And to run this in a way that scales, I need to autoscale a farm of Google Compute Engines up and down, which requires fast Google API access. -- Stephen.

1 3

SSL decoding error regression in Pike 8.1 (works in Pike 8.0)
by Stephen R. van den Berg 24 May '19

24 May '19

The test program below shows an I/O error for Pike 8.1, and it shows Result "{\n" " \"error\": \"invalid_grant\",\n" " \"error_description\": \"Invalid JWT: Token must be a short-lived token (60 minutes) and in a reasonable timeframe. Check your iat and exp values and use a clock with skew to account for clock differences between systems.\"\n" "}" (which is correct) for Pike 8.0. int main(int argc, array(string) argv) { Protocols.HTTP.Query q; q = Protocols.HTTP.do_method("POST", "https://www.googleapis.com/oauth2/v4/token", 0, ([ "user-agent":"Mozilla 4.0 (Pike OAuth2 Client 1.0)", "content-type":"application/x-www-form-urlencoded", ]), 0, "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&" "assertion=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTgyMT" "A3ODIsInNjb3BlIjoiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vYXV0aC9jb21" "wdXRlLnJlYWRvbmx5IiwianRpIjoiYzA3YjM5MDUtNjQ4MC00YjI0LTgyYWMtNzJi" "M2U5MzE3NDE4IiwiaWF0IjoxNTU4MjA3MTgyLCJhdWQiOiJodHRwczovL3d3dy5nb" "29nbGVhcGlzLmNvbS9vYXV0aDIvdjQvdG9rZW4iLCJpc3MiOiJjdWNpYm90QGFuZ2" "VsaWMtaG93bC0yMzYxMjMuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20ifQ.HgT4S-sa" "Cz4hh-7mDdU8LnTLvs1wGCrCzydBiwC7XTFL2EtTsAg0U2HFXr_5kztVhqp6yvAW5" "4xQw5gGSsetSszmdpHEFDwmVmIHfUlvpXeXCLoOx3_Bgphuj4pqigg0Czg2suyl3X" "lpjSNtWC_spJHBtARk8zqXhpx1vnFNU-q1NAMvNRYzXI-aE1od8sOsbxPCFoHDKn-" "pBKs7nNTnNV24BQcYfOCq6Mxiy-ZgYkDZxmM6YbtGastSwcJlGvQt5M9x4BveX-i4" "AqBAMCxZ-d5P_Lc3PqZxSuTVCoxpRGzDGsC6oINEe5tnD08zVFQ1moi-F6aTkaV2W" "m7uODhDmw" ); werror("Result %O\n", q->data()); return 0; } -- Stephen.

2 4

Automatic documentation of functions
by Stephen R. van den Berg 22 May '19

22 May '19

Do we already have some mechanism to get implicit/automatic documentation of function(parameter)s? How about supporting something like this: mixed `->(function(...:mixed), string) Then if there is a sample function: void foo(int a, string b) { } Then using foo->doc would return a mapping or string describing the function and/or its parameters (if known and available). Alternately, one could create a function describe(foo) that does the same. In case of the Google API, it can immediately peruse the discovery information/documentation. For functions that we do not have any information stored/attached, Pike could simply show the function declaration(s). -- Stephen.

2 4

Sample code to access cloud.google.com / google compute engine ?
by Stephen R. van den Berg 19 May '19

19 May '19

Does anyone happen to have some boilerplate code which they can share that uses Pike to get the authorisation tokens required to create and delete google compute engine instances? Basically I need to follow this guide: https://developers.google.com/identity/protocols/OAuth2ServiceAccount Which means that I have a private service key from google, and need to use this to create a JWT, which can then be used to request a token. The token I can then use to call the API. The question is, how do I use Web.Auth to get my hands on the token? The example shown in http://pike.lysator.liu.se/generated/manual/modref/ex/predef_3A_3A/Web/Auth… appears to be geared towards authorisations that require user approval. In my case I only need to call some Google API (server to server), no userdata involved. -- Stephen.

3 3

Pike support for Jupyter interactive computing
by H. William Welliver III 05 May '19

05 May '19

Hi All, I’ve been working on adding support for using Pike language code in Jupyter[1] and thought I’d share some of my progress. Jupyter is an interactive computing environment that’s become quite popular in the education and scientific communities. While Jupyter originated in the Python world and remains heavily python focused, support is available for a fairly wide range of other languages. A “kernel” is the bit of the Jupyter infrastructure that provides support for a given language and with an appropriate kernel implementation, it’s possible to switch between various languages in a fairly seamless manner. A kernel for the pike language has actually existed[2] for a while, using an “IPython” wrapper around hilfe. I wanted to take support some more advanced features so I wrote a native kernel[3] that speaks the juypter kernel protocol. I would consider this code to be proof of concept quality, but I have been using it over the past few months. I’ve learned quite a bit and will probably do a fair amount of restructuring at some point soon. I also had to make some (pretty minor) changes to Hilfe to get everything working. I’ll begin the process of integrating that work back into Pike in the near future. Features: - Code evaluation (the Hilfe we all know and love). - Code auto-completion (ie, hitting tab provides a pop-up of options). - Stdout and stderr are displayed as such in Jupyter. - Support for multiple sessions, a separate pike evaluator is spawned for each one. - Evaluations can return “rich” data, such as HTML or images. This can be used as building blocks for more advanced applications. I’ve also created a demo repository[4] that can be used with juypter-repo2docker to build and launch a Pike-Jupyter session in a docker container. The demo repository includes a “notebook” with some simple demonstrations of the functionality. There is a link in the readme that can be used to spawn a new session in MyBinder.org <http://mybinder.org/>, so you can try it out without any actual work. I like the idea of being able add a little presentation and permanence around my ad-hoc hilfe sessions. Obviously, if you look at some of the Jupyter demos, there’s a lot of missing functionality around fancy presentation of data and UI widgets, but I think that would be fairly simple to add (sadly, it seems the Juypter design pushes that functionality to the Kernel, so every language has to implement all of that). As always, comments, feedback and code contributions are welcome! Bill [1] https://jupyter.org/ [2] https://github.com/kevinior/jupyter-pike-kernel <https://github.com/kevinior/jupyter-pike-kernel>/ [3] https://bitbucket.org/hww3/pike-kernel-jupyter/ [4] https://bitbucket.org/hww3/pike-binder-example/ <https://bitbucket.org/hww3/pike-binder-example/>

1 0

Over-eager type checking?
by ceder (-) Per Cederqvist ＠ Pike (-) developers forum 03 May '19

03 May '19

This seems like type-checking gone wrong: > has_value(({ "a", "b", "c" }), "d"); Compiler Error: 1: Bad argument 2 to has_value. Compiler Error: 1: Expected: string(97..99). Compiler Error: 1: Got : string(100..100). And it is not just an artifact of Hilfe. Attempting to run this program: void main() { write("%d\n", has_value(({ "a", "b", "c" }), "d")); } Gives this error message: /home/ceder/pike-fun/has-value.pike:3:Bad argument 2 to has_value. /home/ceder/pike-fun/has-value.pike:3:Expected: string(97..99). /home/ceder/pike-fun/has-value.pike:3:Got : string(100..100). Pike: Failed to compile script. If I store the array and the string in variables, everything works fine. What I really wanted to do was to read the documentation of has_value(), as I always forget if the haystack is the first or the second argument. And my browser was unresponsive, so I decided to just test it manually. Perhaps I should have used > doc has_value instead, but that seems broken too: Pike v8.0 release 807 running Hilfe v3.5 (Incremental Pike Frontend) > doc has_value Illegal comment "/*! @decl object _static_modules\n" " *!\n" " *! This is an object containing the classes for all static\n" " *! (ie non-dynamic) C-modules.\n" " *!\n" " *! In a typic Pike with support for dynamic modules the contained\n" " *! module classes are:\n" " *! @dl\n" " *! @item @[Builtin]\n" " *! @item @[Gmp]\n" " *! @item @[_Stdio]\n" " *! @item @[_math]\n" " *! @item @[_system]\n" " *! @enddl\n" " *!\n" " *! If the Pike binary lacks support for dynamic modules, all C-modules\n" " *! will show up here.\n" " */" in "_static_modules.Builtin". scope(0,function(object, mixed : int(1bit)) | function(string, int | string : int(1bit)) | function(array((0=mixed)), 0 : int(1bit)) | function(multiset, int : int(1bit)) | function(mapping(mixed:(1=mixed)), 1 : int(1bit))) This is from a freshly compiled Pike from the 8.0 branch, made with "make -j12 tinstall". The has_value issue also exists on the 8.1 branch, but "doc has_value" is slightly better there: > doc has_value scope(0,function(object, mixed : int(1bit)) | function(string, int | string : int(1bit)) | function(array((0=mixed)), 0 : int(1bit)) | function(multiset, int : int(1bit)) | function(mapping(mixed:(1=mixed)), 1 : int(1bit))) While not very readable, I guess it contains the info I needed. :-)

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel May 2019