Pike-devel

pike-devel@lists.lysator.liu.se

1 participants
3045 discussions

Longevity of a struct pike_string str pointer?
by Stephen R. van den Berg 18 Jun '19

18 Jun '19

I'm kneedeep in the Shuffler code again, slightly reworking it, fixing some bugs along the way, but also trying to make sure I don't introduce new ones. The following issue comes up: If I have a struct pike_string with a shared string assigned to it, and I take the value of the str member because I want to access the bytes, how long is this address guaranteed to stay valid? E.g. I see in file.c that we take the str->str value, then do THREADS_ALLOW() and subsequently call fd_write(). Which seems to imply that as long as the pike_string struct we took the address from is still alive, the address of the string it points will not change, even if other pike threads run in the meantime? -- Stephen.

2 1

SEGFAULT in Pike 8.1
by Stephen R. van den Berg 12 Jun '19

12 Jun '19

When I run the following in Pike 8.1, it SEGFAULTs: foo::bar = 1; Gets me: p.pike:4:No inherit or surrounding class foo. Segmentation fault -- Stephen.

2 15

Occasional hiccups during https negotiations with Google
by Stephen R. van den Berg 12 Jun '19

12 Jun '19

Not consistently, but once every 10 tries or so, I get the following backtrace which does not seem to originate in my own code: Can not modify the buffer right now, there are active subbuffers. -:1: _static_modules._Stdio()->Buffer(0 bytes, read=[..-1] data=[0..-1] free=[0.. 5300] allocated (read only))->add("\f\0\0n\3\0\35 \237{å\32B«=B/ûÓ\2\"DmøkFñ 3\35¤\3\37Ò\214+ÜÔ*\4\3\0F0D\2 =Í3]r¿\20CN\222²`ÞG\"Ð«·\23aÆ\r\237{q)Ùß\t\ 206w(\2 {\23$Ò\4vì\b\227ú\31n\24xÈ¢M\v8d F\1/z\211\vÈx¹ï\222") /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:1051: SSL.ClientConnection(handshaking)->got_data("\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\ 17êlU:b5ñwÖäO»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3 \r8\v\0\r4\0\r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½:O5Øj\21ö\rh0\r\ 6\t*\206H\206÷\r\1\1\v\5\u00000T1\v0\t\6\3U\4\6\23\2US1\u001e0\34\6\3"+[3389 ]+"\206w(\2 {\23$Ò\4vì\b\227ú\31n\24xÈ¢M\v8d F\1/z\211\vÈx¹ï\222\26\3\3\0\4\ 16\0\0\0") /usr/local/pike/8.1.13/lib/modules/SSL.pmod/File.pike:1985: SSL.File(Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */), SSL.Cl ientConnection(handshaking))->ssl_read_callback(Stdio.File("socket", "172.21 7.20.106:443", 777 /* fd=43 */),"\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\17êlU:b5ñwÖä O»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3\r8\v\0\r4\0 \r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½"+[3448]+"ú\31n\24xÈ¢M\v8d F \1/z\211\vÈx¹ï\222\26\3\3\0\4\16\0\0\0") /usr/local/pike/8.1.13/lib/modules/Stdio.pmod/module.pmod:1165: Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */)->__stdio_read_ca llback() -:1: Pike.Backend(0)->`()(3600.0) Concurrent.Promise(pending,0): Promise broken. /usr/local/pike/8.1.13/lib/modules/Concurrent.pmod:1176: unknown function() -:1: _static_modules._Stdio()->Buffer(0 bytes, read=[..1279] data=[1280..1279] fr ee=[1280..0] subbuffer)->read_int8() /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:68: Standards.ASN1.Decode.read_identifier(_static_modules._Stdio()->Buffer(0 byt es, read=[..1279] data=[1280..1279] free=[1280..0] subbuffer)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:100: Standards.ASN1.Decode.der_decode((a)0=_static_modules._Stdio()->Buffer(0 bytes , read=[..1279] data=[1280..1279] free=[1280..0] subbuffer),mapping[23]) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:157: Standards.ASN1.Decode.der_decode(@0,@1=mapping[23]) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:245: Standards.ASN1.Decode.secure_der_decode("0\202\5\0\202\20*.googleapis.com\20 2\24*.clients6.google.ae\202\24*.clients6.google.at\202\24*.clients6.google. be\202\24*.clients6.google.ca\202\24*.clients6.google.ch\202\24*.clients6"+[ 1094]+"is.com\202\26cloudendpointsapis.com\202\16googleapis.com",@1) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:703: Standards.X509.TBSCertificate(([ /* 5 elements */ "algorithm": Standards.ASN1.Types.Sequence(({ /* 2 elements */ Standards.ASN1.Types.Identifier(1.2.840.113549.1.1.11), Standards.ASN1.Types.Null() })), "issuer": ([ /* 3 elements */ "commonName": "Google Internet Authority G3", "countryName": "US", "organizationName": "Google Trust Services" ]), "serial": Gmp.mpz(76838582857112318721873235160560176488), "subject": ([ /* 5 elements */ "commonName": "*.googleapis.com", "countryName": "US", "localityName": "Mountain View", "organizationName": "Google LLC", "stateOrProvinceName": "California" ]), "version": 3 ]))->`raw_extensions=(Standards.ASN1.Types.Sequence(...)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:938: Standards.X509.TBSCertificate(([ /* 5 elements */ "algorithm": Standards.ASN1.Types.Sequence(({ /* 2 elements */ Standards.ASN1.Types.Identifier(1.2.840.113549.1.1.11), Standards.ASN1.Types.Null() })), "issuer": ([ /* 3 elements */ "commonName": "Google Internet Authority G3", "countryName": "US", "organizationName": "Google Trust Services" ]), "serial": Gmp.mpz(76838582857112318721873235160560176488), "subject": ([ /* 5 elements */ "commonName": "*.googleapis.com", "countryName": "US", "localityName": "Mountain View", "organizationName": "Google LLC", "stateOrProvinceName": "California" ]), "version": 3 ]))->init(Standards.ASN1.Types.Sequence(...)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:763: Standards.X509.TBSCertificate(([ /* 5 elements */ "algorithm": Standards.ASN1.Types.Sequence(({ /* 2 elements */ Standards.ASN1.Types.Identifier(1.2.840.113549.1.1.11), Standards.ASN1.Types.Null() })), "issuer": ([ /* 3 elements */ "commonName": "Google Internet Authority G3", "countryName": "US", "organizationName": "Google Trust Services" ]), "serial": Gmp.mpz(76838582857112318721873235160560176488), "subject": ([ /* 5 elements */ "commonName": "*.googleapis.com", "countryName": "US", "localityName": "Mountain View", "organizationName": "Google LLC", "stateOrProvinceName": "California" ]), "version": 3 ]))->create(@2=Standards.ASN1.Types.Sequence(...)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:329: Standards.X509.TBSCertificate() /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:1448: Standards.X509.decode_certificate(Standards.PKCS.Signature.Signed(...)) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:283: SSL.ClientConnection(handshaking)->verify_certificate_chain(({"0\202\bÇ0\202 \a¯ \3\2\1\2\2\209Î\222 tü½:O5Øj\21ö\rh0\r\6\t*\206H\206÷\r\1\1\v\5\u00000T1 \v0\t\6\3U\4\6\23\2US1\u001e0\34\6\3U\4\n"+[2160]+"É\vèÍ|ÂPD\201\220.R#Z³\23 7iG(#ç","0\202\4\\0\202\3D \3\2\1\2\2\r\1ã©0\34ür\68?\232S\u001d0\r\6\t*\206 H\206÷\r\1\1\v\5\u00000L1 0\36\6\3U\4\v\23\27GlobalSign Roo"+[1029]+"\234@èÖ Usº<ß\220Ë\0[!\21gÂí2\36Þ"})) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:372: SSL.ClientConnection(handshaking)->handle_certificates(SSL.Buffer(0 bytes, r ead=[..3379] data=[3380..3379] free=[3380..3480] allocated)) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/ClientConnection.pike:876: SSL.ClientConnection(handshaking)->handle_handshake(11,(a)3=SSL.Buffer(0 bytes , read=[..3379] data=[3380..3379] free=[3380..3480] allocated),_static_modul es._Stdio()->Buffer(3384 bytes, read=[..-1] data=[0..3383] free=[3384..0] su bbuffer)) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:1070: SSL.ClientConnection(handshaking)->got_data("\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\ 17êlU:b5ñwÖäO»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3 \r8\v\0\r4\0\r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½:O5Øj\21ö\rh0\r\ 6\t*\206H\206÷\r\1\1\v\5\u00000T1\v0\t\6\3U\4\6\23\2US1\u001e0\34\6\3"+[3389 ]+"\206w(\2 {\23$Ò\4vì\b\227ú\31n\24xÈ¢M\v8d F\1/z\211\vÈx¹ï\222\26\3\3\0\4\ 16\0\0\0") /usr/local/pike/8.1.13/lib/modules/SSL.pmod/File.pike:1985: SSL.File(Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */), SSL.Cl ientConnection(handshaking))->ssl_read_callback(Stdio.File("socket", "172.21 7.20.106:443", 777 /* fd=43 */),"\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\17êlU:b5ñwÖä O»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3\r8\v\0\r4\0 \r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½"+[3448]+"ú\31n\24xÈ¢M\v8d F \1/z\211\vÈx¹ï\222\26\3\3\0\4\16\0\0\0") /usr/local/pike/8.1.13/lib/modules/Stdio.pmod/module.pmod:1165: Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */)->__stdio_read_ca llback() -:1: Pike.Backend(0)->`()(3600.0) -- Stephen.

2 4

Scope runtime bug in Pike 8.0 and 8.1
by Stephen R. van den Berg 08 Jun '19

08 Jun '19

-------------------------------------------------------- class foo { #if 1 protected int bar; // Does not work (and is a bug) #else int bar; // Does work #endif this_program other; protected void create() { other = this; other->bar = 1; } } object x = foo(); -------------------------------------------------------- This gives a runtime error which seems incorrect to me. Flipping the #if causes it to run flawlessly. Is this a problem in Pike, or are my expectations too high? -- Stephen.

2 2

Proposal: named threads
by H. William Welliver III 29 May '19

29 May '19

I’ve been thinking about adding support for named threads and would like to get some feedback on my proposed solution. Named threads can be useful when, for example, debugging or printing a thread dump. There are varying levels of OS support for named threads, ranging from none at all (older versions of illumos, and Solaris before 11.3) to being able to name any thread at any time (glibc, some of the BSDs, etc). I’d like to shoot for a level of functionality that should be possible for all cases: - Ability to set a thread’s name at creation time - Ability to get a thread’s name any time - Add thread name to thread dump, if set A brief survey of the API provided by other languages suggests that most seem to accept a thread name following the function to be run. Since the existing Pike thread API accepts varargs after the function, there’s no way to determine when a thread name has been provided. With this in mind, I suggest the opposite: variant protected void create(string thread_name, function f, mixed … args); public string name(); Assuming `()() can have variants, a similar variant would be created in Thread.Thread as well. On systems that don’t support naming threads, a parameter would be added either to the pike thread object itself or to the underlying OS thread’s thread specific data. In this situation, the thread name would be available through pike, but not in gdb, etc (but it wouldn’t have been anyway, so no big loss there). Some questions: Each OS has its own limit on the length of the thread name. What should the behavior be? Fail if the length is too long, truncate it silently, or set the maximum length but store the full length somewhere so Pike code can see the full length? There seems to be some difference in behavior, with Solaris silently truncating and the others indicating errors. Being able to set thread names after creation seems like a possibly useful feature. Darwin permits setting the thread name only on the current thread. So, if there were a desire to be able to set the thread name outside thread creation, this would be a cross-platform limitation if we wanted to maintain consistency. Is that a desirable behavior, and is such a limitation warranted? Or, do we emulate the capability by insert code somewhere that indicates the name has been changed and when the current thread is about to run, change the name at that time? What else have I missed? As always, comments and suggestions are welcome! Bill

3 2

Revamping googleapis bindings; overloading `[] ?
by Stephen R. van den Berg 29 May '19

29 May '19

After some tinkering with the current Google API integration code, I can conclude the following: a. The currently implemented interface in Pike is skimpy at best (it covers less than 1% of the total available interfaces). b. The current implementation uses outdated interface versions (oauth v1 instead of v2, and token interface v1 instead of v4). c. The current implementation still defaults to Google Plus (which has been closed down). So, I was trying my hand at some Google API integration code that makes use of the JSON-discovery documents Google makes available. This way the interface will cover 100% of all available Google API calls, and will always be current (it will then even support alpha and beta interfaces). In doing so, I tried to get rid of one level of indirection, by overloading the `-> and `[] operators. But this still seems to result in something that does not play well with classes we inherit from. Any suggestions here? Or should I simply stick in a member called "resrc" to make the jump to the dynamically loaded resources and methods? P.S. The reason I'm implementing this is because I'm in the process of creating a webcrawler that crawls like Googlebot (which includes Javascript parsing, but with Googlebot rules). And to run this in a way that scales, I need to autoscale a farm of Google Compute Engines up and down, which requires fast Google API access. -- Stephen.

1 3

SSL decoding error regression in Pike 8.1 (works in Pike 8.0)
by Stephen R. van den Berg 24 May '19

24 May '19

The test program below shows an I/O error for Pike 8.1, and it shows Result "{\n" " \"error\": \"invalid_grant\",\n" " \"error_description\": \"Invalid JWT: Token must be a short-lived token (60 minutes) and in a reasonable timeframe. Check your iat and exp values and use a clock with skew to account for clock differences between systems.\"\n" "}" (which is correct) for Pike 8.0. int main(int argc, array(string) argv) { Protocols.HTTP.Query q; q = Protocols.HTTP.do_method("POST", "https://www.googleapis.com/oauth2/v4/token", 0, ([ "user-agent":"Mozilla 4.0 (Pike OAuth2 Client 1.0)", "content-type":"application/x-www-form-urlencoded", ]), 0, "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&" "assertion=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NTgyMT" "A3ODIsInNjb3BlIjoiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vYXV0aC9jb21" "wdXRlLnJlYWRvbmx5IiwianRpIjoiYzA3YjM5MDUtNjQ4MC00YjI0LTgyYWMtNzJi" "M2U5MzE3NDE4IiwiaWF0IjoxNTU4MjA3MTgyLCJhdWQiOiJodHRwczovL3d3dy5nb" "29nbGVhcGlzLmNvbS9vYXV0aDIvdjQvdG9rZW4iLCJpc3MiOiJjdWNpYm90QGFuZ2" "VsaWMtaG93bC0yMzYxMjMuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20ifQ.HgT4S-sa" "Cz4hh-7mDdU8LnTLvs1wGCrCzydBiwC7XTFL2EtTsAg0U2HFXr_5kztVhqp6yvAW5" "4xQw5gGSsetSszmdpHEFDwmVmIHfUlvpXeXCLoOx3_Bgphuj4pqigg0Czg2suyl3X" "lpjSNtWC_spJHBtARk8zqXhpx1vnFNU-q1NAMvNRYzXI-aE1od8sOsbxPCFoHDKn-" "pBKs7nNTnNV24BQcYfOCq6Mxiy-ZgYkDZxmM6YbtGastSwcJlGvQt5M9x4BveX-i4" "AqBAMCxZ-d5P_Lc3PqZxSuTVCoxpRGzDGsC6oINEe5tnD08zVFQ1moi-F6aTkaV2W" "m7uODhDmw" ); werror("Result %O\n", q->data()); return 0; } -- Stephen.

2 4

Automatic documentation of functions
by Stephen R. van den Berg 21 May '19

21 May '19

Do we already have some mechanism to get implicit/automatic documentation of function(parameter)s? How about supporting something like this: mixed `->(function(...:mixed), string) Then if there is a sample function: void foo(int a, string b) { } Then using foo->doc would return a mapping or string describing the function and/or its parameters (if known and available). Alternately, one could create a function describe(foo) that does the same. In case of the Google API, it can immediately peruse the discovery information/documentation. For functions that we do not have any information stored/attached, Pike could simply show the function declaration(s). -- Stephen.

2 4

Sample code to access cloud.google.com / google compute engine ?
by Stephen R. van den Berg 19 May '19

19 May '19

Does anyone happen to have some boilerplate code which they can share that uses Pike to get the authorisation tokens required to create and delete google compute engine instances? Basically I need to follow this guide: https://developers.google.com/identity/protocols/OAuth2ServiceAccount Which means that I have a private service key from google, and need to use this to create a JWT, which can then be used to request a token. The token I can then use to call the API. The question is, how do I use Web.Auth to get my hands on the token? The example shown in http://pike.lysator.liu.se/generated/manual/modref/ex/predef_3A_3A/Web/Auth… appears to be geared towards authorisations that require user approval. In my case I only need to call some Google API (server to server), no userdata involved. -- Stephen.

3 3

Pike support for Jupyter interactive computing
by H. William Welliver III 05 May '19

05 May '19

Hi All, I’ve been working on adding support for using Pike language code in Jupyter[1] and thought I’d share some of my progress. Jupyter is an interactive computing environment that’s become quite popular in the education and scientific communities. While Jupyter originated in the Python world and remains heavily python focused, support is available for a fairly wide range of other languages. A “kernel” is the bit of the Jupyter infrastructure that provides support for a given language and with an appropriate kernel implementation, it’s possible to switch between various languages in a fairly seamless manner. A kernel for the pike language has actually existed[2] for a while, using an “IPython” wrapper around hilfe. I wanted to take support some more advanced features so I wrote a native kernel[3] that speaks the juypter kernel protocol. I would consider this code to be proof of concept quality, but I have been using it over the past few months. I’ve learned quite a bit and will probably do a fair amount of restructuring at some point soon. I also had to make some (pretty minor) changes to Hilfe to get everything working. I’ll begin the process of integrating that work back into Pike in the near future. Features: - Code evaluation (the Hilfe we all know and love). - Code auto-completion (ie, hitting tab provides a pop-up of options). - Stdout and stderr are displayed as such in Jupyter. - Support for multiple sessions, a separate pike evaluator is spawned for each one. - Evaluations can return “rich” data, such as HTML or images. This can be used as building blocks for more advanced applications. I’ve also created a demo repository[4] that can be used with juypter-repo2docker to build and launch a Pike-Jupyter session in a docker container. The demo repository includes a “notebook” with some simple demonstrations of the functionality. There is a link in the readme that can be used to spawn a new session in MyBinder.org <http://mybinder.org/>, so you can try it out without any actual work. I like the idea of being able add a little presentation and permanence around my ad-hoc hilfe sessions. Obviously, if you look at some of the Jupyter demos, there’s a lot of missing functionality around fancy presentation of data and UI widgets, but I think that would be fairly simple to add (sadly, it seems the Juypter design pushes that functionality to the Kernel, so every language has to implement all of that). As always, comments, feedback and code contributions are welcome! Bill [1] https://jupyter.org/ [2] https://github.com/kevinior/jupyter-pike-kernel <https://github.com/kevinior/jupyter-pike-kernel>/ [3] https://bitbucket.org/hww3/pike-kernel-jupyter/ [4] https://bitbucket.org/hww3/pike-binder-example/ <https://bitbucket.org/hww3/pike-binder-example/>

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel