Pike-devel June 2019

pike-devel@lists.lysator.liu.se

6 participants
21 discussions

Re: Hyperscan regexp engine
by Tomasz Jamroszczak 28 Jun '19

28 Jun '19

On Thu, 27 Jun 2019 15:59:01 +0200, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > So I interpret this as "ure3" (whatever that is) being the thing we > actually want, not this garbage fire? On https://branchfree.org/2019/02/28/paper-hyperscan-a-fast-multi-pattern-rege… the author in the commets writes on March 27, 2019: """ I would like “ure3” to be open. I also will need help. I designed Hyperscan but did not build it all by myself; [...] If I build a new regex matching system it will be a lot smaller and simpler, but it will still be a huge effort. """ And on April 15, 2019: """ if I get to it. """ So well, given the fact that the "ure3" doesn't exist, I doubt it is something we should or even could want. br, tj.

2 1

Re: Hyperscan regexp engine
by Tomasz Jamroszczak 28 Jun '19

28 Jun '19

On Thu, 27 Jun 2019 13:58:02 +0200, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: >> /paper-hyperscan-a-fast-multi-pattern-regex-matcher-for-modern-cpus/ > > For modern CPU:s? Looks more like they are targeting a certain 1970:s > architecture: > > cpuid(1, 0, &eax, &ebx, &ecx, &edx); > > Did you test it on RISC-V? From https://news.ycombinator.com/item?id=19270199: """ I suppose "Some Modern CPUs" was too long-winded a title? As I said, it doesn't take a genius to understand Intel's motivations. They bought the project, after all. Not being @ Intel anymore, I don't have access to the older stuff, and even if I did, the codebase has diverged significantly since. Without going on too much of a tirade - the experience of developing for all those platforms really sucked. Almost all the non-x86 platforms had significant bugs in their toolchains. One of the MIPS variants (particular architecture elided to spare the guilty) had bugs in their gcc intrinsics in a way that suggested that no-one had ever done any significant third-party dev on the platform. Big-endian was also a huge PITA. It was a ton of work to keep all those systems alive, and our machine rack looked like a zoo of dev boards and weirdo devices. In the "ure3" system I mention, I would make retargetability/portability to other systems a first-class goal. One way of achieving this is not having such a huge profusion of methods and complexity. Hyperscan is over-engineered for many use cases if you aren't a network company looking to scan 5,000 complex regexes in streaming mode at hopefully maximal performance. """ > Also: 114 kLOC for a regexp matcher. Srsly? For a 40x speed boost? Worth at least trying. BR, tj

2 2

Hyperscan regexp engine
by Tomasz Jamroszczak 27 Jun '19

27 Jun '19

There's the https://github.com/intel/hyperscan regexp library created over 10 years by algorithm start-up https://branchfree.org/2019/02/28/paper-hyperscan-a-fast-multi-pattern-rege… then bought by Intel and further developed. The development did everything the right way and the lib boasts a lot of optimizations and speedup. Is there a plan to import the lib to Pike? Would it be easy and straight-forward to do so? Best Regards, Tomasz Jamroszczak -- Using Opera's mail client: http://www.opera.com/mail/

3 4

Shuffler.Shuffler, does anyone use it?
by Stephen R. van den Berg 26 Jun '19

26 Jun '19

I though Roxen might be using it, but maybe I remember it wrong. Anyone else? I ask because I'm changing Protocols.HTTP.Server.Request to use the Shuffler, and I'm close to done, but it appears that in the process I have had to fix some quite serious bugs in the current shuffler implementation which would/should have prevented it from working correctly in a lot of normal use cases already. I was then wondering if it is being used successfully anywhere already? -- Stephen.

5 17

set_read_callback()
by Stephen R. van den Berg 26 Jun '19

26 Jun '19

Does anyone know where it is defined in the source? I'm having issues in the Shuffler (which have been there for ages already) where a C-struct is being freed, but then after that, a callback is being called from pike which still references the (now freed) C-struct. The freeing of the C-struct needs to be delayed until all references to the callback function are gone (which probably implies that I need to turn the struct over to the garbage collector, which implies that it needs to be an svalue, I presume). If anyone has better ideas for this, I'm all ears. -- Stephen.

2 2

close callback on FakeFile
by Stephen R. van den Berg 26 Jun '19

26 Jun '19

FakeFile is supposed to emulate a regular file. This would mean that whenever you reach the end of the file/string you are reading, you are supposed to signal a close callback because eof has been reached. I.e. the close callback on FakeFile should therefore not need to wait for an actual close from the writer. Reaching the end of the string (=file) suffices. All this contrasts with FakePipe where this obviously not hold true and you have to wait for an explicit close from the writer side. -- Stephen.

1 0

MozJPEG encoding library
by Stephen R. van den Berg 20 Jun '19

20 Jun '19

I was thinking about writing glue to integrate the MozJPEG encoding library with Pike. I already figured it should live in src/post_modules/_Image_MozJPEG. The question remains what to do about the library itself? - The library itself is a JPEG encoding-only library. - It usually is not system provided (because it's encoding only). - The library itself is available at https://github.com/mozilla/mozjpeg - The library is about 1.5MB xz compressed and about 9MB uncompressed. The options are: a. Make it pluggable, so that if you manually provide the library source tree in the correct src/post_modules/_Image_MozJPEG/mozjpeg location it will automatically detect and compile it in. b. Make it fetch the library using git during the configure run, and then jump to option a. c. Include the library in the Pike source tar. d. Create a package for that library that I submit to all (Linux) distributions. (Not really an option, too much work). I'd think that option b) might be the most prudent one. What do you think? -- Stephen.

2 3

Longevity of a struct pike_string str pointer?
by Stephen R. van den Berg 18 Jun '19

18 Jun '19

I'm kneedeep in the Shuffler code again, slightly reworking it, fixing some bugs along the way, but also trying to make sure I don't introduce new ones. The following issue comes up: If I have a struct pike_string with a shared string assigned to it, and I take the value of the str member because I want to access the bytes, how long is this address guaranteed to stay valid? E.g. I see in file.c that we take the str->str value, then do THREADS_ALLOW() and subsequently call fd_write(). Which seems to imply that as long as the pike_string struct we took the address from is still alive, the address of the string it points will not change, even if other pike threads run in the meantime? -- Stephen.

2 1

SEGFAULT in Pike 8.1
by Stephen R. van den Berg 13 Jun '19

13 Jun '19

When I run the following in Pike 8.1, it SEGFAULTs: foo::bar = 1; Gets me: p.pike:4:No inherit or surrounding class foo. Segmentation fault -- Stephen.

2 15

Occasional hiccups during https negotiations with Google
by Stephen R. van den Berg 13 Jun '19

13 Jun '19

Not consistently, but once every 10 tries or so, I get the following backtrace which does not seem to originate in my own code: Can not modify the buffer right now, there are active subbuffers. -:1: _static_modules._Stdio()->Buffer(0 bytes, read=[..-1] data=[0..-1] free=[0.. 5300] allocated (read only))->add("\f\0\0n\3\0\35 \237{å\32B«=B/ûÓ\2\"DmøkFñ 3\35¤\3\37Ò\214+ÜÔ*\4\3\0F0D\2 =Í3]r¿\20CN\222²`ÞG\"Ð«·\23aÆ\r\237{q)Ùß\t\ 206w(\2 {\23$Ò\4vì\b\227ú\31n\24xÈ¢M\v8d F\1/z\211\vÈx¹ï\222") /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:1051: SSL.ClientConnection(handshaking)->got_data("\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\ 17êlU:b5ñwÖäO»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3 \r8\v\0\r4\0\r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½:O5Øj\21ö\rh0\r\ 6\t*\206H\206÷\r\1\1\v\5\u00000T1\v0\t\6\3U\4\6\23\2US1\u001e0\34\6\3"+[3389 ]+"\206w(\2 {\23$Ò\4vì\b\227ú\31n\24xÈ¢M\v8d F\1/z\211\vÈx¹ï\222\26\3\3\0\4\ 16\0\0\0") /usr/local/pike/8.1.13/lib/modules/SSL.pmod/File.pike:1985: SSL.File(Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */), SSL.Cl ientConnection(handshaking))->ssl_read_callback(Stdio.File("socket", "172.21 7.20.106:443", 777 /* fd=43 */),"\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\17êlU:b5ñwÖä O»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3\r8\v\0\r4\0 \r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½"+[3448]+"ú\31n\24xÈ¢M\v8d F \1/z\211\vÈx¹ï\222\26\3\3\0\4\16\0\0\0") /usr/local/pike/8.1.13/lib/modules/Stdio.pmod/module.pmod:1165: Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */)->__stdio_read_ca llback() -:1: Pike.Backend(0)->`()(3600.0) Concurrent.Promise(pending,0): Promise broken. /usr/local/pike/8.1.13/lib/modules/Concurrent.pmod:1176: unknown function() -:1: _static_modules._Stdio()->Buffer(0 bytes, read=[..1279] data=[1280..1279] fr ee=[1280..0] subbuffer)->read_int8() /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:68: Standards.ASN1.Decode.read_identifier(_static_modules._Stdio()->Buffer(0 byt es, read=[..1279] data=[1280..1279] free=[1280..0] subbuffer)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:100: Standards.ASN1.Decode.der_decode((a)0=_static_modules._Stdio()->Buffer(0 bytes , read=[..1279] data=[1280..1279] free=[1280..0] subbuffer),mapping[23]) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:157: Standards.ASN1.Decode.der_decode(@0,@1=mapping[23]) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/ASN1.pmod/Decode.pmod:245: Standards.ASN1.Decode.secure_der_decode("0\202\5\0\202\20*.googleapis.com\20 2\24*.clients6.google.ae\202\24*.clients6.google.at\202\24*.clients6.google. be\202\24*.clients6.google.ca\202\24*.clients6.google.ch\202\24*.clients6"+[ 1094]+"is.com\202\26cloudendpointsapis.com\202\16googleapis.com",@1) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:703: Standards.X509.TBSCertificate(([ /* 5 elements */ "algorithm": Standards.ASN1.Types.Sequence(({ /* 2 elements */ Standards.ASN1.Types.Identifier(1.2.840.113549.1.1.11), Standards.ASN1.Types.Null() })), "issuer": ([ /* 3 elements */ "commonName": "Google Internet Authority G3", "countryName": "US", "organizationName": "Google Trust Services" ]), "serial": Gmp.mpz(76838582857112318721873235160560176488), "subject": ([ /* 5 elements */ "commonName": "*.googleapis.com", "countryName": "US", "localityName": "Mountain View", "organizationName": "Google LLC", "stateOrProvinceName": "California" ]), "version": 3 ]))->`raw_extensions=(Standards.ASN1.Types.Sequence(...)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:938: Standards.X509.TBSCertificate(([ /* 5 elements */ "algorithm": Standards.ASN1.Types.Sequence(({ /* 2 elements */ Standards.ASN1.Types.Identifier(1.2.840.113549.1.1.11), Standards.ASN1.Types.Null() })), "issuer": ([ /* 3 elements */ "commonName": "Google Internet Authority G3", "countryName": "US", "organizationName": "Google Trust Services" ]), "serial": Gmp.mpz(76838582857112318721873235160560176488), "subject": ([ /* 5 elements */ "commonName": "*.googleapis.com", "countryName": "US", "localityName": "Mountain View", "organizationName": "Google LLC", "stateOrProvinceName": "California" ]), "version": 3 ]))->init(Standards.ASN1.Types.Sequence(...)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:763: Standards.X509.TBSCertificate(([ /* 5 elements */ "algorithm": Standards.ASN1.Types.Sequence(({ /* 2 elements */ Standards.ASN1.Types.Identifier(1.2.840.113549.1.1.11), Standards.ASN1.Types.Null() })), "issuer": ([ /* 3 elements */ "commonName": "Google Internet Authority G3", "countryName": "US", "organizationName": "Google Trust Services" ]), "serial": Gmp.mpz(76838582857112318721873235160560176488), "subject": ([ /* 5 elements */ "commonName": "*.googleapis.com", "countryName": "US", "localityName": "Mountain View", "organizationName": "Google LLC", "stateOrProvinceName": "California" ]), "version": 3 ]))->create(@2=Standards.ASN1.Types.Sequence(...)) /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:329: Standards.X509.TBSCertificate() /usr/local/pike/8.1.13/lib/modules/Standards.pmod/X509.pmod:1448: Standards.X509.decode_certificate(Standards.PKCS.Signature.Signed(...)) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:283: SSL.ClientConnection(handshaking)->verify_certificate_chain(({"0\202\bÇ0\202 \a¯ \3\2\1\2\2\209Î\222 tü½:O5Øj\21ö\rh0\r\6\t*\206H\206÷\r\1\1\v\5\u00000T1 \v0\t\6\3U\4\6\23\2US1\u001e0\34\6\3U\4\n"+[2160]+"É\vèÍ|ÂPD\201\220.R#Z³\23 7iG(#ç","0\202\4\\0\202\3D \3\2\1\2\2\r\1ã©0\34ür\68?\232S\u001d0\r\6\t*\206 H\206÷\r\1\1\v\5\u00000L1 0\36\6\3U\4\v\23\27GlobalSign Roo"+[1029]+"\234@èÖ Usº<ß\220Ë\0[!\21gÂí2\36Þ"})) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:372: SSL.ClientConnection(handshaking)->handle_certificates(SSL.Buffer(0 bytes, r ead=[..3379] data=[3380..3379] free=[3380..3480] allocated)) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/ClientConnection.pike:876: SSL.ClientConnection(handshaking)->handle_handshake(11,(a)3=SSL.Buffer(0 bytes , read=[..3379] data=[3380..3379] free=[3380..3480] allocated),_static_modul es._Stdio()->Buffer(3384 bytes, read=[..-1] data=[0..3383] free=[3384..0] su bbuffer)) /usr/local/pike/8.1.13/lib/modules/SSL.pmod/Connection.pike:1070: SSL.ClientConnection(handshaking)->got_data("\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\ 17êlU:b5ñwÖäO»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3 \r8\v\0\r4\0\r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½:O5Øj\21ö\rh0\r\ 6\t*\206H\206÷\r\1\1\v\5\u00000T1\v0\t\6\3U\4\6\23\2US1\u001e0\34\6\3"+[3389 ]+"\206w(\2 {\23$Ò\4vì\b\227ú\31n\24xÈ¢M\v8d F\1/z\211\vÈx¹ï\222\26\3\3\0\4\ 16\0\0\0") /usr/local/pike/8.1.13/lib/modules/SSL.pmod/File.pike:1985: SSL.File(Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */), SSL.Cl ientConnection(handshaking))->ssl_read_callback(Stdio.File("socket", "172.21 7.20.106:443", 777 /* fd=43 */),"\26\3\3\0;\2\0\u00007\3\3\\ÿP¹\17êlU:b5ñwÖä O»àz£eæ(§DOWNGRD\1\0À+\0\0\17ÿ\1\0\1\0\0\v\0\2\1\0\0#\0\0\26\3\3\r8\v\0\r4\0 \r1\0\bË0\202\bÇ0\202\a¯ \3\2\1\2\2\209Î\222 tü½"+[3448]+"ú\31n\24xÈ¢M\v8d F \1/z\211\vÈx¹ï\222\26\3\3\0\4\16\0\0\0") /usr/local/pike/8.1.13/lib/modules/Stdio.pmod/module.pmod:1165: Stdio.File("socket", "172.217.20.106:443", 777 /* fd=43 */)->__stdio_read_ca llback() -:1: Pike.Backend(0)->`()(3600.0) -- Stephen.

2 4

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel June 2019