I'm taking up an old tradition that haven't been practiced since 2008 as far as I can tell: Making releases on new years eve. Bill has been busy with other things, so I'll jump in to my old role as release manager for now.

Pike 8.0.146 beta for your testing pleasures:

https://pike.lysator.liu.se/pub/pike/beta/8.0.146/Pike-v8.0.146.tar.gz

Pike 8: Changes since Pike 7.8 ----------------------------------------------------------------------

New language features ---------------------

o Added a way to access the local, and not the overloaded, implementation of a symbol.

As an example, given the classes: | class Test | { | int a( ) { return 1; } | int b( ) { return local::a(); } // New | int c( ) { return a(); } | } | | class Test2 | { | int a() { return 42; } | }

Both Test()->b() and Test2()->b() will return 1, but Test2()->a() and Test2()->c() will return 42.

o Added new syntax that can be used to return the current object as if it was a class it is inheriting

The syntax is X::this, where X is the inherited class.

The returned object will not include any symbols not available in the class X, but your overloaded methods (and other identifiers) will be the ones returned when the object is indexed.

o Added a way to easily inherit the implementation of a subclass in an inheriting class.

| inherit ::this_program

The syntax refers to the previous definition of the current class in the inherited class, and is typically used with inherit like:

| inherit Foo; | | // Override the Bar inherited from Foo. | class Bar { | // The new Bar is based on the implementation of Bar in Foo. | inherit ::this_program; | // ... | }

o Added new syntax for referring to and naming inherits.

| inherit "foo.pike"; | inherit "foo/bar/baz.pike" : "foo/bar"; | // ... | "foo.pike"::foo(); | "foo/bar"::baz();

o Look up of named inherits now also looks in indirect inherits.

| class A { int fun() {} } | class B { inherit A; } | class C { | inherit B; | // ... | // The function fun may here be referred to as any of: | B::A::fun(); | B::fun(); | A::fun(); // New.

o Implemented the variant keyword.

The keyword was present before, but did not actually do anything useful.

This is used to do function overloading based on the argument types.

As an example: | class Test1 { int b(); } | class Test2 { int a(); } | | class Foo | { | variant string foo( object q ){ return "Genericfoo"; } | variant string foo( Test1 x ) { return "Test 1"; } | variant string foo( Test2 x ) { return "Test 2"; } | | variant string bar( int(0..0) s ) { return "0"; } | variant float bar( int(1..1) s ) { return 1.0; } | variant int bar( int(2..2) s ) { return 2; } | }

A common use-case is functions that return different types depending on the arguments, such as getenv:

| string|mapping(string:string) getenv( string|void variable );

can now be rewritten as

| variant string getenv( string variable ); | variant mapping(string:string) getenv( );

which gives significantly better type-checking.

o The type-checker has been rewritten.

Mainly it is now much better at finding type errors, and most error messages are more readable.

o Allow '.' to be used as an index operator in the few places it did not work before.

o "Safe" index

Copied from a popular extension to other C-like languages.

X[?ind] is equivalent to ((auto _ = X),(_ && _[ind])) while X?->ind is equivalent to ((auto _ = X),(_ && _->ind))

The main use is 'deep indexing' where some elements can be 0:

| request_id?->user?->misc?->x

vs

| request_id && request_id->user && request_id->user->misc | && request_id->user->misc->x

o Added the '?:' operator for compatibility with other C-like languages. It is identical to '||' in Pike.

o The && operator changed, when doing A && B, and A is false, keep A instead of returning 0.

This is especially useful then A is not actually 0, but an object that evaluates to false, or UNDEFINED.

o Fixed symbol resolution with deep inherits and mixins.

o Added multi-character character constants.

'ILBM' is equivalent to (('I'<<24)|('L'<<16)|('B'<<8)|'M').

Unlike how it works in some C compilers the native byte order is never relevant.

o Added new syntax for literal-string constants

#{, #( and #[ starts a literal string, and it is ended by the corresponding end marker: #}, #) and #] respectively.

No character is modified at all inside the literal string, including newlines, \ " and '.

So, the string #["\n'##] will be equivalent to ""\n\'#".

The main use-case is to write code in code:

| string code = #[ | void main(int c, array v) { | string x = ""; | foreach( v[1..], string elm ) | x += reverse(elm)+","; | write("Testing: %s\n", reverse( x )); | #];

The three different start/end markers might be useful if you write code in code in code, since there is no way to quote the start/end markers.

o Added a shorthand syntax for integer rages: xbit, where x is a number between 1 and 31. This can be used as an example to indicate that a string is 8 bits wide only: string(8bit)

This is equivalent to the range (0..255) but can be easier to parse.

Similarly int(1bit) is an alias for 'bool', and int(12bit) is the same as int(0..4095).

o 'this::x' is now equivalent to 'this_program::x' -- access the identifier x in the current object.

New preprocessor features -------------------------

o Support for the ", ##__VA_ARGS__" cpp feature.

This makes the ‘##’ token paste operator have a special meaning when placed between a comma and a variable argument. If you write

| #define err(format, ...) f(debug)werror("ERROR: "+format, ##__VA_ARGS__)

and the variable argument is left out when the err macro is used, then the comma before the ‘##’ will be deleted. This does not happen if you pass an empty argument, nor does it happen if the token preceding ‘##’ is anything other than a comma.

o The define __COUNTER__ has been added. It is a unique integer value, the first time the macro is expanded it will be 1, the next time 2 etc.

o The preprocessor can now be run with a cpp prefix feature.

This is currently used by the precompiler to run two levels of preprocessing, one using "#cmod_" as the prefix and the other "#".

o Dynamic macros

You can now add programmatic macros. There is currently no syntax that can be used to define these while compiling code, but you can add them from one program before compiling plug-ins/modules.

The main use is macros like DEBUG(...) and IFDEBUG() that would expand to something if a debug setting is enabled in the module but nothing otherwise, or, to take an actual example from the Opera Mini source code:

| add_predefine( "METRIC()", | lambda( string name, string ... code ) | { | string type = type_of( code ); | string aggregate = aggregate_of( code ); | string format = format_of( code ); | code -= ({ 0 }); | return replace( base_metric_class, | (["ID":(string)next_metric_id++, | "NAME":stringify(name), | "TYPE":type, | "AGGREGATE":aggregate, | "FORMAT":format, | "CODE":make_value_function(code), | ])); | });

That is, a macro that needs to do some calculations, and rewrite the code more than is possible in normal macros.

This one expands something along the lines of

| METRIC("requests", Summarize, PerSecond, | floating_average_diff(requests));

into

| class Metric_requests_01 | { | inherit Server.Metric; | constant name = "transcoder:requests"; | constant type = Float; | constant format = PerSecond; | constant aggregate = Summarize; | | float value() { | return floating_average_diff(requests); | } | }

o Dependency declarations

It is now possible to use the CPP directive #require to specify a condition that must be true for the file to be seen by the resolver. This would typically be the inherited C part of a module or a system call.

| #require constant(__WebKit) | inherit __WebKit;

Optimizations -------------

o New significantly faster block allocator

The free in the old one was O(n^2), which means that as an example creating a lot of objects and then free:ing them mainly used CPU in the block allocator.

This fix changed the ordo of one of the tests that did that very thing (binarytrees) from O(n^2) to O(n), and as such is more than a simple percentual speedup in some cases, but it always improves the performance some since the base speed is also faster.

o Power-of-two hashtables are now used for most hashtables

This speeds up mappings and other hashtables a few percent, and also simplifies the code.

o Significantly changed x86-32 and an entirely new AMD64/x86-64 machine-code compilation backend

The main feature with the x86-32 edition is that it is now using normal function call mechanics, which means that it now works with modern GCC:s.

The x86-64 backends has been rewritten so that it is easier to add new instructions (x86-64) and opcodes (Pike) to it, and it is significantly more complete than the x86-32 one.

o Svalue type renumbering

PIKE_T_INT is now type #0, which makes certain things significantly faster.

As an example, clearing of object variables is now done automatically when the object is created since all object memory is set to 0.

The same thing happens when clearing values on the stack.

o Svalue type/subtype setting changed

This halves the number of instructions used to push a value on the stack in most cases.

The speedup is not large, but noticeable.

o And on a related note, we now lie to the compiler about the const:ness of the Pike_interpreter_pointer variable.

This significantly simplifies the code generated by the C-compiler for stack manipulation functions, the stack pointer is now only loaded once per code block, instead of once per stack operation.

This saves a lot of code when using the stack multiple times in a function, and should be safe enough, albeit somewhat unconventional.

If nothing else the binary size shrunk by about 5%.

o string(x..y) (constant) types

The strings now keep track of the min/max values in addition to two new flags: all-uppercase and all-lowercase.

| > typeof("foo"); | (1) Result: string(102..111)

This is used to optimize certain operations, lower_case, upper_case, search and string_has_null for now. It could be added to other places in the future as well.

A fairly common case is where you are doing lower_case or upper_case on an already lower or uppercased string. This is now significantly faster.

o Several other optimizations to execution speed has been done

+ object indexing (cache, generic speedups)

+ lower_apply, changes to apply in general

Taken together these individually small optimizations speeds up at least pike -x benchmark more than 5%.

+ A lot of opcodes implemented in machine-code for x86-64

This speed up the loop benchmarks close to a factor of 3. But then again, most real code is nothing like that benchmark.

+ Several new opcodes added

As an example an observation was that most branch_if_zero is followed by an is_eq, is_lt or is_gt or similar. Those opcodes always return 1 or 0. So, two new opcodes, quick_branch_if_zero and quick_branch_if_non_zero were added that map directly to three x86-64 opcodes, there is no need to check the types, do a real pop_stack etc.

+ Demacroified some code, resulting in smaller code-size

This makes things faster, it is often better to branch to a small block of code than to have it inlined in a lot of different locations on modern architectures.

o Faster hash-function for strings

If your CPU has the crc32 instruction (modern x86 mainly) the hashing is now an order of magnitude faster, albeit the risk of attacks using known crc:s in things like http request headers (to force long hashtable chains and slow things down) is greater, but:

+ Also added siphash24, and use that when the string hash table becomes inefficient.

Deprecated features and modules -------------------------------

o Tools.PEM and Tools.X409 deprecated

Use the corresponding modules in Standards.

o The built in security sandbox is now deprecated

Unless somebody wants to take ownership of the feature and keep it up to date the security system will be removed in the next stable release.

o The compiler now warns about switch statements where the cases aren't enumerable, since these will generate random failures if the code is read from a dump.

o strlen() now only accepts strings

o Gdbm.gdbm is now known as Gdbm.DB

o Yabu.db and Yabu.table renamed to Yabu.DB and Yabu.Table

o The keyword 'static' will now generate deprecation warnings.

Removed features and modules ----------------------------

o Removed facets

The optional (and not enabled by default) support for facet classes has been removed, since it was only partially done and not really used.

o It is no longer possible to compile Pike without libgmp.

Bignums are now a required language feature

o The old low-level 'files' module has been renamed to _Stdio

o The old _PGsql helper module has been removed.

o 'GTK' is now GTK2 if GTK2 support is available, earlier it defaulted to GTK1 as a preference over GTK2.

o Locale.Charset

The charset module is now available on the top level as 'Charset'

o The ancient syntax for arrays (string * was an alias for array(string)) has now been removed completely.

o Compatibility for Pike versions before 7.6 is no longer available.

o decode_value can no longer decode programs using the 'old style' program encoding.

Since the format has been deprecated since feb 2003, and those programs could not be decoded anyway due to other issues it is not much of a loss.

New modules -----------

o Pike.Watchdog

A Watchdog that ensures that the process is not hung for an extended period of time. The definition of 'hung' is: Has not used the default backend.

To use it simply keep an instance of the watchdog around in your application:

| Pike.Watchdog x = Pike.Watchdog( 5 ); // max 5s blocking

An important and useful side-effect of this class is that the process will start to respond to kill -QUIT by printing a lot of debug information to stderr, including memory usage, and if Pike is compiled with profiling, the CPU used since the last time kill -QUIT was called.

o Crypto.Password

A module that can be used to verify and create passwd/ldap style password hashes.

It tries to support most common password hashing schemes.

o Debug.Profiling

Tools useful for collecting and format for output the profiling information generated when Pike is compiled --with-profiling.

o NetUtils

This module contains a lot of functions useful for the storage and processing of network addresses, it supports IPv4 and IPv6.

o ADT.CritBit

Mapping-like key-sorted data structures for string, int and float-keys (ADT.CritBit.Tree, ADT.CritBit.IntTree, ADT.CritBit.FloatTree). Implemented in C.

o Standards.BSON

A module for working with BSON serialized data. See http://bsonspec.org/

o Geography.GeoIP

Does geolocation of IPv4-numbers using databases from maxmind.com or software77.net

o Protocols.WebSocket

An implementation of the WebSocket (RFC 6455) standard, both server and client

o Image.WebP

Encoder and decoder for the WEBP image format. More information about the format can be found on https://developers.google.com/speed/webp/

Requires libwebp.

o Serializer

APIs useful to simplify serialization and deserialization of objects Mainly it allows you to easily iterate over the object variables, including the private ones.

+ Serializer.Encodable A class that can be inherited to make an object easily serializable using encode_value.

o Filesystem.Monitor (and the low level System.Inotify + System.FSEvents)

Basic filesystem monitoring.

This module is intended to be used for incremental scanning of a filesystem.

Supports FSEvents on MacOS X and Inotify on Linux to provide low overhead monitoring; other systems currently use a less efficient polling approach.

o Mysql.SqlTable

This class provides some abstractions on top of an SQL table.

At the core it is generic and could work with any SQL database, but the current implementation is MySQL specific on some points, notably the semantics of AUTO_INCREMENT, the quoting method, knowledge about column types, and some conversion functions. Hence the location in the Mysql module.

Among other things, this class handles some convenient conversions between SQL and Pike data types

o Parser.CSV

This is a parser for line oriented data that is either comma, semi-colon or tab separated. It extends the functionality of the Parser.Tabular with some specific functionality related to a header and record oriented parsing of huge datasets.

o ZXID

ZXID is a library that implements SAML 2.0, Liberty ID-WSF 2.0 and XACML 2.0. Used for single sign-on.

This module implements a wrapper for ZXID. The interface is similar to the C one, but using generally accepted Pike syntax.

o Git

A module for interacting with the Git distributed version control system.

o Val

This module contains special values used by various modules, e.g. a Val.null value used both by Sql and Standards.JSON.

In many ways these values should be considered constant, but it is possible for a program to replace them with extended versions, provided they don't break the behavior of the base classes defined here. Since there is no good mechanism to handle such extending in several steps, Pike libraries should preferably ensure that the base classes defined here provide required functionality directly.

o __builtin

The __builtin module is now a directory module, so that it can provide a suitable namespace for code written in Pike intended for being inherited from modules written in C (cf precompile).

o Web.Auth

Support for federated authentication with OAuth (Web.Auth.OAuth and Web.Auth.OAuth2). There are several ready to go modules for some big OAuth authentication service providers:

+ Web.Auth.Facebook + Web.Auth.Github + Web.Auth.Google + Web.Auth.Instagram + Web.Auth.Linkedin + Web.Auth.Twitter

o Web.Api

Modules and classes for communicating with various RESTful web api's. Currently with ready to go modules for these services:

+ Web.Api.Facebook + Web.Api.Github + Web.Api.Google.Analytics + Web.Api.Google.Plus + Web.Api.Instagram + Web.Api.Linkedin + Web.Api.Twitter

o VCDiff

Glue for the open-vcdiff differential compression library. More information can be found on http://code.google.com/p/open-vcdiff/

Incompatible changes --------------------

o Parser.XML.Tree: Fixed handling of namespaced attributes.

Attribute namespaces starting with "xml" (except for "xmlns") are now handled just like any other attribute namespaces. This means that eg the attribute "xml:lang" will be expanded to "http://www.w3.org/XML/1998/namespacelang" in the mapping returned by get_attributes(), and not kept as "xml:lang" when namespace handling is enabled.

Extensions and new functions ----------------------------

o Bz2.File added

It implements a Stdio.File like API, including support for the same iterator API that Stdio.File has, allowing for convenient line iterations over BZ2 compressed files.

| foreach( Bz2.File("log.bz2")->line_iterator(); int n; string line )

o Both sscanf and sprintf can now handle binary floats in little endian format

%-8F would be a 64 bit IEEE float binary value in little endian order.

o Image.JPEG

+ decode now supports basic CMYK/YCCK support

+ exif_decode is a new function that will rotate the image according to exif information

o Image.BMP now supports some more BMP:s.

+ Added support for vertical mirroring (images starting at the bottom left)

+ Added support for 32-bit (non-alpha) BMP:s.

o String.Buffer

It is possible to add sprintf-formatted data to a String.Buffer object by calling the sprintf() method. This function works just as the normal sprintf(), but writes to the buffer instead.

o String.range(str)

This returns the minimum and maximum character value in the string.

The precision is only 8 bits, so for wide strings only character blocks are known.

o String.filter_non_unicode(str)

This function replaces all non-unicode characters in a Pike string with 0xfffd.

o SDL.Music added to SDL.

Allows the playback of audio/music files. Requires the SDL_mixed library.

o System.TM

Low-level wrapper for struct tm.

This can be used to do (very) simple calendar operations. It is, as it stands, not 100% correct unless the local time is set to GMT, and does mirror functionality already available in gmtime() and localtime() and friends, but in a (perhaps) easier to use API.

o decode_value now throws the error object Error.DecodeError.

Useful to catch format errors in the decode string.

o Process.daemon

The daemon() function is for programs wishing to detach themselves from the controlling terminal and run in the background as system daemons.

o Debug.pp_object_usage()

Pretty-print debug information, useful to get debug information about object counts and memory usage in Pike applications.

Uses the new _object_size lfun, if present in objects, to account for RAM-usage in C-objects that allocate their own memory.

o Mysql

+ Added support more modern client libraries (incl. MariaDB)

+ Mysql.mysql now has methods to query the id or SQLSTATE of the last error.

o pgsql

A complete rewrite of the existing driver. Changes in random order: + Eliminates an obscure and rare Pike-internals corruption bug. + Drops the _PGsql CMOD in favour of pure Pike using Stdio.Buffer. + Fully threaded, event and callback driven operation. + Allows for query setup and row fetches to be spread out over an arbitrary number of threads. + Maximum parallelism over a single filedescriptor (better than before). + New interface: fetch_row_array() and callback driven. + Less filling and faster than the existing driver.

o Protocols.DNS

+ Prevent endless loops in maliciously crafted domain names.

+ Add QTYPE T_ANY to DNS enum EntryType in DNS.pmod.

+ Handle truncated labels

+ TCP client and server support

o Thread no longer inherits Thread.Thread (aka thread_create)

o Thread.Farm now might work

o Cmod precompiler:

+ You can now use #cmod_{define,if,ifdef,include} etc to do preprocessing using CPP before the .cmod file is processed by the precompiler.

This preprocessing is done with the Pike precompiler, while the C-compiler preprocessor will then be used on the generated file (like before)

+ inherit "identifier" -- inherit the program returned by calling master()->resolve() on the specified identifier. Useful to inherit code written in Pike.

+ Object types on return and argument types are now resolved dynamically when needed.

This means that the typing information is sifnificantly more strict.

If you use classes that are not defined in the file (or src/program_id.h) they will be resolved when your .so-file is loaded.

Note: Circular dependencies does not work (currently, at least).

Also, the parser still often have problems with types containing '.'. You can use object(X.Y) instead of X.Y.

o String.levenshtein_distance()

The function calculates the Levenshtein distance between two strings. The Levenshtein distance describes the minimum number of edit operations (insert, delete or substitute a character) to get from one string to the other.

This can be used in approximate string matching to find matches for a short string in many longer texts, when a small number of differences is expected.

o System.sync()

Synchronizes the filesystem on systems where this is possible (currently windows and UNIX-like systems).

o System.getloadavg()

Return the current 1, 5 and 15 minute system load averages as an array.

o access()

Check if a file exist and can also return if it is readable and or writeable for the current process.

o glob()

The glob function has been extended to accept an array of globs as the first (glob pattern) argument.

In this case, if any of the given patterns match the function will return true, or, if the second argument is also an array, all entries that match any glob in the first array.

o Stdio.UDP():

+ added IPv6 multicast support

+ added set_buffer

o Stdio.Port(): + Added client and server support for TCP_FASTCONNECT

To connect using this TCP extension simply pass the data as the fifth argument to connect.

The server support is automatically enabled if possible when a Stdio.Port object is bound to a port.

+ Added support for SO_REUSEPORT. In this mode multiple Stdio.Port instances can bind to the same port (mainly used with the ports being opened in different processes).

This allows automatic load sharing between the processes, incoming connections will only be sent to one of them by the OS.

The presence of these features are indicated by constants in Stdio.Port: Stdio.Port.SO_REUSEPORT_SUPPORT and Stdio.Port.TCP_FASTOPEN_SUPPORT (although the same code should work regardless of the existence of TCP fast open in both server and client mode, it can be useful to know if support is available)

+ Added a query_fd() method

o Stdio.Buffer():

This is a byte buffer (unlike String.Buffer which is a string buffer, and thus contains unicode characters and not bytes) that is optimized for both reading from it and adding data to it at the same time.

The main intent of the class is buffering for binary and non-binary protocol parsing and output, and also buffering when using non-blocking I/O.

o Stdio.File():

+ send_fd and receive_fd

These functions can be used to send and receive an open file-descriptor over another file-descriptor. The functions are only available on some systems, and they generally only work when the file the descriptors are sent over is a UNIX domain socket or a pipe.

+ Changed internally to remove one level of indirection.

The Stdio.File object no longer has a _Stdio.Fd_ref in _fd. They are instead directly inheriting _Stdio.FD.

_fd is still available for compatibility, but internally it is gone.

+ The callbacks now get the file object as the first object if no other id has been set.

+ Added support for buffering via Stdio.Buffer.

This can be used to greatly simplify the writing of non-blocking code.

- When read and/or write buffer is enabled the corresponding callbacks get a buffer object as the second argument

- The only thing the callback has to do is add data to the buffer or read from it, depending on what callback it is.

- The write callback will now only be called when the buffer contains no more output, and data that is not read in one read callback is kept for the next time data arrives.

+ Fixed grantpt() on Solaris failing with EPERM.

o Unicode databases updated to 7.0.0 from 5.1.0

This is the latest released Unicode database from unicode.org.

o The Search search engine module has seen several fixes

+ Added support for padded blobs. This improves performance when incrementally growing blobs. This feature is only enabled if Search.Process.Compactor says this is OK, see the documentation for more information.

+ Several locking optimizations, specifically, avoid locking and unlocking for every single iteration when merging and syncing blobs.

+ Charset conversion fixes

+ Fixes for queries where the same world occur multiple times ('foo and bar and foo')

o pike -x benchmark

+ Output format changed

+ Also added support for JSON output.

+ The results should be more consistent.

+ Added options to allow comparison with a previous run.

o pike -x check_http

This new tool can be used to check http/https connectivity to a host.

o New stand-alone tools added to make it possible to build documentation without the Pike build tree

+ autodoc_to_html AutoDoc XML to HTML converter.

+ autodoc_to_split_html AutoDoc XML to splitted HTML converter.

+ git_export_autodoc Exports a stream of autodoc.xml suitable for git-fast-import. Used on pike-librarian.

o Readline tries to set the charset to the terminal charset

This makes it possible to write non-7bit characters on a terminal if the terminal supports it.

o Fixed units in pike --help=kladdkaka

o Several changes has been done to the GTK2 module

+ GTK2.DrawingArea no longer crash in draw_text if you pass odd parameters.

+ draw_pixbuf can now be passed width and height -1, which makes it take the size from the passed image.

+ GDKEvent no longer crash when you extract strings from them

+ accelerators now work

+ Fixed RadioToolButton

+ signal_connect can now connect a signal in front of the list

+ Several fixes to Tree related objects

+ GTK2.SourceView added

+ GTK2.Spinner added

o A few issues were fixed that were found by Coverity

+ Fixed memory leak in Math.Transform

+ Fixed two compares that were written as assignments (errno checks for EINTR for sockets)

o System.get_home + System.get_user

(mostly) Cross-platform ways to get the user name and home directory.

o System.AllocConsole, System.FreeConsole and System.AttachConsole for Windows NT

These are useful to create or close the console window that is shown for Pike programs.

o Process - forkd

Forkd can be used to more cheaply create new processes on UNIX like systems.

This is done by first starting a sub-process that is then used to create new processes.

If your main process is large, this is significantly faster than using the normal create_process, and does not risk running out of memory for the forked (temporary) copy of the main process that is created.

o MacOSX CoreFoundation support in the backend

This makes it somewhat more likely that native libraries can work with Pike.

o Better IPv6 support.

This includes detection of IPV6 mapped IPV4 addresses (::FFFF:i.p.v.4) and full support for IPv6 in the UDP code.

o Asynchronous Protocols.SNMP client

o Fixes to Process.run, Process.spawn_pike and friends.

+ Support OS/2 path conventions

+ Fixed multiple issues with search_path()/locate_binary() - locate_binary() is now more reliable on Windows - Now invalidates the cached path is PATH is changed - Uses locate_binary to scan the path - spawn_pike() now uses search_path()

+ You can now optionally have System.spawn_pike pass predefines, program and include path to the spawned Pike, in addition to the module path.

o Lots of autodoc fixes

A lot more of the previously existing, but broken, documentation is now readable.

o predef::types

This is equivalent to values and indices, but instead gives the types for each value.

Basically only useful for objects.

o Builtin._get_setter

This function returns a setter for a variable in an object. The setter, when called, will set the variable value to the passed argument.

o Parser.XML.Tree fixes

+ Several namespace improvement and handling fixes

o New charsets

A lot of ISO-IR charsets added: 9-1, 9-2, 31, 232, 234, 231 (aka ANSI/NISO Z39.46, aka ANSEL) 230 (aka TDS 565) 225 (SR 14111:1998), 197/209 (sami) 208 (IS 434:1997) 207 (IS 433:1996), 204,205 and 206 (aka 8859-1, 8859-4 and 8859-13 with euro) 201, 200, 138 (ECMA-121) 198 (ISO 8859-8:1999) 182, 181, 189 (TCVN 5712:1993, aka VSCII) 167, 166 (aka TIS 620-2533 (1990)), 164, 160, 151 (NC 99-10:81), 68 (APL), 59 (CODAR-U), 202 (KPS 9566-97). Fixed CSA_Z242.4

o Several fixes to Protocols.HTTP

+ Improved Protocols.HTTP.Query.PseudoFile (significantly better Stdio.Stream simulation)

+ Do not use hard coded Linux errno:s

+ Case insensitive handling of header overrides in do_method

+ Fixed broken check for URL passwords when querying

+ Add more descriptive HTTP responses along with a mostly complete list of codes

+ Handle non-standards compliant relative redirects

+ Cleaner handling of async DNS failures

+ Handle chunked transfer encoding correctly when doing async queries

+ Fixes for the proxy client support

+ Several keep-alive handling fixes

+ Server: - More forgiving MIME parsing for MSIE - Fixed range header handling - Fixed parsing of broken multipart/form-data data - Added optional error_callback to attach_fd - The response processor (response_and_finish) now treats the reply mapping as read-only. - Support if-none-match (etag:s) - Ignore errors in close when destroying the object - Fixed ordo of sending large chunks of data without using files

o Multiple threads can now call the Backend `() function (the function that waits for events).

The first thread will be the controlling thread, and all callbacks will be called in it, the other threads will wake when the controlling thread is done.

o dtrace support (on MacOSX)

Pike now supports dtrace events on function enter and leaving (and when stack frames are notionally popped, for functions doing tailrecursion).

o Standards.JSON.encode can now get the initial indentation level specified.

This is rather useful for recursive calls encode in pretty print mode (common for objects with encode_json methods).

o Added Pike.identify_cycle(x)

Checks to see if x contains any circular structures.

This can be useful while optimizing to identify reference cycles in Pike data structures, so that the need for garbage collection can be reduced.

o Most math functions (log, pow, exp sin, cos etc) can now take integers as arguments in addition to a floating point number.

The result will still be a float, the argument will be converted.

o The random(), random_string() and random_seed() might be more random

On computers with a hardware pseudo random generator random() can return significantly more random numbers, however, this means that random_seed is a no-op on those machines.

A side-effect of this is that random_string is now actually significantly faster on at least x86 cpu:s with rdrnd.

Note: If you want cryptographically random data please use Crypto.Random.random_string unless you know for sure the random data returned by the RDRND instruction is random enough.

SSL ---

o SSL now supports TLS 1.0 (SSL 3.1), TLS 1.1 and TLS 1.2.

o Several identifiers have changed names:

SSL.alert -> SSL.Alert SSL.connection + SSL.handshake -> SSL.{Client,Server,}Connection SSL.context -> SSL.Context SSL.packet -> SSL.Packet SSL.session -> SSL.Session SSL.sslfile -> SSL.File SSL.sslport -> SSL.Port SSL.state -> SSL.State

o SSL.File: Changed client/server selection API.

Client and server operation is now selected by calling either connect() (client-side) or accept() (server-side) after creating the SSL.File object.

Blocking handshaking mode is selected by calling set_blocking() before either of the above.

o SSL.File: Redesigned I/O state machine.

This reduces the complexity and risk of bugs in the I/O handling.

o SSL support for lots of new cipher suites added:

+ AEADs and modes:

- CCM and CCM-8

- GCM

+ Certificates

- ECDSA

+ Ciphers

- AES and AES-256

- CAMELLIA and CAMELLIA-256

+ Key exchange methods

- DH and DHE

- ECDH and ECDHE

+ All suites currently defined consisting of combinations of the above (and earlier existing) have been added (~190 suites in total).

o TLS Extensions added:

+ ALPN (Application Layer Protocol Negotiation) (RFC 7301).

+ EC Point Formats (RFC 4492).

+ Elliptic Curves (RFC 4492).

+ Encrypt then MAC.

+ Fallback SCSV.

+ Heartbeat (RFC 6520).

+ Max Fragment Length (RFC 6066).

+ Padding.

+ Renegotiation info (RFC 5746).

+ Signature Algorithms (RFC 5246).

+ SNI (Server Name Indicator) for both client and server (RFC 6066).

+ Truncated HMAC (RFC 6066).

o Improved protection against various SSL/TLS attacks:

+ BEAST protection (aka 1/(n-1)).

Improve resilience against the BEAST client-side attack, by splitting the first data packet into two, where the first only contains a single byte of payload.

+ Heartbleed protection.

The protocol code can probe the peer for the Heartbleed vulnerability, and aborts the connection with ALERT_insufficient_protection if so.

+ Lucky 13 protection.

Attempts to have HMAC calculation take the same amount of time regardless of padding size.

o SSL.Context: Improved cipher suite selection:

+ Filtering of weak cipher suites.

+ Suite B (RFC 6460).

o SSL.Context: Support multiple concurrent certificates.

This allows a server to eg have both an RSA and an ECDSA certificate.

Crypto and Nettle -----------------

o Nettle refactored

CBC cipher mode is now twice as fast.

o Nettle 3.0 supported.

The new APIs in Nettle 3.0 are now detected and utilized.

o Crypto.GCM

GCM (Galois Counter Mode) cipher mode added.

o Blowfish and Serpent support fixed in Nettle

o Crypto.PGP

Added support for SHA256, SHA384 and SHA512 as hash functions. Expose the used hash and key types in the out data

o Crypto.Arctwo

The 1-128 bit cipher Arctwo is now provided as a block cipher in Crypto. This cipher is only intended for compatibility with OLD third party code, and should NOT be used for new development.

o Crypto.Camellia

The 128/256 bit cipher Camellia is now available as block cipher in Crypto.

* Crypto.ECC

Elliptic Curve operations are now supported when compiled with Nettle 3.0 or later.

o Crypto.SALSA20 and Crypto.SALSA20R12

The 128/256 bit cipher SALSA20 is now available as a stream cipher in Crypto. SALSA20R12 is SALSA20 reduced to just 12 rounds.

o Crypto.SHA3_224, Crypto.SHA3_256, Crypto.SHA3_384 and Crypto.SHA3_512

The SHA-3 secure hash algorithm has been added in multiple variants.

o Crypto.GOST94 and RIPEMD160

The lesser used hash functions GOST R 34.11-94 (RFC 4357) and RIPEMD160 have been added.

o Crypto.RSA and Crypto.DSA

The key generation for RSA and DSA are now done by Nettle. This results in 90% faster key generation for RSA. Key generation for DSA is 10 times slower, but produces better quality keys.

o Crypto.Hash

Added support for pbkdf1 from PKCS#5v1.5 and pbkdf2 from PKCS#5v2.0.

o Crypto.Random

The underlying algorithm for Crypto.Random has changed from a overly cautious Yarrow implementation to Fortuna on top of system random sources. About 7000x faster than before.

o Standards.PEM

+ Added some support for encrypted PEM files

o Standards.X509

X509 was moved from Tools to Standards and has been refactored and bug fixed. It is now possible to extend both validation and creation of certificates with new cipher and hash algorithms. A range of new algorithms are supported for both RSA and DSA:

RSA MD2 RSA MD5 RSA SHA-1 RSA SHA-2-256 RSA SHA-2-384 RSA SHA-2-512 DSA SHA-1 DSA SHA-2-224 DSA SHA-2-256

Note that the API has changed compared to Tools.X509 and there is now a single make_selfsigned_certificate() method for both RSA and DSA, though it takes the same arguments. In addition a hash function and serial number can be supplied. The hash function defaults to SHA-2-256.

Incompatible C-level API changes --------------------------------

o New svalue layout

The most obvious change is that the macros TYPEOF() and SUBTYPEOF() are now actually needed to directly access the type and subtype of an svalue, the svalues no have 'type' and 'subtype' members.

There are also a few additional macros used to set both the type and subtype of an svalue at the same time:

SVAL_SET_TYPE_SUBTYPE(sval,type,subtype) and SVAL_SET_TYPE_DC(sval,type)

(these are only neede if you do not use the usual push_* macros)

They are useful because they (usually) compiles to significantly more compact code, especially if the type and subtype are compiletime constants. The _DC variant will either leave the subtype unchanged or set it to 0, useful when you do not care about the actual subtype (which is, really, most of the time).

o get_storage() returns void*

There is no need for casts in non-C++ code.

Building and installing -----------------------

o -fvisibility=hidden is now the default

This means that PMOD_EXPORT is now actually needed on systems like Linux and MacOSX. It also means that the binary is slightly smaller and faster.

o clang compilation fixes (bignum overflow checks, misc)

It is now possible to compile Pike using a modern clang compiler.

o Removed bundles

Pike no longer comes with copies of some libraries, and the support for that in the makefile has been removed.

o Several OS/2 and windows compilation fixes

o C89 assumed

The configure tests will not check for functions defined in C89 anymore.

Lots of bug fixes -----------------

o Fixed symbol resolution with deep inherits and mixins

o Fixed PNG 4bpl indexed mode with alpha

o The _sprintf LFUN now works with %F

o foreach( arr[-two()..], string x), where two() returns 2 will no longer iterate over the first element in the array three times or crash.

o Postgres.postgres_result now properly decodes binary blobs and strips trailing spaces from CHAR columns.

o Fixed a typo from 2001-05-05 that caused a lot of countries to reside on the continent ',Europa' instead of "Europa" in Geography.Countries.

Obviously the continent is not that often checked.

o Fixed interresting bug in the constant number parsing for octal numbers in escapes.

For whatever reason 8:s were accepted for every location except the first in octal numbers, so 078 was considered an alias for 0100.

This could cause issues at times, and will result in string parsing being somewhat different:

| Pike v7.8 release 700 running Hilfe v3.5 (Incremental Pike Frontend) | > "\078"; | (1) Result: "@" | Pike v8.0 release 3 running Hilfe v3.5 (Incremental Pike Frontend) | > "\078"; | (1) Result: "\a8"

o A lot more, see the (more or less) full changelog for more info:

http://pike-librarian.lysator.liu.se/index.xml?m=pike.git&start=forever&...

Regards,