I'm taking up an old tradition that haven't been practiced since 2008 as far as
I can tell: Making releases on new years eve. Bill has been busy with other
things, so I'll jump in to my old role as release manager for now.
Pike 8.0.146 beta for your testing pleasures:
https://pike.lysator.liu.se/pub/pike/beta/8.0.146/Pike-v8.0.146.tar.gz
Pike 8: Changes since Pike 7.8
----------------------------------------------------------------------
New language features
---------------------
o Added a way to access the local, and not the overloaded,
implementation of a symbol.
As an example, given the classes:
| class Test
| {
| int a( ) { return 1; }
| int b( ) { return local::a(); } // New
| int c( ) { return a(); }
| }
|
| class Test2
| {
| int a() { return 42; }
| }
Both Test()->b() and Test2()->b() will return 1, but Test2()->a()
and Test2()->c() will return 42.
o Added new syntax that can be used to return the current object as if
it was a class it is inheriting
The syntax is X::this, where X is the inherited class.
The returned object will not include any symbols not available in
the class X, but your overloaded methods (and other identifiers)
will be the ones returned when the object is indexed.
o Added a way to easily inherit the implementation of a subclass in an
inheriting class.
| inherit ::this_program
The syntax refers to the previous definition of the current class in
the inherited class, and is typically used with inherit like:
| inherit Foo;
|
| // Override the Bar inherited from Foo.
| class Bar {
| // The new Bar is based on the implementation of Bar in Foo.
| inherit ::this_program;
| // ...
| }
o Added new syntax for referring to and naming inherits.
| inherit "foo.pike";
| inherit "foo/bar/baz.pike" : "foo/bar";
| // ...
| "foo.pike"::foo();
| "foo/bar"::baz();
o Look up of named inherits now also looks in indirect inherits.
| class A { int fun() {} }
| class B { inherit A; }
| class C {
| inherit B;
| // ...
| // The function fun may here be referred to as any of:
| B::A::fun();
| B::fun();
| A::fun(); // New.
o Implemented the variant keyword.
The keyword was present before, but did not actually do anything
useful.
This is used to do function overloading based on the argument types.
As an example:
| class Test1 { int b(); }
| class Test2 { int a(); }
|
| class Foo
| {
| variant string foo( object q ){ return "Genericfoo"; }
| variant string foo( Test1 x ) { return "Test 1"; }
| variant string foo( Test2 x ) { return "Test 2"; }
|
| variant string bar( int(0..0) s ) { return "0"; }
| variant float bar( int(1..1) s ) { return 1.0; }
| variant int bar( int(2..2) s ) { return 2; }
| }
A common use-case is functions that return different types depending
on the arguments, such as getenv:
| string|mapping(string:string) getenv( string|void variable );
can now be rewritten as
| variant string getenv( string variable );
| variant mapping(string:string) getenv( );
which gives significantly better type-checking.
o The type-checker has been rewritten.
Mainly it is now much better at finding type errors, and most error
messages are more readable.
o Allow '.' to be used as an index operator in the few places it did
not work before.
o "Safe" index
Copied from a popular extension to other C-like languages.
X[?ind] is equivalent to ((auto _ = X),(_ && _[ind]))
while X?->ind is equivalent to ((auto _ = X),(_ && _->ind))
The main use is 'deep indexing' where some elements can be 0:
| request_id?->user?->misc?->x
vs
| request_id && request_id->user && request_id->user->misc
| && request_id->user->misc->x
o Added the '?:' operator for compatibility with other C-like
languages. It is identical to '||' in Pike.
o The && operator changed, when doing A && B, and A is false, keep A
instead of returning 0.
This is especially useful then A is not actually 0, but an object
that evaluates to false, or UNDEFINED.
o Fixed symbol resolution with deep inherits and mixins.
o Added multi-character character constants.
'ILBM' is equivalent to (('I'<<24)|('L'<<16)|('B'<<8)|'M').
Unlike how it works in some C compilers the native byte order is
never relevant.
o Added new syntax for literal-string constants
#{, #( and #[ starts a literal string, and it is ended by the
corresponding end marker: #}, #) and #] respectively.
No character is modified at all inside the literal string, including
newlines, \ " and '.
So, the string #["\n\'##] will be equivalent to "\"\\n\\'#".
The main use-case is to write code in code:
| string code = #[
| void main(int c, array v) {
| string x = "";
| foreach( v[1..], string elm )
| x += reverse(elm)+",";
| write("Testing: %s\n", reverse( x ));
| #];
The three different start/end markers might be useful if you write
code in code in code, since there is no way to quote the start/end
markers.
o Added a shorthand syntax for integer rages: xbit, where x is a
number between 1 and 31. This can be used as an example to indicate
that a string is 8 bits wide only: string(8bit)
This is equivalent to the range (0..255) but can be easier to parse.
Similarly int(1bit) is an alias for 'bool', and int(12bit) is the
same as int(0..4095).
o 'this::x' is now equivalent to 'this_program::x' -- access the
identifier x in the current object.
New preprocessor features
-------------------------
o Support for the ", ##__VA_ARGS__" cpp feature.
This makes the ‘##’ token paste operator have a special meaning when
placed between a comma and a variable argument. If you write
| #define err(format, ...) f(debug)werror("ERROR: "+format, ##__VA_ARGS__)
and the variable argument is left out when the err macro is used,
then the comma before the ‘##’ will be deleted. This does not happen
if you pass an empty argument, nor does it happen if the token
preceding ‘##’ is anything other than a comma.
o The define __COUNTER__ has been added. It is a unique integer value,
the first time the macro is expanded it will be 1, the next time 2
etc.
o The preprocessor can now be run with a cpp prefix feature.
This is currently used by the precompiler to run two levels of
preprocessing, one using "#cmod_" as the prefix and the other "#".
o Dynamic macros
You can now add programmatic macros. There is currently no syntax
that can be used to define these while compiling code, but you can
add them from one program before compiling plug-ins/modules.
The main use is macros like DEBUG(...) and IFDEBUG() that would
expand to something if a debug setting is enabled in the module but
nothing otherwise, or, to take an actual example from the Opera Mini
source code:
| add_predefine( "METRIC()",
| lambda( string name, string ... code )
| {
| string type = type_of( code );
| string aggregate = aggregate_of( code );
| string format = format_of( code );
| code -= ({ 0 });
| return replace( base_metric_class,
| (["ID":(string)next_metric_id++,
| "NAME":stringify(name),
| "TYPE":type,
| "AGGREGATE":aggregate,
| "FORMAT":format,
| "CODE":make_value_function(code),
| ]));
| });
That is, a macro that needs to do some calculations, and rewrite the
code more than is possible in normal macros.
This one expands something along the lines of
| METRIC("requests", Summarize, PerSecond,
| floating_average_diff(requests));
into
| class Metric_requests_01
| {
| inherit Server.Metric;
| constant name = "transcoder:requests";
| constant type = Float;
| constant format = PerSecond;
| constant aggregate = Summarize;
|
| float value() {
| return floating_average_diff(requests);
| }
| }
o Dependency declarations
It is now possible to use the CPP directive #require to specify a
condition that must be true for the file to be seen by the resolver.
This would typically be the inherited C part of a module or a system
call.
| #require constant(__WebKit)
| inherit __WebKit;
Optimizations
-------------
o New significantly faster block allocator
The free in the old one was O(n^2), which means that as an example
creating a lot of objects and then free:ing them mainly used CPU in
the block allocator.
This fix changed the ordo of one of the tests that did that very
thing (binarytrees) from O(n^2) to O(n), and as such is more than a
simple percentual speedup in some cases, but it always improves the
performance some since the base speed is also faster.
o Power-of-two hashtables are now used for most hashtables
This speeds up mappings and other hashtables a few percent, and also
simplifies the code.
o Significantly changed x86-32 and an entirely new AMD64/x86-64
machine-code compilation backend
The main feature with the x86-32 edition is that it is now using
normal function call mechanics, which means that it now works with
modern GCC:s.
The x86-64 backends has been rewritten so that it is easier to add
new instructions (x86-64) and opcodes (Pike) to it, and it is
significantly more complete than the x86-32 one.
o Svalue type renumbering
PIKE_T_INT is now type #0, which makes certain things significantly
faster.
As an example, clearing of object variables is now done
automatically when the object is created since all object memory is
set to 0.
The same thing happens when clearing values on the stack.
o Svalue type/subtype setting changed
This halves the number of instructions used to push a value on the
stack in most cases.
The speedup is not large, but noticeable.
o And on a related note, we now lie to the compiler about the
const:ness of the Pike_interpreter_pointer variable.
This significantly simplifies the code generated by the C-compiler
for stack manipulation functions, the stack pointer is now only
loaded once per code block, instead of once per stack operation.
This saves a lot of code when using the stack multiple times in a
function, and should be safe enough, albeit somewhat unconventional.
If nothing else the binary size shrunk by about 5%.
o string(x..y) (constant) types
The strings now keep track of the min/max values in addition to two
new flags: all-uppercase and all-lowercase.
| > typeof("foo");
| (1) Result: string(102..111)
This is used to optimize certain operations, lower_case, upper_case,
search and string_has_null for now. It could be added to other
places in the future as well.
A fairly common case is where you are doing lower_case or upper_case
on an already lower or uppercased string. This is now significantly
faster.
o Several other optimizations to execution speed has been done
+ object indexing (cache, generic speedups)
+ lower_apply, changes to apply in general
Taken together these individually small optimizations speeds up at
least pike -x benchmark more than 5%.
+ A lot of opcodes implemented in machine-code for x86-64
This speed up the loop benchmarks close to a factor of 3. But then
again, most real code is nothing like that benchmark.
+ Several new opcodes added
As an example an observation was that most branch_if_zero is
followed by an is_eq, is_lt or is_gt or similar. Those opcodes
always return 1 or 0. So, two new opcodes, quick_branch_if_zero
and quick_branch_if_non_zero were added that map directly to three
x86-64 opcodes, there is no need to check the types, do a real
pop_stack etc.
+ Demacroified some code, resulting in smaller code-size
This makes things faster, it is often better to branch to a small
block of code than to have it inlined in a lot of different
locations on modern architectures.
o Faster hash-function for strings
If your CPU has the crc32 instruction (modern x86 mainly) the
hashing is now an order of magnitude faster, albeit the risk of
attacks using known crc:s in things like http request headers (to
force long hashtable chains and slow things down) is greater, but:
+ Also added siphash24, and use that when the string hash table
becomes inefficient.
Deprecated features and modules
-------------------------------
o Tools.PEM and Tools.X409 deprecated
Use the corresponding modules in Standards.
o The built in security sandbox is now deprecated
Unless somebody wants to take ownership of the feature and keep it
up to date the security system will be removed in the next stable
release.
o The compiler now warns about switch statements where the cases
aren't enumerable, since these will generate random failures if the
code is read from a dump.
o strlen() now only accepts strings
o Gdbm.gdbm is now known as Gdbm.DB
o Yabu.db and Yabu.table renamed to Yabu.DB and Yabu.Table
o The keyword 'static' will now generate deprecation warnings.
Removed features and modules
----------------------------
o Removed facets
The optional (and not enabled by default) support for facet classes
has been removed, since it was only partially done and not really
used.
o It is no longer possible to compile Pike without libgmp.
Bignums are now a required language feature
o The old low-level 'files' module has been renamed to _Stdio
o The old _PGsql helper module has been removed.
o 'GTK' is now GTK2 if GTK2 support is available, earlier it defaulted
to GTK1 as a preference over GTK2.
o Locale.Charset
The charset module is now available on the top level as 'Charset'
o The ancient syntax for arrays (string * was an alias for
array(string)) has now been removed completely.
o Compatibility for Pike versions before 7.6 is no longer available.
o decode_value can no longer decode programs using the 'old style'
program encoding.
Since the format has been deprecated since feb 2003, and those
programs could not be decoded anyway due to other issues it is not
much of a loss.
New modules
-----------
o Pike.Watchdog
A Watchdog that ensures that the process is not hung for an extended
period of time. The definition of 'hung' is: Has not used the
default backend.
To use it simply keep an instance of the watchdog around in your
application:
| Pike.Watchdog x = Pike.Watchdog( 5 ); // max 5s blocking
An important and useful side-effect of this class is that the
process will start to respond to kill -QUIT by printing a lot of
debug information to stderr, including memory usage, and if Pike is
compiled with profiling, the CPU used since the last time kill -QUIT
was called.
o Crypto.Password
A module that can be used to verify and create passwd/ldap style
password hashes.
It tries to support most common password hashing schemes.
o Debug.Profiling
Tools useful for collecting and format for output the profiling
information generated when Pike is compiled --with-profiling.
o NetUtils
This module contains a lot of functions useful for the storage and
processing of network addresses, it supports IPv4 and IPv6.
o ADT.CritBit
Mapping-like key-sorted data structures for string, int and
float-keys (ADT.CritBit.Tree, ADT.CritBit.IntTree,
ADT.CritBit.FloatTree). Implemented in C.
o Standards.BSON
A module for working with BSON serialized data. See
http://bsonspec.org/
o Geography.GeoIP
Does geolocation of IPv4-numbers using databases from maxmind.com or
software77.net
o Protocols.WebSocket
An implementation of the WebSocket (RFC 6455) standard, both server
and client
o Image.WebP
Encoder and decoder for the WEBP image format.
More information about the format can be found on
https://developers.google.com/speed/webp/
Requires libwebp.
o Serializer
APIs useful to simplify serialization and deserialization of objects
Mainly it allows you to easily iterate over the object variables,
including the private ones.
+ Serializer.Encodable
A class that can be inherited to make an object easily
serializable using encode_value.
o Filesystem.Monitor (and the low level System.Inotify +
System.FSEvents)
Basic filesystem monitoring.
This module is intended to be used for incremental scanning of a
filesystem.
Supports FSEvents on MacOS X and Inotify on Linux to provide low
overhead monitoring; other systems currently use a less efficient
polling approach.
o Mysql.SqlTable
This class provides some abstractions on top of an SQL table.
At the core it is generic and could work with any SQL database, but
the current implementation is MySQL specific on some points, notably
the semantics of AUTO_INCREMENT, the quoting method, knowledge about
column types, and some conversion functions. Hence the location in
the Mysql module.
Among other things, this class handles some convenient conversions
between SQL and Pike data types
o Parser.CSV
This is a parser for line oriented data that is either comma,
semi-colon or tab separated. It extends the functionality of the
Parser.Tabular with some specific functionality related to a header
and record oriented parsing of huge datasets.
o ZXID
ZXID is a library that implements SAML 2.0, Liberty ID-WSF 2.0
and XACML 2.0. Used for single sign-on.
This module implements a wrapper for ZXID. The interface is similar
to the C one, but using generally accepted Pike syntax.
o Git
A module for interacting with the Git distributed version control
system.
o Val
This module contains special values used by various modules, e.g. a
Val.null value used both by Sql and Standards.JSON.
In many ways these values should be considered constant, but it is
possible for a program to replace them with extended versions,
provided they don't break the behavior of the base classes defined
here. Since there is no good mechanism to handle such extending in
several steps, Pike libraries should preferably ensure that the base
classes defined here provide required functionality directly.
o __builtin
The __builtin module is now a directory module, so that it can
provide a suitable namespace for code written in Pike intended for
being inherited from modules written in C (cf precompile).
o Web.Auth
Support for federated authentication with OAuth (Web.Auth.OAuth and
Web.Auth.OAuth2). There are several ready to go modules for some big
OAuth authentication service providers:
+ Web.Auth.Facebook
+ Web.Auth.Github
+ Web.Auth.Google
+ Web.Auth.Instagram
+ Web.Auth.Linkedin
+ Web.Auth.Twitter
o Web.Api
Modules and classes for communicating with various RESTful web
api's. Currently with ready to go modules for these services:
+ Web.Api.Facebook
+ Web.Api.Github
+ Web.Api.Google.Analytics
+ Web.Api.Google.Plus
+ Web.Api.Instagram
+ Web.Api.Linkedin
+ Web.Api.Twitter
o VCDiff
Glue for the open-vcdiff differential compression library. More
information can be found on http://code.google.com/p/open-vcdiff/
Incompatible changes
--------------------
o Parser.XML.Tree: Fixed handling of namespaced attributes.
Attribute namespaces starting with "xml" (except for "xmlns") are
now handled just like any other attribute namespaces. This means
that eg the attribute "xml:lang" will be expanded to
"http://www.w3.org/XML/1998/namespacelang" in the mapping returned
by get_attributes(), and not kept as "xml:lang" when namespace
handling is enabled.
Extensions and new functions
----------------------------
o Bz2.File added
It implements a Stdio.File like API, including support for the same
iterator API that Stdio.File has, allowing for convenient line
iterations over BZ2 compressed files.
| foreach( Bz2.File("log.bz2")->line_iterator(); int n; string line )
o Both sscanf and sprintf can now handle binary floats in little
endian format
%-8F would be a 64 bit IEEE float binary value in little endian
order.
o Image.JPEG
+ decode now supports basic CMYK/YCCK support
+ exif_decode is a new function that will rotate the image according
to exif information
o Image.BMP now supports some more BMP:s.
+ Added support for vertical mirroring (images starting at the
bottom left)
+ Added support for 32-bit (non-alpha) BMP:s.
o String.Buffer
It is possible to add sprintf-formatted data to a String.Buffer
object by calling the sprintf() method. This function works just as
the normal sprintf(), but writes to the buffer instead.
o String.range(str)
This returns the minimum and maximum character value in the string.
The precision is only 8 bits, so for wide strings only character
blocks are known.
o String.filter_non_unicode(str)
This function replaces all non-unicode characters in a Pike string
with 0xfffd.
o SDL.Music added to SDL.
Allows the playback of audio/music files.
Requires the SDL_mixed library.
o System.TM
Low-level wrapper for struct tm.
This can be used to do (very) simple calendar operations. It is, as
it stands, not 100% correct unless the local time is set to GMT, and
does mirror functionality already available in gmtime() and
localtime() and friends, but in a (perhaps) easier to use API.
o decode_value now throws the error object Error.DecodeError.
Useful to catch format errors in the decode string.
o Process.daemon
The daemon() function is for programs wishing to detach themselves
from the controlling terminal and run in the background as system
daemons.
o Debug.pp_object_usage()
Pretty-print debug information, useful to get debug information
about object counts and memory usage in Pike applications.
Uses the new _object_size lfun, if present in objects, to account
for RAM-usage in C-objects that allocate their own memory.
o Mysql
+ Added support more modern client libraries (incl. MariaDB)
+ Mysql.mysql now has methods to query the id or SQLSTATE of the
last error.
o pgsql
A complete rewrite of the existing driver. Changes in random order:
+ Eliminates an obscure and rare Pike-internals corruption bug.
+ Drops the _PGsql CMOD in favour of pure Pike using Stdio.Buffer.
+ Fully threaded, event and callback driven operation.
+ Allows for query setup and row fetches to be spread out over
an arbitrary number of threads.
+ Maximum parallelism over a single filedescriptor (better than before).
+ New interface: fetch_row_array() and callback driven.
+ Less filling and faster than the existing driver.
o Protocols.DNS
+ Prevent endless loops in maliciously crafted domain names.
+ Add QTYPE T_ANY to DNS enum EntryType in DNS.pmod.
+ Handle truncated labels
+ TCP client and server support
o Thread no longer inherits Thread.Thread (aka thread_create)
o Thread.Farm now might work
o Cmod precompiler:
+ You can now use #cmod_{define,if,ifdef,include} etc to do
preprocessing using CPP before the .cmod file is processed by the
precompiler.
This preprocessing is done with the Pike precompiler, while the
C-compiler preprocessor will then be used on the generated file
(like before)
+ inherit "identifier"
-- inherit the program returned by calling master()->resolve() on
the specified identifier. Useful to inherit code written in Pike.
+ Object types on return and argument types are now resolved
dynamically when needed.
This means that the typing information is sifnificantly more
strict.
If you use classes that are not defined in the file (or
src/program_id.h) they will be resolved when your .so-file is
loaded.
Note: Circular dependencies does not work (currently, at least).
Also, the parser still often have problems with types containing
'.'. You can use object(X.Y) instead of X.Y.
o String.levenshtein_distance()
The function calculates the Levenshtein distance between two
strings. The Levenshtein distance describes the minimum number of
edit operations (insert, delete or substitute a character) to get
from one string to the other.
This can be used in approximate string matching to find matches
for a short string in many longer texts, when a small number of
differences is expected.
o System.sync()
Synchronizes the filesystem on systems where this is possible
(currently windows and UNIX-like systems).
o System.getloadavg()
Return the current 1, 5 and 15 minute system load averages as an
array.
o access()
Check if a file exist and can also return if it is readable and or
writeable for the current process.
o glob()
The glob function has been extended to accept an array of globs as
the first (glob pattern) argument.
In this case, if any of the given patterns match the function will
return true, or, if the second argument is also an array, all
entries that match any glob in the first array.
o Stdio.UDP():
+ added IPv6 multicast support
+ added set_buffer
o Stdio.Port():
+ Added client and server support for TCP_FASTCONNECT
To connect using this TCP extension simply pass the data as the
fifth argument to connect.
The server support is automatically enabled if possible when a
Stdio.Port object is bound to a port.
+ Added support for SO_REUSEPORT. In this mode multiple Stdio.Port
instances can bind to the same port (mainly used with the ports
being opened in different processes).
This allows automatic load sharing between the processes, incoming
connections will only be sent to one of them by the OS.
The presence of these features are indicated by constants in
Stdio.Port: Stdio.Port.SO_REUSEPORT_SUPPORT and
Stdio.Port.TCP_FASTOPEN_SUPPORT (although the same code should
work regardless of the existence of TCP fast open in both server
and client mode, it can be useful to know if support is available)
+ Added a query_fd() method
o Stdio.Buffer():
This is a byte buffer (unlike String.Buffer which is a string
buffer, and thus contains unicode characters and not bytes) that is
optimized for both reading from it and adding data to it at the
same time.
The main intent of the class is buffering for binary and non-binary
protocol parsing and output, and also buffering when using
non-blocking I/O.
o Stdio.File():
+ send_fd and receive_fd
These functions can be used to send and receive an open
file-descriptor over another file-descriptor. The functions are
only available on some systems, and they generally only work when
the file the descriptors are sent over is a UNIX domain socket or
a pipe.
+ Changed internally to remove one level of indirection.
The Stdio.File object no longer has a _Stdio.Fd_ref in _fd. They
are instead directly inheriting _Stdio.FD.
_fd is still available for compatibility, but internally it is gone.
+ The callbacks now get the file object as the first object if no
other id has been set.
+ Added support for buffering via Stdio.Buffer.
This can be used to greatly simplify the writing of non-blocking
code.
- When read and/or write buffer is enabled the corresponding
callbacks get a buffer object as the second argument
- The only thing the callback has to do is add data to the buffer
or read from it, depending on what callback it is.
- The write callback will now only be called when the buffer
contains no more output, and data that is not read in one read
callback is kept for the next time data arrives.
+ Fixed grantpt() on Solaris failing with EPERM.
o Unicode databases updated to 7.0.0 from 5.1.0
This is the latest released Unicode database from unicode.org.
o The Search search engine module has seen several fixes
+ Added support for padded blobs. This improves performance when
incrementally growing blobs. This feature is only enabled if
Search.Process.Compactor says this is OK, see the documentation
for more information.
+ Several locking optimizations, specifically, avoid locking and
unlocking for every single iteration when merging and syncing
blobs.
+ Charset conversion fixes
+ Fixes for queries where the same world occur multiple times
('foo and bar and foo')
o pike -x benchmark
+ Output format changed
+ Also added support for JSON output.
+ The results should be more consistent.
+ Added options to allow comparison with a previous run.
o pike -x check_http
This new tool can be used to check http/https connectivity to a host.
o New stand-alone tools added to make it possible to build
documentation without the Pike build tree
+ autodoc_to_html
AutoDoc XML to HTML converter.
+ autodoc_to_split_html
AutoDoc XML to splitted HTML converter.
+ git_export_autodoc
Exports a stream of autodoc.xml suitable for git-fast-import.
Used on pike-librarian.
o Readline tries to set the charset to the terminal charset
This makes it possible to write non-7bit characters on a terminal if
the terminal supports it.
o Fixed units in pike --help=kladdkaka
o Several changes has been done to the GTK2 module
+ GTK2.DrawingArea no longer crash in draw_text if you pass odd
parameters.
+ draw_pixbuf can now be passed width and height -1, which makes it
take the size from the passed image.
+ GDKEvent no longer crash when you extract strings from them
+ accelerators now work
+ Fixed RadioToolButton
+ signal_connect can now connect a signal in front of the list
+ Several fixes to Tree related objects
+ GTK2.SourceView added
+ GTK2.Spinner added
o A few issues were fixed that were found by Coverity
+ Fixed memory leak in Math.Transform
+ Fixed two compares that were written as assignments (errno checks
for EINTR for sockets)
o System.get_home + System.get_user
(mostly) Cross-platform ways to get the user name and home
directory.
o System.AllocConsole, System.FreeConsole and System.AttachConsole for
Windows NT
These are useful to create or close the console window that is shown
for Pike programs.
o Process - forkd
Forkd can be used to more cheaply create new processes on UNIX like
systems.
This is done by first starting a sub-process that is then used to
create new processes.
If your main process is large, this is significantly faster than
using the normal create_process, and does not risk running out of
memory for the forked (temporary) copy of the main process that is
created.
o MacOSX CoreFoundation support in the backend
This makes it somewhat more likely that native libraries can work
with Pike.
o Better IPv6 support.
This includes detection of IPV6 mapped IPV4 addresses
(::FFFF:i.p.v.4) and full support for IPv6 in the UDP code.
o Asynchronous Protocols.SNMP client
o Fixes to Process.run, Process.spawn_pike and friends.
+ Support OS/2 path conventions
+ Fixed multiple issues with search_path()/locate_binary()
- locate_binary() is now more reliable on Windows
- Now invalidates the cached path is PATH is changed
- Uses locate_binary to scan the path
- spawn_pike() now uses search_path()
+ You can now optionally have System.spawn_pike pass predefines,
program and include path to the spawned Pike, in addition to the
module path.
o Lots of autodoc fixes
A lot more of the previously existing, but broken, documentation is
now readable.
o predef::types
This is equivalent to values and indices, but instead gives the
types for each value.
Basically only useful for objects.
o Builtin._get_setter
This function returns a setter for a variable in an object. The
setter, when called, will set the variable value to the passed
argument.
o Parser.XML.Tree fixes
+ Several namespace improvement and handling fixes
o New charsets
A lot of ISO-IR charsets added:
9-1, 9-2, 31, 232, 234, 231 (aka ANSI/NISO Z39.46, aka ANSEL) 230
(aka TDS 565) 225 (SR 14111:1998), 197/209 (sami) 208 (IS 434:1997)
207 (IS 433:1996), 204,205 and 206 (aka 8859-1, 8859-4 and 8859-13
with euro) 201, 200, 138 (ECMA-121) 198 (ISO 8859-8:1999) 182, 181,
189 (TCVN 5712:1993, aka VSCII) 167, 166 (aka TIS 620-2533 (1990)),
164, 160, 151 (NC 99-10:81), 68 (APL), 59 (CODAR-U), 202 (KPS
9566-97). Fixed CSA_Z242.4
o Several fixes to Protocols.HTTP
+ Improved Protocols.HTTP.Query.PseudoFile
(significantly better Stdio.Stream simulation)
+ Do not use hard coded Linux errno:s
+ Case insensitive handling of header overrides in do_method
+ Fixed broken check for URL passwords when querying
+ Add more descriptive HTTP responses along with a mostly complete
list of codes
+ Handle non-standards compliant relative redirects
+ Cleaner handling of async DNS failures
+ Handle chunked transfer encoding correctly when doing async
queries
+ Fixes for the proxy client support
+ Several keep-alive handling fixes
+ Server:
- More forgiving MIME parsing for MSIE
- Fixed range header handling
- Fixed parsing of broken multipart/form-data data
- Added optional error_callback to attach_fd
- The response processor (response_and_finish) now treats the
reply mapping as read-only.
- Support if-none-match (etag:s)
- Ignore errors in close when destroying the object
- Fixed ordo of sending large chunks of data without using files
o Multiple threads can now call the Backend `() function (the function
that waits for events).
The first thread will be the controlling thread, and all callbacks
will be called in it, the other threads will wake when the
controlling thread is done.
o dtrace support (on MacOSX)
Pike now supports dtrace events on function enter and leaving (and
when stack frames are notionally popped, for functions doing
tailrecursion).
o Standards.JSON.encode can now get the initial indentation level
specified.
This is rather useful for recursive calls encode in pretty print
mode (common for objects with encode_json methods).
o Added Pike.identify_cycle(x)
Checks to see if x contains any circular structures.
This can be useful while optimizing to identify reference cycles in
Pike data structures, so that the need for garbage collection can be
reduced.
o Most math functions (log, pow, exp sin, cos etc) can now take
integers as arguments in addition to a floating point number.
The result will still be a float, the argument will be converted.
o The random(), random_string() and random_seed() might be more random
On computers with a hardware pseudo random generator random() can
return significantly more random numbers, however, this means that
random_seed is a no-op on those machines.
A side-effect of this is that random_string is now actually
significantly faster on at least x86 cpu:s with rdrnd.
Note: If you want cryptographically random data please use
Crypto.Random.random_string unless you know for sure the random data
returned by the RDRND instruction is random enough.
SSL
---
o SSL now supports TLS 1.0 (SSL 3.1), TLS 1.1 and TLS 1.2.
o Several identifiers have changed names:
SSL.alert -> SSL.Alert
SSL.connection + SSL.handshake -> SSL.{Client,Server,}Connection
SSL.context -> SSL.Context
SSL.packet -> SSL.Packet
SSL.session -> SSL.Session
SSL.sslfile -> SSL.File
SSL.sslport -> SSL.Port
SSL.state -> SSL.State
o SSL.File: Changed client/server selection API.
Client and server operation is now selected by calling either
connect() (client-side) or accept() (server-side) after creating the
SSL.File object.
Blocking handshaking mode is selected by calling set_blocking()
before either of the above.
o SSL.File: Redesigned I/O state machine.
This reduces the complexity and risk of bugs in the I/O handling.
o SSL support for lots of new cipher suites added:
+ AEADs and modes:
- CCM and CCM-8
- GCM
+ Certificates
- ECDSA
+ Ciphers
- AES and AES-256
- CAMELLIA and CAMELLIA-256
+ Key exchange methods
- DH and DHE
- ECDH and ECDHE
+ All suites currently defined consisting of combinations of the
above (and earlier existing) have been added (~190 suites in
total).
o TLS Extensions added:
+ ALPN (Application Layer Protocol Negotiation) (RFC 7301).
+ EC Point Formats (RFC 4492).
+ Elliptic Curves (RFC 4492).
+ Encrypt then MAC.
+ Fallback SCSV.
+ Heartbeat (RFC 6520).
+ Max Fragment Length (RFC 6066).
+ Padding.
+ Renegotiation info (RFC 5746).
+ Signature Algorithms (RFC 5246).
+ SNI (Server Name Indicator) for both client and server (RFC 6066).
+ Truncated HMAC (RFC 6066).
o Improved protection against various SSL/TLS attacks:
+ BEAST protection (aka 1/(n-1)).
Improve resilience against the BEAST client-side attack, by
splitting the first data packet into two, where the first only
contains a single byte of payload.
+ Heartbleed protection.
The protocol code can probe the peer for the Heartbleed
vulnerability, and aborts the connection with
ALERT_insufficient_protection if so.
+ Lucky 13 protection.
Attempts to have HMAC calculation take the same amount of time
regardless of padding size.
o SSL.Context: Improved cipher suite selection:
+ Filtering of weak cipher suites.
+ Suite B (RFC 6460).
o SSL.Context: Support multiple concurrent certificates.
This allows a server to eg have both an RSA and an ECDSA
certificate.
Crypto and Nettle
-----------------
o Nettle refactored
CBC cipher mode is now twice as fast.
o Nettle 3.0 supported.
The new APIs in Nettle 3.0 are now detected and utilized.
o Crypto.GCM
GCM (Galois Counter Mode) cipher mode added.
o Blowfish and Serpent support fixed in Nettle
o Crypto.PGP
Added support for SHA256, SHA384 and SHA512 as hash functions.
Expose the used hash and key types in the out data
o Crypto.Arctwo
The 1-128 bit cipher Arctwo is now provided as a block cipher in
Crypto. This cipher is only intended for compatibility with OLD
third party code, and should NOT be used for new development.
o Crypto.Camellia
The 128/256 bit cipher Camellia is now available as block cipher in
Crypto.
* Crypto.ECC
Elliptic Curve operations are now supported when compiled
with Nettle 3.0 or later.
o Crypto.SALSA20 and Crypto.SALSA20R12
The 128/256 bit cipher SALSA20 is now available as a stream cipher
in Crypto. SALSA20R12 is SALSA20 reduced to just 12 rounds.
o Crypto.SHA3_224, Crypto.SHA3_256, Crypto.SHA3_384 and Crypto.SHA3_512
The SHA-3 secure hash algorithm has been added in multiple variants.
o Crypto.GOST94 and RIPEMD160
The lesser used hash functions GOST R 34.11-94 (RFC 4357) and
RIPEMD160 have been added.
o Crypto.RSA and Crypto.DSA
The key generation for RSA and DSA are now done by Nettle. This
results in 90% faster key generation for RSA. Key generation for DSA
is 10 times slower, but produces better quality keys.
o Crypto.Hash
Added support for pbkdf1 from PKCS#5v1.5 and pbkdf2 from PKCS#5v2.0.
o Crypto.Random
The underlying algorithm for Crypto.Random has changed from a overly
cautious Yarrow implementation to Fortuna on top of system random
sources. About 7000x faster than before.
o Standards.PEM
+ Added some support for encrypted PEM files
o Standards.X509
X509 was moved from Tools to Standards and has been refactored and
bug fixed. It is now possible to extend both validation and creation
of certificates with new cipher and hash algorithms. A range of new
algorithms are supported for both RSA and DSA:
RSA MD2
RSA MD5
RSA SHA-1
RSA SHA-2-256
RSA SHA-2-384
RSA SHA-2-512
DSA SHA-1
DSA SHA-2-224
DSA SHA-2-256
Note that the API has changed compared to Tools.X509 and there is
now a single make_selfsigned_certificate() method for both RSA and
DSA, though it takes the same arguments. In addition a hash function
and serial number can be supplied. The hash function defaults to
SHA-2-256.
Incompatible C-level API changes
--------------------------------
o New svalue layout
The most obvious change is that the macros TYPEOF() and SUBTYPEOF()
are now actually needed to directly access the type and subtype of
an svalue, the svalues no have 'type' and 'subtype' members.
There are also a few additional macros used to set both the type and
subtype of an svalue at the same time:
SVAL_SET_TYPE_SUBTYPE(sval,type,subtype) and
SVAL_SET_TYPE_DC(sval,type)
(these are only neede if you do not use the usual push_* macros)
They are useful because they (usually) compiles to significantly
more compact code, especially if the type and subtype are
compiletime constants. The _DC variant will either leave the subtype
unchanged or set it to 0, useful when you do not care about the
actual subtype (which is, really, most of the time).
o get_storage() returns void*
There is no need for casts in non-C++ code.
Building and installing
-----------------------
o -fvisibility=hidden is now the default
This means that PMOD_EXPORT is now actually needed on systems like
Linux and MacOSX. It also means that the binary is slightly smaller
and faster.
o clang compilation fixes (bignum overflow checks, misc)
It is now possible to compile Pike using a modern clang compiler.
o Removed bundles
Pike no longer comes with copies of some libraries, and the support
for that in the makefile has been removed.
o Several OS/2 and windows compilation fixes
o C89 assumed
The configure tests will not check for functions defined in C89
anymore.
Lots of bug fixes
-----------------
o Fixed symbol resolution with deep inherits and mixins
o Fixed PNG 4bpl indexed mode with alpha
o The _sprintf LFUN now works with %F
o foreach( arr[-two()..], string x), where two() returns 2 will no
longer iterate over the first element in the array three times or
crash.
o Postgres.postgres_result now properly decodes binary blobs and
strips trailing spaces from CHAR columns.
o Fixed a typo from 2001-05-05 that caused a lot of countries to
reside on the continent ',Europa' instead of "Europa" in
Geography.Countries.
Obviously the continent is not that often checked.
o Fixed interresting bug in the constant number parsing for octal
numbers in escapes.
For whatever reason 8:s were accepted for every location except the
first in octal numbers, so 078 was considered an alias for 0100.
This could cause issues at times, and will result in string parsing
being somewhat different:
| Pike v7.8 release 700 running Hilfe v3.5 (Incremental Pike Frontend)
| > "\078";
| (1) Result: "@"
| Pike v8.0 release 3 running Hilfe v3.5 (Incremental Pike Frontend)
| > "\078";
| (1) Result: "\a8"
o A lot more, see the (more or less) full changelog for more info:
http://pike-librarian.lysator.liu.se/index.xml?m=pike.git&start=forever&bra…
Regards,
--
Peter Bortas
