There is to much churn in 7.7 to get a release out any time soon if it doesn't stop. I've been tolerating the noice in Postgres because the person making the changes is the only person I know that uses Postgres, but it's time to freeze things now. What's more I'll start backing out changes that are not discussed, documented and added to the ChangeLog. Especially when fiddling with core APIs.
Specificly I'm considering backing out this right now:
http://pike.ida.liu.se/development/cvs/diff.xml?module=Pike&file=7.7/lib...
Discuss.
it may help to differentiate the type and extent of changes.
* incompatible changes to the api * additions to the core api * other additions * incompatible bug fixes * other bug fixes * security fixes * crash fixes * ...
then state which of these are allowed as of now, and what needs to be frozen before a release is possible. (i'd asume anything except security and crash fixes)
also nice would be to clarify which of these will be allowed for 7.8.x (after the 7.8 release)
i have a few hilfe tab-completion fixes which would fit in the other bug fixes category which i'd like to commit before the release, and i hope to be able to continue to work on hilfe during the 7.8 cycle (i still have ideas how to improve completion choices and more stuff that is potentially useful.)
greetings, martin.
it may help to differentiate the type and extent of changes.
- incompatible changes to the api
- additions to the core api
- other additions
- incompatible bug fixes
Discuss on list first. Oh, and I'd better say that refering to a discussion from more than a week ago is not valid.
- other bug fixes
- security fixes
- crash fixes
- ...
Be careful, but check what's needed. Obviously behavioural changes that you think are bugs should be discussed first.
then state which of these are allowed as of now, and what needs to be frozen before a release is possible. (i'd asume anything except security and crash fixes)
I don't expect security fixes to be an issue right now, but for all intents and purposes they are treated in the same category as crash fixes for a new release.
also nice would be to clarify which of these will be allowed for 7.8.x (after the 7.8 release)
Everything excepts incompatible changes to the API is always fair games for new point releases, but just with more care than in minor releases.
i have a few hilfe tab-completion fixes which would fit in the other bug fixes category which i'd like to commit before the release, and i hope to be able to continue to work on hilfe during the 7.8 cycle (i still have ideas how to improve completion choices and more stuff that is potentially useful.)
That is no problem adding to 7.8.x.
7.8 alpha 1 will the cut today, Monday 1500CEST. After that there is a fair chance 7.8/7.9 will be split off unless grubba and mast tells me they have a landslide of patches still to come.
On Mon, Jul 14, 2008 at 12:20:02AM +0000, Peter Bortas @ Pike developers forum wrote:
Discuss on list first. Oh, and I'd better say that refering to a discussion from more than a week ago is not valid.
what if no discussion is happening? should that be taken as acceptance for 7.7?
7.8 alpha 1 will the cut today, Monday 1500CEST. After that there is a fair chance 7.8/7.9 will be split off unless grubba and mast tells me they have a landslide of patches still to come.
the split off has not happened yet or did i miss it? with more questions coming up on what to commit i think having the split would be nice so that we can commit to 7.9 and then discuss whether something is good enough for 7.7/7.8 and backport.
doing the split doesn't even necesarily mean that 7.7 needs to move to 7.8, it could stay 7.7 until it's ready for release, but given pikes tradition that not every x.y.z is a release i guess it would also be fine if 7.8.0 is an alpha and the real release is 7.8.9 or some other value of z.
greetings, martin.
On Mon, Jul 14, 2008 at 12:20:02AM +0000, Peter Bortas @ Pike developers forum wrote:
Discuss on list first. Oh, and I'd better say that refering to a discussion from more than a week ago is not valid.
what if no discussion is happening? should that be taken as acceptance for 7.7?
Not really. Ping the discussion.
7.8 alpha 1 will the cut today, Monday 1500CEST. After that there is a fair chance 7.8/7.9 will be split off unless grubba and mast tells me they have a landslide of patches still to come.
the split off has not happened yet or did i miss it?
Nope, we decided to postpone it while grubba cleaned up some stuff. Then I barely got out the alpha before going on a 1v vacation.
with more questions coming up on what to commit i think having the split would be nice so that we can commit to 7.9 and then discuss whether something is good enough for 7.7/7.8 and backport.
No, that doesn't work. We have to be pretty sure there will be no major backports right after release.
doing the split doesn't even necesarily mean that 7.7 needs to move to 7.8, it could stay 7.7 until it's ready for release, but given pikes tradition that not every x.y.z is a release i guess it would also be fine if 7.8.0 is an alpha and the real release is 7.8.9 or some other value of z.
Sure, the release is unlikely to be called 7.8.0. That is not holding of a split. I'll be talking to grubba and marcus about it later today.
I'm still working out kinks and cleaning stuff up too, but I think that work can continue after the initial release. I've also just now, in blissful disregard of any release policy, committed a new GSS-API glue module. The reason is that Roxen requires it to be there.
Remaining things I intend to do are:
o Contributing my bit to CHANGES. o Fixing the http quoting in Protocols.HTTP since noone has objected.
I've also been working on Windows building and got a reasonably complete package there now (an exception is perhaps Postgres support). There's one problem here that needs to be sorted out before we can distribute any windows packages:
In Windows it's necessary to include most of the libs in the package. Since they commonly are LGPL I reckon that we must also make sure there's a source package (for the libs) available along with the binary installer. I haven't done anything yet to create such a package. It wouldn't be enough to simply have a list of download links to the respective web sites for the libs, right?
In Windows it's necessary to include most of the libs in the package. Since they commonly are LGPL I reckon that we must also make sure there's a source package (for the libs) available along with the binary installer. I haven't done anything yet to create such a package. It wouldn't be enough to simply have a list of download links to the respective web sites for the libs, right?
You are correct that it wouldn't be enough strictly speaking. We must be able to deliver the exact source packages used to produce the libs.
Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
complete package there now (an exception is perhaps Postgres support).
Actually, the pgsql driver I included will work fine under Windows. Its performance is lacking though, I'm in the process of moving some of the core functions to a CMOD though (but I need help there, hint, hint ;-), in order to get it on par with the libpq solution.
On Tue, Jul 22, 2008 at 03:40:02PM +0000, Peter Bortas @ Pike developers forum wrote:
with more questions coming up on what to commit i think having the split would be nice so that we can commit to 7.9 and then discuss whether something is good enough for 7.7/7.8 and backport.
No, that doesn't work. We have to be pretty sure there will be no major backports right after release.
i don't mean after the release but after the split and before the release. if something is supposed to go into 7.7/7.8 can it hurt if it ends up in 7.9 first?
greetings, martin.
And I misspoke too, it's the split that is the problem if it's something convoluted involving pike core. It's not that they can't be backported. They won't.
doing the split doesn't even necesarily mean that 7.7 needs to move to 7.8, it could stay 7.7 until it's ready for release, but given pikes tradition that not every x.y.z is a release i guess it would also be fine if 7.8.0 is an alpha and the real release is 7.8.9 or some other value of z.
Sure, the release is unlikely to be called 7.8.0. That is not holding of a split. I'll be talking to grubba and marcus about it later today.
grubba wants to hold off the split until the crash in the test suite on some machines is fixed in case array.c or some other suspect needs major surgery. That's also the reason for turning on excessive verbosity on the Xenofarm verification logs yesterday.
Peter Bortas @ Pike developers forum wrote:
backing out changes that are not discussed, documented and added to the ChangeLog. Especially when fiddling with core APIs.
Specificly I'm considering backing out this right now:
http://pike.ida.liu.se/development/cvs/diff.xml?module=Pike&file=7.7/lib...
I am/have been working on a Sql.pgsql module (it's nearly finished now), which specifically supports the full PostgreSQL native network API without libpq dependencies (it implements a superset of the existing Sql.postgres module; it allows (amongst other things) multiple simultaneous queries to be run on the same database connection).
The module allows for operation in both interactive and Backend/callout mode, but in order for it to support pushing back the part of the input stream that was already pulled out of the kernel network layer by the Backend, it needs to push back an arbitrary blob into the input stream, to be consumed again.
ungets() doesn't work for that, because it assumes a newline at the end of the pushed back buffer.
The only other option I have is to derive a special class from the Stdio.FILE class which does its own buffering (which is what I need to do if we want to back out the addition).
That's the information I'd like to see now _on the list_ before each checkin that changes behaviour. I know you are doing good work with the Postgres bindings, and I will continue to turn a blind eye to that.
The putback functionality is good to have, and I don't mind including it if I was sure "unget()" was the absolute best name for this function and the documentation was complete¹ and the ChangeLog was updated and this was generally accepted among other developers. Why not "unread()" which is less likely to be confused with *gets*()?
¹ @seealso is now incomplete for all the other related functions.
Peter Bortas @ Pike developers forum wrote:
That's the information I'd like to see now _on the list_ before each checkin that changes behaviour. I know you are doing good work with the Postgres bindings, and I will continue to turn a blind eye to that.
Well, I have fixed and brushed up the existing Postgres driver as much as I needed for the short term (it is stable in its current form and heavily tested in a production environment, I don't think I'll be making much changes there anymore in the future), I'm about to include an alternate driver (pgsql.pike) which is available as the second method to connect to PostgreSQL databases, and as it looks now, it is probably going to be the preferred method; over time the old driver could be deprecated perhaps.
The putback functionality is good to have, and I don't mind including it if I was sure "unget()" was the absolute best name for this function and the documentation was complete? and the ChangeLog was updated and this was generally accepted among other developers. Why
ChangeLog, didn't know I had to update that, but will do so after we've decided on what it is going to be called. The documentation, well, I've had some experience with that now, writing my own pgsql driver from "scratch", but not nearly enough, so I missed some of the fine-points there. I'll see what I can do to brush that up.
not "unread()" which is less likely to be confused with *gets*()?
You might be right. The reason why I didn't pick "unread" is because in English it is usually pronounced as "un-red" (meaning: not read yet), and we actually want "un-reed" (meaning: reverse the operation of read). Then again, maybe the alternate meaning of "not read yet" is not so bad either, because we declare that piece of string to be "unread" as it were. I'll change it to "unread", and wait for comments from the rest here.
? @seealso is now incomplete for all the other related functions.
I'll check it.
Stephen R. van den Berg wrote:
Peter Bortas @ Pike developers forum wrote:
That's the information I'd like to see now _on the list_ before each checkin that changes behaviour. I know you are doing good work with the Postgres bindings, and I will continue to turn a blind eye to that.
Well, I have fixed and brushed up the existing Postgres driver as much as I needed for the short term (it is stable in its current form and heavily tested in a production environment, I don't think I'll be making much changes there anymore in the future), I'm about to include an alternate driver (pgsql.pike) which is available as the second method to connect to PostgreSQL databases, and as it looks now, it is probably going to be the preferred method; over time the old driver could be deprecated perhaps.
I'll make the deadline for 15:00 today, to get the new driver included. It already has been tested quite reasonably (not in a production environment yet, though).
Stephen R. van den Berg wrote:
Stephen R. van den Berg wrote:
Peter Bortas @ Pike developers forum wrote: Well, I have fixed and brushed up the existing Postgres driver as much as I needed for the short term (it is stable in its current form and heavily tested in a production environment, I don't think I'll be making much changes there anymore in the future), I'm about to include an alternate driver (pgsql.pike) which is available as the second method to connect to PostgreSQL databases, and as it looks now, it is probably going to be the preferred method; over time the old driver could be deprecated perhaps.
The driver has been included, including ChangeLog updates. In profiling the driver, I notice a 50% slowdown compared to the libpq based driver. And while profiling the code, I notice that Pike somehow is spending *a lot* of time in regular calls to gettimeofday().
Anyone have any idea which part of Pike does that and why?
The backend does that, to process call-outs.
not "unread()" which is less likely to be confused with *gets*()?
You might be right. The reason why I didn't pick "unread" is because in English it is usually pronounced as "un-red" (meaning: not read yet), and we actually want "un-reed" (meaning: reverse the operation of read). Then again, maybe the alternate meaning of "not read yet" is not so bad either, because we declare that piece of string to be "unread" as it were. I'll change it to "unread", and wait for comments from the rest here.
I'll push up the cutting of the alpha 'till 1900 CEST to give people some more time to comment and to let me fix a blocker I'd forgotten about.
A new Subversion converted repository can be found as ~marcus/pike_svn.dump.gz on Pelix. It's missing a few of todays commits, so I'll start anouther round. This round took 22 hours.
That's the information I'd like to see now _on the list_ before each checkin that changes behaviour. I know you are doing good work with the Postgres bindings, and I will continue to turn a blind eye to that.
The putback functionality is good to have, and I don't mind including it if I was sure "unget()" was the absolute best name for this function and the documentation was complete¹ and the ChangeLog was updated and this was generally accepted among other developers. Why not "unread()" which is less likely to be confused with *gets*()?
I considered using Stdio.FILE for monitoring a logfile today, but I noticed that gets() (and ngets()) destroy their states when they reach EOF. Would anybody complain if I add an extra (optional) argument to them to signal that they shouldn't zap the buffers at EOF?
¹ @seealso is now incomplete for all the other related functions.
On Tue, Jul 22, 2008 at 04:20:02PM +0000, Peter Bortas @ Pike developers forum wrote:
Yes. I want a release this year.
i second that. even more, i want a release before the next pike conference (so that the conference can focus on future development and does not get distracted by trying to get a release out the door)
unlike klingons where software is not released but escapes, leaving a bloddy trail in its wake, releasing pike is more like trying to release a wild animal that has been raised in captivity. pike is so used to living in captivity that it just does not want to leave home and be released ;-)
greetings, martin.
As I said in a discussion yesterday: Any new bugs that people insist on finding are likely to soon be redefined as features.
On Tue, Jul 22, 2008 at 04:10:03PM +0000, Henrik Grubbstr�m (Lysator) @ Pike (-) developers forum wrote:
I considered using Stdio.FILE for monitoring a logfile today, but I noticed that gets() (and ngets()) destroy their states when they reach EOF.
you mean a program like tail can't be written with pike because of this?
Would anybody complain if I add an extra (optional) argument to them to signal that they shouldn't zap the buffers at EOF?
is it possible to add this functionality by subclassing Stdio.FILE?
greetings, martin.
On Tue, Jul 22, 2008 at 04:10:03PM +0000, Henrik Grubbstr�m (Lysator) @ Pike (-) developers forum wrote:
I considered using Stdio.FILE for monitoring a logfile today, but I noticed that gets() (and ngets()) destroy their states when they reach EOF.
you mean a program like tail can't be written with pike because of this?
Correct (or rather not with Stdio.FILE and utilizing gets() et al).
Would anybody complain if I add an extra (optional) argument to them to signal that they shouldn't zap the buffers at EOF?
is it possible to add this functionality by subclassing Stdio.FILE?
Not without reimplementing everything in it once again, since the relevant variables and functions are private.
On Tue, Jul 22, 2008 at 05:30:02PM +0000, Henrik Grubbstr�m (Lysator) @ Pike (-) developers forum wrote:
is it possible to add this functionality by subclassing Stdio.FILE?
Not without reimplementing everything in it once again, since the relevant variables and functions are private.
ok, here is where it gets interesting.
is it necessary for that to be private? is it necessary for anything in pike core to be private? wouldn't protected be enough?
the only reason i can see for something to be private is that it should not be part of the api, but that can be achived with using protected or optional as well. i don't think the pike user should be prevented from accessing anything in order to make use of it and override specific bits to change functionality.
greetings, martin.
I don't think private is very useful either. It can be useful in some special cases: E.g. when you implement a simple cache using a mapping and you have to do some locking (or rely on the interpreter lock). In that case it can be good to declare the mapping itself private, so that you _know_ that the only access to it is right there, through the lookup/set functions you provide. Those access functions are probably protected.
so, should we go through and remove private everywhere?
replace private with protected? (btw there are a number of "private protected" declarations which is kind of redundant)
or better "protected local" (since private implies local)?
or just local? or "local optional" (since not-public means it's not part of the class signature)?
is this change to late for 7.7? (probably)
greetings, martin.
Henrik Grubbstr?m (Lysator) @ Pike (-) developers forum wrote:
I considered using Stdio.FILE for monitoring a logfile today, but I noticed that gets() (and ngets()) destroy their states when they reach EOF. Would anybody complain if I add an extra (optional) argument to them to signal that they shouldn't zap the buffers at EOF?
Excuse me for asking, but what do you need the state for?
Basically, all that's needed is that a seek() resets the state into something that can be used.
Consider the following tail-f style code:
void check_for_new_lines(Stdio.FILE f) { while (string l = f->gets()) { // do something with l. } call_out(check_for_new_lines, 1, f); }
The above currently works fine as long as the file always ends with a line-feed, but if the last line hasn't been fully written to the file, gets() will still return the partial line as if it was complete, with no indication whether the line was partial or not. When the rest of the line is written, gets() will return the remainder, but not the start of the line. With my modified gets():
void check_for_new_lines(Stdio.FILE f) { while (string l = f->gets(1)) { // do something with l. } call_out(check_for_new_lines, 1, f); }
gets() will not return the partial line, but instead keep it in the buffer so that the next time it gets called the complete line can be returned.
why do you need that for tail-f? just write the partial line to the terminal and then continue with the rest. the actual probelm is that getsdoes not tell you if the line was partial and it removes the linebreak, so instead using a method to read the stream that doesn't remove linebreaks should work...
greetings, martin.
It seems better to me that gets solves those problems. Usually you *want* complete lines when you use gets - otherwise you'd use read or something similar; writing your own line splitter isn't particularly hard.
A common application is a side process log parser. Getting incomplete lines are useless.
In the process of combing through Stdio.FILE, I found 2 bugs (fixes already committed), and one small optimisation opportunity. Any objections against checking this one in as well (it basically moves some code around, to make getchar() inlinable)?
commit 9c6d0b0b46b53253119390a76883c962be3e2bf2 Author: Stephen R. van den Berg srb@cuci.nl Date: Mon Jul 14 12:47:28 2008 +0200
Make getchar() less filling, and inlinable
diff --git a/lib/modules/Stdio.pmod/module.pmod b/lib/modules/Stdio.pmod/module.pmod index 8edf78b..fa4aacd 100644 --- a/lib/modules/Stdio.pmod/module.pmod +++ b/lib/modules/Stdio.pmod/module.pmod @@ -1897,6 +1897,22 @@ class FILE bpos=0; }
+ private protected final int getchar_get_data() + { + b = ""; + bpos=0; + return low_get_data(); + } + + private protected final void getchar_updatelinecache() + { + if(sizeof(cached_lines)>lp+1 && sizeof(cached_lines[lp])) + cached_lines = ({cached_lines[lp][1..]})+({cached_lines[lp+1]}); + else + cached_lines = ({}); + lp=0; + } + //! This function returns one character from the input stream. //! //! @returns @@ -1905,16 +1921,13 @@ class FILE //! @note //! Returns an @expr{int@} and not a @expr{string@} of length 1. //! - int getchar() + inline int getchar() { - if(sizeof(b) - bpos <= 0 && !get_data()) + if(sizeof(b) - bpos <= 0 && !getchar_get_data()) return -1;
- if(sizeof(cached_lines)>lp+1 && sizeof(cached_lines[lp])) - cached_lines = ({cached_lines[lp][1..]})+({cached_lines[lp+1]}); - else - cached_lines = ({}); - lp=0; + if(sizeof(cached_lines)) + getchar_updatelinecache();
return b[bpos++]; }
Just a note: There's no reason to use both private and protected at the same time. private is strictly more restrictive, so it implies protected.
pike-devel@lists.lysator.liu.se
-
Henrik Grubbstr�m (Lysator) ＠ Pike (-) developers forum -
Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) ＠ Pike (-) developers forum
-
Martin Baehr -
Martin Bähr
-
Martin Stjernholm, Roxen IS ＠ Pike developers forum
-
Mirar ＠ Pike developers forum
-
Peter Bortas ＠ Pike developers forum
-
Stephen R. van den Berg