Le samedi, 25 jan 2003, à 21:24 Europe/Paris, Martin Baehr a écrit :

On Sat, Jan 25, 2003 at 08:30:01PM +0100, David Hedbor @ Pike developers forum wrote:

...

So what was wrong with the bzip2 module that was in Pexts since a few years back, out of curiosity?

the fact that nobody took the code and inserted it into the main repository. i don't think that allowing pexts to be added to pike is enough, those who wrote pext need to take the code actively merge it.

it's not going to happen by itself...

So maybe we (me, david, or grendel since we have rw access to pike ?) can add all of our modules to 7.5 tree ?

The question is : what is the policy about it ?

For example I'd like to add OpenSSL module there is in pexts since it is working better with current openssl (tested only on FreeBSD's integrated SSL...) and the Ssleay module we currently have.

... my 0,02€ :p

/Xavier

Why not? The ssleay module has been considered junk for a long time afaik.

/ Martin Stjernholm, Roxen IS

Previous text:

2003-01-26 01:03: Subject: Re: Bz2

---

Le samedi, 25 jan 2003, à 21:24 Europe/Paris, Martin Baehr a écrit :

...

On Sat, Jan 25, 2003 at 08:30:01PM +0100, David Hedbor @ Pike developers forum wrote:

...

So what was wrong with the bzip2 module that was in Pexts since a few years back, out of curiosity?

the fact that nobody took the code and inserted it into the main repository. i don't think that allowing pexts to be added to pike is enough, those who wrote pext need to take the code actively merge it.

it's not going to happen by itself...

So maybe we (me, david, or grendel since we have rw access to pike ?) can add all of our modules to 7.5 tree ?

The question is : what is the policy about it ?

For example I'd like to add OpenSSL module there is in pexts since it is working better with current openssl (tested only on FreeBSD's integrated SSL...) and the Ssleay module we currently have.

... my 0,02€ :p

/Xavier

/ Brevbäraren

What would be so irreversible by adding another one or two before that's done? The only issue I can see is the name that gets allocated. A name like Protocols.OpenSSL is perhaps not be the best since the module might not fit snuggly API-wise with the rest, but it could hardly get in the way of anything else.

/ Martin Stjernholm, Roxen IS

Previous text:

2003-01-26 01:17: Subject: Re: Bz2

---

We have been quite reluctant to add new modules recently, since the standard Pike module collection is bit on the big side. The Bz2 module was added more as a kind gesture to Andreas Finnman, who has been working on an evalutation of Pikes module interface as master thesis.

I think the general feeling is that there should be no more modules before we have a proper module system. Unfortunately I think there is quite a bit of work that has to go into designing and implementing a good module system. And that needs skilled and coordinated people that can actually do it.

/ Martin Nilsson (Åskblod)

And as for PExts, there's also the question about licensing.

/ Leif Stensson, Lysator

Previous text:

2003-01-26 01:17: Subject: Re: Bz2

---

We have been quite reluctant to add new modules recently, since the standard Pike module collection is bit on the big side. The Bz2 module was added more as a kind gesture to Andreas Finnman, who has been working on an evalutation of Pikes module interface as master thesis.

I think the general feeling is that there should be no more modules before we have a proper module system. Unfortunately I think there is quite a bit of work that has to go into designing and implementing a good module system. And that needs skilled and coordinated people that can actually do it.

/ Martin Nilsson (Åskblod)

...

...

And as for PExts, there's also the question about licensing.

this question has been answered a long time ago.

Has it? Or, perhaps it has been "answered", but as far as I know, it hasn't been _resolved_: the actual written permissions in a legally valid form has not been obtained yet. It has been speculated that it is "probably" or "almost sure" that all the people involved will give such permissions, but I haven't seen it happen.

But feel free to help making it happen, though. The papers should sign over the copyright to PELAB, and allow PELAB to transfer it to any other organisation, if that becomes desirable. "PELAB" should be spelled out as "PELAB, The Department of Computer Science, Linköping University, Sweden". (IDA is the whole CS department; PELAB the subdepartment that deals with programming languages and environments.)

Make sure to include a list of what stuff it is that the person signing the document has written.

It's probably a good idea to wait a few days for other people's comments about the exact phrasing that should be used in the document, in case I've forgotten something.

Once ready, mail the signed document to:

ATTN: Uwe Aßmann PELAB Department of Computer Science Linköping University SE-581 83 LINKÖPING SWEDEN

/ Leif Stensson, Lysator

Previous text:

2003-01-28 10:57: Subject: Re: Bz2

---

this question has been answered a long time ago.

/ Brevbäraren

I don't really know why we have kept the Ssleay module. I doubt it even works. What's wrong with the SSL module, or more to the point, what is better with OpenSSL?

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 01:03: Subject: Re: Bz2

---

Le samedi, 25 jan 2003, à 21:24 Europe/Paris, Martin Baehr a écrit :

...

On Sat, Jan 25, 2003 at 08:30:01PM +0100, David Hedbor @ Pike developers forum wrote:

...

So what was wrong with the bzip2 module that was in Pexts since a few years back, out of curiosity?

the fact that nobody took the code and inserted it into the main repository. i don't think that allowing pexts to be added to pike is enough, those who wrote pext need to take the code actively merge it.

it's not going to happen by itself...

So maybe we (me, david, or grendel since we have rw access to pike ?) can add all of our modules to 7.5 tree ?

The question is : what is the policy about it ?

For example I'd like to add OpenSSL module there is in pexts since it is working better with current openssl (tested only on FreeBSD's integrated SSL...) and the Ssleay module we currently have.

... my 0,02€ :p

/Xavier

/ Brevbäraren

If major things are planned for the module system in 7.5, then how about adding this to 7.4? This is, after all, not core-pike functionality so I don't see this as an enormous problem.

If/when (allways the pessimist) the new module system is in place, all modules probably need modifying anyway, right?

/ Peter Lundqvist (disjunkt)

Previous text:

2003-01-26 01:45: Subject: Re: Bz2

---

Le dimanche, 26 jan 2003, à 01:25 Europe/Paris, Martin Nilsson (Åskblod) @ Pike (-) developers forum a écrit :

...

I don't really know why we have kept the Ssleay module. I doubt it even works.

For me it segfault.... on FreeBSD. I can add a pikefarm target for you if you'd like..

...

What's wrong with the SSL module, or more to the point, what is better with OpenSSL?

The OpenSSL module that is in the Pexts was submitted in pike@roxen.com ml. It is just a wrapper to OpenSSL calls...

So it can be used very easyly with any openssl implementation.

Now, what do we do ? Stopping adding modules ? Making an hypotical CPAN like for Pike, or just throw all work that have been done we some interressed ppl in Pike ?

This is in general the question I am asking to myself when I have sometimes an idea to add some code to pike... (Licensing issue is not for my code a priority since I give all rights to IDA about it).

Here is some code I really like to add in pike :

o Protocols.POP3.client (pike made) o Protocols.SMTP.server (pike made) (that can be used as Protocols.LMTP.server as well). o PCRE module (C made) o OpenSSL module o Mhash module

/Xavier

...

/ Martin Nilsson (Åskblod)

Previous text:

...

2003-01-26 01:03: Subject: Re: Bz2

---

Le samedi, 25 jan 2003, à 21:24 Europe/Paris, Martin Baehr a écrit :

...

On Sat, Jan 25, 2003 at 08:30:01PM +0100, David Hedbor @ Pike developers forum wrote:

...

So what was wrong with the bzip2 module that was in Pexts since a few years back, out of curiosity?

the fact that nobody took the code and inserted it into the main repository. i don't think that allowing pexts to be added to pike is enough, those who wrote pext need to take the code actively merge it.

it's not going to happen by itself...

So maybe we (me, david, or grendel since we have rw access to pike ?) can add all of our modules to 7.5 tree ?

The question is : what is the policy about it ?

For example I'd like to add OpenSSL module there is in pexts since it is working better with current openssl (tested only on FreeBSD's integrated SSL...) and the Ssleay module we currently have.

... my 0,02€ :p

/Xavier

/ Brevbäraren

/ Brevbäraren

For me it segfault.... on FreeBSD. I can add a pikefarm target for you if you'd like..

Only if someone intends to fix it. It is not part of a default build because it is not maintained (or used). Otherwise removing it in the 7.5 tree sounds like a good idea to me.

...

What's wrong with the SSL module, or more to the point, what is better with OpenSSL?

The OpenSSL module that is in the Pexts was submitted in pike@roxen.com ml. It is just a wrapper to OpenSSL calls... So it can be used very easyly with any openssl implementation.

That's not an answer to my question.

Now, what do we do ? Stopping adding modules ? Making an hypotical CPAN like for Pike, or just throw all work that have been done we some interressed ppl in Pike ?

Hypotical? Yes, as I said I believe that making a module system for Pike is the correct path to take. I have already presented an idea for how such a system could look like (on more than one occasion I think), but so far consensus hasn't been made.

Think of the module tree as an Internet domain name hierarchy. Currently the top domain and everything under it is adnimistrated and managed by IDA. My idea is that every subbranch should be open for people to become manager for. As an example I've said that perhaps some caudium guy would like the top module PExt and everything under it. He in turn could delegate parts of that namespace further down.

This framework would provide a way of administrating the module tree in a way that has the potential to be both flexible and "free" while still preventing namespace mayhem, as has been seen in other langauges.

Now, using the same tree of delegation we could make a tree of trust, where every source code, and possibly binary, can be signed by the module owner and his key is signed by the module owner over him.

Again using the same tree of delegation, the location of the modules can be distributed. When someone tries to download the PExt.Foo.Bar from IDA, the server can just send a referrer to the lowest known module owner site in the module path. Eg. for PExt probably somewhere on a caudium site. With a well defined method for listing modules it is easy to create a "module browser" where you can find modules you didn't know about.

(And why do you propose throwing work away as an option?)

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 01:59: Subject: Re: Bz2

---

If major things are planned for the module system in 7.5, then how about adding this to 7.4? This is, after all, not core-pike functionality so I don't see this as an enormous problem.

If/when (allways the pessimist) the new module system is in place, all modules probably need modifying anyway, right?

/ Peter Lundqvist (disjunkt)

can you provide examples of that mayhem in other languages?

I would say that this appears in pretty much every language that is freely extended. Perhaps C is one of the better examples where there are many lib* with the same name, so you'll have to probe deeper into the "module" to see if it is the one you thought. Many manhours has been spent on solving name clashes in the Pike C source.

...

(And why do you propose throwing work away as an option?)

it's not an option but a possible result if it takes to long to to get some kind of integration of existing code.

So people are that desperate to avoid having a nice way to handle modules?

see bz2: work has been trown away here already.

No. The bz2 module is a by product. The bz2 PExt was know before the thesis work begun, and it has been examined by Andreas.

now, given that the current contributers already have write access to the pike tree, could we maybe start with the factual delegation of nodes by simply creating them somewhere in the pike tree, and then let people write in there, while the implementation of the other necessary features is done in paralell?

It is of course a possibility.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 06:09: Subject: Re: Bz2

---

On Sun, Jan 26, 2003 at 02:40:01AM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

This framework would provide a way of administrating the module tree in a way that has the potential to be both flexible and "free" while still preventing namespace mayhem, as has been seen in other langauges.

can you provide examples of that mayhem in other languages?

...

(And why do you propose throwing work away as an option?)

it's not an option but a possible result if it takes to long to to get some kind of integration of existing code.

see bz2: work has been trown away here already.

now, given that the current contributers already have write access to the pike tree, could we maybe start with the factual delegation of nodes by simply creating them somewhere in the pike tree, and then let people write in there, while the implementation of the other necessary features is done in paralell?

greetings, martin.

/ Brevbäraren

Nothing wrong with that.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 12:51: Subject: Re: Bz2

---

I like the design. However I dislike the amount of implementation necessary to start adding new modules (or moving the PExt modules).

I propose a simpler implementation:

o Decide on a suitable module tree o Decide who is responsible for what parts of that tree o Document this on a web page o Make sure everyone has sufficient access permissions to the cvs o Hope that people can communicate with each other and solve problems o Use the existing packaging and distribution systems

That way you can work on new modules and on the new very cool module system in parallel. Serializing those two processes seems to be a sure way to kill of progress.

/ Mattias Wingstedt (Firefruit)

The problem I'm solving is administration and hosting. You might not percive the convention to put all modules in a big centrally administrated and controlled repository as a problem, but I do. I don't think it is a good idea to create a playground-module namespace, because stuff that gets in there will most likely stick. Other than your second item I don't see that you are contradicting my suggestion in any way.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 18:57: Subject: Re: Bz2

---

Your domain name idea seems like a solution that seeks to fit a problem rather than the reverse. I doubt it's good that an organizational/delegation structure is reflected in the module namespace. That will most likely lead to names that are unnecessarily long from a programmers perspective, and it's easy to do better if there is any sort of central coordination.

The issues that I see currently are:

o Keeping a well designed core library without redundancy and inconsistency. Redundancy is avoided by having a good overview over the current modules so that we readily can point out existing modules when a new one is considered and suggest extending them instead.

Inconsistency is avoided by writing detailed guidelines for libraries and providing a framework to adhere to for basic stuff like collection handling, serialization etc. We already got the framework for serialization and basic collection/iterator stuff (and a more elaborate class hierarchy for it is on the way). What we're lacking is above all an exception system, but also a good interface for streams (both for I/O and internal use).

o Make it fairly easy to add new modules. This can easily conflict with the high standards above. I think a single namespace for all such "uncontrolled" modules is enough. Why not let that area be called "PExts" and plop in the PExts stuff there to begin with?

It should be easy to allocate a name there, so that anyone can get going with a new module without having to raise the interest of some unspecified core developer. I.e. avoid the all too common situation when a request only is met by silence.

The intention should be that this is only a staging area for unfinished modules and that the goal is to extend the core modules instead. It's much easier to use a feature implemented in a consistent way with the rest of the module library than one that exists by itself.

o Improve the C level API. I.e. more docs, make it easier to compile outside the pike tree, reorganize the header files so that the things that modules need are separated from the internal core stuff, etc.

/ Martin Stjernholm, Roxen IS

I think the start should be to make an easy way of building and installing third party modules, for instance with simple tools and pre-made "aclocal.m4" macros.

Namespace problems and controlled repositories should be second, shouldn't it?

There should also be tools to take a bunch of modules and combine with the core to make a pike distribution. I think that the Pike core and the non-necessary modules (Protocols*, Image, etc) could be considered separate projects with their own version numbering.

/ Mirar

Previous text:

2003-01-26 19:24: Subject: Re: Bz2

---

The problem I'm solving is administration and hosting. You might not percive the convention to put all modules in a big centrally administrated and controlled repository as a problem, but I do. I don't think it is a good idea to create a playground-module namespace, because stuff that gets in there will most likely stick. Other than your second item I don't see that you are contradicting my suggestion in any way.

/ Martin Nilsson (Åskblod)

I don't know what that does. (I've never wanted to compile a separate module.) I'm not claiming the problem isn't solved already. :-)

How about a separate installation tree, like site-lisp (site-pike?), that doesn't get erased with installing new pike revisions?

/ Mirar

Previous text:

2003-01-26 19:39: Subject: Re: Bz2

---

...

I think the start should be to make an easy way of building and installing third party modules, for instance with simple tools and pre-made "aclocal.m4" macros.

Doesn't `pike -x module' make it fairly easy already? What's missing?

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Sounds like a good idea. Maybe not for C modules (which may require recompiling to be used with a new pike version), but for undumped pure pike modules (with appropriate #pike declarations).

Maybe C modules just should be tagged for what Pike module API version they work with. It doesn't change that often (hopefully). It could be in the filename - "Image.so.api-7.5a" :)

/ Mirar

Previous text:

2003-01-26 19:46: Subject: Re: Bz2

---

...

I don't know what that does. (I've never wanted to compile a separate module.) I'm not claiming the problem isn't solved already. :-)

It compiles a module in a separate directory. You don't need the main build-tree of the pike, just an installed pike.

...

How about a separate installation tree, like site-lisp (site-pike?), that doesn't get erased with installing new pike revisions?

Sounds like a good idea. Maybe not for C modules (which may require recompiling to be used with a new pike version), but for undumped pure pike modules (with appropriate #pike declarations).

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

B?

The naming idea was to let more then one Pike be installed on the same system, using the same site-pike. But that might be overambitious. ^.^

A libpike.so is a good idea. Should the pike binary use it as well?

/ Mirar

Previous text:

2003-01-26 20:02: Subject: Re: Bz2

---

Actually, dynamic C modules should probably be linked against libpike.so.4711, then they will cease to load with an incompatible libpike.so as long as the major version number is bumped for each incompatible ABI change. Of course, for this to happen we first need to have A) a libpike.so, and B) an ABI. :-)

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Yes, but since I don't know anything about how to handle C-code in a sane way I have not brought it up. What about the "pike -x module" that Comstedt did?

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 19:35: Subject: Re: Bz2

---

I think the start should be to make an easy way of building and installing third party modules, for instance with simple tools and pre-made "aclocal.m4" macros.

Namespace problems and controlled repositories should be second, shouldn't it?

There should also be tools to take a bunch of modules and combine with the core to make a pike distribution. I think that the Pike core and the non-necessary modules (Protocols*, Image, etc) could be considered separate projects with their own version numbering.

/ Mirar

aclocal.m4 contains AC_MODULE_INIT(). That's all that is necessary.

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Previous text:

2003-01-26 19:47: Subject: Re: Bz2

---

For instance an aclocal.m4 that actually contains the macros necessary for separate modules (see Bill Wellivers recent message on the Pike list). It's not clear that it should be the same one that pike uses.

A good start is probably to actually split off some modules as Mirar suggested and then build the tools to make it work, if any are necessary.

/ Martin Stjernholm, Roxen IS

I don't know. I haven't looked at it. Maybe it's trying to solve something other than separate compilation of pike modules. I hope it is, since that problem has already been solved.

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Previous text:

2003-01-26 19:56: Subject: Re: Bz2

---

So Bills effort has no use?

/ Martin Stjernholm, Roxen IS

9634015 talks about "packages that require pike", which presumably means other things than pike modules. In that case it might be something useful, although I can't at the moment think of anything that a simple AC_PATH_PROG(pike) wouldn't solve...

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Previous text:

2003-01-26 20:06: Subject: Re: Bz2

---

He made a short description of it in 9634015. I'm not familiar with the issue, so I can't say whether it solves the same problem.

/ Martin Stjernholm, Roxen IS

Yes, you are right. I'd like to have the solution before the problem arises. It's not a solution looking for a problem, its a solution to avoid the problem.

If I understood you reasoning you wanted to make name management of major parts of the tree a "global activity", and to reduce noise for everyone letting users put modules effortlessly in a subtree.

I want to have management of the core parts of the tree as a collective activity for the core developers and reducing noise by outsourcing management of subtrees with much activity/developer interest. This would hopefully bring committers closer to code managers that knows and cares about the application area at hand.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 19:57: Subject: Re: Bz2

---

No, I tried to describe the problem at hand to begin with. And no, I don't see the need to delegate namespace administration as a problem at hand; we're not anywhere near such a large community yet. Thus my impression that it was more like a solution seeking a problem.

What's necessary is only a registrar for names. IDA doesn't have to provide hosting although it could be nice to offer that for interesting projects.

...

I don't think it is a good idea to create a playground-module namespace, because stuff that gets in there will most likely stick.

How does namespace delegation affect this? I understand that hosting will affect it, or more specifically, what gets included in the standard dists. I didn't touch that issue either.

/ Martin Stjernholm, Roxen IS

o Protocols.POP3.client (pike made) o Protocols.SMTP.server (pike made) (that can be used as Protocols.LMTP.server as well). o PCRE module (C made) o OpenSSL module

I wouldn't mind that, I think. If there will be a new module repository, I'm sure several modules already in the Pike tree would be split away from the main branch anyway (Image, for instance).

o Mhash module

What's that? Is it something that belongs under Crypto?

/ Mirar

Previous text:

2003-01-26 01:45: Subject: Re: Bz2

---

Le dimanche, 26 jan 2003, à 01:25 Europe/Paris, Martin Nilsson (Åskblod) @ Pike (-) developers forum a écrit :

...

I don't really know why we have kept the Ssleay module. I doubt it even works.

For me it segfault.... on FreeBSD. I can add a pikefarm target for you if you'd like..

...

What's wrong with the SSL module, or more to the point, what is better with OpenSSL?

The OpenSSL module that is in the Pexts was submitted in pike@roxen.com ml. It is just a wrapper to OpenSSL calls...

So it can be used very easyly with any openssl implementation.

Now, what do we do ? Stopping adding modules ? Making an hypotical CPAN like for Pike, or just throw all work that have been done we some interressed ppl in Pike ?

This is in general the question I am asking to myself when I have sometimes an idea to add some code to pike... (Licensing issue is not for my code a priority since I give all rights to IDA about it).

Here is some code I really like to add in pike :

o Protocols.POP3.client (pike made) o Protocols.SMTP.server (pike made) (that can be used as Protocols.LMTP.server as well). o PCRE module (C made) o OpenSSL module o Mhash module

/Xavier

...

/ Martin Nilsson (Åskblod)

Previous text:

...

2003-01-26 01:03: Subject: Re: Bz2

---

Le samedi, 25 jan 2003, à 21:24 Europe/Paris, Martin Baehr a écrit :

...

On Sat, Jan 25, 2003 at 08:30:01PM +0100, David Hedbor @ Pike developers forum wrote:

...

So what was wrong with the bzip2 module that was in Pexts since a few years back, out of curiosity?

the fact that nobody took the code and inserted it into the main repository. i don't think that allowing pexts to be added to pike is enough, those who wrote pext need to take the code actively merge it.

it's not going to happen by itself...

So maybe we (me, david, or grendel since we have rw access to pike ?) can add all of our modules to 7.5 tree ?

The question is : what is the policy about it ?

For example I'd like to add OpenSSL module there is in pexts since it is working better with current openssl (tested only on FreeBSD's integrated SSL...) and the Ssleay module we currently have.

... my 0,02€ :p

/Xavier

/ Brevbäraren

/ Brevbäraren

On Sun, Jan 26, 2003 at 01:25:02AM +0100, Martin Nilsson (�skblod) @ Pike (-) developers forum wrote:

What's wrong with the SSL module, or more to the point, what is better with OpenSSL?

What is wrong with SSL: it simply doesn't work (Pike 7.4.10 stock) - a lot of unresolved references. It was fixed somewhere in 7.4.13 and in 7.4.15 it is broken again. So - it is unstable, at least.

Additionally, it doesn't provide all the functionality which is included in OpenSSL, and, again, OpenSSL is long standing, stable project, proven.

A lot of crypto stuff included in OpenSSL is far better and more optimized comparing to original Pike stuff - this is the major point, I think.

Personally, I don't think that this is good idea - to implement in Pike everything just because it can be implemented. Some things are quite ineffective in Pike, even when JIT and optimizer are in use.

Everything which needs speed _must_ be implemented in C, everything else _may_ be implemented in Pike. IMHO, of course :)

Regards, /Al

One obvious disadvantage with OpenSSL though is that it is written in C, and thus is more likely to have bugs causing security holes.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-28 00:41: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

SSL has been stable for years now. Far longer than OpenSSL has been stable or even existing. If it got dented when a lot of stuff was rearranged in preparation for 7.4 it should be fixed.

I want to hear what is better in OpenSSL, not some general fuzzy feelings about going with the flow. Last time I checked the proto-OpenSSL code - several years ago mind you - it was so damned clutteded that I wouldn't trust sending my cats name over it. Niels code on the other hand, while not always easy to follow, is rather clean.

OpenSSL is faster than Pikes SSL module. That is known. I want to know about other differances in OpenSSLs advantage.

/ Peter Bortas

Le mardi, 28 jan 2003, à 08:34 Europe/Paris, Alexander Demenshin a écrit :

On Tue, Jan 28, 2003 at 01:20:01AM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

One obvious disadvantage with OpenSSL though is that it is written in C, and thus is more likely to have bugs causing security holes.

Also the fact that we should support OpenSSL should help us to support hardware crypto cards that for example OpenBSD supports.

This will allows Roxen users and Caudium users to have intensive SSL3 webservers on their machine with such cards. As Caudium maintainer I was sometimes asked if we support hardware accelerators... My reply was always no because of this reasons.

On other hand, the fact that Pike didn't depends to OpenSSL saved us (Roxen & Caudium) our time in the last bugs / problems we had recently and that was good.

I don't ask to replace Pike SSL with OpenSSL API... I just like to add a separate Cmod and the programmer will make the difference...

/Xavier

-- Xavier Beaudouin - Unix System Administrator & Projects Leader. Please visit http://caudium.net/, home of Caudium & Camas projects O ascii ribbon campaign against html email |\ and Microsoft attachments

But memory leaks are not usually a security risk, and they are not invisible bugs.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-28 08:35: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 01:20:01AM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

One obvious disadvantage with OpenSSL though is that it is written in C, and thus is more likely to have bugs causing security holes.

One obvious disadvantage with Pike is that it is used by humans, which tend to make mistakes :) "Guns don't kill people, people kill people".

There is no (and will never be) any _safe_ language, until there are a lot of "unsafe" programmers around. :)

Proper code can be written in C, in Perl, even in asm - if you know what you are doing. Yes, it is extremely difficult (if ever possible) to leave (or create) a buffer overflov like hole in Pike app, but it is quite easy to leave another hole (like memory leak - where objects are never completely dereferenced so GC won't help).

Regards, /Al

/ Brevbäraren

As a note which is unrelated to OpenSSL, I would like to point out that Roxen with SSL (and, I assume Caudium as well or to be exact a webserver using the Pike SSL implementation) performs absolutely horribly. Some time ago (a year or two perhaps - scale hardware to what was the norm back then) Real Networks did a benchmark on SSL and Roxen and the speed was something along the lines of 5-10 requests per second for normal small files (which is too slow even for small/moderate sites) - this compared to 150 req/sec or more for a hardware SSL accelerator (not a fair comparision but still).

One problem with the SSL code in Pike is that, as far as I know, it never has been benchmarked or optimized by developers. Because of that the bad performance is mostly an unknown issue.

/ David Hedbor

Previous text:

2003-01-28 08:35: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 01:20:01AM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

One obvious disadvantage with OpenSSL though is that it is written in C, and thus is more likely to have bugs causing security holes.

One obvious disadvantage with Pike is that it is used by humans, which tend to make mistakes :) "Guns don't kill people, people kill people".

There is no (and will never be) any _safe_ language, until there are a lot of "unsafe" programmers around. :)

Proper code can be written in C, in Perl, even in asm - if you know what you are doing. Yes, it is extremely difficult (if ever possible) to leave (or create) a buffer overflov like hole in Pike app, but it is quite easy to leave another hole (like memory leak - where objects are never completely dereferenced so GC won't help).

Regards, /Al

/ Brevbäraren

I don't even know if there, today, is a 'ab' type application with SSL support. Is there? Also, this was on, probably, something along the lines of a P3-500 or so.

/ David Hedbor

Previous text:

2003-01-28 19:35: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Interesting, but I've suspected that SSL wasn't overly fast. Is it possibly to add an SSL test to the benchmarks?

Most of the work should be done in C and Gmp, so I suspect there isn't too much that can be more optimized...

/ Mirar

Pär Svensson drew the same conclusion (that upgrading GMP would be good) in this thesis work:

http://www.ep.liu.se/exjobb/isy/2002/3308/ http://www.ep.liu.se/exjobb/isy/2002/3308/exjobb.pdf

/ Johan Schön (Firefruit)

Previous text:

2003-01-28 19:36: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

It would be interesting with a current benchmark. At least GMP has undergone heavy optimization in recent years, which matters if it's the rsa computations that are the bottleneck.

/ Niels Möller ()

On Tue, Jan 28, 2003 at 12:45:02AM +0100, Peter Bortas @ Pike developers forum wrote:

SSL has been stable for years now. Far longer than OpenSSL has been stable or even existing.

_Now_ the situation is a bit different, isn't? I agree that in the past it was better (perhaps), but since then many things are changed.

I want to hear what is better in OpenSSL, not some general fuzzy feelings about going with the flow.

Should I post OpenSSL's API description here? And compare to existing Pike stuff? Tell what I need and what I miss?

Last time I checked the proto-OpenSSL code - several years ago mind you -

Serveral _years_. While _current_ Pike's source (no offence) is still (sometimes) very hairy and dirty :)

OpenSSL is faster than Pikes SSL module. That is known. I want to know about other differances in OpenSSLs advantage.

Presence of all popular (and standard) hashing functions and ciphers, for instance. Again - should I compare function-by-function Pike and OpenSSL APIs concerning crypto stuff?

Regards, /Al

On Tue, Jan 28, 2003 at 12:25:07PM +0100, Peter Bortas @ Pike developers forum wrote:

No it isn't. I find Niels code overly object oriented, but not at all hairy and definately not dirty.

I've a different opinion on this point, but that's different story, I guess :)

As I said, if you are missing functionality, state what it is.

OK, I'll. I'll need some time though...

Regards, /Al

I'd still love some Nettle glue, or similar. Alexander does have a point, even if it's hidden among trolls; the Pike SSL code has not seen much tuning over the years, which other crypto toolkits have. Where most other parts have matured over the years, got harmonized naming conventions and evolved with the rest of Pike, Crypto still is pretty much the same, for good and bad.

/ Johan Sundström (a hugging punishment!)

Previous text:

2003-01-28 09:51: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Well, ssleay did exist when I wrote the SSL code, of course. But it was huge and really really ugly. Besides the general uglyness, another showstopper back then was that there was no (documented) way of running it in non-blocking mode.

Openssl has probably gotten better in recent years, but I haven't had any reason to look closely at it. As far as I understand, from talking to openssl developers, I think the API is still more or less broken for non-blocking applications.

/ Niels Möller ()

What is wrong with SSL: it simply doesn't work (Pike 7.4.10 stock) - a lot of unresolved references. It was fixed somewhere in 7.4.13 and in 7.4.15 it is broken again. So - it is unstable, at least.

Well, I've got a nicely working SSL in Pike 7.4.10 and 7.4.15 here, so it sounds to me that you are doing something wrong. You can begin with explaining what you mean with "unresolved references" since that is not an error usually mentioned when discussing Pike problems. (Though it appears when you create the manual, but that just means broken link (a href).)

Additionally, it doesn't provide all the functionality which is included in OpenSSL, and, again, OpenSSL is long standing, stable project, proven.

A lot of crypto stuff included in OpenSSL is far better and more optimized comparing to original Pike stuff - this is the major point, I think.

I want facts, features, figures or I'll disregard from all these arguments except the speed one, which I have already seen measurements and charts for.

Personally, I don't think that this is good idea - to implement in Pike everything just because it can be implemented. Some things are quite ineffective in Pike, even when JIT and optimizer are in use.

Everything which needs speed _must_ be implemented in C, everything else _may_ be implemented in Pike. IMHO, of course :)

The lower langauge the greater potential for high speed and low memory footprint, but at the same time greater portability problems and easier to make harmful bugs. Implementing crypto algorithms in C is a good idea. Implementing low level data management such as ASN1 in C might be a good idea. Implementing SSL in C is probably a bad idea.

Also do note the word 'potential' in the last paragraph. Pike has outperformed C code in several benchmarks because the naive implementation in Pike (which utilizes optimized algorithms deeper down) is faster than the naive implementation in C.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-28 00:15: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Sun, Jan 26, 2003 at 01:25:02AM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

What's wrong with the SSL module, or more to the point, what is better with OpenSSL?

What is wrong with SSL: it simply doesn't work (Pike 7.4.10 stock) - a lot of unresolved references. It was fixed somewhere in 7.4.13 and in 7.4.15 it is broken again. So - it is unstable, at least.

Additionally, it doesn't provide all the functionality which is included in OpenSSL, and, again, OpenSSL is long standing, stable project, proven.

A lot of crypto stuff included in OpenSSL is far better and more optimized comparing to original Pike stuff - this is the major point, I think.

Personally, I don't think that this is good idea - to implement in Pike everything just because it can be implemented. Some things are quite ineffective in Pike, even when JIT and optimizer are in use.

Everything which needs speed _must_ be implemented in C, everything else _may_ be implemented in Pike. IMHO, of course :)

Regards, /Al

/ Brevbäraren

Not to mention that people working on OpenSSL have experience in crypto related areas, while this is not main focus of Pike team. Sorry to say, but I don't completely trust in crypto stuff which is not implemented by someone who has a track in this area already - the IT security is my business, I performed several security projects so I know what I am talking about. To read the spec and to implement it is often not enough.

Why would a programer be better at writing crypto code if he has done a lot of it already? You obviously don't appricate the complexity of crypto-related code. Crypt-algorithms are in them selves often very simple. Their principles are based on diffusion and avalanche effects which are pretty easy concepts to grasp. Nevertheless, the mathematical base for those principles are *very* complex. A programmer does not need to have any deeper knowledge of the mathematical parts to implement a DES algorithm or similar such algorithms. If you require such knowledge, I would stick to implementations from the people who invented the algorithms or found flaws in present ones.

Most of the problems within C/C++ implementations of crypto-algorithms are derived from buffer-overflows, not from flawed algorithm implementations. It would actually be pretty difficult to make a flawed implemementation of an algorithm and still have it work for even testcases. Of course one could implement a poor pseudorandom algorithm, but that is easy to fix.

That said, I want to express my full confidence that the people who have implemented things within the crypto-parts of Pike have more than sufficient knowledge of what they are doing and have created correct implementations of the algorithms present. Mind you that securityholes within OpenSSL are quite frequent and often serious. Please tell me of the last time you found an exploit which would give you root-access within Pike and the SSL within Pike.

/ Marcus Agehall (Trådlös)

Previous text:

2003-01-28 09:06: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 01:50:01AM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

Well, I've got a nicely working SSL in Pike 7.4.10 and 7.4.15 here, so it sounds to me that you are doing something wrong.

The only thing what I a doing wrong is that I run "make sure" after compilation of Pike, and see a lot of in SSL stuff. Well:

--- snip --- Pike v7.4 release 10 running Hilfe v3.5 (Incremental Pike Frontend)

...

object sc = SSL.connection(0);

/usr/local/pike/7.4.10/lib/modules/Calendar.pmod/ISO.pmod:16:Couldn't index module 'Gregorian'. /usr/local/pike/7.4.10/lib/modules/Calendar.pmod/Stardate.pmod:313:Undefined identifier TimeofDay. /usr/local/pike/7.4.10/lib/modules/Calendar.pmod/Stardate.pmod:313:Illegal program identifier (type: int). /usr/local/pike/7.4.10/lib/modules/Calendar.pmod/ISO.pmod:16:Placeholder already has storage! /${PIKE_MODULE_PATH}/Calendar.pmod.Stardate.pmod:-: Warning: Compilation failed: Compilation failed. [... lot of backtrace skipped ...] Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/handshake.pike:978:Failed to index module 'X509' with 'decode_certificate' (module doesn't exist?) Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:26:Error finding program Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:26:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:35:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:36:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:37:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:41:No inherit or surrounding class alert. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:42:No inherit or surrounding class urgent. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:43:No inherit or surrounding class application. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:46:No inherit or surrounding class handshake. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:123:No inherit or surrounding class alert. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:126:No inherit or surrounding class urgent. Compiler Error: 1:Index 'connection' not present in module 'SSL'. --- snip ---

Perhaps I do something wrong, but then "make sure" also does something wrong, but usually I trust tests... Should I don't anymore? :)

...

You can begin with explaining what you mean with "unresolved references" since that is not an error usually mentioned when discussing Pike problems.

It is an error, since I can't use some modules because it references something which doesn't exists. I am wrong?

...

I want facts, features, figures or I'll disregard from all these arguments except the speed one, which I have already seen measurements and charts for.

Ciphers - RC2/RC4/RC5/Blowfish/Idea/Cast, with all possible modes of operation. Tell me please where Blowfish or RC5 in Pike is?

Concerning API - I'll make detailed comparision later. Line-by-line.

Not to mention that people working on OpenSSL have experience in crypto related areas, while this is not main focus of Pike team. Sorry to say, but I don't completely trust in crypto stuff which is not implemented by someone who has a track in this area already - the IT security is my business, I performed several security projects so I know what I am talking about. To read the spec and to implement it is often not enough.

...

footprint, but at the same time greater portability problems and easier to make harmful bugs.

Sure, the lower language is - the greater the chance of mistakes. For newbies. But not for someone who knows what he is doing. Is Pike language for beginners?

...

a good idea. Implementing SSL in C is probably a bad idea.

No difference. Especially because there are several good and stable implementations around. Yes, yes, I know there was no one (or almost no one) long time ago, but we are living _now_ :) Very similar to PCRE stuff - the best (around) implementation of RE is not included in Pike...

...

outperformed C code in several benchmarks because the naive implementation in Pike (which utilizes optimized algorithms deeper down) is faster than the naive implementation in C.

I would like to see how to implement circular list in Pike (pure Pike) and achieve good performance... Comparable to, at least, with Perl.

ADT.List is bad example - lists are implemented with help of arrays, while arrays are getting slow in Pike. Though this is transparent for end-user, it is quite ineffective (I already posted my thoughts about arrays).

And... There is _no way_ to outperform native C implementation, since Pike by itself is implemented in C. Some bad implementation in C would be worse than good (by algorithms) in Pike, but that is all.

Regards, /Al

/ Brevbäraren

While we can never make any language safe (except perhaps ADA) for bad programmers, pike does it's best. The fact that the programmer doesn't have to worry about allocating/freeing memory saves us from a lot of potential problems. Sure, the GC in Pike has it's flaws but I think it does a better job than most C-programmers does.

When it comes to the implementation of cryptoalgorithms in Pike, I do believe that it is reasonably safe. If I were to build a system which required high security (as in a military system or equal), I would not trust the Pike implementations and most certainly I would not trust OpenSSL. A quick search at securityfocus.com reveals that Roxen has 8 documented flaws. They all seem to originate from within roxen. OpenSSL had 10+ *PAGES*! Ok, I know that the userbase for OpenSSL is larger than for Pike, but still. DES, RC{4|5}, AES etc are all welldocumented algorithms which most of my friends would be able to implement in Pike without any major flaws. Considering that the people working on Pike are usualy pretty smart, I don't think we have much of a problem here.

The only thing I see as a problem, is how the key is cleared from memory once the algorithm is done with it. I agree with you that it might be a weak point in the Pike implementation. But that can be fixed with ease.

I think you made my point in your last paragraph. Using OpenSSL won't, by itself, resolv the problem of "safe programming". It's a fact that we've known about most exploits for 20 years and they are still around...

/ Marcus Agehall (Trådlös)

Previous text:

2003-01-28 12:56: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 10:55:03AM +0100, Marcus Agehall (Trådlös) @ Pike (-) developers forum wrote:

...

crypto-related code. Crypt-algorithms are in them selves often very simple. Their principles are based on diffusion and avalanche effects which are pretty easy concepts to grasp.

I am talking about implementation errors, which may lead to weakness.

Poor random number generator is good example of bad implementation, key material which is left in memory awating GC is another one.

...

Most of the problems within C/C++ implementations of crypto-algorithms are derived from buffer-overflows, not from flawed algorithm

Not only. If you don't clear memory used in key generation after use, this is _bad_ practice. And this is exactly what I am talking about.

...

implementations. It would actually be pretty difficult to make a flawed implemementation of an algorithm and still have it work for even

Sure. The algorithm and its implementation is perfect. OK. But what is bad is the fact that one could retrieve some valuable data and use it for analysis. Did you think about such things?

...

That said, I want to express my full confidence that the people who have implemented things within the crypto-parts of Pike have more than sufficient knowledge of what they are doing and have created correct implementations of the algorithms present.

Ok. Tell me a way how I can _clear_ the space allocated for the key in Pike?

To _clear_ (memset(X, 0, sizeof(X)) but not to "mark for GC".

...

within OpenSSL are quite frequent and often serious. Please tell me of the last time you found an exploit which would give you root-access within Pike and the SSL within Pike.

Pike's user base is few hundreths, OpenSSL user base is tens of thousands, see the difference?

When Pike will be widely used, the we will talk about security and related holes. Right now it is too early - we just don't have enough apps and testers to find ones. Though, I remember some flaws in Roxen implementation - just visit securityfocus.com and search for Roxen in Bugtraq :) You might say that it was not Pike's fault - OK, but it is a reminder that Pike by itself won't resolve problem of "safe programming".

Regards, /Al

/ Brevbäraren

On Tue, Jan 28, 2003 at 02:55:04PM +0100, Martin Nilsson (�skblod) @ Pike (-) developers forum wrote:

I don't really see clering out the key as an important weak point.

Clearing of sensitive memory areas is really important, even if those are used for something different than keys. Automatic memory management is not good in this case.

The call like really_destruct() would be useful in such case - with clearing all allocated areas.

Regards, /Al

It's probably easier and less performance intruding to have a secure_destruct function, that works on any data type.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-28 16:36: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

I would rather like to see something similar to the weak-flag on mappings. Perhaps a secure-free-flag on all objects which causes the memory to be freed and zeroed as soon as the refcount goes to 0.

/ Marcus Agehall (Trådlös)

"hej" and "hej". You have to keep track of references yourself, or rather, avoid making a lots of them. I don't see this as a problem. If the key happens to be "Hello World", and it is already a shared string for other purposes, I don't see why it should be removed.

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-28 17:46: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Hmm... And how would that work?

string a="hej"; string b=a; secure_destruct(a); string c="hej";

What would you expect b and c to be?

/ Mirar

How can b be "hej" when it's supposed to be the same value as a, and you haven't sent a as lvalue anywhere to get it changed?

Presumably by having a also be "hej" in this case. If I understand correctly, secure_destruct() should be a no-op if there is more than one reference. Of course, this means that in this code

void foo() { string a="hej"; [...] secure_destruct(a); }

the compiler must make sure to remove the reference stored in a before calling secure_destruct, so that the only reference to "hej" is the one on the stack (provided that there is not other strings in program which are "hej" as well, in which ase there will be more references and secure_destruct a no-op).

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Previous text:

2003-01-28 18:38: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

How could you determine in secure_destruct which references are important?

How can b be "hej" when it's supposed to be the same value as a, and you haven't sent a as lvalue anywhere to get it changed?

I can't see how such a function could work without a secure-free-flag.

/ Mirar

Well, the key just might happen to coincide with some other string used by the program. If you start complaining about that, you give away much more information (i.e. you are _telling_ the would-be attacker that the key is a string which normally occurs in the program as something other than the key) than if you pretend it's raining and do nothing.

Remember that if the key string has more references then one of the following must be true:

1) Some other portion of the program is also processing the key. In this case, that other portion should also do secure_destruct() when it's done, and at this time it should actually be overwritten since it's the last ref.

2) The key string coincides with some unrelated string. In this case, there is no need to overwrite the key. The string occurs in the program as something else than the key, so it's ok for it to be there. It might exist before you even _know_ it happens to be the same string as the key.

In neither case do you get any worse securite than if you had had a C program and done memset(buf, 0, sizeof(buf)) at the point that corresponds to the secure_destruct(). The fact that there are more references means that the key is stored in _some other_ "buf" as well, so the memset would not eradicate the key from the process' memory space in the C case either.

Of course, this is assuming the compiler doesn't let references hang around in dead variables, as I pointed out in the previous post.

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Previous text:

2003-01-28 18:52: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Yes, that is a possibility. But it doesn't feel very secure.

It could throw an error on anything with more then one reference, though, I guess, but it feels like that could be used as an exploit to keep the key in memory :)

/ Mirar

Yes. It does not relieve you from the burden of actually thinking when you write the code. :-)

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Previous text:

2003-01-28 19:06: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Yes, but if you did something stupid like,

class { string key; ... { ... secure_destruct(key); ... } }

you wouldn't even get a warning. I think a zero_when_destroyed flag would work better then. But it's probably best to use a special solution for the key, like System.Memory, if that is possible.

/ Mirar

I was about to suggest a similar thing: Make a SecretString class that overloads sufficient operators to behave reasonably like a string: Adding a string with a SecretString would produce a SecretString, indexing a substring out of a SecretString would produce a SecretString, indexing a single element out of a SecretString could perhaps produce a plain integer since the amount of "secrecy" in any single element could be considered low, doing sprintf("%O",...) on a SecretString would produce "SecretString(CENSORED)", etc.

Especially that last item would be really good to avoid the common pike mistake to leave e.g. passwords in function arguments, which gets printed out in backtraces.

I can't imagine sensitive data being represented as anything but a string, a bignum, and perhaps as an array of integers. So adding a SecretString and a SecretInteger class would probably be enough. If I were to write a new security related module for Pike, I'd start with those as fundamental building blocks.

/ Martin Stjernholm, Roxen IS

Previous text:

2003-01-28 19:18: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Reference counting is too magical to think about when you program Pike, in my opinion. But I'm not too fond of secure destruction in Pike...

It'd be neater if Crypto could take System.Memory objects, so you can use magic flags for keys and other secure data.

/ Mirar

Yes, hm, specialized classes is probably good. But maybe as derivatives of System.Memory, so it becomes natural to use mprotect? Secure.String, or SecureString, or Crypto.SecureString?

But can the Crypto parts handle objects who imitates strings, especially without making a shared string of the object?

/ Mirar

Previous text:

2003-01-28 22:51: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

I was about to suggest a similar thing: Make a SecretString class that overloads sufficient operators to behave reasonably like a string: Adding a string with a SecretString would produce a SecretString, indexing a substring out of a SecretString would produce a SecretString, indexing a single element out of a SecretString could perhaps produce a plain integer since the amount of "secrecy" in any single element could be considered low, doing sprintf("%O",...) on a SecretString would produce "SecretString(CENSORED)", etc.

Especially that last item would be really good to avoid the common pike mistake to leave e.g. passwords in function arguments, which gets printed out in backtraces.

I can't imagine sensitive data being represented as anything but a string, a bignum, and perhaps as an array of integers. So adding a SecretString and a SecretInteger class would probably be enough. If I were to write a new security related module for Pike, I'd start with those as fundamental building blocks.

/ Martin Stjernholm, Roxen IS

I promised myself that I would not involve myself in this discussion, but.

First, many people doing the wrong thing does not make it right. SSLEY was _horrible_. Just plain _horrible_. It was at least 10 times larger than it needed to be, and the code was anything but safe. A causual read-through found 10 static sized buffers that theoretically could be overflowed.

I cannot imagine that OpenSSL is all that much better.

Your arguments boils down to two things: 1> Many people are using OpenSSL, and 2> Clearing memory should always be done when creating keys, and is not done in pike.

I assume you are running Windows only, and IE5.0

Well.

All string allocated by the crypto library tends to be overwritten a few microseconds later. It is of course possible that they are not, but it's very, very, very unlikely. For more or less 100% certainity, using the current interface, do something like this:

string key = Crypto.reasonably_random()->read(length); // Use key here.. key = 0; key = Crypto.reasonably_random()->read(length); // Same size. I have yet to see a malloc that does not use the same // location. This is not really 100% certain, though. See next // solution.

System.Memory key = System.Memory( bits>>8 ); Crypto.reasonably_random( key, bits>>8 ); // requires a trivial modification of the interface. // use key here. key->write( "this is not the real key", bits>>8 ); // Overwrite key here.

Also, generating new keys are not exactly a common problem. Most of the time you use a key you need to explicitly keep it in memory.

Also, a quick check of the openssl keygen program seems to indicate that it does not actually clear the generated keys before it exists. Not that it really matters, since they are saved to disk, but still. :-)

/ Per Hedbor ()

Previous text:

2003-01-28 12:56: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 10:55:03AM +0100, Marcus Agehall (Trådlös) @ Pike (-) developers forum wrote:

...

crypto-related code. Crypt-algorithms are in them selves often very simple. Their principles are based on diffusion and avalanche effects which are pretty easy concepts to grasp.

I am talking about implementation errors, which may lead to weakness.

Poor random number generator is good example of bad implementation, key material which is left in memory awating GC is another one.

...

Most of the problems within C/C++ implementations of crypto-algorithms are derived from buffer-overflows, not from flawed algorithm

Not only. If you don't clear memory used in key generation after use, this is _bad_ practice. And this is exactly what I am talking about.

...

implementations. It would actually be pretty difficult to make a flawed implemementation of an algorithm and still have it work for even

Sure. The algorithm and its implementation is perfect. OK. But what is bad is the fact that one could retrieve some valuable data and use it for analysis. Did you think about such things?

...

That said, I want to express my full confidence that the people who have implemented things within the crypto-parts of Pike have more than sufficient knowledge of what they are doing and have created correct implementations of the algorithms present.

Ok. Tell me a way how I can _clear_ the space allocated for the key in Pike?

To _clear_ (memset(X, 0, sizeof(X)) but not to "mark for GC".

...

within OpenSSL are quite frequent and often serious. Please tell me of the last time you found an exploit which would give you root-access within Pike and the SSL within Pike.

Pike's user base is few hundreths, OpenSSL user base is tens of thousands, see the difference?

When Pike will be widely used, the we will talk about security and related holes. Right now it is too early - we just don't have enough apps and testers to find ones. Though, I remember some flaws in Roxen implementation - just visit securityfocus.com and search for Roxen in Bugtraq :) You might say that it was not Pike's fault - OK, but it is a reminder that Pike by itself won't resolve problem of "safe programming".

Regards, /Al

/ Brevbäraren

Of course we can add OpenSSL.

I am however very strongely against too much module-duplication, we want to avoid the route perl has taken, where there are at least 10 modules for any task, and you never have the right one installed when you want to run a script.

But that is just a general fear, not anything specifically related to openssl.

/ Per Hedbor ()

Previous text:

2003-01-28 13:41: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On which systems is an SSL key in the memory an issue?

Wouldn't that require that 1) you have root access to the system, or similar and 2) that you on the same system at the same time can't do much worse things much simpler then trying to scan the memory real time, like for instance replacing the server with a version that simply mails the keys to you, or edits the transmission to your liking?

When is that applicable? I'm not saying it's not an issue, I'm just curious.

But, *why not* have an OpenSSL glue in Pike? If someone wants to use it, why not? I don't think we should tell people what to use. When I want to use something in Pike, I commit a glue for it. Then I bugfix it (removes bashisms and suchlike) and makes sure it works nicely. I don't see any problem with anyone doing that with an OpenSSL glue.

Maybe an OpenSSL glue could be used to verify the Pike SSL, and vice versa?

/ Mirar

Well, if possible, it would probably be nice if the OpenSSL glue could have a similar interface as the already existing.

/ Mirar

Previous text:

2003-01-28 13:45: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Of course we can add OpenSSL.

I am however very strongely against too much module-duplication, we want to avoid the route perl has taken, where there are at least 10 modules for any task, and you never have the right one installed when you want to run a script.

But that is just a general fear, not anything specifically related to openssl.

/ Per Hedbor ()

Yes, Niels mentioned non-blocking issues... But I did say similar, not identical. :)

It could implement a subset of the interface, then?

/ Mirar

Previous text:

2003-01-28 13:55: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Well, yes, that might be sort of hard to do, though. OpenSSL does not, as an example, allow switching between blocking and non-blocking operation for an already opened socket.

/ Per Hedbor ()

That would be excellent, I think.

By the way, what hardware supports cryptography? On what operative system? How does that work, and should Crypto support it?

/ Mirar

Previous text:

2003-01-28 14:00: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Most likely, yes.

It would probably be better to implement that in pike, not in the glue layer, though.

SSL.set_driver("default") SSL.set_driver("openssl")

:-)

/ Per Hedbor ()

I saw some crypto and hash stuff in a recent kernel, but I don't know what problem it is supposed to solve.

/ Mirar

Previous text:

2003-01-28 14:13: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

I read some openbsd code a while ago. I think they use some special devices and ioctls to install keys, etc. IIRC that was an AES hardware. Public key stuff is a little different, as you may have a smartcard with a private key in it, which is supposed to never leave that smart card.

I don't know what openssl does, I have only second hand information that it is a huge kludge ;-)

It would be cool to add some hardware support to nettle (it would be an orthogonal feature, the application would have to choose between software and hardware). But I don't have niether hardware to play with nor a spec. Does linux have any real support for hardware crypto yet? Does there exist any spec for either of the linux or openbsd api:s?

/ Niels Möller ()

And encrypted swap ought to be easier, because key management is trivial: Generate a random key, use it for a while, destroy it some time later. No need to store it or send it around. No passwords the user must remember. Etc.

/ Niels Möller ()

Previous text:

2003-01-28 14:27: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Encrypted swap is actually a very good idea. I also really want encrypted filesystems.

/ Per Hedbor ()

It's probably related to the loopback device.

Hm. It's possible (and quite easy) to create a loopback device that handles cryptography. You might be able to use that as a swapdevice. :-)

/ Per Hedbor ()

Previous text:

2003-01-28 14:26: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

I saw some crypto and hash stuff in a recent kernel, but I don't know what problem it is supposed to solve.

/ Mirar

I should note that hardware SSL acceleration usually cost like $2500 for one unit. You get A LOT of standard PC hardware for that price.

/ David Hedbor

Previous text:

2003-01-28 14:27: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

Another question about the hardware SSL-accelerators, is it really useful with more-or-less modern CPU:s, except perhaps from a security standpoint (there are hopefully no buffer overruns in the hardware...)?

I mean, a normal 2Ghz pentium can rather easily manage 5.000 handshakes per second using 1024bit RSA and openSSLeay 0.9.5 running apache, and can encrypt rather a lot of data (in excess of 125Mbyte/second, it can fill a Gbit ethernet).

This from an OpenSSL performance whitepaper.

Even a 300Mhz ultra can handle about 600 handshakes per second, and can shuffle some 20Mbyte/second.

This from a Elliptic curve cruptografy for SSL report. :-)

So no real speedup will be achieved using hardware algorithms.

Some cpu-offloading, sure, but most reviews of hardware solutions I have read (admittedly a few years ago) have complained about the fact that a lot of CPU is still needed to send messages over the PCI-buss, handle interrupts, etc.

A most likely cheaper solution is a SSL-accelerator _computer_ that you place in front of the real webserver, that handles the [d]encryption, then you have the webserver on another computer.

A new P4 2.5Ghz computer top-of-the-line (best motherboard, memory etc) computer will set you back for $1000 or so.

/ Per Hedbor ()

Le mardi, 28 jan 2003, à 14:05 Europe/Paris, Per Hedbor () @ Pike (-) developers forum a écrit :

Most likely, yes.

It would probably be better to implement that in pike, not in the glue layer, though.

SSL.set_driver("default") SSL.set_driver("openssl")

I like this idea :p

Maybe in the future :

SSL.set_driver("gnussl");

(Yeah, you heard that there is a GNUSsl :p)

/Xavier

:)

Um. No? :) Where can I find this strange beast?

GNU SSL has been renamed to GNU TLS. It is here http://www.gnu.org/software/gnutls/

It has their own calls and can be used as well in OpenSSL compat mode as well.

/Xavier

-- Xavier Beaudouin - Unix System Administrator & Projects Leader. Please visit http://caudium.net/, home of Caudium & Camas projects O ascii ribbon campaign against html email |\ and Microsoft attachments

When is that applicable? I'm not saying it's not an issue, I'm just curious.

When I break in and steal your swap disk (or get it from a container after you have replaced it). As you say, it's futile to try to protect oneself against a real time attack that requires root privs.

To me, the only sane way to handle the non-real time attack is to make sure that pages are encrypted before they are swapped to disk. Using temporay keys that are destroyed regularly, preferably directly at process death. It's not entirely trivial to do, but at least somebody (Niels Provos?) implemented it for openbsd a few years ago.

It's insane to make application programs responsible for figuring out which of their data need extra protection, and "protect" it. If I'm sending a secret message using pgp, it's not enough that the memory where the keys are stored is protected in various kludgy ways, I also want the emacs buffer where I write my message to be protected.

The issue of memory protection should be addressed by the operating system.

But, *why not* have an OpenSSL glue in Pike?

Sure, I wouldn't object to that, as long as I can continue bashing it ;-)

/ Niels Möller ()

Previous text:

2003-01-28 13:41: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On which systems is an SSL key in the memory an issue?

Wouldn't that require that 1) you have root access to the system, or similar and 2) that you on the same system at the same time can't do much worse things much simpler then trying to scan the memory real time, like for instance replacing the server with a version that simply mails the keys to you, or edits the transmission to your liking?

When is that applicable? I'm not saying it's not an issue, I'm just curious.

But, *why not* have an OpenSSL glue in Pike? If someone wants to use it, why not? I don't think we should tell people what to use. When I want to use something in Pike, I commit a glue for it. Then I bugfix it (removes bashisms and suchlike) and makes sure it works nicely. I don't see any problem with anyone doing that with an OpenSSL glue.

Maybe an OpenSSL glue could be used to verify the Pike SSL, and vice versa?

/ Mirar

I guess it's quite common to have different restrictions on access to both old no-longer-used hardware and tape backups than for physical access to the server room.

As for per-uptime keys, I don't like the idea that if I edit a secret message, then someone who breaks in and gets root some half a year later might be able to get it. The property I'm after is usually called "forward secrecy": The enemy should not be able to get my data by an attack long into the future. One reason to want that is that it's really hard to tell how hard the future attack will be.

It's better if the key is permanently destroyed when I close my editor process, or at least within a few hours or days.

/ Niels Möller ()

Previous text:

2003-01-28 14:24: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

...

When I break in and steal your swap disk (or get it from a container after you have replaced it). As you say, it's futile to try to protect oneself against a real time attack that requires root privs.

But why not steal the system disk where the key is in it's "key.txt" or similar file? Not many server systems requires keys external to the system (ie passphrases and magnetic cards) to get going.

Of course, session keys could be interesting if you want to know what was said in a session, I guess.

Still, if I had that sensitive transmissions I wouldn't leak swap disks. I recall some military installation with a remotely controlled explosive bolt to blow up the sensitive HDD.

...

To me, the only sane way to handle the non-real time attack is to make sure that pages are encrypted before they are swapped to disk. Using temporay keys that are destroyed regularly, preferably directly at process death.

It seems simpler that have one random key per up period, and it would work just as well - all swap data gets worthless once the system goes down, and no non-root system process can read out the key from the kernal anyway.

/ Mirar

On Tue, Jan 28, 2003 at 01:30:01PM +0100, Per Hedbor () @ Pike (-) developers forum wrote:

I cannot imagine that OpenSSL is all that much better.

It is. Just compare to SSLeay. Pike stuff is old enough, BTW :)

Your arguments boils down to two things: 1> Many people are using OpenSSL, and 2> Clearing memory should always be done when creating keys, and is not done in pike.

Not really.

#1: There is an implementation which is stable enough and provides API, ciphers and protocols for almost anything in this area.

#2: It takes care about stuff which leaves less freedom for (potential) attackers.

#3 and so on - to be prepared later on (detailed comparison) :)

I don't propose something only because "many people use it". Usually I use something which is good enough and most people don't use it (examples: Exim instead of sendmail/qmail; Pike instead of Perl :)

I assume you are running Windows only, and IE5.0

Of course not. But I've choice at least :) I use whatever I need to solve problems that I've; I am not sticky to one tool or one environment. So if I see that for particular task OpenSSL is better than Pike's SSL, I would like to have this functionality but I still want to Pike.

Also, generating new keys are not exactly a common problem. Most of the time you use a key you need to explicitly keep it in memory.

Yes, sure. But it should live not longer than needed. It should never be swapped (if possible). Any clues how to do this in Pike? :) Yes, it won't work _everywhere_ (memory locking), but it must work where it can, at least.

that it does not actually clear the generated keys before it exists. Not that it really matters, since they are saved to disk, but still.

If it is intended to be stored on the disk it makes no sense to clear the memory, obviously :)

Regards, /Al

There has never been any Pike-related issues on Bugtraq. Pike did have the same SSL-analysation problem as all SSL implementations, and it is has the same fix.

/ Peter Bortas

Previous text:

2003-01-28 12:56: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 10:55:03AM +0100, Marcus Agehall (Trådlös) @ Pike (-) developers forum wrote:

...

crypto-related code. Crypt-algorithms are in them selves often very simple. Their principles are based on diffusion and avalanche effects which are pretty easy concepts to grasp.

I am talking about implementation errors, which may lead to weakness.

Poor random number generator is good example of bad implementation, key material which is left in memory awating GC is another one.

...

Most of the problems within C/C++ implementations of crypto-algorithms are derived from buffer-overflows, not from flawed algorithm

Not only. If you don't clear memory used in key generation after use, this is _bad_ practice. And this is exactly what I am talking about.

...

implementations. It would actually be pretty difficult to make a flawed implemementation of an algorithm and still have it work for even

Sure. The algorithm and its implementation is perfect. OK. But what is bad is the fact that one could retrieve some valuable data and use it for analysis. Did you think about such things?

...

That said, I want to express my full confidence that the people who have implemented things within the crypto-parts of Pike have more than sufficient knowledge of what they are doing and have created correct implementations of the algorithms present.

Ok. Tell me a way how I can _clear_ the space allocated for the key in Pike?

To _clear_ (memset(X, 0, sizeof(X)) but not to "mark for GC".

...

within OpenSSL are quite frequent and often serious. Please tell me of the last time you found an exploit which would give you root-access within Pike and the SSL within Pike.

Pike's user base is few hundreths, OpenSSL user base is tens of thousands, see the difference?

When Pike will be widely used, the we will talk about security and related holes. Right now it is too early - we just don't have enough apps and testers to find ones. Though, I remember some flaws in Roxen implementation - just visit securityfocus.com and search for Roxen in Bugtraq :) You might say that it was not Pike's fault - OK, but it is a reminder that Pike by itself won't resolve problem of "safe programming".

Regards, /Al

/ Brevbäraren

As you can see in the backtrace, it isn't the SSL module but the Calendar module that isn't working. This might have to do with how and where your Calendar module was dumped.

The Calendar module is a known source for problem due to its design. Mirar, who made the module, didn't care about limitations in Pike, but forced his design to work by creating a private loader and bootstrapper for the module. Have you made any attempts to have the Calendar module load with less magic mirar?

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-28 09:06: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 01:50:01AM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

Well, I've got a nicely working SSL in Pike 7.4.10 and 7.4.15 here, so it sounds to me that you are doing something wrong.

The only thing what I a doing wrong is that I run "make sure" after compilation of Pike, and see a lot of in SSL stuff. Well:

--- snip --- Pike v7.4 release 10 running Hilfe v3.5 (Incremental Pike Frontend)

...

object sc = SSL.connection(0);

/usr/local/pike/7.4.10/lib/modules/Calendar.pmod/ISO.pmod:16:Couldn't index module 'Gregorian'. /usr/local/pike/7.4.10/lib/modules/Calendar.pmod/Stardate.pmod:313:Undefined identifier TimeofDay. /usr/local/pike/7.4.10/lib/modules/Calendar.pmod/Stardate.pmod:313:Illegal program identifier (type: int). /usr/local/pike/7.4.10/lib/modules/Calendar.pmod/ISO.pmod:16:Placeholder already has storage! /${PIKE_MODULE_PATH}/Calendar.pmod.Stardate.pmod:-: Warning: Compilation failed: Compilation failed. [... lot of backtrace skipped ...] Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/handshake.pike:978:Failed to index module 'X509' with 'decode_certificate' (module doesn't exist?) Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:26:Error finding program Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:26:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:35:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:36:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:37:Illegal program pointer. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:41:No inherit or surrounding class alert. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:42:No inherit or surrounding class urgent. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:43:No inherit or surrounding class application. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:46:No inherit or surrounding class handshake. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:123:No inherit or surrounding class alert. Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/connection.pike:126:No inherit or surrounding class urgent. Compiler Error: 1:Index 'connection' not present in module 'SSL'. --- snip ---

Perhaps I do something wrong, but then "make sure" also does something wrong, but usually I trust tests... Should I don't anymore? :)

...

You can begin with explaining what you mean with "unresolved references" since that is not an error usually mentioned when discussing Pike problems.

It is an error, since I can't use some modules because it references something which doesn't exists. I am wrong?

...

I want facts, features, figures or I'll disregard from all these arguments except the speed one, which I have already seen measurements and charts for.

Ciphers - RC2/RC4/RC5/Blowfish/Idea/Cast, with all possible modes of operation. Tell me please where Blowfish or RC5 in Pike is?

Concerning API - I'll make detailed comparision later. Line-by-line.

Not to mention that people working on OpenSSL have experience in crypto related areas, while this is not main focus of Pike team. Sorry to say, but I don't completely trust in crypto stuff which is not implemented by someone who has a track in this area already - the IT security is my business, I performed several security projects so I know what I am talking about. To read the spec and to implement it is often not enough.

...

footprint, but at the same time greater portability problems and easier to make harmful bugs.

Sure, the lower language is - the greater the chance of mistakes. For newbies. But not for someone who knows what he is doing. Is Pike language for beginners?

...

a good idea. Implementing SSL in C is probably a bad idea.

No difference. Especially because there are several good and stable implementations around. Yes, yes, I know there was no one (or almost no one) long time ago, but we are living _now_ :) Very similar to PCRE stuff - the best (around) implementation of RE is not included in Pike...

...

outperformed C code in several benchmarks because the naive implementation in Pike (which utilizes optimized algorithms deeper down) is faster than the naive implementation in C.

I would like to see how to implement circular list in Pike (pure Pike) and achieve good performance... Comparable to, at least, with Perl.

ADT.List is bad example - lists are implemented with help of arrays, while arrays are getting slow in Pike. Though this is transparent for end-user, it is quite ineffective (I already posted my thoughts about arrays).

And... There is _no way_ to outperform native C implementation, since Pike by itself is implemented in C. Some bad implementation in C would be worse than good (by algorithms) in Pike, but that is all.

Regards, /Al

/ Brevbäraren

That sounds buggy. Does make verify pass on your system?

Is this what you would expect?

| > object sc = SSL.connection(0); | Indexing the NULL value with "auth_level". | /usr/local/pike/7.3.18/lib/modules/SSL.pmod/handshake.pike:1117: create(0) | /usr/local/pike/7.3.18/lib/modules/SSL.pmod/connection.pike:47: create(0)

Does https work?

| > Protocols.HTTP.get_url("https://pike.ida.liu.se/development/pikefarm/7.5.xml"); | Result 2: Query(200 OK)

/ Mirar

Previous text:

2003-01-28 18:54: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 06:15:01PM +0100, Martin Nilsson (Åskblod) @ Pike (-) developers forum wrote:

...

As you can see in the backtrace, it isn't the SSL module but the

Look at this:

...

Compiler Error: /${PIKE_MODULE_PATH}/SSL.pmod/handshake.pike:978:Failed to index module 'X509' with 'decode_certificate' (module doesn't exist?)

It is not Calendar, but anyway, SSL and Tools.X509 are dependent (somehow) on Caldendar... :) But in this case reference is to X509 which is actually in Tools.X509 (it was fixed later on but brokem again).

Anyway, I cannot use SSL.* after compilation/installation without tweaking.

"Cannot be used immediately afetr installation" == "Doesn't work properly", or you think different? :)

Regards, /Al

/ Brevbäraren

verification mostly passes except for this part (SSL/Calendar).

I wonder how it could be "green" (passed tests) on Xenofarm... My system is Slackware Linux 8.0, i586.

A) Your system does not participate in the xenofarm builds. B) The testsuite is lacking a test that triggers the problem. C) The machines that participate are not affected by the problem, for some reason - this is more common than you might think.

B) seemed falsified by the first statement (although the Calendar testsuite is *very* limited, compared to the amount of code and features the module offers).

Having a diverse flora of machines in xenofarm is good to find and defeat problems like these; your particular installation could well expose bugs, unmet and undetected dependencies met elsewhere and so on. We welcome any and all contributors in xenofarm, and it is an easy way to help yourself indirectly too. :-)

/ Johan Sundström (a hugging punishment!)

Previous text:

2003-01-28 21:49: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

On Tue, Jan 28, 2003 at 07:05:05PM +0100, Mirar @ Pike developers forum wrote:

...

That sounds buggy. Does make verify pass on your system?

"make sure" is an equivalent for "make verify", so verification mostly passes except for this part (SSL/Calendar).

I wonder how it could be "green" (passed tests) on Xenofarm... My system is Slackware Linux 8.0, i586.

...

Is this what you would expect?

| > object sc = SSL.connection(0); | Indexing the NULL value with "auth_level".

Yes.

...

Does https work?

| > Protocols.HTTP.get_url("https://pike.ida.liu.se/development/pikefarm/7.5.xml"); | Result 2: Query(200 OK)

No, it doesn't:

[a lot of messages, so just one of them] /usr/local/pike/7.4.10/lib/modules/Tools.pmod/X509.pmod:61:Failed to index module 'ISO' with 'Year' (module doesn't exist?) /usr/local/pike/7.4.10/lib/modules/Tools.pmod/X509.pmod:235:Class definition failed. /usr/local/pike/7.4.10/lib/modules/Tools.pmod/X509.pmod:415:Class definition failed. [a bit more messages]

As I said, I use pure 7.4.10, which is available for download as stable release. Most problems trace back to Calendar, but anyway... It is required by some reason for X509 :)

Regards, /Al

/ Brevbäraren

On Tue, Jan 28, 2003 at 08:15:04PM +0100, Martin Nilsson (�skblod) @ Pike (-) developers forum wrote:

There is someone who can not use SSL immediately after installing Pike -> SSL doesn't work properly.

The statement is easy to prove wrong. (black out, broken hard drive, wrong installation procedure, bug in Pike core)

Sorry, I am not a newbie in installations :) I did it right, perhaps more right than most people do, I dodn't use any unusual options, but SSL doesn't work due to some unresolved references (see my other messages).

I _can_ get it to work, but - this is me, not average John Doe whos is not even able to understand what is going on. I hope you get my point and stop being an ass here :) - I just dislike when some stuff which is OK in one version (stable Pike 7.2.xxx) stops working after exactly same installation procedure for stable 7.4.x. Logical or not? :)

Regards, /Al

On Tue, Jan 28, 2003 at 10:10:05PM +0100, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum wrote:

Most people get it to work though. Which kind of speaks against you doing things "more right than most people". :-)

"./configure; make install" supposed to work, right? That is what most people do. Now, since it doesn't in my case (on my system), it might (and eventually will) fail on other systems as well. Not on every, though, but anyway. The problem is that Pike (especially 7.4.10) isn't running or compiled on thousands of systems, otherwise my particular problem wouldn't be so particular :) BTW, why then 7.4.13 (assembled from CVS) was correctly compiled and SSL stuff _did_ work after "./configure; make"? :)

When I say "I can get it to work" it means that I can tweak the source, the configuration or whatever it is and to fix any problems, what most people won't (be able to) do.

In this particular case (SSL/Calendar) I don't need it _now_, so I didn't spend any time trying to figure out where is the problem. I just noticed that it doesn't work, that testsuite fails, that's all.

Also, I would like to mention that CVS version of Pike 7.4 is not really useable when latest version of autoconf is installed (I had to degrade it to very older version), but what the heck... Not everyone is even knows what CVS is, right? :)

Regards, /Al

It's not very important to support the latest autoconf since it isn't used when a source dist is compiled. It's more important that we (the developers) use whatever autoconf versions that suit us, and agree about it.

Yup I agree... Excepted I cannot use generated source in FreeBSD now... The configure script fail miserably.... Dunno why... But redoing a ./run_autoconf fixes the problem for me.

Besides, from what I've gathered things have been changing incompatibly between almost every minor release of autoconf lately. If that's correct then I don't think it's worthwhile to try to support it until it has settled again. Then we can port all the m4 code to it and simply drop support for 2.13.

I really like that anyway. Autoconf 2.13 is not standard in all opensource based systems (linux, *bsd, darwin...) and current autoconf code doesn't generate anything on all systems I use.

So to work I have to do the work... copy it to my slow netra X1, wait 20minutes to ./run_autoconf finish it work.... It is really pain in the ass...

BTW Autoconf 2.54 works like a charm with autoconf 2.5x files now.

It is a shame that 2.5x doesn't support well 2.13 files... but we have to thanks some Epita school people for that...

So maybe to help developpers to work, droping 2.13 compat should help us...

/Xavier

Of course that doesn't always work however. This is what I get with Pike 7.4.13 compiled from the Debian source packages on Sparc Linux (and I believe on all other platforms these particular debs have been compiled on such as mips and x86). However the version I compiled myself from source on x86 works fine - also removing the .o files fixes the problem. Saying that there's no problem is invalid - there is and it's been up many times before. However the problem is not with SSL but rather with the Calendar module.

: 1 neotron@itzlacohuihqui pike Pike v7.4 release 13 running Hilfe v3.5 (Incremental Pike Frontend)

Protocols.HTTP.get_url("https://pike.ida.liu.se/development/pikefarm/7.5.xml");

/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/ISO.pmod:16:Couldn't index module 'Gregorian'. /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate.pmod:313:Undefined identifier TimeofDay. /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate.pmod:313:Illegal program identifier (type: int). /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/ISO.pmod:16:Placeholder already has storage! /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod.Stardate.pmod:-: Warning: Compilation failed: Compilation failed. /usr/lib/pike7.4/7.4.13/master.pike:382: master()->compile_file("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate.pmod",0,Stardate.pmod,0) /usr/lib/pike7.4/7.4.13/master.pike:583: master()->low_findprog("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate.pmod","",0,1) /usr/lib/pike7.4/7.4.13/master.pike:652: master()->findprog("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate.pmod",".pmod",0,1) /usr/lib/pike7.4/7.4.13/master.pike:685: master()->low_cast_to_program("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate","/.",0,1) /usr/lib/pike7.4/7.4.13/master.pike:851: master()->low_cast_to_object("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate.pmod","/.",0) /usr/lib/pike7.4/7.4.13/master.pike:1218: master()->findmodule("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/Stardate",0) [...] Attempt to call the NULL-value Unknown program: 0("UTC") /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/module.pmod:44: Calendar->`[]("_module_value") /usr/lib/pike7.4/7.4.13/master.pike:902: master()->dirnode("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod")->module_checker()->`!() /usr/lib/pike7.4/7.4.13/master.pike:948: master()->dirnode("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod")->ind("Gregorian") /usr/lib/pike7.4/7.4.13/master.pike:999: master()->dirnode("/usr/lib/pike7.4/7.4.13/modules/Calendar.pmod") [...] /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/ISO.pmod:16:Illegal program pointer. /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/ISO.pmod:115:No inherit or surrounding class Gregorian. /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/ISO.pmod:148:Class definition failed. /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/ISO.pmod:152:No inherit or surrounding class Gregorian. /usr/lib/pike7.4/7.4.13/modules/Calendar.pmod/ISO.pmod:185:Class definition failed. /usr/lib/pike7.4/7.4.13/modules/Tools.pmod/X509.pmod.o:-: Warning: Decode failed: Compilation failed.

/usr/lib/pike7.4/7.4.13/modules/Tools.pmod/X509.pmod:61:Failed to index module 'ISO' with 'Year' (module doesn't exist?) /usr/lib/pike7.4/7.4.13/modules/Tools.pmod/X509.pmod:235:Class definition failed. /usr/lib/pike7.4/7.4.13/modules/Tools.pmod/X509.pmod:415:Class definition failed. Protocols.HTTP can't handle "https" or any other protocol than HTTP /usr/lib/pike7.4/7.4.13/modules/Protocols.pmod/HTTP.pmod/module.pmod:28: Protocols.HTTP->do_method("GET",URI("https://pike.ida.liu.se/development/pikefarm/7.5.xml%22),0,(%5B%5D),Protocol...) /usr/lib/pike7.4/7.4.13/modules/Protocols.pmod/HTTP.pmod/module.pmod:87: Protocols.HTTP->get_url("https://pike.ida.liu.se/development/pikefarm/7.5.xml%22,0,0,0)

/ David Hedbor

Previous text:

2003-01-28 22:09: Subject: Re: OpenSSL wrapper vs Pike's SSL (Was: Bz2)

---

...

Sorry, I am not a newbie in installations :) I did it right, perhaps more right than most people do, I dodn't use any unusual options, but SSL doesn't work due to some unresolved references (see my other messages).

Most people get it to work though. Which kind of speaks against you doing things "more right than most people". :-)

gedrix <304> pike Pike v7.4 release 10 running Hilfe v3.5 (Incremental Pike Frontend)

...

Protocols.HTTP.get_url("https://pike.ida.liu.se/development/pikefarm/7.5.xml");

(1) Result: Protocols.HTTP.Query(200 OK)

...

/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)

Didn't we have this discussion already? I think Andreas wanted to write a module and write a paper on it, and this Bz2 module is it.

/ Mirar

Previous text:

2003-01-25 20:26: Subject: Bz2

---

So what was wrong with the bzip2 module that was in Pexts since a few years back, out of curiosity?

/ David Hedbor

And grubba and I was just discussing removing everything foo-bar-int-related the other day... Having it work is of course an even better solution :)

/ Martin Nilsson (Åskblod)

Previous text:

2003-01-26 12:33: Subject: INT_TYPE = INT64 (Bz2)

---

Thanks Grubba for making Bz2 work. I fixed the testsuite too now.

I have another problem, I want Pike to be able to use long long ints in the svalues. It shouldn't be much work to get it to work. But here's where I'm stuck now:

| void debug_push_int_type(INT_TYPE min, INT_TYPE max) | { | *(++Pike_compiler->type_stackp) = mk_type(T_INT, | (void *)(ptrdiff_t)min, | (void *)(ptrdiff_t)max, 0); | | TYPE_STACK_DEBUG("push_int_type"); | }

The Pike int type can't contain ints that take more space then a pointer: (void*).

Suggestions? I'm thinking of moving the car and cdr value type to be a union of some sort rather then these, imh, rather ugly and limited casts.

Grubba?

/ Mirar

In the last episode (Jan 26), Mirar @ Pike developers forum said:

I really think we should get INT_TYPE = INT64 to work. In not so many months there will be 64 bit systems on the market, Intel IA64 and AMD Hammer (?). (Someone else probably have the current estimate.) Having 64-bit ready code then might be nice. :)

Pike on Alpha and Sparc have been working for a long time.

At least Alpha has 64 bit longs (and 32 bit int).

/ David Hedbor

Previous text:

2003-01-27 08:13: Subject: Re: INT_TYPE = INT64 (Bz2)

---

I forgot... is the registers really 64 bit on those processors? IIRC, "long" is 32 bit even on those systems, and that is the default type of integers in Pike.

It would be nice to see tests and benchmarks on Alpha and Sparc with Pike compiled with --with-long-long-int.

/ Mirar

"int" is the default INT_TYPE, so I guess it works on all four or so alphas in the xenofarm. Maybe "long" should be the default INT_TYPE now that it works?

(Why is "long double" only 8 bytes on Alpha?)

/ Mirar

Previous text:

2003-01-27 08:39: Subject: Re: INT_TYPE = INT64 (Bz2)

---

Are you sure? I can't see how that could have worked... Unless maybe if INT32 was 64 bit on Alpha as well - is it?

/ Mirar

Le lundi, 27 jan 2003, à 03:12 Europe/Paris, Dan Nelson a écrit :

In the last episode (Jan 26), Mirar @ Pike developers forum said:

...

I really think we should get INT_TYPE = INT64 to work. In not so many months there will be 64 bit systems on the market, Intel IA64 and AMD Hammer (?). (Someone else probably have the current estimate.) Having 64-bit ready code then might be nice. :)

Pike on Alpha and Sparc have been working for a long time.

On Alpha Linux and FreeBSD yup.... But I was not be able to compile it in full 64bits with Forte C.... :/

/Xavier

-- Xavier Beaudouin - Unix System Administrator & Projects Leader. Please visit http://caudium.net/, home of Caudium & Camas projects O ascii ribbon campaign against html email |\ and Microsoft attachments

If you want to play with it yourself asmodean.lysator and moghedien.lysator are Linux alphas.

/ Peter Bortas

Previous text:

2003-01-27 10:52: Subject: Re: INT_TYPE = INT64 (Bz2)

---

Do you have access to an Alpha? I'd be interested to see how well it works with a 64-bit INT_TYPE after yesterdays bugfixes.

To make sure you got 64 bit INT_TYPE, check the config.info output:

int type............ long long (8 bytes) ^^^^^^^

/ Mirar

Le lundi, 27 jan 2003, à 10:55 Europe/Paris, Mirar @ Pike developers forum a écrit :

Do you have access to an Alpha? I'd be interested to see how well it works with a 64-bit INT_TYPE after yesterdays bugfixes.

To make sure you got 64 bit INT_TYPE, check the config.info output:

int type............ long long (8 bytes) ^^^^^^^

I have access... in fact this is my own Alpha... see alpha.home.oav.net in pike farm.

Humm.. how do I add some INT64 target ???

/Xavier

Le lundi, 27 jan 2003, à 13:40 Europe/Paris, Mirar @ Pike developers forum a écrit :

I have this in my xenofarm pike7.5.cfg (orchid.mirar.org):

test: --with-long-long-int make xenofarm CONFIGUREARGS="--with-long-long-int --with-double-precision"

Test added.... Now waiting for crontab :pp

/Xavier

Le mardi, 28 jan 2003, à 08:20 Europe/Paris, Mirar @ Pike developers forum a écrit :

alpha.home.oav.net xenofarm dies in something that shouldn't have anything to do with large ints, test 762. Something with pipes, open, mutexes and create_process.

int type............ int (4 bytes) Ah, you turned off long ints again? Then that hasn't anything to do with this problem... :ð

No I didn't. The int64 target is still here.... I dunno why it is still in "int (4 bytes)"....

So strange reason the testsuite on freebsd alpha has never be ran correctly...

/Xavier