I don't think it makes much sense with per-certificate callbacks. I think it should be enough to report the kind of problem to the application (certificate chain broken, root cert not trusted, signatures that aren't valid, some certificate in the chain has expired). And then give the application the option of accepting the peer anyway.
I was thinking of one callback at the end of verification. It would have to provide information about all failures, though, not just the first, as it would be possible to have a really messed up chain with bad flags, exirations and more.
The reason I say callback, is because you'd have to get an answer to override before the result of the verification were passed back to ssl.handshake, otherwise the connection would be broken immediately. It is possible that I'm missing an obvious technique.
As for flags, Peter Gutmanns style guide should give some guidance. I don't remember much of the details. Guess one should start with the flags of the "critical" kind, and refuse to accept certificates with unknown or unimplemented critical flags.
One thing I've noticed is that at least for some of the OpenSSL created certificates I've run across, the BasicConstraint extension is invalid. It should contain both a boolean and a constraint, which is present in the comodo certs I've looked at, but the openssl certs seem to be missing the boolean. Does this indicate a bug in the certificate, or is it ok to violate syntax (perhaps that's what the DEFAULT means in the definition) like this?
One important part (although perhaps not very exciting) is testcases that exersize all the failure modes listed above.
Yes, I figured they'd need to be done at somepoint. Any thoughts on the best way to do this (generate our own certs, provide some "bogus" certs to test agains, etc)?
Bill