I'm finally looking into conversion of the lsh repository from CVS to GIT, and this naturally also includes nettle.
A public test repository can be found at lysator's gitorious installation, at http://git.lysator.liu.se/lsh/test-2. The conversion was done by the pcvs2git.pike program (see git://pike-git.lysator.liu.se/pcvs2git.git), and I got some help from Henrik Grubbström to write a config file to handle the few peculiarities in the lsh repository.
The intention is that the new git repository should include all branches and tags from the old cvs repository, and record major merge events, like the 2006-05-16 merge from the experimental branch to the trunk.
Please test, and if all goes well I'll rename this repository to "lsh" (or recreate, if I can't figure out how to do repository renames with gitorious).
If this first steps works out ok, step two is to do a some cleanups (switching to utf-8 for the files, deleting old $Id$ tags, etc). And then step three is to extract the nettle subdirectory as an independent project and repository, using git subtree of git filter-branch or so (I'm not sure what's the best tool for that job). Other sub-projects, e.g., the argp implementation, can be split out later, if desired.
For the few "common files" (e.g., misc/run-tests), they'll simply have to be duplicated in several repositories. If I still want to bundle nettle with the lsh distribution, I'll handle that by setting up some symlink in my working tree. git submodule is probably not a solution, and git subtree is most likely overkill.
The old CVS repository can be considered read-only now.
Regards, /Niels
nisse@lysator.liu.se (Niels Möller) writes:
I'm finally looking into conversion of the lsh repository from CVS to GIT, and this naturally also includes nettle.
That is great. I find that testing and preparing patches is easier with git than CVS. I have been wanting to propose patches for adding the new SHA2/x hashes, and this will be easier for me now.
Please test, and if all goes well I'll rename this repository to "lsh" (or recreate, if I can't figure out how to do repository renames with gitorious).
The nettle part worked fine for me. Let me suggest to populate .gitignore with some files indicate by 'git status' after building Nettle once. For me the output was as below.
/Simon
# INSTALL # aclocal.m4 # aes-decrypt-internal.asm # aes-encrypt-internal.asm # camellia-crypt-internal.asm # config.guess # config.sub # examples/getopt.c # examples/getopt.h # examples/getopt1.c # examples/next-prime # examples/random-prime # examples/run-tests # gcmdata # hogweed.pc # install-sh # machine.m4 # memxor.asm # nettle.pc # serpent-decrypt.asm # serpent-encrypt.asm # sha1-compress.asm # testsuite/camellia-test # testsuite/gcm-test # testsuite/memxor-test # testsuite/meta-armor-test # testsuite/meta-cipher-test # testsuite/meta-hash-test # testsuite/random-prime-test # testsuite/ripemd160-test # testsuite/run-tests # testsuite/sha224-test # testsuite/sha384-test # testsuite/sha512-test # texinfo.tex # tools/getopt.c # tools/getopt.h # tools/getopt1.c # tools/nettle-hash
Hi,
On Tue, 24 Jan 2012, Niels Möller wrote:
I'm finally looking into conversion of the lsh repository from CVS to GIT, and this naturally also includes nettle.
This is great news!
If this first steps works out ok, step two is to do a some cleanups (switching to utf-8 for the files, deleting old $Id$ tags, etc). And then step three is to extract the nettle subdirectory as an independent project and repository, using git subtree of git filter-branch or so (I'm not sure what's the best tool for that job). Other sub-projects, e.g., the argp implementation, can be split out later, if desired.
I'd hope for a round of filter-branch to rewrite author names in git to the original author of patches. I haven't done it myself, but I've seen it done on other projects. One can do some kind of more or less heuristic shell script that e.g. greps the commit message for known strings (either known contributors or strings like "From:" or "contributed by") and based on these override the author name.
If you don't feel up to doing this yourself, I can try to give it a go.
// Martin
Martin Storsjö martin@martin.st writes:
I'd hope for a round of filter-branch to rewrite author names in git to the original author of patches.
I was thinking that it's good enough to convert the information recorded by cvs. But it's certainly possible to add specified author names for certain commits, as a part of the conversion process. The config/Pike-real-authors file (from the git://pike-git.lysator.liu.se/pcvs2git.git repository) is an example of what that input file should look like. Since I haven't used it I'm not sure how it works, but I could try to find out.
But it's going to be some manual work, going through the ChangeLog files looking for contributed changes, and then looking at the cvs log to identify affected revisions and writing the input file for --contributors.
If you don't feel up to doing this yourself, I can try to give it a go.
If you want to give it a try, I'll explain what's needed.
1. Get pike 7.8 release 550 or later, http://pike.ida.liu.se/pub/pike/beta/7.8.550/Pike-v7.8.550.tar.gz
Annoyingly enough, it doesn't support the most recent version of nettle, resulting in compilation errors on the DES glue. I had to #if out the DES and DES3 ciphers from the Nettle module. Alternatively, there should be some configure argument to force Pike to build with the bundled version of nettle, rather than the installed version.
2. Get git://pike-git.lysator.liu.se/pcvs2git.git. This includes Henrik's configuration file in config/lsh.pcvs2git.
3. Get the CVS repository. I'm putting a copy at http://www.lysator.liu.se/~nisse/misc/lsh-cvsroot.tar.gz
4. Create an authors.txt file, containing
nisse=Niels Möller nisse@lysator.liu.se uid444=Niels Möller nisse@lysator.liu.se _cvs_pont=Pontus Freyhult pont@soua.net
5. To do the conversion, I used the command line
pike ../pcvs2git/pcvs2git.pike -c ../pcvs2git/config/lsh.pcvs2git \ -A authors.txt -k -l -C lsh.git -d lsh-cvsroot/lsh
It spawns a git fast-import process which populates the new repository.
The result should be almost identical to the current "test-2" repository (Henrik has committed some tweaks for the generated .gitattributes files).
To use a contributors file, add --contributors contributors.txt early on the above command line (the -d option and argument has to come after other options).
Regards, /Niels
Now I've went through the history, back from the previous millennium, startig from the authors and contributors files that Martin was kind enough to prepare. It's taken a few days. The early ChangeLog entries were a bit messy.
The new repository is now git://git.lysator.liu.se/lsh/test-3.git
And I also found out some things which I had forgotten. E.g.,
* The very first version of the lsh-export-key program was written by Jean-Pierre Stierlin.
* Dan Egnor wrote the base64 code in Nettle, with much effort spent on finding the right interface design.
* Pieces of Balázs Scheidler's now obsolete lsh_proxy program still survives, since current gateway_channel.c evolved from his proxy_channel.c.
The recorded authors are listed below. These are all people I have been able to find who have contributed a larger or smaller change to lsh or nettle (not counting bug reports or sugestions for changes). Change sets which mix a contributed patch with other changes also don't get the contributor listed as author. So we have 37 authors over the years:
Adam Langley alangley@gmail.com Andres Mejia mcitadel@gmail.com Balázs Scheidler bazsi@balabit.hu Dagobert Michelsen dam@opencsw.org Dan Egnor egnor@ofb.net Daniel Kahn Gillmor dkg@fifthhorseman.net David Hoyt hoyt6@llnl.gov Gordon Matzigkeit gord@trick.fig.org Grant Robinson santiago@mr-r.net Henrik Grubbström grubba@lysator.liu.se J.H.M. Dassen jdassen@wi.leidenuniv.nl James Ralston qralston+ml.nettle-bugs@andrew.cmu.edu Jean-Pierre Stierlin jps@macssh.com Joseph Galbraith galb@rt66.com Kalle Olavi Niemitalo tosi@ees2.oulu.fi Karl Berry karl@gnu.org Keresztfalvi Gabor Agoston kg230@hszk.bme.hu Luiz Eduardo Gava leg@terra.com.br Magnus Holmgren holmgren@debian.org Martin Storsjö martin@martin.st Meilof Veeningen meilof@gmail.com Niels Möller nisse@lysator.liu.se Nikos Mavrogiannopoulos nmav@gnutls.org Pavel Roskin proski@gnu.org Per Cederqvist ceder@lysator.liu.se Pontus Freyhult pont@soua.net Pontus Sköld pont@soua.net Rafael Sevilla dido@pacific.net.ph Rafal Maszkowski rzm@icm.edu.pl Ruud de Rooij ruud@debian.org Ryan Schmidt nettle-2011@ryandesign.com Sebastian Reitenbach sebastia@l00-bugdead-prods.de Simon Josefsson simon@josefsson.org Stefan Pfetzing dreamind@dreamind.de Thayne Harbaugh thayne@northsky.com Vincent Torri vincent.torri@gmail.com Volker Zell dr.volker.zell@oracle.com
(Some statistics on first and latest contribution for each author, lines contributed, contributed lines still surviving in the tree, etc, would be interesting).
Testing and general browsing of the history is appreciated. In a few days, I think I'll consider the conversion final.
Regards, /Niels
On Tue, 31 Jan 2012, Niels Möller wrote:
Testing and general browsing of the history is appreciated. In a few days, I think I'll consider the conversion final.
The author mapping looks good to me at least, looking forward to the final repo (including separated nettle)! Thanks for your work on this!
// Martin
On 01/31/2012 09:50 AM, Martin Storsjö wrote:
On Tue, 31 Jan 2012, Niels Möller wrote:
Testing and general browsing of the history is appreciated. In a few days, I think I'll consider the conversion final.
The author mapping looks good to me at least, looking forward to the final repo (including separated nettle)! Thanks for your work on this!
It also looks good to me. Thanks for doing this conversion work, Niels.
--dkg
Martin Storsjö martin@martin.st writes:
The author mapping looks good to me at least, looking forward to the final repo
Now "final". Repo url git://git.lysator.liu.se/lsh/lsh.git. I renamed the old gitorious "project" to get the various test repositories out of the way.
After the conversion, I have pushed some tags to all branches to mark the conversion. On master only, I have eliminatd $Id tags in files, and converted all textfiles to utf-8 (encoding marks in texinfo and html not yet updated, though).
(including separated nettle)!
Not there yet.
Regards, /Niels
nisse@lysator.liu.se (Niels Möller) writes:
Martin Storsjö martin@martin.st writes:
The author mapping looks good to me at least, looking forward to the final repo
Now "final". Repo url git://git.lysator.liu.se/lsh/lsh.git. I renamed the old gitorious "project" to get the various test repositories out of the way.
Great! I added it to my batch job to create statistics for, see output here:
http://gitstats.josefsson.org/lsh/
/Simon
Martin Storsjö martin@martin.st writes:
The author mapping looks good to me at least, looking forward to the final repo (including separated nettle)! Thanks for your work on this!
I now have a first nettle test repo at git://git.lysator.liu.se/lsh-test/nettle-test.git
It was created by first cloning the lsh repository, all branches. Then
git filter-branch --subdirectory-filter nettle --prune-empty -- --all
followed by git update-ref -d on all refs which the previous command warned about being unchanged. Including branches other than master is probably overkill. One file (nettle/tools/nettle-lfib-stream.c) seems to have originated on the lsh-1.4.2 branch, but I think it was copied manually to the main trunk.
However, this conversion loses quite a lot of history, including all nettle-1.15 releases. To actually build old versions, one would need to use the lsh repo (due to the shared files linked by lsh's ./.bootstrap), but it would be nice to still have the history in the nettle repo.
I think the following approach would make some sense, using git filter-branch --tree-filter or possibly --index-filter:
1. Delete everything except nettle, src/nettle and src/symmetric.
2. If nettle exists, move its contents up one level and delete nettle.
3. Otherwise, if src/nettle exists (and is non-empty), move its contents up two levels and delete the original directory.
4. Otherwise, if src/symmetric exists (and is non-empty), move its contents up two levels and delete the original directory.
5. Otherwise, the tree should be empty. It's from the first few weeks of lsh development, not relevant to nettle.
Help from any git guru on the list is appreciated. In particular if the more efficient --index-filter method is to be used, it's not obvious to me how to examine the index and do the right tests and deletions.
One could also try to extract all src/symmetric ChangeLog entries from lsh's ChangeLog and copy to nettle's (prior to spring 2001 or so) . But I think that can be done after the git migration; it's not much use to modify earlier revisions of the ChangeLog.
Regards, /nisse
nisse@lysator.liu.se (Niels Möller) writes:
Martin Storsjö martin@martin.st writes:
The author mapping looks good to me at least, looking forward to the final repo (including separated nettle)! Thanks for your work on this!
I now have a first nettle test repo at git://git.lysator.liu.se/lsh-test/nettle-test.git
It doesn't build:
jas@latte:~/src/nettle-test master$ ./.bootstrap jas@latte:~/src/nettle-test master$ ./configure configure: error: cannot find install-sh, install.sh, or shtool in . "."/. jas@latte:~/src/nettle-test master$
However perhaps that is one of the unresolved issues.
However, this conversion loses quite a lot of history, including all nettle-1.15 releases. To actually build old versions, one would need to use the lsh repo (due to the shared files linked by lsh's ./.bootstrap), but it would be nice to still have the history in the nettle repo.
I think the following approach would make some sense, using git filter-branch --tree-filter or possibly --index-filter:
Delete everything except nettle, src/nettle and src/symmetric.
If nettle exists, move its contents up one level and delete nettle.
Otherwise, if src/nettle exists (and is non-empty), move its contents up two levels and delete the original directory.
Otherwise, if src/symmetric exists (and is non-empty), move its contents up two levels and delete the original directory.
Otherwise, the tree should be empty. It's from the first few weeks of lsh development, not relevant to nettle.
As for these advanced git features, I have no idea.
However, I'm not sure it is that important to make the git repo usable for building older nettle releases -- for anyone that really wants to go back in time and build old things can always look at the 'lsh' repository. That reflects how nettle was developed before, and consequently how everyone need to work with those releases.
For most people, working on master and/or the latest stable release is what's interesting. So I'd say do a git conversion and then make a release from git and then work forward from there.
/Simon
On 02/06/2012 01:42 PM, Simon Josefsson wrote:
For most people, working on master and/or the latest stable release is what's interesting. So I'd say do a git conversion and then make a release from git and then work forward from there.
I'd agree with this. people who are providing longer-term support for older releases can also use the nettle repo to create a branch for the version they're supporting, and cherry-pick any cleanup commits from stable or master that will make the code generally buildable again.
--dkg
Simon Josefsson simon@josefsson.org writes:
configure: error: cannot find install-sh, install.sh, or shtool in . "."/. jas@latte:~/src/nettle-test master$
However perhaps that is one of the unresolved issues.
This is one of the files that was setup using symlinks in the lsh repo. I'll add this and other files to the nettle repo, *after* the conversion of the history is satisfactory.
However, I'm not sure it is that important to make the git repo usable for building older nettle releases
I mostly agree here; you won't be able to build from the nettle repo out of the box (just like you currently can't build HEAD out of the box).
But I still think it's desirable to have older history, before directory renames, available for git log.
Another approach I've been considering is to do a conversion which purges unrelated files (most of src except src/nettle and src/symmetric, most of misc, all contents in doc, argp, sftp, spki, most top-level files). And then use version controlled operations for the remaining cleanup and renames. The aim would then be to make it possible to check out old versions (with their different directory layout) and build them.
But at the moment, I'm leaning towards moving directories in the conversion, since I think that will be less confusing with git log and diff. E.g., I'd expect
git diff master nettle_1.7_release_20030311 -- configure.ac
to show a diff of nettle's configure.ac in different revisions, not a diff between nettle's current configure.ac and a historic configure.ac from lsh.
/nisse
nisse@lysator.liu.se (Niels Möller) writes:
However, this conversion loses quite a lot of history, including all nettle-1.15 releases. To actually build old versions, one would need to use the lsh repo (due to the shared files linked by lsh's ./.bootstrap), but it would be nice to still have the history in the nettle repo.
Now I have a new test repo. Not created using git filter-branch on the lsh git repository, but converted from the same old cvs files, using pcvs2git and a new configuration file written primarily by Henrik Grubbström.
The conversion moves files from nettle, src/nettle and src/symmetric (whichever is found first) to the top level. Also tries to keep the shared files, also putting them at top-level in the new repo (getopt*, config.guess, misc/run-tests, and a few more). Some symlinking still needs to be added to the .bootstrap script before one can build.
See
git://git.lysator.liu.se/lsh-test/nettle-test-2.git
/nisse
ons 2012-02-08 klockan 22:07 +0100 skrev Niels Möller:
git://git.lysator.liu.se/lsh-test/nettle-test-2.git
Looks good to me. It is probably easier to preserve history by going from a cvs repository directly to git. Maybe the CVS repository could even be tweaked somewhat before the conversion to make it look more like a "nettle" cvs repository than a lsh repository...
/Simon
nisse@lysator.liu.se (Niels Möller) writes:
Now I have a new test repo. Not created using git filter-branch on the lsh git repository, but converted from the same old cvs files, using pcvs2git and a new configuration file written primarily by Henrik Grubbström.
Yet another try, and unless I discover any problems with this soon, it's going to be the final one.
git://git.lysator.liu.se/lsh-test/nettle-test-3.git
Difference from the previous one is that I've hacked the pcvs2git.pike program to keep track of the original path to each RCS file (as it moves directories up one or two levels), and use that in the generated Rev: comments. E.g, for git log -- twofish.c (a file at the top-level in the new git repo), git log includes the following commits (among others):
commit f9ec0110676f16e263688b208c3a2e8735723e7d Author: Niels Möller nisse@lysator.liu.se Date: 2010-07-07 21:32:03 +0200
* aes.h (aes_encrypt, aes_decrypt): Declare ctx argument as const. Also updated implementation. * blowfish.h (blowfish_encrypt, blowfish_decrypt): Likewise. * cast128.h (cast128_encrypt, cast128_decrypt): Likewise. * serpent.h (serpent_encrypt, serpent_decrypt): Likewise. * twofish.h (twofish_encrypt, twofish_decrypt): Likewise.
Rev: nettle/ChangeLog:1.91 Rev: nettle/aes-decrypt.c:1.2 Rev: nettle/aes-encrypt.c:1.2 Rev: nettle/aes.h:1.2 Rev: nettle/blowfish.c:1.3 Rev: nettle/blowfish.h:1.3 Rev: nettle/cast128.c:1.2 Rev: nettle/cast128.h:1.2 Rev: nettle/serpent.c:1.2 Rev: nettle/serpent.h:1.2 Rev: nettle/twofish.c:1.2 Rev: nettle/twofish.h:1.2
commit dd06afa4b49913037dabe6cab40ae1b9d49a0344 Author: Niels Möller nisse@lysator.liu.se Date: 2004-10-05 23:46:33 +0200
(q_table): Use a const pointer array.
Rev: src/nettle/twofish.c:1.7
commit 7b6ff2dcdaef64e194c9b83eed3354a5acda9168 Author: Niels Möller nisse@lysator.liu.se Date: 1999-03-08 00:52:36 +0100
Moved UNUSED attribute to be compatible with gcc-2.x, x<8.
Rev: src/symmetric/twofish.c:1.3
Note that the Rev: comments refer to different RCS files, and this information is needed to locate the corresponding change in the old CVS repo.
Regards, /Niels
nisse@lysator.liu.se (Niels Möller) writes:
Yet another try, and unless I discover any problems with this soon, it's going to be the final one.
Turns out I had broken the handling of deleted files. So another try,
git://git.lysator.liu.se/lsh-test/nettle-test-4.git
Regards, /Niels
I think the conversion can be declared final now. New repo at
git://git.lysator.liu.se/nettle/nettle.git
I have fixed the Makefiles so that they work in the new setting. After cloning, you need to run the ./.bootstrap script, then ./configure && make as usual.
Regards, /Niels
nettle-bugs@lists.lysator.liu.se
-
Daniel Kahn Gillmor -
Martin Storsjö -
nisse＠lysator.liu.se -
Simon Josefsson