Hi, the Nettle CI has been running at the mirror repository https://gitlab.com/gnutls/nettle for several years. However, the automatic mirroring stopped working during summer, related to bureaucracy for gitlab's "open source" program. This makes this mirror and CI a lot less useful, and it might be removed at some point.
Instead, CI has now been set up to run directly on git.lysator.liu.se. There's currently one runner serving the nettle project, managed by Simon, thanks!
For now, I have disabled the remote/s390x job, since I've not been able to mark it as conditional depending account variables, and I haven't yet setup new keys (since I want to retire the key used at gitlab.com). If anyone here is more familiar with .gitlab-ci.yml, help is appreciated. I tried using
rules: - if: $SSH_PRIVATE_KEY != "" && $S390X_ACCOUNT != ""
but the job is started (and then fails) even though those variables are unset.
The gnutls job is also disabled for now, let me know when there's a branch in the gnutls repo that aims to be compatible with nettle-4, and I'll reenable this job.
Regards, /Niels
Niels Möller nisse@lysator.liu.se writes:
rules:
- if: $SSH_PRIVATE_KEY != "" && $S390X_ACCOUNT != ""
Shouldn't that be something like this:
rules: - if: $SSH_PRIVATE_KEY != "" && $S390X_ACCOUNT != "" when: never
Note that the following
rules: - if: $SSH_PRIVATE_KEY != "" && $S390X_ACCOUNT != "" - when: never
means something completely different. Each '-' stanza is evaluated on its own, the first one to lead to a job start wins. The first 'if' line is evaluated, and then the default action of when:always applies since nothing else is present, and the job is run. So you actually get kind of the reverse results you wanted.
I'm just guessing, sorry for not catching this earlier.
/Simon
Simon Josefsson simon@josefsson.org writes:
Niels Möller nisse@lysator.liu.se writes:
rules:
- if: $SSH_PRIVATE_KEY != "" && $S390X_ACCOUNT != ""
Shouldn't that be something like this:
rules:
- if: $SSH_PRIVATE_KEY != "" && $S390X_ACCOUNT != "" when: never
No, the intention was that in the current environment without the keys and settings needed for this job, $SSH_PRIVATE_KEY != "" should be false, and rule not match. The additional default rule "- when: never" is redundant, by my reading of the docs, I tried adding it only because I read some recommendation to be explicit about the default behavior (no rule matching).
But I think I really don't get quoting, in yaml in general and in these expressions. I suspected there might be that null != "" is evaluated as true, so to get null values treated as empty strings I tried adding quotes,
- if: "$SSH_PRIVATE_KEY" != "" && "$S390X_ACCOUNT" != ""
However, that results in syntax error from gitlab. Then I tried the opposite, removing both quotes and comparisons,
- if: $SSH_PRIVATE_KEY && $S390X_ACCOUNT
That appears to be syntactically correct, and it evaluates to false. I haven't yet tested what happens if those variables are assigned non-empty values, but I'd hope this is the right way to do it.
Regards, /Niels
On 9/09/25 18:07, Niels Möller wrote:
Simon Josefsson writes:
Niels Möller writes:
...
However, that results in syntax error from gitlab. Then I tried the opposite, removing both quotes and comparisons,
- if: $SSH_PRIVATE_KEY && $S390X_ACCOUNT
That appears to be syntactically correct, and it evaluates to false. I haven't yet tested what happens if those variables are assigned non-empty values, but I'd hope this is the right way to do it.
FWIW, that matches what I understand the github action syntax to require. It seems to like variables that evaluate directly to booleans, but not equivalencies between things.
Cheers AYJ
Niels Möller nisse@lysator.liu.se writes:
For now, I have disabled the remote/s390x job, since I've not been able to mark it as conditional depending account variables, and I haven't yet setup new keys (since I want to retire the key used at gitlab.com).
This is now running again.
I've changed the job conditions to
rules: - if: $SSH_PRIVATE_KEY && $S390X_ACCOUNT
and that seems to work. And I've configured those variables in the gitlab ui (so accessible to me, and with some effort to lysator admins and to Simon who's admin on the runner machine), and that let's the job login to an unprivileged account on a test s390x vm provided by IBM, and run tests. And the ssh key previously used in the same way for runners at gitlab.com no longer has access.
I think the mirror repos at https://gitlab.com/gnutls/nettle and https://gitlab.com/gnutls/nettle-build-images can be removed or archived now.
Regards, /Niels
Niels Möller nisse@lysator.liu.se writes:
I think the mirror repos at https://gitlab.com/gnutls/nettle and https://gitlab.com/gnutls/nettle-build-images can be removed or archived now.
I archived them now, cc'ing gnutls-devel for awareness. If nobody screams within a month I suggest to remove them to reduce confusion and resource usage.
/Simon
The github mirror https://github.com/gnutls/nettle is not working either. Will that be fixed or should that be archived as well?
fooapr@outlook.com writes:
The github mirror https://github.com/gnutls/nettle is not working either. Will that be fixed or should that be archived as well?
That appears to not have pulled any changes for long time, latest commit two and a half years ago. And status of the neighbor libtasn1 repo looks similar.
Archiving or removing makes sense to me, but I don't know anything about this mirror. It looks like maybe Daiki is managing it?
Regards, /Niels
Niels Möller nisse@lysator.liu.se writes:
fooapr@outlook.com writes:
The github mirror https://github.com/gnutls/nettle is not working either. Will that be fixed or should that be archived as well?
That appears to not have pulled any changes for long time, latest commit two and a half years ago. And status of the neighbor libtasn1 repo looks similar.
Archiving or removing makes sense to me, but I don't know anything about this mirror. It looks like maybe Daiki is managing it?
Yes, the current setup (though it's no longer working) is:
git.lysator.liu.se/nettle/nettle -- (1) pull mirroring --> gitlab.com/gnutls/nettle -- (2) push mirroring --> github.com/gnutls/nettle
(2) stopped working since GitHub changed their host key[1]. After that, I wasn't able to make it work again.
(1) has recently stopped working since the GnuTLS org was not able to renew the GitLab.com OSS program[2], as the pull mirroring is one of their premium features.
If anyone knows a better way to mirror git.lysator.liu.se/nettle/nettle to github.com/gnutls/nettle, I would be happy to try.
Regards,
Footnotes: [1] https://forum.gitlab.com/t/push-mirroring-to-github-stopped-working/84621
[2] https://gitlab.com/gitlab-org/gitlab/-/issues/543742
Hi, it seems both the GitLab and GitHub mirrors are working now. Will the Nettle CI move back to GitLab or stay at git.lysator.liu.se?
Cheers Foolbar
fooapr@outlook.com writes:
Hi, it seems both the GitLab and GitHub mirrors are working now. Will the Nettle CI move back to GitLab or stay at git.lysator.liu.se?
I think I'd prefer to keep some or all at lysator, to not have to store the secret key needed for the s390x remote job at gitlab.com, and not have the mirroring delay. But it would be nice to run at both places, if I can figure out what tag(s) to set to make that work.
Regards, /Niels
Replying to an old thread:
Niels Möller nisse@lysator.liu.se writes:
fooapr@outlook.com writes:
Hi, it seems both the GitLab and GitHub mirrors are working now. Will the Nettle CI move back to GitLab or stay at git.lysator.liu.se?
I think I'd prefer to keep some or all at lysator, to not have to store the secret key needed for the s390x remote job at gitlab.com, and not have the mirroring delay. But it would be nice to run at both places, if I can figure out what tag(s) to set to make that work.
I'm not that familiar with how tags work in gitlab (and at the gitlab.com instance in particular). Previously, jobs in nettle's .gitlab-ci.yml used the tag "saas-linux-small-amd64". To work on the runner set up by Simon, which gets jobs from git.lysator.liu.se, I changed that to just "amd64".
Can I add multiple tags (and get jobs to be picked up by runners recognizing any of those tags)? And what is the right tag to have jobs running on .gitlab.com under the umbrella of the gnutls project?
Regards, /Niels
Niels Möller nisse@lysator.liu.se writes:
Replying to an old thread:
Niels Möller nisse@lysator.liu.se writes:
fooapr@outlook.com writes:
Hi, it seems both the GitLab and GitHub mirrors are working now. Will the Nettle CI move back to GitLab or stay at git.lysator.liu.se?
I think I'd prefer to keep some or all at lysator, to not have to store the secret key needed for the s390x remote job at gitlab.com, and not have the mirroring delay. But it would be nice to run at both places, if I can figure out what tag(s) to set to make that work.
I'm not that familiar with how tags work in gitlab (and at the gitlab.com instance in particular). Previously, jobs in nettle's .gitlab-ci.yml used the tag "saas-linux-small-amd64". To work on the runner set up by Simon, which gets jobs from git.lysator.liu.se, I changed that to just "amd64".
Can I add multiple tags (and get jobs to be picked up by runners recognizing any of those tags)? And what is the right tag to have jobs running on .gitlab.com under the umbrella of the gnutls project?
Runner tags is a bit inflexible in GitLab CI/CD. When specified, the .gitlab-ci.yml tags must all be met by the runner. I don't think it is possible to express "run this job on tag X or tag Y". (I would be happy if proven wrong on this!)
I think the GitLab way to deal with this is to add multiple jobs, one for each tag, and enable/disable each of them using a 'rules:' mechanism to figure out if it should run or not. This is complex so I dislike it.
I think you could edit the Runner definition and just add 'saas-linux-small-amd64' to my runner, for your project, and then change .gitlab-ci.yml to use that tag. The tags for a runner can be modified per project or per group. However this may not do the right thing for other jobs, and it is really confusing to steal someone else's tag...
You could also remove any 'tags: ' line and then enable 'Run untagged jobs' for my runner.
I must admit I forgot what you were using my amd64 runner for, but I just got approved for a permenant s390x virtual machine from IBM (the community VM's I had seems to expire after a couple of weeks...) and I will setup a proper GitLab runner on it. Assuming I'm successful with this, I'm happy for nettle to use it too. Then you don't have to store the SSH key at any gitlab site (although they still store runner secrets, which pretty much have the same capabilities, so I'm not sure if this really makes a difference).
I also have a paid GitLab.com subscription on gitlab.com/gsasl/ that I use to mirror some projects for CI. Adding a nettle mirror there would be simple, and then CI would just appear automatically without you doing anything. In fact, I'll see if this works now... yes it works, but stuck because of the tags:
https://gitlab.com/gsasl/nettle/-/pipelines/2264710712
Thus I would recommend to just remove the 'tag: amd64' from your .gitlab-ci.yml, enable 'Run untagged jobs' for my amd64 runner if you want to keep it, and things should work everywhere?
/Simon
Simon Josefsson simon@josefsson.org writes:
I must admit I forgot what you were using my amd64 runner for,
I use if for a dozen build jobs, both native builds in various configurations, and cross builds with qemu tests. And then the special s390x job that uses ssh to my IBM test vm.
just got approved for a permenant s390x virtual machine from IBM (the community VM's I had seems to expire after a couple of weeks...) and I will setup a proper GitLab runner on it. Assuming I'm successful with this, I'm happy for nettle to use it too.
That would be nice!
Thus I would recommend to just remove the 'tag: amd64' from your .gitlab-ci.yml, enable 'Run untagged jobs' for my amd64 runner if you want to keep it, and things should work everywhere?
If I rememer correctly, the reason I originally added tags to these jobs were that at gitlab.com the jobs were sometimes picked up by runners on macos or windows, and then they failed miserably.
I've now tried adding the "Run untagged jobs" for the runner. So if I understand it correctly, the tags don't belong to the runner machinery itself, they are attached on the association between runner machines and gitlab projects?
I've also pushed a branch ci-delete-tags-amd64 where I delete the tags from .gitlab-ci.yml.
But then I think managed to somehow botch something on the gitlab instance related to the nettle project or my account (possible pressing reload at the wrong time when changing the ci/cd settings), and at the moment I get 403 on everything when logged in (if I open a new private browser tab, I get working public view, which goes back to 403 on everything once I login). So I'm a bit stuck there at the moment.
I see that I now get jobs started at gitlab.com, but they fail because
BUILDENV_NATIVE_IMAGE: nettle/build-images:buildenv-native
image: $CI_REGISTRY/$BUILDENV_NATIVE_IMAGE
is expended to a reference that doesn't exist. It probably needs to point explicitly to the registry on git.lysator.liu.se (unless I move image building and hosting elsewhere).
And some of the jobs at git.lysator.liu.se picked up by your runner appear to suffer from a similar 403 problem as interactive access,
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "git.lysator.liu.se:5050/nettle/build-images:buildenv-native" for container build: image pull failed: Back-off pulling image "git.lysator.liu.se:5050/nettle/build-images:buildenv-native": ErrImagePull: Requesting bear token: invalid status code from registry 403 (Forbidden). Check https://docs.gitlab.com/runner/shells/#shell-profile-loading for more information
Weird, and I probably need some assistance from local sysadm.
Regards, /Nies
Niels Möller nisse@lysator.liu.se writes:
I've now tried adding the "Run untagged jobs" for the runner. So if I understand it correctly, the tags don't belong to the runner machinery itself, they are attached on the association between runner machines and gitlab projects?
I've also pushed a branch ci-delete-tags-amd64 where I delete the tags from .gitlab-ci.yml.
Now builds successfully at git.lysator.liu.se (and the weird 403 problem has disappeared).
I see that I now get jobs started at gitlab.com, but they fail because
BUILDENV_NATIVE_IMAGE: nettle/build-images:buildenv-native
image: $CI_REGISTRY/$BUILDENV_NATIVE_IMAGE
I've pushed another change to that branch, replacing $CI_REGISTRY with a reference to git.lysator.liu.se:5050/. I wonder if there are any potential problems with having gitlab.com (and any other mirrors) pull images from that server? One would hope gitlab runners cache images, but I don't know. The server should have plenty of bandwidth, connected to the university network.
And now ci it gitlab.com is working too for this branch, see https://gitlab.com/gnutls/nettle/-/pipelines/2268610930. If I look at a random job, it is picked up by the runner "#12270845 (JLgUopmM) 1-green.saas-linux-small-amd64.runners-manager.gitlab.com/default". So it seems no tagging is required.
All jobs at gitlab.com succeeded except remote/s390x, which failed because I still had CI-variables set with a revoked ssh key. I've now deleted those variables, and job should hopefully be skipped based on job rules on next attempt.
And I also checked that the github mirror appears up-to-date, https://github.com/gnutls/nettle.
Thanks for the advice!
Regards, /Niels
Niels Möller nisse@lysator.liu.se writes:
I must admit I forgot what you were using my amd64 runner for,
I use if for a dozen build jobs, both native builds in various configurations, and cross builds with qemu tests. And then the special s390x job that uses ssh to my IBM test vm.
Is this still true now? I see
https://git.lysator.liu.se/nettle/nettle/-/jobs/4086
that lysator's shared runners appears to be used. I don't know their availability, no problem keeping my runner in the configuration.
just got approved for a permenant s390x virtual machine from IBM (the community VM's I had seems to expire after a couple of weeks...) and I will setup a proper GitLab runner on it. Assuming I'm successful with this, I'm happy for nettle to use it too.
That would be nice!
I'm still waiting for the VM, but at they confirmed I was eligble.
I've now tried adding the "Run untagged jobs" for the runner. So if I understand it correctly, the tags don't belong to the runner machinery itself, they are attached on the association between runner machines and gitlab projects?
Yes. In earlier times the tags came from the runner, not the gitlab project configuration. I think this is still somewhat confused area.
I see that I now get jobs started at gitlab.com, but they fail because
BUILDENV_NATIVE_IMAGE: nettle/build-images:buildenv-native
image: $CI_REGISTRY/$BUILDENV_NATIVE_IMAGE
I've pushed another change to that branch, replacing $CI_REGISTRY with a reference to git.lysator.liu.se:5050/. I wonder if there are any potential problems with having gitlab.com (and any other mirrors) pull images from that server? One would hope gitlab runners cache images, but I don't know. The server should have plenty of bandwidth, connected to the university network.
Runners cache images, but gitlab.com has many runners so in practice the end result vary. I wouldn't worry about this traffic.
And now ci it gitlab.com is working too for this branch, see https://gitlab.com/gnutls/nettle/-/pipelines/2268610930. If I look at a random job, it is picked up by the runner "#12270845 (JLgUopmM) 1-green.saas-linux-small-amd64.runners-manager.gitlab.com/default". So it seems no tagging is required.
All jobs at gitlab.com succeeded except remote/s390x, which failed because I still had CI-variables set with a revoked ssh key. I've now deleted those variables, and job should hopefully be skipped based on job rules on next attempt.
And I also checked that the github mirror appears up-to-date, https://github.com/gnutls/nettle.
The pipeline for the mirror I added is green, without me doing anything - so I think your configuration is good:
https://gitlab.com/gsasl/nettle/-/pipelines
Given that gitlab.com/gnutls finally approved for the OSS Ultimate plan, I suspect there is no point in having a gitlab.com/gsasl/nettle mirror for pipelines, so I will remove the gsasl one.
/Simon
Simon Josefsson simon@josefsson.org writes:
Is this still true now? I see
https://git.lysator.liu.se/nettle/nettle/-/jobs/4086
that lysator's shared runners appears to be used. I don't know their availability, no problem keeping my runner in the configuration.
Ah, I hadn't noticed. When I initially enabled ci at git.lysator.liu.se, that shared runner failed the nettle jobs in ways I didn't understand, and appeared to require additional config. Good that it can be used now!
I think some jobs are still picked up by your runner, most recent being https://git.lysator.liu.se/nettle/nettle/-/jobs/4060. It's good to have some redundancy.
Runners cache images, but gitlab.com has many runners so in practice the end result vary. I wouldn't worry about this traffic.
I would like to understand caching better (but it's more of an issue for gitlab at work; there it would be desirable to also let neighbor runners have a shared cache for all of container images, go packages, debian packages downloaded as part of tests, etc).
The pipeline for the mirror I added is green, without me doing anything
- so I think your configuration is good:
Nice.
Given that gitlab.com/gnutls finally approved for the OSS Ultimate plan, I suspect there is no point in having a gitlab.com/gsasl/nettle mirror for pipelines, so I will remove the gsasl one.
I agree, that seems like a waste of electricity.
Regards, /Niels
nettle-bugs@lists.lysator.liu.se
-
Amos Jeffries -
Daiki Ueno -
fooapr＠outlook.com -
Niels Möller -
Simon Josefsson