Special Interest Groups

Special Interest Groups¶

CentOS clearly defines what a SIG (Special Interest Group) is : a group of people sharing interest into a specific task/tech area, and willing to build/test/deliver artifacts (like RPM packages) that CentOS users can then consume.

Each SIG must be approved by the CentOS board, and no action should be taken before that. Once approved and CentOS board meeting minutes published, we can (Infra and Releng) proceed with next steps :

SIG onboarding¶

Creating SIG authentication Group¶

We first need to create the SIG group in IPA (if it doesn't exist yet) and it shouldn't of course conflict with any existing. Requirements:

a user with admin right in IPA to be able to create the group
a shell account on an enrolled machine (like ipa-shell.iad2.centos.org but can be any enrolled node within the @FEDORAPROJECT.ORG REALM (verify that you have a valid krb5 ticket or next steps will fail)
a SIG name and description
at least one SIG member that will be the chair (meaning that he'll be himself then able to add/remove other SIG users in that group

We can use either the IPA UI, or for automation purposes,just use the ipa cli. Basicall three commands:

ipa group-add : to create the group
ipa group-add-member : to add the first user
ipa group-add-member-manager : to grant "sponsor" right to sig chair, himself then able to add users

Here is a full output/example about creating the sig-kmods with one chair (valid IPA/FAS account) :

# Defininig first some variables
sig_name="sig-kmods"
sig_description="The CentOS kmods SIG"
sig_chair="pjgeorg"

# Now creating group and granting rights
ipa group-add --desc="${sig_description}" --fasgroup ${sig_name}
-----------------------
Added group "sig-kmods"
-----------------------
  Group name: sig-kmods
  Description: The CentOS kmods SIG
  GID: 1494370508
  FAS group: True

ipa group-add-member ${sig_name} --users=${sig_chair}
  Group name: sig-kmods
  Description: The CentOS kmods SIG
  GID: 1494370508
  Member users: pjgeorg
  Membership managed by users: pjgeorg
-------------------------
Number of members added 1
-------------------------

ipa group-add-member-manager ${sig_name} --users=${sig_chair}
  Group name: sig-kmods
  Description: The CentOS kmods SIG
  GID: 1494370508
  Membership managed by users: pjgeorg
-------------------------
Number of members added 1
-------------------------

If you need to add later more "sponsor" user, just repeat the ipa group-add-member-manager operation with new user.

At this stage, the group should appear throug portal on https://accounts.centos.org

Worth knowing also that in the next minutes, an automatic email alias is created for the group following the naming convention sig-<name>@centosproject.org and that will automatically include all sig members email addresses, once added (and removed when removed)

Same for SIG members : an email alias following the <user>@centosproject.org alias is also created for all SIG members (and pointing to the email address they used when they registered their ACO/FAS account) More informations about this in the Ansible postfix role

Don't forget to also add the new sig- to the centosproject-email-aliases group as indirect member so that all new SIG members will be granted an email alias, based ont he role described above.

Creating SIG namespace on git.centos.org¶

Each SIG can push to specific branches on git.centos.org (computed automatically from some sig prefixes and sig name). To allow a new SIG group to have the automatic right in git.centos.org (and reflect group membership there though centralized authentication), we need to just add sig-<name> to the existing pagure_supported_sigs ansible list/vars in Ansible inventory.

Once pushed and applied (automatically by Ansible), when a SIG member will login again, his group membership will be synced and such user will be allowed to :

push to some specific branches
push to lookaside

More informations about this in the Git section.

Welcome email¶

While not fully needed, it's now a good idea to send mail to that new SIG to welcome them. Something like this :

To: <sig_name>@centosproject.org
Subject: Welcome to the new <sig_name> SIG !

Hi,

This welcome email means that your new group <sig_name> was created and granted some management rights to at least the SIG Chair.
The group should appear now on https://accounts.centos.org/group/<sig_name>/

Also worth knowing that each SIG member is also automatically granted an email alias following this convention : <user>@centosproject.org

If you have any needs, requests, feel free to contact the CentOS Infra and Releng team through https://pagure.io/centos-infra/issues.

Kind Regards,

Security GPG key for packages signing¶

Each SIG building RPM packages through https://cbs.centos.org community build service will need a GPG security key that will be then automatically be used by the signing service

Danger

Due to sensitive information ,the following steps are restricted to a very limited number of people and on a restrictive node that doesn't have any direct inbound connection. Worth knowing that such node isn't reachable directly by the central ansible automation system, so ansible automation has to be ran "on demand" by people taking following actions.

We first need to create a new GPG key on the signing machine , protected with a strong passphrase, that will be needed to unlock signing service/agent.

To generate a strong passphrase you can for example use openssl rand -base64 30 (or any other strong/long passphrase generator)

Let's use the sig-kmods SIG as an example for the next steps (and of course adapt on each operation, manual step) We first need to ssh as the signing user (important for TTY) on the signing service machine. Then we can proceed like this :

# Let's define some variables first
sig_name="CentOS Kmods SIG"
sig_url="https://wiki.centos.org/SpecialInterestGroup/Kmods"
sig_key_passphrase="Of_Course_A_Real_Passphrase_generated_as_described_above" 
# Creating the key in keyring
gpg --batch --gen-key <<EOF
Key-Type: 1
Key-Length: 2048
Subkey-Type: 1
Subkey-Length: 2048
Name-Real: ${sig_name}
Name-Comment: ${sig_url}
Name-Email: security@centos.org
Passphrase: ${sig_key_passphrase}
Expire-Date: 0
EOF

gpg: key 7AE06D54 marked as ultimately trusted

As we can see the key was created but we have first to unlock it now (in gpg-agent) with the generated passphrase, otherwise CBS signing service will be blocked (replace obviously gpg key id with the output above) :

# Unlocking the key
key_id=$(gpg --list-keys | egrep -B 1 "${sig_name}"|head -n1|tr -d [:blank:])
test_file=$(mktemp)
timeout 30 gpg --batch --sign -b --use-agent --local-user ${key_id}! ${test_file} && rm ${test_file} && echo "Signing OK" || echo "Signing failed"

Now that CBS signing service is ready, we have to export the new key and so push it to the website (so that people can see it publicly to verify and also have it embedded in a centos-release-* rpm pkg :

# extract key
gpg --export --armor ${key_id} > /var/tmp/RPM-GPG-KEY-CentOS-SIG-$(echo ${sig_name/CentOS/}|sed 's/SIG//')
gpg --with-fingerprint /var/tmp/RPM-GPG-KEY-CentOS-SIG-$(echo ${sig_name/CentOS/}|sed 's/SIG//')

pub  2048R/7AE06D54 2021-06-22 CentOS Kmods SIG (https://wiki.centos.org/SpecialInterestGroup/Kmods) <security@centos.org>
      Key fingerprint = 48EF 712E C5DD B68B 5280  BE45 5B8E 1A76 7AE0 6D54
sub  2048R/E46DCBCB 2021-06-22

Now either submit a PR or directly do a git commit/push to CentOS.org git repository and key will be available on website automatically after. Simply drop the RPM-GPG-KEY-CentOS-SIG-<name> under /keys/ directory and update /keys.md to add the link to download the key and show fingerprint

Here is the Kmods SIG commit given a reference

Warning

Don't forget to also put the gpg passphrase in the git-crypted pkistore as it will be needed each time that we'll have to unlock the gpg keys in keyring (like on a reboot of the signing machine/service)

Last step is to then play the stylo role with ansible after having updated the signing node stylo_gpg_sigs list in host_vars. Attention that gpg key id has to be written in lowercase though :

update ansible inventory with new gpg keyid / sig name
ansible-playbook playbooks/role-stylo.yml --tags "config"

CBS Koji tags¶

Overview¶

CBS uses a specific naming convention for the koji tags that will be created. It's important that we follow that specific convention because the signing/pushing automated process parses the tags to know which gpg SIG key to use, and where to push content that will appear on the mirror network.

Tag structure:

SIG name
CentOS Linux/Stream release (abbreviated)
SIG project
SIG project release
tag level that can be either candidate , testing or release

What's the influence of the tag level mentioned above ?

candidate : ignored by signing service, build stays in koji, used to validate build itself or just internal unit-test happening through ci.centos.org for SIGs doing their tests in that environment
testing : processed by signing service but packages aren't signed, and just land on https://buildlogs.centos.org CDN, so just to have publicly available pkgs that can be promoted to release by SIG later
release : processed by signing service : packages are gpg signed with SIG key, and content appear on the mirror CDN and can be consummed by all

SIG koji tags¶

Creating tags¶

To create tags, we have in place a tool that is maintained through ansible-role-kojihub and distributed on the kojihub node itself. That means that to be able to use it, you need to have local shell access and kick the following commands as the koji user , or eventually have admin rights and kick the tool from outside

/opt/cbs-tools/scripts/create_sig.sh 
usage: /opt/cbs-tools/scripts/create_sig.sh -d <distribution> -s <signame(s)> -t <tag(s)>

This script generate new build target in koji for SIGS.

OPTIONS:
   -a   ARCHES                   : Force arches e.g : "x86_64","aarch64 i686", etc...
   -s   SIG NAME                 : cloud
   -d   DISTRIBUTION             : 7 8 8s 9 9s (8 and 9 will use RHEL buildroots)
   -p   SIG PROJECT NAME         : cloud6-<openstack>, sclo-<mariadb100>, etc...
   -r   SIG PROJECT RELEASE NAME : cloud6-openstack-<juno>
   -t   DISTTAGS                 : el7 el7.centos el8_0 el8s el9s el9
   -e   EXTREPOS         : Additional repositories to add , e.g 'epel8 epel8-next' 
   -i   INHERITANCE              : Additional tag-inheritance , e.g 'nfv9s-openvswitch-2-release'
   -b                            : Enable non public bootstrap repo (SCLO SIG only)
   -c   COLLECTION               : Enable collection in the buildroot e.g : mariadb100
   -x                            : delete old -build tag and then recreate.

Let's use the Kmods SIG again as an example :

/opt/cbs-tools/scripts/create_sig.sh -a "x86_64" -s kmods -d 8s -p packages -r main -t el8
* Checking distribution el8s configuration...
 -> Checking kmods config...
Creating user : kmods
Using default options for kmods/packages
Creating tag  : kmods8s-packages-main-candidate
Creating tag  : kmods8s-packages-main-testing
Creating tag  : kmods8s-packages-main-release
 -> creating kmods8s-packages-main-el8
Added external repo centos8s-cr to tag kmods8s-packages-main-el8-build (priority 5)
Added external repo centos8s-extras to tag kmods8s-packages-main-el8-build (priority 10)
Added external repo centos8s-powertools to tag kmods8s-packages-main-el8-build (priority 15)
Added external repo centos8s-appstream to tag kmods8s-packages-main-el8-build (priority 20)
Added external repo centos8s-baseos to tag kmods8s-packages-main-el8-build (priority 25)

Note

The -e and -i (external repos and inheritance) parameters are optional but useful when some SIGs are asking to depend directly on additional existing tags/repos.

Danger

If you have to create tags for SCLs, see the note below as we have to consolidate tags into specific one for signing purpose. We'll eventually backport that into automation script later

SCLs specific tags¶

The Software Collection SIG has some specific requirements and pkgs that need to be tagged in buildroot and also tags being consolidated when processed for signing (all lands into one repository and not multiple ones, even if there quite some SCL tags.

Example below that we need to do when we have a request for a new SCL , and let's assume that it's a RH collection.We have already created the tag as in the process above, but we'll need to tune these tags a little bit and let's so first declare some variables :

tag="sclo7-rh-redis6-rh"
collection="rh-redis6"
origin="rh" # can be sclo or rh, needed below
koji add-group-pkg ${tag}-el7-build srpm-build scl-utils-build ${collection}-build
koji add-group-pkg ${tag}-el7-build build scl-utils-build ${collection}-build

for level in testing release ;  do
  last_priority=$(koji taginfo sclo7-all-${origin}-${level}|tail -n 1 |awk '{print $1}')
  koji add-tag-inheritance sclo7-all-${origin}-${level} ${tag}-${level}  --priority $(( $last_priority + 5 ))
done

Modifying existing tags¶

It's always possible to modify some existing tags through standard koji (or cbs alias) CLI calls. Example : let's add additional architectures for the Kmods SIG :

for i in el8-build candidate testing release ; do koji edit-tag --arches="x86_64 ppc64le aarch64" kmods8s-packages-main-${i}; koji taginfo kmods8s-packages-main-${i}|grep Arches ; done

If some SIGs want to depend on external repo, like to get all -devel packages from CentOS Stream, we can also add it :

externalrepo_name="centos8s-buildroot"
externalrepo_url="http://kojifiles.rdu2.centos.org/kojifiles/repos/dist-c8-stream-build/latest/\$arch"
sig_tag="hyperscale8s-packages-experimental-el8-build"
koji add-external-repo --mode=bare --tag ${sig_tag} ${externalrepo_name} ${externalrepo_url}
koji taginfo ${sig_tag}

If external repo is already defined, we just need to add it to tags and no need to redefine it :

sig_tag="hyperscale8s-packages-main-el8-build"
externalrepo_name="epel8"
koji add-external-repo --mode bare --tag=${sig_tag} ${externalrepo_name}

We can also modify the rpm dist tag through rpm macro that will be added to the specific tag :

tag="automotive9s-packages-experimental-el9s-build"
cbs edit-tag ${tag} --extra rpm.macro.dist='.el9iv'
# confirm with `cbs taginfo $tag`

Media spins with Kiwi¶

If SIGs are asking to be able to produce media with kiwi plugin, we have to ensure that there is at least one (or more) kojid builders which has/have the kiwi plugin enabled (through role). Then at the SIGs tag level, we need to ensure that we add the mandatory kiwi-build group. (RPM packages list can vary depending on SIG needs but kiwi-cli and kiwi-systemdeps are mandatory)

You can now use the /opt/cbs-tools/scripts/cbs-add-kiwi-support (interactive script asking for needed informations) script to do this for you

Warning

Worth knowing that some of these deps are satisfied only through Epel repository so it should first be enabled as external repo to the build tag, otherwise they'll not be installed in the buildroot

That also means that one usually have to import comps.xml files (defining groups/packages) so that they'd be available in kickstart definitions. To add comps to a build tag, you can proceed like this (example for epel repository):

/opt/cbs-tools/scripts/cbs-import-comps -t altimage9s-images-live-el9s-build -r epel