Overview
Overview¶
Duffy is the middle layer running ci.centos.org that manages the provisioning, maintenance and teardown / rebuild of the hosts that are used to run the tests in the CI Cluster.
Stack¶
- Rest API
- beanstalkd message bus
- Ansible playbooks, IPMI, PXE
- API client python-cicoclient
- Reserve a bare-metal for 6 hours
- We maintain multiple versions of CentOS several different architectures for baremetal nodes in a pool ready to be consumed
- eg: CentOS 9s, 10s
Pools¶
Architecture node counts in pool:
duffy=> select pool, count(*) from nodes where state = 'ready' group by pool;
pool | count
---------------------------------+-------
virt-ec2-t2-centos-9s-x86_64 | 8
metal-ec2-c5n-centos-10s-x86_64 | 3
virt-ec2-c6g-centos-9s-aarch64 | 5
metal-ec2-c5n-centos-9s-x86_64 | 6
virt-ec2-c6g-centos-10s-aarch64 | 2
virt-ec2-t2-centos-10s-x86_64 | 8
Useful links¶
- Duffy runs on: admin.ci.centos.org
- https://github.com/centos/duffy
- https://github.com/CentOS/ansible-role-duffy
Duffy SOPs¶
Create/migrate a Duffy CI tenant¶
Create a dedicated ssh keypair
Note
we'll start using the pkistore git-crypted git repo for this, so be sure
to have that under git control
We'll just create the new ssh keypair directly into the CI pkistore repository:
project="samba"
ssh-keygen -f ocp/ssh/${project} -C ${project}@CI
Important
Don't forget to add and git commit && git push new keys in pkistore/ocp/ssh repo
This will create both private and public ssh keys, and you can now copy the public key to be inserted into Duffy DB (see below)
Create new Duffy tenant
Login to the Duffy server:
ssh duffy.ci.centos.org
Change to duffy user:
sudo su - duffy
Create tenant with the command below and save api key somewhere safe, it
outputs the key as <tenant name>: <API key>
duffy admin create-tenant <tenant name> "ssh-rsa <ssh pub-key_created_above>"
Artifacts storage box
CI tenants are allowed to upload artifacts to one storage box, so we need to
allow them to upload/rsync to it. It's all controlled by the
artifacts_projects_list list in ansible (host_vars) so don't forget to also
add project and ssh public keys there too
Duffy client configuration (external)
Connect to the host that will have duffy client:
ssh <host user>@<target host>
Install duffy client using pip:
pip3.8 install --user duffy[client]
In the home path of the user, create .config directory if it doesn’t exist
and create .config/duffy with the following content:
client:
url: https://duffy.ci.centos.org/api/v1
auth:
name: <tenant name>
key: <API key>
(Optional) To create a session, the name of the pool is required. Check the pool available executing the command:
duffy client list-pools
Request a session:
duffy client request-session pool=<name of the pool>,quantity=<number of sessions wanted>
By default this command outputs a json, but it's possible to change the
format to yaml or flat using --format. Under "node" key it's possible to
find the hostname to be used. Log in to it as root user, using ssh.
{
...output ommited...
"nodes": [
{
"hostname": "<hostname>.ci.centos.org",
"ipaddr": "<ip address>",
...output ommited...
}
When needed to retire the session, connect to your duffy client host and execute the command:
duffy client retire-session <session id>
It's possible to check the session id either when the session is requested, in the output under "session" key, or using the following command:
duffy client list-sessions
Create a duffy API/SSH key¶
This SOP covers the process of creating an API key for duffy, and adding it to the duffy database table
Requirements
- project name
Duffy Database Schemas
MariaDB [duffy]> show tables;
+-----------------+
| Tables_in_duffy |
+-----------------+
| alembic_version |
| session_archive |
| session_hosts |
| sessions |
| stock |
| userkeys |
| users |
+-----------------+
7 rows in set (0.00 sec)
MariaDB [duffy]> describe stock;
+--------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| hostname | varchar(20) | YES | | NULL | |
| ip | varchar(15) | YES | | NULL | |
| chassis | varchar(20) | YES | | NULL | |
| used_count | int(11) | YES | | NULL | |
| state | varchar(20) | YES | | NULL | |
| comment | varchar(255) | YES | | NULL | |
| distro | varchar(20) | YES | | NULL | |
| rel | varchar(10) | YES | | NULL | |
| ver | varchar(10) | YES | | NULL | |
| arch | varchar(10) | YES | | NULL | |
| pool | int(11) | YES | | NULL | |
| console_port | int(11) | YES | | NULL | |
| flavor | varchar(20) | YES | | NULL | |
| session_id | varchar(37) | YES | MUL | NULL | |
| next_state | varchar(20) | YES | | NULL | |
+--------------+--------------+------+-----+---------+-------+
16 rows in set (0.01 sec)
MariaDB [duffy]> describe users;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| apikey | varchar(37) | NO | PRI | | |
| projectname | varchar(50) | YES | | NULL | |
| jobname | varchar(50) | YES | | NULL | |
| createdat | date | YES | | NULL | |
| limitnodes | int(11) | YES | | NULL | |
+-------------+-------------+------+-----+---------+-------+
5 rows in set (0.00 sec)
MariaDB [duffy]> describe userkeys;
+------------+---------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| project_id | varchar(37) | YES | MUL | NULL | |
| key | varchar(8192) | YES | | NULL | |
+------------+---------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)
MariaDB [duffy]> describe users;
+-----------+----------------------+----------------------+------------+-------------+
| apikey | projectname | jobname | createdat | limitnodes |
+-----------+----------------------+----------------------+------------+-------------+
| xxxx-yyyy | nfs-ganesha | nfs-ganesha | 2016-02-24 | 10 |
| zzzz-aaaa | CentOS | centos_arrfab | 2015-04-17 | 10 |
+-----------+----------------------+----------------------+------------+-------------+
Steps to create a new duffy SSH key
Note
we'll start using the pkistore git-crypted git repo for this, so be sure
to have that under git control
We'll just create the new ssh keypair directly into the pkistore repository:
project="samba"
ssh-keygen -f ocp/ssh/${project} -C ${project}@CI
Important
Don't forget to add and git commit && git push new keys in pkistore/ocp/ssh repo
This will create both private and public ssh keys, and you can now copy the public key to be inserted into Duffy DB
Steps to create a new duffy API key
- The Duffy database runs on the admin.ci node:
ssh admin.ci.centos.organd once in the admin node, login in mysql - duffy db - Create user in usertable:
insert into users values(UUID(), '<projectname>', '<projectname>', NOW(), 5);
- Retrieve the api key from the users table:
select * from users where projectname="projectname";
- Copy the API key somewhere handy (we will need it)
- Using that api-key/UUID as project_id, enter ssh key of a user from the
project so that they can ssh into the machines. This process must be repeated
for every user we wish to add access to via SSH.
insert into userkeys (project_id, key) values(<project-UUID-var>, <ssh-key-var>);
This ssh key is pushed to duffy nodes - authorized keys when a tenant requests the node through api key.
Retiring a node pool¶
Duffy's config is controlled by Ansible, but there is data in the DB that needs to be cleaned up. Follow these steps:
- Update
70_nodepools.yamlin the Ansible filestore to remove the pool - Deploy the new config, so the pool is no longer in use
- Run
/home/duffy/bin/remove-pool.sh <poolname>to cleanup the DB
This will remove the historical data of that pool from the DB, which also means Zabbix will stop monitoring the pool (since it uses the DB as a source of truth).