Skip to content


This how-to is WIP and based on

The setup is as follows: * Currently we are testing mbs 2.32 * CentOS-8 installed * Firewall allowing communication between the hosts * Public network for access and private network for koji, psql and kerberos communication between hosts * hosts mbs and fedmsg * SSL is setup for web facing services using let's encrypt and there is a directory services setup, you need to sort this one out

Installing mbs and fedmsg

!!!THIS IS A QUICK GUIDE FOR TESTING ONLY!!! * Currently we are installing using pip from the source, however the latest version is available at: * Enable the current temp rockylinux-tools repo by running dnf copr enable nalika/rockylinux-tools * Install the needed packages: dnf install epel-release -y && dnf install fedmsg python3-gssapi git httpd mod_ssl python3-mod_wsgi python3-solv python3-pungi python3-psycopg2 mod_auth_gssapi -y

git clone
cd fm-orchestrator
git checkout v2.32.0
pip3 install .

Starting fedmsg

We need fedmsg-hub and fedmsg-relay to get things up and running "as far as I can tell" but I disable fedora's incoming messages and message signing * Edit /etc/fedmsg.d/ and comment out "tcp://" * Edit /etc/fedmsg.d/ as following "leave everything else as is":

            # "tcp://"
    # Start of code signing configuration
    'sign_messages': False,
    'validate_signatures': False,
* Edit /etc/fedmsg.d/ and make sure validate_signatures is set to false
config = dict(
* Edit /etc/fedmsg.d/ and set the topic_prefix="org.gnulab" and environment="prod" * Start fedmsg-hub and fedmsg-relay service
systemctl enable fedmsg-hub --now
systemctl enable fedmsg-relay --now

Apache configuration for mbs-frontend

  • Create a new file /etc/httpd/conf.d/mbs.conf with the following:
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
      WSGIDaemonProcess mbs user=mbs group=mbs threads=5
        WSGIScriptAlias / /etc/module-build-service/mbs.wsgi
      WSGIPassAuthorization on
        <Directory /etc/module-build-service>
            WSGIProcessGroup mbs
            WSGIApplicationGroup %{GLOBAL}
            Require all granted
        <Location />
            AuthType GSSAPI
            AuthName "GSSAPI Single Sign On Login"
            GssapiCredStore keytab:/etc/koji.keytab
            Require valid-user
    SSLCertificateFile /etc/letsencrypt/live/
    SSLCertificateKeyFile /etc/letsencrypt/live/
    Include /etc/letsencrypt/options-ssl-apache.conf
  • Create the wsgi file /etc/module-build-service/mbs.wsgi as follows:
    import logging
    from module_build_service import app as application

Kerberos settings

  • Make sure that /etc/krb5.conf as the correct realm and settings as following:
    # To opt out of the system crypto-policies configuration of krb5, remove the
    # symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated.
    includedir /etc/krb5.conf.d/
        default = FILE:/var/log/krb5libs.log
        kdc = FILE:/var/log/krb5kdc.log
        admin_server = FILE:/var/log/kadmind.log
        dns_lookup_realm = false
        ticket_lifetime = 24h
        renew_lifetime = 7d
        forwardable = true
        rdns = false
        pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
        spake_preauth_groups = edwards25519
        default_realm = GNULAB.ORG
        default_ccache_name = KEYRING:persistent:%{uid}
     GNULAB.ORG = {
         kdc =
         admin_server =
    [domain_realm] = GNULAB.ORG = GNULAB.ORG

General mbs's setup

  • Create mbs user and set a password for it as following:
    useradd mbs
    passwd mbs
  • Fix the permissions for /etc/module-build-service/ to mbs: chown -R mbs:mbs /etc/module-build-service/

Postgresql configuration

I am using the same database server for koji which is hosted on * Create the required access by editing /var/lib/pgsql/data/pg_hba.conf as the following:

# IPv4 local connections:
host    mbs             mbs              md5
host    all             all               ident
* Edit /var/lib/pgsql/data/postgresql.conf to allow listening on network listen_addresses = 'localhost,' * Create mbs pgsql user and database
createuser --no-superuser --no-createrole --no-createdb mbs
createdb -O mbs mbs
psql  -c "alter user mbs with encrypted password 'mysupersecretepasswordmbs';"
* Restart pgsql systemctl restart postgresql

mbs configuration

  • Edit /etc/module-build-service/koji.conf as the follows:
    ;configuration for koji cli tool
    ;url of XMLRPC server
    server =
    ;url of web interface
    weburl =
    ;url of package download site
    topurl =
    authtype = kerberos
    krb_rdns = false
    use_fast_upload = true
    server =
    weburl =
    topurl =
    authtype = kerberos
    krb_rdns = false
    use_fast_upload = true
  • Edit /etc/module-build-service/ as follows:
    # -*- coding: utf-8 -*-
    # SPDX-License-Identifier: MIT
    from os import environ, path
    # FIXME: workaround for this moment till confdir, dbdir (installdir etc.) are
    # declared properly somewhere/somehow
    confdir = path.abspath(path.dirname(__file__))
    # use parent dir as dbdir else fallback to current dir
    dbdir = path.abspath(path.join(confdir, "..")) if confdir.endswith("conf") else confdir
    class ProdConfiguration(object):
        DEBUG = True
        # Make this random (used to generate session keys)
        SECRET_KEY = "74d9e9f9cd40e66fc6c4c2e9987dce48df3ce98542529126"
        #SQLALCHEMY_DATABASE_URI = "sqlite:///{0}".format(path.join(dbdir, "module_build_service.db"))
        SQLALCHEMY_DATABASE_URI = 'postgresql://'
        # Where we should run when running " run" directly.
        HOST = ""
        PORT = 5000
        # Global network-related values, in seconds
        NET_TIMEOUT = 120
        #DISTGITS = {"git+": ("git clone {repo_path}", "")}
        SYSTEM = "koji"
        MESSAGING = "fedmsg"  # or amq
        KOJI_CONFIG = "/etc/module-build-service/koji.conf"
        KOJI_PROFILE = "koji"
        ARCHES = ["x86_64"]
        KOJI_TAG_PREFIXES = ["module", "scrmod"]
        CHECK_FOR_EOL = False
        PDC_URL = ""
        PDC_INSECURE = False
        PDC_DEVELOP = True
        SCMURLS = ["git+", ""]
        # How often should we resort to polling, in seconds
        # Set to zero to disable polling
        POLLING_INTERVAL = 600
        # Determines how many builds that can be submitted to the builder
        # and be in the build state at a time. Set this to 0 for no restrictions
        #RPMS_DEFAULT_CACHE = ""
        RPMS_ALLOW_CACHE = False
        # Available backends are: console and file
        LOG_BACKEND = "file"
        # Path to log file when LOG_BACKEND is set to "file".
        LOG_FILE = "/tmp/module_build_service.log"
        # Available log levels are: debug, info, warn, error.
        LOG_LEVEL = "debug"
        # Allow stream override
        # Settings for Kerberos
        KRB_KEYTAB = "/etc/mbs.keytab"
        # AMQ prefixed variables are required only while using 'amq' as messaging backend
        # Addresses to listen to
        # Address for sending messages
        AMQ_DEST_ADDRESS = \
        AMQ_CERT_FILE = "/etc/module_build_service/msg-m8y-client.crt"
        AMQ_PRIVATE_KEY_FILE = "/etc/module_build_service/msg-m8y-client.key"
        AMQ_TRUSTED_CERT_FILE = "/etc/module_build_service/Root-CA.crt"
        # Disable Client Authorization
        NO_AUTH = False
        AUTH_METHOD = "kerberos"
        LDAP_URI = "ldap://"
        LDAP_GROUPS_DN = "ou=group,dc=gnulab,dc=org"
        ADMIN_GROUPS = {"packageradmin"}
        ALLOWED_GROUPS = {"packager"}
        KOJI_PROXYUSER = True
        REBUILD_STRATEGY = 'only-changed'
        KOJI_CG_BUILD_TAG_TEMPLATE = "{}-modular-updates-candidate"
        KOJI_CG_DEFAULT_BUILD_TAG = "modular-updates-candidate"
        # Extra options set for newly created Koji tags
            "mock.package_manager": "dnf",
            # This is needed to include all the Koji builds (and therefore
            # all the packages) from all inherited tags into this tag.
            # See and
            # for background.
            "repo_include_all": True,
            # Has been requested by Fedora infra in
            # Disables systemd-nspawn for chroot.
            "mock.new_chroot": 0,
            # Works around fail-safe mechanism added in DNF 4.2.7
            "mock.yum.module_hotfixes": 1,
        # DEFAULT_DIST_TAG_PREFIX = 'module_'
  • Create the logging file /tmp/module_build_service.log and set the correct permission:
    touch /tmp/module_build_service.log
    chown mbs:fedmsg /tmp/module_build_service.log
    chmod 664 /tmp/module_build_service.log
  • Restart services :
    systemctl restart fedmsg-hub
    systemctl restart fedmsg-relay
    systemctl restart httpd

Testing the module build service

We will need to create a few tags for this to work, so on the koji admin machine, run the following commands: * Creating tags:

koji add-tag module-centos-8.2.0-build
koji add-tag module-centos-8.3.0-build
* Adding the external mirrors for the build tags:
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-baseos-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-appstream-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-devel-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-ha-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-pt-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-cp-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-cr-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-extras-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-fasttrack-external\$arch/os
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-debuginfo-external\$arch/
koji add-external-repo -m bare -t module-centos-8.2.0-build module-cent-8.2-sheriflocalrepo-external\$arch/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-baseos-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-appstream-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-devel-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-ha-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-pt-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-cp-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-cr-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-extras-external\$arch/os/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-fasttrack-external\$arch/os
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-debuginfo-external\$arch/
koji add-external-repo -m bare -t module-centos-8.3.0-build module-cent-8.3-sheriflocalrepo-external\$arch/
* Add mbs user:
koji add-user mbs
* Fix the content generating for koji On Koji server, run the follwing commands
koji call addBType module
koji grant-cg-access mbs module-build-service --new

Now, on mbs server, we will need to do the following: * Upgrade mbs db mbs-manager db upgrade`` * Download the platform modulesgit clone* Switch to the needed branche and loading it

git branch -a
git checkout c8-stream-el8.2.0
mbs-manager import_module platform.yaml
git checkout c8-stream-el8.3.0
mbs-manager import_module platform.yaml
* Create the module build json file, for example **redis.json**
    "scmurl": "",
    "branch": "c8-stream-5"
* Generate your kerberos ticket
kinit snagy* Submit the job curl -X POST -H "Content-Type: application/json" -u : --negotiate -d @redis.json``` * Check Tasks on koji