Local Modularity Environment

Setting up a Modularity environment

If you decide to get involved in the Modularity project, sooner or later you'll be adding features or fixing bugs in the software components used to build modules. Every change should be tested locally and also have a testcase for automated testing. But setting up a local environment for Modularity can be challenging as there are a couple of components involved:

• fedpkg You need a special version that understands the module-build parameter. It is possible to circumvent this by sending a specially formatted string to a web URL where rida listens. fedpkg also handles certificates that are used to decide if this user is allowed to build a module.
• pdc pdc or product-definition-center gets fed by fedmsg notices and is a repository and API for storing and querying product metadata
• fedmsg fedmsg needs to be reconfigured to listen to messages from the Fedora staging environment
• rida The so called 'orchestrator' that accepts input from fedpkg, checks permissions, gets a list of build time dependencies, sets up a buildroot in koji and submits a build in koji.
• ridad ridad is part of the orchestrator and listens to messages on the fedmsg bus and acts on them
• koji koji creates buildroots and builds packages and modules in them. Setting up koji is not part of this exercise as you would need to download all binary and source packages of a given distribution and import them into a local koji. Instead of doing this, we'll use Fedora's staging koji environment, koji.stg.fedoraproject.org. Unfortunately this means at the moment that only certain users with admin privileges can build modules. Work is under way to fix this issue.

PDC

product-definition-center in a development environment needs to be configured to bypass all authentications. This can be done by adding a file pdc/settings_local.py in the git checkout directory of pdc with the following content:

import os.path
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
DEBUG = True
DEBUG_USER = 'superuser'
DISABLE_RESOURCE_PERMISSION_CHECK = True
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        # 'USER': '',
        # 'PASSWORD': '',
        # 'HOST': '',
        # 'PORT': '',
    }
}
REST_API_URL = 'rest_api/'
REST_API_VERSION = 'v1'
BROWSABLE_DOCUMENT_MACROS = {
    'HOST_NAME': 'http://localhost:8000',
    'API_PATH': '%s%s' % (REST_API_URL, REST_API_VERSION),
}
def get_setting(setting):
    import settings
    return getattr(settings, setting)
ADMINS = (('PDC Dev', 'pdc@example.com'),)
EMAIL_HOST = 'smtp.example.com'
SERVER_EMAIL = 'noreply@example.com'
EMAIL_SUBJECT_PREFIX = '[PDC]'
del get_setting('REST_FRAMEWORK')['DEFAULT_PERMISSION_CLASSES']"

Note the DEBUG_USER line, this user needs to be created after initializing the pdc database with

python manage.py migrate

with

python manage.py createsuperuser

(follow the instructions and enter 'superuser' as name and a password and emailaddress

fedmsg

the modularity devel environent requires that fedmsg listens to messages coming from the fedora staging environment. Change /etc/fedmsg.d/endpoints.py so that the fedora-infrastructure endpoint points at tcp://stg.fedoraproject.org:9940  :

config = dict(
    endpoints={
        "fedora-infrastructure": [
            #"tcp://hub.fedoraproject.org:9940",
            "tcp://stg.fedoraproject.org:9940",
        ],
    },
)

For debugging it is also useful to change all fedmsg debugging levels in /etc/fedmsg.d/logging.py to 'DEBUG'

fedmsg also needs to relay the incoming messages to the local system. This can be done by modifying /etc/fedmsg.d/relay.py so that it looks like this:

config = dict(
    endpoints={
        "relay_outbound": [
            "tcp://0.0.0.0:4001",
        ],
    },
    relay_inbound=[
        "tcp://127.0.0.1:2003",
    ],
)

I've also set validate_signatures in /etc/fedmsg.d/ssl.py to False, although I'm not sure if that is really required.

rida (orchestrator)

Rida needs to get metadata from the local pdc. Therefore config.py in the rida source directory needs to be changed. Replace

PDC_URL = 'http://modularity.fedorainfracloud.org:8080/rest_api/v1'

with

PDC_URL = 'http://127.0.0.1:8000/rest_api/v1'

This URL also needs to be changed in manage.py, replace

cfg.pdc_url = "http://modularity.fedorainfracloud.org:8080/rest_api/v1"

with

cfg.pdc_url = "http://127.0.0.1:8000/rest_api/v1"

Rida looks for its config files in /etc/rida /etc/rida/rida.conf should look like this:

[DEFAULT]
system = koji
messaging = fedmsg
koji_config = /etc/rida/koji.conf
koji_profile = koji
db = sqlite:///rida.db
pdc_url = http://127.0.0.1:8000/rest_api/v1
pdc_insecure = True
pdc_develop = True
scmurls = ["git://pkgs.stg.fedoraproject.org/modules/"]
# Where we should run when running rida.py directly.
host = 127.0.0.1
port = 5000
# How often should we resort to polling, in seconds
# Set to zero to disable polling
polling_interval = 60
rpms_default_repository = git://pkgs.fedoraproject.org/rpms/
rpms_allow_repository = False
rpms_default_cache = http://pkgs.fedoraproject.org/repo/pkgs/
rpms_allow_cache = False
ssl_enabled = True
ssl_certificate_file = server.crt
ssl_certificate_key_file = server.key
ssl_ca_certificate_file = cacert.pem
pkgdb_api_url = https://admin.stg.fedoraproject.org/pkgdb/api
# Available backends are: console, file, journal.
log_backend = journal
# Path to log file when log_backend is set to "file".
log_file = rida.log
# Available log levels are: debug, info, warn, error.
log_level = info
ridaurl = http://127.0.0.1:5000

This references /etc/rida/koji.conf which looks like this:

[koji]
;configuration for koji cli tool
;url of XMLRPC server
server = http://koji.stg.fedoraproject.org/kojihub
;url of web interface
weburl = http://koji.stg.fedoraproject.org/koji
;url of package download site
topurl = http://kojipkgs.fedoraproject.org/
;configuration for SSL authentication
authtype = ssl
; XXXX These are currently special certs with admin priviledges. If you have access, grab them
; from modularity.fedorainfracloud.org. This is supposed to work with the normal 
; fedora certs later on
;client certificate
cert = /home/karsten/Modularity/fm-orchestrator/.fedora.cert
;certificate of the CA that issued the client certificate
ca = /home/karsten/Modularity/fm-orchestrator/.fedora-server-ca.cert
;certificate of the CA that issued the HTTP server certificate
serverca = /home/karsten/Modularity/fm-orchestrator/.fedora-server-ca.cert

minor suggestions, not required

You might also want to restrict your builds to only one arch to speed things up and lessen the load on the Fedora staging infrastructure. Remove all but one (x86_64) arch from KOJI_ARCHES in config.py
It might also help to change the LOG_LEVEL to debug in the same file.