From Fedora Project Wiki

< Docs‎ | Drafts

mNo edit summary
No edit summary
Line 1: Line 1:
<!-- page was renamed from MockSetupUsingLocalMirror
-->
<!-- page was renamed from ParagNemade/MockSetupUsingLocalMirror
-->
= Mock Setup Using Local Mirror =
= Mock Setup Using Local Mirror =


Line 42: Line 38:
#!/usr/bin/python
#!/usr/bin/python


<!--# License: GPL
# License: GPL
-->
# Author:  Rudolf Kastl
<!--# Author:  Rudolf Kastl
# default target: /mirror_repo
-->
# how?: use config file and special mirror .repo files


<!--# default target: /mirror_repo
# todo:
-->
<!--# how?: use config file and special mirror .repo files
-->


<!--# todo:
## code:
-->
# - ConfigParser
# - logging
# - legacy and updates-testing awareness
# - fix hierachy into the fedora mirror dir scheme
# - use separate configs for each release
# - progressbars
# - eta (complete, per repo, per file(?!))
## support files/packaging/integration:
# - creation of specially crafted .repo files
# - cron integration
# - 0 conf apache serving as split package
# - mock config files as split package
# - yum localmirror config files as split package
# - discuss access rights on /mirror_repo


<!--## code:
-->
<!--# - ConfigParser
-->
<!--# - logging
-->
<!--# - legacy and updates-testing awareness
-->
<!--# - fix hierachy into the fedora mirror dir scheme
-->
<!--# - use separate configs for each release
-->
<!--# - progressbars
-->
<!--# - eta (complete, per repo, per file(?!))
-->
<!--## support files/packaging/integration:
-->
<!--# - creation of specially crafted .repo files
-->
<!--# - cron integration
-->
<!--# - 0 conf apache serving as split package
-->
<!--# - mock config files as split package
-->
<!--# - yum localmirror config files as split package
-->
<!--# - discuss access rights on /mirror_repo
-->
<!--################
-->
<!--# special cases:
-->
<!--# - the fedora core repos differ in logic -> core repos have updates, updates-testing, legacy -> to be investigated
-->
<!--# - development is another special case not having updates updates-testing and legacy
-->
<!--# - other repos (3rd party) never have updates and updates-testing
-->
<!--# - SRPMS are another special case -> same for each arch of the same release
-->
<!--###########################
-->


################
# special cases:
# - the fedora core repos differ in logic -> core repos have updates, updates-testing, legacy -> to be investigated
# - development is another special case not having updates updates-testing and legacy
# - other repos (3rd party) never have updates and updates-testing
# - SRPMS are another special case -> same for each arch of the same release
###########################




Line 111: Line 78:
subflavourlist=["binary"]  
subflavourlist=["binary"]  


<!--### block for parsing commandline switches
### block for parsing commandline switches
-->
from optparse import OptionParser
from optparse import OptionParser


parser = OptionParser()
parser = OptionParser()
<!--# var can be accessed with options.filename
# var can be accessed with options.filename
-->
 
parser.add_option("-d", "--dest", dest="destination", help="Sync Destination", metavar="dir", default="/mirror_repo")
parser.add_option("-d", "--dest", dest="destination", help="Sync Destination", metavar="dir", default="/mirror_repo")
parser.add_option("-n", "--newonly", action="store_true", dest="newonly", help="Sync Destination", default=False)
parser.add_option("-n", "--newonly", action="store_true", dest="newonly", help="Sync Destination", default=False)
Line 124: Line 90:


(options, args) = parser.parse_args()
(options, args) = parser.parse_args()
<!--### end bloc
### end bloc
-->
 


targetlocation = options.destination
targetlocation = options.destination
Line 132: Line 98:
print "syncing: Fedora " + version, flavour, arch, subflavour, "to", targetlocation
print "syncing: Fedora " + version, flavour, arch, subflavour, "to", targetlocation


<!--# handling the special cases
# handling the special cases
-->
 
if subflavour == "binary":
if subflavour == "binary":
subflavourid = ""
subflavourid = ""
Line 144: Line 110:
subflavourpath = "-"+subflavour
subflavourpath = "-"+subflavour


<!--# devel is special
# devel is special
-->
 
if version == "development":
if version == "development":
repoid="mirror-"+flavour+"-"+version+subflavourid
repoid="mirror-"+flavour+"-"+version+subflavourid
Line 154: Line 120:
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+subflavourpath
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+subflavourpath
else:
else:
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+"/"+arch+subflavourpath
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+"/"+arch+subflavourpath# handling the newonly option of reposync


<!--# handling the newonly option of reposync
-->
if options.newonly == True:
if options.newonly == True:
newonly = " -n"
newonly = " -n"
Line 163: Line 127:
newonly = ""
newonly = ""


<!--# building the command for syncing a single repo
# building the command for syncing a single repo
-->
syncdistro="mirrorreleasever="+version+" mirrorbasearch="+arch+" /usr/bin/reposync"+newonly+" -r "+repoid+" -p "+targetdir
syncdistro="mirrorreleasever="+version+" mirrorbasearch="+arch+" /usr/bin/reposync"+newonly+" -r "+repoid+" -p "+targetdir


<!--# for testing we just print instead of sync
# for testing we just print instead of sync
-->
 
if options.testonly == True:
if options.testonly == True:
print syncdistro
print syncdistro
else:
else:
<!--# create potentially missing paths
# create potentially missing paths
-->
 
try:
try:
if os.path.isdir(targetdir) == False:
if os.path.isdir(targetdir) == False:
Line 182: Line 145:
os.system(syncdistro)
os.system(syncdistro)


<!--## main loop
## main loop
-->
 
for version in versionlist:
for version in versionlist:
for flavour in flavourlist:
for flavour in flavourlist:
for arch in archlist:
for arch in archlist:
for subflavour in subflavourlist:
for subflavour in subflavourlist:
<!--# stuff that just doesent happen
# stuff that just doesent happen
-->
 
if version == "development" and flavour == "updates":
if version == "development" and flavour == "updates":
continue
continue

Revision as of 08:03, 26 November 2008

Mock Setup Using Local Mirror

Author: Parag Nemade

Revision: 0.4

Initial Draft: Friday Jun 30, 2006

Last Revised: Saturday Aug 21, 2006

Note: This page describes my personal mock setup using local mirror setup for FC6.


Old page
This page has been marked as "old", and likely contains content that is irrelevant or incorrect. If you can, please update this page. This page will be deleted if action is not taken.
This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

Setting Local Mirrors For Fedora Development version

Prior required steps are to have at least 10GB free space on your hard drive for setting Fedora development repository.

To setup local mirror, create following files

1)copy /etc/yum.repos.d/fedora-development.repo to /etc/yum.repos.d/mirror-fedora-development.repo

2)copy /etc/yum.repos.d/fedora-extras.repo to /etc/yum.repos.d/mirror-fedora-extras.repo

Say if you have /repo directory having minimum free space 10GB then download fedora-mirror.py from http://www.fedoraproject.org/wiki/Projects/fedora-mirror to /mirror_repo.

Change /srv/mirror to /mirror_repo in fedora-mirror.py file. For development version we need only development, extras and groups repository. We don't need any updates repository as its not exist.

Modified version of fedora-mirror.py

Final Modified version of fedora-mirror.py for development version is

#!/usr/bin/python

# License: GPL
# Author:  Rudolf Kastl
# default target: /mirror_repo
# how?: use config file and special mirror .repo files

# todo:

## code:
# - ConfigParser
# - logging
# - legacy and updates-testing awareness
# - fix hierachy into the fedora mirror dir scheme
# - use separate configs for each release
# - progressbars
# - eta (complete, per repo, per file(?!))
## support files/packaging/integration:
# - creation of specially crafted .repo files
# - cron integration
# - 0 conf apache serving as split package
# - mock config files as split package
# - yum localmirror config files as split package
# - discuss access rights on /mirror_repo


################
# special cases:
# - the fedora core repos differ in logic -> core repos have updates, updates-testing, legacy -> to be investigated
# - development is another special case not having updates updates-testing and legacy
# - other repos (3rd party) never have updates and updates-testing
# - SRPMS are another special case -> same for each arch of the same release
###########################


import os

flavourlist=["core", "extras", "updates"] 
versionlist=["development"] 
archlist=["i386"] 
subflavourlist=["binary"] 

### block for parsing commandline switches
from optparse import OptionParser

parser = OptionParser()
# var can be accessed with options.filename

parser.add_option("-d", "--dest", dest="destination", help="Sync Destination", metavar="dir", default="/mirror_repo")
parser.add_option("-n", "--newonly", action="store_true", dest="newonly", help="Sync Destination", default=False)
parser.add_option("-t", "--testonly" , action="store_true", dest="testonly", help="No Sync Transaction", default=False)
parser.add_option("-q", "--quiet", action="store_false", dest="verbose", default=True, help="don't print status msgs to stdout")

(options, args) = parser.parse_args()
### end bloc


targetlocation = options.destination

def syncMirror (version, flavour, arch, subflavour, targetlocation):
print "syncing: Fedora " + version, flavour, arch, subflavour, "to", targetlocation

# handling the special cases

if subflavour == "binary":
subflavourid = ""
subflavourpath = ""
elif subflavour == "source":
subflavourid = "-"+subflavour
subflavourpath = "/SRPMS"
else:
subflavourid = "-"+subflavour
subflavourpath = "-"+subflavour

# devel is special

if version == "development":
repoid="mirror-"+flavour+"-"+version+subflavourid
else:
repoid="mirror-"+flavour+subflavourid

if subflavour == "source":
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+subflavourpath
else:
targetdir=targetlocation+"/fedora/"+version+"/"+flavour+"/"+arch+subflavourpath# handling the newonly option of reposync

if options.newonly == True:
newonly = " -n"
else:
newonly = ""

# building the command for syncing a single repo
syncdistro="mirrorreleasever="+version+" mirrorbasearch="+arch+" /usr/bin/reposync"+newonly+" -r "+repoid+" -p "+targetdir

# for testing we just print instead of sync

if options.testonly == True:
print syncdistro
else:
# create potentially missing paths

try:
if os.path.isdir(targetdir) == False:
os.makedirs(targetdir)
except:
print "Error: Cannot create targetdirectory:", targetdir

os.system(syncdistro)

## main loop

for version in versionlist:
for flavour in flavourlist:
for arch in archlist:
for subflavour in subflavourlist:
# stuff that just doesent happen

if version == "development" and flavour == "updates":
continue
if version == "development" and flavour == "legacy":
continue
if version == "development" and flavour == "updates-testing":
continue
else:
syncMirror(version, flavour, arch, subflavour, targetlocation)


Creating Directories for Local Repository Path

Now start downloading all development repository to /mirror_repo by giving

python fedora-mirror.py

Once all core and extras development repository downloaded, Now move core and extras repository to parent directory /mirror_repo.

fedora-mirror.py will download core repository at

/mirror_repo/fedora/development/core/i386/mirror-core-development

fedora-mirror.py will download extras repository at

/mirror_repo/fedora/development/extras/i386/mirror-extras-development

Create following directories

/mirror_repo/fedora/development

/mirror_repo/fedora/development/i386/

/mirror_repo/fedora/development/i386/core

/mirror_repo/fedora/development/i386/extras

/mirror_repo/fedora/development/i386/groups

Setting up core repository directory

Then create clean i386 repository by doing following

mv /mirror_repo/fedora/development/core/i386/mirror-core-development/* /mirror_repo/fedora/development/i386/core

cd /mirror_repo/fedora/development/i386/core

createrepo .

Setting up extras repository directory

mv /mirror_repo/fedora/development/extras/i386/mirror-extras-development /mirror_repo/fedora/development/i386/extras

cd /mirror_repo/fedora/development/i386/extras

createrepo .

Setting up groups repository directory

cd /mirror_repo/fedora/development/i386/groups

create minimal.xml and add its contents as

<?xml version="1.0"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">

<comps>
<group>
<id>build</id>
<uservisible>true</uservisible>
<name>Minimal Install</name>
<packagelist>
<packagereq type="mandatory">bash</packagereq>
<packagereq type="mandatory">bzip2</packagereq>
<packagereq type="mandatory">coreutils</packagereq>
<packagereq type="mandatory">cpio</packagereq>
<packagereq type="mandatory">diffutils</packagereq>
<packagereq type="mandatory">fedora-release</packagereq>
<packagereq type="mandatory">gcc</packagereq>
<packagereq type="mandatory">gcc-c++</packagereq>
<packagereq type="mandatory">gzip</packagereq>
<packagereq type="mandatory">make</packagereq>
<packagereq type="mandatory">patch</packagereq>
<packagereq type="mandatory">perl</packagereq>
<packagereq type="mandatory">rpm-build</packagereq>
<packagereq type="mandatory">redhat-rpm-config</packagereq>
<packagereq type="mandatory">sed</packagereq>
<packagereq type="mandatory">tar</packagereq>
<packagereq type="mandatory">unzip</packagereq>
<packagereq type="mandatory">buildsys-macros</packagereq>
</packagelist>
</group>
</comps>

download buildsys-build and buildsys-macors rpms under /mirror_repo/fedora/development/i386/groups

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-build-0.5-1.noarch.rpm

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-build-0.5-3.fc6.noarch.rpm

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-macros-6-2.fc6.noarch.rpm

wget http://buildsys.fedoraproject.org/buildgroups/development/i386/buildsys-macros-6-2.fc6.src.rpm

Final contents of directory groups will look as

ls /mirror_repo/fedora/development/i386/groups

minimal.xml buildsys-macros-6-2.fc6.noarch.rpm buildsys-macros-6-2.fc6.src.rpm buildsys-build-0.5-1.noarch.rpm buildsys-build-0.5-3.fc6.noarch.rpm

now do createrepo -g minimal.xml . under /mirror_repo/fedora/development/i386/groups

Thus you will find all repository set under /mirror_repo/fedora/development/i386.


Mock Setup for Local Mirror

I have setup mock environment for development(FC6) version. My mock configuration for i386 is /etc/mock/fedora-devel-i386-core.cfg


import os
config_opts['root']  = 'fedora-development-i386-core'
config_opts['target_arch']  = 'i386'

config_opts['yum.conf']  = """
[main] 
cachedir=/var/cache/yum
debuglevel=1
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1

[core] 
name=core
baseurl=file:///mirror_repo/fedora/development/i386/core

[updates-released] 
name=updates
baseurl=file:///mirror_repo/fedora/development/i386/core

[groups] 
name=groups
baseurl=file:///mirror_repo/fedora/development/i386/groups

[extras] 
name=extras
baseurl=file:///mirror_repo/fedora/development/i386/extras

[local] 
name=local
baseurl=http://extras64.linux.duke.edu/plague-results/fedora-5-extras
enabled=0

"""


Building packages in Mock Environment

To build packages in mock you need to go through http://fedoraproject.org/wiki/PackageMaintainers/MockTricks

Troubleshooting for fedora-mirror.py

If you face any problems using fedora-mirror.py script then you can use /usr/bin/reposync command to download binary RPMS for extras repository.

Core repository can be setup by copying FC6 installation DVD/CDROMs to /mirror_repo/fedora/development/i386/core. Detailed Procedure to do this can be found at http://www.city-fan.org/tips/YumRepoFromImages

reposync command without any options will download extras repository. Just move all those RPMS to /mirror_repo/fedora/development/i386/extras and use createrepo . command

Thus still you can download and create local repository without fedora-mirror.py script.