From Fedora Project Wiki
m (与英文版本同步,更新翻译)
 
(28 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{autolang}}{{header|infra}}
 
{{autolang}}{{header|infra}}
 +
Fedora 项目'''已在全球拥有超过 200 个镜像站点 [https://admin.fedoraproject.org/mirrormanager/mirrors 点击查看]''' ,这十分有助于 Fedora 的传播。我们十分感谢他们!
  
Fedora 项目'''在全球拥有超过 200 个志愿镜像站点 [http://mirrors.fedoraproject.org/publiclist 点击查看]''' ,这有助于分发 Fedora 。我们十分感谢他们!
+
{{:Legal:Export}}
  
 
== 联系我们 ==
 
== 联系我们 ==
* 邮件列表: [http://www.redhat.com/mailman/listinfo/mirror-list mirror-list] (仅发布镜像变更声明) 和 [http://www.redhat.com/mailman/listinfo/mirror-list-d mirror-list-d] (讨论专用)
+
* 邮件列表: [https://lists.fedoraproject.org/admin/lists/mirror-admin.lists.fedoraproject.org/ mirror-admins]
* IRC 频道: <code>{{fpchat|#fedora-admin}}</code>
+
* IRC 频道: Freenode 上的 <code>{{fpchat|#fedora-admin}}</code>
 
* 管理员: <code>mirror-admin@fedoraproject.org</code>
 
* 管理员: <code>mirror-admin@fedoraproject.org</code>
  
== 镜像大约有多大? ==
+
== 在你决定搭建镜像站点之前 ==
 +
 
 +
=== 镜像大约有多大? ===
  
 
* 详见:http://dl.fedoraproject.org/pub/DIRECTORY_SIZES.txt
 
* 详见:http://dl.fedoraproject.org/pub/DIRECTORY_SIZES.txt
 
请仔细阅读。
 
请仔细阅读。
  
== 出口原则 ==
+
=== 如何搭建一个公共镜像? ===
  
By downloading Fedora software, you acknowledge that you understand all of the following: Fedora software and technical information may be subject to the U.S. Export Administration Regulations (the “EAR”) and other U.S. and foreign laws and may not be exported, re-exported or transferred (a) to any country listed in Country Group E:1 in Supplement No. 1 to part 740 of the EAR (currently, Cuba, Iran, North Korea, Sudan & Syria); (b) to any prohibited destination or to any end user who has been prohibited from participating in U.S. export transactions by any federal agency of the U.S. government; or (c) for use in connection with the design, development or production of nuclear, chemical or biological weapons, or rocket systems, space launch vehicles, or sounding rockets, or unmanned air vehicle systems. You may not download Fedora software or technical information if you are located in one of these countries or otherwise subject to these restrictions. You may not provide Fedora software or technical information to individuals or entities located in one of these countries or otherwise subject to these restrictions. You are also responsible for compliance with foreign law requirements applicable to the import, export and use of Fedora software and technical information.
+
搭建公共镜像非常简单,而且将越来越简单。确保您的网站拥有足够的带宽和磁盘空间来承担负载,这就是我们的全部要求。
  
== 如何搭建一个公共镜像? ==
+
==== 磁盘空间 ====
  
搭建公共镜像非常简单,而且将越来越容易。确保您的网站拥有足够的带宽和磁盘空间来承担负载,这就是我们的全部要求。每个 Fedora 发行版都会消耗200GB以上的磁盘空间,并且有可能下载者会耗尽您的全部带宽。镜像站点应至少有100Mbps的带宽连接到互联网,现有许多站点是基于千兆或更高的带宽工作的。以 Fedora 8 发行版为例,其在主服务器上消耗的总空间(也是镜像站点要消耗的空间)达到了1.1TB并且仍在增长。1-2TB的空间适合于建立长期镜像。这是基于硬链接的空间概念,如果您不能使用硬链接(例如,使用AFS),您将需要更多的磁盘空间。实际磁盘空间消耗在该文件中已给出:
+
每个 Fedora 发行版都会消耗250GB以上的磁盘空间。 从Fedora 26版本开始,主服务器上消耗的总空间(即镜像可以消耗的空间)为1.5TB并且在不断增长。 较旧的版本会定期存档,因此3-4TB的磁盘空间适合建立长期镜像。 这是基于硬链接的空间概念,如果您不能使用硬链接(例如,使用AFS),您将需要更多的磁盘空间。 实际磁盘空间消耗记录在[https://dl.fedoraproject.org/pub/DIRECTORY_SIZES.txt DIRECTORY_SIZES.txt]中。
  
http://dl.fedoraproject.org/pub/DIRECTORY_SIZES.txt
+
==== 带宽 ====
  
== 如何搭建一个私有镜像? ==
+
你需要了解下载者可能会耗尽你的全部带宽。镜像站点应至少有 100Mbit/sec* 的带宽连接到互联网,现有许多站点是基于千兆、万兆或更高的带宽。100Mbit/sec 是针对已经有充足镜像站覆盖的国家而言的。我们希望在具有较少镜像的国家的新镜像能有更大的带宽。我们总是欢迎拥有Internet2、National Lambda Rail、GEANET2、 RedIRIS 或者类似的高速科研网络连接、教育网络的镜像站。
 +
 
 +
=== 如何搭建一个私有镜像? ===
  
 
私有镜像是那些只能被内部人员访问的某些组织或者机构专有的镜像,例如公司,学校等等。这些镜像是为了在机构内部加快 Fedora 的分发而被开设的。当然,这些镜像所处的网络对当地的带宽消耗远少于连接互联网的。
 
私有镜像是那些只能被内部人员访问的某些组织或者机构专有的镜像,例如公司,学校等等。这些镜像是为了在机构内部加快 Fedora 的分发而被开设的。当然,这些镜像所处的网络对当地的带宽消耗远少于连接互联网的。
Line 29: Line 34:
 
私有镜像运行与公共镜像基本无异,除了以下几点:
 
私有镜像运行与公共镜像基本无异,除了以下几点:
 
* 镜像管理器的 publiclist 不会有该镜像的显示。
 
* 镜像管理器的 publiclist 不会有该镜像的显示。
* 无法从本社区镜像获取同步文件,只能寻求其他公共镜像。
+
* 无法从官方镜像目录同步,只能寻求其他公共镜像。
* Private mirrors should include IP netblocks in their MirrorManager configuration.  This allows your network-local users to be automatically redirected to your mirror.  You may list IP netblocks (e.g. 18.0.0.0/8), or if your network is NAT'd, the hostname of your NAT gateway.
+
* 私有镜像必须在镜像管理器设置中包含你的 IP 段。这将会允许你的内网用户访问镜像站点时自动重定向至你的镜像。你可以将 IP 段定为类似括号中的 (18.0.0.0/8) 格式;如果你的网络是 NAT 结构,则请输入你的 NAT 网关主机名。
* 私有镜像不会被镜像管理器的蜘蛛爬行。
+
* 私有镜像不会被镜像管理器的蜘蛛爬行,也就是不会被索引。
* Private mirrors must run report_mirror to inform the MirrorManager database of their content.  If you don't run report_mirror, your clients will not be automatically redirected.
+
* 私有镜像必须运行 report_mirror 工具以告知 MirrorManager 数据库。如果你不运行,你的客户端原本想使用你的镜像,但是将不会自动跳转到你的镜像地址。
 
You may also find it more beneficial to run an [https://fedorahosted.org/intelligentmirror/ IntelligentMirror] instead of a full rsync mirror.  In this way, only the updates your local users actually need will be cached on your local mirror, saving you the bandwidth from downloading updates you don't actually need.
 
  
 
== 镜像管理器:Fedora 项目的镜像管理系统 ==
 
== 镜像管理器:Fedora 项目的镜像管理系统 ==
[http://mirrormanager.org 镜像管理器]跟踪所有的镜像而不需要很多的人工的文件编辑。镜像站点的管理员一点要确保 <code>report_mirror</code> 脚本和 <code>mirrormanager-client</code> 在每次 <code>rsync</code> 之后被运行以更新镜像管理数据库。
+
 
 +
[https://github.com/fedora-infra/mirrormanager2/ MirrorManager2]软件可以跟踪所有镜像,无需大量手动文本文件编辑。 Fedora的这个应用程序的实例位于[https://admin.fedoraproject.org/mirrormanager https://admin.fedoraproject.org/mirrormanager]
 +
 
 +
=== 镜像状态 ===
 +
 
 +
MirrorManager keeps track of all the mirrors with regular scans of the content of all mirrors. The content of the mirrors is scanned by the crawler which either scans/crawls the mirrors using HTTP or RSYNC. We strongly recommend providing a RSYNC URL (at least for our crawler) as crawling via RSYNC is much faster as it only requires a single network connection for each mirrored category. If only HTTP access for Fedora's mirror crawler is possible it is important to enable [[Infrastructure/Mirroring#Keepalives|HTTP Keepalives]] to reduce the number of required network connections.
 +
 
 +
In addition, admins of the mirrors should ensure that the <code>report_mirror</code> script available with the <code>mirrormanager-client</code> is run after each <code>rsync</code> run to update the Mirrormanager database.
  
 
=== 注册 ===
 
=== 注册 ===
  
 
==== Fedora 账户系统 ====
 
==== Fedora 账户系统 ====
* 您必须有一个 [https://admin.fedoraproject.org/accounts/ Fedora 账户]不过您不必为了镜像内容而去签署Contributors License Agreement,但是如果您想要在其他领域做贡献,您必须签署。
+
* 您必须有一个 [https://admin.fedoraproject.org/accounts/ Fedora 账户](在[https://fedoraproject.org/wiki/AccountSystem 这里]获取更多信息)。 不过您不必为了镜像内容而去签署Contributors License Agreement,但是如果您想要在其他领域做贡献,您必须签署。
* 如果是公共镜像,您必须要发送一封邮件至 <code>mirror-admin@fedoraproject.org</code> 来介绍一下您的镜像。同时这也是您想成为公共镜像的声明。注意这封邮件必须涵盖了您的镜像的 IP 地址,所处国家或地区以及该镜像的流量和带宽。
+
* 如果是公共镜像,您必须要发送一封邮件至 <code>mirror-admin@fedoraproject.org</code> 来介绍一下您的镜像。同时这也是您想成为公共镜像的声明。注意这封邮件必须涵盖了您的镜像的 IP 地址,所处国家或地区以及该镜像的流量和带宽。如果是私有镜像则不必了,如果您想介绍一下自己,请简短。
如果是私有镜像则不必了,如果您想介绍一下自己,请简短。
+
* 您必须订阅 [https://lists.fedoraproject.org/admin/lists/mirror-admin.lists.fedoraproject.org/ mirror-admin 邮件列表] ,新发行版的同步通知会在此发布。
* 您必须订阅 [http://www.redhat.com/mailman/listinfo/mirror-list mirror 邮件列表] ,可选择订阅 [http://www.redhat.com/mailman/listinfo/mirror-list-d 镜像讨论] 列表,新发行版的同步通知会在此发布。
 
  
 
==== 在 MirrorManager 中注册 ====
 
==== 在 MirrorManager 中注册 ====
* 使用您的 FAS 账户登录 [https://admin.fedoraproject.org/mirrormanager mirrormanager] ;
 
* 创建一个新站点;
 
* create a new Host, and sign up that host for the Categories of content you'll carry, any other site administrators you want, your site's IP addresses used for our Access Control List, and the other details listed there if applicable to you
 
* Please run <code>report_mirror</code> after each rsync run.
 
* You may list your site's IP address ranges (Netblocks).  Clients coming from an IP address within your netblock will be automatically redirected to your mirror for any content you carry.
 
* You may list your site's BGP Autonomous System Number (ASN).  Clients on your ASN will be automatically redirected to your mirror for any content you carry.  One way to lookup up your ASN is to query it from the routeviews.org DNS servers.  It is like a PTR record lookup, but at a specific server.  For example, to look up 143.166.1.1, type:
 
$ dig txt 1.1.166.143.asn.routeviews.org @archive.routeviews.org
 
;; ANSWER SECTION:
 
1.1.166.143.asn.routeviews.org. 86400 IN TXT "3614" "143.166.0.0" "16"
 
Here, the answer is in the TXT record, the first value, 3614.
 
  
=== 搭建 ===
+
* 使用你的 FAS 账户登录 [https://admin.fedoraproject.org/mirrormanager mirrormanager] ;
最简单的做法,也是最通用的,就是用 <code>rsync</code> 工具来同步。注意该工具的 <code>-H</code> (硬链接)<code>--delay-updates</code>, <code>--numeric-ids</code> 和 <code>--delete-after</code> 选项是必须要启用的。示例:
+
* Create a new Site (a site is a group of Hosts belonging to the same organization).
 +
* Create a new Host, and sign up that host for the Categories of content you'll carry (under the ''formally optional'' section '''Categories Carrried'''), any other site administrators you want, your site's IP addresses used for our Access Control List, list of countries that are allowed to access your mirror, and the other details listed there if applicable to you.
 +
** You may list your site's IP address ranges (Netblocks).  Clients coming from an IP address within your netblock will be automatically redirected to your mirror for any content you carry.
 +
** You may list your site's BGP Autonomous System Number (ASN).  Clients on your ASN will be automatically redirected to your mirror for any content you carry.  One way to lookup up your ASN is to query it from the routeviews.org DNS servers.  It is like a PTR record lookup, but at a specific server.  For example, to look up 143.166.1.1, type:<br />
 +
**:  <pre>$ dig txt 1.1.166.143.asn.routeviews.org @archive.routeviews.org
 +
**:: ;; ANSWER SECTION:
 +
**:: 1.1.166.143.asn.routeviews.org. 86400 IN TXT "3614" "143.166.0.0" "16"</pre>
 +
**:: Here, the answer is in the TXT record, the first value, 3614.
 +
** For a mirror to get listed in the MirrorManager for the category of content you have selected to carry, you ''must'' add at least one URL via which the category of content on your mirror can be accessed (under '''Categories Carried'''). The URLs you add can be of type HTTP, HTTPS, and rsync. Fedora's MirrorManager instance does not support FTP URLs and therefore it is not possible to add a FTP URL anymore.
 +
* Please run <code>report_mirror</code> after each rsync run. ('''Note:''' this is not necessary if you use quick-fedora-mirror, see below for details.)
 +
 
 +
=== Mirroring ===
 +
 
 +
* Please use the [https://pagure.io/quick-fedora-mirror quick-fedora-mirror] zsh script if at all possible. This script uses rsync and some informational files on the mirrors to allow you to only sync those files you need and saves lots of time and file seeking. '''Note:''' make sure that the mirror you intend to use provides the fedora-buffet rsync module, as otherwise the script will not work.
 +
 
 +
* Please pull from one of the [[Infrastructure/Mirroring/Tiering#Tier_1_Mirrors|Tier 1 mirrors]]. Instead of using one of the Tier 1 servers, you may wish to pull from another fast mirror that's closer to you. Contact the respective mirror admins to be added to their ACL.
  
rsync -vaH --exclude-from=${EXCLUDES} --numeric-ids --delete --delete-after --delay-updates \
+
* You may exclude any content you desire, such as architectures or releases, using an EXCLUDES file or <code>--exclude</code> parameter.
  rsync://dl.fedoraproject.org/fedora-enchilada ${LOCAL_DIR}
 
  
* 要同步的内容由您决定,不必同步所有内容,比如某些特殊架构的专用版您可以选择性同步。
+
==== Tools to avoid ====
  
* 请从1级镜像站点同步。请见 [[Infrastructure/Mirroring/Tiering]]。除了以上方法,您也可以选择从一个比较快的镜像同步。请联系相关站点管理员以便将您的站点添加到他们的访问控制列表中。
+
Please don't use tools like 'lftp mirror' to mirror content from the master mirrors. It's places a heavy load on our master mirrors and is slow for both you and us. Please use [https://pagure.io/quick-fedora-mirror quick-fedora-mirror] instead.
  
* 您应该在08:00 UTC(此时是 rawhide 版本发布)后同步, 14:00 UTC (when bitflips occur), 且一天同步次数为3-5次。
+
==== Mirror Frequency====
  
* 你应该设置您的站点同步时间为一个随机值以便拉平上游镜像的负载。最好写一个cron job:
+
* If you are using quick-fedora-mirror you can sync as often as every 10minutes in a loop. If there is no new content the script should return almost immediately. Make sure to use locking so you are only running one instance at a time.
  45 */6 * * * perl -le 'sleep rand 1800' && bash -l ~/mirror-fedora > /dev/null
 
  
* 如果您使用 rsync 3.0以上版本,您可以使用 <code>--delete-delay</code> 选项替换 <code>--delete-after</code>,这种情况下可以提高性能。具体资料请见:[http://lists.debian.org/debian-mirrors/2009/04/msg00017.html 这份报告]。
+
* If you are not using quick-fedora-mirror you should sync a few times a day and consider switching.  
  
 
==== 运行 report_mirror ====
 
==== 运行 report_mirror ====
镜像管理器包含了一个工具 <code>report_mirror</code> ,which can upload to the mirror database that you completed a run and what content you've got.  This makes generating the yum mirrorlists and all other pages much much simpler.请在每次同步完成后运行 <code>report_mirror</code>。请通过以下方法安装获取:
 
  
yum install mirrormanager-client
+
'''Note:''' quick-fedora-mirror can also check in for you and has no need of the <code>report_mirror</code> package/conf. To enable it, set the correct values for <code>CHECKIN_SITE</code> and <code>CHECKIN_PASSWORD</code> variables. In most cases, setting <code>CHECKIN_HOST</code> is not necessary. The following applies only if you are not using quick-fedora-mirror, and, as said above, you should consider switching.
 +
 
 +
镜像管理器包含了一个工具 <code>report_mirror</code> ,该工具可以在你完成同步任务后通报官方你的镜像本次同步的详细信息。这对我们的日常管理和用户的安装很有帮助。请在每次同步完成后运行 <code>report_mirror</code>。请通过以下方法安装获取:
  
或者访问 [http://git.fedorahosted.org/git/?p=mirrormanager;a=tree;f=client;hb=HEAD report_mirror files]获取代码。您也可以使用 git 来获取:
+
dnf install mirrormanager2-client
  
git clone git://git.fedorahosted.org/git/mirrormanager
+
or yum (in Fedora 21 or earlier, RHEL and CentOS):
  或者
 
git clone http://git.fedorahosted.org/git/mirrormanager/
 
  
 +
yum install mirrormanager2-client
  
您需要 report_mirror 和 report_mirror.conf,且必须配置 report_mirror.conf 以便识别您的磁盘上所有镜像文件位置。
+
或者直接从 [https://github.com/fedora-infra/mirrormanager2/ git 仓库]获取代码:
 +
 
 +
git clone https://github.com/fedora-infra/mirrormanager2.git
 +
 
 +
你需要 report_mirror 和 report_mirror.conf,且必须配置 report_mirror.conf 以便识别你的磁盘上所有镜像文件位置。
  
 
==== 可用内容 ====
 
==== 可用内容 ====
  
请要注意同步的内容:
+
这些是可以用 rysnc 进行的内容模块,以及在目录树中的节点:
  
 
===== 建议的同步的部分 =====
 
===== 建议的同步的部分 =====
 
{| border="1"
 
{| border="1"
 
|-
 
|-
|'''rsync module'''||'''Description'''||'''path on master server'''||Comments
+
|'''rsync 模块名称'''||'''描述'''||'''主镜像服务器上的路径'''||备注
 
|-
 
|-
|fedora-enchilada0||Fedora - the whole enchilada||/pub/fedora||Please use this if you can, it provides the all current Fedora content, including pre-bitflip content.  This is open to specific mirrors by request.  Mirrors participating in our tiering should use this.
+
|fedora-buffet||Fedora - The whole buffet. All you can eat.||/pub||Please use this if you can, it provides the all current Fedora content, including pre-bitflip content.  This is open to specific mirrors by request.  Mirrors participating in our tiering should use this.  Mirrors syncing both fedora-enchilada and fedora-archive should use this, as we can now hardlink across both of those trees under fedora-buffet0.
 
|-
 
|-
|fedora-epel0||Extra Packages for Enterprise Linux||/pub/epel||Please use this to mirror EPEL
+
|fedora-enchilada||Fedora - the whole enchilada||/pub/fedora||Please use this if you can, it provides the all current Fedora content, including pre-bitflip content.  This is open to specific mirrors by request.  Mirrors participating in our tiering should use this.
 +
|-
 +
|fedora-epel||Extra Packages for Enterprise Linux||/pub/epel||Please use this to mirror EPEL
 
|-
 
|-
 
|fedora-archive||Historical Fedora releases||/pub/archive||Fedora Core 1-6 and Extras 3-6, and obsolete releases 7 and higher
 
|fedora-archive||Historical Fedora releases||/pub/archive||Fedora Core 1-6 and Extras 3-6, and obsolete releases 7 and higher
 
|-
 
|-
|fedora-secondary||Fedora Secondary Arches||/pub/fedora-secondary
+
|fedora-secondary||Fedora Secondary Arches||/pub/fedora-secondary||
 
|-
 
|-
|fedora-alt||Fedora Other||/pub/alt
+
|fedora-alt||Fedora Other||/pub/alt||
 
|}
 
|}
  
  
以下内容请不要同步,它们是特定时期的产物。
+
以下模块请不要同步,它们是特定时期的产物。
  
 
{| border="1"
 
{| border="1"
 
|-
 
|-
|'''同步的目录'''||'''描述'''||'''path on master server'''||备注
+
|'''模块名称'''||'''描述'''||'''主镜像服务器上的路径'''||备注
 
|-
 
|-
|fedora-enchilada||Fedora - the whole enchilada||/pub/fedora||Please use this if you can, it provides the all Fedora content except pre-bitflip content.  This is open for the general public to use.
+
|fedora-linux-releases||Fedora Linux Releases||/pub/fedora/linux/releases||
 
|-
 
|-
|fedora-linux-releases||Fedora Linux Releases||/pub/fedora/linux/releases
+
|fedora-linux-development||Fedora Linux Development (Rawhide)||/pub/fedora/linux/development||
 
|-
 
|-
|fedora-linux-development||Fedora Linux Development (Rawhide)||/pub/fedora/linux/development
+
|fedora-linux-updates||Fedora Linux Updates||/pub/fedora/linux/updates||
|-
 
|fedora-linux-updates||Fedora Linux Updates||/pub/fedora/linux/updates
 
|-
 
|fedora-epel||Extra Packages for Enterprise Linux||/pub/epel||EPEL doesn't do the bitflip trick, so this is the same as fedora-epel0 above.
 
 
|-
 
|-
 
|}
 
|}
  
===== Red Hat Enterprise Linux Beta =====
+
==== Rsync 配置(示例) ====
RHEL beta 版本也通过 MirrorManager 分发。Fedora 项目的一级镜像可以同步这些内容。
 
 
 
{| border="1"
 
|-
 
|'''同步的目录'''||'''描述'''||'''path on master server'''||备注
 
|-
 
|rhel-beta||RHEL||/pub/redhat||on rhm3.redhat.com
 
|}
 
 
 
==== DVD,CD 和 exploded trees ====
 
When a new release is available, it can be bandwidth-efficient to download only the ISOs first (say, the DVD ISOs), then explode those into the directory structure, then run a full normal rsync run.  This lets you avoid downloading the same RPMs twice (both on ISOs and as plain RPMs).  There's a tool somewhere to help do this.
 
 
 
==== Regular hardlink runs ====
 
While the Fedora release maintainers try to keep as little redundant packaging around as possible, there are some duplicate packages in the tree.  For example, when a Fedora Test release comes out, the package set included there looks remarkably like that of the development tree from a few days before.  By copying the development tree over into the new Test directory before starting your rsync run, and using <code>rsync -H</code>, you can avoid downloading all that content a second time.
 
 
 
In addition, it's good practice to run a tool like <code>hardlink++</code> on your tree occasionally (say, weekly), to ensure as much of your tree as possible is hardlinked.
 
 
 
==== Pre-Release: Copying Development tree to new release directory ====
 
In the days leading up to a release, either test or final, the development tree will stop taking new packages, and will closely resemble what winds up in the new release.  As a mirror, you can avoid downloading content that already is in your copy of the development tree that matches what's in the release tree by copying those packages using hardlinks, such as:
 
cp -lr fedora/linux/development/13/i386/os fedora/linux/releases/13/Fedora/i386/
 
cp -lr fedora/linux/development/13/x86_64/os fedora/linux/releases/13/Fedora/x86_64/
 
cp -lr fedora/linux/development/13/source fedora/linux/releases/13/Fedora/
 
 
 
and then start the rsync process, which will clean up any changes and fix up the timestamps.
 
 
 
==== Rsync Configuration (sample) ====
 
 
Larger mirrors, like kernel.org, have slightly custom front-ends to rsync (mainly so that they can have a single rsync instance and have multiple ip based vhost configuration files)  That said what follows is a sample rsync configuration file for public syncing (this is not intended for private pre-bitflip mirroring)
 
Larger mirrors, like kernel.org, have slightly custom front-ends to rsync (mainly so that they can have a single rsync instance and have multiple ip based vhost configuration files)  That said what follows is a sample rsync configuration file for public syncing (this is not intended for private pre-bitflip mirroring)
  
Line 193: Line 181:
 
Other http servers such as lighttpd have keepalives enabled by default.
 
Other http servers such as lighttpd have keepalives enabled by default.
  
===== Caching of metadata =====
+
===== metadata 缓存 =====
 
We don't want caching proxy servers between our mirrors and our end user systems to cache our yum repository metadata.  So, add explicit metadata handling.  (Suggested by the OpenSUSE download redirector.)
 
We don't want caching proxy servers between our mirrors and our end user systems to cache our yum repository metadata.  So, add explicit metadata handling.  (Suggested by the OpenSUSE download redirector.)
  
Line 202: Line 190:
 
       ExpiresDefault "now"
 
       ExpiresDefault "now"
 
   </LocationMatch>
 
   </LocationMatch>
</pre>
 
 
===== Redirecting ISO downloads to FTP =====
 
While no longer a recommended practice, the following mod_rewrite rules will force all *.iso files to be downloaded via FTP.  In this example HEAD requests are not redirected, so the MirrorManager crawler is not disrupted.
 
 
<pre>
 
RewriteCond    %{REQUEST_METHOD} GET
 
RewriteRule    ^(.*\.iso)$ ftp://myserver/$1  [L,R=301]
 
 
</pre>
 
</pre>
  
Line 222: Line 202:
 
</pre>
 
</pre>
  
===== Limiting Download Accelerators =====
+
===== 限制某些加速下载工具 =====
  
 
Download accelerators will try to open the same file many times, and request chunks, hoping to download them in parallel.  This can overload heavily loaded mirror servers, especially on release day.  Here are some tricks to thwart such activities.
 
Download accelerators will try to open the same file many times, and request chunks, hoping to download them in parallel.  This can overload heavily loaded mirror servers, especially on release day.  Here are some tricks to thwart such activities.
Line 274: Line 254:
 
</pre>
 
</pre>
  
===== Serving content to other mirrors =====
+
===== 向其它镜像提供内容 =====
Tier 1 mirrors will necessarily need to share content to Tier 2 mirrors before the bitflip.  This is done by running another instance of the rsync daemon, on a different port (e.g. 874), with an Access Control List to prevent public downloads, running as a user in the same group as downloaded the content (e.g. group mirror).  This could be user mirror, group mirror, who has group read/execute permissions on the still-private content.
+
 
 +
[[Infrastructure/Mirroring/Tiering#Tier_1_Mirrors|Tier 1 mirrors]] will necessarily need to share content to Tier 2 mirrors before the bitflip.  This is done by running another instance of the rsync daemon, on a different port (e.g. 874), with an Access Control List to prevent public downloads, running as a user in the same group as downloaded the content (e.g. group mirror).  This could be user mirror, group mirror, who has group read/execute permissions on the still-private content.
  
 
Tier 1 mirrors have a tendency to use different authentication methods for granting access to these non-public downloads, they vary from maintaining IP based ACL's to assigning username/password combinations to mirrors wishing to sync from them.  Each method has advantages / disadvantages, the IP list is 'simpler' from a mirrormanager perspective as mirrormanager can give you the list of IP's but from an automation standpoint can be more difficult (as rsync's configuration file does not allow that ACL list to be stored in a separate file).  Username / passwords can be more versatile as sites mirroring can change IPs without notifying you, but it's easier for those credentials to leak out and get miss-used.
 
Tier 1 mirrors have a tendency to use different authentication methods for granting access to these non-public downloads, they vary from maintaining IP based ACL's to assigning username/password combinations to mirrors wishing to sync from them.  Each method has advantages / disadvantages, the IP list is 'simpler' from a mirrormanager perspective as mirrormanager can give you the list of IP's but from an automation standpoint can be more difficult (as rsync's configuration file does not allow that ACL list to be stored in a separate file).  Username / passwords can be more versatile as sites mirroring can change IPs without notifying you, but it's easier for those credentials to leak out and get miss-used.
  
== 全球镜像分布地图 ==
+
=== Other MirrorManager features ===
http://fedoraproject.org/maps/mirrors.png
+
 
 +
==== Statistics ====
 +
 
 +
MirrorManager tracks [https://admin.fedoraproject.org/mirrormanager/statistics the number of accesses to the mirror list] and breaks it down by country, architecture, and repository.
 +
 
 +
==== Mirror Maps ====
 +
 
 +
[https://admin.fedoraproject.org/mirrormanager/maps The map of all public mirrors] is updated daily; if it doesn't display the correct data, please try again later.
  
每日更新。如果没有显示请稍后再试。
+
===== Recognition =====
  
=== 感谢 ===
+
This product includes [https://dev.maxmind.com/geoip/legacy/geolite/ GeoLite data] created by [https://www.maxmind.com/ MaxMind].
本页包含由 MaxMind 旗下产品 GeoLite 的数据。详情请前往:http://www.maxmind.com/ 查看。
 
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Latest revision as of 15:35, 4 September 2019

Fedora 项目已在全球拥有超过 200 个镜像站点 点击查看 ,这十分有助于 Fedora 的传播。我们十分感谢他们!

Fedora Export Compliance/Customs Information

By downloading Fedora software, you acknowledge that you understand all of the following: Fedora software and technical information may be subject to the U.S. Export Administration Regulations (the “EAR”) and other U.S. and foreign laws and may not be exported, re-exported or transferred (a) to a prohibited destination country under the EAR or U.S. sanctions regulations (currently Cuba, Iran, North Korea, Sudan, Syria, and the Crimea Region of Ukraine, subject to change as posted by the United States government); (b) to any prohibited destination or to any end user who has been prohibited from participating in U.S. export transactions by any federal agency of the U.S. government; or (c) for use in connection with the design, development or production of nuclear, chemical or biological weapons, or rocket systems, space launch vehicles, or sounding rockets, or unmanned air vehicle systems. You may not download Fedora software or technical information if you are located in one of these countries or otherwise subject to these restrictions. You may not provide Fedora software or technical information to individuals or entities located in one of these countries or otherwise subject to these restrictions. You are also responsible for compliance with foreign law requirements applicable to the import, export and use of Fedora software and technical information.

Fedora software in source code and binary code form are publicly available and are not subject to the EAR in accordance with §742.15(b).


联系我们

在你决定搭建镜像站点之前

镜像大约有多大?

请仔细阅读。

如何搭建一个公共镜像?

搭建公共镜像非常简单,而且将越来越简单。确保您的网站拥有足够的带宽和磁盘空间来承担负载,这就是我们的全部要求。

磁盘空间

每个 Fedora 发行版都会消耗250GB以上的磁盘空间。 从Fedora 26版本开始,主服务器上消耗的总空间(即镜像可以消耗的空间)为1.5TB并且在不断增长。 较旧的版本会定期存档,因此3-4TB的磁盘空间适合建立长期镜像。 这是基于硬链接的空间概念,如果您不能使用硬链接(例如,使用AFS),您将需要更多的磁盘空间。 实际磁盘空间消耗记录在DIRECTORY_SIZES.txt中。

带宽

你需要了解下载者可能会耗尽你的全部带宽。镜像站点应至少有 100Mbit/sec* 的带宽连接到互联网,现有许多站点是基于千兆、万兆或更高的带宽。100Mbit/sec 是针对已经有充足镜像站覆盖的国家而言的。我们希望在具有较少镜像的国家的新镜像能有更大的带宽。我们总是欢迎拥有Internet2、National Lambda Rail、GEANET2、 RedIRIS 或者类似的高速科研网络连接、教育网络的镜像站。

如何搭建一个私有镜像?

私有镜像是那些只能被内部人员访问的某些组织或者机构专有的镜像,例如公司,学校等等。这些镜像是为了在机构内部加快 Fedora 的分发而被开设的。当然,这些镜像所处的网络对当地的带宽消耗远少于连接互联网的。

私有镜像运行与公共镜像基本无异,除了以下几点:

  • 镜像管理器的 publiclist 不会有该镜像的显示。
  • 无法从官方镜像目录同步,只能寻求其他公共镜像。
  • 私有镜像必须在镜像管理器设置中包含你的 IP 段。这将会允许你的内网用户访问镜像站点时自动重定向至你的镜像。你可以将 IP 段定为类似括号中的 (18.0.0.0/8) 格式;如果你的网络是 NAT 结构,则请输入你的 NAT 网关主机名。
  • 私有镜像不会被镜像管理器的蜘蛛爬行,也就是不会被索引。
  • 私有镜像必须运行 report_mirror 工具以告知 MirrorManager 数据库。如果你不运行,你的客户端原本想使用你的镜像,但是将不会自动跳转到你的镜像地址。

镜像管理器:Fedora 项目的镜像管理系统

MirrorManager2软件可以跟踪所有镜像,无需大量手动文本文件编辑。 Fedora的这个应用程序的实例位于https://admin.fedoraproject.org/mirrormanager

镜像状态

MirrorManager keeps track of all the mirrors with regular scans of the content of all mirrors. The content of the mirrors is scanned by the crawler which either scans/crawls the mirrors using HTTP or RSYNC. We strongly recommend providing a RSYNC URL (at least for our crawler) as crawling via RSYNC is much faster as it only requires a single network connection for each mirrored category. If only HTTP access for Fedora's mirror crawler is possible it is important to enable HTTP Keepalives to reduce the number of required network connections.

In addition, admins of the mirrors should ensure that the report_mirror script available with the mirrormanager-client is run after each rsync run to update the Mirrormanager database.

注册

Fedora 账户系统

  • 您必须有一个 Fedora 账户(在这里获取更多信息)。 不过您不必为了镜像内容而去签署Contributors License Agreement,但是如果您想要在其他领域做贡献,您必须签署。
  • 如果是公共镜像,您必须要发送一封邮件至 mirror-admin@fedoraproject.org 来介绍一下您的镜像。同时这也是您想成为公共镜像的声明。注意这封邮件必须涵盖了您的镜像的 IP 地址,所处国家或地区以及该镜像的流量和带宽。如果是私有镜像则不必了,如果您想介绍一下自己,请简短。
  • 您必须订阅 mirror-admin 邮件列表 ,新发行版的同步通知会在此发布。

在 MirrorManager 中注册

  • 使用你的 FAS 账户登录 mirrormanager
  • Create a new Site (a site is a group of Hosts belonging to the same organization).
  • Create a new Host, and sign up that host for the Categories of content you'll carry (under the formally optional section Categories Carrried), any other site administrators you want, your site's IP addresses used for our Access Control List, list of countries that are allowed to access your mirror, and the other details listed there if applicable to you.
    • You may list your site's IP address ranges (Netblocks). Clients coming from an IP address within your netblock will be automatically redirected to your mirror for any content you carry.
    • You may list your site's BGP Autonomous System Number (ASN). Clients on your ASN will be automatically redirected to your mirror for any content you carry. One way to lookup up your ASN is to query it from the routeviews.org DNS servers. It is like a PTR record lookup, but at a specific server. For example, to look up 143.166.1.1, type:
      $ dig txt 1.1.166.143.asn.routeviews.org @archive.routeviews.org
      
      ;; ANSWER SECTION:
      1.1.166.143.asn.routeviews.org. 86400 IN TXT "3614" "143.166.0.0" "16"
      Here, the answer is in the TXT record, the first value, 3614.
    • For a mirror to get listed in the MirrorManager for the category of content you have selected to carry, you must add at least one URL via which the category of content on your mirror can be accessed (under Categories Carried). The URLs you add can be of type HTTP, HTTPS, and rsync. Fedora's MirrorManager instance does not support FTP URLs and therefore it is not possible to add a FTP URL anymore.
  • Please run report_mirror after each rsync run. (Note: this is not necessary if you use quick-fedora-mirror, see below for details.)

Mirroring

  • Please use the quick-fedora-mirror zsh script if at all possible. This script uses rsync and some informational files on the mirrors to allow you to only sync those files you need and saves lots of time and file seeking. Note: make sure that the mirror you intend to use provides the fedora-buffet rsync module, as otherwise the script will not work.
  • Please pull from one of the Tier 1 mirrors. Instead of using one of the Tier 1 servers, you may wish to pull from another fast mirror that's closer to you. Contact the respective mirror admins to be added to their ACL.
  • You may exclude any content you desire, such as architectures or releases, using an EXCLUDES file or --exclude parameter.

Tools to avoid

Please don't use tools like 'lftp mirror' to mirror content from the master mirrors. It's places a heavy load on our master mirrors and is slow for both you and us. Please use quick-fedora-mirror instead.

Mirror Frequency

  • If you are using quick-fedora-mirror you can sync as often as every 10minutes in a loop. If there is no new content the script should return almost immediately. Make sure to use locking so you are only running one instance at a time.
  • If you are not using quick-fedora-mirror you should sync a few times a day and consider switching.

运行 report_mirror

Note: quick-fedora-mirror can also check in for you and has no need of the report_mirror package/conf. To enable it, set the correct values for CHECKIN_SITE and CHECKIN_PASSWORD variables. In most cases, setting CHECKIN_HOST is not necessary. The following applies only if you are not using quick-fedora-mirror, and, as said above, you should consider switching.

镜像管理器包含了一个工具 report_mirror ,该工具可以在你完成同步任务后通报官方你的镜像本次同步的详细信息。这对我们的日常管理和用户的安装很有帮助。请在每次同步完成后运行 report_mirror。请通过以下方法安装获取:

dnf install mirrormanager2-client

or yum (in Fedora 21 or earlier, RHEL and CentOS):

yum install mirrormanager2-client

或者直接从 git 仓库获取代码:

git clone https://github.com/fedora-infra/mirrormanager2.git

你需要 report_mirror 和 report_mirror.conf,且必须配置 report_mirror.conf 以便识别你的磁盘上所有镜像文件位置。

可用内容

这些是可以用 rysnc 进行的内容模块,以及在目录树中的节点:

建议的同步的部分
rsync 模块名称 描述 主镜像服务器上的路径 备注
fedora-buffet Fedora - The whole buffet. All you can eat. /pub Please use this if you can, it provides the all current Fedora content, including pre-bitflip content. This is open to specific mirrors by request. Mirrors participating in our tiering should use this. Mirrors syncing both fedora-enchilada and fedora-archive should use this, as we can now hardlink across both of those trees under fedora-buffet0.
fedora-enchilada Fedora - the whole enchilada /pub/fedora Please use this if you can, it provides the all current Fedora content, including pre-bitflip content. This is open to specific mirrors by request. Mirrors participating in our tiering should use this.
fedora-epel Extra Packages for Enterprise Linux /pub/epel Please use this to mirror EPEL
fedora-archive Historical Fedora releases /pub/archive Fedora Core 1-6 and Extras 3-6, and obsolete releases 7 and higher
fedora-secondary Fedora Secondary Arches /pub/fedora-secondary
fedora-alt Fedora Other /pub/alt


以下模块请不要同步,它们是特定时期的产物。

模块名称 描述 主镜像服务器上的路径 备注
fedora-linux-releases Fedora Linux Releases /pub/fedora/linux/releases
fedora-linux-development Fedora Linux Development (Rawhide) /pub/fedora/linux/development
fedora-linux-updates Fedora Linux Updates /pub/fedora/linux/updates

Rsync 配置(示例)

Larger mirrors, like kernel.org, have slightly custom front-ends to rsync (mainly so that they can have a single rsync instance and have multiple ip based vhost configuration files) That said what follows is a sample rsync configuration file for public syncing (this is not intended for private pre-bitflip mirroring)

[fedora]
        comment         = Fedora - RedHat community project
        path            = <path to your fedora directory>
        exclude         = lost+found/
        read only       = true
        max connections = 100
        lock file       = /var/run/rsyncd-mirrors.lock
        uid             = <user id (numeric, or textual) of an anonymous style user who should have read access>
        gid             = <group id (numeric, or textual) of an anonymous style user who should have read access>
        transfer logging = yes
        timeout         = 900
        ignore nonreadable = yes
        dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2
        refuse options = checksum

Things to explicitly note:

  • The path above should be a full path to your fedora directory
  • You should *really* want to leave this read-only
  • Make sure your uid/gid are set to public users, not to the user that you run as your sync agent. If you set this to the user who does your syncs you will be inadvertently giving the public full pre-bitflip access.
  • Make sure you have the 'refuse options' set to checksum, your server will be *MUCH* happier with this set, as it will prevent public users from performing a checksum run against you. This can be incredibly I/O abusive, so should not be available to the general public.

HTTPd 配置

Keepalives

HTTP Keepalives should be enabled on your mirror server to speed up client downloads. By default, Fedora's Apache httpd package has keepalives disabled. They should be enabled, with a timeout of at least 2 seconds (the default of 15 seconds might be too high for a heavily loaded mirror server, but 2 seconds is sufficient and appropriate for yum).

KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 100

Other http servers such as lighttpd have keepalives enabled by default.

metadata 缓存

We don't want caching proxy servers between our mirrors and our end user systems to cache our yum repository metadata. So, add explicit metadata handling. (Suggested by the OpenSUSE download redirector.)

   <LocationMatch "\.(xml|xml\.gz|xml\.asc|sqlite)$">
       Header set Cache-Control "must-revalidate"
       ExpiresActive On
       ExpiresDefault "now"
   </LocationMatch>
Content Types

ISO and RPM files should be served using MIME Content-Type: application/octet-stream. In Apache, this can be done inside a VirtualHost or similar section:

<VirtualHost *:80>
AddType application/octet-stream .iso
AddType application/octet-stream .rpm
</VirtualHost>
限制某些加速下载工具

Download accelerators will try to open the same file many times, and request chunks, hoping to download them in parallel. This can overload heavily loaded mirror servers, especially on release day. Here are some tricks to thwart such activities.

To limit connections to ISO dirs by some amount per IP:

<IfModule mod_limitipconn.c>
MaxConnPerIP 6
</IfModule>

To block ranged requests as this is what download accelerators do indeed:

RewriteEngine on
RewriteCond %{HTTP:Range} [0-9] $
RewriteRule \.iso$ / [F,L] 

Similar things can be done with iptables and the recent module, which might give you a little more ability to control what is being done, either by limiting new connections or by dropping 50% of a users packets.

Logging Partial Content Downloads

Partial content can be logged correctly using apache:

# this includes actual counts of actual bytes received (%I) and
# sent (%O); this requires the mod_logio module to be loaded.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" %I %O \"%{User-Agent}i\"" combined

Pre-bitflip mirroring

Several days before each public release, the content will be staged to the master mirror servers, but with restricted permissions on the directories (generally mode 0750), specifically, not world readable.

Mirror servers should have several different user/group accounts on their server, for running the different public services. Typically you find:

  • HTTP server runs as user apache, group apache
  • FTP server runs as user ftp, group ftp
  • RSYNC server runs as user rsync, group rsync
  • a user account for downloading content from the masters (e.g. user mirror, group mirror).

The user account used to download content from the masters must be not be the same as the HTTP, FTP, or RSYNC server accounts. This guarantees that content downloaded with permissions 0750 will not be made available via your public servers yet.

On the morning of the public release, the permissions on the directories on the master servers will change to 0755 - world readable. This is called the bitflip.

Mirrors may either rsync one more time to pick up these new permissions (but won't have to download all the data again), or preferably, can schedule a batch job to bitflip:

$ echo "chmod a+rx /pub/fedora/linux/releases/9" | at '14:45 UTC May 13 2008'
向其它镜像提供内容

Tier 1 mirrors will necessarily need to share content to Tier 2 mirrors before the bitflip. This is done by running another instance of the rsync daemon, on a different port (e.g. 874), with an Access Control List to prevent public downloads, running as a user in the same group as downloaded the content (e.g. group mirror). This could be user mirror, group mirror, who has group read/execute permissions on the still-private content.

Tier 1 mirrors have a tendency to use different authentication methods for granting access to these non-public downloads, they vary from maintaining IP based ACL's to assigning username/password combinations to mirrors wishing to sync from them. Each method has advantages / disadvantages, the IP list is 'simpler' from a mirrormanager perspective as mirrormanager can give you the list of IP's but from an automation standpoint can be more difficult (as rsync's configuration file does not allow that ACL list to be stored in a separate file). Username / passwords can be more versatile as sites mirroring can change IPs without notifying you, but it's easier for those credentials to leak out and get miss-used.

Other MirrorManager features

Statistics

MirrorManager tracks the number of accesses to the mirror list and breaks it down by country, architecture, and repository.

Mirror Maps

The map of all public mirrors is updated daily; if it doesn't display the correct data, please try again later.

Recognition

This product includes GeoLite data created by MaxMind.