From Fedora Project Wiki

(Update a bit.)
Line 9: Line 9:
Contact: #fedora-admin
Contact: #fedora-admin


Persons: abadger1999, ricky, lmacken
Persons: sysadmin-web


Location: Phoenix
Location: Phoenix


Servers: app3 and app4, puppet1
Servers: bapp1, app1, app2, app3 and app4, puppet1


Purpose: Provide In-House Web Applications for our users
Purpose: Provide In-House Web Applications for our users
Line 27: Line 27:


=== Add RPMs to the Fedora Infrastructure Repo ===
=== Add RPMs to the Fedora Infrastructure Repo ===
1. Copy the rpms to lockbox
1. Copy the rpms to puppet1
2. Sign the rpms with the Fedora Infrastructure Key
2. Sign the rpms with the Fedora Infrastructure Key
<pre>
<pre>
Line 34: Line 34:
3. Copy the rpms to the repo directory
3. Copy the rpms to the repo directory
<pre>
<pre>
mv foo-1.0-1.el5.src.rpm /netapp/app/fi-repo/el/5/SRPMS/
mv foo-1.0-1.el5.src.rpm /mnt/fedora/app/fi-repo/5/SRPMS/
mv foo-1.0-1.el5.x86_64.rpm /netapp/app/fi-repo/el/5/x86_64/
mv foo-1.0-1.el5.x86_64.rpm /mnt/fedora/app/fi-repo/5/x86_64/
</pre>
</pre>
4. Run createrepo to regenerate the repo metadata
4. Run createrepo to regenerate the repo metadata
<pre>
<pre>
cd /netapp/app/fi-repo/el/5/SRPMS/
cd /mnt/fedora/app/fi-repo/el/5/SRPMS/
sudo createrepo .
sudo createrepo --update .
cd /netapp/app/fi-repo/el/5/x86_64/
cd /mnt/fedora/app/fi-repo/el/5/x86_64/
sudo createrepo .
sudo createrepo --update .
</pre>
</pre>


Line 51: Line 51:
First log into puppet1 and checkout the repositories our configs are stored in:
First log into puppet1 and checkout the repositories our configs are stored in:
<pre>
<pre>
$ CVSROOT=/cvs/puppet cvs co manifests
git clone /git/puppet
$ CVSROOT=/cvs/puppet cvs co configs
</pre>
</pre>


==== Create the manifest ====
==== Create a module ====
1. cd manifests/services
1. cd modules; mkdir -p packagename/{files,manifests,templates}
2. create a file named myapp.pp with something similar to the following:
2. create a file named manifests/init.pp with something similar to the following:
<pre>
<pre>
class myapp-proxy inherits httpd {
class myapp::app {
apachefile { "/etc/httpd/conf.d/admin.fedoraproject.org/myapp.conf":
    include httpd::app
source => 'web/myapp-proxy.conf'
 
}
    package { "mypackage":
        ensure => installed,
    }
 
    file { "/etc/myapp/myapp.cfg":
        owner  => "root",
        group  => "root",
        mode    => 0600,
        content => template("myapp/myapp-prod.cfg"),
        notify  => Service["httpd"],
        require => Package["mypackage"],
    }
 
    # ... and similar setup for all files needed
 
    file { "/etc/httpd/conf.d/myapp.conf":
        owner  => "root",
        group  => "root",
        mode    => 0644,
        source => "file:///myapp/myapp-app.conf",
        notify  => Service["httpd"],
        require => Package["httpd"],
    }
}
}
</pre>
</pre>
This defines a class that we'll add to the proxy servers to send requests to the application running on the app servers.
 
This defines a server class that we'll add to the app servers.  The package definition uses the name of your application's rpm package to install from a yum repo and get required dependencies.  If you are developing and building the application yourself and have control over when new releases make it to the yum repo, set <code>ensure => latest</code> to automatically get the latest version otherwise set <code>ensure => present</code> so we can vette the latest releases before installing them on the server.
 
<!-- Perhaps we could mention something about adding passwords in the private repo.
<!-- Perhaps we could mention something about adding passwords in the private repo.
-->
-->
3. Continue editing myapp.pp and add something like the following:
3. Continue editing myapp.pp and add something like the following:
<pre>
<pre>
class myapp-server inherits turbogears {
define myapp::proxy(
$myappDatabasePassword='XXXXXXXXXXXXX'
    $website,
    $path,
    $proxyurl
) {
    include httpd::proxy


include supervisor
    file { "/etc/httpd/conf.d/$website/myapp.conf":
 
        owner  => "root",
package { myapp:
        group  => "root",
ensure => latest,
        mode    => 0644,
}
        content => template("myapp/myapp-proxy.conf.erb"),
templatefile { '/etc/myapp.cfg':
        notify => Service["httpd"],
content => template('/var/lib/puppet/config/web/applications/myapp-prod.cfg.erb'),
        require => Httpd::Website[$website],
notify => Service['supervisord'] ,
    }
owner => 48,
mode => '640'
}
}
}
</pre>
</pre>
This defines a server class that we'll add to the app servers.  The package definition uses the name of your application's rpm package to install from a yum repo and get required dependencies.  If you are developing and building the application yourself and have control over when new releases make it to the yum repo, set <code>ensure => latest</code> to automatically get the latest version otherwise set <code>ensure => present</code> so we can vette the latest releases before installing them on the server.
 
This defines a class that we'll add to the proxy servers to send requests to the application running on the app servers.


Now that we've defined the files and packages our app uses we need to define which machines the files and packages belong on.
Now that we've defined the files and packages our app uses we need to define which machines the files and packages belong on.


1. cd ~/manifests/servergroups
1. cd ~/manifests/servergroups
2. If this application is going to run on the RHEL app servers edit appRhel.pp; if it's going to run on the Fedora app servers edit appFc.pp.  In either case we're just including the new server class in the file:
2. edit appRhel.pp to include your myapp::app class.
<pre>
<pre>
class appRhel {
class appRhel {
[...]  
[...]  
include pkgdb-server
include pkgdb::app
include myapp-server
include myapp::app
</pre>
</pre>
3. Next edit the manifest for the proxy servers, proxy.pp:
3. Next edit the manifest for the proxy servers, proxy.pp:
Line 102: Line 127:
class proxy {
class proxy {
[...]  
[...]  
include pkgdb-proxy
    myapp { "admin.fedoraproject.org/myapp":
include myapp-proxy
        website  => "admin.fedoraproject.org",
        path    => "/myapp",
        proxyurl => "http://localhost:10014",
    }
</pre>
</pre>


That's it for the manifests, now we need to create the config files we reference in the manifest file.
That's it for the manifests, now we need to create the config files we reference in the manifest file.


==== Create the proxy config ====
==== Create the app config ====
1. cd ~/puppet/modules/myapp/files
2. create a myapp-app.conf that may look something like this:
<pre>
WSGISocketPrefix run/wsgi
 
# TG implements its own signal handler.
WSGIRestrictSignal Off


1. cd ~/configs/web
# These are the real tunables
2. create myapp-proxy.conf and put the following into the file:
WSGIDaemonProcess myapp processes=8 threads=2 maximum-requests=50000 user=apache group=apache display-name=fas inactivity-timeout=300 shutdown-timeout=10
<pre>
WSGIPythonOptimize 1
<Location /myapp>
RequestHeader set CP-Location /myapp
</Location>


<Location ~ /myapp/(static|tg_js)>
WSGIScriptAlias /myapp /usr/lib/python2.4/site-packages/myapp/myapp.wsgi/myapp
Header unset Set-Cookie
</Location>


RewriteEngine On
<Directory /usr/lib/python2.4/site-packages/myapp>
RewriteRule ^/myapp(.*)      balancer://myappCluster/myapp$1 [P]
    WSGIProcessGroup myapp
    Order deny,allow
    Allow from all
</Directory>
</pre>
</pre>
The first section tells CherryPy that it's running under the /myapp/ directory.


The second, unsets cookies when requesting static resources.  If you have other directories of all static files (images, css, javascript, raw html, etc) include them in the regexp.  This will allow us to setup caching of these directories in the next step.
==== Create the proxy config ====


{{admon/warning|See also the [[ISOP:HAPROXY| HAProxy SOP]] for the new way to balance requests between app server.}}
1. cd ~/puppet/modules/myapp/templates
 
2. create myapp-proxy.erb.conf and put the following into the file:
The last section makes all requests with /myapp as the base directory go to the servers setup in the balancer config file.
3. Edit balancer.conf to tell the proxy server what app servers to send requests to.  Add something like this:
<pre>
<pre>
<Proxy balancer://myappCluster>
ProxyPass <%= path %> <%= proxyurl %>/myapp
BalancerMember http://app3.fedora.phx.redhat.com:8089 timeout=3
ProxyPassReverse <%= path %> <%= proxyurl %>/myapp
BalancerMember http://app4.fedora.phx.redhat.com:8089 timeout=3
</Proxy>
</pre>
</pre>
Currently we have two app servers running RHEL and two servers running Fedora.  If your application is going to run on the RHEL servers, use app1 and app2.  If it's going to run on Fedora, use app3 and app4.  The port number is the one that your TurboGears app is listening on.  If you haven't allocated one yet, look at the [[InfrastructurePrivate/PortRegistry|  PortRegistry]]  to see what's available.  This port may also need to be added to the iptables rules in appFc.pp or appRhel.pp.


==== Caching ====
3. Follow the [[ISOP:HAPROXY| HAProxy SOP]] to add your app there. This is the addres that you give for proxyurl in proxy.pp.
As mentioned in the last section, we have the ability to cache static files for our TurboGears apps.
1. cd ~/configs/web/admin.fedoraproject.org/
2. edit modcache.conf and add a CacheEnable line for every directory we can cache like so:
<pre>
CacheEnable disk /myapp/tg_js/
CacheEnable disk /myapp/static/
</pre>
Remember that if you list a directory in this file, you *must* unset any cookies on the page in the myapp-proxy.conf file.  If you don't the cache will distribute cookies for people's sessions to the wrong clients leading to people being logged in as someone else.


==== Application config file ====
==== Application config file ====
The final piece is to create a config file template for your app.
The final piece is to create a config file template for your app.
1. cd ~/web/applications/
1. cd ~/puppet/modules/myapp/tepmlates
2. edit myapp-prod.cfg.erb
2. edit myapp-prod.cfg.erb


Line 161: Line 180:
</pre>
</pre>
will substitute the password from the config file into the template.  This keeps passwords out of the configs repository and thus keeps them from being logged to a publicly readable list.
will substitute the password from the config file into the template.  This keeps passwords out of the configs repository and thus keeps them from being logged to a publicly readable list.
* server.socket_port should be set to the same port you used in balancer.conf
* The following settings seem to yield reasonable performance.  These are good defaults until you have a chance to test and refine the settings:
<pre>
server.thread_pool=50
server.socket_queue_size=30
#sqlalchemy.pool_size=5
sqlalchemy.max_overflow=21
</pre>
* Remember to set <code>server.environment="production"</code> instead of <code>"development"</code>.
* Since the app will be running under /myapp, and behind a proxy, make sure the following are set correctly:
<pre>
server.webpath="/myapp"
base_url_filter.on = True
base_url_filter.use_x_forwarded_host = True
base_url_filter.base_url = "https://admin.fedoraproject.org/myapp"
</pre>
=== Configure supervisor ===
{{admon/note|We are slowly moving from deploying behind supervisor to deploying via mod_wsgi}}
Supervisor starts our applications.
1. Log into puppet1
3. cd configs/web/applications
4. edit supervisord.conf.  You want to add a new entry similar to this:
<pre>
[program:MYAPP]
command=/usr/local/bin/startTurboGearsApp.sh /usr/sbin/start-MYAPP /etc/MYAPP.cfg
priority=9
autostart=true
autorestart=true
startsecs=10
startretries=5
stopsignal=TERM
stopwaitsecs=10
user=apache
</pre>
Modify the MYAPP entries to fit your application.
<code>[program:MYAPP] </code> should contain a short, lowercase version of your program name.  Supervisor commands will use this to identify your program (Like <code>supervisorctl restart myapp</code>).  For more information about these commands, see the [[Infrastructure/SOP/Supervisor|  Supervisor SOP]] .
<code>/usr/sbin/start-MYAPP</code> should be the path to the script you use to start your application.
<code>/etc/MYAPP.cfg</code> is the path to the config file you use with your application.


== Upgrading an App ==
== Upgrading an App ==
Line 213: Line 186:
Then on puppet1 run:
Then on puppet1 run:
<pre>
<pre>
sudo func 'app[1-5].fedora*' call command run 'yum clean metadata'
sudo func '*app[1-6].fedora*' call command run 'yum clean metadata'
sudo func 'app[1-5].fedora*' call command run 'yum -y upgrade APPPKGNAME'
sudo func '*app[1-6].fedora*' call command run 'yum -y upgrade APPPKGNAME'
sudo func 'app[1-2].fedora*' call command run 'supervisorctl restart packagedb'
sudo func '*app[1-2].fedora*' call command run '/etc/init.d/httpd graceful'
sudo func 'app[3-5].fedora*' call command run 'supervisorctl restart packagedb'
sudo func '*app[3-6].fedora*' call command run '/etc/init.d/httpd graceful'
</pre>
</pre>
'''
'''
Line 224: Line 197:
The first two commands upgrade the package on the app server.
The first two commands upgrade the package on the app server.


The second two commands restart the app.  We do it in two parts so that we always have some app servers ready to handle requests.  This should avoid downtime.
The second two commands restart apache.  We do it in two parts so that we always have some app servers ready to handle requests.  This should avoid downtime.


After restarting the servers it may be necessary to clean the cache of static files.  This is because javscript, css, and other static files are cached.  If those reference things that are not available in the new server, then we will get errors.  Cleaning the cache is done by rm'ing the cache on the proxy servers.
After restarting the servers it may be necessary to clean the cache of static files.  This is because javscript, css, and other static files are cached.  If those reference things that are not available in the new server, then we will get errors.  Cleaning the cache is done by rm'ing the cache on the proxy servers.


<pre>
<pre>
ssh proxy1
ssh proxy[1-5]
sudo su -
sudo su -
rm -rf /srv/cache/mod_cache/*
rm -rf /srv/cache/mod_cache/*

Revision as of 02:21, 2 July 2009

Shortcut:
ISOP:TG

We have many TurboGears applications deployed in our infrastructure. This SOP, the Supervisor SOP, and HaProxy SOP explain how TurboGears apps are deployed.

Contact Information

Owner: Fedora Infrastructure Team

Contact: #fedora-admin

Persons: sysadmin-web

Location: Phoenix

Servers: bapp1, app1, app2, app3 and app4, puppet1

Purpose: Provide In-House Web Applications for our users

Deploying a new App

These instructions will help you setup a load balanced turbogears application that runs on a URL of the form:

https://admin.fedoraproject.org/myapp

Configuration of the new application is done on puppet1. If you need to drop rpms of the application into the fedora infrastructure repository (because they are not available in Fedora), that presently occurs on lockbox.

Add RPMs to the Fedora Infrastructure Repo

1. Copy the rpms to puppet1 2. Sign the rpms with the Fedora Infrastructure Key

rpm --add-sign foo-1.0-1.el5.*.rpm

3. Copy the rpms to the repo directory

mv foo-1.0-1.el5.src.rpm /mnt/fedora/app/fi-repo/5/SRPMS/
mv foo-1.0-1.el5.x86_64.rpm /mnt/fedora/app/fi-repo/5/x86_64/

4. Run createrepo to regenerate the repo metadata

cd /mnt/fedora/app/fi-repo/el/5/SRPMS/
sudo createrepo --update .
cd /mnt/fedora/app/fi-repo/el/5/x86_64/
sudo createrepo --update .
There are some notes on packaging TG applications on the wiki.

Configure the application

First log into puppet1 and checkout the repositories our configs are stored in:

git clone /git/puppet

Create a module

1. cd modules; mkdir -p packagename/{files,manifests,templates} 2. create a file named manifests/init.pp with something similar to the following:

class myapp::app {
    include httpd::app

    package { "mypackage":
        ensure => installed,
    }

    file { "/etc/myapp/myapp.cfg":
        owner   => "root",
        group   => "root",
        mode    => 0600,
        content => template("myapp/myapp-prod.cfg"),
        notify  => Service["httpd"],
        require => Package["mypackage"],
    }

    # ... and similar setup for all files needed

    file { "/etc/httpd/conf.d/myapp.conf":
        owner   => "root",
        group   => "root",
        mode    => 0644,
        source  => "file:///myapp/myapp-app.conf",
        notify  => Service["httpd"],
        require => Package["httpd"],
    }
}

This defines a server class that we'll add to the app servers. The package definition uses the name of your application's rpm package to install from a yum repo and get required dependencies. If you are developing and building the application yourself and have control over when new releases make it to the yum repo, set ensure => latest to automatically get the latest version otherwise set ensure => present so we can vette the latest releases before installing them on the server.

3. Continue editing myapp.pp and add something like the following:

define myapp::proxy(
    $website,
    $path,
    $proxyurl
) {
    include httpd::proxy

    file { "/etc/httpd/conf.d/$website/myapp.conf":
        owner   => "root",
        group   => "root",
        mode    => 0644,
        content => template("myapp/myapp-proxy.conf.erb"),
        notify  => Service["httpd"],
        require => Httpd::Website[$website],
    }
}

This defines a class that we'll add to the proxy servers to send requests to the application running on the app servers.

Now that we've defined the files and packages our app uses we need to define which machines the files and packages belong on.

1. cd ~/manifests/servergroups 2. edit appRhel.pp to include your myapp::app class.

class appRhel {
[...] 
include pkgdb::app
include myapp::app

3. Next edit the manifest for the proxy servers, proxy.pp:

class proxy {
[...] 
    myapp { "admin.fedoraproject.org/myapp":
        website  => "admin.fedoraproject.org",
        path     => "/myapp",
        proxyurl => "http://localhost:10014",
    }

That's it for the manifests, now we need to create the config files we reference in the manifest file.

Create the app config

1. cd ~/puppet/modules/myapp/files 2. create a myapp-app.conf that may look something like this:

WSGISocketPrefix run/wsgi

# TG implements its own signal handler.
WSGIRestrictSignal Off

# These are the real tunables
WSGIDaemonProcess myapp processes=8 threads=2 maximum-requests=50000 user=apache group=apache display-name=fas inactivity-timeout=300 shutdown-timeout=10
WSGIPythonOptimize 1

WSGIScriptAlias /myapp /usr/lib/python2.4/site-packages/myapp/myapp.wsgi/myapp

<Directory /usr/lib/python2.4/site-packages/myapp>
    WSGIProcessGroup myapp
    Order deny,allow
    Allow from all
</Directory>

Create the proxy config

1. cd ~/puppet/modules/myapp/templates 2. create myapp-proxy.erb.conf and put the following into the file:

ProxyPass <%= path %> <%= proxyurl %>/myapp
ProxyPassReverse <%= path %> <%= proxyurl %>/myapp

3. Follow the HAProxy SOP to add your app there. This is the addres that you give for proxyurl in proxy.pp.

Application config file

The final piece is to create a config file template for your app. 1. cd ~/puppet/modules/myapp/tepmlates 2. edit myapp-prod.cfg.erb

You should look at other application's config files and the one you've been using for testing locally. A few things to note:

  • This file is a template. So using:
<%= myappDatabasePassword %>

will substitute the password from the config file into the template. This keeps passwords out of the configs repository and thus keeps them from being logged to a publicly readable list.

Upgrading an App

First put the new packages in the infrastructure repo as noted above.

Then on puppet1 run:

sudo func '*app[1-6].fedora*' call command run 'yum clean metadata'
sudo func '*app[1-6].fedora*' call command run 'yum -y upgrade APPPKGNAME'
sudo func '*app[1-2].fedora*' call command run '/etc/init.d/httpd graceful'
sudo func '*app[3-6].fedora*' call command run '/etc/init.d/httpd graceful'

When running yum upgrade, make sure you specify the APPPKGNAME! We don't want to have yum upgrade every package on the box as, in many cases, we need to review the packages that will be updated instead of blindly applying them.

The first two commands upgrade the package on the app server.

The second two commands restart apache. We do it in two parts so that we always have some app servers ready to handle requests. This should avoid downtime.

After restarting the servers it may be necessary to clean the cache of static files. This is because javscript, css, and other static files are cached. If those reference things that are not available in the new server, then we will get errors. Cleaning the cache is done by rm'ing the cache on the proxy servers.

ssh proxy[1-5]
sudo su -
rm -rf /srv/cache/mod_cache/*


Troubleshooting and Resolution

[COMMON ISSUES AND HOW TO FIX THEM]