From Fedora Project Wiki
(45 intermediate revisions by 9 users not shown)
Line 2: Line 2:
Fedora Cloud Product Requirements Document.
Fedora Cloud Product Requirements Document.
= Document Purpose and Overview =
= Document Purpose and Overview =
== What this document describes ==
This is the [http://en.wikipedia.org/wiki/Product_requirements_document Product Requirements Document] for the Fedora Cloud SIG. It:
This is the [http://en.wikipedia.org/wiki/Product_requirements_document wikipedia Product Requirements Document] for the Fedora Cloud SIG. It:
* Provides a high-level market of the cloud computing market as it pertains to the Fedora Cloud SIG; this includes overviews of things which may not be within our actual scope/ability to accomplish at the current time.
* Provides a high-level market of the cloud computing market as it pertains to the Fedora Cloud SIG; this includes overviews of things which may not be within our actual scope/ability to accomplish at the current time.
* Provides deeper understanding of the types of users who could use Fedora for their cloud computing needs. This includes describing their main day-to-day tasks, common problems, etc. The perspective here is not necessarily limited to system administrators, or developers, but a combination of many types of users and roles.
* Provides deeper understanding of the types of users who could use Fedora for their cloud computing needs. This includes describing their main day-to-day tasks, common problems, and so on. The perspective here is not necessarily limited to system administrators, or developers, but a combination of many types of users and roles.
* Ties common issues and needs of potential users/consumers of the Fedora Cloud product to high-level product needs, from a "functional" standpoint
* Ties common issues and needs of potential users and consumers of the Fedora Cloud product to high-level product needs, from a "functional" standpoint
* Provides solutions, in the form of "changes" or "features" which will provide the functionality described as needs for the potential users.
* Provides solutions, in the form of "changes" or "features" which will provide the functionality described as needs for the potential users.
= Release/Product Overview =
= Release/Product Overview =
Fedora Cloud provides a customizable base image and tools for developing scale out applications on public and private clouds, as well as a small number of images pre-configured for specific uses.
Fedora Cloud provides a customizable base image and tools for developing scale out applications on public and private clouds, as well as two to four images pre-configured for what we expect to be the most popular scenarios/use cases for using Fedora for cloud computing.
Public and private cloud adoption is taking off, and the requirements for an image OS differ significantly from the requirements for a desktop or server OS. In these environments, much or all of the instance lifecycle — from the creation of the image and addition of software or configuration specific to the instance, to the teardown of the image — will be automated.  Systems are designed to scale out via many identical nodes rather than scale up with carefully-tended individual servers. Individual uptime (mean time between failure) is not as important as the ability to get a new instance running quickly (mean time to recovery).
Public and private cloud adoption is taking off, and the requirements for an image OS differ significantly from the requirements for a desktop or server OS. In these environments, much or all of the instance lifecycle — from the creation of the image and addition of software or configuration specific to the instance, to the teardown of the image — will be automated.  Systems are designed to scale out via many identical nodes rather than scale up with carefully-tended individual servers. Individual uptime (mean time between failure) is not as important as the ability to get a new instance running quickly (mean time to recovery).
With that in mind, we're tailoring a release specifically for cloud environments.  
With that in mind, we're tailoring a release specifically for cloud environments.  
Line 16: Line 15:
In the next two to three years, we expect to see a great deal of growth in adoption and still see a number of emerging players where no clear favorites have emerged (for instance, Google Compute Engine). Additionally, some platforms (Amazon Web Services) have matured to a point where a large number of companies are relying upon the technology for their full infrastructure. While this is not a widespread practice, AWS is seeing a great deal of adoption and will likely start eating into "traditional" workloads that currently live behind the firewall.
In the next two to three years, we expect to see a great deal of growth in adoption and still see a number of emerging players where no clear favorites have emerged (for instance, Google Compute Engine). Additionally, some platforms (Amazon Web Services) have matured to a point where a large number of companies are relying upon the technology for their full infrastructure. While this is not a widespread practice, AWS is seeing a great deal of adoption and will likely start eating into "traditional" workloads that currently live behind the firewall.
In short, there's an enormous opportunity for Fedora to become an instance-OS of choice if the project moves quickly, develops or adopts the right technologies, and succeeds in educating the market about its existence. A failure on any of those three points means that the Fedora Cloud product will have little chance in taking a significant portion of the new market or taking any of the existing market.  
In short, there's an enormous opportunity for Fedora to become an instance-OS of choice if the project moves quickly, develops or adopts the right technologies, and succeeds in educating the market about its existence. A failure on any of those three points means that the Fedora Cloud product will have little chance in taking a significant portion of the new market or taking any of the existing market.  
== Product Objectives ==
 
The Fedora Cloud product consists of an image to be used to run one or more instances in a public cloud, and a set of tools for creating and modifying images. We will also provide two to four additional images that are pre-configured for what we expect to be the most popular scenarios/use cases for using Fedora in public or private cloud.
=== Major Release Themes ===
=== Major Release Themes ===
Cloud computing in general is the transition of computing power from individual hand-tended resources to a ubiquitous utility. Fedora fits into this at several levels, from the infrastructure service software we include (like OpenStack and Eucalyptus) to end-user tools.
Cloud computing in general is the transition of computing power from individual hand-tended resources to a ubiquitous utility. Fedora fits in at several levels, from the infrastructure service software we include (like OpenStack and Eucalyptus) to end-user tools.
The Fedora Cloud image fits into middle of this, providing a guest OS image to run on Infrastructure as a Service systems, on which platform and application services can be deployed. We targets use cases which fit the "cattle" side of the "pets vs. cattle" metaphor for computing.
 
The Fedora Cloud image fits into middle of this, providing a guest OS image to run on Infrastructure as a Service systems, on which platform and application services can be deployed. We targets use cases which fit the "cattle" side of the "[http://www.slideshare.net/randybias/architectures-for-open-and-scalable-clouds pets vs. cattle]" metaphor for computing.
 
We also need to address demand for Linux containers. Docker, Rocket (rkt), systemd-nspawn, as well as orchestration technologies (Kubernetes, Mesos, etc.) are going to play a huge role in developing, deploying, and managing cloud services. The Cloud Working Group needs to address these technologies and help guide their development within Fedora.
 
=== Secondary Objectives ===
=== Secondary Objectives ===
Aside from adoption and development of applications on top of the Fedora Cloud images, we have a few secondary goals that should be helped by wider adoption:
Aside from adoption and development of applications on top of the Fedora Cloud images, we have a few secondary goals that will be helped by wider adoption:
* More testing of Fedora images with additional bug reports.
* More testing of Fedora images with additional bug reports.
* Better feedback about how the product should improve. This is separate from "bug reports" in that we hope to engage the audience and receive detailed feedback about use cases, desired features, developing trends in cloud management, etc.
* Better feedback about how the product should improve. This is separate from "bug reports" in that we hope to engage the audience and receive detailed feedback about use cases, desired features, developing trends in cloud management, etc.
* Patches and contributions that will help improve the product, and Fedora in general. Assuming we're successful, some users should take an interest in helping to develop our product.
* Patches and contributions that will help improve the product, and Fedora in general. As we are increasingly successful, more users will take an interest in helping to develop our product.
 
== Target Market / Audience ==
== Target Market / Audience ==
Developers creating scale out applications on top of public and private clouds, and organizations and users running those applications.
Developers and operators creating scale out applications on top of public and private clouds, and organizations and users running those applications. Fedora is particularly interested in organizations that might want to contribute back to Fedora and be involved in helping find/report/fix bugs, and develop new features.  
== Delivery Mechanisms ==
== Delivery Mechanisms ==
We need to ensure that images are as easy to consume as possible.
Cloud images must be easy to consume as part of a public or private cloud. Because we target these environments, we won't be worrying about physical media at all. The cloud images also won't be "installed" in the same manner that users are accustomed to with desktop or server images. The cloud image will simply boot in its target environment ready to run, or for further customization and configuration via a boostrapping service.
Since Fedora Cloud images are meant to be consumed as part of a public or private cloud, we won't be worrying about physical media at all. The cloud images also won't be "installed" in the same manner that users are accustomed to with desktop or server images. The cloud image will simply boot in its target environment ready to run and/or for further customization/configuration.
=== Where to obtain ===
=== Where to obtain ===
Users will be able to obtain the images for public clouds via download ''or'' via the usual marketplace for those images. For instance, we publish Amazon AMIs on Amazon directly. Users are able to launch new instances with Fedora without having to obtain the images directly from the Fedora Project and then upload to Amazon.
Users will be able to obtain the images for public clouds via download ''or'' via the usual marketplace for those images. For instance, we publish AMIs directly to the [https://aws.amazon.com/marketplace AWS Marketplace]. Users are able to launch new instances with Fedora without having to obtain the images directly from the Fedora Project and then upload to AWS.
Users will be able to download appropriate images for Apache CloudStack, Eucalyptus, OpenStack, and other IaaS platforms.
Users will be able to download appropriate images for Apache CloudStack, Eucalyptus, OpenStack, OpenNebula, and other IaaS platforms.
 
=== Delivery Format ===
=== Delivery Format ===
Images will be delivered as AMIs on Amazon EC2, and as downloadable images in qcow2 and raw.xz formats. We may add other public cloud images and other downloadable formats to meet demand or anticipated need.
Images will be delivered as AMIs on Amazon EC2, and as downloadable images in qcow2 and raw.xz formats. We may add other public cloud images and other downloadable formats to meet demand or anticipated need.
=== Architectures ===
Starting F24, we will be targeting 64-bit x86 and drop 32-bit x86. As ARM-based cloud systems become available, we will add ARM images.
=== Updates ===
=== Updates ===
Fedora Cloud images can be updated using yum as normal. We also intend to produce periodic respun images with security updates, once the infrastructure is in place to support that.
Fedora Cloud images can be updated using [[dnf|DNF]] or [[yum|YUM]] as normal. We also intend to produce periodic respun images with security updates, once the infrastructure is in place to support that.
 
=== Image Creation Toolkit ===
=== Image Creation Toolkit ===
We will also maintain a set of tools that can be used to generate, modify, and configure Fedora instances for use with public and private clouds.
We will also maintain a set of tools that can be used to generate, modify, and configure Fedora instances for use with public and private clouds. Initial efforts will focus on creating official images in Fedora's build system. This effort is in parallel to that and does not block the main release. The Fedora Cloud SIG is also interested in a curated library of image templates.
Unofficial templates will be made available through a peer-reviewed portal, so that users can use these templates with confidence to bootstrap their own images.
 
== Measuring Success ==
== Measuring Success ==
Currently, Fedora is not a widely used option for instances on public and private clouds. We know there's some usage, but it's not one of the top three or four OSes on Amazon or (likely) for private clouds.
Currently, Fedora is not a widely used option for instances on public and private clouds. We know there's some usage, but it's not one of the top three or four OSes on Amazon or (likely) for private clouds.
Line 47: Line 57:
* Increased contribution and participation in the Fedora Cloud WG and Fedora Project in general.
* Increased contribution and participation in the Fedora Cloud WG and Fedora Project in general.
= User Profiles, Goals, and Primary Use Cases =
= User Profiles, Goals, and Primary Use Cases =
Still working out some logic on this section. But forging ahead with what I have for the moment.
Goal of this section is to provide insight into either or both of:
* Primary Use Cases: What are the situations / environments in which we expect a Fedora Cloud Product to be used
* User Profiles and Goals: This is more like “personas” work, or could be done as “user stories” (more along the lines of agile, see http://en.wikipedia.org/wiki/User_story )
... and then ensure that for each type of user or use case, we have features/changes that make the Fedora Cloud product useful.
== User Profiles ==
== User Profiles ==
Based on [https://docs.google.com/document/d/16rkiXWxxgzGT47_Wc6hzIPzO2-s2JWAPEKD0gP2mt7E OpenStack personas] (licensed under CC-By 3.0).
There are three cloud user roles which describe the tasks of the people who interact with any cloud-based system:
Three Cloud User Roles (based on “Description and Application of Core Cloud User Roles” ACM CHIMIT 2011, December 4 2011) that describe the tasks of the people who interact with any cloud-based Information Technology system:
# Cloud Service Creator: Develops the technical and business aspects of a (simple or high-level) cloud service
# Cloud Service Creator: Develop the technical and business aspects of a (simple or high-level) cloud service
# Cloud Service Provider: Provides all types of services to a Service Consumer
# Cloud Service Provider: Provide all types of services (SPI, etc.) to a Service Consumer
# Cloud Service Consumer: Consumes all types of services offered by a Service Provider
# Cloud Service Consumer: Consume all types of services (SPI) offered by a Service Provider
We will use a set of personas  to describe our target users and their respective needs. These users are primarily in the cloud service creator and cloud service provider roles.
=== Persona #1: Alan the AWS enthusiast ===
This document lists the personas in summary form, with detailed  explanations of each one available in a separate document. https://fedoraproject.org/wiki/Cloud_Personas. These roles include aspects of cloud service creators, providers, and consumer.
Alan was a very early AWS adopter. He writes and maintains a number of AWS-deployed applications, including staging and production, from the same data source. He needs to monitor his applications for cost, resource consumption, demand peaks and any crashes. He works for a small start-up. Puppet master.
# AWS enthusiast and Early Adopter
* Adoption curve: Innovator
#* Writes and maintains a number of AWS-deployed applications including staging and production.
* Skills: Experienced web developer & operations guy - not on his first cloud application architecture, mastery of continuous integration (CI) and continuous delivery (CD).
#* Works for a small start-up, primarily on his own. Automates as much as possible.
* Behavior patterns: Wants to concentrate on application development, not architecture. Change management - applying security updates to guest systems - takes a chunk of his time. He spends a little time every few months evaluating the cloud architecture to consider alternative components to increase price/performance.
# DevOps Team Member
* Goals: To deliver excellent scalable web-deployed applications with a web interface and robust API for a mobile application front-end. To deliver incremental, tested updates to the applications on a regular basis.
#* Uses the cloud to do Ruby on Rails development utilizing virtual machines.
* Needs: Application lifecycle management, monitoring, to make his apps faster. Makes heavy use of cloud APIs and services to minimize work he has to do setting up his own services.
#* Works as part of a DevOps team responsible for all aspects of a set of applications.
* Main Tasks: writing and deploying code. Anything that requires a significant distraction from that is a time sink, and Alan has little tolerance for mucking with infrastructure more than he has to.  
#* Development, QA, and Production environments need to be identical.
* Attitudes: Alan loves AWS. He's unlikely to consider an alternative public cloud due to dependencies on AWS services and the fact that migration would consume too many cycles that he doesn't have.
# Web Developer
* Beliefs: Continuous deployment is king. Automate all the things. Do the hardest things often, until they’re easy. etc.
#* Concentrates on app development, not architecture, deployment, or operations.
* Motivations: getting the code running in production
#* Unlikely to care deeply about OS internals, cares that they can easily deploy the frameworks they want to use.
* Frustrations: poor command line tools,
# Large-Environment Sysadmin
* Environment: Macbook for development, AWS for development, testing, and production environments.
#* Interested in deploying self-service PaaS to lighten operational load
* Interface Usage Tendencies:
#* Will work closely with development team on deployment and development.  
** GUI: Medium
# Data Scientist
** CLI: Medium
#* Works with large data sets
** API: High
#* Interested in big data tools and dedicated scripting languages (R, Octave, etc.)
* Collaborates With: Alan is pretty much working alone. He may work with other folks in the start-up, but not on a daily or regular basis. His work flow is tooled specifically for his convenience.
# HPC Scientist
=== Persona #2: Erin the Rails devops team member ===
#* Moving from traditional batch and grid technology to the cloud.
Erin uses the cloud to do Ruby on Rails development utilizing virtual machines.
#* Interested in hybrid cloud (infrastructure overflow) for some massive computation campaigns.
* Adoption curve: Early adopter
* Skills: Familiar with Linux systems. Several years' programming experience, comfortable with Linux and other environments.
* Behavior patterns: Will choose from the range of available flavors of VMs to run. Will use OpenStack images to launch instances, also create and save her own.
* Goals: To set up and use cloud guests for her application with minimal effort. Erin is focussed on the application and regards the IaaS services as resources to be used with a minimum of administrative overhead. Would prefer if her language stack were just there and just worked in the cloud guests of her choice.
* Needs: Easy way to create cloud guests configured with the versions of Ruby and Rails used for her various projects.
* Main Tasks: Deploying and updating code in development, testing, and production environments. Maintains the guest images used as a base in all three.
* Attitudes: Hates it when things change just for change's sake. Wants the latest of the things her team needs, wants the rest
* Beliefs: The development, test, and production enviroments need to be identical.
* Motivations: provisionning environments quickly and easily
* Frustrations: lacking guest images preconfigured with her application framework
* Workflow: Will mostly use the GUI and command line to set up and interact with OpenStack services, and will ensure that rolling out the application will be automated as much as possible. She will use the dashboard to monitor consumption of resources on an ongoing basis. Some else in her group has primary responsibility for managing the IaaS layer, but she can pitch in with a basic understanding if need be.
* Environment: Linux development system. Primarily launches cloud instances via the OpenStack GUI. Part of a devops team which manages the a small OpenStack deployment for their own use.
* Interface Usage Tendencies:
** GUI: Medium-High
** CLI: Medium
** API: Low
* Collaborates With: Other members of her development and production operations team.
=== Persona #3: Walter the Web Developer ===
Walter develops feature rich HTML5 applications using Symfony and Twitter Bootstrap, and a range of other web technologies. He has no real system administration skills, and develops on a single desktop computer or his MacBook. Walter wants the app to Just Work when he’s finished, and doesn’t really think a lot about caching, sharding, proxies, or making his app scalable. He works in a medium sized development shop which has people who take care of deployment and change management to the web applications he produces.
By-line: “Not a bug: Works for me”
* Adoption curve: Early majority (Early adopter for web technologies)
* Skills: PHP, Ruby, Python, Javascript, CSS, multiple MVC frameworks, experienced Linux & MacOS X user (but not admin), comfortable setting up LAMP, github.
* Behavior patterns: Wants to concentrate on application development, not architecture. Spends most of his time in TextMate, and testing how app behaves and looks in VirtualBox.
* Goals: To deliver excellent web applications and robust RESTful API for mobile apps.
* Needs: Up to date application development stack, good developer tools, someone to actually deploy his app.
* Main Tasks:
* Attitudes: Walter would like to know more about the cloud but there’s so much to learn, he has no real idea where to start, and just keeping on top of new web development trends takes all his research time.
* Beliefs: “This stuff should be easier to get a handle on”
* Environment: Macbook for development, a Linux desktop for test deployment, VirtualBox for testing on different browsers.
* Interface Usage Tendencies:
** GUI: High
** CLI: Low
** API: Low
* Collaborates With:
=== Persona #4: Martina the Senior Sysadmin ===
Martina is a senior systems administrator at a large university. She has to manage more systems than she really has time for, and much of her time is interrupted by requests from grad students and other members of the university for resources. The IT department has some automated systems for deploying new sites with Drupal, Wordpress, MediaWiki or the like, but it needs to be done by hand.
Martina's group has recently deployed an OpenStack environment, allowing some groups
to provision virtual machines by themselves. This is some help, but is a level
too low to take care of most users' real needs. She's interested in taking this
one step further and providing a full Platform as a Service offering.
By-line: "I could automate this, if I weren't so busy."
* Adoption curve: Early adopter
* Skills: Shell scripting expert, hacks at Python, Perl, and just about anything else. Good at putting all the parts together.
* Behavior patterns: Would like to work on interesting emerging technologies that she believes will really help her employer and her users, but keeps getting interrupted by break/fix and user help requests.
* Goals: Provide a self-service application infrastructure for users.
* Needs: Robust deployment tools, simple automation stack, ability to rapidly provision large number of hetergenous systems at once, and then efficiently monitor and change them.
* Main Tasks: Provision new systems on both the public cloud and the private OpenStack cloud her company runs. Work with developers to deploy code as often as possible.
* Attitudes: -
* Beliefs: “Whenever a sysadmin has to interact with a user directly, something has gone wrong”
* Environment: Fedora on her desktop, Android on a phone
* Interface Usage Tendencies:
** UI: Low
** CLI: High
** API: Medium-Low
* Collaborates With: Developers, junior administrators, CTO, finance organization
=== Persona #5: Sarah the Data Scientist ===
Sarah is a data scientist in a biotech startup. She works with large data sets and intends to
process them with map-reduce. She plans to test this in her office and then scale out to
local cloud resources and to spot instances in Amazon EC2
By-line: "Crunching a large set of data, get it done and done efficiently"
* Adoption curve: innovator
* Skills: Python, Scala, Octave, R, Hadoop, Cassandra, and Pig.
* Behavior patterns: Most interested in results, but not afraid of tackling new technology to get there.
* Goals: Process data quickly and efficiently, at the lowest cost possible. Since Sarah is potentially working with medical data, she cares about data strorage, retention policies, and ethical issues around her data - not just reliability and cost.
* Needs: -Reliable images with the latest improvements in her toolchain. Does not want to compile her own Hadoop, Pig, Mahout, etc.
* Main Tasks: collecting/crunching data, creating visualizations from data, managing data storage/retention.
* Attitudes: she cares about performance and is energy-conscious. Cares about security and the ethics of data management.
* Beliefs: "The tools are only a means to discover interesting scientific and user data"
* Environment: Fedora on her workstations, CentOS servers in the data center (including an OpenStack deployment),  uses public clouds like Rackspace, the HP cloud, or AWS spot instances  to spin up many servers at once for a short period of time, a Mac laptop
* Interface Usage Tendencies:
** GUI: Medium
** CLI: Medium
** API: Medium
* Collaborates With: Other researchs and scientists, lab technicians, HPC techs, more 'pure' developers
=== Persona #6: Jody the HPC Scientist ===
Jody is a researcher at a major university. She previously used grids to solve meteorological simulations using
batch schedulers (SGE, Torque).
Since she needs to run many parallels simulations, she thinks about using cloud ressources for temporary infrastructure overflow.
By-line: "I need a bunch of machines to run my computational tasks, cloud or not."
* Adoption curve: Early majority
* Skills: Shell, Python, Fortran
* Behavior: Most interested in results, but not afraid of tackling new technology to get there.
* Goals: Run her computational tasks
* Needs: access to distributed filesystems (NFS, GlusterFS)
* Main Tasks: Designing experiments and running them
* Attitudes: She is not a computer nerd, but someone who likes to get things done
* Beliefs: "The tools are only a means to validate my theories and give them real-life applications"
* Environment: Fedora on her lab workstations, CentOS servers in the university data center (including an OpenStack deployment), uses public clouds like Rackspace, the HP cloud, or AWS spot instances to spin up many servers at once for a short period of time, a Mac laptop
* Interface Usage Tendencies:
** GUI: Medium
** CLI: Medium
** API: Medium
* Collaborates With: other scientists, few research engineers (sysadmin or developers)
== Primary Use Cases ==
== Primary Use Cases ==
=== Web Application Deployment in a Public Cloud ===  
=== Web Application Deployment in a Public Cloud ===  
Modern web applications are deployed as a collection of interconnected
Modern web applications are deployed as a collection of interconnected services, including parts like web servers, application servers, databases, and caching layers. Fault tolerance is handled at the overall orchestration level rather than by individual instances. Fedora Cloud can be the base of each of these parts, providing recent libraries, server software, and language toolchains. Each system will be managed using the public cloud's own management tools plus a configuration management system like Puppet, Chef, Salt, or Ansible.
services, including parts like web servers, application servers,
Code may be developed against different language stacks; if these are readily available to users, that's a win. In the real world, libraries and other code we don't include may also be required and we should make that as painless as possible.
databases, and caching layers. Fault tolerance is handled at the overall
Fedora Cloud will provide access to the following stacks (not exhaustive):
orchestration level rather than by individual instances Fedora Cloud can
* Ruby: Rails, Sinatra
be the base of each of these parts, providing recent libraries, server
* Python 2 & 3: Django, Pyramid, Flask
software, and language toolchains. Each system will be managed using the
* PHP 5.x: Symfony2, ZendFramework 2.x
public cloud's own management tools plus a configuration management system
* Node.js
like Puppet, Chef, Salt, or Ansible.
* Java
* Perl 5.x
* Go
 
=== Web Application Deployment in Private Cloud ===  
=== Web Application Deployment in Private Cloud ===  
As above, but in a locally-deployed and managed private cloud system.
As above, but in a locally-deployed and managed private cloud system.
=== Web Application Deployment in Hybrid Cloud ===  
=== Web Application Deployment in Hybrid Cloud ===  
As above, but rather than a single cloud provider, the application
As above, but rather than a single cloud provider, the application seamlessly takes advantage of resources in both a private and public cloud.
seamlessly takes advantage of resources in both a private and public
=== Hybrid Cloud as Staging Area ===
cloud.
Application development and testing happens in-house on a private cloud, while production goes to a public cloud.
=== Big Data / Number-Crunching ===
Possible scenarios: Using an open source IaaS stack internally and externally (e.g. OpenStack internally, Rackspace externally; Eucalyptus internally, AWS externally; CloudStack internally, any of the CloudStack providers or AWS externally; or using a library like jClouds to smooth out the bumps.) This may include identical software stacks which run in different clouds, or cloud-specific tailored images onto which an application is deployed, or even very different environments under which the same Docker container is run.
Deploy scale-out application for data processing to public, private, or hybrid cloud.
=== Simple Deployment of Code from Development to Production ===
=== Docker Container Host ===
As a developer, I want to ensure that my code is easily deployable from my development environment to a QA environment and finally a production environment, without encountering compatibility issues or surprises from differences in the environment. A solution where an identical customized image or container (such as Docker) is used in all three environments is ideal.  
Docker is a technology for running applications inside a protected
 
container. These containers run under the host kernel, but otherwise
are self-contained. The Fedora Cloud image running in either a private
or public cloud can provide the host level, including basic docker
management tools plus tools for security and for access to storage.
Workflow may involve developing and testing Docker containers on
local system and pushing to cloud for production.
=== OpenShift Origin System ===
OpenShift Origin is an open source platform-as-a-service system already included in Fedora.
A Fedora Cloud image focused on OpenShift would make it easy for users
to run their own PaaS.
=== Simple Deployment of code to from dev to production ===
As a developer, I want to ensure that my code is easily deployable from my development environment to a production environment, without encountering issues of non-compatibility.
Example: A feature described as “Improved deployment of code to a cloud” might be fully described in the Features section, and as a result, Vagrant might be listed as a requirement in the non-functional requirements section, and cross-coordinated with the workstation working group. There are likely numerous other requirements as well.
=== Development as part of a team ===
As a member of a development team, I would like to develop in an environment where code can go through unit or functional testing, and be approved or accepted by other members of the team.
Example: A feature described as “Continuous Integration platform” may be listed in the Features section, and the various tools available to implement  would be enumerated and described in the “non-functional requirements” section, and cross-coordinated with the server working group.
=== Development using libraries not included in the distribution ===
As a developer, my toolchain includes libraries or dependencies on other packages using libraries not in the Fedora distribution, and I may be developing for distributions not limited to Fedora.
Example: A feature described as “Portable code” (REALLY POOR DESCRIPTION, sorry) may be listed in the Features section, and “Software Collections” might be listed as a requirement.
Note that this may have some cross-over with a possible additional story/use case, “Deployment of code using libraries not included in the distribution.”
=  Target IaaS environments =
=  Target IaaS environments =
The Fedora Cloud product can be used as a guest/VM/image under many IaaS services and providers. For projects that are not currently packaged within Fedora/EPEL, we may need to locate a kind person to ensure testing. These include:
The Fedora Cloud product can be used as a guest/VM/image under many IaaS services and providers. For projects that are not currently packaged within Fedora/EPEL, we may need to locate a kind person to ensure testing. These include:
Line 227: Line 122:
* Linode
* Linode
= Features =
= Features =
Features here address the primary and secondary use cases, product or secondary objectives,  market opportunities from above.
== Feature: Ready to run in Public and Private Clouds ==
Features should provide functional requirements (“what it should do”) preferably in a narrative fashion - more of a story / solution description, rather than “package XYZ” - the features (the ways to meet a user's objectives?) each likely consist of more than one package/enhancement, and those packages should be detailed in the “Detailed requirements” section of this document, and each of those detailed requirements should refer back to which feature it supports.
The Fedora cloud image is ready to go out of the box in the public and private cloud environments we target. It includes cloud-init, the defacto standard for boot-time configuration for cloud instances, and heat_cfgtools package, which is used for handling guest initialization with OpenStack Heat.
== Feature #0 ==
Feature description should be described in the line saying “Feature #1/2/etc.”.
Describe the feature in more detail, specifically addressing how it addresses user scenarios, primary or secondary use cases / objectives of the product.
Use a table to indicate the following items:
Priority (Must, Should, NTH)
Citation of use cases addressed
As work continues and specific detailed requirements are developed, reference the detailed requirements within this document helping to fulfill this feature. This helps to ensure awareness around “do we still have a feature if some of the detailed requirements are not fulfilled, and thus are not able to address the specific use case needs / user objectives.
== Feature: Cloud Bootstrap Tools ==
The Fedora Cloud guest image includes cloud-init, the defacto standard for boot-time configuration for cloud instances. It is a critical tool in getting public and private cloud instances properly configured based on metadata present to the hypervisor, like the desired hostname and filesystem configuration of the guest.
Heat, one of the OpenStack sub-projects is also playing an important role in the orchestration landscape. We include the client tools for Heat bootstrapping.
The cloud landscape is rapidly changing and we will consider additional tools to help with guest initialization and setup as they appear.
== Feature: Ready Access to Fedora Collection of Packages ==
== Feature: Ready Access to Fedora Collection of Packages ==
language and application stacks to enable whatever you need... blah blah blah FIXME
* Access to the standard Fedora package repositories
== Feature: Docker support ==
* Standard language stacks (like Python, Ruby, NodeJS, PHP)
Docker is an easy to use interface for running application containers on Linux. Fedora is uniquely positioned to provide the best platform for Docker, since this container technology is not a security solution, but can be made reasonably secure when wrapped with SELinux. This includes adding libvirt support to the image, which is more heavyweight than many users of the image for other purposes will want, so we will produce a dedicated image specfically tailored for Docker.
* Modernized language and application-specific stacks, perhaps in the form of SCLs, for running userspace applications
== Feature: Big Data Tools ==
== Feature: Updated Images ==
... FIXME. .... Exact contents in collaboration with Big Data SIG ....
Currently, the Fedora Cloud images are only produced twice-yearly as part of the general Fedora release. Because Fedora updates constantly, this means that after several months, it is often the case that a newly-launched image has pending updates which effectively replace the entire content. This is slow and wastes bandwidth, and in many cases means that users simply never update.
Instead, we will produce images with security updates and critical bugfixes rolled in. Our initial target is monthly releases, with special updates produced for critical security flaws.
 
== Feature: Cloud -> Server ==
== Feature: Cloud -> Server ==
Easy path from base cloud image to Fedora Server
The Fedora Server product targets more traditional server roles, where systems have a more unique identity. The Fedora Cloud image supports this by providing a path to go from our base to a Fedora Server role, in effect taming one of your cloud computing "cattle" and making it into a "pet" traditional server — but running in a cloud environment.
== Feature: OS Tree ==
 
??? FOR DISCUSSION ???
== Feature: Atomic Host ==
The Fedora Cloud image includes OSTree for safe, atomic updates.....
The Atomic Host provides an easy means to keep your container infrastructure up to date. Atomic provides git-like updates and roll backs of your host packages so your containers can keep running, secure and stable. The Atomic spin will provide updates every two weeks, ensuring that your host receives the latest system updates.
 
== Feature: Dockerfiles ==
Fedora Dockerfiles provides a curated list of Dockerfiles designed to work with Fedora - making both development and deployment of your Docker infrastructure easier. By maintaining a list of supported Dockerfiles, we're able to better saturate the container ecosystem, making it easier for developers to use Fedora in their stack from end to end.
 
== Future Features ==
The following areas are unlikely to be completely addressed in the upcoming iteration of Fedora Cloud, but we intend to work on them in the future.
=== Image Generation Tools ===
The official Fedora Cloud images will be produced in the Fedora build system (Koji) using kickstart scripts, Anaconda, Oz, and ImageFactory. We also want to provide tools and documentation so users can easily respin their own images. This will likely be using ImageFactory as well. We may provide a web-based service for user-created images.
=== Image or Image Template Library ===
In addition to the basic tooling for image creation, we want to enable sharing of user-created images and their recipies. This may apply to Docker or other container images in addition to images targetted at full virtualization.
=== Higher-Level Cloud Tooling ===
Initial efforts focus on individual cloud images. In the future, we want to address higher level concerns as well. These areas include:
# Orchestration
# Performance / Scalability / Failover
# Logging / Auditing
# Monitoring / Notification
# Database requirements


= Requirements =
= Requirements =
'''''TODO: change these things into actual requirements in the lovely format shown below.'''''
== Requirement #1: Reducing Image Footprint ==
* Support for smallification: a) kernel (reqs for kernel team), b) lang & docs (reqs for packaging tools team), c) cloud-init refactoring
Making the Fedora Cloud image fit in a smaller space. Storage is a significant cost element on cloud platforms.
* Docker security ( = strong selling point): Libvirt/SELinux integration for Docker (planned but not done)
=== Feature Addressed ===
* Infrastructure for automatic production and upload of cloud images for updates
Ready to run in Public and Private Clouds.
* easy software stacks -- install your language of choice, preferably your version of things of choice
=== Priority ===
* integration with Fedora Server roles
Nice to Have. The current cloud image is reasonably sized when compared to our competitors. However, smaller footprint has several advantages. Fewer packages means fewer updates and a smaller target for security problems. It makes it faster to download and deploy images across networks where the image storage and compute nodes run separately. And, reducing things our users don't really need gives more room to include things that they do.
* tools for user creation of images -- imagefactory?
=== Specific Areas ===
** repository for these images, or at least their definitions?
This requirement has several areas where effort will yield meaningful results. Each has its own level of effort, stakeholders, and dependencies.
** dockerfiles? same or different?
==== Cloud-Focused Base Kernel ====
** DOCUMENTATION
The cloud-focused base kernel will be a minimal kernel without many of the drivers traditionally necessary on hardware, like most PCI devices. The goal of having a custom kernel package is ultimately to reduce the size of the images which will reduce storage costs, make the images easier to move across networks between compute and image storage nodes.
* web site design for selecting version to launch or download -- specs to web team
 
Note that the kernel binaries will be shared across all of Fedora; the difference will be that not all modules will be packaged and included by default in the cloud image. This also means that it will be simple to add the other drivers if they are needed for special cases.
 
Effort is moderate; it includes defining the exact requirements of the smaller kernel, and creation and maintenance of that separation by the Fedora kernel team.
 
==== Internationalization / Localization ====
Fedora Cloud base images should only include a single default locale since it is meant to be used as an deployment target to save space. Other locales should be available through repositories. The idea is to rely on langpacks rather than the current RPM kludges which leave out internationalization, because that approach does not allow one to easily add language support later.
Effort required is fairly high, because the packaging tools do not currently have such support. It will also likely have some impact on all Fedora packagers, depending on the implementation, and may also involve changes to the packaging guidelines.
==== Included Documentation ====
Documentation has the same issue as internationalization: it is often not needed on each image but takes significant space, and cannot be easily added back if RPM is configured to exclude it initially.
Effort is similar to that for internationalization, although significant impact could be made by breaking out documentation from key packages included in the base image.
==== Refactoring Cloud-Init ====
Cloud Init was initially designed for a different distribution and is only loosely tailored for our needs. As it stands, it pulls in a rather large set of packages not used for other things. It is also written in Python, itself a large subsystem which it would eventually be nice to leave out of the base.
Effort is moderate, with some low-hanging fruit which may be addressed easily.
== Requirement #2: Improved Docker Integration ==
Docker is a popular tool for launching and managing application containers. This runs on bare metal, in virtualization on developers' machines, or in cloud guests.
Right now, Docker can be added to the cloud base image but isn't included by default. Docker itself is small, but infrastructure to support it includes Libvirt, which is not small. In order to make it easier to get started and to speed up use in production, we will produce a Docker-specific image.
Additionally, we want to create a library of Dockerfiles and possibly our own registry of images.
=== Feature Addressed ===
Docker support.
=== Priority ===
Should. This feature is an important change to support a variety of different teams working on PaaS (like OpenShift) and end-user delivery issues. That being said, since there is very little usage of Docker in those tools right now and the package is available in the repositories already, it won't be terribly detrimental to not have the image complete.
=== Effort required ===
High. We can produce the basic image with minor work, but SELinux is a key feature which will make us stand out from other systems on which Docker can be run. Additionally, producing official Docker container images depends on changes to the build system.
=== Stakeholders / Owners / Major Dependencies ===
The Cloud SIG and our users are the major stakeholders. The SELinux team is instrumental in delivering the functionality we would like to see. Fedora Infrastructure and Release Engineering are involved in updating the build system. And the official images will be produced through Release Engineering.
Major dependencies include:
* Better SElinux MCS separation incorporated in the policies shipped in Fedora.
* Creation of an agreed upon set of Dockerfiles for different tasks.
* ImageFactory support for creating Docker images, and that support integrated into Koji.
 
== Requirement #3: Automatic Production and Upload of Images ==
We will develop Fedora Cloud images using text-based descriptive files and use an automated toolchain that generates images for our targets and upload them where appropriate. Currently, this is done by hand. Automation is necessary to scale to multiple supported images and a growing set of targets.
Fedora images must pass automated QA before publication to ensure a minimum level of quality. In order to release updated images with confidence, automated testing is required, because we cannot expect the level of human attention which can be brought to a twice-yearly release.
=== Feature Addressed ===
Updated Images.
=== Priority ===
Must. We simply need this to be useful to our users, and therefore to be competitive.
=== Effort required ===
High. Work is currently ongoing with Release Engineering and QA to facilitate this process.
=== Stakeholders / Owners / Major Dependencies ===
The Fedora Cloud Working Group will work with Release Engineering in implementing and supporting the needed functionality.


== Requirement #1 (Short Description of Requirement) ==
== Requirement #4: Software Stacks for Various Languages ==
People developing and deploying code on top of Fedora need access to the relevant language stacks. Fedora already provides the latest versions of many of these, but often real-world version dependencies do not match the Fedora cycle. We need some way of offering multiple versions of languages and software stacks. Additionally, it is ideal if support for a given version lasts longer than the 13-month Fedora lifecycle, so that moving from one release of the base to the next can be less painful.
The Cloud SIG plan to support the major languages:
* Ruby
* Python 2 & 3
* PHP 5.x
* Java
* Node.js
* Perl 5.x
* Go
We will provide infrastructure and support to volunteers who want to complete the supported stacks or bring new ones.
=== Feature Addressed ===
Ready Access to Fedora Collection of Packages.
=== Priority ===
Must. This addresses a key need of userbase.
=== Effort required ===
High. Access to existing Fedora packages is simple (we simply won't break what currently works), but Fedora does not currently have a good infrastructure for multiple versions of stacks or for decoupled lifecycles.
=== Stakeholders / Owners / Major Dependencies ===
Software in this area will be the responsibility of the Environments and Stacks working group.
 
== Requirement #5: Integration with Fedora Server Roles ==
We simply want to make sure that the tools to convert a Fedora Cloud base image to a Fedora Server role are available and function.
=== Feature Addressed ===
Cloud -> Server
=== Priority ===
Should.  This helps tie the Fedora products together and makes the blurry lines  between the Cloud and Server product more understandable.
=== Effort required ===
Medium. Effort is primarily in coordination and testing.
=== Stakeholders / Owners / Major Dependencies ===
This effort will be done in collaboration with the Fedora Server Working Group.
== Requirement #6:  Tools for User Image Creation and Image / Template Library ==
We do not have a standardized process for end-user image creation, although we have several tools which can be used. We plan to develop this process and document it. This may also include an image library or a library of image creation templates, or both.
=== Features Addressed ===
Image Generation Tools,  Image or Image Template Library
=== Priority ===
Nice to have. This is not required for our initial launch.
=== Effort required ===
High.
=== Stakeholders / Owners / Major Dependencies ===
Image creation tools (Oz/ImageFactory, etc.), Fedora Websites, Fedora Documentation, Cloud SIG users in general.
 
== Requirement #7: Updated Web Site for Obtaining Cloud Images ==
As we provides images for various use cases (ie: Big Data, PaaS hosting, scale-out apps, etc.) and various target platforms, we need an updated website.
Fedora provides its build toolchain and encourage the community to use it to build new products. The updated website should also reflect that by allowing folks to share and review the work done by their peers.
=== Feature(s) Addressed ===
=== Feature(s) Addressed ===
Refer to which previously described Features, Use Cases this requirement helps to fulfill.
Easy access to Fedora Cloud products and encouraging sharing in the community.
=== Priority ===
=== Priority ===
Must, should, NTH
Must. A simple access to the cloud products is also a key argument to their success.
Should.  Providing a sharing platform to the community (as in not supported by  the Cloud SIG) and peer-review based system will allow to service better  our users and foster innovation.
=== Effort required ===
=== Effort required ===
High, Medium, Low
High. We need to define precisely what we want to show, and maintain the website.
=== Stakeholders / Owners ===
=== Stakeholders / Owners / Major Dependencies ===
=== Major Dependencies ===
The Cloud SIG will work closely with Fedora Websites/Infrastructure and possibly Fedora Engineering if we need to develop specific apps and/or integrate with the existing Fedora infrastructure (like fedmsg).
Any major dependencies, including things that may require any cross-working-group coordination, should be called out here. Any process changes required within Fedora should be documented here as well.
=== Testing ===
Level of testing required; is it a blocker to release?
Is the testing automate-able?
=== Other Documentation ===
* Existing BZ:
* Upstream webpage / wiki page / github page(s):
== Requirement #2 (Short Description of Requirement) ==


= Non-functional requirements =
These are the requirements needed that are not necessarily part of implementation of the product itself, but are still required as part of either making the product more attractive/useful, compatibility requirements for a user's workflow (ie: “works with Puppet,”) or things needing to be done/coordinated in other areas of the project (in other working groups?) to ensure a well-rounded solution.
For each, as was done with Features in a previous section, we should call out some additional information, assuming doing so makes sense for the requirement:
* Priority (Must, Should, NTH)
* Effort required
* Major dependencies / Process Changes needed
* Stakeholders/Owners
* Existing Documentation: BZ #, pointers to upstream project docs
== Image Creation ==
How to create images. Do not be shocked if there are 48 ways to do this. Also: not sure if we want to include containers in this section.
=== Creation of “official” Fedora Project images ===
New images. Updated images (ie: providing newer images for a release that have updates already included) might also be included here?
=== Images created by users ===
== Technical requirements ==
=== Configuration Management ===
=== Migration / Upgrades ===
=== Performance / Scalability / Failover ===
=  General requirements =
== Documentation ==
== Documentation ==
We aim to provide a comprehensive documentation covering users to developers.
Should. Provide narrative documentation to make it easier to use cloud images and use them as foundations to new products.
NTH. On a volunteer basis, we will maintain a knowledge base.
=== Fedora Project Documentation ===
=== Fedora Project Documentation ===
The Cloud SIG will cooperate with the Documentation Project to provide documentation to users, and developers of Fedora Cloud Products.
=== Open Source Projects documentation ===
=== Open Source Projects documentation ===
Ensuring that Fedora is well-represented, up-to-date in other open source project documentation...
Ensuring that Fedora is well-represented, up-to-date in other open source project documentation...
== Release Criteria ==
= Technical requirements =
== Maintainability ==
== Support Requirements ==
=== Architectures Supported ===
* x86
* x86_64
* at some point: ARM (which variants ?)
=== Supported (platforms?) ===
Virtualization types? Container types?
== Internationalization / Localization ==
* NTH: Cloud Product base images should only includes en.us_US locale since it is meant to be used as an deployment target to save space. Other locales should be available through repositories.
The idea is to rely on langpacks rather than RPM wizardry.
== Logging / Auditing ==
== Monitoring / Notification ==
== Database requirements ==
== Security ==
= About this Document =
= About this Document =
This PRD (Product Requirements Document) is an evolving document, created by the Fedora Cloud SIG Working Group as part of the process for designing the Fedora Cloud product. The framework for the PRD itself is currently in a draft state.
This PRD (Product Requirements Document) is an evolving document, created by the Fedora Cloud SIG Working Group as part of the process for designing the Fedora Cloud product. The framework for the PRD itself is currently in a draft state.
Line 331: Line 281:
* [[User:skottler|Sam Kottler]]
* [[User:skottler|Sam Kottler]]
* [[User:Hguemar| Haïkel Guémar]]
* [[User:Hguemar| Haïkel Guémar]]
* [[User:jzb| Joe Brockmeier]]
* [[User:frankieonuonga|Frankie Onuonga]]
* [[User:roshi|Mike Ruckman]]
== Reviewers & Contributors ==
== Reviewers & Contributors ==
The following people have contributed to the development of this document, through feedback on IRC, mailing lists, and other points of contact.  
The following people have contributed to the development of this document, through feedback on IRC, mailing lists, and other points of contact.  
* [[User:mattdm|Matthew Miller]]
* [[User:mattdm|Matthew Miller]]
* [[User:skottler|Sam Kottler]]
* [[User:Hguemar| Haïkel Guémar]]
* [[User:jzb|Joe Brockmeier]]
* [[User:frankieonuonga|Frankie Onuonga]]
==  Credits ==
Some of the Cloud User Roles are based on “Description and Application of Core Cloud User Roles” ACM CHIMIT 2011, December 4 2011.
Some aspects of Fedora Cloud personas are based on [https://docs.google.com/document/d/16rkiXWxxgzGT47_Wc6hzIPzO2-s2JWAPEKD0gP2mt7E OpenStack personas] (licensed under CC-By 3.0)
==  Community Information ==
==  Community Information ==
The Fedora Cloud SIG is one of many teams within the Fedora Project.
The Fedora Cloud SIG is one of many teams within the Fedora Project.
Line 341: Line 301:
Over time, it is expected that this document will undergo various rounds of review, approval, and editing; in the future, it may be rewritten for different releases of Fedora.  
Over time, it is expected that this document will undergo various rounds of review, approval, and editing; in the future, it may be rewritten for different releases of Fedora.  
While one can review the history of a wiki document (by clicking the "history" tab), it is useful to provide explicit indicators of any major format changes, approvals, or indications of it being in a “final” state, in a list that can allow someone to quickly see that all of the prescribed layers of approval have occured.  
While one can review the history of a wiki document (by clicking the "history" tab), it is useful to provide explicit indicators of any major format changes, approvals, or indications of it being in a “final” state, in a list that can allow someone to quickly see that all of the prescribed layers of approval have occured.  
* January 22, 2014: [http://meetbot.fedoraproject.org/teams/fesco/fesco.2014-01-22-18.00.log.txt Approval by FESCo]
* January  8, 2014: Collaborative hackfest day.
* January  8, 2014: Collaborative hackfest day.
* October 28, 2013: [http://fedoraproject.org/w/index.php?title=Cloud_PRD&oldid=358260 Initial Draft of template.]
* October 28, 2013: [http://fedoraproject.org/w/index.php?title=Cloud_PRD&oldid=358260 Initial Draft of template.]
* FutureDate: Approval by SomeGroup; link to any pertinent mail announcement and/or meeting minutes
* FutureDate: Approval by SomeGroup; link to any pertinent mail announcement and/or meeting minutes
==Tracking of Progress==
 
This document contains numerous descriptions of use cases, descriptions of feature sets to address the use cases, and the requirements to enable those features. Numerous Fedora self-contained and systemwide changess (in addition to updates to individual packages) may combine to address those use cases and feature sets. Thus, as a single release, or even series of releases, undergoes development, it is useful to easily track how an entire use case or feature set may be progressing.
While Fedora uses the [[Changes/Policy | Changes Process]] to track changes in the distribution, those changes are typically described as details of changes to a specific package, or the introduction of a specific package, rather than as a piece of a larger feature set.
This document could possibly be used to do any or a number of the following things:
* Provide a secondary location where changes are tracked (which seems like a lot of overhead to me)
* Provide a location where overall Feature Progress is tracked, via periodic cross-checking against Change pages; this could be either in a standalone section, or simply attached to each Feature description.
* Scope out  how features are expected to progress over a number of releases.
* None of these things.
When we more fully determine how to most efficiently track progress, the pointer to where that tracking is done, and/or the description of or process by which we do the tracking is formalized, should be documented in this section in lieu of what is currently written here.
== Document Conventions==
== Document Conventions==
=== Definitions and Acronyms ===
=== Definitions and Acronyms ===
* AWS: Amazon Web Services
* AWS: [https://aws.amazon.com/ Amazon Web Services]
* Amazon EC2: Amazon Elastic Compute Cloud, a popular public IaaS
* Amazon EC2: Amazon Elastic Compute Cloud, a popular public IaaS, part of AWS
* IaaS: Infrastructure as a Service
* IaaS: Infrastructure as a Service
* PaaS: Platform as a Service
* PaaS: Platform as a Service
Line 371: Line 324:
* CLI: Command Line Interface
* CLI: Command Line Interface
* API: Application Programming Interface
* API: Application Programming Interface
=== Indication of prioritization ===
* RDBMS: Relational DataBase Management System
=== Other ===
* [[FESCo]]: Fedora Engineering Steering Committee
* Items marked with a ? and following question in italicized lettering are open questions.
 
 
<references />
[[Category:Product_Requirements_Document]]

Revision as of 11:57, 8 October 2015

The Fedora Cloud Special Interest Group

Fedora Cloud Product Requirements Document.

Document Purpose and Overview

This is the Product Requirements Document for the Fedora Cloud SIG. It:

  • Provides a high-level market of the cloud computing market as it pertains to the Fedora Cloud SIG; this includes overviews of things which may not be within our actual scope/ability to accomplish at the current time.
  • Provides deeper understanding of the types of users who could use Fedora for their cloud computing needs. This includes describing their main day-to-day tasks, common problems, and so on. The perspective here is not necessarily limited to system administrators, or developers, but a combination of many types of users and roles.
  • Ties common issues and needs of potential users and consumers of the Fedora Cloud product to high-level product needs, from a "functional" standpoint
  • Provides solutions, in the form of "changes" or "features" which will provide the functionality described as needs for the potential users.

Release/Product Overview

Fedora Cloud provides a customizable base image and tools for developing scale out applications on public and private clouds, as well as two to four images pre-configured for what we expect to be the most popular scenarios/use cases for using Fedora for cloud computing. Public and private cloud adoption is taking off, and the requirements for an image OS differ significantly from the requirements for a desktop or server OS. In these environments, much or all of the instance lifecycle — from the creation of the image and addition of software or configuration specific to the instance, to the teardown of the image — will be automated. Systems are designed to scale out via many identical nodes rather than scale up with carefully-tended individual servers. Individual uptime (mean time between failure) is not as important as the ability to get a new instance running quickly (mean time to recovery). With that in mind, we're tailoring a release specifically for cloud environments.

Market Opportunity

Public and private cloud adoption is happening rapidly, but the market is not yet mature and is relatively ripe for disruption even though some favorites have emerged as early leaders. In the next two to three years, we expect to see a great deal of growth in adoption and still see a number of emerging players where no clear favorites have emerged (for instance, Google Compute Engine). Additionally, some platforms (Amazon Web Services) have matured to a point where a large number of companies are relying upon the technology for their full infrastructure. While this is not a widespread practice, AWS is seeing a great deal of adoption and will likely start eating into "traditional" workloads that currently live behind the firewall. In short, there's an enormous opportunity for Fedora to become an instance-OS of choice if the project moves quickly, develops or adopts the right technologies, and succeeds in educating the market about its existence. A failure on any of those three points means that the Fedora Cloud product will have little chance in taking a significant portion of the new market or taking any of the existing market.

Major Release Themes

Cloud computing in general is the transition of computing power from individual hand-tended resources to a ubiquitous utility. Fedora fits in at several levels, from the infrastructure service software we include (like OpenStack and Eucalyptus) to end-user tools.

The Fedora Cloud image fits into middle of this, providing a guest OS image to run on Infrastructure as a Service systems, on which platform and application services can be deployed. We targets use cases which fit the "cattle" side of the "pets vs. cattle" metaphor for computing.

We also need to address demand for Linux containers. Docker, Rocket (rkt), systemd-nspawn, as well as orchestration technologies (Kubernetes, Mesos, etc.) are going to play a huge role in developing, deploying, and managing cloud services. The Cloud Working Group needs to address these technologies and help guide their development within Fedora.

Secondary Objectives

Aside from adoption and development of applications on top of the Fedora Cloud images, we have a few secondary goals that will be helped by wider adoption:

  • More testing of Fedora images with additional bug reports.
  • Better feedback about how the product should improve. This is separate from "bug reports" in that we hope to engage the audience and receive detailed feedback about use cases, desired features, developing trends in cloud management, etc.
  • Patches and contributions that will help improve the product, and Fedora in general. As we are increasingly successful, more users will take an interest in helping to develop our product.

Target Market / Audience

Developers and operators creating scale out applications on top of public and private clouds, and organizations and users running those applications. Fedora is particularly interested in organizations that might want to contribute back to Fedora and be involved in helping find/report/fix bugs, and develop new features.

Delivery Mechanisms

Cloud images must be easy to consume as part of a public or private cloud. Because we target these environments, we won't be worrying about physical media at all. The cloud images also won't be "installed" in the same manner that users are accustomed to with desktop or server images. The cloud image will simply boot in its target environment ready to run, or for further customization and configuration via a boostrapping service.

Where to obtain

Users will be able to obtain the images for public clouds via download or via the usual marketplace for those images. For instance, we publish AMIs directly to the AWS Marketplace. Users are able to launch new instances with Fedora without having to obtain the images directly from the Fedora Project and then upload to AWS. Users will be able to download appropriate images for Apache CloudStack, Eucalyptus, OpenStack, OpenNebula, and other IaaS platforms.

Delivery Format

Images will be delivered as AMIs on Amazon EC2, and as downloadable images in qcow2 and raw.xz formats. We may add other public cloud images and other downloadable formats to meet demand or anticipated need.

Architectures

Starting F24, we will be targeting 64-bit x86 and drop 32-bit x86. As ARM-based cloud systems become available, we will add ARM images.

Updates

Fedora Cloud images can be updated using DNF or YUM as normal. We also intend to produce periodic respun images with security updates, once the infrastructure is in place to support that.

Image Creation Toolkit

We will also maintain a set of tools that can be used to generate, modify, and configure Fedora instances for use with public and private clouds. Initial efforts will focus on creating official images in Fedora's build system. This effort is in parallel to that and does not block the main release. The Fedora Cloud SIG is also interested in a curated library of image templates. Unofficial templates will be made available through a peer-reviewed portal, so that users can use these templates with confidence to bootstrap their own images.

Measuring Success

Currently, Fedora is not a widely used option for instances on public and private clouds. We know there's some usage, but it's not one of the top three or four OSes on Amazon or (likely) for private clouds. Success looks like:

  • Increase in adoption.
  • Third party support / targeting of Fedora Cloud as a platform.
  • Increased contribution and participation in the Fedora Cloud WG and Fedora Project in general.

User Profiles, Goals, and Primary Use Cases

User Profiles

There are three cloud user roles which describe the tasks of the people who interact with any cloud-based system:

  1. Cloud Service Creator: Develops the technical and business aspects of a (simple or high-level) cloud service
  2. Cloud Service Provider: Provides all types of services to a Service Consumer
  3. Cloud Service Consumer: Consumes all types of services offered by a Service Provider

We will use a set of personas to describe our target users and their respective needs. These users are primarily in the cloud service creator and cloud service provider roles. This document lists the personas in summary form, with detailed explanations of each one available in a separate document. https://fedoraproject.org/wiki/Cloud_Personas. These roles include aspects of cloud service creators, providers, and consumer.

  1. AWS enthusiast and Early Adopter
    • Writes and maintains a number of AWS-deployed applications including staging and production.
    • Works for a small start-up, primarily on his own. Automates as much as possible.
  2. DevOps Team Member
    • Uses the cloud to do Ruby on Rails development utilizing virtual machines.
    • Works as part of a DevOps team responsible for all aspects of a set of applications.
    • Development, QA, and Production environments need to be identical.
  3. Web Developer
    • Concentrates on app development, not architecture, deployment, or operations.
    • Unlikely to care deeply about OS internals, cares that they can easily deploy the frameworks they want to use.
  4. Large-Environment Sysadmin
    • Interested in deploying self-service PaaS to lighten operational load
    • Will work closely with development team on deployment and development.
  5. Data Scientist
    • Works with large data sets
    • Interested in big data tools and dedicated scripting languages (R, Octave, etc.)
  6. HPC Scientist
    • Moving from traditional batch and grid technology to the cloud.
    • Interested in hybrid cloud (infrastructure overflow) for some massive computation campaigns.

Primary Use Cases

Web Application Deployment in a Public Cloud

Modern web applications are deployed as a collection of interconnected services, including parts like web servers, application servers, databases, and caching layers. Fault tolerance is handled at the overall orchestration level rather than by individual instances. Fedora Cloud can be the base of each of these parts, providing recent libraries, server software, and language toolchains. Each system will be managed using the public cloud's own management tools plus a configuration management system like Puppet, Chef, Salt, or Ansible. Code may be developed against different language stacks; if these are readily available to users, that's a win. In the real world, libraries and other code we don't include may also be required and we should make that as painless as possible. Fedora Cloud will provide access to the following stacks (not exhaustive):

  • Ruby: Rails, Sinatra
  • Python 2 & 3: Django, Pyramid, Flask
  • PHP 5.x: Symfony2, ZendFramework 2.x
  • Node.js
  • Java
  • Perl 5.x
  • Go

Web Application Deployment in Private Cloud

As above, but in a locally-deployed and managed private cloud system.

Web Application Deployment in Hybrid Cloud

As above, but rather than a single cloud provider, the application seamlessly takes advantage of resources in both a private and public cloud.

Hybrid Cloud as Staging Area

Application development and testing happens in-house on a private cloud, while production goes to a public cloud. Possible scenarios: Using an open source IaaS stack internally and externally (e.g. OpenStack internally, Rackspace externally; Eucalyptus internally, AWS externally; CloudStack internally, any of the CloudStack providers or AWS externally; or using a library like jClouds to smooth out the bumps.) This may include identical software stacks which run in different clouds, or cloud-specific tailored images onto which an application is deployed, or even very different environments under which the same Docker container is run.

Simple Deployment of Code from Development to Production

As a developer, I want to ensure that my code is easily deployable from my development environment to a QA environment and finally a production environment, without encountering compatibility issues or surprises from differences in the environment. A solution where an identical customized image or container (such as Docker) is used in all three environments is ideal.

Target IaaS environments

The Fedora Cloud product can be used as a guest/VM/image under many IaaS services and providers. For projects that are not currently packaged within Fedora/EPEL, we may need to locate a kind person to ensure testing. These include: Open source IaaS systems:

  • OpenStack
  • Eucalyptus
  • Apache Cloudstack
  • OpenNebula
  • oVirt

Public clouds:

  • Amazon EC2
  • Google Compute Engine
  • HP Cloud
  • Rackspace
  • Digital Ocean
  • Linode

Features

Feature: Ready to run in Public and Private Clouds

The Fedora cloud image is ready to go out of the box in the public and private cloud environments we target. It includes cloud-init, the defacto standard for boot-time configuration for cloud instances, and heat_cfgtools package, which is used for handling guest initialization with OpenStack Heat.

Feature: Ready Access to Fedora Collection of Packages

  • Access to the standard Fedora package repositories
  • Standard language stacks (like Python, Ruby, NodeJS, PHP)
  • Modernized language and application-specific stacks, perhaps in the form of SCLs, for running userspace applications

Feature: Updated Images

Currently, the Fedora Cloud images are only produced twice-yearly as part of the general Fedora release. Because Fedora updates constantly, this means that after several months, it is often the case that a newly-launched image has pending updates which effectively replace the entire content. This is slow and wastes bandwidth, and in many cases means that users simply never update. Instead, we will produce images with security updates and critical bugfixes rolled in. Our initial target is monthly releases, with special updates produced for critical security flaws.

Feature: Cloud -> Server

The Fedora Server product targets more traditional server roles, where systems have a more unique identity. The Fedora Cloud image supports this by providing a path to go from our base to a Fedora Server role, in effect taming one of your cloud computing "cattle" and making it into a "pet" traditional server — but running in a cloud environment.

Feature: Atomic Host

The Atomic Host provides an easy means to keep your container infrastructure up to date. Atomic provides git-like updates and roll backs of your host packages so your containers can keep running, secure and stable. The Atomic spin will provide updates every two weeks, ensuring that your host receives the latest system updates.

Feature: Dockerfiles

Fedora Dockerfiles provides a curated list of Dockerfiles designed to work with Fedora - making both development and deployment of your Docker infrastructure easier. By maintaining a list of supported Dockerfiles, we're able to better saturate the container ecosystem, making it easier for developers to use Fedora in their stack from end to end.

Future Features

The following areas are unlikely to be completely addressed in the upcoming iteration of Fedora Cloud, but we intend to work on them in the future.

Image Generation Tools

The official Fedora Cloud images will be produced in the Fedora build system (Koji) using kickstart scripts, Anaconda, Oz, and ImageFactory. We also want to provide tools and documentation so users can easily respin their own images. This will likely be using ImageFactory as well. We may provide a web-based service for user-created images.

Image or Image Template Library

In addition to the basic tooling for image creation, we want to enable sharing of user-created images and their recipies. This may apply to Docker or other container images in addition to images targetted at full virtualization.

Higher-Level Cloud Tooling

Initial efforts focus on individual cloud images. In the future, we want to address higher level concerns as well. These areas include:

  1. Orchestration
  2. Performance / Scalability / Failover
  3. Logging / Auditing
  4. Monitoring / Notification
  5. Database requirements

Requirements

Requirement #1: Reducing Image Footprint

Making the Fedora Cloud image fit in a smaller space. Storage is a significant cost element on cloud platforms.

Feature Addressed

Ready to run in Public and Private Clouds.

Priority

Nice to Have. The current cloud image is reasonably sized when compared to our competitors. However, smaller footprint has several advantages. Fewer packages means fewer updates and a smaller target for security problems. It makes it faster to download and deploy images across networks where the image storage and compute nodes run separately. And, reducing things our users don't really need gives more room to include things that they do.

Specific Areas

This requirement has several areas where effort will yield meaningful results. Each has its own level of effort, stakeholders, and dependencies.

Cloud-Focused Base Kernel

The cloud-focused base kernel will be a minimal kernel without many of the drivers traditionally necessary on hardware, like most PCI devices. The goal of having a custom kernel package is ultimately to reduce the size of the images which will reduce storage costs, make the images easier to move across networks between compute and image storage nodes.

Note that the kernel binaries will be shared across all of Fedora; the difference will be that not all modules will be packaged and included by default in the cloud image. This also means that it will be simple to add the other drivers if they are needed for special cases.

Effort is moderate; it includes defining the exact requirements of the smaller kernel, and creation and maintenance of that separation by the Fedora kernel team.

Internationalization / Localization

Fedora Cloud base images should only include a single default locale since it is meant to be used as an deployment target to save space. Other locales should be available through repositories. The idea is to rely on langpacks rather than the current RPM kludges which leave out internationalization, because that approach does not allow one to easily add language support later. Effort required is fairly high, because the packaging tools do not currently have such support. It will also likely have some impact on all Fedora packagers, depending on the implementation, and may also involve changes to the packaging guidelines.

Included Documentation

Documentation has the same issue as internationalization: it is often not needed on each image but takes significant space, and cannot be easily added back if RPM is configured to exclude it initially. Effort is similar to that for internationalization, although significant impact could be made by breaking out documentation from key packages included in the base image.

Refactoring Cloud-Init

Cloud Init was initially designed for a different distribution and is only loosely tailored for our needs. As it stands, it pulls in a rather large set of packages not used for other things. It is also written in Python, itself a large subsystem which it would eventually be nice to leave out of the base. Effort is moderate, with some low-hanging fruit which may be addressed easily.

Requirement #2: Improved Docker Integration

Docker is a popular tool for launching and managing application containers. This runs on bare metal, in virtualization on developers' machines, or in cloud guests. Right now, Docker can be added to the cloud base image but isn't included by default. Docker itself is small, but infrastructure to support it includes Libvirt, which is not small. In order to make it easier to get started and to speed up use in production, we will produce a Docker-specific image. Additionally, we want to create a library of Dockerfiles and possibly our own registry of images.

Feature Addressed

Docker support.

Priority

Should. This feature is an important change to support a variety of different teams working on PaaS (like OpenShift) and end-user delivery issues. That being said, since there is very little usage of Docker in those tools right now and the package is available in the repositories already, it won't be terribly detrimental to not have the image complete.

Effort required

High. We can produce the basic image with minor work, but SELinux is a key feature which will make us stand out from other systems on which Docker can be run. Additionally, producing official Docker container images depends on changes to the build system.

Stakeholders / Owners / Major Dependencies

The Cloud SIG and our users are the major stakeholders. The SELinux team is instrumental in delivering the functionality we would like to see. Fedora Infrastructure and Release Engineering are involved in updating the build system. And the official images will be produced through Release Engineering. Major dependencies include:

  • Better SElinux MCS separation incorporated in the policies shipped in Fedora.
  • Creation of an agreed upon set of Dockerfiles for different tasks.
  • ImageFactory support for creating Docker images, and that support integrated into Koji.

Requirement #3: Automatic Production and Upload of Images

We will develop Fedora Cloud images using text-based descriptive files and use an automated toolchain that generates images for our targets and upload them where appropriate. Currently, this is done by hand. Automation is necessary to scale to multiple supported images and a growing set of targets. Fedora images must pass automated QA before publication to ensure a minimum level of quality. In order to release updated images with confidence, automated testing is required, because we cannot expect the level of human attention which can be brought to a twice-yearly release.

Feature Addressed

Updated Images.

Priority

Must. We simply need this to be useful to our users, and therefore to be competitive.

Effort required

High. Work is currently ongoing with Release Engineering and QA to facilitate this process.

Stakeholders / Owners / Major Dependencies

The Fedora Cloud Working Group will work with Release Engineering in implementing and supporting the needed functionality.

Requirement #4: Software Stacks for Various Languages

People developing and deploying code on top of Fedora need access to the relevant language stacks. Fedora already provides the latest versions of many of these, but often real-world version dependencies do not match the Fedora cycle. We need some way of offering multiple versions of languages and software stacks. Additionally, it is ideal if support for a given version lasts longer than the 13-month Fedora lifecycle, so that moving from one release of the base to the next can be less painful. The Cloud SIG plan to support the major languages:

  • Ruby
  • Python 2 & 3
  • PHP 5.x
  • Java
  • Node.js
  • Perl 5.x
  • Go

We will provide infrastructure and support to volunteers who want to complete the supported stacks or bring new ones.

Feature Addressed

Ready Access to Fedora Collection of Packages.

Priority

Must. This addresses a key need of userbase.

Effort required

High. Access to existing Fedora packages is simple (we simply won't break what currently works), but Fedora does not currently have a good infrastructure for multiple versions of stacks or for decoupled lifecycles.

Stakeholders / Owners / Major Dependencies

Software in this area will be the responsibility of the Environments and Stacks working group.

Requirement #5: Integration with Fedora Server Roles

We simply want to make sure that the tools to convert a Fedora Cloud base image to a Fedora Server role are available and function.

Feature Addressed

Cloud -> Server

Priority

Should. This helps tie the Fedora products together and makes the blurry lines between the Cloud and Server product more understandable.

Effort required

Medium. Effort is primarily in coordination and testing.

Stakeholders / Owners / Major Dependencies

This effort will be done in collaboration with the Fedora Server Working Group.

Requirement #6: Tools for User Image Creation and Image / Template Library

We do not have a standardized process for end-user image creation, although we have several tools which can be used. We plan to develop this process and document it. This may also include an image library or a library of image creation templates, or both.

Features Addressed

Image Generation Tools, Image or Image Template Library

Priority

Nice to have. This is not required for our initial launch.

Effort required

High.

Stakeholders / Owners / Major Dependencies

Image creation tools (Oz/ImageFactory, etc.), Fedora Websites, Fedora Documentation, Cloud SIG users in general.

Requirement #7: Updated Web Site for Obtaining Cloud Images

As we provides images for various use cases (ie: Big Data, PaaS hosting, scale-out apps, etc.) and various target platforms, we need an updated website. Fedora provides its build toolchain and encourage the community to use it to build new products. The updated website should also reflect that by allowing folks to share and review the work done by their peers.

Feature(s) Addressed

Easy access to Fedora Cloud products and encouraging sharing in the community.

Priority

Must. A simple access to the cloud products is also a key argument to their success. Should. Providing a sharing platform to the community (as in not supported by the Cloud SIG) and peer-review based system will allow to service better our users and foster innovation.

Effort required

High. We need to define precisely what we want to show, and maintain the website.

Stakeholders / Owners / Major Dependencies

The Cloud SIG will work closely with Fedora Websites/Infrastructure and possibly Fedora Engineering if we need to develop specific apps and/or integrate with the existing Fedora infrastructure (like fedmsg).

Documentation

We aim to provide a comprehensive documentation covering users to developers. Should. Provide narrative documentation to make it easier to use cloud images and use them as foundations to new products. NTH. On a volunteer basis, we will maintain a knowledge base.

Fedora Project Documentation

The Cloud SIG will cooperate with the Documentation Project to provide documentation to users, and developers of Fedora Cloud Products.

Open Source Projects documentation

Ensuring that Fedora is well-represented, up-to-date in other open source project documentation...

About this Document

This PRD (Product Requirements Document) is an evolving document, created by the Fedora Cloud SIG Working Group as part of the process for designing the Fedora Cloud product. The framework for the PRD itself is currently in a draft state. This document will evolve over time as the purpose of the SIG continues to be determined as the working group process gets under way and initial products start to get produced.

Authors

Contributors to this document include:

Reviewers & Contributors

The following people have contributed to the development of this document, through feedback on IRC, mailing lists, and other points of contact.

Credits

Some of the Cloud User Roles are based on “Description and Application of Core Cloud User Roles” ACM CHIMIT 2011, December 4 2011. Some aspects of Fedora Cloud personas are based on OpenStack personas (licensed under CC-By 3.0)

Community Information

The Fedora Cloud SIG is one of many teams within the Fedora Project. The Cloud SIG mailing list is located here. Minutes and logs from IRC meetings related to the development of this document should be listed here as the document evolves.

Approval History

Over time, it is expected that this document will undergo various rounds of review, approval, and editing; in the future, it may be rewritten for different releases of Fedora. While one can review the history of a wiki document (by clicking the "history" tab), it is useful to provide explicit indicators of any major format changes, approvals, or indications of it being in a “final” state, in a list that can allow someone to quickly see that all of the prescribed layers of approval have occured.

  • January 22, 2014: Approval by FESCo
  • January 8, 2014: Collaborative hackfest day.
  • October 28, 2013: Initial Draft of template.
  • FutureDate: Approval by SomeGroup; link to any pertinent mail announcement and/or meeting minutes

Document Conventions

Definitions and Acronyms

  • AWS: Amazon Web Services
  • Amazon EC2: Amazon Elastic Compute Cloud, a popular public IaaS, part of AWS
  • IaaS: Infrastructure as a Service
  • PaaS: Platform as a Service
  • SaaS: Software as a Service
  • PRD: Product Requirements Document
  • EPEL: Extra Packages for Enterprise Linux
  • CI: Continuous Integration
  • CD: Continuous Delivery or Continuous Deployment
  • SCL: Software Collections
    teams in charge of some aspects of Fedora Project
  • NTH: Nice-to-have
  • BZ: Bugzilla
  • GUI: Graphical User Interface
  • CLI: Command Line Interface
  • API: Application Programming Interface
  • RDBMS: Relational DataBase Management System
  • FESCo: Fedora Engineering Steering Committee