Welcome to my tutorials on how to leverage Docker to deploy your application! This is meant as much to help me learn, so it doubles as a “what worked for me” series.
In the previous lesson, we covered how to write a docker-compose.yml file to manage the desired configuration of multiple conatiners and how to use the docker-compose command to build, start, and stop those containers.
This lesson, we will examine how to leverage Docker for development and deployment, including making a full image that could be used for production and how to upgrade and revert your production container. We will also cover how to clone a Docker volume.
For application developers, one of the most challenging problems is ensuring that what you develop and test works the same way in production where your customers are using it. I’ve seen it plenty of times where you have a “testing environment” that has all the components of your production environment, but the software versions are out of date, the data in production is not what’s in test, and even “test stopped working”, usually because of a lack of ongoing maintenance in that environment. Normal means of combating this is to make a copy of the production environment that can be used to develop on. However, when you’re looking at cloning a database, 6 application servers and their supporting processes, things can get messy.
However, using Docker can change this paradigm in two ways. You can at anytime create a new image using the docker container commit command, making a full image of the container that can then be run on any other Docker host. Also, Docker volumes can be cloned by mounting the named container volume into a different container and using the tar command to make a full copy (per https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes). Second, if efforts are made to segregate data from the application being worked on, it becomes possible to update the application using the volume attachment method we used in making the MySQL database from the last two lessons.
So how do we achieve development zen? Divide and conquer, baby.
First, you want to ensure that all application data is stored in its own volume. The problem with docker container commit is the data in your application gets baked into the image. While this might sound appealing because you’ll always have a point to go back to, when you want to test your app against fresh data, you do not have a clean way to inject it into your app. You end up running docker container commit again and then either get and copy the fresh data out into your development container, or copy your application changes into the new clone. Both options can be very ugly on a good day.
Second, you want to isolate your application (the code you added to the image to make it an application) from the base image you are developing with. This does not mean another volume, necessarily. Rather, you want to be in a position that you can identify specific directories (not files) that you need to populate for your application to work.
To demonstrate, let’s build a PHP application that allows you to upload and download files. Note that using an app like this in real life is HIGHLY insecure, but the development concepts here can be applied easily. We will use the php:apache image, identify the /var/www/html direcotry as the place that your application will live, and create the directory /var/www/files as where the application data will live, backed by a mounted volume.
So let’s start by creating two of the three volumes:
docker volume create --name files_dev docker volume create --name files_test
Defining your environments:
files_dev is the development environment used while building your application
files_test is the testing environment and will be a copy of production when we are testing how an upgrade will work.
files_prod is the production environment and is what your users see today (and hopefully see tomorrow…). This will get created when we make a production container.
Now let’s build your development environment. Start by creating a project directory, a php directory in there, and an empty index.php file in the php directory. We don’t need a custom image, we’ll just use php:apache. But attach your data volume at /var/www/files, and attach the php directory to /var/www/html. Unfortunately, the command is different if you ware on Windows vs Linux. Run these commands from inside your project directory.
Linux:
docker run -d -v files_dev:/var/www/files -v ./php:/var/www/html -p 8080:80 --name php_dev php:apache
Windows:
docker run -d -v files_dev:/var/www/files -v "<FULL_PATH_TO>\php:/var/www/html" -p 8080:80 --name php_dev php:apache
Connect to the container shell with the following command:
docker exec -it php_dev /bin/bash
You can now run ls /var/www/html and see the empty PHP file there. However, if you run ls -l /var/www, you will notice that the owner of your html and files directories is none other than root. This will break your webserver, so quickly run the following to fix ownership:
chown www-data:www-data /var/www/*
Now exit the container shell and save the following in your index.php file:
<?php
$files_path = '/var/html/files';
$add_path = isset($_GET['dir'])?$_GET['dir']:'/';
?>
<html>
<head>
<title>File Share</title>
<style>
table, th, td { border: 2px solid black }
span { border: 2px solid black; padding:4px 2px}
</style>
</head>
<body>
<h1>Location: <?=$add_path?></h1>
<form method="POST">
<span><input type="file" name="file" /><input type="submit" name="action" value="Add File" /></span>
</form>
<table>
<thead><tr><th>Name</th><th>Size</th><th>Date</th></tr></thead>
</table>
</body>
</html>If you go to http://localhost:8080, you should see your bare-bones webpage. There is no PHP code that has any real effect, so you should see an empty table and a file upload bar.
Now for the real reason we mounted the local directory: you can work on your app to your heart’s content and don’t need to create a new image in order to test code changes. There’s nothing holding you back, and you’re developing on the same software you will deploy to. If you discover that you need more software in your core image, you add to it like we did in previous lessons, build, and redeploy the upgraded image with the same mounts, and then development continues.
So let’s finish version 1 of the app (entire file):
<?php
$files_path = '/var/www/files';
$add_path = isset($_GET['path'])?$_GET['path']:'';
$action = isset($_POST['action'])?$_POST['action']:'';
$path = $files_path . $add_path;
$realpath = realpath($path);
$status = '';
if ($realpath != $path ) {
$add_path = '';
$path = $files_path . $add_path;
$action='';
}
switch ($action) {
case 'Add File':
$struct = $_FILES['file'];
if ($struct['error'] == UPLOAD_ERR_OK) {
$tmp_name = $struct['tmp_name'];
// basename() may prevent filesystem traversal attacks;
// further validation/sanitation of the filename may be appropriate
$name = basename($struct['name']);
if (move_uploaded_file($tmp_name, "$path/$name")) {
$status = "<span class='ok'>File $name uploaded.</span>";
} else {
$status = "<span class='bad'>File $name not uploaded.</span>";
}
}
break;
case 'Remove File':
if (!isset($_POST['tgt_path']) || !$_POST['tgt_path']) { break; }
$tgt_path = $_POST['tgt_path'];
$name = basename($tgt_path);
$full_path = realpath($files_path . $tgt_path);
if (!$full_path) { break; }
if (unlink($full_path)) {
$status = "<span class='ok'>File $name deleted.</span>";
}
else {
$status = "<span class='bad'>File $name not deleted.</span>";
}
break;
default:
if ($realpath && is_file($realpath) && !is_dir($realpath)) {
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename="' . basename($realpath) .'"');
readfile($realpath);
exit;
}
}
?>
<html>
<head>
<title>File Share</title>
<style>
table, tr { border: 1px solid black }
span { border: 2px solid black; padding:4px 2px}
form { margin: 10px }
.r { text-align:right }
.g { background-color: lightgray }
.ok { background-color: limegreen; color: green }
.bad { background-color: coral; color: red }
</style>
</head>
<body>
<?=$status?>
<h1>Location: <?=$add_path?></h1>
<form enctype="multipart/form-data" method="POST">
<span><input type="file" name="file" /><input type="submit" name="action" value="Add File" /></span>
</form>
<table>
<thead><tr><th>Name</th><th>Size</th><th>Date</th><th>Action</th></tr></thead><tbody>
<?php
$count = 0;
foreach (scandir($path) as $filename) {
$bg = (++$count % 2) ? 'g' : '';
if ($filename == '.' || (!$add_path && $filename == '..')) { continue; }
$filepath = realpath($path . '/' . $filename);
$newpath = str_replace($files_path, '', $filepath);
$size = filesize($filepath);
$mod = strftime('%Y-%m-%d %H:%M:%S',filemtime($filepath));
$value = 'Remove File';
echo "<tr class='$bg'><td><a target='_blank' href='?path=$newpath'>$filename</a></td><td class='r'>$size</td><td>$mod</td><td><form method='POST'><input type='hidden' name='tgt_path' value='$newpath'><input type='submit' name='action' value='$value' /></form></td></tr>";
}
?>
</tbody></table>
</body>
</html>This working PHP program lists and allows for the upload of files. It will also delete them. Uploading a file with the same name replaces the file. So for now, upload some files, delete one or two, and make sure it works.
Now we’re ready to deploy to production. Create a Dockerfile and save the following into it:
FROM php:apache
COPY ./php/* /var/www/html
RUN mkdir /var/www/files && \
chown -R www-data:www-data /var/www && \
chmod a-w,o-rwx /var/www/html && \
chmod go-rwx /var/www/filesThis grabs your base image, copies your PHP application into the correct directory in the image, pre-creates your files directory, and sets file ownership permissions on all folders that need them. It also specifically removes write access for all accounts and any access for users not in the www-data group from your webserver directory. Let’s build the image now:
docker build -t my_php_app:1.0 .
This builds your image using the Dockerfile as my_php_app and tags it as version 1.0. Since we’re also deploying to production, let’s also create a docker-compose.yml file:
version: "3.8"
volumes:
files_prod: {}
services:
php-container:
image: my_php_app:prod
restart: always
ports:
- "80:80"
volumes:
- "files_prod:/var/www/files"If you are paying attention, the tag on your image should have caught your eye. Your docker-compose.yml file is looking for a my_php_app:prod image, but we just made a my_php_app:1.0 image. Why do this? The answer has to do with recovering from a failed upgrade. We are going to use the “prod” tag to identify the specific image we want to run in prod, even if it is not the latest one. Your docker-compose.yml file is the authoritative configuration for how your production environment should work, and it has been my experience that configuration files like this should not be touched as much as possible. Also, images can have multiple tags, but a specific tag can only be on one image. So if your new prod image does not work as you want, you can set the prod tag back on the old image and rerun “docker-compose up” to put things back the way they were before. So let’s add the prod tag to your image:
docker tag my_php_app:1.0 my_php_app:prod
Now lets build and deploy your production server:
docker-compose up -d
A look at http://localhost should give you your app in all its file-storing spendor. Watch out Google! Let’s add some files to this container, but make sure that they are not the same files that you uploaded with your dev instance. Also, your production volume was created. Let’s see what’s out there:
docker volume ls
If you’ve noticed, there are already a few volumes here. Most of them are from prior lessons. However, the one we are looking for ends with “_files_prod”. The name of the volume starts with the name of the project folder that your docker-compose.yml file was located in, without spaces and all lowercase. We will need the full name of that volume in a moment.
Google sent us a message: Drive is way superior because users can make folders to organize their files. Guess we need to upgrade your app!
Go back to your index.php file and update it to the following:
<?php
$files_path = '/var/www/files';
$add_path = isset($_GET['path'])?$_GET['path']:'';
$action = isset($_POST['action'])?$_POST['action']:'';
$path = $files_path . $add_path;
$realpath = realpath($path);
$status = '';
if ($realpath != $path ) {
$add_path = '';
$path = $files_path . $add_path;
$action='';
}
switch ($action) {
case 'Add File':
$struct = $_FILES['file'];
if ($struct['error'] == UPLOAD_ERR_OK) {
$tmp_name = $struct['tmp_name'];
// basename() may prevent filesystem traversal attacks;
// further validation/sanitation of the filename may be appropriate
$name = basename($struct['name']);
if (move_uploaded_file($tmp_name, "$path/$name")) {
$status = "<span class='ok'>File $name uploaded.</span>";
} else {
$status = "<span class='bad'>File $name not uploaded.</span>";
}
}
break;
case 'New Dir':
if (!isset($_POST['dir']) || !$_POST['dir']) { break; }
$dir = $_POST['dir'];
$name = basename($dir);
$full_path = realpath($files_path);
if ($dir != $name || !$full_path) { break; }
mkdir ($full_path . '/' . $name);
break;
case 'Remove Dir':
if (!isset($_POST['tgt_path']) || !$_POST['tgt_path']) { break; }
$tgt_path = $_POST['tgt_path'];
$name = basename($tgt_path);
$full_path = realpath($files_path . $tgt_path);
if (!$full_path) { break; }
if (count(scandir($full_path)) != 2) {
$status = "<span class='bad'>Directory $name not deleted, is not empty.</span>";
break;
}
if ($full_path && rmdir($full_path)) {
$status = "<span class='ok'>Directory $name deleted.</span>";
}
else {
$status = "<span class='bad'>Directory $name not deleted.</span>";
}
break;
case 'Remove File':
if (!isset($_POST['tgt_path']) || !$_POST['tgt_path']) { break; }
$tgt_path = $_POST['tgt_path'];
$name = basename($tgt_path);
$full_path = realpath($files_path . $tgt_path);
if (!$full_path) { break; }
if (unlink($full_path)) {
$status = "<span class='ok'>File $name deleted.</span>";
}
else {
$status = "<span class='bad'>File $name not deleted.</span>";
}
break;
default:
if ($realpath && is_file($realpath) && !is_dir($realpath)) {
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename="' . basename($realpath) .'"');
readfile($realpath);
exit;
}
}
?>
<html>
<head>
<title>File Share</title>
<style>
table, tr { border: 1px solid black }
span { border: 2px solid black; padding:4px 2px}
form { margin: 10px }
.r { text-align:right }
.g { background-color: lightgray }
.ok { background-color: limegreen; color: green }
.bad { background-color: coral; color: red }
</style>
</head>
<body>
<?=$status?>
<h1>Location: <?=$add_path?></h1>
<form enctype="multipart/form-data" method="POST">
<span><input type="file" name="file" /><input type="submit" name="action" value="Add File" /></span>
</form>
<form method="POST">
<span><label for="dir">New directory name</label><input id="dir" name="dir" size="20" max-size="20" /><input type="submit" name="action" value="New Dir" /></span>
</form>
<table>
<thead><tr><th>Name</th><th>Size</th><th>Date</th><th>Action</th></tr></thead><tbody>
<?php
$count = 0;
foreach (scandir($path) as $filename) {
$bg = (++$count % 2) ? 'g' : '';
if ($filename == '.' || (!$add_path && $filename == '..')) { continue; }
$filepath = realpath($path . '/' . $filename);
$newpath = str_replace($files_path, '', $filepath);
$is_dir = is_dir($filepath);
if ($is_dir){
$size = '';
$mod = '';
$button = ($filename != '..') ? "<input type='submit' name='action' value='Remove Dir' />" : '';
}
else {
$size = filesize($filepath);
$mod = strftime('%Y-%m-%d %H:%M:%S',filemtime($filepath));
$button = "<input type='submit' name='action' value='Remove File' />";
}
echo "<tr class='$bg'><td><a target='_blank' href='?path=$newpath'>$filename</a></td><td class='r'>$size</td><td>$mod</td><td><form method='POST'><input type='hidden' name='tgt_path' value='$newpath'>$button</form></td></tr>";
}
?>
</tbody>
</table>
</body>
</html>Now let’s turn this into an image candidate and test it out. This will take a few steps. First, let’s make the image:
docker build --name my_php_app:2.0 .
Now things get tricky. We are going to create a new container that clones the files_prod data into files_test. What’s tricky is that the volume for files_prod is in use, so while we are copying the data, the data can change, or worse, the application can hang. This is a challenge for any application. The safest way is to stop the main application so no data is in use while copying. The way your application is configured may also allow for the use of storage snapshots or backups to recover data in a usable state. However, in this Docker environment, we can safely copy your files over because unless you are editing a file while the copy is happening, there will be no problems. First get the name of volume that has your production file data:
docker volume ls
Now, run the following to clone your data:
docker run --rm -v ":/mnt/prod" -v "files_test:/mnt/test" ubuntu /bin/bash -c "rm -fr /mnt/test/* ; tar -cf - -C /mnt/prod . | tar -xvf - -C /mnt/test"
The Docker website offers the –volumes-from option to grab the volumes. This is a quicker way to mount the volumes on your container, but this will mount all volumes from the specified container in the same locations on your container. I prefer having control of where the volumes are mounted. Also, the Docker website commands will write a tarball on the Docker host. This is normally preferable to a straight volume-to-volume copy, as you can use the tarball file to “start over” if needed. This is doubly helpful if you have to stop your prod environment in order to get your production data. However, since this is a tutorial and Windows users would need to jump through additional hoops, I opted for a direct copy.
Last, create a testing container:
docker run -d -v "files_test:/var/www/files" -p "4444:80" --name test_my_php_app my_php_app:2.0
Notice that I used port 4444 to test with. I tried to make the port number as different from prod and the development container as possible to ensure that I know which container I’m working with. Now visit http://localhost:4444 and see if it works. You should see what looks like a copy of your production environment, but with the ability to create a directory. Make a directory or two. Excellent.
Now let’s upgrade your production website. Apply the prod tag to your newest app image, and rerun docker-compose:
docker tag my_php_app:2.0 my_php_app:prod docker-compose up -d
You should get a notice about rebuilding the container. Remember that while the container is being rebuilt, the app is down. Once it’s done, go to http://localhost, and test your upgraded application.
If you want to simulate a rollback, tag the old version and rerun docker-compose:
docker tag my_php_app:1.0 my_php_app:prod docker-compose up -d
Now if you created a directory with version 2.0 in production then went back, you will notice that the app does not work exactly as planned. That is because the v1.0 app did not do anything with directories, but they are now in your file data. This brings up an important point: you are still responsible for fixing any new data that conflicts with your old app. In this case, we could connect to the container shell and remove any directories that were created. However, sometimes this is a lot harder, like in the case of a failed database upgrade. You will still need to plan for what to do if that happens. Docker only makes it easier to revert your application code in this scenario.
To summarize, we will saw how to use Docker for development and deployment, including how to make a full image that could be used for production based on your development environment and how to upgrade and revert a production container. We will also learned a couple different ways to clone a Docker volume, including some of the challenges of cloning a live volume.
For the next lesson, we will create a private Docker Registry. We will also use the OpenStack Swift storage driver to store the registry in a storage bucket.