Command Line Website Deployment

CLWD

Overview

  • Goal: Introduce some basic, useful command line tools in a web context
  • Motivation (somewhat): Yahoo's YSlow and Google's Pagespeed tools
  • Motivation (ulterior): Help me improve my site deploy script

Introduction

Welcome

The web is text

  • The web is just a bunch of text files maybe linking to images or video of maybe naked people
  • Linux and other *nix systems are good at manipulating text
I know Unix

Unix is text

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
Ken Thomson, quoted in A Quarter Century of Unix

The web is boring

Much of the work in building a website is performing various, repetitive actions on many files

  • verifying correctness of HTML
  • compressing your images
  • minifying your text files
  • finding and replacing text in files

Outline!

  1. Registering your domain name
  2. Verifying your HTML
  3. Measuring page weight
  4. Compressing your images
  5. Checking for broken links
  6. Uploading your site

Registering your domain name

Just for fun, this is how you could theoretically register a domain here in South Africa. 


curl -O http://co.za/coza_reg.txt
	      
$EDITOR coza_reg.txt
	      
cat coza_reg.txt | mail -s "plz register jozilug.co.za" coza-admin@co.za

Check ur HTML

H.T.M.L.
  • HTML Tidy is an HTML syntax checker

# apt-get install tidy
	      
find $SITELOC -name "*.html" -exec echo {} \; -exec tidy -quiet -errors {} \;

Measure page weight

  • Not completely accurate, but a good first approximation of page weight

wget -o ~/wget-output.txt --page-requisites http://127.0.0.1 && 
grep ^Downloaded ~/wget-output.txt

# Downloaded: 12 files, 188K in 0.001s (247 MB/s)

Compress those images!

  • The default output graphic programs generate is not suitable for use on websites
  • Commandline tools like jpegoptim and optipng can perform lossless compression and remove metadata

find $SITELOC -name *.jpg | xargs -P 4 -n 1 jpegoptim --strip-all 

find $SITELOC -name *.png | xargs -P 4 -n 1 optipng -quiet -o7 -strip all

Instead of find's -exec switch, I pipe to xargs; - P 4 runs 4 commands in parallel!

Check for broken links!


wget -o ~/wget_errors.txt --spider --execute="robots=off" --recursive --page-requisites http://127.0.0.1:4000
	      
EXIT_CODE=$?
if [ $EXIT_CODE -gt 0 ]; then
echo "ERROR: Found broken link(s)"
grep --before-context=2 '404' ~/wget_errors.txt
exit 1
fi

rm ~/wget_errors.txt

Upload your site!


rsync --quiet --copy-dirlinks --delete --archive --verbose --update ~/public_html $REMOTE

#lftp -u user,password -e "mirror --delete --verbose --only-newer --continue --reverse $LOCAL $REMOTE; bye" www.yourdomainyo.co.za

Commands

  • grep
  • wget
  • curl
  • find
  • rsync
  • jpegoptim
  • optipng

Links

FIN

Daniel Fowler

Liz Lemon