matthew manning

Sun Jar Prototyping

matt.manning@gmail.com (Matt Manning) — Sun, 31 Mar 2013 00:00:00 +0000

Sun jars are solar-powered LED night lights. You can find them for sale all over the web and even at some brick-and-mortar retail stores. Making one looked like a fun project, so I started Googling and checking the normal places like Instructables for project ideas. Surprisingly, all the DIY sun jar guides I could find centered around hacking up a pre-built solar garden light and stuffing the guts into a jar.

That didn’t sound like any fun to me. This had to be a really simple circuit. Why not build it from scratch and learn something along the way? I decided that would be my project.

Designing the circuit

My first step was reading about battery-charging circuits. I wanted this to be a simple project, and I knew I’d be working with a low current from a solar panel, so trickle-charging seemed like the best option. After Googling around for a bit, I settled on this circuit configuration:

Click on the circuit image above to visit the CircuitLab page and read a full explanation of how it works.

Building the circuit

The next step was to start spec’ing out the parts. Most LEDs have a forward voltage — also called a voltage drop — of around 2V, with some blues dropping as much as 3.8V. This meant I needed a battery of 3V or more. A normal AA or AAA cell is about 1.2V, and I had some rechargeables around, so I strung 3 AAAs in series and made myself a 3.6V battery pack.

To trickle charge a 3.6V battery I needed a solar panel that could produce a higher voltage than that and also would fit in the lid of a jar. After a lot of searching I found these inexpensive 5V solar “badges” from Adafruit.

Next, I built the circuit on a breadboard to test it out. Here’s a video of me demonstrating the light turning on when the solar panel gets dark:

I could tell right away that the battery would light the LED and voltage from the solar panel would control the transistor, but it took some time to test charging the battery. I observed with a multimeter that the battery voltage was higher after several hours in the sun, but I also left the breadboard in a sunny window for a couple of weeks as a real test, and the LED kept lighting up every night.

Fitting it in a jar

Now that I’d demonstrated that the circuit worked, the next step was to build a prototype. The 3AAA battery pack was too big to fit in a jar, so I went searching for a better battery. I found these cool 3.6V rechargeable lithium ion coin cells

at Sparkfun which ended up being perfect, so I ordered several of them along with some matching holders.

Here’s where ordering stuff online can bite you. I didn’t realize until they arrived that the holders are only the positive contact for the battery. They’re intended for use on a PCB where a solder pad on the board provides the negative contact. Bummer! So I had to wait a few more days for some breakout boards to arrive.

Once I received the breakout boards, I soldered one up and used it to replace the 3AAA battery pack in my window circuit. I let that run for several more weeks as a battery test.

Lithium ion batteries are really sensitive to overcharging, and you can easily ruin them that way. Most LI-charging circuits employ circuitry to cut off the charger once the battery has reached a certain float voltage. I didn’t think too much overcharging would happen in my circuit, so I decided to gamble $3 and find out what would happen. So far everything seems to work fine. The LED lights up brightly after sunny days and runs for several hours. On overcast days it’s somewhat dimmer and doesn’t last as long. I haven’t measured a battery voltage above the 3.6V rating, so I don’t think it’s overcharging. I guess it could be different in areas that get more intense direct sunlight for longer periods of time.

Making the first real prototype

With a working circuit and a smaller battery I next built the actual prototype sun jar. Most commercial sun jars put the solar panel underneath a hinged glass lid. Since I had weatherproof solar panels that could go on the outside of a jar, I decided to go for a slightly different look, and I ordered these 8-ounce Ball Jars from Amazon.

I also picked up some frosted glass spray.

I soldered a couple of wires to the solder pads on the back of the solar panel and drilled a hole in the jar lid. I routed the wires through the hole and then adhered the panel to the lid with some RTV Silicone sealant.

This stuff takes patience because you have to clamp it overnight, but it makes a good weatherproof seal, so the finished jar can be left outside.

I took the other circuit components off the breadboard and soldered them down to a little piece of protobard. Once the silicone had cured on the lid I wired the solar panel to the rest of the board.

Adhering the board to the underside of the jar lid proved pretty tricky. I couldn’t get silicone to hold, so I went to hot glue. It seemed to work at first, but then popped apart a couple of hours later. These jars are made for food, and it turns out the non-stick stuff they coat the underside of the lid with is pretty damn non-stick. The third time was the charm, though. Gorilla glue held really well.

The jars actually look pretty nice after being frosted with the spray. Here’s one complete jar and a couple I’m still working on.

Here’s how the first completed jar looks in the dark. Pretty good if I do say so myself.

Designing a custom PCB

Building the sun jar was really fun, and I want to make a lot more with different size jars and different color LEDs, so it would be really nice to get this circuit on a PCB. That way I could just drop in the parts and solder, rather than buying more battery breakout boards and cobbling everything together on protobard.

First I downloaded Eagle. It seemed to be the DIY community standard and Sparkfun has open source Eagle libraries that include the battery holder wanted to use.

I used these two really great tutorials, also by Sparkfun, to first create the circuit schematic and then the PCB layout.

Eagle: Schematics

Eagle: PCB Layout

If you’d like to play with this design, you can fork it on Github.

The tutorials were really great and comprehensive. I didn’t have to do much Googling outside of them to figure things out. The only real issues were that the Windows-oriented keyboard shortcuts didn’t work for me on OS X, so I had to dig through the menus for a couple of things, but it wasn’t too bad.

Fabricating the PCB

With the layouts done and Gerber files generated, I needed a place to get the PCB printed. Adafruit has some good reviews and recommendations for board houses, but even the cheapest of those charge a $35 setup fee and it looked like I might have to spend time on the phone to get the order done.

Just when I started to feel discouraged, Sparkfun came to the rescue again with their brilliant BatchPCB service. You simply upload a zip of your Gerber files and it does some quick checks to verify the files. Once they clear, your board shows up on the Marketplace view where you can order it. The set up fee is only $10, so I’ve ordered a prototype of my board for under $20 including shipping.

It’ll take a few weeks for the first board to get to me, but if it works, I’ll post an update here in case anyone wants to buy one. I think this might also make a fun “learn to solder” kit since it only has a few parts and they’re all of the through-hole type. If you’d be interested in buying a kit of these parts including a PCB, please leave a comment here or let me know at matt.manning@gmail.com.

Geospoofing with the Raspberry Pi

matt.manning@gmail.com (Matt Manning) — Mon, 11 Feb 2013 00:00:00 +0000

Many online services alter their content availability based on your geographic location, which they detect using your IP address. For example, some movie streaming services offer different movies to users outside of the United States, and live sports streaming services enforce regional blackouts to protect TV contracts. These types of practices violate the spirit of the free and open Internet.

The good news is that you can circumvent these unfair practices using a virtual private network (VPN). VPN technology is used heavily in the corporate world to provide secure remote access to a local networks. A side effect of VPN usage is that outbound requests appear to originate from the VPN’s IP address, rather than your own, and you can use this to your advantage to “geospoof” your physical location.

This works great if you’re spoofing on a general-purpose computing device like a laptop. You can just fire up your VPN client and go to town. But what if you want to connect a closed device, like an AppleTV, a game console, or a network-connected TV to a VPN? These devices won’t let you install arbitrary software like a VPN client or let you have too much control over networking.

What you need is a router on your local network that can forward traffic from these devices through a VPN. You could do this by installing open source firmware like TomatoUSB or ddwrt onto a hardware router, but that process can be tedious, and good hardware routers tend to be a bit pricey. I wanted something easy and inexpensive. Enter the Raspberry Pi.

The Raspberry Pi is a credit-card sized, low power computer on a single board that costs about $35. It was originally designed to be a cheap computer for education, but it’s seen a boom in the maker market, too. My friend, Mike, gave me one for Christmas, and it’s been really fun to tinker with.

The Raspberry Pi Foundation releases a special version of Debian Linux called Raspbian which is optimized for the Raspberry Pi. A $35 Linux computer is hard to beat, and it can be turned into a geospoofing VPN gateway on your local network pretty easily. Here’s a step-by-step guide.

Setting up the Raspberry Pi

First you need to start with a fresh install of Raspian. You can download it here and find a guide for writing the image to an SD card here.

The first time you boot the computer, you’ll be taken to the configuration tool, raspi-config. It looks something like this:

Since you’ll be using this computer as a networking device, you should change the default password for security reasons. Turning on the SSH server is also necessary unless you intend to keep the Pi hooked up to a monitor, keyboard, and mouse.

Next, give your Raspberry Pi a static IP address outside your router’s DHCP range. You’ll use this address as the router or gateway when you configure the other devices on your local network.

It’s a good idea to upgrade all the software on your Pi using the commands:

sudo apt-get update
sudo apt-get upgrade

The upgrade command will take quite a while to complete, so you might want to kick it off at night before you go to bed or before you go to work in the morning.

Installing and configuring OpenVPN

The next step is to install a VPN client. You can install OpenVPN with the following command:

sudo apt-get install openvpn

Now you’ll want to get access to a VPN server in a geographically beneficial area. I’ve had good results with PrivateTunnel. The service is inexpensive, and they have servers in San Jose, Chicago, London, Zurich, and Montreal. Once you choose a server location you can download its related configuration file. Save this file on the Raspberry Pi as /etc/openvpn/server.conf.

It’s necessary to make a couple of edits to this file. So open it with nano:

sudo nano /etc/openvpn/server.conf

Right before the first certificate add the line:

redirect-gateway

and further down in the file, change the line:

dev tun

to:

dev tun0

Configuring iptables

Now that you have VPN software set up, the next step is to turn the Pi into a router. You can do this using the Linux utility, iptables. Download this file and save it on your Raspberry Pi as /etc/iptables.test.rules. Load its rules using the command:

sudo iptables-restore < /etc/iptables.test.rules

Normally you would now test the functionality of your iptables rules. We know that these work, so go ahead and save them into your permanent config file with the command:

sudo iptables-save > /etc/iptables.up.rules

You want these rules to be loaded every time the Pi boots, so create a new file:

sudo nano /etc/network/if-pre-up.d/iptables

and add these lines to it:

#!/bin/bash
/sbin/iptables-restore < /etc/iptables.up.rules

Make it executable:

sudo chmod +x /etc/network/if-pre-up.d/iptables

Enabling IP forwarding

Finally, we need to enable IP forwarding by editing one more config file. Open up the file:

sudo nano /etc/sysctl.conf

and uncomment the line:

net.ipv4.ip_forward = 1

Now reboot your Raspberry Pi with the command:

sudo reboot

Testing

If everything was done correctly, you should now be able to use the Raspberry Pi as a VPN gateway. On a computer connected to the same local network as the Pi, visit WhatIsMyIPAddress.com. Note your current IP address and physical location on the map.

Now, change your computer’s router to the Raspberry Pi’s IP address. If you don’t know its address, you can find it by running the command:

ifconfig

on the Pi. I’m on a Mac and my Pi’s IP address is 10.0.1.202, so my network config looks like this:

If you’re on Windows I think “Router” is called “Default Gateway”. Now, reload WhatIsMyIPAddress, and you should see the IP address of your VPN and its location reflected on the map. Congratulations! You’re geospoofing!

Now all you have to do is set the router (or gateway) on your closed devices like AppleTV and Xbox 360 to the Raspberry Pi’s IP address and you can access content available only in your VPN’s location.

This is what my AppleTV config looked like after setting it to use the Pi as a router:

Using the gateway selectively

PrivateTunnel is reasonably priced, but if you’re streaming video, it’s easy to use up your data transfer allotment very quickly. Luckily, many services use only a single authentication request to determine your location. If you find this to be the case with your service, you can configure OpenVPN to send requests through the VPN only to certain IP addresses.

To do that, open up the config file:

sudo nano /etc/openvpn/server.conf

Comment out the line:

#redirect-gateway

and right below it add these lines:

# Ignore routes given to us by OpenVPN Server:
route-nopull
# Route authentication IP through VPN:
route 184.72.247.194 255.255.255.255

Now restart OpenVPN:

sudo /etc/init.d/openvpn restart

In this example, I’ve used the IP address of another IP address checking site, myipaddress.com. If you’ve set everything up correctly, you should be able to reload WhatIsMyIPAddress.com and see your actual IP address, but load myipaddress.com and see the VPN’s IP address. Once you get this working, you can use the route command to send requests to any IP addresses you want through the VPN!

Credits

I learned all the stuff to write this post from:

RPI as VPN gateway
iptables - Debian Wiki
Bugging smart guy Blake Gentry via email

Cloud Goodness With the DNSimple ALIAS Record

matt.manning@gmail.com (Matt Manning) — Wed, 18 Jan 2012 00:00:00 +0000

This is the third in a series of posts about setting up this Jekyll-backed blog on Heroku.

Cloud computing and second-level domains a.k.a. root domains (example.com) don’t play nicely together. This is because of RFC 1034 section 3.6.2: “If a CNAME RR is present at a node, no other data should be present; this ensures that the data for a canonical name and its aliases cannot be different.” Since root domains tend to have several types of records – “other data” – like NS or MX records, you can’t use a CNAME for them. That means you’re left with A records, which must specify IP addresses. Hence the rub with cloud computing, which tries to abstract away servers and IP addresses.

Heroku deals with root domains by providing 3 public IP addresses for routing web requests. For SSL, the extremely cloud-unfriendly IP-based SSL add-on is used, which provisions a dedicated server instance to serve your certificate. While these solutions work, they’re not really optimal. For example, if Heroku ever changes those IP addresses, as would be necessary in the case of a large-scale DDoS attack, or a datacenter failure, your root domain would no longer resolve. The IP-based SSL add-on has these same downsides, plus it’s a scaling bottleneck and costs $100/mo. Bummer.

DNSimple has introduced a really clever mechanism for dealing with this problem, the ALIAS record. An ALIAS record is set up much like a CNAME, but it resolves the name it points to and returns its IP addresses as if you had set up A records for them. This provides the flexibility of a CNAME without violating the RFC. You can see it working for this site using the host command:

[~/code/blog] host mwmanning.com
mwmanning.com has address 107.20.209.232
mwmanning.com has address 107.22.180.220
mwmanning.com has address 107.20.154.48
mwmanning.com has address 107.22.181.229
mwmanning.com has address 107.22.173.156
mwmanning.com has address 50.17.208.142
mwmanning.com has address 107.22.180.255
mwmanning.com has address 107.22.178.183

Besides the routing flexibility afforded by using an ALIAS record, you can take advantage of Heroku’s new HTTP stack, which is only available at [appname].herokuapp.com, and not at the 3 IPs mentioned previously. You can also use an ALIAS record to provide SSL at your root domain without using the IP-SSL add-on. ~~If you use the Endpoint SSL add-on (available in the Beta program), you can actually have SSL at your root domain for free!~~ SSL Endpoint was free in early beta, but is now priced at $20/mo. More info here.

Jekyll on Heroku Part 2: Rack 'Em

matt.manning@gmail.com (Matt Manning) — Sun, 04 Dec 2011 00:00:00 +0000

This is a follow up to my previous post, Run Your Jekyll Site On Heroku.

Ditching rack-jekyll

The previous post we got going as a Rack app using rack-jekyll. While rack-jekyll is a good project, it does some checking that isn’t really necessary since we’re precompiling the site and don’t need to build anything at runtime.

I first tried replacing rack-jekyll with Rack::Static, but it only works with explicit filenames. For example, with rack static the URL http://mwmanning.com/index.html would work fine, but http://mwmanning.com would return a 404. Luckily I found this blog post by Chris Continanza which clued me in to Rack::TryStatic, which is part of rack-contrib.

Making this switch is pretty easy. Just remove the rack-jekyll gem and add the rack-contrib gem, so your Gemfile looks like this:

source :rubygems

gem 'jekyll'
gem 'rack-contrib'
gem 'RedCloth'
gem 'thin'

Then change your config.ru file to use Rack::TryStatic. The result should look something like this:

require 'rack/contrib/try_static'

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

Rewriting URLs

One of the reasons I really wanted Rack was for rewriting URLs. That’s something the Jekyll server can’t do. My old blog was a Wordpress site hosted by A Small Orange, and its URL scheme was slightly different. I had a few links and some Google results pointing to my old URLs, and I didn’t want to throw all of that away, so I turned to Rack::Rewrite. With Rack:Rewrite you get a little DSL that lets you specify routes that are either rewritten or issue redirects. Regular expressions are available for route matching, which suited my needs perfectly for supporting the old blog URL scheme.

Since I don’t plan to go back to the old scheme from the Wordpress blog, I dedided to use 301 - Moved Permanently redirects.

I added the gem to my Gemfile:

source :rubygems

gem 'jekyll'
gem 'rack-contrib'
gem 'rack-rewrite'
gem 'RedCloth'
gem 'thin'

and set up the appropriate redirects in my config.ru:

require 'rack/contrib/try_static'
require 'rack/rewrite'

# Support links to old Wordpress site
use Rack::Rewrite do
  r301 '/2010/11/29/ec2-micro-instance-as-a-remote-bittorrent-client/', '/2010/11/29/EC2-Micro-Instance-as-a-Remote-Bittorrent-Client.html'
  r301 '/2009/01/13/timemachine-ubuntu-nas/', '/2009/01/13/TimeMachine-Ubuntu-NAS.html'
  r301 '/projects/jack-o-led/', '/projects/jack-o-LED.html'
  r301 %r{/projects/(\S+)/}, '/projects/$1.html'
end

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

Caching

The final piece I wanted for my Rack app was caching. Older Heroku stacks (Aspen and Bamboo, which only support Ruby apps) use Varnish and nginx in front of the app server and can be blazing fast. The Cedar stack does not, because they conflict with some advanced HTTP features it supports.

Luckily, Heroku offers a hosted memcache add-on which can be used with the Rack::Cache middleware. The free version of the add-on gives you 5MB of memory, which is pretty good for my purposes since my whole app (gems included) is only a little over 7MB. To get this working I first added the add-on with this command:

heroku addons:add memcache:5mb

Next, I needed a couple of gems, dalli and rack-cache. With those installed my Gemfile looked like this:

source :rubygems

gem 'dalli'
gem 'jekyll'
gem 'rack-cache'
gem 'rack-contrib'
gem 'rack-rewrite'
gem 'RedCloth'
gem 'thin'

The last bit was to set up Rack::Cache in the config.ru file.

require 'dalli'
require 'rack/cache'
require 'rack/contrib/try_static'
require 'rack/rewrite'

$cache = Dalli::Client.new
use Rack::Cache,
  :verbose => true,
  :metastore => $cache,
  :entitystore => $cache

# Support links to old Wordpress site
use Rack::Rewrite do
  r301 '/2010/11/29/ec2-micro-instance-as-a-remote-bittorrent-client/', '/2010/11/29/EC2-Micro-Instance-as-a-Remote-Bittorrent-Client.html'
  r301 '/2009/01/13/timemachine-ubuntu-nas/', '/2009/01/13/TimeMachine-Ubuntu-NAS.html'
  r301 '/projects/jack-o-led/', '/projects/jack-o-LED.html'
  r301 %r{/projects/(\S+)/}, '/projects/$1.html'
end

use Rack::TryStatic,
    :root => "_site",
    :urls => %w[/],
    :try => ['.html', 'index.html', '/index.html']

run lambda { [404, {'Content-Type' => 'text/html'}, ['Not Found']]}

I set this up without really thinking too much about it. Caching Is Good™, right? Well I decided to do a quick test with ApacheBench to see what kind of average response times I got. (If you’re having trouble with ApacheBench on OS X Lion, here’s how to fix it). First I wanted to get a baseline number without caching, so I removed that part from config.ru and ran this test on one dyno.

[~/code/blog] ab -n 1000 -c 5 http://www.mwmanning.com/2010/11/29/EC2-Micro-Instance-as-a-Remote-Bittorrent-Client.html
This is ApacheBench, Version 2.3 <$Revision: 1178079 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.mwmanning.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        thin
Server Hostname:        www.mwmanning.com
Server Port:            80

Document Path:          /2010/11/29/EC2-Micro-Instance-as-a-Remote-Bittorrent-Client.html
Document Length:        11274 bytes

Concurrency Level:      5
Time taken for tests:   42.249 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      11451000 bytes
HTML transferred:       11274000 bytes
Requests per second:    23.67 [#/sec] (mean)
Time per request:       211.247 [ms] (mean)
Time per request:       42.249 [ms] (mean, across all concurrent requests)
Transfer rate:          264.68 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       29   43  12.6     39     117
Processing:    75  168 127.0    140    1870
Waiting:       32   59  27.4     53     422
Total:        109  211 127.5    184    1913

Percentage of the requests served within a certain time (ms)
  50%    184
  66%    199
  75%    209
  80%    218
  90%    255
  95%    310
  98%    582
  99%    853
 100%   1913 (longest request)

Not bad. The average response time was about 211ms, with 2/3 of requests taking less than 200ms. Next I turned caching back on and ran the same test.

[~/code/blog] ab -n 1000 -c 5 http://www.mwmanning.com/2010/11/29/EC2-Micro-Instance-as-a-Remote-Bittorrent-Client.html
This is ApacheBench, Version 2.3 <$Revision: 1178079 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.mwmanning.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        thin
Server Hostname:        www.mwmanning.com
Server Port:            80

Document Path:          /2010/11/29/EC2-Micro-Instance-as-a-Remote-Bittorrent-Client.html
Document Length:        11274 bytes

Concurrency Level:      5
Time taken for tests:   49.630 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      11593000 bytes
HTML transferred:       11274000 bytes
Requests per second:    20.15 [#/sec] (mean)
Time per request:       248.152 [ms] (mean)
Time per request:       49.630 [ms] (mean, across all concurrent requests)
Transfer rate:          228.11 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       28   43  60.9     35    1142
Processing:    76  205 262.7    142    3830
Waiting:       41   77 140.4     59    2418
Total:        110  248 285.4    183    3932

Percentage of the requests served within a certain time (ms)
  50%    183
  66%    199
  75%    214
  80%    228
  90%    392
  95%    548
  98%    970
  99%   1279
 100%   3932 (longest request)

What’s this? Not only were the caching results not any better, the averages were slightly worse. This was a bit of a facepalm moment. The main purpose of caching is to prevent calculations or database transactions being made by the web app backend, but in this case the web app backend is doing very little other than serving static files off of disk. Plus the Memcached instance being used here is network-attached, so there’s some unavoidable latency in that connection. In this case serving files off of disk is faster than serving out of memory over a network connection, so I disabled caching. Hooray for the scientific method!

Run Your Jekyll Site On Heroku

matt.manning@gmail.com (Matt Manning) — Tue, 29 Nov 2011 00:00:00 +0000

A few months back I decided I wanted to blog like a hacker, so I started revamping my old Wordpress blog into a more minimal site generated by Jekyll. You’re reading the result of that work right now.

I was happy with how easy Jekyll was to set up and use, and it’s a relief to have my post archives as markdown files under Git version control instead of in a proprietary blog engine format in some database.

One of the cool things about a Jekyll site is that you can push it to Github, and Github will serve it. Not just the source, but the actual site itself. You can see it working with this site. Here is the source, and here it is running. While this is pretty cool, Github isn’t really an app platform. I wanted to run my site on Heroku.

Through some searching and tinkering I figured out a couple of different ways of getting my site running on Heroku. I’ll share both of them here. The first method is great for its simplicity but is short on configurability and efficiency. The second method is more complicated but is performant and endlessly tweakable.

Method 1: Running the Jekyll Server on Heroku

Heroku’s latest application stack, Cedar, provides an amazing amount of flexibility with some very simple configuration techniques. It makes running your Jekyll site on Heroku as simple as running it on your own computer – you install some gems and run the server.

I used this example app created by Mark Pundsack as a guide. Its REAME explains the setup process very well. Here is a simple series of steps to get up and running for free on Heroku, assuming you already have a working Jekyll site.

Create a Heroku app on the Cedar stack

From the top level of your Jekyll site’s repo, create the Heroku site. This will provision the Heroku site and add a git remote named “heroku”.

heroku create -s cedar

Install the gems

Create a Gemfile that looks like this:

source :rubygems

gem 'jekyll'
gem 'RedCloth'

Then run:

bundle install

to install the gems and create a Gemfile.lock file.

Configure the Jekyll server

Apps on the Cedar stack define their process types via a Procfile. Your Procfile needs one line defining the Jekyll server like this:

web: bundle exec jekyll --server -p $PORT

Deploy to Heroku

git push heroku master

That’s it! Now your Jekyll site is running on Heroku.

Method 2: Running Your Site As a Rack App

While the method above is simple and straightforward, it’s not the ideal way to run the site. The Jekyll server isn’t meant for production as it generates pages on the fly for every request. I wanted to take advantage of the real power of Jekyll, its ability to generate a static site that can be served efficiently.

Instead of using the Jekyll server, I really wanted to serve my site as a Rack app. That way I could choose my own app server and have a great deal more control over things like caching using Rack middleware.

Setting Up rack-jekyll

Luckily this can be done quite easily using the rack-jekyll project. Here’s how to set it up.

First add the gem to your Gemfile. I also added the Thin gem, so I could use it as my server. Now your Gemfile should look something like this:

source :rubygems

gem 'jekyll'
gem 'rack-jekyll'
gem 'RedCloth'
gem 'thin'

Next set up a config.ru file that looks like this:

require 'rack/jekyll'

run Rack::Jekyll.new

Finally, modify your Procfile to run your new Rack app. With Thin, mine looks like this:

web: bundle exec thin start -p $PORT -e $RACK_ENV

Building Your Site the Right Way

The documentation for rack-jekyll suggests building the files and then committing the _site directory to your repo before deploying to Heroku. While that will work, it’s a bit messy. I really didn’t want builds to be checked into version control.

An alternative would have been letting Jekyll build the site on the first request to each dyno, but that’s no good either, because it mixes the build and run phases. Plus, depending on writes across multiple requests is not the intended use of Heroku’s ephemeral writable filesystem and could lead to bugs.

What I really needed was to build the site during Heroku’s build phase. This can actually be done by leveraging a new experimental feature known as a “custom buildpack.” Buildpacks are the key to Heroku’s polyglot platform. Each language or framework supported on Heroku such as Ruby, Node.js, or Clojure has it’s own buildpack containing the tools necessary to build an app.

In a normal deployment Heroku automatically detects what buildpack to apply, but you can actually tell Heroku to use a specific buildpack. Either pass an option for it when you create the app:

heroku create -s cedar --buildpack [URL]

or provide the URL for the buildpack repo using the BUILDPACK_URL config variable.

Since it already had all the tools for Rack apps, I forked the Ruby buildpack and added a few lines so that my Jekyll site will be built by Heroku during slug compilation. Finally I set my BUILDPACK_URL with the following command:

heroku config:add BUILDPACK_URL=http://github.com/mattmanning/heroku-buildpack-ruby-jekyll.git

Now when I deploy I see an extra “Building jekyll site” line in the output,

[~/code/blog] git push heroku master
Counting objects: 12, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 1.10 KiB, done.
Total 8 (delta 4), reused 0 (delta 0)

-----> Heroku receiving push
-----> Fetching custom build pack... done
-----> Ruby/Rack app detected
-----> Installing dependencies using Bundler version 1.1.rc
       Running: bundle install --without development:test --path vendor/bundle --binstubs bin/ --deployment
       Fetching gem metadata from http://rubygems.org/.......
       Using RedCloth (4.2.8)
       Using posix-spawn (0.3.6)
       Using albino (1.3.3)
       Using fast-stemmer (1.0.0)
       Using classifier (1.3.3)
       Using daemons (1.1.4)
       Using directory_watcher (1.4.1)
       Using eventmachine (0.12.10)
       Using kramdown (0.13.3)
       Using liquid (2.3.0)
       Using syntax (1.0.0)
       Using maruku (0.6.0)
       Using jekyll (0.11.0)
       Using rack (1.3.5)
       Using thin (1.3.1)
       Using bundler (1.1.rc)
       Your bundle is complete! It was installed into ./vendor/bundle
       Cleaning up the bundler cache.
       Building jekyll site
-----> Discovering process types
       Procfile declares types     -> web
       Default types for Ruby/Rack -> console, rake
-----> Compiled slug size is 7.2MB
-----> Launching... done, v47
-----> Deploy hooks scheduled, check output in your logs
       http://www.mwmanning.com deployed to Heroku

To git@heroku.com:mattmanning.git
   8f84bc4..9350a12  master -> master

and my site runs as a Rack app without polluting the repo with build files!

EC2 Micro Instance as a Remote Bittorrent Client

matt.manning@gmail.com (Matt Manning) — Mon, 29 Nov 2010 00:00:00 +0000

Bittorrent is an efficient, fault-tolerant way to distribute files across the Internet. When a file is popular, it can get shared really quickly as lots of users swarm to download and upload the file simultaneously. When a file is less popular, however, it can take hours or even days to download because of a diminished number of seeders. If you use a desktop computer that runs 24/7 and is always connected to a reliable Internet connection this is no problem, but if your primary machine is a laptop and especially if you use it on the go, slow torrents can turn into a problem.

Wouldn’t it be nice if you had a remote Bittorrent client running on a reliable server somewhere? You could start a download any time from any computer, then pull down the file over a fast, reliable connection when it’s finished. You could even be nice and seed the file for a few days, all the while using (or more importantly NOT using) your laptop as normal.

Amazon’s new, cheap — or in limited cases free — “micro” EC2 instances make this a possibility. Below you’ll find a step-by-step guide to set up your own remote bittorrent client using the Transmission application on an EC2 micro instance running Ubuntu.

Amazon Web Services

First you’ll need to create an Amazon Web Services (AWS) account. If you already have an AWS account you can skip ahead to the server creation. If not head on over to http://aws.amazon.com/free and sign up for an account and get your free Linux-based micro instance (this is what we’ll be using to run our client).

Creating and Booting an Instance

Now that you’re all signed up, we need to create and boot an EC2 micro instance to run our client. EC2 instances are based on Amazon Machine Image (AMI) files. For this guide, we’ll be using one of the official Ubuntu 10.4 LTS AMIs found here. You should chose a 32-bit, EBS-based AMI from a region geographically close to you. I chose ami-480df921.

Once you have the ID of the AMI you want to use, create a new instance based on it. Log into the AWS management console and click on the “Instances” link in the left navbar, then click the “Launch Instance” button.

Click the “Community Instances” tab in the wizard that pops up and paste your AMI ID into the search field. Click the “Select” button next the the search result for your image.

Change the “Instance Type” to “Micro (t1.micro, 613 MB) and click “Continue.”

Click “Continue” on the next screen to use the default Kernel ID and RAM Disk ID.

On the next screen, give your instance a name. I called mine “Remote Bittorrent.” Click “Continue.”

If you don’t already have a key pair set up, click the “Create New Key Pair” radio button and follow the directions to create a new key pair.

On the next screen, click the “Create a New Security Group” radio button. Here we will open port 22 so we can SSH into the server for administration. Add an entry for SSH by choosing it from the select in the “Application” column and clicking the “Add Rule” button. Click “Continue.”

Finally click the “Launch” button to start your instance. It may take a few minutes to launch your instance for the first time.

Return to the management console and click “Security Groups” in the left navbar. Click the name of the security group we just created; we need to a few more rules to it.

First we’ll open port 9091, which is the port used by the Transmission web client. To do this choose “Custom” from the “Connection Method” select box. Chose TCP as the Protocol, From Port and To Port should be 9091, and the source IP should be 0.0.0.0/0. Click the “Add Rule” button. If you don’t see the new rule in the list right away, click the “Refresh” button near the top of the page.

Next we’ll open a range of ports for the actual bittorrent application to use. Again chose “Custom” as the Connection Method, TCP as the Protocol, 49152 as the From Port, 65535 as the To Port, and 0.0.0.0/0 as the Source IP. Click “Add Rule.”

Update/Installation

Now we’ll connect to our new server and configure all of the software we’ll be using. Click “Instances” in the left navbar. Select your instance and select “Connect” under the “Instance Actions” select box. Follow the directions to SSH into your server, but use the user “ubuntu” instead of “root”.

ssh -i bittorrent.pem ubuntu@ec2-184-72-148-12.compute-1.amazonaws.com

Now we’ll upgrade all of the installed packages. Issue the following commands:

ubuntu@ip-10-203-65-125:~$ sudo aptitude update
ubuntu@ip-10-203-65-125:~$ sudo aptitude upgrade

Next, install the Transmission packages

ubuntu@ip-10-203-65-125:~$ sudo aptitude install transmission-cli transmission-daemon

Configuration

Now we want to stop transmission-daemon so we can edit its config.

ubuntu@ip-10-203-65-125:~$ sudo /etc/init.d/transmission-daemon stop

Open the settings.json config file with your favorite editor.

ubuntu@ip-10-203-65-125:~$ sudo nano /etc/transmission-daemon/settings.json

Save downloaded files in a more convenient place: “download-dir”: “/home/ubuntu/Downloads”

Come up with a password to protect your server: “rpc-password”: “mysecretpassword”. (Don’t worry. This password will be obscured to a hash once we restart the transmission-daemon.) Turn off the rpc-whitelist so we can access this client from any IP: “rpc-whitelist-enabled”: false,

Now start transmission-daemon back up:

ubuntu@ip-10-203-65-125:~$ sudo /etc/init.d/transmission-daemon start

Finally, we need to create the /home/ubuntu/Downloads directory and give transmission-daemon the ability to write to it.

ubuntu@ip-10-203-65-125:~$ mkdir /home/ubuntu/Downloads/
ubuntu@ip-10-203-65-125:~$ chown ubuntu:debian-transmission /home/ubuntu/Downloads/
ubuntu@ip-10-203-65-125:~$ sudo chmod g+w /home/ubuntu/Downloads/

And we’re done!

Now point your browser at your instance on port 9091. You can find the URL by clicking on your instance in the AWS management console and checking the value of the “Public DNS” field. For my instance, the URL is http://ec2-184-72-148-12.compute-1.amazonaws.com:9091. You will be prompted to log in. The username is “transmission” (unless you changed it in settings.json) and the password is whatever password you chose in settings.json.

Now you have a client that you can browse to at any time on any computer. You can upload torrent files directly, or you can simply enter the URL of the torrent file and have the client do the rest.

Amazon gets pretty decent download speeds too :)

TimeMachine + Ubuntu NAS

matt.manning@gmail.com (Matt Manning) — Tue, 13 Jan 2009 00:00:00 +0000

So after a few months of hem-hawing around, I finally implemented a respectable backup solution for my Mac. It’s getting old enough (almost 4 years) that I’m starting to worry about random hardware failures, so having a reliable backup is very important.

At home, I have a computer running Ubuntu Linux that I use as a file/media/print server. It has four 250 GB drives in a RAID5 configuration. Since RAID5 can be rebuilt if a drive dies, I feel like this is a pretty safe place to store backups, although eventually I hope to have an offsite backup (ex: Amazon S3) as well.

A while back a little command that would let TimeMachine use network drive started floating around the Internet:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I already had SAMBA set up, but I had some problems, so I ended up following this tutorial — which I highly recommend — and setting up netatalk so I could share the volume over AFP. Looking back, I don’t think SAMBA was the problem, I think the issue was with creating the initial backup disk image. I had to make the image locally and copy it over, as is explained in the troubleshooting section, but after that everything worked perfectly.