Lincoln Stoll

I ran

1

history|awk '{a[$2]++} END{for(i in a){printf "%5d\t%s\n ",a[i],i}}'|sort -rn|head

and got this:

1
2
3
4
5
6
7
8
9
10

100        cd
62        ls
60        git
28        ssh
26        hg
19        mate
17        psql
15        sudo
15        mv
14        rake

Looks like I’ve spent too much time looking for things, and not enough time working on them.

This is a quick post, as you may notice there is a new Notebook option in the sidebar. I’ve decided to open up some of my notes from now on in a public format, because a lot of the good stuff is there – useful information, that I never get time to write up. So check it out, and beware – it can be pretty raw. Big props to evernote for this – their new beta is a pretty good notetaking app in its own right, but combined with the online service and syncing – it becomes awesome, and makes something like this possible. I also plan on integrating search with it somehow.

Which leads me on to one of my favorite topics – information management. I seem to go through so much information and generate so many notes it’s not funny, and trying to look after it is difficult. I’ve settled on Evernote and EagleFiler for now. Evernote is great for note taking and jotting things down wherever I am, and EagleFiler works well for storing larger documents, manuals etc. I also use a bunch of other tools around this – I plan on writing up some more of my processes here real soon, so keep an eye out. Also, if you have any tools or methods you find indispensable please let me know – I’d love to hear.

I got an email from Amazon yesterday, talking about persistent storage for EC2 – Based on the email (full text below). From what I can see this works almost like a SAN, which is awesomely cool – you can store you database on here, and if your DB server dies, just start a new instance, and mount the same filesystem on it. It also supports snapshot backups to S3 – both MySQL and PostgreSQL can work with this. It’s adding a SAN service to go with the storage and hosting services that amazon already provide. Can’t wait to see the pricing..

I think this takes EC2 away from being a cool idea, and turns it into a serious hosting platform. Bring it on!

Amazon email:

Dear Amazon EC2 Developer, Many Amazon EC2 customers have been requesting that we let them know ahead of time about features that are currently under development so that they can better plan for how that functionality might integrate with their applications. To that end, we would like to share some details about a major upcoming feature that many of you have requested – persistent storage for EC2.

This new feature provides reliable, persistent storage volumes, for use with Amazon EC2 instances. These volumes exist independently from any Amazon EC2 instances, and will behave like raw, unformatted hard drives or block devices, which may then be formatted and configured based on the needs of your application. The volumes will be significantly more durable than the local disks within an Amazon EC2 instance. Additionally, our persistent storage feature will enable you to automatically create snapshots of your volumes and back them up to Amazon S3 for even greater reliability.

You will be able to create volumes ranging in size from 1 GB to 1 TB, and will be able to attach multiple volumes to a single instance. Volumes are designed for high throughput, low latency access from Amazon EC2, and can be attached to any running EC2 instance where they will show up as a device inside of the instance. This feature will make it even easier to run everything from relational databases to distributed file systems to Hadoop processing clusters using Amazon EC2.

When persistent storage is launched, Amazon EC2 will be adding several new APIs to support the persistent storage feature. Included will be calls to manage your volume (CreateVolume, DeleteVolume), mount your volume to your instance (AttachVolume, DetachVolume) and save snapshots to Amazon S3 (CreateSnapshot, DeleteSnapshot).

This new functionality is already being used privately by a handful of EC2 customers, and will be publicly available later this year. We will be expanding the private offering as we get closer to launch. Please sign up if you are interested in participating.

We hope this information is useful to you as you plan, design and deploy your applications in Amazon EC2.

Sincerely, The Amazon EC2 Team

I’ve been working with the iPhone since it came out, and I’m amazed at how far everything has come, back from coding up the first TurboSIM hardware hack and all the messing around for jaikbreak and activation, to now where it’s just point a click. Really, only two steps are needed: The first is to unlock and activate the phone which I won’t cover here, as there are good references elsewhere, the second is to tweak it for Australian Carriers (Telstra,. Optus, Vodafone and Virgin have been tested by myself)

First, unlock according to this guide: Mac Windows Then, tap on installer on the home screen. Tap on sources at the bottom right. Tap edit, then add. Enter http://wpool.com/iphone , then tap OK. Wait for it to refresh the sources, then tap on Install on the bottom of the screen. Scroll down to the localisation (1.1.4) menu, then for each item in there tap on it, then tap install. Once they are all down hold down the power button to turn the phone off, then turn it back on – and enjoy!

Sorry if this is a little short – kinda busy right now!

Passenger is now available for download – http://www.modrails.com . If you haven’t heard of it, is basically an apache module for running rails apps – nice and easily. I’ve just moved it over to this site, and it was incredibly easy. The install on my ubuntu server went like this:

1
2
3

sudo apt-get install apache2-prefork-dev libapr1-dev
sudo gem install passenger
passenger-install-apache2-module

I then added the apache config lines to the end of the apache config, and set my site up like:

1
2
3
4
5
6
7

<VirtualHost *:80>
ServerName lstoll.net
DocumentRoot /home/xxx/sites/lstoll.net/current/public
ServerAdmin xxx@xxx.net
ErrorLog /home/xxx/logs/lstoll.net_error.log
CustomLog /home/xxx/logs/lstoll.net_access.log combined
</VirtualHost>

Then, when apache started up it automatically started the rails app, and it just worked. It’s simple to use with capistrano, here’s my deploy tasks from deploy.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

namespace :deploy do
  desc "Custom restart task for passenger"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "touch #{deploy_to}/current/tmp/restart.txt"
  end
  
  desc "Custom start task for passenger"
  task :start, :roles => :app do
    # Don't need to do anything, it's automatic
  end
  
  desc "Custom stop task for passenger"
  task :stop, :roles => :app do
    # currently no way I know of to 'stop' the app
  end
  
end

Once that is done, capistrano can flag to passenger to restart the app.

A bit of casual benchmarking shows it’s fast too. I’m very happy this is out there, it makes running rails apps as easy as PHP – saving all the messing around with mongrel/thin/monit. It also works with rails caching, so so more messing around with rewrite rules to get cached files to be read right off disk. This should hopefully make rails better on shared hosts, and lower the barrier of entry. Only downside is that this is currently rails only – so for merb/camping/anything else the old way is still needed. I’m hoping that soon passenger will support rack, or mod_rack/mod_rubinius will come out, so this ease is brought to all ruby frameworks.

Currently working on a merb project using datamapper – nice framework, but the auto-migrate feature is fairly limited – So I decided to hook up ActiveRecord’s migrations into the app. This code could be equally as easily adapted in to anything else.

First, I created a folder for the migrations to live in:

1

mkdir -p db/ar_migrations

I then appended a few tasks ripped out of Rails and slightly tweaked to the Rakefile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

APP_BASE = File.dirname(File.expand_path(__FILE__))

namespace :db do
  task :ar_init do
    # Load the database config
    require 'active_record'
    database_yml = YAML::load(File.open(APP_BASE + "/config/database.yml"))
    init_env == "rake" ? current_env = "development".to_sym : current_env = init_env.to_sym # Default to dev env, symbolize
    ActiveRecord::Base.establish_connection(database_yml[current_env])
    # set a logger for STDOUT
    ActiveRecord::Base.logger = Logger.new(STDOUT)
  end
  
  desc "Migrate the database through scripts in db/migrate. Target specific version with VERSION=x. Turn off output with VERBOSE=false."
  task :migrate => :ar_init  do
    ActiveRecord::Migration.verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] == "true" : true
    ActiveRecord::Migrator.migrate(APP_BASE + "/db/ar_migrations/", ENV["VERSION"] ? ENV["VERSION"].to_i : nil)
    Rake::Task["db:schema:dump"].invoke if ActiveRecord::Base.schema_format == :ruby
  end
  
  namespace :schema do
    desc "Create a db/ar_schema.rb file that can be portably used against any DB supported by AR"
    task :dump => :ar_init do
      require 'active_record/schema_dumper'
      File.open(ENV['SCHEMA'] || APP_BASE + "/db/ar_schema.rb", "w") do |file|
        ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
      end
    end

    desc "Load a ar_schema.rb file into the database"
    task :load => :ar_init do
      file = ENV['SCHEMA'] || APP_BASE + "/db/ar_schema.rb"
      load(file)
    end
  end
end

These will read from database.yml in config/ , so make sure you’ve got datamapper set up.

Lastly, Let’s create an example migration to create the datamapper sessions table in db/ar_migrations/001_create_dm_session_table.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

class CreateDmSessionTable < ActiveRecord::Migration
  def self.up
    create_table(:sessions, :id => false) do |t|
      t.string :session_id, :null => false
      t.text :data
      t.timestamp :updated_at
    end
    # Need this because ar in unhappy with a varchar primary key
    sql = "ALTER TABLE sessions ADD CONSTRAINT sessions_pkey PRIMARY KEY  ( session_id )"
    execute(sql)

    add_index :sessions, :session_id
    add_index :sessions, :updated_at
  end

  def self.down
    drop_table :sessions
  end
end

Once this is done, you can run rake db:migrate , and the DB will get the sessions table added. The db:schema:load task is also there, so you should use this when you first deploy your app to a new server. The naming is also consistent with rails, so capistrano should interact with it out of the box, with the exception that the environment to load/migrate needs to be called MERB_ENV instead of RAILS_ENV

When creating additional migrations you will need to follow the ‘proper’ naming scheme: the filename should start with an incrementing three digit number, and the rest should be a ‘flattened’ version of the class name. (i.e migration class AddPostTable becomes add_post_table).

I recently upgraded to mercurial 1.0 using macports, and when I tried to communicate with a HTTPS repository I got the following error:

1

abort: Python support for SSL and HTTPS is not installed

The solution was simple:

1

sudo port install py25-socket-ssl

After that, HTTPS operations worked fine.

I loved github, but prefer using mercurial – it just feels nicer to me, has better platform & IDE support, and has some really nice features. But github is a great place to mange repo’s, work with other peoples code and the like.

Enter hghub – a site I’m currently developing to be github, but for those of us who use mercurial (and with some nice extra’s like a great interface for tracking patch queues for other people’s projects)

If you are interested please register for the beta at http://hghub.com , and if you have any ideas/comments/questions/suggestions please let me know – I want this to be the best place for keeping and working with code.

EnterpriseDB have done some reshuffling, and have release a package called Postgres Plus – contains their tweaked version of postgresql bundled with some other pieces for a bunch of platforms including mac – well recommended.

I did have a few problems getting the postgres gem working with it – here’s what I had to do

Mac OS X:

1
2
3

PATH=$PATH:/Library/PostgresPlus/8.3/bin/
sudo gem install postgres -- --with-pg-include=/Library/PostgresPlus/8.3/include \r
--with-pg-lib=/Library/PostgresPlus/8.3/lib/ ARCHFLAGS='-arch i386'

Linux:

1
2
3
4
5

PATH=$PATH:/Library/PostgresPlus/8.3/bin/
sudo ln -s /usr/lib/libssl.so.0.9.8 /usr/lib/libssl.so.4
sudo ln -s /usr/lib/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.4
sudo gem install postgres -- --with-pg-include=/Library/PostgresPlus/8.3/include \r
--with-pg-lib=/Library/PostgresPlus/8.3/lib

You’ll probably want to make the PATH change permanent for access to psql.

And that should get you going. Those of you who use postgres should definately check it out at http://www.enterprisedb.com/ – and for those using any other DB it’s worth checking out too.

Finally got this back up and running – keep an eye out, expect more soon!

Linc

I’ve finally decided to bite the bullet and move to a distributed VCS – after looking around it seems that mercurial is the way to go. There is a TextMate bundle for it, and NetBeans plugin, Trac plugin, and seems to be well supported. I also looked around and particularly at git – however even though its name comes up often, it doesn’t seem to have anywhere near the support.

Getting the basic system up and running

Start a root shell & install install the software:

1
2

sudo -s
apt-get install mercurial

Set up the repository storage

1
2
3

mkdir -p /var/lib/hg/htdocs
mkdir /var/lib/hg/repos
cp /usr/share/doc/mercurial/examples/hgwebdir.cgi /var/lib/hg/htdocs/

Create the apache config file /etc/apache2/sites-available/hg.lstoll.net and make it look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

<VirtualHost *:80>
ServerName hg.lstoll.net
RewriteEngine On
RewriteRule ^/(.*)$ https://hg.lstoll.net/$1 [R]
</VirtualHost>

<VirtualHost *:443>
ServerName hg.lstoll.net
DocumentRoot /var/lib/hg/htdocs
ServerAdmin lstoll@lstoll.net
ErrorLog /var/log/apache2/hg.lstoll.net_ssl_error.log
CustomLog /var/log/apache2/hg.lstoll.net_ssl_access.log combined

RewriteEngine on
RewriteRule ^/(.*) /hgwebdir.cgi/$1

<Directory /var/lib/hg/htdocs>
DirectoryIndex hgwebdir.cgi
AddHandler cgi-script .cgi
Options ExecCGI
AllowOverride None
</Directory>

<Location />
AuthUserFile /etc/apache2/projects.htpasswd
AuthType Basic
AuthName "hg.lstoll.net"
Require valid-user
Order allow,deny
allow from all
</Location>
</VirtualHost>

Create the hg web directory config file

1
2
3
4
5
6

cat > /var/lib/hg/htdocs/hgweb.config << EOF
[collections]
/var/lib/hg/repos/ = /var/lib/hg/repos
[web]
style = gitweb
EOF

Correct permissions, and enable the site in apache

1
2
3
4

chown -R www-data:www-data /var/lib/hg
chmod +x /var/lib/hg/htdocs/hgwebdir.cgi
a2ensite hg.lstoll.net
apache2ctl graceful

You should now be able to browse to the site, authenticate, and see that their is no repositories available. So lets create one:

1

su www-data -c "hg init /var/lib/hg/repos/hgtest"

You should now have a basic mercurial environment up and running.

If like me you used user@host for the htpasswd usernames previously (so you can do things like have trac send email notifications), you will need to also add usernames without the @ symbol – mercurial doesn’t seem to handle these well when trying to save a username and password in a clone operation. I now maintain both a user@host and user accounts for all users.

Migrating a SVN repository into your new setup

Install hgsvn

1
2

apt-get install python-setuptools
easy_install hgsvn

change to the repo dir, and import the repository

1
2
3
4
5

cd /var/lib/hg/repos
su www-data -c "hgimportsvn file:///var/lib/svn/test"
cd test
su www-data -c "hgpullsvn"
su www-data -c "hg update"

Clean out the svn remnants

1
2

find . -name .svn | xargs rm -fr
find . -name .hgignore | xargs rm -fr

Setting up the trac integration

1
2
3
4
5
6

cd /usr/share/trac/plugins
svn co http://svn.edgewall.com/repos/trac/sandbox/mercurial-plugin
cd mercurial-plugin
python setup.py bdist_egg
cd dist
easy_install TracMercurial-0.10.0.2-py2.5.egg

(note – the egg filename may differ for you) Edit the config file for the project you are moving to mercurial

1

nano /var/lib/trac/test/conf/trac.ini

enable the plugin by adding something like this:

1
2

[components]
tracvc.hg.* = enabled

You will also need to change your repository config to something like this:

1
2
3

[trac]
repository_type = hg
repository_dir = /var/lib/hg/repos/test

Installing the client on Mac OS X, and checking it out

Install mercurial

1

sudo port install mercurial

Create a local copy to work with:

1

hg clone http://username:password@hg.lstoll.net/test

Replace username and password with the username and password you set up. Quick note – your password will be stored in an insecure format, and will be echoed on the screen when performing an operation against the central repository – be aware of this (Optional) Set up the Mercurial TextMate bundle

1
2
3

mkdir -p ~/Library/Application Support/TextMate/Bundles
cd ~/Library/Application Support/TextMate/Bundles
svn co http://macromates.com/svn/Bundles/trunk/Bundles/Mercurial.tmbundle

Sources: http://ww2.samhart.com/book/export/html/49 http://www.selenic.com/mercurial/wiki/index.cgi/HgWebDirStepByStep

The MySQL Gem won’t install directly on a Leopard machine using a Macports install of mysql (or any other MySQL install method I can think of) because by default gem will try built a universal gem, however the MySQL install will only have libraries for your current platform. So we need to force the architecture the gem is built for.

To install on a Intel Machine:

1
2

sudo bash -c "ARCHFLAGS='-arch i386' gem install mysql -- \r
--with-mysql-config=/opt/local/bin/mysql_config5"

To install on a PowerPC Machine:

1
2

sudo bash -c "ARCHFLAGS='-arch i386' gem install mysql -- \r
--with-mysql-config=/opt/local/bin/mysql_config5"

Make sure you select the newest ruby (not mswin32) version.
If you are not using a Macports MySQL Install, find where mysql_config lives and update the above command.

More info on the bundled Ruby under Leopard can be found here: http://trac.macosforge.org/projects/ruby/wiki/WhatsNewInLeopard

I wanted to telnet to a SSL service (specifically gmail’s IMAP servers) to mess around, and wanted a quick and easy command to do it. Turns out openssl can do it, which is handy because its pretty much everywhere. Here’s the command

1

openssl s_client -connect imap.gmail.com:993 -state -debug

I set up a new Ubuntu XEN VM, and installed a few packages.. then started getting errors like this all over the place:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = “en_AU.UTF-8”
are supported and installed on your system.
perl: warning: Falling back to the standard locale (“C”).
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Seems that at some point all the entries get screwed up for setting locales. I ran sudo apt-get install language-pack-en-base , and then everything was fine

I tried installing VMware Server 1.0.3 onto OpenSuse 10.2, and came across an issue – It wouldn’t let me log in as root (or anything), telling me that I had an invalid username or password. To get it working I had to use yast to install pam-modules-32bit, then edit /etc/vmware/pam.d/vmware-authd , and replace pamdir with /usr/lib/vmware/lib/libpam.so.0/security . You should then be right to go, straight away!