Curlbomb - Server Automation for the Wayfaring Hacker

May 19, 2016 at 10:13 AM | categories: curlbomb | View Comments

The industry is full of configuration frameworks: chef, puppet, fabric, salt, ansible, etc. I've also personally worked on several proprietary ones too. Most of these frameworks are pretty good at what they do. It's a good time to be alive if you're into devops.

All of these frameworks require some amount of setup to get going (some more than others.) In this way, they are geared towards running in a preconfigured environment, and are not easily used with fresh hardware that is outside of this environment. Mostly I mean having SSH keys installed, software dependencies setup, and (usually) the client daemon for your chosen automation framework running:

  • chef - Requires chef-client preinstalled on the client.
  • puppet - Requires puppet agent preinstalled on the client.
  • fabric - Runs entirely through SSH, no client is necessary, but does require SSH keys to be installed.
  • salt - Requires salt-minion preinstalled, or to use salt-ssh which requires SSH keys to be preinstalled.
  • ansible - Runs via SSH, so requires SSH keys to be preinstalled.

If you're running an enterprise, this extra setup is not a problem. You just bake the setup into your OS image, including the SSH keys, software dependencies, and a systemd service file (I don't get the hate) to run your client. Spin up a new instance with that image, and chef, puppet, or whatever can see it and control it.

But I'm a wayfaring hacker. I encounter new systems, on new networks, all the time. Here's a few examples where it is not so easy to setup these tools:

  • Installing Linux on a computer using a Live CD. Live CDs typically have a bare minimum of packages installed and don't have an SSH service running, let alone have your SSH keys installed.
  • Being given hardware from an IT organization, or from a partner company. In these cases I've usually been told to use IPMI or I have been given a passphrase rather than an SSH key.
  • Using an Open Source image for your virtualization platform. Maybe you don't trust yourself to perform security updates on your images. Maybe you feel safer running the base image provided by the distribution.
  • Cloud instances where you forgot to give it your SSH key. Yea, typically I just smack my head and redeploy with the key, but this happens to me fairly regularly.
  • Embedded systems, or Virtual Machines, where an SSH service and automation client is overkill.
  • You're at someone elses desk, they're logged into the system with their own key. Maybe you don't even have an account.

It's not that hard to copy your SSH key into a ~/.ssh/authorized_keys file, so what am I ranting on about here? I'm about to propose something even easier.

The wayfaring hacker is very different than the enterprise. He has more limited resources, more limited time, and is doing things very often in a one-off fashion. The wayfaring hacker doesn't have time to create images for all the systems he uses. He may not have the resources to dedicate a server to configuration managment and prefers to just use his laptop. Wayfaring hackers still find homes within enterprises, but his default way of working is to ignore enterprise standards of setting up data centers and creating system images. The wayfaring hacker bootstraps from default Linux distros. The wayfaring hacker regularly tries novel things on cloud instances for only a few hours, then shuts them down, only to rebuild them again a bit later.

Ad-hoc server automation

I've been using the following method of scripting the bootstrap of new machines for several years now. I have to assume I'm not the only one who does it this way, but I don't see it referenced too often. It doesn't use any of the above mentioned configuration management tools. When you're doing something simple, on just one machine, they're overkill. I just do this:

gpg -d ~/git/machine_scripts/my_server_setup.sh.gpg | ssh my_server

This one-liner is great. Let me enumerate the ways:

  • It uses tools commonly installed almost everywhere.
  • It uses ssh, so everything is encrypted on the wire.
  • The script is executed directly, without saving it on the remote client.
  • The script is safely stored, gpg encrypted, on my laptop. It is only ever decrypted right before being sent over the wire.
  • I edit the script in Emacs, which supports gpg encryption out of the box.
  • I store the script in a git repository, so it's always ready to use. If I need to redeploy, just power cycle the server, wipe the drives (or spin up a new cloud instance), and rerun the script.
  • Passphrases or other credentials can be put directly into the script because of the security this affords.

This makes it really easy for me to write a script for a specific machine, and keep that script safe on my own laptop (where it's easy to edit with any editor I have installed locally I might add.) I can redeploy my cloud instance, and rerun my script on it to do whatever I want. It's got none of the niceties (or error handling) of the big configuration management tools, but it's great for simple scripts (including scripts that kick off jobs for those other tools.)

But this still has the same problem as before, the machine still requires an SSH server, and it requires that my key is setup.

Server automation with curlbomb

The gpg | ssh trick is very similar to another well-known trick: curl | bash. It's probably the most popular way of installing open source software these days. It's everywhere (a few examples: docker, sandstorm, even Salt.)

The great thing about curl | bash is that it has very minimal requirements on the client end... you guessed it: curl and bash. You don't need an SSH server (nor ssh keys). All you have to do is type the command and stuff happens. Magic! There's a lot of reasons to not do that though (and some that make a pretty good argument it's ok), which this article doesn't really get into, I'm just planting the idea.

curlbomb is a twist on this idea. Instead of hosting an install script on a public webserver (that anyone can download and run), curlbomb starts a private webserver that requires an authentication token to download a script. You run this on your personal computer, where you have stored a script:

curlbomb run ~/git/machine_scripts/my_server.sh.gpg

Which outputs the following text to the screen:

Paste this command on the client:

  KNOCK=KfG2LkxEkrj4 bash <(curl -LSs http://192.0.2.100:56423)

curlbomb stays running until you paste that bit of text into another computer's shell. That computer will connect to your server, provide the preshared key (KNOCK), download the script, and pipe it into bash. Once the script has been served one time, the server shuts down. If you ever want to run the script again, you have to restart curlbomb. In essence, curlbomb is a private, one-time-use, curl | bash that only you can run. You write the script, and curlbomb ensures that it's transported securely to your client machine over TLS (SSL.) You gain security benefits similar to the gpg | ssh trick (GnuPG+OpenSSL+knock vs GnuPG+OpenSSH), and you also gain the dependency-free nature of curl | bash.

curlbomb can forward it's HTTP port through another server via SSH. This allows you to run the server on your laptop, from anywhere in the world, as long as you can ssh to another server you control. Any client, that can access the internet, can then download the script from your laptop by using the server as a proxy.

Let's look at the same scenarios I said was hard to automate with traditional configuration tools, this time with curlbomb:

  • Installing Linux on a computer using a Live CD - Stick the usb drive/CD into the computer, setup the network connection, and type a curl command.
  • Being given hardware from an IT organization, or from a partner company - Log in via IPMI, type a curl command.
  • Using an Open Source image for your virtualization platform - Log in, type a curl command.
  • Cloud instances where you forgot to give it your SSH key - Log in via the vendor provided console, type a curl command.
  • Embedded systems, or Virtual Machines, where an SSH service and automation client is overkill - Log in, type a curl command.
  • You're at someone elses desk, they're logged into the system with their own key - Type a curl command.

In essence, your dependencies for curlbomb on the client are:

  • a network connection.
  • curl installed.
  • a way to type the curl command.

Isn't that easy? No opening up ports to the client. No running daemons on the client. No passwords or SSH keys need setting up. No maintaining images. No need to ask questions on the client to fill in passwords or other setup questions (just put them in the script.)

Consider the following script I use to do the initial setup for a server. This creates my user account, sets my password, sets up my SSH key, and installs salt-minion so that I can now control the machine with saltstack:

#!/bin/bash
user=ryan
pass=hunter2
sshkey="ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAmo9sQ6iAHDH8Gg1vd1js3R...."

# Create user account:
create_user() {
    useradd $user
    mkdir /home/$user
    chown $user:$user /home/$user
    chpasswd <<EOF
$user:$pass
EOF
    # Install SSH Key:
    mkdir /home/$user/.ssh
    chmod 700 /home/$user/.ssh
    echo "$sshkey" > /home/$user/.ssh/authorized_keys
    chmod 600 /home/$user/.ssh/authorized_keys
    chown -R $user:$user /home/$user/.ssh
}


install_salt_minion() {
    # Saltstack GPG key from https://repo.saltstack.com/apt/debian/8/amd64/latest/SALTSTACK-GPG-KEY.pub
    cat <<EOF | apt-key add -
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2.0.22 (GNU/Linux)

mQENBFOpvpgBCADkP656H41i8fpplEEB8IeLhugyC2rTEwwSclb8tQNYtUiGdna9
m38kb0OS2DDrEdtdQb2hWCnswxaAkUunb2qq18vd3dBvlnI+C4/xu5ksZZkRj+fW
tArNR18V+2jkwcG26m8AxIrT+m4M6/bgnSfHTBtT5adNfVcTHqiT1JtCbQcXmwVw
WbqS6v/LhcsBE//SHne4uBCK/GHxZHhQ5jz5h+3vWeV4gvxS3Xu6v1IlIpLDwUts
kT1DumfynYnnZmWTGc6SYyIFXTPJLtnoWDb9OBdWgZxXfHEcBsKGha+bXO+m2tHA
gNneN9i5f8oNxo5njrL8jkCckOpNpng18BKXABEBAAG0MlNhbHRTdGFjayBQYWNr
YWdpbmcgVGVhbSA8cGFja2FnaW5nQHNhbHRzdGFjay5jb20+iQE4BBMBAgAiBQJT
qb6YAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRAOCKFJ3le/vhkqB/0Q
WzELZf4d87WApzolLG+zpsJKtt/ueXL1W1KA7JILhXB1uyvVORt8uA9FjmE083o1
yE66wCya7V8hjNn2lkLXboOUd1UTErlRg1GYbIt++VPscTxHxwpjDGxDB1/fiX2o
nK5SEpuj4IeIPJVE/uLNAwZyfX8DArLVJ5h8lknwiHlQLGlnOu9ulEAejwAKt9CU
4oYTszYM4xrbtjB/fR+mPnYh2fBoQO4d/NQiejIEyd9IEEMd/03AJQBuMux62tjA
/NwvQ9eqNgLw9NisFNHRWtP4jhAOsshv1WW+zPzu3ozoO+lLHixUIz7fqRk38q8Q
9oNR31KvrkSNrFbA3D89uQENBFOpvpgBCADJ79iH10AfAfpTBEQwa6vzUI3Eltqb
9aZ0xbZV8V/8pnuU7rqM7Z+nJgldibFk4gFG2bHCG1C5aEH/FmcOMvTKDhJSFQUx
uhgxttMArXm2c22OSy1hpsnVG68G32Nag/QFEJ++3hNnbyGZpHnPiYgej3FrerQJ
zv456wIsxRDMvJ1NZQB3twoCqwapC6FJE2hukSdWB5yCYpWlZJXBKzlYz/gwD/Fr
GL578WrLhKw3UvnJmlpqQaDKwmV2s7MsoZogC6wkHE92kGPG2GmoRD3ALjmCvN1E
PsIsQGnwpcXsRpYVCoW7e2nW4wUf7IkFZ94yOCmUq6WreWI4NggRcFC5ABEBAAGJ
AR8EGAECAAkFAlOpvpgCGwwACgkQDgihSd5Xv74/NggA08kEdBkiWWwJZUZEy7cK
WWcgjnRuOHd4rPeT+vQbOWGu6x4bxuVf9aTiYkf7ZjVF2lPn97EXOEGFWPZeZbH4
vdRFH9jMtP+rrLt6+3c9j0M8SIJYwBL1+CNpEC/BuHj/Ra/cmnG5ZNhYebm76h5f
T9iPW9fFww36FzFka4VPlvA4oB7ebBtquFg3sdQNU/MmTVV4jPFWXxh4oRDDR+8N
1bcPnbB11b5ary99F/mqr7RgQ+YFF0uKRE3SKa7a+6cIuHEZ7Za+zhPaQlzAOZlx
fuBmScum8uQTrEF5+Um5zkwC7EXTdH1co/+/V/fpOtxIg4XO4kcugZefVm5ERfVS
MA==
=dtMN
-----END PGP PUBLIC KEY BLOCK-----
EOF

    cat <<EOF > /etc/apt/sources.list.d/saltstack.list
deb http://repo.saltstack.com/apt/debian/8/amd64/latest jessie main
EOF

    apt-get update
    apt-get install -y salt-minion

}

main() {
    create_user
    install_salt_minion

    systemctl start salt-minion
}

# It's a really good idea to create main functions and call them at
# the very bottom of the file whenever you are streaming a shell script
# from a remote location into your shell. What happens if you want to
# run the command "rm -rf /home/guy_who_doesnt_work_here_anymore" and
# the connection dies right as it gets to the first slash? I'm developing 
# a buffering feature into curlbomb, to prevent this, but it's not done yet 
# (although the encryption features do accomplish that as a side effect.)

main 

curlbomb is useful for these kinds of quick scripts that setup the environment for a more robust configuration management tool. It's also a way to keep your entire server setup automated and documented, without having to resort to system images.

This article only scratched the surface of what curlbomb can do. Learn about it in more detail at the curlbomb github project page.

Read and Post Comments

PHP Code Compliance In Emacs

July 01, 2011 at 09:00 AM | categories: php, emacs | View Comments

My job has me working on a project in PHP right now -- it sure isn't Python, but PHP has grown up considerably since the last time I used it, which has been awhile. Consequently, I have no Emacs config for PHP setup, other than vanilla php-mode, so I went in search of one. Of course, I knew Sacha Chua would have a great one, so that's where I've started.

My code is required to conform to the PEAR coding standards which is a bit pedantic in places, but it's generally a good reference to make clean and readable code. The only real pain in conforming to a coding standard is if your editor doesn't pick up on your mistakes right away, you get no automatic feedback and you're left with the task of cleaning up your code later at an inconvenient time. In my case, a subversion pre-commit hook checks for compliance and prevents me from checking in non-conforming code just when I thought I was ready to go home!

Sacha's configuration only checks for general syntax errors, it doesn't check for code compliance. For that, I'm using PHP_CodeSniffer which performs a static analysis of a PHP file and notes any deviations for a given standard. They even support Emacs compile mode out of the box. Compile mode still doesn't give me automatic feedback though, for that I still wanted to use flymake like Sacha has.

So here's my elisp for configuring Emacs to automatically highlight both syntax errors and coding standard deviations for PHP:

(require 'php-mode)
(require 'flymake)

;; Pear coding standards : http://pear.php.net/manual/en/standards.indenting.php
(defun pear/php-mode-init ()
  "Set some buffer-local variables."
  (setq case-fold-search t)
  (setq indent-tabs-mode nil)
  (setq fill-column 78)
  (setq c-basic-offset 4)
  (c-set-offset 'arglist-cont 0)
  (c-set-offset 'arglist-intro '+)
  (c-set-offset 'case-label 2)
  (c-set-offset 'arglist-close 0))
(add-hook 'php-mode-hook 'pear/php-mode-init)

(defun my-php-hook-function ()
  (set (make-local-variable 'compile-command) (format "php_lint %s" (buffer-file-name))))
(add-hook 'php-mode-hook 'my-php-hook-function)

(defun flymake-php-init ()
  "Use php and phpcs to check the syntax and code compliance of the current file."
  (let* ((temp (flymake-init-create-temp-buffer-copy 'flymake-create-temp-inplace))
     (local (file-relative-name temp (file-name-directory buffer-file-name))))
    (list "php_lint" (list local))))

;;This is the error format for : php -f somefile.php -l 
(add-to-list 'flymake-err-line-patterns
  '("\\(Parse\\|Fatal\\) error: +\\(.*?\\) in \\(.*?\\) on line \\([0-9]+\\)$" 3 4 nil 2))

(add-to-list 'flymake-allowed-file-name-masks '("\\.php$" flymake-php-init))
(add-hook 'php-mode-hook (lambda () (flymake-mode 1)))

This depends on a small BASH helper script which I have stored in my $HOME/bin directory (which is on my PATH by default):

!/bin/bash
#This does standard PHP syntax checking
php -f $1 -l
#This does coding standard checking
phpcs --standard=PEAR --report=emacs $1
#Always exit with status code 0 otherwise flymake complains
exit 0

Now I get nice red highlighting for when I forget a doc tag or to put a space between parens and brackets. Isn't pedantry great? :/

Read and Post Comments

Searching GameFAQs within Steam games

September 22, 2010 at 08:30 PM | categories: python, games | View Comments

To take the edge off work, I've been playing Fallout 3 this week. Let me just say this: post apocalyptic Washington DC is huge! Thankfully, there's GameFAQs, because I don't have 40 hours a week to devote to playing games the hard way.

The problem is, 3D games are notoriously bad at recovering after you Alt-Tab out of the game to use your web browser. Luckily, Steam has a built in web browser, so that you don't need to leave the game in order to check GameFAQs, which is great... except for the fact that the Steam browser is totally lame.

See, GameFAQs are plain text files. I think that's cool; plain text is an age old geek tradition, plus it's the ultimate portable file format. However, that also makes things difficult when you're talking about a guide that is over 1MB and broken into hundreds of sections with no hyperlinks to help navigate. GameFAQ authors are a smart bunch though, they have the habit of creating unique codes for each section, essentially creating ad-hoc hyperlinks. You just use your browser's search feature to search for the code and you go right to the section you want.

And what browser doesn't have a search feature?

Oh... right, the Steam browser.

Seriously?

Seriously!?

Alright fine, I'm a hacker, I can get around this. My solution was to write a GameFAQs proxy that downloads FAQs and injects them into a page that uses the jQuery highlight plugin. Slapping on a header to every page for easy navigation, gives us our very own search-in-the-browser add-on for the feature-gimped Steam browser.

Download a standalone executable and try it yourself. Yea, I probably could have made a public server, but I didn't want to get into a copyright hassle, so for now you'll just need to run it yourself on your own computer. I've tried to make it as easy as possible. If you're a Python geek and want to check out the source (or just paranoid about running a pre-compiled exe), it's over on github. If you're interested in how I compiled the Pylons application into a standalone executable with bbfreeze, checkout make.py (although it's pretty cludgy).

One more thing, this proxy is essentially scraping the GameFAQs.com website, so it's possible things could break if they change their site too much. Let me know in the comments if this thing doesn't work anymore.

Instructions for the standalone executable:

  • Download the exe (Tested on Windows 7, 32 & 64 bit)
  • Run the application. It's a self extracting archive, extract it wherever you want. The desktop is fine (it's just one .bat file and a directory.)
  • Run start_steam_gamefaqs.bat and wait for the window to pop up.
  • Leaving that window open, start your steam game.
  • In the game, press Shift-Tab to bring up the Steam community page.
  • Click on the web browser.
  • Enter the URL : http://localhost:5000
  • Search for your desired game and FAQ.
  • Search the page by pressing Ctrl-F and typing in the "Search This Page" box. Pressing Enter multiple times in that box will find the next match for that search term.

Read and Post Comments

My Model M Keyboard Dyed

July 17, 2010 at 07:13 PM | categories: cool stuff | View Comments

I've been using the same IBM Model M keyboard for about 6 years now. I never thought I would see the day when it would quit working. I love this keyboard -- the buckling springs deliberately and assuredly clicking and clacking beneath my fingers blow the pants off any cheap-ass keyboard you get for "free" with your factory Dell PC. It's amazing that the best keyboard ever made was actually one of the very first. Created in 1985, the Model M set the standard to beat, but the quality of PC keyboards over the years has only gone downhill. Thank goodness Unicomp still makes these things. Otherwise, I'm not sure what I would ever replace this keyboard with... maybe a Das or a Deck.

Luckily, I don't have to replace it. It hasn't died yet, and probably never will.

Chris McDonough recently wrote an excellent article on the 25 years of the Model M. Since reading it, I've been thinking quite a bit about my trusty Model M and I decided to give it an early 25th birthday present: some new colors!






I followed a guide from Overclock.net describing the process of applying rit dye to the keys. I've used other keyboards that had paint jobs on them before, but the rit dye process is soo much nicer. With paint, you always get a sticky paint feel on the keys and of course it also covers up the letters on the keys permanently. The rit dye, on the other hand, permeates the plastic and leaves no residue on the surface at all. In fact, you can still see the letters quite well.

I liked the red and brown combination used in the Overclock.net guide, although I left the original color differentiation of the model M intact: letters and numbers red, and the meta keys and some of the F keys brown. The red keys turned out fantastic, the letters on them are very distinctive. However, the brown turned out a bit differently; being much darker, almost black, it all but obscures the markings on the keys. No big deal though, I guess my new Model M is half-way to a Das Ultimate after all. :)

Read and Post Comments

Audacious Dynamic Playlist powered by inotify

July 13, 2010 at 12:22 AM | categories: python, linux | View Comments

I was playing around with StreamRipper today to record a shoutcast stream I enjoy, and I thought to myself: wouldn't it be nice to be able to continuously play all the files I've downloaded without having to manually queue the new files in Audacious?

So, I scratched an itch. With a little DBUS, pyinotify, and an optparse wrapper, I now have a tool to automatically add the tracks to my Audacious playlist. You can grab the latest version on github.

Streamripper does have a relay option (-r) to allow you to listen to the same stream as you're ripping it, and if that's what you want, the following script is superfluous. However, what I wanted was something slightly different: I didn't necessarily want to just listen to the stream live, instead, I wanted the ability to jump around between the tracks already downloaded, exploring different songs, but with the list of tracks ever expanding. That's what this script allows.

You'll need python-dbus and setuptools installed, then just install with:

easy_install http://github.com/EnigmaCurry/audacious-plugins/zipball/master

Startup StreamRipper with your favorite audio stream and point the tool at your target directory:

python -m ec_audacious.dynamic_filesystem_playlist /path/to/your/streamripper_location

Now when StreamRipper creates a new file, it will get automatically (instantaeneously actually, thanks to inotify!) to your Audacious playlist.

Alternatively, you can use the streamripper wrapper script that I've included to start both the filesystem monitor as well as streamripper in one go:

Put the following in your .bashrc file or somewhere equivalent:

alias streamripper="python -m ec_audacious.streamripper"

Then whenever you run streamripper, you'll actually be running the wrapper script instead:

streamripper http://your-cool-stream.com:8000 --audacious

By aliasing streamripper to point to the ec_audacious.streamripper wrapper script, we're effectively adding a new option to streamripper called --audacious which spawns our filesystem monitor.

Read and Post Comments

Next Page »