I Wanted Orange (The machine would not make a mistake.)


Refreshing my LAMP stack with Docker

Over the years my home desktop has acquired many layers of development tools, to the extent where I keep things around "just in case" but in reality the project or experiment is definitely over. (Who needs disk-images of a Slackware server from 2001?) I've decided to do some spring-cleaning and install new versions.

Part of that means (re)installing Docker on Windows 7, and I've decided to journal this process mainly so I can refer to what I did (or didn't) do later.  I'm going to assume that if you're still reading this, you have a vague idea what Docker is and where it stands relative to Vagrant, Virtualbox, and the development benefits of virtual-machinery in general.

Step one: Install Docker Toolbox (and VirtualBox)

For Windows 7 you'll want Docker Toolbox, since it seems the flagship Docker Platform is designed around features in Windows 10. (And for various reasons, I myself do not want to upgrade.)

When you install Docker Toolbox (in my case, v1.12.2) it will automatically include VirtualBox (5.1.6) as a workhorse behind the scenes. Effectively it has to hose a teeny tiny VM which, in turn, can host the core components of Docker. In the process I was prompted to install some device drivers from Oracle for things like USB support and network drivers.

When it's done, you should have Start Menu entries for VirtualBox (which can be used on its own without Docker) and a shortcut called "Docker Quickstart Terminal" should exist on your desktop. Run it.

You may be prompted to allow VirtualBox to make some changes -- it's probably setting up all the network-adapter magic that lets your virtual machines talk to one-another and reach out to download things from the internet. (Often necessary, when installing.)

Once it's done, you should see an ASCII-art whale (with ominous growths on its back) and a prompt. So let's run the its standard "Hello World" test image in a new container:

User@Desktop MINGW64 ~/
$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:

For more examples and ideas, visit:

Step two: A simple docker-compose

Create a new folder somewhere for the project, and create a new file called docker-compose.yml  , which will define which host(s) we need for a full installation of, well, whatever.

Initially, let's choose one PHP web node, running a pre-release copy of PHP-7.1, and let's structure it so that we can customize that image a little bit.

version: '2'
      context: ./image_web
       - "8000:80"
     restart: always
       - ./src/:/var/www

This sample defines a new host called webserver, and in order to create it docker-compose will go try to build ./image_web/Dockerfile , and when it runs the data in ./src/ will be mounted.

So go ahead and create the folder image_web, and inside place a Dockerfile that just says:

FROM php:7.1-rc-apache

At the moment this is a pointless level of indirection, but later it'll help us layer on customizations.

Next, create the folders and file for src/html/index.php , the "html" portion is due to the server setup defined by the existing php:7.1-rc-apache image. To test that PHP is running right, try:

Hello dockerized world!

Step three: Bringing everything up and testing

Back to the command-line, let's check that we have nothing currently running:
User@Desktop MINGW64 ~/docker-demo
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED
STATUS              PORTS               NAMES

So let's ask docker-compose to build everything and start up our tiny world of one webserver. You'll probably see much much more output than this as your computer downloads what it needs for the php:7.1-rc-apache image.

User@Desktop MINGW64 ~/docker-demo
$ docker-compose up -d
Building webserver
Step 1 : FROM php:7.1-rc-apache
 ---> dd39d6dff6fd
Successfully built dd39d6dff6fd
WARNING: Image for service webserver was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
dockerphp_webserver_1 is up-to-date

To figure out what URL to use in your browser, first we should find the IP address that the docker virtual-machine (one-layer out from the containers) is using. You can do that with:

User@Desktop MINGW64 ~/docker-demo
$ docker-machine.exe ip

As for the port number, remember the docker-compose.yml  file? It contained a line saying that the container's port-80 should be mapper to port-8000 on the outside, so that means our combined URL is

Visiting that, you ought to see "Hello dockerized world!" and a PHPInfo output.

Now, we used the -d argument to docker-compose which means the webserver will keep running. To turn it off, simply go:

User@Desktop MINGW64 ~/docker-demo
$ docker-compose stop
Stopping dockerphp_webserver_1 ... done

Epilogue: Cleaning up

What if you want to get rid of this stuff and try some things over? First, you can get a list of the images docker knows about like so:

User@Desktop MINGW64 ~/docker-demo/image_web
$ docker images
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
dockerphp_webserver   latest              dd39d6dff6fd        2 days ago          406.3 MB
php                   7.1-rc-apache       dd39d6dff6fd        2 days ago          406.3 MB
hello-world           latest              c54a2cc56cbb        4 months ago        1.848 kB

Let's try removing the hello-world stuff from before:

User@Desktop MINGW64 ~/docker-demo
$ docker rmi hello-world:latest
Error response from daemon: conflict: unable to remove repository reference "hello-world:latest" (must force) - container 560e895d696c is using its referenced i
mage c54a2cc56cbb

Uh oh, what happened? Well, the hello-world container we created is still around, even if it isn't actively running. We can see it here:

User@Desktop MINGW64 ~/docker-demo
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS                     PORTS               NAMES
cf5dbd9936a7        dd39d6dff6fd        "apache2-foreground"   22 minutes ago      Exited (0) 6 minutes ago                       dockerphp_webserver_1
560e895d696c        hello-world         "/hello"               46 hours ago        Exited (0) 46 hours ago                        small_pasteur

And remove it with:

User@Desktop MINGW64 ~/docker-demo
$ docker rm 560e895d696c

Now when we remove the image, things are smoother:

User@Desktop MINGW64 ~/docker-demo
$ docker rmi hello-world:latest
Untagged: hello-world:latest
Untagged: hello-world@sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Deleted: sha256:c54a2cc56cbb2f04003c1cd4507e118af7c0d340fe7e2720f70976c4b75237dc
Deleted: sha256:a02596fdd012f22b03af6ad7d11fa590c57507558357b079c3e8cebceb4262d7
Tagged as: , Comments Off

Python script: Easily import screenshots to Steam

Recently I wanted to upload some very old TF2 screenshots with Steam's screenshot-tool, only to realize that it isn't as simple as dropping them into a folder. You've got to rename them correctly, generate thumbnails, and (sometimes) restart the Steam-client so it sees them. Rather than manually making dozens of specially-named thumbnails, I decided to automate it. Fortunately I already had the Python Imaging Library installed, which makes the thumbnail-generation easy.

I've uploaded the script to this Github gist.

Assuming you have Python and PIL, just drop the script into the Steam screenshots folder you want to populate, for example C:\Program Files\Steam\userdata\{userid}\780\remote\{gameid}\screenshots\. Then you can just drag-and-drop image files onto it (or pass them as command-line arguments) and it will do the rest. (780 is the app-id for the Steam screenshot tool.)

Steam Screenshot MigratorSteam Screenshot Migrator

Once the new files exist, you may need to restart Steam for it to notice. Unfortunately, all the effort of preserving the original date-information doesn't seem to matter after uploading: Only an "uploaded on" date is visible through Steam's website. Oh well, at least my local records are in-order.

Tagged as: , Comments Off

Spaceballs: The Python Script

"Before you die, there is something you should know about us, Lone-Star..."

def unfoldFilter(unfoldFunction, filterFunction, iterable):
    for item in iterable:           
        for result in unfoldFunction(item):            
            if filterFunction is None or apply(filterFunction, [result]):
                yield result

assert darkHelmet in unfoldFilter(lambda p: p.roommates, lambda p: p.isFormer(), 
    unfoldFilter(lambda p: p.children, None, 
        unfoldFilter(lambda p: p.siblings, None, 
            unfoldFilter(lambda p: p.parents, None, 
                unfoldFilter(lambda p: p.children, lambda p: p.isMale(), 
                    unfoldFilter(lambda p: p.siblings, None, 
                        unfoldFilter(lambda p: p.siblings, lambda p: p.isMale(), 
                            unfoldFilter(lambda p: p.parents, lambda p: p.isMale(), list(loneStar))

assert makesThem(darkHelmet, loneStar) is None

For those who don't get the joke, the movie "Spaceballs" self-parodied itself with a series of whimsical merchandising options and contained a gag referencing Star Wars' iconic "I am your father", except... well, a little more complex.

Tagged as: Comments Off

Uncertain Base64 – Overwatch Puzzle

As I mentioned in yesterday's post, there's an Overwatch graphic of some base64 text with an unclear font... But what's the point of writing something to solve a puzzle when you aren't sure you have the right question?
Overwatch Base64 Issues
So I made this little function which takes a base64-string and generates visually-similar versions:

import itertools
import binascii

def B64Variants(val, confusions = None):
    if confusions is None:
        confusions = ["Il1","O0"] # Use defaults
    val = val.translate(None, " \n\t") # Remove whitespace
    sections = []        
    for char in val:
        confused = False
        for group in confusions:
            if char in group:
                confused = True                
        if not confused:   
            if (len(sections) > 0) and isinstance(sections[-1],str):
                sections[-1] = sections[-1] + char
    # Tidy up so we have a consistent list-of-lists
    for i,v in enumerate(sections):
        if isinstance(v,str):
            sections[i] = [v]                
    for varBits in apply(itertools.product, sections):
        yield "".join(varBits)

if __name__ == "__main__":
    sample = binascii.b2a_base64("Hello, World!")
    print "given", sample
    for var in B64Variants(sample):
        print "maybe", var    

Example output:

given SGVsbG8sIFdvcmxkIQ==

maybe SGVsbG8sIFdvcmxkIQ==
maybe SGVsbG8sIFdvcmxklQ==
maybe SGVsbG8sIFdvcmxk1Q==
maybe SGVsbG8slFdvcmxkIQ==
maybe SGVsbG8slFdvcmxklQ==
maybe SGVsbG8slFdvcmxk1Q==
maybe SGVsbG8s1FdvcmxkIQ==
maybe SGVsbG8s1FdvcmxklQ==
maybe SGVsbG8s1Fdvcmxk1Q==

I think the next step is to try hooking this up to code which tries to decode the cipher-stream byte by byte, skipping to a new password (or a new Base64 source-string) when the decoded byte goes outside conventional ASCII. (I'm gambling that it'll contain an ASCII message, if it doesn't then it's hard to know if you've successfully cracked it.)

Finally, what happens if we apply this to one of the transcriptions people have made from the Overwatch Summer Games video?


78,732 variations in total. Uh oh. I'm happy with the output, but whatever I use to go through these, it needs to be able to memoize or somehow reuse whatever progress it can from variant-to-variant, rather than starting over with each new string.


A more-realistic number would be 6,561, since we know that the first 1 is good because it's part of the OpenSSL header, and because the letter O is visibly wider than zero.


Experiments in decryption – Overwatch puzzle

As part of it's "Summer Games" update, Overwatch put an easter-egg into their video, embedding some base64-encoded text into a scene. Overwatch Base64 hint


Unfortunately there may be errors since the font doesn't clearly distinguish between certain characters such as 1/l/I, but one interpretation comes out to binary data like:

>>> binascii.b2a_qp(binascii.a2b_base64("U2FsdGVkX1+vupppZksvRf5pq5g5XjFRlipRkwB0K1Y96Qsv2L m+31cmzaAILwytX/z66ZVWEQM/ccf1g+9m5Ubu1+sit+A9cenD xxqklaxbm4cMeh2oKhqlHhdaBKOi6XX2XDWpa6+P5o9MQw=="))

This seems to be from an OpenSSL command-line encryption utility, since it begins with the bytes for "Salted__". I'm really no crypto expert, but this is interesting. Maybe I can at least run a dictionary-attack against it?

The first thing to note about the puzzle is that it has an odd number of bytes, implying that that a "stream" cipher was used rather than a "block" cipher, or at least a block cipher being used in a streaming mode. Our chances of guessing the right one are somewhat slim anyway, so why not start with a simple stream-cipher, RC4? First we need to figure out how it's implemented in the OpenSSL command-line tools. Let's try a simple example using "x" (one byte) as our secret message, and "test" as our key:

$ hexdump temp.txt
0000000 0078

$ openssl RC4 -S FFFFFFFFFFFFFFFF -k "test" -p -in temp.txt -a


By forcing the salt to a known 8-byte pattern (-S) and by telling OpenSSL to show us the key it created (-p) we know that somehow all those FF's and "test" combined to make D7BA581CCB7DBAFD5BD1C7DF8BDDE4E3. This is useful, because with a small amount of trial-and-error I can figure out how it is generated, leading to this Python script which gives the same key-output:

import binascii
from Crypto.Hash import MD5

password = "test"
salt = binascii.a2b_hex("FFFFFFFFFFFFFFFF")
key = MD5.new(password+salt).digest()
print binascii.b2a_hex(key) # Output: d7ba581ccb7dbafd5bd1c7df8bdde4e3

Now we're one step on the way to scripting up a compatible RC4 encoding routine, which ends up looking like:

import binascii
from Crypto.Hash import MD5
from Crypto.Cipher import ARC4

def encryptString(self, in_str, password):
    salt =  binascii.a2b_hex('FF' * 8)
    tempkey = MD5.new(password+salt).digest()
    print binascii.b2a_hex(tempkey)
    cipher = ARC4.new(tempkey)
    enc = cipher.encrypt(in_str)
    return 'Salted__' + salt + enc

Great! Now we can implement decoding, and after a little more tinkering, and we've got a little Python class which can encrypt/decrypt the same as the command-line tool:

import binascii
from Crypto.Hash import MD5
from Crypto.Cipher import ARC4
from Crypto import Random

class SimpleRc4:
    def __init__(self):        
        self.random = Random.new()       
        self.header = "Salted__"
        self.saltLen = 8

    def encryptString(self, in_str, password):        
        salt =  self.random.read(self.saltLen)
        tempkey = MD5.new(password+salt).digest()
        cipher = ARC4.new(tempkey)
        enc = cipher.encrypt(in_str)
        return self.header + salt + enc
    def decryptString(self, in_str, password):        
        salt = in_str[len(self.header) : len(self.header)+self.saltLen]
        body = in_str[len(self.header)+self.saltLen:]
        tempkey = MD5.new(password+salt).digest()        
        cipher = ARC4.new(tempkey)
        dec = cipher.decrypt(body)
        return dec

    def selftest(self):
        password = "selftest"
        a = "Content"                    
        b = self.encryptString(a,password)                                        
        c = self.decryptString(b,password)        
        assert(a == c)

So what's this kind of thing useful for? Quickly trying lots and lots of decodings. Here, let's build up a brute-forcing framework, and prove that we can crack a simple RC4-encoded message... (Github Gist)

import sys
import itertools
import binascii
import StringIO

from Crypto.Hash import SHA, MD5
from Crypto.Cipher import AES, ARC4
from Crypto import Random

class Breaker:
    def __init__(self,e,puzzle):
        self.e = e
        self.puzzle = puzzle
        self.last = None
    def attempt(self, password):
        self.last = password
        result = self.e.decryptString(self.puzzle, password)                
        return result
    def comboAttack(self, sequence, tester):
        for pw in sequence:            
            result = b.attempt(pw)
            if tester(result):
                yield (pw, result)
    def CheckBoringAscii(result):
        for c in result:
            d = ord(c)
            if d > 127:
                return False
            elif d < 32:
                return False
        return True

    def GenPasswordList(passwordFile):        
        with open(passwordFile,'rb') as pwdict:
            for line in pwdict:            
                pw = line.strip()
                yield pw

    def GenBrute(charset, maxlength):        
        for i in range(1, maxlength + 1):            
            for c in itertools.product(charset,repeat=i):
                yield ''.join(c)       

class SimpleRc4:
    def __init__(self):        
        self.random = Random.new()       
        self.header = "Salted__"
        self.saltLen = 8

    def encryptString(self, in_str, password):        
        salt =  self.random.read(self.saltLen)
        tempkey = MD5.new(password+salt).digest()
        cipher = ARC4.new(tempkey)
        enc = cipher.encrypt(in_str)
        return self.header + salt + enc
    def decryptString(self, in_str, password):        
        salt = in_str[len(self.header) : len(self.header)+self.saltLen]
        body = in_str[len(self.header)+self.saltLen:]
        tempkey = MD5.new(password+salt).digest()        
        cipher = ARC4.new(tempkey)
        dec = cipher.decrypt(body)
        return dec

    def selftest(self):
        password = "selftest"
        a = "Content"                    
        b = self.encryptString(a,password)                                        
        c = self.decryptString(b,password)        
        assert(a == c)

if __name__ == "__main__":
    e = SimpleRc4()    
    sample = e.encryptString("brute decode challenge", "test")
    b = Breaker(e, sample)
    source = Breaker.GenBrute('abcdefghijklmnopqrstuvwxyz',4)
    tester = Breaker.CheckBoringAscii
    for pw,result in b.comboAttack(source, tester):
        print pw, result 
    #Output: test brute decode challenge

And... Yes! It manages to crack the sample.

I'll explore applying this framework to the actual Overwatch puzzle in a follow-up post. I'll need to put in some fuzzier logic to handle the fact that the source-data -- the ciphertext -- may be subtly corrupted by errors by humans who have to write it down from inside a video-frame.


CTF 2Fort Revamp B3 uploaded to Steam Workshop

After coming across another stolen/plagiarized copy, I decided to try uploading the map to the new (comparatively) Steam Workshop: CTF 2Fort Revamp

Dealing with the Source SDK again, I imagine this is a bit like Chell felt at the start of Portal 2: You realize a whole bunch of time passed, and all the nice humming machinery you remember is in disrepair while a few new weird mechanics exist.

This also marks an attempt to overcome a sort of writer's block that uses the excuse of "it's not good enough to share yet", which usually ends up with all of my creative-energy going into pun-filled Reddit comments. We'll see.


Chase_fixer project now up on GitHub

Following up on my previous post, I've worked on packaging the scripts up a little more nicely and the result is now available on github.

So far it's handled all the weird cases that I see from my own financial history, but I expect there are a few more oddball scenarios (like wire-transfers or refunds) which may require additional tweaking as time goes on.

Tagged as: , Comments Off

Chase’s malformed transaction records

The problem:

Last month, I tried to import some bank-account records (QFX/OFX formats) into the “You Need A Budget” accounting software, which involves telling it how to recognize certain transactions “groceries” and “gas” etc. This did not go as smoothly as I expected, even for an accounting chore, because many of the payee-name and memo fields had ridiculous values! Manually fixing a lot of scrambled data every month wasn't what I had in mind when it came to simplify my budgeting, so I decided to investigate.

<MEMO>01/20 Purchase $9.41 Cash Back $

I believe whatever steam-powered mainframes JP Morgan Chase uses don't seem to have caught up with the current century: Payee fields and memo fields are combined, truncated, arbitrarily split, and whitespace-trimmed, all presumably as sacrifice to some dark and ancient internal decision that 32 characters (for name) and 32 characters (for memo) were long enough for anybody. Some folks say it's the data-format's problem, but I disagree: Chase's datafile says it complies with OFX v1.02, but if you crack open the spec (dated 1997) it clearly says that at least the memo-field should support 256 characters, not 32.


Payee and Memo don't really contain the right thing:

Payee: "Online Payment 1234567890 To Cap"
 Memo: "ital One Bank"

The rest of the memo would have had my cash-back amount (which might be handy in budgeting software) but is truncated:

Payee: "Grocer &amp; Sons Inc. 12345 Exampl"
 Memo: "e road 01/18 Purchase $20.11 Cash b"

The split here occurs between two words, but the whitespace was trimmed! There's no automatic way to know this is "Park lane" vs. "Park lane":

Payee: "Marios Pizza and Plumbing 5442 Park" 
 Memo: "lane NW"

Current progress

Right now I have a series of Python classes  which:

  • Parses the original OFX file(example)
  • Translates it into a much-more-convenient XML file with similar structure
  • Visits every transaction in the XML file and applies custom logic to fix it up
  • Writes the XML file back out as OFX

So far I'm pretty happy with the result: All I have to do is code logic for a few of the common cases, and run the scripts after I download the OFX files from chase. Here's an example of a super-basic statement visitor that just tries to combine Payee and Memo.

    def visitStatement(self, values):
        name = values.get("NAME", "")
        memo = values.get("MEMO", "")

        if len(name) < 32:
            # When the split occurred, there was whitespace which got trimmed, re-add it
            combined = name + " " + memo
        elif len(name) == 32:
            # The split was forced due to some size limit, and we don't really know if there's
            # a space between them or not...
            combined = name + memo
            pass #TODO warning, larger than ever expected

        values["NAME"] = combined
        values["MEMO"] = "" # No more memo data, it's all inside Payee

From this humble beginning I can branch out into recognizing common patterns (like transfers) and payees and clean up the data appropriately. After generating a new QFX file, the YNAB software seems to handle the longer payee names just fine.

Future work

The biggest problem left is that the data still isn't clean enough: Anything over 64 characters has been lost, and it's not always clear if I need to reinsert whitespace between Payee and Memo. Fortunately, Chase does offer a CSV download, which isn't as useful for importing into accounting applications but does contain the entire original. I just need to find some way to cross-reference between the two, perhaps based on dates, amounts, and some sort of non-whitespace similarity.

Once I have things a little more polished I plan to put them up on Github, but at the moment there are still a lot of hardcoded data-file paths and stuff.

Tagged as: , Comments Off

Still alive in 2012

Sorry, dear readers. Or reader, more likely. I've been slacking off for the last few months and now my self-guilt compels me to update.

My work on a branch of PackBSP to handle more game-engines didn't go so well. I painted myself into some of the same "massive rewrite" corners I vowed to avoid, and then spent so long doing other things that it's hard to pick up again. On the other hand, I think I've worked my way through some architectural problems, and the silver-lining of having no other code-contributors is that I don't need to worry about backwards-compatibility very much.

Currently my interest is on the Netbeans Platform, and how I might be able to use it to streamline PackBSP and break away from the limitations of a wizard-centric interface. Actually, my daydream is to create a bunch of Netbeans plug-ins that turn the Netbeans IDE into a Source-engine-related powerhouse, but with the VIDE making a surprise return from the dead, perhaps that niche is already well on its way towards being filled.

Lastly, as a matter of interest, I temporarily fixed my video card with the "oven trick" (8 minutes at 375 Fahrenheit) but a week or two later it failed again, so I may just put it up on Craigslist as a challenge to anyone who thinks they can make it stick.


Hardware failure: Need to RMA my video card.

Well, it seems my graphics card (which has been limping along for years with intermittent glitches under load) has finally given up the ghost and my computer will no longer boot. Fortunately, this occurred after I finished wringing many hours of enjoyment and completionism from Deus Ex: Human Revolution, or else I would feel incredibly annoyed at the interruption. (Aside: It's a good game. A worthy successor to Deus Ex.)

Since the card (an Nvidia 8800GT variant) is still decently-powerful and has no obvious damage, I'm going to see what I can do under the limited-lifetime warranty. Compared to its earlier foibles, RMAing now ought to be unambiguous and straightforward, given that the computer now refuses to even POST if the card is present.

Tagged as: Comments Off