I Wanted Orange (The machine would not make a mistake.)

14Aug/16Off

Python script: Easily import screenshots to Steam

Recently I wanted to upload some very old TF2 screenshots with Steam's screenshot-tool, only to realize that it isn't as simple as dropping them into a folder. You've got to rename them correctly, generate thumbnails, and (sometimes) restart the Steam-client so it sees them. Rather than manually making dozens of specially-named thumbnails, I decided to automate it. Fortunately I already had the Python Imaging Library installed, which makes the thumbnail-generation easy.

I've uploaded the script to this Github gist.

Assuming you have Python and PIL, just drop the script into the Steam screenshots folder you want to populate, for example C:\Program Files\Steam\userdata\{userid}\780\remote\{gameid}\screenshots\. Then you can just drag-and-drop image files onto it (or pass them as command-line arguments) and it will do the rest. (780 is the app-id for the Steam screenshot tool.)

Steam Screenshot MigratorSteam Screenshot Migrator

Once the new files exist, you may need to restart Steam for it to notice. Unfortunately, all the effort of preserving the original date-information doesn't seem to matter after uploading: Only an "uploaded on" date is visible through Steam's website. Oh well, at least my local records are in-order.

Tagged as: , Comments Off
11Aug/16Off

Spaceballs: The Python Script

"Before you die, there is something you should know about us, Lone-Star..."

def unfoldFilter(unfoldFunction, filterFunction, iterable):
    for item in iterable:           
        for result in unfoldFunction(item):            
            if filterFunction is None or apply(filterFunction, [result]):
                yield result

assert darkHelmet in unfoldFilter(lambda p: p.roommates, lambda p: p.isFormer(), 
    unfoldFilter(lambda p: p.children, None, 
        unfoldFilter(lambda p: p.siblings, None, 
            unfoldFilter(lambda p: p.parents, None, 
                unfoldFilter(lambda p: p.children, lambda p: p.isMale(), 
                    unfoldFilter(lambda p: p.siblings, None, 
                        unfoldFilter(lambda p: p.siblings, lambda p: p.isMale(), 
                            unfoldFilter(lambda p: p.parents, lambda p: p.isMale(), list(loneStar))
                        )
                    )
                )
            )
        )
    )

assert makesThem(darkHelmet, loneStar) is None

For those who don't get the joke, the movie "Spaceballs" self-parodied itself with a series of whimsical merchandising options and contained a gag referencing Star Wars' iconic "I am your father", except... well, a little more complex.

Tagged as: Comments Off
4Aug/16Off

Uncertain Base64 – Overwatch Puzzle

As I mentioned in yesterday's post, there's an Overwatch graphic of some base64 text with an unclear font... But what's the point of writing something to solve a puzzle when you aren't sure you have the right question?
Overwatch Base64 Issues
So I made this little function which takes a base64-string and generates visually-similar versions:

#!/usr/bin/python
import itertools
import binascii

def B64Variants(val, confusions = None):
    if confusions is None:
        confusions = ["Il1","O0"] # Use defaults
    val = val.translate(None, " \n\t") # Remove whitespace
    sections = []        
    for char in val:
        confused = False
        for group in confusions:
            if char in group:
                confused = True                
                sections.append(list(group))
                
        if not confused:   
            if (len(sections) > 0) and isinstance(sections[-1],str):
                sections[-1] = sections[-1] + char
            else:
                sections.append(char)       
    # Tidy up so we have a consistent list-of-lists
    for i,v in enumerate(sections):
        if isinstance(v,str):
            sections[i] = [v]                
    for varBits in apply(itertools.product, sections):
        yield "".join(varBits)

if __name__ == "__main__":
    sample = binascii.b2a_base64("Hello, World!")
    print "given", sample
    for var in B64Variants(sample):
        print "maybe", var    

Example output:

given SGVsbG8sIFdvcmxkIQ==

maybe SGVsbG8sIFdvcmxkIQ==
maybe SGVsbG8sIFdvcmxklQ==
maybe SGVsbG8sIFdvcmxk1Q==
maybe SGVsbG8slFdvcmxkIQ==
maybe SGVsbG8slFdvcmxklQ==
maybe SGVsbG8slFdvcmxk1Q==
maybe SGVsbG8s1FdvcmxkIQ==
maybe SGVsbG8s1FdvcmxklQ==
maybe SGVsbG8s1Fdvcmxk1Q==

I think the next step is to try hooking this up to code which tries to decode the cipher-stream byte by byte, skipping to a new password (or a new Base64 source-string) when the decoded byte goes outside conventional ASCII. (I'm gambling that it'll contain an ASCII message, if it doesn't then it's hard to know if you've successfully cracked it.)

Finally, what happens if we apply this to one of the transcriptions people have made from the Overwatch Summer Games video?

U2FsdGVkXI+vupppZksvRf5pq5g5XjFRIipRkwBOKIY96Qsv2Lm+3IcmzaAILwytX/z66ZVWEQMccfIg+9m5UbuI+sit+A9cenDxxqkIaxbm4cMeh2oKhqIHhdaBKOi6XX2XDWpa6+P5o9MQw==
U2FsdGVkXI+vupppZksvRf5pq5g5XjFRIipRkwBOKIY96Qsv2Lm+3IcmzaAILwytX/z66ZVWEQMccfIg+9m5UbuI+sit+A9cenDxxqkIaxbm4cMeh2oKhqIHhdaBK0i6XX2XDWpa6+P5o9MQw==
U2FsdGVkXI+vupppZksvRf5pq5g5XjFRIipRkwBOKIY96Qsv2Lm+3IcmzaAILwytX/z66ZVWEQMccfIg+9m5UbuI+sit+A9cenDxxqkIaxbm4cMeh2oKhqlHhdaBKOi6XX2XDWpa6+P5o9MQw==
U2FsdGVkXI+vupppZksvRf5pq5g5XjFRIipRkwBOKIY96Qsv2Lm+3IcmzaAILwytX/z66ZVWEQMccfIg+9m5UbuI+sit+A9cenDxxqkIaxbm4cMeh2oKhqlHhdaBK0i6XX2XDWpa6+P5o9MQw==
U2FsdGVkXI+vupppZksvRf5pq5g5XjFRIipRkwBOKIY96Qsv2Lm+3IcmzaAILwytX/z66ZVWEQMccfIg+9m5UbuI+sit+A9cenDxxqkIaxbm4cMeh2oKhq1HhdaBKOi6XX2XDWpa6+P5o9MQw==
..snip..

78,732 variations in total. Uh oh. I'm happy with the output, but whatever I use to go through these, it needs to be able to memoize or somehow reuse whatever progress it can from variant-to-variant, rather than starting over with each new string.

P.S.

A more-realistic number would be 6,561, since we know that the first 1 is good because it's part of the OpenSSL header, and because the letter O is visibly wider than zero.

16Feb/14Off

Chase_fixer project now up on GitHub

Following up on my previous post, I've worked on packaging the scripts up a little more nicely and the result is now available on github.

So far it's handled all the weird cases that I see from my own financial history, but I expect there are a few more oddball scenarios (like wire-transfers or refunds) which may require additional tweaking as time goes on.

Tagged as: , Comments Off
4Feb/14Off

Chase’s malformed transaction records

The problem:

Last month, I tried to import some bank-account records (QFX/OFX formats) into the “You Need A Budget” accounting software, which involves telling it how to recognize certain transactions “groceries” and “gas” etc. This did not go as smoothly as I expected, even for an accounting chore, because many of the payee-name and memo fields had ridiculous values! Manually fixing a lot of scrambled data every month wasn't what I had in mind when it came to simplify my budgeting, so I decided to investigate.

<STMTTRN>
<TRNTYPE>DEBIT
<DTPOSTED>20140120120000[0:GMT]
<TRNAMT>-29.41
<FITID>201401020
<NAME>SAFEWAY  STORE  1234 MYCITY ST
<MEMO>01/20 Purchase $9.41 Cash Back $
</STMTTRN>

I believe whatever steam-powered mainframes JP Morgan Chase uses don't seem to have caught up with the current century: Payee fields and memo fields are combined, truncated, arbitrarily split, and whitespace-trimmed, all presumably as sacrifice to some dark and ancient internal decision that 32 characters (for name) and 32 characters (for memo) were long enough for anybody. Some folks say it's the data-format's problem, but I disagree: Chase's datafile says it complies with OFX v1.02, but if you crack open the spec (dated 1997) it clearly says that at least the memo-field should support 256 characters, not 32.

Examples:

Payee and Memo don't really contain the right thing:

Payee: "Online Payment 1234567890 To Cap"
 Memo: "ital One Bank"

The rest of the memo would have had my cash-back amount (which might be handy in budgeting software) but is truncated:

Payee: "Grocer &amp; Sons Inc. 12345 Exampl"
 Memo: "e road 01/18 Purchase $20.11 Cash b"

The split here occurs between two words, but the whitespace was trimmed! There's no automatic way to know this is "Park lane" vs. "Park lane":

Payee: "Marios Pizza and Plumbing 5442 Park" 
 Memo: "lane NW"

Current progress

Right now I have a series of Python classes  which:

  • Parses the original OFX file(example)
  • Translates it into a much-more-convenient XML file with similar structure
  • Visits every transaction in the XML file and applies custom logic to fix it up
  • Writes the XML file back out as OFX

So far I'm pretty happy with the result: All I have to do is code logic for a few of the common cases, and run the scripts after I download the OFX files from chase. Here's an example of a super-basic statement visitor that just tries to combine Payee and Memo.

    def visitStatement(self, values):
        name = values.get("NAME", "")
        memo = values.get("MEMO", "")

        if len(name) < 32:
            # When the split occurred, there was whitespace which got trimmed, re-add it
            combined = name + " " + memo
        elif len(name) == 32:
            # The split was forced due to some size limit, and we don't really know if there's
            # a space between them or not...
            combined = name + memo
        else:
            pass #TODO warning, larger than ever expected

        values["NAME"] = combined
        values["MEMO"] = "" # No more memo data, it's all inside Payee

From this humble beginning I can branch out into recognizing common patterns (like transfers) and payees and clean up the data appropriately. After generating a new QFX file, the YNAB software seems to handle the longer payee names just fine.

Future work

The biggest problem left is that the data still isn't clean enough: Anything over 64 characters has been lost, and it's not always clear if I need to reinsert whitespace between Payee and Memo. Fortunately, Chase does offer a CSV download, which isn't as useful for importing into accounting applications but does contain the entire original. I just need to find some way to cross-reference between the two, perhaps based on dates, amounts, and some sort of non-whitespace similarity.

Once I have things a little more polished I plan to put them up on Github, but at the moment there are still a lot of hardcoded data-file paths and stuff.

Tagged as: , Comments Off