Trendy Stephen

Save Flash Video under OS X (firefox)

2011-09-25T21:30:00.001-07:00

Once upon a time, in Linux, it was possible to copy the files being downloaded and played by the Flash plugin out of /tmp and to wherever for later viewing. I was pretty sure this was possible under OS X. And wouldn't you know it, it is. But the files were harder to find. They're stored somewhere under /private/var/folders. I did this to locate them:


find /private/var/folders/ -type f -exec file {} \; 2>/dev/null | grep Macromedia

To figure out the top-level folder that contained them, a google query showed me this thread which clued me in. I don't know if someone pointed out the exact location later (tl;dr) but when I saw the bits about /private/var/folders I did a find in a terminal.

Removing Linux from OS X (to redo Bootcamp)

2011-09-24T06:50:00.000-07:00

I wanted to remove Linux and try Windows (for some steam games) on my MacBook Air, but I had a little trouble. Using the Disk Utility (graphical program), I was only able to merge the existing JHFS+ file system with my Linux partition, but It wouldn't touch the swap. You can't just launch Boot Camp at this point because it will see the multiple partitions and refuse to run.

I tried to merge using the diskUtil command line, but it gave me the error: Merging partitions encountered error "Couldn't read partition map (-69876)".

So, I erased the volume and replaced it with a HFS+ with journaling. Then I launched the Disk Utility and it refused to let me merge them. For posterity, this is the command I used to erase:
diskutil eraseVolume JHFS+ new /dev/disk0s3

The result is that I had two Journaled HFS+ partitions that still couldn't be merged, but once I erased the volume from within the Disk Utility, it let me grow the remaining partition to fill the space. Boot Camp happily ran and created a new Windows Partition after that.

Fuck You, OS X!

2011-07-25T22:14:00.001-07:00

The joys of a commercial operating system! I was happy to play Diablo 2 on my shiny new MacBook Air. No, not one of the cool new-new MacBook Airs, the slightly older MacBook air that still has discreet graphics. Anyway, I decided to finally bite the bullet and pay the $5 for the latest Xcode. I was happy to discover that it's now free. Then I noticed that it was free for Lion, only. So I paid the $30 and upgraded. Then I discovered that I couldn't play Diablo 2 anymore. Then I discovered that I probably could have gotten the upgrade to Lion for free, through Apple, because I bought my system so close to the release date of Lion.

Well, bitterness aside, I decided to grab wine for OS X and use that to launch a windows install of Diablo 2. Then once that was working, I decided to package it up in a .app file so that I could at least pretend it was a native app. After that, I discovered there's something called wineskin that will let you package stuff up into .app bundles. Eh, whatever, I already made the damn script, put the right files in the right places, and through some guesswork figured out how to get it to all play nicely. Then I tweaked my script so that it would search for any old wine installation within the bundle, in case I upgrade the version of wine it's using. Then I tweaked the script a bit further to have it configure the 800x600 desktop so that the game would stop fucking up my graphics and displaying incorrectly in the process. It was weird, the game would launch, it looked great, but the window was getting pushed off the screen, ever so slightly. The emulated desktop fixed that.

Since I had it configure wine, I decided to remove the stupid gecko prompt for HTML support in windows applications. Maybe battle.net makes use of the HTML display, I don't know, the game certainly didn't need it for multi-player.

Anyway, here's the bizarre stuff that I learned in the process of creating the .app. I knew that .apps were folders, but I didn't know what the structure needed to look like, so I took a peek at Torchlight. Nice and simple. I mimicked the directory structure so that I had the following:

Diablo2.app/Contents/MacOS
Diablo2.app/Contents/MacOS/Info.plist
Diablo2.app/Contents/MacOS/diablo2.icns  (stole from somewhere)
Diablo2.app/Contents/MacOS/diablo2.sh    (script I wrote)
Diablo2.app/Contents/Resources
Diablo2.app/Contents/Resources/diablo2.icns ->
 ../MacOS/diablo2.icns
Diablo2.app/Contents/PkgInfo
Diablo2.app/Contents/Info.plist -> ./MacOS/Info.plist

When it was all said and done, I had the following additional items:

Diablo2.app/Contents/MacOS/1.3.24     (wine distro)
Diablo2.app/Contents/MacOS/Diablo II  (from windows)
Diablo2.app/Contents/MacOS/prefix     (generated)

[Edit: Here's where I got the icon set: http://www.iconeasy.com/icon/diablo-ii-icon/ No idea where they got it from, so I don't know that this is really proper attribution.]

So, here's the cool part. Parameter Expansion. The diablo2.sh I created was modeled after the Steam.app steam.sh script. And when I say modeled after, what I really mean is that I read the first executable line of the script and said, "What the hell is this?"

Here's the line in question:
STEAMROOT=$(cd "${0%/*}" && echo $PWD)

I gathered that PWD was feeding the PWD to the variable STEAMROOT, so I figured the ${0%/*} must be doing something crazy to $0. So the first thing I did was start issuing echo commands and trying things with different variables. Now, I could see that the ${0%/*} was taking whatever the value of the variable was and chopping off the trailing file name but I didn't understand what sorcery was allowing this. Turns out bash has some built-in awesomeness that I just didn't bother to notice in the 10 years or so that I've been using it. I'm so so sorry for ignoring you, bash! If you look at the man page for bash, it will explain (among other things) ${VAR%pattern}, ${VAR#pattern}, and ${VAR/pattern/substitute/}. They work much like you can imagine. The ${VAR%pattern) removes the pattern, starting its search from the right side of the value and searching towards the left, going for the least-greedy approach. The ${VAR#pattern} similarly removes the pattern, but starting from the left. The ${VAR/pattern/substitute} is something that I've needed ever since I started using Linux back in high school. It's like a sed expression. It replaces the specified pattern with the specified substition. It's all so bloody awesome.

Up until now, I had been doing this to get the directory part of a file name:
DIR="`echo $FULLPATH| rev | cut -f1- -d/ | rev`"

I just needed to do this:
DIR="${FULLPATH%/*}"

I cannot count the number of times that I've shelled out to imagemagick to process a directory and have done something foolish like this:
for f in *.jpg ; do NEW_NAME="`echo $f | rev | cut -f2- -d'/' | rev`.png" convert "$f" "$NEW_NAME" ; done

And I could have just done this:
for f in *.jpg ; do convert "${f}" "${f/\.jpg/.png}; done

So, getting back to the Diablo stuff. I don't know why the steam.sh does the cd and pwd when it could just evaluate the variable, but whatever; maybe they use a side-effect later. Here's my diablo2.sh file:

On to the other package details...

Info.plist, ties everything in the package together:

PkgInfo only contains this line:

APPL????

Here's a crazy script I threw together, simply invoke it and it will base64 decode the last line of it's content, reconstitute a tar.bz2 file, md5 it for integrity, and if all went well, it will extract a copy of my Diablo2.app folder, minus the wine and minus my Diablo 2 install folder. Just get any one of the pre-built wine for OS X from here: http://wine.playonlinux.com/darwin-i386/ and extract it into the MacOS folder of the Diablo2.app. Then, copy in your Diablo II from a windows install, and the script will locate all the pieces and launch. Be prepared for sounds, because the script speaks its status messages.

OS X Right Click Menu Additions

2011-07-03T07:25:00.001-07:00

I'm sure this is documented somewhere, but I didn't find it while looking. If you use the Automator to create a service that receives files or folders in any application, it will get added to the right click menu. Now I can right click documents and choose to edit them with Vim.

I noticed this while trying to add a hotkey to the service. The hotkey didn't work, but then I noticed my edit option in the menu, so that's almost as good.

So here's how I created a right-click option for Edit with Vim under Snow Leopard:

Launch Automator

Choose The Service Template

Change "Service receives selected" pulldown option from "text" to "files or folders."

Drag "Run Shell Script" from the library into the current workflow

Change the "Pass input" pulldown from "to stdin" to "as arguments"

Delete the text from the edit box and enter the following:

Save it as something concise, but descriptive, like: Edit with single Vim

This time I embedded everything for the command in the shell script of the service because I'll only ever need to call this from the gui.

Edit: Update: OS X Lion treats Applications like services. If you create an Application with the Automator, it will likewise show up in your services menu.

OS X DMG Creation from a Folder

2011-07-02T20:12:00.000-07:00

Edit: As of OS X Lion (10.7) Applications show up in your services menu, so you get a right click menu for free.

The hdiutil command is awesome. Amongst its many features is the ability to create a DMG file from a specified folder. For some reason, there's no right click option for creating DMG files from folders. Strangely enough, there is a right click compress option that will create a zip, but that's really not sufficient. If you create a zip, you can't access individual files from it like you can a compressed DMG.

Edit: For anyone who just wants the command and not fuss with the scripts, here's how you might create a zip compressed dmg file from a folder called Movies:

hdiutil create -srcfolder Movies -volname Movies -fs HFS+ \
 -fsargs -c c=64,a=16,e=16 -format UDBZ ./Movies.dmg

Once again, I've used Automator to invoke a bash script. Now I can drop folders on my app and generate a compressed, read-only DMG file. I don't have lots of free space, so I use this for applications I've downloaded that come as a tar.gz/tar.bz/zip/whatever.

I'd like to point out that while it is possible to embed all of the bash script content into the Automator "Run Shell Script" feature, but then I wouldn't be able to easily invoke it from the command line and I use the command line alot.

Here's the content of my Automator "Run Shell Script" box (CreateDMGFromFolder.app):

Here's the content of my shell script (CreateDMGFromFolder):

Edit: I've updated the script in light of reading the bash man page. Parameter Expansion is bad-ass.

Here's the new shell script to replace the aforementioned (CreateDMGFromFolder):

OS X Open With Vim

2011-07-01T20:39:00.000-07:00

Edit: After my second OS X post, I went back and created a service to perform Edit with Vim. I actually forgot about the .app that I made with automator. Then I upgraded to Lion and was all sorts of confused when two entries for Edit with Vim popped up in my context menu. Wouldn't you know it? Lion puts all automator Services and Applications into the right click context menu. So I've actually deleted this script and am using the one mentioned here.

OS X comes with vim preinstalled. Nice. It's not that retarded version Ubuntu comes with by default, either. It's the full vim. So then, why the hell can't I right click something, choose open with, and then choose vim? Why can't I pass arguments to the terminal to tell it to run arbitrary commands (such as vim) from the open with dialog?

It appears that OS X only lets you pick Apps as the target application for performing an "Open with". So, to get what I want, I have to use Automator in conjunction with a bash shell script in order to run osascript in order to pop up a new terminal with vim in it and whatever files are selected (or dropped on the app).

Here's what I have in my bash script, aptly named TellTerminal:

This script sends whatever it's arguments are to the Terminal.app and tells it to run them. Then it tells the Terminal.app to activate itself so that it jumps to the foreground. Without this, the terminal will still open, but there's a chance that it will be sitting in the background somewhere. A really, really good chance.

To invoke this script, I had to create an Automator Application (not a workspace, only the .app can be invoked as an application) and add an action 'Run shell script'. This is the shell script that it runs:

This script (which only exists in the Automator app that I've named EditWithVim.app,) loops over all the arguments that were passed to it in order to invoke vim and pass it the quoted file names. This is to avoid having a space within a file name break my script. Also, this script looks for a BashScripts directory in my user's Library directory within the user's home.

Now I can set things to be opened with vim. It's extremely convoluted, but it does seem to be working.

Thanks, FCC!

2011-04-20T19:37:00.000-07:00

Apparently you can just download the FCC database of Ham users. Then you can just grep the list instead of doing something lame like scraping a 3rd party website. Thanks, FCC.

Link here.

Ham Lookup Bash Script (fragile)

2011-04-17T20:08:00.000-07:00

Earlier today I wanted to quickly find some info on some Hams, but I couldn't remember the FCC link. Then I found the QRZ site. Not bad, but I didn't want to have to launch Firefox just to look someone up, so I thought maybe I'd write a python script. Except then I couldn't think of an easy Regex to parse the HTML So I fetched the page source, found the form that looks up names and callsigns, pulled down my own info, and before I knew it, I had a series of ugly greps and seds pulling out the relevant information. Then something strange happened. I lost interest in pursuing the python angle. I dumped my two queries into two separate bash scripts named lookupname and lookupsign. It was quick and painless, even though the "code" is horribly fragile and entirely dependant on the HTML coming back, but it (currently) works so I'm calling it good for now.

A few minutes ago I was sitting on a different computer and wanted to lookup a callsign, again, so I dumped the two separate bash scripts into a single script and I guess I'll share that in case I want it again in the future. Here's the script:

Here's the same script. Invoke it with either a callsign or a person's name. Basically, if you want to do a callsign lookup make sure there are no spaces in the invokation, ex: lookupham AB2CDE. If you want to do a name lookup then make sure you provide a first and last or if you're clever make sure there is a space in the parameter list. ex: lookupham John Doe.

vim + xxd = easy binary patching

2011-03-27T09:55:00.001-07:00

I know this is somewhat common knowledge and can be found elsewhere, but it's just so awesome that I have to repost it. I had always used xxd to generate a listing, edited that listing with vim, and then used xxd -r to re-create the binary file. Instead, you can use vim to open the binary file, shell out to xxd using the current buffer as input, modify the binary content by editing the hex digits (that's right, no need to edit the ascii, that's just for show,) and then shell out to xxd with the current buffer again to get the binary representation.

Once more for posterity...

vim -b binaryfile

:%!xxd

Edit the hex column in the center.

:%!xxd -r

Save the file and it's patched.

For some extra goodness (prettier columns):
set syntax=xxd

So, that was pretty cool, but also check out :help xxd. Vim's built-in help has a nice bit of code to put in your .vimrc and then vim -b will automatically handle shelling out to xxd when you open a binary file and then shell out to xxd when you save the modified buffer.

Revised Rich Header Parser

2011-03-15T20:40:00.000-07:00

I've revised the rich.py script. It's much better but doesn't actually do anything more than it used to. I'll be making some more additions in the very near future, but I wanted to post this before I misplaced it.

New script here: rich.py

Fujitsu Lifebook B3020D Touchscreen and Linux

2011-01-08T18:25:00.000-08:00

(I actually did this at the tail end of 2009, then I revised the python script it in 2010. I've been thinking of attacking the problem with un-smooth lines when the cursor is dragged...)

A friend gave me one of these Fujitsu B3020D Lifebooks. It's quite beat up. No, really, the screen is cracked in the lower left corner and a failing hard drive or bad RAM requires that I reboot the thing periodically. It's currently running Windows 7 and Skype with a Zoom adapter so I can plug cordless telephone bases into it. But before that, it was running Ubuntu. It ran Ubuntu for a good week to a week and a half. The touchscreen didn't work on it, and the screen was cracked even then, but it wasn't a bad little system. The battery life was good, the wireless worked without having to fight with it, and I thought it would be fun to reverse engineer the touchscreen and write some kind of mouse driver.

It didn't take me long to see that the touchscreen is actually exposed to the system as a serial device. That's damn cool. If you just start cating out the serial ports at the console and then touching the screen, you'll eventually see it spew a bunch of numbers. That's how I figured out which serial port it was using. After that you can pretty much tell what the values mean just by trial and error. There's a byte that indicates the type of event (push/release/drag) and then a series of bytes indicating the x and y coordinates. I suggest piping the output of cat through xxd in order to see the byte values in hex. For example: cat /dev/ttyS2 | xxd -

I don't recall from memory what the different byte values were, but you can kind of tell from the python script I wrote for it. This script reads the touch screen and will tell X11 to move the mouse around the screen and click the left mouse button in pretty much the way you would expect it. However, if you run the gimp or some other paint program, you'll notice that touching and dragging results in a rather imperfect line. Instead of nice diagonal lines, you get these crazy little semi-circles. If I ever get a second system set up to run my Skype phone (or can borrow another one of these laptops,) I'll revisit the script and see if I can fix it, but otherwise it works okay.

Here's the script via pastebin: touchmy.py

SEN-08634 Magnetic Card Reader/Writer

2011-01-08T17:44:00.000-08:00

The SEN-08634 Magnetic Card Reader/Writer is exactly what it sounds like: it reads and writes to magnetic strip cards. SparkFun has them.

This device will connect to your host as a serial device (/dev/ttyUSB0, for example) but if you try to cat it out, it will immediately return. The damn thing doesn't buffer! Well, okay, that's not true, it will buffer, but it throws the buffer away after a very short period of time. So, if you wanted to use the device with some clever bash scripting, you would have to do something like while [ "1" ] ; do echo -n `cat /dev/ttyUSB0` ; done

I didn't actually try that command while I had the device, it didn't occur to me until just now.

Anyway, I borrowed one of these and wrote a python script that can use it. It reads high coercivity cards and reads and writes low coercivity cards, but that's about the only thing that works. For some reason, all the commands that the documentation claims should work don't return anything. My script is heavily commented and I don't recommend simply running it unless you're ready to lose the data on whatever card you swipe through it. The main body of the script will attempt to write hello world to track 1 and then a couple strings of numbers to track 2 and 3. It will not backup what's on the card before writing. That didn't occur to me either. I don't have the device any more, so I'm not going to update the script.

Here's the script via pastebin: magstripe.py.
If you want to see documentation for it, you can do something like this: doxygen -g magstrip.doxy ; doxygen magstrip.doxy

Diablo 2 Cloner GUI

2009-11-13T20:53:00.000-08:00

I threw this together just for practice, though not very good practice since it spits most of the output to the terminal... It uses GtkBuilder under python so it requires the Gtk runtime in addition to python.

This is the file to run: d2gui.py
Here's the file that does the actual work:d2functions.py
And here's the interface xml document: d2gui.glade

Wordpress is no better

2009-11-13T20:52:00.001-08:00

Wordpress didn't turn out to be any better. Perhaps if I had my own domain it would be more configurable, but I'm far too lazy for that.

So now I have both blogs and I'll have to make a few more posts here to see which one I want to keep. After that I'll consolidate my posts to one location and tear down the old one.

So long and thanks for all the fish

2008-02-23T12:49:00.000-08:00

Blogger was a pain. I'm still not satisfied with the layout and complete lack of ability to upload files. I'm going to try wordpress for a while. If google ever fixes things, maybe I'll return. Or, if it turns out that WordPress is even worse, I might return.

WordPress Blog

Rich Header

2008-01-14T00:57:00.001-08:00

The Rich header is that bit of data in a Portable Executable (PE) File^[1] that follows the DOS Stub, but preceeds the actual PE Header. It's format is only documented in a handfull of places on the searchable net, and that documentation isn't terribly in-depth.^[2] This post tries to shed some light on the Rich Header, but I fear it poses more questions than it answers.

While looking into the format of the PE Header, I became curious about an odd collection of consistent data in PE files produced by Visual Studio. It always began with the word Rich, and it wasn't in any of the documents made public by Microsoft. After spending some quality time with google, I found a bunch of references to the Rich header, but nothing described it in depth. Most just said it wasn't used by Windows and could be safely removed to save a few bytes. So I gave up.

Later on, I was reading something about anti-disassembling that discussed some fun modifications that one could do to a PE File. I don't recall where I saw this document^[3], as it occurred quite some time ago, but it basically dealt with crashing various debuggers and disassemblers. At one point, this document talked about how the Rich header had been used in some court case to attribute a virus to its creator. That was all the document said, and THAT got my attention.

Some additional quality time with google eventually led me to the forum posting^[2] by lifewire, who I'm assuming is some russian hacker or something. It described, with a couple of confusing typos, a little bit more information about the Rich header. In fact, most of what will be described here can be ascertained by reading his rich.txt file. (I'd post that file, but I haven't figured out how to put up attachments.) Unfortunately, I couldn't get the details through my thick skull without taking his document and firing up a free copy of IDA Pro^[4] to disassemble the link.exe file provided in Microsoft Visual Studio .Net 2003.

Using IDA Pro and frequently referring to lifewire's rich.txt file, I learned that the Rich header begins with a marker, a checksum that is repeated three times, some encoded values, the word Rich in ascii, and finally ends with the checksum value, again.

The encoded values are "comp.id" fields from Visual Studio. I don't entirely understand them, and they're really key in making the Rich header forensically useful. For every library file that is linked against, for every object file that is compiled in, and for every tool in Visual Studio that touches data in the build process, there is a compid that is recorded.^[5] Some of the Visual Studio tools have an associated compid. These compids are encoded along with an occurrence count. So the format of the encoded data in the Rich header is compid XOR checksum followed by the number of times that compid is referenced XORed with the checksum. That is, the compid is XORed with the checksum and then the occurrence count is XORed with the checksum, and then these values are written to the header.

The checksum itself is computed by iterating over every byte of the DOS Header, skipping the elfanew field, copying the byte into a 32-bit field, rotating the field left by the field's offset in the PE header, and then adding that to a sum. But that's not all. Next, that sum is then added to each compid XORed with its occurrence count. That is, the 32-bit sum is added to the value resulting from compid XOR occurrence, for every compid in the list.

The last detail I have is that the marker that appears at the beginning of the Rich header is the checksum value that was computed XORed with 0x536e6144. (Note that it's DanS in ascii; another name to put with MZ and Rich.)

So what does the Rich header tell us about the executable? It can tell us how many different object files, libraries, and tools where used to create the executable. What additional information could it tell us? If one were to create a table of compids and the corresponding components in Microsoft's tools, the Rich header would tell us what tools were used to create that executable. Additionally, by verifying the checksum, one can make guesses about whether the PE file were modified. I suspect that a table of components used to build the PE might provide enough details to extrapolate whether the executable itself were modified manually by testing to see if the object files and tools that were specified by the compids actually left a mark on the rest of the PE file.

In closing, here's some python (forgive it's rough edges, I'm new to python) that parses compids from PE files: rich.py via GoogleDocs

EDIT: here's a revised one via pastebin: rich.py

Sources:
[1] PE Coff Format Spec
[2] Forum Posting of rich.txt by lifewire
[3] If anyone recognized this description, please post a comment!
[4] IDA Pro Freeware Download
[5] This was speculation incorrectly included as though it were fact -- and it's false