EsoErik

I'm Erik Hvatum, and you are reading my blog. Now that we know each other, I invite you to learn from my posts. Programming concepts esoteric and sundry interest me, and if you've read this far, likely you.

Jeg Erik Hvatum, og du leser bloggen min. Nå som vi kjenner hverandre, jeg inviterer deg til å lære av mine innlegg. Programmering konsepter esoteriske og enhver interesse meg, og hvis du har lest så langt, sannsynligvis deg.

I noticed that when iTunes writes a track to an iPod, it hashes the track filename. For example, an MP3 on an iPod might be called HQXA.mp3 and located in one of many directories with two letter numeric names. Incidentally, this style of naming files is similar to that used by CCache, squid, and other applications that store a large number of files.

I desired to import numerous such files from an iPod into iTunes. iTunes supports automatic organization of imported tracks by the information stored in each track (such as artist name, album name, track number, disc number). Thus, for tracks imported into iTunes, filenames are irrelevant - provided that those tracks include tag information. Some of the tracks that I wanted to import included no useful information in their tags. The original tack filenames was descriptive and obviously stored somewhere on the iPod - when the iPod played these tracks, it showed their original filenames (minus extension). I browsed through the data files on the iPod and quickly found that iPod_Control/iTunes/iTunesDB appeared to contain track names in little endian UCS-16 or UTF-16 format in addition to a significant amount of other data.

On a flight from Boston to Reykjavik, I wore down my laptop battery exploring the iTunesDB file from an iPod containing the music I wished to recover. Looking at the file in a hex editor, I quickly recognized that the iTunesDB is definitely a database composed of a hierarchical record structure. Records are delimited by plain ASCII strings beginning with "mh"; offsets and sizes are specified as 32 bit little endian integers that are probably unsigned (I would need a 2GiB+ iTunesDB to verify this - mine was just 500KiB, and the DB does not use virtual addressing).

To keep myself occupied as I overcame jetlag upon reaching Norway, I wrote a ruby module and front end application to retrieve original filenames from an iTunes DB and to give hashed filenames their original names. It worked well with my data; that said, I haven't tested it with any other, so be careful if you use my application.


fix_iPod_filenames.rb (the frontend application)

#! /usr/bin/env ruby19
#
# (c) Erik Hvatum, 2009
#

require 'pathname'
require 'FileUtils'
require 'iTunesDB'

if ARGV.size != 2
    abort "Usage: fix_iPod_filenames.rb <directory containing iPod_Control> <directory in which to place renamed music files>"
end

iPodRoot = Pathname.new(File.expand_path(ARGV[0]))
dbPath = iPodRoot + "iPod_Control/iTunes/iTunesDB"
if !dbPath.exist?
    abort "iPod database \"#{dbPath}\" does not exist or is inaccessible."
end
destPath = Pathname.new(File.expand_path(ARGV[1]))
if !destPath.exist?
    abort "Destination path \"#{destPath}\" does not exist or is inaccessible."
end

db = MiTunesDB::CiTunesDB.new
db.open(dbPath)
tracks = db.tracks
db.close
db = nil

tracks.sort! {|l, r| l.location <=> r.location}
tracks.each do |track|
    needSep = false
    dstFn = ""
    doField = Proc.new do |field|
        v = eval "track.#{field}"
        if v != nil
            if needSep
                dstFn << "-"
            else
                needSep = true
            end
            dstFn << v.to_s
        end
    end
    doField.call("album")
    doField.call("discNumber")
    doField.call("trackNumber")
    doField.call("artist")
    doField.call("title")
    dstFn.gsub!(/[*!?\\\/:%]/, "_")
    src = iPodRoot.to_s + track.location.gsub(/:/, "/")
    dst = destPath.to_s + "/" + dstFn[0, 250] + "." + track.format.downcase.gsub(/ $/, "")
    FileUtils.mv(src, dst)
end

iTunesDB.rb (the module used by the frontend application)

# (c) Erik Hvatum, 2009

module MiTunesDB

    # Reads the binary DB written by iTunes Windows to an iPod circa late 2008
    class CiTunesDB
        Track = Struct.new(:title,
                           :location,
                           :album,
                           :artist,
                           :genre,
                           :fileType,
                           :comment,
                           :composer,
                           :grouping,
                           :description,
                           :albumArtist,
                           :format,
                           :trackNumber,
                           :discNumber)
        attr :mhbd
        attr :file

        def initialize
            @mhbd = nil
            @file = nil
        end

        # Opens the iTunesDB specified by filename, loading its contents into memory
        def open(fileName)
            @file = File.open(fileName, "rb")
            @mhbd = Cmhbd.new(self)
        end

        # Closes the iTunesDB
        def close()
            @mhbd = nil
            @file = nil
        end

        # Returns all known tracks as an array of Track structures (defined above)
        def tracks()
            ts = Array.new
            if @mhbd
                @mhbd.children.each do |mhsd|
                    if mhsd.children
                        mhsd.children.each do |sdChild|
                            if sdChild.is_a? Cmhlt
                                sdChild.children.each do |mhit|
                                    t = Track.new
                                    ts << t
                                    mhit.children.each do |mhod|
                                        if Cmhod::Types.has_key? mhod.type
                                            sym = (Cmhod::Types[mhod.type].to_s + '=').to_sym
                                            t.method(sym).call(mhod.str)
                                        end
                                    end
                                    t.format = mhit.format
                                    t.trackNumber = mhit.trackNumber
                                    t.discNumber = mhit.discNumber
                                end
                            end
                        end
                    end
                end
            end
            return ts
        end

        # Base class for objects in the iTunes DB
        class Cmh_base
            attr :addr
            attr :depth
            attr :len
            attr :recordName
            attr :children

            def initialize(addr, db, depth)
                @addr = addr
                @depth = depth
                @len = nil
                @db = db
                @children = nil
                load()
            end

            def to_s()
                indent = "\t" * @depth
                return "#{indent}#{@recordName}:\n#{indent} len: #@len\n"
            end

            protected

            def load(loadLen = true)
                if @db.file.tell != @addr
                    @db.file.seek(@addr)
                end
                recordName = @db.file.read(@recordName.length())
                if recordName != @recordName
                    raise "Invalid record identifier in DB: expected \"#@recordName\" but read \"#{recordName}\"."
                end
                @len = readUInt32() if loadLen
            end

            def loadChildren()
                @children = []
            end

            def readUInt32()
                return @db.file.read(4).unpack("V")[0]
            end

            def children_to_s()
                ret = String.new
                if @children
                    @children.each { |child| ret << child.to_s() }
                end
                return ret
            end
        end

        # Record describing the database.  This is the first record in the database and is located at the beginning
        # of the database file.
        class Cmhbd < Cmh_base
            attr :dbVersion
            attr :dbLen

            def initialize(file)
                @dbVersion = nil
                @dbLen = nil
                @recordName = "mhbd"
                super(0, file, 0)
            end

            def to_s()
                indent = "\t" * @depth
                return super() << "#{indent} dbVersion: #@dbVersion\n#{indent} dbLen: #@dbLen\n" << children_to_s()
            end

            protected

            def load()
                super
                seekTo = @addr + @recordName.length() + 4
                if @db.file.tell != seekTo
                    @db.file.seek(seekTo)
                end
                @dbLen = readUInt32()
                @dbVersion = Array.new(3) { readUInt32() }
                loadChildren()
            end

            def loadChildren()
                super
                childAddr = @addr + @len
                while childAddr < @dbLen
                    @children << Cmhsd.new(childAddr, @db, @depth + 1)
                    childAddr += @children[-1].childSize
                end
            end
        end

        # Record describing a dataset.
        class Cmhsd < Cmh_base
            # Adding childSize to @addr gives the addr of the next mhsd record
            attr :childSize
            attr :childType

            def initialize(addr, db, depth)
                @recordName = "mhsd"
                @childSize = nil
                @childType = nil
                super
            end

            def to_s()
                indent = "\t" * @depth
                return super() << "#{indent} childSize: #@childSize\n#{indent} childType: #@childType\n" << children_to_s()
            end

            protected

            def load()
                super
                seekTo = @addr + @recordName.length() + 4
                if @db.file.tell != seekTo
                    @db.file.seek(seekTo)
                end
                @childSize = readUInt32()
                @childType = readUInt32()
                loadChildren()
            end

            def loadChildren()
                super
                if @childType == 1
                    @children << Cmhlt.new(@addr + @len, @db, @depth + 1)
                end
            end
        end

        # Record describing a track list
        class Cmhlt < Cmh_base
            attr :numChildren

            def initialize(addr, db, depth)
                @numChildren = nil
                @recordName = "mhlt"
                super
            end

            def to_s()
                indent = "\t" * @depth
                return super() << "#{indent} numChildren: #@numChildren\n" << children_to_s()
            end

            protected

            def load()
                super
                seekTo = @addr + @recordName.length() + 4
                @db.file.seek(seekTo)
                @numChildren = readUInt32()
                loadChildren()
            end

            def loadChildren()
                super
                addr = @addr + @len
                @numChildren.times do
                    mhit = Cmhit.new(addr, @db, depth + 1)
                    @children << mhit
                    addr += mhit.totalLen
                end
            end
        end

        # Record describing a track item
        class Cmhit < Cmh_base
            attr :totalLen
            attr :numStrMhods
            attr :id
            attr :format
            attr :trackNumber
            attr :discNumber

            def initialize(addr, db, depth)
                @totalLen = nil
                @numStrMhods = nil
                @id = nil
                @format = nil
                @recordName = "mhit"
                @trackNumber = nil
                @discNumber = nil
                super
            end

            def to_s()
                indent = "\t" * @depth
                str = super
                str << "#{indent} totalLen: #@totalLen\n"
                str << "#{indent} numStrMhods: #@numStrMhods\n"
                str << "#{indent} id: #@id\n"
                str << "#{indent} format: #@format\n"
                str << "#{indent} trackNumber: #@trackNumber\n"
                str << "#{indent} discNumber: #@discNumber\n"
                str << children_to_s()
                return str
            end

            protected

            def load()
                super
                seekTo = @addr + @recordName.length() + 4
                @db.file.seek(seekTo)
                @totalLen = readUInt32()
                @numStrMhods = readUInt32()
                @id = readUInt32()
                @db.file.seek(4, IO::SEEK_CUR)
                @format = @db.file.read(4).reverse()
                @db.file.seek(@addr + 44)
                trackNumber = readUInt32()
                @trackNumber = trackNumber if trackNumber > 0
                discNumber = readUInt32()
                @discNumber = discNumber if discNumber > 0
                loadChildren()
            end

            def loadChildren()
                super
                addr = @addr + @len
                numStrMhods.times do
                    mhod = Cmhod.new(addr, @db, depth + 1)
                    children << mhod
                    addr += mhod.len
                end
            end
        end

        # Record describing a data object
        class Cmhod < Cmh_base
            Types = {1 => :title,
                     2 => :location,
                     3 => :album,
                     4 => :artist,
                     5 => :genre,
                     6 => :fileType,
                     8 => :comment,
                     12 => :composer,
                     13 => :grouping,
                     14 => :description,
                     22 => :albumArtist}.freeze
            attr :headerLen
            attr :type
            attr :strLen
            attr :str

            def initialize(addr, db, depth)
                @headerLen = nil
                @type = nil
                @str = nil
                @strLen = nil
                @recordName = "mhod"
                super
            end

            def to_s()
                indent = "\t" * @depth
                str = super
                str << "#{indent} headerLen: #@headerLen\n"
                str << "#{indent} type: #@type\n"
                str << "#{indent} strLen: #@strLen\n"
                str << "#{indent} str: #@str\n" if @str
                return str
            end

            protected

            def load()
                super(false)
                @db.file.seek(@addr + @recordName.length())
                @headerLen = readUInt32()
                @len = readUInt32()
                @type = readUInt32()
                @db.file.seek(12, IO::SEEK_CUR)
                @strLen = readUInt32()
                if @strLen > 0 && @strLen % 2 == 0
                    @db.file.seek(8, IO::SEEK_CUR)
                    begin
                        str = @db.file.read(@strLen).force_encoding("UTF-16LE").encode("US-ASCII")
                    rescue
                    else
                        @str = str
                    end
                end
            end
        end
    end

end

Labels: coding, iPod, iTunesDB, ruby

OS X 10.5.7 includes ruby 1.8.6 with a number of outdated gems. When updating, the gem command line program installs newer gems alongside the outdated ones. To tidy up, gem supports the "cleanup" command; this would normally blow away all gems that are obsoleted by newer installed gems. However, because the gems included with OS X are installed to a path that is not the default installation location for new gems, "cleanup" fails.

In order to work around this, I wrote a ruby script that:

• Calls gem cleanup in dry run mode to identify those gems that are outdated and should be deleted.
• Calls gem list in details mode to determine the installation path of the outdated gems.
• Calls gem uninstall on each gem targeted for deletion, providing the path to the gem when it does so.
  

On my laptop, this uninstalled all the obsolete gems, just as "gem cleanup" would have if it had worked correctly. A caveat: I call "gem uninstall" with the no executables parameter as the bin directory in the Apple gem directory did not exist for me. Newer gems replace the frontend scripts to these binaries, and I don't know where the old binaries actually dwelt. Hopefully they were in directories of the uninstalled gems and were therefore deleted. If not, they are evidentally not in the default shell path and merely waste a little disk space.

#! /usr/bin/env ruby
#
# (c) Erik Hvatum 2009
#

require 'rubygems'
require 'optparse'
require 'open4'

#* Parse command line options

options = {}
optparser = OptionParser.new
optparser.banner = "Usage: #{$0} [options]"

options[:dryrun] = false
optparser.on('-d', '--dry', 'Dry run; print the calls to gem that would be executed') do
    options[:dryrun] = true
end

options[:gemexec] = 'gem'
optparser.on('-g', '--gem EXECUTABLE', 'Specify gem executable name; ie, gem19 or /usr/local/bin/gem19') do |gem|
    options[:gemexec] = gem
end

optparser.on('-h', '--help', 'Display help information') do
    puts optparser
    exit
end

options[:verbose] = false
optparser.on('-v', '--verbose', 'Output more information') do
    options[:verbose] = true
end

begin
    optparser.parse! ARGV
rescue OptionParser::ParseError => e
    abort e
end

if ARGV.size > 0
    abort "invalid option: #{ARGV[0]}"
end

#* Make list of outdated gems to remove

GemInfo = Struct.new(:nameAndVersion, :name, :version, :path)
gemsToUninstall = []

gemCleanDryCmd = "#{options[:gemexec]} cleanup -d"
if options[:verbose]
    puts "To determine which gems are outdated and should be removed, executing shell command: #{gemCleanDryCmd}"
end

pid, stdin, stdout, stderr = Open4::popen4(gemCleanDryCmd)
ignored, status = Process::waitpid2(pid)
if status != 0
    abort "Error: #{gemCleanDryCmd} returned #{status} instead of zero, with stdout output\n#{stdout.read.strip}\nand stderr output\n#{stderr.read.strip}"
end

l = stdout.readline()
unless l =~ /^\s*Cleaning up installed gems\.\.\.\s*$/i
    abort "Error: First line of #{gemCleanDryCmd} stdout is not \"Cleaning up installed gems...\""
end

while 1
    l = stdout.readline()

    if l == nil
        abort "Error: Premature end of output from #{gemCleanDryCmd}"
    end

    if l =~ /^\s*Clean Up Complete\s*$/i
        break
    end

    if l =~ /^\s*Dry Run Mode: Would uninstall (\S+)\s*$/i
        if options[:verbose]
            puts "Found obsolete gem to uninstall: #{$1}"
        end
        gemsToUninstall << GemInfo.new($1)
    else
        abort "Error: Failed to parse output from #{gemCleanDryCmd}: #{l}"
    end
end

for gem in gemsToUninstall
    if gem.nameAndVersion =~ /^(.*)-((?:\d+\.?)+)$/
        gem.name, gem.version = $1, $2
    else
        abort "Error: Failed to parse gem name: #{gem.nameAndVersion}"
    end
end

#* Get paths of identified gems

for gem in gemsToUninstall
    cmd = "#{options[:gemexec]} list -d -l #{gem.name}"
    if options[:verbose]
        puts cmd
    end
    pid, stdin, stdout, stderr = Open4::popen4(cmd)
    ignored, status = Process::waitpid2(pid)
    if status != 0
        abort "Error: #{cmd} returned #{status} instead of zero, with stdout output\n#{stdout.read.strip}\nand stderr output\n#{stderr.read.strip}"
    end
    o = stdout.read()
    unless o =~ /Installed at /
        abort "Error: failed to parse output of #{cmd}"
    end
    while $' =~ /\s*\(((?:\d+\.?)+)\): (.*)$/
        if $1 == gem.version
            gem.path = $2
            break
        end
    end
end

#* Perform removal

for gem in gemsToUninstall
    cmd = "#{options[:gemexec]} uninstall -I --no-executables
         --install-dir #{gem.path} --version \"=#{gem.version}\" #{gem.name}"
    if options[:dryrun] || options[:verbose]
        puts cmd
    end
    unless options[:dryrun]
        system(cmd)
    end
end

Labels: coding, ruby