I was wrong.
That’s not something anyone likes to admit, but when I’m wrong, I’m wrong, I’ll admit it.
Some quick background for any readers who are experiencing my personal saga with Macintosh Finance software for the first time. Back in February of 2010, I was terribly frustrated with the direction that Intuit was taking with Quicken for Macintosh, or the lack thereof. I vented that frustration in my article titled, Quicken Essentials 2010 for Mac. Why Bother? I fired up the Macintosh Quicken community. So much so that I got the attention of senior people inside the Intuit organization. I ended up interviewing Aaron Patzer, the GM/VP of Personal Finance at Intuit, and the founder of Mint.com, which had just been acquired by Intuit. You can read about that interview in my article Intuit Responds – Quicken Essentials 2010 for Mac. That was when I found iBank. I was looking for alternatives to Quicken, and iBank seemed like a good choice. I wrote a review of iBank 3: iBank – Your Quicken Alternative. Finally, I had a decision to make: Quicken Essentials or iBank 3? You can read my conclusion in the article Quicken Essentials for Mac – The Bare Minimum. I recommended that previous Quicken users go ahead and upgrade to Quicken Essentials, which I did. That’s where I was wrong.
This decision has been a year in the making.
Goodbye Quicken, Hello iBank
There’s only one problem. Intuit, with Quicken Essentials 2010 for Macintosh, has stolen your data. They are preventing you, once you’ve converted to Quicken Essentials, from ever exporting your data again for use in any other software, including their own! (no exporting into TurboTax anymore) Intuit is locking you into the Quicken family and there is very little reason why they would ever provide a way back out, other than consumer demand.
Until now.
I’ve spent the last three weeks doing just that. Moving out of Quicken Essentials and into iBank. It wasn’t easy for me, but it will be a lot easier for you, thanks to some programming magic. Some background on why this is so difficult. I’ve eluded to it in my previous article The Dirty Little Secret in Quicken Essentials…You Lose Your Data, but I want to take a moment to just outline how complex this undertaking was for me. First, Intuit only allows for exporting a single account at a time, and then, only in a “CSV” format. I’ve put “CSV” in quotes, because what Intuit exports isn’t even valid CSV (Comma Separated Values) format. If anyone at Intuit is reading this, please fix the format, it would make it so much easier. Worse than exporting invalid CSV, they’ve also filled the file with lines of data that were not actually in the source file. Things like the type of file, dates, subtotals, totals, balances, filter criteria, basically garbage. The export they perform is more of a Register Report than an actual export of transactional data. That makes it very difficult to programmatically parse. I’m sure that was the whole point of them doing it, but it frustrated me. Finally, they actually fail to export certain key pieces of data. Do you have a split transaction? Does that split have a parent transaction that may have a different Memo from each of the child transactions? Too bad, it’s gone. Quicken Essentials only exports the child transactions in a split, not the parent. I have to re-build the parent based on the totals of the children. Even worse, Quicken Essentials might take the parent transaction Memo and put it on one of the children if that child transaction doesn’t have a Memo, it’s crazy!
Okay, so you know this was hard. Fortunately for you, and me, it works. Let’s get started moving you from Quicken Essentials to iBank 4.
Exporting from Quicken Essentials
The first step in exporting your Quicken Essentials data is to take stock of where you are in the process. I converted all of my Quicken 2007 data into Quicken Essentials. That amounted to 15 years of financial data. I have 78 different accounts and over 20,000 transactions. I did not want to export those a single account at a time. So, my first step was locating my old Quicken 2007 data file, which I did (thank you iDisk!). I then fired up Quicken 2007, opened the old file, and exported the whole thing as a QIF format file, then imported it into iBank 4. This is all a relatively straight-forward exercise, because back when Quicken 2007 came out, Intuit actually supported Macintosh users, sort of. Hopefully you either have your original Quicken 2007 (or earlier) data file or you have the QIF export you made when you converted to Quicken Essentials. If you were a new user to Quicken Essentials and only have about a year’s worth of data in Quicken Essentials, then you can just follow these instructions. If you’re using an old QIF file, go ahead and import that into iBank 4 to get yourself established there. The rest of these instructions assume you have a mixture of data in each account that you want to move into a valid QIF file for importing elsewhere.
Step 1
Select the account from which you wish to export transactions from the left-hand navigation menu in Quicken Essentials. Then set the Transactions being viewed to All Dates:
Select the transactions you wish to export. You have two choices here: one, select only the newer transactions if you have already imported some of them into your other program; or two, skip this step entirely to export all of the transactions available in the account.
Step 3
Export the transactions by selecting Export from the File menu. During this step, for reasons unknown to everyone, the Export menu option will only be available if you have a transaction selected, regardless of which account you have selected. If you’re exporting the whole account, just select any single transaction. If you’re exporting just certain transactions, then you already have them selected.
Set the options for the transaction export. If you are exporting the entire account, make sure it says All visible transactions. If you are only exporting your selected transactions, make sure it says Selected transactions only. Set your export location and file name. Unfortunately, you cannot select any other format for the export besides CSV (Comma Separated Value), even though Intuit made this a drop-down box, no idea why. Click Save when you’re ready to perform the export.
Enter qecsv2qif.pl
And now without further ado the source code for qecsv2qif.pl:
[perl collapse=”true” gutter=”true”]
!/usr/bin/perl
qecsv2qif.pl – Convert Quicken Essentials 2010 for Mac CSV output to valid QIF file
Copyright (C) 2011 Robert A. Pickering Jr.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see http:</http:>.
use strict;
use warnings;
use Getopt::Std;
use Switch;
use Text::CSV;
use Tie::File;
use Tie::Handle::CSV;
$Carp::Verbose = 1;
$|=1;
Declare basic variables
use vars qw/$VERSION $VERBOSE $csv $line %opt/;
Delclare subroutines
sub processSplit;
set up usage info
$VERSION=”1.0″;
sub main::VERSION_MESSAGE { print $0.’, Version ‘.$main::VERSION.”n” }
sub main::HELP_MESSAGE { print “Usage: $0 -f
$Getopt::Std::STANDARD_HELP_VERSION = 1;
Accept the following options: v, w, c, f, and t – f and t require parameters
getopts(‘vwcf:t:’,%opt);
$VERBOSE = $opt{v} ? 1 : 0;
Print GPL Warranty information with -w, then exit program
if ($opt{w}) {
print “15. Disclaimer of Warranty.n”;
print “THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.n”;
print “EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIESn”;
print “PROVIDE THE PROGRAM ÒAS ISÓ WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED ORn”;
print “IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITYn”;
print “AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCEn”;
print “OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COSTn”;
print “OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.nn”;
print “16. Limitation of Liability.n”;
print “IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANYn”;
print “COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM ASn”;
print “PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL,n”;
print “INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USEn”;
print “THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDEREDn”;
print “INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAMn”;
print “TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEENn”;
print “ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.nn”;
print “17. Interpretation of Sections 15 and 16.n”;
print “If the disclaimer of warranty and limitation of liability provided above cannot ben”;
print “given local legal effect according to their terms, reviewing courts shall apply localn”;
print “law that most closely approximates an absolute waiver of all civil liability inn”;
print “connection with the Program, unless a warranty or assumption of liability accompaniesn”;
print “a copy of the Program in return for a fee.n”;
exit;
}
Print GPL Distribution information with -c, then exit program
if ($opt{c}) {
print “4. Conveying Verbatim Copies.n”;
print “You may convey verbatim copies of the Program’s source code as you receive it, in anyn”;
print “medium, provided that you conspicuously and appropriately publish on each copy ann”;
print “appropriate copyright notice; keep intact all notices stating that this License and anyn”;
print “non-permissive terms added in accord with section 7 apply to the code; keep intact alln”;
print “notices of the absence of any warranty; and give all recipients a copy of this Licensen”;
print “along with the Program.nn”;
print “You may charge any price or no price for each copy that you convey, and you may offern”;
print “support or warranty protection for a fee.n”;
exit;
}
Required parameters
die “You must specify your
unless $opt{f};
die “You must specify the account type being imported with -t.n”
unless $opt{t};
Output Licensing Information
print “qecsv2qif.pl Copyright (C) 2011 Robert A. Pickering Jr.n”;
print “This program comes with ABSOLUTELY NO WARRANTY; for details use -w’.n”; print “This is free software, and you are welcome to redistribute itn”; print “under certain conditions; use -c’ for details.n”;
Programatically determine output file name (same as input but .qif
my $outfile = substr($opt{f}, 0, rindex($opt{f}, ‘.’)) . “.qif”;
The first several lines of the Quicken Essentials CSV format are useless
Unfortunately, the number varies based on the Export done, so I go find the
header row, move it to the top of the file, and overwrite the old header line.
tie my @array, ‘Tie::File’, $opt{f} or die “$opt{f} $!”;
for (my $i = 0; $i # Header line is on line 0, only added so I can do multiple runs on same file.
last if ($array[0] =~ m/Payee/);
Header line not on line 0
if ($array[$i] =~ m/Payee/) {
switch ($opt{t}) {
case “Cash” {$array[$i] =~ s/Receive/Deposit/;$array[$i] =~ s/Spend/Payment/;}
case “Bank” {$array[$i] =~ s/Withdrawal/Payment/;}
case “CCard” {$array[$i] =~ s/Payment/Deposit/;$array[$i] =~ s/Charge/Payment/;}
case “Invst” {}
case “OthA” {}
case “OthL” {}
else {die “Type not recognized: $opt{t}. Need: Cash,Bank,CCard,Invst,OthA,OthL”}
}
$array[0] = $array[$i]; # Move Header Row, to say, the first row, where it belongs
$array[$i] = “,,”; # Set the old line to valid CSV
last;
}
}
my $lineCount = 0;
for (@array) { # Intuit creates malformed CSV lines with double quotes (doesn’t quote the field)
s/””/”/g; # replace with something valid, not perfect, but what we have
$lineCount++;
}
$lineCount = $lineCount – 13; # Approximate number of valid records in file, used later
untie @array;
Open the input (modified above) and output files
open (CSV, “ open (QIF, “>”, $outfile) or die “Cannot open QIF file [$outfile]”;
my $csv = Tie::Handle::CSV->new($opt{f},key_case => ‘lower’);
Account Type
Only valid types are:
Cash, Bank, CCard, Invst, Oth A, Oth L
switch ($opt{t}) {
case “Cash” {print QIF “!Type:Cashn”}
case “Bank” {print QIF “!Type:Bankn”}
case “CCard” {print QIF “!Type:CCardn”}
case “Invst” {print QIF “!Type:Invstn”}
case “OthA” {print QIF “!Type:Oth An”}
case “OthL” {print QIF “!Type:Oth Ln”}
else {die “Type not recognized: $opt{t}. Need: Cash,Bank,CCard,Invst,OthA,OthL”}
}
my $transactions=0;
my $splitTransNum=0;
my $myLineNum=0;
Main processing loop
while (my $csv_line = ) {
Skip Transactions without a Payee
next if ((exists $csv_line->{‘payee’}) && ($csv_line->{‘payee’} eq “”));
Skip Scheduled Transactions
next if ((exists $csv_line->{‘scheduled’}) && ($csv_line->{‘scheduled’} ne “”));
Split Transaction, should be easy, but thanks to Intuit are not
while ((exists $csv_line->{‘split’}) && ($csv_line->{‘split’} eq “S”)) {
$csv_line = processSplit($csv,*QIF,$csv_line);
}
End Split Transaction Processing
Date
print QIF “D” . $csv_line->{‘date’} . “n”;
Payee
print QIF “P” . $csv_line->{‘payee’} . “n”;
Category or Transfer
if ((exists $csv_line->{‘category’}) && ($csv_line->{‘category’} eq “”)) {
if ((exists $csv_line->{‘transfer’}) && ($csv_line->{‘transfer’} ne “”)) {
print QIF “L[” . $csv_line->{‘transfer’} . “]n”;
}
} else {
print QIF “L” . $csv_line->{‘category’} . “n”;
}
Amount
if ((exists $csv_line->{‘check #’}) && ($csv_line->{‘check #’} ne “”)) {
print QIF “N” . $csv_line->{‘check #’} . “n”;
}
Amount
if ((exists $csv_line->{‘payment’}) && ($csv_line->{‘payment’} eq “”)) {
$csv_line->{‘deposit’} =~ tr/$//d;
$csv_line->{‘deposit’} =~ tr/,//d;
print QIF “T” . $csv_line->{‘deposit’} . “n”;
} else {
$csv_line->{‘payment’} =~ tr/$//d;
$csv_line->{‘payment’} =~ tr/,//d;
print QIF “T” . $csv_line->{‘payment’} . “n”;
}
Tags, Quicken Essential supports them, iBank doesn’t, so preserve in Memo if Memo blank
if ((exists $csv_line->{‘tags’}) && ($csv_line->{‘tags’} ne “”)) {
if ((exists $csv_line->{‘memo/notes’}) && ($csv_line->{‘memo/notes’} eq “”)) {
print QIF “M” . $csv_line->{‘tags’} . “n”;
} else {
Memo has something, so Tags will be lost
print QIF “M” . $csv_line->{‘memo/notes’} . “n”;
}
} else {
Tags were blank, so print memo
print QIF “M” . $csv_line->{‘memo/notes’} . “n”;
}
Transaction Status
print QIF “C” . $csv_line->{‘reconcile’} . “n”;
Transaction Close
print QIF “^n”;
$transactions++;
} # end Main processing loop
print “Total CSV Transaction Lines (approximate): ” . $lineCount . “n”;
print “Total Transactions Written to QIF: ” . $transactions . ” (“. $splitTransNum . ” splits)n”;
close CSV;
close QIF;
sub processSplit {
my @splitTrans;
my $numSplits=0;
my $j=0;
my $splitTotal=0;
my ($csvFile, $QIF, $csv_line) = @_; # Parameters from call
Record all of the splits for this transaction
while ($csv_line->{‘split’} eq “S”) { # Need to deal with case of back-to-back splits
$splitTrans[$numSplits]->{‘date’} = $csv_line->{‘date’};
$splitTrans[$numSplits]->{‘payee’} = $csv_line->{‘payee’};
$splitTrans[$numSplits]->{‘payment’} = $csv_line->{‘payment’};
$splitTrans[$numSplits]->{‘deposit’} = $csv_line->{‘deposit’};
$splitTrans[$numSplits]->{‘check #’} = $csv_line->{‘check #’};
$splitTrans[$numSplits]->{‘category’} = $csv_line->{‘category’};
$splitTrans[$numSplits]->{‘transfer’} = $csv_line->{‘transfer’};
$splitTrans[$numSplits]->{‘memo/notes’} = $csv_line->{‘memo/notes’};
$splitTrans[$numSplits]->{‘reconcile’} = $csv_line->{‘reconcile’};
$numSplits++;
$csv_line = ;
last if ($csv_line->{‘payee’} ne $splitTrans[$numSplits-1]->{‘payee’});
}
Figure out how much the split total should be
for (my $j = 0; $j if ((exists $splitTrans[$j]->{‘payment’}) && ($splitTrans[$j]->{‘payment’} ne “”)) {
Oh, so happy that Intuit put dollar signs and commas in my CSV amounts
$splitTrans[$j]->{‘payment’} =~ tr/$//d;
$splitTrans[$j]->{‘payment’} =~ tr/,//d;
$splitTotal += $splitTrans[$j]->{‘payment’};
}
if ((exists $splitTrans[$j]->{‘deposit’}) && ($splitTrans[$j]->{‘deposit’} ne “”)) {
Oh, so happy that Intuit put dollar signs and commas in my CSV amounts
$splitTrans[$j]->{‘deposit’} =~ tr/$//d;
$splitTrans[$j]->{‘deposit’} =~ tr/,//d;
$splitTotal += $splitTrans[$j]->{‘deposit’};
}
}
Write the top-level transaction
Date of parent transaction, not used in splits
print $QIF “D” . $splitTrans[0]->{‘date’} . “n”;
Payee of parent transaction, not used in splits
Unfortunately, this means the Payee of the transaction will always be the same
as the payee of the first split. Can’t be helped.
print $QIF “P” . $splitTrans[0]->{‘payee’} . “n”;
Total of parent transaction
print $QIF “T” . $splitTotal . “n”;
Transaction parent status
print $QIF “C” . $splitTrans[0]->{‘reconcile’} . “n”;
Print each of the split transactions
for (my $j = 0; $j # Print category of split
if ((exists $splitTrans[$j]->{‘category’}) && ($splitTrans[$j]->{‘category’} eq “”)) {
if ((exists $splitTrans[$j]->{‘transfer’}) && ($splitTrans[$j]->{‘transfer’} ne “”)) {
print $QIF “S[” . $splitTrans[$j]->{‘transfer’} . “]n”;
}
} else {
print $QIF “S” . $splitTrans[$j]->{‘category’} . “n”;
}
Memo of Split (unfortunately, cannot do memo of parent)
Though Intuit is happy to put the parent memo on a child split that is blank, idiots.
Tags, Quicken Essential supports them, iBank doesn’t, so preserve in Memo if Memo blank
if ((exists $splitTrans[$j]->{‘tags’}) && ($splitTrans[$j]->{‘tags’} ne “”)) { # This never works, not sure why
if ((exists $splitTrans[$j]->{‘memo/notes’}) && ($splitTrans[$j]->{‘memo/notes’} eq “”)) {
print $QIF “E” . $splitTrans[$j]->{‘tags’} . “n”;
} else {
Memo has something, so Tags will be lost
print $QIF “E” . $splitTrans[$j]->{‘memo/notes’} . “n”;
}
} else {
Tags were blank, so print memo
print $QIF “E” . $splitTrans[$j]->{‘memo/notes’} . “n”;
}
if ((exists $splitTrans[$j]->{‘check #’}) && ($splitTrans[$j]->{‘check #’} ne “”)) {
print $QIF “N” . $splitTrans[$j]->{‘check #’} . “n”;
}
Print amount of split
if ((exists $splitTrans[$j]->{‘payment’}) && ($splitTrans[$j]->{‘payment’} eq “”)) {
print $QIF “$” . $splitTrans[$j]->{‘deposit’} . “n”;
} else {
print $QIF “$” . $splitTrans[$j]->{‘payment’} . “n”;
}
}
Transaction Close
print $QIF “^n”;
$transactions++;
$splitTransNum++;
return $csv_line;
} # end processSplit()
[/perl]
How to use qecsv2qif.pl
To use my program, you’ll need to first have a working Perl installation. You’ll also need the following Perl modules:
Once you have everything necessary to run the converter, you may use the following syntax:
./qecsv2qif.pl -f FILENAME -t ACCT
Where FILENAME is the CSV formatted file you exported and ACCT is the type of account you exported. The account type can be one of the following: Cash, Bank, CCard, Invst, OthA, OthL (only Cash, Bank, and CCard have been implemented, because only these are able to be used in Quicken Essentials). It is very important you select the proper account type, otherwise, the conversion will not complete correctly.
That’s about it. I hope this helps many people escape the binds that Intuit has forced upon us with Quicken Essentials. You need to communicate your dissatisfaction with your wallets. Go try out iBank 4.
If you’re patient, I’m also working on a web service that will implement my code and allow you to convert files here on my blog. That will take a little more time.
Update – 20110220
I just found out that iBank 4 actually supports CSV file importing. So, after you make your CSV files per account, try it out as it may save you some time downloading and getting my Perl script running. Here’s a quick tutorial on it:
First select the account you’re going to import the transactions into. Then select Import… from the File menu.
There are several issues using the built-in CSV Import tool with iBank that my program does’t have:
- Quicken Essentials puts Categories in one column and Transfers in another. So, you have to pick one (pick Categories). Unfortunately that means NONE of your transfers will import correctly and you’ll have to manually edit all of them.
- iBank doesn’t recognize the Split column. So all of your splits will import as separate transactions.
- iBank doesn’t support Tags, so if you used Tags in Quicken, you’ll lose them all. My program will save the Tags if you didn’t put in a Memo for that transaction.
- For some reason iBank doesn’t expose the Reconcile column, so none of your transaction statuses will be saved.
If the above issues don’t bother you, than give that a try, as it will certainly save you some time in getting my Perl program running.
Update 2 – 20110221
There seems to be some confusion about an update that Intuit did for Quicken Essentials 2010 for Mac last week. That update, version 1.5.2, allowed provided the following: “You can now export Quicken Essentials for Mac data using the .QXF file transfer format”. You can read the full release notes for Quicken Essentials v1.5.2 here.
The first thing you need to know is that the “.QXF file transfer format” is proprietary to Intuit / Quicken. It is not a file transfer format that anyone else supports. It’s essentially worthless unless you wish to move your data from one Quicken product to another (like from the Quicken Essentials 2010 for Mac to Quicken Premier 2010 for Windows).
The .QFX file transfer format is a relatively open file format based on .OFX. The .QFX format and the .QXF format are totally different. You can read a little bit about QFX here. If Intuit allowed you to export as QFX there would be no reason for this article, but they do not.
Update 3 – 20110307
A reader of this article, Chris, wrote me to point out that Intuit’s Quicken Essentials data format (.quickendata) is a SQLite database. I did a little digging using Mike T’s SQLite Database App and sure enough, the data is all stored in 37 different SQLite tables. You cannot open the .quickendata file directly, as that file is actually just a container for a bunch of data. The SQLite database is actually a file called data inside of that file, and if you use Mike T’s SQLite Database App to open it, then you’ll have to extract it out of the container first (just right-click on it and select Show Package Contents). This opens up several possibilities for developers to reverse engineer the database schema and get a much more accurate translation of your transactions. Stay tuned!