AlexKuo.info

TXF Converter for 2010 ThinkOrSwim and Ameritrade Excel to CSV Files

Alex Kuo — Tue, 22 Feb 2011 22:57:39 GMT

After attempting to import my equity trades into Intuit's TurboTax 2010 Home and Business edition, I discovered that support for importing CSV files has been removed. So, I decided to roll my own program.

You can download the source here or download the Windows executable here. Please note, the .NET 4.0 framework is required. After executing the program, you need to enter in the full file path [directory and file name] in the file text box and the full path for the directory where you want the file to be created. A 'TXF' file should be created afterwards in the specified location.

You can import the TXF file in TurboTax 2010 by opening your current tax return and selecting File -> Import -> From Accounting Software. Check the details before finishing the import to make sure everything looks right.

If you want more details about the TXF file format or the CSV reader I used in the program. Please check out the following links.

TXF Format
LumenWorks CSV Reader

I've only tested the TXF files from this program on Turbo Tax 2010 Business and Home edition. This software is provided without warranty or support.

ASP.NET: Modifying DetailsView Select and Updating Events

Alex Kuo — Sun, 09 Aug 2009 18:45:49 GMT

Recently I was tasked with encrypting/decrypting some data that could be edited using ASP.NET's DetailsView control. Along with the details view control, the legacy code used a SQL Data Source to select and update everything. This means the data access layer was bypassed completely, so any business logic that dealt with encryption on this page had to use a different execution path. So, how would you encrypt and decrypt data?

The problem could be divided into two tasks:

Create a filter to decrypt data in any bound columns.
Edit the DetailsView event, ItemUpdating and figure out which columns needed to be updated during the event.

The first task was the easier of the two, as it just involved referencing the encryption library from the code behind page and calling the function inline in the aspx page. So, the method in the code behind page would look like something like this:

protected string Decrypt(object input)
{
     return EncryptionHelper.Decrypt(input.ToString());
}

While calling the decrypt function in the DetailsView child Fields tags on the aspx page:

<asp:templatefield headertext="Decrypted Data" sortexpression="EncryptedData>
    <edititemtemplate>     
     <asp:TextBox ID="TextBox1" runat="Server" Text='<%#Decrypt(Eval("EncryptedData"))%>' />
    </edititemtemplate>
    <itemtemplate>
      <asp:label runat="server" text='<%# Decrypt(Eval("EncryptedData")) %>' id="Label1"></asp:label>
 </itemtemplate>
</asp:templatefield>

However, decrypting the data using a templatefield tag posed another problem, extracting new values from the 'TextBox1' control. This will be solved when calling the ItemUpdating event.

After setting the DetailsView's ItemUpdating method, you will need to modify the SQL Data Source's UpdateCommand, UpdateCommandType, and UpdateParameters. The legacy code in the SQL data source used a update statement with parameters. Normally the details view control automatically detects whether to pass a null in the parameter. Instead when setting the UpdateParameters in the codebehind, the control will update the database with the parameter names entered. For example '@Parameter', will be sent in the update to the record instead of null. To solve this, you would need to automatically detect whether a value is null and then add it to the UpdateParameters list. This was done using the 'IsFieldNull' method [shown below].

Initializing the SQL data source also had another problem - because certain data needed to be re-encrypted before being sent back to the database. You will need to single out any columns that need to be encrypted. On top of that, the table being updated had over 50 columns, so typing in 50+ lines just to initialize the parameters was required, but I decided to do something else instead.

The parameters being sent to the stored procedure were identically named to the column names. Using this protocol, I decided to simply grab all the field names, which were identical to the column names and add the '@' character next during initialization. So, the method to get the column names looked like the following:

private List<string> GetColumnNameList()
{
  List<string> Ret = new List<string>();

  for (int i = 0; i < DetailsView.Rows.Count; i++)
  {
    if (!String.IsNullOrEmpty(DetailsView.Rows[i].Cells[0].Text))
    {
      Ret.Add(DetailsView.Rows[i].Cells[0].Text);
    }
  }
  return Ret;
}

Initializing the update parameters looked like the following:

private void InitSqlDataSource()
{
   SQLDataSource.UpdateCommand = "Update";
   SQLDataSource.UpdateParameters.Clear();

   List<string> paramList = GetColumnNameList();

   foreach (string col in paramList)
   {
      if (!IsFieldNull(col))
      {
         if (col != "EncryptedData")
         {
           SQLDataSource.UpdateParameters.Add(col, "@" + col);
         }
         else
         {
           SQLDataSource.UpdateParameters.Add("EncrptedData", 
                 EncryptionHeper.Encrypt(GetDecryptedValue()));
         }
      }
   }  
}

The method checks to make sure the data being sent isn't blank or null by executing the 'IsFieldNull' method. This method uses the 'ExtractValuesFromCell' method to grab values from the DetailsView control. This method isn't exactly well documented on MSDN, so it's not obvious at first that the passed parameter, IOrderedDictionary dictionary, is being used as a referenced parameter, not as a copy. [another weird quirk about the .NET framework] The method, GetValues, used is identical to what is covered at the article written at David Fowler's blog.

private bool IsFieldNull(string fieldName)
{
    OrderedDictionary vals = GetValues(DetailsView) as OrderedDictionary;

    if (vals[fieldName] == null)
    {
        return true;
    }

    return string.IsNullOrEmpty(vals[fieldName].ToString());
}

public IDictionary GetValues(DetailsView detailsView)
{
    IOrderedDictionary values = new OrderedDictionary();

    foreach (DetailsViewRow row in detailsView.Rows)
    {
        if (row.RowType != DataControlRowType.DataRow)
        {
            continue;
        }

        DataControlFieldCell dataCell = (DataControlFieldCell)row.Cells[0];
 
        if (dataCell.ContainingField.ShowHeader)
        {
            dataCell = (DataControlFieldCell)row.Cells[1];
        }

        dataCell.ContainingField.ExtractValuesFromCell(values, dataCell, row.RowState, true);
    }

    return values;
}

Angelo's BBQ in Ft. Worth

Alex Kuo — Thu, 06 Aug 2009 05:09:01 GMT

Back Online

Alex Kuo — Mon, 29 Jun 2009 06:28:37 GMT

After a long hiatus, I finally got my blog back online. I've been looking into quite a few new technologies the past few months and will certainly start writing about them this week.

Ubuntu Server 7.10: Installing Proftpd and 'warning: unable to determine IP address of'

Thu, 06 Mar 2008 23:54:24 GMT

I had some trouble installing proftpd today on a Linode.com VPS server. For anyone that hasn't used linode, it's a colo service that lets people lease different grades of Linux VPS servers at reasonable prices. I've been using them for 4 months with few problems.

Installing proftpd in Ubuntu is easy. All you have to do is open a Terminal window or SSH into your server and type:

sudo apt-get install proftpd

However for some reason, the Ubuntu distros mounted on Linode's servers have a quirk or two that require a few tweaks to get the server working. The first thing is whether to run the server in standalone or xinetd mode. The server won't be able access port 21 unless it's operating in standalone mode. So, set it to standalone mode.

The second quirk, which is not unique to Linode VPS servers, regards resolving the host's address. You must add the server's static address to the hosts file. For example, if your machine's name is 'johnny' and the static IP address is 79.221.23.12.

To edit your hosts file type

sudo vi /etc/hosts

Your hosts file should look like this:

127.0.0.1       localhost
79.221.23.12	johnny

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

You can determine your machine's host name by opening the file, '/etc/hostname'.

Aftewards, check your proftpd configuration. To do this, type:

sudo proftpd -td5

If you would like more information on setting up Proftpd or Ubuntu's network configuration. Please consult the following:

Baking Ribs

Mon, 04 Feb 2008 02:55:59 GMT

For those of you enjoying your Super Bowl Sunday, I would like to extend to you a quick tip about preparing pork ribs. Baking ribs isn't that hard. Just do the following:

Preheat the oven to 200 degrees.
Defrost ribs. If they're still in those water / air tight plastic packages, you can defrost them quickly by submerging the plastic package in warm water.
Season ribs and wrap the slabs in foil.
After oven is preheated, put them on a cookie sheet and let them bake in the oven for 3 to 3.5 hours.

After baking them, just put BBQ sauce on it and sear the ribs either on a grill or hot skillet to get the BBQ sauce nice and toasted. For sauce, I usually use Original K.C. Masterpeice, and for seasoning, one of those dry rub seasonings made by Weber (yes, the same company that makes BBQ Grills).

PostgreSQL 8.2: Recovering from a Corrupted Database

Tue, 27 Nov 2007 01:50:31 GMT

Today I encountered my second database corruption or crash on my development machine. The problem is that my IDE drive sucks, an old IBM 75GXP. For those of you who don't remember, these drives were notorious for having all sorts of problems when they were released. Most of the problems revolved around dying after six months. Regardless, I'm still using this thing for another month.

You can tell if your database became corrupted by restarting the postmaster or typing:

sudo /etc/init.d/posrgresql-8.2 restart

The result of the command will look similar to the following:

2007-11-26 13:14:48 CST LOG:  server process (PID 16312) was terminated by signal 11
2007-11-26 13:14:48 CST LOG:  terminating any other active server processes
2007-11-26 13:14:48 CST LOG:  all server processes terminated; reinitializing
2007-11-26 13:14:48 CST LOG:  database system was interrupted at 2007-11-26 13:14:47 CST
2007-11-26 13:14:48 CST LOG:  checkpoint record is at 0/965E4130
2007-11-26 13:14:48 CST LOG:  redo record is at 0/965E4130; undo record is at 0/0; shutdown TRUE
2007-11-26 13:14:48 CST LOG:  next transaction ID: 0/73629; next OID: 10952256
2007-11-26 13:14:48 CST LOG:  next MultiXactId: 1; next MultiXactOffset: 0
2007-11-26 13:14:48 CST LOG:  database system was not properly shut down; automatic recovery in progress
2007-11-26 13:14:48 CST LOG:  record with zero length at 0/965E4178

Before initiating any recovery options, I recommend backing up your $PGDATA directory. The following directions will attempt to reset the transaction log, and you may loose the latest changes to your databases. The first option for recovering from a crash is to open your '/etc/postgresql/8.2/main/postgresql.conf' file and turn on the following option:

...
zero_damaged_pages = true

After saving the changes, type the following to restart postgresql:

sudo /etc/init.d/postgresql-8.2 restart

If you still see error messages, you may need to restart the entire computer in order to get postgresql to start.

After restarting postgresql, reopen the '/etc/postgresql/8.2/main/postgresql.conf' file and comment out the zero_damaged_pages option. The line should look similar to the following:

#zero_damaged_pages = true

After reading a few posts, it seems that IDE drives in general should not be used when working with databases because of corruption issues. If you are using an IDE drive, I recommend adding the following options to the '/etc/postgresql/8.2/main/postgresql.conf' file.

fsync = on				# turns forced synchronization on or off
wal_sync_method = fsync

The problem with turning on fsync is that it slows down the performance of Postgresql, however, it should prevent further corruption issues with your database.

You can find more information about these configuration options in the Postgresql 8.2 documentation.

Reference:

pg_resetxlog - Tool used as a last resort for resetting the transaction log

When Not to Run on an Open Source Platform

Thu, 22 Nov 2007 09:07:54 GMT

The other day I had a nice chat with a colleague about the benefits and pitfalls of using an open source platform. I know this subject has been discussed deeply on various forums articles and blogs. I've personally done research for Alcatel-Lucent and the University of Texas at Dallas on the subject. For businesses, the bottom line is cost. Their are certain scenarios in a business where switching platforms, whether its open source or not, does not make sense. The scenario I discussed with my colleague revolved around the cost of switching from a Microsoft platform to a Linux platform for its custom server applications.

My colleague's business faces two main hurdles before considering a migration: the time it will take to migrate custom applications and the expenses incurred during the migration. If the current system is meeting customer needs and does not need further changes, then migrating over to an open source platform may not be the best choice, especially if the lifetime of the system is not close to expiring. Open source software is best used when continuous scaling is demanded. Because of the licensing fees incurred when scaling can become astronomical, development of the system needs to consider the benefits given to a proprietary solution over a open source solution. For example, Microsoft's Visual Studio has excellent tools for building .NET applications. If the time saved from developing on this platform is justifiable and the scaling needs do not supercede the cost saved using an open source platform, then using a Microsoft solution may be warranted.

Despite my overall excellent experience developing on the Microsoft platform, I still have a hard time recommending it as the platform of choice, mainly because of cost and time savings. These savings are not obvious to people who have not used an open platform before. One way to explain the cost and time savings incurred is the notion of barriers or hurdles towards completing a goal. When developing or deploying a system, one of the hurdles towards completion is paying for licenses, which incurs some amount of expense in time and money. If the entire application stack is free, then you eliminate that expense altogether for the rest of the system's lifetime. This allows the complete application stack to be cloned or deployed multitudes of times, whether in a test system, developer system, or production system, without incurring the expense of time and money you would encounter in a purchased product. Removing this hurdle has changed the way entire systems are deployed and general deployment of systems in both open source and commercial projects. One simple example is Debian's software distribution, 'apt'. A user can script out the default software configuration for a server with one command line. For example, installing a web server, office suite, browser, a couple of games, a couple of compilers, and a IDE with one shell command. Removing the purchasing barrier also paves the way towards completely automating the scaling of a system's infrastructure. For the small software vendor, you just can't do that on a Windows platform.

Ubuntu Desktop 7.10: Setup HP 1200 Printer

Wed, 24 Oct 2007 10:09:03 GMT

After a short search through the Ubuntu forums, I ran into this post that went into details about setting up a HP printing device. After briefly reading through the instructions, I ran a utility called 'HPLIP'. 'HPLIP' is a program that will automatically download and compile all the necessary files to activate your printer. The program will ask you a few questions about your computer and request you replug-in your printer at the end of the installation. After doing this, I printed a test page and golly... it actually worked.

Ubuntu Linux: Syncing Documents between Different Computers Using NFS and Unison

Thu, 18 Oct 2007 07:22:50 GMT

The other day I successfully made a full transition from my laptop to my desktop as my primary development environment. The biggest hurdle before completing this transition was transferring and syncing documents between my two laptop and desktop. For quick file transfers, I created a network file share, or NFS, on my desktop, while mounting the drive on my laptop. For a quick overview on how to setup and mount NFS, consult this thread on Ubuntu forums.

I also wanted to sync and compare documents from a centralized server and have the ability to compare differences between a client and a centralized 'master' copy. (think Subversion - but without all the permissions and change logging) After a quick Google search, I found a wonderful program called 'Unison'. This program will allow a user to define a master directory on a server and slave directory on a client. Master being the label for the directory where all clients compare their files and slave as clients that send new files or receive files copied from other clients to the master directory. For directions on installing unison, consult this article on howtoforge.com.

ASP.NET and Retrieving Different Sections of the Current URL using Request.Url

Fri, 07 Sep 2007 10:13:29 GMT

While working on a project in ASP.NET, I needed a function that would retrieve the domain of the current url, however, I also wanted the function to also retrieve the correct ASP.NET development web server path when developing in Visual Studio. After consulting Google, I ran into this old post on Rick Strahl's blog about the Request.Url object.

After some experimenting, I created a web page in ASP.NET 2.0 that showed what parts of the URL could be returned using different calls. Consult the following in the page load event.

    protected void Page_Load(object sender, EventArgs e)
    {
            Response.Write("Request.Url.AbsolutePath= " + Request.Url.AbsoluteUri);
	    Response.Write("<br>");
            Response.Write("Request.Url.AbsoluteUri= " + Request.Url.AbsoluteUri);
            Response.Write("<br>");
            Response.Write("Request.Url.GetLeftPart(UriPartial.Authority)= " + Request.Url.GetLeftPart(UriPartial.Authority));
            Response.Write("<br>");
            Response.Write("Request.Url.GetLeftPart(UriPartial.Path)= " + Request.Url.GetLeftPart(UriPartial.Path));
            Response.Write("<br>");
            Response.Write("Request.Url.GetLeftPart(UriPartial.Scheme)= " + Request.Url.GetLeftPart(UriPartial.Scheme));
            Response.Write("<br>");	
            Response.Write("Request.RawUrl= " + Request.RawUrl);
            Response.Write("<br>");
            Response.Write("Request.Path= " + Request.Path);
            Response.Write("<br>");
            Response.Write("Request.ApplicationPath= " + Request.ApplicationPath);
            Response.Write("<br>");
            Response.Write("Request.ResolveUrl= " + ResolveUrl("~/dealer/default.aspx"));
            Response.Write("<br>");
            Response.Write("GetAuthorityApplicationPath= " + GetAuthorityApplicationPath());
    }

    private String GetAuthorityApplicationPath()
    {
        return String.Concat(Request.Url.GetLeftPart(UriPartial.Authority), Request.ApplicationPath);
    }

The function GetAuthorityApplicationPath() is what I needed in the end to dynamically retrieve either the domain in a production environment or the development web server url while running Visual Studio (eg. 'http://localhost:1234/WebDirectory')

Developing on a Open Source Platform

Mon, 03 Sep 2007 02:07:26 GMT

Lately I've been tasked with writing an Administration interface that's very client heavy for a web application. It uses the Extjs framework and its widgets for building the GUI, Django + Python for the application tier, and PostgreSQL for the database end. We're using Apache and Ubuntu Server as our platform. The entire application stack is open source, so the acquisition costs for starting development is nil.

In the past few weeks, I've developed more insights into the advantages of developing on a completely open source stack. The newest pro I've discovered is the documentation and active communities in the larger and popular projects. I know a lot of MS developers moan about the lack of adequate support available on some open source projects, but it's not true of all them out there. When choosing components for your system, it's almost a given that strong community support is a requirement. Fortunately in OSS projects, the utility of a project and the general following behind it go hand in hand.

I can recommend with confidence that the community support behind our application stack (Extjs, Django, Python, PostgreSQL) for our system is strong and adequate for any web application projects that you may want to pursue

Ubuntu Desktop: Unplugging/Replugging your Network Cable on your Laptop and Requesting a New Address from DHCP.

Fri, 10 Aug 2007 03:05:53 GMT

Sometimes when working on my laptop, which has Ubuntu Desktop installed on it, I have to move it around and therefore unplug the network cable and switch to wireless or vice versa. Unplugging and replugging your laptop into a network sometimes results in the laptop's inability to renew its IP address or re-establish a connection with the internet. After a quick Google search, I found this post about the problem.

In order to issue a command similar to 'ipconfig -renew' in Windows, open your shell and type the following:

sudo ifdown eth0
sudo ifup eth0

These two commands will renew your IP address and should fix the connection problem. However, there's a program called 'ifplugd' that monitors your network connection and automatically renews your address if this problem occurs. To install this program, open your shell and type:

sudo apt-get install ifplugd

Script tags and Javascript Arrays in IE and Firefox

Tue, 07 Aug 2007 12:10:16 GMT

Tonight I was wrapping up a deployment for a feature that was heavily dependent on javascript. After running my first batch of tests on Firefox, everything passed without a hitch. Then I ran my tests on Internet Explorer 7 and.... nothing.. literally a blank page rendered on my screen. Thus began my latest saga of fixing compatibility issues between browsers.

The first problem was that my scripts were either being downloaded and not executing, or were not downloading at all. I checked if the javascript files were being retrieved by using program called Fiddler2. This program intercepts all HTTP requests and responses from Internet Explorer. After running the program, it confirmed that my files were being downloaded. So, this means the files were not being executed. After staring at my html code which looked like the following:

...
<body>
<script type='text/javascript' src='/file.js' />
</body>
....

I immediately remembered that Internet Explorer doesn't render javascript references unless you specify the complete tag. Why? I suspect its because I didn't specify the doctype in the html tags. So to fix that problem, I changed the script tag to look like this:

...
<body>
<script type='text/javascript' src='/file.js'></script>
</body>

However, that was not the end of my javascript problems with Internet Explorer. After running the javascript, I ran into an error regarding an undefined element in an array. My first thought was that this couldn't be. The script ran flawlessly in Firefox. After some debugging, I concluded that the Javascript engine in IE interprets text representations for arrays differently than Firefox. The evidence that lead to this conclusion came from comparing the array length returned in Firefox versus Internet Explorer. Firefox's length was 3, while Internet Explorer's was 4.

My Firefox-only-array looked something like this:

array = [ {id:1,name:'hi'}, {id:2, name:'hello'}, {id:3, name: 'greetings'}, ]

Turns out that last comma in the array throws off Internet Explorer and causes it to increment that length an extra tick. The fix would be to remove that last comma. The fixed code looks like this:

array = [ {id:1,name:'hi'}, {id:2, name:'hello'}, {id:3, name: 'greetings'} ]

Developing Client Heavy Applications

Mon, 06 Aug 2007 10:57:00 GMT

After sitting down and creating a javascript client, I have to say - this really sucks. I guess moving to a heavier client tier is a natural progression of web development. Back in 1996 or so when I was still in grade school, I started messing around with HTML and basic CGI scripts for creating dynamic web pages. About 2 years later, MySQL became really popular and the first database generated web sites started to appear. The big argument back then was whether application logic should be done in the database end or in the application tier. The languages and platforms I was aware of in the open source world were Perl and PHP3 - both were equally terrible web development tools. These weren't the only open source tools out there at the time, rather quite the opposite. There was a huge plethora of different web platforms out there. Some still exist but still obscure - others just obscure. Let's just say things were weird because everyone was still trying to figure out the best way to develop web front ends. For example, it wasn't unheard of to hear SQL being used to generate HTML.

After PHP4 came out, there was an explosion in web development around the PHP, MySQL, and Apache application stack. About this time, most of the dynamic page generation got moved to the application tier. Fast forward about 6 years later to the present, we're seeing more of the page generation executed in the web browser using Javascript or some other platform built into the browser. There are numerous advantages that can be gained by moving page generation to the client: lower load on the servers, improved visual enhancements, and faster execution speed. However, developing interfaces in javascript still feels immature. To me, client side development in javascript is about equivalent to what PHP4 was back in 2001 - it does the job but has a lot of room for improvement.

This doesn't mean the javascript engines found in IE and Firefox aren't mature, its actually quite the opposite. Since Javascript has been incorporated in the browser since the 1990s, the engine is very mature and stable. It's just new requirements and forms of usage for javascript has changed since its original inception, hence why I think the current implementation seems incomplete. By incomplete, I mean issues such as browser compatibility, the occasional memory leak, and I still think development tools have room for improvement, like code completion and debugging.

The current application I'm working on uses the ExtJs framework for generating the Admin UI and Django as the middle tier for parsing requests between the client and the database. So far, the combination is working well.

Ubuntu Desktop: How to Find Your Application Files that Store Your Personal Preferences

Fri, 27 Jul 2007 02:10:01 GMT

Today I was trying to find the location of my chat logs for gaim in Ubuntu and noticed that none of my logs were being found through the File Search program ('Places' -> 'Search for Files'). After digging through a few Google queries, I ran into a blurb about hidden folders prefixed with a period. It turns out all personal preferences user specific files for your applications are stored in directories with this notation '/home/user_name/.program_name'. So all my personal files and settings related to gaim would be stored in '/home/alex/.gaim'.

The File Search program ignores hidden files by default. In order to search hidden files, click on the 'Available Options' drop down list and select 'Show hidden and backup files' and press the 'Add' button. This should include all hidden files in your search.

To see a list of the hidden directories in your home folder, open the shell and type the following:

ls -a

By default, your command prompt should open in your home folder. So you shouldn't need to navigate to '/home/user_name/'.

Mirroring an FTP site in Ubuntu Server

Mon, 09 Jul 2007 12:11:27 GMT

The other day I was tasked with mirroring a FTP site, about 5+ gigs of files, on our local server. Mirroring directories is a fairly common task when administering servers, however, the main differences when tasked with this job are the protocols available, whether the job is bi-directional or one way, and the how fast the mirroring needs to occur.

Lucky for me, this job did not demand instantaneous sync and the job was only one way - meaning changes from a server were reflected only from one server. The biggest problem was this job was limited to using only the FTP protocol for mirroring the site. This immediately removed rsync, a popular server/client for syncing directories remotely, as an option. After a quick search through the Ubuntu forums, I stumbled upon a post that detailed several programs on mirroring an FTP site using FTP protocol only. I chose a program called 'ftpmirror'.

Ftpmirror is a program that lets a user define 'packages', which are configuration details for mirroring an FTP site, and scheduling these 'packages' to be run daily, monthly, or weekly. To install this program, I typed

sudo apt-get install ftpmirror

If you're using Ubuntu Server, the configuration files should reside in the '/etc/ftpmirror/' directory. Upon browsing through the directory, you will find a file called 'ftpmirror.cf-sample'. This file contains a few example 'packages' that can be used as a template. The user puts any active 'packages' in the 'ftpmirror.cf' file. My 'ftpmirror.cf' file looks like this:

package = alexkuo_media
 ftp-server = master.alexkuo.info
 ftp-user = mirror
 ftp-pass = password
 remote-directory = /media/pics/
 local-directory = /home/deploy/media/pics

This package uses the directory '/media/pics/' as the root directory on the ftp server 'master.alexkuo.info' and uses the login/password, mirror/password, to login into the remote server. All files, directories, and subdirectories found in the '/media/pics' directory are then downloaded into the '/home/deploy/media/pics' directory on the local machine once the package is activated.

I decided to run this job once a week, so I added the package to the '/etc/ftpmirror/list.weekly' file. To do this, open the 'list.weekly' file with a text editor. Mine looks like this:

alexkuo_media

Pretty plain huh? I removed the comments that originally came with the file, so it looks pretty bare. Adding another package involves defining another package in the ftpmirror.cf file and appending the package name on a new line in one of the *.list files.

Ubuntu Server 7.04 Fiesty: Mounting or Accessing a USB Drive in Bash Shell

Fri, 29 Jun 2007 02:32:15 GMT

I ran into a problem today accessing a USB Drive in Ubuntu Server. In Windows and in Ubuntu Standard, plug and play works fine. My laptop, which has Ubuntu installed, immediately recognized the USB drive once it was plugged in. I accessed its contents in the Bash shell by typing:

ls -l /media
ls -l /media/ExternalHDD

Typing the same thing in Ubuntu Server will not yield the same results. Ubuntu 7.04 server does not come with automatic mounting. After plugging in your drive, you need to run a few commands before access to the drive will be allowed. The first command is to install a program called 'pmount'. Type:

sudo apt-get install pmount

After installing the pmount, you need to figure out what the drive is called on your system. Type:

fdisk -l

You should see output that looks like this:

Disk /dev/sdf: 300.0 GB, 300069052416 bytes
255 heads, 63 sectors/track, 36481 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf4   *           1       36481   293033601    7  HPFS/NTFS

The USB drive I am using is called '/dev/sdf4'. To mount the drive, you're need to create a directory to alias the drive and use pmount to reference the alias. To do this I type:

sudo mkdir /media/usbdrive
pmount /dev/sdf4 /media/usbdrive

Afterwards, I copied a bunch of files on the USB drive and set the ownership of the files from root to myself. I typed:

sudo cp -R /media/usbdrive/filesToCopy /home/alex/
cd /home/alex
sudo chown -R alex:alex filesToCopy

New Article

CruiseControl Server and Django: How to Setup CruiseControl to Automatically Deploy Django Applications

Ubuntu Server 7.04 Fiesty: Installation Problems and Changing Network Settings from DHCP to Static IP

Tue, 26 Jun 2007 13:12:31 GMT

Tonight I deployed another installation of Ubuntu Server 7.04 Fiesty. The biggest problem I ran into was setting up the network card. Apparently, the Ubuntu setup wizard tries to contact the network in at least two points during the installation wizard. In one of the points of the installation, failure to contact a server on the internet results in the installation hanging. I got around this by running the installation while connected to my residential DHCP network and then redeploying the server to the DMZ, which requires defining a static IP.

This created another problem - How do you change the network settings from DHCP to static IP address? In order to solve this problem, you need to update your interfaces file to use an assigned static IP address and configure the server to use your network's DNS servers.

To change your network settings from DHCP to static IP, type:

sudo vi /etc/network/interfaces

The file displayed should have a line that says ' iface eth0 inet dhcp' or something similar. Change the file to look like the following:

....
....
....
auto eth0
iface eth0 inet static
	address 71.164.212.40
	netmask 255.255.255.0
	network 71.164.212.0
	broadcast 71.164.0.255
	gateway 71.164.212.1

Afterwards, restart your network interface by typing:

sudo /etc/init.d/networking restart

Next, you need to configure your DNS servers. You need at least two entries. To edit your DNS entries type:

sudo vi /etc/resolv.conf

My file looked like this:

nameserver 4.2.2.1
nameserver 4.2.2.2

After saving the file, you should be able to connect to the network.

If you would like to edit your hosts file, type the following in your shell.

sudo vi /etc/hosts

You can add more references to your localhost in here. Mine looks like this:

127.0.0.1       localhost
127.0.0.1	alex-server
127.0.0.1	sandbox-server

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

If you would like to change the hostname of your server, type the following:

sudo vi /etc/hostname

After typing in your new hostname, you need to run a script to implement the change. Type:

sudo /etc/init.d/hostname.sh start

References

Ubuntugeek.com - Ubuntu 7.04 Fiesty Fawn LAMP Server Setup

Removing Graffiti on my Car

Sun, 24 Jun 2007 07:57:44 GMT

The other day somebody in the neighborhood dribbled touch up paint all over my car. Judging from a bottle of paint that was left near the crime scene, I think it's lacquer-based. After pondering what to do, I consulted Google and ran into this post. It explained that I should use something called Goof-Off for removing the offending paint. This stuff really does work, however, it takes quite a bit of elbow grease to get the paint off.

At first I used Q-tips to remove the splotches of paint, but I found that it was easier to just use a soft paper towel. It took about 2.5 hours to remove all the dribbles and spots. The only word of caution I can offer to anyone out there when using 'Goof-off' is to take it easy on the pin stripes. This stuff is powerful enough to take the stripes off (which is bad), but not powerful enough to get through the clear coat (which is good).

After 2.5 hours of rubbing and cleaning, the paint was taken off without any problems.

Killing processes and PostgreSQL Queries

Mon, 18 Jun 2007 11:20:02 GMT

I've been working on issues related to processes and PostgreSQL these past few days. Minor as they were, I think these are common issues that aren't documented very well. I recently had a problem canceling queries issued from PGAdmin. I decided to cancel the queries by looking up the process id and canceling the process thats running the query.

You can look up the process id, or the 'pid' column, by executing the following query in psql or PGAdmin:

select * from pg_stat_activity;

This query will return a list of all processes currently being run by your server. After finding the query you want to cancel, go to the BASH prompt and type:

sudo kill pid_number

Another problem that I ran into the other day was figuring out which processes a java server was running on. I did this by querying processes by name or browsing through all of the server's processes. To query processes by a name, I typed:

ps -o user,pid,ppid,command -ax | grep java

If you still can't find the process, you can browse all processes by typing:

procinfo

New Article

PostgreSQL: Backing up and Restoring a Database

Milestone: First Django Application Ever

Wed, 13 Jun 2007 03:19:11 GMT

I finally deployed a Django application to a public facing Linux server. You can read all about it in a write up below. For those of you who don't know what I'm talking about, Django is another 'rapid application development' web framework that comes with all sorts of useful tools to help web application developers get their software finished faster. Some of these time saving features include an auto generated administration tool, an ORM object mapper, and a template engine.

So how does developing on Django compare with development on ASP.NET? Well, for non-MS Office related development I would say it is faster and more suitable for custom applications. Alot of this has to do with the object relational mapper, which accomplishes the same things as Microsoft's Dlinq, and the generated admin tool. These two features alone completely remove a large amount of coding that usually has to be done when creating an ASP.NET application.

I highly recommend anyone out there give Django a try. Check out this tutorial for a quick introduction.

New Article

Python: How to Setup a Django Application on Apache 2.0 with mod_python on Ubuntu Server 7.04 Fiesty

The Garage Door is Broken

Mon, 11 Jun 2007 03:07:22 GMT

In any engineering project, finding the cause of a problem can be just as hard as fixing the actual problem. Today our garage door broke, so my roommates and I tried to fix it. At first we thought the problem was caused by a broken axle on one of the track wheels. After inspecting the plate, we quickly found out that the axle was just fine.

Then we decided to inspect one of the guide wire wheels. The guide wires were definitely incorrectly configured. We found that the wire had not re-wrapped itself properly, causing the motor to push harder on the garage door when closing.

This still didn't explain why the door wouldn't close though. Stepping back to look at the garage door, we quickly realized that the two sides were not falling down at the same rate, causing one side to close faster than the other - warping one of the guide rails.

After examining the brackets that secure the track wheels to the door, we found one of the brackets had cracked. This created a harder 'push' on one side than the other when closing.

The end result is this:

At this point, we decided to call the repairman.

New Articles

Goodbye to Three Hour Programs

Wed, 06 Jun 2007 23:26:13 GMT

Back in the day, I use to sling code in 2-3 hour blocks, taking nice long breaks in between sessions. This allowed my brain to think about problems deeper and come up with more elegant solutions. One of the reasons is that the house I am in turns into a noise making, hard rocking, garage band around 5 PM when all my roommates come home. Homes with children experience a similar event. In any case, I code late at night or in the mornings when no one is home. In fact, I try and code for as long as I can because I know these periods of peace and quiet don't last. I guess the 2-3 hour programming sessions will be on ice until 'quiet time' becomes the norm again.

New Article

Getting Around Linux: PostgreSQL's 'COPY FROM' and Permissions in the Home Directory

Beta Testing Joost

Mon, 04 Jun 2007 12:45:45 GMT

Joost is like YouTube except with complete programming instead of one to three minute clips of random shows and amateur videos, requires a Windows client, smaller selection of content categories, no top ten lists, and with commercials that are spliced randomly throughout shows. The sound and video quality is better than most clips posted on YouTube. Programming isn't viewed on a schedule like on traditional television, instead, a user searches for programming, just like the search box used on YouTube or Google Video.

Since its almost 3 am over here, I thought to myself that their must be something better to watch on Joost than on satellite TV. As most people don't know, I love documentaries. When I'm watching TV, the first channels I surf are the History Channel, Game Play HD, and the Discovery Channel.

I did a few random searches for documentaries, cartoons, and music videos. In terms of pickings, the documentaries list is pretty sparse. All I found were a bunch of amateur videos of 'famous' parties around the world and some guy with a show called 'Death Dealer' – strangely most of these shows appeared to be from Europe.

Next I searched for cartoons. A few shows popped up, not much variety though. A few included Adult Swim, which I suspect is just reruns of Aqua Team Hunger Force running 24/7, and something called 'Harveytoons Compilation'. I also saw some 'Ren and Stimpy' and some other random stuff that I never heard of.

My search for music videos also returned some very general search results. Tip for anyone trying to find something to watch, search for specifics. Searching for the term 'music videos' won't return a large search results list. I suspect this is because popularity or views hasn't been integrated into the search results yet.

So what's the verdict? Well their's definitely a huge variety of content, a bunch of random words searched came up with lots of programming. For example, searching for the word 'jump' yielded 40 episodes of the Guinness World Records Show. After watching an episode, I still haven't figured out the relevancy between the show and the word 'jump'. When compared to YouTube or Google Video, this service definitely could serve as a welcomed channel of distribution for video. But what about Bit Torrent or torrent trackers?

As sad as I care to admit this, torrent trackers still do a better job of offering users content that they're looking for. Even though Joost is still in beta and legal, the fact that a free semi-legal website serving illegal content does a better job then a multi-million dollar backed service says a lot about how far commercialization needs to go before it will be seen as successful and equivalent to its illegal counter-part. This doesn't mean I don't see potential in Joost. The biggest thing Joost offers over YouTube, torrent trackers, or any illegal download service, is better audio/video quality, better search integration, and most of all - instantaneous access. If Joost successfully gets these three elements executed, I think they will be able to make a niche for themselves in the expanding online video market (IPTV, streaming video, whatever you want to call it).