Installing voter data

Voter data installation - general comments

 

AdvoKit's voterfile installation process is a multi-step affair. To further complicate matters, there are two possible paths you can take to getting voter data installed. In each case, you must first tell AdvoKit how you want to map your voter data fields into AdvoKit's voter data tables, and then run a perl script from a command line environment to actually insert the voter data into your AdvoKit database. The method that was first developed is decidedly inferior to the second, and so we will only describe the second in detail. The basic steps are:

  • Prepare your voterfile offline and upload it to the voterfiles directory in your AdvoKit installation
  • Log in as campaign leader, map your voterfile fields to AdvoKit standard and custom voter data fields and generate a mapping file
  • From a shell account on the server, run the import.pl perl script to actually import the data

 

Voterfile preparation

Your voter data needs to be carefully prepared and have been uploaded to the server. Installing voter data is not an easy process when you are doing it for the first time, and if there are problems with the data, you will have to go into MySQL to purge the bad data before re-running the import. Therefore, it is extremely important to have your voter data prepared well to be used in AdvoKit.

The right way to go about this is to use an offline (desktop) application such as Microsoft Access to inspect your voter data for problems, if necessary manipulate the way some of your data is represented, and then export to a comma separated value file.

What is a comma separated value file?:

  • A text file with one voter record per line.
  • Each value is surrounded by double-quotes and delimited by commas (e.g. "value 1","value 2","value 3").
  • The first line of the file must contain the field names, and field names may not contain non-alphanumeric characters or white space.

(More details on the CSV file specification can be found here.)

AdvoKit has built-in support for a number of typical voter data fields. See the following links for information:

AdvoKit does not depend upon having data in most fields. It works best however if you provide at least a fully parsed name, address and date of birth for each voter (by "parsed" we mean separated into parts, e.g. first name, last name, middle initial). Most advocacy operations rely upon other fields for voter filtering, such as "political address" (ward, precinct, etc.), and party affiliation.

 

Some pitfalls we have seen:

  • Check the voterfile carefully for non-standard characters embedded within the data. Some databases are happy with e.g. linefeed characters within a field - e.g. there will be an "address" field, with linefeed characters separating line1 from line2. The AdvoKit import utility is not smart about handling these, and you will get undesirable results that may not be obvious to you at first..
  • The import utility crashes if no county name is provided for each voter - you can put anything in here, as long as it is not nothing
  • Dates of birth are only imported correctly if they are brought in as separate year, month and day of month - we recommend processing your birth dates so that year, month and date are in separate fields in your voterfile.
 
CSV voterfile upload to server

Upload this file (by e.g. using an ftp client) to your server. It should be placed in the "voterfiles" folder, which is typically located in the folder where all your installed advokit system files are stored.

As you will see in the next screen, it is possible to skip the step of manually uploading the voterfile to your server. However it only works for relatively small voter files, and is quite slow. We recommend not using this option.

 

 
Voterfile mapping

When you click on the "Upload Voterfile" link (you must be logged in as the campaign leader), you will first be asked where the voterfile is found:

In this example, we had uploaded a file called "vf.csv" to our voterfiles directory and picked its name from the drop-down list. (Again, we recommend against using the direct upload option.)

When you click on the "Choose/Upload File" button, AdvoKit will look at the file, read the field names, and then present you with a screen where you can indicate how your voterfile fields will be related to fields and tables in the AdvoKit voter database.

Each field in your voter file is listed in the left-hand column. For each voterfile field, you should choose the field name in the corresponding AdvoKit data table that you want it mapped to. Note that not all voterfile fields need to be mapped.

What to do with voterfile data that you need to use, but for which there is no corresponding AdvoKit field? This is what custom fields are for. The first step is to map the fields that you can, as above, to AdvoKit fields. Then click "Proceed >"

At this point, we can now choose among the unmapped fields from the voterfile to put into custom fields, as can be seen here. Be sure to choose a field type as well as give the new field a name.

IMPORTANT: We strongly recommend that you check the box at the top of the screen to have a mapping file generated for use with the command line import utility. Whether or not you use this option, you will still need to execute the voterfile import from the command line. Using a generated mapping file with the import utility is many times faster, and much more reliable, compared with having AdvoKit generate the data import script.

Assuming you did check the generate mapping file box at the top of the screen, clicking "Proceed >" will cause a mapping file to be generated and placed into your voterfiles folder on the server. It's useful to write down the name of the mapping file since you will have to type it into the command line in the import step.

 

Running import.pl

To actually import the voter data into your AdvoKit voter database, you will need to be able to execute a Perl script on your server. The script is called "import.pl" and the original can be found in the advokit-installer/utils directory. It probably makes most sense to extract this file on your PC and edit it there before uploading it to the server.

You will need to edit import.pl to include information that the script will need to connect to your MySQL database. This is the same information as was provided when you did the first installation step. Starting on line 20 of the import.pl script, you will see:

my $CFG_DBHOST = 'localhost';
my $CFG_DBPORT = 3306;
my $CFG_DBNAME = 'advokit';
my $CFG_DBUSER = 'root';
my $CFG_DBPASS = 'wayne';
my $CFG_DBPREFIX = 'ak_';

Edit these values to provide the correct host, database name, database username, password and table prefix.

Save the file and upload it to your voterfiles directory on the server.

Now log into a command window on the server (consult with your host if you do not know how to do this) and go to your voterfiles directory.

The syntax for running the import script (assuming you are in the voterfiles directory) is:

perl import.pl <mapping file name> <voterfile name>

The import.pl script will now execute. Feedback is provided as to how many records have been imported.

When complete, it is a good idea to inspect the data using e.g. phpMyAdmin.In particular, check to see if the number of ak_voter records is correct (should be the same as the number of original records in the voterfile). If there have been any problems with the import, you will probably need to delete the records in several tables in the database and re-do the import.

 

 
   

If you have a LOT of users or a SLOW server

Instead of " perl import.pl [mapping file name] [voterfile name] "
You can go with " nohup perl import.pl [mapping file name] [voterfile name] & "

This will make the script run in the background and log everything to a file called nohup.out.

You can then do " tail -f nohup.out" to keep up with the progress.

On our server, it's been running for a day now and we're just up to 200,000 voters.