More exercises in swirl

Once you’ve installed swirl and completed the recommended modules in R Programming in swirl, there are other swirl modules that will help you with course content.

Getting and Cleaning Data (dyplr, tidr, lubridate)

Data cleaning

swirl::install_course(“Getting and Cleaning Data”)

Statistical inference (probability, distributions, hypothesis testing)

probabilities

swirl::install_course(“Statistical Inference”)

Regression Models

regression models

swirl::install_course(“Regression Models”)

Exploratory Data Analysis

visual data exploration

swirl::install_course(“Exploratory Data Analysis”)

More swirl modules available

Online: http://swirlstats.com/scn/title.html

Setting up a Zotero workflow on MacOS

Zotero will only allow you to sync your library using either their service (not enough space) or via WebDAV. My Dreamhost backups storage doesn’t support webDAV. I tinkered around with some options (e.g., storagemadeeasy) to turn either my Dreamhost or my NAS shares into WebDAV folders. But, then, I learned that Mac OS doesn’t play well (or at least easily) with WebDAV, so I abandoned that strategy. Also, though my NAS has an iPad app and related tools to be able to access files remotely (via an encrypted tunnel and using a myCloud app), in practice, they haven’t worked seamlessly or at all.

At every turn, it seems that Dropbox was the only solid solution. Since I’m tired of moving files around to not fill up my free space, I’ve resigned myself to paying for Dropbox storage. Seems silly because I have lots of storage at home and 50GB at Dreamhost and I can get them to work together with lots of ad hoc tools and arrangements, but in this instance, I decided that convenience and reliability was more important than cost. Also, I’m really good at rationalizing money spent on technology.

I set up Zotero with my data directory in my home folder, but my document folder in my Dropbox. My group library files are synced using the Zotero service. I also have copied these to my personal library as separate collections. I installed the ZotFile plugin to manage the documents. It took some back and forth and tweaking to get the settings right, so I’m going to put them all here for future reference.

[In the process of researching how to do this, I consulted these sources, which all had some part of the necessary information. I did try to get things set up with Zotpad, but it kept crashing and wouldn’t work. Maybe someday there will be an updated, stable version.]

For now, the Zotero + Zotfile + Dropbox combo works. The goal is to keep a library of original .pdfs and also copies labeled _md.pdf that have been annotated. The tricky part is making sure both versions are linked to the Zotero database, which is what the Zotfile “send to tablet” features do within Zotero.

Essentially, when I add .pdfs to entries in Zotero, I also need to send them to the ipad folder (highlight all items, right click, manage attachments, send to tablet). This makes a second copy of the file in the ipad subfolder within my Zotero document folder inside Dropbox. Then to read the file on my ipad, I open iAnnotate, use the Dropbox connection to browse to the ipad folder and read/annotate the file. When, I’m done, I can save the file back to Dropbox. Then, next time I want to write/cite, I open Zotero. Zotfile creates to special search folders that will let you see if you have any tablet files that need to be reconnected (Tablet files modified) to their parent items in the Zotero database. So, before writing, I open that search folder to get the annotated files from the tablet back into the database and main library folder (highlight all items, right click, get from tablet). The result is one library with both my original .pdf plus the _md.pdf annotated .pdf. Zotfile also copies all the annotations into the Zotero notes field, too, which is nice. Also, informally, I can treat the ipad folder as a to-read folder, for those times when I’m sitting around bored with my ipad and nothing to do. [<- Ha! That’s what email is for. But, really, I’m never sitting around bored, and I never have nothing to do.] It sounds a bit clunky, but actually isn’t too bad if you just remember to send all new items to your tablet when you add them and then re-get them from the tablet the next time you open Zotero (hence the need for the full 100GB in Dropbox…. though you could set it up so just the ipad folder is in your Dropbox folder). The downside is that this workflow assumes a) you’re adding items to your library from your pc (not tablet) and b) you always send new .pdfs to your tablet from Zotero+Zotfile standalone.

Installing R & RStudio

If you have not used or installed R previously, follow the instructions on this page to install R, RStudio, and swirl before the first day of class. Note: If you are a Windows user, you must have an administrative account to install software.

We will be using R & RStudio during class and for all class assignments. If you do not have a laptop or use a tablet (with iOS or Android) for your academic work, please see me during office hours so that I can direct you to web-based alternatives.

Overview. The steps are generally:

  1. Install “R”, which is the underlying statistical software program that performs all the data manipulation and analysis.
  2. Install “RStudio Desktop”, which is a software program that provides a nice looking user interface (aka GUI) that has additional features that make analysis easier.

Install R. Instructions for installing R are online. Choose your Operating System (e.g., Mac or Win) and follow the instructions for installing the base R package.

If you are unsure which version of the software to download and install, post a question to the class discussion board. Include with your question a link to the page you are asking about so that we can see which choices you have. This StackOverflow page has some useful guidance about how to post a good question, and I would recommend trying to follow this guidance when you post to the class discussion board.

Download and install R before you go on to the next step.

Install RStudio Desktop. Visit this page to download the free version of RStudio Desktop.

Download and install RStudio Desktop before you go on to install the swirl package inside RStudio. The instructions include a list of the swirl learning modules that I recommend for new R users, particularly those who have not used a command script with a statistical software program before.

Installing and exploring R with swirl

swirl is an interactive learning tool that works inside RStudio. I recommend swirl exercises for all R users, but especially for students who have never used a command script with a software analysis program before.

Install the swirl package inside RStudio. You are now ready to install the swirl package in RStudio. A package is a small program that runs inside R/RStudio. Follow the steps below.

Launch or open RStudio. In the “Console” after the > type:

install.packages(“swirl”)

It should look like this:

screenshot of RStudio
Screenshot of RStudio interface

Launch the swirl package inside RStudio by typing in the Console:

library(swirl)

swirl()

If you choose Option 1. R Programming, you should see these options in the Console

swirl R Programming menu options

For our class, I recommend completing the following units for the first week: 1 – 2

If you get stuck, you can always type info() to get a list of additional commands, as illustrated below:

After you are done with a session, type

bye()

You can then either exit RStudio or continue your data analysis.

Running legacy Win-only software on Mac OS

One of the headaches of moving to Mac has been that I have one piece of statistical software that is Windows only. I could abandon the software and learn R, but I didn’t want to do anything rash. 🙂 Besides, at the Midwest, Fred assured me that he was able to get everything to run on his Mac. Armed with optimism and patience, I figured it out.

My goal: To run HLM 6 + Stata 10 on my MBP without having a dual boot or full out windows emulator.

Random aside: I use these together because Stata will just not estimate complex models with random effects / coefficients. I can get a simple model to run in a few minutes in Stata, but I’ve left a model with 2-3 cross-level interactions run overnight with no convergence. Using HLM 6 + Stata 10 (+ the hlm .ado file), I can use Stata to pass the model to HLM, which then passes the results back to Stata for post-estimation export/analysis.

The process: Install Crossover.

Install both HLM 6 and Stata 10 into the same Vista Bottle. [Maybe I could have put them in an XP bottle, but to be safe I went with Vista.] It took me a bit of reading to figure out that they need to be in the same bottle if they are going to interact with each other. Because neither HLM nor Stata are supported, you have to install them as “unsupported” apps. I used the HLM 6.04 cd-rom to install HLM with no problem, and then I used the old upgrade patch to 6.08 to upgrade the version just fine. Stata 10 was a bit more of a pain. I have a cd-rom and a perpetual license. However, the cd-rom has both Windows and Mac versions of the install, and my Mac wouldn’t recognize or mount the Windows files. The Crossover help files don’t really explain what to do if this is the case, and the various instructions that I found online to mount a Windows cd-rom from the Mac terminal didn’t work for me. Instead, I fired up my old pc and copied the install cd-rom to my NAS. [An important lesson here about switching from PC to Mac… copy any old software cd’s you might need/want to an external drive before you burn your PC. It will save you headaches later.]

Once they are both installed, you need to follow the instructions to enable HLM to talk with Stata 10. In this setup, this means that I needed to add the following line:

"PATH"="C:/Program Files/HLM6/WHLM.exe"

to the very end of the file located at

/Users/username/Library/Application Support/CrossOver/Bottles/HLM6/cxbottle.conf

I got it to work once, but since then have had issues getting it to work with new files due to versioning of the stata .dta file.

Moving on up….

… to Maclandia. We’ve replaced our Linux and Windows pcs with Macbooks. The world is a scary but shiny new place. We also upgraded our NAS to 4 TB. It’s taken me a couple of weekends, but I’ve finally gotten all my files moved, various tools installed, and a full range of backup provisions configured. I’m mainly putting everything down here so that I can remember all the steps for setting up B’s tools and backups and also in the event of harddrive armageddon.

First lesson

Finder doesn’t play nice with my old workflow. In the past, I would sometimes put a working folder in my Dropbox, and then when the project was finished, copy it back into my home working directory. Windows had a nice way of merging or updating the changed files, prompting me if a file already existed. In Mac OS, I learned quite quickly that if you copy a folder over an existing folder of the same name, it will delete the old contents and replace all the files with the new contents. As a result, I will need to become proficient with rsync and the terminal to do this merging effectively. [NOTE to self: Figure out the correct options/commands and post here.]

Setting up a new backup or redundancy plan

When I lived in PClandia, I never backed up my entire OS, based on the philosophy that if something happened to my harddrive and/or OS, I would want to reinstall my entire OS anyway (esp. since Windows tends to get bogged down over time with installs and uninstalls, registry edits, etc.). My old strategy was to back up an exact copy of my user folder (within which all my user profiles, data files, and copies of downloaded software, etc.) to our NAS (RAID 1, 2TB) drive, and then have automatic backups run to a 3TB USB connected drive nightly. I’m still not sure about all the backup lingo, but I essentially had it set up so that the NAS was a mirror copy (including deletions of my pc) but the USB backup would not delete any files but add new copies/modifications to the drive, renaming the old versions of files that had been updated.

A couple of things prompted me to rethink this strategy now that I live in Maclandia. First, there’s Time Machine. I understand it can be buggy, so I don’t want to rely on it alone. At the same time, if there were a hardware failure, it would be nice to be able to just reload everything to a new machine (and since Mac OS doesn’t have the bogged down with time/installs problem of Windows – or so I’m told – then I might not want to reinstall everything). This means I wanted to set up TM on the NAS, but also have my data backed up the old way, too. I want my home directory backed up separately in the event that my OS gets corrupted (again, remember, I’ve lived in PClandia for a long time), I can easily reload all my data to a new OS install.

My strategy
I created a user share on my NAS for my TM backups based on these instructions. I had initially used the built in TM features of the NAS, but got a couple of errors (e.g., the TM had problems ‘verifying’ the backup and then would start a new one) when we were trying to back up 2 MBPs to the same TM share. Rather than try to troubleshoot, I figured it might be safer just to keep them in separate NAS shares anyway. [Downside: In my version of NAS firmware, there’s not an option in the user interface to apply a quota to these user shares, though I can see a user quota file in the admin directory that I can probably edit to apply a quota. So, that’s a to-do item.] These TM backups won’t be backed up, since they are the backup.

Then, I set up rsync + Automator to backup my user home folder to another NAS share nightly. Right now, I followed these simple instructions. The downside of the simplicity is that it doesn’t create a log file or email me the status of the backup, or errors. It looks like these more complicated instructions would do that, but I think I’m going to see how the simple version works for a bit before tinkering with it. I’m also still using the NAS’s backup tools to do an incremental backup to the USB drive like before.

Off-site backups and access to files from iPad

In addition, I want to save my work files offsite and also access them from the iPad. I have two options here: my backups user on Dreamhost (50GB) and/or Dropbox (free 5GB or paid 100GB). This got a little complicated because I also wanted to integrate Zotero (on MBP) and iAnnotate (on the iPad) into my workflow.

Zotero will only allow you to sync your library using either their service (not enough space) or via WebDAV. My Dreamhost backups storage doesn’t support webDAV. I tinkered around with some options (e.g., storagemadeeasy) to turn either my Dreamhost or my NAS shares into WebDAV folders. But, then, I learned that Mac OS doesn’t play well (or at least easily) with WebDAV, so I abandoned that strategy. Also, though my NAS has an iPad app and related tools to be able to access files remotely (via an encrypted tunnel and using a myCloud app), in practice, they haven’t worked seamlessly or at all.

At every turn, it seems that Dropbox was the only solid solution. Since I’m tired of moving files around to not fill up my free space, I’ve resigned myself to paying for Dropbox storage. Seems silly because I have lots of storage at home and 50GB at Dreamhost and I can get them to work together with lots of ad hoc tools and arrangements, but in this instance, I decided that convenience and reliability was more important than cost. Also, I’m really good at rationalizing money spent on technology.

I set up Zotero with my data directory in my home folder, but my document folder in my Dropbox. My group library files are synced using the Zotero service. I also have copied these to my personal library as separate collections. I installed the ZotFile plugin to manage the documents. It took some back and forth and tweaking to get the settings right, so I’m going to put them all here for future reference.

screenshot

screenshot1

screenshot2

screenshot3

screenshot4

[In the process of researching how to do this, I consulted these sources, which all had some part of the necessary information. [I did try to get things set up with Zotpad, but it kept crashing and wouldn’t work. Maybe someday there will be an updated, stable version.]

For now, the Zotero + iAnnotate workflow will do what I want (though if Zotpad worked, it would be closer to the ideal). The goal is to keep a library of original .pdfs and also copies labeled _md.pdf that have been annotated. The tricky part is making sure both versions are linked to the Zotero database, which is what the Zotfile “send to tablet” features do within Zotero. [Zotpad would do this at the tablet, rather than in Zotero, but since it doesn’t work, it’s not really an option.]

Essentially, when I add .pdfs to entries in Zotero, I also need to send them to the ipad folder (highlight all items, right click, manage attachments, send to tablet). This makes a second copy of the file in the ipad subfolder within my Zotero document folder inside Dropbox. Then to read the file on my ipad, I open iAnnotate, use the Dropbox connection to browse to the ipad folder and read/annotate the file. When, I’m done, I can save the file back to Dropbox. Then, next time I want to write/cite, I open Zotero. Zotfile creates to special search folders that will let you see if you have any tablet files that need to be reconnected (Tablet files modified) to their parent items in the Zotero database. So, before writing, I open that search folder to get the annotated files from the tablet back into the database and main library folder (highlight all items, right click, get from tablet). The result is one library with both my original .pdf plus the _md.pdf annotated .pdf. Zotfile also copies all the annotations into the Zotero notes field, too, which is nice. Also, informally, I can treat the ipad folder as a to-read folder, for those times when I’m sitting around bored with my ipad and nothing to do. [<- Ha! That's what email is for. But, really, I'm never sitting around bored, and I never have nothing to do.] It sounds a bit clunky, but actually isn't too bad if you just remember to send all new items to your tablet when you add them and then re-get them from the tablet the next time you open Zotero (hence the need for the full 100GB in Dropbox.... though you could set it up so just the ipad folder is in your Dropbox folder). The downside is that this workflow assumes a) you're adding items to your library from your pc (not tablet) and b) you always send new .pdfs to your tablet from Zotero+Zotfile standalone. [Again, if Zotpad worked, you could use it to do the linking of files to database tablet-side rather than pc-side, which would add the flexibility to add/edit Zotero library items on the ipad.] One more off-site backup, just because I’m paranoid

My home folder is synced to my NAS, and then backed up to an external USB. My Dropbox folder includes all my work files, including .pdf library, and so everything is also synced to Dropbox. [My Dropbox folder also includes some other shared folders with Brian and others that are not part of my work directory.] Because my work files are the most important, I’ve also set them to back up (less my .pdf library) to the Dreamhost backups user using rsync and automator. If I worked at a University that provided secure data backups for faculty (ahem!), I could have tweaked these instructions to use that space. [If my university provided secure data backups + WebDav, I probably could avoid Dropbox altogether. (sigh)] Here’s the instructions I followed to backup just my work folder to Dreamhost, with a variation on the iCal/Automator step at the end:
http://www.commandlineidiot.com/2007/how-to-backup-your-mac-to-a-dreamhost-server-part-1-of-2/
http://www.commandlineidiot.com/2007/how-to-backup-your-mac-to-a-dreamhost-server-part-2-of-2/
http://esilva.net/2013/07/automator2/

Christmas stories

I’m sure every family has a collection of stories they share around Christmas. One of the favorites in my mom’s family is about the time, when my mom was in elementary school, her dad told her they weren’t having Christmas so that she would stop bugging them about her presents. He told her they couldn’t afford Christmas that year.

My mom told her teacher that they weren’t having Christmas because they didn’t have any money. So, my mom was allowed to drag the classroom tree home, and apparently it was a sad, pitiful thing. Grandma and Grandpa didn’t understand why the teacher gave the tree to my mom.

They didn’t understand later, either, when various church and charity groups from around Flint brought baskets of food and gifts to their house. And, despite my Grandma’s protestations that they didn’t need charity, everyone insisted they accept the goods because everyone deserves to have Christmas.

Eventually, they figured out what had happened.

Customizing Emacs

I figure one way to become more familiar with Emacs is to customize some of its settings. Of the various pages I’ve found on the web that describe different customizations, I’ve found these two to be the most useful.

First, I wandered around the Emacs customization menu by clicking the Customize Setup link on the original home screen. There were so many options there that it was a bit overwhelming, even if you browse using the M-x customize-browse command. I did use the menu to “Save for future sessions,” which created a virtually empty .emacs file for me. I figured that way the file would be saved in the correct place (i.e., my home directory).

Next, I thought I’d browse the options for changing the color scheme. These instructions were the most helpful for installation. Once installed, I typed M-x color-theme-select and then clicked/RET on the different themes until I found one I liked. Not too complicated. Seems you can customize your own theme using this tool, too.

I poked around online and found several examples of .emacs.el or init.el files. (Seems like it’s a “thing” to post your Emacs customizations online…. kind of like the equivalent of a g33k bicep flex.) I also read more about the pros/cons of Emacs vs. WinEdt. It seems like one of the big cons of Emacs for Windows users is that several of the keyboard shortcuts and/or default behaviors are so different in Emacs that it creates problems if you’re moving back and forth across tools. Sure, you could customize Emacs to get rid of some of those annoying default behaviors, but, boy, does that seem like a lot of work. And, while it’s easy in linux to install packages from the command line, in Windows it’s a several step process.

What I’ve learned so far? I could be a super g33k and use Emacs if I really wanted to, but I’m not sure I really want to. As far as tools for writing and editing text files (research articles and R or Stata code) go, there are a range of options from g33kiest (Emacs+AUCTeX, with apologies to Chris) to smart but pragmatic g33k (WinEdt) to lazy g33k (LyX, with apologies to Justin) to least g33k (MSWord). [Actually, I guess a real geek would write LaTeX so natively they could just use Notepad2, or heavens forbid, the notepad that comes with Win. Or, I guess a real geek would be using a linux box anyway, but I digress…]

So, while I am confident I could learn to use Emacs, and maybe it really would be transformative and change my life forever, but I think I’m going to be more pragmatic and move on to WinEdt for now.

Updated to add: FWIW, I’ve saved the most useful pages about setting up Emacs on Windows to my delicious.

Follow-up to post about LaTeX, etc.

Yesterday, I provided a rough outline of the steps for installing Emacs + AUCTeX in Windows 7.

Upon further reflection, I realized a few things about the process. First, most of the documentation for these tools is outdated and/or poorly written (at least for recent versions of Windows). Second, it’s way easier to install this stuff than the documentation would lead you to believe. Third, using the software itself is also way easier than the documentation would lead you to believe or than you might think yourself.

Specific examples?

Installing Emacs in Windows 7 is a three step process: Download file. Unzip file into preferred directory (e.g., C:\Program Files\emacs\). Set the HOME property to point to %USERPROFILE%.

Installing AUCTeX? Download file. Unzip file into the Emacs directory (above). However, if you read the AUCTeX manual, you’d think this was a 30 step process.

Everything seems to work ok. AUCTeX loads when a .tex file is being edited, and works. This setup even comes with a spell checker already installed, even though some of the [outdated] online documentation would lead you to believe that you have to install one separately. Perhaps its broken in a way I’m too n00bish to recognize, but so far, so good.

I went through the Emacs tutorial and the first 2 of these tutorials, and everything seemed pretty straightforward. These ref cards for AUCTeX and Emacs seem to have all the essentials. It’s just a matter of learning different keyboard shortcuts than the ones I’ve been using in Notepad2 or Word.

On the other hand, there’s this mystery .emacs file that is supposed to be where I put my personalized settings for using Emacs. I see an .emacs folder, but not a file. So, that’s a mystery to me, but I’m sure I can figure it out. So that’s the next step: Customize my environment, export my existing bibliography files to a .bib file, and see if I can create a custom formatting .bst, or whatever it’s called.

Installing and learning LaTeX

Long on my to-do list has been to use LaTeX, mainly for its BibTeX features. I’ve avoided it for so long because I’m pretty adept at MSWord. However, with Office2010 and that blasted ribbon, they broke all my keyboard shortcuts for menu items (imagine grumpy old professor shaking hand at the gods and yelling).

As a first step, I emailed Chris (not to be confused with this CL) to ask for suggestions, which he graciously provided. Then, having narrowed things to Emacs+AUCTex (Chris’s strategy) and WinEdt, I turned to FB, which yielded additional suggestions including Sublime Text and LyX. The latter is apparently a good GUI tool, but the exported raw .tex can be clunky according to those who have used it. (I imagine it’s not unlike creating a webpage in Word, which you can do, but I wouldn’t recommend.)

I decided to install Emacs, LyX, and WinEdt, but I think I’m going to start with Emacs and try LyX and WinEdt when I inevitably get stuck.

Figuring out which version and how to install Emacs also took a bit of effort, particularly because a lot of the how-tos, which may include links to packages, aren’t necessarily always linking to the latest stable version. I found these instructions to be the clearest. However, I went to the original source for the install files. It seems that AUCTeX isn’t compatible with the most recent version of Emacs, so I had to go back and re-install Emacs. Typical (for me at least).

Here are the (abbreviated) steps I went through:
.5. Read several introductions to LaTeX, including some recommended by Chris.
1. Install MiKTeX.
2. Install WinEdt.
3. Install LyX. Since I already installed MiKTeX, I had to navigate to the tex.exe file in the miktex\bin\x64 folder during LyX install. It seemed to find and get all the missing packages and automatically install them in the correct MiKTeX directory (thank goodness).
4. Install latest version of Emacs.
5. Set HOME property.
6. Install version of Emacs that works with AUCTeX.

Next step(s): Check on spellchecking features, setup preferences, etc.

Edited to add: Here’s my first document. 😉