Tuning AWS SQS Job Queues

On a project recently, we were debugging a slow user experience during file upload and after investigating, found that the culprit was mainly our queue configuration. We were using Amazon’s Simple Queue Service (SQS) for queueing and this post goes over our debugging process and the lessons learned for tuning SQS along with some more general take aways about background jobs and queue design. Our use case The user flow in question here is a contract signing flow where the user uploads a file to be prepared for signing. Read On →

Find and Replace in Vim Without Plugins

Find and replace across files is one of those features I frequently see Vim users reverting back to something like Sublime or another more GUI driven editor. This apparent weakness in Vim bothered me so I went in search of how to find and replace across multiple files or directories in a project without leaving Vim. It turns out there are built in building blocks we can use to build up this command. Read On →

Adding a Boolean Column to an Existing Table with Alembic and SQLAlchemy

Ever find yourself needing to add a new boolean column to an existing table? I do! Recently I’ve been using SQLAlchemy and Alembic to manage migrations and I frequently find myself looking up how to achieve this task. Below is a recipe for adding a boolean column to an existing table. We want to avoid the three state boolean problem so we’ll be making this column null: false. Since we’re adding this column to an existing table we have existing rows that will have empty values for this new column. Read On →

HATEOAS with Ember Data

This post was originally published on thoughtbot’s blog. Ember Data introduced strong conventions around how to structure API responses. While these conventions allow us to move quickly, there are additional steps we can take to minimize the coupling between the front end and back end. Using concepts from HATEOAS (Hypermedia as the Engine of Application State) we can make our Ember applications more flexible and resilient to changes on the server. Read On →

Datetime Parsing with Go

While working on a side project recently (https://github.com/srt32/sfpd_crime) I wrote a CSV parsing script in Go and I realized I did not understand how to parse a non-standard string into a time.Time type. Here’s what I learned. My date data looked like the following: 12/18/2014 07:00:00 but the Go reference time is in the following format: Mon Jan 2 15:04:05 -0700 MST 2006. So, how do we go from my format to the required format? Read On →

Speed Up JavaScript Capybara Specs by Blacklisting URLs

This post was originally published on thoughtbot’s blog. On a project recently, our full test suite began to crawl (taking ~9 minutes instead of less than 1) on our local machines running OS X but ran normally on CI. This slowdown took our productivity to near zero. We discovered our Capybara specs with js: true set were the culprit but we couldn’t figure out why. read more…

Postgres Window Functions

This post was originally published on thoughtbot’s blog. We recently ran into a case where a join was getting out of hand and we were struggling with how to rein in the size of it. Then we found window functions. Window functions (Oracle calls them analytic functions) are a part of the SQL standard and this post will explore how to use them in Postgres. Let’s see how they work and what kind of problems they can help us solve. Read On →

An Introduction to Go Tools and Clean Code

I’ve begun learning Go and while the language has been fun to learn and use, figuring out all the tooling has been less than simple. Here’s a quick overview of a sample tool chain to get you started with community acceptable code. Step 1: the compiler Coming from a dynamically typed background, types and compile errors have been enjoyable (I hope the feeling lasts). Step 2: go fmt This tool automatically formats Go source code and as the docs explain, it: removes minor formatting concerns makes code consistent across the community allows diffs to show only real changes removes the ability to squabble over spaces or tabs It’s important to realize that this tool manipulates your code. Read On →

Writing Functional SQL in PostreSQL: Don't do it

During some recent PostgreSQL rabbit hole digging I discovered the concept of ‘functional query notation’ and it mildly blew my mind. The Question Assuming we have two tables: contacts table having columns: id and name conversations table having columns: id and contact_id (FK to contacts.id) The following two queries return the same data: dot notation (pretty typical): SELECT contacts.name, contacts.id, conversations.id FROM contacts INNER JOIN conversations ON contacts.id = conversations.contact_id; and functional notation (pretty atypical): SELECT contacts.name, contacts.id, conversations.id FROM contacts INNER JOIN conversations ON id(contacts) = contact_id(conversations); This second approach raised some questions: Why and how does this second approach work? Read On →

Why Postgres Won't Always Use an Index

This post was originally published on thoughtbot’s blog. We were analyzing slow responses on a project recently and found ourselves questioning the performance of some PostgreSQL queries. As is typical when digging into queries, we had some questions around what the query was doing. So, we used our handy EXPLAIN command to shed some light on the database’s behavior. Upon inspecting the query it turned out an index we had created was not being used. Read On →