Tuesday, November 5, 2013

Using Sass with Compass and a large development team

Sass, the Syntatically Awesome Stylesheets, has been a great help to our user experience team.  We've grown from a one-man PHP shop to a team of 8 with 2 focused on UX.  This growth has forced us to improve our processes.  Two major improvements were the move from subversion to git and from hand-coded CSS to using Sass.  Unfortunately, these improvements did not come easily.

Sass is similar to compiled programming languages in that you write in a different language than the computer executes.  In the case of Sass, that executable format is CSS, where Java would be bytecode and C would be machine language.  However, in Sass, changing a single source file usually results in multiple output files being updated.  And updating 2 source files can update the same output file.  So, source code management can be much more difficult.

We had to choose whether to keep the compiled CSS in git, or to ask each developer to install and run compass.  We decided to keep the compiled CSS in git.  This was for a number of reasons.  Primarily, we wanted to have repeatable code to test from.  I also wanted a simple release process, and to not have to maintain another compiler on production.  Finally, we wanted to keep the developer setup as simple as possible, to reduce cost to new hires and to reduce the number of moving parts.

The Problem - In a NutShell

Here's what would happen; developers A and B would both pull.  They would both make changes to the sass, but not in the same sass files.  Compass would automatically compile the CSS.  Developer A would check in his Sass and his CSS.  Developer B would pull and get conflicts in the CSS.  So, he or she would have to turn off compass, back-out the local changes, re-get (and fight with rebasing) the source, turn compass back on, re-compile, then check-in.  Uuuggghhhh.

This are easy to manage if it happened a few times a month, but it would happen on every checkin.  Not good.

We looked for git fixes.  Carwin Young came up with a workable solution using git - http://carwinyoung.com/2013/01/23/avoiding-git-conflicts-involving-compiled-sass/.  The short directions are:

git add some-sass.scss
git commit -m "A helpful and descriptive message."
git checkout -- compiled-css.css (this one checks out the desired file from HEAD)
git pull --rebase
compass watch / sass watch scss:css / some gui stuff (whatever)
git add compiled-css.css
git commit --amend
git push remote branch
Uuuggghhhh.  This solution is, in my opinion, one of the reasons that people hate git.  Our UX developers don't have the time to become git experts.  Even our software developers won't do this.  Heck, even our SCM dude won't do this.  No disrespect meant for Carwin Young, who is obviously a super-smart guy and went through a lot of work to teach us something useful.

The Desired Solution

We want a solution where the UX designers can use compass to compile, but everyone else can just get the CSS from git.  We want to have a consistent compilation routine, so that we can have good assurances that the tested code will act the same in production.  Beyond that, we want to be able to test the final release with code that is exactly the same in production.

Also a top priority was ease-of-use.  I don't mind spending some time to get everything set up on the server.  I'd like to keep the workstation setup to a minimum as well, but I didn't mind 10 minutes for each new employee.  However, I want on-going development, test and deployment to be as simple as it was before.

The Solution - A Submodule and a Hook

We decided to store the compiled CSS in a separate git repository.  The UX designers would simply not mount the repo, and instead would use compass to compile the sass to CSS on the fly.  Everyone else would mount that repo on their CSS directory.  That only left two problems:

  • We'd have to remember to update the CSS project separately from the main project.  Which means 2 pulls instead of one, every time.  I want to avoid extra process for the devs, deployment and QA.
  • We'd need a way to keep the CSS updated properly

Our Directory Structure

We use the play framework, which uses a standard layout:

You can see that the css directory, which is under public/stylesheets/css, is a repo, not a normal directory.

The Submodule

With git, you can mount a repo as a "submodule" of another repo.  This allows you to treat the child repo as part of the parent, so stuff like pulls and fetches work transparently.  To do this, we did:
  • git submodule add -b develop git-url public/stylesheets/css
  • Add this to your .git/config
    • [fetch]
    • recursesubmodules = true
    • [pull]
    • recursesubmodules = true

The first command adds it as a submodle.  The next part tells git to always use recursesubmodules when fetching or pulling.  These together give you an almost seamless experience.  This is the only thing you have to run on every workstation.

The Hook

We setup a hook on the server side.  The purpose of the hook is to perform these actions

  • Find the branch and comment for the current push
  • Checkout the right branch of the code
  • Compile the sass into CSS using compass
  • Commit and push the compiled CSS to the git repo

We also needed to configure the build environment.  We have 3 important directories:

  • The git repo for the main project
  • The git repo for the project's css
  • A directory which we'll use for compiling, with the build script and the checked-out project inside it

This is the contents of the post-receive hook


# read the arguments from stdin
read oldrev newrev refname

# get the branch and the message from the commit ID, which is stored in $newrev
branch=$(git branch --contains $newrev|sed s/^\ *//|sed s/\ *g//)
message=$(git log -n 1 $branch $newrev|tail -1|sed s/^\ *//|sed s/\ *$//)

cd /usr/local/build
echo "sudo /usr/local/build/run_compass.sh $branch \"$message\""|at now

All it does is gather the branch and checkin message and then executes run-compass.sh.  Note that it pipes the command to the unix "at" command.  Compass is slow to run and made pushes painfully slow.  I had problems getting git to run in the background until I came upon this hack on stackoverflow.

This is the contents of run-compass.sh


DIR=`dirname $0`

# basic environment setup
cd $DIR/project
umask 0022
env --unset=GIT_DIR

# get the correct branch and make sure we don't have any local changes
git checkout -t remotes/origin/$BRANCH || git checkout $BRANCH
git reset --hard
git pull

# do a clean compile
compass clean -c $DIR/compass.rb
compass compile -c $DIR/compass.rb

# add the files then commit and push
cd public/stylesheets/css/
find . -type f -name "*css" -exec git add '{}' \;
git commit -m "$MESSAGE" .
git push

The Final Flow

A UX developer makes a change to his or her sass file.  Compass automatically compiles it and reloads the page.  When it all looks good, the developer commits and pushes the change to the develop branch in the project repo.  He or she marks the ticket as done in our bug tracking package.

In the background, the server switches to the develop branch and pulls. It compiles the sass to CSS.  It then commits and pushes the CSS to the develop branch in the project-css repo.

Other developers pull regularly.  When they do the next pull, it automatically pulls the new sass and the new CSS just like you would expect.  If they hadn't heard me talking about it, they would never know that something fancy is happening.

QA picks up the ticket.  He or she cherry picks the sass file change into the release branch and pushes.  In the background, the server switches to the release branch, pulls, compiles the sass and pushes it to the release branch in the project-css repo.  Now the CSS changes for that ticket are in both develop and release.  QA has to pull again, but that is the only change to the process for them.

When QA is ready to deploy the release, he or she merges into master.  In the background, the server switches to the master branch, pulls, compiles, pushes to the master branch on project-css.  QA switches locally to master and pulls, just like normal.  They do a couple days of testing before we deploy.

When we deploy, we pull the latest changes, just like before.  We go through our normal deploy steps just as before with no change to the process.

(This is a little simplified.  We have jenkins doing tests, developers fixing issues from the test, and a ful suite before deployment.  We have a deployment process.  We have bug reviews and code reviews and iteration planning, etc.)

The only changes to the developer, QA and deployment process are that QA has to do an extra pull after merge, and you still have to change the branch in each area.

What isn't Working

The submodules are working fine for pulling, but they don't follow when you switch branches.  Maybe I'm missing a setting in .git/config

Work Left to Do

There are race conditions galore when multiple developers check in simultaneously.  Luckily, we're a small team, and it fixes itself on the next check-in.

User management is not working.  In the post-receive hook, I run the compile script as root.  This is a bad idea, but I haven't had time to fix it yet. I would love to have the CSS checked in under the same user as the sass was.

Better error handling.  I have it executing unnecessary commands so that setup is easier.  For instance, I should make it check for an existing local branch.  Also, we just blithely continue on after errors.  I am not an expert bash programmer.

Monday, July 16, 2012

How QuickBase Google Contacts Works

QuickBase to Google Contacts is our most advanced plugin yet!  It is actually a whole new webpage which loads your QuickBase contacts push them into your Google Mail's addressbook.

We use your existing QuickBase session to authenticate you against QB.  We use Google's OAuth to authenticate you against Google, and to let you choose between different accounts.  Once this is all setup, QB2GContacts will load your contacts from the configured report and match them against Google Contacts records.  You'll have the chance to select contacts, and twiddle mappings.  Then, click "Go!" and you're done.

We're hoping to use this structure for more applications.  Google Calendar is an obvious example, as is a bulk emailer like MailChimp. 

How QuickBase Form works

QuickBase has an excellent Form Wizard.  If you don't like the Form Wizard, it's pretty easy to code up an HTML form using their API.  But, these approaches have some downsides. 

For one, you need to modify the form on your website.  Many websites are built using a CMS, which makes using the form wizard tough.  Sometimes you've spent thousands of dollars working with a web design firm, and you're not psyched about having to change their work. 

For another, the form has to actually post to QuickBase.  So, your customers get sent to QuickBase instead of your own landing page. You can supply a "return url", which will tell QuickBase to send your customers back to your page.  Which is totally transparent and works...most of the time.  Except if QuickBase is having an (infrequent) outage.  Or you have permissions setup wrong.  Or someone deleted the target database. 

What if you have fields  that you want to record locally and not keep in QuickBase?  Perhaps there's secure information, or a file upload, or voluminous analytical data.  With the form wizard this is impossible, and even with the API it's tough without running server-side code in PHP or ASP. 

QuickBase Form is a simple way around these issues.  It works as a JavaScript plugin, using the popular jQuery javascript framework.  It's simple to install, just copy some boilerplate to the of your HTML document and set up a simple field mapping.  It doesn't change the form or any CSS or HTML. 

It works by intercepting the form's submit.  When you click your forms "submit" button, QBForm bundles the data according to your mapping and submits it to QuickBase.  Then, whether it succeeds or fails, the form is submitted like normal. 

Rebario QuickBase Tools

I've been talking about cloud integrations for a while.  Due to job changes, I had to take a hiatus from it for a couple years.  But, it's back now!

Introducing Rebario, my new company which is focused on delivering QuickBase extensions.  We are currently in beta and looking for beta testers.  We have 3 products:
  • QuickBase to Google Contacts - a system to export your QuickBase contact records to your Google Mail's addressbook.
  • QuickBase Form - an easier way to integrate a website's "contact me" form
  • Change Logger for QuickBase - an extension to log changes to your QuickBase databases

Rebario is built upon a dream I've had for a long time.  It's an SaaS-style service.  That means it's
  • Self-Service - A normal business user should be able to sign-up, install and run the software without having to purchase an installation contract or any professional services
  • Monthly Billing - All billing is automatic, with a simple month-by-month plan.
  • Affordable - Since we don't have a ton of expensive professional services required, we can charge a small monthly fee.  Our most expensive (change logger) is $25/month.  Compare that to the large fees of the professional services firms, which typically run $2000 per application plus a large yearly maintenance fee.

Please come check us out at http://www.rebario.com

Friday, April 9, 2010

Email Monitoring with Mailive

Notification has the occasional hiccup with email. Usually, it's an upstream problem. Maybe the SMTP mail router is down, or there's a network problem. Sometimes there's an actual problem, like the one where Oracle Enterprise Manager sends non-compliant email which chokes JavaMail. Sometimes it's just that someone wants the warm and fuzzies, but the logs don't give it to them.

Mailive is an email monitoring system, similar to Gomez or AlertSite. But, it goes a bit deeper. Instead of just hitting the site and expecting a positive response, Mailive sends and email and expects a response.

You can set this up with Reliable Response Notification like so:
  • Create a new user for Mailive.
  • Add an email device which sends to mailive@knowledgefront.com
  • Find the Mailive user's email address. You can find this in the "Send a New Notification" screen. Search for a user, then click the blue arrow to find the email.
  • Give this email to Mailive.

You should be good to go. This will provide a full monitor; everything from MX records to network connectivity to Notification's POP or SMTP service, to the core Notification engine.

Wednesday, January 27, 2010

QuickBase Sync in Progress - Google Contacts, Google Calendar and QuickBooks Sync

I'm working on a new product. It's a SaaS application which connects your QuickBase CRM and Project Management tools to external data sources. My first project is Google Contacts. I actually have that running, and a good first stab at Google Calendar. I'm working on the web frontend now. After that is Chargify integration.

If anyone is interested in synchronizing your QuickBase database to Google Contacts or Google Calendar, let me know.

After I have these up and running, my next task will be QuickBooks. There have been a lot of attempts to do QuickBooks->QuickBase integration. There have been some successes, but nothing cheap and/or easy. My plan is to provide a very basic, good enough for 80% of use cases, integration. Let me know if you want to participate in feature requirements.

Monday, December 21, 2009

QuickBase PowerTools

Reliable Response is proud to release a groundbreaking new tool for users of the QuickBase platform. QuickBase PowerTools is an extension to your Firefox and Chrome browsers which add exciting new capabilities, including

  • Currency Conversion

  • Company lookup with Dunn & Bradstreet

  • One-click access to Google Maps for all addresses

  • Click-to-Conference using Twilio conferencing

Click-to-Conference is a tool for sales professionals who need quick and easy telephone conferences with their prospects and customers. It allows users o add customers to a conference with a simple click of the button. Click-to-Conference is free with advertising or ad-free with a small monthly subscription.

Visit http://www.reliableresponse.net/drupal/node/7 to install QuickBase PowerTools now.

Because of its innovative design, QuickBase PowerTools works with any QuickBase application or table with no changes to your QuickBase. It seamlessly adds new capabilities on the fly with no setup or maintenance costs.

Please visit us at www.reliableresponse.net for more QuickBase applications, tools and services.