Dev Workstation Setup

This page describes how to setup a Mac OS X workstation to work on Narrative's projects.

Common System Tools

Homebrew

Homebrew is the de-facto package manager for Mac OS X. Most other setup instructions depend on having brew installed.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Setup ~/.bash_profile.d

The goal is to make it easier to customize environment variables by adding/overwriting files in the ~/.bash_profile.d directory. Most setup instructions depend on all files in the ~/.bash_profile.d directory to be automatically sourced when the shell is started

mkdir -p ~/.bash_profile.d
cat > ~/.bash_profile << 'EOF'
for f in $(find $HOME/.bash_profile.d -type f | sort) ; do
  source "$f"
done
EOF

bash-completion

Programmable completion functions for bash

Most commands (git, ...) provide advanced tab-completions to make your life easier. To make use of this mechanism, it needs to be activated.

Install

brew install bash-completion

Enable

cat  > ~/.bash_profile.d/bash_completion << 'EOF'
if [ -f `brew --prefix`/etc/bash_completion ]; then
    . `brew --prefix`/etc/bash_completion
fi
EOF
source ~/.bash_profile.d/bash_completion

asdf-vm for Python, Ruby, NodeJS, Terraform

asdf-vm is a CLI tool that can manage multiple language runtime versions on a per-project basis. It is like gvm, nvm, rbenv & pyenv (and more) all in one!

Each project manages its own .tool-versions and specifies the required versions it depends on, but let's install default versions for the major languages we depend on.

brew install asdf
echo "source $(brew --prefix asdf)/asdf.sh" > ~/.bash_profile.d/zzz_asdf
source $(brew --prefix asdf)/asdf.sh
echo "legacy_version_file = yes" > ~/.asdfrc

# Install Python
asdf plugin-add python https://github.com/danhper/asdf-python.git
asdf install python 3.9.2
asdf global python 3.9.2

# Install Ruby: still required for working with Jekyll doc that we
# are phasing out in favor of nuxt-content
asdf plugin-add ruby https://github.com/asdf-vm/asdf-ruby.git
asdf install ruby 2.7.2
asdf global ruby 2.7.2

# Install Node
brew install coreutils gpg
asdf plugin-add nodejs https://github.com/asdf-vm/asdf-nodejs.git
bash ~/.asdf/plugins/nodejs/bin/import-release-team-keyring
asdf install nodejs 14.16.0
asdf global nodejs 14.16.0

# Install terraform
asdf plugin-add terraform https://github.com/Banno/asdf-hashicorp.git
# some projects still depend on v0.12, but we are migrating to v0.14
asdf install terraform 0.12.30 
asdf install terraform 0.14.7
asdf global terraform 0.14.7

Docker

Docker is used for instance to run local PostgreSQL instances.

brew install docker docker-compose docker-machine-driver-xhyve
brew cask install virtualbox
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

PS: docker-machine is now in maintenance mode, so we need to migrate to Docker Desktop at some point.

Git

Git is a distributed version control system

Install

brew install git

Configure

The minimum configuration would look like

git config --global user.name "Your Name"
git config --global user.email you@narrative.io

Generate Github SSH Key

ssh-keygen

Then manually upload ~/.ssh/id_rsa.pub to github

Install hub

Hub provides github-related shortcuts

brew install hub
echo 'alias git=hub' > ~/.bash_profile.d/hub
source ~/.bash_profile.d/hub

direnv

direnv is an environment switcher for the shell. It knows how to hook into bash, zsh, tcsh and fish shell to load or unload environment variables depending on the current directory. This allows project-specific environment variables without cluttering the ~/.profile file.

brew install direnv
# Load direnv after everything else has been loaded. In particular after `rbenv` and other shell extensions that manipulate the prompt
echo 'eval "$(direnv hook bash)"' > ~/.bash_profile.d/zzz_direnv
source ~/.bash_profile.d/zzz_direnv

AWS Tools

AWS Command-Line Tools

First create create AWS credentials via the console, or speak to one of the other devs to have some created for you.

Install awscli

brew install awscli aws-shell

Configure awscli using the generated Access Key

$ aws configure
> AWS Access Key ID [None]: <SECRET>
> AWS Secret Access Key [None]: <SECRET>
> Default region name [None]: us-east-1
> Default output format [None]: json

Some Hadoop/S3 libraries look for the credentials in the AWS_* environment variables instead of ~/.aws/credentials or ~/.aws_credentials, so you may want to define AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Note: these environment variables do not play well with assume-role, so they should not be automatically sourced. Source them at your convenience.

ln -s ~/.aws/credentials ~/.aws_credentials
cat  > ~/.aws/env_config << 'EOF'
export AWS_ACCESS_KEY_ID=<TO_REPLACE>
export AWS_SECRET_ACCESS_KEY=<TO_REPLACE>
EOF

Test the config

aws ec2 describe-instances
aws s3 ls narrative-artifact-releases

Configure FoxyProxy] to access EMR UIs

FoxyProxy lets you use an SSH tunnel as a proxy to browse the Hadoop/Spark EMR admin tools.

  • Browse to the EMR console
  • Click on any cluster
  • Click on Enable Web Connections and follow the instructions to configure FoxyProxy

Install Packer

We use Packer to build custom AWS AMIs.

brew install packer

assume-role

assume-role will request and set temporary credentials in your shell environment variables for a given (AWS IAM) role.

We use it to temporarily acquire a role while performing certain tasks that require additional permissions.

brew install remind101/formulae/assume-role

Backend Application Dev

Each project has its own setup instructions and may require installing additional tools, but this section serves as a general guide to install the main tools needed for backend development.

Java SDK

brew install openjdk

sbt

sbt is the Scala Build Tool

Install

brew install sbt

Scala Console

The Scala Console REPL is included in the scala toolchain.

brew install scala

Coursier

Coursier provides numerous benefits over sbt's default ivy resolvers

  • downloading artifacts in parallel
  • better offline mode
  • non obfuscated cache
  • no global lock

It is automatically used by the SBT builds. No need to manually configure it.

IntelliJ CE

brew cask install intellij-idea-ce

On first launch:

  • Choose Mac OS X 10.5+ keymap
  • Install Scala plugin
  • Use the newly installed JDK while importing projects
  • Navigate to Preferences / Editor / Code Style / Scala
  • Import the Narrative I/O Scala Code Style Scheme.

PostgreSQL Client

We install both the official PostgreSQL client to get access to tools like pg_dump as well as pgcli for a more feature-rich CLI tool.

brew install postgresql pgcli

Redis Client

The Redis command-line client.

brew install redis

Thrift

We use Thrift + Parquet as a serialization mechanism for some of the backend jobs.

brew install thrift

Apache Spark

Apache Spark is a unified analytics engine for large-scale data processing.

brew install apache-spark

Frontend Application Dev

Each project has its own setup instructions and may require installing additional tools, but this section serves as a general guide to install the main tools needed for backend development.

PostgreSQL Client

We install both the official PostgreSQL client to get access to tools like pg_dump as well as pgcli for a more feature-rich CLI tool.

brew install postgresql pgcli

Install jq

Used by the legacy frontend build

brew install jq

Vue.js Devtools

Vue.js devtools is a great Chrome extension for debugging Vue Apps.

Augury

Augury is a great Chrome extension for debugging Angular apps, which we use for the legacy frontend

Edit this page on GitHub Updated at Mon, Mar 1, 2021