Back-end Engineering Articles

I write and talk about backend stuff like Ruby, Ruby On Rails, Databases, Testing, Architecture / Infrastructure / System Design, Cloud, DevOps, Backgroud Jobs, and more...

Twitter:
@daniel_moralesp

2020-01-10

Commands inside Digital Ocean + Rails 5.2

Link to original: https://gist.github.com/danielmoralesp/ddf73ee8e2183b010ba9e290c87f407f

## Change file size on production
Enter to server
sudo vim /etc/nginx/nginx.conf
Inside of http section add client_max_body_size 10M;
Then restart the server with: sudo service nginx restart

http {

        ##
        # Basic Settings
        ##
        client_max_body_size 10M;
        sendfile on;
  

## setting database and connect with app
/brokercurve/shared/config

production:
  adapter: postgresql
  host: 127.0.0.1
  database: project_name
  username: user
  password: pass
  encoding: unicode
  pool: 5

# commands to run database VIA CAPISTRANO
# https://gitlab.com/ydkn/capistrano-rails-console
gem 'capistrano-rails-console', require: false
#Capfile
require 'capistrano/rails/console'
#Run a remote rails console with:
$ cap production rails:console
$ cap production rails:c
You can also start a sandbox session:
$ cap production rails:console sandbox=1
Or run a dbconsole:
$ cap production rails:dbconsole
$ cap production rails:db

### im using capistrano, and this commands doesnt work directly on the server
#ssh [email protected]
#cd brokercurve/current/
#bundle exec rake db:create
#bundle exec rake db:schema:load
#bundle exec rails db:migrate
#bundle exec rails c
#User.all

#### Elastic search: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-ubuntu-16-04
# instal java + elasticsearch
# java as dependency of elasticsearch
sudo apt-get install default-jdk
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

# elasticsearch
sudo apt-get update
wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.1/elasticsearch-2.3.1.deb
sudo dpkg -i elasticsearch-2.3.1.deb
sudo systemctl enable elasticsearch.service

# config elasticsearch
sudo vim /etc/elasticsearch/elasticsearch.yml
cluster.name: mycluster1
node.name: "My First Node"
node.master: true
node.data: true

sudo systemctl start elasticsearch
sudo systemctl restart elasticsearch
sudo journalctl --unit elasticsearch
#testing inside server
curl -X GET 'http://localhost:9200'

# run elasticsearch
# inside proyect in terminal
cap production rails:console
# inside rails console, run the model that yoy wan to search inside
Model.reindex

## Digital Ocean + Cloudflare
# to add domain on digital ocean
https://www.youtube.com/watch?v=QduZyoy_eFA
# to connect via cloudflare
https://www.youtube.com/watch?v=2X_Tp_G7aTs
## resume
A = @ or hostname = @ and points to @IPADDRESS and select the droplet
CNAME = @ or hostname = www - is an alias of brokercurve.com

# whenever gem to run simple task with cronjobs
#Capfile
require 'whenever/capistrano'
# config/deploy.rb
set :whenever_roles, -> { [:app, :db, :web] }
# terminal
cap production deploy
# server
crontab -l
### working https://stackoverflow.com/questions/28495711/setting-up-secret-key-base-on-deploy-with-capistrano-3/30809747
gem 'capistrano-secrets-yml', '~> 1.0.0'
# make sure your local config/secrets.yml is not git tracked. It should be on the disk, but gitignored.
# populate production secrets in local config/secrets.yml:

production:
  secret_key_base: d6ced...

#add to Capfile:
require 'capistrano/secrets_yml'

#create secrets.yml file on the remote server by executing this task:
$ bundle exec cap production setup

# Paperclip error with imagemagick
# in server run
sudo apt-get update
sudo apt-get install imagemagick
sudo service nginx restart

## Deploy sideqik and Redis to background Jobs
sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install tcl8.5

wget http://download.redis.io/releases/redis-stable.tar.gz
tar xzf redis-stable.tar.gz
cd redis-stable
make
make test
sudo make install
cd utils
sudo ./install_server.sh
#output
#Port           : 6379
#Config file    : /etc/redis/6379.conf
#Log file       : /var/log/redis_6379.log
#Data dir       : /var/lib/redis/6379
#Executable     : /usr/local/bin/redis-server
#Cli Executable : /usr/local/bin/redis-cli


## You can start and stop redis with these commands (the number depends on the port you set during the installation. 6379 is the default port setting):

sudo service redis_6379 start
sudo service redis_6379 stop


#You can then access the redis database by typing the following command:
redis-cli
#To set Redis to automatically start at boot, run:
sudo update-rc.d redis_6379 defaults
export LANGUAGE=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_TYPE=en_US.UTF-8

## securing redis
sudo apt-get install ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable (+ yes)
sudo ufw allow http
sudo ufw allow https

# instaling sideqik
/lib/systemd/system
sudo touch sidekiq.service
# inside

#####################
#
# systemd unit file for CentOS 7, Ubuntu 15.04
#
# Customize this file based on your bundler location, app directory, etc.
# Put this in /usr/lib/systemd/system (CentOS) or /lib/systemd/system (Ubuntu).
# Run:
#   - systemctl enable sidekiq
#   - systemctl {start,stop,restart} sidekiq
#
# This file corresponds to a single Sidekiq process.  Add multiple copies
# to run multiple processes (sidekiq-1, sidekiq-2, etc).
#
# See Inspeqtor's Systemd wiki page for more detail about Systemd:
# https://github.com/mperham/inspeqtor/wiki/Systemd
#
[Unit]
Description=sidekiq
# start us only once the network and logging subsystems are available,
# consider adding redis-server.service if Redis is local and systemd-managed.
After=syslog.target network.target

# See these pages for lots of options:
# http://0pointer.de/public/systemd-man/systemd.service.html
# http://0pointer.de/public/systemd-man/systemd.exec.html
[Service]
Type=simple
WorkingDirectory=/home/deploy/myapp/current
# If you use rbenv:
ExecStart=/home/deploy/.rbenv/shims/bundle exec "sidekiq -e production"
User=deploy
Group=deploy
UMask=0002

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=multi-user.target

#####################

#After setting up configuration, enable the Sidekiq service with:
systemctl enable sidekiq

#other useful commands for controlling the service are:
systemctl {start,stop,status,restart} sidekiq
# or
service sidekiq {start,stop,status,restart}

## Restart sidekiq after deploy
# config/deploy

## For sidekiq
set :pty, true
namespace :sidekiq do
  task :quiet do
    on roles(:app) do
      puts capture("pgrep -f 'sidekiq' | xargs kill -TSTP")
    end
  end
  task :restart do
    on roles(:app) do
      execute :sudo, :systemctl, :restart, :sidekiq
    end
  end
end

after 'deploy:starting', 'sidekiq:quiet'
after 'deploy:reverted', 'sidekiq:restart'
after 'deploy:published', 'sidekiq:restart'

## Open the sudoers file on your server with sudo visudo. On Ubuntu, this will open the file with the nano text editor. To allow passwordless sudo for our Sidekiq restart task, add this to the bottom of the file and save /exit with ctrl-x.

%deploy ALL=NOPASSWD:/bin/systemctl restart sidekiq

## Cockpit to monitor server from Browser
ssh [email protected]
sudo apt-get install cockpit
type sudo password
sudo ufw allow 9090 (this is to allow this port. Digital Ocean automatically set it)
https://IPADDRESS:9090/ (allow connection)
username: deploy
password: xxxx
Check as true the button: Reuse my password for privileged tasks

## Adding MX from google gmail
- This setup is following instructions from google
- The change of MXs is from Cloudflare, not from Digital ocean
- Copy and paste the MX inside cloudflare DNS

##  Adding Staging Environment
https://itnext.io/deploy-staging-and-production-applications-to-single-server-using-capistrano-rails-1d5ab558d44f
- Step 1
Go to cloudflare and/or digital ocean to crea an A Server as
Type A
Hostname: staging
Contento: IPADDRESS (Hosting server)
TTL: auto

Then, if I'm having www-redirections
Type: CNAME --> www.staging
Target: bookcademy.com
TTL; auto

- Step 2
$> ssh [email protected]
$> cd /etc/nginx/sites-enabled
$> sudo vim default (defaul is the name of current app)
$> sudo vim staging /this create an empty doc)

Inside staging
server {
  listen 80;
  listen [::]:80;

  server_name datasource.ai staging.datasource.ai;
  root /home/deploy/staging/current/public;

  passenger_enabled on;
  passenger_app_env staging;

  location /cable {
    passenger_app_group_name staging_websocket;
    passenger_force_max_concurrent_requests_per_process 0;
  }

  # Allow uploads up to 100MB in size
  client_max_body_size 100m;

  location ~ ^/(assets|packs) {
    expires max;
    gzip_static on;
  }
}

The default file is now this

server {
        listen 80;
        listen [::]:80 ipv6only=on;

        server_name IPADDRESS;
        passenger_enabled on;

        rails_env    production;
        root         /home/deploy/shelf/current/public;

        # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
        
        location /cable {
                passenger_app_group_name production_websocket;
                passenger_force_max_concurrent_requests_per_process 0;
        }

        
        # Allow uploads up to 100MB in size
        client_max_body_size 100m;

        location ~ ^/(assets|packs) {
                expires max;
                gzip_static on;
        }
}

After updating both files, check nginx configuration and restart succeeds with following commands:

$> sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$> sudo service nginx start

Step 3 - Change deply scripts

## config/deploy.rb
# config valid for current version and patch releases of Capistrano
lock "~> 3.11.0"

set :application, "shelf"
set :repo_url, "[email protected]:danielmoralesp/bookcademy.git"

set :passenger_restart_with_touch, true

append :linked_files, "config/database.yml", "config/master.key", "config/application.yml"
append :linked_dirs, "log", "tmp/pids", "tmp/cache", "tmp/sockets", "vendor/bundle", "public/system", "public/uploads"

load 'lib/capistrano/tasks/seed.rake'

## config/deploy/production.rb
# Replace 127.0.0.1 with your server's IP address!
server 'IPADDRESS', user: 'deploy', roles: %w{app db web}

set :branch, 'master'
set :deploy_to, '/home/deploy/shelf'
set :stage, :production

## config/deploy/staging.rb
server 'IPADDRESS', user: 'deploy', roles: %w{app db web}

set :branch, 'staging'
set :deploy_to, '/home/deploy/staging'
set :stage, :staging


Step 4 - Staging database Configuration

# setup database
sudo su
su postgres
psql
postgres=# create database db_name_staging with owner = deploy;
\q
exit
exit (come back to deploy user)
psql --user deploy --password db_name_staging
password_db
# it should print the name of database
\q

## inside database.yml in server add staging like this

staging:
  adapter: postgresql
  host: 127.0.0.1
  database: db
  username: user
  password: pass
  encoding: unicode
  pool: 5
  
## inside project rails add this to database.yml
staging:
  <<: *default
  database: db/staging.sqlite3

Step 5 - Create other environments and githhub branch
## environment staging is the same as production, so copy them

$> cp config/environments/production.rb config/environments/staging.rb

# first upload actual changes to master
git add .
git commit ...
git push origin master

## then we create the brach staging
git checkout -b staging
git push origin staging

Step 6 - Deploy
## inside deploy branch
$> cap staging deploy

## The first time running, you might get an error saying linked files are missing. As we have database.yml and master.key files shared in shared/config folder. Please create or update those files with the content from your application, if you get that error message

ERROR linked file /home/deploy/staging/shared/config/database.yml does not exist on 167.71.87.122

## copy and paste same as production
touch application.yml
touch database.yml
touch master.key

## paste the same as production and then
$> cap production deploy

## Finally, the code in routes.rb if I wanted to redirect, is this:
constraints(subdomain: '') do
  match '(*any)', via: :all, to: redirect { |params, request|
   URI.parse(request.url).tap { |x| x.host = "www.#{x.host}" }.to_s
  }
end

### INSTALLING RSPEC AND TRAVIS CI
app/.travis.yml

language: ruby
rvm: 2.3.3
script: bundle exec rake spec

install gems
...
group :development, :test do
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
  # Use sqlite3 as the database for Active Record
  gem 'sqlite3'
  ## Annotate models
  gem 'annotate'
  # Test Suite Part 1
  # RSpec
  gem 'rspec-rails', '~> 3.7'
  # Devise Test
  gem 'rails-controller-testing'
  gem 'faker'
end

# Test Suite Part 2
gem 'simplecov', require: false, group: :test
gem 'factory_bot_rails'

group :development do
...

run this commands
bundle install
rails generate rspec:install

## create first folder "controllers" and file: home_controller_spec.rb
require 'rails_helper'

RSpec.describe HomeController, type: :controller do

  describe "GET #index" do
    it "returns http success" do
      get :index
      expect(response).to have_http_status(:success)
    end
  end
end

## run command
rspec spec/controllers/home_controller_spec.rb

## travis badge
- inside travis CI webpage add repo

## Corriendo test
Build status via Travis CI
[![Build Status](https://travis-ci.com/danielmoralesp/datasource.svg?token=MgzPTs7cYofsdNWxWqp7&branch=staging)](https://travis-ci.com/danielmoralesp/datasource)

* Cuando necesite saltarse el build (casi siempre) se ejecuta con el comando

```
git add .
git commit -m "Ticket #ticket_number - Comentario [ci skip]"
git push origin my_custom_branch_name
```