Back-end Engineering Articles
I write and talk about backend stuff like Ruby, Ruby On Rails, Databases, Testing, Architecture / Infrastructure / System Design, Cloud, DevOps, Backgroud Jobs, and more...
2020-01-10
## Change file size on production Enter to server sudo vim /etc/nginx/nginx.conf Inside of http section add client_max_body_size 10M; Then restart the server with: sudo service nginx restart http { ## # Basic Settings ## client_max_body_size 10M; sendfile on; ## setting database and connect with app /brokercurve/shared/config production: adapter: postgresql host: 127.0.0.1 database: project_name username: user password: pass encoding: unicode pool: 5 # commands to run database VIA CAPISTRANO # https://gitlab.com/ydkn/capistrano-rails-console gem 'capistrano-rails-console', require: false #Capfile require 'capistrano/rails/console' #Run a remote rails console with: $ cap production rails:console $ cap production rails:c You can also start a sandbox session: $ cap production rails:console sandbox=1 Or run a dbconsole: $ cap production rails:dbconsole $ cap production rails:db ### im using capistrano, and this commands doesnt work directly on the server #ssh [email protected] #cd brokercurve/current/ #bundle exec rake db:create #bundle exec rake db:schema:load #bundle exec rails db:migrate #bundle exec rails c #User.all #### Elastic search: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-elasticsearch-on-ubuntu-16-04 # instal java + elasticsearch # java as dependency of elasticsearch sudo apt-get install default-jdk sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer # elasticsearch sudo apt-get update wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.1/elasticsearch-2.3.1.deb sudo dpkg -i elasticsearch-2.3.1.deb sudo systemctl enable elasticsearch.service # config elasticsearch sudo vim /etc/elasticsearch/elasticsearch.yml cluster.name: mycluster1 node.name: "My First Node" node.master: true node.data: true sudo systemctl start elasticsearch sudo systemctl restart elasticsearch sudo journalctl --unit elasticsearch #testing inside server curl -X GET 'http://localhost:9200' # run elasticsearch # inside proyect in terminal cap production rails:console # inside rails console, run the model that yoy wan to search inside Model.reindex ## Digital Ocean + Cloudflare # to add domain on digital ocean https://www.youtube.com/watch?v=QduZyoy_eFA # to connect via cloudflare https://www.youtube.com/watch?v=2X_Tp_G7aTs ## resume A = @ or hostname = @ and points to @IPADDRESS and select the droplet CNAME = @ or hostname = www - is an alias of brokercurve.com # whenever gem to run simple task with cronjobs #Capfile require 'whenever/capistrano' # config/deploy.rb set :whenever_roles, -> { [:app, :db, :web] } # terminal cap production deploy # server crontab -l ### working https://stackoverflow.com/questions/28495711/setting-up-secret-key-base-on-deploy-with-capistrano-3/30809747 gem 'capistrano-secrets-yml', '~> 1.0.0' # make sure your local config/secrets.yml is not git tracked. It should be on the disk, but gitignored. # populate production secrets in local config/secrets.yml: production: secret_key_base: d6ced... #add to Capfile: require 'capistrano/secrets_yml' #create secrets.yml file on the remote server by executing this task: $ bundle exec cap production setup # Paperclip error with imagemagick # in server run sudo apt-get update sudo apt-get install imagemagick sudo service nginx restart ## Deploy sideqik and Redis to background Jobs sudo apt-get update sudo apt-get install build-essential sudo apt-get install tcl8.5 wget http://download.redis.io/releases/redis-stable.tar.gz tar xzf redis-stable.tar.gz cd redis-stable make make test sudo make install cd utils sudo ./install_server.sh #output #Port : 6379 #Config file : /etc/redis/6379.conf #Log file : /var/log/redis_6379.log #Data dir : /var/lib/redis/6379 #Executable : /usr/local/bin/redis-server #Cli Executable : /usr/local/bin/redis-cli ## You can start and stop redis with these commands (the number depends on the port you set during the installation. 6379 is the default port setting): sudo service redis_6379 start sudo service redis_6379 stop #You can then access the redis database by typing the following command: redis-cli #To set Redis to automatically start at boot, run: sudo update-rc.d redis_6379 defaults export LANGUAGE=en_US.UTF-8 export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 export LC_TYPE=en_US.UTF-8 ## securing redis sudo apt-get install ufw sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow ssh sudo ufw enable (+ yes) sudo ufw allow http sudo ufw allow https # instaling sideqik /lib/systemd/system sudo touch sidekiq.service # inside ##################### # # systemd unit file for CentOS 7, Ubuntu 15.04 # # Customize this file based on your bundler location, app directory, etc. # Put this in /usr/lib/systemd/system (CentOS) or /lib/systemd/system (Ubuntu). # Run: # - systemctl enable sidekiq # - systemctl {start,stop,restart} sidekiq # # This file corresponds to a single Sidekiq process. Add multiple copies # to run multiple processes (sidekiq-1, sidekiq-2, etc). # # See Inspeqtor's Systemd wiki page for more detail about Systemd: # https://github.com/mperham/inspeqtor/wiki/Systemd # [Unit] Description=sidekiq # start us only once the network and logging subsystems are available, # consider adding redis-server.service if Redis is local and systemd-managed. After=syslog.target network.target # See these pages for lots of options: # http://0pointer.de/public/systemd-man/systemd.service.html # http://0pointer.de/public/systemd-man/systemd.exec.html [Service] Type=simple WorkingDirectory=/home/deploy/myapp/current # If you use rbenv: ExecStart=/home/deploy/.rbenv/shims/bundle exec "sidekiq -e production" User=deploy Group=deploy UMask=0002 # if we crash, restart RestartSec=1 Restart=on-failure # output goes to /var/log/syslog StandardOutput=syslog StandardError=syslog # This will default to "bundler" if we don't specify it SyslogIdentifier=sidekiq [Install] WantedBy=multi-user.target ##################### #After setting up configuration, enable the Sidekiq service with: systemctl enable sidekiq #other useful commands for controlling the service are: systemctl {start,stop,status,restart} sidekiq # or service sidekiq {start,stop,status,restart} ## Restart sidekiq after deploy # config/deploy ## For sidekiq set :pty, true namespace :sidekiq do task :quiet do on roles(:app) do puts capture("pgrep -f 'sidekiq' | xargs kill -TSTP") end end task :restart do on roles(:app) do execute :sudo, :systemctl, :restart, :sidekiq end end end after 'deploy:starting', 'sidekiq:quiet' after 'deploy:reverted', 'sidekiq:restart' after 'deploy:published', 'sidekiq:restart' ## Open the sudoers file on your server with sudo visudo. On Ubuntu, this will open the file with the nano text editor. To allow passwordless sudo for our Sidekiq restart task, add this to the bottom of the file and save /exit with ctrl-x. %deploy ALL=NOPASSWD:/bin/systemctl restart sidekiq ## Cockpit to monitor server from Browser ssh [email protected] sudo apt-get install cockpit type sudo password sudo ufw allow 9090 (this is to allow this port. Digital Ocean automatically set it) https://IPADDRESS:9090/ (allow connection) username: deploy password: xxxx Check as true the button: Reuse my password for privileged tasks ## Adding MX from google gmail - This setup is following instructions from google - The change of MXs is from Cloudflare, not from Digital ocean - Copy and paste the MX inside cloudflare DNS ## Adding Staging Environment https://itnext.io/deploy-staging-and-production-applications-to-single-server-using-capistrano-rails-1d5ab558d44f - Step 1 Go to cloudflare and/or digital ocean to crea an A Server as Type A Hostname: staging Contento: IPADDRESS (Hosting server) TTL: auto Then, if I'm having www-redirections Type: CNAME --> www.staging Target: bookcademy.com TTL; auto - Step 2 $> ssh [email protected] $> cd /etc/nginx/sites-enabled $> sudo vim default (defaul is the name of current app) $> sudo vim staging /this create an empty doc) Inside staging server { listen 80; listen [::]:80; server_name datasource.ai staging.datasource.ai; root /home/deploy/staging/current/public; passenger_enabled on; passenger_app_env staging; location /cable { passenger_app_group_name staging_websocket; passenger_force_max_concurrent_requests_per_process 0; } # Allow uploads up to 100MB in size client_max_body_size 100m; location ~ ^/(assets|packs) { expires max; gzip_static on; } } The default file is now this server { listen 80; listen [::]:80 ipv6only=on; server_name IPADDRESS; passenger_enabled on; rails_env production; root /home/deploy/shelf/current/public; # redirect server error pages to the static page /50x.html error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } location /cable { passenger_app_group_name production_websocket; passenger_force_max_concurrent_requests_per_process 0; } # Allow uploads up to 100MB in size client_max_body_size 100m; location ~ ^/(assets|packs) { expires max; gzip_static on; } } After updating both files, check nginx configuration and restart succeeds with following commands: $> sudo nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful $> sudo service nginx start Step 3 - Change deply scripts ## config/deploy.rb # config valid for current version and patch releases of Capistrano lock "~> 3.11.0" set :application, "shelf" set :repo_url, "[email protected]:danielmoralesp/bookcademy.git" set :passenger_restart_with_touch, true append :linked_files, "config/database.yml", "config/master.key", "config/application.yml" append :linked_dirs, "log", "tmp/pids", "tmp/cache", "tmp/sockets", "vendor/bundle", "public/system", "public/uploads" load 'lib/capistrano/tasks/seed.rake' ## config/deploy/production.rb # Replace 127.0.0.1 with your server's IP address! server 'IPADDRESS', user: 'deploy', roles: %w{app db web} set :branch, 'master' set :deploy_to, '/home/deploy/shelf' set :stage, :production ## config/deploy/staging.rb server 'IPADDRESS', user: 'deploy', roles: %w{app db web} set :branch, 'staging' set :deploy_to, '/home/deploy/staging' set :stage, :staging Step 4 - Staging database Configuration # setup database sudo su su postgres psql postgres=# create database db_name_staging with owner = deploy; \q exit exit (come back to deploy user) psql --user deploy --password db_name_staging password_db # it should print the name of database \q ## inside database.yml in server add staging like this staging: adapter: postgresql host: 127.0.0.1 database: db username: user password: pass encoding: unicode pool: 5 ## inside project rails add this to database.yml staging: <<: *default database: db/staging.sqlite3 Step 5 - Create other environments and githhub branch ## environment staging is the same as production, so copy them $> cp config/environments/production.rb config/environments/staging.rb # first upload actual changes to master git add . git commit ... git push origin master ## then we create the brach staging git checkout -b staging git push origin staging Step 6 - Deploy ## inside deploy branch $> cap staging deploy ## The first time running, you might get an error saying linked files are missing. As we have database.yml and master.key files shared in shared/config folder. Please create or update those files with the content from your application, if you get that error message ERROR linked file /home/deploy/staging/shared/config/database.yml does not exist on 167.71.87.122 ## copy and paste same as production touch application.yml touch database.yml touch master.key ## paste the same as production and then $> cap production deploy ## Finally, the code in routes.rb if I wanted to redirect, is this: constraints(subdomain: '') do match '(*any)', via: :all, to: redirect { |params, request| URI.parse(request.url).tap { |x| x.host = "www.#{x.host}" }.to_s } end ### INSTALLING RSPEC AND TRAVIS CI app/.travis.yml language: ruby rvm: 2.3.3 script: bundle exec rake spec install gems ... group :development, :test do # Call 'byebug' anywhere in the code to stop execution and get a debugger console gem 'byebug', platforms: [:mri, :mingw, :x64_mingw] # Use sqlite3 as the database for Active Record gem 'sqlite3' ## Annotate models gem 'annotate' # Test Suite Part 1 # RSpec gem 'rspec-rails', '~> 3.7' # Devise Test gem 'rails-controller-testing' gem 'faker' end # Test Suite Part 2 gem 'simplecov', require: false, group: :test gem 'factory_bot_rails' group :development do ... run this commands bundle install rails generate rspec:install ## create first folder "controllers" and file: home_controller_spec.rb require 'rails_helper' RSpec.describe HomeController, type: :controller do describe "GET #index" do it "returns http success" do get :index expect(response).to have_http_status(:success) end end end ## run command rspec spec/controllers/home_controller_spec.rb ## travis badge - inside travis CI webpage add repo ## Corriendo test Build status via Travis CI [](https://travis-ci.com/danielmoralesp/datasource) * Cuando necesite saltarse el build (casi siempre) se ejecuta con el comando ``` git add . git commit -m "Ticket #ticket_number - Comentario [ci skip]" git push origin my_custom_branch_name ```