Categories
Linux

Benchmarking SSDs with fio

Fio which stands for Flexible I/O Tester is a free and open source disk I/O tool used both for benchmark and stress/hardware verification that i mainly use for benchmarking ceph or specific ssd harware.

When using an SSD make sure it's pre-warmed. This can be done using the dd command:

dd if=/dev/zero of=/dev/xvdb bs=100M &

After this you can start performance measurement with fio. My advice is to run this test for 6 to 8 hours in order to get real data out of it.

fio --filename=/dev/nvmeXnXpX --direct=1 --rw=randwrite --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=128k --iodepth=16 --numjobs=1 --time_based --runtime=86400 --group_reporting –-name=benchtest

This command will run for 24 hours and perform write-only workload of 128k blocks on a single process.

Random Read test

sudo fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=0 --size=512M --numjobs=4 --runtime=240 --group_reporting

This will use 4 processes, run for 2 minutes and only perform read iops.

Random Write test

sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

This will to a read/write test on a 4 GB file.

Categories
Ansible Linux

Ansible through Ubuntu (WSL) on Windows 10

Windows Subsystem for Linux (WSL) allows you to run Linux straight from your Windows Desktop. I use this on a daily basis for running Ansible scripts without having to install VM's. Make sure you installed al latest updates.

Enable WSL feature

Open up a Powershell box as Administrator (search powershell, right click and run as Administrator).

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

This will initiate the installation and once finished ask if you would like to reboot your system. Go ahead and do that. When the reboot is done search for 'bash' and open that, it will first require a few anwsers. Simply fill out all the questions and once that is done you will have Ubuntu up and running.

Install Ansible

Now you are basicly in a Linux environment so you can install Ansible the typical way. Again, in the 'bash' window of course, use these instructions:

sudo apt-get -y install python-pip python-dev libffi-dev libssl-dev
sudo pip install ansible

Should you get any permission errors (i did not have this time, but given the nature of how WSL works that could happen) install pip with the --user flag. This will cause it to install ansible in the users home dir, not globally.

You are done. Using the following command you can check what ansible version is now installed:

ansible --version

If you need the most recent version check out my other post here.

Categories
Ansible Linux

Install latest version ansible on Ubuntu 16.04 / 18.04

Ubuntu doesn't ship with the newest version of ansible out of the box, sadly. You have to manually configure the PPA on your system in order to upgrade to the stable version. Follow these commands to install the PPA:

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible

Hit Enter when asked, and once the process is done update your apt repos:

$ sudo apt update

Now you can either upgrade or simply install ansible:

$ sudo apt install ansible

This should be all, use the following to verify the ansible version:

$ ansible --version
Categories
Linux

Ubuntu 18.04 resize/expand (root) filesystem

Running Ubuntu 18.04 i ran out of disk space on my main partition. I increased disk space in VMware and needed to expand the partitions from within Ubuntu. Start with scanning for changes on your disk

echo 1 > /sys/class/block/sda/device/rescan

Verify that you can see the new (correct) disk space using:

fdisk -l

Create a new partition using "cfdisk", navigate to the free space and hit "new". After that hit Write to make sure the partition table gets written. Close cfdisk and either reboot or rescan to update your partition table. Now it's time to add the disk space. First, find the new partition number:

fdisk -l

In my case sda3 was created to i'm going to create a new volume on it

pvcreate /dev/sda3

now extent my volume group with the newly added volume:

vgextend ubuntu-dev-box /dev/sda3

Extend the volume with all available (new) disk space:

lvextend -l+100%FREE /dev/ubuntu-dev-box/root

Now resize the filesystem:

resize2fs /dev/mapper/ubuntu--dev--box--root
Categories
Linux Networking Security

SSH Tunnel to watch Netflix

I often use a 'hopping server' when connecting to clients, that means i need to login twice each time. To make my life easier i sometimes use an SSH tunnel so i can connect to clients directly.

SSH Tunnel can also be usefull when your office blocks netflix 😉

Local Port Forwarding

This will allow you to access remote servers direcly from your local computer. Let's assume you want to use RDP (3389) to a clients hosts (10.0.1.1) and your hopping server is 'hopping.server'

ssh -L 6000:10.0.1.1:3389 [email protected]

Now you can open Remote Desktop and connect to 'localhost:6000', directing you through the tunnel!

Remote Port Forwarding

This will make your local service/port acccessible from a remote host. Sometimes i use this to keep a 'backdoor' and login remotely (home or whatever).

Let's say you want to make a webapplication (TCP 443) availible at port 6000 on the remote SSH server

ssh -R 6000:localhost:443 [email protected]

Now you should be able to connect to port 6000 on the remote host (bontekoe.technology)

Dynamic Forwarding (Proxy)

This is ideal for people who want to use the internet safely/anonymous or for offices where Netflix is blocked 😉

Use a remote server to tunnel all web traffic (eg. home server), connect through SSH to it using the -D flag

ssh -D 6000 [email protected]

Now open up your browser settings, navigate to the connection properties and enter a Proxy server (manually using SOCKS). Use 127.0.0.1 as host and 6000 as port. The tunnel will remain open as long as you are connected through SSH.

Categories
Linux

mod_pagespeed Module on Ubuntu 18.04

mod_pagespeed is an open-source Apache module created by Google to help Make the Web Faster by rewriting web pages to reduce latency and bandwidth. mod_pagespeed releases are available as precompiled linux packages or as source. (See Release Notes for information about bugs fixed)

Installation

  1. Update system

    apt update -y
    apt upgrade -y

  2. Install Apache

    apt-get install apache2 -y

  3. Enable Apache Startup

    systemctl start apache2
    systemctl enable apache2

  4. Install mod_pagespeed

    wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_amd64.deb
    dpkg -i mod-pagespeed-stable_current_amd64.deb
    systemctl restart apache2

  5. Verify mod_pagespeed is running

    curl -D- localhost | head | grep pagespeed

Web Interface

mod_pagespeed has a very simple web-interface to see statistics. If you do not case, skip this step.

nano /etc/apache2/mods-available/pagespeed.conf

Add these lines to it:

<Location /ps_admin>
    Order allow,deny
    Allow from localhost
    Allow from 127.0.0.1
    Allow from all
    SetHandler ps-admin
</Location>

<Location /ps_global_admin>
    Order allow,deny
    Allow from localhost
    Allow from 127.0.0.1
    Allow from all
    SetHandler ps_global_admin
</Location>

After restarting apache you can go to http://<your url>/ps_admin

Categories
Linux Networking

Test internet speed using speedtest-cli

Speedtest-cli is a great tool to test your internet speed using the Speedtest servers, make sure you have Python installed before installing Speedtest-cli.

Installing and Using Speedtest-CLI

  1. Update APT and install packages

    apt-get update; apt-get install python-pip speedtest-cli

  2. Test your speed!

    speedtest-cli
    Testing download speed........................................
    Download: 913.12 Mbit/s
    Testing upload speed..................................................
    Upload: 524.12 Mbit/s

  3. Share your speed 🙂

    speedtest-cli --share
    This will provide you with an image to share proving the speed.

Categories
Linux Networking

Load Balancing Remote Desktop

Using HAProxy to loadbalance between RDS servers is usefull if you have more then one RDS servers and want users to connect to a single IP.

  1. Install Haproxy

    sudo apt-get update
    sudo apt-get install haproxy

  2. Add the RDP VIP (virtual IP) and RDP hosts

    defaults
    clitimeout 1h
    srvtimeout 1h
    listen VIP1 193.x.x.x:3389
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if RDP_COOKIE
    persist rdp-cookie
    balance rdp-cookie
    option tcpka
    option tcplog
    server win2k19-A 192.168.10.5:3389 weight 10 check inter 2000 rise 2 fall 3
    server win2k19-B 192.168.10.6:3389 weight 10 check inter 2000 rise 2 fall 3
    option redispatch

Now we have HAProxy running on a 193.x.x.x ip address, when you connect to that IP it will direct you to one of the Windows 2019 machines. If one dies, it will remove it and you can reconnect to the last one that is online.

Categories
Linux Networking

Installing phpIPAM

phpipam is an open-source web IP address management application (IPAM). Its goal is to provide light, modern and useful IP address management. It has lots of features and can be integrated with PowerDNS!

  1. Install Apache, PHP 7.2, MariaDB and GIT client

    apt-get install apache2 mariadb-server php7.2 libapache2-mod-php7.2 php7.2-curl php7.2-mysql php7.2-curl php7.2-gd php7.2-intl php-pear php7.2-imap php-memcache php7.2-pspell php7.2-recode php7.2-tidy php7.2-xmlrpc php7.2-mbstring php-gettext php7.2-gmp php7.2-json php7.2-xml git wget -y

  2. Run the secure installation for MariaDB

    mysql_secure_installation
    Enter current password for root (enter for none):
    Set root password? [Y/n]: N
    Remove anonymous users? [Y/n]: Y
    Disallow root login remotely? [Y/n]: Y
    Remove test database and access to it? [Y/n]: N
    Reload privilege tables now? [Y/n]: Y

  3. Create database and user

    MariaDB [(none)]> create database phpipam;
    MariaDB [(none)]> grant all on phpipam.* to [email protected] identified by 'bontekoe123';

    MariaDB [(none)]> FLUSH PRIVILEGES;
    MariaDB [(none)]> EXIT;

  4. Clone it from Github

    cd /var/www/html
    git clone https://github.com/phpipam/phpipam.git /var/www/html/phpipam/
    cd /var/www/html/phpipam
    git checkout 1.3
    git submodule update --init --recursive

  5. Edit the config file

    cp config.dist.php config.php
    nano config.php

    Add your MariaDB database, username and password there.

  6. Import PHP IPADM Database

    mysql -u root -p phpipam < db/SCHEMA.sql

  7. Set the correct permissions

    chown -R www-data:www-data /var/www/html/phpipam
    chmod -R 755 /var/www/html/phpipam

  8. Apache configuration

    Either you use the default apache configuration or you can create a virtual host specificly for this. That i leave up to you.

  9. Finish the installer

    Go to http://x.x.x.x/phpipam and you will see the wizard, asking you to finish the installation. Follow the easy steps. After that you can login with admin/admin and change the default password.

  10. Create your first subnet

    Here you can create your first subnet. If can automaticly help you by discovery, checking hosts status and more.

Categories
Linux

Browser Caching via .htaccess

A browser retrieves many resources from the webserver (css/js, etc). Cache allows websites to store this files in temporary storage to allow faster retrieval next time the file is needed.

mod_expires headers

Using these statements we can inform the browser that it can cache files for a longer period. Make sure mod_headers is enabled in apache.

<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType text/css "access 1 month"
ExpiresByType text/html "access 1 month"
ExpiresByType image/gif "access 1 year"
ExpiresByType image/png "access 1 year"
ExpiresByType image/jpg "access 1 year"
ExpiresByType image/jpeg "access 1 year"
ExpiresByType image/x-icon "access 1 year"
ExpiresByType application/pdf "access 1 month"
ExpiresByType application/javascript "access 1 month"
ExpiresByType text/x-javascript "access 1 month"
ExpiresByType application/x-shockwave-flash "access 1 month"
ExpiresDefault "access 1 month"
</IfModule>

Cache control in mod_headers

<ifModule mod_headers.c>
<filesMatch "\.(ico|jpe?g|png|gif|swf)$">
Header set Cache-Control "public"
</filesMatch>
<filesMatch "\.(css)$">
Header set Cache-Control "public"
</filesMatch>
<filesMatch "\.(js)$">
Header set Cache-Control "private"
</filesMatch>
<filesMatch "\.(x?html?|php)$">
Header set Cache-Control "private, must-revalidate"
</filesMatch>
</ifModule>

Turn off ETags

By removing the ETag header, you disable caches and browsers from being able to validate files, so they are forced to rely on your Cache-Control and Expires header.

<IfModule mod_headers.c>
   Header unset Etag
   Header set Connection keep-alive
</IfModule>
FileETag None

Deflating compression

Compression is implemented by the DEFLATE filter. The following directive will enable compression for documents in the container where it is placed; again, make sure the module is enabled in Apache.

<IfModule mod_deflate.c>
  # Compress HTML, CSS, JavaScript, Text, XML and fonts
  AddOutputFilterByType DEFLATE application/javascript
  AddOutputFilterByType DEFLATE application/rss+xml
  AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
  AddOutputFilterByType DEFLATE application/x-font
  AddOutputFilterByType DEFLATE application/x-font-opentype
  AddOutputFilterByType DEFLATE application/x-font-otf
  AddOutputFilterByType DEFLATE application/x-font-truetype
  AddOutputFilterByType DEFLATE application/x-font-ttf
  AddOutputFilterByType DEFLATE application/x-javascript
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE font/opentype
  AddOutputFilterByType DEFLATE font/otf
  AddOutputFilterByType DEFLATE font/ttf
  AddOutputFilterByType DEFLATE image/svg+xml
  AddOutputFilterByType DEFLATE image/x-icon
  AddOutputFilterByType DEFLATE text/css
  AddOutputFilterByType DEFLATE text/html
  AddOutputFilterByType DEFLATE text/javascript
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/xml
  # Remove browser bugs (only needed for really old browsers)
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
  Header append Vary User-Agent
</IfModule>