Easy install of a LMDE repository mirror

Write tutorials here
More tutorials here http://community.linuxmint.com/tutorial/welcome
Forum rules
Do not start a support topic here please, Before you post please read this

Easy install of a LMDE repository mirror

Postby ketoth on Thu Nov 08, 2012 8:05 am

The goal of this tutorial is to prepare and maintain a repository mirror with quick and easy steps without requiring advanced Linux sysadmin skills. Once done, the mirror URL can be send to the Mint team for addition on the mirrors page, ready to be used by Linux Mint Debian users around the world. Please note that in the moment of writing, the current Update Pack is number 5.

Choose server and system

What you need
  • a dedicated server (rented or purchased) with the following:
  • 100 Mb/s internet connection, or faster, unmetered traffic or cheap/not so much slowed down traffic above monthly 5 TB
  • 750 GB or more disk space
  • 512 MB RAM or more
  • 700 MHz CPU or faster
  • basic service, 99.99% uptime guarantee..
  • Ubuntu Server 12.10 "Quantal Quetzal" with OpenSSH access

As you can see the hardware requirements are quite low, the only difficult one is the hard disk: often dedicated servers come with 120 or 360 GB hard disk, while LMDE mirroring requires at very least 500 GB free space, and a recommended 600 GB. I recommend you 750 GB, so that you won't lack of free space. The cheapest I found in France is a server from OVH's "Kimsufi" class, the "KS 2G" edition: 17.93 € with VAT. From other hosts you start to get 750+ GB at around 100 $ monthly, together with pointless-powerful CPU and RAM: if you can, ask to customize the server config (CPU and RAM doesn't really matter, but big hard disk is mandatory).

Server OS installation
The installing of Ubuntu Server really depends on your hosting provider. Most of the time for rented servers, the hosting provider has a management interface that lets you easily select the system and disk partitioning. If you're housing your own server, you may have to call the datacenter team to be your "eyes and hands" to install Ubuntu Server from ISO. Just go for a basic install, only make sure you get OpenSSH access (or you won't be able to administrate your server, obviously).

Disk partitioning: don't waste any free space, I say. Put a 2 GB swap partition, use the rest for / (root partition).
Image

Once done, you should have a server running Ubuntu Server 12.10, waiting for you to work on it through OpenSSH (root account with password authentication).
In the next post, we gonna do basic configuration (usual user account, access rights, basic security..)
Last edited by ketoth on Thu Nov 08, 2012 11:49 am, edited 1 time in total.
Proud Linux Mint Debian user since 04/2011
mirror.rts-informatique.fr mirror tech maintainer

Likes GPL and Creative Commons and people using them
Somewhere between Linus Torvalds and Richard Stallman
ketoth
Level 2
Level 2
 
Posts: 81
Joined: Sat Dec 31, 2011 8:01 pm

Linux Mint is funded by ads and donations.
 

Re: Easy install of a LMDE repository mirror (write in progr

Postby ketoth on Thu Nov 08, 2012 10:09 am

Basic setup

Time to log in !
Code: Select all
ssh root@IP

(replace IP by your server's IP or FQDN)

Confirm the fingerprint authenticity, type password, and here you are: shell as root.
First things first:

Using a separate account

If you know a bit about GNU/Linux, you know that root is the "full access" account, the one that can do everything on the system. With so much power, you can guess that many bots and other intrusion attempts are trying to log as root. So we gonna create a "normal" user account, give it sudo power, and then deactivate root login from OpenSSH.

Code: Select all
adduser hermes

(replace "hermes" by the user name of your choice, try to avoid "server", "admin" and other common account names)
When asked for the password, choose a good password: at least 6 characters long, mixing upper and lower case, with numbers and special characters. "I am the Master of the Universe!" is a good one, your birth date or pet's name isn't 8) Of course you have to remember the password, don't write it anywhere. Leave the rest blank (full name, room number,..).

Code: Select all
adduser hermes sudo

This adds your user account to the "sudo" group, which will allow it to execute administrative commands.

Code: Select all
exit

Log out from server.

Code: Select all
ssh hermes@IP

Login as the created user account.

Code: Select all
sudo ls /

Listing of the root folder, this is just to make sure the account can use "sudo" commands.

Deactivating remote root login

Once you're sure you can run "sudo" commands, it's time to prevent root login on OpenSSH. To do this we will use VIM: a console text editor.

Code: Select all
sudo vim /etc/ssh/sshd_config


To activate the "insertion" mode, press the "i" key. To exit the mode, press the "Esc" key. To write to disk (save), enter ":w" and press Enter. To exit VIM, enter ":q" and press Enter. Get it ? :)
Now look for this line with the arrow keys:
Code: Select all
PermitRootLogin yes

Activate insertion mode, change "yes" into "no", exit mode, save, exit VIM. On the keyboard: "i del del del 'no' Esc :w Enter :q Enter". Take your time :)

Upgrading the system

To make sure your server is running the newest software, upgrade it.
Code: Select all
sudo apt-get update && sudo apt-get dist-upgrade -y


Reinforcing server security

Ubuntu provides quite new and safe software, but the server has to stay up-to-date and should be capable to deny flooding login attempts. Let's prepare it for the next step too (PPA).
Code: Select all
sudo apt-get install unattended-upgrades fail2ban software-properties-common

Now, let's configure the unattended-upgrades.
Code: Select all
sudo dpkg-reconfigure -plow unattended-upgrades

Select "yes" for updating stable packages.
Code: Select all
sudo vim /etc/apt/apt.conf.d/50unattended-upgrades

Look for this line:
Code: Select all
//      "${distro_id}:${distro_codename}-updates";

Change it like this (just remove the // ):
Code: Select all
      "${distro_id}:${distro_codename}-updates";

You can set autoreboot to true if you want to, but I don't recommend it since it makes the server reboot while it can be very much needed. Most of the time reboot is required for kernel updates, but this happens only once in 2-4 months anyway.

By default fail2ban denies connections for 10 minutes after 6 wrong login attempts, together with a non-standard username and good password, getting into your server would take.. at least a few centuries, I guess :mrgreen: If you want to set it even stronger, edit "/etc/fail2ban/jail.conf".

So you have installed "software-properties-common": this brings in "python-software-properties", and these 2 are needed to run the next command:

Code: Select all
sudo add-apt-repository ppa:cherokee-webserver/ppa

Confirm the repository addition. Let's finish this step with this:
Code: Select all
sudo apt-get update

and a little reboot:
Code: Select all
sudo reboot


In the next post, we gonna install and configure the web server: Cherokee !
Proud Linux Mint Debian user since 04/2011
mirror.rts-informatique.fr mirror tech maintainer

Likes GPL and Creative Commons and people using them
Somewhere between Linus Torvalds and Richard Stallman
ketoth
Level 2
Level 2
 
Posts: 81
Joined: Sat Dec 31, 2011 8:01 pm

Re: Easy install of a LMDE repository mirror (write in progr

Postby ketoth on Thu Nov 08, 2012 10:39 am

Cherokee web server with RRD graphs
Why Cherokee instead of Apache, Lighttpd, Nginx or any other HTTP server ? Well I guess because it's easier to configure since it shows you all what can be done in a nice web interface :wink: It was in the "precise" repository but ain't in the "quantal" ones, therefor the PPA.

Code: Select all
sudo apt-get install cherokee libcherokee-mod-rrd rrdtool php5-cgi


One thing: instead of "www-data", we gonna use the normal username. So for graph generation, we have to pass the ownership to the normal username, or Cherokee won't be able to generate a damn graph. Like this.
Code: Select all
sudo chown -R hermes:hermes /var/lib/cherokee/graphs


Now, let's create a "webroot" directory named "apt-mirror".
Code: Select all
mkdir -p ~/"apt-mirror"

In this folder you will put all public files to be accessed. Since you're the owner of the folder, you can modify them at will using a SFTP client (FileZilla works great for this).

Ok, time to setup Cherokee.
Code: Select all
sudo cherokee-admin -b

It will give you a "one-time password", select it and copy it. Now open your web browser and call your server. In my case the URL is "h++p://pukab.rtsinfo.fr:9090/", of course you have to change it according to your server's hostname or IP.
Image
Isn't this nice ? :)

Go to "General" tab, "network" subtab. For "Server tokens", select "Product + minor version". For "Graphs type", select "RRDTool Graphs".
Now go to the "Permissions" subtab. Here for user and group, enter your normal username and group (in my case, hermes and hermes).
Click on the "Save" button, click on "hard restart" (since nobody else should be connected to the server, it doesn't matter).

Now, go to the "vServers" tab. The "default" vServer refers to the default behavior of the server, typically when entering the IP in the URL. By default the server should provide NOTHING, so we will make these changes:
[Basics] Document root: enter "/dev/null"
[Behavior] click on "rule management". For every directory on the left, click on the cross icon to remove the behavior rule. For the "Default" behavior rule, go to the "Handler" tab and select "Drop connection".
Click on save.

Go back to the "vServers" tab. Time to add the good virtual server :)
Click on the "+" button on the left (add new virtual server). Choose Languages > PHP >> Add >> Next. Now enter the document root, in my case: /home/hermes/apt-mirror/
Next screen, New host name: enter your server's FQDN (in my case: pukab.rtsinfo.fr) and click on create. Done !
Image

The default options are ok, but you can experiment behavior rules, caching options, gzip support, logging, and so on.
When done, click on save and close your browser. Back in the Terminal, press Ctrl+C to close cherokee-admin. Run a well deserved reboot:
Code: Select all
sudo reboot


Next post: repository synchronization !
Proud Linux Mint Debian user since 04/2011
mirror.rts-informatique.fr mirror tech maintainer

Likes GPL and Creative Commons and people using them
Somewhere between Linus Torvalds and Richard Stallman
ketoth
Level 2
Level 2
 
Posts: 81
Joined: Sat Dec 31, 2011 8:01 pm

Re: Easy install of a LMDE repository mirror (write in progr

Postby ketoth on Thu Nov 08, 2012 11:46 am

Feeding the mirror

First of all, let's check if everything's fine with Cherokee.
Code: Select all
echo "Hello World" >> ~/apt-mirror/index.html

When you open up your browser and enter your server's domain name, you will see a simple "Hello World" page. Good.

First retrieval

For the first time, your server has to retrieve not less than 460 GB of data from Mint's main server. This will take days. We need to do it smart, using crontab and preventing multiple works. Let's script this with verbosity.
Code: Select all
#!/bin/bash
cd "/home/hermes/apt-mirror/"
if [ -f "cron_deb.lock" ]
then
   echo "* cron job already running, exit *"
   exit
fi
echo "" > "cron_deb.lock"
echo ""
echo "*********************"
echo "* Linux Mint Debian *"
echo "*********************"
   echo "syncing..."
   remote="debian.linuxmint.com::debian"
   local="/home/hermes/apt-mirror/linuxmint/debian/"
   echo "`date +%x-%T` - Doing: rsync -aP --delete-after --no-inc-recursive $remote $local"
   rsync -aP --delete-after --no-inc-recursive $remote $local
   echo "- `date +%x-%T` finished -"
rm "cron_deb.lock"
echo ""
echo "`date +%x-%T` - ALL DONE"
SELF=$(cd $(dirname $0); pwd -P)/$(basename $0)
touch $SELF
exit 0


My script, explained: first it checks for "cron_deb.lock": if this file is found, the script exits immediately. If it isn't the script continues. This prevents the script to be run multiple times simultaneously by cron: more than one rsync on the same files that's not good at all, believe me. This lockfile makes sure only one script process downloads the archive.

"remote" is the remote server's address, here the Mint main server. If you're in Europe, you may want to change this to the faster "debian.lth.se::lmde". "local" tells the output directory.
rsync -aP --delete-after --no-inc-recursive: archive mode with progress bar, old file deletion at the end, file count before download.
Once done, the script removes the lockfile, and "touches" itself: you can then easily see with it's modtime the last time the script synced successfully.

How to set the script on your server:
Code: Select all
vim ~/sync

Press "i" to enter Insertion mode, copy-paste the full script, save and exit VIM. Ah, one more thing: make it executable.
Code: Select all
chmod +x sync


Prepare the folder structure:
Code: Select all
mkdir -p apt-mirror/linuxmint/debian


.. and let the script stuff it. Different ways to do it.

I want to run it now and manually !

Code: Select all
./sync


.. but don't want to stay on my PC for days

Code: Select all
screen -dmS mirror ./sync

To reattach the screen:
Code: Select all
screen -r mirror

To detach: Press Ctrl + A + D

No, I prefer let cron handle it
Code: Select all
crontab -e

(wanna select vim-basic as editor ? Do it :) )

The Mint team recommends a daily sync for the mirror. So, to run the script daily, it's pretty simple; add this line:
Code: Select all
@daily /home/hermes/sync

Another example, every 3 hours:
Code: Select all
* */3 * * * /home/hermes/sync

Refer to the crontab documentation to set more precise conditions, for example week days, working hours, minute-precise scheduling...


That's it, your mirror is now syncing, and you can check with your web browser:
http://{your domain name}/linuxmint/debian/latest/update-pack-info.txt

Good ? Good ! Tell the Mint team a new mirror is on duty :) (mail to root [at] linuxmint.com)

Customize the webroot "/" and the "/linuxmint" folder as much as you want: by default Cherokee outputs a directory listing, but feel free to start writing HTML or PHP. Pure aesthetics, it doesn't affect the way APT uses your mirror. With the above setup, Cherokee does generate usage graphs but doesn't write access or error logs. My advice: only activate logging for short periods when it's really necessary, you end up quite quickly with gigabyte-heavy log files because a single update pack upgrade involves more than a thousand downloads.


Got questions, need help ? Feel free to post :wink:
-- end of tutorial --
Proud Linux Mint Debian user since 04/2011
mirror.rts-informatique.fr mirror tech maintainer

Likes GPL and Creative Commons and people using them
Somewhere between Linus Torvalds and Richard Stallman
ketoth
Level 2
Level 2
 
Posts: 81
Joined: Sat Dec 31, 2011 8:01 pm


Return to Tutorials

Who is online

Users browsing this forum: No registered users and 4 guests