Easy install of a LMDE repository mirror

Archived topics about LMDE 1 and LMDE 2
Locked
ketoth

Easy install of a LMDE repository mirror

Post by ketoth »

The goal of this tutorial is to prepare and maintain a repository mirror with quick and easy steps without requiring advanced Linux sysadmin skills. Once done, the mirror URL can be send to the Mint team for addition on the mirrors page, ready to be used by Linux Mint Debian users around the world. Please note that in the moment of writing, the current Update Pack is number 5.

Choose server and system

What you need
  • a dedicated server (rented or purchased) with the following:
  • 100 Mb/s internet connection, or faster, unmetered traffic or cheap/not so much slowed down traffic above monthly 5 TB
  • 750 GB or more disk space
  • 512 MB RAM or more
  • 700 MHz CPU or faster
  • basic service, 99.99% uptime guarantee..
  • Ubuntu Server 12.10 "Quantal Quetzal" with OpenSSH access
As you can see the hardware requirements are quite low, the only difficult one is the hard disk: often dedicated servers come with 120 or 360 GB hard disk, while LMDE mirroring requires at very least 500 GB free space, and a recommended 600 GB. I recommend you 750 GB, so that you won't lack of free space. The cheapest I found in France is a server from OVH's "Kimsufi" class, the "KS 2G" edition: 17.93 € with VAT. From other hosts you start to get 750+ GB at around 100 $ monthly, together with pointless-powerful CPU and RAM: if you can, ask to customize the server config (CPU and RAM doesn't really matter, but big hard disk is mandatory).

Server OS installation
The installing of Ubuntu Server really depends on your hosting provider. Most of the time for rented servers, the hosting provider has a management interface that lets you easily select the system and disk partitioning. If you're housing your own server, you may have to call the datacenter team to be your "eyes and hands" to install Ubuntu Server from ISO. Just go for a basic install, only make sure you get OpenSSH access (or you won't be able to administrate your server, obviously).

Disk partitioning: don't waste any free space, I say. Put a 2 GB swap partition, use the rest for / (root partition).
Image

Once done, you should have a server running Ubuntu Server 12.10, waiting for you to work on it through OpenSSH (root account with password authentication).
In the next post, we gonna do basic configuration (usual user account, access rights, basic security..)
Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 2 times in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
ketoth

Re: Easy install of a LMDE repository mirror (write in progr

Post by ketoth »

Basic setup

Time to log in !

Code: Select all

ssh root@IP
(replace IP by your server's IP or FQDN)

Confirm the fingerprint authenticity, type password, and here you are: shell as root.
First things first:

Using a separate account

If you know a bit about GNU/Linux, you know that root is the "full access" account, the one that can do everything on the system. With so much power, you can guess that many bots and other intrusion attempts are trying to log as root. So we gonna create a "normal" user account, give it sudo power, and then deactivate root login from OpenSSH.

Code: Select all

adduser hermes
(replace "hermes" by the user name of your choice, try to avoid "server", "admin" and other common account names)
When asked for the password, choose a good password: at least 6 characters long, mixing upper and lower case, with numbers and special characters. "I am the Master of the Universe!" is a good one, your birth date or pet's name isn't 8) Of course you have to remember the password, don't write it anywhere. Leave the rest blank (full name, room number,..).

Code: Select all

adduser hermes sudo
This adds your user account to the "sudo" group, which will allow it to execute administrative commands.

Code: Select all

exit
Log out from server.

Code: Select all

ssh hermes@IP
Login as the created user account.

Code: Select all

sudo ls /
Listing of the root folder, this is just to make sure the account can use "sudo" commands.

Deactivating remote root login

Once you're sure you can run "sudo" commands, it's time to prevent root login on OpenSSH. To do this we will use VIM: a console text editor.

Code: Select all

sudo vim /etc/ssh/sshd_config
To activate the "insertion" mode, press the "i" key. To exit the mode, press the "Esc" key. To write to disk (save), enter ":w" and press Enter. To exit VIM, enter ":q" and press Enter. Get it ? :)
Now look for this line with the arrow keys:

Code: Select all

PermitRootLogin yes
Activate insertion mode, change "yes" into "no", exit mode, save, exit VIM. On the keyboard: "i del del del 'no' Esc :w Enter :q Enter". Take your time :)

Upgrading the system

To make sure your server is running the newest software, upgrade it.

Code: Select all

sudo apt-get update && sudo apt-get dist-upgrade -y
Reinforcing server security

Ubuntu provides quite new and safe software, but the server has to stay up-to-date and should be capable to deny flooding login attempts. Let's prepare it for the next step too (PPA).

Code: Select all

sudo apt-get install unattended-upgrades fail2ban software-properties-common
Now, let's configure the unattended-upgrades.

Code: Select all

sudo dpkg-reconfigure -plow unattended-upgrades

Select "yes" for updating stable packages.

Code: Select all

sudo vim /etc/apt/apt.conf.d/50unattended-upgrades
Look for this line:

Code: Select all

//      "${distro_id}:${distro_codename}-updates";
Change it like this (just remove the // ):

Code: Select all

      "${distro_id}:${distro_codename}-updates";
You can set autoreboot to true if you want to, but I don't recommend it since it makes the server reboot while it can be very much needed. Most of the time reboot is required for kernel updates, but this happens only once in 2-4 months anyway.

By default fail2ban denies connections for 10 minutes after 6 wrong login attempts, together with a non-standard username and good password, getting into your server would take.. at least a few centuries, I guess :mrgreen: If you want to set it even stronger, edit "/etc/fail2ban/jail.conf".

So you have installed "software-properties-common": this brings in "python-software-properties", and these 2 are needed to run the next command:

Code: Select all

sudo add-apt-repository ppa:cherokee-webserver/ppa
Confirm the repository addition. Let's finish this step with this:

Code: Select all

sudo apt-get update
and a little reboot:

Code: Select all

sudo reboot
In the next post, we gonna install and configure the web server: Cherokee !
ketoth

Re: Easy install of a LMDE repository mirror (write in progr

Post by ketoth »

Cherokee web server with RRD graphs
Why Cherokee instead of Apache, Lighttpd, Nginx or any other HTTP server ? Well I guess because it's easier to configure since it shows you all what can be done in a nice web interface :wink: It was in the "precise" repository but ain't in the "quantal" ones, therefor the PPA.

Code: Select all

sudo apt-get install cherokee libcherokee-mod-rrd rrdtool php5-cgi
One thing: instead of "www-data", we gonna use the normal username. So for graph generation, we have to pass the ownership to the normal username, or Cherokee won't be able to generate a damn graph. Like this.

Code: Select all

sudo chown -R hermes:hermes /var/lib/cherokee/graphs
Now, let's create a "webroot" directory named "apt-mirror".

Code: Select all

mkdir -p ~/"apt-mirror"
In this folder you will put all public files to be accessed. Since you're the owner of the folder, you can modify them at will using a SFTP client (FileZilla works great for this).

Ok, time to setup Cherokee.

Code: Select all

sudo cherokee-admin -b
It will give you a "one-time password", select it and copy it. Now open your web browser and call your server. In my case the URL is "h++p://pukab.rtsinfo.fr:9090/", of course you have to change it according to your server's hostname or IP.
Image
Isn't this nice ? :)

Go to "General" tab, "network" subtab. For "Server tokens", select "Product + minor version". For "Graphs type", select "RRDTool Graphs".
Now go to the "Permissions" subtab. Here for user and group, enter your normal username and group (in my case, hermes and hermes).
Click on the "Save" button, click on "hard restart" (since nobody else should be connected to the server, it doesn't matter).

Now, go to the "vServers" tab. The "default" vServer refers to the default behavior of the server, typically when entering the IP in the URL. By default the server should provide NOTHING, so we will make these changes:
[Basics] Document root: enter "/dev/null"
[Behavior] click on "rule management". For every directory on the left, click on the cross icon to remove the behavior rule. For the "Default" behavior rule, go to the "Handler" tab and select "Drop connection".
Click on save.

Go back to the "vServers" tab. Time to add the good virtual server :)
Click on the "+" button on the left (add new virtual server). Choose Languages > PHP >> Add >> Next. Now enter the document root, in my case: /home/hermes/apt-mirror/
Next screen, New host name: enter your server's FQDN (in my case: pukab.rtsinfo.fr) and click on create. Done !
Image

The default options are ok, but you can experiment behavior rules, caching options, gzip support, logging, and so on.
When done, click on save and close your browser. Back in the Terminal, press Ctrl+C to close cherokee-admin. Run a well deserved reboot:

Code: Select all

sudo reboot
Next post: repository synchronization !
ketoth

Re: Easy install of a LMDE repository mirror (write in progr

Post by ketoth »

Feeding the mirror

First of all, let's check if everything's fine with Cherokee.

Code: Select all

echo "Hello World" >> ~/apt-mirror/index.html
When you open up your browser and enter your server's domain name, you will see a simple "Hello World" page. Good.

First retrieval

For the first time, your server has to retrieve not less than 460 GB of data from Mint's main server. This will take days. We need to do it smart, using crontab and preventing multiple works. Let's script this with verbosity.

Code: Select all

#!/bin/bash
cd "/home/hermes/apt-mirror/"
if [ -f "cron_deb.lock" ]
then
	echo "* cron job already running, exit *"
	exit
fi
echo "" > "cron_deb.lock"
echo ""
echo "*********************"
echo "* Linux Mint Debian *"
echo "*********************"
	echo "syncing..."
	remote="debian.linuxmint.com::debian"
	local="/home/hermes/apt-mirror/linuxmint/debian/"
	echo "`date +%x-%T` - Doing: rsync -aP --delete-after --no-inc-recursive $remote $local"
	rsync -aP --delete-after --no-inc-recursive $remote $local
	echo "- `date +%x-%T` finished -"
rm "cron_deb.lock"
echo ""
echo "`date +%x-%T` - ALL DONE"
SELF=$(cd $(dirname $0); pwd -P)/$(basename $0) 
touch $SELF
exit 0
My script, explained: first it checks for "cron_deb.lock": if this file is found, the script exits immediately. If it isn't the script continues. This prevents the script to be run multiple times simultaneously by cron: more than one rsync on the same files that's not good at all, believe me. This lockfile makes sure only one script process downloads the archive.

"remote" is the remote server's address, here the Mint main server. If you're in Europe, you may want to change this to the faster "debian.lth.se::lmde". "local" tells the output directory.
rsync -aP --delete-after --no-inc-recursive: archive mode with progress bar, old file deletion at the end, file count before download.
Once done, the script removes the lockfile, and "touches" itself: you can then easily see with it's modtime the last time the script synced successfully.

How to set the script on your server:

Code: Select all

vim ~/sync
Press "i" to enter Insertion mode, copy-paste the full script, save and exit VIM. Ah, one more thing: make it executable.

Code: Select all

chmod +x sync
Prepare the folder structure:

Code: Select all

mkdir -p apt-mirror/linuxmint/debian
.. and let the script stuff it. Different ways to do it.

I want to run it now and manually !

Code: Select all

./sync
.. but don't want to stay on my PC for days

Code: Select all

screen -dmS mirror ./sync
To reattach the screen:

Code: Select all

screen -r mirror
To detach: Press Ctrl + A + D

No, I prefer let cron handle it

Code: Select all

crontab -e
(wanna select vim-basic as editor ? Do it :) )

The Mint team recommends a daily sync for the mirror. So, to run the script daily, it's pretty simple; add this line:

Code: Select all

@daily /home/hermes/sync
Another example, every 3 hours:

Code: Select all

* */3 * * * /home/hermes/sync
Refer to the crontab documentation to set more precise conditions, for example week days, working hours, minute-precise scheduling...


That's it, your mirror is now syncing, and you can check with your web browser:
http://{your domain name}/linuxmint/debian/latest/update-pack-info.txt

Good ? Good ! Tell the Mint team a new mirror is on duty :) (mail to root [at] linuxmint.com)

Customize the webroot "/" and the "/linuxmint" folder as much as you want: by default Cherokee outputs a directory listing, but feel free to start writing HTML or PHP. Pure aesthetics, it doesn't affect the way APT uses your mirror. With the above setup, Cherokee does generate usage graphs but doesn't write access or error logs. My advice: only activate logging for short periods when it's really necessary, you end up quite quickly with gigabyte-heavy log files because a single update pack upgrade involves more than a thousand downloads.


Got questions, need help ? Feel free to post :wink:
-- end of tutorial --
Locked

Return to “LMDE Archive”