6 Mar 2014

Run your own repository with Apt-Mirror

I have been using Ubuntu 10.04 (Lucid) and 12.04 (Precise) for a while now and as I like the Gnome 2 type desktop (Classic) I would love to continue using them into the future. But there will come a time (Precise ends April 2017, Lucid already stopped last year) when support will seize and quite possibly the repositories of their software packages will fold up as well; meaning that we won't be able to install new applications any longer.

One solution is to create your own repositories of all the software packages they offer. They can then be stored on your own computer or external hard drive. I won't get into the possibility of creating it for Internet or LAN streaming; but just for your own use.

The basic idea is to sync the repositories of your choice with the ones on your own computer. The application apt-mirror does just that. It downloads the package lists. Then compares it with the existing files you already have and eventually downloads the new one. There is also the option to delete previous versions of those new packages.


Creating our own copy of repository with apt-mirror


Apt-Mirror was developed by Russian developer Dmitriy Khramtsov. This application is available from the Ubuntu universe repositories and therefore already part of what is on offer via Synaptic or command: sudo apt-get install apt-mirror
Once installed we can make some changes. The original file /etc/apt/mirror.list looks like this:

--------------
############# config ##################
#
# set base_path /var/spool/apt-mirror
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads 20
set _tilde 0
#
############# end config ##############
deb http://archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
#deb http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu precise main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu precise-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu precise-updates main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu precise-proposed main restricted universe multiverse
#deb-src http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu
 
--------------

It could be left like this. The repositories would be downloaded into /var/spool/apt-mirror as set by line 3: set base_path

Two facts may be against that path.
  1. First: it needs to be handled as a super user.
  2. Secondly you may not have enough space on your basic Ubuntu partition. 
My Precise repositories which also include non-Ubuntu sites is now over 100 GB (I have kept previous versions too).

So we can change that to any location we wish. For example using an external HDD named DATA.

In there we create a folder called say: UbuntuRepos. It's address therefore will be added after set base_path. We could add this to line 4 and uncomment it first (delete the # sign) looking like:

set base_path /media/DATA/UbuntuRepos/ 

If we want to keep previous file versions we will have to comment (what is still) line 29 like this:

# clean http://archive.ubuntu.com/ubuntu

Otherwise we'll leave it uncommented; but two more things! It assumes that we use the given mirror at http://archive.ubuntu.com/ubuntu. This is the main server and could get very busy and it may not be the fastest mirror for your area. For me the best mirror is: http://mirror.optus.net/ubuntu/. That meant changing the addresses after ###### end config ##### from:

deb http://archive.ubuntu.com/ubuntu precise main restricted universe multiverse

to:

deb http://mirror.optus.net/ubuntu/ precise main restricted universe multiverse

To find out what is your best mirror you can use Synaptic > Settings > Repositories > Ubuntu Software > Download from: > Other > Choose a Download Server > Click on: Select Best Server and it will test approx. 300 repositories around the globe and offers the fastest one and that would be the address to use. It's best to actually go to the site and look at the structure to make sure you get the address right. Some may include the name "archive" and others may not. All the Ubuntu lines (-security, -updates, -proposed and -backports) will have to be changed as well.

The other thing is that the clean command (if using it) needs to be there for each repository, e.g.

clean http://mirror.optus.net/ubuntu/
clean http://archive.canonical.com/ubuntu
clean http://mirrors.dotsrc.org/getdeb/ubuntu
and so on.

Default architecture i386 for 32 bit machines or amd64 for 64 bit machines (also non-amd CPUs) gets automatically set to what your computer runs. Although if you get it with a 32 bit machine and then use it on a 64 bit machine it won't match. Therefore line 9 may be uncommented and add your architecture to get: set defaultarch amd-64 and if we want both architectures we'll leave that commented and add the architecture to the address lines, one for 32 bit and a copy of it for 64 bit: 

deb-i386 http://mirror.optus.net/ubuntu/ precise main restricted universe multiverse
deb-amd64 http://mirror.optus.net/ubuntu/ precise main restricted universe multiverse

The rest of the configuration section can be left as is. Below it we can add more repos like: 

deb-amd64 http://archive.canonical.com/ubuntu precise partner
deb-i386 http://archive.canonical.com/ubuntu precise partner
deb-amd64 http://mirrors.dotsrc.org/getdeb/ubuntu precise-getdeb apps games
deb-i386 http://mirrors.dotsrc.org/getdeb/ubuntu precise-getdeb apps games
deb-amd64 http://download.virtualbox.org/virtualbox/debian precise contrib
deb-i386 http://download.virtualbox.org/virtualbox/debian precise contrib
deb-amd64 http://extras.ubuntu.com/ubuntu precise main
deb-i386 http://extras.ubuntu.com/ubuntu precise main
deb-amd64 http://ppa.launchpad.net/ubuntu-wine/ppa/ubuntu precise main
deb-i386 http://ppa.launchpad.net/ubuntu-wine/ppa/ubuntu precise main
deb-amd64 http://ppa.launchpad.net/gezakovacs/ppa/ubuntu precise main
deb-i386 http://dl.google.com/linux/earth/deb/ stable main
deb-amd64 http://dl.google.com/linux/earth/deb/ stable main
deb-i386 http://deb.opera.com/opera/ stable non-free
deb-amd64 http://deb.opera.com/opera/ stable non-free
deb-i386 http://ppa.launchpad.net/alexmurray/indicator-sensors/ubuntu precise main
deb-amd64 http://ppa.launchpad.net/alexmurray/indicator-sensors/ubuntu precise main
deb-i386 http://ppa.launchpad.net/nilarimogard/webupd8/ubuntu precise main
deb-amd64 http://ppa.launchpad.net/nilarimogard/webupd8/ubuntu precise main

To update your local repository is as simple as using the terminal: apt-mirror. I noticed that with my satellite provider it doesn't always download all it says it will. So I usually repeat and do a few passes until it says 0.0 bytes need to be downloaded.

When using the apt-mirror update command and off-line downloads we have to make sure that the partition or external HDD, where it is located, is already mounted. If it is not set to automatically mount we can add the device to the file /etc/fstab.

In my case the added line looks like: 

/dev/disk/by-uuid/46D7-BEE8 /media/DATA vfat rw,noatime,user,umask=0 0 0

It is recommended to use the UUID of a device for mounting. To find out the UUID of your partition or connected external drive in terminal type: sudo blkid. For above drive it would say:

/dev/sdb1: LABEL="DATA" UUID="46D7-BEE8" TYPE="vfat"

Use command: sudo gedit /etc/fstab to tweak this file (Caution: make backup of file first!)

How to use apt-mirror when off-line

Sources.list is where apt (and therefore Synaptic) looks for repository addresses, contacts the sites and downloads lists of available software packages. Then after we make our choices apt-get will download the needed packages and their dependencies and installs them. We can manually add more addresses to this file. It's a marvellous system!

The file /etc/apt/sources.list needs to be changed. First create a back up of the original from terminal:

sudo cp /etc/apt/sources.list /etc/apt/sources.list.online

To open and tweak the file:

sudo gedit /etc/apt/sources.list

Instead of lines including http:// it needs to be changed to file:///

So the original: 

deb http://mirror.optus.net/ubuntu/ precise multiverse universe restricted

gets changed to:

deb file:///media/DATA/UbuntuRepos/mirror/mirror.optus.net/ubuntu/ precise multiverse universe restricted

Now we can make a copy of the file and call it offline. In a terminal type:

sudo cp /etc/apt/sources.list /etc/apt/sources.list.offline

Then later if we want to change from on-line to off-line use we simply type: 

sudo cp /etc/apt/sources.list.offline /etc/apt/sources.list

and when on-line change it back to the respective file: 

sudo cp /etc/apt/sources.list.online /etc/apt/sources.list

Once the sources.list had been changed we have to update apt so it knows. In a terminal type: sudo apt-get update

Now we can start Synaptic and use it for off-line use. "Software Center" may need the internet for some functions so one better not use it for off-line usage.

Notes


It may pay to regularly check all the addresses in mirror.list to see if they still exist. For example last year http://packages.medibuntu.org folded up and slowed everything down because apt-mirror could not connect to this non-existent address!

I can still use it in the sources.list.offline because it exists on my media partition; but I commented (add # in front of line) it in mirror.list so it won't look for it!

In mirror.list I had to change line 10 from:

# set postmirror_script $var_path/postmirror.sh

to: 

set postmirror_script $/var/spool/apt-mirror/var/postmirror.sh

because the file ended up being installed in this place, please check this location first.

Also line 8 then would become:

set cleanscript $/var/spool/apt-mirror/var/clean.sh

P.S.1 If the clean.sh script is on an external HDD, it cannot be executed.

Copy it from the …/var/ folder of your local apt-mirror repository to your Desktop. To do so, in a terminal issue:

cd Desktop
sudo chmod a+x clean.sh
sudo su
(give your password)
./clean.sh


P.S.2 In Ubuntu Trusty the /media folder has the other drives shown behind yet another folder with the user name; in my case they are mounted in: /media/user/, so above example would then be:

deb file:///media/user/DATA/UbuntuRepos/mirror/mirror.optus.net/ubuntu/ precise multiverse universe restricted

Please check your /media/folder to see if this applies!

This is a guest post by Rolf Sommerhalder.

0 comments:

Post a Comment