Firefox Sync Server on a Raspberry Pi

Author: akeil
Date: 2016-09-14
Version: 1.2
updated: 2016-09-15

The Firefox browser can use a Sync Service to synchronize settings, bookmarks and other stuff across multiple Firefox installations. By default, Mozilla's public sync server is used but it is possible to run your own sync server [1].

We will install piCore Linux and the Firefox Sync Server on a new Raspberry Pi.

Install piCore OS

We will use piCore [2], the Raspberry Pi version of Tiny Core Linux [3]. The benefit of using piCore is that it runs entirely from memory which should reduce strain on the SD card.

piCore is installed by downloading the image and putting it on an SD card.

$ wget http://tinycorelinux.net/8.x/armv6/releases/RPi/piCore-8.0.zip
$ unzip piCore-8.0.zip
$ sudo dd if=piCore-8.0.img of=/dev/sdX bs=1M
$ sudo sync

Where /dev/sdX is the empty SD card. Don't forget to unmount all partitions from the SD card before writing to it.

After that, put the SD card into the Raspberry Pi, plug in power and network and log in with user tc and password piCore.

Extend Partition

TinyCore Linux knows two modes - "Cloud Mode" and "Mounted Mode". Mounted Mode allows to have some persistence, e.g. for installed applications. For this, a second partition is used. The second partition is already available after installation but it should be enlarged.

Use fdisk to delete the existing partition and create a new, larger one:

The second partition will normally be mounted. Unmount it first.

$ sudo umount /dev/mmcblk0p2
$ sudo fdisk -u /dev/mmcblk0

List partitions with p and note down the start and end sectors of the second partition. Delete the second partition with d, then create a new one with n. Use the same start sector as previously but different end sector. The default end sector uses all available space. Save with w.

After a reboot, log in again an expand the filesystem to fill up the second partition:

$ sudo reboot
$ sudo resize2fs /dev/mmcblk0p2

This should leave us with two partitions on the SD card. A small partition 1 (mmcblk0p1) which holds the OS files and will be unmounted after boot and a larger partition 2 (mmcblk0p2) which takes most of the SD cards capacity and holds installed applications and data.

Add Additional Storage

We will use the No. 2 partition to persist applications and settings on the SD card. We will also set up a removable USB drive to hold bulk data. We create a filesystem on it and set it up to be mounted at boot.

Plug the USB storage into the Raspberry Pi. It should show up in /proc/partitions, blkid can be used to get more info.

$ cat /proc/partitions
$ blkid /dev/sdX

We will put an ext2 filesystem on it. ext2 is non-journaling which should generate less writes and extend the life of the flash drive

First, we create a single partition for the whole disk:

$ sudo fdisk /dev/sdX
> d
> n
> p
> 1
> w

Then, create an ext2 filesystem [4]:

$ mke2fs /dev/sdX1

Since the device is a plugged in drive, we mount it by its UUID. Find the UUID with blkid /dev/sdX1. Also we choose a different directory name for the mountpoint.

To mount it automatically on boot, add the following lines to /opt/bootlocal.sh:

mkdir /mnt/storage
chmod 777 /mnt/storage
mount UUID=b0ee2fe8-c54a-4c67-9a40-a9c2c4cb734c /mnt/storage

And the respective unmount command to /opt/shutdown.sh:

umount /mnt/storage

Tiny Core Persistence

By default, TinyCore Linux has no persistence. That means all changes made in configuration files or newly installed applications are lost when the system reboots. There are two options to keep data between two boots:

  • backup
  • persistent partitions

Software packages for TinyCore Linux are called "extensions" and they are stored in the /tce directory.

The preconfigured image already comes with the /tce directory on the second partition so there is not much to do here.

Verify that /mnt/mmcblk0p2/tce exists; it should contain some preinstalled applications (like openssh). Look at /etc/fstab to see how /dev/mmcblk0p2 is mounted.

The backup utility compresses selected files or directories and stores them as mydata.tgz inside the /tce directory.

The file /opt/.filetool.lst defines which files or directories will be included. /opt/.xfiletool.lst can be used to exclude previously included files (e.g. include a complete directory tree but exclude single large files).

The default list includes the /opt and /home directories, ssh host keys and files related to user management (/etc/passwd for example).

Warning

The backup script does not run automatically. One must include a call to filetool.sh -b in /opt/shutdown.sh. Also, shutdown.sh will only run when the system is shut down with exitcheck.sh.

Install Firefox Sync

The server is installed into a python virtualenv [5] which is located in the syncserver installation directory. The server will later run as a Pyramid [6] app inside a Gunicorn [7] application server.

There is no prebuilt package for Firefox Sync on piCore so we need to download the source and build the software locally.

Install Prerequisites

Start by installing prerequisites:

$ tce-load -wi python
$ tce-load -wo python-dev
$ tce-load -wo make
$ tce-load -wo git
$ tce-load -wo gcc
$ tce-load -wo compiletc
$ tce-load -i python-dev make git gcc compiletc

The -wi switch adds the extension to the OnBoot list which means it is loaded on every boot. The -wo switch creates an OnDemand item which we can load manually with tce-load -i. We install only the base python with the OnBoot option because it is the only package we need for running the app. All other packages are only required for the build process.

The installation additionally requires Python virtualenv for which we do not have a piCore package. But we can install virtualenv from source [8]:

$ wget https://pypi.python.org/packages/8b/2c/c0d3e47709d0458816167002e1aa3d64d03bdeb2a9d57c5bd18448fd24cd/virtualenv-15.0.3.tar.gz
$ tar -xzf virtualenv-15.0.3.tar.gz
$ cd virtualenv-15.0.3
$ sudo python setup.py install
$ cd ..
$ rm virtualenv-15.0.3.tar.gz
$ rm -r virtualenv-15.0.3

The download URL can be obtained from PyPi.

Note

Installing virtualenv in this way places installation files in their default locations. They will be lost on shutdown. This does not matter (much) as we only need virtualenv for the installation, not for running the application.

Create an ffsync User

We want to run the sync service under a dedicated ffsync user. Create it like so:

$ sudo adduser ffsync -SH

Installation Location

Decide upon an installation location.

The default installation procedure is to clone the syncserver git repository [9] and build the application inside the repo directory. Building the application entails creation of a virtual Python environment. The virtualenv is basically a copy of the system-wide Python install plus additional packages installed in a local directory, in this case the syncserver installation directory.

This results in a large number of files and directories. If we keep the installation inside the /home directory, we have the benefit of running it from memory but the disadvantage that the backup and restore for the home directory takes quite long.

If we install to a persistent partition, we lose the "running from memory" advantage.

The proper(?) solution is to create a TinyCore Linux extension for it. This is described in detail in chapter 14 and 15 of the Tiny Core book [10] [PDF]. The basic idea is to create the desired directory structure and files in some temporary directory and then "pack" it with squashfs. TinyCore Linux will later mount/symlink it into the desired location.

That is, if we want to have a file /usr/share/myfile, create /tmp/myextension/usr/share/myfile and then use squashfs to create myextension.tcz from /tmp/myextension.

The syncserver build process (more precisely: virtualenv) will generate some files with hardcoded paths so it is tricky to move the installation to some place different than where it was created. There is an option to make a relocatable virtualenv but it does not seem fully supported and we are not going to use it.

The good thing is that the virtualenv installation keeps all required files under a single base directory. This makes it possible to install the application at the desired location then pack it up and remove the original install.

We will install under /usr/local/syncserver.

First create the required directory and clone the syncserver repo into it:

$ cd /usr/local
$ sudo mkdir syncserver
$ sudo chown syncserver tc:staff
$ git clone --depth=1 https://github.com/mozilla-services/syncserver syncserver
$ cd syncserver
$ make build

Note

Root permissions are only required to create a new directory in /usr/local. The installation itself works with normal permissions.

Use git clone with --depth=1 to retrieve as little history as possible, but still keep it as a git repo.

Delete the syncserver/.git directory to save additional space.

Next, we need to install the Python database driver. This is done using the Python package manager (pip) that is part of the virtualenv. So, from the syncserver directory:

$ ./local/bin/pip install pysqlite

Install pysqlite for Usage with SQLite, PyMySQL for MySQL/MariaDB.

Note

The make build and the installation of pysqlite/PyMySQL with pip will leave several files in ~/.cache/pip. You may want to delete those so they will not be included in the backup.

We should now have a complete installation at /usr/share/syncserver. Running sudo make serve from that directory should start the server with default settings on port 5000.

Look at ./local/bin/gunicorn. On the top it should say:

#!/usr/local/syncserver/local/bin/python2

Config

The default setup assumes that the syncserver is started from inside the installation directory with make serve and that the configuration file is kept there (it is referenced with ./syncserver.ini). If we want to change config separately from the installation, we need to copy it to another location:

$ cp syncserver.ini /etc

And also include it in /opt/.filetool.lst:

etc/syncserver.ini

We will later use a custom command like this to run the service:

$ ./local/bin/gunicorn --paste /etc/syncserver.ini

Create the Extension

First off, install squashfs-tools:

$ tce-load -wo squashfs-tools

To produce a TinyCore Linux extension, we need to transfer everything to a temporary directory structure, pack it and move the resulting .tcz file to the tce directory:

$ cd /tmp
$ mkdir -p syncserver/usr/local
$ sudo mv /usr/local/syncserver syncserver/usr/local
$ sudo chown root:root -R syncserver/
$ mksquashfs syncserver syncserver.tcz
$ mv syncserver.tcz /etc/sysconfig/tcedir/optional
$ sudo rm -r syncserver/

Note

It is a good idea to place a backup of the syncserver.tcz file somewhere other then the tce/optional directory.

We can now start the application with:

$ tce-load -i syncserver

You should see the syncserver installation (as a collection of symlinks) under /usr/local/syncserver.

Note that tce-run will not actually start the service. It will only make the installed application available. Running sudo make serve from the installation directory should now work as before.

To have the application available on boot, add it to the onboot.lst (inside the tce directory):

$ echo syncserver.tcz >> /etc/sysconfig/tcedir/onboot.lst

Start and Stop Script

We need a custom script to start the service. The start script is included in bootlocal.sh to start the service on boot and performs three tasks:

  1. change into the correct working directory (/usr/local/syncserver)
  2. switch to the ffsync user
  3. start the syncserver with our configuration file at /etc/syncserver.ini
BASEDIR=/usr/local/syncserver
GUNICORN=$BASEDIR/local/bin/gunicorn
CONFIG=/etc/syncserver.ini
COMMAND="$GUNICORN --paste $CONFIG --log-file /tmp/syncserver.log > /dev/null &"
(cd $BASEDIR && su ffsync -c "$COMMAND" -s /bin/sh)

We still need to change the working directory before we execute the sync server. If we do this inside the bootlocal.sh script we might affect subsequent actions in that script. One way to avoid this, invoke everything in a subshell.

Also, since the gunicorn command will not exit, "disown" it afterwards with "&".

To stop the service, use:

pkill syncserver

(not pretty)

Configuration

Edit the config file to reflect your settings:

[syncserver]
public_url = http://box:5000/
sqluri = sqlite:////mnt/storage/syncserver/syncserver.db
secret = xxx
allow_new_users = false  # set to true to create initial user

public_url is set to what the client sees as the URL of the service.

If you do not configure a sqluri, the default applies - which is an in-memory database (which is lost when the service is shut down).

The secret is used to generate authentication tokens. It can be any random string. Generate one like this:

$ head -c20 /dev/urandom | sha1sum

allow_new_users should be set to true initially. When the service is fully installed and all required Firefox accounts are registered, it can be set to false.

Install a Database

The syncserver works with SQLite, MySQL/MariaDB or PostgreSQL. There are piCore packages for SQLite and MariaDB.

SQLite

For SQLite, make sure the sqlite3 package is installed. Then decide where the database file should live and update syncserver.ini accordingly:

[syncserver]
...
sqluri = sqlite:////mnt/storage/syncserver/syncserver.db

Remember to make the database directory accessible for the ffsync user.

$ sudo mkdir /mnt/storage/syncserver
$ sudo chown ffsync:staff /mnt/storage/syncserver

MariaDB

There is a (somewhat outdated?) MariaDB installation guide [11] in the TinyCore Linux wiki.

Start by installing MariaDB

$ tce-load -wi mariadb
$ tce-load -wi mariadb-client

Next, we make sure that the DB service starts on boot and stops on shutdown. The package comes only with a partially complete configuration and some effort is required to make it work.

Start by creating a system account to run mariadb:

$ sudo adduser -SH mysql

Next, add commands to start and stop MariaDB [12] to bootlocal.sh and shutdown.sh:

# bootlocal.sh
/usr/local/share/mysql/mysql.server start

# shutdown.sh
/usr/local/share/mysql/mysql.server stop

Note

Make sure that the database service is started before the syncserver, and shut down after the syncserver.

Then create a minimal configuration file for MariaDB [13] file at /etc/my.cnf:

[client]
socket = /tmp/mysql.sock

[mysqld]
socket = /tmp/mysql.sock
basedir = /usr/local
user = mysql
datadir = /usr/local/mysql/data
tmpdir = /tmp
log_error = /tmp/mysql.error.log
pid_file = /tmp/mysql.pid

It is important to set the socket option in the [server] and [client] sections. Otherwise tools like mysqladmin will not see the socket option and will fail to connect.

Make sure that the datadir is accessible for the mysql user:

$ sudo chown -R mysql:nogroup /usr/local/mysql/data

Manually starting and stopping mysqld should now work:

$ sudo /usr/local/share/mysql/mysql.server start
Starting MySQL. SUCCESS!
$ sudo /usr/local/share/mysql/mysql.server stop
Shutting down MySQL.. SUCCESS!

Persistence

For now, the config file and data directory only exist in memory. To persist the config file between reboots, add it to the backup list in /opt/.filetool.lst:

etc/my.cnf

The data directory is where the database files are stored. By default, this is /usr/local/mysql/data.

If this would be kept in memory and backed up/restored when the system restarts, we have a risk of losing data when the system shuts down unexpectedly and the backup script does not run (e.g. when power is unplugged).

To avoid this, keep the database on a separate partition and create a symlink in the original location which points to the new location. Stop the mysqld service before moving the data directory.

$ sudo /usr/local/share/mysql/mysql.server stop
$ sudo mkdir /mnt/storage/mysql
$ sudo chown mysql:nogroup /mnt/storage/mysql
$ sudo mv /usr/local/mysql/data /mnt/storage/mysql
$ sudo ln -s /mnt/storage/mysql/data /usr/local/mysql
$ sudo /usr/local/share/mysql/mysql.server start

We need to repeat parts of this on every reboot. So add the following to bootlocal.sh (before starting mysqld):

# replace the standard datadir with persisted data
MYSQL_DATADIR=/usr/local/mysql/data
rm -rf $MYSQL_DATADIR
ln -s /mnt/storage/mysql/data $MYSQL_DATADIR

Setup a Database User

First, set a password for mysql admin

$ mysqladmin -u root password <secret>

Login to mysql console to create the user and database for ffsync:

$ mysql -u root -p
(enter password when prompted)
mysql> CREATE USER 'ffsync'@'localhost' IDENTIFIED BY '<secret>';
mysql> CREATE DATABASE ffsync;
mysql> GRANT ALL ON ffsync.* TO 'ffsync'@'localhost';

Finally, configure Firefox Sync Server to use the MariaDB database:

[syncserver]
...
sqluri = pymysql://ffsync:<secret>@localhost/ffsync

Configure Firefox

Go to about:config and set:

identity.sync.tokenserver.uri = http://box:5000/token/1.0/sync/1.5

(the default is https://token.services.mozilla.com/1.0/sync/1.5)

You will still need a firefox account to use the sync feature.

Troubleshooting

syncserver complains that public_url does not match application url. Public URL is set to "http://hostname:5000/", error says that app url is "http://hostname/". Change public_url in config file to hostname w/o port, get the same error but the other way round. Set force_wsgi_environ = true fixed this.

With SQLite - errors when clients try to sync:

(OperationalError) cannot start a transaction within a transaction

Tried to limit gunicorn to a single worker thread - no help. Installed MariaDB to work around this.

Note

Type about:sync-log into Firefox address bar to see ...sync logs.

Some additional links: - http://blog.sysbite.org/run-a-firefox-sync-server-on-the-raspberry-pi/ - http://www.raspberry-pi-geek.com/Archive/2015/11/The-new-Firefox-synchronizer - https://wiki.archlinux.org/index.php/Mozilla_Firefox_Sync_Server


[1] https://docs.services.mozilla.com/howtos/run-sync-1.5.html
[2] http://tinycorelinux.net/8.x/armv6/releases/RPi/README
[3] http://tinycorelinux.net/
[4] http://www.tldp.org/HOWTO/Flash-Memory-HOWTO/ext2.html
[5] https://virtualenv.pypa.io/en/stable/
[6] https://trypyramid.com/
[7] http://gunicorn.org/
[8] https://virtualenv.pypa.io/en/stable/installation/
[9] https://github.com/mozilla-services/syncserver
[10] http://tinycorelinux.net/corebook.pdf
[11] http://wiki.tinycorelinux.net/wiki:mysql_persistence_guide
[12] https://mariadb.com/kb/en/mariadb/starting-and-stopping-mariadb-automatically/
[13] https://mariadb.com/kb/en/mariadb/configuring-mariadb-with-mycnf/