diff --git a/README.md b/README.md index e98ee1c..fd85c60 100644 --- a/README.md +++ b/README.md @@ -1,343 +1,351 @@ # Micro Backup script This is a keep it simple and stupid backup script. ## Installation Choose a directory and then clone this repository: ``` git clone URL ``` Example: ``` sudo -i cd /opt git clone https://gitpull.it/source/micro-backup-script/ ``` Please change the URL to point on your own fork (if you have one). ## Configuration Enter in the cloned directory and run: ``` cp backup-instructions-example.conf backup-instructions.conf cp options-example.conf options.conf ``` Then edit these files with your favorite text editor: * `options.conf` * `backup-instructions.conf` ## Usage After you configured the script for your needs, there are no arguments. Just run this: ``` sudo ./backup-everything.sh ``` You can schedule it from your crontab. Example: ``` $ sudo crontab -e # on-site backup at night #m h dom mon dow command 00 1 * * * /opt/micro-backup-script/backup-everything.sh ``` ## Options documentation (`options.conf`) The file `options.conf` is designed to store important options (like `BOX` and `BASE`). The file `options.conf` can be copied from the example called `options-example.conf`. ### Option `BOX` The `BOX` option sets the human name for your local computer. Default: current `hostname`. The `BOX` option is recommended in order to be independent from the hostname and have more stable backup pathnames. The `BOX` option is used to create a directory with the same name. Here 3 examples: ``` BOX=gargantua BOX=my-local-nice-machine BOX=server001.example.com ``` NOTE: The `BOX` option is used to build the final pathname of your on-site backups. See below. ### Option `BASE` The `BASE` option sets the base pathname for on-site backups. Default: `/home/backups` for historical reasons. The `BASE` option is strongly suggested in order to be independent from the system default. The `BASE` option should **not** end with a slash. The `BASE` option should contain a valid filesystem pathname. Here 3 examples: ``` BASE=/var/backups/stark-industries BASE=/mnt/stark-industries BASE=/tmp/test-backups ``` For example if you set `BASE=/var/backups/stark-industries` and `BOX=gargantua`, your on-site backups will be placed in `/var/backups/stark-industries/gargantua`. ### Option `PORCELAIN` The option `PORCELAIN` allows to debug everything and do nothing. It's useful to see what would be done without executing any command. The option `PORCELAIN` accepts an empty value (default) or `1`. When `1` the flag is activated. Here 2 examples: ``` # run all instructions normally (default) PORCELAIN= # do not run any instruction but just print them PORCELAIN=1 ``` The option `PORCELAIN` will skip the following actions when set to `1`: * skip any database dump (avoiding to run `mysqldump`) * skip any data transfer via rsync (adding a `--dry-run') * skip any systemd stop/start command ### Option `NO_DISSERVICE` The option `NO_DISSERVICE` is a flag that can avoid any command related to a systemd service. It's disabled as default. The option `NO_DISSERVICE` is only useful if you use some backup instructions related to systemd and you want to debug them. The option `NO_DISSERVICE` accepts an empty value (default) or `1`. When `1` the flag is activated. Here 2 examples: ``` # run all systemd-related instructions normally (default) NO_DISSERVICE= # do not run any systemd-related instruction NO_DISSERVICE=1 ``` +### Option `HOURS_INTERVAL` + +The option `HOURS_INTERVAL` can be used to set a desired minimum time window (in hours) between each execution. + +The option `HOURS_INTERVAL` is particularly effective to mitigate daylight saving issues and race conditions in general. + +The option `HOURS_INTERVAL` defaults to `12` hours. It can be disabled setting an empty string. + ## Instructions documentation (`backup-instructions.conf`) The file `backup-instructions.conf` contains the backup commands (which databases should be saved, which pathnames, etc.) The file `backup-instructions.conf` can be copied from an example called `backup-instructions-example.conf`. ### Instruction `backup_path` The instruction `backup_path` instruction does an on-site copy of a directory or a single file. Examples: ``` # save the Unix configuration files backup_path /etc # save all user homes backup_path /home # save a copy of this specific log file backup_path /mnt/something/log.err ``` The data will be saved in a sub-directory of `$BASE/$BOX/daily/files` keeping the original structure. For example the path `/mnt/something/log.err` will be stored under `$BASE/$BOX/daily/files/mnt/something/log.err`. ### Instruction `backup_paths` The instruction `backup_paths` (note it ends with an "s") allows to save multiple pathnames or use Bash globs to capture multiple pathnames. Example: ``` # backup only the log files who start with "Kern" backup_paths /var/log/Kern* # backup these pathnames backup_paths /home/mario /home/wario ``` ### Instruction `backup_last_log_lines` The instruction `backup_last_log_lines` saves the last lines of a long txt file. Example: ``` backup_last_log_lines /var/log/secure ``` ### Instruction `backup_database` The instruction `backup_database` runs a `mysqldump` on a specific database and compress it with `gzip`. The instruction `backup_database` saves the database under `$BASE/$BOX/daily/databases/$DATABASE_NAME.sql.gzip`. Examples: ``` # first, backup a database with a specific name backup_database wordpress_testing # then, backup another database backup_database wordpress_production ``` ### Instruction `backup_every_database` The instruction `backup_every_database` runs a `mysqldump` for every database (but not on the skipped ones). The instruction `backup_every_database` skips as default `information_schema` and `performance_schema` and more databases can be ignored using the instruction `skip_database`. Example: ``` # skip 2 databases skip_database "^BIG_DATABASE_PRODUCTION_ALPHA$" skip_database "^BIG_DATABASE_PRODUCTION_BETA$" # backup all the others backup_every_database ``` ### Instruction `skip_database` The instruction `skip_database` adds another database name from the exclusion list of `backup_every_database`. The instruction `skip_database` accepts only one argument expressed as a regular expression and has no effect if it's executed after `backup_every_database` or without a `backup_every_database` in your instructions. Examples: ``` # skip this specific database skip_database "^BIG_DATABASE_PRODUCTION_ALPHA$" # skip also all databases starting with the prefix 'OLD_' skip_database "$OLD_.*$" # backup all the remaining databases backup_every_database ``` ### Instruction `skip_database_table_data` The instruction `skip_database_table_data` can be used to skip the table data. The system automagically saves that table schema (and table triggers, etc.) in a separate file. The instruction can be used multiple times to specify more tables to be ignored. The instruction must be specified before any `backup_every_database` and before `backup_database` (on that specific database at least). ``` skip_database_table_data DBNAME1 TABLENAME1 skip_database_table_data DBNAME1 TABLENAME2 skip_database_table_data DBNAME2 TABLENAMEANOTHER ``` ### Instruction `backup_service_and_database` The instruction `backup_service_and_database` can be used to stop a service, backup its database, and restart the service. The instruction `backup_service_and_database` does not try to stop a not-running service and does not try to start it if it was not running. If it's not running it just backups the expressed database. Example: ``` backup_service_and_database tomcat9.service WEBAPPDB ``` ### Instruction `backup_phabricator` The instruction `backup_phabricator` puts a Phabricator installation in maintenance mode, then it dumps all its databases, and then it removes maintenance mode. Examples: ``` backup_phabricator /var/www/phabricator DATABASEPREFIX_ ``` ### Instruction `push_path_host` The instruction `push_path_host` sends local files (e.g. `/home/foo`) to a remote host (e.g. `example.com`). Example: ``` push_path_host /home/foo backupuser@example.com:/var/backups/remote-destination ``` The instruction `push_path_host` works running an `rsync` command and connecting to SSH to the remote host. So your local Unix user will run `ssh backupuser@example.com`. So, if it does not work, and if you have no idea how SSH works, just run these and press enter 10 times from your local Unix user: ``` ssh-keygen ssh-copy-id backupuser@example.com ``` If it still does not work and you don't know how to configure SSH or how to use rsync, trust me, RTMF about SSH and rsync. ### Instruction `push_daily_directory` The instruction `push_daily_directory` sends your local daily backup to a remote host (e.g. `example.com`). Example: ``` push_daily_directory backupuser@example.com:/var/backups/remote-destination ``` The instruction `push_daily_directory` internally uses the `push_path_host` passing the pathname `$BASE/$BOX/daily` as first argument. ## Utility `rotate.sh` The utility `rotate.sh` is like logrotate applied to a directory. It's designed to configure backup data retention. The utility `rotate.sh` does not suffer from timezone changes. It has a mechanism to avoid to be launched twice by mistake. The utility `rotate.sh` has this help menu: ``` ./rotate.sh PATH DAYS MAX_ROTATIONS ``` * `PATH`: the directory to be rotated * `DAYS`: the minimum amount of days between each rotation * `MAX_ROTATIONS`: the maximum allowed rotation (the next one will be dropped) The utility `rotate.sh` creates directories named like `PATH` but with a suffix like `.1` and `.2` etc. up to `MAX_ROTATIONS`. The utility `rotate.sh` can be used to rotate a directory (e.g. `/var/backups`) every 1 day up to 30 days and automatically drop older rotations. Example: ``` $ sudo crontab -e # every 1 day at 2:00 rotate my latest backups (/var/backups) for max 30 times # NOTE: this creates /var/backups.{1..30} where .1 is the most recent and .30 the oldest #m h dom mon dow command 0 2 * * * /opt/micro-backup-script/rotate.sh /var/backups 1 30 ``` The utility `rotate.sh` allows to have longer times between rotations: ``` $ sudo crontab -e 00 1 * * * /opt/my-rotate.sh ``` ``` name=/opt/my-rotate.sh #!/bin/sh # rotate my backups every day for 30 days to have /var/backups.{1..30} # NOTE: this creates /var/backups.{1..30} /opt/micro-backup-script/rotate.sh /var/backups 1 30 # then rotate by oldest backup (/var/backups.30) every week for 10 times # NOTE: this creates /var/backups.30.{1.10} /opt/micro-backup-script/rotate.sh /var/backups.30 7 10 ``` ## License 2020-2024 Valerio Bozzolan, ER Informatica, contributors MIT License https://mit-license.org/ ## Contact For EVERY question, feel free to contact Valerio Bozzolan: https://boz.reyboz.it diff --git a/backup-everything.sh b/backup-everything.sh index 955dd9e..7567ac0 100755 --- a/backup-everything.sh +++ b/backup-everything.sh @@ -1,722 +1,739 @@ #!/bin/bash ### # Micro script to backup some stuff # # IMPORTANT: do not edit this file # and see 'backup-instructions.conf' instead! # # This file defines some functions and then it executes # your 'backup-instructions.conf' # # Author: 2020, 2021 Valerio Bozzolan # License: MIT ## # try to proceed even in case of errors (to do not skip any backup) # set -e # current directory MYDIR="$(dirname "$(realpath "$0")")" # load useful stuff . "$MYDIR"/bootstrap.sh # no instructions no party if [ ! -f "$INSTRUCTIONS" ]; then echo "missing instructions expected in $INSTRUCTIONS" exit 1 fi ### # Create a directory with safe permissions if it does not exist # # @param string path Pathname # write_safe_dir_if_unexisting() { local path="$1" # try to create the directory if it does not exists if [ ! -d "$path" ]; then warn "creating missing $path with 750" mkdir --parents "$path" chmod 750 "$path" fi # it must exists now require_existing_dir "$path" } ### # Create a directory for a filename with safe permissions if it does not exist # # @param string path Pathname to a filename # @param int is_file_mode Check if we are in file mode (default: directory mode) # write_safe_basedir_if_unexisting() { # first function argument local path="$1" # extract just the sub-directory local basepath basepath=$(dirname "$path") # create that sub-directory write_safe_dir_if_unexisting "$basepath" } ### # Require an existing directory or die # # @param string Directory path # require_existing_dir() { if [ ! -d "$1" ]; then error "unexisting directory $1" exit 1 fi } # create these pathnames if they do not exist write_safe_dir_if_unexisting "$BASE" write_safe_dir_if_unexisting "$DAILY" write_safe_dir_if_unexisting "$DAILY_FILES" write_safe_dir_if_unexisting "$DAILY_DATABASES" # create or clean the last log file if [ "$WRITELOG" = 1 ]; then cat /dev/null > "$DAILY_LASTLOG" fi # # Dump and compress a database by its name. # # @param db string Database name # @param spaces string Cosmetic spaces # backup_database() { local db="$1" local spaces="$2" local path="$DAILY_DATABASES"/"$db".sql.gz # Log the most appropriate message. if [ -n "${TABLES_TO_BE_SKIPPED["$db"]}" ]; then # The user can ignore some tables. Better to have that information in an explicit way. log "$spaces""dumping database $db (skipping tables: ${TABLES_TO_BE_SKIPPED["$db"]})" else log "$spaces""dumping database $db" fi # no database no party if ! $MYSQL "$db" -e exit > /dev/null 2>&1; then warn "$spaces skip unexisting database" return fi # # MySQL dump with table skipper # # Eventually generate something like: # 'mysqldump --ignore-table=DB.table1 --ignore-table=DB.table2' etc. local mysqldump="$MYSQLDUMP" for table in ${TABLES_TO_BE_SKIPPED["$db"]}; do mysqldump+=" --ignore-table=$db.$table" done # Perform the operation only if porcelain mode is disabled. if [ "$PORCELAIN" = 1 ]; then # Debug the dump without doing it. log "$spaces [PORCELAIN] $mysqldump $db > $path" else # Perform the dump. $mysqldump "$db" | gzip > "$path" fi # If we ignored some tables, let's also dump the schema of these tables, separately. # So the backup just ignores the data, and not the schema, and tables are easy to be recovered. # So this is just an extra 'mysqldump' with '--no-data'. # We also use '--skip-dump-date' since the schema has possibility to never change, # and so, also the file should be (binary) the same. # So, without dates, we save bytes on your backup storage, and also some transfer bandwidth. for table in ${TABLES_TO_BE_SKIPPED["$db"]}; do local path_dump_nodata="$DAILY_DATABASES"/"$db"."$table".nodata.sql.gz local mysqldump_nodata="$MYSQLDUMP --no-data --skip-dump-date" log "$spaces saving schema of table $db.$table" # Perform the operation only if porcelain mode is disabled. if [ "$PORCELAIN" = 1 ]; then log "$spaces [PORCELAIN] $mysqldump_nodata $db $table > $path_dump_nodata" else $mysqldump_nodata "$db" "$table" | gzip > "$path_dump_nodata" fi done } # # Dump and compress some databases # # @param database... string Database names # backup_databases() { local something_done= for database in $@; do something_done=1 backup_database "$database" done if ! [ "$something_done" = 1 ]; then error "Bad usage: backup_databases DATABASE_NAME1 DATABASE_NAME2" error " Have you confused this instruction with this one?" error " backup_every_database" exit 1 fi } # # Dump some databases with a shared prefix # # @param prefix string Database prefix # @param spaces string Cosmetic spaces # backup_databases_prefixed() { local prefix="$1" local spaces="$2" log "$spaces""backup all databases prefixed with: $prefix" databases=$($MYSQL --skip-column-names --execute='SHOW DATABASES' | grep "$prefix") for database in $databases; do backup_database "$database" "$spaces " done } # default databases to be skipped DATABASE_SKIP_LIST=("^information_schema$" "^performance_schema$") # database tables indexed by database name declare -A TABLES_TO_BE_SKIPPED # # Call this to skip a database # # @param string database Database pattern (Bash regex) # # This should be called before calling backup_every_database. # skip_database() { local db="$1" DATABASE_SKIP_LIST+=( "$db" ) } # # Skip data of a specific database table. # # We will do our best to skip the table schema instead. # This must be called before calling 'backup_every_database' # and also before 'backup_database DBNAME' to have effect. # # To skip multiple tables, just call this method for each table. # # @param string database Database name (exact name) # @param string table Database table (exact name) # @return void # skip_database_table_data() { local db="$1" local table="$2" # Eventually pre-pend a space. Since the tables are just a space-separated string. if [ -n "${TABLES_TO_BE_SKIPPED["$db"]}" ]; then TABLES_TO_BE_SKIPPED["$db"]+=" " fi # Add the table to be ignored. TABLES_TO_BE_SKIPPED["$db"]+="$table" } # # Backup every single database. That's easy. # backup_every_database() { # backup every damn database databases=$($MYSQL -e 'SHOW DATABASES') for database in $databases; do local do_backup=1 # just skip the information_schema and the performance_schema that cannot be locked for skip_entry in "${DATABASE_SKIP_LIST[@]}"; do if [[ "$database" =~ $skip_entry ]]; then # show a cute message log "skippin database $database matching blacklist" # do not backup this do_backup=0 # do not check other entries break fi done # backup if it does not match the above skip entries if [ "$do_backup" = 1 ]; then backup_database "$database" fi done } # # Backup a database that is used by a service # # The service will be stopped before dumping. # # @param service string Systemd unit file # @param db string Database name # backup_service_and_database() { local service="$1" local db="$2" local is_active= log "start backup service '$service' with database '$db'" # check if the service is running if systemctl is-active "$service" > /dev/null; then is_active=1 fi # eventually stop the service if [ $is_active == 1 ]; then if [ "$NO_DISSERVICE" == 1 ]; then warn " NOT stopping service: $service (to avoid any disservice)" else log " stopping service: $service" if [ "$PORCELAIN" != 1 ]; then systemctl stop "$service" fi fi else log " service already inactive: $service" fi # backup the database now that the service is down backup_database "$db" " " # eventually start again the service if [ $is_active == 1 ]; then if [ "$NO_DISSERVICE" == 1 ]; then warn " NOT starting again service: $service (to avoid any disservice)" else log " starting again service: $service" if [ "$PORCELAIN" != 1 ]; then systemctl start "$service" fi fi fi } # # Backup some databases used by a service # # The service will be stopped before dumping the databases # # @param service string Systemd unit file # @param db... string Database names # backup_service_and_databases() { backup_service_and_database $@ } # # Backup phabricator # # @param path string Phabricator webroot # @param dbprefix string Database prefix # backup_phabricator() { local path="$1" local dbprefix="$2" local repos_full_path="$3" local configset="$path"/bin/config log "backup Phabricator databases" log " set Phabricator read only mode (and wait some time to make it effective)" $configset set cluster.read-only true > /dev/null sleep 5 backup_databases_prefixed "$dbprefix" " " # if specified, backup the repositories if [ -n "$repos_full_path" ]; then log " start Phabricator repositories backup" backup_path "$repos_full_path" else log " skipping repositories (be sure they are under another backup)" fi log " revert Phabricator read only mode" $configset set cluster.read-only false > /dev/null } # # Backup a directory or a filename # # @param path string Pathname to be backupped # @param identifier string Optional identifier of this pathname # backup_path() { local path="$1" local identifier="$2" # create a default identifier if [ -z "$identifier" ]; then identifier="$path" fi # destination directory # note that the identifier may start with a slash but this is good, just don't care dest="$DAILY_FILES/$identifier" # tell were the backup will go if [ "$path" = "$identifier" ]; then log "backup $path" else log "backup $path with identifier $identifier" fi # check if the path exists if [ -e "$path" ]; then # check if it's a directory if [ -d "$path" ]; then # this is a directory # eventually create the destination if it does not exist write_safe_dir_if_unexisting "$dest" # backup this # force the source to end with a slash in order to copy the files inside the directory $RSYNC "$path/" "$dest" else # this is a filename # eventually create the base destination if it does not exist write_safe_basedir_if_unexisting "$dest" # backup this filename $RSYNC "$path" "$dest" fi else # no path no party warn " path not found: $path" fi } # # Backup last lines of a filename and compress them # # @param path string Pathname to be backupped # @param identifier string Optional identifier of this pathname (it's considered a file) # @param lines string Lines to be saved # backup_last_log_lines() { local path="$1" local identifier="$2" local lines="$3" # create a default identifier if [ -z "$identifier" ]; then identifier="$path" fi # default lines if [ -z "$lines" ]; then lines="$BACKUP_LAST_LOG_LINES" fi # destination directory # note that the identifier may start with a slash but this is good, just don't care dest="$DAILY_FILES/$identifier" # tell were the backup will go if [ "$path" = "$identifier" ]; then log "backup $path last $lines log lines" else log "backup $path last $lines log lines with identifier $identifier" fi # check if the path exists if [ -e "$path" ]; then # this is a filename # eventually create the base destination if it does not exist write_safe_basedir_if_unexisting "$dest" # backup this file tail -n "$lines" "$path" > "$dest" log "compressing $dest" gzip --force "$dest" else # no path no party warn " path not found: $path" fi } # # Backup some filenames ending with whatever # # @param path[...] string Pathname to be backupped # backup_paths() { # pathnames local paths="$@" # process every pathname for path in $paths; do # backup every single pathname backup_path "$path" done } ## # Validate the backup host arguments # # validate_backup_host_args() { # function parameters local port="$3" # no host no party if [ -z "$1" ] || [ -z "$2" ]; then error "Bad usage: backup_host REMOTE_HOST:REMOTE_PATH IDENTIFIER [PORT]" exit 1 fi # tell what we will do if [ "$port" = "22" ]; then log "backup $1 in $2" else log "backup $1 (:$port) in $2" fi } ## # Backup another host directory via rsync # # @param string Rsync host # @param string Rsync path to a directory # @param string Rsync host port # backup_host_dir() { # function parameters local hostlink="$1" local identifier="$2" local port="$3" # eventually set default rsync port if [ -z "$port" ]; then port=22 fi # validate the arguments validate_backup_host_args "$hostlink" "$identifier" "$port" # destination path local dest="$BASE/$identifier" # create the destination if it does not exist write_safe_dir_if_unexisting "$dest" # backup everything # force the source to end with a slash to just copy the files inside it $RSYNC_REMOTE --rsh="ssh -p $port" "$hostlink/" "$dest" } ## # Backup another host directory via rsync # # @param string Rsync host # @param string Rsync path to a directory # push_path_host() { # function parameters local source="$1" local hostlink="$2" local port="$3" # eventually set default rsync port if [ -z "$port" ]; then port=22 fi # validate the arguments and print message validate_backup_host_args "$source" "$hostlink" "$port" # backup everything # force the source to end with a slash to just copy the files inside it $RSYNC_REMOTE --rsh="ssh -p $port" "$source" "$hostlink" } ## # Push some pathnames into an host # # @param string Rsync host # @param string Rsync port # @param string... Rsync path to some directories # push_host_port_paths() { # function parameters local hostlink="$1" local port="$2" # strip the first two arguments shift shift local path="" # process every pathname while [[ $# -gt 0 ]]; do path="$1" shift # backup every single pathname push_path_host "$path" "$hostlink" "$port" done } ## # Push the local "daily/" directory into a remote host # # @param string Rsync host # @param string Rsync path to a directory # push_daily_directory() { # function parameters local hostlink="$1" local port="$2" push_path_host "$DAILY/" "$hostlink" "$port" } ## # Backup another host file via rsync # # @param string hostlink Rsync source host and pathname e.g. ravotti94:/tmp/asd.txt # @param string identifier Rsync destination directory # backup_host_file() { # function parameters local hostlink="$1" local identifier="$2" local port="$3" # filename local filename= filename=$(basename "$hostlink") # eventually set default rsync port if [ -z "$port" ]; then port=22 fi # validate the arguments validate_backup_host_args "$hostlink" "$identifier" "$port" # destination path local dest="$BASE/$identifier/$filename" # create the destination directory if it does not exist write_safe_basedir_if_unexisting "$dest" # backup everything # force the source to end with a slash to just copy the files inside it $RSYNC_REMOTE --rsh="ssh -p $port" "$hostlink" "$dest" } # # Send a date to a Zabbix item # # This will be done only if Zabbix is installed. # # The Zabbix item must be of type "Zabbix trapper" # # If the "zabbix_sender" command is not installed, nothing will be done. # # Item definition: # - Type: Zabbix trapper # - Unit: unixtime # - Type: Numeric # zabbix_sender_date_if_installed() { local zabbix_item_key="$1" # check if Zabbix sender is already installed if which zabbix_sender > /dev/null; then local current_time current_time="$(date +%s)" # note that the Zabbix sender can fail # note that the Zabbix output is unuseful # if you have problems, remove this part: > /dev/null if zabbix_sender \ --config /etc/zabbix/zabbix_agentd.conf \ --key "$zabbix_item_key" \ --value "$current_time" > /dev/null; then log "sent Zabbix value for key $zabbix_item_key" else error "fail sending Zabbix value for key $zabbix_item_key - exit status: $?" fi fi } +# do not proceed if not enough time passed since last execution +# this avoids daylight saving time change problems +# this also avoids race conditions when starting parallel executions by mistake +if [ -n "$DAILY_STARTTIME" ] && [ -n "$HOURS_INTERVAL" ]; then + if ! are_enough_hours_passed "$DAILY_STARTTIME" "$HOURS_INTERVAL"; then + warn "doing nothing: last backup was too recent: last-now $(date +%s)-$(< "$DAILY_STARTTIME") - expected at least HOURS_INTERVAL: $HOURS_INTERVAL" + exit 0 + fi +fi + +# Remember when we started. +# This also avoids race confitions on daylight saving changes +# or other unexpecteyd and potentially dangerous parallel executions. +# An example race condition may happen if the backup script does not conclude +# before the next crontab call. +write_timestamp "$DAILY_STARTTIME" + log "init backup in $DAILY" # try to record when we started zabbix_sender_date_if_installed micro_backup_date_start . "$INSTRUCTIONS" # try to record when we concluded zabbix_sender_date_if_installed micro_backup_date_stop log "backup successfully concluded" # remember the last successfully concluded backup timestamp write_timestamp "$DAILY_LASTTIME" diff --git a/bootstrap.sh b/bootstrap.sh index 01ed1d0..d86c0a9 100644 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -1,308 +1,313 @@ #!/bin/bash ### # Part of a stupid script to backup some stuff # # This bootstrap.sh file does nothing by itself but loads useful stuff. # # This file is loaded from 'backup-everything.sh' or 'rotate.sh' # # Author: 2020-2024 Valerio Bozzolan, contributors # License: MIT ## # current directory export DIR="${BASH_SOURCE%/*}" if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi # check if the standard input is not a terminal export INTERACTIVE= if [ -t 0 ]; then INTERACTIVE=1 fi # # Check if this is the quiet mode # # Default - not quite. # # Actually we are in quiet mode if it's not interactive. # This lazy behavior is to avoid stupid emails from the crontab # without the need to specify some --quiet etc. # Note that in quiet mode only WARN and ERROR messages are shown. # I've not created a --quiet flag because nobody is needing it. # # Edit your options - do not edit here. # #QUIET= # # Eventually write a log file # # Default - write a log file. # # Edit your options - do not edit here. # export WRITELOG=1 # path to the instructions file export INSTRUCTIONS="$DIR/backup-instructions.conf" # path to the configuration file export CONFIG="$DIR/options.conf" # no config no party if [ ! -f "$CONFIG" ]; then echo "missing options expected in $CONFIG" exit 1 fi # default mysql commands # --batch: avoid fancy columns (auto-enabled, but better to specify it) # --silent: avoid the column name to be included export MYSQL="mysql --batch --silent" export MYSQLDUMP="mysqldump --routines --triggers" # default rsync command # --archive: Try to keep all the properties # --fuzzy: Try to check if a file was renamed instead of delete and download a new one # It's efficient for example with log rotated files. # --delete: Delete the destination files if not present in the source # NOTE: we want this behaviour but it's not a good idea toghether with --fuzzy # that's why we do not use --delete but we use the next flags # --delay-updates Put all updated files into place at end (useful with fuzzy and delete modes) # --delete-delay Delete after everything (useful with fuzzy and delete modes) # NOTE: sometime some data is kept in damn .~tmp~ directories # So we are deprecating --delete-delay, and going back to --delete # and so removing --fuzzy # --hard-links Try to look for hard links during the transfer to do not copy separate files #RSYNC="rsync --archive --fuzzy --delay-updates --delete-delay --hard-links" # default rsync command # --archive: Try to keep all the properties # --delete: Delete the destination files if not present in the source # --hard-links Try to look for hard links during the transfer to do not copy separate files export RSYNC="rsync --archive --delete --hard-links" # rsync used in remote transfers # --compress Use more CPU to save network bandwidth export RSYNC_REMOTE="$RSYNC --compress" # default base backup directory for all backups export BASE="/home/backups" # default box name BOX="$(hostname)" export BOX # set to 1 to avoid any disservice (e.g. systemctl stop/start) export NO_DISSERVICE= # set to 1 to do nothing export PORCELAIN= +# How many hours should pass between each execution. +# This is just a sane default to avoid daylight saving issues. +export HOURS_INTERVAL=12 + # include the configuration to eventually override some options # shellcheck source=config.sh . "$CONFIG" # as default, if not interactive, set quite mode if [ -z "$QUIET" ] && [ "$INTERACTIVE" != 1 ]; then QUIET=1 fi # full pathnames to the backup directories export BASEBOX="$BASE/$BOX" export DAILY="$BASEBOX/daily" export DAILY_FILES="$DAILY/files" export DAILY_DATABASES="$DAILY/databases" export DAILY_LASTLOG="$DAILY/last.log" export DAILY_LASTTIME="$DAILY/last.timestamp" +export DAILY_STARTTIME="$DAILY/start.timestamp" # apply the porcelain to the rsync command if [ "$PORCELAIN" = 1 ]; then RSYNC="$RSYNC --dry-run" RSYNC_REMOTE="$RSYNC_REMOTE --dry-run" fi # set default backup_last_log() lines if [ -z "$BACKUP_LAST_LOG_LINES" ]; then BACKUP_LAST_LOG_LINES=8000 fi ## # Receive in input a file path, and a number of hours, and check whenever -# enough time (in days) was passed or not. +# enough time (in hours) was passed or not. # # If the file was never created, we assume that enough time was passed. # # @param string timestamp_file -# @param int days +# @param int hours # -function are_enough_days_passed() { +function are_enough_hours_passed() { # No args, no party. # Note that the file argument will be checked later. local timestamp_file="$1" - local expected_days="$2" - if [ -z "$expected_days" ]; then - echo "Error: Missing argument expected days." + local expected_hours="$2" + if [ -z "$expected_hours" ]; then + echo "Error: Missing argument expected hours." exit 2 fi - local expected_seconds=$((expected_days * 24 * 3600)) + local expected_seconds=$((expected_hours * 3600)) - are_enough_seconds_passed "$timestamp_file" "$expected_hours" + are_enough_seconds_passed "$timestamp_file" "$expected_seconds" } ## # Receive in input a file path, and a number of hours, and check whenever # enough time (in seconds) was passed or not. # # If the file was never created, we assume that enough time was passed. # # @param string timestamp_file # @param int seconds # function are_enough_seconds_passed() { # No args, no party. local timestamp_file="$1" local expected_hours="$2" if [ -z "$timestamp_file" ]; then echo "Error: Missing argument timestamp file." exit 2 fi if [ -z "$expected_hours" ]; then echo "Error: Missing argument expected hours." exit 2 fi if [ -f "$timestamp_file" ]; then # Read the file, if it has sense. local last_timestamp=$(<"$timestamp_file") if [ "$last_timestamp" -lt 1000 ]; then echo "Error: Bad format in file $timestamp_file" exit 2 fi local current_timestamp=$(date +%s) local diff_seconds=$((current_timestamp - last_timestamp)) # If enough time is passed, return true. [ "$diff_seconds" -ge "$expected_seconds" ]; fi # The file doesn't exist. Return nothing special (0, that is True). } ## # Receive in input a file path, and write there the current timestamp. # # @param string timestamp_file # function write_timestamp() { # No arg, no party. local timestamp_file="$1" if [ -z "$timestamp_file" ]; then echo "Error: Missing timestamp file argument." exit 1 fi # Write the current Unix timestamp. date +%s > "$timestamp_file" } ### # Print something # # It also put the message in the backup directory # # @param string severity # @param string message # function printthis() { local msg msg="[$(date)][$1] $2" # print to standard output if it's not in quiet mode if [ "$QUIET" != 1 ]; then printf "%s\n" "$msg" fi # put in the log file if possible if [ -f "$DAILY_LASTLOG" ] && [ "$WRITELOG" = 1 ]; then printf "%s\n" "$msg" >> "$DAILY_LASTLOG" fi } ### # Run an rsync # function copy() { # show what we are doing log "copy $*" # run the rsync command if [ "$PORCELAIN" != 1 ]; then $RSYNC $@ fi } ### # Remove a pathname # function drop() { # show what we are doing log "drop $*" # well, proceed... finger crossed... with some protections if [ "$PORCELAIN" != 1 ]; then rm --recursive --force --one-file-system --preserve-root -- $@ fi } ### # Move something somewhere # function move() { # show what we are doing log "move $*" if [ "$PORCELAIN" != 1 ]; then mv --force $@ fi } ### # Print a information message # # @param msg Message # function log() { printthis INFO "$1" } ### # Print a warning message # # @param msg Message # function warn() { printthis WARN "$1" } ### # Print an error message # # @param msg Message # function error() { printthis ERROR "$1" } diff --git a/options-example.conf b/options-example.conf index 97c7b5a..2d2d9c6 100644 --- a/options-example.conf +++ b/options-example.conf @@ -1,34 +1,38 @@ # # Generic options for 'micro-backup-script' # # See: # https://gitpull.it/source/micro-backup-script/ # # Notes: # If you have not an options.conf then copy it from options-example.conf # # This file is loaded after 'bootstrap.sh' ####################################################################### # please uncomment and set here your hostname or something like that #BOX=my-important-server # eventually change the place where you want to put backups (no trailing slash) #BASE=/home/backups +# uncomment this if your script may be executed often (daily?) but you only want +# a weekly backup. +#HOURS_INTERVAL=128 + # eventually uncomment these if you have custom mysql commands to be root #MYSQL="mysql --defaults-file=/etc/mysql/debian.cnf" #MYSQLDUMP="mysqldump --defaults-file=/etc/mysql/debian.cnf" # uncomment to set custom flags on local copies with rsync (e.g. verbose mode) #RSYNC="$RSYNC --verbose" # uncomment to set custom bandwidth in KBps and other flags in remote copies with rsyncs # Note: remote copy already use compression so NO need to add here again #RSYNC_REMOTE="$RSYNC_REMOTE --bwlimit=900" # eventually uncomment to avoid any disservice (e.g. avoid systemctl stop/start) #NO_DISSERVICE=1 # eventually uncomment to test things and do nothing #PORCELAIN=1