diff --git a/README.md b/README.md index 33cd1fd..f708d65 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,8 @@ rsync -aH --exclude '*.sqlite3*' storage/ backup/ # uploaded files (-H keeps th Alternatively, stop the container and copy the volume. +To restore, stop the app and run `bin/rails backups:restore` (the newest nightly copy, or a file such as one downloaded from Google Drive); it checks the copy first and keeps the current database aside. Then `bin/rails records:check` verifies the history and signatures. `bin/rails sam:doctor` gives a health report of the whole installation. All commands are in the [administration guide](docs/guide/administration.md#maintenance-commands). + The app also makes a verified copy of the main database every night in `storage/backups` (Admin → About shows the last one), so a backup of the storage volume always contains a consistent database snapshot. For an off-site copy without extra tools, **Admin → Off-site Backup** connects a Google Drive account: each night the verified database copy and any new uploaded files are copied there, and a restore test checks them. Setup and restoring are in the [administration guide](docs/guide/administration.md#off-site-backup-to-google-drive). Keep `SECRET_KEY_BASE` somewhere safe off the server: stored secrets (API keys, the Drive connection) and electronic signature seals depend on it. diff --git a/app/jobs/offsite_backup_job.rb b/app/jobs/offsite_backup_job.rb index c97fc45..6a6b5f5 100644 --- a/app/jobs/offsite_backup_job.rb +++ b/app/jobs/offsite_backup_job.rb @@ -7,14 +7,14 @@ class OffsiteBackupJob < ApplicationJob def perform(kind = "backup") return unless OffsiteConnection.connected? - run = kind == "restore_test" ? OffsiteBackup.restore_test! : backup + run = kind == "restore_test" ? OffsiteBackup.restore_test! : self.class.backup return if run.nil? || run.ok || !MailSettings.configured? User.active.where(admin: true).find_each { AdminMailer.offsite_alert(_1, run).deliver_now } end - private - - def backup + # Copies the latest verified backup (making one if there's none from the + # last day). Returns the OffsiteRun. Also used by bin/rails offsite:backup. + def self.backup latest = BackupRun.where(ok: true).order(:created_at, :id).last latest = DatabaseBackup.run! unless latest && latest.created_at > 1.day.ago && latest.path.present? && File.exist?(latest.path) return OffsiteRun.create!(kind: "backup", ok: false, message: "No verified database backup to copy: #{latest.message}") unless latest.ok diff --git a/app/mailers/admin_mailer.rb b/app/mailers/admin_mailer.rb index c624089..85f8a79 100644 --- a/app/mailers/admin_mailer.rb +++ b/app/mailers/admin_mailer.rb @@ -10,8 +10,9 @@ class AdminMailer < ApplicationMailer mail subject: run.kind == "restore_test" ? "Off-site restore test failed" : "Off-site backup failed", to: user.email_address end - def test_email(user) + def test_email(user, via: "from Admin → About") @user = user + @via = via mail subject: "Test email from Simple Aircraft Manager", to: user.email_address end end diff --git a/app/services/account_recovery.rb b/app/services/account_recovery.rb index 633df2d..dc1eb06 100644 --- a/app/services/account_recovery.rb +++ b/app/services/account_recovery.rb @@ -65,6 +65,14 @@ class AccountRecovery [ user, password ] end + # Ends every session of every user (suspected compromise): everyone signs + # in again, through the lockout and two-factor checks. + def self.sign_out_all! + count = Session.delete_all + Rails.logger.warn("[account recovery] all #{count} sessions ended from the command line") + count + end + def self.generate_password = SecureRandom.alphanumeric(20).scan(/.{5}/).join("-") def self.log(user, what) = Rails.logger.warn("[account recovery] #{user.email_address}: #{what} from the command line") diff --git a/app/services/database_restore.rb b/app/services/database_restore.rb new file mode 100644 index 0000000..fd3b0b7 --- /dev/null +++ b/app/services/database_restore.rb @@ -0,0 +1,79 @@ +require "zlib" + +# Puts a database backup in place of the live database (bin/rails +# backups:restore), with the app stopped: a nightly copy from storage/backups, +# or a copy downloaded from Google Drive (.sqlite3.gz). The copy is checked +# first, and the current database is moved aside, never deleted. +class DatabaseRestore + class Error < StandardError; end + + attr_reader :source, :target + + def self.newest_backup = Dir[DatabaseBackup::DIR.join("sam-*.sqlite3")].max_by { File.mtime(_1) } + + def self.live_database = File.expand_path(ActiveRecord::Base.connection_db_config.database.to_s, Rails.root) + + def initialize(source: nil, target: self.class.live_database, pidfile: Rails.root.join("tmp/pids/server.pid")) + @pidfile = Pathname(pidfile) + @source = source.presence || self.class.newest_backup or raise Error, "No backup found in #{DatabaseBackup::DIR}; give a file: backups:restore[path]" + @target = File.expand_path(target.to_s, Rails.root) + raise Error, "#{@source} doesn't exist" unless File.file?(@source) + end + + # Opens the copy (unzipping a .gz to a temporary file) and checks it. + # Returns a description; raises if it isn't a usable database. + def check + with_plain_copy { |path| describe(path) } + end + + # Swaps the copy in. Returns the folder the previous database was moved to. + def restore! + raise Error, "The app seems to be running (#{running_reason}). Stop it first; with Docker: docker compose stop web, then docker compose run --rm web bin/rails backups:restore" if running_reason + with_plain_copy do |path| + describe(path) + ActiveRecord::Base.connection_handler.clear_all_connections! if @target == self.class.live_database + aside = File.join(File.dirname(@target), "pre-restore-#{Time.current.utc.strftime('%Y%m%d-%H%M%S')}") + FileUtils.mkdir_p(aside) + [ "", "-wal", "-shm" ].each { |ext| FileUtils.mv("#{@target}#{ext}", aside) if File.exist?("#{@target}#{ext}") } + FileUtils.cp(path, @target) + Rails.logger.warn("[restore] #{@target} replaced by #{@source}; previous database in #{aside}") + aside + end + end + + private + + def with_plain_copy + return yield(@source) unless @source.end_with?(".gz") + Dir.mktmpdir("sam-restore") do |dir| + path = File.join(dir, "restore.sqlite3") + Zlib::GzipReader.open(@source) { |gz| File.open(path, "wb") { IO.copy_stream(gz, _1) } } + yield path + end + rescue Zlib::GzipFile::Error => e + raise Error, "#{@source} isn't a valid .gz file (#{e.message})" + end + + def describe(path) + db = SQLite3::Database.new(path, readonly: true) + result = db.get_first_value("PRAGMA integrity_check") + raise Error, "integrity check failed: #{result}" unless result == "ok" + counts = %w[aircraft logbook_entries entry_signatures users].to_h { [ _1, db.get_first_value("SELECT COUNT(*) FROM #{_1}") ] } + version = db.get_first_value("SELECT MAX(version) FROM schema_migrations") + "#{File.basename(@source)}: SQLite integrity check ok; #{counts['aircraft']} aircraft, #{counts['logbook_entries']} logbook entries, " \ + "#{counts['entry_signatures']} signatures, #{counts['users']} users; schema #{version}" + rescue SQLite3::Exception => e + raise Error, "#{File.basename(@source)} isn't a usable database (#{e.message})" + ensure + db&.close + end + + # A live server process (Puma's pid file) means the app is running. + def running_reason + return unless @pidfile.exist? + pid = @pidfile.read.to_i + Process.kill(0, pid) && "server process #{pid}" if pid.positive? + rescue Errno::ESRCH, Errno::EPERM + nil + end +end diff --git a/app/services/system_doctor.rb b/app/services/system_doctor.rb new file mode 100644 index 0000000..8a2d0b8 --- /dev/null +++ b/app/services/system_doctor.rb @@ -0,0 +1,117 @@ +require "net/http" +require "socket" +require "open3" + +# A health report for the installation (bin/rails sam:doctor): settings, +# connections to email, single sign-on and Google Drive, migrations, disk +# space, background jobs, and how recent the backups, checks and audit are. +class SystemDoctor + Check = Data.define(:status, :name, :detail) # status: :ok, :warn, :fail + + def checks + [ secret_key, host, https, email, sso, password_login, migrations, disk, jobs, backup, integrity, offsite, procedures, audit ].flatten.compact + end + + def self.report + doctor = new + list = doctor.checks + lines = list.map { |c| "#{{ ok: ' ok ', warn: ' WARN', fail: ' FAIL' }[c.status]} #{c.name.ljust(22)} #{c.detail}" } + [ "Simple Aircraft Manager #{SystemInfo.new.revision} (#{Rails.env})", *lines ].join("\n").then { [ _1, list ] } + end + + private + + def ok(name, detail) = Check.new(:ok, name, detail) + def warning(name, detail) = Check.new(:warn, name, detail) + def failed(name, detail) = Check.new(:fail, name, detail) + def production? = Rails.env.production? + + def secret_key + return ok("Secret key", "development/test key") unless production? + ENV["SECRET_KEY_BASE"].present? || ENV["RAILS_MASTER_KEY"].present? ? ok("Secret key", "set (keep a copy off the server)") : warning("Secret key", "SECRET_KEY_BASE isn't set in the environment") + end + + def host + url = Sso.app_url + production? && ENV["APP_HOST"].blank? && ENV["APP_URL"].blank? ? warning("Address", "APP_HOST/APP_URL not set: email links point at #{url}") : ok("Address", url) + end + + def https = Rails.application.config.force_ssl ? ok("HTTPS", "enforced (FORCE_SSL)") : warning("HTTPS", "not enforced by the app (FORCE_SSL=false): only behind a TLS proxy or on a private network") + + def email + return warning("Email", "not configured (set SMTP_ADDRESS): no password resets, sign-in links or alerts") unless MailSettings.configured? + s = ActionMailer::Base.smtp_settings + return ok("Email", "#{ActionMailer::Base.delivery_method} delivery") unless ActionMailer::Base.delivery_method == :smtp + Socket.tcp(s[:address], s[:port], connect_timeout: 5).close + ok("Email", "SMTP #{s[:address]}:#{s[:port]} reachable (send one with bin/rails \"email:test[you@example.com]\")") + rescue StandardError => e + failed("Email", "can't reach SMTP #{s[:address]}:#{s[:port]} (#{e.class.name.demodulize})") + end + + def sso + return unless Sso.enabled? + uri = URI("#{ENV['OIDC_ISSUER'].to_s.chomp('/')}/.well-known/openid-configuration") + res = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https", open_timeout: 5, read_timeout: 5) { _1.get(uri.request_uri) } + res.is_a?(Net::HTTPSuccess) ? ok("Single sign-on", "#{Sso.label} issuer reachable") : failed("Single sign-on", "#{uri} answered #{res.code}") + rescue StandardError => e + failed("Single sign-on", "can't reach #{Sso.label} (#{e.class.name.demodulize}: #{e.message.truncate(80)})") + end + + def password_login + return ok("Password sign-in", "on") if Sso.password_login? + ok("Password sign-in", "off (single sign-on only; invited mechanics can't create accounts)") + end + + def migrations + pending = ActiveRecord::Base.connection_pool.migration_context.needs_migration? + pending ? failed("Database", "migrations pending: run bin/rails db:migrate (or restart the container)") : ok("Database", "schema up to date") + end + + def disk + out = Open3.capture2("df", "-Pk", Rails.root.join("storage").to_s).first.lines.last.to_s.split + return warning("Disk space", "couldn't check") if out.size < 5 + free_gb = out[3].to_f / 1024 / 1024 + used = out[4].to_i + detail = "#{free_gb.round(1)} GB free on the storage volume (#{used}% used)" + free_gb < 1 || used > 95 ? failed("Disk space", detail) : (free_gb < 5 || used > 85 ? warning("Disk space", detail) : ok("Disk space", detail)) + end + + def jobs + return unless defined?(SolidQueue::Process) + count = SolidQueue::Process.where(last_heartbeat_at: 5.minutes.ago..).count + count.positive? ? ok("Background jobs", "#{count} running (nightly backup, reminders, AD search)") : warning("Background jobs", "none seen in the last 5 minutes: is the app running with SOLID_QUEUE_IN_PUMA?") + rescue StandardError => e + warning("Background jobs", "couldn't check (#{e.class.name.demodulize})") + end + + def backup = age_check("Last backup", BackupRun.order(:created_at).last) { "#{_1.ok ? 'ok' : 'FAILED'}: #{_1.message.to_s.truncate(70)}" } + def integrity = age_check("Records check", IntegrityCheck.order(:created_at).last) { _1.ok ? "no problems" : "#{Array(_1.problems).size} problem(s)" } + + def offsite + return warning("Off-site backup", "Google Drive not connected (Admin → Off-site Backup), or describe another off-site copy in the procedures") unless OffsiteConnection.connected? + OffsiteConnection.current.client.account_email + run = OffsiteRun.latest_backup + run ? age_check("Off-site backup", run) { "#{_1.ok ? 'ok' : 'FAILED'}: Google Drive #{OffsiteConnection.current.account_email}" } : warning("Off-site backup", "Google Drive connected; no copy yet") + rescue GoogleDriveClient::Error => e + failed("Off-site backup", "Google Drive: #{e.message.truncate(90)}") + end + + def procedures + r = ProceduresRevision.current + r ? ok("Procedures", "revision #{r.number}, adopted #{r.adopted_at.to_date}") : warning("Procedures", "not adopted (Admin → Recordkeeping Procedures)") + end + + def audit + return warning("Records audit", "never done (Admin → Records Audits)") unless AuditRun.latest + due = AuditRun.next_due + detail = "next due #{due} (last #{AuditRun.latest.performed_at.to_date})" + due < Date.current ? warning("Records audit", "overdue: #{detail}") : ok("Records audit", detail) + end + + def age_check(name, record) + return warning(name, "never run") unless record + detail = "#{record.created_at.to_date}, #{yield(record)}" + return failed(name, detail) unless record.ok + record.created_at < 2.days.ago ? warning(name, "#{detail} (more than 2 days ago)") : ok(name, detail) + end +end diff --git a/app/views/admin_mailer/test_email.html.erb b/app/views/admin_mailer/test_email.html.erb index 7730fbd..cd4a821 100644 --- a/app/views/admin_mailer/test_email.html.erb +++ b/app/views/admin_mailer/test_email.html.erb @@ -1,2 +1,2 @@ -
This is a test email from Simple Aircraft Manager, sent by <%= @user.email_address %> from Admin → About.
+This is a test email from Simple Aircraft Manager, sent by <%= @user.email_address %> <%= @via %>.
Email is working: password resets and sign-in links will arrive.
diff --git a/app/views/admin_mailer/test_email.text.erb b/app/views/admin_mailer/test_email.text.erb index 989945b..beab18f 100644 --- a/app/views/admin_mailer/test_email.text.erb +++ b/app/views/admin_mailer/test_email.text.erb @@ -1,3 +1,3 @@ -This is a test email from Simple Aircraft Manager, sent by <%= @user.email_address %> from Admin → About. +This is a test email from Simple Aircraft Manager, sent by <%= @user.email_address %> <%= @via %>. Email is working: password resets and sign-in links will arrive. diff --git a/docs/guide/administration.md b/docs/guide/administration.md index 2717077..d71a98d 100644 --- a/docs/guide/administration.md +++ b/docs/guide/administration.md @@ -88,10 +88,28 @@ The first backup starts straight away. It copies every uploaded file, so it can If the server is lost: 1. Set up the app again (see the main README) with the **same `SECRET_KEY_BASE`**, but don't start it yet. -2. From Google Drive, download the newest file in **Simple Aircraft Manager backups → database**, unzip it (`gunzip sam-….sqlite3.gz`), and put it in place of `storage/production.sqlite3` (delete any `production.sqlite3-wal` and `-shm` files next to it). -3. Start the app and sign in. The records are back, but uploaded files are still missing. -4. Restore the files: `bin/rails offsite:restore_files` downloads them from Drive (the connection is in the restored database). If that doesn't work (for example the `SECRET_KEY_BASE` is lost), download the **files** folder from Drive, unzip it on the server and run `bin/rails "offsite:restore_files[/path/to/files]"`. -5. Check the About page (run **Back up and check now**) and open a few entries and documents. +2. From Google Drive, download the newest file in **Simple Aircraft Manager backups → database** (`sam-….sqlite3.gz`) and copy it onto the server. +3. Put it in place: `bin/rails "backups:restore[/path/to/sam-….sqlite3.gz]"`. With Docker, run it in a one-off container while the app is stopped: `docker compose run --rm web bin/rails "backups:restore[/rails/storage/sam-….sqlite3.gz]"` (copy the file into the storage volume first). It checks the copy, keeps any existing database aside, and asks before replacing. +4. Start the app and sign in. The records are back, but uploaded files are still missing. +5. Restore the files: `bin/rails offsite:restore_files` downloads them from Drive (the connection is in the restored database). If that doesn't work (for example the `SECRET_KEY_BASE` is lost), download the **files** folder from Drive, unzip it on the server and run `bin/rails "offsite:restore_files[/path/to/files]"`. +6. Check everything: `bin/rails records:check` (history and signatures intact) and `bin/rails sam:doctor`, then open a few entries and documents. + +The same `backups:restore` (with no file) puts back the newest nightly copy from `storage/backups`, for example after a mistake that can't be undone in the app; `bin/rails backups:list` shows the copies. + +## Maintenance commands + +Run on the server like the account recovery commands (`docker compose exec web` in front, or the container's Console in Portainer). + +| Command | What it does | +|---|---| +| `bin/rails sam:doctor` | Health report: secret key, address, HTTPS, email (SMTP reachable), single sign-on (issuer reachable), Google Drive, migrations, disk space, background jobs, and how recent the backup, records check, off-site copy, procedures and audit are. Exits non-zero if something fails | +| `bin/rails records:check` | Backs up the database now and checks the record history and every signature; prints any problems and exits non-zero | +| `bin/rails backups:list` | Lists the nightly database copies | +| `bin/rails "backups:restore[file]"` | Replaces the database with a backup (newest nightly copy if no file; a Google Drive `.sqlite3.gz` works too). The app must be stopped: `docker compose stop web`, then `docker compose run --rm web bin/rails backups:restore`. The current database is moved to `storage/pre-restore-…`, never deleted. `CONFIRM=yes` skips the question | +| `bin/rails users:sign_out_all` | Signs every user out of every device, if you think an account or session was compromised | +| `bin/rails "sam:records_package[N12345]"` | Writes the aircraft's records package to a zip, for when the web side is down | +| `bin/rails "email:test[you@example.com]"` | Sends a test email and shows the mail settings | +| `bin/rails offsite:backup` / `offsite:restore_test` | Runs the Google Drive copy or a restore test now and prints the result | ## Recordkeeping procedures diff --git a/lib/tasks/maintenance.rake b/lib/tasks/maintenance.rake new file mode 100644 index 0000000..117a9cf --- /dev/null +++ b/lib/tasks/maintenance.rake @@ -0,0 +1,99 @@ +# Maintenance and recovery commands. In Docker: docker compose exec web bin/rails