Bulk document upload and deduplicated file storage

- Bulk Upload page: pick many files or a folder; each file becomes a
  document named after it, and files sharing a name become one multi-page
  document. Files go up as direct uploads with per-file progress.
- Direct uploads require a signed-in user and check type/size up front;
  only fresh, unattached blobs can be attached; abandoned uploads are
  purged nightly.
- DedupDisk storage service: each distinct file is stored once under its
  SHA-256 and blobs hard-link to it; nightly sweep (and storage:dedupe)
  links pre-existing duplicates and drops unused objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rickyandClaude Opus 5.5 committed 2026-11-07 08:20:22 -05:00
1 parent 230a569b77
commit 2bfcb845ca
15 files changed
+735 -5

No files matched your search

+10
View File
@@ -0,0 +1,10 @@
namespace :storage do
desc "Store each distinct uploaded file once (hard links) and drop unused copies"
task dedupe: :environment do
service = ActiveStorage::Blob.service
abort "The #{service.class.name.demodulize} service doesn't deduplicate." unless service.respond_to?(:sweep)
result = service.sweep
puts "Linked #{result[:linked]} duplicate files, removed #{result[:removed]} unused copies; " \
"#{ActiveSupport::NumberHelper.number_to_human_size(result[:saved_bytes])} saved by sharing."
end
end