Bulk document upload and deduplicated file storage

- Bulk Upload page: pick many files or a folder; each file becomes a
  document named after it, and files sharing a name become one multi-page
  document. Files go up as direct uploads with per-file progress.
- Direct uploads require a signed-in user and check type/size up front;
  only fresh, unattached blobs can be attached; abandoned uploads are
  purged nightly.
- DedupDisk storage service: each distinct file is stored once under its
  SHA-256 and blobs hard-link to it; nightly sweep (and storage:dedupe)
  links pre-existing duplicates and drops unused objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rickyandClaude Opus 5.5 committed 2026-11-07 08:20:22 -05:00
1 parent 230a569b77
commit 2bfcb845ca
15 files changed
+735 -5

No files matched your search

@@ -0,0 +1,25 @@
# Direct uploads (the browser sends each file straight to storage, then submits
# only its signed id). Active Storage's own endpoint lets anyone create blobs;
# this one requires a signed-in user and applies the upload rules up front.
class DirectUploadsController < ActiveStorage::DirectUploadsController
include Authentication
before_action :check_upload, only: :create
private
def blob_args
@blob_args ||= super.tap do |args|
args[:content_type] = Marcel::MimeType.for(name: args[:filename].to_s) if args[:content_type].blank?
end
end
def check_upload
error =
if !UploadValidation::ALLOWED_CONTENT_TYPES.include?(blob_args[:content_type])
"#{blob_args[:filename]} isn't an allowed file type"
elsif blob_args[:byte_size].to_i > UploadValidation::MAX_SIZE
"#{blob_args[:filename]} is over #{UploadValidation::MAX_SIZE / 1.megabyte} MB"
end
render json: { error: error }, status: :unprocessable_entity if error
end
end