v2.0.0: merge dev into main

CHANGELOG.md

@@ -25,6 +25,40 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 ### Security
 -->
 
+## [v2.0.0] - Shift to saxonche, improve error handling, refactor Omeka S posting, add post options
+[v2.0.0]: https://github.com/CDRH/datura/compare/v1.1.0...v2.0.0
+
+### Added
+- `--csv-rows` filter option added to enable line-by-line transformation of CSV data
+- `--json-output` option (Omeka specific) added to output JSON instead of posting to Omeka S
+- `--proceed` option added to enable restarting of post from a checkpoint
+
+### Changed
+- Omeka S posting pipeline refactored to, among other things: 
+  - introduce `OmekaContext` dataclass to encapsulate shared runtime state into a single object
+  - restructure overrides and update example files
+  - add filter from Ruby side (`--update`)
+  - remove all output JSON as part of a complete post to Omeka S
+  - adjust params for Omeka S API calls (see [`omeka_s_tools` fork](https://github.com/CDRH/omeka_s_tools))
+  - update dependency list and create `requirements.in` file from which `requirements.txt` can be auto-generated
+  - improve logging and error handling
+  - fix bugs
+- Updated Ruby and Python dependencies where possible, removed unused/deprecated dependencies
+- Improved error handling: clarify XML/CSV parse errors and environment errors, show all errors at end of post
+- Created option and fall-back to `full` for iiif URL creation
+
+### Removed
+- Removed `RestClient` in favor of built-in `net-http` library
+
+### Migration
+- Shifted to use Python `saxonche` package instead of Saxon calls to eliminate per-file JVM startup
+
+### Security
+- `safe_load` added for YAML
+- SSL verification and regex validation added
+
+Note that several of the above changes and some of the language and description in the documentation were created with Claude Code. All code and documentation has been reviewed and updated by CDRH staff. 
+
 ## [v1.1.0] - Omeka S Posting
 [v1.1.0]: https://github.com/CDRH/datura/compare/v1.0.1...v1.1.0
 

Gemfile.lock

@@ -2,9 +2,8 @@ PATH
   remote: .
   specs:
     datura (1.1.0)
-      byebug (~> 11.0)
-      colorize (~> 0.8.1)
-      nokogiri (~> 1.10)
+      colorize (~> 1.0)
+      nokogiri (~> 1.18)
       pdf-reader (~> 2.12)
       rest-client (~> 2.1)
 
@@ -15,7 +14,7 @@ GEM
     afm (0.2.2)
     bigdecimal (3.3.1)
     byebug (11.1.3)
-    colorize (0.8.1)
+    colorize (1.1.0)
     domain_name (0.6.20240107)
     hashery (2.1.2)
     http-accept (1.7.0)
@@ -26,7 +25,6 @@ GEM
       logger
       mime-types-data (~> 3.2025, >= 3.2025.0507)
     mime-types-data (3.2026.0414)
-    mini_portile2 (2.8.9)
     minitest (5.27.0)
     netrc (0.11.0)
     nokogiri (1.18.10)
@@ -56,7 +54,7 @@ PLATFORMS
   x86_64-darwin-20
 
 DEPENDENCIES
-  bundler (>= 1.16.0, < 3.0)
+  bundler (>= 2.0, < 5.0)
   datura!
   minitest (~> 5.0)
   rake (~> 13.0)

README.md

@@ -18,9 +18,7 @@ source "https://rubygems.org"
 gem "datura", git: "https://github.com/CDRH/datura.git", tag: "v0.0.0"
 ```
 
-If this is the first datura repository on your machine, install saxon as a system wide executable. [Saxon setup documentation](docs/4_developers/saxon.md).
-
-Then, in the directory with the Gemfile, run the following:
+After that, in the directory with the Gemfile, run the following:
 
 ```
 gem install bundler
@@ -29,7 +27,10 @@ bundle install
 bundle exec setup
 ```
 
-The last step should add files and some basic directories. Have a look at the [setup instructions](/docs/1_setup/collection_setup.md) to learn how to add your files and start working with the data!
+The setup step should add files and some basic directories. If this is the first datura repository on your machine, you will need to install Python 3. You will also need to install `saxonche`. `saxonche` is included in `requirements.txt`, which should have been added as part of setup, so you can install it by following the Omeka setup [instructions](/docs/1_setup/omeka_setup.md#enabling-a-virtual-environment) for installing Python dependencies. 
+
+Now you are ready to have a look at the [setup instructions](/docs/1_setup/collection_setup.md) to learn how to add your files and start working with the data!
+
 
 ### RVM
 

bin/post

@@ -2,5 +2,10 @@
 
 require "datura"
 
-manager = Datura::DataManager.new
-manager.run
+begin
+  manager = Datura::DataManager.new
+  manager.run
+rescue => e
+  puts e.message.red
+  exit 1
+end

bin/post_omeka

@@ -1,9 +1,7 @@
 #!/usr/bin/env ruby
 
 require "datura"
-require "byebug"
 require "optparse"
-require "shellwords"
 
 @usage = "Usage: post_omeka -[options]..."
 
@@ -13,48 +11,133 @@ options = {}
 optparse = OptionParser.new do |opts|
     # Set a banner
     opts.banner = @usage
-    opts.on('-s', '--skip', 'skip generation step and just post to Omeka') do
-        generate_es = false
+    opts.on('-c', '--csv-rows [input]', 'only process CSV rows matching this identifier regex') do |input|
+        if input && input.length > 0
+            options["csv_rows"] = input
+        end
     end
 
-    opts.on('-e', '--environment [input]', 'environment (development, production)') do |input|
+    opts.on('-e', '--environment [input]', 'environment (default: development, or production)') do |input|
         if input && input.length > 0
             options["environment"] = input
         end
     end
+
+    opts.on('-f', '--format [input]', 'only post files of this format (tei, csv, vra, ead, html, pdf, webs)') do |input|
+        if input && input.length > 0
+            options["format"] = input
+        end
+    end
+
+    opts.on('-j', '--json-output', 'write Omeka S item payloads to output/<environment>/omeka/ instead of posting to the API (API connection still required for property ID lookups)') do
+        options["json_output"] = true
+    end
+
+    opts.on('-p', '--proceed [input]', 'proceed from the file matching this regex, or from the last checkpoint if no value given') do |input|
+        options["proceed"] = input
+    end
 
     opts.on('-r', '--regex [input]', 'only generate and post files matching this regex') do |input|
         if input && input.length > 0
             options["regex"] = input
         end
     end
 
+    opts.on('-s', '--skip', 'skip generation step and just post to Omeka') do
+        generate_es = false
+    end
+
+    opts.on('-u', '--update [input]', 'only post files updated after this date (today, 2015-01-01, or 2015-01-01T18:24)') do |input|
+        if input && input.length > 0
+            options["update_time"] = input
+        end
+    end
+
     opts.on('-[!-~]', '-[!-~] [input]', 'arguments for the main Datura script') do |name, value|
         #does nothing, this is a placeholder to catch all arguments not needed now and pass them along to the main script
     end
 end
 #parse, but do not consume, command line arguments (the usual parse! would consume them)
 optparse.parse(ARGV)
+Datura::Helpers.validate_regex(options["regex"], "--regex") if options["regex"]
+Datura::Helpers.validate_regex(options["csv_rows"], "--csv-rows") if options["csv_rows"]
+#remove options not used in the main script
+ARGV.delete("-s")
+ARGV.delete("--skip")
+#-p checkpointing belongs to the Omeka posting step (Python); strip it so
+#DataManager does not trigger the Ruby proceed/checkpoint logic instead
+ARGV.delete("-p")
+ARGV.delete("--proceed")
+ARGV.reject! { |arg| arg.start_with?("--proceed=") || (arg.start_with?("-p") && arg.length > 2) }
+#if -p was given, resolve the proceed value from the Python checkpoint and
+#re-inject into ARGV so DataManager also starts from the same file, preventing
+#all source files from being transformed unnecessarily
+if options.key?("proceed")
+  proceed_value = options["proceed"]
+  if proceed_value.nil?
+    env = options["environment"] || "development"
+    checkpoint_file = File.join("logs", "proceed_omeka_#{env}")
+    if File.exist?(checkpoint_file)
+      last = File.read(checkpoint_file).strip
+      unless last.empty?
+        print "Continue from #{last}? (y/n): "
+        $stdout.flush
+        response = $stdin.gets&.chomp&.downcase || ""
+        if response == "y"
+          proceed_value = last
+        else
+          puts "Exiting."
+          exit 0
+        end
+      end
+    end
+    if proceed_value.nil?
+      puts "ERROR: --proceed given with no value but no checkpoint file found at #{checkpoint_file}"
+      exit 1
+    end
+  end
+  ARGV.push("--proceed=#{proceed_value}")
+  options["proceed"] = proceed_value
+end
+ARGV.delete("-j")
+ARGV.delete("--json-output")
+#if -c/--csv-rows is set and no format was specified, restrict generation to CSV only
+if options["csv_rows"] && !options["format"] && !ARGV.any? { |a| a == "-f" || a == "--format" || a.start_with?("--format=") }
+  ARGV.push("-f", "csv")
+end
 #add options to output a json file instead of posting it to Elasticsearch
 ARGV.unshift("-x", "es", "-o", "-t")
-#skip generation step with option -s
+#create DataManager before conditional run
+manager = Datura::DataManager.new
+# On full runs (no limiting flags), clear all ES output before transforms so
+# that deleted source files do not persist in the output directory. This makes
+# the output directory a faithful reflection of the current source tree.
+# -s (skip transforms) is supported for limited runs only; combining -s with
+# an unrestricted full run would clear the output and leave nothing to post.
+is_limited_run = options["regex"] || options["format"] || options["proceed"] || options["csv_rows"]
+if generate_es && !is_limited_run
+  env = options.fetch("environment", "development")
+  out_es = File.join(Dir.pwd, "output", env, "es")
+  if Dir.exist?(out_es)
+    files = Dir.glob("#{out_es}/*.json")
+    cleared = 0
+
+    files.each do |f|
+        begin
+            File.delete(f)
+            cleared += 1
+        rescue => e
+            puts "There was an error deleting a file: #{e.message}".red
+        end
+    end
+    puts "Cleared #{cleared} file(s) from output/#{env}/es"
+  end
+end
 if generate_es
-    manager = Datura::DataManager.new
     manager.run
 end
 datura_dir = File.join(File.dirname(__FILE__), "..")
 # path to the gem's config files
 python_script_path = File.join(datura_dir, "lib", "datura", "python", "json_to_omeka.py")
 #run posting script into Omeka S
-if File.exist?("#{python_script_path}")
-    command = ["python3", python_script_path]
-    if options["environment"]
-        command.append("-e", Shellwords.escape(options["environment"]))
-    end
-    if options["regex"]
-        command.append("-r", Shellwords.escape(options["regex"]))
-    end
-    system(*command)
-else
-    puts("Omeka script not found at #{python_script_path}".red)
-end
+Datura::Helpers.run_omeka_script(python_script_path, options)