diff --git a/CHANGELOG.md b/CHANGELOG.md index b12b25c3d..1437c0a4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,40 @@ Versioning](https://semver.org/spec/v2.0.0.html). ### Security --> +## [v2.0.0] - Shift to saxonche, improve error handling, refactor Omeka S posting, add post options +[v2.0.0]: https://github.com/CDRH/datura/compare/v1.1.0...v2.0.0 + +### Added +- `--csv-rows` filter option added to enable line-by-line transformation of CSV data +- `--json-output` option (Omeka specific) added to output JSON instead of posting to Omeka S +- `--proceed` option added to enable restarting of post from a checkpoint + +### Changed +- Omeka S posting pipeline refactored to, among other things: + - introduce `OmekaContext` dataclass to encapsulate shared runtime state into a single object + - restructure overrides and update example files + - add filter from Ruby side (`--update`) + - remove all output JSON as part of a complete post to Omeka S + - adjust params for Omeka S API calls (see [`omeka_s_tools` fork](https://github.com/CDRH/omeka_s_tools)) + - update dependency list and create `requirements.in` file from which `requirements.txt` can be auto-generated + - improve logging and error handling + - fix bugs +- Updated Ruby and Python dependencies where possible, removed unused/deprecated dependencies +- Improved error handling: clarify XML/CSV parse errors and environment errors, show all errors at end of post +- Created option and fall-back to `full` for iiif URL creation + +### Removed +- Removed `RestClient` in favor of built-in `net-http` library + +### Migration +- Shifted to use Python `saxonche` package instead of Saxon calls to eliminate per-file JVM startup + +### Security +- `safe_load` added for YAML +- SSL verification and regex validation added + +Note that several of the above changes and some of the language and description in the documentation were created with Claude Code. All code and documentation has been reviewed and updated by CDRH staff. + ## [v1.1.0] - Omeka S Posting [v1.1.0]: https://github.com/CDRH/datura/compare/v1.0.1...v1.1.0 diff --git a/Gemfile.lock b/Gemfile.lock index 8ef855c6e..94a602d61 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -2,9 +2,8 @@ PATH remote: . specs: datura (1.1.0) - byebug (~> 11.0) - colorize (~> 0.8.1) - nokogiri (~> 1.10) + colorize (~> 1.0) + nokogiri (~> 1.18) pdf-reader (~> 2.12) rest-client (~> 2.1) @@ -15,7 +14,7 @@ GEM afm (0.2.2) bigdecimal (3.3.1) byebug (11.1.3) - colorize (0.8.1) + colorize (1.1.0) domain_name (0.6.20240107) hashery (2.1.2) http-accept (1.7.0) @@ -26,7 +25,6 @@ GEM logger mime-types-data (~> 3.2025, >= 3.2025.0507) mime-types-data (3.2026.0414) - mini_portile2 (2.8.9) minitest (5.27.0) netrc (0.11.0) nokogiri (1.18.10) @@ -56,7 +54,7 @@ PLATFORMS x86_64-darwin-20 DEPENDENCIES - bundler (>= 1.16.0, < 3.0) + bundler (>= 2.0, < 5.0) datura! minitest (~> 5.0) rake (~> 13.0) diff --git a/README.md b/README.md index 550142d6f..436fc1f3c 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,7 @@ source "https://rubygems.org" gem "datura", git: "https://github.com/CDRH/datura.git", tag: "v0.0.0" ``` -If this is the first datura repository on your machine, install saxon as a system wide executable. [Saxon setup documentation](docs/4_developers/saxon.md). - -Then, in the directory with the Gemfile, run the following: +After that, in the directory with the Gemfile, run the following: ``` gem install bundler @@ -29,7 +27,10 @@ bundle install bundle exec setup ``` -The last step should add files and some basic directories. Have a look at the [setup instructions](/docs/1_setup/collection_setup.md) to learn how to add your files and start working with the data! +The setup step should add files and some basic directories. If this is the first datura repository on your machine, you will need to install Python 3. You will also need to install `saxonche`. `saxonche` is included in `requirements.txt`, which should have been added as part of setup, so you can install it by following the Omeka setup [instructions](/docs/1_setup/omeka_setup.md#enabling-a-virtual-environment) for installing Python dependencies. + +Now you are ready to have a look at the [setup instructions](/docs/1_setup/collection_setup.md) to learn how to add your files and start working with the data! + ### RVM diff --git a/bin/post b/bin/post index 026b183f8..b0ec665be 100755 --- a/bin/post +++ b/bin/post @@ -2,5 +2,10 @@ require "datura" -manager = Datura::DataManager.new -manager.run +begin + manager = Datura::DataManager.new + manager.run +rescue => e + puts e.message.red + exit 1 +end \ No newline at end of file diff --git a/bin/post_omeka b/bin/post_omeka index bba61370a..23ffacc34 100755 --- a/bin/post_omeka +++ b/bin/post_omeka @@ -1,9 +1,7 @@ #!/usr/bin/env ruby require "datura" -require "byebug" require "optparse" -require "shellwords" @usage = "Usage: post_omeka -[options]..." @@ -13,15 +11,31 @@ options = {} optparse = OptionParser.new do |opts| # Set a banner opts.banner = @usage - opts.on('-s', '--skip', 'skip generation step and just post to Omeka') do - generate_es = false + opts.on('-c', '--csv-rows [input]', 'only process CSV rows matching this identifier regex') do |input| + if input && input.length > 0 + options["csv_rows"] = input + end end - opts.on('-e', '--environment [input]', 'environment (development, production)') do |input| + opts.on('-e', '--environment [input]', 'environment (default: development, or production)') do |input| if input && input.length > 0 options["environment"] = input end end + + opts.on('-f', '--format [input]', 'only post files of this format (tei, csv, vra, ead, html, pdf, webs)') do |input| + if input && input.length > 0 + options["format"] = input + end + end + + opts.on('-j', '--json-output', 'write Omeka S item payloads to output//omeka/ instead of posting to the API (API connection still required for property ID lookups)') do + options["json_output"] = true + end + + opts.on('-p', '--proceed [input]', 'proceed from the file matching this regex, or from the last checkpoint if no value given') do |input| + options["proceed"] = input + end opts.on('-r', '--regex [input]', 'only generate and post files matching this regex') do |input| if input && input.length > 0 @@ -29,32 +43,101 @@ optparse = OptionParser.new do |opts| end end + opts.on('-s', '--skip', 'skip generation step and just post to Omeka') do + generate_es = false + end + + opts.on('-u', '--update [input]', 'only post files updated after this date (today, 2015-01-01, or 2015-01-01T18:24)') do |input| + if input && input.length > 0 + options["update_time"] = input + end + end + opts.on('-[!-~]', '-[!-~] [input]', 'arguments for the main Datura script') do |name, value| #does nothing, this is a placeholder to catch all arguments not needed now and pass them along to the main script end end #parse, but do not consume, command line arguments (the usual parse! would consume them) optparse.parse(ARGV) +Datura::Helpers.validate_regex(options["regex"], "--regex") if options["regex"] +Datura::Helpers.validate_regex(options["csv_rows"], "--csv-rows") if options["csv_rows"] +#remove options not used in the main script +ARGV.delete("-s") +ARGV.delete("--skip") +#-p checkpointing belongs to the Omeka posting step (Python); strip it so +#DataManager does not trigger the Ruby proceed/checkpoint logic instead +ARGV.delete("-p") +ARGV.delete("--proceed") +ARGV.reject! { |arg| arg.start_with?("--proceed=") || (arg.start_with?("-p") && arg.length > 2) } +#if -p was given, resolve the proceed value from the Python checkpoint and +#re-inject into ARGV so DataManager also starts from the same file, preventing +#all source files from being transformed unnecessarily +if options.key?("proceed") + proceed_value = options["proceed"] + if proceed_value.nil? + env = options["environment"] || "development" + checkpoint_file = File.join("logs", "proceed_omeka_#{env}") + if File.exist?(checkpoint_file) + last = File.read(checkpoint_file).strip + unless last.empty? + print "Continue from #{last}? (y/n): " + $stdout.flush + response = $stdin.gets&.chomp&.downcase || "" + if response == "y" + proceed_value = last + else + puts "Exiting." + exit 0 + end + end + end + if proceed_value.nil? + puts "ERROR: --proceed given with no value but no checkpoint file found at #{checkpoint_file}" + exit 1 + end + end + ARGV.push("--proceed=#{proceed_value}") + options["proceed"] = proceed_value +end +ARGV.delete("-j") +ARGV.delete("--json-output") +#if -c/--csv-rows is set and no format was specified, restrict generation to CSV only +if options["csv_rows"] && !options["format"] && !ARGV.any? { |a| a == "-f" || a == "--format" || a.start_with?("--format=") } + ARGV.push("-f", "csv") +end #add options to output a json file instead of posting it to Elasticsearch ARGV.unshift("-x", "es", "-o", "-t") -#skip generation step with option -s +#create DataManager before conditional run +manager = Datura::DataManager.new +# On full runs (no limiting flags), clear all ES output before transforms so +# that deleted source files do not persist in the output directory. This makes +# the output directory a faithful reflection of the current source tree. +# -s (skip transforms) is supported for limited runs only; combining -s with +# an unrestricted full run would clear the output and leave nothing to post. +is_limited_run = options["regex"] || options["format"] || options["proceed"] || options["csv_rows"] +if generate_es && !is_limited_run + env = options.fetch("environment", "development") + out_es = File.join(Dir.pwd, "output", env, "es") + if Dir.exist?(out_es) + files = Dir.glob("#{out_es}/*.json") + cleared = 0 + + files.each do |f| + begin + File.delete(f) + cleared += 1 + rescue => e + puts "There was an error deleting a file: #{e.message}".red + end + end + puts "Cleared #{cleared} file(s) from output/#{env}/es" + end +end if generate_es - manager = Datura::DataManager.new manager.run end datura_dir = File.join(File.dirname(__FILE__), "..") # path to the gem's config files python_script_path = File.join(datura_dir, "lib", "datura", "python", "json_to_omeka.py") #run posting script into Omeka S -if File.exist?("#{python_script_path}") - command = ["python3", python_script_path] - if options["environment"] - command.append("-e", Shellwords.escape(options["environment"])) - end - if options["regex"] - command.append("-r", Shellwords.escape(options["regex"])) - end - system(*command) -else - puts("Omeka script not found at #{python_script_path}".red) -end \ No newline at end of file +Datura::Helpers.run_omeka_script(python_script_path, options) diff --git a/bin/post_omeka_html b/bin/post_omeka_html index 27f3a3522..e01dbd6cd 100755 --- a/bin/post_omeka_html +++ b/bin/post_omeka_html @@ -1,26 +1,36 @@ #!/usr/bin/env ruby require "datura" -require "byebug" -require "shellwords" generate_es = true options = {} optparse = OptionParser.new do |opts| # Set a banner opts.banner = @usage - opts.on('-s', '--skip', 'skip generation step and just post to Omeka') do - generate_es = false + opts.on('-c', '--csv-rows [input]', 'only process CSV rows matching this identifier regex') do |input| + if input && input.length > 0 + options["csv_rows"] = input + end + end + + opts.on('-e', '--environment [input]', 'environment (default: development, or production)') do |input| + if input && input.length > 0 + options["environment"] = input + end + end + + opts.on('-f', '--format [input]', 'only post files of this format (tei, csv, vra, ead, html, pdf, webs)') do |input| + if input && input.length > 0 + options["format"] = input + end end opts.on('-m', '--media_skip', 'skip deleting and regenerating media') do options["media_skip"] = true end - opts.on('-e', '--environment [input]', 'environment (development, production)') do |input| - if input && input.length > 0 - options["environment"] = input - end + opts.on('-p', '--proceed [input]', 'proceed from the file matching this regex, or from the last checkpoint if no value given') do |input| + options["proceed"] = input end opts.on('-r', '--regex [input]', 'only generate and post files matching this regex') do |input| @@ -29,19 +39,74 @@ optparse = OptionParser.new do |opts| end end + opts.on('-s', '--skip', 'skip generation step and just post to Omeka') do + generate_es = false + end + + opts.on('-u', '--update [input]', 'only post files updated after this date (today, 2015-01-01, or 2015-01-01T18:24)') do |input| + if input && input.length > 0 + options["update_time"] = input + end + end + opts.on('-[!-~]', '-[!-~] [input]', 'arguments for the main Datura script') do |name, value| #does nothing, this is a placeholder to catch all arguments not needed now and pass them along to the main script end end #parse, but do not consume, command line arguments (the usual parse! would consume them) optparse.parse(ARGV) -#remove option not used in the main script +Datura::Helpers.validate_regex(options["regex"], "--regex") if options["regex"] +Datura::Helpers.validate_regex(options["csv_rows"], "--csv-rows") if options["csv_rows"] +#remove options not used in the main script ARGV.delete("-m") +ARGV.delete("--media_skip") +ARGV.delete("-s") +ARGV.delete("--skip") +#-p checkpointing belongs to the Omeka posting step (Python); strip it so +#DataManager does not trigger the Ruby proceed/checkpoint logic instead +ARGV.delete("-p") +ARGV.delete("--proceed") +ARGV.reject! { |arg| arg.start_with?("--proceed=") || (arg.start_with?("-p") && arg.length > 2) } +#if -p was given, resolve the proceed value from the Python checkpoint and +#re-inject into ARGV so DataManager also starts from the same file, preventing +#all source files (including CSVs) from being transformed unnecessarily +if options.key?("proceed") + proceed_value = options["proceed"] + if proceed_value.nil? + env = options["environment"] || "development" + checkpoint_file = File.join("logs", "proceed_omeka_html_#{env}") + if File.exist?(checkpoint_file) + last = File.read(checkpoint_file).strip + unless last.empty? + print "Continue from #{last}? (y/n): " + $stdout.flush + response = $stdin.gets&.chomp&.downcase || "" + if response == "y" + proceed_value = last + else + puts "Exiting." + exit 0 + end + end + end + if proceed_value.nil? + puts "ERROR: --proceed given with no value but no checkpoint file found at #{checkpoint_file}" + exit 1 + end + end + ARGV.push("--proceed=#{proceed_value}") + options["proceed"] = proceed_value +end +#if -c/--csv-rows is set and no format was specified, restrict generation to CSV only +if options["csv_rows"] && !options["format"] && !ARGV.any? { |a| a == "-f" || a == "--format" || a.start_with?("--format=") } + ARGV.push("-f", "csv") +end #add option to generate html ARGV.unshift("-x", "html") +#create DataManager before conditional run +manager = Datura::DataManager.new #skip generation step with option -s if generate_es - manager = Datura::DataManager.new manager.run end @@ -49,18 +114,4 @@ datura_dir = File.join(File.dirname(__FILE__), "..") # path to the gem's config files python_script_path = File.join(datura_dir, "lib", "datura", "python", "html_and_media_ingest.py") #run posting script into Omeka S -if File.exist?("#{python_script_path}") - command = ["python3", python_script_path] - if options["environment"] - command.append("-e", Shellwords.escape(options["environment"])) - end - if options["regex"] - command.append("-r", Shellwords.escape(options["regex"])) - end - if options["media_skip"] - command.append("-m") - end - system(*command) -else - puts("Omeka script not found at #{python_script_path}".red) -end \ No newline at end of file +Datura::Helpers.run_omeka_script(python_script_path, options) diff --git a/bin/setup b/bin/setup index 61d5d80a3..689968737 100755 --- a/bin/setup +++ b/bin/setup @@ -25,12 +25,24 @@ File.open(File.join(coll, "config", "private.yml"), "w") do |file| default: # number of files processed at the same time, increase for more powerful computers threads: 5 + api_version: "2.0" + + # Omeka S posting (required only if using post_omeka / post_omeka_html) + # omeka_server: servername.unl.edu/path/to/api + # key_identity: your_api_key_identity + # key_credential: your_api_key_credential + # resource_template: 1 + # omeka_data_base: path/to/tei + # iiif_server: servername.unl.edu # optional, only needed for media ingest + # iiif_collection: collection_name # optional, only needed for media ingest development: es_path: https://edit_private_config/elastic es_index: edit_private_config + # item_set: 123 production: es_path: https://edit_private_config/elastic es_index: edit_private_config + # item_set: 456 TEXT file.write(text) @@ -81,11 +93,17 @@ end # SCRIPTS -puts "-- Place XSLT, Ruby, and Python overrides in scripts/overrides" +puts "-- Place XSLT and Ruby overrides in scripts/overrides" FileUtils.mkdir_p(File.join(coll, "scripts", "overrides")) FileUtils.touch(File.join(coll, "scripts", "overrides", ".keep")) -FileUtils.cp(File.join(datura, "lib", "datura", "python", "omeka_overrides_example.py"), File.join(coll, "scripts", "overrides", "omeka_overrides_example.py")) +puts "-- Place Python overrides in scripts/python" +FileUtils.mkdir_p(File.join(coll, "scripts", "python")) +FileUtils.cp(File.join(datura, "lib", "datura", "python", "field_overrides_example.py"), File.join(coll, "scripts", "python", "field_overrides_example.py")) +FileUtils.cp(File.join(datura, "lib", "datura", "python", "process_overrides_example.py"), File.join(coll, "scripts", "python", "process_overrides_example.py")) + +puts "-- Copying requirements.txt for Python/Omeka pipeline".green +FileUtils.cp(File.join(datura, "requirements.txt"), File.join(coll, "requirements.txt")) # SOURCE @@ -104,6 +122,7 @@ FileUtils.touch(File.join(src, "drafts", "tei", ".keep")) File.open(File.join(coll, ".gitignore"), "w") do |file| text = < 0.8.1" - spec.add_runtime_dependency "nokogiri", "~> 1.10" - spec.add_runtime_dependency "rest-client", "~> 2.1" - spec.add_runtime_dependency "pdf-reader", "~> 2.12" - spec.add_runtime_dependency "byebug", "~> 11.0" - spec.add_development_dependency "bundler", ">= 1.16.0", "< 3.0" + spec.add_runtime_dependency "colorize", "~> 1.1" + spec.add_runtime_dependency "nokogiri", "~> 1.18" + spec.add_runtime_dependency "pdf-reader", "~> 2.15" + spec.add_development_dependency "byebug", "~> 11.0" + spec.add_development_dependency "bundler", ">= 2.0", "< 5.0" spec.add_development_dependency "minitest", "~> 5.0" spec.add_development_dependency "rake", "~> 13.0" end diff --git a/docs/1_setup/omeka_setup.md b/docs/1_setup/omeka_setup.md index 23857fb76..77cc24877 100644 --- a/docs/1_setup/omeka_setup.md +++ b/docs/1_setup/omeka_setup.md @@ -1,18 +1,53 @@ ## Set up for Omeka S posting -### Setting up data repo for Omeka +### Step 1: Set up a data repository for Omeka -In your Gemfile, make sure that Datura is on the right branch for Omeka posting. Currently the line should read `gem "datura", git: "https://github.com/CDRH/datura", branch: "omeka_posting_generalized"` (soon it might be incorporated into a formal release). Change `.ruby-gemset` to `datura-omeka` or something similar if this is not the version of Datura you usually use. Run `cd .` and then `bundle install`. Then run `setup`. Copy `omeka_overrides_examples.py` to `omeka_overrides.py` and make desired changes. Alternatively, if you don't want to set up a repo from scratch, clone the `https://github.com/CDRH/datura` repo, switch to the `https://github.com/CDRH/datura` branch, and copy the `lib/datura/python/omeka_overrides_example.py` to `[collection-directory]/scripts/omeka_overrides.py`. +#### If you would like to create a new repository -### Enabling a virtual environment +Follow [the steps](https://github.com/CDRH/datura/blob/dev/docs/1_setup/collection_setup.md#step-1--create-a-new-collection-directory) in the `collection_setup` documentation for Datura, specifying any recent release of Datura in your Gemfile. If you know out of the gate that you plan to create overrides for any Omeka processes or fields, copy `process_overrides_examples.py` and/or `field_overrides_examples.py` (in the `/scripts/python` directory) to `process_overrides.py` or `field_overrides.py`. -In your collection repo, first exit any virtual environemt if one currently enabled (this may be indicated by `(.venv)` or similar text before your command prompt) with `deactivate`. If you have not previously created a virtual environment, type `python3 -m venv .venv`. The environment will be installed in the `.venv` folder in the root of the collection repo. This folder should not be committed. To enter the virtual environment once it has been created, run `source .venv/bin/activate`. Then run `pip3 install -r requirements.txt` to install the dependencies. These two steps are necessary to get the `post_omeka` and `post_omeka_html` scripts to run. +#### If you are working with an existing data repository -If running the script results in an error that a dependency is missing (i.e. `ModuleNotFound`) run `pip3 install [dependency]`. (It may be necessary to do an Internet search to determine the name of the needed package, which may differ between the `import` statement and the `pip3 install` command; e.g. `import dotenv` but `pip3 install python-dotenv`). After installing all necessary dependencies, you can run `pip3 freeze > requirements.txt`, and commit the `requirements.txt` file within the data repo. +In your Gemfile, make sure that Datura is on the right branch for Omeka posting. This functionality should be included in any recent release. Change `.ruby-gemset` to `datura-omeka` or something similar if this is not the version of Datura you usually use. Run: -### Config for Omeka S posting +```bash +cd . +bundle install +``` -The following settings should be placed in `config/private.yml` (in addition to the config that is already included for Datura): +If you plan to create overrides for any Omeka fields, you can copy the `lib/datura/python/omeka_overrides_example.py` in the Datura library (at `/lib/datura/python/`) to your repository as `[collection-directory]/scripts/python/omeka_overrides.py`. + +### Step 2: Enable a virtual environment + +In your collection repo, first exit any virtual environment if one is currently enabled (this may be indicated by `(.venv)` or similar text before your command prompt) with `deactivate`. If you have not previously created a virtual environment, run: + +```bash +python3 -m venv .venv +``` + +The environment will be installed in the `.venv` folder in the root of the collection repo. This folder should not be committed. If you are working in a newly created repo, it should already be added to the `.gitignore` file. If you are working with an existing data repository, you may have to add the `.venv` directory to `.gitignore`. + +To enter the virtual environment once it has been created, run + +```bash +source .venv/bin/activate +``` + +You should now see a `(.venv)` at the front of the command line prompt. You will need to activate this environment every time you post to Omeka S. + +### Step 3: Install Python dependencies + +Next, confirm you have a `requirements.txt` file in the root directory of your collection. If you are working with an existing repository, you may need to copy this over from Datura. Then, to install the dependencies, run: + +```bash +pip3 install -r requirements.txt +``` + +The `packaging` and `setuptools` libraries may need to be installed separately if the `omeka_s_tools` installation (part of the `requirements.txt` list) does not install correctly. If installation fails, `pip` may need to be upgraded (the error message should advise this). + +### Step 4: Set up config for Omeka S posting + +If you have a newly created repo, you should see some omeka-related config values in your auto-generated `config/private.yml` config file. Uncomment these and fill in the values. If you are working with an existing repo, the following settings should be placed in `config/private.yml` (in addition to the config that is already included for Datura): ```yaml default: @@ -21,14 +56,17 @@ default: key_credential: ***** resource_template: ## omeka_data_base: desired/base/url/for/tei/files - iiif_server: servername.unl.edu + iiif_server: servername.unl.edu # optional, if the collection uses the image server + iiif_collection: collection_name # optional, if the collection uses the image server development: item_set: ## -production +production: item_set: ## ``` -- (for developers) `json_dir`, `html_id`, and `iiif_dir` are set within the script and correspond to the standard Datura output folders. +All values not listed as optional are required for the omeka scripts to run. + +- (for developers) `json_dir`, `html_id`, and `iiif_dir` are set within the Omeka S scripts and correspond to the standard Datura output folders. The `key_identity` and `key_credential` fields should correspond to the generated API key credentials. which you can generate on your Omeka S user page (click "Edit user" and then the API key). Make sure to copy the credentials down right away after generating the key. @@ -36,8 +74,10 @@ Make sure that config is pointing to the right `resource_template` for the data `omeka_data_base` is necessary to indicate the URL to the TEI data documents. It should have a format like `https://github.com/CDRH/[repo_name]/blob/[env]/source/tei` or specify a similar relative path. The Omeka script adds the filename at the end. Make sure you have the right repo to make this a valid url. -For HTML posting, set `iiif_server` to the base url of the IIIF image server. +For media posting, set `iiif_server` to the base url of the IIIF image server and `iiif_collection` to the name of the collection or whatever name is used for the collection's iiif directory on the image server. + +`item_set` should be specified by environment in `private.yml` in order to categorize items by environment on Omeka S. The proper item_set id can be found in Omeka if you append `admin/item-set` to the base Omeka site URL. Look for `Environment--Development` or something similar; the id will appear at the end of the URL if you click the link. Not all projects have environments. -`item_set` should be specified by environment in `private.yml` in order to categorize items by environment on Omeka S. The proper item_set id can be found in Omeka if you append `admin/item-set` to the base Omeka site URL. Look for `Environment--Development` or something similar; the id will appear at the end of the URL if you click the link. Not all projects have environments and specifying an item set is not necessary to post. +### Step 5: Prepare to post! -See [post_omeka instructions](../3_manage/post_omeka.md) and [post_omeka_html instructions](../3_manage/post_omeka_html.md) for more information. \ No newline at end of file +See [post_omeka instructions](../3_manage/post_omeka.md) and [post_omeka_html instructions](../3_manage/post_omeka_html.md) for information about posting to Omeka S. \ No newline at end of file diff --git a/docs/2_customization/omeka_overrides.md b/docs/2_customization/omeka_overrides.md index 202cd1b2e..01c7d74a5 100644 --- a/docs/2_customization/omeka_overrides.md +++ b/docs/2_customization/omeka_overrides.md @@ -2,13 +2,13 @@ ### Standard definitions of fields -Each Omeka field is updated by the method in [api_fields.py](../../../lib/datura/python/api_fields.py) to compile the Omeka S JSON. This method takes the form `update_item_value(item, key, value, datatype="literal")`, the first argumment is the json hash with the API data, the second argument corresponds to the field in the resource template, and the third is the return value the corresponding function of `field_definitions.py`. Optionally, you can pass in the datatype, as the fourth argument. The default definitions of Omeka fields are in [field_definitions.py](../../../lib/datura/python/field_definitions.py). The Omeka API fields defined here must correspond with the Omeka resource template you are using, and the return value should be compatible with the data type.If you do not specify it, it will be set to "literal". For example `update_item_value(built_item, "dcterms:date", fields.date(json), "numeric:timestamp")`. +Each Omeka field is updated by the method in [api_fields.py](../../../lib/datura/python/api_fields.py) to compile the Omeka S JSON. This method takes the form `update_item_value(ctx, item, key, value, datatype="literal")`. The first argument is the context parameter (providing access to config values, OmekaAPIClient, and property ID cache), the second argument is the json hash with the API data, the third argument corresponds to the field in the resource template, and the fourth is the return value of the corresponding function of `field_definitions.py`. Optionally, you can pass in the datatype, as the fifth argument. The default definitions of Omeka fields are in [field_definitions.py](../../../lib/datura/python/field_definitions.py). The Omeka API fields defined here must correspond with the Omeka resource template you are using, and the return value should be compatible with the data type. If you do not specify it, it will be set to "literal". For example `update_item_value(ctx, built_item, "dcterms:date", fields.date(json), "numeric:timestamp")`. ### Overriding fields -To override the field definitions, copy the file [omeka_overrides_example.py](../../../lib/datura/python/omeka_overrides_example.py) to [omeka_overrides.rb](../../../lib/datura/python/omeka_overrides.py) in the `scripts/overrides` file of the project directory. Then override each method as needed, using the existing definitions in `field_definitions.py` as examples. For an example, see https://github.com/CDRH/data_stories_humanity/blob/omeka_s_ingest/scripts/python/omeka_overrides.py (not currently field). +To override the field definitions, copy the file [field_overrides_example.py](../../../lib/datura/python/field_overrides_example.py) to [field_overrides.py](../../../lib/datura/python/field_overrides.py) in the `scripts/python` file of the project directory. Then override each method as needed, using the commented patterns in the example overrides file as a guide. -Each overriden method needs to take the arguments `self` (a Python placeholder for a class instance) and `json` (representing the generated JSON) and to match the methods defined on `field_definitions.py`. (The same goes for adding new methods to `field_definitions.py`.) +Each overridden method needs to take the arguments `self` (a Python placeholder for a class instance) and `json` (representing the generated JSON) and to match the methods defined in `field_definitions.py`. (The same goes for adding new methods to `field_definitions.py`.) For instance: ```python @@ -20,17 +20,21 @@ For instance: return person_names ``` -First retrieve the value from the Elasticsearch `json` (keeping in mind that it is sometimes nil), then do any manipulations needed before returning the desired value. The return value must be either an list or single value. For single values, usually this will be the same as the value in the JSON. But Unlike the Elasticsearch-based API, it is not possible to ingest nested fields into Omeka S, so they must be reduced into array form. See [field_definitions.py](../../../lib/datura/python/field_definitions.py)for examples of how to retrieve single and nested values from the JSON, manipulate them and return the proper values for Omeka S. +First retrieve the value from the Elasticsearch `json` (keeping in mind that it is sometimes nil), then do any manipulations needed before returning the desired value. The return value must be either an list or single value. For single values, usually this will be the same as the value in the JSON. But unlike the ElasticSearch-based API, it is not possible to ingest nested fields into Omeka S, so they must be reduced into array form. See [field_definitions.py](../../../lib/datura/python/field_definitions.py)for examples of how to retrieve single and nested values from the JSON, manipulate them and return the proper values for Omeka S. + +### Overriding processes + +Some core functions involved in the Omeka posting can also be overridden. To do this, copy the file [process_overrides_example.py](../../../lib/datura/python/process_overrides_example.py) to [process_overrides.py](../../../lib/datura/python/process_overrides.py) in the `scripts/python` file of the project directory. Then override each method as needed, using the commented patterns in the example overrides file as a guide. ### Linking items -Any new fields that link to the id of another item should be added to the `link_item` in [api_fields.py](../../../lib/datura/python/api_fields.py). `link_item_record` works in the same way as `update_item_value` but the value of the ES field must be a CDRH ID. +Any new fields that link to the id of another item should be added to the `link_records` in `process_overrides.py`. `link_item_record` works in the same way as `update_item_value` but the value of the ES field must be a CDRH ID. ```python try: part_ids = [part['id'] for part in json_item["has_part"]] - link_item_record(existing_item, "dcterms:hasPart", part_ids) - except Exception: + link_item_record(ctx, existing_item, "dcterms:hasPart", part_ids) + except (KeyError, TypeError): pass ``` diff --git a/docs/3_manage/post.md b/docs/3_manage/post.md index 2a013f4c5..482e31a97 100644 --- a/docs/3_manage/post.md +++ b/docs/3_manage/post.md @@ -30,6 +30,14 @@ The above does the following: Displays usage and list of options +```bash +-c, --csv-rows [input] +``` + +Transforms / posts only csv lines whose identifier (id/identifier column) matches a specific regular expression. + +Examples: `post -c cat_001 (exact), cat_ (prefix), 'cat_00[1-3]' (range), 'cat_\d' (use digit character)` + ```bash -e, --environment [input] ``` @@ -54,6 +62,21 @@ Format options include: If you do not select any, all the formats found will be executed. +```bash +-j, --json-output +``` + +*Default setting: false* + +Omeka specific, this writes Omeka S item payloads to `output//omeka/` instead of posting to the API. This is useful for debugging and inspection purposes. + +```bash +-m, --no-media +``` + +Omeka specific, this will skip the step of deleting and regenerating media if both HTML and cover image are present in Omeka S. + + ```bash -n, --no-commit ``` @@ -70,6 +93,26 @@ Solr specific, this will post documents but will not "commit" them to the index. Outputs transformed files to a collection's `output/[environment]/[type]`. This is useful for debugging and inspection purposes. +```bash +-p, --proceed [input] +``` + +Proceed with posting from (and including) the file matching this regex. Files are sorted alphabetically before the proceed point is located. The regex must match exactly one file or the script will exit with an error. Can be combined with `-r` to proceed within a filtered set. + +Example: `post -p let0050` (post all files from `let0050` onward) + +Example: `post -r let -p let0050` (post all `let` files from `let0050` onward) + +**Checkpoint file**: After each batch of files is posted (not in `--transform-only` mode), Datura writes the last posted filename to `logs/proceed_{environment}` in your collection directory. + +**Interactive resume**: When `-p` is given with no value (`post -p`), Datura reads the checkpoint file and prompts: + +``` +Continue from ? (y/n): +``` + +Enter `y` to resume posting from that file, or `n` to exit. If no checkpoint file exists yet, the script exits with an error directing you to run `post` at least once first. + ```bash -r, --regex [input] ``` @@ -78,6 +121,12 @@ Transforms / posts only files matching a specific regular expression. DO NOT in Example: `post -r let0001` +```bash +-s, --skip +``` + +Omeka specific, this skips the JSON or HTML generation step and only posts to Omeka S (it can be used with both `post_omeka` and `post_omeka_html`). Note however in the case of `post_omeka` that it should only be used with a filtered subset of files (otherwise nothing will be posted, because JSON files are deleted as part of unfiltered posts). + ```bash -t, --transform-only ``` @@ -127,4 +176,4 @@ If you get an error when you run `bundle install`, try running the suggested com ### Posting to Omeka -For posting to Omeka, see [post_omeka.md] and [post_omeka_html.md] +For posting to Omeka, see [post_omeka.md](post_omeka.md) and [post_omeka_html.md](post_omeka_html.md) diff --git a/docs/3_manage/post_omeka.md b/docs/3_manage/post_omeka.md index 9c6ec706f..a0f9eb13d 100644 --- a/docs/3_manage/post_omeka.md +++ b/docs/3_manage/post_omeka.md @@ -1,22 +1,22 @@ ## Instructions for posting data into Omeka API -See (omeka setup instructions)[../1_setup/omeka_setup.md] for how to prepare your repo, config, and activate the Python virtual environment. +See (omeka setup instructions)[../1_setup/omeka_setup.md] for how to prepare and configure your data repository and activate the Python virtual environment. -Running the `post_omeka` script will first run the Datura scripts to generate JSON files with the standard fields and values of the CDRH API (this is what is normally sent to Elasticsearch when you run `post`). This first step is equivalent to running `post -x es -o -t`. It then sends the generated JSON to the Python scripts to be ingested into Omeka S. +Running the `post_omeka` script will first run the Datura scripts to generate JSON files with the standard fields and values of the CDRH API (this is what is normally sent to ElasticSearch when you run `post`). This first step is equivalent to running `post -x es -o -t`. It then sends the generated JSON to the Python scripts to be ingested into Omeka S. Note that this command also deletes existing content from the output `es` directory so that any files that have been deleted from source do not populate from stale JSON. -Use the `-s` option to skip the generation step and only post to Omeka S (requires that you have already generated the needed documents by running `post_omeka` normally). +NOTE: if and only if you are posting a subset of items, you may use the `-s` option to skip the JSON generation step and only post to Omeka S (requires that you have already generated the needed documents by running `post_omeka` normally). If you run `post_omeka` with `-s` and no filter, the script will delete all JSON output in the `es` directory and will therefore have nothing to post to Omeka. -It is possible to run `post_omeka` with Datura's other command line options as described in [post.md] (for instance `-f` to filter by file type and `-r` and filter by regex), but it is not recommended to override the default options such as `-x es` +It is possible to run `post_omeka` with Datura's other command line options as described in [post.md] (for instance `-f` to filter by format and `-r` to filter by regex), but it is not recommended to override the default options such as `-x es`. You can specify the environment with `-e [environment]` but you must set an `item_set` with the desired environment in `config/private.yml.` See (omeka setup instructions)[../1_setup/omeka_setup.md] for more details. -For information on how to override field definitions, see [Omeka Overrides](../2_customization/omeka_overrides.md). +For information on how to override processes and field definitions, see [Omeka Overrides](../2_customization/omeka_overrides.md). ## Troubleshooting -### notes on debugging +### Notes on debugging -The standard way to debug Python scripts is with `breakpoint()`, equivalent to `byebug` in Ruby. Execution will pause and then you can check the contents of variables and try snippets of code from the prompt. Sometimes putting a debugger within an except clause (especially within a loop) makes it difficult to break out of the script and halt execution even if you type `quit`. CTRL-C sometimes works in these cases. If CTRL-C also fails to halt the script, try running `os._exit(0)` (this is the reason for `import os` in some of the scripts, and if you get an error that the module is not found, then `import os` should be run from the debugger prompt first). For more details on a particular error, look at the error message which is printed in some of the except clauses, check online documentation to see if this error message has additional methods (depending on the error), or use `traceback.print_exc()` to print out the full stack trace. In the case of errors in the HTTP reponse, it may be necessary to look in the logs on the Omeka site. +The standard way to debug Python scripts is with `breakpoint()`, equivalent to `byebug` in Ruby. Execution will pause and then you can check the contents of variables and try snippets of code from the prompt. Sometimes putting a debugger within an except clause (especially within a loop) makes it difficult to break out of the script and halt execution even if you type `quit`. `CTRL-C` sometimes works in these cases. If `CTRL-C` also fails to halt the script, try running `os._exit(0)` (this is the reason for `import os` in some of the scripts, and if you get an error that the module is not found, then `import os` should be run from the debugger prompt first). For more details on a particular error, look at the error message which is printed in some of the except clauses, check online documentation to see if this error message has additional methods (depending on the error), or use `traceback.print_exc()` to print out the full stack trace. You can also check the logs at `/logs/python.log`. In the case of errors in the HTTP response, it may be necessary to look in the logs on the Omeka site. ### 500 error @@ -24,7 +24,7 @@ Look in the error log on the Omeka site. This may indicate a configuration probl ### Data type not allowed in template -Check your script in api_fields.py to make sure you are passing the right data types, as specified in the resource template you are using. The default data type is "literal". If the resource template changes you must change the data types in your script, too. +Check your script in `api_fields.py` to make sure you are passing the right data types, as specified in the resource template you are using. The default data type is "literal". If the resource template changes you must change the data types in your script, too. ### Term not in template diff --git a/docs/3_manage/post_omeka_html.md b/docs/3_manage/post_omeka_html.md index 3593f9501..dd54c9656 100644 --- a/docs/3_manage/post_omeka_html.md +++ b/docs/3_manage/post_omeka_html.md @@ -6,7 +6,7 @@ Use the `-s` option to skip the generation step and only post to Omeka S (requir Use the `-m` option to skip the step of deleting and regenerating for media items that have already been ingested. (It will only do this if both the image and html are uploaded already). -You can specify the environment with `-e [environment]` but you must set an `item_set` with the desired environment in config/private.yml. See [post_omeka instructions](docs/3_manage/post_omeka.md) for instructions. +You can specify the environment with `-e [environment]` but you must set an `item_set` with the desired environment in `config/private.yml`. See [post_omeka instructions](docs/3_manage/post_omeka.md) for instructions. Instructions for setting up the Python virtual enviroment and config can be found in (omeka setup instructions)[../1_setup/omeka_setup.md]. @@ -14,14 +14,14 @@ For information on how to override field definitions, see [Omeka Overrides](../2 ### Media ingesters (for developers) -The media payload, set in `html_and_media_ingest.py`, must be structured in a specific way to add items. It is different in the case of html and iiif images. +The media payload, set in `html_and_media_ingest.py`, must be structured in a specific way to add items. It is different in the case of html and iiif images. For an html field: ```json { "o:is_public": True, "data": { - "html": html_content + "html": html_content, }, "o:ingester": "html" } @@ -31,7 +31,7 @@ For a file upload (i.e. to upload): { "o:is_public": True, "data": { - "upload": html_content + "upload": html_content, }, "o:ingester": "upload" } @@ -47,13 +47,13 @@ For a file upload (i.e. to upload): } ``` The IIIF URL should be in the format https://servername/iiif/2/collection_name%2Fitem_id.jpg/info.json. It should not point to a specific image. - `o:source`, set in the `add_media_to_item` in omeka.py, either corersponds to the filename or to the remote path if the ingester requires a remote URL. + `o:source`, set in `add_media_to_item` in the `omeka_s_tools` library, either corresponds to the filename or to the remote path if the ingester requires a remote URL. ## Troubleshooting -Sometimes running the script will return error code 422 (unprocessable content) or error code 500. These error messages can be investigated in the Omeka S logs found on the admin page. To investigate the errors, you can check the stack traces in these logs against the Omeka S source code found in GitHub. (Note that the base Omeka code, powering the website, is in PHP). +Sometimes running the script will return error code 422 (unprocessable content) or error code 500. These error messages can be investigated in the Omeka S logs found on the admin page. To investigate the errors, you can check the stack traces in these logs against the Omeka S source code found in GitHub or the Python logs in `/logs/python`. (Note that the base Omeka code, powering the website, is in PHP). - The ingester expects a full URL, not a local file path, in `o:source`, which is set when you post media items. - A malformed URL may cause an error that the script is unable to connect to a server. Make sure that it includes slashes in the proper places. - Internal SQL errors are likely also caused by sending bad data, not corresponding to the designated format. - There is a known issue where Omeka S raises an error when deleting media items and unlinking them, even though the action is performed successfully. -- Forbidden (403) errors may indicate that API credentials are missing or incorrect \ No newline at end of file +- Forbidden (403) errors may indicate that API credentials are missing or incorrect. \ No newline at end of file diff --git a/docs/4_developers/saxon.md b/docs/4_developers/saxon.md deleted file mode 100644 index 303295bea..000000000 --- a/docs/4_developers/saxon.md +++ /dev/null @@ -1,67 +0,0 @@ -**Contents** -- [Install System JAR](#install-system-jar) -- [Bash Executable](#bash-executable) -- [Reference](#reference) - -A system Saxon JAR is used with scripting via a [Bash executable](#bash-executable) - -## Install System JAR - -Please use the same version of Saxon used by Oxygen as noted in -server documentation tracking versions of Saxon used across servers: -https://github.com/CDRH/cdrh-technical-documentation/blob/main/pages/Saxon.md#versions - -```bash -# Make directory named for the version: -sudo mkdir /usr/local/share/saxon-(version) - -# Symlink to versioned directory for quick switching in the future -sudo ln -s /usr/local/share/saxon-(version) /usr/local/share/saxon -``` - -Copy the Saxon JAR file (e.g. `saxon9he.jar`) from another server with `scp` or `rsync` - -or - -Download Saxon-HE from [Saxonica Open Source](https://sourceforge.net/projects/saxon/files/Saxon-HE/) - -Find saxon9he.jar and move it to usr/local/share/saxon-(version) (make sure to have the Finder show hidden files, if you are on a Mac). - -## Bash Executable -This bash executable passes its arguments through to and runs the Saxon JAR - -`sudo vim /usr/local/bin/saxon`: -```bash -#!/usr/bin/env bash - -JAVA_BIN="$(which java)" -SAXON_JAR="/usr/local/share/saxon/saxon9he.jar" - -# If JAVA_HOME is defined, call java from ENV-based path -if [[ -n "${JAVA_HOME}" ]]; then - JAVA_BIN="${JAVA_HOME}/bin/java" -fi - -# Pass all bash arguments ($@) through to the Saxon JAR -exec "${JAVA_BIN}" -jar "${SAXON_JAR}" "$@" - -# Exit with an error status if the exec shell exits back to this script -exit 1 -``` - -Add the executable permission to the Bash script:
-`sudo chmod +x /usr/local/bin/saxon` - -If you locate your Saxon JAR somewhere other than `/usr/local/share/saxon/saxon9he.jar`, make sure to update the path stored in the `SAXON_JAR` variable at the top of the script. - -Now you and others may run `saxon (arguments)` from anywhere on your system to save typing and run Saxon within scripts. - -If you get an error about locating a Java Runtime, this is probably because macOS no longer comes installed with a Java Runtime Environment. Run `brew install java`. - - -## Reference -- Download / Support Info: http://saxonica.com/download/opensource.xml -- Release Announcements: https://saxonica.plan.io/news/1 -- Bug Fixes: In release(version #).txt file on [SourceForge](https://sourceforge.net/projects/saxon/files/Saxon-HE/) -- Change Info: http://www.saxonica.com/documentation/#!changes/serialization/9.2-9.3 -- Issues: https://saxonica.plan.io/projects/saxon/issues diff --git a/docs/README.md b/docs/README.md index 75618aa9d..3d7b0c2e7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -35,7 +35,6 @@ The files are parsed and formatted into documents appropriate for Solr, IIIF, El - Remove / destroy index - Developers - [Installation](4_developers/installation.md) - - [Saxon setup](4_developers/saxon.md) - [Ruby / Gems](4_developers/ruby_gems.md) - Class organization - [Tests](4_developers/test.md) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 78067b0a0..aebe1bd38 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -22,7 +22,7 @@ Do you have the correct permissions to run all of the files? Check that the TEI ### XSLT -If you type `saxon` into the command line and hit enter, does it find the command? Saxon will need to be able to run from the command line. See the [saxon docs](saxon.md) for setup information. +Check that Python 3 is available and `saxonche` has been installed (`pip3 install -r requirements.txt`). ### Tests diff --git a/lib/datura/common_xml.rb b/lib/datura/common_xml.rb index 8fb7465db..1131634f2 100644 --- a/lib/datura/common_xml.rb +++ b/lib/datura/common_xml.rb @@ -45,14 +45,13 @@ def self.create_xml_object(filepath, remove_ns=true) file_xml end + # TODO: this method is deprecated with switch to saxonche + # but leaving it in place until testing is complete # saxon accepts params in following manner # fw=true pb=true figures=false - def self.stringify_params(param_hash) - params = "" - if param_hash - params = param_hash.map{ |k, v| "#{k}=#{v}" }.join(" ") - end - params + def self.arrayify_params(param_hash) + return [] unless param_hash + param_hash.map { |k, v| "#{k}=#{v}"} end def self.sub_corrections(aXml) diff --git a/lib/datura/data_manager.rb b/lib/datura/data_manager.rb index 24f4898e9..9bcd1754e 100644 --- a/lib/datura/data_manager.rb +++ b/lib/datura/data_manager.rb @@ -1,7 +1,6 @@ require "colorize" require "logger" require "yaml" -require "byebug" require_relative "./requirer.rb" class Datura::DataManager @@ -12,6 +11,7 @@ class Datura::DataManager attr_accessor :error_html attr_accessor :error_iiif attr_accessor :error_solr + attr_accessor :skipped_es attr_accessor :files attr_accessor :options @@ -38,6 +38,7 @@ def initialize @error_html = [] @error_iiif = [] @error_solr = [] + @skipped_es = [] # combine user input and config files params = Datura::Parser.post_params @@ -64,6 +65,13 @@ def load_collection_classes end end + def check_omeka_options + %w[omeka_server key_identity key_credential iiif_server + resource_template omeka_data_base item_set].each do |opt| + assert_option(opt) + end + end + def print_options pretty = JSON.pretty_generate(@options) puts "Options: #{pretty}" @@ -80,6 +88,7 @@ def run @log.info(msg) puts msg pre_file_preparation + handle_proceed_prompt @files = prepare_files pre_batch_processing batch_process_files @@ -109,7 +118,6 @@ def allowed_files(all_files) files end - # TODO should this move to Options class? def assert_option(opt) if !@options.key?(opt) puts "Option #{opt} was not found! Check config files and add #{opt} to continue".red @@ -128,6 +136,16 @@ def batch_process_files end # wait for all the files to process before moving on with the next chunk threads.each { |t| t.join } + # save checkpoint after chunk completes + Datura::Helpers.write_checkpoint(files_subset.last.filename(false), @options) + end + # clear checkpoint if all files in the source directories were posted (not a filtered subset) + unless @files.empty? + last_overall = Datura::DataManager.format_to_class.keys.filter_map { |fmt| + found = Datura::Helpers.get_directory_files(File.join(@options["collection_dir"], "source", fmt)) + found&.map { |f| File.basename(f, ".*") }&.sort&.last + }.last + Datura::Helpers.clear_checkpoint(@options) if @files.last.filename(false) == last_overall end end @@ -157,9 +175,27 @@ def end_run error_msg << "#{@error_html.length} HTML transform error(s)\n" error_msg << "#{@error_iiif.length} IIIF Manifest transform error(s)\n" error_msg << "#{@error_solr.length} Solr transform / post error(s)\n" + error_msg << "#{@skipped_es.length} ES item(s) skipped (missing id or title)\n" puts error_msg @log.info(error_msg) + all_errors = { + "ES" => @error_es, + "HTML" => @error_html, + "IIIF" => @error_iiif, + "Solr" => @error_solr, + "ES skipped" => @skipped_es + }.reject { |_, v| v.empty? } + + if all_errors.any? + puts "\n--- Error details ---".red + all_errors.each do |type, errors| + errors.each { |e| puts "[#{type}] #{e}".red } + end + puts "---------------------".red + @log.error("Error details: #{all_errors.inspect}") + end + # figure time for running @time << Time.now dur = @time[1] - @time[0] @@ -179,6 +215,10 @@ def get_files formats = [] if @options["format"] formats = [@options["format"]] + elsif @options["csv_rows"] + msg = "csv_rows filter set (-c); restricting processing to CSV format only" + puts msg.cyan + formats = ["csv"] else formats = Datura::DataManager.format_to_class.keys end @@ -190,12 +230,39 @@ def get_files files end + def handle_proceed_prompt + # Only act when -p was given with no value (proceed is nil, not false) + return unless @options["proceed"].nil? + + checkpoint = Datura::Helpers.read_checkpoint(@options) + if checkpoint.nil? + path = Datura::Helpers.checkpoint_path(@options) + msg = "ERROR: --proceed given with no value but no checkpoint file found at #{path}. Run post at least once without -p to create a checkpoint.".red + puts msg + @log.error(msg) + exit 1 + end + + print "Continue from #{checkpoint}? (y/n): " + STDOUT.flush + response = STDIN.gets&.chomp&.downcase || "" + if response == "y" + @options["proceed"] = checkpoint + msg = "Resuming from checkpoint: #{checkpoint}" + puts msg + @log.info(msg) + else + puts "Exiting." + exit 0 + end + end + def options_msg msg = "Start Time: #{Time.now}\n" msg << "Running script with following options:\n" msg << "collection: #{@options['collection']}\n" msg << "Environment: #{@options['environment']}\n" - msg << "Posting to: #{@es.index_url}\n\n" if should_post?("es") + msg << "Posting to: #{@es.index_url}\n\n" if should_post?("es") && @es msg << "Posting to: #{@solr_url}\n\n" if should_post?("solr") msg << "Format: #{@options['format']}\n" if @options["format"] msg << "Regex: #{@options['regex']}\n" if @options["regex"] @@ -233,8 +300,19 @@ def prepare_files allowed = allowed_files(files) # filter by regex regexed = Datura::Helpers.regex_files(allowed, @options["regex"]) + if @options["regex"] && regexed.empty? + msg = "No files matched regex: #{@options['regex']}" + puts msg.yellow + @log.warn(msg) + end + # proceed from (and including) a specific file + proceeded = if @options["proceed"] + Datura::Helpers.proceed_files(regexed, @options["proceed"]) + else + regexed + end # filter by date - filtered = regexed.select { |f| Datura::Helpers.should_update?(f, @options["update_time"]) } + filtered = proceeded.select { |f| Datura::Helpers.should_update?(f, @options["update_time"]) } file_classes = [] @log.info("After filters (regex, update time), #{filtered.length}/#{files.length} files remaining") @@ -264,7 +342,13 @@ def prepare_xslt if !t1 || !t2 || t1 > t2 puts "Copying datura XSLT default scripts into collection" - FileUtils.cp_r(datura_xslt, dest) + begin + FileUtils.cp_r(datura_xslt, dest) + rescue Errno::ENOENT + raise "Could not copy XSLT scripts into the collection. " \ + "Confirm you are running this command from the root of the collection repository, " \ + "not from a subdirectory." + end end end @@ -281,16 +365,36 @@ def set_up_logger ) end + def check_xslt_dependency + _, err, status = Open3.capture3("python3", "-c", "import saxonche") + unless status.success? + puts "saxonche Python module is not installed. Install it with: pip install saxonche\n#{err}".red + exit + end + end + def set_up_services if should_post?("es") - # set up elasticsearch instance - @es = Datura::Elasticsearch::Index.new(@options, schema_mapping: true) + begin + # set up elasticsearch instance + @es = Datura::Elasticsearch::Index.new(@options, schema_mapping: true) + rescue Errno::ECONNREFUSED, SocketError, Errno::ETIMEDOUT + msg = "Could not connect to Elasticsearch at #{File.join(@options['es_path'], @options['es_index'])}. " \ + "Confirm you have specified the correct environment " \ + "(currently: #{@options['environment']}). Use -e to specify an environment." + error_with_transform_and_post(msg, @error_es) + @es = nil + end end if should_post?("solr") # set up posting URLs @solr_url = File.join(options["solr_path"], options["solr_core"], "update") end + + if should_transform?("html") || should_transform?("solr") + check_xslt_dependency + end end def should_post?(type) @@ -304,22 +408,21 @@ def should_transform?(type) def transform_and_post(file) # elasticsearch - if should_transform?("es") - if @options["transform_only"] - # TODO transformation is not treated the same way here as in - # most post methods, so having to use try catch block - begin + begin + if should_transform?("es") + if @options["transform_only"] res_es = file.transform_es - rescue => e - error_with_transform_and_post("#{e}", @error_es) - end - else - res_es = file.post_es(@es) - if res_es && res_es.has_key?("error") - error_with_transform_and_post(res_es["error"], @error_es) + elsif @es + res_es = file.post_es(@es) + if res_es && res_es.has_key?("error") + error_with_transform_and_post(res_es["error"], @error_es) + end end end + rescue => e + error_with_transform_and_post("#{e}", @error_es) end + @skipped_es.concat(file.skipped_es) if file.skipped_es.any? # html begin diff --git a/lib/datura/elasticsearch/alias.rb b/lib/datura/elasticsearch/alias.rb index 4d6a3a118..d28751bee 100644 --- a/lib/datura/elasticsearch/alias.rb +++ b/lib/datura/elasticsearch/alias.rb @@ -1,5 +1,4 @@ require "json" -require "rest-client" require_relative "./../elasticsearch.rb" @@ -20,15 +19,17 @@ def self.add { add: { alias: ali, index: idx } } ] } - RestClient.post(base_url, data.to_json, @auth_header.merge({ content_type: :json })) { |res, req, result| - if result.code == "200" - puts res - puts "Successfully added alias #{ali}. Current alias list:" - puts list - else - raise "#{result.code} error managing aliases: #{res}" - end - } + auth = Datura::Helpers.construct_auth_header(options) + response = Datura::Helpers.es_http_request("POST", base_url, + body: data.to_json, + headers: auth.merge("Content-Type" => "application/json")) + if response.code == "200" + puts response.body + puts "Successfully added alias #{ali}. Current alias list:" + puts list + else + raise "#{response.code} error managing aliases: #{response.body}" + end end def self.delete @@ -40,16 +41,20 @@ def self.delete url = File.join(options["es_path"], idx, "_alias", ali) - res = JSON.parse(RestClient.delete(url, @auth_header)) - puts JSON.pretty_generate(res) + auth = Datura::Helpers.construct_auth_header(options) + response = Datura::Helpers.es_http_request("DELETE", url, + headers: auth) + puts JSON.pretty_generate(JSON.parse(response.body)) list end def self.list options = Datura::Options.new({}).all - res = RestClient.get(File.join(options["es_path"], "_aliases"), ) - JSON.pretty_generate(JSON.parse(res)) + auth = Datura::Helpers.construct_auth_header(options) + response = Datura::Helpers.es_http_request("GET", File.join(options["es_path"], "_aliases"), + headers: auth) + JSON.pretty_generate(JSON.parse(response.body)) end end diff --git a/lib/datura/elasticsearch/index.rb b/lib/datura/elasticsearch/index.rb index 09828d82f..b4ffe0a44 100644 --- a/lib/datura/elasticsearch/index.rb +++ b/lib/datura/elasticsearch/index.rb @@ -1,5 +1,4 @@ require "json" -require "rest-client" require "yaml" require "base64" @@ -25,7 +24,7 @@ def initialize(options = nil, schema_mapping: false) @mapping_url = File.join(@index_url, "_mapping?pretty=true") # yaml settings (if exist) and mappings - @requested_schema = YAML.load_file(@options["es_schema"]) + @requested_schema = YAML.safe_load_file(@options["es_schema"], permitted_classes: [Symbol]) @auth_header = Datura::Helpers.construct_auth_header(@options) # if requested, grab the mapping currently associated with this index # otherwise wait until after the requested schema is loaded @@ -36,42 +35,36 @@ def create json = @requested_schema["settings"].to_json puts "Creating ES index for API version #{@options["api_version"]}: #{@pretty_url}" if json && json != "null" - RestClient.put(@pretty_url, json, @auth_header.merge({ content_type: :json })) { |res, req, result| - if result.code == "200" - puts res - else - raise "#{result.code} error creating Elasticsearch index: #{res}" - end - } + response = Datura::Helpers.es_http_request("PUT", @pretty_url, + body: json, + headers: @auth_header.merge("Content-Type" => "application/json")) else - RestClient.put(@pretty_url, nil, @auth_header) { |res, req, result| - if result.code == "200" - puts res - else - raise "#{result.code} error creating Elasticsearch index: #{res}" - end - } + response = Datura::Helpers.es_http_request("PUT", @pretty_url, + headers: @auth_header) + end + if response.code == "200" + puts response.body + else + raise "#{response.code} error creating Elasticsearch index: #{response.body}" end end def delete puts "Deleting #{@options["es_index"]} via url #{@pretty_url}" - RestClient.delete(@pretty_url, @auth_header) { |res, req, result| - if result.code != "200" - raise "#{result.code} error deleting Elasticsearch index: #{res}" - end - } + response = Datura::Helpers.es_http_request("DELETE", @pretty_url, + headers: @auth_header) + raise "#{response.code} error deleting Elasticsearch index: #{response.body}" if response.code != "200" end def get_schema - RestClient.get(@mapping_url, @auth_header) { |res, req, result| - if result.code == "200" - JSON.parse(res) - else - raise "#{result.code} error getting Elasticsearch schema: #{res}" - end - } + response = Datura::Helpers.es_http_request("GET", @mapping_url, + headers: @auth_header) + if response.code == "200" + JSON.parse(response.body) + else + raise "#{response.code} error getting Elasticsearch schema: #{response.body}" + end end def get_schema_mapping @@ -114,13 +107,14 @@ def set_schema json = @requested_schema["mappings"].to_json puts "Setting schema: #{@mapping_url}" - RestClient.put(@mapping_url, json, @auth_header.merge({ content_type: :json })) { |res, req, result| - if result.code == "200" - puts res - else - raise "#{result.code} error setting Elasticsearch schema: #{res}" - end - } + response = Datura::Helpers.es_http_request("PUT", @mapping_url, + body: json, + headers: @auth_header.merge("Content-Type" => "application/json")) + if response.code == "200" + puts response.body + else + raise "#{response.code} error setting Elasticsearch schema: #{response.body}" + end end # doc: ruby hash corresponding with Elasticsearch document JSON @@ -213,13 +207,14 @@ def self.clear_all(options) url = File.join(options["es_path"], options["es_index"], "_delete_by_query?pretty=true") auth_header = Datura::Helpers.construct_auth_header(options) json = { "query" => { "match_all" => {} } } - RestClient.post(url, json.to_json, auth_header.merge({ content_type: :json })) { |res, req, result| - if result.code == "200" - puts res - else - raise "#{result.code} error when clearing entire index: #{res}" - end - } + response = Datura::Helpers.es_http_request("POST", url, + body: json.to_json, + headers: auth_header.merge("Content-Type" => "application/json")) + if response.code == "200" + puts response.body + else + raise "#{response.code} error when clearing entire index: #{response.body}" + end else puts "You typed '#{confirm}'. This is incorrect, exiting program" exit @@ -233,13 +228,14 @@ def self.clear_index(options) if confirmation data = self.build_clear_data(options) auth_header = Datura::Helpers.construct_auth_header(options) - RestClient.post(url, data.to_json, auth_header.merge({content_type: :json })) { |res, req, result| - if result.code == "200" || result.code == "201" - puts res - else - raise "#{result.code} error when clearing index: #{res}" - end - } + response = Datura::Helpers.es_http_request("POST", url, + body: data.to_json, + headers: auth_header.merge("Content-Type" => "application/json")) + if response.code == "200" || response.code == "201" + puts response.body + else + raise "#{response.code} error when clearing index: #{response.body}" + end else puts "come back anytime!" exit diff --git a/lib/datura/file_type.rb b/lib/datura/file_type.rb index 2e53fd174..a88ed496c 100644 --- a/lib/datura/file_type.rb +++ b/lib/datura/file_type.rb @@ -6,6 +6,7 @@ class FileType # general information about file attr_reader :file_location attr_reader :options + attr_reader :skipped_es # script locations attr_accessor :script_es @@ -22,6 +23,7 @@ class FileType def initialize(location, options) @file_location = location @options = options + @skipped_es = [] add_xsl_params_options # set output directories output = File.join(@options["collection_dir"], "output", @options["environment"]) @@ -35,6 +37,12 @@ def initialize(location, options) # script locations set in child classes end + def debug_info(error) + if options["verbose"] + puts "Backtrace: " << error.backtrace.join("\n ") + end + end + def filename(ext=true) if ext File.basename(@file_location) @@ -56,13 +64,15 @@ def post_es(es) begin transformed = transform_es rescue => e + debug_info(e) return { "error" => "Error transforming ES for #{self.filename(false)}: #{e.full_message}" } end if transformed && transformed.length > 0 transformed.each do |doc| if doc["identifier"].to_s.empty? || doc["title"].to_s.empty? - puts "skipping item without id or title".red - puts "check line ".red + doc.values.join("; ").strip.red + msg = "Skipping item without id or title: #{doc.values.join('; ').strip[0..100]}" + puts msg.yellow + @skipped_es << msg next end id = doc["identifier"] @@ -74,9 +84,17 @@ def post_es(es) # NOTE: If you need to do partial updates rather than replacement of doc # you will need to add _update at the end of this URL begin - RestClient.put("#{es.index_url}/_doc/#{id}", doc.to_json, @auth_header.merge({:content_type => :json }) ) + response = Datura::Helpers.es_http_request("PUT", "#{es.index_url}/_doc/#{id}", + body: doc.to_json, + headers: @auth_header.merge("Content-Type" => "application/json")) + raise "#{response.code} error posting to Elasticsearch: #{response.body}" unless response.code.start_with?("2") + rescue Errno::ECONNREFUSED, SocketError, Errno::ETIMEDOUT => e + error = "Could not connect to ElasticSearch at #{es.index_url}. " \ + "Confirm you have specified the correct environment " \ + "(currently: #{@options['environment']}. Use -e to specify an environment." rescue => e - error = "Error transforming or posting to ES for #{self.filename(false)}: #{e}" + debug_info(e) + error = "Error transforming or posting to ES for #{self.filename(false)}: #{e.message}" end else error = "Document #{id} did not validate against the elasticsearch schema" @@ -109,7 +127,8 @@ def post_solr(url=nil) return { "error" => "Error posting to Solr for #{self.filename}: #{res.body}" } end rescue => e - return { "error" => "Error posting to Solr for #{self.filename}: #{e.inspect}" } + debug_info(e) + return { "error" => "Error posting to Solr for #{self.filename}: #{e.message}" } end end @@ -147,9 +166,6 @@ def transform_es end return es_req rescue => e - puts "something went wrong transforming #{self.filename}" - puts e - puts e.backtrace raise e end end @@ -187,26 +203,35 @@ def add_xsl_params_options end end - # TODO can remove most of these parameters and grab them from instance variables def exec_xsl(input, xsl, ext, outpath=nil, params=nil) - saxon_params = CommonXml.stringify_params(params) - cmd = "saxon -s:#{input} -xsl:#{xsl}" - # TODO which way would we rather do this? - # cmd << " -o:#{outpath}/#{filename(false)}.#{ext}" if outpath - cmd << " #{saxon_params}" - cmd << " | tee #{outpath}/#{filename(false)}.#{ext}" if outpath - puts "using command #{cmd}" if @options["verbose"] - Open3.popen3(cmd) do |stdin, stdout, stderr| - out = stdout.read - err = stderr.read - if err.length > 0 - msg = "There was an error transforming #{filename}: #{err}" - return { "error" => msg } - else - puts "Successfully transformed #{filename}" - return { "doc" => out } + # build the python script path + python_script = File.join( + @options["datura_dir"], "lib", "datura", "python", "xslt_transform.py" + ) + # initialize the command as an array, then apppend xslt params + cmd = ["python3", python_script, "--input", input, "--xsl", xsl] + if params + params.each do |k, v| + cmd += ["--param", k.to_s, v.to_s] end end + # append output path and base output URI for xsl:result-document secondary outputs if configured + if outpath + cmd += ["--output", File.join(outpath, filename(false) + "." + ext)] + cmd += ["--base-output-uri", outpath] + end + puts "using command #{cmd.inspect}" if @options["verbose"] + # run the command + out, err, status = Open3.capture3(*cmd) + + # check for errors + if !status.success? || err.length > 0 + msg = "There was an error transforming #{filename}: #{err}" + return { "error" => msg } + else + puts "Successfully transformed #{filename}" + return { "doc" => out } + end end def pretty_json(json) @@ -222,4 +247,4 @@ def subdoc_xpaths # } end -end +end \ No newline at end of file diff --git a/lib/datura/file_types/file_csv.rb b/lib/datura/file_types/file_csv.rb index c3336cb54..532c26e3e 100644 --- a/lib/datura/file_types/file_csv.rb +++ b/lib/datura/file_types/file_csv.rb @@ -7,9 +7,19 @@ def initialize(file_location, options) @csv = read_csv(file_location, options["csv_encoding"]) end + # Builds one HTML file per CSV row. Respects the --csv-rows filter option: + # if @options["csv_rows"] is set, only rows with identifier matching that + # regex are written. Note: if overriding this method in a collection, call + # build_csv_row_filter / row_matches_filter? to preserve filter behavior. def build_html_from_csv + row_filter = build_csv_row_filter + if row_filter + puts "csv_rows filter active: only processing rows matching /#{@options["csv_rows"]}/".cyan + end @csv.each_with_index do |row, index| next if row.header_row? + # Skip rows that don't match the identifier filter (if one is active) + next if row_filter && !row_matches_filter?(row, row_filter) # Note: if overriding this function, it's recommended to use # a more specific identifier for each row of the CSV # but since this is a generic version, simply using the current iteration number @@ -64,16 +74,23 @@ def transform_es puts "transforming #{self.filename}" es_doc = [] + row_filter = build_csv_row_filter + if row_filter + puts "csv_rows filter active: only processing rows matching /#{@options["csv_rows"]}/".cyan + end + @csv.each do |row| - if !row.header_row? - row_to_es = row_to_es(@csv.headers, row) - if !row_to_es["identifier"].to_s.empty? && !row_to_es["title"].to_s.empty? - es_doc << row_to_es - else - puts "skipping item without id or title".red - puts "check line ".red + row.to_s.strip[0..400].red - next - end + next if row.header_row? + next if row_filter && !row_matches_filter?(row, row_filter) + + row_to_es = row_to_es(@csv.headers, row) + if !row_to_es["identifier"].to_s.empty? && !row_to_es["title"].to_s.empty? + es_doc << row_to_es + else + msg = "Skipping item without id or title: check line #{row.to_s.strip[0..200]}" + puts msg.yellow + @skipped_es << msg + next end end if @options["output"] @@ -89,6 +106,7 @@ def transform_iiif def transform_html puts "transforming #{self.filename} to HTML subdocuments" + # build_html_from_csv handles the --csv-rows filter internally build_html_from_csv # transform_html method is expected to send back a hash # but already wrote to filesystem so just sending back empty @@ -100,14 +118,20 @@ def transform_html # it will have to do! (transmississippi only collection so far) def transform_solr puts "transforming #{self.filename}" + # Build and apply the identifier filter, if --csv-rows was passed + row_filter = build_csv_row_filter + if row_filter + puts "csv_rows filter active: only processing rows matching /#{@options["csv_rows"]}/".cyan + end solr_doc = Nokogiri::XML("") @csv.each do |row| - if !row.header_row? - doc = Nokogiri::XML::Node.new("doc", solr_doc) - # row_to_solr should return an XML::Node object with children - doc = row_to_solr(doc, @csv.headers, row) - solr_doc.at_css("add").add_child(doc) - end + next if row.header_row? + # Skip rows that don't match the identifier filter (if one is active) + next if row_filter && !row_matches_filter?(row, row_filter) + doc = Nokogiri::XML::Node.new("doc", solr_doc) + # row_to_solr should return an XML::Node object with children + doc = row_to_solr(doc, @csv.headers, row) + solr_doc.at_css("add").add_child(doc) end # Uncomment to debug # puts solr_doc.root.to_xml @@ -123,4 +147,21 @@ def write_html_to_file(builder, index) puts "writing to #{filepath}" if @options["verbose"] File.open(filepath, "w") { |f| f.write(builder.to_xml) } end + + private + + def build_csv_row_filter + return nil unless @options["csv_rows"] + + begin + Regexp.new(@options["csv_rows"]) + rescue RegexpError => e + raise ArgumentError, "Invalid regex '#{options["csv_rows"]}': #{e.message}" + end + end + + def row_matches_filter?(row, filter) + id = row["id"] || row["identifier"] || row["Identifier"] || "" + !!filter.match(id) + end end diff --git a/lib/datura/file_types/file_custom.rb b/lib/datura/file_types/file_custom.rb index 28725d02b..3fba3cfac 100644 --- a/lib/datura/file_types/file_custom.rb +++ b/lib/datura/file_types/file_custom.rb @@ -1,8 +1,6 @@ require_relative "../helpers.rb" require_relative "../file_type.rb" -require "rest-client" - class FileCustom < FileType attr_reader :es_req, :format diff --git a/lib/datura/file_types/file_ead.rb b/lib/datura/file_types/file_ead.rb index a809ab9b4..7c0fe5c81 100644 --- a/lib/datura/file_types/file_ead.rb +++ b/lib/datura/file_types/file_ead.rb @@ -1,7 +1,6 @@ require_relative "../helpers.rb" require_relative "../file_type.rb" require_relative "../solr_poster.rb" -require "rest-client" class FileEad < FileType # TODO we could include the tei_to_es and other modules directly here diff --git a/lib/datura/file_types/file_html.rb b/lib/datura/file_types/file_html.rb index 4835ba40c..cd820c8f5 100644 --- a/lib/datura/file_types/file_html.rb +++ b/lib/datura/file_types/file_html.rb @@ -1,7 +1,6 @@ require_relative "../helpers.rb" require_relative "../file_type.rb" require_relative "../solr_poster.rb" -require "rest-client" class FileHtml < FileType attr_reader :es_req diff --git a/lib/datura/file_types/file_tei.rb b/lib/datura/file_types/file_tei.rb index d756450f2..463b1ba48 100644 --- a/lib/datura/file_types/file_tei.rb +++ b/lib/datura/file_types/file_tei.rb @@ -1,7 +1,6 @@ require_relative "../helpers.rb" require_relative "../file_type.rb" require_relative "../solr_poster.rb" -require "rest-client" class FileTei < FileType # TODO we could include the tei_to_es and other modules directly here diff --git a/lib/datura/file_types/file_webs.rb b/lib/datura/file_types/file_webs.rb index 62c8dc5d9..28bfcee49 100644 --- a/lib/datura/file_types/file_webs.rb +++ b/lib/datura/file_types/file_webs.rb @@ -1,8 +1,6 @@ require_relative "../helpers.rb" require_relative "../file_type.rb" -require "rest-client" - class FileWebs < FileType attr_reader :es_req diff --git a/lib/datura/helpers.rb b/lib/datura/helpers.rb index bcc245fff..baf9f9724 100644 --- a/lib/datura/helpers.rb +++ b/lib/datura/helpers.rb @@ -1,7 +1,9 @@ require 'fileutils' require 'net/http' require 'nokogiri' +require 'shellwords' require 'yaml' +require 'uri' module Datura::Helpers @@ -74,8 +76,10 @@ def self.get_input(original_input, msg) # get_url # sends a request to a given url def self.get_url(url) - url = URI.parse(url) - Net::HTTP.get_response(url) + uri = URI.parse(url) + http = Net::HTTP.new(uri.host, uri.port) + http.use_ssl = true if uri.scheme == "https" + http.request(Net::HTTP::Get.new(uri.request_uri)) end # make_dirs @@ -109,7 +113,7 @@ def self.normalize_space(abnormal) def self.regex_files(files, regex=nil) array = files.nil? ? [] : files if !files.nil? && !regex.nil? - exp = Regexp.new(regex) + exp = validate_regex(regex, "--regex") array = files.select do |file| file_name = File.basename(file, ".*") match = exp.match(file_name) @@ -119,6 +123,70 @@ def self.regex_files(files, regex=nil) array end + # proceed_files + # returns all files from the first file matching the proceed regex onward (inclusive), + # preserving directory order and sorting alphabetically within each directory. + # Exits with an error if the regex matches zero or more than one file. + # params: files (array of file paths), regex (string) + # returns: array + def self.proceed_files(files, regex) + # Preserve directory order from input list; sort alphabetically within each directory + sorted = files.group_by { |f| File.dirname(f) } + .sort_by { |dir, _| dir } + .flat_map { |_, fs| fs.sort_by { |f| File.basename(f, ".*") } } + exp = validate_regex(regex, "--proceed") + matches = sorted.select { |f| exp.match(File.basename(f, ".*")) } + + if matches.empty? + puts "ERROR: --proceed regex '#{regex}' matched no files. Exiting.".red + exit 1 + elsif matches.length > 1 + names = matches.map { |f| File.basename(f, ".*") }.join(", ") + puts "ERROR: --proceed regex '#{regex}' matched #{matches.length} files (#{names}). Refine your regex to match exactly one file. Exiting.".red + exit 1 + end + + proceed_index = sorted.index(matches.first) + sorted[proceed_index..] + end + + # checkpoint_path + # returns the full path to the proceed checkpoint file + # params: options (hash with "collection_dir" and "environment" keys) + # returns: string + def self.checkpoint_path(options) + File.join(options["collection_dir"], "logs", "proceed_#{options["environment"]}") + end + + # read_checkpoint + # reads the proceed checkpoint file and returns its contents + # params: options (hash) + # returns: string (basename without extension) or nil if file missing/empty + def self.read_checkpoint(options) + path = checkpoint_path(options) + return nil unless File.exist?(path) + content = File.read(path).strip + content.empty? ? nil : content + end + + # write_checkpoint + # writes the basename of the last posted file to the checkpoint file + # params: basename (string, filename without extension), options (hash) + # returns: nil + def self.write_checkpoint(basename, options) + path = checkpoint_path(options) + File.write(path, "#{basename}\n") + end + + # clear_checkpoint + # writes empty content to the checkpoint file, signaling no resume point + # params: options (hash) + # returns: nil + def self.clear_checkpoint(options) + path = checkpoint_path(options) + File.write(path, "") + end + # should_update? # determines if a user has changed a file since specified date # params: file (string path), since_date (Time format or nil) @@ -134,10 +202,76 @@ def self.should_update?(file, since_date=nil) end end + # validate_regex + # compiles a regex string; prints a readable error and exits if invalid + # params: regex (string), flag (string, e.g. "--regex" or "--proceed") + # returns: Regexp + def self.validate_regex(regex, flag) + Regexp.new(regex) + rescue RegexpError => e + puts "ERROR: Invalid regex for #{flag} '#{regex}': #{e.message}".red + exit 1 + end + def self.construct_auth_header(options) username = options["es_user"] password = options["es_password"] - { "Authorization" => "Basic #{Base64::encode64("#{username}:#{password}")}" } + + if (username || password) && options["es_path"]&.start_with?("http://") + warn "[SECURITY WARNING] ES credentials are set but es_path uses unencrypted HTTP. " \ + "Credentials will be transmitted in cleartext. Use HTTPS in production." + end + + { "Authorization" => "Basic #{Base64::strict_encode64("#{username}:#{password}")}" } + end + + def self.es_http_request(method, url, body: nil, headers: {}) + uri = URI.parse(url) + http = Net::HTTP.new(uri.host, uri.port) + http.use_ssl = (uri.scheme == "https") + req_class = { + "GET" => Net::HTTP::Get, + "PUT" => Net::HTTP::Put, + "POST" => Net::HTTP::Post, + "DELETE" => Net::HTTP::Delete + }.fetch(method.upcase) + req = req_class.new(uri.request_uri) + headers.each { |k, v| req[k.to_s] = v } + req.body = body if body + + http.request(req) + end + + def self.run_omeka_script(script_path, options) + ''' + Build and run a Python Omeka posting script. + + Handles common CLI flag forwarding (-e, -r, -m). + Called by bin/post_omeka and bin/post_omeka_html. + + Parameters: + * script_path - absolute path to the Python script to run + * options - hash of parsed CLI options ("environment", "regex", "media_skip") + ''' + unless File.exist?(script_path) + puts "Omeka script not found at #{script_path}".red + return + end + command = ["python3", script_path] + command.append("-c", Shellwords.escape(options["csv_rows"])) if options["csv_rows"] + command.append("-e", Shellwords.escape(options["environment"])) if options["environment"] + command.append("-f", Shellwords.escape(options["format"])) if options["format"] + command.append("-r", Shellwords.escape(options["regex"])) if options["regex"] + command.append("-u", Shellwords.escape(options["update_time"])) if options["update_time"] + command.append("-m") if options["media_skip"] + # -p may be given with no value (nil, meaning "use last checkpoint") or + # with a regex string. key? distinguishes "not provided" from nil. + if options.key?("proceed") + command.append("-p") + command.append(Shellwords.escape(options["proceed"])) if options["proceed"] + end + command.append("-j") if options["json_output"] + system(*command) end end diff --git a/lib/datura/options.rb b/lib/datura/options.rb index c478ced42..c37db9916 100644 --- a/lib/datura/options.rb +++ b/lib/datura/options.rb @@ -64,7 +64,7 @@ def read_all_configs(general, collection) def read_config(path) if File.file?(path) begin - return YAML.load_file(path) + return YAML.safe_load_file(path, permitted_classes: [Symbol]) rescue Exception => e puts "There was an error reading config file #{path}: #{e.message}" end diff --git a/lib/datura/parser_options/post.rb b/lib/datura/parser_options/post.rb index b6e48880f..f3eb06793 100644 --- a/lib/datura/parser_options/post.rb +++ b/lib/datura/parser_options/post.rb @@ -39,11 +39,22 @@ def self.post_params options["commit"] = false end + options["csv_rows"] = nil + opts.on('-c', '--csv-rows [input]', + 'Only process CSV rows whose identifier (id/identifier column) matches this regex.') do |input| + options["csv_rows"] = input + end + options["output"] = false opts.on('-o', '--output', 'Write solr and elasticsearch docs to file') do options["output"] = true end + options["proceed"] = false + opts.on('-p', '--proceed [input]', 'Proceed with posting from (and including) the file matching this regex') do |input| + options["proceed"] = input + end + options["regex"] = nil opts.on('-r', '--regex [input]', 'Only post files matching this regex') do |input| options["regex"] = input @@ -61,7 +72,6 @@ def self.post_params puts "'today', date (2015-01-01), or date and time (2015-01-01T18:24)".light_yellow exit else - # TODO should verify that this is a correct date and turn it into a time object datetime = timify(input) if datetime.nil? exit diff --git a/lib/datura/python/api_fields.py b/lib/datura/python/api_fields.py index 9273fee9b..c604983ee 100644 --- a/lib/datura/python/api_fields.py +++ b/lib/datura/python/api_fields.py @@ -1,285 +1,331 @@ +""" +api_fields.py + +Transforms a single Datura-generated JSON item into the format expected by the +Omeka S REST API, and resolves inter-item relationships (links) by looking up +CDRH identifiers in the live Omeka instance. + +The two primary entry points called by json_to_omeka.py are: + prepare_item(ctx, json_item, existing_item) — build or update item metadata + link_records(ctx, json_item, existing_item) — resolve and attach relationships + +All functions that need API access or configuration now receive a ctx +(OmekaContext) parameter. The property ID cache on ctx (ctx.get_property_id()) +means each Omeka term is looked up only once per run rather than once per field per item. +""" + import json +import logging import re +import sys + import omeka from datetime import datetime -from field_definitions import get_fields -def build_item_dict(json, existing_item): - """Takes in JSON with CDRH API fields and an existing API item from Omeka S in format. - Returns Omeka API item (in JSON format) updated with values from new CDHR schema for Omeka.""" +# Module-level logger so that log records from this module are identifiable +# by name in the output stream. +logger = logging.getLogger(__name__) + + +def prepare_item(ctx, json_item, existing_item=None): + """ + Build a complete Omeka item dict from a Datura JSON record. + + Iterates over the field manifest declared on ctx.fields, where each entry + specifies an Omeka property term, the extractor method name to call on + ctx.fields, and the Omeka datatype. The manifest is defined by + FieldDefinitions.field_manifest and may be extended by a collection's + CustomFields subclass to add collection-specific properties. + + Parameters: + * ctx - OmekaContext providing config and the property ID cache + * json_item - raw JSON item dict from the Datura ES output + * existing_item - existing Omeka item dict to update in-place, or None + when creating a new item (an empty dict is used instead) + + Returns the built or updated item dict, ready for payload preparation. + Raises ValueError (caught by the caller) if field extraction fails in an + unexpected way. + """ try: - fields = get_fields() built_item = existing_item if existing_item else {} - update_item_value(built_item, "dcterms:title", fields.title(json)) - update_item_value(built_item, "dcterms:identifier", fields.identifier(json)) - update_item_value(built_item, "dh:collection", fields.collection(json)) - update_item_value(built_item, "dh:category", fields.category(json)) - update_item_value(built_item, "dh:category2", fields.category2(json)) - update_item_value(built_item, "dh:uriData", fields.uriData(json), "uri") - update_item_value(built_item, "dcterms:type", fields.dcterms_type(json)) - update_item_value(built_item, "dcterms:creator", fields.creator(json)) - update_item_value(built_item, "dcterms:contributor", fields.contributor(json)) - update_item_value(built_item, "dcterms:date", fields.date(json), "numeric:timestamp") - update_item_value(built_item, "dh:dateDisplay", fields.dateDisplay(json)) - update_item_value(built_item, "dh:dateYear", fields.dateYear(json)) - update_item_value(built_item, "dcterms:description", fields.description(json)) - update_item_value(built_item, "dcterms:format", fields.dcterms_format(json)) - update_item_value(built_item, "dcterms:relation", fields.relation(json)) - update_item_value(built_item, "dcterms:publisher", fields.publisher(json)) - update_item_value(built_item, "dh:biblID", fields.biblID(json)) - update_item_value(built_item, "tei:biblTitle", fields.biblTitle(json)) - update_item_value(built_item, "tei:biblPubPlace", fields.biblPubPlace(json)) - update_item_value(built_item, "bibo:issue", fields.issue(json)) - update_item_value(built_item, "bibo:pageStart", fields.pageStart(json)) - update_item_value(built_item, "bibo:pageEnd", fields.pageEnd(json)) - update_item_value(built_item, "bibo:section", fields.section(json)) - update_item_value(built_item, "bibo:volume", fields.volume(json)) - update_item_value(built_item, "tei:biblTitleA", fields.biblTitleA(json)) - update_item_value(built_item, "tei:biblTitleM", fields.biblTitleM(json)) - update_item_value(built_item, "tei:biblTitleJ", fields.biblTitleJ(json)) - update_item_value(built_item, "dcterms:rightsHolder", fields.rightsHolder(json)) - update_item_value(built_item, "dcterms:license", fields.license(json)) - update_item_value(built_item, "dcterms:subject", fields.subject(json)) - update_item_value(built_item, "dh:topic", fields.topic(json)) - update_item_value(built_item, "dh:category3", fields.category3(json)) - update_item_value(built_item, "dh:category4", fields.category4(json)) - update_item_value(built_item, "dh:category5", fields.category5(json)) - update_item_value(built_item, "dh:note", fields.note(json)) - update_item_value(built_item, "dcterms:abstract", fields.abstract(json)) - update_item_value(built_item, "dh:keyword", fields.keyword(json)) - update_item_value(built_item, "dh:keyword2", fields.keyword2(json)) - update_item_value(built_item, "dh:keyword3", fields.keyword3(json)) - update_item_value(built_item, "dh:keyword4", fields.keyword4(json)) - update_item_value(built_item, "dh:keyword5", fields.keyword5(json)) - update_item_value(built_item, "dcterms:source", fields.source(json)) - update_item_value(built_item, "dcterms:medium", fields.medium(json)) - update_item_value(built_item, "dcterms:extent", fields.extent(json)) - update_item_value(built_item, "dcterms:language", fields.language(json)) - update_item_value(built_item, "dh:box", fields.box(json)) - update_item_value(built_item, "dh:folder", fields.folder(json)) - update_item_value(built_item, "foaf:name", fields.name(json)) - update_item_value(built_item, "dh:spatial_short_name", fields.spatial_short_name(json)) - update_item_value(built_item, "tei:correspSentName", fields.correspSentName(json)) - update_item_value(built_item, "tei:correspSentPlace", fields.correspSentPlace(json)) - update_item_value(built_item, "tei:correspSentDate", fields.correspSentDate(json), "numeric:timestamp") - update_item_value(built_item, "tei:correspDeliveredName", fields.correspDeliveredName(json)) - update_item_value(built_item, "tei:correspDeliveredPlace", fields.correspDeliveredPlace(json)) - update_item_value(built_item, "tei:correspDeliveredDate", fields.correspDeliveredDate(json), "numeric:timestamp") - update_item_value(built_item, "tei:distributor", fields.distributor(json)) - update_item_value(built_item, "tei:authority", fields.authority(json)) - update_item_value(built_item, "tei:biblNote", fields.biblNote(json)) - update_item_value(built_item, "dh:annotationsText", fields.annotationsText(json)) - update_item_value(built_item, "dh:itemText", fields.itemText(json)) + _update = ctx._fn_update_item_value or update_item_value + for omeka_term, method_name, datatype in ctx.fields.field_manifest(): + value = getattr(ctx.fields, method_name)(json_item) + _update(ctx, built_item, omeka_term, value, datatype) return built_item - except ValueError: - breakpoint() + except ValueError as e: + logger.error("ValueError building item dict: %s", e) + raise + + +def link_records(ctx, json_item, existing_item): + """ + Resolve inter-item relationships for a single item and attach them to the + existing Omeka item dict. + + Each relationship type (has_part, is_part_of, etc.) is handled in its own + try/except block. A missing or None field in the JSON is expected for most + items — these are caught as KeyError or TypeError and logged at DEBUG level + so they do not pollute the run log. Genuine API failures in + link_item_record() will surface as exceptions and should be caught by the + caller in json_to_omeka.py. + + Parameters: + * ctx - OmekaContext providing the API client and item_set_id + * json_item - raw JSON item dict from the Datura ES output + * existing_item - the current Omeka item dict, deepcopied by the caller + + Returns the updated existing_item dict with relationship fields populated. -#TODO change item linking for JSON and new API -def link_item(json_item, existing_item): + """ + identifier = json_item.get("identifier") + _link = ctx._fn_link_item_record or link_item_record - #has_part try: part_ids = [part['id'] for part in json_item["has_part"]] - link_item_record(existing_item, "dcterms:hasPart", part_ids) - except Exception: - pass - #is_part_of + _link(ctx, existing_item, "dcterms:hasPart", part_ids) + except (KeyError, TypeError) as e: + logger.debug("No has_part data for %s: %s", identifier, e) + try: - link_item_record(existing_item, "dcterms:isPartOf", json_item["is_part_of"]["id"]) - except Exception: - pass - #has_relation + _link(ctx, existing_item, "dcterms:isPartOf", json_item["is_part_of"]["id"]) + except (KeyError, TypeError) as e: + logger.debug("No is_part_of data for %s: %s", identifier, e) + try: - link_item_record(existing_item, "dcterms:relation", json_item["has_relation"]["id"]) - except Exception: - pass - #previous + _link(ctx, existing_item, "dcterms:relation", json_item["has_relation"]["id"]) + except (KeyError, TypeError) as e: + logger.debug("No has_relation data for %s: %s", identifier, e) + try: - link_item_record(existing_item, "dh:orderPrev", json_item["previous_item"]["id"]) - except Exception: - pass - #next + _link(ctx, existing_item, "dh:orderPrev", json_item["previous_item"]["id"]) + except (KeyError, TypeError) as e: + logger.debug("No previous_item data for %s: %s", identifier, e) + try: - link_item_record(existing_item, "dh:orderNext", json_item["next_item"]["id"]) - except Exception: - pass + _link(ctx, existing_item, "dh:orderNext", json_item["next_item"]["id"]) + except (KeyError, TypeError) as e: + logger.debug("No next_item data for %s: %s", identifier, e) + try: - link_item_record(existing_item, "tei:correspNext", json_item["correspNext_omeka_s"]) - except Exception: - pass + _link(ctx, existing_item, "tei:correspNext", json_item["correspNext_omeka_s"]) + except (KeyError, TypeError) as e: + logger.debug("No correspNext_omeka_s data for %s: %s", identifier, e) + try: - link_item_record(existing_item, "tei:correspPrev", json_item["correspPrev_omeka_s"]) - except Exception: - pass + _link(ctx, existing_item, "tei:correspPrev", json_item["correspPrev_omeka_s"]) + except (KeyError, TypeError) as e: + logger.debug("No correspPrev_omeka_s data for %s: %s", identifier, e) + return existing_item -def prepare_item(row, existing_item = None): - item_dict = build_item_dict(row, existing_item) - # TODO add conditional logic for different templates? - return item_dict - -def link_records(row, existing_item): - item_dict = link_item(row, existing_item) - # TODO add conditional logic? - return item_dict - -def get_json_value(row, name): - if len(row[name]) > 0: - if row[name].startswith('["'): - return json.loads(row[name]) - elif ";;;" in row[name]: - return row[name].split(";;;") - else: - return row[name] - else: - return row[name] - -def update_item_value(item, key, value, datatype="literal"): + +def update_item_value(ctx, item, key, value, datatype="literal"): """ - takes in JSON representation of API item, the field name, the value to add or update, and a datatype (defaults to "literal") - value may be in string format or list. Should be able to modify existing values and update new ones. (Note that there are still issues with updating fields - returns the JSON hash with the updated value + Set or replace a property on an Omeka item dict. + + Clears the existing value list for the key (if any) and writes the new + value(s). This ensures that re-running the ingest for an existing item + replaces stale values rather than appending duplicates. + + If value is None or an empty list, the key is initialised to [] and no + formatted values are added — this effectively clears the field in Omeka + when the item is PUT back. + + Parameters: + * ctx - OmekaContext (passed through to add_formatted_value) + * item - the Omeka item dict being built + * key - Omeka property term string, e.g. "dcterms:title" + * value - the value to set; may be a string, int, float, or list. + None and empty list are treated as "no value". + * datatype - Omeka data type string (default "literal"). Use + "uri" for URLs or "numeric:timestamp" for dates. """ - #clear the existing values of the key, or initialize it if it is new + # Always reset the key so that old values from a prior ingest are not + # carried forward when the source data no longer has a value for this field. item[key] = [] - if type(value) in [str, int, float]: - item = add_formatted_value(item, key, value, datatype) - elif type(value) == list: - # make sure values are unique - value = list(set(value)) - value = [v for v in value if v is not None] # remove None values from values + + if isinstance(value, (str, int, float)): + item = add_formatted_value(ctx, item, key, value, datatype) + elif isinstance(value, list): + # List entries may be strings or dicts (e.g. contributor returns + # [{"name": "...", "id": "..."}]). For dicts, the dedup key compares + # based on id when id is present or creates a sorted tuple of all + # items so that two dicts are only considered duplicates when every + # key-value pair matches. + seen = {} for v in value: - item = add_formatted_value(item, key, v, datatype) - -def add_formatted_value(item, key, value, datatype, label=""): - # takes in item, key, value, and datatype, returns item with key set or added to value, and formatted in the format Omeka S - # expects as indicated in the template - # used when adding a new value that is not already in the Omeka JSON, so that Omeka will properly update the value - # this comes up - # literal values should be str + if v is None: + continue + dedup_key = v.get("id") or tuple(sorted(v.items())) if isinstance(v,dict) else v + if dedup_key not in seen: + seen[dedup_key] = v + for v in seen.values(): + display = v.get("name") if isinstance(v,dict) else v + if display is None: + continue + else: + item = add_formatted_value(ctx, item, key, display, datatype) + + +def add_formatted_value(ctx, item, key, value, datatype, label=""): + """ + Format a single value and append it to the property list on an item dict. + + Calls ctx.get_property_id() to obtain the numeric Omeka property ID for + the given term. The cache on ctx ensures this API call is made at most once + per unique term per run. + + Parameters: + * ctx - OmekaContext; provides get_property_id() and the API client + * item - the Omeka item dict being built + * key - Omeka property term string, e.g. "dcterms:title" + * value - the scalar value to format + * datatype - Omeka data type string, e.g. "literal", "uri", + "numeric:timestamp" + * label - optional display label for URI values + + Returns the item dict with the new value appended to item[key]. + """ + # Coerce to string for literal values to avoid sending a bare int or float + # in the API payload, which Omeka S may reject. if datatype == "literal": value = str(value) - prop_id = omeka.omeka_auth.get_property_id(key) + + # Look up the property ID via the cache. + prop_id = ctx.get_property_id(key) + prop_value = { "value": value, - "type": datatype + "type": datatype, } - formatted = omeka.prepare_property_value(prop_value, prop_id, label) - if key in item and type(item[key]) == list: + formatted = ctx.client.prepare_property_value(prop_value, prop_id, label) + + if key in ctx.fields.private_fields(): + formatted["is_public"] = False + + if key in item and isinstance(item[key],list): item[key].append(formatted) else: item[key] = [formatted] + return item -def get_matching_ids_from_markdown(row, field): - # takes in an array of strings in markdown format, which include CDRH IDs - # returns an array of just the IDs - if row[field]: - markdown_values = sorted(get_json_value(row, field)) - ids = [] - if markdown_values: - #should be either single value or array - if type(markdown_values) == str: - match = re.search(r"\]\((.*)\)", markdown_values) - if match: - id_no = match.group(1) - ids.append(id_no) - else: - for value in markdown_values: - #parse with regex to get ids - match = re.search(r"\]\((.*)\)", value) - if match: - id_no = match.group(1) - ids.append(id_no) - if len(ids) > 1: - ids = list(filter(None, ids)) - return ids - - else: - return [] - -def get_matching_names_from_markdown(row, field): - # takes in an array of strings in markdown format, which include names - # returns an array of just the names - # filters out the ones that have a corresponding id, it is not necessary to get their names - if row[field]: - markdown_values = get_json_value(row, field) - names = [] - - if markdown_values: - #should be either single value or array - if type(markdown_values) == str: - name_match = re.search(r"\[(.*?)\]", markdown_values) - # filter out entries that have ids - id_match = re.search(r"\]\((.*)\)", markdown_values) - if name_match and not id_match.group(1): - name = name_match.group(1) - names.append(name) - else: - for value in markdown_values: - #parse with regex to get ids - name_match = re.search(r"\[(.*?)\]", value) - id_match = re.search(r"\]\((.*)\)", value) - if name_match and not id_match.group(1): - name = name_match.group(1) - names.append(name) - return names - else: - return [] +def get_omeka_ids(ctx, lookup_values, filter_property, item_set_id="ctx_default"): + """ + Resolve a list of lookup values to Omeka numeric item IDs. + + For each lookup value, queries the Omeka API to find the matching item. + Used during the linking pass to convert CDRH identifiers into the Omeka IDs + required for resource:item links. -def get_omeka_ids(lookup_values, filter_property, item_set_id = None): - item_set_id = omeka.get_item_set() + Parameters: + * ctx - OmekaContext providing the API client and item_set_id + * lookup_values - a single value or list of values to look up; typically + CDRH identifier strings but may be Omeka IDs directly + when filter_property is "o:id" + * filter_property - the Omeka property to match against, e.g. + "dcterms:identifier" or "o:id" + * item_set_id - restricts the search to a specific Omeka item set. + Defaults to ctx.item_set_id (the current collection). + Pass None to search across all item sets — useful when + the target items (e.g. a personography) live in a + separate item set from the collection being ingested. + + Returns a list of integer Omeka item IDs for all successfully resolved values. + Logs a warning for values that cannot be resolved. + """ omeka_ids = [] - #lookup_values are usually a list of cdrh_ids, but may be another value + + # Resolve the sentinel to ctx.item_set_id so existing callers are unaffected. + resolved_item_set_id = ctx.item_set_id if item_set_id == "ctx_default" else item_set_id + + # Normalise a single value to a list for uniform iteration. lookup_values = [lookup_values] if not isinstance(lookup_values, list) else lookup_values + for lookup_value in lookup_values: + # Skip blank or None values — these are common when optional relation + # fields are absent in some records but not others. if not lookup_value or lookup_value == '': continue + if filter_property == "o:id": + # The value is already an Omeka ID; cast to int and add directly. omeka_ids.append(int(lookup_value)) else: - match = omeka.omeka_auth.filter_items_by_property(filter_property = filter_property, filter_value = lookup_value, item_set_id=item_set_id) + match = ctx.client.filter_items_by_property( + filter_property=filter_property, + filter_value=lookup_value, + item_set_id=resolved_item_set_id, + ) if match["total_results"] >= 1: if match["total_results"] > 1: - print(f"warning: multiple matches for {lookup_value}, taking first match") - breakpoint() - omeka_id = match['results'][0]["o:id"] - omeka_ids.append(omeka_id) + # Multiple matches indicate a data integrity issue; take + # the first result and log a warning for investigation. + logger.warning( + "Multiple matches for %r, taking first result", lookup_value + ) + omeka_ids.append(match['results'][0]["o:id"]) else: - print(f"Unable to link {lookup_value}, no matches") + logger.warning("Unable to link %r: no matching items found", lookup_value) + return omeka_ids +def link_item_record(ctx, item, key, values, item_set=False, filter_property="dcterms:identifier"): + """ + Resolve lookup values to Omeka IDs and attach them as resource links on + the item dict. + + Clears the existing value list for the key before writing, so re-running + this function replaces stale links rather than appending duplicates. -def link_item_record(item, key, values, item_set=False, filter_property = "dcterms:identifier"): - omeka_ids = values if item_set else get_omeka_ids(values, filter_property) - #dedupe + Parameters: + * ctx - OmekaContext providing the API client + * item - the Omeka item dict being built + * key - Omeka property term for this relationship, + e.g. "dcterms:hasPart" or "dh:orderNext" + * values - lookup values to resolve; either already-resolved Omeka + IDs (when item_set=True) or CDRH identifiers to look up + * item_set - if True, treat values as Omeka item set IDs rather than + item IDs; uses "resource:itemset" type and adds the + extra fields required by the item-sets plugin + * filter_property - the Omeka property to use when looking up items by value; + defaults to "dcterms:identifier" + + Returns the updated item dict. + """ + # When item_set=True the caller has already resolved the IDs; otherwise + # resolve them from CDRH identifiers via the API. + omeka_ids = values if item_set else get_omeka_ids(ctx, values, filter_property) + + # Deduplicate while preserving order (dict.fromkeys is stable in Python 3.7+). omeka_ids = list(dict.fromkeys(omeka_ids)) - prop_id = omeka.omeka_auth.get_property_id(key) - #always clear items + + # Look up the property ID via the cache. + prop_id = ctx.get_property_id(key) + + # Always clear the existing values for this relationship field so that + # stale links from a prior ingest are removed. item[key] = [] resource_type = "resource:itemset" if item_set else "resource:item" + for omeka_id in omeka_ids: - #make sure item isn't already linked, to avoid duplicates - if not item[key] or not omeka_id in [value.get("value_resource_id") for value in item[key]]: - prop_value = { - "type": resource_type, - "value": omeka_id - } - formatted = omeka.omeka_auth.prepare_property_value(prop_value, prop_id) - #different format for item sets, plugin doesn't do it automatically - if item_set: - formatted['@id'] = f'{omeka.omeka_auth.api_url}/item_sets/{omeka_id}' - formatted['value_resource_id'] = omeka_id - formatted["value_resource_name"] = "item_sets" - item[key].append(formatted) - return item + prop_value = { + "type": resource_type, + "value": omeka_id, + } + formatted = ctx.client.prepare_property_value(prop_value, prop_id) + + if item_set: + # The item-sets plugin requires these extra fields in addition + # to what prepare_property_value generates. + formatted["@id"] = f"{ctx.client.api_url}/item_sets/{omeka_id}" + formatted["value_resource_id"] = omeka_id + formatted["value_resource_name"] = "item_sets" + + item[key].append(formatted) -def build_citation(row): - # TODO format the date better - if row["publisher"]: - return f""" - "{row["title"]}", {json.loads(row["publisher"])[0]}, {row["Article Date (formatted)"]}, {row["Source page no"]}. - Accessed {row["Source access date"]}. {row["Source link"]}. - """ + return item \ No newline at end of file diff --git a/lib/datura/python/field_definitions.py b/lib/datura/python/field_definitions.py index 9dc1e851e..34f16ce5e 100644 --- a/lib/datura/python/field_definitions.py +++ b/lib/datura/python/field_definitions.py @@ -1,10 +1,121 @@ -import sys -import os +import importlib.util +import logging from datetime import datetime -import omeka +from pathlib import Path + +logger = logging.getLogger(__name__) class FieldDefinitions: - #these are the default field definitions, which may be overridden in specific projects + """ + Default field extraction patterns for the Omeka S ingestion pipeline. + + Each method receives the raw JSON item dict (a single record from the + Datura-generated ES output file) and returns the value to be posted for + that Omeka property, or None if the field is absent. + + To override any method for a specific collection, copy + omeka_overrides_example.py to scripts/python/omeka_overrides.py in the + collection repository and subclass FieldDefinitions there. The get_fields() + factory below will load the override class automatically. + """ + + def __init__(self, omeka_data_base=""): + """ + Parameters: + * omeka_data_base - base URL used to construct media URIs in uriData(). + Passed in from OmekaContext.omeka_data_base. + Defaults to "" (empty string) so that instantiation + without arguments is safe in tests. + """ + # Stored as a private attribute and accessed only by uriData(). + self._omeka_data_base = omeka_data_base + + def private_fields(self): + """ + Return a set of Omeka property terms whose values should be posted with + is_public=False. Override in field_overrides.py to mark specific fields + as private for this collection. + + Example: + def private_fields(self): + return {"dcterms:identifier", "dcterms:source"} + """ + return set() + + def field_manifest(self): + """ + Declare the ordered list of Omeka property mappings for this collection. + + Each entry is a (omeka_term, method_name, datatype) triple: + * omeka_term - Omeka S property term string, e.g. "dcterms:title" + * method_name - name of the extractor method on this FieldDefinitions + instance to call for each JSON item + * datatype - Omeka data type string + + prepare_item() in api_fields.py iterates this manifest to build the + item payload, calling getattr(ctx.fields, method_name)(json_item) for each + entry. Overriding field_manifest in a CustomFields subclass is the + recommended way to add collection-specific Omeka properties without + touching prepare_item() itself. + """ + return [ + ("dcterms:title", "title", "literal"), + ("dcterms:identifier", "identifier", "literal"), + ("dh:collection", "collection", "literal"), + ("dh:category", "category", "literal"), + ("dh:category2", "category2", "literal"), + ("dh:uriData", "uriData", "uri"), + ("dcterms:type", "dcterms_type", "literal"), + ("dcterms:creator", "creator", "literal"), + ("dcterms:contributor", "contributor", "literal"), + ("dcterms:date", "date", "numeric:timestamp"), + ("dh:dateDisplay", "dateDisplay", "literal"), + ("dh:dateYear", "dateYear", "literal"), + ("dcterms:description", "description", "literal"), + ("dcterms:format", "dcterms_format", "literal"), + ("dcterms:relation", "relation", "literal"), + ("dcterms:publisher", "publisher", "literal"), + ("dh:biblID", "biblID", "literal"), + ("tei:biblTitle", "biblTitle", "literal"), + ("tei:biblPubPlace", "biblPubPlace", "literal"), + ("bibo:issue", "issue", "literal"), + ("bibo:pageStart", "pageStart", "literal"), + ("bibo:pageEnd", "pageEnd", "literal"), + ("bibo:section", "section", "literal"), + ("bibo:volume", "volume", "literal"), + ("tei:biblTitleA", "biblTitleA", "literal"), + ("tei:biblTitleM", "biblTitleM", "literal"), + ("tei:biblTitleJ", "biblTitleJ", "literal"), + ("dcterms:rightsHolder", "rightsHolder", "literal"), + ("dcterms:license", "license", "literal"), + ("dcterms:subject", "subject", "literal"), + ("dh:topic", "topic", "literal"), + ("dh:category3", "category3", "literal"), + ("dh:category4", "category4", "literal"), + ("dh:category5", "category5", "literal"), + ("dh:note", "note", "literal"), + ("dcterms:abstract", "abstract", "literal"), + ("dh:keyword", "keyword", "literal"), + ("dh:keyword2", "keyword2", "literal"), + ("dh:keyword3", "keyword3", "literal"), + ("dh:keyword4", "keyword4", "literal"), + ("dh:keyword5", "keyword5", "literal"), + ("dcterms:source", "source", "literal"), + ("dcterms:medium", "medium", "literal"), + ("dcterms:extent", "extent", "literal"), + ("dcterms:language", "language", "literal"), + ("dh:box", "box", "literal"), + ("dh:folder", "folder", "literal"), + ("foaf:name", "name", "literal"), + ("dh:spatial_short_name", "spatial_short_name", "literal"), + ("dh:annotationsText", "annotationsText", "literal"), + ("dh:itemText", "itemText", "literal"), + ] + + def _get_citation(self, json): + """Return the citation sub-dict, or {} if absent or null.""" + return json.get("citation") or {} + def title(self, json): return json.get("title", None) @@ -23,9 +134,10 @@ def category2(self, json): def uriData(self, json): uri_data = json.get("uri_data", None) if uri_data: + # Strip the original path and reconstruct the URI under the + # collection's configured media base URL. filename = uri_data.split("/")[-1] - omeka_data_base = omeka.omeka_data_base - new_uri_data = f"{omeka_data_base}/{filename}" + new_uri_data = f"{self._omeka_data_base}/{filename}" return new_uri_data def dcterms_type(self, json): @@ -33,27 +145,30 @@ def dcterms_type(self, json): return json.get("type", None) def creator(self, json): - creator_names = [creator['name'] for creator in json.get("creator") or [] if 'name' in creator] - return creator_names + # Return dicts with name+id so the dedup in api_fields preserves + # entries that share a name but have different ids. + return [{"name": creator["name"], "id": creator.get("id", "")} + for creator in json.get("creator") or [] if "name" in creator] def contributor(self, json): - contributor_names = [contributor['name'] for contributor in json.get("contributor") or [] if 'name' in contributor] - return contributor_names + # Return dicts with name+id so the dedup in api_fields preserves + # entries that share a name but have different ids. + return [{"name": contrib["name"], "id": contrib.get("id", "")} + for contrib in json.get("contributor") or [] if "name" in contrib] + # NOTE: use Pattern 5 in omeka_overrides if automatic conversion of dates with year or month only to yyyy-01-01 + # back to yyyy for Omeka is desired def date(self, json): date_to_parse = json.get("date", None) - if date_to_parse and "-01-01" in date_to_parse: - #dates are automatically converted by Datura to yyyy-01-01 if month and date are missing - #convert such dates back to yyyy for Omeka S (since it is allowed by the date parser) - return datetime.strptime(date_to_parse, "%Y-%m-%d").year - else: - return date_to_parse + return date_to_parse def dateYear(self, json): date_to_parse = json.get("date", None) if date_to_parse: - year = datetime.strptime(date_to_parse, "%Y-%m-%d").year - return year + try: + return datetime.strptime(date_to_parse, "%Y-%m-%d").year + except ValueError: + return None def dateDisplay(self, json): return json.get("date_display", None) @@ -66,48 +181,46 @@ def dcterms_format(self, json): return json.get("format", None) def relation(self, json): - relation_ids = [relation['id'] for relation in json.get("has_relation") or [] if 'id' in relation] + relations = json.get("has_relation") or {} + relation_ids = [relations['id']] if relations.get('id') is not None else [] return relation_ids - - #citation fields - #TODO is citation always single-valued? if array might need to add code to deal with that def publisher(self, json): - return (json.get("citation") or {}).get("publisher", None) + return self._get_citation(json).get("publisher", None) def biblID(self, json): #note: this field is not yet implemented in the schema - return (json.get("citation") or {}).get("id", None) + return self._get_citation(json).get("id", None) def biblTitle(self, json): - return (json.get("citation") or {}).get("title", None) + return self._get_citation(json).get("title", None) def biblPubPlace(self, json): - return (json.get("citation") or {}).get("pubplace", None) + return self._get_citation(json).get("pubplace", None) def issue(self, json): - return (json.get("citation") or {}).get("issue", None) + return self._get_citation(json).get("issue", None) def pageStart(self, json): - return (json.get("citation") or {}).get("page_start", None) + return self._get_citation(json).get("page_start", None) def pageEnd(self, json): - return (json.get("citation") or {}).get("page_end", None) + return self._get_citation(json).get("page_end", None) def section(self, json): - return (json.get("citation") or {}).get("section", None) + return self._get_citation(json).get("section", None) def volume(self, json): - return (json.get("citation") or {}).get("volume", None) + return self._get_citation(json).get("volume", None) def biblTitleA(self, json): - return (json.get("citation") or {}).get("title_a", None) + return self._get_citation(json).get("title_a", None) def biblTitleM(self, json): - return (json.get("citation") or {}).get("title_m", None) + return self._get_citation(json).get("title_m", None) def biblTitleJ(self, json): - return (json.get("citation") or {}).get("title_j", None) + return self._get_citation(json).get("title_j", None) def rightsHolder(self, json): return json.get("rights_holder", None) @@ -152,7 +265,7 @@ def keyword5(self, json): return json.get("keywords5", None) def source(self, json): - return json.get("has_source") and json.get("has_source", {}).get("title") + return (json.get("has_source") or {}).get("title") def medium(self, json): return json.get("medium", None) @@ -170,8 +283,10 @@ def folder(self, json): return json.get("container_folder", None) def name(self, json): - person_names = [person['name'] for person in json.get("person") or [] if 'name' in person] - return person_names + # Return dicts with name+id so the dedup in api_fields preserves + # entries that share a name but have different ids. + return [{"name": person["name"], "id": person.get("id", "")} + for person in json.get("person") or [] if "name" in person] def spatial_short_name(self, json): spatial = json.get("spatial") @@ -180,48 +295,58 @@ def spatial_short_name(self, json): places = [spatial] if isinstance(spatial, dict) else spatial short_names = [place['short_name'] for place in places if 'short_name' in place] return short_names - - def correspSentName(self, json): - return json.get("correspSentName_omeka_s", None) - - def correspSentPlace(self, json): - return json.get("correspSentPlace_omeka_s", None) - - def correspSentDate(self, json): - return json.get("correspSentDate_omeka_s", None) - - def correspDeliveredName(self, json): - return json.get("correspDeliveredName_omeka_s", None) - - def correspDeliveredPlace(self, json): - return json.get("correspDeliveredPlace_omeka_s", None) - - def correspDeliveredDate(self, json): - return json.get("correspDeliveredDate_omeka_s", None) - - def distributor(self, json): - return json.get("distributor_omeka_s", None) - - def authority(self, json): - return json.get("authority_omeka_s", None) - - def biblNote(self, json): - return json.get("biblNote_omeka_s", None) def annotationsText(self, json): return json.get("annotations_text", None) def itemText(self, json): text = json.get("text", None) - if json.get("data_type"): - text += (" " + self.identifier(json)) + if text and json.get("data_type"): + identifier = self.identifier(json) + if identifier: + text += (" " + identifier) return text -def get_fields(): +def get_fields(omeka_data_base=""): + """ + Return the appropriate FieldDefinitions instance for this collection. + + Looks for a CustomFields class in scripts/python/field_overrides.py in + the collection directory (resolved from the current working directory). + If that file does not exist, falls back to the default FieldDefinitions + class. If the file exists but cannot be loaded, raises RuntimeError. + + Parameters: + * omeka_data_base - passed through to the FieldDefinitions constructor + so that uriData() can build correct media URIs. + Callers should pass ctx.omeka_data_base. + + Returns a FieldDefinitions instance (or a CustomFields subclass of it). + """ + override_path = Path.cwd() / "scripts" / "python" / "field_overrides.py" + override_relative_path = "scripts/python/field_overrides.py" + if not override_path.exists(): + return FieldDefinitions(omeka_data_base=omeka_data_base) + try: - #make sure it can override from the right directly - sys.path.insert(0, './scripts/python') - from omeka_overrides import CustomFields - return CustomFields() - except ImportError: - return FieldDefinitions() \ No newline at end of file + spec = importlib.util.spec_from_file_location("field_overrides", override_path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + except Exception as e: + raise RuntimeError( + f"Failed to load field overrides from {override_relative_path}: {e}" + ) from e + + CustomFields = getattr(module, "CustomFields", None) + if CustomFields is None: + # File exists but defines no CustomFields class — use defaults. + return FieldDefinitions(omeka_data_base=omeka_data_base) + + # CustomFields inherits __init__ from FieldDefinitions, so + # omeka_data_base is passed through automatically. Override __init__ + # in CustomFields only if you need additional constructor logic. + logger.warning( + "Field overrides found at %s; custom field mappings will be applied.", + override_relative_path, + ) + return CustomFields(omeka_data_base=omeka_data_base) \ No newline at end of file diff --git a/lib/datura/python/field_overrides_example.py b/lib/datura/python/field_overrides_example.py new file mode 100644 index 000000000..0720328ab --- /dev/null +++ b/lib/datura/python/field_overrides_example.py @@ -0,0 +1,46 @@ +#copy this file to field_overrides.py in your scripts/python directory. Edit the return values as needed + +from field_definitions import FieldDefinitions + +class CustomFields(FieldDefinitions): + """ + Override only the methods whose behavior differs from the defaults in + FieldDefinitions. The following patterns address common override categories. + """ + + # Pattern 1: read from a different ES key + # def title(self, json): + # return json.get("preferred_title") or json.get("title") + + # Pattern 2: citation sub-field (base class handles null/array automatically) + # def publisher(self, json): + # return self._get_citation(json).get("publisher", None) + + # Pattern 3: combine multiple ES fields into one Omeka value + # def creator(self, json): + # creators = json.get("creator") or [] + # return [ + # f"{c['name']} ({c['role']})" if c.get("role") else c["name"] + # for c in creators if c.get("name") + # ] + + # Pattern 4: transform a value (e.g. reformat a date or strip whitespace) + # def dateDisplay(self, json): + # raw = json.get("date_display", None) + # return raw.strip() if raw else None + + # Pattern 5: date display fix for -01-01 + # dates are automatically converted by Datura to yyyy-01-01 if month and date are missing + # use the below to convert such dates back to yyyy for Omeka S (since it is allowed by the date parser) + # def date(self, json): + # date_to_parse = json.get("date", None) + # if date_to_parse and "-01-01" in date_to_parse: + # return datetime.strptime(date_to_parse, "%Y-%m-%d").year + # else: + # return date_to_parse + + # Pattern 6: mark specific metadata fields as private at the value level + # values for these terms will be posted with is_public: false in the Omeka S API payload, + # hiding them from public view regardless of item-level visibility + # def private_fields(self): + # return {"dcterms:identifier", "dcterms:source"} \ No newline at end of file diff --git a/lib/datura/python/html_and_media_ingest.py b/lib/datura/python/html_and_media_ingest.py index 6dbbce353..554b10274 100644 --- a/lib/datura/python/html_and_media_ingest.py +++ b/lib/datura/python/html_and_media_ingest.py @@ -1,136 +1,602 @@ -import omeka -from pathlib import Path +""" +html_and_media_ingest.py + +Entrypoint script: attaches HTML and IIIF thumbnail media objects to items +that have already been posted to Omeka S by json_to_omeka.py. + +For each JSON record this script: +1. Looks up the Omeka item by its CDRH identifier. +2. Optionally deletes any existing media objects (unless --media-skip is set + and the item already has 2+ media objects). +3. Downloads the IIIF thumbnail from the configured iiif_server and uploads + it to Omeka as the primary media object (so Omeka designates it the + primary_media for the item). +4. Reads the pre-rendered HTML file from output//html/ and uploads it + as an HTML media object. + +Usage (from collection root directory): + python3 html_and_media_ingest.py # defaults to development + python3 html_and_media_ingest.py -e production -r "some_pattern" + python3 html_and_media_ingest.py -m # skip items that already have media + python3 html_and_media_ingest.py --log-level DEBUG + +The script is invoked by bin/post_omeka_html in the Datura gem. The Ruby +wrapper passes -e, -r, and -m arguments from its own CLI. +""" + +import argparse import json -# not used, but needed for debugging +import logging +import os import sys -import traceback -import requests -from requests.exceptions import HTTPError -from copy import deepcopy - -#look for the output folder: /output/development/* -json_dir = omeka.get_dir("output/development/es") -pathlist = list(Path(json_dir).glob('**/*.json')) -regex = omeka.get_regex() -if regex: - pathlist = omeka.filter_items(regex, pathlist) -html_dir = omeka.get_dir("output/development/html") -iiif_dir = omeka.get_dir("output/development/iiif") - -item_set_id = omeka.get_item_set() - -def delete_media_items(matching_item): - if len(matching_item["o:media"]) >= 1: - for media_item in matching_item["o:media"]: - try: - print("deleting media item " + str(media_item["o:id"])) - omeka.omeka_auth.delete_resource(media_item["o:id"], "media") - except HTTPError as err: - if err.response.status_code == 500: - continue - else: - print("error deleting media item: " + str(err)) - raise - -def ingest_thumbnail(json_item, matching_item): - ## IIIF THUMBNAIL INGEST - # note that thumbnail ingest should be done first so that thumbnails are designated primary_media - - collection_name = json_item["collection"] - cover_image = json_item.get("cover_image", None) +import time +from pathlib import Path + +try: + import requests + from requests.exceptions import HTTPError + + import omeka + from omeka import filter_items, filter_items_by_date, filter_items_by_format + from omeka_context import ( + OmekaAPIError, + OmekaAuthError, + OmekaConfigError, + OmekaContext, + OmekaMediaError, + checkpoint_path, + configure_logging, + finish_run, + read_checkpoint, + validate_regex_arg, + write_checkpoint, + ) +except ModuleNotFoundError as err: + raise SystemExit( + f"\033[31m ERROR: {err}\n" + "A required Python package could not be found. " + "You may need to ensure the virtual environment is activated before running this script.\n" + "You may also need to be connected to the VPN.\033[0m" + ) from err + +# Module-level logger. Records from this module appear as +# "html_and_media_ingest" in log output. +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# CLI argument parsing +# --------------------------------------------------------------------------- + +def _parse_args(): + """ + Parse command-line arguments for the HTML/media ingest entrypoint. + + Returns an argparse.Namespace with: + * csv_rows - optional identifier regex for -c item filter, or None + * environment - "development" or "production" (default: "development") + * format_filter - optional format string for -f (directory-based) filter, or None + * media_skip - bool; True skips items that already have 2+ media objects + * proceed - False (not given), None (-p with no value), or a regex + string (-p "pattern") for checkpoint-based resumption + * regex - optional file-filter pattern string, or None + * update_time - optional date/time string for -u filter, or None + * media_skip - bool; True skips items that already have 2+ media objects + * log_level - logging level string, default "INFO" + """ + parser = argparse.ArgumentParser( + description="Attach HTML and IIIF thumbnail media to existing Omeka S items." + ) + parser.add_argument( + "-c", "--csv-rows", + default=None, + dest="csv_rows", + help=( + "Only process items whose identifier matches this regex. " + "Mirrors the Ruby -c flag, which filters CSV rows by identifier." + ), + ) + parser.add_argument( + "-e", "--environment", + default="development", + help="Target environment: 'development' or 'production' (default: development).", + ) + parser.add_argument( + "-f", "--format", + default=None, + dest="format_filter", + help="Only post files of this format (tei, csv, vra, ead, html, pdf, webs).", + ) + parser.add_argument( + "-m", "--media-skip", + action="store_true", + dest="media_skip", + help=( + "Skip re-ingesting media for items that already have 2 or more " + "media objects (thumbnail + HTML). Useful when re-running the " + "script after a partial failure to avoid re-uploading media that " + "was already successfully ingested." + ), + ) + parser.add_argument( + "-p", "--proceed", + nargs="?", + default=False, + const=None, + dest="proceed", + help=( + "Proceed with media ingest from (and including) the JSON file " + "matching this regex. If given without a value, resumes from " + "the last checkpoint saved in logs/proceed_omeka_html_{environment}." + ), + ) + parser.add_argument( + "-r", "--regex", + default=None, + help=( + "Optional regex pattern to restrict processing to matching " + "file paths. Example: -r 'abc123' processes only files whose " + "path contains 'abc123'." + ), + ) + parser.add_argument( + "-u", "--update", + default=None, + dest="update_time", + help=( + "Only process items whose source file was modified at or after " + "this date/time. Accepts 'today', a date (2015-01-01), or " + "date-time (2015-01-01T18:24)." + ), + ) + parser.add_argument( + "--log-level", + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR"], + dest="log_level", + help="Set the logging verbosity (default: INFO).", + ) + return parser.parse_args() + + +# --------------------------------------------------------------------------- +# Media operations +# --------------------------------------------------------------------------- + +def delete_media_items(ctx, matching_item): + """ + Delete all media objects currently attached to an Omeka item. + + Called before re-uploading thumbnail and HTML so that the item does not + accumulate duplicate media objects across repeated script runs. + + HTTP 500 responses from the delete endpoint are treated as non-fatal — + the Omeka S API occasionally returns 500 for media items that have already + been removed in a prior step or that reference missing files on disk. + All other HTTP errors are recorded as OmekaMediaError and processing + continues with the next media object. + + Parameters: + * ctx - OmekaContext providing the authenticated API client + * matching_item - dict: the Omeka item JSON-LD object whose media to delete + """ + for media_item in matching_item.get("o:media", []): + media_id = media_item["o:id"] + try: + logger.info("Deleting media item %s", media_id) + ctx.client.delete_resource(media_id, "media") + except HTTPError as err: + if err.response.status_code == 401 or err.response.status_code == 403: + raise OmekaAuthError( + "Omeka S returned 401 Unauthorized or 403 Forbidden. " + "Check that key_identity and key_credential in config/private.yml are correct. " + "You may also need to be logged onto the VPN." + ) from err + elif err.response.status_code == 500: + # 500 on DELETE is treated as "already gone" by convention. + # Log at DEBUG so it does not clutter normal output. + logger.debug( + "HTTP 500 deleting media %s (may already be absent); continuing", + media_id, + ) + else: + ctx.record_error( + OmekaMediaError(f"HTTP {err.response.status_code} deleting media {media_id}: {err}") + ) + except Exception as err: + ctx.record_error( + OmekaMediaError(f"Unexpected error deleting media {media_id}: {err}") + ) + +def build_thumbnail_url(ctx, json_item): + """ + Construct the remote IIIF URL and local cache filename for this item's thumbnail. + + Returns a (remote_url, local_filename) tuple, or None if the item has no + cover_image. local_filename is a string suitable for joining with iiif_dir; + ingest_thumbnail() appends it to the iiif_dir path. + + Parameters: + * ctx - OmekaContext (provides iiif_server, iiif_collection) + * json_item - dict: one record from a Datura ES JSON file + + """ + + collection_name = ctx.iiif_collection if ctx.iiif_collection else json_item.get("collection", "") + cover_image = json_item.get("cover_image") + if not cover_image: + return None + + # Parse any existing extension from the cover_image name. Image identifiers + # often contain dots that are not extensions (e.g. loc.00001, ccda.let00001), + # so only treat the suffix as an extension if it is a known image format. + _KNOWN_IMAGE_EXTS = {".jpg", ".jpeg", ".png"} + _stem, _ext = os.path.splitext(cover_image) + if _ext.lower() in _KNOWN_IMAGE_EXTS: + stem, image_ext = _stem, _ext + else: + stem, image_ext = cover_image, ".jpg" + + # Construct the IIIF Image API URL for the thumbnail. + # The !200,200 size specifier requests a thumbnail that fits within a + # 200×200 bounding box while preserving aspect ratio. + remote = ( + f"{ctx.iiif_server}/iiif/2/{collection_name}%2F{stem}{image_ext}/full/!200,200/0/default.jpg" + ) + # Cache the thumbnail locally using the same URL-encoded filename so that + # re-runs can be inspected on disk if needed. + local_name = f"{collection_name}%2F{stem}{image_ext}" + + return remote, local_name + + +def ingest_thumbnail(ctx, json_item, matching_item, iiif_dir): + """ + Download a IIIF thumbnail and upload it to Omeka as the item's primary media. + + Thumbnail ingest is performed before HTML ingest so that Omeka designates + the image as the item's primary_media (Omeka S uses the first media object + as the primary). + + URL construction is delegated to build_thumbnail_url(), which can be + overridden independently. + + If build_thumbnail_url() returns None (no cover_image on the item), or if + the download or upload fails, the function returns — the HTML ingest still + proceeds. + + Parameters: + * ctx - OmekaContext (provides iiif_server, client, item_set_id) + * json_item - dict: one record from a Datura ES JSON file + * matching_item - dict: the Omeka item to attach the thumbnail to + * iiif_dir - pathlib.Path pointing to the local IIIF output directory + where the downloaded thumbnail is cached temporarily + """ + identifier = json_item.get("identifier", "unknown") + + _build_thumbnail_url = ctx._fn_build_thumbnail_url or build_thumbnail_url + result = _build_thumbnail_url(ctx, json_item) + if result is None: + # No thumbnail for this item — nothing to do. + logger.debug("No cover_image for %r; skipping thumbnail ingest", identifier) return - # download thumbnail from iiif server - thumbnail_remote = f"{omeka.config['iiif_server']}/iiif/2/{collection_name}%2F{cover_image}.jpg/full/!200,200/0/default.jpg" - thumbnail_local = f"{iiif_dir}/{collection_name}%2F{cover_image}.jpg" + + thumbnail_remote, local_name = result + thumbnail_local = iiif_dir / local_name + + # --- Download --- try: - print(f"downloading thumbnail for {json_item['identifier']}") - response = requests.get(thumbnail_remote) + logger.info("Downloading thumbnail for %r", identifier) + response = requests.get(thumbnail_remote, timeout=30) response.raise_for_status() - with open (thumbnail_local, "wb") as thumb_file: + with open(thumbnail_local, "wb") as thumb_file: thumb_file.write(response.content) except Exception as err: - print(err) - print(f"error downloading thumbnail for {json_item['identifier']}, omitting") + logger.warning( + "Could not download thumbnail for %r: %s; skipping thumbnail ingest", + identifier, + err, + ) return - # attach thumbnail to api item - try: - with open(thumbnail_local, "rb") as thumb_file: - media_payload = { - "o:is_public": True, - "data": { - "upload": thumbnail_local, - "dcterms:title": omeka.prepare_property_value(json_item["title"], omeka.omeka_auth.get_property_id("dcterms:title")) - }, - "o:ingester": "upload" - } - print(f"posting thumbnail for {json_item['identifier']}") - try: - omeka.add_media_to_item(matching_item["o:id"], thumbnail_local, payload=media_payload) - except Exception as err: - print(err) - print(f"error adding image file for {json_item['identifier']}, omitting") - except FileNotFoundError: - print(f"file {thumbnail_local} not found, skipping thumbnail") -def ingest_html(json_item, matching_item): - #get desired path - file_path = f"{html_dir}/{json_item['identifier']}.html" - # get data from html + # --- Upload --- + # The title property ID is fetched via the cache so repeated calls for + # the same term do not make redundant API requests. try: - with open(file_path, "r") as file: - html_content = file.read() media_payload = { "o:is_public": True, "data": { - "html": html_content + "upload": str(thumbnail_local), + "dcterms:title": ctx.client.prepare_property_value( + json_item.get("title", ""), + ctx.get_property_id("dcterms:title"), + ), }, - "o:ingester": "html" + "o:ingester": "upload", } - print(f"posting html for {json_item['identifier']}") - try: - omeka.add_media_to_item(matching_item["o:id"], file_path, payload=media_payload) - except Exception as err: - print(err) - print(f"error adding html file for {json_item['identifier']}, omitting") - traceback.print_exc() + logger.info("Posting thumbnail for %r", identifier) + ctx.client.add_media_to_item(matching_item["o:id"], thumbnail_local, payload=media_payload) except FileNotFoundError: - print(f"file {file_path} not found, skipping item") + # The download step wrote the file, but something removed it between + # download and upload. Unlikely in practice but handled explicitly + # so the error message is clear. + logger.warning( + "Thumbnail file %s not found at upload time; skipping", + thumbnail_local, + ) + except Exception as err: + ctx.record_error( + OmekaMediaError( + f"Error posting thumbnail for {identifier!r}: {err}" + ) + ) + + +def ingest_html(ctx, json_item, matching_item, html_dir): + """ + Read a pre-rendered HTML file and upload it to Omeka as an HTML media object. + + The HTML ingester reads content from payload["data"]["html"] rather than + from the uploaded file bytes; the file is opened only to read its content + into memory. The Omeka S "html" ingester stores the markup directly in + the database, making it searchable and renderable within Omeka. -#iterate through each file -for path in pathlist: - filename = str(path) - with open(filename) as jsonfile: - json_items = json.load(jsonfile) + Skips silently if: + * The .html file does not exist at html_dir/.html. + * The file exists but is empty or contains only whitespace. + + Parameters: + * ctx - OmekaContext + * json_item - dict: one record from a Datura ES JSON file + * matching_item - dict: the Omeka item to attach the HTML to + * html_dir - pathlib.Path pointing to the HTML output directory + """ + identifier = json_item.get("identifier", "unknown") + file_path = html_dir / f"{identifier}.html" + + try: + with open(file_path, "r", encoding="utf-8") as file: + html_content = file.read() + except FileNotFoundError: + # A missing HTML file is common for items that have no text + # representation (e.g. pure image records). Log at INFO so users + # can see which items were skipped without it being alarming. + logger.info("HTML file %s not found; skipping", file_path) + return + + # Guard against empty or whitespace-only files. + if not html_content.strip(): + logger.warning( + "HTML file for %r is empty; skipping. " + "Check whether the XSLT transform produced output for this item.", + identifier, + ) + return + + media_payload = { + "o:is_public": True, + "data": { + "html": html_content, + }, + "o:ingester": "html", + } + + try: + logger.info("Posting HTML for %r", identifier) + ctx.client.add_media_to_item(matching_item["o:id"], file_path, payload=media_payload) + except Exception as err: + ctx.record_error( + OmekaMediaError( + f"Error posting HTML for {identifier!r}: {err}" + ) + ) + + +# --------------------------------------------------------------------------- +# Main processing loop +# --------------------------------------------------------------------------- + +def ingest_item_media(ctx, json_item, matching_item, html_dir, iiif_dir): + """ + Run the full media pipeline for a single Omeka item. + + Encapsulates the delete-then-reingest sequence so that collections needing + a different media pipeline (e.g. adding a PDF step, skipping thumbnails for + certain item types) can override this function without touching the item-lookup and + skip logic in process_items(). + + Order matters: thumbnail must be uploaded before HTML so that Omeka + designates the image as primary_media (Omeka S uses the first media object + attached to an item as its primary). + + Parameters: + * ctx - OmekaContext + * json_item - dict: one record from a Datura ES JSON file + * matching_item - dict: the current Omeka item retrieved from the API + * html_dir - pathlib.Path to output//html/ + * iiif_dir - pathlib.Path to output//iiif/ + """ + delete_media_items(ctx, matching_item) + ingest_thumbnail(ctx, json_item, matching_item, iiif_dir) + ingest_html(ctx, json_item, matching_item, html_dir) + +def process_items(ctx, pathlist, html_dir, iiif_dir): + """ + For each JSON record, look up the Omeka item and ingest its media. + + Logic: + * Skip items with no identifier (cannot look up in Omeka). + * Skip items with 0 or >1 Omeka matches (not posted / data integrity issue). + * If --media-skip is set and the item already has 2+ media objects + (thumbnail + HTML), skip re-ingestion to avoid unnecessary deletions. + * Otherwise: delete existing media, ingest thumbnail, ingest HTML. + + Parameters: + * ctx - OmekaContext + * pathlist - list of pathlib.Path objects for ES JSON files + * html_dir - pathlib.Path to output//html/ + * iiif_dir - pathlib.Path to output//iiif/ + """ + for path in pathlist: + filename = str(path) + rel = path.relative_to(Path.cwd()) + with open(filename) as jsonfile: + json_items = json.load(jsonfile) + # Apply --csv-rows identifier filter if provided. + if ctx.csv_rows: + json_items = omeka.filter_items_by_identifier(ctx.csv_rows, json_items) for json_item in json_items: - if not json_item["identifier"]: + identifier = json_item.get("identifier") + if not identifier: + logger.warning("Skipping item without identifier in %s", rel) + continue + + title = json_item.get("title") + if not title: + logger.warning("Skipping item without title in %s", rel) + continue + + # --- Look up the item in Omeka --- + try: + matching_items = ctx.client.filter_items_by_property( + filter_property="dcterms:identifier", + filter_value=identifier, + item_set_id=ctx.item_set_id, + ) + except Exception as err: + ctx.record_error(OmekaAPIError(identifier, "filter_items (media)", err)) + continue + + if not matching_items: + logger.warning( + "Unexpected empty response from filter_items for %r; skipping", + identifier, + ) continue - matching_items = omeka.omeka_auth.filter_items_by_property(filter_property = "dcterms:identifier", filter_value = json_item["identifier"], item_set_id=item_set_id) - if matching_items: - if matching_items["total_results"] == 1: - matching_item = matching_items["results"][0] - media_count = len(matching_item["o:media"]) - elif matching_items["total_results"] > 1: - print("multiple items found for " + json_item["identifier"] + ", check admin site") - continue - else: - print("no matching items for " + json_item["identifier"] + ", skipping") - continue - #check for existing media items, to avoid duplicates - #skip with -m flag - if not(omeka.args.media_skip and media_count >=2): - delete_media_items(matching_item) - - ## IIIF THUMBNAIL INGEST - #if -m flag, ingest only if not already present - # note that thumbnail ingest should be done first so that thumbnails are designated primary_media - ingest_thumbnail(json_item, matching_item) - - ## HTML INGEST - #if -m flag, ingest only if not already present - ingest_html(json_item, matching_item) - - ##TODO add other media ingest as needed - else: - print("skipping media for " + json_item["identifier"] + ", already ingested.") + total = matching_items.get("total_results", 0) + if total == 0: + logger.warning("No Omeka item found for %r; skipping media ingest", identifier) + continue + if total > 1: + logger.warning( + "Multiple Omeka items (%d) found for %r; check admin site and skip", + total, + identifier, + ) + continue + + matching_item = matching_items["results"][0] + media_count = len(matching_item.get("o:media", [])) + + # --media-skip: if the item already has 2+ media objects + # (thumbnail + HTML), assume it was already fully ingested and + # skip it to avoid redundant deletion and re-upload. + if ctx.media_skip and media_count >= 2: + logger.info( + "Skipping media for %r: already has %d media object(s)", + identifier, + media_count, + ) + continue + + # --- Media pipeline --- + ingest_item_media(ctx, json_item, matching_item, html_dir, iiif_dir) + + # Record the last-processed file so that -p (no value) can resume + # from this point on the next run. + write_checkpoint(path.stem, ctx, "omeka_html") + +# --------------------------------------------------------------------------- +# Entrypoint +# --------------------------------------------------------------------------- + +def main(): + """ + Entrypoint: parse arguments, build context, run media ingest, report. + + Execution order: + 1. Parse CLI arguments. + 2. Configure root logger. + 3. Build OmekaContext (loads config, validates keys, creates API client). + 4. Resolve output directories for the requested environment. + 5. Discover JSON files; apply csv (-c), format (-f), regex (-r), and update-time (-u) filters. + 6. Apply proceed filter (-p): resume from a checkpoint or a named file. + 7. Run media ingest for all items; writes a checkpoint after each JSON file. + 8. Report errors; exit 1 if any failures, 0 if clean. + """ + args = _parse_args() + start_time = time.time() + configure_logging(args.log_level) + + # Validate regex for -r and -c option input + if args.regex: + validate_regex_arg(args.regex, "--regex") + if args.csv_rows: + validate_regex_arg(args.csv_rows, "--csv-rows") + + # OmekaConfigError propagates here as a fatal error — missing or broken + # config means no API access is possible. + ctx = OmekaContext.from_args(args) + + # Resolve all three environment-specific directories using the requested + # environment so that -e production reads from output/production/ rather + # than always defaulting to output/development/. + json_dir = ctx.resolve_path(f"output/{ctx.environment}/es") + html_dir = ctx.resolve_path(f"output/{ctx.environment}/html") + iiif_dir = ctx.resolve_path(f"output/{ctx.environment}/iiif") + + pathlist = list(Path(json_dir).glob("**/*.json")) + + if ctx.format_filter: + pathlist = filter_items_by_format(ctx.format_filter, pathlist) + if ctx.regex: + pathlist = filter_items(ctx.regex, pathlist) + if ctx.update_time: + pathlist = filter_items_by_date(ctx.update_time, pathlist) + + # Apply -p / --proceed: resume from a specific file or the last checkpoint. + # args.proceed is False (not given), None (-p with no value), or a string. + proceed = args.proceed + if proceed is None: + # -p given with no value — prompt to resume from the saved checkpoint. + last = read_checkpoint(ctx, "omeka_html") + if last is None: + print( + "ERROR: --proceed given with no value but no checkpoint file " + "found at {}.".format(checkpoint_path(ctx, "omeka_html")) + ) + sys.exit(1) + response = input("Continue from {}? (y/n): ".format(last)).strip().lower() + if response == "y": + proceed = last + else: + print("Exiting.") + sys.exit(0) + if proceed: + pathlist = omeka.proceed_files(proceed, pathlist) + + logger.info( + "Found %d JSON file(s) in %s (environment=%r, media_skip=%s)", + len(pathlist), + f"output/{ctx.environment}/es", + ctx.environment, + ctx.media_skip, + ) + + process_items(ctx, pathlist, html_dir, iiif_dir) + + finish_run(ctx, args, start_time) + + +if __name__ == "__main__": + try: + main() + except KeyboardInterrupt: + print("\nInterrupted. Exiting.") + sys.exit(1) + except OmekaConfigError as err: + logger.debug("Fatal configuration error:", exc_info=True) + print(f"ERROR: {err}", file=sys.stderr) + sys.exit(1) diff --git a/lib/datura/python/json_to_omeka.py b/lib/datura/python/json_to_omeka.py index 5d2f63d8d..729037539 100644 --- a/lib/datura/python/json_to_omeka.py +++ b/lib/datura/python/json_to_omeka.py @@ -1,116 +1,607 @@ -#import necessary modules, including omeka, and the fields -import omeka -import api_fields +""" +json_to_omeka.py +Entrypoint script: reads Datura-generated ElasticSearch JSON files and posts +each item to an Omeka S instance. + +The script runs in two sequential passes: + + Pass 1 — Item posting (post_items): + For each JSON item, check whether the Omeka identifier already exists. + * If found (one result): update the existing Omeka item's metadata. + * If not found: create a new Omeka item. + * If multiple results: warn and skip (data integrity problem). + + Pass 2 — Item linking (link_items): + After all items exist in Omeka, re-read the same JSON files and populate + relational fields (has_part, has_source, etc.) by resolving Omeka IDs for + the referenced items. A separate pass is required because items must exist + before they can be linked to one another. + +Usage (from collection root directory): + python3 json_to_omeka.py # defaults to development + python3 json_to_omeka.py -e production -r "some_pattern" + python3 json_to_omeka.py --log-level DEBUG + +The script is invoked by bin/post_omeka in the Datura gem. The Ruby wrapper +passes -e and -r arguments from its own CLI. No changes to bin/post_omeka +are required to support the --log-level flag (it defaults to INFO). +""" + +import argparse import copy import json -from pathlib import Path -#needed for debugging purposes and path -import sys -import traceback +import logging import os +import sys +import time +from pathlib import Path + +try: + import api_fields + import omeka + from omeka_context import ( + OmekaAPIError, + OmekaConfigError, + OmekaContext, + checkpoint_path, + configure_logging, + finish_run, + read_checkpoint, + validate_regex_arg, + write_checkpoint, + ) + from omeka import filter_items, filter_items_by_date, filter_items_by_format, prepare_item_payload_using_template +except ModuleNotFoundError as err: + raise SystemExit( + f"\033[31m ERROR: {err}\n" + "A required Python package could not be found. " + "You may need to ensure the virtual environment is activated before running this script.\n" + "You may also need to be connected to the VPN.\033[0m" + ) from err + +# Module-level logger. Records from this module appear as "json_to_omeka" +# in log output so they can be filtered independently from other modules. +logger = logging.getLogger(__name__) + -def post_items(pathlist): - #iterate through each file +# --------------------------------------------------------------------------- +# CLI argument parsing +# --------------------------------------------------------------------------- + +def _parse_args(): + """ + Parse command-line arguments for the JSON-to-Omeka entrypoint. + + Returns an argparse.Namespace with: + * environment - "development" or "production" (default: "development") + * format_filter - optional format string for -f (directory-based) filter, or None + * json_output - bool; True skips post to Omeka API and instead writes data to files as JSON + * regex - optional file-filter pattern string, or None + * csv_rows - optional identifier regex for -c item filter, or None + * proceed - False (not given), None (-p with no value), or a regex + string (-p "pattern") for checkpoint-based resumption + * update_time - optional date/time string for -u filter, or None + * log_level - logging level string, default "INFO" + + """ + parser = argparse.ArgumentParser( + description="Post Datura ES JSON output to an Omeka S instance." + ) + parser.add_argument( + "-c", "--csv-rows", + default=None, + dest="csv_rows", + help=( + "Only process items whose identifier matches this regex. " + "Mirrors the Ruby -c flag, which filters CSV rows by identifier." + ), + ) + parser.add_argument( + "-e", "--environment", + default="development", + help="Target environment: 'development' or 'production' (default: development).", + ) + parser.add_argument( + "-f", "--format", + default=None, + dest="format_filter", + help="Only post files of this format (tei, csv, vra, ead, html, pdf, webs).", + ) + parser.add_argument( + "-j", "--json-output", + action="store_true", + dest="json_output", + default=False, + help=( + "Write Omeka S item payloads to output//omeka/ instead " + "of posting to the API. No items are created or updated in Omeka. " + "An API connection is still required for property ID lookups and " + "template validation. The link pass is skipped because no live " + "Omeka item IDs are available." + ), + ) + parser.add_argument( + "-p", "--proceed", + nargs="?", + default=False, + const=None, + dest="proceed", + help=( + "Proceed with posting from (and including) the JSON file matching " + "this regex. If given without a value, resumes from the last " + "checkpoint saved in logs/proceed_omeka_{environment}." + ), + ) + parser.add_argument( + "-r", "--regex", + default=None, + help=( + "Optional regex pattern to restrict processing to matching " + "file paths. Example: -r 'abc123' processes only files whose " + "path contains 'abc123'." + ), + ) + parser.add_argument( + "-u", "--update", + default=None, + dest="update_time", + help=( + "Only process items whose source file was modified at or after " + "this date/time. Accepts 'today', a date (2015-01-01), or " + "date-time (2015-01-01T18:24)." + ), + ) + parser.add_argument( + "--log-level", + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR"], + dest="log_level", + help="Set the logging verbosity (default: INFO).", + ) + return parser.parse_args() + + +# --------------------------------------------------------------------------- +# Pass 1: item creation / update +# --------------------------------------------------------------------------- + +def post_items(ctx, pathlist, json_output_dir=None): + """ + First pass: create or update Omeka items for every JSON record. + + Iterates over all JSON files in pathlist. For each record: + * Items whose identifier already exists in Omeka are updated in place. + * Items not yet in Omeka are created from scratch. + * Items that return multiple Omeka matches are skipped with a warning + (duplicate identifiers indicate a data integrity problem that must be + resolved in the Omeka admin UI before the item can be re-posted). + * Items whose identifier field is falsy are skipped with a warning. + + Per-item errors (API failures, malformed payload) are recorded via + ctx.record_error() and do NOT halt the run. Fatal errors (wrong + credentials, missing config) raise exceptions that propagate to main(). + + Parameters: + * ctx - OmekaContext providing API client, config, and error log + * pathlist - list of pathlib.Path objects pointing to ES JSON files + * json_output_dir - optional Path; when set, write Omeka S item payloads to + this directory instead of posting to the API + """ for path in pathlist: filename = str(path) + rel = path.relative_to(Path.cwd()) with open(filename) as jsonfile: json_items = json.load(jsonfile) - # TODO change template_number to actual number, account for other schemas is necessary - template_number = omeka.template_number - for json_item in json_items: + # Apply --csv-rows identifier filter if provided. + if ctx.csv_rows: + json_items = omeka.filter_items_by_identifier(ctx.csv_rows, json_items) + # template_number is stable for the lifetime of a run — read from + # ctx rather than re-reading from config for every item. + template_number = ctx.template_number + + for json_item in json_items: + identifier = json_item.get("identifier") + if not identifier: + # Records without an identifier cannot be matched or created. + logger.warning("Skipping item without identifier in %s", rel) + continue + + title = json_item.get("title") + if not title: + logger.warning("Skipping item without title in %s", rel) + continue + + if json_output_dir is not None: + # JSON output mode: build the payload and write it to disk + # rather than to the Omeka API. try: - if not json_item["identifier"]: - breakpoint() - print("skipping item without identifier") - continue - except TypeError as e: - breakpoint() - matching_items = omeka.omeka_auth.filter_items_by_property(filter_property = "dcterms:identifier", filter_value = json_item["identifier"], item_set_id=item_set_id) - if matching_items: - #if item exists, update item - if matching_items["total_results"] == 1: - update_existing_item(json_item, matching_items) - #add new item if item does not exist - elif matching_items["total_results"] == 0: - add_new_item(json_item, template_number) - #if multiple matches, warn but don't ingest - else: - print(f"multiple matches for {json_item['identifier']}, please check Omeka admin site") - -def link_items(pathlist): - #go through tables again to link records + new_item = api_fields.prepare_item(ctx, json_item) + except Exception as err: + ctx.record_error(OmekaAPIError(identifier, "prepare_item", err)) + continue + if not new_item: + logger.warning("Could not prepare payload for %r; skipping", identifier) + continue + try: + payload = prepare_item_payload_using_template(ctx, new_item, template_number) + except Exception as err: + ctx.record_error(OmekaAPIError(identifier, "prepare_template_payload", err)) + continue + out_path = json_output_dir / f"{identifier}.json" + relative_path = f"output/{ctx.environment}/{identifier}.json" + logger.info("Writing Omeka payload for %r to %s", identifier, relative_path) + with open(out_path, "w") as f: + json.dump(payload, f, indent=2) + continue + + try: + matching_items = ctx.client.filter_items_by_property( + filter_property="dcterms:identifier", + filter_value=identifier, + item_set_id=ctx.item_set_id, + ) + except Exception as err: + ctx.record_error(OmekaAPIError(identifier, "filter_items", err)) + continue + + if not matching_items: + # API returned no response object at all — treat as a lookup + # failure rather than "zero results". + logger.warning( + "Unexpected empty response from filter_items for %r; skipping", + identifier, + ) + continue + + total = matching_items.get("total_results", 0) + + if total == 1: + # Item exists — refresh its metadata. + update_existing_item(ctx, json_item, matching_items) + elif total == 0: + # Item is new — create it. + add_new_item(ctx, json_item, template_number) + else: + # More than one match — cannot determine which to update. + logger.warning( + "Multiple matches (%d) for %r; check Omeka admin site", + total, + identifier, + ) + + # Record the last-processed file so that -p (no value) can resume + # from this point on the next run. + write_checkpoint(path.stem, ctx, "omeka") + + +def add_new_item(ctx, json_item, template_number): + """ + Build the Omeka payload for a new item and POST it to the API. + + Calls api_fields.prepare_item() to extract and format each field from + the Datura JSON record. If preparation produces no payload (e.g. all + fields were absent), the item is skipped with a warning rather than + POSTing an empty object. + + Parameters: + * ctx - OmekaContext + * json_item - dict: one record from a Datura ES JSON file + * template_number - Omeka resource template numeric ID (from ctx.template_number) + """ + identifier = json_item.get("identifier", "unknown") + + new_item = api_fields.prepare_item(ctx, json_item) + if not new_item: + logger.warning("Could not prepare payload for %r; skipping", identifier) + return + + # Log the identifier we are about to create. Use .get() with a default + # rather than direct dict access so a missing dcterms:identifier key + # does not raise KeyError and halt the run. + title_val = ( + new_item.get("dcterms:identifier", [{}])[0].get("@value", identifier) + ) + logger.info("Creating item %r", title_val) + + # Validate terms against the resource template and wrap values in the + # JSON-LD structure Omeka S expects. + payload = prepare_item_payload_using_template(ctx, new_item, template_number) + + try: + ctx.client.add_item( + payload, + template_id=template_number, + item_set_id=ctx.item_set_id, + is_public=ctx.is_public, + ) + except Exception as err: + ctx.record_error(OmekaAPIError(identifier, "add_item", err)) + + +def update_existing_item(ctx, json_item, matching_items): + """ + Re-prepare an existing Omeka item's metadata and PATCH it via the API. + + Fetches the current Omeka item, merges it with fresh values from the + Datura JSON record, and calls update_resource() to apply the changes. + + Parameters: + * ctx - OmekaContext + * json_item - dict: one record from a Datura ES JSON file + * matching_items - dict: Omeka API response containing the existing item + under matching_items["results"][0] + """ + identifier = json_item.get("identifier", "unknown") + + # Log the Omeka-side identifier to make it easy to correlate log lines + # with records in the Omeka admin UI. + omeka_id_display = ( + matching_items["results"][0] + .get("dcterms:identifier", [{}])[0] + .get("@value", identifier) + ) + logger.info("Updating item %r", omeka_id_display) + + # Deep-copy to avoid mutating the dict returned by the API client; the + # original might be referenced elsewhere (e.g. in link_items). + item_to_update = copy.deepcopy(matching_items["results"][0]) + updated_item = api_fields.prepare_item(ctx, json_item, item_to_update) + if not updated_item: + logger.warning("Could not prepare update payload for %r; skipping", identifier) + return + + try: + ctx.client.update_resource(updated_item, "items") + except Exception as err: + ctx.record_error(OmekaAPIError(identifier, "update_resource", err)) + + +# --------------------------------------------------------------------------- +# Pass 2: record linking +# --------------------------------------------------------------------------- + +def link_items(ctx, pathlist): + """ + Second pass: populate relational fields between Omeka items. + + Re-reads the same JSON files processed in pass 1. For each record, + resolves the Omeka item IDs of related items (has_part, has_source, etc.) + and PATCHes the item with link values. + + A separate pass is necessary because linked items must already exist in + Omeka before they can be referenced. Running this pass after all items + have been created (or updated) in pass 1 guarantees that the target items + are present. + + Items that cannot be found or that have multiple matches are skipped with + a warning; they will be logged in the run summary. + + Parameters: + * ctx - OmekaContext + * pathlist - list of pathlib.Path objects (same list used in post_items) + """ for path in pathlist: filename = str(path) + rel = path.relative_to(Path.cwd()) with open(filename) as jsonfile: json_items = json.load(jsonfile) - for json_item in json_items: - if not json_item["identifier"]: - print("skipping item without identifier") - continue - matching_items = omeka.omeka_auth.filter_items_by_property(filter_property = "dcterms:identifier", filter_value = json_item["identifier"], item_set_id=item_set_id) - if matching_items and matching_items["total_results"] == 1: - #if item exists, update item with linked records - link_item(json_item, matching_items) - else: - #if multiple matches or item not found, display warning - print(f"skipping {json_item['identifier']}, item not properly ingested") - -def link_item(json_item, matching_items): - item_id = matching_items["results"][0]["dcterms:identifier"][0]["@value"] - print(f"linking records for {item_id}") + if ctx.csv_rows: + json_items = omeka.filter_items_by_identifier(ctx.csv_rows, json_items) + + for json_item in json_items: + identifier = json_item.get("identifier") + if not identifier: + logger.debug("Skipping item without identifier in %s", rel) + continue + + title = json_item.get("title") + if not title: + logger.warning("Skipping item without title in %s", rel) + continue + + try: + matching_items = ctx.client.filter_items_by_property( + filter_property="dcterms:identifier", + filter_value=identifier, + item_set_id=ctx.item_set_id, + ) + except Exception as err: + ctx.record_error(OmekaAPIError(identifier, "filter_items (link pass)", err)) + continue + + if not matching_items: + logger.warning( + "Unexpected empty response from filter_items for %r during link pass; skipping", + identifier, + ) + continue + + total = matching_items.get("total_results", 0) + if total == 1: + _link_item(ctx, json_item, matching_items) + else: + # total == 0: item was not successfully posted in pass 1. + # total > 1: data integrity problem (duplicate identifiers). + # In both cases, linking is impossible. + logger.warning( + "Skipping link pass for %r: expected 1 match, got %d", + identifier, + total, + ) + + +def _link_item(ctx, json_item, matching_items): + """ + Resolve and attach relational fields for a single Omeka item. + + Extracts the Omeka item from matching_items, calls api_fields.link_records() + to populate relation fields, and PATCHes the result back to the API. + + Named with a leading underscore to signal that it is an internal helper + called only by link_items(); external code should use link_items(). + + Parameters: + * ctx - OmekaContext + * json_item - dict: one record from a Datura ES JSON file + * matching_items - dict: Omeka API response containing the item to link + """ + # Pull the human-readable identifier from the Omeka item for log messages. + item_id = ( + matching_items["results"][0] + .get("dcterms:identifier", [{}])[0] + .get("@value", json_item.get("identifier", "unknown")) + ) + logger.info("Linking records for %r", item_id) + + # Deep-copy so that api_fields.link_records() can modify the item dict + # without affecting the in-memory copy used elsewhere in this pass. item_to_link = copy.deepcopy(matching_items["results"][0]) - linked_item = api_fields.link_records(json_item, item_to_link) + try: - omeka.omeka_auth.update_resource(linked_item, "items") + _link_records = ctx._fn_link_records or api_fields.link_records + linked_item = _link_records(ctx, json_item, item_to_link) except Exception as err: - print(str(err)) - traceback.print_exc - print(f"Error updating item {item_id}") - breakpoint() - pass - -def add_new_item(json_item, template_number): - new_item = api_fields.prepare_item(json_item) - if new_item: - try: - print(f"creating item {new_item['dcterms:identifier'][0]['@value']}") - except KeyError as err: - print(err) - breakpoint() - payload = omeka.prepare_item_payload_using_template(new_item, template_number) - # add item set - try: - omeka.omeka_auth.add_item(payload, template_id=template_number, item_set_id=item_set_id, is_public=is_public) - except Exception as err: - print(err) - breakpoint() - else: - print(f"error preparing item {json_item['identifier']}") - -def update_existing_item(json_item, matching_items): - print(f"updating item {matching_items['results'][0]['dcterms:identifier'][0]['@value']}") - item_to_update = copy.deepcopy(matching_items["results"][0]) - updated_item = api_fields.prepare_item(json_item, item_to_update) - if updated_item: - try: - omeka.omeka_auth.update_resource(updated_item, "items") - except Exception as err: - print(err) - breakpoint() - -#look for the output folder: /output/development/es and get all items -json_dir = omeka.get_dir("output/development/es") -regex = omeka.get_regex() -pathlist = list(Path(json_dir).glob('**/*.json')) -if regex: - pathlist = omeka.filter_items(regex, pathlist) -item_set_id = omeka.get_item_set() -is_public = True if omeka.get_environment() == "production" else False -#enables importing of overrides -sys.path.append(os.path.join(os.getcwd(), "scripts/overrides")) -post_items(pathlist) -#need to query the API again at this point so that records can be linked -omeka.reset() -link_items(pathlist) + ctx.record_error(OmekaAPIError(item_id, "link_records", err)) + return + + try: + ctx.client.update_resource(linked_item, "items") + except Exception as err: + # Log the full traceback at DEBUG level so that --log-level DEBUG + # reveals the exact API response; WARNING is shown by default. + logger.debug("Traceback for update_resource failure:", exc_info=True) + ctx.record_error(OmekaAPIError(item_id, "update_resource (link pass)", err)) + + +# --------------------------------------------------------------------------- +# Entrypoint +# --------------------------------------------------------------------------- + +def main(): + """ + Entrypoint: parse arguments, build context, run both passes, report. + + Execution order: + 1. Parse CLI arguments via _parse_args(). + 2. Configure the root logger (before any other work so all output is + captured at the right level). + 3. Build OmekaContext — loads config/private.yml, validates required keys, + initialises the authenticated API client. Exits with a descriptive + error message if the config is missing or malformed (OmekaConfigError). + 4. Discover JSON files under output//es/. + 5. Apply csv (-c), format (-f), regex (-r), and update-time (-u) filters. + 6. Apply proceed filter (-p): resume from a checkpoint or a named file. + 7. Add scripts/python to sys.path so that collection-specific + overrides can be imported by field_definitions.get_fields(). + 8. Run pass 1 (post_items); writes a checkpoint after each JSON file. + 9. Reset the API client between passes for a clean connection. + 10. Run pass 2 (link_items); always processes the full filtered pathlist. + 11. Print run summary; exit 1 if any per-item errors were recorded, + """ + args = _parse_args() + start_time = time.time() + + # Configure root logger first so that even OmekaContext initialisation + # errors are captured at the correct level. + configure_logging(args.log_level) + + # Validate regex for -r and -c option input + if args.regex: + validate_regex_arg(args.regex, "--regex") + if args.csv_rows: + validate_regex_arg(args.csv_rows, "--csv-rows") + + # OmekaContext.from_args() raises OmekaConfigError (a subclass of + # OmekaError) if config/private.yml is missing, unparseable, or missing + # a required key. Let this propagate to the top level — configuration + # errors are fatal and should produce a clear traceback. + ctx = OmekaContext.from_args(args) + # Resolve the ES output directory for the requested environment. + # ctx.resolve_path() returns an absolute Path relative to cwd (the + # collection root), so passing -e production reads from output/production/ + # rather than always defaulting to output/development/. + json_dir = ctx.resolve_path(f"output/{ctx.environment}/es") + pathlist = list(Path(json_dir).glob("**/*.json")) + + if ctx.format_filter: + pathlist = filter_items_by_format(ctx.format_filter, pathlist) + if ctx.regex: + pathlist = filter_items(ctx.regex, pathlist) + if ctx.update_time: + pathlist = filter_items_by_date(ctx.update_time, pathlist) + + # Apply -p / --proceed: resume from a specific file or the last checkpoint. + # args.proceed is False (not given), None (-p with no value), or a string. + proceed = args.proceed + if proceed is None: + # -p given with no value — prompt to resume from the saved checkpoint. + last = read_checkpoint(ctx, "omeka") + if last is None: + print( + "ERROR: --proceed given with no value but no checkpoint file " + "found at {}.".format(checkpoint_path(ctx, "omeka")) + ) + sys.exit(1) + response = input("Continue from {}? (y/n): ".format(last)).strip().lower() + if response == "y": + proceed = last + else: + print("Exiting.") + sys.exit(0) + if proceed: + pathlist = omeka.proceed_files(proceed, pathlist) + + logger.info( + "Found %d JSON file(s) in %s (environment=%r)", + len(pathlist), + f"output/{ctx.environment}/es", + ctx.environment, + ) + + if not pathlist: + logger.warning(f"No JSON files found in output/{ctx.environment}/es — nothing to post or link. Exiting.") + sys.exit(0) + + # --- JSON output mode (-j / --json-output) --- + if args.json_output: + relative_dir = f"output/{ctx.environment}/omeka" + omeka_out_dir = ctx.resolve_path(relative_dir) + Path(omeka_out_dir).mkdir(parents=True, exist_ok=True) + logger.info( + "JSON output mode: writing Omeka S payloads to %s (API will not be called)", + relative_dir, + ) + post_items(ctx, pathlist, json_output_dir=Path(omeka_out_dir)) + finish_run(ctx, args, start_time) + return + + # --- Pass 1: create / update items --- + logger.info("Starting pass 1: item posting") + post_items(ctx, pathlist) + + # Reset the API client between passes. ctx.reset_client() re-instantiates + # OmekaAPIClient with the same credentials, giving a fresh connection for + # the second round of requests. The property ID cache is preserved because + # term-to-ID mappings do not change between passes. + logger.info("Pass 1 complete. Resetting API client for pass 2.") + ctx.reset_client() + + # --- Pass 2: link related items --- + logger.info("Starting pass 2: record linking") + link_items(ctx, pathlist) + + finish_run(ctx, args, start_time) + + +if __name__ == "__main__": + try: + main() + except KeyboardInterrupt: + print("\nInterrupted. Exiting.") + sys.exit(1) + except OmekaConfigError as err: + logger.debug("Fatal configuration error:", exc_info=True) + print(f"ERROR: {err}", file=sys.stderr) + sys.exit(1) diff --git a/lib/datura/python/omeka.py b/lib/datura/python/omeka.py index 5b079d04f..78bcbe596 100644 --- a/lib/datura/python/omeka.py +++ b/lib/datura/python/omeka.py @@ -1,223 +1,222 @@ +""" +omeka.py + +Utility functions for the Omeka S ingestion pipeline. + +""" + +from datetime import datetime from pathlib import Path -import json -from omeka_s_tools.api import OmekaAPIClient -import math -import yaml -import argparse +import logging +import os import re -#needed for debugging purposes -import traceback -import os +logger = logging.getLogger(__name__) -# custom methods - -def get_dir(relative_path): - cwd = Path.cwd() - return (cwd / relative_path).resolve() - -def get_config(path, env='default'): - with open(path) as stream: - try: - contents = yaml.safe_load(stream) - return(contents[env]) - except yaml.YAMLError as exc: - print(exc) - -def reset(): - omeka = OmekaAPIClient(config['omeka_server']) - omeka_auth = OmekaAPIClient( - api_url = config['omeka_server'], - key_identity = config['key_identity'], - key_credential = config['key_credential'] - ) - -def item_sets(): - pages = math.ceil(omeka.get_resources("item_sets")["total_results"] / 5) - item_sets = [] - for i in range(pages): - item_sets += omeka.get_resources("item_sets", page=i)["results"] - return item_sets - -def get_item_set(): - env = get_environment() - if env == "production": - item_set = prod_config["item_set"] - elif env == "development": - item_set = dev_config["item_set"] - else: - item_set == None - return item_set - -def get_environment(): - environment = args.environment - return environment - -def get_regex(): - regex = args.regex - return regex - -def add_media_to_item(item_id, media_file, payload={}, template_id=None, class_id=None): - # copied from the module to modify with different ingester - ''' - Upload a media file and associate it with an existing item. - Parameters: - * `item_id` - the Omeka id of the item this media file should be added to - * `media_path` - a path to an image/media file (string or pathlib Path) - * `payload` (optional) - metadata to attach to media object, either - a dict generated by `prepare_item_payload()` or `prepare_item_payload_using_template()`, - or a string which is used as the value for `dcterms:title`. - * `template_id` - internal Omeka identifier of a resource template you want to attach to this item - * `class_id` - internal Omeka identifier of a resource class you want to attach to this item - - Returns: - * a dict providing a JSON-LD representation of the new media object - ''' - files = {} - # For backwards compatibility - if isinstance(media_file, dict): - path = media_file['path'] - payload = media_file['title'] - # Make sure path is a Path object - path = Path(media_file) - if isinstance(payload, str): - payload = omeka.omeka_auth.prepare_item_payload({'dcterms:title': [payload]}) - if template_id: - payload['o:resource_template'] = omeka.omeka_auth.format_resource_id(template_id, 'resource_templates') - if not class_id: - template = omeka.omeka_auth.get_resource_by_id(template_id, 'resource_templates') - class_id = template['o:resource_class']['o:id'] - if class_id: - payload['o:resource_class'] = omeka.omeka_auth.format_resource_id(class_id, 'resource_classes') - #add option to change ingester - ingester = payload["o:ingester"] if payload["o:ingester"] else "upload" - file_data = { - 'o:ingester': ingester, - 'file_index': '0', - 'o:source': path.name, - 'o:item': {'o:id': item_id} - } - payload.update(file_data) - files[f'file[0]'] = path.read_bytes() - files['data'] = (None, json.dumps(payload), 'application/json') - response = omeka_auth.s.post(f'{omeka_auth.api_url}/media', files=files, params=omeka_auth.credentials) - data = omeka_auth.process_response(response) - return data - -def prepare_item_payload_using_template(terms, template_id): - ''' - Prepare an item payload, checking the supplied terms and values against the specified template. - Note: - * terms that are not in the template will generate a warning and be dropped from the payload - * data types that don't match the template definitions will generate a warning and the term will be dropped from the payload - * if no data type is supplied, a type that conforms with the template definition will be used +def prepare_item_payload_using_template(ctx, terms, template_id): + """ + Build an item payload, validating terms and values against a resource template. - Parameters: - * `terms`: a dict of terms, values, and (optionally) data types - * `template_id`: Omeka's internal numeric identifier for the template + Behavior: + - Terms not present in the template are logged and dropped from the payload. + - Values whose data type does not match the template definition are dropped. + - If no data type is supplied for a value, the template default is used, + or "literal" if the template allows it and no single default exists. - Returns: - * the payload dict - ''' - template_properties = omeka_auth.get_template_properties(template_id) + Parameters: + * ctx - OmekaContext providing the authenticated API client + * terms - dict mapping Omeka property term strings to lists of value + dicts, e.g. {"dcterms:title": [{"value": "My Title"}]} + * template_id - Omeka's internal numeric ID for the resource template + + Returns a payload dict suitable for passing to ctx.client.add_item(). + """ + # Fetch the template's property definitions once; this dict maps term + # strings to their allowed types and property IDs. + template_properties = ctx.client.get_template_properties(template_id) payload = {} + for term, values in terms.items(): - if term in template_properties: - property_details = template_properties[term] - payload[term] = [] - for value in values: - if not isinstance(value, dict): - value = {'value': value} - # The supplied data type doesn't match the template - if 'type' in value and value['type'] not in property_details['type']: - print(f'Data type "{value["type"]}" for term "{term}" not allowed by template') - break - elif 'type' not in value: - # Use default datatype from template if none is supplied - if len(property_details['type']) == 1: - value['type'] = property_details['type'][0] - # Use literal if allowed by template and data type not supplied - elif 'literal' in property_details['type']: - value['type'] = 'literal' - # Don't know what data type to use - else: - print(f'Specify data type for term "{term}"') - break - if "property_id" in value: - #don't format values that have already been formatted - payload[term].append(value) + if term not in template_properties: + # Terms outside the template are intentionally dropped — each + # collection defines which fields are relevant to its template. + logger.warning("Term %r not in template; skipping", term) + continue + + property_details = template_properties[term] + payload[term] = [] + + for value in values: + # Ensure value is a dict with at least a "value" key. + if not isinstance(value, dict): + value = {'value': value} + + # Validate the supplied data type against the template's allowed types. + if 'type' in value and value['type'] not in property_details['type']: + logger.warning( + "Data type %r for term %r not allowed by template; skipping value", + value['type'], term + ) + continue + + if 'type' not in value: + # Infer a data type from the template definition. + if len(property_details['type']) == 1: + # Only one type allowed — use it. + value['type'] = property_details['type'][0] + elif 'literal' in property_details['type']: + # Multiple types allowed; prefer "literal" as the default. + value['type'] = 'literal' else: - # Add a value formatted according to the data type - payload[term].append(omeka_auth.prepare_property_value(value, property_details['property_id'])) - # The supplied term is not in the template - else: - print(f'Term {term} not in template') + # Cannot determine a type; skip this value. + logger.warning("Cannot determine data type for term %r; skipping value",term) + continue + + if "property_id" in value: + # Value was already formatted by a prior call; append as-is to + # avoid double-formatting. + payload[term].append(value) + else: + # Format the value according to the template property definition. + payload[term].append( + ctx.client.prepare_property_value( + value, property_details['property_id'] + ) + ) + return payload -def prepare_property_value(value, property_id, label = ""): - ''' - Formats a property value according to its datatype as expected by Omeka. - The formatted value can be used in a payload to create a new item. +def filter_items(regex, pathlist): + """ + Filter a list of file paths to those matching a regex pattern. - Parameters: - * `value` - a dict containing a `value` and (optionally) a `type` - * `property_id` - the numeric identifier of the property - * `label` - a text label for the URI if `type` is "uri" - - Note that is no `type` is supplied, 'literal' will be used by default. - - Returns: - * a dict with values for `property_id`, `type`, and either `@id` or `@value`. - ''' - if not isinstance(value, dict): - value = {'value': value} - - try: - data_type = value['type'] - except KeyError: - data_type = 'literal' - - property_value = { - 'property_id': property_id, - 'type': data_type - } - - if data_type == 'resource:item': - property_value['@id'] = f'{self.api_url}/items/{value["value"]}' - property_value['value_resource_id'] = value['value'] - property_value['value_resource_name'] = 'items' - elif data_type == 'uri': - property_value['@id'] = value['value'] - if label == "": - property_value["o:label"] = value["value"].split("/")[-1] - else: - property_value["o:label"] = label - else: - property_value['@value'] = value['value'] - return property_value + Used by both entrypoint scripts to restrict processing to a subset of + files when the -r / --regex flag is passed on the command line. + Parameters: + * regex - regex pattern string, compiled with re.compile() + * pathlist - iterable of pathlib.Path or string paths to filter -def filter_items(regex, pathlist): + Returns a list containing only the paths whose stem (identifier, without + extension or directory components) matches the pattern. + """ reg = re.compile(regex) - return [p for p in pathlist if reg.search(str(p))] - -conf_path = get_dir("config/private.yml") -config = get_config(conf_path) -dev_config = get_config(conf_path, "development") -prod_config = get_config(conf_path, "production") - -omeka = OmekaAPIClient(config['omeka_server']) -omeka_auth = OmekaAPIClient( - api_url = config['omeka_server'], - key_identity = config['key_identity'], - key_credential = config['key_credential'] -) -parser = argparse.ArgumentParser() -parser.add_argument('-e', '--environment', required=False, default="development") -parser.add_argument('-m', '--media-skip', action='store_true', - help='Only ingest media not already ingested') -parser.add_argument('-r', '--regex', required=False, help = "Filter files with regex") -args = parser.parse_args() -template_number = config["resource_template"] -omeka_data_base = config["omeka_data_base"] \ No newline at end of file + return [p for p in pathlist if reg.search(p.stem)] + + +def filter_items_by_date(update_time, pathlist): + """ + Filter a list of output JSON file paths to those whose corresponding source + file has a modification time at or after update_time. + + Source files are expected at source//. relative to + the collection root (cwd), matching the Datura convention where the output + JSON stem equals the source filename stem (e.g. source/tei/abc123.xml -> + output/development/es/abc123.json). Items with no locatable source file are + included unconditionally so they are not silently dropped. + + Parameters: + * update_time - datetime object; only items with source mtime >= this are kept + * pathlist - iterable of pathlib.Path or string paths to filter + """ + source_base = Path.cwd() / "source" + # Build index once: stem -> list of source paths + source_index = {} + for sf in source_base.glob("*/*.*"): + source_index.setdefault(sf.stem, []).append(sf) + + result = [] + for p in pathlist: + identifier = Path(p).stem + source_files = source_index.get(identifier, []) + if not source_files: + logger.debug( + "No source file found for %r; including without date filter", identifier + ) + result.append(p) + continue + source_mtime = max( + datetime.fromtimestamp(sf.stat().st_mtime) for sf in source_files + ) + if source_mtime >= update_time: + result.append(p) + return result + + +def filter_items_by_format(format_type, pathlist): + """ + Filter a list of output JSON file paths to those whose source file lives + in the source// directory. + + Mirrors the Ruby DataManager convention: source files for a given format + are stored under source// (e.g. source/csv/, source/tei/). A JSON + output file belongs to that format if and only if a corresponding source + file exists at source//.*. + + Files with no match in source// are excluded. This correctly + drops stale JSON left in the output directory from earlier runs of a + different format. + + Parameters: + * format_type - string, e.g. "tei" or "csv" + * pathlist - iterable of pathlib.Path or string paths to filter + """ + source_dir = Path.cwd() / "source" / format_type + # Build the set of stems once + source_stems = {sf.stem for sf in source_dir.glob("*")} + return [p for p in pathlist if Path(p).stem in source_stems] + + +def filter_items_by_identifier(csv_rows, json_items): + """ + Filter a list of JSON item dicts to those whose identifier matches a regex. + + Used by both entrypoint scripts when -c / --csv-rows is passed on the + command line. Mirrors the Ruby FileCsv row filter: the regex is matched + against the item's "identifier" field (equivalent to the CSV id/identifier + column). + + Parameters: + * csv_rows - regex pattern string, compiled with re.compile() + * json_items - list of item dicts loaded from a Datura ES JSON file + + Returns a (possibly shorter) list containing only the items whose + "identifier" value matches the pattern. + """ + pat = re.compile(csv_rows) + return [item for item in json_items if pat.search(item.get("identifier") or "")] + + +def proceed_files(regex, pathlist): + """ + Filter a sorted list of file paths to those from the first match onward. + + Used by both entrypoint scripts when -p / --proceed is given with a value. + Mirrors the Ruby Helpers.proceed_files() behavior: sorts the list by stem, + finds exactly one matching file, and returns every file from that point to + the end of the list. + + Parameters: + * regex - regex pattern string to locate the starting file by stem + * pathlist - iterable of pathlib.Path objects to filter + + Exits with a descriptive error message if the regex matches zero files + (typo or stale checkpoint) or more than one file (ambiguous — refine the + regex so it matches exactly one starting point). + """ + sorted_paths = sorted(pathlist, key=lambda p: p.stem) + pat = re.compile(regex) + matches = [p for p in sorted_paths if pat.search(p.stem)] + if not matches: + print(f"ERROR: --proceed regex '{regex}' matched no files. Exiting.") + sys.exit(1) + if len(matches) > 1: + names = ", ".join(p.stem for p in matches) + print( + f"ERROR: --proceed regex '{regex}' matched {len(matches)} files ({names}). " + "Refine your regex to match exactly one file. Exiting." + ) + sys.exit(1) + idx = sorted_paths.index(matches[0]) + return sorted_paths[idx:] \ No newline at end of file diff --git a/lib/datura/python/omeka_context.py b/lib/datura/python/omeka_context.py new file mode 100644 index 000000000..a8ca24379 --- /dev/null +++ b/lib/datura/python/omeka_context.py @@ -0,0 +1,702 @@ +""" +omeka_context.py + +Central context object and exception hierarchy for the Omeka S ingestion pipeline. + +""" + +import importlib.util +import logging +from logging.handlers import RotatingFileHandler +import os +import re +import sys + +from field_definitions import get_fields +import time +from datetime import date, datetime +from pathlib import Path +from typing import Dict, List, Optional + +import yaml +from omeka_s_tools.api import OmekaAPIClient + +# Module-level logger. Using __name__ means log records from this module +# appear as "omeka_context" in the output, making it easy to filter. +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Logging setup +# --------------------------------------------------------------------------- + +# --- Colored console handler --- +CYAN = "\033[36m" +GREEN = "\033[32m" +RED = "\033[31m" +BRIGHTRED = "\033[1;31m" +YELLOW = "\033[33m" +RESET = "\033[0m" + +class ColoredConsoleHandler(logging.StreamHandler): + COLORS = { + logging.DEBUG: CYAN, + logging.INFO: GREEN, + logging.WARNING: YELLOW, + logging.ERROR: RED, + logging.CRITICAL: BRIGHTRED, + } + RESET = RESET + def emit(self, record): + color = self.COLORS.get(record.levelno, self.RESET) + record.msg = f"{color}{record.msg}{self.RESET}" + super().emit(record) + +def configure_logging(level="INFO"): + numeric_level = getattr(logging, level.upper(), logging.INFO) + + os.makedirs("logs", exist_ok=True) + + # File handler — verbose + file_handler = RotatingFileHandler( + "logs/python.log", maxBytes=5 * 1024 * 1024, backupCount=3 + ) + file_handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)-8s] %(name)s: %(message)s")) + file_handler.setLevel(logging.DEBUG) # always save everything to logs + + # Console handler — respects the requested level + console_handler = ColoredConsoleHandler() + console_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)-8s %(name)s: %(message)s")) + console_handler.setLevel(numeric_level) + + # Root logger + root = logging.getLogger() + root.setLevel(logging.DEBUG) # let handlers decide what to filter + root.addHandler(file_handler) + root.addHandler(console_handler) + +# --------------------------------------------------------------------------- +# Exception hierarchy +# --------------------------------------------------------------------------- + +class OmekaError(Exception): + """ + Base class for all Omeka pipeline errors. + + Catch this to handle any pipeline error generically, or catch a subclass + to handle a specific failure mode. All subclasses produce a descriptive + human-readable message so that log entries are self-explanatory without + requiring a full traceback. + """ + + +class OmekaConfigError(OmekaError): + """ + Raised when the pipeline cannot start due to a configuration problem. + + Common causes: + - config/private.yml does not exist in the collection directory + - The YAML file is malformed and cannot be parsed + - A required key (e.g. "omeka_server", "key_identity") is absent + + This is always a fatal error: the pipeline cannot connect to or + authenticate with Omeka S without a valid configuration, so the process + exits immediately rather than attempting to continue. + """ + +class OmekaAuthError(OmekaConfigError): + """ + Raised when the Omeka S API returns 401 Unauthorized or 403 Forbidden. + + This is always fatal: if credentials are wrong every API call will fail, + so the process exits immediately rather than accumulating per-item failures. + + Common causes: + - key_identity or key_credential is wrong or has been revoked + - The Omeka S instance URL points to the wrong server + """ + + +def _is_unauthorized(err): + """Return True if err is an HTTP 401 or 403 response error from the requests library.""" + try: + from requests.exceptions import HTTPError + return isinstance(err, HTTPError) and getattr( + getattr(err, "response", None), "status_code", None + ) in {401, 403} + except ImportError: + return False + + +class OmekaAPIError(OmekaError): + """ + Raised when an Omeka S API call fails for a specific item or resource. + + Unlike OmekaConfigError, this is typically a per-item failure. The run + continues and the error is accumulated via ctx.record_error() so that + a full summary is printed at the end of the run. + + Attributes: + * identifier - the CDRH identifier string of the item that failed, + or "unknown" if the identifier could not be determined + * operation - short description of the failing operation, e.g. "add_item" + * cause - the original exception raised by the API client + """ + def __init__(self, identifier, operation, cause): + self.identifier = identifier + self.operation = operation + self.cause = cause + super().__init__( + f"{operation} failed for {identifier!r}: {cause}" + ) + + +class OmekaMediaError(OmekaError): + """ + Raised when a media upload or deletion operation fails. + + Separated from OmekaAPIError so that callers can distinguish between failures + on item metadata (OmekaAPIError) and failures on associated media objects + (OmekaMediaError), which may warrant different recovery strategies. + """ + + +# --------------------------------------------------------------------------- +# Date parsing +# --------------------------------------------------------------------------- + +def parse_update_time(s): + """ + Parse a -u / --update date string into a datetime object. + + Accepts the same formats as the Ruby Datura -u flag: + * "today" - midnight of the current local date + * "2015-01-01" - date only + * "2015-01-01T18:24" - date and time + + Raises OmekaConfigError with a descriptive message if the string does not + match any expected format. + """ + if s == "today": + d = date.today() + return datetime(d.year, d.month, d.day) + for fmt in ("%Y-%m-%dT%H:%M", "%Y-%m-%d"): + try: + return datetime.strptime(s, fmt) + except ValueError: + continue + raise OmekaConfigError( + f"{RED}Invalid --update value {s!r}. " + f"Expected 'today', a date (2015-01-01), or date-time (2015-01-01T18:24).{RESET}" + ) + +# --------------------------------------------------------------------------- +# Regex parsing +# --------------------------------------------------------------------------- + +def validate_regex_arg(pattern, flag): + """ + Compile a regex string, raising OmekaConfigError immediately if invalid. + + Mirrors the Ruby Datura::Helpers.validate_regex check. Called in each + entrypoint's main() before OmekaContext is built, so that an invalid + -r or -c value exits with a clean error message rather than a traceback. + + Parameters: + * pattern - the regex string from the CLI argument + * flag - the flag name for the error message, e.g. "--regex" + """ + try: + re.compile(pattern) + except re.error as e: + raise OmekaConfigError( + f"Invalid regex for {flag} {pattern!r}: {e}" + ) + +# --------------------------------------------------------------------------- +# Context +# --------------------------------------------------------------------------- + +class OmekaContext: + """ + Encapsulates all configuration and shared state for one pipeline run. + + Create exactly one OmekaContext per entrypoint invocation using the class + method from_args(). Pass the resulting context object to every pipeline + function that needs configuration or API access. + + The context holds: + - Parsed, validated configuration from config/private.yml + - A single authenticated OmekaAPIClient (self.client) + - A property ID cache to avoid repeated API lookups per field per item + - An error accumulator that collects per-item failures without halting the run + + """ + + @classmethod + def from_args(cls, args): + """ + Build an OmekaContext from a parsed argparse.Namespace. + + Loads config/private.yml from the current working directory (the + collection root), validates that all required keys are present, and + initialises the authenticated API client. + + Parameters: + * args - argparse.Namespace produced by an entrypoint's _parse_args(). + Expected attributes: + .environment str "development" or "production" + .format_filter str optional format string for -f (directory-based) + filter, or None + .regex str optional file-filter pattern, or None + .update_time str optional date/time string for -u filter, or None + .csv_rows str optional identifier regex for -c filter, or None + .media_skip bool skip re-ingesting existing media + (html_and_media_ingest only; absent on + json_to_omeka args, defaults to False) + + Raises OmekaConfigError if the config file is missing, unparseable, + or is missing a required key. + """ + conf_path = Path.cwd() / "config" / "private.yml" + logger.debug("Loading config from %s", conf_path) + + # Load the top-level "default" section, which holds credentials and + # settings common to all environments. + default_config = cls._load_config(conf_path, env="default") + + # Load the environment-specific section (primarily contains item_set). + # If the section is absent (e.g. an unrecognised environment name was + # passed), raise an error and exit. + env_config = cls._load_config(conf_path, env=args.environment) + + # Merge so that environment-specific values override defaults, giving + # collections the ability to override any default key (e.g. + # resource_template, omeka_data_base) on a per-environment basis. + env_config = {**default_config, **env_config} + + raw_update = getattr(args, "update_time", None) + return cls( + env_config=env_config, + environment=args.environment, + # getattr with a default handles entrypoints that don't define + # every flag (e.g. json_to_omeka.py has no --media-skip). + regex=getattr(args, "regex", None), + media_skip=getattr(args, "media_skip", False), + update_time=parse_update_time(raw_update) if raw_update else None, + format_filter=getattr(args, "format_filter", None), + csv_rows=getattr(args, "csv_rows", None), + ) + + @staticmethod + def _load_config(path, env): + """ + Load a single environment section from a YAML config file. + + Parameters: + * path - pathlib.Path pointing to the YAML file + * env - the top-level key to extract, e.g. "default" or "development" + + Returns the dict for that section. + + Raises OmekaConfigError with a descriptive message on any I/O or parse + failure, so that operators know exactly what to fix without reading a + Python traceback. + """ + try: + with open(path) as f: + contents = yaml.safe_load(f) + except FileNotFoundError: + raise OmekaConfigError( + f"{RED}Config file not found: {path}. " + "Ensure config/private.yml exists in the collection directory " + f"and that you are running the script from the collection root.{RESET}" + ) + except yaml.YAMLError as exc: + raise OmekaConfigError( + f"{RED}Could not parse YAML in {path}: {exc}{RESET}" + ) + + if env not in contents: + raise OmekaConfigError(RED + + f"{RED}Environment section {env!r} not found in {path}. " + f"Available sections: {list(contents.keys())}{RESET}" + ) + + return contents[env] + + def __init__(self, env_config, environment, regex, media_skip, update_time=None, format_filter=None, csv_rows=None): + """ + Initialise the context. Prefer OmekaContext.from_args() over calling + this constructor directly except in tests. + + Parameters: + * env_config - merged dict: the "default" section of private.yml + overlaid with the environment-specific section so that + per-environment values take precedence over defaults. + Must contain omeka_server, key_identity, key_credential, + resource_template, omeka_data_base, and item_set. + * environment - "development" or "production" + * format_filter - optional format string to filter files by format (directory); + None means process all files in the output directory + * regex - optional regex string to filter input file paths; + None means process all files in the output directory + * media_skip - if True, items that already have 2+ media objects + (thumbnail + HTML) are skipped during media ingest + * update_time - optional datetime; if set, only items whose source file + mtime >= this value are processed (mirrors the -u flag + from the main Datura post command) + * csv_rows - optional regex string; if set, only items whose + "identifier" field matches are processed (mirrors the + -c flag from the main Datura post command) + """ + # ---- Validate required config keys -------------------------------- + # Validate up front so that failures are immediate and descriptive. + # Checks for all keys so user is alerted to any missing key at the outset. + required_keys = [ + "omeka_server", + "key_identity", + "key_credential", + "resource_template", + "omeka_data_base", + ] + missing_keys = [key for key in required_keys if key not in env_config] + if missing_keys: + raise OmekaConfigError( + f"{RED}Missing required config key(s): {missing_keys}. " + f"Check the 'default' or {environment!r} section of config/private.yml.{RESET}" + ) + + # ---- Validate environment-specific item_set --------------------------- + if "item_set" not in env_config: + raise OmekaConfigError( + f"{RED}Missing 'item_set' for environment {environment!r} in config/private.yml.\n" + "Add the item set ID for this environment before running. Example:\n\n" + f" {environment}:\n" + " item_set: 123\n\n" + "To find your item set ID, log into the Omeka S admin and navigate " + f"to Items > Item Sets.{RESET}" + ) + + # ---- Runtime flags ------------------------------------------------ + self.environment = environment + self.regex = regex + self.media_skip = media_skip + self.update_time = update_time + self.format_filter = format_filter + self.csv_rows = csv_rows + + # ---- Config values ------------------------------------------------ + self.template_number = env_config["resource_template"] + self.omeka_data_base = env_config["omeka_data_base"] + # iiif_server is optional — not all collections ingest thumbnails. + self.iiif_server = env_config.get("iiif_server", "") + # iiif_collection is optional — not all collections have different iiif collection names. + self.iiif_collection = env_config.get("iiif_collection", "") + + # Keep the merged config dict for the item_set_id property. + self._env_config = env_config + + # ---- Credentials (stored for reset_client) ------------------------ + # Stored privately so that credential strings are not accidentally + # printed, logged, or serialised through the public interface. + self._api_url = env_config["omeka_server"] + self._key_identity = env_config["key_identity"] + self._key_credential = env_config["key_credential"] + + # ---- API client --------------------------------------------------- + # Single authenticated client used for all API operations. + self.client = OmekaAPIClient( + api_url=self._api_url, + key_identity=self._key_identity, + key_credential=self._key_credential, + ) + logger.debug( + "OmekaContext initialised (environment=%r, template=%s)", + self.environment, + self.template_number, + ) + + # ---- Property ID cache -------------------------------------------- + # Maps Omeka term strings (e.g. "dcterms:title") to their numeric IDs. + # IDs are stable within a single Omeka S instance for the lifetime of + # a run, so fetching each term once is sufficient. + self._property_id_cache = {} # type: Dict[str, int] + + # ---- Error accumulator -------------------------------------------- + # Non-fatal per-item errors are appended here rather than aborting the + # run. report_errors() logs a consolidated summary at the end. + self._errors = [] # type: List[OmekaError] + + # ---- Field definitions -------------------------------------------- + # Load collection-specific field mappings once here. get_fields() returns + # a CustomFields subclass if scripts/python/field_overrides.py is present; + # otherwise the default FieldDefinitions instance. + self.fields = get_fields(omeka_data_base=self.omeka_data_base) + + # ---- Process function overrides ----------------------------------- + # Load collection-specific replacements for four pipeline functions: + # link_records, update_item_value, link_item_record (api_fields.py) + # build_thumbnail_url (html_and_media_ingest.py) + # + # Each attribute is None when no override is present; call sites + # resolve the default via: (ctx._fn_X or module.X)(ctx, ...) + # + # importlib.util is used to load the file by explicit path so that + # sys.path is not mutated and the module is not cached in sys.modules, + # keeping each context init independent. + self._fn_link_records = None + self._fn_update_item_value = None + self._fn_link_item_record = None + self._fn_build_thumbnail_url = None + + _process_override_path = Path.cwd() / "scripts" / "python" / "process_overrides.py" + _process_override_relative_path = "scripts/python/process_overrides.py" + if _process_override_path.exists(): + try: + _spec = importlib.util.spec_from_file_location( + "process_overrides", _process_override_path + ) + _po = importlib.util.module_from_spec(_spec) + _spec.loader.exec_module(_po) + self._fn_link_records = getattr(_po, "link_records", None) + self._fn_update_item_value = getattr(_po, "update_item_value", None) + self._fn_link_item_record = getattr(_po, "link_item_record", None) + self._fn_build_thumbnail_url = getattr(_po, "build_thumbnail_url", None) + _active = [ + n for n, f in [ + ("link_records", self._fn_link_records), + ("update_item_value", self._fn_update_item_value), + ("link_item_record", self._fn_link_item_record), + ("build_thumbnail_url", self._fn_build_thumbnail_url), + ] if f is not None + ] + if _active: + logger.warning( + "Process overrides found at %s; active overrides: %s", + _process_override_relative_path, + _active, + ) + except Exception as e: + raise OmekaConfigError( + f"Failed to load process overrides from {_process_override_relative_path}: {e}" + ) from e + + + # ----------------------------------------------------------------------- + # Properties + # ----------------------------------------------------------------------- + + @property + def item_set_id(self): + # type: () -> Optional[int] + """ + The Omeka item set ID for the current environment. + + Stored in the environment-specific config section so that development + and production ingests can target different item sets. + """ + return self._env_config.get("item_set") + + @property + def is_public(self): + # type: () -> bool + """ + Always returns False. Items are posted as private (visible only to logged-in Omeka + admins) regardless of environment. Visibility can be changed manually in the Omeka S + admin interface after ingest if needed. + + If it is desired to change this behavior at some future point such that items posted + to the production environment will be public by default, replace `return False` below + with `return self.environment == "production"`. + + """ + return False + + # ----------------------------------------------------------------------- + # API helpers + # ----------------------------------------------------------------------- + + def get_property_id(self, term): + # type: (str) -> int + """ + Return the numeric Omeka property ID for a term, using a per-run cache. + + The first call for a given term makes one API request and stores the + result. All subsequent calls return the cached integer immediately. + The cache is preserved across reset_client() calls because term-to-ID + mappings are stable within a single Omeka S instance. + + Parameters: + * term - Omeka property term string, e.g. "dcterms:title", "dh:collection" + """ + if term not in self._property_id_cache: + logger.debug("Fetching property ID for term %r (not yet cached)", term) + self._property_id_cache[term] = self.client.get_property_id(term) + return self._property_id_cache[term] + + def reset_client(self): + """ + Re-instantiate the authenticated API client with a fresh connection + to clear its HTTP response cache. (Otherwise Pass 2 would return the + stale responses from GET requests in Pass 1, per the OmekaAPIClient's + requests_cache session wrapper.) + + The property ID cache is intentionally preserved: term-to-ID mappings + do not change between passes, so clearing and re-fetching them would + waste API calls without any benefit. + """ + logger.debug("Resetting API client (property ID cache preserved)") + self.client = OmekaAPIClient( + api_url=self._api_url, + key_identity=self._key_identity, + key_credential=self._key_credential, + ) + + # ----------------------------------------------------------------------- + # Path resolution + # ----------------------------------------------------------------------- + + def resolve_path(self, relative): + # type: (str) -> Path + """ + Resolve a path relative to the current working directory (collection root). + Callers interpolate the environment into the path template: + + json_dir = ctx.resolve_path(f"output/{ctx.environment}/es") + + This ensures that passing -e production reads from output/production/ + rather than always using output/development/. + + Parameters: + * relative - path string relative to cwd, e.g. "output/development/es" + + Returns an absolute pathlib.Path. + """ + return (Path.cwd() / relative).resolve() + + # ----------------------------------------------------------------------- + # Error accumulation + # ----------------------------------------------------------------------- + + def record_error(self, err): + # type: (OmekaError) -> None + """ + Record a non-fatal per-item error without halting the run. + + Use for item-level failures (API errors, missing files, malformed data) + where the correct behaviour is to log the problem, skip the affected + item, and continue processing the rest of the batch. + + Fatal errors that make the entire run impossible (wrong credentials, + missing config file) should raise OmekaConfigError directly and let + the process exit with a traceback. + + Parameters: + * err - an OmekaError (or subclass) instance describing the failure + """ + cause = getattr(err, "cause", None) + if _is_unauthorized(cause): + raise OmekaAuthError(RED + + "Omeka S returned 401 Unauthorized or 403 Forbidden. " + "Check that key_identity and key_credential in config/private.yml are correct. " + "You may also need to be logged onto the VPN." + + RESET + ) from cause + logger.error(str(err)) + self._errors.append(err) + + def report_errors(self): + """ + Log a consolidated summary of all errors recorded during the run. + + Called by entrypoint scripts just before sys.exit(). If any errors + were recorded, the entrypoint should exit with code 1 so that the + calling Ruby process (system() in bin/post_omeka or bin/post_omeka_html) + can detect that the run completed with failures. + + If no errors were recorded, logs a single success message. + """ + if self._errors: + logger.warning("Run completed with %d error(s):", len(self._errors)) + for err in self._errors: + logger.warning(" %s", err) + else: + logger.info("Run completed successfully with no errors.") + +def finish_run(ctx, args, start_time): + """ + Report errors, print error count and timing, and exit. + + Called at the end of each Omeka entrypoint script's main() function. + Exits 0 on success, 1 if any errors were recorded. + + Parameters: + * ctx - OmekaContext whose _errors list is inspected + * args - argparse.Namespace (unused; kept for call-site compatibility) + * start_time - float from time.time() captured at the top of main() + """ + ctx.report_errors() + print(f"{len(ctx._errors)} Omeka posting error(s)") + elapsed = int(time.time() - start_time) + hours, rem = divmod(elapsed, 3600) + mins, secs = divmod(rem, 60) + print(f"{CYAN}Script finished in {hours:02d} hrs {mins:02d} mins {secs:02d} secs{RESET}") + sys.exit(1 if ctx._errors else 0) + +# --------------------------------------------------------------------------- +# Checkpoint helpers (-p / --proceed support) +# --------------------------------------------------------------------------- + +def checkpoint_path(ctx, label): + """ + Return the Path to the checkpoint file for this environment and pipeline. + + The checkpoint file records the stem of the last JSON file that was + successfully processed so that a subsequent run with -p (no value) can + resume from the same point rather than restarting from the beginning. + + Parameters: + * ctx - OmekaContext providing the current environment string + * label - pipeline label used to keep each script's checkpoint separate, + e.g. "omeka" for json_to_omeka or "omeka_html" for + html_and_media_ingest + """ + return Path.cwd() / "logs" / f"proceed_{label}_{ctx.environment}" + + +def read_checkpoint(ctx, label): + """ + Read the last-saved checkpoint stem for this pipeline and environment. + + Returns the identifier stem string written by the most recent + write_checkpoint() call, or None if no checkpoint file exists or the + file is empty (e.g. first run, or file was manually cleared). + + Parameters: + * ctx - OmekaContext + * label - pipeline label (see checkpoint_path) + """ + path = checkpoint_path(ctx, label) + if not path.exists(): + return None + content = path.read_text().strip() + return content if content else None + + +def write_checkpoint(stem, ctx, label): + """ + Save stem as the most recently processed item identifier for this pipeline. + + Called after each JSON file is processed so that a run interrupted + mid-way can be resumed with -p. Creates the logs/ directory if it does + not yet exist. + + Parameters: + * stem - identifier string (JSON file stem) to record, e.g. "abc123" + * ctx - OmekaContext + * label - pipeline label (see checkpoint_path) + """ + path = checkpoint_path(ctx, label) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f"{stem}\n") \ No newline at end of file diff --git a/lib/datura/python/omeka_overrides_example.py b/lib/datura/python/omeka_overrides_example.py deleted file mode 100644 index 58d16e828..000000000 --- a/lib/datura/python/omeka_overrides_example.py +++ /dev/null @@ -1,193 +0,0 @@ -#copy this file to omeka_overrides.py in your scripts/overrides file. Edit the return values as needed - -from field_definitions import FieldDefinitions - -class CustomFields(FieldDefinitions): - #these are the default field definitions, which may be overridden in specific projects - def title(self, json): - return json.get("title", None) - - def identifier(self, json): - return json.get("identifier", None) - - def collection(self, json): - return json.get("collection", None) - - def category(self, json): - return json.get("category", None) - - def category2(self, json): - return json.get("category2", None) - - def uriData(self, json): - return json.get("uri_data", None) - - def dcterms_type(self, json): - #note that "type" is a builtin function in Python - return json.get("type", None) - - def creator(self, json): - creator_names = [creator['name'] for creator in json.get("creator") or [] if 'name' in creator] - return creator_names - - def contributor(self, json): - contributor_names = [contributor['name'] for contributor in json.get("contributor") or [] if 'name' in contributor] - return contributor_names - - def date(self, json): - return json.get("date", None) - - def description(self, json): - return json.get("description", None) - - def dcterms_format(self, json): - #note that "format" is a builtin function in Python - return json.get("format", None) - - def relation(self, json): - relation_ids = [relation['id'] for relation in json.get("has_relation") or [] if 'name' in relation] - return relation_ids - - #citation fields - #TODO is citation always single-valued? if array might need to add code to deal with that - - def publisher(self, json): - return json.get("citation", {}).get("publisher", None) - - def biblID(self, json): - #note: this field is not yet implemented in the schema - return json.get("citation", {}).get("id", None) - - def biblTitle(self, json): - return json.get("citation", {}).get("title", None) - - def biblPubPlace(self, json): - return json.get("citation", {}).get("pubplace", None) - - def issue(self, json): - return json.get("citation", {}).get("issue", None) - - def pageStart(self, json): - return json.get("citation", {}).get("page_start", None) - - def pageEnd(self, json): - return json.get("citation", {}).get("page_end", None) - - def section(self, json): - return json.get("citation", {}).get("section", None) - - def volume(self, json): - return json.get("citation", {}).get("volume", None) - - def biblTitleA(self, json): - return json.get("citation", {}).get("title_a", None) - - def biblTitleM(self, json): - return json.get("citation", {}).get("title_m", None) - - def biblTitleJ(self, json): - return json.get("citation", {}).get("title_j", None) - - def rightsHolder(self, json): - return json.get("rights_holder", None) - - def license(self, json): - return json.get("rights", None) - - def subject(self, json): - return json.get("subjects", None) - - def topic(self, json): - return json.get("topics", None) - - def category3(self, json): - return json.get("category3", None) - - def category4(self, json): - return json.get("category4", None) - - def category5(self, json): - return json.get("category5", None) - - def note(self, json): - return json.get("notes", None) - - def abstract(self, json): - return json.get("abstract", None) - - def keyword(self, json): - return json.get("keywords", None) - - def keyword2(self, json): - return json.get("keywords2", None) - - def keyword3(self, json): - return json.get("keywords3", None) - - def keyword4(self, json): - return json.get("keywords4", None) - - def keyword5(self, json): - return json.get("keywords5", None) - - def source(self, json): - return json.get("has_source") and json.get("has_source", {}).get("title") - - - def medium(self, json): - return json.get("medium", None) - - def extent(self, json): - return json.get("extent", None) - - def language(self, json): - return json.get("language", None) - - def box(self, json): - return json.get("container_box", None) - - def folder(self, json): - return json.get("container_folder", None) - - def name(self, json): - person_names = [person['name'] for person in json.get("person") or [] if 'name' in person] - return person_names - - def spatial_short_name(self, json): - places = [json["spatial"]] if isinstance(json["spatial"], dict) else json["spatial"] - if places: - place_names = [place['short_name'] for place in places or [] if 'short_name' in place] - return place_names - - def correspSentName(self, json): - return json.get("correspSentName_omeka_s", None) - - def correspSentPlace(self, json): - return json.get("correspSentPlace_omeka_s", None) - - def correspSentDate(self, json): - return json.get("correspSentDate_omeka_s", None) - - def correspDeliveredName(self, json): - return json.get("correspDeliveredName_omeka_s", None) - - def correspDeliveredPlace(self, json): - return json.get("correspDeliveredPlace_omeka_s", None) - - def correspDeliveredDate(self, json): - return json.get("correspDeliveredDate_omeka_s", None) - - def distributor(self, json): - return json.get("distributor_omeka_s", None) - - def authority(self, json): - return json.get("authority_omeka_s", None) - - def biblNote(self, json): - return json.get("biblNote_omeka_s", None) - - def annotationsText(self, json): - return json.get("annotations_text", None) - - def itemText(self, json): - return json.get("text", None) diff --git a/lib/datura/python/process_overrides_example.py b/lib/datura/python/process_overrides_example.py new file mode 100644 index 000000000..90bcf4ccf --- /dev/null +++ b/lib/datura/python/process_overrides_example.py @@ -0,0 +1,54 @@ +# Copy this file to process_overrides.py in your scripts/python directory. +# Define only the functions whose behavior differs from the defaults in +# api_fields.py (link_records, update_item_value, link_item_record) and +# html_and_media_ingest.py (build_thumbnail_url). Functions not defined +# here fall back to the default implementations automatically. +# +# Each function must match the signature of its default counterpart exactly. +# To extend rather than replace a default, import it directly: +# +# from api_fields import link_item_record as default_link_item_record +# +# Overriding link_item_record here is automatically picked up by the default +# link_records without needing to override link_records as well. + + +# Pattern 1: override build_thumbnail_url to use a different IIIF path format +# def build_thumbnail_url(ctx, json_item): +# collection = ctx.iiif_collection or json_item.get("collection", "") +# cover_image = json_item.get("cover_image") +# if not cover_image: +# return None +# remote = f"{ctx.iiif_server}/iiif/2/{collection}%2F{cover_image}/full/!150,150/0/default.jpg" +# local_name = f"{collection}_{cover_image}" +# return remote, local_name + + +# Pattern 2: override link_item_record to use a custom filter property +# def link_item_record(ctx, item, key, values, item_set=False, filter_property="dcterms:identifier"): +# from api_fields import link_item_record as default_link_item_record +# # Use a collection-specific property for lookups instead of dcterms:identifier +# return default_link_item_record(ctx, item, key, values, item_set, filter_property="dh:slug") + + +# Pattern 3: override update_item_value to skip a field under certain conditions +# def update_item_value(ctx, item, key, value, datatype="literal"): +# from api_fields import update_item_value as default_update_item_value +# # Do not post empty string values for any field +# if value == "": +# value = None +# return default_update_item_value(ctx, item, key, value, datatype) + + +# Pattern 4: override link_records to add a custom relationship type +# def link_records(ctx, row, existing_item): +# from api_fields import link_records as default_link_records +# existing_item = default_link_records(ctx, row, existing_item) +# # Add collection-specific "dh:relatedProject" links +# try: +# from api_fields import link_item_record +# project_ids = [p["id"] for p in row["related_projects"]] +# link_item_record(ctx, existing_item, "dh:relatedProject", project_ids) +# except (KeyError, TypeError): +# pass +# return existing_item \ No newline at end of file diff --git a/lib/datura/python/xslt_transform.py b/lib/datura/python/xslt_transform.py new file mode 100644 index 000000000..3295d1aac --- /dev/null +++ b/lib/datura/python/xslt_transform.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +import argparse +import sys +from pathlib import Path + +try: + import saxonche +# this should now be redundant with the addition of check_xslt_dependency to data_manager.rb +# but I am leaving it here for now as a backstop and in case we decide to return to this approach +# (the downside is that it will print an error for every call) +except ImportError as e: + print(f"XSLT transformation error: {e}", file=sys.stderr) + sys.exit(1) + +def parse_args(): + # create the parser + parser = argparse.ArgumentParser(description="XSLT transform using saxonche") + # declare the arguments + parser.add_argument("--input", required=True, help="Path to source XML file") + parser.add_argument("--xsl", required=True, help="Path to XSL stylesheet") + parser.add_argument("--output", required=False, help="Path to write output file") + parser.add_argument("--param", action="append", default=[], nargs=2, metavar=("KEY", "VALUE"), + help="XSL parameter as separate key and value (repeatable)") + parser.add_argument("--base-output-uri", required=False, dest="base_output_uri", + help="Base URI for xsl:result-document secondary outputs (file:// URI or directory path)") + # parse sys.argv, validate args, return namespace object with attributes + return parser.parse_args() + + +def run_transform(input_path, xsl_path, params, output_path=None, base_output_uri=None): + # create Saxon processor using Home Edition tier + with saxonche.PySaxonProcessor(license=False) as proc: + # create XSLT 3.0 processor + xslt_proc = proc.new_xslt30_processor() + # iterate list of [key, value] pairs + for key, value in params: + xslt_proc.set_parameter(key, proc.make_string_value(value)) + # parse and compile xsl file into executable + executable = xslt_proc.compile_stylesheet(stylesheet_file=xsl_path) + # set base output URI for xsl:result-document secondary outputs if provided; + # convert plain directory path to file:// URI if needed + if base_output_uri: + if not base_output_uri.startswith("file://"): + base_output_uri = Path(base_output_uri).resolve().as_uri() + "/" + executable.set_base_output_uri(base_output_uri) + # run transformation and return output as string + result = executable.transform_to_string(source_file=input_path) + # when a base output URI is set, None primary output is expected — all output + # went to xsl:result-document secondary files; only error when neither was produced + if result is None: + if base_output_uri: + return + raise RuntimeError("Transformation produced no output") + if output_path: + Path(output_path).write_text(result, encoding="utf-8") + print(result, end="") + + +def main(): + args = parse_args() + # catch exceptions + try: + run_transform( + input_path=args.input, + xsl_path=args.xsl, + params=args.param, + output_path=args.output, + base_output_uri=args.base_output_uri, + ) + except Exception as e: + print(f"XSLT transformation error: {e}", file=sys.stderr) + sys.exit(1) + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/lib/datura/solr_poster.rb b/lib/datura/solr_poster.rb index eb4434a88..c9698e294 100644 --- a/lib/datura/solr_poster.rb +++ b/lib/datura/solr_poster.rb @@ -13,8 +13,6 @@ def initialize(url, commit=true) end end - # TODO this is very similar to the below _by_regex function - # so could stick them together later def clear_index del_str = "*:*" res = post_xml(del_str) @@ -53,11 +51,11 @@ def commit_solr end def post(content, type) - url = URI.parse(@url) - http = Net::HTTP.new(url.host, url.port) - http.use_ssl = @url[/^https/] ? true : false + uri = URI.parse(@url) + http = Net::HTTP.new(uri.host, uri.port) + http.use_ssl = true if uri.scheme == "https" http.open_timeout = 10 - request = Net::HTTP::Post.new(url.request_uri) + request = Net::HTTP::Post.new(uri.request_uri) request.body = content request["Content-Type"] = type http.request(request) @@ -65,7 +63,6 @@ def post(content, type) # post_file # assumes xml file because of historical usage of this script - # TODO refactor? def post_file(file_location) file = IO.read(file_location) post_xml(file) diff --git a/lib/datura/to_es/csv_to_es.rb b/lib/datura/to_es/csv_to_es.rb index 5625c6e84..9ca504084 100644 --- a/lib/datura/to_es/csv_to_es.rb +++ b/lib/datura/to_es/csv_to_es.rb @@ -41,7 +41,7 @@ def create_json end def get_id - @row["id"] || @row["identifier"] || "" + @row["id"] || @row["identifier"] || @row["Identifier"] || "" end def preprocessing diff --git a/lib/datura/to_es/csv_to_es/fields.rb b/lib/datura/to_es/csv_to_es/fields.rb index 6adcfc23c..b447b6762 100644 --- a/lib/datura/to_es/csv_to_es/fields.rb +++ b/lib/datura/to_es/csv_to_es/fields.rb @@ -196,7 +196,7 @@ def text_additional end def title - @row["title"] + @row["title"] || @row["Title"] end def title_sort diff --git a/lib/xslt/tei_to_html/lib/formatting.xsl b/lib/xslt/tei_to_html/lib/formatting.xsl index f26b408be..96dd1b4c5 100644 --- a/lib/xslt/tei_to_html/lib/formatting.xsl +++ b/lib/xslt/tei_to_html/lib/formatting.xsl @@ -558,10 +558,16 @@ %2F - .jpg/full/! - - , - + .jpg/full/ + + + ! + + , + + + full + /0/default.jpg @@ -577,10 +583,16 @@ %252F - .jpg/full/! - - , - + .jpg/full/ + + + ! + + , + + + full + /0/default.jpg diff --git a/lib/xslt/tei_to_html/lib/personography_encyclopedia.xsl b/lib/xslt/tei_to_html/lib/personography_encyclopedia.xsl index d15066563..c833f082e 100644 --- a/lib/xslt/tei_to_html/lib/personography_encyclopedia.xsl +++ b/lib/xslt/tei_to_html/lib/personography_encyclopedia.xsl @@ -39,9 +39,8 @@ - - - + + diff --git a/requirements.in b/requirements.in new file mode 100644 index 000000000..4ac4912ab --- /dev/null +++ b/requirements.in @@ -0,0 +1,4 @@ +# requirements.in +omeka_s_tools @ git+https://github.com/CDRH/omeka_s_tools.git@will_changes +pyyaml>=6.0.2,<7.0 +saxonche>=12.9,<13.0 # 12.5.0 wheels unavailable for recent Python/platform updates; 12.9 is latest stable 12.x per Saxonica docs diff --git a/requirements.txt b/requirements.txt index 9cb662327..e0616ca8d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,16 +1,44 @@ -attrs==25.3.0 -cattrs==25.1.1 -certifi==2025.8.3 -charset-normalizer==3.4.3 -dotenv==0.9.9 -idna==3.10 -Markdown==3.8.2 -omeka-s-tools==0.3.0 -packaging==25.0 -platformdirs==4.4.0 -python-dotenv==1.1.1 -requests==2.32.5 -requests-cache==1.2.1 -typing_extensions==4.15.0 -url-normalize==2.2.1 -urllib3==2.5.0 +# +# This file is autogenerated by pip-compile with Python 3.14 +# by the following command: +# +# pip-compile requirements.in +# +attrs==26.1.0 + # via + # cattrs + # requests-cache +cattrs==26.1.0 + # via requests-cache +certifi==2026.6.17 + # via requests +charset-normalizer==3.4.9 + # via requests +idna==3.18 + # via + # requests + # url-normalize +omeka-s-tools @ git+https://github.com/CDRH/omeka_s_tools.git@will_changes + # via -r requirements.in +packaging==26.2 + # via omeka-s-tools +platformdirs==4.10.0 + # via requests-cache +pyyaml==6.0.3 + # via -r requirements.in +requests==2.34.2 + # via + # omeka-s-tools + # requests-cache +requests-cache==1.3.3 + # via omeka-s-tools +saxonche==12.9.0 + # via -r requirements.in +typing-extensions==4.16.0 + # via cattrs +url-normalize==3.0.0 + # via requests-cache +urllib3==2.7.0 + # via + # requests + # requests-cache diff --git a/test/helpers_test.rb b/test/helpers_test.rb index 0fa3ca13f..3fa51c168 100644 --- a/test/helpers_test.rb +++ b/test/helpers_test.rb @@ -103,6 +103,84 @@ def test_regex_files # return a specific id files = Datura::Helpers.regex_files(test_files, "cat.let0001") assert_equal 1, files.length + + # invalid regex: exits with error + assert_raises(SystemExit) do + Datura::Helpers.regex_files(test_files, "[unclosed") + end + end + + def test_proceed_files + test_files = %w[ + /path/to/cody.book.002.xml + /path/to/cat.let0001.xml + /path/to/cody.book.001.xml + /path/to/transmiss.mem.001.xml + /path/to/cody.news.001.xml + ] + + # exact match on one file: returns that file and all alphabetically after it + # alphabetical order: cat.let0001, cody.book.001, cody.book.002, cody.news.001, transmiss.mem.001 + files = Datura::Helpers.proceed_files(test_files, "cody\.book\.002") + basenames = files.map { |f| File.basename(f, ".*") } + assert_equal %w[cody.book.002 cody.news.001 transmiss.mem.001], basenames + + # match on first file alphabetically: returns all files + files = Datura::Helpers.proceed_files(test_files, "cat\.let0001") + assert_equal 5, files.length + + # match on last file: returns only that file + files = Datura::Helpers.proceed_files(test_files, "transmiss\.mem\.001") + assert_equal 1, files.length + assert_equal "transmiss.mem.001", File.basename(files.first, ".*") + + # no match: exits with error + assert_raises(SystemExit) do + Datura::Helpers.proceed_files(test_files, "zzz_no_such_file") + end + + # multiple matches: exits with error + assert_raises(SystemExit) do + Datura::Helpers.proceed_files(test_files, "cody") + end + + # invalid regex: exits with error + assert_raises(SystemExit) do + Datura::Helpers.proceed_files(test_files, "[unclosed") + end + end + + def test_checkpoint_helpers + Dir.mktmpdir do |tmpdir| + opts = { + "collection_dir" => tmpdir, + "environment" => "test" + } + log_dir = File.join(tmpdir, "logs") + FileUtils.mkdir_p(log_dir) + expected_path = File.join(log_dir, "proceed_test") + + assert_equal expected_path, Datura::Helpers.checkpoint_path(opts) + + # returns nil when file does not exist + assert_nil Datura::Helpers.read_checkpoint(opts) + + # creates file with correct content + Datura::Helpers.write_checkpoint("let0050", opts) + assert File.exist?(expected_path) + assert_equal "let0050", File.read(expected_path).strip + + # returns the stored basename + assert_equal "let0050", Datura::Helpers.read_checkpoint(opts) + + # overwrites on second write + Datura::Helpers.write_checkpoint("let0100", opts) + assert_equal "let0100", Datura::Helpers.read_checkpoint(opts) + + # returns nil for empty/whitespace file + File.write(expected_path, " \n") + assert_nil Datura::Helpers.read_checkpoint(opts) + end end def test_should_update?