Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
7830dfc
core: guard copy_or_move when no location path matches dupe parent
SAY-5 May 12, 2026
2520c42
Set the application DesktopFileName
nbeaver May 10, 2026
bf0a1d1
Start PyQt5 -> PyQt6 migration
arsenetar Jul 7, 2022
05cd156
More updates for pyqt6
arsenetar Jul 8, 2022
a1a0447
Next batch of qt6 upgrades
arsenetar Jul 9, 2022
0cc04c2
More updates mainly in preferences
arsenetar Jul 9, 2022
d43b0b2
Port Qt frontend from PyQt5 to PyQt6
bgermann May 9, 2026
a47665d
Patches from <glubsy@users.noreply.github.com>
Aug 2, 2026
a7a5a17
Add StartupWMClass to desktop files to ensure DupeGuru icon is shown …
nbeaver May 5, 2026
eea76f4
perf: 16x faster picture scanning via pre-scaling and parallel prepar…
codingfrog Feb 15, 2026
d0ac873
feat: add scan speed preference for picture mode (Accurate/Balanced/T…
codingfrog Feb 15, 2026
418a6e6
fix: prevent SQL injection in cache modules
codingfrog Feb 14, 2026
0edf4cc
fix: prevent XXE attacks in XML parsing
codingfrog Feb 14, 2026
f4488bc
fix: prevent command injection in custom command feature
codingfrog Feb 14, 2026
abc0287
fix: Update columns.po
ojczeo Jul 4, 2024
0b3b886
fix: Update core.po
ojczeo Jul 4, 2024
c54d1af
Update core.po
microniko Mar 1, 2024
b25bf3c
Update columns.po
microniko Mar 1, 2024
e8ff386
Update ui.po
microniko Mar 1, 2024
4f1326a
Update Russian language
PushKK Jun 24, 2023
a6d6048
add dupeguru namespace to avoid naming conflicts
Aug 2, 2026
7a07a3d
force Qt6 to use the X11 interfaces since Fedora now runs wayland
Aug 2, 2026
2a031c8
allow send2trash 2.1.0 which is in f45
Aug 2, 2026
cf4d0b4
move core,hscommon,qt down one level
Aug 2, 2026
e8f50a9
add dupeguru namespace in all imports
Aug 2, 2026
d6e2978
update changelog
Aug 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
name: Build Cpp
run: |
sudo apt-get update
sudo apt-get install python3-pyqt5
sudo apt-get install python3-pyqt6
make modules
- if: matrix.language == 'python'
name: Autobuild
Expand Down
18 changes: 10 additions & 8 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
PYTHON ?= python3
PYTHON_VERSION_MINOR := $(shell ${PYTHON} -c "import sys; print(sys.version_info.minor)")
PYRCC5 ?= pyrcc5
RCC ?= rcc
REQ_MINOR_VERSION = 7
PREFIX ?= /usr/local

Expand Down Expand Up @@ -35,15 +35,15 @@ endif
# Our build scripts are not very "make like" yet and perform their task in a bundle. For now, we
# use one of each file to act as a representative, a target, of these groups.

packages = hscommon core qt
packages = dupeguru/hscommon dupeguru/core dupeguru/qt
localedirs = $(wildcard locale/*/LC_MESSAGES)
pofiles = $(wildcard locale/*/LC_MESSAGES/*.po)
mofiles = $(patsubst %.po,%.mo,$(pofiles))

vpath %.po $(localedirs)
vpath %.mo $(localedirs)

all: | env i18n modules qt/dg_rc.py
all: | env i18n modules dupeguru/qt/dg_rc.py
@echo "Build complete! You can run dupeGuru with 'make run'"

run:
Expand All @@ -60,8 +60,8 @@ ifndef NO_VENV
@${PYTHON} -m venv -h > /dev/null || \
echo "Creation of our virtualenv failed. If you're on Ubuntu, you probably need python3-venv."
endif
@${PYTHON} -c 'import PyQt5' >/dev/null 2>&1 || \
{ echo "PyQt 5.4+ required. Install it and try again. Aborting"; exit 1; }
@${PYTHON} -c 'import PyQt6' >/dev/null 2>&1 || \
{ echo "PyQt 6.3+ required. Install it and try again. Aborting"; exit 1; }

env: | reqs
ifndef NO_VENV
Expand All @@ -76,8 +76,10 @@ endif
build/help: | env
$(VENV_PYTHON) build.py --doc

qt/dg_rc.py: qt/dg.qrc
$(PYRCC5) qt/dg.qrc > qt/dg_rc.py
dupeguru/qt/dg_rc.py: dupeguru/qt/dg.qrc
$(RCC) -g python dupeguru/qt/dg.qrc -o dupeguru/qt/dg_rc.py
sed -i 's/from PySide2/from PyQt6/' dupeguru/qt/dg_rc.py
sed -i 's/from PySide6/from PyQt6/' dupeguru/qt/dg_rc.py

i18n: $(mofiles)

Expand Down Expand Up @@ -118,6 +120,6 @@ uninstall:
clean:
-rm -rf build
-rm locale/*/LC_MESSAGES/*.mo
-rm core/pe/*.$(SO) qt/pe/*.$(SO)
-rm dupeguru/core/pe/*.$(SO) dupeguru/qt/pe/*.$(SO)

.PHONY: clean normpo mergepot modules i18n reqs run pyc install uninstall all
7 changes: 2 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,18 +32,15 @@ For macos instructions (qt version) see the [macOS Instructions](macos.md).

### Prerequisites
* [Python 3.7+][python]
* PyQt5
* PyQt6

### System Setup
When running in a linux based environment the following system packages or equivalents are needed to build:
* python3-pyqt5
* pyqt5-dev-tools (on some systems, see note)
* python3-pyqt6
* python3-venv (only if using a virtual environment)
* python3-dev
* build-essential

Note: On some linux systems pyrcc5 is not put on the path when installing python3-pyqt5, this will cause some issues with the resource files (and icons). These systems should have a respective pyqt5-dev-tools package, which should also be installed. The presence of pyrcc5 can be checked with `which pyrcc5`. Debian based systems need the extra package, and Arch does not.

To create packages the following are also needed:
* python3-setuptools
* debhelper
Expand Down
2 changes: 1 addition & 1 deletion Windows.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ To build with a different python version 3.7 vs 3.8 or 32 bit vs 64 bit specify
### With makefile
It is possible to build dupeGuru with the makefile on windows using a compatable POSIX environment. The following steps have been tested using [msys2][msys2]. Before running make:
1. Install msys2 or other POSIX environment
2. Install PyQt5 globally via pip
2. Install PyQt6 globally via pip
3. Use the respective console for msys2 it is `msys2 msys`

Then the following execution of the makefile should work. Pass the correct value for PYTHON to the makefile if not on the path as python3.
Expand Down
8 changes: 4 additions & 4 deletions build.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@
import shutil
from multiprocessing import Pool

from hscommon import sphinxgen
from hscommon.build import (
from dupeguru.hscommon import sphinxgen
from dupeguru.hscommon.build import (
add_to_pythonpath,
print_and_do,
fix_qt_resource_file,
)
from hscommon import loc
from dupeguru.hscommon import loc
import subprocess


Expand Down Expand Up @@ -130,7 +130,7 @@ def build_normal():
build_localizations()
print("Building Qt stuff")
Path("qt", "dg_rc.py").unlink(missing_ok=True)
print_and_do("pyrcc5 {} > {}".format(Path("qt", "dg.qrc"), Path("qt", "dg_rc.py")))
print_and_do("rcc -g python {} -o {}".format(Path("qt", "dg.qrc"), Path("qt", "dg_rc.py")))
fix_qt_resource_file(Path("qt", "dg_rc.py"))
build_help()

Expand Down
1 change: 0 additions & 1 deletion core/me/__init__.py

This file was deleted.

1 change: 0 additions & 1 deletion core/se/__init__.py

This file was deleted.

1 change: 0 additions & 1 deletion core/tests/conftest.py

This file was deleted.

File renamed without changes.
85 changes: 43 additions & 42 deletions core/app.py → dupeguru/core/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,34 +9,34 @@
import os
import os.path as op
import logging
import shlex
import subprocess
import re
import shutil
from pathlib import Path

from send2trash import send2trash
from hscommon.jobprogress import job
from hscommon.notify import Broadcaster
from hscommon.conflict import smart_move, smart_copy
from hscommon.gui.progress_window import ProgressWindow
from hscommon.util import delete_if_empty, first, escape, nonone, allsame
from hscommon.trans import tr
from hscommon import desktop

from core import se, me, pe
from core.pe.photo import get_delta_dimensions
from core.util import cmp_value, fix_surrogate_encoding
from core import directories, results, export, fs, prioritize
from core.ignore import IgnoreList
from core.exclude import ExcludeDict as ExcludeList
from core.scanner import ScanType
from core.gui.deletion_options import DeletionOptions
from core.gui.details_panel import DetailsPanel
from core.gui.directory_tree import DirectoryTree
from core.gui.ignore_list_dialog import IgnoreListDialog
from core.gui.exclude_list_dialog import ExcludeListDialogCore
from core.gui.problem_dialog import ProblemDialog
from core.gui.stats_label import StatsLabel
from dupeguru.hscommon.jobprogress import job
from dupeguru.hscommon.notify import Broadcaster
from dupeguru.hscommon.conflict import smart_move, smart_copy
from dupeguru.hscommon.gui.progress_window import ProgressWindow
from dupeguru.hscommon.util import delete_if_empty, first, escape, nonone, allsame
from dupeguru.hscommon.trans import tr
from dupeguru.hscommon import desktop

from dupeguru.core import se, me, pe
from dupeguru.core.pe.photo import get_delta_dimensions
from dupeguru.core.util import cmp_value, fix_surrogate_encoding
from dupeguru.core import directories, results, export, fs, prioritize
from dupeguru.core.ignore import IgnoreList
from dupeguru.core.exclude import ExcludeDict as ExcludeList
from dupeguru.core.scanner import ScanType
from dupeguru.core.gui.deletion_options import DeletionOptions
from dupeguru.core.gui.details_panel import DetailsPanel
from dupeguru.core.gui.directory_tree import DirectoryTree
from dupeguru.core.gui.ignore_list_dialog import IgnoreListDialog
from dupeguru.core.gui.exclude_list_dialog import ExcludeListDialogCore
from dupeguru.core.gui.problem_dialog import ProblemDialog
from dupeguru.core.gui.stats_label import StatsLabel

HAD_FIRST_LAUNCH_PREFERENCE = "HadFirstLaunch"
DEBUG_MODE_PREFERENCE = "DebugMode"
Expand Down Expand Up @@ -182,7 +182,10 @@
self.view.create_results_window()

def _get_picture_cache_path(self):
cache_name = "cached_pictures.db"
prescale = self.options.get("picture_prescale", 1)
# Balanced (1) = no suffix for backward compat with existing caches
suffixes = ["_accurate", "", "_turbo"]
cache_name = f"cached_pictures{suffixes[prescale]}.db"
return op.join(self.appdata, cache_name)

def _get_dupe_sort_key(self, dupe, get_group, key, delta):
Expand Down Expand Up @@ -433,7 +436,7 @@
if dest_type in {DestType.RELATIVE, DestType.ABSOLUTE}:
# no filename, no windows drive letter
source_base = source_path.relative_to(source_path.anchor).parent
if dest_type == DestType.RELATIVE:
if dest_type == DestType.RELATIVE and location_path is not None:
source_base = source_base.relative_to(location_path.relative_to(location_path.anchor))
dest_path = dest_path.joinpath(source_base)
if not dest_path.exists():
Expand Down Expand Up @@ -544,24 +547,22 @@
dupes = self.selected_dupes
refs = [self.results.get_group_of_duplicate(dupe).ref for dupe in dupes]
for dupe, ref in zip(dupes, refs):
dupe_cmd = cmd.replace("%d", str(dupe.path))
dupe_cmd = dupe_cmd.replace("%r", str(ref.path))
match = re.match(r'"([^"]+)"(.*)', dupe_cmd)
if match is not None:
# This code here is because subprocess. Popen doesn't seem to accept, under Windows,
# executable paths with spaces in it, *even* when they're enclosed in "". So this is
# a workaround to make the damn thing work.
exepath, args = match.groups()
path, exename = op.split(exepath)
p = subprocess.Popen(
exename + args, shell=True, cwd=path, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
output = p.stdout.read()
logging.info("Custom command %s %s: %s", exename, args, output)
else:
p = subprocess.Popen(dupe_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
# Tokenize the command template safely before substituting paths
try:
cmd_parts = shlex.split(cmd)
except ValueError as e:
logging.error("Invalid custom command syntax: %s", e)

Check failure on line 554 in dupeguru/core/app.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Use "logging.exception()" instead.

See more on https://sonarcloud.io/project/issues?id=arsenetar_dupeguru&issues=AZ_EQbZJVL7xxLuooKpS&open=AZ_EQbZJVL7xxLuooKpS&pullRequest=1407
self.view.show_message(tr("Invalid custom command syntax: {}").format(str(e)))
return
# Replace placeholders in each token individually to prevent injection
cmd_parts = [part.replace("%d", str(dupe.path)).replace("%r", str(ref.path)) for part in cmd_parts]
try:
p = subprocess.Popen(cmd_parts, shell=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.stdout.read()
logging.info("Custom command %s: %s", dupe_cmd, output)
logging.info("Custom command %s: %s", cmd_parts, output)
except OSError as e:
logging.error("Failed to execute custom command %s: %s", cmd_parts, e)

Check failure on line 564 in dupeguru/core/app.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Use "logging.exception()" instead.

See more on https://sonarcloud.io/project/issues?id=arsenetar_dupeguru&issues=AZ_EQbZJVL7xxLuooKpT&open=AZ_EQbZJVL7xxLuooKpT&pullRequest=1407
self.view.show_message(tr("Failed to execute custom command: {}").format(str(e)))

def load(self):
"""Load directory selection and ignore list from files in appdata.
Expand Down
11 changes: 6 additions & 5 deletions core/directories.py → dupeguru/core/directories.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,15 @@

import os
from xml.etree import ElementTree as ET
from defusedxml.ElementTree import parse as safe_parse
import logging
from pathlib import Path

from hscommon.jobprogress import job
from hscommon.util import FileOrPath
from hscommon.trans import tr
from dupeguru.hscommon.jobprogress import job
from dupeguru.hscommon.util import FileOrPath
from dupeguru.hscommon.trans import tr

from core import fs
from dupeguru.core import fs

__all__ = [
"Directories",
Expand Down Expand Up @@ -246,7 +247,7 @@ def load_from_file(self, infile):
:param file infile: path or file pointer to XML generated through :meth:`save_to_file`
"""
try:
root = ET.parse(infile).getroot()
root = safe_parse(infile).getroot()
except Exception:
return
for rdn in root.iter("root_directory"):
Expand Down
6 changes: 3 additions & 3 deletions core/engine.py → dupeguru/core/engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@
from collections import defaultdict, namedtuple
from unicodedata import normalize

from hscommon.util import flatten, multi_replace
from hscommon.trans import tr
from hscommon.jobprogress import job
from dupeguru.hscommon.util import flatten, multi_replace
from dupeguru.hscommon.trans import tr
from dupeguru.hscommon.jobprogress import job

(
WEIGHT_WORDS,
Expand Down
9 changes: 5 additions & 4 deletions core/exclude.py → dupeguru/core/exclude.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
# which should be included with this package. The terms are also available at
# http://www.gnu.org/licenses/gpl-3.0.html

from core.markable import Markable
from dupeguru.core.markable import Markable
from xml.etree import ElementTree as ET
from defusedxml.ElementTree import parse as safe_parse

# TODO: perhaps use regex module for better Unicode support? https://pypi.org/project/regex/
# also https://pypi.org/project/re2/
Expand All @@ -12,8 +13,8 @@
from os import sep
import logging
import functools
from hscommon.util import FileOrPath
from hscommon.plat import ISWINDOWS
from dupeguru.hscommon.util import FileOrPath
from dupeguru.hscommon.plat import ISWINDOWS
import time

default_regexes = [
Expand Down Expand Up @@ -321,7 +322,7 @@
infile can be a file object or a filename.
"""
try:
root = ET.parse(infile).getroot()
root = safe_parse(infile).getroot()
except Exception as e:
logging.warning(f"Error while loading {infile}: {e}")
self.restore_defaults()
Expand Down Expand Up @@ -415,7 +416,7 @@
try:
self._excluded_compiled.add(self._excluded.get(regex).get("compiled"))
except Exception as e:
logging.error(f"Exception while adding regex {regex} to compiled set: {e}")

Check failure on line 419 in dupeguru/core/exclude.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Use "logging.exception()" instead.

See more on https://sonarcloud.io/project/issues?id=arsenetar_dupeguru&issues=AZ_EQbXvVL7xxLuooKpR&open=AZ_EQbXvVL7xxLuooKpR&pullRequest=1407
return

def is_compilable(self, regex):
Expand Down
File renamed without changes.
9 changes: 8 additions & 1 deletion core/fs.py → dupeguru/core/fs.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from typing import Any, AnyStr, Union, Callable

from pathlib import Path
from hscommon.util import nonone, get_file_ext
from dupeguru.hscommon.util import nonone, get_file_ext

hasher: Callable
try:
Expand Down Expand Up @@ -112,6 +112,9 @@ class FilesDB:
ON CONFLICT(path) DO UPDATE SET size=:size, mtime_ns=:mtime_ns, entry_dt=datetime('now'), {key}=:value;
"""

# Whitelist of valid column names to prevent SQL injection via .format(key=key)
VALID_KEYS = {"digest", "digest_partial", "digest_samples"}

ignore_mtime = False

def __init__(self):
Expand Down Expand Up @@ -150,6 +153,8 @@ def clear(self) -> None:
conn.execute(self.create_table_query)

def get(self, path: Path, key: str) -> Union[bytes, None]:
if key not in self.VALID_KEYS:
raise ValueError(f"Invalid cache key: {key}")
stat = path.stat()
size = stat.st_size
mtime_ns = stat.st_mtime_ns
Expand All @@ -175,6 +180,8 @@ def get(self, path: Path, key: str) -> Union[bytes, None]:
return None

def put(self, path: Path, key: str, value: Any) -> None:
if key not in self.VALID_KEYS:
raise ValueError(f"Invalid cache key: {key}")
stat = path.stat()
size = stat.st_size
mtime_ns = stat.st_mtime_ns
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion core/gui/base.py → dupeguru/core/gui/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# which should be included with this package. The terms are also available at
# http://www.gnu.org/licenses/gpl-3.0.html

from hscommon.notify import Listener
from dupeguru.hscommon.notify import Listener


class DupeGuruGUIObject(Listener):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@

import os

from hscommon.gui.base import GUIObject
from hscommon.trans import tr
from dupeguru.hscommon.gui.base import GUIObject
from dupeguru.hscommon.trans import tr


class DeletionOptionsView:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
# which should be included with this package. The terms are also available at
# http://www.gnu.org/licenses/gpl-3.0.html

from hscommon.gui.base import GUIObject
from core.gui.base import DupeGuruGUIObject
from dupeguru.hscommon.gui.base import GUIObject
from dupeguru.core.gui.base import DupeGuruGUIObject


class DetailsPanel(GUIObject, DupeGuruGUIObject):
Expand Down
Loading