Managing millions of pictures

I take lots of pictures. And, it is a challenge to organize, sort and continuously de-dup them. Over the years, I have developed a pipeline to automate this process as much as it is possible. My pictures get automatically synced to a central location where a cron job periodically starts a docker container to run exiftool to rename the files and place them in the right directory structure based on year, month and day. Once the pictures are placed in the right place, Czkawka runs to move duplicates to a holding place where they get eventually deleted.

You can simply download my docker image and get started right away. Here is an example docker-compose.yml file to get started:

version: '2'
services:
  exiftool:
    stdin_open: true
    container_name: exiftool
    tty: true
    image: ahmadfs/exiftool
    volumes:
      - ./logs:/logs
      - ./INPUT:/source:ro     # Make sure you provide a valid input directory
      - ./OUTPUT:/destination  # Make sure you provide a valid output directory

Make sure to edit the variables to point to the right directories for input and output sources. You can start your container by using this command where you have the docker-compose.yml file:

docker compose up -d

Depending on the size of your source directory, this may take time to complete. You can tail the logs directory to see the progress and watch for errors or warnings.

If you wish to build the container yourself, here is the Dockerfile and necessary script:

# DOCKERFILE
FROM alpine:latest
RUN apk update && \
    apk upgrade && \
    apk add exiftool bash
COPY ./run_exiftool.sh /run_exiftool.sh
entrypoint ["/bin/bash", "-c", "/run_exiftool.sh"]

Here is the actual exiftool script that does the renaming and copying of pictures:

# EXIFTOOL SHELL SCRIPT
#!/bin/bash
# EXIFTOOl Documentation: https://exiftool.org/exiftool_pod.html
# You can adjust the output directory and filename structure below

exiftool -r -P -o . \
'-FileName<$FileModifyDate/${FileModifyDate#;DateFmt("%Y-%m-%d_%H%M%S")}_%f%-c.%e' \
'-FileName<$DateTimeOriginal/${DateTimeOriginal#;DateFmt("%Y-%m-%d_%H%M%S")}_%f%-c.%e'  \
'-FileName<$FileModifyDate/${model;}/${FileModifyDate#;DateFmt("%Y-%m-%d_%H%M%S%%-c")}.%e' \
'-FileName<$DateTimeOriginal/${model;}/${DateTimeOriginal#;DateFmt("%Y-%m-%d_%H%M%S%%-c")}.%e' \
-d '/destination/%Y/%B/%d' /source/ >/logs/`date '+%Y%m%d_%H%M'`.log 2>&1