gallery-dl-archive-manager

Scripts to manage a (currently twitter only) archive using gallery-dl. Much of the code came from a need to augment pre-existing, outdated archives that were originally created from the twittermediadownloader browser extension.

Config

This repo uses its own config.json in order to save media in the same format as twittermediadownloader. The scripts depend on the media being saved in this format.

Scripts

node run-initDb.js --path={/path/to/your/archive} --args={gallery-dl args}

Initializes a user database from existing folders. Useful if you have a pre-existing archive of users.

node run-downloadDb.js --path={/path/to/your/archive} --threads={#} --args={gallery-dl args}

Args

Standard args:

--path={/path/to/your/archive}

The path to the archive. This is a parent directory with a list of child directories which correspond to users.

--threads={#}

Max number of concurrent download threads. Only this number of concurrent gallery-dl download threads will run at a given time, other remaining users will be queued.

--args={gallery-dl args}

Additional args to pass to gallery-dl. Note that these aren't currently checked for duplicates that may be used by this repo.

TODO

run-addUsers.js --path={/path/to/your/archive} --users={string array of user(s)} --threads={#}

Should add a new user to the db and initiate a full download similar to run-downloadDb.js

run-updateDb.js --path={/path/to/your/archive} --mode={search|media} --threads={#} --args={gallery-dl args}

Should pull from the user database and update the archive without doing a full download.

• --mode
  • =search: The DB should save with a lastUpdated field. This should be used as a date for the /search API. Preferred if it's been a long time since an update has happened for a user and/or the user has uploaded a significant amount of media since lastUpdated.
  • =media: This will run /media and stop after hitting skipped files, as skipped files indicate hitting lastUpdated. Preferred in normal circumstances.