Wurker Documentation

• Colony: a cluster of Wurker nodes.
• Queen: the controller node of a Colony. May be the only node.
• Drone: a worker node of a Colony.
• Bee: a single job, either the job definition, or the running process, itself.
• Command: a single command (script or Python module) that Bees reference.
• Cron: a single scheduled or supervised definition of how and when to run.
• Hive: the set of currently-running Bees.

• Sting: inject dependencies into Wurker, including wurker.toml, bees, settings, models, seeds, and secret data, such as data/user.json.
• Seed: declare Commands, Bees, Crons and Maps to be added, updated, or deleted.
• Schedule: run short-lived Bees as scheduled on cron-type schedules.
• Supervise: run long-lived Bees as supervised and restart the process if it dies.
• Ondemand: run any enabled Bees one time immediately.
• CLI: easily add, remove, and edit Bees, Crons, and Maps from the command-line using JSON.
• UI: serve a dashboard to view and enable or disable Bees, including the Hive.
• API: perform many of the same actions as CLI or UI via API request. Used by the Queen to talk to Drones.

Wurker requires a minimum Python version of 3.10. A virtual environment is strongly recommended.

Wurker uses two databases:

1. Remote Configuration DB: by default a sqlite3 file defined in etc/wurker.toml.
2. Local DBs: sqlite3 file in data/wurker.db keep track of Wurker processes and Wurker UI users.

Wurker is configured via a database connection managed with SQLAlchemy. To use the default MySQL DBAPI, first install the Python 3 and MySQL development headers and libraries:

If you are deploying to production and want to use the web app UI, you will need to install nginx.

Further instructions may be found in the Production section.

Wurker may be installed from PyPi using Pip or similar package manager. A Python virtual environment is strongly recommended:

pip install wurker

Alternatively, Wurker may be cloned and run as a Hatch project.

• Install pipx
• Install hatch

Next, install system dependencies, as documented in System Dependencies.

After hatch and all system dependencies are successfully installed, cd into the wurker directory and do:

hatch env create
hatch shell

Existing version 0.0.X Wurker installs will need to meet new system requirements.

You may use Stingers to inject additional Python requirements into etc/requirements.txt. You, or your CI/CD, will need to then perform additional package installation:

pip install -r etc/requirements.txt

You can check that installation worked by executing the following commands:

wurk --help

Use the default Stinger to inject basic dependencies, including test Seeds and Bees:

wurk admin stinger sting

This will inject local resources. For custom dependency-injection, see the Stingers section.

The default database set in etc/wurker.toml will be a SQLite file named wurker.db. See the Configuration section for details.

After injecting resources, can Seed the local and remote configuration databases (the same file, by default):

wurk admin seed

Finally, and optionally, you can enable some of the test Bees:

wurk edit bee test_scheduled_bee <<EOF
{"enabled": true}
EOF

You can see that test_scheduled_bee was enabled with the command:

wurk show

And you can run the Hive in the foreground with the command, while logging to stderr:

wurk run

If you choose to log to syslog, instead, you can open a separate terminal and do:

journalctl -f

You can also override the wurker.toml log setting from the command-line. Type C-c to stop running, then run again but this time, force logging to dump to stderr:

wurk --log stderr run

Running Wurker this way is only for development and devops use. In production, Wurker will run as a systemd service, and we will use wurk in parallel to access the same runtime and configuration data as the service.

For example, when running Wurker as a service, we can follow all the logs this way:

journalctl -u wurker -f

But when running wurk run we cannot follow -u wurker because that is the name of a systemd unit which does not exist. Instead, we must follow a text name, like -t Wurker.

If all else fails, follow and filter logs like:

journalctl -f | grep Wurker

Since our Test Run example is a Queen (all singleton nodes are Queens) we can also serve the Dashboard GUI.

Open a new terminal, so you don't interrupt the running Wurker. Then in the new terminal, activate your virtual environment and run the command:

wurker-ui-flask run

Then browse to specified URL.

The default development credentials are wurkeradmin/wurkeradmin.

To create your own Wurker UI credentials, see the Stingers section.

To configure Wurker locally, it is enough to use the default wurker.toml file:

[wurker]
name = "Wurker"
app_id = "6b018c2b-77b7-40ca-bb82-cc8378252f67" # Must be UUID
env = "local"                                   # 'local', 'development', or 'production'
log = "stderr"                                  # 'stderr' or 'syslog'
loglevel = "debug"
is_queen = true
debug = true

[ui]
domain = ""                                     # Default: localhost
workers = 3                                     # Default: 3

[local_db]
hostname = ""                                   # Default: local sqlite file
username = ""                                   # Default: local sqlite file
password = ""                                   # Default: local sqlite file
database = "wurker.db"                          # Default: local sqlite file 'wurker.db'
dbdriver = "sqlite"                             # Default: 'sqlite'
port = 0

[remote_db]
hostname = ""                                   # Default: local sqlite file
username = ""                                   # Default: local sqlite file
password = ""                                   # Default: local sqlite file
database = ""                                   # Default: local sqlite file 'wurker.db'
dbdriver = ""                                   # Default: 'sqlite'
port = 0

While the wurker.toml file alone is sufficient for a few Bees abuzz, Wurker operators may additionally configure custom Python and JSON dependencies using Stingers.

Wurker configuration of Commands, Crons, and Bees are defined in remote database tables within the schema specified by [remote_db] in the previous Local Config section.

• A Command is either a path to a command (ex: /usr/bin/ls), or the name of a module in bees/.
• A Supervised Cron is a wurker_cron that has a null cron field.
• A Scheduled Cron is a wurker_cron with a valid cron field in crontab notation.
• An Orphaned Bee is any Bee that is not mapped to a Cron, either supervised or scheduled.
• A Disabled Bee is any Bee that is not enabled. Bees may be both orphaned and disabled.

Because Wurker permits the use of internal PyBees, it also permits internal configuration of PyBees and PyBee-compatible modules.

Wurker and the Wurker UI run as systemd services. In addition, Wurker UI requires an nginx service.

After configuring Wurker and Bees, you may also configure the systemd services using the service command. Then either or both of wurker and wurker-ui services will be running in the background.

Their logs will be available using journalctl.

See the Service section for full details.

The following are only simple examples for pre-configured services:

wurk admin service wurker
wurk admin service ui

Some subcommands are only available via the Command-Line Interface (CLI). In particular, you can only perform dependency injection, seed the database, configure services, and run Bees via the CLI.

The other subcommands are available via CLI or API. Most API subcommands are also available via the UI in a more limited fashion.

Seed files are JSON files that used to migrate data without dropping the remote database schema. Seed files can be used to add, update, or delete zero or more Commands, Crons, Bees, and/or Maps.

A Seed file is a JSON object with at least one of wurker_command, wurker_cron, wurker_bee, or wurker_bee_cron items. All of these are JSON arrays containing objects, except wurker_bee_cron, which is an object.

See Default Seed JSON for a real example of adding and scheduling Commands, Crons, and Bees.

See Seed JSON Schema for a formal declaration that can be used to validate Seed JSON data.

Stingers are a powerful tool that may include Seeds, as well as additional dependencies to be injected, like PyBee modules, custom DB models, custom Python libraries, etc.

The Seeds permit us to declare a complete configuration for all our Commands, Crons, and Bees that are injected during the sting operation.

The wurk admin stinger sting options should be familiar concepts, by now:

• ROOT: Required: the root of the Stingers you want to choose from.
• SLUG: Required: the directory of the Stinger you want to inject.
• NODE: Optional: the directory of the node (queen, drone1, drone2, etc) you want to inject.
• ENV: Optional: the directory of the environment (production, development, etc) you want to inject.

Colonies are Wurker clusters: multiple hosts (hardware or virtual machines) running in tandem.

Each separate host in a Colony runs a Wurker instance, identified by the app_id in the wurker.toml file. All nodes in a Colony share the same database schema. Only one of these instances in the Colony can be the Queen. The others are Drones.

Stingers make it much easier to declare Queens and Drones, and when used in conjunction with version control and a CI/CD system, they provide a highly reproducible solution to scaling Wurker on conventional, low-cost cloud infrastructure.

Consider the final example from the previous section on Stingers, and note the colony directory.

Like the wurker directory, the colony slug is special. It indicates that one or more NODE directories will be found inside. And each NODE directory may have a wurker directory, or one or more ENV directories. Or both wurker and ENV directories.

Every Wurker instance may serve an API, and it may also serve the Wurker Dashboard UI, if it's a Queen.

During the startup phase, all versions were considered BUILD versions until feature completeness is achieved.

That means we set MAJOR = 0 and MINOR = 0 for all early versions.

After v1.0.0, apply the following criteria when determining which version value to increment:

• MAJOR: A new subcommand or integration is added, or major code revisions have occurred. Increment rarely.
• MINOR: Any release candidate that has been deemed "production-ready". Increment often.
• BUILD: For development use only. It should be excluded in releases, except released patches.

The road map items should be moved to the Change Log as they are completed.

First fully-functional transitional version from startup v0.0 to v1.0. Too many changes to document in detail, but consistent with the road map for v0.1 to v1.0.

Added:

• Created a PyPi repository and setup sufficient for pip install wurker in test.pypi.org.
• Published documentation to Gitlab Pages.
• New wurk admin config {set,get,reset} command that allows CLI reading and editing of the new wurker.toml, which replaces the old .env file.
• New wurk admin stinger refactor command to refactor seeds and .env. Does not refactor PyBees or other Python code that imports from legacy paths.
• Plugin system instead of legacy package injection.

Changed:

• Forked and rewrote Wurker, preserving wurker.core logic and (mostly) Bee and PyBee implementations using the Hatch project management framework for simple PyPi build and publish.
• Refactored CLI to use Click instead of Python's built-in argparse package.
• Moved administrative commands to a nested wurk admin command to simplify top-level CLI.
• Moved working directory out of the source directory into a first-run installed directory in $HOME/.config/wurker/.
• Default configuration database is now the same as the local database, so that wurk admin stinger sting will work on installation, and wurk admin seed will add test Bees only.

Removed:

• The CLI test subcommand has been removed, for now. Unit tests should in instead use unittest directly on the cloned tests directory.

Added:

• Add overrun column to wurker_cron table (model).
• Refactor UIs, seeder and seeds. Make optional everywhere.
• Add runcount field to wurker_bee_report.status_info for display.
• Wurkers should know if they're Queens or Drones. Both should have an API, but only Queens have a GUI.
• The Queen's Court should provide the same intuitive display and control of Drone data.
• Wurker UI should have a login screen and session.
• Wurker UI and API should support HTTPS.
• After login, the Queen may use Basic HTTP Auth to access Drone APIs.
• User logins should be configured via Stinger.
• Accessing Wurker UI or API (nginx) should be via HTTPS, not HTTP. Should support self-signed certs by default.
• Add wurk report subcommand with filtering.
• Add Report view to Wurker UI (separate page) with same filters as CLI.
• Add Action buttons per Bee row, and Cron table, for pre-filtered Report views.
• Implement Alembic migrations in place of current create schema logic.

Changed:

• Derive timeout instead of setting it. If the Bee has a scheduled Cron, then use that as the timeout. If it is a supervised Cron, then it should have no timeout. Ondemand should use the same timeout logic.
• Derive runcount from Cron to BaseBee.runcount for runtime reference.
• Upon timeout in _run_scheduled():
  • If bee.overrun == 0 then timeout as normal. Set report status to TIMEOUT.
  • If bee.overrun > 0 and bee.runcount < bee.overrun then increment bee.runcount, and let this Cron overrun the next Cron. Set report status to OVERRUN.
  • If bee.overrun > 0 and bee.runcount >= bee.overrun then TIMEOUT the overrun.
• Deleting a Bee or Cron should cascade into Report table.
• Add pid and timestamp Hive columns, and set them when the Bee is added to the Hive.
• Add nullable duration Hive column, which is set only when the Cron ends.
• Change wurker_bee_report.status enum to: PROCESSING, OVERRUN, COMPLETED, TIMEOUT, ERROR
• Insert record on bee.run() as PROCESSING
• Update status to OVERRUN upon allowed overrun, or to TIMEOUT if not allowed.
• Record stdout errors from Bees, and PyBee exceptions into status_info JSON.
• Every Cron ending should cause a final write, setting duration, status, and status_info.

Changed:

• Major revisions to Wurkman._run_scheduled() to speed up the method, especially under heavy load when there are many Bees starting at the same scheduled time. Now utilizes ThreadPoolExecutor to process concurrently.

Added

• Added support for update and delete cron operations in Seed files.
• Added support for delete wurker_bee_cron mappings in Seed files.

Changed:

• Added logic to BaseBee.NextRun to self-correct when self.next_run is more than a minute older than now.

Changed:

• Upgraded sqlalchemy queries to use v2.0 syntax (should still work with latest 1.4).
• Small but impactful improvements to scheduler behavior.

Added:

• Added support for update and delete operations in Seed files.

Changed:

• Simplified Bee.kill() and PyBee.kill() using fewer nested if-else statements, making the logic easier to understand.

Added:

• Added Sting and local SQLite DB Seed support for Basic HTTP Auth via data/api_auth.json, with a new Flask app.auth decorator @api_auth_required.
• Added settings.env.ApiAuth property which autoloads the first item in data/api_auth.json for requests via DroneClient. Each subcommand must handle passing auth keyword argument to DroneClient.__init__().
• Added api_auth_required() and check_api_auth() to decorate and validate API Basic HTTP auth. If no users defined in local api_auth table, then the API is considered public, and requires no authentication. Otherwise checks that the username / password combination exists in the local SQLite3 DB.

Changed:

• Changed seed subcommand to seed data/api_auth.json into SQLite3 api_auth table, for now just the same as the user table.
• Changed DroneClient.request() to add auth to underlying requests request, if it exists.

Added:

• Added Wurker CLI service [--ssl-dir SSL_DIR] [--ssl-certs CRT KEY] options to customize nginx SSL.
• Added ssl as one of the Stinger directories to inject into etc/nginx/ssl/. For example, wurker-stingers/default/wurker/ssl/ with certs wurkzen.com.crt and wurkzen.com.key.
• Added logic to service subcommand to generate self-signed certs if they don't exist.

Changed:

• Changed etc/nginx/wurker-ui.conf.skel to listen on 443 instead of 80, and use configured SSL certs.
• Changed default DroneClient.protocol to https.

Added:

• Added Wurker CLI report subcommand with Queen and Drone options.
• Added Wurker UI /report view (Queen only) and /api/report endpoint (all nodes), using report subcommand.
• Added Grid.js frontend rendering for report tables, with server-side API responses for pagination (but not sorting, yet… still working on server-side sorting).

Changed:

• Switched JS JSON syntax hightlighter library from deprecated code-prettify to Rainbow.js, including for status in the index view.

Added:

• Added Wurker Documentation as a repo, serving GitLab Pages.
• Added more test cases to test_00_core.py including init Wurkman and 'public' methods except run/kill.
• Added ./wurk test -v option to implement pytest -v via CLI.

Changed:

• Moved the README.org from Wurker repository to new wurker-docs.git repo on GitLab. Wurker README.org now refers to that document and the new Wurker Documentation GitLab Pages.

Added:

• Added wurker_hive.runcount column in hive.db to keep track of how many Crons a Bee has lived through.
• Added Hive.incr_runcount() convenience method to increment the new column for the given Bee.
• Added wurker_cron.overrun column (default 0) to set the max run count.
• Added logic to Wurkman._run_scheduled() handling overrun and runcount

Changed:

• Updated Hive.add() to set runcount to 0.
• Refactored timeout logic to use value computed in BaseBee._init_timeout(), as follows:
  • timeout = duration - 10 if duration < 5 minutes
  • timeout = duration - 30 if duration between 5 and 10 minutes
  • timeout = duration - 60 if duration > 10 minutes
• Improved kill() logic and logging to work with overruns. Tweaked scheduler to play well.

Removed:

• Dropped wurker_bee.timeout column.

Added:

• Added enhanced functionality to Wurker UI, including Add Bee, Manage Cron, and Manage Command logic.
• Added sting --unsting --drop-schema option, making it not the default behavior. Equivalent to seed --drop-schema-only followed by sting --unsting.

Changed:

• Improved SQLAlchemy session handling to keep DB connections from Flask down, but still too damn high!
• Renamed wurker-server.service to wurker-ui.service to make it less confusing (and shorter).
• Further improvements to SQLAlchemy/Flask sessions by using scoped_session() and session.remove()

Added:

• Added auth views and logic from Flask tutorial.
• Added service subcommand to configure and install systemd services for wurker and wurker-server.
• Added test subcommand to run pytest suite all CI/CD friendly.
• Added wurker.drone.DroneClient class to encapsulate API requests from Queen to Drones.
• Added wurker.db.get_drone_status function to obtain wurker_status records for the Queen's Drones.
• Added --drone option to status, control, show, add, edit, rm, kill for Queens only, to request the subcommand via the Drone API(s).
• Added all functionality to Wurker UI, except Add Bee, Manage Cron, and Manage Command logic.

Changed:

• Switched from Bottle to Flask for micro-framework.
• Modified routing to serve API separately from Dashboard.
• Refactored wurker_control.id to wurker_control.app_id for sanity's sake.
• Updated tests/ test suite with refactored app_id.
• Refactored show, run, kill subcommands to use new --filter and --slug options instead of flags.
• Changed names of settings in .env from WURKER_APP_* to WURKER_*, except for WURKER_APP_ID.

Added:

• Added logic to seed to resolve relative command paths to abspath.
• Added unseeding logic to sting --unsting. Also does equivalent of seed --drop-only.
• Added sting --extract option to copy out a Stinger, respecting existing configuration.

Added:

• Added app_id column as FK into wurker_control.id, for all RemoteDB tables except wurker_bee_cron.
• Added wurker_control.is_queen flag to model, and WURKER_APP_IS_QUEEN setting to .env.
• Added --control-only option to seed subcommand.
• Added built-in bees.dispatch PyBee, which imports from new bees.dispatchers submodule.
• Added new sting subcommand to inject dependencies such as .env, seeds, bees, models, and settings from a structured dependency tree, permitting simpler CI/CD integration.
• Added sting --unsting option to delete the injected files, including .env.
• Enhanced sting command to also inject requirements.txt into etc/.

Changed:

• Changed wurker.commands module to wurker.subcommands to disambiguate them from wurker_command table.
• Refactored all subcommands to include app_id in queries.
• Fixed a bug in core.Wurkman._run_ondemand() that was making noise in the log.

Added:

• Added --drop-only and --local-drop-only options to seed command.
• Added add ondemand to add a list of Bee slugs to the wurker_queue table for ondemand processing.
• Added buttons to add ondemand Bees via the serve command. They work.
• Added buttons to kill and rm Bees via the serve command. Also added buttons to add Bees and Crons. They don't work yet. Most of the editable fields do not work yet, either, except the .enabled.toggle column.
• Added logic to automatically record status and status_info in the wurker_bee_report table.

Changed:

• Changed control stop to control pause since it can only be stopped as a systemd service.
• Tore down the whole mk_bee_cfg() logic in core.py and made BaseBee and children 1st class objects. Now, when we want a dict representation of a Bee or PyBee, we just ask for the bee.AsDict property.

Added:

• Added wurker_command table and WurkerCommand model to encapsulate command and module definitions.
• Added show --command to view the existing commands.
• Added command as a choice to add, edit, and rm commands.
• Added wurker_bee.command_slug as a FK to wurker_command.slug PK.
• Added pause and reload buttons to the dashboard (don't work yet).

Changed:

• Removed wurker_bee.command and module columns. Now use bee.command.is_module instead.
• Lots of changes to commands to accomodate new wurker_command objects and config.
• Fixes for the dashboard an lots of improvements to make it prettier and easier to use.
• Updated this README.org to include new wurker_command usage and discussion.

Added:

• Added serve command to serve a simple browser-based dashboard.
• Added JSON-based seed option (in addition to original raw SQL).
• Added wurker_control.seed column to keep track of current seed number.
• Added quiet_restart config option for supervised Bees.
• Added --ondemand option to show, run and kill commands with new wurker_queue table.

Changed:

• Moved database config from settings/database.py to .env file.
• Enhanced seed to intelligently seed only new data when run again.
• Fixed hive logic to do INSERT OR IGNORE and added logic to reconcile with psutils.
• Enhanced Bee (command) processes to capture errors and log them in Wurker's log.

Added:

• Completed interface with implementations of seed, status, kill, add, and edit commands.
• Added local process DB using Sqlite3 to permit management of running processes via command-line.
• Moved wurker.models to a sibling models module with separate models.wurker and models.wurkzen. This permits Wurker to have an independent database schema from PyBees, or to be installed in an app database.
• New hive.Hive class to manage the local process DB operations we need.

Changed:

• core.Wurker became core.Wurkman and logic was added to add and remove Bees from the Hive.

Added:

• Implemented essential subcommands and funtionality (show and run).
• BaseBee, Bee, and PyBee classes implemented with test scripts and modules.
• radagast PyBee module implemented for initial release, with an eye toward Vishnu compliance.

• Wurker project initialized with README.md