ben/cluster-state
1
0
Fork 0
Code Issues 1 Pull requests Activity Actions

feat(matrix): add Nomad job specification for Matrix service

Browse source 
feat(matrix): update resource allocation in Nomad job specification

feat(matrix): onboard element service to traefik

feat(matrix): add port configuration for Element service

chore(matrix): reformat

feat(matrix): update resource allocation in Nomad job specification

fix(matrix): minimum MemoryMB value is 10

feat(matrix): update resource allocation in Nomad job specification

feat(matrix): split server and clients into seperate groups

feat(matrix): well known to be served by nginx

fix(matrix): add well known route for all hosts

feat(matrix): use separate traefik router for well known

feat(matrix): migrate config.yaml for mas

feat(matrix): divide mas config between nomad and volume

feat(matrix): split cinny and element task groups

refactor(media-centre): Migrate media-centre job spec to Nomad HCL format

fix(media-centre): remove json from resource nomad job

fix(media-centre): update media-centre job spec to use Nomad HCL format

feat(media-centre): add downloader group

- Added a new group called "downloaders" to handle proxy tasks for downloading media files.
- Configured the proxy task with necessary settings and environment variables.

fix(media-centre): use OPENVPN_USER env variable in proxy task

fix(media-centre): Add /dev/net/tun device to proxy task

feat(media-centre): Add resource limits to proxy task

feat(media-centre): Add Plex task to media-centre job spec

fix(media-centre): add constraints to media-centre job spec

fix(media-centre): nomad doesn't allow sharing devices

fix(media-centre): disable change config dir ownership

fix(media-centre): plex process user is set using env vars

fix(media-centre): update PLEX_GID in job spec

fix(media-centre): update PLEX_GID in job spec

fix(media-centre): update PLEX_UID in job spec

feat(media-centre): enable nvidia gpu capabilities

feat(media-centre): add Tautulli service to media-centre job spec

fix(media-centre): update tautulli volumes

feat(plextraktsync): add plextraktsync module

fix(plextraktsync): update plextraktsync job spec "type" to "batch"

feat(plextraktsync): update resource allocation

fix(plextraktsync): fix cron schedule in plextraktsync job spec

feat(nfs-csi): add nfs-csi module

chore: update .gitignore to include .env file

chore: format files

feat(seedbox): add seedbox module

feat(seedbox): add qbittorrent module and NFS volume

feat(seedbox): add timezone configuration for seedbox job

fix(seedbox): vuetorrent-lsio-mod image env var

feat(seedbox): add HTTP_PORT environment variable for qbittorrent module

feat(seedbox): update access mode for NFS volume

feat(seedbox): add node constraint for seedbox job

feat(seedbox): add subdirectories for NFS volumes

feat(seedbox): add nolock mount flag for NFS volumes

feat(seedbox): Update NFS volume configuration

feat(seedbox): update Docker image and enable force pull

feat(seedbox): pause container network definition

feat(elk): create kibana

feat(elk): update kibana cpu allocation

feat(elk): add elasticsearch container to elk job

This commit adds a new task "elasticsearch" to the "elk" job in the "node" group. The task uses the "podman" driver and pulls the "docker.elastic.co/elasticsearch/elasticsearch:8.15.2" image with force pull enabled. It exposes the "transport" port and mounts the "/mnt/docker/elastic/elasticsearch/config" and "/mnt/docker/elastic/elasticsearch/data" volumes. The task is allocated with 500 CPU and 1024 memory resources.

feat(seedbox): update resource allocation in seedbox job

fix(elk): remove ulimit from elk job

See: https://github.com/hashicorp/nomad-driver-podman/issues/341

fix(elk): add selinuxlabel to volume mounts in elk job

refactor(modules): remove unused modules and jobspecs

refactor(elk): update CPU allocation in elk job

feat(media-centre): Plex to use host network

feat(elk): add 9200 port to es node

feat(elk): allocate more ram to node

feat(elk): allocate even more ram to node

feat(media-centre): reduce memory allocation of tautulli

feat(elk): revert memory allocation after shard tidy-up

feat(media-centre): set memory soft limit

feat(media-centre): update memory hard limit for tautulli

feat(elk): tweak node mem alloc

See: https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#_example_11

feat(seedbox): add memory soft limit to vpn client

feat(seedbox): update memory hard limit for vpn client

fix(matrix): increase whatsapp-bridge memory allocation

refactor(elk): update elastic and kibana image versions in elk job

feat: add latest image versions and add force pull

feat: enable force pull for all podman driver tasks

feat(matrix): increase syncv3 memory allocation

feat: migrate podman memory allocation to nomad max memory

fix: nomad max memory is defined by memory_max

feat(matrix): add ecs fields to task metadata

refactor(matrix): migrate shared meta to parent

refactor(matrix): update resource allocation in jobspec.nomad.hcl

refactor(matrix): update resource allocation in jobspec.nomad.hcl

refactor(matrix): update resource allocation in jobspec.nomad.hcl

refactor(plextraktsync): update resource allocation in jobspec.nomad.hcl

refactor(plextraktsync): remove task node constraint

refactor: migrate podman tasks to docker tasks

feat(elk): update ulimit for elasticsearch container

refactor(elk): update volume paths in jobspec.nomad.hcl

feat(seedbox): remove pause container

feat(elk): update kibana count in jobspec.nomad.hcl

refactor(elk): remove node constraint from kibana

refactor(elk): add spread attribute to kibana

refactor(elk): update port configuration in jobspec.nomad.hcl

refactor(dummy): migrate json jobspec to hcl

feat(dummy): update service provider to consul

fix(dummy): add port label to port definition

refactor(dummy): rename jobspec to match standard

feat(dummy): migrate to service mesh

chore(dummy): update Nomad provider version to 2.4.0

chore(dummy): update Nomad provider version to 2.4.0

feat(dummy): configure traefik

refactor(dummy): update provider to use consul instead of nomad

feat(renovate): create module for automated dependency updates

Add renovate.json

fix(renovate): increase memory allocation

feat(renovate): add GITHUB_COM_TOKEN variable

refactor(renovate): pin version

feat(renovate): enable dependency dashboard

refactor(matrix): use bridge netowrking for server group

refactor(matrix): update URLs to use allocated addresses

refactor(matrix): remove host.docker.internal host

fix(matrix): update SYNCV3_BINDADDR

fix(matrix): update SYNCV3_BINDADDR port to 8009

fix(elk): increase memory allocation

feat(elk): disable co-located kibana allocations

refactor(jobspec): update provider to consul for elk and media-centre services

feat(media-centre): reduce memory allocation from 4096 to 1024

fix(jobspec): replace constraints with new neto client id

feat(elk): update data volume path to use unique node name

feat(elk): migrate elastic config to nfs

feat(elk): add Nyx

refactor(workflows): reformat (#17)

Reviewed-on: #17

fix(elk): increase memory allocation to 2048 MB

refactor(matrix): remove specific node constraint from job specification

feat(matrix): implement consul service mesh

feat(elk): use allocation index for node state location

refactor(media-centre): remove deprecated NVIDIA_DRIVER_CAPABILITIES

fix(media-centre): plex transcode dir not writable

fix(media-centre): set transcode dir to world writable

fix(media-centre): set transcode dir to world writable

feat(media-centre): replace plex transcode dir with a persistent volume

feat(media-centre): increase plex memory limit

For caching

chore(elk): promote elastic version

feat(elk): remove force_pull option from Elasticsearch and Kibana configurations

style(jobspec): improve formatting in HCL files

feat(elk): add health check

feat(media-centre): add NVIDIA visible devices for Jellyfin and Plex

fix(media-centre): increase max memory for tautulli

feat(plugin-csi): add NFS CSI driver jobspec and main configuration

feat(main.tf): add plugin-csi module to main configuration

fix(plugin-csi): refactor NFS job specifications into separate files for controller and node

fix(plugin-csi): add NFS path variable for controller and node resources

fix(plugin-csi): add NFS path variable to controller and node job specifications

fix(plugin-csi): add provisioner name to NFS job specifications for controller and node

fix(plugin-csi): update NFS job specifications

feat(seedbox): restructure job specifications and add NFS volume registrations for media and qbittorrent config

feat(workflows): add lint workflow for Terraform and Nomad formatting

fix(seedbox): add attachment and access modes for media and qbittorrent_config volumes

feat(seedbox): remove node constraint

Update modules/seedbox/main.tf

fix(seedbox): add mount options with nolock flag for media and qbittorrent_config volumes

fix(seedbox): update share paths to use lowercase in media and qbittorrent_config volumes

fix(seedbox): remove unused device configuration from jobspec

feat(matrix): add health check configuration

feat(matrix): add health check ports for synapse, mas, and nginx

fix(matrix): remove health check configuration for synapse, mas, and nginx

feat(main.tf): remove unused and broken seedbox module

feat(renovate): use JSON log format

chore(elk): upgrade version to latest

feat(elk): use 2 kibana replicas

feat(elk): add on_update ignore option to ready check configuration

fix(elk): update volume paths to use node unique name for configuration and data

feat(matrix): add envoy_metrics port and update service metadata for Consul integration

feat(matrix): add health check configuration to synapse job

feat(matrix): add /metrics endpoint exposure for envoy_metrics

fix(matrix): update service port configurations to use static port numbers

feat(matrix): restructure ingress groups and enhance service configurations for improved routing

fix(matrix): update whatsapp bridge tokens and change push to receive ephemeral

feat(media-centre): remove node constraint from tautulli task configuration

feat(elk): onboard hestia node to nomad

feat(elk): enhance job specification with Envoy metrics and update service configurations

feat(renovate): onboard nomad docker image updates

chore(deps): update ghcr.io/renovatebot/renovate docker tag to v38.142.7

chore(jobspec): use explicit image version tags where possible

fix(jobspec): formatting

chore(deps): update busybox docker tag to v1.37.0

chore(deps): update docker.io/library/nginx docker tag to v1.27.3

chore(deps): update ghcr.io/renovatebot/renovate docker tag to v39

chore(deps): update ghcr.io/renovatebot/renovate docker tag to v39.59.0

chore(deps): update ghcr.io/renovatebot/renovate docker tag to v39.60.0

chore(matrix): format multiline string in jobspec.nomad.hcl for improved readability

chore(secrets): refactor jobspecs to use templates for sensitive environment variables
This commit is contained in:
Ben Martin 2024-09-26 18:17:46 +00:00
parent ea6c8893b6
commit 9af9846216
Signed by: ben
GPG key ID: 859A655FCD290E4A
28 changed files with 1428 additions and 296 deletions
Download patch file Download diff file Expand all files Collapse all files

203
modules/media-centre/jobspec.nomad.hcl Normal file
View file

@ -0,0 +1,203 @@
job "media-centre" {
group "jellyfin" {
task "jellyfin" {
user = "985"
driver = "docker"
constraint {
attribute = "${node.unique.id}"
value = "3f6d897a-f755-5677-27c3-e3f0af1dfb7e"
}
config {
image = "ghcr.io/jellyfin/jellyfin:10.10.3"
runtime = "nvidia"
group_add = ["997"]
ports = ["jellyfin"]
mount {
type = "volume"
target = "/media"
volume_options {
driver_config {
name = "local"
options {
type = "nfs"
o = "addr=martinibar.lan,nolock,soft,rw"
device = ":/volume1/docker"
}
}
}
}
mount {
type = "volume"
target = "/config"
source = "jellyfin-config"
}
}
env {
JELLYFIN_PublishedServerUrl = "https://jellyfin.brmartin.co.uk"
NVIDIA_DRIVER_CAPABILITIES = "all"
NVIDIA_VISIBLE_DEVICES = "all"
}
resources {
cpu = 1200
memory = 4096
}
}
service {
name = "Jellyfin"
provider = "consul"
port = "jellyfin"
tags = [
"traefik.enable=true",
"traefik.http.routers.jellyfin.entrypoints=websecure",
"traefik.http.routers.jellyfin.rule=Host(`jellyfin.brmartin.co.uk`)"
]
}
network {
port "jellyfin" {
to = 8096
}
}
}
group "plex" {
task "plex" {
driver = "docker"
constraint {
attribute = "${node.unique.id}"
value = "3f6d897a-f755-5677-27c3-e3f0af1dfb7e"
}
config {
image = "plexinc/pms-docker:latest"
runtime = "nvidia"
ports = ["plex"]
network_mode = "host"
mount {
type = "volume"
target = "/data"
volume_options {
driver_config {
name = "local"
options {
type = "nfs"
o = "addr=martinibar.lan,nolock,soft,rw"
device = ":/volume1/docker"
}
}
}
}
mount {
type = "volume"
target = "/share"
volume_options {
driver_config {
name = "local"
options {
type = "nfs"
o = "addr=martinibar.lan,nolock,soft,rw"
device = ":/volume1/Share"
}
}
}
}
mount {
type = "volume"
target = "/config"
source = "plex-config"
}
mount {
type = "volume"
target = "/transcode"
source = "plex-transcode"
}
}
env {
TZ = "Europe/London"
CHANGE_CONFIG_DIR_OWNERSHIP = "false"
PLEX_UID = "990"
PLEX_GID = "997"
NVIDIA_DRIVER_CAPABILITIES = "all"
NVIDIA_VISIBLE_DEVICES = "all"
}
resources {
cpu = 1200
memory = 4096
}
}
service {
name = "Plex"
provider = "consul"
port = "plex"
tags = [
"traefik.enable=true",
"traefik.http.routers.plex.entrypoints=websecure",
"traefik.http.routers.plex.rule=Host(`plex.brmartin.co.uk`)"
]
}
network {
port "plex" {
static = 32400
}
}
}
group "tautulli" {
task "tautulli" {
driver = "docker"
config {
image = "ghcr.io/tautulli/tautulli:v2.15.0"
ports = ["tautulli"]
volumes = [
"/mnt/docker/downloads/config/tautulli:/config",
]
}
env {
PUID = "994"
PGID = "997"
TZ = "Europe/London"
}
resources {
cpu = 100
memory = 128
memory_max = 256
}
}
service {
provider = "consul"
port = "tautulli"
tags = [
"traefik.enable=true",
"traefik.http.routers.tautulli.entrypoints=websecure",
"traefik.http.routers.tautulli.rule=Host(`tautulli.brmartin.co.uk`)"
]
}
network {
port "tautulli" {
to = 8181
}
}
}
}