Web Infrastructure: Difference between revisions

From The Munich Maker Lab's Wiki
Jump to navigation Jump to search
No edit summary
mNo edit summary
 
(32 intermediate revisions by 2 users not shown)
Line 7: Line 7:
* jupiter.munichmakerlab.de
* jupiter.munichmakerlab.de
* saturn.munichmakerlab.de
* saturn.munichmakerlab.de
=== Saturn ===
* docker containers are started via systemd
* cronjob added for cleanup of old docker images
* Firewall: ufw => check via ''sudo ufw status verbose''
* [https://github.com/fail2ban/fail2ban fail2ban] to ban hosts with too many authentication failures


== Services ==
== Services ==


{| class="wikitable sortable"
{| class="wikitable sortable"
! Service Name !! Hostname !! Server !! native/docker !! status
! Service Name !! Hostname  
!Functionality!! Server !! native/docker !! status
!Source
!Source
|-
|-
| Website || [https://www.munichmakerlab.de www.munichmakerlab.de] || saturn || docker || productive
| Website || [https://www.munichmakerlab.de www.munichmakerlab.de]  
|
|Just Website, compare Github for details|| saturn || docker || productive
|-
| Wiki || [https://wiki.munichmakerlab.de wiki.munichmakerlab.de] || jupiter || native || productive
|[https://github.com/munichmakerlab/website Github Website]
|[https://github.com/munichmakerlab/website Github Website]
|-
|-
| Nodered || [https://nodered.munichmakerlab.de nodered.munichmakerlab.de] || jupiter || docker || productive
| Wiki || [https://wiki.munichmakerlab.de wiki.munichmakerlab.de]  
|Media wiki for knowledge sharing and documentation|| jupiter || native || productive
|
|
|-
|-
| Log || [https://log.munichmakerlab.de log.munichmakerlab.de] || Tumblr || - || productive
| Nodered || [https://nodered.munichmakerlab.de nodered.munichmakerlab.de]Admin: [https://nodered.munichmakerlab.de/admin/ https://nodered.munichmakerlab.de/admi]<nowiki/>[https://nodered.munichmakerlab.de/admin/ n/]
|Automation like spa<nowiki/>cestatus, Slack Bots etc.|| jupiter || docker || productive
|
|
|-
|-
| Etherpad || [https://pad.munichmakerlab.de pad.munichmakerlab.de] || mars || docker || productive
| Log || [https://log.munichmakerlab.de log.munichmakerlab.de]  
|Blog|| Tumblr || - || productive
|
|
|-
|-
| Mailinglisten || [https://lists.munichmakerlab.de lists.munichmakerlab.de] || mars || native || productive
| Etherpad || [https://pad.munichmakerlab.de pad.munichmakerlab.de]
|Colaboration text tool|| saturn||docker||productive
|Latest version: [https://github.com/ether/etherpad-lite ether/etherpad-lite]
|-
| Mailinglisten||[https://lists.munichmakerlab.de lists.munichmakerlab.de]
|Mailsystem||mars||native||productive
|
|
|-
|-
| Roombooking || [https://rooms.munichmakerlab.de rooms.munichmakerlab.de] || jupiter || docker || productive
|Roombooking
|
|[https://rooms.munichmakerlab.de rooms.munichmakerlab.de]
|Original for reserving rooms during covid||jupiter||docker||deactivated
|[https://github.com/LibreBooking/app BookedSchedular]
|-
|-
| Slack Inviter || [https://slack.munichmakerlab.de slack.munichmakerlab.de] || jupiter || docker || productive
| Slack Inviter||[https://slack.munichmakerlab.de slack.munichmakerlab.de]
|
|Self invite capability for our slack||saturn||docker ||productive
|[https://github.com/rauchg/slackin rauchg/slackin]
|-
|-
| Space Status || [https://status.munichmakerlab.de status.munichmakerlab.de] || saturn || docker || productive
|Space Status
|[https://status.munichmakerlab.de status.munichmakerlab.de]
|Button in the lab to mark space as open/closed on slack/homepage||saturn||docker||productive
|[https://github.com/munichmakerlab/spacestatus Github Spacestatus]
|[https://github.com/munichmakerlab/spacestatus Github Spacestatus]
|-
|-
| MQTT || [https://mqtt.munichmakerlab.de mqtt.munichmakerlab.de] || jupiter || native || productive
|Eclipse Mosquitto (MQTT)||[https://mqtt.munichmakerlab.de mqtt.munichmakerlab.de]
|MQTT to use for other servicesservices like status etc. Compare [[MuMaBus]]||saturn||docker||productive
|[https://hub.docker.com/_/eclipse-mosquitto Eclipse Mosquitto]
|-
|Nextcloud||[https://nextcloud.munichmakerlab.de nextcloud.munichmakerlab.de]
|Document sharing, calendar||saturn||docker||experimental
|
|
|-
|-
| Nextcloud || [https://nextcloud.munichmakerlab.de nextcloud.munichmakerlab.de] || saturn || docker || experimental
|Traefik Reverseproxy||[https://saturn.munichmakerlab.de/dashboard/ saturn.munichmakerlab.de/dashboard/] <br> (might be disabled)
|Reverse proxy for other services||saturn||docker||productive
|
|
|-
|-
| Traefik Reverseproxy || [https://saturn.munichmakerlab.de/dashboard/ saturn.munichmakerlab.de/dashboard/] </br> (might be disabled) || saturn || docker || productive
|Tickets (old)
|tickets.mumalab.org
|Ticket system for workshops and events
|German
| -
|productive
|https://github.com/pretix/pretix
|-
|Tickets
|tickets.munichmakerlab.de
|Ticket system for workshops and events
|saturn
|docker
|Todo
|https://github.com/pretix/pretix
|-
|Wiki Staging
|wiki-staging.munichmakerlab.de
|Wiki for testing (temporary)
|saturn
|docker
|Todo
|
|
|-
|Influx DB
|influxdb.munichmakerlab.de
|DB for particles sensor (temporary)
|saturn
|docker
|Todo
|https://hub.docker.com/_/influxdb
|-
|ToolJet
|tooljet.munichmakerlab.de
|Store member and token, who has which safet course etc. Might be replaced by authentik directly
|saturn
|docker
|Todo
|https://github.com/ToolJet/ToolJet
|}
|}


=== Website ===
===Website===
Static website at https://munichmakerlab.de
Static website at https://munichmakerlab.de


=== Wiki ===
===Wiki===
MediaWiki at https://wiki.munichmakerlab.de/
MediaWiki at https://wiki.munichmakerlab.de/  
* Create your own account, needs to be confirmed by an admin
*Create your own account, needs to be confirmed by an admin


==== Maintenance ====
====Maintenance====
We currently have a bit of a spam problem, easiest way to fix it currently is to clean them up in the database directly.
We currently have a bit of a spam problem, easiest way to fix it currently is to clean them up in the database directly.
<pre>
<pre>
Line 68: Line 130:
</pre>
</pre>


=== Status ===
===Status===
Space status at https://status.munichmakerlab.de
Space status at https://status.munichmakerlab.de
* Details at [[StartYourEngines]]
*Details at [[StartYourEngines]]


=== MuMaBus ===
===MuMaBus===
Space Automation, see [[MuMaBus]] for details
Space Automation, see [[MuMaBus]] for details
* MQTT at jupiter.munichmakerlab.de
*MQTT at jupiter.munichmakerlab.de


=== Slack ===
===Slack ===
Chat, with bridge to IRC
Chat, with bridge to IRC
* Application in itself  is SaaS, we have a 50 seat community license. Talk to [[User:Tarwin|tarwin]] or [[User:Tiefpunkt|tiefpunkt]]
*Application in itself  is SaaS. Talk to [[User:Tarwin|tarwin]] or [[User:Tiefpunkt|tiefpunkt]]
* IRC bridge is powered by [https://github.com/munichmakerlab/RelayBot RelayBot], hosted on ???
* IRC bridge is powered by [https://github.com/munichmakerlab/RelayBot RelayBot], hosted on ???


=== Additional Services ===
===Additional Services===
* [https://munichmakerlab.de/calendar.ics Calendar as iCal]
*[https://munichmakerlab.de/calendar.ics Calendar as iCal]
* ical2email. Sends reminder emails for events to mailing list, using the wordpress calendar. Python script running daily on vps02.thearrow.de
*ical2email. Sends reminder emails for events to mailing list, using the wordpress calendar. Python script running daily on vps02.thearrow.de


== Access ==
==Access==
The following people currently have admin access to the infrastructure:
The following people currently have admin access to the infrastructure:
* [[User:Milian|Milian]]
*[[User:Milian|Milian]]
* [[User:Phier|Phier]]
*[[User:Phier|Phier]]
* [[User:Tiefpunkt|tiefpunkt]]
*[[User:Tiefpunkt|tiefpunkt]]


== Migration and Optimization 2024 ==
==Migration and Optimization 2024==
We're planning to consolidate services into a standard deployment model, consolidate external services, and maybe add some new ones.
We're planning to consolidate services into a standard deployment model, consolidate external services, and maybe add some new ones.
=== Ideas ===
Consolidate:
*Wiki: Containerize => Challenges: php modules; updating php/wiki; ggf. make it easier to include the plugins (maybe php compose module handling)
*Mailsystem: ??


Externally hosted, and to be transfered into MuMaLab Infrastructure
Externally hosted, and to be transfered into MuMaLab Infrastructure
* Tickets (https://tickets.mumalab.org/courses/)
*Tickets (https://tickets.mumalab.org/courses/)
* Calendar -> Google Calendar -> NextCloud
*Calendar -> Google Calendar -> NextCloud
* [https://tooljet.yt.gl/ ToolJet] (OpenUnitState)
*[https://tooljet.yt.gl/ ToolJet] (via [https://github.com/homeofmaking/OpenUnitState/tree/master OpenUnitState])


Planned Services
Planned Services
* NextCloud
*NextCloud
* evtl Ticket System
*evtl Ticket System
*Single Sign On: e.g. login to wiki either locally or via SSO. Later only SSO. Can be used e.g. for nextcloud or other services as well


Details unclear
Details unclear
* InfoBeamer
*InfoBeamer


==== Lightburn VM ====
2. Licence for lightburn is already available. Would be nice to give members the chance to prepare Laser stuff remote and just come to the lab to laser.


Challenges: No Linux support https://forum.lightburnsoftware.com/t/linux-support-to-end-after-v1-7/144605; exposing remote desktop in secure way
Remote Systems?
https://guacamole.apache.org/
==== SSO ====
IDPs
#Option: [https://goauthentik.io/ Authentik]
#Option: https://git.cccv.de/uffd/uffd
#Option: ...?
Auth:
[https://blog.cubieserver.de/2022/complete-guide-to-nextcloud-oidc-authentication-with-authentik/ Complete guide to Nextcloud OIDC authentication with Authentik]
[https://docs.goauthentik.io/integrations/services/nextcloud/ Integrate Authentik and Nextcloud]
Wiki Plugins for OIDC etc.: [https://www.mediawiki.org/wiki/Extension:PluggableAuth Plugable Auth]
===ToDos===
{| class="wikitable"
|+
!Topic
!Tasks
!Who is on it/wants to do it?
!Notes
|-
|SSO
|
*test authentik on saturn => done
*test connecting authentik and nextcloud  => done
*authentic password reset => in progress
*IT group in authentic
*test connecting wiki to authentik
*IaC configuration of nextcloud e.g. https://docs.goauthentik.io/integrations/services/nextcloud/#nextcloud-1
|Phier
|
|-
|Wiki
|
* containerize wiki (build on gitlab) and migrate to saturn as staging wiki
*update wiki
|open
|
|-
|Lightburn Remote VM
|
* Setup second lightburn licence on VM
* Expose VM with some secure remote connection
|Phier
|
|-
|Migrate Node Red
|Migrate to saturn
Old version 2.0.6
Latest: 4.0.5
|open
|https://hub.docker.com/r/nodered/node-red
|-
|Migrate Mailsetup
|
* Old setup is on Mailman 2, prevents Debian update
* Setup on saturn with Mailman 3
* Migrate existing stuff
|open
|
|-
|Migrate Ticket System
|Replace external https://tickets.mumalab.org/courses/ with Pretix instance on our server with ticket.munichmakerlab.de
* Setup DNS => Done
* Setup Pretix
* Connect to Authentic
|open
|Setup new, without migration
|-
|Migrate Token DB
|<s>Deploy [https://tooljet.yt.gl/ ToolJet]  on our server</s> (might be obsolete and using other approach)
Existing setup https://github.com/homeofmaking/OpenUnitState/tree/master
Migrate existing token from TBD
Adjust Lasercutter and door(?) to this DB
* Setup FQDN tooljet.munichmakerlab.de => Done
* Check how existing setup is working
...
|open
|Contact German for old DB/Automation setup
|-
|Setup Nextcloud
|
* Setup nextcloud
* Create shared folders e.g. for password safe
* Create calender
* Replace google calender with next cloud calender
* integrate new calender on homepage, kreativquartier, ticket system etc.
|Phier, Severin
|
|-
|Security
|
*Setup Firewall => Done
*Add fail2ban => in progress
*Add firewall to ansible: https://github.com/munichmakerlab/infrastructure/tree/debian-security-ansible
*update docker networks for better separation?
|Milian
|
|-
|IaC
|
* Setup Ansible in Repo => Done by Severin
*Playbook for Server => Done
*Playbook for Docker => in progress
|Milian
|
|-
|IoT Setup Lab
|Local https://www.home-assistant.io/ setup to have a plattform for additional functions like power monitoring or controlling of the devices in the lab
* wipe and re-install local server with proxmox => vulpix.intern.munichmakerlab.de
* setup home-assistant
|Adrian
|
|-
|Cleanup
|Check MQTT and other IT devices. Which are still up to date, which can be fixed and which are not existent anymore.
Compare [[Network]] and [[MuMaBus]]
check remaining stuff in the lab, if something depends on old ports: [[MuMaBus]] ; Cleanup also acl.conf
|Adrian
|
|-
|Backup
|Check Backup of Doorlok DB
=> old local server was removed
|
|
|}
=== DONE ===
{| class="wikitable"
!Topic
!Tasks
!Who is on it/wants to do it?
!Notes
|-
|Update Apps
| e.g. Etherpad
|Severin
|Done
|-
|MQTT
|Migrate to saturn and update to latest version.
Connect with Adrian => https://munichmakerlab.slack.com/archives/C79T8NFU7/p1731197933279969
* migrate /etc/system/systemd/docker-traefik.service to use config file => done
* adjust new /data/traefik/config/traefik.yml to integrate MQTT => done
* create mosquito config /data/mqtt/ with old config and new requirements => done
* migrate db /var/lib/mosquitto/mosquitto.db => done
* create /etc/system/systemd/docker-mosquitto.service => done
* test to start new mqqt service and restart traefik => done
* add new ports to ufw => done
* add new ports to ansible ufw: <nowiki>https://github.com/munichmakerlab/infrastructure/tree/debian-security-ansible=</nowiki>> done
* change FQDN to saturn and test => Done
* Test migrated Broker => Done
|Mili
|DONE
|}
[[Category:Infrastructure]]
[[Category:Infrastructure]]

Latest revision as of 23:47, 18 November 2024

Some documentation on MuMaLab's web infrastructure stuff.

Hosts

We currently have 3 VMs at Hetzner:

  • mars.munichmakerlab.de (Mars)
  • jupiter.munichmakerlab.de
  • saturn.munichmakerlab.de

Saturn

  • docker containers are started via systemd
  • cronjob added for cleanup of old docker images
  • Firewall: ufw => check via sudo ufw status verbose
  • fail2ban to ban hosts with too many authentication failures

Services

Service Name Hostname Functionality Server native/docker status Source
Website www.munichmakerlab.de Just Website, compare Github for details saturn docker productive Github Website
Wiki wiki.munichmakerlab.de Media wiki for knowledge sharing and documentation jupiter native productive
Nodered nodered.munichmakerlab.deAdmin: https://nodered.munichmakerlab.de/admin/ Automation like spacestatus, Slack Bots etc. jupiter docker productive
Log log.munichmakerlab.de Blog Tumblr - productive
Etherpad pad.munichmakerlab.de Colaboration text tool saturn docker productive Latest version: ether/etherpad-lite
Mailinglisten lists.munichmakerlab.de Mailsystem mars native productive
Roombooking rooms.munichmakerlab.de Original for reserving rooms during covid jupiter docker deactivated BookedSchedular
Slack Inviter slack.munichmakerlab.de Self invite capability for our slack saturn docker productive rauchg/slackin
Space Status status.munichmakerlab.de Button in the lab to mark space as open/closed on slack/homepage saturn docker productive Github Spacestatus
Eclipse Mosquitto (MQTT) mqtt.munichmakerlab.de MQTT to use for other servicesservices like status etc. Compare MuMaBus saturn docker productive Eclipse Mosquitto
Nextcloud nextcloud.munichmakerlab.de Document sharing, calendar saturn docker experimental
Traefik Reverseproxy saturn.munichmakerlab.de/dashboard/
(might be disabled)
Reverse proxy for other services saturn docker productive
Tickets (old) tickets.mumalab.org Ticket system for workshops and events German - productive https://github.com/pretix/pretix
Tickets tickets.munichmakerlab.de Ticket system for workshops and events saturn docker Todo https://github.com/pretix/pretix
Wiki Staging wiki-staging.munichmakerlab.de Wiki for testing (temporary) saturn docker Todo
Influx DB influxdb.munichmakerlab.de DB for particles sensor (temporary) saturn docker Todo https://hub.docker.com/_/influxdb
ToolJet tooljet.munichmakerlab.de Store member and token, who has which safet course etc. Might be replaced by authentik directly saturn docker Todo https://github.com/ToolJet/ToolJet

Website

Static website at https://munichmakerlab.de

Wiki

MediaWiki at https://wiki.munichmakerlab.de/

  • Create your own account, needs to be confirmed by an admin

Maintenance

We currently have a bit of a spam problem, easiest way to fix it currently is to clean them up in the database directly.

update mw_account_requests set acr_rejected = DATE_FORMAT(NOW(),"%Y%m%d%H%i%S"), acr_user = 1, acr_comment = "Spam, no confirmed mail address", acr_deleted = 1 where acr_email_authenticated is null and acr_rejected is null and acr_registration < now() - interval 7 day;

update mw_account_requests set acr_rejected = DATE_FORMAT(NOW(),"%Y%m%d%H%i%S"), acr_user = 1, acr_comment = "Spam", acr_deleted = 1 where acr_rejected is null and acr_registration < now() - interval 7 day;

update mw_account_requests set acr_rejected = DATE_FORMAT(NOW(),"%Y%m%d%H%i%S"), acr_user = 1, acr_comment = "Spam", acr_deleted = 1 where acr_rejected is null;

Status

Space status at https://status.munichmakerlab.de

MuMaBus

Space Automation, see MuMaBus for details

  • MQTT at jupiter.munichmakerlab.de

Slack

Chat, with bridge to IRC

Additional Services

  • Calendar as iCal
  • ical2email. Sends reminder emails for events to mailing list, using the wordpress calendar. Python script running daily on vps02.thearrow.de

Access

The following people currently have admin access to the infrastructure:

Migration and Optimization 2024

We're planning to consolidate services into a standard deployment model, consolidate external services, and maybe add some new ones.

Ideas

Consolidate:

  • Wiki: Containerize => Challenges: php modules; updating php/wiki; ggf. make it easier to include the plugins (maybe php compose module handling)
  • Mailsystem: ??

Externally hosted, and to be transfered into MuMaLab Infrastructure

Planned Services

  • NextCloud
  • evtl Ticket System
  • Single Sign On: e.g. login to wiki either locally or via SSO. Later only SSO. Can be used e.g. for nextcloud or other services as well

Details unclear

  • InfoBeamer

Lightburn VM

2. Licence for lightburn is already available. Would be nice to give members the chance to prepare Laser stuff remote and just come to the lab to laser.

Challenges: No Linux support https://forum.lightburnsoftware.com/t/linux-support-to-end-after-v1-7/144605; exposing remote desktop in secure way


Remote Systems?

https://guacamole.apache.org/

SSO

IDPs

  1. Option: Authentik
  2. Option: https://git.cccv.de/uffd/uffd
  3. Option: ...?

Auth:

Complete guide to Nextcloud OIDC authentication with Authentik

Integrate Authentik and Nextcloud

Wiki Plugins for OIDC etc.: Plugable Auth

ToDos

Topic Tasks Who is on it/wants to do it? Notes
SSO Phier
Wiki
  • containerize wiki (build on gitlab) and migrate to saturn as staging wiki
  • update wiki
open
Lightburn Remote VM
  • Setup second lightburn licence on VM
  • Expose VM with some secure remote connection
Phier
Migrate Node Red Migrate to saturn

Old version 2.0.6 Latest: 4.0.5

open https://hub.docker.com/r/nodered/node-red
Migrate Mailsetup
  • Old setup is on Mailman 2, prevents Debian update
  • Setup on saturn with Mailman 3
  • Migrate existing stuff
open
Migrate Ticket System Replace external https://tickets.mumalab.org/courses/ with Pretix instance on our server with ticket.munichmakerlab.de
  • Setup DNS => Done
  • Setup Pretix
  • Connect to Authentic
open Setup new, without migration
Migrate Token DB Deploy ToolJet on our server (might be obsolete and using other approach)

Existing setup https://github.com/homeofmaking/OpenUnitState/tree/master

Migrate existing token from TBD Adjust Lasercutter and door(?) to this DB

  • Setup FQDN tooljet.munichmakerlab.de => Done
  • Check how existing setup is working

...

open Contact German for old DB/Automation setup
Setup Nextcloud
  • Setup nextcloud
  • Create shared folders e.g. for password safe
  • Create calender
  • Replace google calender with next cloud calender
  • integrate new calender on homepage, kreativquartier, ticket system etc.
Phier, Severin
Security Milian
IaC
  • Setup Ansible in Repo => Done by Severin
  • Playbook for Server => Done
  • Playbook for Docker => in progress
Milian
IoT Setup Lab Local https://www.home-assistant.io/ setup to have a plattform for additional functions like power monitoring or controlling of the devices in the lab
  • wipe and re-install local server with proxmox => vulpix.intern.munichmakerlab.de
  • setup home-assistant
Adrian
Cleanup Check MQTT and other IT devices. Which are still up to date, which can be fixed and which are not existent anymore.

Compare Network and MuMaBus

check remaining stuff in the lab, if something depends on old ports: MuMaBus ; Cleanup also acl.conf

Adrian
Backup Check Backup of Doorlok DB

=> old local server was removed

DONE

Topic Tasks Who is on it/wants to do it? Notes
Update Apps e.g. Etherpad Severin Done
MQTT Migrate to saturn and update to latest version.

Connect with Adrian => https://munichmakerlab.slack.com/archives/C79T8NFU7/p1731197933279969

  • migrate /etc/system/systemd/docker-traefik.service to use config file => done
  • adjust new /data/traefik/config/traefik.yml to integrate MQTT => done
  • create mosquito config /data/mqtt/ with old config and new requirements => done
  • migrate db /var/lib/mosquitto/mosquitto.db => done
  • create /etc/system/systemd/docker-mosquitto.service => done
  • test to start new mqqt service and restart traefik => done
  • add new ports to ufw => done
  • add new ports to ansible ufw: https://github.com/munichmakerlab/infrastructure/tree/debian-security-ansible=> done
  • change FQDN to saturn and test => Done
  • Test migrated Broker => Done
Mili DONE