Puzzle ITC - Notice history

All systems operational

Operational

OpenShift 4 Cluster cloudscale.ch - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 99.91%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

OpenShift 4 Cluster LPG1 cloudscale.ch - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021
Operational

Puzzle Website (puzzle.ch) - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Mail - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 99.25%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Rocket.Chat - Operational

100% - uptime
May 2021 · 99.77%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Taiga - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

PuzzleTime - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Wiki - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Puzzle SSO (Keycloak) - Operational

100% - uptime
May 2021 · 99.77%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

CodiMD - Operational

100% - uptime
May 2021 · 99.77%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

OpenProject - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Nextcloud - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

PuzzleSkills - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

GitLab - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Jenkins - Operational

98% - uptime
May 2021 · 95.38%Jun · 97.50%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Quay (registry.puzzle.ch ) - Operational

100% - uptime
May 2021 · 99.78%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Harbor (harbor.puzzle.ch) - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Grafana - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Puzzle News Site (news.puzzle.ch) - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Cryptopus - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Artifactory - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

SonarQube - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Puzzle Meet - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Zammad (help.puzzle.ch) - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Sentry - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

CardDAV - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Webmail - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Argo CD - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Youtrack - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

OKR - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Vault - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Loki - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

LDAP - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Puzzle Docs (docs.puzzle.ch) - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021
Operational

Network Office-Bern - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Network Office-Zürich - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Google Meet Hardware - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Telephone Setup - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021
Operational

cloudscale.ch - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Amazon AWS - Operational

100% - uptime
May 2021 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2021
Jun 2021
Jul 2021

Notice history

Jul 2021

Update CSI drivers OpenShift CloudScale Production
  • Update
    30. July 2021 at 11:35
    Completed
    30. July 2021 at 11:35

    Maintenance has completed successfully.

  • Completed
    30. July 2021 at 11:23
    Completed
    30. July 2021 at 11:23

    We are planning to update the CSI drivers on OpenShift CloudScale Production. We do not expect any impact or downtime during this time. During maintenance, requesting a new persistent volume may take longer.

Sentry Migration
  • Completed
    29. July 2021 at 16:9
    Completed
    29. July 2021 at 16:9

    Maintenance has completed successfully.

  • In progress
    29. July 2021 at 15:31
    In progress
    29. July 2021 at 15:31

    Maintenance is now in progress.

  • Planned
    29. July 2021 at 15:30
    Planned
    29. July 2021 at 15:30

    We will be migrating Sentry from Openshift 3 to 4 tonight. During the migration Sentry will be temporarily unavailable.

Webmail 504 Gateway Time-out
  • Resolved
    Resolved

    We just resolved the issue!

  • Investigating
    Investigating

    We are currently investigating this incident.

/sys Wartungsfenster
  • Completed
    22. July 2021 at 20:2
    Completed
    22. July 2021 at 20:2

    Maintenance has completed successfully.

  • Planned
    22. July 2021 at 16:0
    Planned
    22. July 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

  • In progress
    22. July 2021 at 15:54
    In progress
    22. July 2021 at 15:54

    Maintenance is now in progress.

OpenShift 4 maintenance
  • Completed
    7. July 2021 at 17:7
    Completed
    7. July 2021 at 17:7

    Maintenance has completed successfully.

  • In progress
    7. July 2021 at 16:11
    In progress
    7. July 2021 at 16:11

    Maintenance is now in progress.

  • Planned
    7. July 2021 at 16:0
    Planned
    7. July 2021 at 16:0

    We are planning for a scheduled maintenance during that time. The following production-labelled pods are going to encounter a short downtime due to rescheduling:

    • Pod "cm-acme-http-solver-bq6vm" in namespace "pitc-lightning-prod"
    • Pod "pfeedback-waf-1-gwfg9" in namespace "pitc-pfeedback-prod"
    • Pod "rails-5df8d96f8b-cjbj4" in namespace "pitc-skills-prod"

Jun 2021

Rancher & Kubernetes Maintenance
  • Completed
    1. July 2021 at 18:15
    Completed
    1. July 2021 at 18:15

    Maintenance has completed successfully.

    Everything is up and running again

  • In progress
    1. July 2021 at 15:52
    In progress
    1. July 2021 at 15:52

    Maintenance is now in progress.

  • Planned
    30. June 2021 at 16:0
    Planned
    30. June 2021 at 16:0

    We are planning for a scheduled maintenance during that time for:

    • OS patching
    • Ingress controller upgrade
    • worker02 node replacement/rebuild
Cryptopus Release
  • Completed
    30. June 2021 at 11:23
    Completed
    30. June 2021 at 11:23

    Maintenance has completed successfully.

  • In progress
    30. June 2021 at 11:20
    In progress
    30. June 2021 at 11:20

    Maintenance is now in progress.

  • Planned
    30. June 2021 at 11:15
    Planned
    30. June 2021 at 11:15

    We're updating Cryptopus. There will be some interruptions. Please use http://cryptopus.fallback.puzzle.ch/ (read only, vpn) during that time. check https://puzzle-status.ch for more details

/sys Wartungsfenster
  • Completed
    25. June 2021 at 15:40
    Completed
    25. June 2021 at 15:40

    We completed this Maintenance yesterday at 23:30 but somehow forgot to update this status, sorry for that. Have a nice Weekend

  • In progress
    24. June 2021 at 16:0
    In progress
    24. June 2021 at 16:0

    Maintenance is now in progress.

  • Planned
    24. June 2021 at 16:0
    Planned
    24. June 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Major Gitlab Outage
  • Resolved
    Resolved

    This has been resolved last week during our maintenance window

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result. Another smaller fix is in the making, and we will inform you as soon as we know more.

  • Investigating
    Investigating

    We are currently investigating this incident.

OpenShift 4 Updates
  • Completed
    7. June 2021 at 19:31
    Completed
    7. June 2021 at 19:31

    Maintenance has completed successfully.

  • In progress
    7. June 2021 at 16:1
    In progress
    7. June 2021 at 16:1

    Maintenance is now in progress.

  • Planned
    7. June 2021 at 16:0
    Planned
    7. June 2021 at 16:0

    We are planning for a scheduled maintenance during that time. The OpenShift 4 clusters on AWS and cloudscale are going to be updated to the latest version.

May 2021

/sys Wartungsfenster
  • Completed
    27. May 2021 at 20:19
    Completed
    27. May 2021 at 20:19

    Maintenance has completed successfully.

  • In progress
    27. May 2021 at 16:24
    In progress
    27. May 2021 at 16:24

    Maintenance is now in progress.

  • Planned
    27. May 2021 at 16:0
    Planned
    27. May 2021 at 16:0

    As usual we will be conducting our monthly maintenance window to apply updates and configuration enhancements. Services may experience a short interruption and/ or periods of degraded performance.

Jenkins freestyle job configuration
  • Resolved
    Resolved

    We just resolved the issue!

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    We found the Plugin which causes the Issue. The Hudson Locks and Latches plugin is no longer compatible with the newest Jenkins Version.

    We will delete the plugin and restart Jenkins today at 12:30.

  • Investigating
    Investigating

    Saving freestyle job configuration changes does not work.

    We are currently investigating this incident.

Partial outage of PROD Rancher cluster worker 08
  • Resolved
    Resolved

    We just resolved the issue!

  • Identified
    Identified

    Cordoned, drained & rebooted node. The "flapping" behavior seems to be caused by the "broken" runC version we are currently running on our Rancher nodes: https://github.com/kubernetes/kubernetes/issues/45419#issuecomment-819574343

    [root@k8s-worker08 ~]# runc --version runc version 1.0.0-rc93 commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec spec: 1.0.2-dev go: go1.13.15 libseccomp: 2.3.1

Jenkins Gitlab Plugin Update
  • Completed
    6. May 2021 at 16:7
    Completed
    6. May 2021 at 16:7

    Maintenance has completed successfully.

  • Planned
    6. May 2021 at 16:0
    Planned
    6. May 2021 at 16:0

    We are planning for a scheduled maintenance during that time. Jenkins will be restartet during this time.

  • In progress
    6. May 2021 at 15:59
    In progress
    6. May 2021 at 15:59

    Maintenance is now in progress.

Kubernetes Worker down
  • Resolved
    Resolved

    I checked the resource request/limits and optimized them a bit. Furthermore there is a new worker node running. I'm closing this incident all services are up and running.

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result.

  • Identified
    Identified

    The node is up and running again. But we need to identify what caused the problem, as earlier there was already a node outage. Current guess is, not enough memory, nodes are too crowed and resource management is probably not the best. We are adding a new node (maybe temporary), to mitigate the problem

  • Investigating
    Investigating

    Anonther Kubernetes Worker is unfortunatly down

May 2021 to Jul 2021

Next