Cloudsatck Overload
Chers utilisateurs CI,
Depuis quelques temps nous constatons des difficultĂ©s pour dĂ©marer des VMs Cloudstack utilisant un grand nombre de coeurs CPU. Le dĂ©marage est en erreur avec un message “Unable to start a VM due to insufficient capacity”. Nous cherchons un moyen d’accroitre les capacitĂ©s CPU du service mais celĂ va prendre du temps.
En attendant, pour que chacun puisse profiter pleinement de Cloudstack nous vous enjoignons fortement Ă Ă©teindre toutes les VMs qui ne servent pas. Vous pouvez Ă©gelement utiliser un quadriciel comme opentofu pour crĂ©er et supprimer des VMs Ă la demande. Vous trouverez un exemple d’utilisation d’opentofu dans la gallery d’exemples CI ici.
Cordialement,
L’Ă©quipe CI
Dear CI users,
For some time now we’ve been experiencing difficulties starting Cloudstack VMs using a large number of CPU cores. The startup error is “Unable to start a VM due to insufficient capacity”. We’re looking for a way to increase the service’s CPU capacity, but it’s going to take some time.
In the meantime, so that everyone can take full advantage of Cloudstack we strongly urge you to turn off all VMs that are not in use. You can also use a framework like opentofu to create and delete VMs on demand. You’ll find an example of how to use opentofu in the CI example gallery here.
Best,
The CI Team
Deletion of old Jenkins servers
Chers utilisateurs CI,
Suite aux rappels envoyés par e-mail début mai et début juin, nous vous informons que les serveurs Jenkins dont la version est antérieure à la version 2.440.3 LTS au 9 juillet 2024 seront supprimés des serveurs.
Cordialement,
L’Ă©quipe CI
PS: si vous avez explicitement demandé un délai pour la montée en version, ne tenez pas compte de ce message.
Dear CI users,
Further to reminders sent by e-mail at the beginning of May and June, we would like to inform you that Jenkins servers with versions prior to version 2.440.3 LTS on July 9, 2024 will be removed from the servers.
Best,
The CI Team
PS: if you have explicitly requested an upgrade deadline, please disregard this message.
Jenkins deactivation for new projects
Chers utilisateurs CI,
Rencontrant pas mal de problèmes de maintenance sur les serveurs Jenkins, nous avons dĂ©cidĂ© d’enlever de l’interface utilisateur la possibilitĂ© d’activer Jenkins Ă la crĂ©ation d’un projet CI. Cependant, vous pourrez toujours crĂ©er un ticket helpdesk (https://helpdesk.inria.fr/categories/320/submit) si vous souhaitez avoir Jenkins.
Cordialement,
L’Ă©quipe CI
Dear CI users,
As we are regularly facing maintenance issues with Jenkins servers, we have decided to remove from the user interface the option to activate Jenkins when creating a CI project. However, you will still be able to create a helpdesk ticket (https://helpdesk.inria.fr/categories/320/submit) if you wish to use Jenkins.
Best regards,
The CI Team
Reboot of the Jenkins servers
A maintenance operation requires us to stop all CI jenkins servers. This operation will take place tomorrow, Thursday December 14 at 1:30 pm. It should last half an hour.
Once the servers have been restarted, your jenkins projects may remain switched off. You may have to restart them manually on ci.inria.fr.
We apologize for the inconvenience.
CI plateform interruption
We will be updating and restarting some machines in charge of our infrastructure tomorrow, Tuesday, November 21, between 1:30 pm and 2:00 pm. This update is expected to be brief. While the virtual machines should not be impacted, all services related to user accounts, including login, will be temporarily unavailable.
We apologize for any inconvenience and will endeavor to minimize the downtime. Thank you for your understanding.
Planned shutdown for Apple hypervisors: June 26 6pm till June 27 9pm
Apple hypervisors will be shutdown from Monday 26 June 6pm till Tuesday 27 June 9pm for maintainance.
Sorry for the inconvenience.
Login failure on CI portal and CloudStack on 2023/04/05: LDAP certificate expired
Dear CI users,
The LDAP certificate for ci.inria.fr has been renewed: the certificate was expired on 2023/04/05 and login was impossible on that day on CI portal (ci.inria.fr) and CloudStack (login on virtual machines still worked). You should no longer have difficulties to login on CI anymore.
Sorry for the inconvenience.
Featured template update for Ubuntu 20.04 LTS amd64
There is a new featured template for Ubuntu 20.04 LTS amd64
(ubuntu-20.04-lts
). This template subsumes the former template
Ubuntu 20.04 amd64
, which is no longer featured.
Changes with respect to the former template:
- the initialization service for ‘~ci/.ssh/authorized_keys` is fixed, and the contents of this file will no longer be reset on each boot;
- cloud-init is now configured to consider CloudStack provided user-data by default;
- packages are upgraded to their last available version.
Thank you very much for using CI.
CloudStack is now updated to 4.16
CloudStack is now updated to 4.16. You should now be able to start your virtual machines again via the portal ci.inria.fr or via CloudStack API or web interface.
You may refer to the portal documentation: https://inria-ci.gitlabpages.inria.fr/doc/page/web_portal_tutorial/#slaves
Thanks to this update, Mac OS Catalina and Mojave are supported, and nested KVM is enabled.
Thank you very much for using CI.
CloudStack is now updated to 4.16
CloudStack is now updated to 4.16. You should now be able to start your virtual machines again via the portal ci.inria.fr.
You may refer to the portal documentation: https://inria-ci.gitlabpages.inria.fr/doc/page/web_portal_tutorial/#slaves
We still have a problem that prevents users to log in to CloudStack by the web interface. Only the CloudStack web interface seems to be affected: you are still able to manage your virtual machines via the portal or the CloudStack API. We will fix the connection to the web interface as soon as possible.
Thank you and sorry for the inconvenience.
VMs will be interrupted on Tuesday afternoon 2023-02-07, 2pm-6pm, for CloudStack maintenance
All VMs hosted on ci.inria.fr will be interrupted on Tuesday afternoon 2023-02-07, 2pm-6pm, for CloudStack maintenance. CloudStack will be upgraded to 4.16, and nested KVM will be enabled. The actual interruption may be shorter, we will publish an announce when the platform will be back. You will have to restart the VMs manually via ci.inria.fr or CloudStack. Thank you!
Jenkins option `--ajpPort13=-1` removed
The option --ajpPort13=-1
has been removed for launching
Jenkins. This option was necessary for very old version of Jenkins,
but was ignored by all currently deployed Jenkins instance, and is no
longer supported since Jenkins 2.375 (released on 2022-10-25). If you
experienced failure while updating Jenkins recently, you should have a
try now. Thank you!
Happy new year!
Jenkins storage is back to normal
The shared storage for Jenkins instance is back to normal since last Thursday. If your Jenkins instance is down due to disk space outage, restarting the instance should now work again. Sorry for the inconvenience, we will try in the next few days to take counter-measures to prevent these kinds of problems in the future.
Jenkins storage is full
The shared storage for Jenkins instance is full: this probably prevents all Jenkins instances to work properly. We are currently investigating why we have difficulties to free space and I hope we will be able to restore the service very soon!
LDAP certificate renewal
The certificate of the LDAP server was expired this morning and has been renewed. You may have had some difficulties to connect to ci.inria.fr, CloudStack and Jenkins instances this morning but everything should work again now! Sorry for the inconvenience.
VMs expiration date extension
You may have received this morning some emails inviting you to extend
the expiration date of your CI virtual machines. As you may have
noticed, the URLs were wrong: /prolongation
was missing at the
end. You may either use the fixed URL, or the “Extends” button in the
slave table in the web interface of your CI project.
Sorry for this inconvenience, and thank you for extending your VMs!
Shared docker runners for gitlab.inria.fr
We are pleased to announce that shared docker runners are now available for gitlab.inria.fr!
Thanks to everyone who helped testing the service during the (very long!) beta period. You can right now enable the shared runners in the CI/CD settings of your Gitlab projects.
See the shared runners documentation for more details. The three following docker runners are available:
- small (1 CPU, 2048 MB RAM), tags: ci.inria.fr, linux, small
- medium (2 CPUs, 4096 MB RAM), tags: ci.inria.fr, linux, medium
- large (4 CPUs, 8096 MB RAM), tags: ci.inria.fr, linux, large
The runners will handle jobs declared in the .gitlab-ci.yml
file of
the project repository, at the condition that the jobs have at least
one tag among ci.inria.fr
, linux
and either small
, medium
or
large
(these three last tags are exclusive: they cannot be used
together for the same job since no runner will match more than one of
them).
We would like to send a big thank you to Anthony Baire for all his efforts in developping this service, and we would like to thank also the CI infrastructure team of the Inria DSI who helped in deploying the platform.
Thank you very much for using ci.inria.fr, and happy hacking!
Mac OS X VMs keep rebooting
For a few days and for an unknown reason, Mac OS X VMs are continuously rebooting when they are newly created or when their offer is changed. We are aware of the problem and are trying to fix it as soon as possible.
Thank you for your understanding!
LDAP server down
Update 2021-04-13T17:18:52+02:00: The server is now running again! Thank you for your understanding.
The LDAP server is down and we are currently looking for a solution. Since LDAP is used for authentification for portal, logins, and SSH access to VMs, all accesses to these services are currently blocked, even if VMs are still running. We hope that this problem will be solved as soon as possible.
Thank you for your understanding!
ci-ssh upgrade, 13 April 2021 9:30am-10:00am
The ci-ssh
machine will be upgraded on 13 April 2021 between 9:30am and 10:00am,
and will be unavailable for about 5 minutes during this interval.
This machine is the proxy used for connecting CI VMs by ssh, therefore the VMs will be unavailable as well during this short period of time.
This upgrade will provide a more recent version of OpenSSH
(8.0p
instead of 5.3p1
), which will bring support for ed25519 keys and
other new cyphering algorithms.
Thank you for your understanding!
Web portal maintenance, 6 April 2021 6pm-6:30pm
The web portal ci.inria.fr may be unavailable on 6 April 2021 from 6pm till 6:30pm for maintenance purposes (LDAP). It concerns only the web portal. CloudStack, virtual machines, Jenkins and gitlab-ci shall not be impacted.
Thank you for your understanding!
Web portal maintenance, the 30th of March 2021 18:00-18:30
The web portal ci.inria.fr will be unavailable the 30th of March 2021 from 18:00 till 18:30 for maintenance purposes. It concerns only the web portal. CloudStack, virtual machines, Jenkins and gitlab-ci shall not be impacted.
Thank you for your understanding!
VM expiration: 1 year, renewable
Each VM hosted by ci.inria.fr has now an expiration date, fixed in one year from now for current VMs, and one year after the creation date for future VMs. This rule will make us able to clean unused VMs and save ressources for new projects.
These expiration dates are freely renewable, using the dedicated button “Extend” in the VM list associated to a project in the portal ci.inria.fr.
When VMs are about to expire, project admins will be reminded three times by email (with a link to extend the expiration date) starting three weeks before the expiration. Expired VMs will be deleted and project admins will be notified (there will be a 24h delay before expunging, during which the VM can be recovered).
Thank you for your understanding!
CI Service interruption for Mac OS VMs on Wednesday 17 - Thursday 18 March
In order to increase the Mac OS virtualization capacities of the CI cloud (more CPU cores and more RAM available for Mac OS virtual machines), new Mac OS hosts will be deployed on Wednesday 17 - Thursday 18 March.
During these two days, it will be impossible to create new Mac OS virtual machines, and existing Mac OS virtual machines may become unavailable or restarted without notice. We are sorry for these inconveniences.
Thank you for your understanding!
CI portal unavailable on Tuesday 16 March: cloudstack upgrade v4.11→v4.13
The cloudstack software that manages CI virtual machines will be upgraded on Tuesday 16 March (from v4.11 to v4.13): the CI portal will likely be unavailable on that day, and in particular creating new virtual machines or projects will be impossible. However, existing virtual machines and Jenkins instances will still be available as usual. If you need new virtual machines, please create them before or wait until the end of the upgrade!
The upgrade will make ISO image uploading easier and will allow virtual machines to be created with “custom” offers, by choosing manually and independently the number of cores and the amount of RAM.
As a side note, thanks to many efforts performed by our users, a notable amount of memory has been freed on the VMware hypervisor: if you were blocked for creating new Mac OS virtual machines, you can now try again.
Thank you for using CI!
Windows 10 template
The Windows 10 template windows10-choco-java-virtio2
is available
for more than a year now, and provides minimal Windows settings with
the native OpenSSH server preconfigured and the package manager
Chocolatey for installing software.
The template comes with a documentation, which gives commands to install Visual Studio, Cygwin, and other common software. https://ci.inria.fr/doc/page/windows10-template/
You may even have a look on how the template has been configured over a vanilla Windows 10 installation. https://ci.inria.fr/doc/page/windows10-template-create/
Portal update: creation of projects that do not use Jenkins
The maintenance on ci.inria.fr scheduled on Tuesday, September 22nd was successful and the CI web portal is now fully working again. This maintenance work enabled us to introduce new functionalities to the portal, some of them are visible to anyone, and others will allow the CI administrators to make the platform more robust.
All Inria users can now choose, on the creation of a new project, whether the project will be associated to a Jenkins instance or not. Projects without Jenkins instances are useful to create virtual machines on Cloudstack to be used by other CI software such as Gitlab CI. These projects will not have to worry about Jenkins maintenance anymore, and the resource consumption will be reduced.
Therefore, do not hesitate to create new projects without Jenkins or to stop unused Jenkins instances! It will help us to reduce memory consumption.
Concerning more internal changes, the back-end is now driven by ansible instead of puppet: this change will make the portal easier to maintain and evolve. Moreover, CI administrators can now have finer control on the memory consumption of Jenkins instances: they can set in particular the allowed amount of RAM dedicated to each instance on demand. Besides, they can now balance the workload among the servers that run the Jenkins instances by moving an instance from one server to another if necessary.
Saving resources for MacOS X slaves
Please read this message if you use or intent to use a slave running MacOSX on CloudStack.
We are getting short in RAM on the VMWare infrastructure used to provide MacOS slaves on CloudStack. We are doing our best to get more RAM, but it would also help if the unused MacOS slaves would be deleted by their owners (until a bit more RAM becomes available, it will be impossible to create new MacOSX slaves on CloudStack).
So, if you have created a MacOSX slave on CloudStack and it turns out you do not really need it, we would be grateful if you could take just a few minutes to remove it through the CI portal:
Go to the page of your project, then choose “Manage slaves”, stop the slave you would like to delete and finally delete it.
Many thanks for your understanding.
We will send an update as soon as it becomes available to us.