Deploy Second Scheduler scheduler-plugins in a Cluster¶
This page describes how to deploy a second scheduler-plugins in a cluster.
Why do we need scheduler-plugins?¶
The cluster created through the platform will install the native K8s scheduler-plugin, but the native scheduler-plugin has many limitations:
- The native scheduler-plugin cannot meet scheduling requirements, so you can use either CoScheduling, CapacityScheduling or other types of scheduler-plugins.
- In special scenarios, a new scheduler-plugin is needed to complete scheduling tasks without affecting the process of the native scheduler-plugin.
- Distinguish scheduler-plugins with different functionalities and achieve different scheduling scenarios by switching scheduler-plugin names.
This page takes the scenario of using the vgpu scheduler-plugin while combining the coscheduling plugin capability of scheduler-plugins as an example to introduce how to install and use scheduler-plugins.
Installing scheduler-plugins¶
Prerequisites¶
- kubean is a new feature introduced in v0.13.0, please ensure that your version is v0.13.0 or higher.
- The installation version of scheduler-plugins is v0.27.8, please ensure that the cluster version is compatible with it. Refer to the document Compatibility Matrix.
Installation Process¶
-
Add the scheduler-plugins parameter in Create Cluster -> Advanced Settings -> Custom Parameters.
scheduler_plugins_enabled:true scheduler_plugins_plugin_config: - name: Coscheduling args: permitWaitingTimeSeconds: 10 # default is 60
Parameters:
scheduler_plugins_enabled
Set to true to enable the scheduler-plugins capability.- You can enable or disable certain plugins by setting the
scheduler_plugins_enabled_plugins
orscheduler_plugins_disabled_plugins
options. See K8s Official Plugin Names for reference. - If you need to set parameters for custom plugins, please configure
scheduler_plugins_plugin_config
, for example: set thepermitWaitingTimeoutSeconds
parameter for coscheduling. See K8s Official Plugin Configuration for reference.
-
After successful cluster creation, the system will automatically install the scheduler-plugins and controller component loads. You can check the load status in the proper cluster's deployment.
Using scheduler-plugins¶
Here is an example of how to use scheduler-plugins by demonstrating a scenario where the vgpu scheduler is used in combination with the coscheduling plugin capability of scheduler-plugins.
-
Install vgpu in the Helm Charts and set the values.yaml parameters.
schedulerName: scheduler-plugins-scheduler
: This is the scheduler name for scheduler-plugins installed by kubean, and currently cannot be modified.scheduler.kubeScheduler.enabled: false
: Do not install kube-scheduler and use vgpu-scheduler as a separate extender.
-
Extend vgpu-scheduler on scheduler-plugins.
[root@master01 charts]# kubectl get cm -n scheduler-plugins scheduler-config -ojsonpath="{.data.scheduler-config\.yaml}"
apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration leaderElection: leaderElect: false profiles: # Compose all plugins in one profile - schedulerName: scheduler-plugins-scheduler plugins: multiPoint: enabled: - name: Coscheduling - name: CapacityScheduling - name: NodeResourceTopologyMatch - name: NodeResourcesAllocatable disabled: - name: PrioritySort pluginConfig: - args: permitWaitingTimeSeconds: 10 name: Coscheduling
Modify configmap of scheduler-config for scheduler-plugins:
[root@master01 charts]# kubectl get cm -n scheduler-plugins scheduler-config -ojsonpath="{.data.scheduler-config\.yaml}"
apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration leaderElection: leaderElect: false profiles: # Compose all plugins in one profile - schedulerName: scheduler-plugins-scheduler plugins: multiPoint: enabled: - name: Coscheduling - name: CapacityScheduling - name: NodeResourceTopologyMatch - name: NodeResourcesAllocatable disabled: - name: PrioritySort pluginConfig: - args: permitWaitingTimeSeconds: 10 name: Coscheduling extenders: - urlPrefix: "${urlPrefix}" filterVerb: filter bindVerb: bind nodeCacheCapable: true ignorable: true httpTimeout: 30s weight: 1 enableHTTPS: true tlsConfig: insecure: true managedResources: - name: nvidia.com/vgpu ignoredByScheduler: true - name: nvidia.com/gpumem ignoredByScheduler: true - name: nvidia.com/gpucores ignoredByScheduler: true - name: nvidia.com/gpumem-percentage ignoredByScheduler: true - name: nvidia.com/priority ignoredByScheduler: true - name: cambricon.com/mlunum ignoredByScheduler: true
-
After installing vgpu-scheduler, the system will automatically create a service (svc), and the urlPrefix specifies the URL of the svc.
-
Restart the scheduler pod of scheduler-plugins to load the new configuration file.
Note
When creating a vgpu application, you do not need to specify the name of a scheduler-plugin. The vgpu-scheduler webhook will automatically change the scheduler's name to "scheduler-plugins-scheduler" without manual specification.