其他
mlr3：超参数调优
主要包括4个部分的内容，今天学习第一部分。
模型调优
调整超参数
方法一：通过`tuninginstancesinglecrite`和`tuner`训练模型
方法二：通过`autotuner`训练模型
超参数设定的方法
参数依赖
嵌套重抽样
进行嵌套重抽样
评价模型
把超参数应用于模型
Hyperband调参
特征选择
filters
计算分数
计算变量重要性
组合方法（wrapper methods）
自动选择
模型调优
当你对你的模型表现不满意时，你可能希望调高你的模型表现，可通过超参数调整或者尝试一个更加适合你的模型，本篇将介绍这些操作。
本章主要包括3个部分的内容：
超参数调整
机器学习模型都有默认的超参数，但是这些超参数不能根据数据自动调整，往往不能得到更好的性能表现。但是手动调整往往也不能获得最佳的表现，mlr3包含自动调参的策略，在此包中实现自动调参，需要指定：搜索空间（search_space），优化算法（调参方法），评估方法（重抽样策略），评价指标。
特征选择
主要是通过mlr3filter和mlr3select包进行。
嵌套重抽样
调整超参数
很多人戏称调参的过程就像是"炼丹"！确实差不多，而且很多时候你调整后的结果可能还不如默认的结果好！这就好比打游戏，"一顿操作猛如虎，一看战绩0比5"！
模型调优一定要基于对算法和数据的理解进行，不是随便调的。
我们使用著名的糖尿病数据集进行演示，首先创建任务
library(mlr3verse)
## 载入需要的程辑包：mlr3
task <- tsk("pima")
print(task)
## <TaskClassif:pima> (768 x 9)
## * Target: diabetes
## * Properties: twoclass
## * Features (8):
## - dbl (8): age, glucose, insulin, mass, pedigree, pregnant, pressure,
## triceps
选择算法,查看算法支持的超参数
learner <- lrn("classif.rpart")
learner$param_set
## <ParamSet>
## id class lower upper nlevels default value
## 1: cp ParamDbl 0 1 Inf 0.01
## 2: keep_model ParamLgl NA NA 2 FALSE
## 3: maxcompete ParamInt 0 Inf Inf 4
## 4: maxdepth ParamInt 1 30 30 30
## 5: maxsurrogate ParamInt 0 Inf Inf 5
## 6: minbucket ParamInt 1 Inf Inf <NoDefault[3]>
## 7: minsplit ParamInt 1 Inf Inf 20
## 8: surrogatestyle ParamInt 0 1 2 0
## 9: usesurrogate ParamInt 0 2 3 2
## 10: xval ParamInt 0 Inf Inf 10 0
在这里我们选择调整复杂度参数cp和最小分支参数minsplit，并设定超参数的调整范围：
search_space <- ps(
cp = p_dbl(lower = 0.001, upper = 0.1),
minsplit = p_int(lower = 1, upper = 10)
)
search_space
## <ParamSet>
## id class lower upper nlevels default value
## 1: cp ParamDbl 0.001 0.1 Inf <NoDefault[3]>
## 2: minsplit ParamInt 1.000 10.0 10 <NoDefault[3]>
然后选择重抽样方法和性能指标
hout <- rsmp("holdout", ratio = 0.7)
measure <- msr("classif.ce")
接下来进行调参有两种方法。
方法一：通过tuninginstancesinglecrite和tuner训练模型
library(mlr3tuning)
## 载入需要的程辑包：paradox
evals20 <- trm("evals", n_evals = 20) # 设定何时停止训练
# 统一放入instance中
instance <- TuningInstanceSingleCrit$new(
task = task,
learner = learner,
resampling = hout,
measure = measure,
terminator = evals20,
search_space = search_space
)
instance
## <TuningInstanceSingleCrit>
## * State: Not optimized
## * Objective: <ObjectiveTuning:classif.rpart_on_pima>
## * Search Space:
## <ParamSet>
## id class lower upper nlevels default value
## 1: cp ParamDbl 0.001 0.1 Inf <NoDefault[3]>
## 2: minsplit ParamInt 1.000 10.0 10 <NoDefault[3]>
## * Terminator: <TerminatorEvals>
## * Terminated: FALSE
## * Archive:
## <ArchiveTuning>
## Null data.table (0 rows and 0 cols)
关于何时停止训练，mlr3给出了5种方法：
Terminate after a given time：一定时间后停止 Terninate after a given number of iterations：特定迭代次数后停止 Terminate after a specific performance has been reached：达到特定性能指标后停止 Terminate when tuning dose find a better configuration for a given number of iterations：在给定迭代次数中确实找到表现很好的参数组合后停止 A combination of above in ALL or ANY fashon：上面几种方法组合
然后还需要设置超参数搜索的方法：
mlr3tuning目前支持以下超参数搜索的方法：
Grid search：网格搜索 Random search：随机搜索 Generalized simulated annealing Non-Linear optimization
# 这里选择网格搜索
tuner <- tnr("grid_search", resolution = 5) # 网格搜索
接下来就是进行训练模型，上面我们设置了网格搜索的分辨率是5，我们有2个超参数需要调整，所以理论上一共有5 * 5 = 25个组合，但是在前面的停止搜索的方法中我们选择了n_evals = 20，所有实际上在评价完20个组合后就会停止了！
#lgr::get_logger("mlr3")$set_threshold("warn")
#lgr::get_logger("bbotk")$set_threshold("warn") # 减少屏幕打印内容
tuner$optimize(instance)
## INFO [20:51:28.312] [bbotk] Starting to optimize 2 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [20:51:28.331] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.344] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.378] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.401] [mlr3] Finished benchmark
## INFO [20:51:28.421] [bbotk] Result of batch 1:
## INFO [20:51:28.422] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.422] [bbotk] 0.02575 3 0.2130435 0 0 0.02
## INFO [20:51:28.422] [bbotk] uhash
## INFO [20:51:28.422] [bbotk] 3862f1f2-138e-40e2-907e-1db19c255134
## INFO [20:51:28.423] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.430] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.433] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.445] [mlr3] Finished benchmark
## INFO [20:51:28.464] [bbotk] Result of batch 2:
## INFO [20:51:28.465] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.465] [bbotk] 0.07525 8 0.2130435 0 0 0
## INFO [20:51:28.465] [bbotk] uhash
## INFO [20:51:28.465] [bbotk] 67bfb6fc-b5a2-494b-bd1d-700d979763fc
## INFO [20:51:28.466] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.473] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.476] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.488] [mlr3] Finished benchmark
## INFO [20:51:28.511] [bbotk] Result of batch 3:
## INFO [20:51:28.512] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.512] [bbotk] 0.0505 5 0.2130435 0 0 0
## INFO [20:51:28.512] [bbotk] uhash
## INFO [20:51:28.512] [bbotk] dc9a75ad-c027-40f3-a06b-c62c12cb5642
## INFO [20:51:28.513] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.519] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.523] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.536] [mlr3] Finished benchmark
## INFO [20:51:28.555] [bbotk] Result of batch 4:
## INFO [20:51:28.556] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.556] [bbotk] 0.001 1 0.3 0 0 0
## INFO [20:51:28.556] [bbotk] uhash
## INFO [20:51:28.556] [bbotk] 80066053-7bcb-4602-a08d-aaf6e0eb77bb
## INFO [20:51:28.557] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.564] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.567] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.579] [mlr3] Finished benchmark
## INFO [20:51:28.598] [bbotk] Result of batch 5:
## INFO [20:51:28.599] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.599] [bbotk] 0.1 3 0.2130435 0 0 0.02
## INFO [20:51:28.599] [bbotk] uhash
## INFO [20:51:28.599] [bbotk] fe9e4732-a8d0-4168-a92f-5685a0eda1da
## INFO [20:51:28.600] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.607] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.610] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.625] [mlr3] Finished benchmark
## INFO [20:51:28.643] [bbotk] Result of batch 6:
## INFO [20:51:28.644] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.644] [bbotk] 0.02575 5 0.2130435 0 0 0.02
## INFO [20:51:28.644] [bbotk] uhash
## INFO [20:51:28.644] [bbotk] c0fc6e51-2a52-481d-b227-43c8fcf6567c
## INFO [20:51:28.645] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.652] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.655] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.667] [mlr3] Finished benchmark
## INFO [20:51:28.686] [bbotk] Result of batch 7:
## INFO [20:51:28.687] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.687] [bbotk] 0.1 8 0.2130435 0 0 0.01
## INFO [20:51:28.687] [bbotk] uhash
## INFO [20:51:28.687] [bbotk] 3941b23d-30db-4ef7-b92e-c64239c87a06
## INFO [20:51:28.688] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.695] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.698] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.710] [mlr3] Finished benchmark
## INFO [20:51:28.732] [bbotk] Result of batch 8:
## INFO [20:51:28.733] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.733] [bbotk] 0.001 8 0.2695652 0 0 0
## INFO [20:51:28.733] [bbotk] uhash
## INFO [20:51:28.733] [bbotk] fc6eed5f-55ed-4947-844b-819f850b7993
## INFO [20:51:28.734] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.740] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.743] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.756] [mlr3] Finished benchmark
## INFO [20:51:28.774] [bbotk] Result of batch 9:
## INFO [20:51:28.775] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.775] [bbotk] 0.001 5 0.2782609 0 0 0
## INFO [20:51:28.775] [bbotk] uhash
## INFO [20:51:28.775] [bbotk] a3e4cc2a-4482-40b7-a49a-4a67cbd2ee46
## INFO [20:51:28.776] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.783] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.786] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.797] [mlr3] Finished benchmark
## INFO [20:51:28.816] [bbotk] Result of batch 10:
## INFO [20:51:28.817] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.817] [bbotk] 0.1 5 0.2130435 0 0 0.02
## INFO [20:51:28.817] [bbotk] uhash
## INFO [20:51:28.817] [bbotk] c29929fb-44a8-4057-8415-44beb946bfb2
## INFO [20:51:28.818] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.825] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.828] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.842] [mlr3] Finished benchmark
## INFO [20:51:28.861] [bbotk] Result of batch 11:
## INFO [20:51:28.862] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.862] [bbotk] 0.07525 10 0.2130435 0 0 0
## INFO [20:51:28.862] [bbotk] uhash
## INFO [20:51:28.862] [bbotk] ef591734-880d-453d-8d92-48a509362e0b
## INFO [20:51:28.862] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.869] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.872] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.884] [mlr3] Finished benchmark
## INFO [20:51:28.903] [bbotk] Result of batch 12:
## INFO [20:51:28.904] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.904] [bbotk] 0.0505 10 0.2130435 0 0 0.01
## INFO [20:51:28.904] [bbotk] uhash
## INFO [20:51:28.904] [bbotk] 362f322a-e033-42f8-9eba-7e1585668661
## INFO [20:51:28.905] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:28.911] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:28.914] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:28.926] [mlr3] Finished benchmark
## INFO [20:51:28.945] [bbotk] Result of batch 13:
## INFO [20:51:28.946] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:28.946] [bbotk] 0.07525 5 0.2130435 0 0 0
## INFO [20:51:28.946] [bbotk] uhash
## INFO [20:51:28.946] [bbotk] f08882f8-939f-42e6-b6ab-fb3b27389601
## INFO [20:51:28.947] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.010] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.013] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.026] [mlr3] Finished benchmark
## INFO [20:51:29.045] [bbotk] Result of batch 14:
## INFO [20:51:29.046] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.046] [bbotk] 0.0505 8 0.2130435 0 0 0.01
## INFO [20:51:29.046] [bbotk] uhash
## INFO [20:51:29.046] [bbotk] e66cd07f-6f47-4a5b-9aeb-590a5ddbdc58
## INFO [20:51:29.047] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.054] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.057] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.069] [mlr3] Finished benchmark
## INFO [20:51:29.088] [bbotk] Result of batch 15:
## INFO [20:51:29.089] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.089] [bbotk] 0.001 10 0.2608696 0 0 0
## INFO [20:51:29.089] [bbotk] uhash
## INFO [20:51:29.089] [bbotk] 0fe38ee8-7989-4831-b1cb-136de8877961
## INFO [20:51:29.090] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.097] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.100] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.112] [mlr3] Finished benchmark
## INFO [20:51:29.133] [bbotk] Result of batch 16:
## INFO [20:51:29.134] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.134] [bbotk] 0.0505 3 0.2130435 0 0 0
## INFO [20:51:29.134] [bbotk] uhash
## INFO [20:51:29.134] [bbotk] 14add2e3-3f68-45c8-80e1-5ef4ba2d44e5
## INFO [20:51:29.135] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.142] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.145] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.157] [mlr3] Finished benchmark
## INFO [20:51:29.176] [bbotk] Result of batch 17:
## INFO [20:51:29.177] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.177] [bbotk] 0.0505 1 0.2130435 0 0 0.02
## INFO [20:51:29.177] [bbotk] uhash
## INFO [20:51:29.177] [bbotk] 4c4c7ce3-c6e3-4ee5-9f3e-91e5afa6bd04
## INFO [20:51:29.177] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.184] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.187] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.199] [mlr3] Finished benchmark
## INFO [20:51:29.217] [bbotk] Result of batch 18:
## INFO [20:51:29.218] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.218] [bbotk] 0.1 10 0.2130435 0 0 0
## INFO [20:51:29.218] [bbotk] uhash
## INFO [20:51:29.218] [bbotk] 2f752bae-5fa7-4c0f-9e5b-2b5b23f126c5
## INFO [20:51:29.219] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.225] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.229] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.244] [mlr3] Finished benchmark
## INFO [20:51:29.262] [bbotk] Result of batch 19:
## INFO [20:51:29.263] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.263] [bbotk] 0.07525 1 0.2130435 0 0 0.01
## INFO [20:51:29.263] [bbotk] uhash
## INFO [20:51:29.263] [bbotk] e7eba34c-b7f6-4b72-8cfd-e96c970e863b
## INFO [20:51:29.264] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.271] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.274] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.286] [mlr3] Finished benchmark
## INFO [20:51:29.305] [bbotk] Result of batch 20:
## INFO [20:51:29.306] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.306] [bbotk] 0.02575 1 0.2130435 0 0 0
## INFO [20:51:29.306] [bbotk] uhash
## INFO [20:51:29.306] [bbotk] 58eb421d-f0ed-4246-8430-3c1832ae615c
## INFO [20:51:29.309] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [20:51:29.310] [bbotk] Result:
## INFO [20:51:29.310] [bbotk] cp minsplit learner_param_vals x_domain classif.ce
## INFO [20:51:29.310] [bbotk] 0.02575 3 <list[3]> <list[2]> 0.2130435
## cp minsplit learner_param_vals x_domain classif.ce
## 1: 0.02575 3 <list[3]> <list[2]> 0.2130435
查看调整好的超参数：
instance$result_learner_param_vals
## $xval
## [1] 0
##
## $cp
## [1] 0.02575
##
## $minsplit
## [1] 3
查看模型性能：
instance$result_y
## classif.ce
## 0.2130435
查看每一次迭代的结果，只有20个：
instance$archive
## <ArchiveTuning>
## cp minsplit classif.ce runtime_learners timestamp batch_nr
## 1: 0.026 3 0.21 0.02 2022-02-27 20:51:28 1
## 2: 0.075 8 0.21 0.00 2022-02-27 20:51:28 2
## 3: 0.050 5 0.21 0.00 2022-02-27 20:51:28 3
## 4: 0.001 1 0.30 0.00 2022-02-27 20:51:28 4
## 5: 0.100 3 0.21 0.02 2022-02-27 20:51:28 5
## 6: 0.026 5 0.21 0.02 2022-02-27 20:51:28 6
## 7: 0.100 8 0.21 0.01 2022-02-27 20:51:28 7
## 8: 0.001 8 0.27 0.00 2022-02-27 20:51:28 8
## 9: 0.001 5 0.28 0.00 2022-02-27 20:51:28 9
## 10: 0.100 5 0.21 0.02 2022-02-27 20:51:28 10
## 11: 0.075 10 0.21 0.00 2022-02-27 20:51:28 11
## 12: 0.050 10 0.21 0.01 2022-02-27 20:51:28 12
## 13: 0.075 5 0.21 0.00 2022-02-27 20:51:28 13
## 14: 0.050 8 0.21 0.01 2022-02-27 20:51:29 14
## 15: 0.001 10 0.26 0.00 2022-02-27 20:51:29 15
## 16: 0.050 3 0.21 0.00 2022-02-27 20:51:29 16
## 17: 0.050 1 0.21 0.02 2022-02-27 20:51:29 17
## 18: 0.100 10 0.21 0.00 2022-02-27 20:51:29 18
## 19: 0.075 1 0.21 0.01 2022-02-27 20:51:29 19
## 20: 0.026 1 0.21 0.00 2022-02-27 20:51:29 20
## warnings errors resample_result
## 1: 0 0 <ResampleResult[22]>
## 2: 0 0 <ResampleResult[22]>
## 3: 0 0 <ResampleResult[22]>
## 4: 0 0 <ResampleResult[22]>
## 5: 0 0 <ResampleResult[22]>
## 6: 0 0 <ResampleResult[22]>
## 7: 0 0 <ResampleResult[22]>
## 8: 0 0 <ResampleResult[22]>
## 9: 0 0 <ResampleResult[22]>
## 10: 0 0 <ResampleResult[22]>
## 11: 0 0 <ResampleResult[22]>
## 12: 0 0 <ResampleResult[22]>
## 13: 0 0 <ResampleResult[22]>
## 14: 0 0 <ResampleResult[22]>
## 15: 0 0 <ResampleResult[22]>
## 16: 0 0 <ResampleResult[22]>
## 17: 0 0 <ResampleResult[22]>
## 18: 0 0 <ResampleResult[22]>
## 19: 0 0 <ResampleResult[22]>
## 20: 0 0 <ResampleResult[22]>
接下来就可以把训练好的超参数应用于模型，重新应用于数据：
learner$param_set$values <- instance$result_learner_param_vals
learner$train(task)
这个训练好的模型就可以用于预测了，使用learner$predict()即可！
以上步骤写起来有些复杂，与tidymodels相比不够简洁好理解，我刚开始学习的时候经常记不住，后来版本更新后终于有了简便写法：
instance <- tune(
task = task,
learner = learner,
resampling = hout,
measure = measure,
search_space = search_space,
method = "grid_search",
resolution = 5,
term_evals = 25
)
## INFO [20:51:29.402] [bbotk] Starting to optimize 2 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=25, k=0]'
## INFO [20:51:29.403] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.411] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.415] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.427] [mlr3] Finished benchmark
## INFO [20:51:29.443] [bbotk] Result of batch 1:
## INFO [20:51:29.444] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.444] [bbotk] 0.07525 10 0.2956522 0 0 0
## INFO [20:51:29.444] [bbotk] uhash
## INFO [20:51:29.444] [bbotk] 414742f4-dcdd-42de-aebe-39ec74e0fd03
## INFO [20:51:29.445] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.453] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.456] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.468] [mlr3] Finished benchmark
## INFO [20:51:29.490] [bbotk] Result of batch 2:
## INFO [20:51:29.491] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.491] [bbotk] 0.02575 10 0.2347826 0 0 0.02
## INFO [20:51:29.491] [bbotk] uhash
## INFO [20:51:29.491] [bbotk] 62ab2f99-3f3e-4653-8cf5-8b1a0264ccac
## INFO [20:51:29.492] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.499] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.503] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.515] [mlr3] Finished benchmark
## INFO [20:51:29.533] [bbotk] Result of batch 3:
## INFO [20:51:29.534] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.534] [bbotk] 0.1 8 0.2956522 0 0 0
## INFO [20:51:29.534] [bbotk] uhash
## INFO [20:51:29.534] [bbotk] 3ad78954-c9d9-4bae-8296-2be9ee120a20
## INFO [20:51:29.535] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.542] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.545] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.557] [mlr3] Finished benchmark
## INFO [20:51:29.576] [bbotk] Result of batch 4:
## INFO [20:51:29.577] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.577] [bbotk] 0.07525 5 0.2956522 0 0 0.01
## INFO [20:51:29.577] [bbotk] uhash
## INFO [20:51:29.577] [bbotk] b7c1b381-5d6a-4db9-a5f0-92e75d919007
## INFO [20:51:29.578] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.589] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.592] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.604] [mlr3] Finished benchmark
## INFO [20:51:29.623] [bbotk] Result of batch 5:
## INFO [20:51:29.624] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.624] [bbotk] 0.0505 1 0.2347826 0 0 0.01
## INFO [20:51:29.624] [bbotk] uhash
## INFO [20:51:29.624] [bbotk] 6d96a811-3321-4ddd-a8bd-f9f90355e7d3
## INFO [20:51:29.625] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.632] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.635] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.647] [mlr3] Finished benchmark
## INFO [20:51:29.666] [bbotk] Result of batch 6:
## INFO [20:51:29.667] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.667] [bbotk] 0.02575 8 0.2347826 0 0 0
## INFO [20:51:29.667] [bbotk] uhash
## INFO [20:51:29.667] [bbotk] 60a4ddca-8b03-4d84-a25b-f756747cf7ed
## INFO [20:51:29.668] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.675] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.678] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.694] [mlr3] Finished benchmark
## INFO [20:51:29.713] [bbotk] Result of batch 7:
## INFO [20:51:29.713] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.713] [bbotk] 0.02575 5 0.2347826 0 0 0
## INFO [20:51:29.713] [bbotk] uhash
## INFO [20:51:29.713] [bbotk] 09a8814f-1469-43ea-902e-57426c9d2bec
## INFO [20:51:29.714] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.722] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.725] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.738] [mlr3] Finished benchmark
## INFO [20:51:29.757] [bbotk] Result of batch 8:
## INFO [20:51:29.758] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.758] [bbotk] 0.001 10 0.2695652 0 0 0
## INFO [20:51:29.758] [bbotk] uhash
## INFO [20:51:29.758] [bbotk] 170ef854-40c1-4ff1-b1e7-18bf9909c194
## INFO [20:51:29.758] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.766] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.769] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.781] [mlr3] Finished benchmark
## INFO [20:51:29.800] [bbotk] Result of batch 9:
## INFO [20:51:29.801] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.801] [bbotk] 0.0505 3 0.2347826 0 0 0.01
## INFO [20:51:29.801] [bbotk] uhash
## INFO [20:51:29.801] [bbotk] 50df53a8-a638-4ab9-b019-45a10c0b8c8c
## INFO [20:51:29.805] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.813] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.816] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.828] [mlr3] Finished benchmark
## INFO [20:51:29.847] [bbotk] Result of batch 10:
## INFO [20:51:29.848] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.848] [bbotk] 0.07525 1 0.2956522 0 0 0.02
## INFO [20:51:29.848] [bbotk] uhash
## INFO [20:51:29.848] [bbotk] 54d10423-0a65-4466-a171-6c953adccb8e
## INFO [20:51:29.849] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.856] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.859] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.871] [mlr3] Finished benchmark
## INFO [20:51:29.890] [bbotk] Result of batch 11:
## INFO [20:51:29.891] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.891] [bbotk] 0.0505 10 0.2347826 0 0 0
## INFO [20:51:29.891] [bbotk] uhash
## INFO [20:51:29.891] [bbotk] e948aaa1-2f71-4887-9798-0d6d3e5704a6
## INFO [20:51:29.892] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.900] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.903] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.918] [mlr3] Finished benchmark
## INFO [20:51:29.937] [bbotk] Result of batch 12:
## INFO [20:51:29.938] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.938] [bbotk] 0.0505 5 0.2347826 0 0 0
## INFO [20:51:29.938] [bbotk] uhash
## INFO [20:51:29.938] [bbotk] a03e64ae-cbf2-4aa8-965e-f89f8d940ae1
## INFO [20:51:29.938] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.946] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.949] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:29.961] [mlr3] Finished benchmark
## INFO [20:51:29.981] [bbotk] Result of batch 13:
## INFO [20:51:29.982] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:29.982] [bbotk] 0.1 10 0.2956522 0 0 0.02
## INFO [20:51:29.982] [bbotk] uhash
## INFO [20:51:29.982] [bbotk] d254cd1a-1887-45f5-a6e2-acc0ec58e87e
## INFO [20:51:29.982] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:29.990] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:29.993] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.006] [mlr3] Finished benchmark
## INFO [20:51:30.029] [bbotk] Result of batch 14:
## INFO [20:51:30.030] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.030] [bbotk] 0.001 1 0.3391304 0 0 0
## INFO [20:51:30.030] [bbotk] uhash
## INFO [20:51:30.030] [bbotk] 81c14e1c-9fe1-48fc-9e85-e66b127958a5
## INFO [20:51:30.030] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.038] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.041] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.053] [mlr3] Finished benchmark
## INFO [20:51:30.072] [bbotk] Result of batch 15:
## INFO [20:51:30.073] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.073] [bbotk] 0.02575 3 0.2347826 0 0 0
## INFO [20:51:30.073] [bbotk] uhash
## INFO [20:51:30.073] [bbotk] a4df42bc-0586-41d1-b24c-0ee017f46849
## INFO [20:51:30.074] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.082] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.085] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.097] [mlr3] Finished benchmark
## INFO [20:51:30.116] [bbotk] Result of batch 16:
## INFO [20:51:30.117] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.117] [bbotk] 0.1 1 0.2956522 0 0 0
## INFO [20:51:30.117] [bbotk] uhash
## INFO [20:51:30.117] [bbotk] 33c28e2d-dd0f-440b-89a0-3c81edb9068f
## INFO [20:51:30.118] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.125] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.132] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.145] [mlr3] Finished benchmark
## INFO [20:51:30.163] [bbotk] Result of batch 17:
## INFO [20:51:30.164] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.164] [bbotk] 0.001 3 0.3434783 0 0 0
## INFO [20:51:30.164] [bbotk] uhash
## INFO [20:51:30.164] [bbotk] 366c572b-4157-4d7f-86f7-8ecdfa9e5e95
## INFO [20:51:30.165] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.173] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.176] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.188] [mlr3] Finished benchmark
## INFO [20:51:30.207] [bbotk] Result of batch 18:
## INFO [20:51:30.208] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.208] [bbotk] 0.07525 8 0.2956522 0 0 0.02
## INFO [20:51:30.208] [bbotk] uhash
## INFO [20:51:30.208] [bbotk] 7b0aeb6f-b465-4675-8480-f8b3b0d325b7
## INFO [20:51:30.209] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.217] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.220] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.232] [mlr3] Finished benchmark
## INFO [20:51:30.255] [bbotk] Result of batch 19:
## INFO [20:51:30.256] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.256] [bbotk] 0.1 3 0.2956522 0 0 0
## INFO [20:51:30.256] [bbotk] uhash
## INFO [20:51:30.256] [bbotk] 5aaf6481-da75-4cf0-a8d7-9e9579c0ecaf
## INFO [20:51:30.257] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.264] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.268] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.280] [mlr3] Finished benchmark
## INFO [20:51:30.300] [bbotk] Result of batch 20:
## INFO [20:51:30.301] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.301] [bbotk] 0.001 8 0.2782609 0 0 0.01
## INFO [20:51:30.301] [bbotk] uhash
## INFO [20:51:30.301] [bbotk] c861abc0-2cb2-4ed8-9a75-73eeec3c2d70
## INFO [20:51:30.302] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.309] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.313] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.325] [mlr3] Finished benchmark
## INFO [20:51:30.345] [bbotk] Result of batch 21:
## INFO [20:51:30.346] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.346] [bbotk] 0.07525 3 0.2956522 0 0 0
## INFO [20:51:30.346] [bbotk] uhash
## INFO [20:51:30.346] [bbotk] 27b35d46-e907-449b-affb-b1d2f2a4ea83
## INFO [20:51:30.347] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.358] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.361] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.374] [mlr3] Finished benchmark
## INFO [20:51:30.394] [bbotk] Result of batch 22:
## INFO [20:51:30.395] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.395] [bbotk] 0.001 5 0.2869565 0 0 0.02
## INFO [20:51:30.395] [bbotk] uhash
## INFO [20:51:30.395] [bbotk] b3e6b2a6-4b9c-4e1d-b194-3c59d753e27b
## INFO [20:51:30.395] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.403] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.406] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.418] [mlr3] Finished benchmark
## INFO [20:51:30.438] [bbotk] Result of batch 23:
## INFO [20:51:30.439] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.439] [bbotk] 0.02575 1 0.2347826 0 0 0
## INFO [20:51:30.439] [bbotk] uhash
## INFO [20:51:30.439] [bbotk] 5b05beb1-7149-4f18-a360-016197e143bf
## INFO [20:51:30.440] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.448] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.451] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.467] [mlr3] Finished benchmark
## INFO [20:51:30.486] [bbotk] Result of batch 24:
## INFO [20:51:30.487] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.487] [bbotk] 0.0505 8 0.2347826 0 0 0
## INFO [20:51:30.487] [bbotk] uhash
## INFO [20:51:30.487] [bbotk] 48c09c49-824d-44fc-8da4-22f439f871b3
## INFO [20:51:30.487] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.495] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.499] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.511] [mlr3] Finished benchmark
## INFO [20:51:30.530] [bbotk] Result of batch 25:
## INFO [20:51:30.531] [bbotk] cp minsplit classif.ce warnings errors runtime_learners
## INFO [20:51:30.531] [bbotk] 0.1 5 0.2956522 0 0 0.02
## INFO [20:51:30.531] [bbotk] uhash
## INFO [20:51:30.531] [bbotk] 8260cb91-8e11-4e47-8818-ec08d0e289ba
## INFO [20:51:30.534] [bbotk] Finished optimizing after 25 evaluation(s)
## INFO [20:51:30.534] [bbotk] Result:
## INFO [20:51:30.535] [bbotk] cp minsplit learner_param_vals x_domain classif.ce
## INFO [20:51:30.535] [bbotk] 0.02575 10 <list[3]> <list[2]> 0.2347826
instance$result_learner_param_vals
## $xval
## [1] 0
##
## $cp
## [1] 0.02575
##
## $minsplit
## [1] 10
instance$result_y
## classif.ce
## 0.2347826
learner$param_set$values <- instance$result_learner_param_vals
learner$train(task)
mlr3也支持同时设定多个性能指标：
measures <- msrs(c("classif.ce","time_train")) # 设定多个评价指标
evals20 <- trm("evals", n_evals = 20)
instance <- TuningInstanceMultiCrit$new(
task = task,
learner = learner,
resampling = hout,
measures = measures,
search_space = search_space,
terminator = evals20
)
tuner$optimize(instance)
## INFO [20:51:30.595] [bbotk] Starting to optimize 2 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [20:51:30.597] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.605] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.608] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.620] [mlr3] Finished benchmark
## INFO [20:51:30.642] [bbotk] Result of batch 1:
## INFO [20:51:30.643] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:30.643] [bbotk] 0.0505 1 0.2347826 0 0 0 0.02
## INFO [20:51:30.643] [bbotk] uhash
## INFO [20:51:30.643] [bbotk] 81d97917-9255-4229-9bd1-035c772b5580
## INFO [20:51:30.644] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.651] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.655] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.670] [mlr3] Finished benchmark
## INFO [20:51:30.694] [bbotk] Result of batch 2:
## INFO [20:51:30.695] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:30.695] [bbotk] 0.07525 1 0.2347826 0 0 0 0.01
## INFO [20:51:30.695] [bbotk] uhash
## INFO [20:51:30.695] [bbotk] 5252f6ce-07df-4f30-9661-2dc60a3f1d78
## INFO [20:51:30.696] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.704] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.707] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.719] [mlr3] Finished benchmark
## INFO [20:51:30.744] [bbotk] Result of batch 3:
## INFO [20:51:30.745] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:30.745] [bbotk] 0.07525 10 0.2347826 0 0 0 0.02
## INFO [20:51:30.745] [bbotk] uhash
## INFO [20:51:30.745] [bbotk] f3ebba00-dc40-4e29-8740-14ca886a63bd
## INFO [20:51:30.745] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.753] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.756] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.768] [mlr3] Finished benchmark
## INFO [20:51:30.796] [bbotk] Result of batch 4:
## INFO [20:51:30.797] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:30.797] [bbotk] 0.1 8 0.2347826 0 0 0 0
## INFO [20:51:30.797] [bbotk] uhash
## INFO [20:51:30.797] [bbotk] 84a5a7d4-5946-4899-ab08-18ca4a63f8f3
## INFO [20:51:30.797] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.805] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.808] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.821] [mlr3] Finished benchmark
## INFO [20:51:30.845] [bbotk] Result of batch 5:
## INFO [20:51:30.846] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:30.846] [bbotk] 0.02575 3 0.2347826 0 0 0 0.02
## INFO [20:51:30.846] [bbotk] uhash
## INFO [20:51:30.846] [bbotk] ceee10f8-a4f1-4244-9ea7-c2bb6438970a
## INFO [20:51:30.847] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.854] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.857] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.870] [mlr3] Finished benchmark
## INFO [20:51:30.897] [bbotk] Result of batch 6:
## INFO [20:51:30.898] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:30.898] [bbotk] 0.07525 8 0.2347826 0 0 0 0.02
## INFO [20:51:30.898] [bbotk] uhash
## INFO [20:51:30.898] [bbotk] 41667418-fd16-4dd0-9811-f5f3f2b8b8d2
## INFO [20:51:30.899] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.906] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.910] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.922] [mlr3] Finished benchmark
## INFO [20:51:30.946] [bbotk] Result of batch 7:
## INFO [20:51:30.947] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:30.947] [bbotk] 0.1 3 0.2347826 0 0 0 0.01
## INFO [20:51:30.947] [bbotk] uhash
## INFO [20:51:30.947] [bbotk] 3c6ec6d1-2e27-4b38-a73a-9ad774128801
## INFO [20:51:30.948] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:30.955] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:30.959] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:30.971] [mlr3] Finished benchmark
## INFO [20:51:31.008] [bbotk] Result of batch 8:
## INFO [20:51:31.009] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.009] [bbotk] 0.1 5 0.2347826 0 0 0 0
## INFO [20:51:31.009] [bbotk] uhash
## INFO [20:51:31.009] [bbotk] 72a0b254-705c-47d2-8215-8f97d79da14b
## INFO [20:51:31.010] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.017] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.020] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.033] [mlr3] Finished benchmark
## INFO [20:51:31.057] [bbotk] Result of batch 9:
## INFO [20:51:31.058] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.058] [bbotk] 0.02575 5 0.2347826 0 0 0 0.01
## INFO [20:51:31.058] [bbotk] uhash
## INFO [20:51:31.058] [bbotk] c65515ce-4f05-4a88-88f8-2cd252efdc09
## INFO [20:51:31.059] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.067] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.070] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.083] [mlr3] Finished benchmark
## INFO [20:51:31.107] [bbotk] Result of batch 10:
## INFO [20:51:31.108] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.108] [bbotk] 0.001 5 0.2695652 0 0 0 0
## INFO [20:51:31.108] [bbotk] uhash
## INFO [20:51:31.108] [bbotk] a6e8ea84-4bbb-4ee8-94f1-b69fe2b80a45
## INFO [20:51:31.109] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.122] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.125] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.138] [mlr3] Finished benchmark
## INFO [20:51:31.162] [bbotk] Result of batch 11:
## INFO [20:51:31.163] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.163] [bbotk] 0.001 3 0.2782609 0 0 0 0
## INFO [20:51:31.163] [bbotk] uhash
## INFO [20:51:31.163] [bbotk] d85e0607-d82c-472d-b9b4-16f40a0a6afd
## INFO [20:51:31.164] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.171] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.175] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.186] [mlr3] Finished benchmark
## INFO [20:51:31.210] [bbotk] Result of batch 12:
## INFO [20:51:31.211] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.211] [bbotk] 0.07525 5 0.2347826 0 0 0 0.02
## INFO [20:51:31.211] [bbotk] uhash
## INFO [20:51:31.211] [bbotk] 63511c99-3a81-45b6-9d1e-c8c102b0a9f2
## INFO [20:51:31.212] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.220] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.223] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.241] [mlr3] Finished benchmark
## INFO [20:51:31.264] [bbotk] Result of batch 13:
## INFO [20:51:31.265] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.265] [bbotk] 0.0505 8 0.2347826 0 0 0 0
## INFO [20:51:31.265] [bbotk] uhash
## INFO [20:51:31.265] [bbotk] 130f3856-bfd0-4292-b635-fc80d4d2af49
## INFO [20:51:31.266] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.274] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.277] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.290] [mlr3] Finished benchmark
## INFO [20:51:31.314] [bbotk] Result of batch 14:
## INFO [20:51:31.315] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.315] [bbotk] 0.001 10 0.2434783 0 0 0 0.02
## INFO [20:51:31.315] [bbotk] uhash
## INFO [20:51:31.315] [bbotk] a3ec5b97-44f5-4315-8faf-859ef886f303
## INFO [20:51:31.315] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.323] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.326] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.338] [mlr3] Finished benchmark
## INFO [20:51:31.368] [bbotk] Result of batch 15:
## INFO [20:51:31.369] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.369] [bbotk] 0.0505 3 0.2347826 0 0 0 0.02
## INFO [20:51:31.369] [bbotk] uhash
## INFO [20:51:31.369] [bbotk] d3b6df82-3fc0-43c2-aa29-f26bb6a83586
## INFO [20:51:31.370] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.377] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.381] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.393] [mlr3] Finished benchmark
## INFO [20:51:31.417] [bbotk] Result of batch 16:
## INFO [20:51:31.418] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.418] [bbotk] 0.07525 3 0.2347826 0 0 0 0
## INFO [20:51:31.418] [bbotk] uhash
## INFO [20:51:31.418] [bbotk] 49c4f9fb-3f24-4ae2-9b19-e9e1a1cff98a
## INFO [20:51:31.419] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.426] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.429] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.442] [mlr3] Finished benchmark
## INFO [20:51:31.619] [bbotk] Result of batch 17:
## INFO [20:51:31.620] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.620] [bbotk] 0.001 8 0.2434783 0 0 0 0
## INFO [20:51:31.620] [bbotk] uhash
## INFO [20:51:31.620] [bbotk] 04906760-4c96-4725-a357-49389b61bb65
## INFO [20:51:31.621] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.629] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.632] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.644] [mlr3] Finished benchmark
## INFO [20:51:31.667] [bbotk] Result of batch 18:
## INFO [20:51:31.668] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.668] [bbotk] 0.0505 5 0.2347826 0 0 0 0
## INFO [20:51:31.668] [bbotk] uhash
## INFO [20:51:31.668] [bbotk] 966fc33e-eb74-4395-9f0d-3a20cfadbe50
## INFO [20:51:31.669] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.676] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.680] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.691] [mlr3] Finished benchmark
## INFO [20:51:31.715] [bbotk] Result of batch 19:
## INFO [20:51:31.715] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.715] [bbotk] 0.02575 1 0.2347826 0 0 0 0
## INFO [20:51:31.715] [bbotk] uhash
## INFO [20:51:31.715] [bbotk] 14930e19-9909-4518-ad70-fa231fd174ce
## INFO [20:51:31.716] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.724] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.727] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.739] [mlr3] Finished benchmark
## INFO [20:51:31.763] [bbotk] Result of batch 20:
## INFO [20:51:31.764] [bbotk] cp minsplit classif.ce time_train warnings errors runtime_learners
## INFO [20:51:31.764] [bbotk] 0.001 1 0.273913 0 0 0 0
## INFO [20:51:31.764] [bbotk] uhash
## INFO [20:51:31.764] [bbotk] 2d461519-627e-4694-bf12-aa689a1164b5
## INFO [20:51:31.768] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [20:51:31.768] [bbotk] Result:
## INFO [20:51:31.769] [bbotk] cp minsplit learner_param_vals x_domain classif.ce time_train
## INFO [20:51:31.769] [bbotk] 0.05050 1 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.07525 1 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.07525 10 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.10000 8 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.02575 3 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.07525 8 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.10000 3 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.10000 5 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.02575 5 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.07525 5 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.05050 8 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.05050 3 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.07525 3 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.05050 5 <list[3]> <list[2]> 0.2347826 0
## INFO [20:51:31.769] [bbotk] 0.02575 1 <list[3]> <list[2]> 0.2347826 0
## cp minsplit learner_param_vals x_domain classif.ce time_train
## 1: 0.05050 1 <list[3]> <list[2]> 0.2347826 0
## 2: 0.07525 1 <list[3]> <list[2]> 0.2347826 0
## 3: 0.07525 10 <list[3]> <list[2]> 0.2347826 0
## 4: 0.10000 8 <list[3]> <list[2]> 0.2347826 0
## 5: 0.02575 3 <list[3]> <list[2]> 0.2347826 0
## 6: 0.07525 8 <list[3]> <list[2]> 0.2347826 0
## 7: 0.10000 3 <list[3]> <list[2]> 0.2347826 0
## 8: 0.10000 5 <list[3]> <list[2]> 0.2347826 0
## 9: 0.02575 5 <list[3]> <list[2]> 0.2347826 0
## 10: 0.07525 5 <list[3]> <list[2]> 0.2347826 0
## 11: 0.05050 8 <list[3]> <list[2]> 0.2347826 0
## 12: 0.05050 3 <list[3]> <list[2]> 0.2347826 0
## 13: 0.07525 3 <list[3]> <list[2]> 0.2347826 0
## 14: 0.05050 5 <list[3]> <list[2]> 0.2347826 0
## 15: 0.02575 1 <list[3]> <list[2]> 0.2347826 0
查看结果：
instance$result_learner_param_vals
## [[1]]
## [[1]]$xval
## [1] 0
##
## [[1]]$cp
## [1] 0.0505
##
## [[1]]$minsplit
## [1] 1
##
##
## [[2]]
## [[2]]$xval
## [1] 0
##
## [[2]]$cp
## [1] 0.07525
##
## [[2]]$minsplit
## [1] 1
##
##
## [[3]]
## [[3]]$xval
## [1] 0
##
## [[3]]$cp
## [1] 0.07525
##
## [[3]]$minsplit
## [1] 10
##
##
## [[4]]
## [[4]]$xval
## [1] 0
##
## [[4]]$cp
## [1] 0.1
##
## [[4]]$minsplit
## [1] 8
##
##
## [[5]]
## [[5]]$xval
## [1] 0
##
## [[5]]$cp
## [1] 0.02575
##
## [[5]]$minsplit
## [1] 3
##
##
## [[6]]
## [[6]]$xval
## [1] 0
##
## [[6]]$cp
## [1] 0.07525
##
## [[6]]$minsplit
## [1] 8
##
##
## [[7]]
## [[7]]$xval
## [1] 0
##
## [[7]]$cp
## [1] 0.1
##
## [[7]]$minsplit
## [1] 3
##
##
## [[8]]
## [[8]]$xval
## [1] 0
##
## [[8]]$cp
## [1] 0.1
##
## [[8]]$minsplit
## [1] 5
##
##
## [[9]]
## [[9]]$xval
## [1] 0
##
## [[9]]$cp
## [1] 0.02575
##
## [[9]]$minsplit
## [1] 5
##
##
## [[10]]
## [[10]]$xval
## [1] 0
##
## [[10]]$cp
## [1] 0.07525
##
## [[10]]$minsplit
## [1] 5
##
##
## [[11]]
## [[11]]$xval
## [1] 0
##
## [[11]]$cp
## [1] 0.0505
##
## [[11]]$minsplit
## [1] 8
##
##
## [[12]]
## [[12]]$xval
## [1] 0
##
## [[12]]$cp
## [1] 0.0505
##
## [[12]]$minsplit
## [1] 3
##
##
## [[13]]
## [[13]]$xval
## [1] 0
##
## [[13]]$cp
## [1] 0.07525
##
## [[13]]$minsplit
## [1] 3
##
##
## [[14]]
## [[14]]$xval
## [1] 0
##
## [[14]]$cp
## [1] 0.0505
##
## [[14]]$minsplit
## [1] 5
##
##
## [[15]]
## [[15]]$xval
## [1] 0
##
## [[15]]$cp
## [1] 0.02575
##
## [[15]]$minsplit
## [1] 1
instance$rusult_y
## NULL
以上就是第一种方法，接下来介绍第二种方法。
方法二：通过autotuner训练模型
这种方式方法把调整参数、将调整好的参数应用于模型放到一起了，但是也需要提前设定好各种需要的参数。
task <- tsk("pima") # 创建任务
leanrer <- lrn("classif.rpart") # 选择学习器
search_space <- ps(
cp = p_dbl(0.001, 0.1),
minsplit = p_int(1,10)
) # 设定搜索范围
terminator <- trm("evals", n_evals = 10) # 设定停止标志
tuner <- tnr("random_search") # 选择搜索方法
resampling <- rsmp("holdout") # 选择重抽样方法
measure <- msr("classif.acc") # 选择评价指标
# 训练
at <- AutoTuner$new(
learner = learner,
resampling = resampling,
search_space = search_space,
measure = measure,
tuner = tuner,
terminator = terminator
)
自动选择最优参数并作用于数据：
at$train(task)
## INFO [20:51:31.873] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=10, k=0]'
## INFO [20:51:31.882] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.891] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.895] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.908] [mlr3] Finished benchmark
## INFO [20:51:31.926] [bbotk] Result of batch 1:
## INFO [20:51:31.927] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:31.927] [bbotk] 0.02278977 3 0.7695312 0 0 0.02
## INFO [20:51:31.927] [bbotk] uhash
## INFO [20:51:31.927] [bbotk] 3f90df2e-bc2d-4829-be96-65f617caad41
## INFO [20:51:31.929] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.941] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.945] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.956] [mlr3] Finished benchmark
## INFO [20:51:31.975] [bbotk] Result of batch 2:
## INFO [20:51:31.976] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:31.976] [bbotk] 0.05287745 5 0.7382812 0 0 0
## INFO [20:51:31.976] [bbotk] uhash
## INFO [20:51:31.976] [bbotk] 4112d036-8de7-4856-83d5-6730313376f3
## INFO [20:51:31.977] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:31.985] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:31.988] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:31.999] [mlr3] Finished benchmark
## INFO [20:51:32.017] [bbotk] Result of batch 3:
## INFO [20:51:32.018] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.018] [bbotk] 0.03784078 9 0.7382812 0 0 0.01
## INFO [20:51:32.018] [bbotk] uhash
## INFO [20:51:32.018] [bbotk] 20022363-818b-4b22-960b-fe5335cdcc72
## INFO [20:51:32.020] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.028] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.031] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.042] [mlr3] Finished benchmark
## INFO [20:51:32.061] [bbotk] Result of batch 4:
## INFO [20:51:32.062] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.062] [bbotk] 0.05691762 2 0.7382812 0 0 0.02
## INFO [20:51:32.062] [bbotk] uhash
## INFO [20:51:32.062] [bbotk] 37043db7-8dcb-44f0-910f-4af97015e3e6
## INFO [20:51:32.064] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.071] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.074] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.086] [mlr3] Finished benchmark
## INFO [20:51:32.105] [bbotk] Result of batch 5:
## INFO [20:51:32.106] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.106] [bbotk] 0.06826047 1 0.7382812 0 0 0.02
## INFO [20:51:32.106] [bbotk] uhash
## INFO [20:51:32.106] [bbotk] dff3724e-8121-4639-95b2-0559a9264529
## INFO [20:51:32.108] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.115] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.119] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.131] [mlr3] Finished benchmark
## INFO [20:51:32.150] [bbotk] Result of batch 6:
## INFO [20:51:32.151] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.151] [bbotk] 0.003273522 1 0.7148438 0 0 0
## INFO [20:51:32.151] [bbotk] uhash
## INFO [20:51:32.151] [bbotk] e479a07c-6925-42a5-a4ed-f57c1c34331e
## INFO [20:51:32.153] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.161] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.164] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.176] [mlr3] Finished benchmark
## INFO [20:51:32.197] [bbotk] Result of batch 7:
## INFO [20:51:32.198] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.198] [bbotk] 0.05795381 6 0.7382812 0 0 0
## INFO [20:51:32.198] [bbotk] uhash
## INFO [20:51:32.198] [bbotk] cdcfeb0d-3a80-4af6-a937-7851c8106bba
## INFO [20:51:32.200] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.207] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.210] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.222] [mlr3] Finished benchmark
## INFO [20:51:32.240] [bbotk] Result of batch 8:
## INFO [20:51:32.241] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.241] [bbotk] 0.04561714 1 0.7382812 0 0 0
## INFO [20:51:32.241] [bbotk] uhash
## INFO [20:51:32.241] [bbotk] 0e505278-1204-4e16-b9d0-d26e153a385f
## INFO [20:51:32.243] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.250] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.253] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.264] [mlr3] Finished benchmark
## INFO [20:51:32.282] [bbotk] Result of batch 9:
## INFO [20:51:32.283] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.283] [bbotk] 0.06368355 1 0.7382812 0 0 0.02
## INFO [20:51:32.283] [bbotk] uhash
## INFO [20:51:32.283] [bbotk] 27a6cefa-2095-4c34-af8d-ee5b5130872f
## INFO [20:51:32.285] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.293] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.296] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.308] [mlr3] Finished benchmark
## INFO [20:51:32.326] [bbotk] Result of batch 10:
## INFO [20:51:32.327] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.327] [bbotk] 0.06798473 5 0.7382812 0 0 0.01
## INFO [20:51:32.327] [bbotk] uhash
## INFO [20:51:32.327] [bbotk] 4b8a679c-29fc-45c7-a765-926cf1e0e09d
## INFO [20:51:32.331] [bbotk] Finished optimizing after 10 evaluation(s)
## INFO [20:51:32.332] [bbotk] Result:
## INFO [20:51:32.332] [bbotk] cp minsplit learner_param_vals x_domain classif.acc
## INFO [20:51:32.332] [bbotk] 0.02278977 3 <list[3]> <list[2]> 0.7695312
at$predict(task)
## <PredictionClassif> for 768 observations:
## row_ids truth response
## 1 pos pos
## 2 neg neg
## 3 pos neg
## ---
## 766 neg neg
## 767 pos neg
## 768 neg neg
这个方法也有个简便写法：
auto_learner <- auto_tuner(
learner = learner,
resampling = resampling,
measure = measure,
search_space = search_space,
method = "random_search",
term_evals = 10
)
auto_learner$train(task)
## INFO [20:51:32.407] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerRandomSearch>' and '<TerminatorEvals> [n_evals=10, k=0]'
## INFO [20:51:32.414] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.421] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.425] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.437] [mlr3] Finished benchmark
## INFO [20:51:32.457] [bbotk] Result of batch 1:
## INFO [20:51:32.457] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.457] [bbotk] 0.0178747 6 0.7382812 0 0 0.02
## INFO [20:51:32.457] [bbotk] uhash
## INFO [20:51:32.457] [bbotk] 2eb118ca-943b-4f2c-83f0-1fd7d395e163
## INFO [20:51:32.459] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.467] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.470] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.482] [mlr3] Finished benchmark
## INFO [20:51:32.500] [bbotk] Result of batch 2:
## INFO [20:51:32.500] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.500] [bbotk] 0.02922122 8 0.7539062 0 0 0
## INFO [20:51:32.500] [bbotk] uhash
## INFO [20:51:32.500] [bbotk] f2043af0-1528-4765-8f49-c292b30413bf
## INFO [20:51:32.502] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.510] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.513] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.524] [mlr3] Finished benchmark
## INFO [20:51:32.543] [bbotk] Result of batch 3:
## INFO [20:51:32.544] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.544] [bbotk] 0.00829102 1 0.7070312 0 0 0.01
## INFO [20:51:32.544] [bbotk] uhash
## INFO [20:51:32.544] [bbotk] c0ce1d3a-8d92-4cf6-bc1c-d48115ec00bf
## INFO [20:51:32.545] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.553] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.556] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.568] [mlr3] Finished benchmark
## INFO [20:51:32.588] [bbotk] Result of batch 4:
## INFO [20:51:32.589] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.589] [bbotk] 0.004526872 3 0.7109375 0 0 0
## INFO [20:51:32.589] [bbotk] uhash
## INFO [20:51:32.589] [bbotk] fda2c932-d259-4a47-8d8f-8aab90019b48
## INFO [20:51:32.591] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.599] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.602] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.613] [mlr3] Finished benchmark
## INFO [20:51:32.632] [bbotk] Result of batch 5:
## INFO [20:51:32.633] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.633] [bbotk] 0.0861106 5 0.7539062 0 0 0
## INFO [20:51:32.633] [bbotk] uhash
## INFO [20:51:32.633] [bbotk] 241bb5e4-8259-4673-aba5-4deb98fe6aff
## INFO [20:51:32.635] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.643] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.646] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.658] [mlr3] Finished benchmark
## INFO [20:51:32.677] [bbotk] Result of batch 6:
## INFO [20:51:32.678] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.678] [bbotk] 0.01523008 4 0.7382812 0 0 0.02
## INFO [20:51:32.678] [bbotk] uhash
## INFO [20:51:32.678] [bbotk] bd043464-f412-4873-8d6d-b93d563176e3
## INFO [20:51:32.680] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.687] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.690] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.705] [mlr3] Finished benchmark
## INFO [20:51:32.723] [bbotk] Result of batch 7:
## INFO [20:51:32.724] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.724] [bbotk] 0.09650509 5 0.7226562 0 0 0.01
## INFO [20:51:32.724] [bbotk] uhash
## INFO [20:51:32.724] [bbotk] 1cecea5a-245a-4fb7-809a-7e7902467072
## INFO [20:51:32.726] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.733] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.736] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.748] [mlr3] Finished benchmark
## INFO [20:51:32.766] [bbotk] Result of batch 8:
## INFO [20:51:32.767] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.767] [bbotk] 0.06900854 10 0.7539062 0 0 0
## INFO [20:51:32.767] [bbotk] uhash
## INFO [20:51:32.767] [bbotk] 84d40f61-9a67-4bb3-b20f-96d7aa88be5a
## INFO [20:51:32.768] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.776] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.779] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.790] [mlr3] Finished benchmark
## INFO [20:51:32.809] [bbotk] Result of batch 9:
## INFO [20:51:32.810] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.810] [bbotk] 0.08973989 4 0.7539062 0 0 0.02
## INFO [20:51:32.810] [bbotk] uhash
## INFO [20:51:32.810] [bbotk] 68874363-065e-4e7d-b546-3f859fa1063c
## INFO [20:51:32.811] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:32.819] [mlr3] Running benchmark with 1 resampling iterations
## INFO [20:51:32.822] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/1)
## INFO [20:51:32.834] [mlr3] Finished benchmark
## INFO [20:51:32.853] [bbotk] Result of batch 10:
## INFO [20:51:32.854] [bbotk] cp minsplit classif.acc warnings errors runtime_learners
## INFO [20:51:32.854] [bbotk] 0.02565367 7 0.7265625 0 0 0
## INFO [20:51:32.854] [bbotk] uhash
## INFO [20:51:32.854] [bbotk] 9b8f4dd5-cd07-4cb2-a5c0-a1ae480ffc1a
## INFO [20:51:32.858] [bbotk] Finished optimizing after 10 evaluation(s)
## INFO [20:51:32.859] [bbotk] Result:
## INFO [20:51:32.859] [bbotk] cp minsplit learner_param_vals x_domain classif.acc
## INFO [20:51:32.859] [bbotk] 0.02922122 8 <list[3]> <list[2]> 0.7539062
auto_learner$predict(task)
## <PredictionClassif> for 768 observations:
## row_ids truth response
## 1 pos pos
## 2 neg neg
## 3 pos neg
## ---
## 766 neg neg
## 767 pos neg
## 768 neg neg
超参数设定的方法
每次单独设置超参数的范围等可能会显得比较笨重无聊，mlr3也提供另外一种可以在选择学习器时进行设定超参数的方法。
# 在选择学习器时设置超参数范围
learner <- lrn("classif.svm")
learner$param_set$values$kernel <- "polynomial"
learner$param_set$values$degree <- to_tune(lower = 1, upper = 3)
print(learner$param_set$search_space())
## <ParamSet>
## id class lower upper nlevels default value
## 1: degree ParamInt 1 3 3 <NoDefault[3]>
但其实这样也有问题，这个方法要求你对算法很熟悉，能够记住所有超参数记忆它们在mlr3中的拼写！但很显然这有点困难，所有我还是推荐第一种，每次单独设置，记不住还可以查看一下具体的超参数。
参数依赖
某些超参数只有在某些条件下才有效，比如支持向量机（SVM），它的degree参数只有在kernel是polynomial时才有效，这种情况也可以在mlr3中设置好。
library(data.table)
search_space = ps(
cost = p_dbl(-1, 1, trafo = function(x) 10^x), # 可进行数据变换
kernel = p_fct(c("polynomial", "radial")),
degree = p_int(1, 3, depends = kernel == "polynomial") # 设置参数依赖
)
rbindlist(generate_design_grid(search_space, 3)$transpose(), fill = TRUE)
## cost kernel degree
## 1: 0.1 polynomial 1
## 2: 0.1 polynomial 2
## 3: 0.1 polynomial 3
## 4: 0.1 radial NA
## 5: 1.0 polynomial 1
## 6: 1.0 polynomial 2
## 7: 1.0 polynomial 3
## 8: 1.0 radial NA
## 9: 10.0 polynomial 1
## 10: 10.0 polynomial 2
## 11: 10.0 polynomial 3
## 12: 10.0 radial NA
进行以上设置后在进行后面的操作时不会出错，自动处理。
以上就是今天的内容，希望对你有帮助哦！欢迎点赞、在看、关注、转发！
欢迎在评论区留言或直接添加我的微信！
欢迎关注我的公众号：医学和生信笔记
“医学和生信笔记 公众号主要分享：1.医学小知识、肛肠科小知识；2.R语言和Python相关的数据分析、可视化、机器学习等；3.生物信息学学习资料和自己的学习笔记！