[2026-04-10 09:49:47,915][ INFO][PID:603075] Marking build as starting [2026-04-10 09:49:47,983][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:49:47,984][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:49:47,998][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2083, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:49:53,006][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:49:58,013][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:03,014][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:08,016][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:13,019][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:18,022][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:23,025][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:28,026][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:33,035][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:38,036][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:43,038][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:48,040][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:53,042][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:50:58,045][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:03,047][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:08,048][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:13,052][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:18,056][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:23,059][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:28,060][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:33,062][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:38,071][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:43,076][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:48,077][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:53,078][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:51:58,081][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:03,085][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:08,086][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:13,089][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:18,092][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:23,093][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:28,094][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:33,096][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:38,097][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:43,100][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:48,103][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:53,109][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:52:58,113][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:03,116][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:08,120][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:13,122][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:18,123][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:23,125][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:28,126][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:33,132][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:38,133][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:43,134][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:48,136][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:53,141][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:53:58,144][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:03,148][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:06,195][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2083, hostname=2620:52:6:1161:dead:beef:cafe:c902, name=vmhost_p09_03_dev_02146169_20260410_094929, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:54:06,198][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:54:06,199][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:54:07,850][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:54:07,850][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2083, hostname=2620:52:6:1161:dead:beef:cafe:c902, name=vmhost_p09_03_dev_02146169_20260410_094929, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:54:07,853][ INFO][PID:603075] VM Release request [2026-04-10 09:54:07,892][ INFO][PID:603075] Retry #1 (on other host) [2026-04-10 09:54:07,895][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:07,896][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:54:07,912][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2114, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:54:12,916][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:17,917][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:22,922][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:27,929][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:32,931][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:37,933][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:42,937][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:47,939][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:52,942][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:54:57,944][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:02,947][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:07,949][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:12,954][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:17,957][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:22,962][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:27,963][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:32,967][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:37,970][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:42,971][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:47,974][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:52,977][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:55:57,979][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:02,980][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:07,988][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:12,991][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:17,994][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:22,998][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:26,120][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2114, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:26,121][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:56:26,126][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:56:27,266][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:56:27,268][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2114, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:56:27,271][ INFO][PID:603075] VM Release request [2026-04-10 09:56:27,317][ INFO][PID:603075] Retry #2 (on other host) [2026-04-10 09:56:27,320][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:27,325][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:56:27,375][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2185, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:30,425][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2185, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:30,427][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:56:30,430][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:56:30,675][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:56:30,678][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2185, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:56:30,680][ INFO][PID:603075] VM Release request [2026-04-10 09:56:30,705][ INFO][PID:603075] Retry #3 (on other host) [2026-04-10 09:56:30,709][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:30,712][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:56:30,764][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2188, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:33,836][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2188, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:33,843][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:56:33,846][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:56:34,071][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:56:34,071][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2188, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:56:34,075][ INFO][PID:603075] VM Release request [2026-04-10 09:56:34,116][ INFO][PID:603075] Retry #4 (on other host) [2026-04-10 09:56:34,119][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:34,124][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:56:34,160][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2193, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:37,207][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2193, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:37,208][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:56:37,210][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:56:37,397][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:56:37,400][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2193, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:56:37,401][ INFO][PID:603075] VM Release request [2026-04-10 09:56:37,521][ INFO][PID:603075] Retry #5 (on other host) [2026-04-10 09:56:37,524][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:37,530][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:56:37,603][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2197, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:40,678][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2197, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:40,682][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:56:40,682][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:56:40,866][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:56:40,867][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2197, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:56:40,868][ INFO][PID:603075] VM Release request [2026-04-10 09:56:40,894][ INFO][PID:603075] Retry #6 (on other host) [2026-04-10 09:56:40,896][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:40,899][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:56:40,924][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2202, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:43,973][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2202, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:43,974][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:56:43,974][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:56:44,105][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:56:44,107][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2202, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02146175_20260410_095230, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:56:44,109][ INFO][PID:603075] VM Release request [2026-04-10 09:56:44,129][ INFO][PID:603075] Retry #7 (on other host) [2026-04-10 09:56:44,133][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:44,136][ INFO][PID:603075] VM allocation process starts [2026-04-10 09:56:44,158][ INFO][PID:603075] Trying to allocate VM: ResallocHost, ticket_id=2205, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:56:49,163][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:54,166][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:56:59,170][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:04,172][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:09,175][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:14,177][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:19,179][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:24,181][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:29,184][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:34,194][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:39,196][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:44,198][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:49,201][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:54,203][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:57:59,205][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:04,208][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:09,211][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:14,214][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:19,216][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:24,220][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:29,222][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:34,225][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:39,230][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:44,231][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:49,232][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:54,235][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:58:59,237][ INFO][PID:603075] Checking for cancel request [2026-04-10 09:59:02,353][ INFO][PID:603075] Allocated host ResallocHost, ticket_id=2205, hostname=2620:52:6:1161:dead:beef:cafe:c907, name=vmhost_x86_04_dev_02146187_20260410_095621, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-10 09:59:02,355][ INFO][PID:603075] Allocating ssh connection to builder [2026-04-10 09:59:02,356][ INFO][PID:603075] Checking that builder machine is OK [2026-04-10 09:59:02,928][ INFO][PID:603075] Installed copr-rpmbuild version: 1.6 [2026-04-10 09:59:02,929][ ERROR][PID:603075] Re-try request for task on 'ResallocHost, ticket_id=2205, hostname=2620:52:6:1161:dead:beef:cafe:c907, name=vmhost_x86_04_dev_02146187_20260410_095621, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-10 09:59:02,930][ INFO][PID:603075] VM Release request [2026-04-10 09:59:02,946][ ERROR][PID:603075] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c900', '2620:52:6:1161:dead:beef:cafe:c907', '2620:52:6:1161:dead:beef:cafe:c902'} [2026-04-10 09:59:02,950][WARNING][PID:603075] Switching not-finished job state to 'failed' [2026-04-10 09:59:02,954][ INFO][PID:603075] Worker failed build, took 555.0407085418701 [2026-04-10 09:59:02,958][ INFO][PID:603075] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.stg.fedoraproject.org", "memory_reqs": null, "enable_net": true, "project_owner": "@copr", "project_name": "TEST1775814189149811860-storage-backend", "project_dirname": "TEST1775814189149811860-storage-backend", "submitter": "frostyx", "ended_on": 1775815142.95076, "started_on": 1775814587.9100513, "submitted_on": null, "status": 0, "chroot": "srpm-builds", "arch": null, "buildroot_pkgs": null, "task_id": "2924923", "build_id": 2924923, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 1, "source_json": "{\"url\": \"https://github.com/fedora-copr/copr-test-sources/raw/main/hello-2.8-1.src.rpm\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@copr/TEST1775814189149811860-storage-backend--frostyx", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "distributions_in_build": [], "distributions_in_project": [ "fedora-43" ], "destdir": "/var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend", "results_repo_url": "https://download.copr-dev.fedorainfracloud.org/results/@copr/TEST1775814189149811860-storage-backend", "result_dir": "02924923", "built_packages": "", "tags": [ "arch_noarch" ], "id": 2924923, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "TEST1775814189149811860-storage-backend", "vendor": "Fedora Project COPR (@copr/TEST1775814189149811860-storage-backend)" } } ] } [2026-04-10 09:59:03,073][ INFO][PID:603075] Sending fedora-messaging bus message in build.end [2026-04-10 09:59:05,664][ INFO][PID:603075] Compressing /var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend/srpm-builds/02924923/builder-live.log by gzip [2026-04-10 09:59:05,677][ INFO][PID:603075] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend/srpm-builds/02924923/builder-live.log' as PID 623174 [2026-04-10 09:59:05,682][ INFO][PID:603075] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend/srpm-builds/02924923/builder-live.log) stdout: stderr: gzip: /var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend/srpm-builds/02924923/builder-live.log: No such file or directory [2026-04-10 09:59:05,693][ ERROR][PID:603075] Unable to compress file /var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend/srpm-builds/02924923/builder-live.log [2026-04-10 09:59:05,695][ INFO][PID:603075] Compressing /var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend/srpm-builds/02924923/backend.log by gzip [2026-04-10 09:59:05,699][ INFO][PID:603075] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1775814189149811860-storage-backend/srpm-builds/02924923/backend.log' as PID 623176