Message boards : Puzzles and science : KAMPYLOS
Message board moderation
Previous · 1 · 2 · 3 · Next
| Author | Message |
|---|---|
ConanSend message Joined: 29 Sep 26 Posts: 38 Credit: 19,720,099 RAC: 239,916 |
In reply to Drago75's message of 5 Oct 2026: I am getting mostly download errors for Kampylos, but strangely not all of them. RTX 3060m on Linux Mint 22.3: Yes I just had a number of these as well. Only 2 successes out of 8 or 9 downloads. Potamos does not have this issue. Conan |
|
25 years old Send message Joined: 27 Sep 26 Posts: 349 Credit: 5,210,003 RAC: 137,017 |
The problem has been resolved.
|
|
51 years old Send message Joined: 28 Sep 26 Posts: 67 Credit: 45,851,691 RAC: 928,453 |
I am running Kampylos Nvidia on my RTX 3060m Laptop, Linux Mint 22.3 On this machine one of the GPU tasks uses up to 4 CPU threads. Just to let everyone know to check their CPU utilization from time to time to avoid a frozen and unresponsive host. |
dskagcommunity46 years old Send message Joined: 29 Sep 26 Posts: 64 Credit: 21,977,365 RAC: 136,334 |
OMG thx for looking for that. I had the same though but never had the time to look for it. Should be in the app then as 4C/1NV as expamle from projectside. ---------- 24/7 Crunching since 2011 ----------- DSKAG Austria: http://www.dskag.at
|
|
Send message Joined: 17 Oct 26 Posts: 2 Credit: 13,290,867 RAC: 486,633 |
There is a few Kampylos tasks that takes a lot longer than others. I have a couple of tasks that has reached 4%, 8% and 11% on 32+ hrs runtime - is this expected? I will let them run unless you recommend otherwise. |
|
25 years old Send message Joined: 27 Sep 26 Posts: 349 Credit: 5,210,003 RAC: 137,017 |
Hi Oggy Keep it running. We apply a credit adjustment for long-running tasks.
|
|
72 years old Send message Joined: 28 Sep 26 Posts: 6 Credit: 8,019,775 RAC: 42,311 |
Hi! Oggy: Maybe You will beat this one, ( https://bitboinc.athena.org.tr/workunit.php?wuid=4147056 ) let it run and get "some" credits😁😉👍 Hans S. |
|
Send message Joined: 17 Oct 26 Posts: 2 Credit: 13,290,867 RAC: 486,633 |
Challenge accepted 😎
|
Steve DoddSend message Joined: 28 Sep 26 Posts: 27 Credit: 53,008,805 RAC: 218,076 |
My biggest concern is they won't complete by deadline. |
|
Send message Joined: 1 Oct 26 Posts: 35 Credit: 86,317,024 RAC: 555,648 |
Once the WU starts, the deadline is almost meaningless. It will continue to run to completion. The resend that would be sent out to another user would take the same amount of time (more or less) as your WU and you would have a 10 to 12 day head start. Most likely, you WU will end first. I just aborted 5 WUs that had been running 10 to 12 days and were still in the single digits for completion percentage. They supposedly wanted another 250 to 476 days to complete and I wasn't inclined to run a WU for close to 2 years. Even Climate Prediction work runs less than that. |
|
51 years old Send message Joined: 28 Sep 26 Posts: 67 Credit: 45,851,691 RAC: 928,453 |
When you run Kampylos check your cpu utilisation. At least the GPU version uses up to 4 cpu threads. Maybe the cpu version does that too which would lead to a massiv over-assignment of your cpu and a good explanation why some tasks take that long |
|
Send message Joined: 1 Oct 26 Posts: 35 Credit: 86,317,024 RAC: 555,648 |
In this particular case I don't think that is the problem. These were running on a 128-thread 256GB server |
|
51 years old Send message Joined: 28 Sep 26 Posts: 67 Credit: 45,851,691 RAC: 928,453 |
You are spoiled! :-) |
|
51 years old Send message Joined: 28 Sep 26 Posts: 67 Credit: 45,851,691 RAC: 928,453 |
Hey Alperen! Great that you add more FP64 support! It`s only that my laptop AMD iGPU is not FP64-capable so the task failed. Maybe the hardware detection of your scheduler thinks it`s a full grown discrete GPU? Bug report: Kampylos v0.09 assigned to an OpenCL device without FP64 support Kampylos v0.09 (opencl_ati) failed on my ASUS laptop’s AMD integrated GPU under Linux. Result ID: 7600303 Workunit ID: 7380234 Host ID: 89 GPU: AMD Radeon Graphics (renoir) OpenCL: Mesa Rusticl / radeonsi BOINC: 8.3.0 The application selected the correct AMD GPU, but kernel compilation failed: error: use of type 'double' requires cl_khr_fp64 support warning: unsupported OpenCL extension 'cl_khr_fp64' - ignoring opencl_fit_select_device(0) failed: kernel build failed clinfo confirms that this device currently exposes no double-precision support. The task exited with code 1 before GPU computation could start. Could you add an FP64 requirement to the server-side plan class? BOINC provides <double_precision_fp/> for this purpose. An application-side capability check would also avoid the lengthy compiler error output. https://bitboinc.athena.org.tr/result.php?resultid=7600303 |
|
25 years old Send message Joined: 27 Sep 26 Posts: 349 Credit: 5,210,003 RAC: 137,017 |
I will fix this. I had made this change a long time ago due to complaints about tasks; I wasn't aware that the iGPUs were being disabled.
|
|
51 years old Send message Joined: 28 Sep 26 Posts: 67 Credit: 45,851,691 RAC: 928,453 |
Here is another issue. It looks like the time window a task is granted to complete is at times too strict. **Subject: Kampylos v0.12 – EXIT_TIME_LIMIT_EXCEEDED (197) on long-running FSPL+LD tasks** Hi Alperen, I've been investigating several Kampylos v0.12 CPU tasks that failed with exit code 197 (`EXIT_TIME_LIMIT_EXCEEDED`) on different Linux hosts. To understand the issue, I also analyzed 100 successfully completed Kampylos tasks on my Windows AceMagic F3A (host 156). **1. Results from 100 successful tasks** All 100 tasks were reported as successful and valid. | Initial model | Tasks | Average runtime | Longest runtime | | PSPL | 18 | 37.3 min | 4h 17m | | FSPL | 42 | 49.9 min | 5h 29m | | FSPL+LD | 40 | 63.0 min | 9h 02m | The runtime varies considerably, even between tasks using the same dataset and initial model. **2. Direct comparison using dataset ev14764** Two tasks processed the same dataset (`ev14764`, 1,472 data points), with identical single-lens initialization results: - PSPL chi2 = 2648.823 - FSPL chi2 = 2648.823 - FSPL+LD chi2 = 2648.823 - Selected model: `best=fspl_ld` - tE = 2999.9668 **Successful task – Windows AceMagic F3A** - Result: [8752232](https://bitboinc.athena.org.tr/result.php?resultid=8752232) - Workunit: 8435867 - Host: 156 - Grid: 1×1 - log10_s = -0.8000 - log10_q = -1.6667 - Runtime: 9h 02m 22s - CPU time: 8h 59m 21s - Exit status: 0 - Validation: Valid The task completed its single grid cell and called `boinc_finish(0)`. **Failed task – Linux ROG Ally** - Result: [8752202](https://bitboinc.athena.org.tr/result.php?resultid=8752202) - Workunit: 8435837 - Host: 995 - Grid: 1×1 - log10_s = -0.8000 - log10_q = -3.3333 - Runtime before termination: approximately 5h 31m - Exit status: 197 (`EXIT_TIME_LIMIT_EXCEEDED`) BOINC reported: ``` exceeded elapsed time limit 19870.92 (1200000.00G/60.39G) ``` The application output ended at: ``` Resuming from cell 0/1 ``` There was no completed-cell output before termination. These are different grid-search parameter points, so they are not identical calculations. However, they demonstrate that tasks from the same dataset can have very different computational requirements. **3. Another interesting comparison: 3,173 data points** Two FSPL+LD tasks completed successfully on the Windows host: - [8751906](https://bitboinc.athena.org.tr/result.php?resultid=8751906) – 1h 45m 50s - [8751862](https://bitboinc.athena.org.tr/result.php?resultid=8751862) – 3h 16m 29s Another FSPL+LD task with 3,173 data points failed on a Linux NUC: - [8751908](https://bitboinc.athena.org.tr/result.php?resultid=8751908) – exit 197 after approximately 8h 50m **4. Possible issue with the BOINC runtime limit** The affected tasks appear to have an `rsc_fpops_bound` of approximately 1.2 × 10^15 FLOPs. The ROG Ally failure message indicates that BOINC calculated an elapsed-time limit of only 19,870.92 seconds. This is particularly interesting because other Kampylos tasks can legitimately require more than nine hours of computation. My suspicion is that the current `rsc_fpops_bound`, or the resulting BOINC elapsed-time limit, may be too restrictive for some computationally demanding grid cells. Of course, the available evidence does not prove that the failed tasks would eventually have completed successfully. They may also involve unusually difficult numerical optimization cases. **Could you please check:** 1. Whether the current `rsc_fpops_bound` is sufficient for the worst-case Kampylos v0.12 CPU workloads. 2. Whether some grid-search parameter combinations require substantially more computation than others. 3. Whether increasing the runtime allowance or adjusting the workunit limits would prevent unnecessary exit-197 failures. I would be happy to provide additional task results or logs if needed. Thanks for your work on Kampylos and BitBOINC! |
|
25 years old Send message Joined: 27 Sep 26 Posts: 349 Credit: 5,210,003 RAC: 137,017 |
Thanks for the detailed analysis, you were right. The time limit BOINC computed for these tasks was too tight. Cause: The limit is derived from rsc_fpops_bound. For lighter cells the generator set it to a floor of 1.2e15 FLOPs. On a ~60 GFLOPS host that works out to about 19,870 s (~5.5 h). Run time within one dataset varies a lot, and FSPL+LD cells can run well past 9 h, so those tasks were cut off with exit 197. Over the last 14 days, 188 of about 52,800 Kampylos results ended this way. Fix (live now): The bound multiplier went from 40× to 400× the estimate, and the floor from 1e15 to 1e16 FLOPs. On a ~60 GFLOPS host the limit is now roughly 46 h, so it should only catch genuinely stuck tasks. I raised the bound on all work units still waiting to be sent, so new downloads get the higher limit too. Tasks that were already downloaded keep their old limit. If one of those fails with 197, it will be re-sent with the new limit. Credit and scheduling are not affected. If you still see exit 197 on newly downloaded tasks, please send the result IDs.
|
|
51 years old Send message Joined: 28 Sep 26 Posts: 67 Credit: 45,851,691 RAC: 928,453 |
**Subject: Kampylos CPU task runtimes, checkpoint reliability and possible GPU-only workloads** Hi Alperen, lately my hosts received cpu versions more suitable for cpu but sometimes there are a few big ones in between. I've analyzed my Kampylos results to investigate which workunit types are suitable for CPU processing and which may be too computationally expensive. The dataset contains **3,811 successful and 94 unsuccessful results**, covering several Kampylos task types. ### 1. CPU runtime analysis Using four hours (14,400 seconds) as a practical target: - 3,579 successful tasks (93.9%) finished within four hours. - 232 successful tasks (6.1%) exceeded four hours. - The regular `cpu_row0`, `cpu_row1` and `cpu_row2` tasks generally perform well, with approximately 95% completing within four hours. - The `fast_row6` through `fast_row9` tasks are significantly more demanding. For the successful `fast_row` tasks: | Task type | Successful | Over 4 hours | | fast_row6 | 3 | 3 | | fast_row7 | 14 | 13 | | fast_row8 | 1 | 1 | | fast_row9 | 13 | 11 | | **Total** | **31** | **28** | Although the sample size is relatively small, these results suggest that the current `fast_row` workloads are generally too large for a four-hour CPU runtime target. ### 2. Checkpoint reliability Another concern is checkpoint handling. While monitoring several long-running `fast_row7` and `fast_row9` tasks on my Ryzen 9 7950X3D, I noticed behavior suggesting that checkpoint recovery may not always preserve all completed work. In some cases, the stderr output showed that later grid cells had already been processed, but after restarting, the application appeared to resume from cell 0. I have not yet established whether this is caused by incomplete checkpoint restoration, the way progress is logged, or another issue. However, reliable checkpoint recovery is particularly important for tasks requiring many hours or even days of CPU time. Otherwise, restarting BOINC or rebooting a computer may result in substantial duplicated computation. It would be useful to verify that checkpoint files correctly preserve the current cell and seed position and that the application resumes from the saved state. ### 3. Possible GPU-only workloads Rather than removing computationally demanding tasks from the project, perhaps the larger workloads could be handled differently. Some possible approaches: - Keep the regular `cpu_row0` to `cpu_row2` tasks available for CPU hosts. - Split particularly large `fast_row` tasks into smaller CPU workunits, ideally targeting completion within four hours. - Alternatively, offer the most computationally demanding task types exclusively to GPU hosts. This could reduce excessively long CPU runtimes while preserving the scientific calculations. ### 4. Runtime limits I would suggest treating four hours as a target rather than immediately enforcing a strict four-hour execution limit. A hard cutoff could cause additional wasted computation, particularly while checkpoint behavior remains uncertain. Some older tasks already accumulated hundreds of CPU hours before being aborted or reaching their reporting deadlines. **The main objective would be to improve workunit sizing and checkpoint reliability, rather than discard scientifically valuable workloads.** I hope these statistics are useful for further Kampylos development.
|
|
25 years old Send message Joined: 27 Sep 26 Posts: 349 Credit: 5,210,003 RAC: 137,017 |
I'm tired of constantly revising Kampylos. I'll probably uninstall the CPU version altogether soon.
|
rilianSend message Joined: 28 Sep 26 Posts: 261 Credit: 43,482,494 RAC: 474,864 |
|