woj/woj-server
1
0
Fork 0
Code Issues 2 Pull Requests Packages Projects Releases Wiki Activity

使用 cgroups 统计资源 #9

New Issue
Open
opened 2024-01-31 12:03:31 +08:00 by Paul · 6 comments
Paul commented 2024-01-31 12:03:31 +08:00
Owner
Copy Link

目前使用 wait4 统计内存、时间开销，但如果子进程涉及多线程，统计结果将不可靠：

I don't have a reference for this, but you can easily verify it yourself. The CPU time of a grand-child is only included in the CPU time of a child if the grand-child terminated before the child and the child has waited for the termination of the grand-child with one of the wait system calls. Otherwise the CPU time of the grand child is not yet included or even completely lost.

stackexchange

这一点在 pypy3 上得到验证：A+B 问题中 RLIMIT 设置的极限内存（76MB）与统计的内存（~50MB）存在过大差距

Originally posted by @Paul in woj/woj-sandbox#3 (comment)

目前使用 `wait4` 统计内存、时间开销，但如果子进程涉及多线程，统计结果将不可靠： > I don't have a reference for this, but you can easily verify it yourself. The CPU time of a grand-child is only included in the CPU time of a child if the grand-child terminated before the child and the child has waited for the termination of the grand-child with one of the wait system calls. Otherwise the CPU time of the grand child is not yet included or even completely lost. > > [stackexchange](https://unix.stackexchange.com/questions/61244/cpu-time-measurement-with-wait4-vs-cpuacct-cgroup) 这一点在 `pypy3` 上得到验证：`A+B` 问题中 `RLIMIT` 设置的极限内存（`76MB`）与统计的内存（`~50MB`）存在过大差距 _Originally posted by @Paul in https://git.0x7f.app/woj/woj-sandbox/issues/3#issue-22_
Paul commented 2024-01-31 12:06:38 +08:00
Author
Owner
Copy Link

memory.peak 统计在 a1235a506b 支持，对应于 containerd/cgroups@v3.0.3

而 containerd 最早在版本 v2.0.0-beta.2 中更新 cgroups 版本

`memory.peak` 统计在 https://github.com/containerd/cgroups/commit/a1235a506bb9732067abf3fb1fdd9d0bced81008 支持，对应于 `containerd/cgroups@v3.0.3` 而 `containerd` 最早在版本 [v2.0.0-beta.2](https://github.com/containerd/containerd/releases/tag/v2.0.0-beta.2) 中更新 `cgroups` 版本
Paul added this to the 1.4.0 milestone 2024-01-31 12:06:46 +08:00
Paul added the
bug
enhancement
 labels 2024-01-31 12:06:54 +08:00
Paul added reference develop 2024-01-31 12:06:57 +08:00
Paul self-assigned this 2024-01-31 12:07:01 +08:00
Paul commented 2024-02-15 12:56:54 +08:00
Author
Owner
Copy Link

6956fe4ee1 添加了 Metrics 功能，尚未测试

6956fe4ee1b33d072d5c79f9bbc6b369758e2e53 添加了 `Metrics` 功能，尚**未**测试
Paul commented 2024-03-13 20:04:13 +08:00
Author
Owner
Copy Link

ff98b7b05f 使用 nsjail 替换了 containerd

ff98b7b05f 使用 `nsjail` 替换了 `containerd`
Paul commented 2024-04-28 01:22:59 +08:00
Author
Owner
Copy Link

目前使用 wait4 统计内存、时间开销，但如果子进程涉及多线程，统计结果将不可靠：

I don't have a reference for this, but you can easily verify it yourself. The CPU time of a grand-child is only included in the CPU time of a child if the grand-child terminated before the child and the child has waited for the termination of the grand-child with one of the wait system calls. Otherwise the CPU time of the grand child is not yet included or even completely lost.

stackexchange

这一点在 pypy3 上得到验证：A+B 问题中 RLIMIT 设置的极限内存（76MB）与统计的内存（~50MB）存在过大差距

Originally posted by @Paul in woj/woj-sandbox#3 (comment)

举例有误，pypy3 的差距并不来自于多线程（NPROC 设置的是 1）

> 目前使用 `wait4` 统计内存、时间开销，但如果子进程涉及多线程，统计结果将不可靠： > > > I don't have a reference for this, but you can easily verify it yourself. The CPU time of a grand-child is only included in the CPU time of a child if the grand-child terminated before the child and the child has waited for the termination of the grand-child with one of the wait system calls. Otherwise the CPU time of the grand child is not yet included or even completely lost. > > > > [stackexchange](https://unix.stackexchange.com/questions/61244/cpu-time-measurement-with-wait4-vs-cpuacct-cgroup) > > 这一点在 `pypy3` 上得到验证：`A+B` 问题中 `RLIMIT` 设置的极限内存（`76MB`）与统计的内存（`~50MB`）存在过大差距 > > _Originally posted by @Paul in https://git.0x7f.app/woj/woj-sandbox/issues/3#issue-22_ 举例有误，`pypy3` 的差距并不来自于多线程（`NPROC` 设置的是 `1`）
Paul commented 2024-04-28 01:23:56 +08:00
Author
Owner
Copy Link

0f026aa180 引入了 SoftMemoryLimit 用于处理 Go 等喜欢预留大量内存的语言

0f026aa18074a2467464060c658b14cc05d334a2 引入了 `SoftMemoryLimit` 用于处理 `Go` 等喜欢预留大量内存的语言
Paul commented 2024-04-28 01:24:51 +08:00
Author
Owner
Copy Link
  • 配置 cgroups 时考虑限制 swap 使用 （或者不允许使用 swap ?）
- [ ] 配置 `cgroups` 时考虑限制 `swap` 使用 （或者不允许使用 `swap` ?）
Sign in to join this conversation.
develop
Branches Tags
Labels
Clear labels   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
question

More information is needed

  
wontfix

This won't be fixed

No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
1.4.0
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
Paul
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: woj/woj-server#9
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?