2017-05-07 04:33:13 +08:00
- [What is it? ](#what-is-it- )
- [What form of isolation does it provide? ](#what-type-of-isolation-does-this-tool-provide- )
2017-05-07 04:34:33 +08:00
- [Which use-cases are supported? ](#which-use-cases-are-supported )
* [Isolation of network services (inetd-style) ](#isolation-of-network-services--inetd-style )
* [Isolation, with access to a private, cloned interface (requires euid==0) ](#isolation--with-access-to-a-private--cloned-interface--requires-euid--0 )
2017-01-16 05:17:14 +08:00
* [Isolation of local processes ](#isolation-of-local-processes )
2017-05-07 04:34:33 +08:00
* [Isolation of local processes (and re-running them) ](#isolation-of-local-processes--and-re-running-them )
2017-01-16 05:17:14 +08:00
* [Bash in a minimal file-system with uid==0 and access to /dev/urandom ](#bash-in-a-minimal-file-system-with-uid--0-and-access-to--dev-urandom )
2017-05-07 04:34:33 +08:00
* [Even more contrained shell (with seccomp-bpf policies) ](#even-more-contrained-shell--with-seccomp-bpf-policies )
- [More info? ](#more-info )
2017-05-07 04:33:13 +08:00
- [Launching in Docker ](#launching-in-docker )
2017-01-16 05:17:14 +08:00
2015-05-15 05:44:48 +08:00
### WHAT IS IT?
2015-05-16 11:18:23 +08:00
NsJail is a process isolation tool for Linux. It makes use of the the namespacing, resource control, and seccomp-bpf syscall filter subsystems of the Linux kernel.
2017-05-07 04:33:13 +08:00
It can help among others, with:
2015-08-15 22:20:48 +08:00
* Securing networking services (e.g. web, time, DNS), by isolating them from the rest of the OS
2015-05-16 11:18:23 +08:00
* Hosting computer security challenges (so-called CTFs)
* Containing invasive syscall-level OS fuzzers
2015-05-15 05:44:48 +08:00
2017-05-07 04:33:13 +08:00
Features:
* It offers three distinct operation modes. See [this section ](#which-use-cases-are-supported- ) for more info.
* Can use [kafel seccomp-bpf configuration language ](https://github.com/google/kafel/ ) for syscall policy creation.
* It's rock-solid.
2015-05-15 08:08:13 +08:00
2016-06-20 01:43:10 +08:00
### WHAT TYPE OF ISOLATION DOES THIS TOOL PROVIDE?
2015-05-16 11:18:23 +08:00
1. Linux namespaces: UTS (hostname), MOUNT (chroot), PID (separate PID tree), IPC, NET (separate networking context), USER
2015-05-16 07:55:25 +08:00
2. FS constraints: chroot(), pivot_root(), RO-remounting
2015-05-16 11:18:23 +08:00
3. Resource limits (wall-time/CPU time limits, VM/mem address space limits, etc.)
4. Programmable seccomp-bpf syscall filters
2016-03-02 10:10:08 +08:00
5. Cloned and separated Ethernet interfaces
2016-06-20 01:41:11 +08:00
6. Cgroups for memory utilization control
2015-05-15 05:44:48 +08:00
2016-06-20 01:43:10 +08:00
### WHICH USE-CASES ARE SUPPORTED?
2016-03-02 10:16:50 +08:00
#### Isolation of network services (inetd-style)
2015-05-15 08:25:55 +08:00
2017-05-07 04:33:13 +08:00
This is NOT an official Google product.
2015-05-15 08:25:55 +08:00
+ Server:
2015-05-15 08:27:51 +08:00
```
2015-05-15 05:44:48 +08:00
$ ./nsjail -Ml --port 9000 --chroot /chroot/ --user 99999 --group 99999 -- /bin/sh -i
2015-05-15 08:27:51 +08:00
```
2015-05-15 05:44:48 +08:00
2015-05-15 08:25:55 +08:00
+ Client:
```
2015-05-16 11:10:13 +08:00
$ nc 127.0.0.1 9000
/ $ ifconfig
/ $ ifconfig -a
lo Link encap:Local Loopback
LOOPBACK MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ $ ps wuax
PID USER COMMAND
1 99999 /bin/sh -i
2015-05-16 11:18:23 +08:00
3 99999 {busybox} ps wuax
2015-07-08 01:54:36 +08:00
/ $
2015-05-16 11:10:13 +08:00
2015-05-15 08:25:55 +08:00
```
2015-05-15 05:44:48 +08:00
2016-03-02 10:16:50 +08:00
#### Isolation, with access to a private, cloned interface (requires euid==0)
```
$ sudo ./nsjail --user 9999 --group 9999 --iface eth0 --chroot /chroot/ -Mo --iface_vs_ip 192.168.0.44 --iface_vs_nm 255.255.255.0 --iface_vs_gw 192.168.0.1 -- /bin/sh -i
/ $ id
uid=9999 gid=9999
/ $ ip addr sh
1: lo: < LOOPBACK , UP , LOWER_UP > mtu 65536 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: vs: < BROADCAST , MULTICAST , UP , LOWER_UP > mtu 1500 qdisc noqueue
link/ether ca:a2:69:21:33:66 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.44/24 brd 192.168.0.255 scope global vs
valid_lft forever preferred_lft forever
inet6 fe80::c8a2:69ff:fe21:cd66/64 scope link
valid_lft forever preferred_lft forever
/ $ nc 217.146.165.209 80
GET / HTTP/1.0
HTTP/1.0 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.ch/?gfe_rd=cr& ei=cEzWVrG2CeTI8ge88ofwDA
Content-Length: 258
Date: Wed, 02 Mar 2016 02:14:08 GMT
< HTML > < HEAD > < meta http-equiv = "content-type" content = "text/html;charset=utf-8" >
< TITLE > 302 Moved< / TITLE > < / HEAD > < BODY >
< H1 > 302 Moved< / H1 >
The document has moved
< A HREF = "http://www.google.ch/?gfe_rd=cr&ei=cEzWVrG2CeTI8ge88ofwDA" > here< / A > .
< / BODY > < / HTML >
/ $
```
2015-05-16 07:55:25 +08:00
#### Isolation of local processes
2015-05-15 08:25:55 +08:00
```
2015-05-15 05:44:48 +08:00
$ ./nsjail -Mo --chroot /chroot/ --user 99999 --group 99999 -- /bin/sh -i
2015-05-16 11:10:13 +08:00
/ $ ifconfig -a
lo Link encap:Local Loopback
LOOPBACK MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
2015-05-15 05:44:48 +08:00
/ $ id
uid=99999 gid=99999
2015-05-16 11:10:13 +08:00
/ $ ps wuax
PID USER COMMAND
1 99999 /bin/sh -i
2015-05-16 11:18:23 +08:00
4 99999 {busybox} ps wuax
2015-05-15 05:44:48 +08:00
/ $exit
$
2015-05-15 08:25:55 +08:00
```
2015-05-15 05:44:48 +08:00
2015-05-16 07:55:25 +08:00
#### Isolation of local processes (and re-running them)
2015-05-15 08:25:55 +08:00
```
2015-05-15 05:44:48 +08:00
$ ./nsjail -Mr --chroot /chroot/ --user 99999 --group 99999 -- /bin/sh -i
BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
2015-05-16 11:10:13 +08:00
/ $ ps wuax
PID USER COMMAND
1 99999 /bin/sh -i
2 99999 {busybox} ps wuax
2015-05-15 05:44:48 +08:00
/ $ exit
BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
2015-05-16 11:10:13 +08:00
/ $ ps wuax
PID USER COMMAND
1 99999 /bin/sh -i
2 99999 {busybox} ps wuax
2015-05-15 05:44:48 +08:00
/ $
2015-05-15 08:25:55 +08:00
```
2015-05-15 05:44:48 +08:00
2016-06-19 20:32:27 +08:00
#### Bash in a minimal file-system with uid==0 and access to /dev/urandom
```
2016-12-11 00:11:55 +08:00
$ ./nsjail -Mo --user 0 --group 99999 -R /bin/ -R /lib -R /lib64/ -R /usr/ -R /sbin/ -T /dev -R /dev/urandom --keep_caps -- /bin/bash -i
2016-06-19 20:32:27 +08:00
bash-4.3# ls -l /
total 40
drwxr-xr-x 2 65534 65534 12288 Jun 17 23:27 bin
drwxrwxrwt 2 0 99999 60 Jun 19 12:31 dev
drwxr-xr-x 25 65534 65534 4096 Jun 9 18:29 lib
drwxr-xr-x 2 65534 65534 4096 Apr 15 22:27 lib64
dr-xr-xr-x 260 65534 65534 0 Jun 19 12:31 proc
drwxr-xr-x 2 65534 65534 16384 Jun 11 21:03 sbin
drwxr-xr-x 21 65534 65534 4096 Apr 24 16:13 usr
bash-4.3# ls -l /dev/
total 0
crw-rw-rw- 1 65534 65534 1, 9 Jun 9 18:33 urandom
bash-4.3# id
uid=0 gid=99999 groups=99999,65534
```
2017-01-16 05:00:55 +08:00
#### Even more contrained shell (with seccomp-bpf policies)
```
$ ./nsjail --chroot / --seccomp_string 'POLICY a { ALLOW { write, execve, brk, access, mmap, open, newfstat, close, read, mprotect, arch_prctl, munmap, getuid, getgid, getpid, rt_sigaction, geteuid, getppid, getcwd, getegid, ioctl, fcntl, newstat, clone, wait4, rt_sigreturn, exit_group } } USE a DEFAULT KILL' -- /bin/sh -i
[2017-01-15T21:53:08+0100] Mode: STANDALONE_ONCE
[2017-01-15T21:53:08+0100] Jail parameters: hostname:'NSJAIL', chroot:'/', process:'/bin/sh', bind:[::]:0, max_conns_per_ip:0, uid:(ns:1000, global:1000), gid:(ns:1000, global:1000), time_limit:0, personality:0, daemonize:false, clone_newnet:true, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clonew_newuts:true, clone_newcgroup:false, keep_caps:false, tmpfs_size:4194304, disable_no_new_privs:false, pivot_root_only:false
[2017-01-15T21:53:08+0100] Mount point: src:'/' dst:'/' type:'' flags:0x5001 options:''
[2017-01-15T21:53:08+0100] Mount point: src:'(null)' dst:'/proc' type:'proc' flags:0x0 options:''
[2017-01-15T21:53:08+0100] PID: 18873 about to execute '/bin/sh' for [STANDALONE_MODE]
/bin/sh: 0: can't access tty; job control turned off
$ set
IFS='
'
OPTIND='1'
PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PPID='0'
PS1='$ '
PS2='> '
PS4='+ '
PWD='/'
$ id
Bad system call
$ exit
[2017-01-15T21:53:17+0100] PID: 18873 exited with status: 159, (PIDs left: 0)
```
2015-05-15 05:44:48 +08:00
### MORE INFO?
2016-12-11 03:47:20 +08:00
To see the options, simply type:
2015-05-15 08:30:53 +08:00
```
2016-12-11 03:47:20 +08:00
./nsjail --help
2015-05-15 08:30:53 +08:00
```
2016-12-11 00:17:01 +08:00
2016-12-11 03:47:20 +08:00
The commandline options should be reasonably well-documented
2016-12-11 00:17:01 +08:00
2015-05-16 07:55:25 +08:00
```
Usage: ./nsjail [options] -- path_to_command [args]
Options:
2016-03-02 10:10:08 +08:00
--help|-h
2015-05-16 07:55:25 +08:00
Help plz..
2016-06-19 20:32:27 +08:00
--mode|-M VALUE
2016-12-11 00:17:01 +08:00
Execution mode (default: o [MODE_STANDALONE_ONCE]):
2016-03-02 10:10:08 +08:00
l: Wait for connections on a TCP port (specified with --port) [MODE_LISTEN_TCP]
o: Immediately launch a single process on a console using clone/execve [MODE_STANDALONE_ONCE]
e: Immediately launch a single process on a console using execve [MODE_STANDALONE_EXECVE]
2015-05-16 07:55:25 +08:00
r: Immediately launch a single process on a console, keep doing it forever [MODE_STANDALONE_RERUN]
2016-06-19 20:32:27 +08:00
--chroot|-c VALUE
2016-12-11 00:17:01 +08:00
Directory containing / of the jail (default: none)
2016-03-02 10:10:08 +08:00
--rw
Mount / as RW (default: RO)
2016-06-19 20:32:27 +08:00
--user|-u VALUE
2016-03-02 10:10:08 +08:00
Username/uid of processess inside the jail (default: your current uid). You can also use inside_ns_uid:outside_ns_uid convention here
2016-06-19 20:32:27 +08:00
--group|-g VALUE
2016-03-02 10:10:08 +08:00
Groupname/gid of processess inside the jail (default: your current gid). You can also use inside_ns_gid:global_ns_gid convention here
2016-06-19 20:32:27 +08:00
--hostname|-H VALUE
2015-05-16 07:55:25 +08:00
UTS name (hostname) of the jail (default: 'NSJAIL')
2016-06-19 20:32:27 +08:00
--cwd|-D VALUE
2016-03-02 10:10:08 +08:00
Directory in the namespace the process will run (default: '/')
2016-06-19 20:32:27 +08:00
--port|-p VALUE
2016-12-11 00:17:01 +08:00
TCP port to bind to (enables MODE_LISTEN_TCP) (default: 0)
2016-06-19 20:32:27 +08:00
--bindhost VALUE
IP address port to bind to (only in [MODE_LISTEN_TCP]), '::ffff:127.0.0.1' for locahost (default: '::')
--max_conns_per_ip|-i VALUE
2015-05-16 07:55:25 +08:00
Maximum number of connections per one IP (default: 0 (unlimited))
2016-06-19 20:32:27 +08:00
--log|-l VALUE
2016-03-02 10:10:08 +08:00
Log file (default: /proc/self/fd/2)
2016-06-19 20:32:27 +08:00
--time_limit|-t VALUE
2015-07-08 04:17:44 +08:00
Maximum time that a jail can exist, in seconds (default: 600)
2016-03-02 10:10:08 +08:00
--daemon|-d
Daemonize after start
--verbose|-v
Verbose output
--keep_env|-e
Should all environment variables be passed to the child?
2016-06-19 20:32:27 +08:00
--env|-E VALUE
2016-03-02 10:10:08 +08:00
Environment variable (can be used multiple times)
--keep_caps
Don't drop capabilities (DANGEROUS)
--silent
Redirect child's fd:0/1/2 to /dev/null
--skip_setsid
Don't call setsid(), allows for terminal signal handling in the sandboxed process
2016-06-19 20:32:27 +08:00
--pass_fd VALUE
Don't close this FD before executing child (can be specified multiple times), by default: 0/1/2 are kept open
2016-12-11 00:17:01 +08:00
--pivot_root_only
Only perform pivot_root, no chroot. This will enable nested namespaces
--disable_no_new_privs
Don't set the prctl(NO_NEW_PRIVS, 1) (DANGEROUS)
2016-06-19 20:32:27 +08:00
--rlimit_as VALUE
2015-05-16 07:55:25 +08:00
RLIMIT_AS in MB, 'max' for RLIM_INFINITY, 'def' for the current value (default: 512)
2016-06-19 20:32:27 +08:00
--rlimit_core VALUE
2015-05-16 07:55:25 +08:00
RLIMIT_CORE in MB, 'max' for RLIM_INFINITY, 'def' for the current value (default: 0)
2016-06-19 20:32:27 +08:00
--rlimit_cpu VALUE
2015-05-16 07:55:25 +08:00
RLIMIT_CPU, 'max' for RLIM_INFINITY, 'def' for the current value (default: 600)
2016-06-19 20:32:27 +08:00
--rlimit_fsize VALUE
2015-05-16 07:55:25 +08:00
RLIMIT_FSIZE in MB, 'max' for RLIM_INFINITY, 'def' for the current value (default: 1)
2016-06-19 20:32:27 +08:00
--rlimit_nofile VALUE
2015-05-16 07:55:25 +08:00
RLIMIT_NOFILE, 'max' for RLIM_INFINITY, 'def' for the current value (default: 32)
2016-06-19 20:32:27 +08:00
--rlimit_nproc VALUE
2015-05-16 07:55:25 +08:00
RLIMIT_NPROC, 'max' for RLIM_INFINITY, 'def' for the current value (default: 'def')
2016-06-19 20:32:27 +08:00
--rlimit_stack VALUE
2015-05-16 07:55:25 +08:00
RLIMIT_STACK in MB, 'max' for RLIM_INFINITY, 'def' for the current value (default: 'def')
2016-03-02 10:10:08 +08:00
--persona_addr_compat_layout
personality(ADDR_COMPAT_LAYOUT)
--persona_mmap_page_zero
personality(MMAP_PAGE_ZERO)
--persona_read_implies_exec
personality(READ_IMPLIES_EXEC)
--persona_addr_limit_3gb
personality(ADDR_LIMIT_3GB)
--persona_addr_no_randomize
personality(ADDR_NO_RANDOMIZE)
--disable_clone_newnet|-N
Don't use CLONE_NEWNET. Enable networking inside the jail
--disable_clone_newuser
Don't use CLONE_NEWUSER. Requires euid==0
--disable_clone_newns
Don't use CLONE_NEWNS
--disable_clone_newpid
Don't use CLONE_NEWPID
--disable_clone_newipc
Don't use CLONE_NEWIPC
--disable_clone_newuts
Don't use CLONE_NEWUTS
2016-06-19 20:32:27 +08:00
--enable_clone_newcgroup
Use CLONE_NEWCGROUP
2016-12-11 00:17:01 +08:00
--uid_mapping|-U VALUE
Add a custom uid mapping of the form inside_uid:outside_uid:count. Setting this requires newuidmap to be present
--gid_mapping|-G VALUE
Add a custom gid mapping of the form inside_gid:outside_gid:count. Setting this requires newuidmap to be present
2016-06-19 20:32:27 +08:00
--bindmount_ro|-R VALUE
2016-03-02 10:10:08 +08:00
List of mountpoints to be mounted --bind (ro) inside the container. Can be specified multiple times. Supports 'source' syntax, or 'source:dest'
2016-06-19 20:32:27 +08:00
--bindmount|-B VALUE
2016-03-02 10:10:08 +08:00
List of mountpoints to be mounted --bind (rw) inside the container. Can be specified multiple times. Supports 'source' syntax, or 'source:dest'
2016-06-19 20:32:27 +08:00
--tmpfsmount|-T VALUE
2016-03-02 10:10:08 +08:00
List of mountpoints to be mounted as RW/tmpfs inside the container. Can be specified multiple times. Supports 'dest' syntax
2016-06-19 20:32:27 +08:00
--tmpfs_size VALUE
2016-03-02 10:10:08 +08:00
Number of bytes to allocate for tmpfsmounts (default: 4194304)
--disable_proc
Disable mounting /proc in the jail
2016-12-11 00:17:01 +08:00
--seccomp_policy|-P VALUE
Path to file containing seccomp-bpf policy (see kafel/)
--seccomp_string VALUE
String with kafel seccomp-bpf policy (see kafel/)
2016-06-21 03:54:05 +08:00
--cgroup_mem_max VALUE
Maximum number of bytes to use in the group (default: '0' - disabled)
2016-06-19 20:32:27 +08:00
--cgroup_mem_mount VALUE
2016-06-21 03:54:05 +08:00
Location of memory cgroup FS (default: '/sys/fs/cgroup/memory')
2016-06-20 01:41:11 +08:00
--cgroup_mem_parent VALUE
2016-06-21 03:54:05 +08:00
Which pre-existing memory cgroup to use as a parent (default: 'NSJAIL')
2016-03-02 10:10:08 +08:00
--iface_no_lo
Don't bring up the 'lo' interface
2016-06-19 20:32:27 +08:00
--iface|-I VALUE
Interface which will be cloned (MACVLAN) and put inside the subprocess' namespace as 'vs'
--iface_vs_ip VALUE
2016-03-02 10:10:08 +08:00
IP of the 'vs' interface
2016-06-19 20:32:27 +08:00
--iface_vs_nm VALUE
2016-03-02 10:10:08 +08:00
Netmask of the 'vs' interface
2016-06-19 20:32:27 +08:00
--iface_vs_gw VALUE
2016-03-02 10:10:08 +08:00
Default GW for the 'vs' interface
2016-06-19 20:32:27 +08:00
Examples:
Wait on a port 31337 for connections, and run /bin/sh
nsjail -Ml --port 31337 --chroot / -- /bin/sh -i
Re-run echo command as a sub-process
nsjail -Mr --chroot / -- /bin/echo "ABC"
Run echo command once only, as a sub-process
nsjail -Mo --chroot / -- /bin/echo "ABC"
2016-06-21 03:54:05 +08:00
Execute echo command directly, without a supervising process
nsjail -Me --chroot / --disable_proc -- /bin/echo "ABC"
2015-05-16 07:55:25 +08:00
```
2017-04-21 21:35:50 +08:00
### LAUNCHING IN DOCKER
To launch nsjail in a docker container clone the repository and build the docker image:
```bash
docker build . -t nsjail
```
This will build up an image containing njsail and kafel.
From now you can either use it in another Dockerfile (`FROM nsjail`) or directly:
```bash
docker run --rm -it nsjail nsjail --user 99999 --group 99999 --disable_proc --chroot / --time_limit 30 /bin/bash
```
