This PR integrates an Intel SGX feature called Intel Protection File System Library (IPFS)
into the runtime to create, operate and delete files inside the enclave, while guaranteeing
the confidentiality and integrity of the data persisted. IPFS can be referred to here:
https://www.intel.com/content/www/us/en/developer/articles/technical/overview-of-intel-protected-file-system-library-using-software-guard-extensions.html
Introduce a cmake variable `WAMR_BUILD_SGX_IPFS`, when enabled, the files interaction
API of WASI will leverage IPFS, instead of the regular POSIX OCALLs. The implementation
has been written with light changes to sgx platform layer, so all the security aspects
WAMR relies on are conserved.
In addition to this integration, the following changes have been made:
- The CI workflow has been adapted to test the compilation of the runtime and sample
with the flag `WAMR_BUILD_SGX_IPFS` set to true
- Introduction of a new sample that demonstrates the interaction of the files (called `file`),
- Documentation of this new feature
Fix the issue reported by #1484:
Platform ESP-IDF broken for WAMR 1.0.0 with ESP-IDF 4.4.2
Let the dummy ftruncate only work with ESP-IDF earlier than 4.4.2
Memory num_bytes_per_page was incorrectly set in memory enlarging for
shared memory, we fix it. And don't set memory_data_size again for shared
memory.
Implement more socket APIs, refer to #1336 and below PRs:
- Implement wasi_addr_resolve function (#1319)
- Fix socket-api byte order issue when host/network order are the same (#1327)
- Enhance sock_addr_local syscall (#1320)
- Implement sock_addr_remote syscall (#1360)
- Add support for IPv6 in WAMR (#1411)
- Implement ns lookup allowlist (#1420)
- Implement sock_send_to and sock_recv_from system calls (#1457)
- Added http downloader and multicast socket options (#1467)
- Fix `bind()` calls to receive the correct size of `sockaddr` structure (#1490)
- Assert on correct parameters (#1505)
- Copy only received bytes from socket recv buffer into the app buffer (#1497)
Co-authored-by: Marcin Kolny <mkolny@amazon.com>
Co-authored-by: Marcin Kolny <marcin.kolny@gmail.com>
Co-authored-by: Callum Macmillan <callumimacmillan@gmail.com>
Fix two issues of building WAMR on Windows:
- The build_llvm.py script calls itself, spawning instances faster than they expire,
which makes Python3 eat up the entire RAM in a pretty short time.
- The MSVC compiler doesn't support preprocessor statements inside macro expressions.
Two places inside bh_assert() were found.
If WASM app has called pthread_detach() to detach a thread, it will be detached again
when thread exits. Attempting to detach an already detached thread may result in crash
in musl-libc. This patch fixes it.
And enable classic interpreter instead fast interpreter when llvm jit is enabled,
so as to fix the issue that llvm jit cannot handle opcode drop_64/select_64.
Remove handling opcode DROP_64/SELECT_64 in loader stage
prepare_bytecode, as they are the modified opcodes of DROP/SELECT
for optimization purpose, but not the opcodes defined by spec.
Add more checks for Fast JIT to fix the issues reported by instrument test:
- add check for the jit_value before pushing it into the stack
- add check at the end of form_and_translate_func
- add checks after each jit pass
Fix multi-module issue:
don't call the sub module's function with "$sub_module_name$func_name"
Fix the aot_call_function free argv1 issue
Modify some API comments in wasm_export.h
Fix the wamrc help info
Some configurations (eg. esp32/nuttx) have limited space for BSS,
0x20000 byte buffer is huge on embedded systems, change to
allocate the buffer dynamically.
Destroy Fast-JIT compiler after destroying the modules loaded by
multi-module feature, since the Fast JIT's code cache allocator may
be used by these modules. If the Fast JIT's code cache allocator was
destroyed, then runtime will fail to destroy these modules.
And fix the issue of destroying import module's memory instance.
Use the semantic versioning (https://semver.org) to replace the current date
versioning system, which is more general and is requested by some developers,
e.g. issue #1357.
There are three parts in the new version string:
- major. Any incompatible modification on ABIs and APIs will lead to an increment
in the value of major, which mainly includes: AOT calling conventions, AOT file
format, wasm_export.h, wasm_c_api.h, and so on.
- minor. It represents new features, including MVP/POST-MVP features, libraries,
WAMR private ones, and so one.
- patch. It represents patches.
The new version will start from 1.0.0. Update the help info and version showing for
iwasm and wamrc.
Normalize wasm types, for the two wasm types, if their parameter types
and result types are the same, we only save one copy, so as to reduce
the footprint and simplify the type comparison in opcode CALL_INDIRECT.
And fix issue in interpreter globals_instantiate, and remove used codes.
esp-idf: Make esp-idf support Libc WASI
1. Support to get WASM APP libs' DIR from upper layer
2. Add SSP support for esp-idf platform
3. Change the errno of readlinkat
Reserve one pointer size for fast-interp code_compiled_size: if the last opcode of
current function is to be dropped (e.g. OP_DROP), the peak memory usage will
be larger than the final code_compiled_size, we record the peak size to ensure
there won't be invalid memory access during the second traversing.
Remove some unused fields in module instance and the related codes,
which are introduced by emsdk some special mode (-DSIDE_MODULE=1),
and are not required now.
Add a new option WAMR_BUILD_STACK_GUARD_SIZE to set the custom
stack guard size. For most RTOS systems, we use the native stack base
address as the check boundary which may be not safe as POSIX based
systems (like Linux).
Thread data should not be destroyed when thread exits, or other thread
may not be able to join it. This PR saves the thread data into thread data
list when thread exits, sets thread status and stores the return value, so
that other thread can join it.
Also set MEM_TOP_DOWN flag for Windows VirtualAlloc to yield LLVM
JIT relocation error.
And set opt/size level to 3 for LLVM JIT for future use, currently the flags
are not used by LLVM JIT.
ASSERT_NOT_IMPLEMENTED is bh_assert, which might be no-op.
in that case, it's better to fall back to the "default" case,
which reports an error properly.
Should not clear last label's polymorphic state after current label is popped
Fix invalid func_idx check in opcode REF_FUNC
Add check when there are extra unneeded bytecodes for a wasm function
Fix dump call stack issue in interpreter introduced by hw bound check:
the call stack isn't dumped if the exception is thrown and caught by
signal handler.
And restore the wasm stack frame to the original status after calling a
wasm function.
Lookup table for i32.const and i64.const for xtensa XIP
Lookup const offset from table for load/store opcodes for xtensa XIP
Fill capability flags for xtensa XIP
Enable lower switch pass for xtensa XIP
Import WAMR Fast JIT which is a lightweight JIT with quick startup, small footprint,
relatively good performance (~40% to ~50% of LLVM JIT) and good portability.
Platforms supported: Linux, MacOS and Linux SGX.
Arch supported: x86-64.
Since legacy binding for loop unswitch pass was removed and we can't get
it back. Implement its equivalent in `aot_llvm_extra.cpp` and use it in
`aot_compiler.c`.
Follow up to #1183.
Before resolving the module function's export in wasm_mini_loader,
"module->retain_function" need to be initialized, otherwise,
the "__new" function export will lead to abort.
issue: https://github.com/bytecodealliance/wasm-micro-runtime/issues/1332
Co-authored-by: yaozhongxiao <yaozhongxiao@bytedance.com>
Fix build script to enable hw bound check for interpreter when
AOT is disabled, so as to enable spec cases test for interp with
hw bound check. And fix the issues found.
Implement boundary check with hardware trap for interpreter on
64-bit platforms:
- To improve the performance of interpreter and Fast JIT
- To prepare for multi-tier compilation for the feature
Linux/MacOS/Windows 64-bit are enabled.
The existing validation didn't work as expected; e.g. for address
pool: 8.8.8.8/24 the application had access to the 127.0.0.1 address (which
should not).
Fix the issue reported in #1282.
When i32/i64 rotate (rotl/rotr) with 0, the LLVM IRs translated are:
left<<0 | left>>64 and left >>0 | left<<64
The value of left >> 64 and left <<64 in LLVM are treated as poison,
which causes invalid result when executing the aot function.
Directly return left when right is 0 to fix the issue.
Sub module's auxiliary stack boundary and bottom may be different from
main module's counterpart, so when calling sub module, its aux stack info
should be gotten and set to exec_env firstly, or aux stack overflow and out
of bounds memory access exception may be thrown when calling sub
module's function.
Fix the issue reported in PR #1278.
Enhance the hw bound check reported in #1262:
When registering signal handlers for SIGSEGV & SIGBUS in boundary
check with hardware trap, preserve the previous handlers for signal
SIGSEGV and SIGBUS, and forward the signal to the preserved signal
handlers if it isn't handled by hw bound check.
Enable aot compiler and jit based on llvm-14.0 and llvm-15.0git,
replace LLVMBuildLoad/LLVMBuildInBoundsGEP/LLVMBuildCall with
LLVMBuildLoad2/LLVMBuildInBoundsGEP2/LLVMBuildCall2, and pass
them with related types, so as to meet the requirements of opaque
pointers.
And fix several compilation errors for llvm-14.0/15.0git.
Most spec cases and standalone cases are tested.
Fix the following warning introduced by the recent change.
```
In file included from wamr/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:17,
from wamr/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.h:18,
from wamr/core/iwasm/aot/../common/wasm_runtime_common.h:18,
from wamr/core/iwasm/aot/aot_runtime.h:10,
from wamr/core/iwasm/aot/aot_loader.c:6:
wamr/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/ssp_config.h:55: error: "CONFIG_HAS_ISATTY" redefined [-Werror]
#define CONFIG_HAS_ISATTY 1
In file included from wamr/core/shared/utils/../platform/include/platform_common.h:13,
from wamr/core/shared/utils/bh_platform.h:9,
from wamr/core/iwasm/aot/aot_runtime.h:9,
from wamr/core/iwasm/aot/aot_loader.c:6:
wamr/core/shared/platform/nuttx/platform_internal.h:75: note: this is the location of the previous definition
#define CONFIG_HAS_ISATTY 0
```
When using clang compiler, the f32/f64 return value might be
invalid when calling invokeNative asm code. Declare the return
type of invokeNative as void, and set volatile for the converted
function pointers to resolve the issue.
Enable dump call stack to a buffer, use API
`wasm_runtime_get_call_stack_buf_size` to get the required buffer size
and use API
`wasm_runtime_dump_call_stack_to_buf` to dump call stack to a buffer
Support integrating 3rd-party toolchain llc compiler or asm compiler
into wamrc by setting environment variable WAMRC_LLC_COMPILER
or WAMRC_ASM_COMPILER, wamrc will use these tools to generate
object file from LLVM IR firstly, and then refactor the object file into
aot file.
Implement Go binding APIs of runtime, module and instance
Add sample, build scripts and update the document
Co-authored-by: venus-taibai <97893654+venus-taibai@users.noreply.github.com>
This header file is supposed to be used by user code, which is not
a part of WAMR. Usually WAMR configuration is not available there,
remove DEBUG_INTERP macro control in it.
Re-implement socket api send/recv in an atomic-like way, fix the return value
check in posix.c.
And fix wasm-c-api sample callback_chain calling malloc issue.
Automatically dump memory/performance profiling data in
wasm_application_execute_main and wasm_application_execute_func when
the related feature is enabled.
And remove unused aot_compile_wasm_file func declaration in aot_compiler.h.
wasm_c_api.c: add more checks, fix LOG_WARNING invalid specifier
aot_emit_aot_file: fix strncpy max size length to copy
posix.c: fix potential socket not close issue
wasm-c-api samples: add return value checks for fseek/ftell
cJSON.c: remove dead code
module_wasm_app.c: add return value check for wasm_runtime_call_wasm
aot_runtime.c: add return value check for aot_get_default_memory
aot_runtime.c: add return value check before calling wasm app malloc/free func
wasm_runtime_common.c: fix dead code warning in wasm_runtime_load_from_sections
aot_emit_memory.c: fix potential integer overflow issue
wasm_runtime.c: remove dead code in memory_instantiate, add assertion for globals
samples simple/gui/littlevgl: fix fields of struct sigaction initialization issue
host-tool: add return value check for sendto
runtime_sensor.c: add return value check for os_mutex_init
fix find_sensor_client
sensor_mgr_ref.c: add return value check for init_sensor_framework
app_manager_host.c: add return value check for app_manager_host_init
module_wasm_app.c: add bh_assert for m_data
fix mkdir potential issue
sample littlevgl/gui/simple: add return value check for init_sensor_framework
host_tool: add more check for g_conn_fd
Fix the symbol resolving failure with recent version of wamrc:
```
AOT module load failed: resolve symbol .Lswitch.table.aot _func#82.2 failed
```
Replace the relocations for such symbols with .rodata section.
Add assertion for BH_MALLOC/BH_FREE in wasm_runtime_common.c,
when building runtime, the BH_MALLOC/BH_FREE macros should be
defined as wasm_runtime_malloc/wasm_runtime_free.
Fix allocate zero size memory warning reported by wasm_runtime_malloc
when allocating the import fun pointers if the import func count is 0:
`warning: wasm_runtime_malloc with size zero`
Fix wasm_application_execute_main/wasm_application_execute_func not waiting for
other threads to terminate in multi-thread mode, which causes that the exception
thrown by other threads may haven't been spreaded to current main thread, and
cannot be detected by the caller, as reported in #1131.
Refine opcode br_table for classic interpreter as there may be a lot of
leb128 decoding when the br count is big:
1. Use the bytecode itself to store the decoded leb br depths if each
decoded depth can be stored with one byte
2. Create br_table cache to store the decode leb br depths if the decoded
depth cannot be stored with one byte
After the optimization, the class interpreter can access the br depths array
with index, no need to decode the leb128 again.
And fix function record_fast_op() return value unchecked issue in source
debugging feature.
Fix fault modification on data segment in wasm module which leads to
fail to instantiate wasm module next time, reported by #1115.
Co-authored-by: yangwenming <yangwenming@bytedance.com>
Fix an UBSan complaint introduced by recent change by adding more checks
to word_copy:
```
wasm_interp_fast.c:792:9: runtime error: applying zero offset to null pointer
```
Fix the following warning when loading an aot file without relocations:
```
[20:19:00:528 - 1119F1600]: warning: wasm_runtime_malloc with size zero
```
Fix issues in PR "Refine interp/aot string storage and emitting (#820)",
which had a few issues:
- It looks a wrong byte to mark the flag
- It doesn't work for long strings (>= 0x80 in case of little endian)
This commit fixes them by maintaining a list of loaded symbols while loading
relocation section to avoid reading a string repeatedly, and no need to mark
the flag again.
Add aot relocation for ".rodata.str" symbol to support more cases
Fix some coding style issues
Fix aot block/value stack destroy issue
Refine classic/fast interpreter codes
Clear compile warning of libc_builtin_wrapper.c in 32-bit platform
Fix compile warnings in libc-wasi posix.c:
```
posix.c:880:41: warning: comparison of integers of different signs:
'unsigned long' and 'ssize_t' (aka 'long') [-Wsign-compare]
if (bufoff + iov[i].buf_len < len) {
posix.c:1359:32: warning: comparison of integers of different signs:
'off_t' (aka 'long long') and 'unsigned long long' [-Wsign-compare]
if (ret == 0 && sb.st_size < offset + len)
```
The format specifiers of %ld/%lu/%li/%lx/%lp/%lo (or %z* and %t* like) should
be treated as 32-bit integer in wasm.
And merge the handle of unsigned/signed for %u/%d and %llu/%lld specifier
when calling snprintf.
During debugging, the debug client may request to malloc a memory space
to evaluate the user expressions. If we malloc memory from the linear memory,
it may fail when the thread is in stop status. We preserve a buffer during
creating debug instance, and use a simple bump pointer allocator to serve lldb's
memory request.
Fix the potential integer overflow of const index in const space of fast interpreter,
emit i32/i64.const opcode when the const index is larger than INT32_MAX.
And add check for the function local cell num.
Implement Berkeley Socket API for Intel SGX
- bring Berkeley socket API in Intel SGX enclaves,
- adapt the documentation of the socket API to mention Intel SGX enclaves,
- adapt _iwasm_ in the mini-product _linux-sgx_ to support the same option as the one for _linux_,
- tested on the socket sample as provided by WAMR (the TCP client/server).
Fix handle OP_TABLE_COPY issue
Fix loader handle OP_BLOCK/IF/LOOP issue if type_index is larger than 256
Fix loader handle OP_GET_GLOBAL, allow to change to GET_GLOBAL_64 for
aot compiler similiar to handling OP_SET_GLOBAL
Refine loader handle OP_GET/SET/TEE_LOCAL, disable changing opcode when
source debugging is enabled, so as no need to record the change of opcode
Refine wasm_interp_interp_frame_size to reduce the wasm operand stack usage
Signed-off-by: Wenyong Huang <wenyong.huang@intel.com>
Remove the `const` flag for the first argument `buf` of wasm_runtime_load as
it might be modified by runtime for footprint and performance purpose, and
update the related functions and document.
This patch allows safer (note: safer, not safe) embedding in a plugin
environment where multiple instances of the engine could be needed.
Original code initializes and tears down the full runtime during
wasm_engine_new() and wasm_engine_delete() respectively. After this
update the C API implementation keeps track of engine instances count
and inits/deinits the runtime only when needed.
This allows for example to call wasm_engine_new() twice and then call
wasm_engine_delete() once without rendering the first engine instance
invalid.
In thread_manager.c, `wasm_cluster_spawn_exec_env` creates a new module
instance but not sets wasi_ctx, so when the new exec_env calls wasm function
which uses WASI API, the WASI API functions in `libc_wasi_wrapper.c` will get
null result in calling `get_wasi_ctx` and then return `wasi_errno`.
Signed-off-by: HangedFish <bravohangedman@outlook.com>
Refer to [Networking API design](https://github.com/WebAssembly/WASI/issues/370)
and [feat(socket): berkeley socket API v2](https://github.com/WebAssembly/WASI/pull/459):
- Support the socket API of synchronous mode, including `socket/bind/listen/accept/send/recv/close/shutdown`,
the asynchronous mode isn't supported yet.
- Support adding `--addr-pool=<pool1,pool2,..>` argument for command line to identify the valid ip address range
- Add socket-api sample and update the document
Fix several issues in wasm-c-api wasm_module_imports function:
1. Two of the if branches never set the module_name and name fields which are later passed as arguments to wasm_importtype_new, and eventually might cause double-free and/or use-after-free
2. Should zero module_name/name/extern_type at the start of loop iteration, and destroy their resources when failed at the end of loop iteration
2. No need to check `if (!extern_type) { continue; }`, as extern_type is converted from type and type is already checked
3. No need to wasm_importtype_vec_delete(out) when failed, as it is passed from outside and should be destroyed by outside
When calling native function from AOT code, current implementation is to return
back to runtime to call aot_invoke_native, which calls wasm_runtime_invoke_native
and the latter calls assembly code. We did it before as there may be pointer and
string arguments to check and convert if the native function's registered signature
has character '*' and '$'.
As the built-in native function's signatures can be gotten in compilation time, we
check the pointer/string arguments and convert them into native address in AOT
code, and then invoke the native function directly, so as to improve performance.
In some Linux systems whose kernel version is smaller than 2.6.38, the macro
MADV_HUGEPAGE isn't introduced yet which causes compilation error.
Add macro control to fix the compilation error.
The littlevgl library had changed its name and domain to [LVGL](https://lvgl.io).
See https://blog.lvgl.io/2020-06-01/announcement
We change some names and links accordingly.
Also remove the cloning for tlsf library as it isn't used now.
Increase default/min native stack size when UVWASI is enabled as
UVWASI requires larger native stack size.
Increase the reserved bytes to the native thread stack boundary to
better detect the native stack overflow.
Set WASM_DISABLE_HW_BOUND_CHECK to 0 when interpreter is
enabled and AOT is disabled, as memory access boundary check
with hardware trap is only enabled in AOT/JIT mode.
Allow compilation on Windows MinGW, see build_wamr.md for more details.
Note that WASI and some other smallish details are still not supported, but
we have a starting point. See more discussion at #993
Implement pthread_cond_broadcast wrapper for lib-pthread
- support pthread_cond_broadcast wrapper for posix/linux-sgx/windows
- update document for building multi-thread wasm app with emcc
Use LLVM new pass manager for wamrc to replace the legacy pass manger,
so as to gain better performance and reduce the compilation time.
Reference links:
- https://llvm.org/docs/NewPassManager.html
- https://blog.llvm.org/posts/2021-03-26-the-new-pass-manager
And add an option to use the legacy pm mode when building wamrc:
cmake .. -DWAMR_BUILD_LLVM_LEGACY_PM=1
For JIT mode, keep it unchanged as it only runs several function passes and
using new pass manager will increase the compilation time.
And refactor the codes of applying LLVM passes.
Refactor LLVM Orc JIT to actually enable the lazy compilation and speedup
the launching process:
https://llvm.org/docs/ORCv2.html#laziness
Main modifications:
- Create LLVM module for each wasm function, wrap it with thread safe module
so that the modules can be compiled parallelly
- Lookup function from aot module instance's func_ptrs but not directly call the
function to decouple the module relationship
- Compile the function when it is first called and hasn't been compiled
- Create threads to pre-compile the WASM functions parallelly when loading
- Set Lazy JIT as default, update document and build/test scripts
Currently when calling wasm_runtime_call_wasm() to invoke wasm function
with externref type argument from runtime embedder, developer needs to
use wasm_externref_obj2ref() to convert externref obj into an internal ref
index firstly, which is not convenient to developer.
To align with GC feature in which all the references passed to
wasm_runtime_call_wasm() can be object pointers directly, we change the
interface of wasm_runtime_call_wasm() to allow to pass object pointer
directly for the externref argument, and refactor the related codes, update
the related samples and the document.
Add log info for heap and stack like wasm loader in aot loader
Signed-off-by: Huang Qi <huangqi3@xiaomi.com>
Change-Id: I349848d75f1a26cde29217c14cfb6e779c976a8b
Refine is_xip_file check, when e_type isn't E_TYPE_XIP, just return false
and no need to go through all the other sections of the AOT file.
Refine pointer range check, convert pointer to uintptr_t type before
comparison to yield possible sanitizer pointer overflow error.
Fix __wasi_subscription_t structure definition inconsistent with wasi-libc
definition issue, reported by #961, tested with sleep, poll API and other
wasi cases on x86-64, x86-32 and arm32 targets.
The return address of pthread_get_stackaddr_np() in MacOS and NuttX
may be the base address or the end (boundary) address of the native stack,
if it is the end address, we get the base address according to it and the
stack size, so as to get the actual stack boundary address correctly.
Signed-off-by: Huang Qi <huangqi3@xiaomi.com>
Fix the issue that joining a detached thread might result in joining hang,
resolve the issue by adding wait_count for a thread's exec_env to indicate
whether a thread needs to detach itself or not when it exits.
And add checks for the input exec_env for cluster's join/detach/cancel thread.
