This reverts commit 788bbf4f261fc558b714bdedd4122d7115efc940.
Reason for revert: fixes a conflict with upcoming upstream BPF changes.
Bug: 145210207
Change-Id: I3bbc1279fc613be0d2e833008413ad3561b851df
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This reverts commit d9756ba082c13d928d6904bc3741a1c08b4bd783.
Reason for revert: fixes a conflict with upcoming upstream BPF changes.
Bug: 145210207
Change-Id: I0e7e76c117ab3608b6dd5a1bc6b949b9e109038f
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
BPF dispatcher functions are patched at runtime to perform direct
instead of indirect calls. Disable CFI for the dispatcher functions
to avoid conflicts.
Bug: 145210207
Change-Id: Iea72f5a9fe09dd5adbb90b0174945707f42594b0
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
BPF dispatcher functions are patched at runtime to perform direct
instead of indirect calls. Disable CFI for the dispatcher functions
to avoid conflicts.
(re-add due to merge conflicts previously)
Bug: 145210207
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I7edf6052e121d16ccb0f3d3492ff4eefedfa509e
With CFI, the compiler replaces function references with pointers
to the CFI jump table. This breaks passing these addresses to
code running at EL2, where the jump tables are not valid. Add a
__va_function macro similarly to the earlier __pa_function to take
address of the actual function in inline assembly and use that in
kvm_ksym_ref instead.
Bug: 163385976
Change-Id: I097b99409995512c00786300e7d18fe42c720a1b
(cherry picked from commit 2f4d6c9fd77c88ad0500aad4bf1f64aaf2654c49)
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
After the switch to non-canonical CFI jump tables, the jump table
sections were placed after the .text section. Merge these sections
into .text to fix issues with error injection and kallsyms.
Bug: 225079388
Bug: 190422440
Change-Id: I6c81b3e4dbba62739f7fc5f6b45271c54f278c8f
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
[nathan: This change is necessary in android-4.19 to boot after upstream
LLVM commit 7b346357db30 ("[ELF] Orphan placement: prefer the
last similar section when its rank <= orphan's rank"), which
changes how sections not described in linker scripts are laid
out in the final binary]
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
commit b744b43f79cc758127042e71f9ad7b1afda30f84 upstream.
Similarly to the CC_IS_CLANG config, add LD_IS_LLD to avoid GNU ld
specific logic such as ld-version or ld-ifversion and gain the
ability to select potential features that depend on the linker at
configuration time such as LTO.
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Masahiro Yamada <masahiroy@kernel.org>
[nc: Reword commit message]
Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Reviewed-by: Sedat Dilek <sedat.dilek@gmail.com>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
e820__mapped_all() is passed as a callback to is_mmconf_reserved(),
which expects a function of type:
typedef bool (*check_reserved_t)(u64 start, u64 end, unsigned type);
However, e820__mapped_all() accepts enum e820_type as the last argument
and this type mismatch trips indirect call checking with Clang's
Control-Flow Integrity (CFI).
As is_mmconf_reserved() only passes enum e820_type values for the
type argument, change the typedef and the unused type argument in
is_acpi_reserved() to enum e820_type to fix the type mismatch.
Bug: 210705746
(cherry picked from commit 83321c335dccba262a57378361d63da96b8166d6)
Reported-by: Sedat Dilek <sedat.dilek@gmail.com>
Suggested-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20201130193900.456726-1-samitolvanen@google.com
Change-Id: I92f78710ac55d5f567ad73053fc9f8c7cef68f71
(cherry picked from commit 7d1d5848183bd1d9086d0572f9af431d3ded407f)
(cherry picked from commit 8f54ba9ae7e38e315df32f5338665a1e44cd1a4b)
After LLVM commit rG8f8b5e5587c3 ("[MC] Error for .globl/.local which
change the symbol binding and warn for .weak"), using cond_syscall()
for IA32 compat syscalls produces the following errors with LTO:
error: __ia32_compat_sys_sysctl changed binding to STB_GLOBAL
Avoid conflicting .weak/.globl statements by using a weak function for
the stub instead.
Bug: 180726982
Link: https://github.com/ClangBuiltLinux/linux/issues/1269
Change-Id: I52446c153155f40cefecf6230cbc0d229de02041
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
commit e0d5896bd356cd577f9710a02d7a474cdf58426b upstream.
Unlike gcc, clang considers each inline assembly block to be independent
and therefore, when using the integrated assembler for inline assembly,
any preambles that enable features must be repeated in each block.
This change defines __LSE_PREAMBLE and adds it to each inline assembly
block that has LSE instructions, which allows them to be compiled also
with clang's assembler.
Link: https://github.com/ClangBuiltLinux/linux/issues/671
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Tested-by: Andrew Murray <andrew.murray@arm.com>
Tested-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Andrew Murray <andrew.murray@arm.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
[nd: backport adjusted due to missing:
commit addfc38672c7 ("arm64: atomics: avoid out-of-line ll/sc atomics")]
Signed-off-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
After LLVM rG9e33c096476a ("[ELF] Keep orphan section names
(.rodata.foo .text.foo) unchanged if !hasSectionsCommand"), LLD splits
compound literals to separate sections with -fdata-sections, which
it always enables with LTO. Merge these sections to allow LLVM to be
upgraded.
Bug: 174047799
Change-Id: I858c5fcc48283d5528c1d3e2b2d7a3c72f9e03d6
Link: https://github.com/ClangBuiltLinux/linux/issues/958
Suggested-by: Danny Lin <danny@kdrag0n.dev>
Suggested-by: Fangrui Song <maskray@google.com>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
[ Upstream commit 966a0acce2fca776391823381dba95c40e03c339 ]
Commit f7b93d42945c ("arm64/alternatives: use subsections for replacement
sequences") breaks LLVM's integrated assembler, because due to its
one-pass design, it cannot compute instruction sequence lengths before the
layout for the subsection has been finalized. This change fixes the build
by moving the .org directives inside the subsection, so they are processed
after the subsection layout is known.
Fixes: f7b93d42945c ("arm64/alternatives: use subsections for replacement sequences")
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Link: https://github.com/ClangBuiltLinux/linux/issues/1078
Link: https://lore.kernel.org/r/20200730153701.3892953-1-samitolvanen@google.com
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Merge the following new data sections generated by current
LLVM versions:
.bss..L* .bss..compoundliteral*
.data..L* .data..compoundliteral*
.rodata..L* .rodata..compoundliteral*
Also drop the unnecessary rules to merge .rela sections.
Bug: 158605670
Change-Id: I098212bc74dfb6b54d4fd60d90957fea29122a6c
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Instead of disabling LTO for all the code in the directory, only
disable it for the VDSO itself.
Bug: 145297900
Change-Id: I95a9c68ee76fdc8f4cee5c5bd1356fc90de1680c
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
(cherry picked from commit 74d07ccc220bb807a590c7a260fa8f388ccd2af7)
Signed-off-by: Mark Salyzyn <salyzyn@google.com>
Bug: 154668398
On modules with no executable code, LLVM generates a __cfi_check stub,
but won't align it to page size as expected. This change ensures the
function is at the beginning of the .text section and correctly aligned
for the CFI shadow.
Bug: 148458318
Change-Id: I85ea31fa851bc23988f649b021b3ac7e9d9dcb38
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Sections with double dots, e.g. .data..percpu are named intentionally
to avoid matching rules that apply to .section.*. Change module section
merging rules to skip these.
Bug: 151981957
Change-Id: I23787aa40d69da1c6ca622a5f111704d2459e163
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
KBUILD_LDS_MODULE didn't exist in 4.19, it was added in upstream commit
10df06385582 ("kbuild: rebuild modules when module linker scripts are
updated"), which means the module-lto.lds linker script is not actually
passed to the linker. Append the flags directly to KBUILD_LDFLAGS_MODULE
instead.
Bug: 151700304
Fixes: 6cea04778e ("ANDROID: kbuild: merge module sections with LTO")
Change-Id: I600db54d2ff9cd4e287913e8ddd463a20741a4a3
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With LTO, modules with a large number of compilation units maybe end
up exceeding the for loop argument list in the shell. Reduce the
probability for this happening by including only the modules that have
exported symbols.
Bug: 150234396
Change-Id: I4a289aff47e1444aca28d1bd00b125628f39bcd5
Suggested-by: Hsiu-Chang Chen <hsiuchangchen@google.com>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With non-canonical CFI, LLVM generates jump table entries for external
symbols in modules and as a result, a function pointer passed from a
module to the core kernel will have a different address.
Disable the warning for now.
Bug: 145210207
Change-Id: Ifdcee3479280f7b97abdee6b4c746f447e0944e6
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Signed-off-by: Alistair Delva <adelva@google.com>
[ Upstream commit c54f90c2627cc316d365e3073614731e17dbc631 ]
LLVM's integrated assembler fails with the following error when
building KVM:
<inline asm>:12:6: error: expected absolute expression
.if kvm_update_va_mask == 0
^
<inline asm>:21:6: error: expected absolute expression
.if kvm_update_va_mask == 0
^
<inline asm>:24:2: error: unrecognized instruction mnemonic
NOT_AN_INSTRUCTION
^
LLVM ERROR: Error parsing inline asm
These errors come from ALTERNATIVE_CB and __ALTERNATIVE_CFG,
which test for the existence of the callback parameter in inline
assembly using the following expression:
" .if " __stringify(cb) " == 0\n"
This works with GNU as, but isn't supported by LLVM. This change
splits __ALTERNATIVE_CFG and ALTINSTR_ENTRY into separate macros
to fix the LLVM build.
Link: https://github.com/ClangBuiltLinux/linux/issues/472
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
With CONFIG_THINLTO and CFI both enabled, LLVM appends a hash to the
names of all static functions. This breaks userspace tools, so strip
out the hash from output.
Bug: 147422318
Change-Id: Ibea6be089d530e92dcd191481cb02549041203f6
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Allow CFI enabled entry code to make indirect calls by also mapping
CFI jump tables, and add a check to ensure the jump table section is
not empty.
Bug: 145297900
Change-Id: I1204c50a139ba62234f3bb4699c50921a831162b
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Also ignore these relocations when loading modules.
Bug: 145297900
Change-Id: I3d00f93deb3cf370ce25f8672f90e410e87ab79e
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Leaf changes summary: 20 artifacts changed
Changed leaf types summary: 8 leaf types changed
Removed/Changed/Added functions summary: 0 Removed, 3 Changed, 5 Added functions
Removed/Changed/Added variables summary: 0 Removed, 0 Changed, 1 Added variable
3 functions with some sub-type change:
5 Added functions:
[A] 'function void __cfi_slowpath(uint64_t, void*, void*)'
[A] 'function void __ubsan_handle_cfi_check_fail_abort(void*, void*, void*)'
[A] 'function bool arch_bpf_jit_check_func(const bpf_prog*)'
[A] 'function void cfi_module_add(module*, unsigned long int, unsigned long int)'
[A] 'function void cfi_module_remove(module*, unsigned long int, unsigned long int)'
'enum zone_stat_item at mmzone.h:132:1' changed:
type size hasn't changed
1 enumerator insertion:
'zone_stat_item::NR_KERNEL_SCS_BYTES' value '10'
4 enumerator changes:
'zone_stat_item::NR_BOUNCE' from value '10' to '11' at mmzone.h:132:1
'zone_stat_item::NR_ZSPAGES' from value '11' to '12' at mmzone.h:132:1
'zone_stat_item::NR_FREE_CMA_PAGES' from value '12' to '13' at mmzone.h:132:1
'zone_stat_item::NR_VM_ZONE_STAT_ITEMS' from value '13' to '14' at mmzone.h:132:1
6 impacted interfaces
'struct kernel_symbol at export.h:61:1' changed:
type size changed from 64 to 128 (in bits)
1 data member deletion:
'int kernel_symbol::name_offset', at offset 32 (in bits) at export.h:63:1
1 data member insertion:
'const char* kernel_symbol::name', at offset 64 (in bits) at export.h:73:1
there are data member changes:
type 'int' of 'kernel_symbol::value_offset' changed:
type name changed from 'int' to 'unsigned long int'
type size changed from 32 to 64 (in bits)
name of 'kernel_symbol::value_offset' changed to 'kernel_symbol::value' at export.h:72:1, size changed from 32 to 64 (in bits) (by +32 bits)
6889 impacted interfaces
'struct module at module.h:331:1' changed:
type size hasn't changed
1 data member insertion:
'cfi_check_fn module::cfi_check', at offset 1856 (in bits) at module.h:353:1
there are data member changes:
'mutex module::param_lock' offset changed from 1856 to 1920 (in bits) (by +64 bits)
'kernel_param* module::kp' offset changed from 2112 to 2176 (in bits) (by +64 bits)
'unsigned int module::num_kp' offset changed from 2176 to 2240 (in bits) (by +64 bits)
'unsigned int module::num_gpl_syms' offset changed from 2208 to 2272 (in bits) (by +64 bits)
'const kernel_symbol* module::gpl_syms' offset changed from 2240 to 2304 (in bits) (by +64 bits)
'const s32* module::gpl_crcs' offset changed from 2304 to 2368 (in bits) (by +64 bits)
'bool module::sig_ok' offset changed from 2368 to 2432 (in bits) (by +64 bits)
'bool module::async_probe_requested' offset changed from 2376 to 2440 (in bits) (by +64 bits)
'const kernel_symbol* module::gpl_future_syms' offset changed from 2432 to 2496 (in bits) (by +64 bits)
'const s32* module::gpl_future_crcs' offset changed from 2496 to 2560 (in bits) (by +64 bits)
'unsigned int module::num_gpl_future_syms' offset changed from 2560 to 2624 (in bits) (by +64 bits)
'unsigned int module::num_exentries' offset changed from 2592 to 2656 (in bits) (by +64 bits)
'exception_table_entry* module::extable' offset changed from 2624 to 2688 (in bits) (by +64 bits)
'void ()* module::init' offset changed from 2688 to 2752 (in bits) (by +64 bits)
type 'tracepoint_ptr_t*' of 'module::tracepoints_ptrs' changed:
pointer type changed from: 'tracepoint_ptr_t*' to: 'tracepoint_ptr_t*'
6889 impacted interfaces
'struct per_cpu_pageset at mmzone.h:287:1' changed:
type size hasn't changed
there are data member changes:
type 's8[13]' of 'per_cpu_pageset::vm_stat_diff' changed:
type name changed from 's8[13]' to 's8[14]'
array type size changed from 104 to 112
array type subrange 1 changed length from 13 to 14
, size changed from 104 to 112 (in bits) (by +8 bits)
778 impacted interfaces
'struct pglist_data at mmzone.h:628:1' changed:
type size hasn't changed
there are data member changes:
778 impacted interfaces
'struct task_struct at sched.h:584:1' changed:
type size hasn't changed
there are data member changes:
type 'struct thread_info' of 'task_struct::thread_info' changed:
type size changed from 192 to 256 (in bits)
1 data member insertion:
'void* thread_info::shadow_call_stack', at offset 192 (in bits) at thread_info.h:47:1
7192 impacted interfaces
'volatile long int task_struct::state' offset changed from 192 to 256 (in bits) (by +64 bits)
'void* task_struct::stack' offset changed from 256 to 320 (in bits) (by +64 bits)
'atomic_t task_struct::usage' offset changed from 320 to 384 (in bits) (by +64 bits)
'unsigned int task_struct::flags' offset changed from 352 to 416 (in bits) (by +64 bits)
'unsigned int task_struct::ptrace' offset changed from 384 to 448 (in bits) (by +64 bits)
'llist_node task_struct::wake_entry' offset changed from 448 to 512 (in bits) (by +64 bits)
'int task_struct::on_cpu' offset changed from 512 to 576 (in bits) (by +64 bits)
'unsigned int task_struct::cpu' offset changed from 544 to 608 (in bits) (by +64 bits)
'unsigned int task_struct::wakee_flips' offset changed from 576 to 640 (in bits) (by +64 bits)
'unsigned long int task_struct::wakee_flip_decay_ts' offset changed from 640 to 704 (in bits) (by +64 bits)
'task_struct* task_struct::last_wakee' offset changed from 704 to 768 (in bits) (by +64 bits)
'int task_struct::recent_used_cpu' offset changed from 768 to 832 (in bits) (by +64 bits)
'int task_struct::wake_cpu' offset changed from 800 to 864 (in bits) (by +64 bits)
'int task_struct::on_rq' offset changed from 832 to 896 (in bits) (by +64 bits)
'int task_struct::prio' offset changed from 864 to 928 (in bits) (by +64 bits)
'int task_struct::static_prio' offset changed from 896 to 960 (in bits) (by +64 bits)
'int task_struct::normal_prio' offset changed from 928 to 992 (in bits) (by +64 bits)
'unsigned int task_struct::rt_priority' offset changed from 960 to 1024 (in bits) (by +64 bits)
'const sched_class* task_struct::sched_class' offset changed from 1024 to 1088 (in bits) (by +64 bits)
7192 impacted interfaces
'struct thread_info at thread_info.h:39:1' changed:
details were reported earlier
'struct zone at mmzone.h:363:1' changed:
type size hasn't changed
there are data member changes:
type 'atomic_long_t[13]' of 'zone::vm_stat' changed:
type name changed from 'atomic_long_t[13]' to 'atomic_long_t[14]'
array type size changed from 832 to 896
array type subrange 1 changed length from 13 to 14
, size changed from 832 to 896 (in bits) (by +64 bits)
'atomic_long_t zone::vm_numa_stat[]' offset changed from 12096 to 12160 (in bits) (by +64 bits)
778 impacted interfaces
Bug: 145210207
Change-Id: I5d573192b5ac192f342fd0a15bdb0dc467f756bb
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change limits function inlining across translation unit
boundaries in order to reduce the binary size with LTO.
The -import-instr-limit flag defines a size limit, as the number
of LLVM IR instructions, for importing functions from other TUs.
The default value is 100, and decreasing it to 5 reduces the size
of a stripped arm64 defconfig vmlinux by 11%.
Bug: 145297228
Change-Id: Iaf366f843582972a5dfadc4695abb8f9c59882af
Suggested-by: George Burgess IV <gbiv@google.com>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
LLD always splits sections with LTO, which increases module sizes. This
change adds a linker script that merges the split sections in the final
module.
Bug: 145297228
Change-Id: I247e8bd029bd0f98a4fa1cd4db7f6398467b8e55
Suggested-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Exporting a static function breaks the ABI toolchain when CFI is
enabled. Remove the static identifier to fix the issue.
Bug: 145210207
Change-Id: Ib23d594ae05718dcf119b8ed0b5a8251aee2561c
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
If we detect a corrupted x18 and SCS is enabled, restore the register
before jumping back to instrumented code. This is safe, because the
wrapper is called with preemption disabled and a separate shadow stack
is used for interrupt handling.
Bug: 145210207
Change-Id: Ie83578cef27e1656a234d1500581eaf9de6c2365
(am from https://lore.kernel.org/patchwork/patch/1149060/)
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reserve the x18 register from general allocation when SCS is enabled,
because the compiler uses the register to store the current task's
shadow stack pointer. Note that all external kernel modules must also be
compiled with -ffixed-x18 if the kernel has SCS enabled.
Bug: 145210207
Change-Id: If2315ace9a879b3dd2a85f6ba43eddadc4430595
(am from https://lore.kernel.org/patchwork/patch/1149058/)
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
The graph tracer hooks returns by modifying frame records on the
(regular) stack, but with SCS the return address is taken from the
shadow stack, and the value in the frame record has no effect. As we
don't currently have a mechanism to determine the corresponding slot
on the shadow stack (and to pass this through the ftrace
infrastructure), for now let's disable the graph tracer when SCS is
enabled.
Bug: 145210207
Change-Id: I6fdca3eee60bb8594401920a420cd3c1e23cabce
(am from https://lore.kernel.org/patchwork/patch/1149057/)
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change adds generic support for Clang's Shadow Call Stack,
which uses a shadow stack to protect return addresses from being
overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that security guarantees in the kernel differ from the
ones documented for user space. The kernel must store addresses
of shadow stacks used by other tasks and interrupt handlers in
memory, which means an attacker capable reading and writing
arbitrary memory may be able to locate them and hijack control
flow by modifying shadow stacks that are not currently in use.
Bug: 145210207
Change-Id: Ia5f1650593fa95da4efcf86f84830a20989f161c
(am from https://lore.kernel.org/patchwork/patch/1149054/)
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
__apply_alternatives makes indirect calls to functions whose address is
taken in assembly code using the alternative_cb macro. With CFI enabled
using non-canonical jump tables, the compiler isn't able to replace the
function reference with the jump table reference, which trips CFI.
Bug: 145210207
Change-Id: I088faf94b64ac3068ac798dd8e5029cb2ab92903
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
We use non-canonical CFI jump tables with CONFIG_CFI_CLANG, which
means the compiler replaces function address references with the
address of the function's CFI jump table entry. This results in
__pa_symbol(function) returning the physical address of the jump
table entry, which can lead to address space confusion since the
jump table points to a virtual address.
This change adds a __pa_function macro, which uses inline assembly
to take the actual function address instead.
Bug: 145210207
Change-Id: I14995e522365ad09a5c9bd676e1203b2b642cd5a
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
LLVM's integrated assembler fails with the following error when
building KVM:
<inline asm>:12:6: error: expected absolute expression
.if kvm_update_va_mask == 0
^
<inline asm>:21:6: error: expected absolute expression
.if kvm_update_va_mask == 0
^
<inline asm>:24:2: error: unrecognized instruction mnemonic
NOT_AN_INSTRUCTION
^
LLVM ERROR: Error parsing inline asm
These errors come from ALTERNATIVE_CB and __ALTERNATIVE_CFG,
which test for the existence of the callback parameter in inline
assembly using the following expression:
" .if " __stringify(cb) " == 0\n"
This works with GNU as, but isn't supported by LLVM. This change
splits __ALTERNATIVE_CFG and ALTINSTR_ENTRY into separate macros
to fix the LLVM build.
Bug: 145210207
Change-Id: I62dd786277bcaf3c730a7489c4b5a2b437f722ce
(am from https://lore.kernel.org/patchwork/patch/1136803/)
Link: https://github.com/ClangBuiltLinux/linux/issues/472
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With LTO, the compiler doesn't necessarily obey link order for
initcalls, and the initcall variable needs to be globally unique
to avoid naming collisions.
In order to preserve the correct order, we add each variable
into its own section and generate a linker script (in
scripts/link-vmlinux.sh) to ensure the order remains correct. We
also add a __COUNTER__ prefix to the name, so we can retain the
order of initcalls within each compilation unit, and __LINE__ to
make the names more unique.
Bug: 145210207
Change-Id: Iddda881a52b7942781713b188d810b6100159a2b
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This reverts commit 9730a6289c.
This is change not sufficient for ThinLTO, it's replaced by
Iddda881a52b7942781713b188d810b6100159a2b
Bug: 145210207
Change-Id: Ie67f90a19bc78dbf8b61f13dcc441f0f58c60444
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change adds support for ThinLTO, which greatly improves build times
over full LTO while retaining most of the performance benefits:
https://clang.llvm.org/docs/ThinLTO.html
Bug: 145210207
Change-Id: I8bfc19028266077be2bc1fb5c2bc001b599d3214
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Remove CONFIG_FUNCTION_TRACER as it's not needed. Note that
CONFIG_FTRACE is still enabled by default.
Change-Id: Iaf1452937c72325871c1a49b85ba37b4bf136d2e
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Implement arch_bpf_jit_check_func to check that pointers to jited BPF
functions are correctly aligned and point to the BPF JIT region. This
narrows down the attack surface on the stored pointer.
Bug: 140377409
Change-Id: I10c448eda6a8b0bf4c16ee591fc65974696216b9
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
(cherry picked from commit c10baf7606)
With CONFIG_BPF_JIT, the kernel makes indirect calls to dynamically
generated code, which the compile-time Control-Flow Integrity (CFI)
checking cannot validate. This change adds basic sanity checking to
ensure we are jumping to a valid location, which narrows down the
attack surface on the stored pointer.
In addition, this change adds a weak arch_bpf_jit_check_func function,
which architectures that implement BPF JIT can override to perform
additional validation, such as verifying that the pointer points to
the correct memory region.
Bug: 140377409
Change-Id: I8ebac6637ab6bd9db44716b1c742add267298669
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
(cherry picked from commit 9a11e8da57)
Instead of invoking struct bpf_prog::bpf_func directly, use the
BPF_PROG_RUN macro.
Bug: 140377409
Change-Id: I26abeccc8d25af0f412935ed97aebb5c64f52a2a
(cherry picked from commit a2c11b034142 ("kcm: use BPF_PROG_RUN"))
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Yonghong Song <yhs@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
(cherry picked from commit 59c019fdd1)
Implement arch_bpf_jit_check_func to check that pointers to jited BPF
functions are correctly aligned and point to the BPF JIT region. This
narrows down the attack surface on the stored pointer.
Bug: 140377409
Change-Id: I10c448eda6a8b0bf4c16ee591fc65974696216b9
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
With CONFIG_BPF_JIT, the kernel makes indirect calls to dynamically
generated code, which the compile-time Control-Flow Integrity (CFI)
checking cannot validate. This change adds basic sanity checking to
ensure we are jumping to a valid location, which narrows down the
attack surface on the stored pointer.
In addition, this change adds a weak arch_bpf_jit_check_func function,
which architectures that implement BPF JIT can override to perform
additional validation, such as verifying that the pointer points to
the correct memory region.
Bug: 140377409
Change-Id: I8ebac6637ab6bd9db44716b1c742add267298669
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Instead of invoking struct bpf_prog::bpf_func directly, use the
BPF_PROG_RUN macro.
Bug: 140377409
Change-Id: I26abeccc8d25af0f412935ed97aebb5c64f52a2a
(cherry picked from commit a2c11b034142 ("kcm: use BPF_PROG_RUN"))
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Acked-by: Yonghong Song <yhs@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Define a weak function for COND_SYSCALL instead of creating a weak
alias to a function with an incompatible type. This fixes indirect
call mismatches with Control-Flow Integrity.
Bug: 138254717
Change-Id: I3a9a8162f30ff63cea2002304f70152b7cc8f746
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Define a weak function for COND_SYSCALL instead of creating a weak
alias to a function with an incompatible type. This fixes indirect
call mismatches with Control-Flow Integrity.
Bug: 138254717
Change-Id: I3effbbf5c86107103cf29a40dfb37b268e59bfd8
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
sys_ni.c compiles fine with CONFIG_LTO_CLANG, and disabling LTO
for it breaks indirect call checking to functions in this file.
Bug: 138254717
Change-Id: I7947cf3d0283ad37431860739665fee7fb0dfbdb
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
LKDTM was refactored in 4.17 and lkdtm_rodata.o was moved to
lkdtm/rodata.o. Fix CFLAGS to apply to the correct object file,
so LKDTM can be built with LTO.
Bug: 133186739
Change-Id: I906db4aecabf42dec905a8b03933689732ab5b52
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
[ Upstream commit 1e29ab3186e33c77dbb2d7566172a205b59fa390 ]
Calling sys_ni_syscall through a syscall_fn_t pointer trips indirect
call Control-Flow Integrity checking due to a function type
mismatch. Use SYSCALL_DEFINE0 for __arm64_sys_ni_syscall instead and
remove the now unnecessary casts.
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
[ Upstream commit 0e358bd7b7ebd27e491dabed938eae254c17fe3b ]
Although a syscall defined using SYSCALL_DEFINE0 doesn't accept
parameters, use the correct function type to avoid indirect call
type mismatches with Control-Flow Integrity checking.
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
[ Upstream commit 8ef8f368ce72b5e17f7c1f1ef15c38dcfd0fef64 ]
Syscall wrappers in <asm/syscall_wrapper.h> use const struct pt_regs *
as the argument type. Use const in syscall_fn_t as well to fix indirect
call type mismatches with Control-Flow Integrity checking.
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Use the correct function type to avoid tripping Control-Flow
Integrity checking.
Change-Id: I6a145ba075f8902a5883e6374328643b3771b1eb
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Although a syscall defined using SYSCALL_DEFINE0 doesn't accept
parameters, use the correct function type to avoid type mismatches
with Control-Flow Integrity checking.
Change-Id: I246d27e4418695231979f4f9a6d98c8eb6776167
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Disable HAVE_ARCH_PREL32_RELOCATIONS to stop LLVM from reordering
initcalls.
Change-Id: Ie8f58b53fe8dccb961573f3aa80ab6d8aae16f4d
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Instead of disabling LTO for all the code in the directory, only
disable it for the VDSO itself.
Change-Id: I02e34b75e022982f9884e238aec89486890dd4bb
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
LLVM's assembler doesn't accept the short form inl (%%dx) instruction,
but instead insists on the output register to be explicitly specified:
<inline asm>:1:7: error: invalid operand for instruction
inl (%dx)
^
LLVM ERROR: Error parsing inline asm
Change-Id: I0519034f4a66bd72f23d206d4638578836a49ff5
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Calling a stand-alone assembly function trips clang's Control-Flow
Integrity checking. Add a static inline stub to convert indirect calls
to direct calls for pmull_ghash_update_p(64|8).
Change-Id: Ib4899cdae60d83961f78ed3451a919f7c9c713bc
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Calling a stand-alone assembly function trips clang's Control-Flow
Integrity checking. Add static inline stubs to convert indirect calls
to direct calls for sha[12]_ce_transform
Bug: 67506682
Change-Id: If872f30095994206bc768eee13670be552b2a247
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Disable CFI for code that runs at EL2 because __cfi_check only
understands EL1 addresses.
Bug: 67506682
Change-Id: Ia582943be0b31669d88464fd99228a5368b1aa6a
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Disable CFI to allow an indirect call to a physical address.
Bug: 67506682
Change-Id: I0ec38f34245a4ad52f508f6989093526d3bf442f
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Calling sys_ni_syscall through a syscall_fn_t pointer trips indirect
call Control-Flow Integrity checking due to a function type
mismatch. Use SYSCALL_DEFINE0 for __arm64_sys_ni_syscall instead and
remove the now unnecessary casts.
Change-Id: Id01f505b333b864477335e87e0c3100fa60be1eb
Link: https://lkml.org/lkml/2019/5/3/690
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Although a syscall defined using SYSCALL_DEFINE0 doesn't accept
parameters, use the correct function type to avoid indirect call
type mismatches with Control-Flow Integrity checking.
Change-Id: I45d1235f4b93c69451ed3610b676d0948cef5ee1
Link: https://lkml.org/lkml/2019/5/3/691
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Syscall wrappers in <asm/syscall_wrapper.h> use const struct pt_regs *
as the argument type. Use const in syscall_fn_t as well to fix indirect
call type mismatches with Control-Flow Integrity checking.
Change-Id: I23d26aef5107985631a9aac500f2ceefc0b25419
Link: https://lkml.org/lkml/2019/5/3/689
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
When CONFIG_CFI_CLANG is enabled, LLVM renames all address taken
functions by appending a .cfi postfix to their names, and creates
function stubs with the original names. The compiler always injects
these stubs to the text section, even if the function itself is
placed into init or exit sections, which creates modpost warnings.
This commit adds a modpost exception for CFI stubs to prevent the
warnings.
Bug: 117237524
Change-Id: Ieb8bf20d0c3ad7b7295c535f598370220598cdb0
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change fixes indirect call mismatches with function and function
graph tracing, which trip Control-Flow Integrity (CFI) checking.
Bug: 79510107
Bug: 67506682
Change-Id: I5de08c113fb970ffefedce93c58e0161f22c7ca2
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Fix the callback 9p passes to read_cache_page to actually have the
proper type expected. Casting around function pointers can easily
hide typing bugs, and defeats control flow protection.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Change-Id: I3738b7c0c7f77f85cd7007141400c36516dac6b8
Link: https://lkml.org/lkml/2019/5/1/374
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Use struct device_attribute instead of struct idletimer_tg_attr, and
the correct callback function type to avoid indirect call mismatches
with Control Flow Integrity checking.
Bug: 117299373
Change-Id: I5cc50171e01a5162ef504a726fe3ed4b51941b34
(cherry-picked from commit 20fdaf6e1e313b4efa48243d5250526eb43ea70d)
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
With CFI enabled, LLVM appends .cfi to most function names, which
potentially breaks user space tools. While stripping the postfix is
not optimal either, this should at least create less confusion.
Bug: 67506682
Bug: 73328469
Change-Id: I253f34a562629032ddd792b8498e171109ea7cbc
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
This change adds the CONFIG_CFI_CLANG option, CFI error handling,
and a faster look-up table for cross module CFI checks.
Bug: 67506682
Change-Id: Ic009f0a629b552a0eb16e6d89808c7029e91447d
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Disable HAVE_ARCH_PREL32_RELOCATIONS to stop LLVM from reordering
initcalls.
Change-Id: I8fd9e24c89986b5a41813e23d28778839dd3abdf
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
__ll_sc_* functions are only referenced in inline assembly, which means lld
won't see them when CONFIG_LTO_CLANG is enabled due to a bug in LLVM:
https://bugs.llvm.org/show_bug.cgi?id=35841
When LTO and CONFIG_ARM64_LSE_ATOMICS are both enabled, linking fails with
the following type of errors:
ld.lld: error: relocation R_AARCH64_CALL26 cannot refer to absolute
symbol: __ll_sc_atomic64_add_return_acquire
>>> defined in vmlinux.o
>>> referenced by atomic_lse.h:299 (arch/arm64/include/asm/atomic_lse.h:299)
>>> vmlinux.o:(kernel_init_freeable)
...
This change works around the problem by always linking in atomic_ll_sc.o to
vmlinux, instead of placing it in arch/arm64/lib/lib.a.
Bug: 63740206
Bug: 117299373
Bug: 124318741
Change-Id: Idf2211034cce9cb0b6b2007bbdb1dfc1c1a1053d
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
LLVM's integrated assembler is always used for inline assembly with
CONFIG_LTO_CLANG. Unlike gcc, LLVM considers each inline assembly
block to be independent and therefore, any preambles that enable
features must be included in each block.
This change adds the necessary preamble to ARM64_LSE_ATOMIC_INSN to
allow CONFIG_ARM64_LSE_ATOMICS to be enabled with LTO.
Bug: 117299373
Change-Id: Icc06361dc2a2dba0f5f967d7f540cac2753b3e9c
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Due to a bug in clang, vdso fails to build when both LTO_CLANG and
CC_OPTIMIZE_FOR_SIZE are enabled:
https://bugs.llvm.org/show_bug.cgi?id=32155
Disable LTO for vdso to work around the problem.
Bug: 62093296
Bug: 67506682
Change-Id: I1d0279535fd389db4c829e4556f9ef728f240a34
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Clang's integrated assembler does not allow assembly macros defined
in one inline asm block using the .macro directive to be used across
separate asm blocks. LLVM developers consider this a feature and not a
bug, recommending code refactoring:
https://bugs.llvm.org/show_bug.cgi?id=19749
As binutils doesn't allow macros to be redefined, this change uses
UNDEFINE_MRS_S and UNDEFINE_MSR_S to define corresponding macros
in-place and workaround gcc and clang limitations on redefining macros
across different assembler blocks.
Specifically, the current state after preprocessing looks like this:
asm volatile(".macro mXX_s ... .endm");
void f()
{
asm volatile("mXX_s a, b");
}
With GCC, it gives macro redefinition error because sysreg.h is included
in multiple source files, and assembler code for all of them is later
combined for LTO (I've seen an intermediate file with hundreds of
identical definitions).
With clang, it gives macro undefined error because clang doesn't allow
sharing macros between inline asm statements.
I also seem to remember catching another sort of undefined error with
GCC due to reordering of macro definition asm statement and generated
asm code for function that uses the macro.
The solution with defining and undefining for each use, while certainly
not elegant, satisfies both GCC and clang, LTO and non-LTO.
Co-developed-by: Alex Matveev <alxmtvv@gmail.com>
Co-developed-by: Yury Norov <ynorov@caviumnetworks.com>
Co-developed-by: Sami Tolvanen <samitolvanen@google.com>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
(cherry picked from commit be604c616ca71cbf5c860d0cfa4595128ab74189)
Change-Id: I803fff57f639b0921ef81f90ec4befe802e7eecf
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Sami Tolvanen