050dbce2343bf334a27d9c6a3a557c9325204e64
890732
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Michael Bestas
|
050dbce234 |
Merge remote-tracking branch 'sm8250/lineage-20' into lineage-23.2
* sm8250/lineage-20: (296 commits) defconfigs: Enable CONFIG_HIDRAW to satiate FCM 7 requirements arm64: configs: vendor: reduce KFENCE pool size arm64: configs: vendor: Enable KFENCE by setting the sample interval to 500ms arm64: configs: vendor: Enable KFENCE UPSTREAM: random: split initialization into early step and later step UPSTREAM: kfence: avoid passing -g for test UPSTREAM: kfence: fix memory leak when cat kfence objects UPSTREAM: kfence: unconditionally use unbound work queue UPSTREAM: kfence: use TASK_IDLE when awaiting allocation FROMLIST: kfence: skip all GFP_ZONEMASK allocations FROMLIST: kfence: move the size check to the beginning of __kfence_alloc() ANDROID: kfence: clean up unused variables FROMGIT: kfence: use power-efficient work queue to run delayed work FROMGIT: kfence: maximize allocation wait timeout duration FROMGIT: kfence: await for allocation using wait_event FROMGIT: kfence: zero guard page after out-of-bounds access UPSTREAM: kfence: make compatible with kmemleak UPSTREAM: kfence: fix reports if constant function prefixes exist UPSTREAM: kfence: fix printk format for ptrdiff_t UPSTREAM: kfence: report sensitive information based on no_hash_pointers ... Change-Id: Ia5322fb5697db2c0fcbd3031aae8c4cc3cfb4a80 |
||
|
Nolen Johnson
|
03832f9734 |
defconfigs: Enable CONFIG_HIDRAW to satiate FCM 7 requirements
* Already in kona, but not the rest. Change-Id: I738f6ae4ea62ebd76e17438730050260ef52c799 |
||
|
Alexander Potapenko
|
db70df8dd9 |
arm64: configs: vendor: reduce KFENCE pool size
CONFIG_KFENCE_NUM_OBJECTS controls the constant memory overhead that KFENCE introduces for its memory pool. By default it is 255 objects (2Mb extra memory), but since concerns have been raised that low-memory devices may not afford that, we are lowering the number of objects to 63 (512Kb extra memory). So far we haven't seen Android devices allocate more than 50 KFENCE objects. Should the kernel exhaust the pool, KFENCE will stop allocating new objects and fall back to SLAB/SLUB until one of the objects is freed. An immediate consequence of reducing the pool size is that a freed KFENCE object will be reused 4x times faster, effectively reducing the probability of detecting a use-after-free. Since KFENCE is a best-effort error detection tool, not a use-after-free mitigation mechanism, we believe this should not be problematic. Bug: 181822795 Test: run KFENCE on an Android device Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I88fca80eea8b382115de03dbb104fd305b35476f |
||
|
Alexander Potapenko
|
ef2f7f5748 |
arm64: configs: vendor: Enable KFENCE by setting the sample interval to 500ms
This change effectively enables KFENCE in GKI kernels with a sample interval of 500ms. It is still possible to disable KFENCE at boot time using kfence.sample_interval=0. Bug: 176228452 Test: kernel builds, device boots normally Test: TreeHugger Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I69cb0fc8fdebba36c9166b275ad245240314d811 |
||
|
Alistair Delva
|
ed4c9c5aa1 |
arm64: configs: vendor: Enable KFENCE
Set KFENCE_SAMPLE_INTERVAL to zero to runtime disable KFENCE for now. The feature can be activated with kfence.sample_interval=<time ms>. Bug: 177201466 Change-Id: I8f8f3fcd578f414eeea6ef4d922e110b31325b27 Signed-off-by: Alistair Delva <adelva@google.com> |
||
|
Jason A. Donenfeld
|
582a30d0a0 |
UPSTREAM: random: split initialization into early step and later step
The full RNG initialization relies on some timestamps, made possible with initialization functions like time_init() and timekeeping_init(). However, these are only available rather late in initialization. Meanwhile, other things, such as memory allocator functions, make use of the RNG much earlier. So split RNG initialization into two phases. We can provide arch randomness very early on, and then later, after timekeeping and such are available, initialize the rest. This ensures that, for example, slabs are properly randomized if RDRAND is available. Without this, CONFIG_SLAB_FREELIST_RANDOM=y loses a degree of its security, because its random seed is potentially deterministic, since it hasn't yet incorporated RDRAND. It also makes it possible to use a better seed in kfence, which currently relies on only the cycle counter. Another positive consequence is that on systems with RDRAND, running with CONFIG_WARN_ALL_UNSEEDED_RANDOM=y results in no warnings at all. One subtle side effect of this change is that on systems with no RDRAND, RDTSC is now only queried by random_init() once, committing the moment of the function call, instead of multiple times as before. This is intentional, as the multiple RDTSCs in a loop before weren't accomplishing very much, with jitter being better provided by try_to_generate_entropy(). Plus, filling blocks with RDTSC is still being done in extract_entropy(), which is necessarily called before random bytes are served anyway. Cc: Andrew Morton <akpm@linux-foundation.org> Reviewed-by: Kees Cook <keescook@chromium.org> Reviewed-by: Dominik Brodowski <linux@dominikbrodowski.net> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> (cherry picked from commit f62384995e4cb7703e5295779c44135c5311770d) Change-Id: Ibf5ca2ddea8d853c108c6191803046cae8808dfa [dereference23: Backport to 5.4] Signed-off-by: Alexander Winkowski <dereference23@outlook.com> |
||
|
Marco Elver
|
492a4c8eec |
UPSTREAM: kfence: avoid passing -g for test
Nathan reported that when building with GNU as and a version of clang that defaults to DWARF5: $ make -skj"$(nproc)" ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu- \ LLVM=1 LLVM_IAS=0 O=build \ mrproper allmodconfig mm/kfence/kfence_test.o /tmp/kfence_test-08a0a0.s: Assembler messages: /tmp/kfence_test-08a0a0.s:14627: Error: non-constant .uleb128 is not supported /tmp/kfence_test-08a0a0.s:14628: Error: non-constant .uleb128 is not supported /tmp/kfence_test-08a0a0.s:14632: Error: non-constant .uleb128 is not supported /tmp/kfence_test-08a0a0.s:14633: Error: non-constant .uleb128 is not supported /tmp/kfence_test-08a0a0.s:14639: Error: non-constant .uleb128 is not supported ... This is because `-g` defaults to the compiler debug info default. If the assembler does not support some of the directives used, the above errors occur. To fix, remove the explicit passing of `-g`. All the test wants is that stack traces print valid function names, and debug info is not required for that. (I currently cannot recall why I added the explicit `-g`.) Bug: 254441685 Link: https://lkml.kernel.org/r/20230316224705.709984-1-elver@google.com Fixes: bc8fbc5f305a ("kfence: add test suite") Signed-off-by: Marco Elver <elver@google.com> Reported-by: Nathan Chancellor <nathan@kernel.org> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 2e08ca1802441224f5b7cc6bffbb687f7406de95) Signed-off-by: Lee Jones <joneslee@google.com> Change-Id: I5646d8667f119147b92f4b86b4cf86cbc47dbdde |
||
|
Baokun Li
|
4001711846 |
UPSTREAM: kfence: fix memory leak when cat kfence objects
Hulk robot reported a kmemleak problem:
unreferenced object 0xffff93d1d8cc02e8 (size 248):
comm "cat", pid 23327, jiffies 4624670141 (age 495992.217s)
hex dump (first 32 bytes):
00 40 85 19 d4 93 ff ff 00 10 00 00 00 00 00 00 .@..............
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
seq_open+0x2a/0x80
full_proxy_open+0x167/0x1e0
do_dentry_open+0x1e1/0x3a0
path_openat+0x961/0xa20
do_filp_open+0xae/0x120
do_sys_openat2+0x216/0x2f0
do_sys_open+0x57/0x80
do_syscall_64+0x33/0x40
entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff93d419854000 (size 4096):
comm "cat", pid 23327, jiffies 4624670141 (age 495992.217s)
hex dump (first 32 bytes):
6b 66 65 6e 63 65 2d 23 32 35 30 3a 20 30 78 30 kfence-#250: 0x0
30 30 30 30 30 30 30 37 35 34 62 64 61 31 32 2d 0000000754bda12-
backtrace:
seq_read_iter+0x313/0x440
seq_read+0x14b/0x1a0
full_proxy_read+0x56/0x80
vfs_read+0xa5/0x1b0
ksys_read+0xa0/0xf0
do_syscall_64+0x33/0x40
entry_SYSCALL_64_after_hwframe+0x44/0xa9
I find that we can easily reproduce this problem with the following
commands:
cat /sys/kernel/debug/kfence/objects
echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak
The leaked memory is allocated in the stack below:
do_syscall_64
do_sys_open
do_dentry_open
full_proxy_open
seq_open ---> alloc seq_file
vfs_read
full_proxy_read
seq_read
seq_read_iter
traverse ---> alloc seq_buf
And it should have been released in the following process:
do_syscall_64
syscall_exit_to_user_mode
exit_to_user_mode_prepare
task_work_run
____fput
__fput
full_proxy_release ---> free here
However, the release function corresponding to file_operations is not
implemented in kfence. As a result, a memory leak occurs. Therefore,
the solution to this problem is to implement the corresponding release
function.
Link: https://lkml.kernel.org/r/20211206133628.2822545-1-libaokun1@huawei.com
Fixes: 0ce20dd84089 ("mm: add Kernel Electric-Fence infrastructure")
Signed-off-by: Baokun Li <libaokun1@huawei.com>
Reported-by: Hulk Robot <hulkci@huawei.com>
Acked-by: Marco Elver <elver@google.com>
Reviewed-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Yu Kuai <yukuai3@huawei.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 0129ab1f268b6cf88825eae819b9b84aa0a85634)
Bug: 187129171
Signed-off-by: Connor O'Brien <connoro@google.com>
Change-Id: If4235f6dbdf89c45f832fab827eab4b6e0271190
(cherry picked from commit 64fe36c410b16e8bbf32bb2268296692eabda900)
Signed-off-by: Mark-PK Tsai <mark-pk.tsai@mediatek.com>
|
||
|
Marco Elver
|
8672460abd |
UPSTREAM: kfence: unconditionally use unbound work queue
Unconditionally use unbound work queue, and not just if wq_power_efficient is true. Because if the system is idle, KFENCE may wait, and by being run on the unbound work queue, we permit the scheduler to make better scheduling decisions and not require pinning KFENCE to the same CPU upon waking up. Link: https://lkml.kernel.org/r/20210521111630.472579-1-elver@google.com Fixes: 36f0b35d0894 ("kfence: use power-efficient work queue to run delayed work") Signed-off-by: Marco Elver <elver@google.com> Reported-by: Hillf Danton <hdanton@sina.com> Reviewed-by: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit ff06e45d3aace3f93d23956c1e655224f363ebe2) Bug: 187129171 Signed-off-by: Connor O'Brien <connoro@google.com> Change-Id: Ib0f1a18a42b71e57e8ec095be0b72bd298e8aec5 |
||
|
Marco Elver
|
b9c183d17a |
UPSTREAM: kfence: use TASK_IDLE when awaiting allocation
Since wait_event() uses TASK_UNINTERRUPTIBLE by default, waiting for an allocation counts towards load. However, for KFENCE, this does not make any sense, since there is no busy work we're awaiting. Instead, use TASK_IDLE via wait_event_idle() to not count towards load. BugLink: https://bugzilla.suse.com/show_bug.cgi?id=1185565 Link: https://lkml.kernel.org/r/20210521083209.3740269-1-elver@google.com Fixes: 407f1d8c1b5f ("kfence: await for allocation using wait_event") Signed-off-by: Marco Elver <elver@google.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: David Laight <David.Laight@ACULAB.COM> Cc: Hillf Danton <hdanton@sina.com> Cc: <stable@vger.kernel.org> [5.12+] Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit 8fd0e995cc7b6a7a8a40bc03d52a2cd445beeff4) Bug: 187129171 Signed-off-by: Connor O'Brien <connoro@google.com> Change-Id: Ida5cea821a5a8ef538d314a6e80680faf8aacbdc |
||
|
Alexander Potapenko
|
8955213d7d |
FROMLIST: kfence: skip all GFP_ZONEMASK allocations
Allocation requests outside ZONE_NORMAL (MOVABLE, HIGHMEM or DMA) cannot
be fulfilled by KFENCE, because KFENCE memory pool is located in a
zone different from the requested one.
Because callers of kmem_cache_alloc() may actually rely on the
allocation to reside in the requested zone (e.g. memory allocations done
with __GFP_DMA must be DMAable), skip all allocations done with
GFP_ZONEMASK and/or respective SLAB flags (SLAB_CACHE_DMA and
SLAB_CACHE_DMA32).
Fixes: 0ce20dd84089 ("mm: add Kernel Electric-Fence infrastructure")
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Marco Elver <elver@google.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Souptick Joarder <jrdr.linux@gmail.com>
Cc: stable@vger.kernel.org # 5.12+
Signed-off-by: Alexander Potapenko <glider@google.com>
Reviewed-by: Marco Elver <elver@google.com>
Acked-by: Souptick Joarder <jrdr.linux@gmail.com>
Bug: 192294212
Test: ran KFENCE test suite with __GFP_DMA on QEMU
Link: https://lore.kernel.org/linux-mm/20210714092222.1890268-2-glider@google.com/
Change-Id: I72d2e24185e213d0ce60155d5a78f29bca8c88cc
Signed-off-by: Alexander Potapenko <glider@google.com>
(cherry picked from commit 0f27e1d31707e09cf7bac781a08e5a46a9b6cfd1)
|
||
|
Alexander Potapenko
|
d93bb7f110 |
FROMLIST: kfence: move the size check to the beginning of __kfence_alloc()
Check the allocation size before toggling kfence_allocation_gate. This way allocations that can't be served by KFENCE will not result in waiting for another CONFIG_KFENCE_SAMPLE_INTERVAL without allocating anything. Suggested-by: Marco Elver <elver@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Marco Elver <elver@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: stable@vger.kernel.org # 5.12+ Signed-off-by: Alexander Potapenko <glider@google.com> Reviewed-by: Marco Elver <elver@google.com> Bug: 192294212 Test: ran KFENCE test suite with __GFP_DMA on QEMU Link: https://lore.kernel.org/linux-mm/20210714092222.1890268-1-glider@google.com/ Change-Id: Ie69c6134fc1c3e68238457b146e26355c17f5295 Signed-off-by: Alexander Potapenko <glider@google.com> (cherry picked from commit 8478d8dc5381464999ac17a93e639f869d9b514f) |
||
|
Carlos Llamas
|
f3daf7cd4e |
ANDROID: kfence: clean up unused variables
Commit 62fa525d1e50 ("BACKPORT: kfence: add test suite") removed memcg
sanity checks from original change as part of the backport. However,
this left a couple variables unused. This patch removes such variables
fixing the following warnings:
mm/kfence/kfence_test.c:270:23: warning: unused variable ‘s’ [-Wunused-variable]
270 | struct kmem_cache *s = test_cache ?: kmalloc_caches[kmalloc_type(GFP_KERNEL)][kmalloc_index(size)];
| ^
mm/kfence/kfence_test.c:269:17: warning: unused variable ‘page’ [-Wunused-variable]
269 | struct page *page = virt_to_head_page(alloc);
| ^~~~
Bug: 183339614
Fixes: 62fa525d1e50 ("BACKPORT: kfence: add test suite")
Reported-by: kernelci.org bot <bot@kernelci.org>
Signed-off-by: Carlos Llamas <cmllamas@google.com>
Change-Id: Ifac52f9021be61840cb43bc8d20c9728f3beb948
|
||
|
Marco Elver
|
c212b2f5d2 |
FROMGIT: kfence: use power-efficient work queue to run delayed work
Use the power-efficient work queue, to avoid the pathological case where we keep pinning ourselves on the same possibly idle CPU on systems that want to be power-efficient (https://lwn.net/Articles/731052/). Link: https://lkml.kernel.org/r/20210421105132.3965998-4-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Hillf Danton <hdanton@sina.com> Cc: Jann Horn <jannh@google.com> Cc: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Bug: 172317151 Bug: 185280916 Test: power team confirmed there's no regression (cherry picked from commit e5bcc059aa2321383325e98628d65258bdcb306f https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git akpm) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I53a2869987cdac165840e3878ff73735a03438fa |
||
|
Marco Elver
|
83058cf90a |
FROMGIT: kfence: maximize allocation wait timeout duration
The allocation wait timeout was initially added because of warnings due to CONFIG_DETECT_HUNG_TASK=y [1]. While the 1 sec timeout is sufficient to resolve the warnings (given the hung task timeout must be 1 sec or larger) it may cause unnecessary wake-ups if the system is idle. [1] https://lkml.kernel.org/r/CADYN=9J0DQhizAGB0-jz4HOBBh+05kMBXb4c0cXMS7Qi5NAJiw@mail.gmail.com Fix it by computing the timeout duration in terms of the current sysctl_hung_task_timeout_secs value. Link: https://lkml.kernel.org/r/20210421105132.3965998-3-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Hillf Danton <hdanton@sina.com> Cc: Jann Horn <jannh@google.com> Cc: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Bug: 172317151 Bug: 185280916 Test: power team confirmed there's no regression (cherry picked from commit 5d744ff5acbb40f3b8f1703cb433432889f88fbd https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git akpm) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I1a8da1844aaded4827bda904ca621a27bbead4bc |
||
|
Marco Elver
|
f884d4a99b |
FROMGIT: kfence: await for allocation using wait_event
Patch series "kfence: optimize timer scheduling", v2. We have observed that mostly-idle systems with KFENCE enabled wake up otherwise idle CPUs, preventing such to enter a lower power state. Debugging revealed that KFENCE spends too much active time in toggle_allocation_gate(). While the first version of KFENCE was using all the right bits to be scheduling optimal, and thus power efficient, by simply using wait_event() + wake_up(), that code was unfortunately removed. As KFENCE was exposed to various different configs and tests, the scheduling optimal code slowly disappeared. First because of hung task warnings, and finally because of deadlocks when an allocation is made by timer code with debug objects enabled. Clearly, the "fixes" were not too friendly for devices that want to be power efficient. Therefore, let's try a little harder to fix the hung task and deadlock problems that we have with wait_event() + wake_up(), while remaining as scheduling friendly and power efficient as possible. Crucially, we need to defer the wake_up() to an irq_work, avoiding any potential for deadlock. The result with this series is that on the devices where we observed a power regression, power usage returns back to baseline levels. This patch (of 3): On mostly-idle systems, we have observed that toggle_allocation_gate() is a cause of frequent wake-ups, preventing an otherwise idle CPU to go into a lower power state. A late change in KFENCE's development, due to a potential deadlock [1], required changing the scheduling-friendly wait_event_timeout() and wake_up() to an open-coded wait-loop using schedule_timeout(). [1] https://lkml.kernel.org/r/000000000000c0645805b7f982e4@google.com To avoid unnecessary wake-ups, switch to using wait_event_timeout(). Unfortunately, we still cannot use a version with direct wake_up() in __kfence_alloc() due to the same potential for deadlock as in [1]. Instead, add a level of indirection via an irq_work that is scheduled if we determine that the kfence_timer requires a wake_up(). Link: https://lkml.kernel.org/r/20210421105132.3965998-1-elver@google.com Link: https://lkml.kernel.org/r/20210421105132.3965998-2-elver@google.com Fixes: 0ce20dd84089 ("mm: add Kernel Electric-Fence infrastructure") Signed-off-by: Marco Elver <elver@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jann Horn <jannh@google.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Hillf Danton <hdanton@sina.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Bug: 172317151 Bug: 185280916 Test: power team confirmed there's no regression (cherry picked from commit 0ac66fbbadde5547db190e9d87ff2e77d245f9a2 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git akpm) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Idba39683f0b76eb3f70df1113e43d94845fab5bf |
||
|
Marco Elver
|
d4eabce327 |
FROMGIT: kfence: zero guard page after out-of-bounds access
After an out-of-bounds accesses, zero the guard page before re-protecting in kfence_guarded_free(). On one hand this helps make the failure mode of subsequent out-of-bounds accesses more deterministic, but could also prevent certain information leaks. Link: https://lkml.kernel.org/r/20210312121653.348518-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Acked-by: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Bug: 172317151 Test: build and run on an ARM64 device (cherry picked from commit 35eaef74e16dcc85b0659db1c7b4a8d83f7a34ef https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git akpm) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I2bad112b6cbf457892b1dd1a90b909cd88eee2c6 |
||
|
Marco Elver
|
484a723c10 |
UPSTREAM: kfence: make compatible with kmemleak
Because memblock allocations are registered with kmemleak, the KFENCE pool was seen by kmemleak as one large object. Later allocations through kfence_alloc() that were registered with kmemleak via slab_post_alloc_hook() would then overlap and trigger a warning. Therefore, once the pool is initialized, we can remove (free) it from kmemleak again, since it should be treated as allocator-internal and be seen as "free memory". The second problem is that kmemleak is passed the rounded size, and not the originally requested size, which is also the size of KFENCE objects. To avoid kmemleak scanning past the end of an object and trigger a KFENCE out-of-bounds error, fix the size if it is a KFENCE object. For simplicity, to avoid a call to kfence_ksize() in slab_post_alloc_hook() (and avoid new IS_ENABLED(CONFIG_DEBUG_KMEMLEAK) guard), just call kfence_ksize() in mm/kmemleak.c:create_object(). Link: https://lkml.kernel.org/r/20210317084740.3099921-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Reported-by: Luis Henriques <lhenriques@suse.de> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Tested-by: Luis Henriques <lhenriques@suse.de> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Bug: 172317151 Test: build and run on an ARM64 device (cherry picked from commit 9551158069ba8fcc893798d42dc4f978b62ef60f) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Ida4d747d3b81b5e8a6f1047b864da89e8e110e61 |
||
|
Marco Elver
|
6dadba5f6c |
UPSTREAM: kfence: fix reports if constant function prefixes exist
Some architectures prefix all functions with a constant string ('.' on
ppc64). Add ARCH_FUNC_PREFIX, which may optionally be defined in
<asm/kfence.h>, so that get_stack_skipnr() can work properly.
Link: https://lkml.kernel.org/r/f036c53d-7e81-763c-47f4-6024c6c5f058@csgroup.eu
Link: https://lkml.kernel.org/r/20210304144000.1148590-1-elver@google.com
Signed-off-by: Marco Elver <elver@google.com>
Reported-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Tested-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Alexander Potapenko <glider@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrey Konovalov <andreyknvl@google.com>
Cc: Jann Horn <jannh@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Bug: 172317151
Test: build and run on an ARM64 device
(cherry picked from commit 0aa41cae92c1e2e61ae5b3a2dde8e674172e40ac)
Signed-off-by: Alexander Potapenko <glider@google.com>
Change-Id: Iad99dd75be77c26359723d1336e2d1b25beb5a26
|
||
|
Marco Elver
|
6198ef7cdd |
UPSTREAM: kfence: fix printk format for ptrdiff_t
Use %td for ptrdiff_t. Link: https://lkml.kernel.org/r/3abbe4c9-16ad-c168-a90f-087978ccd8f7@csgroup.eu Link: https://lkml.kernel.org/r/20210303121157.3430807-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Reported-by: Christophe Leroy <christophe.leroy@csgroup.eu> Reviewed-by: Alexander Potapenko <glider@google.com> Cc: Dmitriy Vyukov <dvyukov@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Jann Horn <jannh@google.com> Cc: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Bug: 172317151 Test: build and run on an ARM64 device (cherry picked from commit 702b16d724a61cb97461f403d7a2da29324471b3) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I49c7a8ed6dfecceafd4e25e979c6dd58b1d2dbe2 |
||
|
Marco Elver
|
0bcabb04b2 |
UPSTREAM: kfence: report sensitive information based on no_hash_pointers
We cannot rely on CONFIG_DEBUG_KERNEL to decide if we're running a "debug kernel" where we can safely show potentially sensitive information in the kernel log. Instead, simply rely on the newly introduced "no_hash_pointers" to print unhashed kernel pointers, as well as decide if our reports can include other potentially sensitive information such as registers and corrupted bytes. Link: https://lkml.kernel.org/r/20210223082043.1972742-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Cc: Timur Tabi <timur@kernel.org> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Bug: 177201466 Bug: 180086542 (cherry picked from commit 35beccf0926d42ee0d56e41979ec8cdf814c4769) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Ie76403ee1b902742a6ac300ad13bd117a02f93f1 |
||
|
Timur Tabi
|
fe718b03ac |
UPSTREAM: lib/vsprintf: no_hash_pointers prints all addresses as unhashed
If the no_hash_pointers command line parameter is set, then
printk("%p") will print pointers as unhashed, which is useful for
debugging purposes. This change applies to any function that uses
vsprintf, such as print_hex_dump() and seq_buf_printf().
A large warning message is displayed if this option is enabled.
Unhashed pointers expose kernel addresses, which can be a security
risk.
Also update test_printf to skip the hashed pointer tests if the
command-line option is set.
Signed-off-by: Timur Tabi <timur@kernel.org>
Acked-by: Petr Mladek <pmladek@suse.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Acked-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Marco Elver <elver@google.com>
Signed-off-by: Petr Mladek <pmladek@suse.com>
Link: https://lore.kernel.org/r/20210214161348.369023-4-timur@kernel.org
Bug: 177201466
Bug: 180086542
(cherry picked from commit 5ead723a20e0447bc7db33dc3070b420e5f80aa6)
Signed-off-by: Alexander Potapenko <glider@google.com>
Change-Id: I9680bb5e076da9f766a66e6076e5c39291188e40
|
||
|
Timur Tabi
|
b466062109 |
UPSTREAM: lib: use KSTM_MODULE_GLOBALS macro in kselftest drivers
Instead of defining the total/failed test counters manually, test drivers that are clients of kselftest should use the macro created for this purpose. Signed-off-by: Timur Tabi <timur@kernel.org> Reviewed-by: Petr Mladek <pmladek@suse.com> Acked-by: Marco Elver <elver@google.com> Signed-off-by: Petr Mladek <pmladek@suse.com> Link: https://lore.kernel.org/r/20210214161348.369023-2-timur@kernel.org Bug: 177201466 Bug: 180086542 (cherry picked from commit 4e89a78779647ca7ee2967551c599633fe9d3647) Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I5d60024e8214548ef1ee3a4e48b0dbc50a8adde9 |
||
|
Alexander Potapenko
|
2596d68c80 |
BACKPORT: kasan: use error_report_end tracepoint
Make it possible to trace KASAN error reporting. A good usecase is watching for trace events from the userspace to detect and process memory corruption reports from the kernel. Link: https://lkml.kernel.org/r/20210121131915.1331302-4-glider@google.com Signed-off-by: Alexander Potapenko <glider@google.com> Suggested-by: Marco Elver <elver@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Petr Mladek <pmladek@suse.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 99bbc9fba9b0c8956b817dd21ecb510da8b676dc https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: resolved conflicts in mm/kasan/report.c] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I60ca062373d13f08ed4ed05bee341a8c28619407 |
||
|
Alexander Potapenko
|
415bb6b4c4 |
FROMGIT: kfence: use error_report_end tracepoint
Make it possible to trace KFENCE error reporting. A good usecase is watching for trace events from the userspace to detect and process memory corruption reports from the kernel. Link: https://lkml.kernel.org/r/20210121131915.1331302-3-glider@google.com Signed-off-by: Alexander Potapenko <glider@google.com> Suggested-by: Marco Elver <elver@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Petr Mladek <pmladek@suse.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 2eb9559621418b844b29d2b56d9962ac019f600b https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I991cd1eee48d107bcdf5e1c0d9194e6a386b9bdf |
||
|
Alexander Potapenko
|
d3b772dc9a |
FROMGIT: tracing: add error_report_end trace point
Patch series "Add error_report_end tracepoint to KFENCE and KASAN", v3. This patchset adds a tracepoint, error_repor_end, that is to be used by KFENCE, KASAN, and potentially other bug detection tools, when they print an error report. One of the possible use cases is userspace collection of kernel error reports: interested parties can subscribe to the tracing event via tracefs, and get notified when an error report occurs. This patch (of 3): Introduce error_report_end tracepoint. It can be used in debugging tools like KASAN, KFENCE, etc. to provide extensions to the error reporting mechanisms (e.g. allow tests hook into error reporting, ease error report collection from production kernels). Another benefit would be making use of ftrace for debugging or benchmarking the tools themselves. Should we need it, the tracepoint name leaves us with the possibility to introduce a complementary error_report_start tracepoint in the future. Link: https://lkml.kernel.org/r/20210121131915.1331302-1-glider@google.com Link: https://lkml.kernel.org/r/20210121131915.1331302-2-glider@google.com Signed-off-by: Alexander Potapenko <glider@google.com> Suggested-by: Marco Elver <elver@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Petr Mladek <pmladek@suse.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit ba7612c00686f204f7bca4ceb7394a9e705e84bd https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Ic86e29982c04dad4b3b7889a424f37b22cc5f22b |
||
|
Marco Elver
|
cc59eafba6 |
FROMGIT: kfence: show access type in report
Show the access type in KFENCE reports by plumbing through read/write information from the page fault handler. Update the documentation and test accordingly. Link: https://lkml.kernel.org/r/20210111091544.3287013-2-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Suggested-by: Jörn Engel <joern@purestorage.com> Reviewed-by: Jörn Engel <joern@purestorage.com> Cc: Alexander Potapenko <glider@google.com> Cc: Marco Elver <elver@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit e29117c1fbf30d27d5afe41cf34263e1fd8e4f04 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I2e9bb224292cf92ac828232c51cd57024ac56d7d |
||
|
Marco Elver
|
cf4d20195e |
FROMGIT: kfence: fix typo in test
Fix a typo/accidental copy-paste that resulted in the obviously incorrect 'GFP_KERNEL * 2' expression. Link: https://lkml.kernel.org/r/X9lHQExmHGvETxY4@elver.google.com Signed-off-by: Marco Elver <elver@google.com> Reported-by: kernel test robot <lkp@intel.com> Acked-by: Alexander Potapenko <glider@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 161c8770c371ca3992565d8f1db1ad4a88a562e9 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I4c26617c64fd2e6d410e6e793bb0eebf8fc87e55 |
||
|
Marco Elver
|
79d23f598b |
BACKPORT: kfence: add test suite
Add KFENCE test suite, testing various error detection scenarios. Makes use of KUnit for test organization. Since KFENCE's interface to obtain error reports is via the console, the test verifies that KFENCE outputs expected reports to the console. Link: https://lkml.kernel.org/r/20201103175841.3495947-9-elver@google.com Signed-off-by: Alexander Potapenko <glider@google.com> Signed-off-by: Marco Elver <elver@google.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Co-developed-by: Alexander Potapenko <glider@google.com> Reviewed-by: Jann Horn <jannh@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christopher Lameter <cl@linux.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Rientjes <rientjes@google.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Hillf Danton <hdanton@sina.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Joern Engel <joern@purestorage.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Kees Cook <keescook@chromium.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Pekka Enberg <penberg@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: SeongJae Park <sjpark@amazon.de> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit d6364119849bb0432e9a46e9699519ea9ff1bb77 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: removed memcg sanity checks from the tests] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I733090d4109a795c078fe8090c46b19cdfe9413f |
||
|
Marco Elver
|
90c3900ad6 |
FROMGIT: kfence: add missing copyright header to documentation
Add missing copyright header to KFENCE documentation. Link: https://lkml.kernel.org/r/20210118092159.145934-4-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Reviewed-by: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 14c8d0148b7b44c549c8892a93a800a9650fa31c https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Iaf953c9bf1f220054b1ec4068f2ba0bd9a7e9fc4 |
||
|
Marco Elver
|
1817d6edcf |
BACKPORT: kfence, Documentation: add KFENCE documentation
Add KFENCE documentation in dev-tools/kfence.rst, and add to index. Link: https://lkml.kernel.org/r/20201103175841.3495947-8-elver@google.com Signed-off-by: Alexander Potapenko <glider@google.com> Signed-off-by: Marco Elver <elver@google.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Co-developed-by: Alexander Potapenko <glider@google.com> Reviewed-by: Jann Horn <jannh@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christopher Lameter <cl@linux.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Rientjes <rientjes@google.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Hillf Danton <hdanton@sina.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Joern Engel <joern@purestorage.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Kees Cook <keescook@chromium.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Pekka Enberg <penberg@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: SeongJae Park <sjpark@amazon.de> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit f63eeaffb09b57d4c9cb9c92c4925ee3bd3df457 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: fixed a minor conflict in Documentation/dev-tools/index.rst] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Iaedafbf20944d68c87229f6c8b2a6e5155fcee60 |
||
|
Alexander Potapenko
|
6f3edbf1cb |
BACKPORT: kfence, kasan: make KFENCE compatible with KASAN
Make KFENCE compatible with KASAN. Currently this helps test KFENCE itself, where KASAN can catch potential corruptions to KFENCE state, or other corruptions that may be a result of freepointer corruptions in the main allocators. Link: https://lkml.kernel.org/r/20201103175841.3495947-7-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Alexander Potapenko <glider@google.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Jann Horn <jannh@google.com> Co-developed-by: Marco Elver <elver@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christopher Lameter <cl@linux.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Rientjes <rientjes@google.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Hillf Danton <hdanton@sina.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Joern Engel <joern@purestorage.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Kees Cook <keescook@chromium.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Pekka Enberg <penberg@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: SeongJae Park <sjpark@amazon.de> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 8ab944ae627dc9fb165bff68acc465751a0b8de2 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: resolved a conflict in mm/kasan/generic.c] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I2f862c2e514e7fcff50a019048c8f0d22f46e6c4 |
||
|
Alexander Potapenko
|
c8cbb13941 |
BACKPORT: mm, kfence: insert KFENCE hooks for SLUB
Inserts KFENCE hooks into the SLUB allocator. To pass the originally requested size to KFENCE, add an argument 'orig_size' to slab_alloc*(). The additional argument is required to preserve the requested original size for kmalloc() allocations, which uses size classes (e.g. an allocation of 272 bytes will return an object of size 512). Therefore, kmem_cache::size does not represent the kmalloc-caller's requested size, and we must introduce the argument 'orig_size' to propagate the originally requested size to KFENCE. Without the originally requested size, we would not be able to detect out-of-bounds accesses for objects placed at the end of a KFENCE object page if that object is not equal to the kmalloc-size class it was bucketed into. When KFENCE is disabled, there is no additional overhead, since slab_alloc*() functions are __always_inline. Link: https://lkml.kernel.org/r/20201103175841.3495947-6-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Alexander Potapenko <glider@google.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Jann Horn <jannh@google.com> Co-developed-by: Marco Elver <elver@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christopher Lameter <cl@linux.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Rientjes <rientjes@google.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Hillf Danton <hdanton@sina.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Joern Engel <joern@purestorage.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Kees Cook <keescook@chromium.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Pekka Enberg <penberg@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: SeongJae Park <sjpark@amazon.de> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 5aa4d4f541f500cd63f814ae39bea55024ed5c6b https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: dropped changes to include/linux/slub_def.h, fixed conflict in mm/slub.c, always pass root cache pointer to kfence_alloc()] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Iacfc22088087cb920107a595f71b09e21d5f2f47 |
||
|
Alexander Potapenko
|
2fe50ef221 |
BACKPORT: mm, kfence: insert KFENCE hooks for SLAB
Inserts KFENCE hooks into the SLAB allocator. To pass the originally requested size to KFENCE, add an argument 'orig_size' to slab_alloc*(). The additional argument is required to preserve the requested original size for kmalloc() allocations, which uses size classes (e.g. an allocation of 272 bytes will return an object of size 512). Therefore, kmem_cache::size does not represent the kmalloc-caller's requested size, and we must introduce the argument 'orig_size' to propagate the originally requested size to KFENCE. Without the originally requested size, we would not be able to detect out-of-bounds accesses for objects placed at the end of a KFENCE object page if that object is not equal to the kmalloc-size class it was bucketed into. When KFENCE is disabled, there is no additional overhead, since slab_alloc*() functions are __always_inline. Link: https://lkml.kernel.org/r/20201103175841.3495947-5-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Alexander Potapenko <glider@google.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Co-developed-by: Marco Elver <elver@google.com> Cc: Christoph Lameter <cl@linux.com> Cc: Pekka Enberg <penberg@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Hillf Danton <hdanton@sina.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Jann Horn <jannh@google.com> Cc: Joern Engel <joern@purestorage.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kees Cook <keescook@chromium.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: SeongJae Park <sjpark@amazon.de> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 840c0553e89413319d67971a321bcc07114da9b8 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: resolved minor API change in mm/slab_common.c, dropped changes to include/linux/slab_def.h, resolved conflicts in mm/slab.c, always pass root cache pointer to kfence_alloc] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Iab2ba9c7b06b9a234d93ba892be639941861f8ab |
||
|
Marco Elver
|
aa44dfbaab |
UPSTREAM: mm/kasan: add object validation in ksize()
ksize() has been unconditionally unpoisoning the whole shadow memory region associated with an allocation. This can lead to various undetected bugs, for example, double-kzfree(). Specifically, kzfree() uses ksize() to determine the actual allocation size, and subsequently zeroes the memory. Since ksize() used to just unpoison the whole shadow memory region, no invalid free was detected. This patch addresses this as follows: 1. Add a check in ksize(), and only then unpoison the memory region. 2. Preserve kasan_unpoison_slab() semantics by explicitly unpoisoning the shadow memory region using the size obtained from __ksize(). Tested: 1. With SLAB allocator: a) normal boot without warnings; b) verified the added double-kzfree() is detected. 2. With SLUB allocator: a) normal boot without warnings; b) verified the added double-kzfree() is detected. [elver@google.com: s/BUG_ON/WARN_ON_ONCE/, per Kees] Link: http://lkml.kernel.org/r/20190627094445.216365-6-elver@google.com Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=199359 Link: http://lkml.kernel.org/r/20190626142014.141844-6-elver@google.com Change-Id: I74fb5904ad52c6f5555d86af33b40050c8faf5d2 Signed-off-by: Marco Elver <elver@google.com> Acked-by: Kees Cook <keescook@chromium.org> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Christoph Lameter <cl@linux.com> Cc: Pekka Enberg <penberg@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> |
||
|
Marco Elver
|
afaa642262 |
UPSTREAM: mm/kasan: change kasan_check_{read,write} to return boolean
This changes {,__}kasan_check_{read,write} functions to return a boolean
denoting if the access was valid or not.
[sfr@canb.auug.org.au: include types.h for "bool"]
Link: http://lkml.kernel.org/r/20190705184949.13cdd021@canb.auug.org.au
Link: http://lkml.kernel.org/r/20190626142014.141844-3-elver@google.com
Change-Id: Ib725b6caf26995121bccedf8561cbe43094569be
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@google.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Kees Cook <keescook@chromium.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
|
||
|
Marco Elver
|
8e3089d59b |
UPSTREAM: mm/kasan: introduce __kasan_check_{read,write}
Patch series "mm/kasan: Add object validation in ksize()", v3.
This patch (of 5):
This introduces __kasan_check_{read,write}. __kasan_check functions may
be used from anywhere, even compilation units that disable instrumentation
selectively.
This change eliminates the need for the __KASAN_INTERNAL definition.
[elver@google.com: v5]
Link: http://lkml.kernel.org/r/20190708170706.174189-2-elver@google.com
Link: http://lkml.kernel.org/r/20190626142014.141844-2-elver@google.com
Change-Id: Iddfae00044e339498cd71fcddb0bec2b6345b907
Signed-off-by: Marco Elver <elver@google.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@google.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Kees Cook <keescook@chromium.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
|
||
|
Marco Elver
|
0e1432322b |
UPSTREAM: mm/slab: refactor common ksize KASAN logic into slab_common.c
This refactors common code of ksize() between the various allocators into slab_common.c: __ksize() is the allocator-specific implementation without instrumentation, whereas ksize() includes the required KASAN logic. Link: http://lkml.kernel.org/r/20190626142014.141844-5-elver@google.com Change-Id: Ie1ed4a70559f9bbd395a1c0774a02776d7bc48f6 Signed-off-by: Marco Elver <elver@google.com> Acked-by: Christoph Lameter <cl@linux.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Pekka Enberg <penberg@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Kees Cook <keescook@chromium.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> |
||
|
Kees Cook
|
bdcefa1e51 |
UPSTREAM: mm/slab: sanity-check page type when looking up cache
This avoids any possible type confusion when looking up an object. For example, if a non-slab were to be passed to kfree(), the invalid slab_cache pointer (i.e. overlapped with some other value from the struct page union) would be used for subsequent slab manipulations that could lead to further memory corruption. Since the page is already in cache, adding the PageSlab() check will have nearly zero cost, so add a check and WARN() to virt_to_cache(). Additionally replaces an open-coded virt_to_cache(). To support the failure mode this also updates all callers of virt_to_cache() and cache_from_obj() to handle a NULL cache pointer return value (though note that several already handle this case gracefully). [dan.carpenter@oracle.com: restore IRQs in kfree()] Link: http://lkml.kernel.org/r/20190613065637.GE16334@mwanda Link: http://lkml.kernel.org/r/20190530045017.15252-3-keescook@chromium.org Change-Id: I7c98ef20b38af63ff58dc80fb5655e8be40cd617 Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com> Cc: Alexander Popov <alex.popov@linux.com> Cc: Alexander Potapenko <glider@google.com> Cc: Christoph Lameter <cl@linux.com> Cc: David Rientjes <rientjes@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Pekka Enberg <penberg@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> |
||
|
Kees Cook
|
54280aefda |
UPSTREAM: mm/slab: validate cache membership under freelist hardening
Patch series "mm/slab: Improved sanity checking". This adds defenses against slab cache confusion (as seen in real-world exploits[1]) and gracefully handles type confusions when trying to look up slab caches from an arbitrary page. (Also is patch 3: new LKDTM tests for these defenses as well as for the existing double-free detection. This patch (of 3): When building under CONFIG_SLAB_FREELIST_HARDENING, it makes sense to perform sanity-checking on the assumed slab cache during kmem_cache_free() to make sure the kernel doesn't mix freelists across slab caches and corrupt memory (as seen in the exploitation of flaws like CVE-2018-9568[1]). Note that the prior code might WARN() but still corrupt memory (i.e. return the assumed cache instead of the owned cache). There is no noticeable performance impact (changes are within noise). Measuring parallel kernel builds, I saw the following with CONFIG_SLAB_FREELIST_HARDENED, before and after this patch: before: Run times: 288.85 286.53 287.09 287.07 287.21 Min: 286.53 Max: 288.85 Mean: 287.35 Std Dev: 0.79 after: Run times: 289.58 287.40 286.97 287.20 287.01 Min: 286.97 Max: 289.58 Mean: 287.63 Std Dev: 0.99 Delta: 0.1% which is well below the standard deviation [1] https://github.com/ThomasKing2014/slides/raw/master/Building%20universal%20Android%20rooting%20with%20a%20type%20confusion%20vulnerability.pdf Link: http://lkml.kernel.org/r/20190530045017.15252-2-keescook@chromium.org Change-Id: Ibb7e2ef88efb2fb9cd30fa8707a2bc0a759c2ede Signed-off-by: Kees Cook <keescook@chromium.org> Cc: Christoph Lameter <cl@linux.com> Cc: Pekka Enberg <penberg@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Alexander Popov <alex.popov@linux.com> Cc: Alexander Potapenko <glider@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> |
||
|
Alexander Popov
|
dcc3522b15 |
FROMGIT: mm/slab: rerform init_on_free earlier
Currently in CONFIG_SLAB init_on_free happens too late, and heap objects go to the heap quarantine not being erased. Lets move init_on_free clearing before calling kasan_slab_free(). In that case heap quarantine will store erased objects, similarly to CONFIG_SLUB=y behavior. Link: https://lkml.kernel.org/r/20201210183729.1261524-1-alex.popov@linux.com Signed-off-by: Alexander Popov <alex.popov@linux.com> Reviewed-by: Alexander Potapenko <glider@google.com> Acked-by: David Rientjes <rientjes@google.com> Acked-by: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Christoph Lameter <cl@linux.com> Cc: Pekka Enberg <penberg@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Bug: 177201466 (cherry picked from commit a32d654db543843a5ffb248feaec1a909718addd https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I2bf5e70c1524619526efd792bbdd959b813af1e4 |
||
|
Marco Elver
|
66fb355247 |
FROMGIT: kfence: use pt_regs to generate stack trace on faults
Instead of removing the fault handling portion of the stack trace based on the fault handler's name, just use struct pt_regs directly. Change kfence_handle_page_fault() to take a struct pt_regs, and plumb it through to kfence_report_error() for out-of-bounds, use-after-free, or invalid access errors, where pt_regs is used to generate the stack trace. If the kernel is a DEBUG_KERNEL, also show registers for more information. Link: https://lkml.kernel.org/r/20201105092133.2075331-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Suggested-by: Mark Rutland <mark.rutland@arm.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Alexander Potapenko <glider@google.com> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 54a5abe9b5d542ee71836439cc662efe178c8211 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I3a60060b24f0efb4faee2e6c953973bc1263e8d1 |
||
|
Marco Elver
|
399a3b5b04 |
FROMGIT: kfence, arm64: add missing copyright and description header
Add missing copyright and description header to KFENCE source file. Link: https://lkml.kernel.org/r/20210118092159.145934-3-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Reviewed-by: Alexander Potapenko <glider@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit ce19707fadfe62b7441f2a9455b6866161fbc1f0 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I75c06e57e4bb906626c13318d00cfc7d47b76bde |
||
|
Marco Elver
|
2c94402fda |
BACKPORT: arm64, kfence: enable KFENCE for ARM64
Add architecture specific implementation details for KFENCE and enable KFENCE for the arm64 architecture. In particular, this implements the required interface in <asm/kfence.h>. KFENCE requires that attributes for pages from its memory pool can individually be set. Therefore, force the entire linear map to be mapped at page granularity. Doing so may result in extra memory allocated for page tables in case rodata=full is not set; however, currently CONFIG_RODATA_FULL_DEFAULT_ENABLED=y is the default, and the common case is therefore not affected by this change. Link: https://lkml.kernel.org/r/20201103175841.3495947-4-elver@google.com Signed-off-by: Alexander Potapenko <glider@google.com> Signed-off-by: Marco Elver <elver@google.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Co-developed-by: Alexander Potapenko <glider@google.com> Reviewed-by: Jann Horn <jannh@google.com> Reviewed-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christopher Lameter <cl@linux.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Rientjes <rientjes@google.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Hillf Danton <hdanton@sina.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Joern Engel <joern@purestorage.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Kees Cook <keescook@chromium.org> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Pekka Enberg <penberg@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: SeongJae Park <sjpark@amazon.de> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit d3cb0555da9b469c3f9ebe663d4c0d6265757175 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: resolved a minor conflict in arch/arm64/Kconfig] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: Ibe9d26c2c9088862ba8855c18e0c5210a68c7f50 |
||
|
Marco Elver
|
280fcbae34 |
FROMGIT: kfence, x86: add missing copyright and description header
Add missing copyright and description header to KFENCE source file. Link: https://lkml.kernel.org/r/20210118092159.145934-2-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Reviewed-by: Alexander Potapenko <glider@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 8470d53956b7e1b6f3f2dd17f277b9d2bb1472ae https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I90aef6f80e7b5a46f5168d1a24c02a5737785831 |
||
|
Alexander Potapenko
|
6853b2999b |
BACKPORT: x86, kfence: enable KFENCE for x86
Add architecture specific implementation details for KFENCE and enable KFENCE for the x86 architecture. In particular, this implements the required interface in <asm/kfence.h> for setting up the pool and providing helper functions for protecting and unprotecting pages. For x86, we need to ensure that the pool uses 4K pages, which is done using the set_memory_4k() helper function. Link: https://lkml.kernel.org/r/20201103175841.3495947-3-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Alexander Potapenko <glider@google.com> Reviewed-by: Dmitry Vyukov <dvyukov@google.com> Co-developed-by: Marco Elver <elver@google.com> Reviewed-by: Jann Horn <jannh@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Borislav Petkov <bp@alien8.de> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christopher Lameter <cl@linux.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: David Rientjes <rientjes@google.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Hillf Danton <hdanton@sina.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Joern Engel <joern@purestorage.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Kees Cook <keescook@chromium.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Pekka Enberg <penberg@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: SeongJae Park <sjpark@amazon.de> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit cb99fcd83140d0d58ea36db6c1c2034abc95f983 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) [glider: resolved a minor conflict in arch/x86/Kconfig, s/flush_tlb_one_kernel/__flush_tlb_one_kernel] Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I111caffb0b88c34ed9ff57b95f127b08eacedcb9 |
||
|
Marco Elver
|
f9e10d18fe |
FROMGIT: kfence: add missing copyright and description headers
Add missing copyright and description headers to KFENCE source files. Link: https://lkml.kernel.org/r/20210118092159.145934-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Reviewed-by: Alexander Potapenko <glider@google.com> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit b86d1ed1155ce1d2420057bfbdcc62b9fd53c1d6 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I95eb6756baaa0d8ca1dbc656708667372cff546d |
||
|
Marco Elver
|
ebc5b341bb |
FROMGIT: kfence: add option to use KFENCE without static keys
For certain usecases, specifically where the sample interval is always set to a very low value such as 1ms, it can make sense to use a dynamic branch instead of static branches due to the overhead of toggling a static branch. Therefore, add a new Kconfig option to remove the static branches and instead check kfence_allocation_gate if a KFENCE allocation should be set up. Link: https://lkml.kernel.org/r/20210111091544.3287013-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Suggested-by: Jörn Engel <joern@purestorage.com> Reviewed-by: Jörn Engel <joern@purestorage.com> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit c01761611b325c1e4ec7d3e236cc9db003cb82fd https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I68a112a8ff68fa24742b198e036f130a9757c27f |
||
|
Marco Elver
|
f0699a2c5c |
FROMGIT: kfence: fix potential deadlock due to wake_up()
Lockdep reports that we may deadlock when calling wake_up() in
__kfence_alloc(), because we may already hold base->lock. This can happen
if debug objects are enabled:
...
__kfence_alloc+0xa0/0xbc0 mm/kfence/core.c:710
kfence_alloc include/linux/kfence.h:108 [inline]
...
kmem_cache_zalloc include/linux/slab.h:672 [inline]
fill_pool+0x264/0x5c0 lib/debugobjects.c:171
__debug_object_init+0x7a/0xd10 lib/debugobjects.c:560
debug_object_init lib/debugobjects.c:615 [inline]
debug_object_activate+0x32c/0x3e0 lib/debugobjects.c:701
debug_timer_activate kernel/time/timer.c:727 [inline]
__mod_timer+0x77d/0xe30 kernel/time/timer.c:1048
...
Therefore, switch to an open-coded wait loop. The difference to before is
that the waiter wakes up and rechecks the condition after 1 jiffy;
however, given the infrequency of kfence allocations, the difference is
insignificant.
Link: https://lkml.kernel.org/r/000000000000c0645805b7f982e4@google.com
Link: https://lkml.kernel.org/r/20210104130749.1768991-1-elver@google.com
Reported-by: syzbot+8983d6d4f7df556be565@syzkaller.appspotmail.com
Signed-off-by: Marco Elver <elver@google.com>
Suggested-by: Hillf Danton <hdanton@sina.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Bug: 177201466
(cherry picked from commit c5fb1ab1a3c6d0ee02d1054a10d51ffcac57aed5
https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10)
Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish
Signed-off-by: Alexander Potapenko <glider@google.com>
Change-Id: Iee40e9f216afbc3fce8e43c0e2a4bc807fdddf39
|
||
|
Marco Elver
|
c862f6124f |
FROMGIT: kfence: avoid stalling work queue task without allocations
To toggle the allocation gates, we set up a delayed work that calls toggle_allocation_gate(). Here we use wait_event() to await an allocation and subsequently disable the static branch again. However, if the kernel has stopped doing allocations entirely, we'd wait indefinitely, and stall the worker task. This may also result in the appropriate warnings if CONFIG_DETECT_HUNG_TASK=y. Therefore, introduce a 1 second timeout and use wait_event_timeout(). If the timeout is reached, the static branch is disabled and a new delayed work is scheduled to try setting up an allocation at a later time. Note that, this scenario is very unlikely during normal workloads once the kernel has booted and user space tasks are running. It can, however, happen during early boot after KFENCE has been enabled, when e.g. running tests that do not result in any allocations. Link: https://lkml.kernel.org/r/CADYN=9J0DQhizAGB0-jz4HOBBh+05kMBXb4c0cXMS7Qi5NAJiw@mail.gmail.com Link: https://lkml.kernel.org/r/20201110135320.3309507-1-elver@google.com Signed-off-by: Marco Elver <elver@google.com> Reported-by: Anders Roxell <anders.roxell@linaro.org> Cc: Alexander Potapenko <glider@google.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: SeongJae Park <sjpark@amazon.de> Cc: Jann Horn <jannh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Bug: 177201466 (cherry picked from commit 80d4693491f6f20de01437319b081fdda2079e67 https://github.com/hnaz/linux-mm v5.11-rc4-mmots-2021-01-21-20-10) Test: CONFIG_KFENCE_KUNIT_TEST=y passes on Cuttlefish Signed-off-by: Alexander Potapenko <glider@google.com> Change-Id: I2332ff8144b8bce5c4574b01ea2863e0e71e6124 |