aboutsummaryrefslogtreecommitdiff
path: root/sysdeps/x86_64
AgeCommit message (Collapse)Author
2017-06-15x86-64: Implement memset family IFUNC selectors in CH.J. Lu
Implement memset family IFUNC selectors in C. All internal calls within libc.so can use IFUNC on x86-64 since unlike x86, x86-64 supports PC-relative addressing to access the GOT entry so that it can call via PLT without using an extra register. For libc.a, we can't use IFUNC for functions which are called before IFUNC has been initialized. Use IFUNC internally reduces the icache footprint since libc.so and other codes in the process use the same implementations. This patch uses IFUNC for memset functions within libc. 2017-06-07 H.J. Lu <hongjiu.lu@intel.com> Erich Elsen <eriche@google.com> * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add memset-sse2-unaligned-erms, and memset_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add test for __memset_chk_erms. Update comments. * sysdeps/x86_64/multiarch/ifunc-memset.h: New file. * sysdeps/x86_64/multiarch/memset-sse2-unaligned-erms.S: Likewise. * sysdeps/x86_64/multiarch/memset.c: Likewise. * sysdeps/x86_64/multiarch/memset_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/memset_chk.c: Likewise. * sysdeps/x86_64/multiarch/memset.S: Removed. * sysdeps/x86_64/multiarch/memset_chk.S: Likewise. * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S (__memset_chk_erms): New function.
2017-06-14x86-64: Implement memmove family IFUNC selectors in CH.J. Lu
Implement memmove family IFUNC selectors in C. All internal calls within libc.so can use IFUNC on x86-64 since unlike x86, x86-64 supports PC-relative addressing to access the GOT entry so that it can call via PLT without using an extra register. For libc.a, we can't use IFUNC for functions which are called before IFUNC has been initialized. Use IFUNC internally reduces the icache footprint since libc.so and other codes in the process use the same implementations. This patch uses IFUNC for memmove family functions within libc. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add memmove-sse2-unaligned-erms, memcpy_chk-nonshared, mempcpy_chk-nonshared and memmove_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add tests for __memmove_chk_erms, __memcpy_chk_erms and __mempcpy_chk_erms. Update comments. * sysdeps/x86_64/multiarch/ifunc-memmove.h: New file. * sysdeps/x86_64/multiarch/memcpy.c: Likewise. * sysdeps/x86_64/multiarch/memcpy_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/memcpy_chk.c: Likewise. * sysdeps/x86_64/multiarch/memmove-sse2-unaligned-erms.S: Likewise. * sysdeps/x86_64/multiarch/memmove.c: Likewise. * sysdeps/x86_64/multiarch/memmove_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/memmove_chk.c: Likewise. * sysdeps/x86_64/multiarch/mempcpy.c: Likewise. * sysdeps/x86_64/multiarch/mempcpy_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/mempcpy_chk.c: Likewise. * sysdeps/x86_64/multiarch/memcpy.S: Removed. * sysdeps/x86_64/multiarch/memcpy_chk.S: Likewise. * sysdeps/x86_64/multiarch/memmove.S: Likewise. * sysdeps/x86_64/multiarch/memmove_chk.S: Likewise. * sysdeps/x86_64/multiarch/mempcpy.S: Likewise. * sysdeps/x86_64/multiarch/mempcpy_chk.S: Likewise. * sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S (__mempcpy_chk_erms): New function. (__memmove_chk_erms): Likewise. (__memcpy_chk_erms): New alias.
2017-06-14Remove __need macros from errno.h (__need_Emath, __need_error_t).Zack Weinberg
This is fairly complicated, not because the users of __need_Emath and __need_error_t have complicated requirements, but because the core changes had a lot of fallout. __need_error_t exists for gnulib compatibility in argz.h and argp.h. error_t itself is a Hurdism, an enum containing all the E-constants, so you can do 'p (error_t) errno' in gdb and get a symbolic value. argz.h and argp.h use it for function return values, and they want to fall back to 'int' when that's not available. There is no reason why these nonstandard headers cannot just go ahead and include all of errno.h; so we do that. __need_Emath is defined only by .S files; what they _really_ need is for errno.h to avoid declaring anything other than the E-constants (e.g. 'extern int __errno_location(void);' is a syntax error in assembly language). This is replaced with a check for __ASSEMBLER__ in errno.h, plus a carefully documented requirement for bits/errno.h not to define anything other than macros. That in turn has the consequence that bits/errno.h must not define errno - fortunately, all live ports use the same definition of errno, so I've moved it to errno.h. The Hurd bits/errno.h must also take care not to define error_t when __ASSEMBLER__ is defined, which involves repeating all of the definitions twice, but it's a generated file so that's okay. * stdlib/errno.h: Remove __need_Emath and __need_error_t logic. Reorganize file. Declare errno here. When __ASSEMBLER__ is defined, don't declare anything other than the E-constants. * include/errno.h: Change conditional for exposing internal declarations to (not _ISOMAC and not __ASSEMBLER__). * bits/errno.h: Remove logic for __need_Emath. Document requirements for a port-specific bits/errno.h. * sysdeps/unix/sysv/linux/bits/errno.h * sysdeps/unix/sysv/linux/alpha/bits/errno.h * sysdeps/unix/sysv/linux/hppa/bits/errno.h * sysdeps/unix/sysv/linux/mips/bits/errno.h * sysdeps/unix/sysv/linux/sparc/bits/errno.h: Add multiple-include guard and check against improper inclusion. Remove __need_Emath logic. Don't declare errno here. Ensure all constants are defined as simple integer literals. Consistent formatting. * sysdeps/mach/hurd/errnos.awk: Likewise. Only define error_t and enum __error_t_codes if __ASSEMBLER__ is not defined. * sysdeps/mach/hurd/bits/errno.h: Regenerate. * argp/argp.h, string/argz.h: Don't define __need_error_t before including errno.h. * sysdeps/i386/i686/fpu/multiarch/s_cosf-sse2.S * sysdeps/i386/i686/fpu/multiarch/s_sincosf-sse2.S * sysdeps/i386/i686/fpu/multiarch/s_sinf-sse2.S * sysdeps/x86_64/fpu/s_cosf.S * sysdeps/x86_64/fpu/s_sincosf.S * sysdeps/x86_64/fpu/s_sinf.S: Just include errno.h; don't define __need_Emath or include bits/errno.h directly.
2017-06-14PowerPC64 ELFv2 PPC64_OPT_LOCALENTRYAlan Modra
ELFv2 functions with localentry:0 are those with a single entry point, ie. global entry == local entry, that have no requirement on r2 or r12 and guarantee r2 is unchanged on return. Such an external function can be called via the PLT without saving r2 or restoring it on return, avoiding a common load-hit-store for small functions. This patch implements the ld.so changes necessary for this optimization. ld.so needs to check that an optimized plt call sequence is in fact calling a function implemented with localentry:0, end emit a fatal error otherwise. The elf/testobj6.c change is to stop "error while loading shared libraries: expected localentry:0 `preload'" when running elf/preloadtest, which we'd get otherwise. * elf/elf.h (PPC64_OPT_LOCALENTRY): Define. * sysdeps/alpha/dl-machine.h (elf_machine_fixup_plt): Add refsym and sym parameters. Adjust callers. * sysdeps/aarch64/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/arm/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/generic/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/hppa/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/i386/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/ia64/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/m68k/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/microblaze/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/mips/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/nios2/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/powerpc/powerpc32/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/s390/s390-32/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/s390/s390-64/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/sh/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/sparc/sparc32/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/sparc/sparc64/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/tile/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/x86_64/dl-machine.h (elf_machine_fixup_plt): Likewise. * sysdeps/powerpc/powerpc64/dl-machine.c (_dl_error_localentry): New. (_dl_reloc_overflow): Increase buffser size. Formatting. * sysdeps/powerpc/powerpc64/dl-machine.h (ppc64_local_entry_offset): Delete reloc param, add refsym and sym. Check optimized plt call stubs for localentry:0 functions. Adjust callers. (elf_machine_fixup_plt, elf_machine_plt_conflict): Add refsym and sym parameters. Adjust callers. (_dl_reloc_overflow): Move attribute. (_dl_error_localentry): Declare. * elf/dl-runtime.c (_dl_fixup): Save original sym. Pass refsym and sym to elf_machine_fixup_plt. * elf/testobj6.c (preload): Call printf.
2017-06-12x86-64: Implement strcpy family IFUNC selectors in CH.J. Lu
Implement strcpy family IFUNC selectors in C. All internal calls within libc.so can use IFUNC on x86-64 since unlike x86, x86-64 supports PC-relative addressing to access the GOT entry so that it can call via PLT without using an extra register. For libc.a, we can't use IFUNC for functions which are called before IFUNC has been initialized. Use IFUNC internally reduces the icache footprint since libc.so and other codes in the process use the same implementations. This patch uses IFUNC for strcpy family functions within libc. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add strcpy-sse2 and stpcpy-sse2. * sysdeps/x86_64/multiarch/ifunc-unaligned-ssse3.h: New file. * sysdeps/x86_64/multiarch/stpcpy-sse2.S: Likewise. * sysdeps/x86_64/multiarch/stpcpy.c: Likewise. * sysdeps/x86_64/multiarch/stpncpy.c: Likewise. * sysdeps/x86_64/multiarch/strcpy-sse2.S: Likewise. * sysdeps/x86_64/multiarch/strcpy.c: Likewise. * sysdeps/x86_64/multiarch/strncpy.c: Likewise. * sysdeps/x86_64/multiarch/stpcpy.S: Removed. * sysdeps/x86_64/multiarch/stpncpy.S: Likewise. * sysdeps/x86_64/multiarch/strcpy.S: Likewise. * sysdeps/x86_64/multiarch/strncpy.S: Likewise. * sysdeps/x86_64/multiarch/stpncpy-c.c (weak_alias): New. (libc_hidden_def): Always defined as empty. * sysdeps/x86_64/multiarch/strncpy-c.c (libc_hidden_builtin_def): Always Defined as empty.
2017-06-09x86-64: Correct comments in ifunc-impl-list.cH.J. Lu
* sysdeps/x86_64/multiarch/ifunc-impl-list.c: Correct comments.
2017-06-09x86-64: Optimize strrchr/wcsrchr with AVX2H.J. Lu
Optimize strrchr/wcsrchr with AVX2 to check 32 bytes with vector instructions. It is as fast as SSE2 version for small data sizes and up to 1X faster for large data sizes on Haswell. Select AVX2 version on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add strrchr-sse2, strrchr-avx2, wcsrchr-sse2 and wcsrchr-avx2. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add tests for __strrchr_avx2, __strrchr_sse2, __wcsrchr_avx2 and __wcsrchr_sse2. * sysdeps/x86_64/multiarch/strrchr-avx2.S: New file. * sysdeps/x86_64/multiarch/strrchr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/strrchr.c: Likewise. * sysdeps/x86_64/multiarch/wcsrchr-avx2.S: Likewise. * sysdeps/x86_64/multiarch/wcsrchr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/wcsrchr.c: Likewise.
2017-06-09x86-64: Optimize memrchr with AVX2H.J. Lu
Optimize memrchr with AVX2 to search 32 bytes with a single vector compare instruction. It is as fast as SSE2 memrchr for small data sizes and up to 1X faster for large data sizes on Haswell. Select AVX2 memrchr on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add memrchr-sse2 and memrchr-avx2. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add tests for __memrchr_avx2 and __memrchr_sse2. * sysdeps/x86_64/multiarch/memrchr-avx2.S: New file. * sysdeps/x86_64/multiarch/memrchr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/memrchr.c: Likewise.
2017-06-09x86-64: Optimize strchr/strchrnul/wcschr with AVX2H.J. Lu
Optimize strchr/strchrnul/wcschr with AVX2 to search 32 bytes with vector instructions. It is as fast as SSE2 versions for size <= 16 bytes and up to 1X faster for or size > 16 bytes on Haswell. Select AVX2 version on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast. NB: It uses TZCNT instead of BSF since TZCNT produces the same result as BSF for non-zero input. TZCNT is faster than BSF and is executed as BSF if machine doesn't support TZCNT. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add strchr-sse2, strchrnul-sse2, strchr-avx2, strchrnul-avx2, wcschr-sse2 and wcschr-avx2. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add tests for __strchr_avx2, __strchrnul_avx2, __strchrnul_sse2, __wcschr_avx2 and __wcschr_sse2. * sysdeps/x86_64/multiarch/strchr-avx2.S: New file. * sysdeps/x86_64/multiarch/strchr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/strchr.c: Likewise. * sysdeps/x86_64/multiarch/strchrnul-avx2.S: Likewise. * sysdeps/x86_64/multiarch/strchrnul-sse2.S: Likewise. * sysdeps/x86_64/multiarch/strchrnul.c: Likewise. * sysdeps/x86_64/multiarch/wcschr-avx2.S: Likewise. * sysdeps/x86_64/multiarch/wcschr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/wcschr.c: Likewise. * sysdeps/x86_64/multiarch/strchr.S: Removed.
2017-06-09x86-64: Optimize strlen/strnlen/wcslen/wcsnlen with AVX2H.J. Lu
Optimize strlen/strnlen/wcslen/wcsnlen with AVX2 to check 32 bytes with a single vector compare instruction. It is as fast as SSE2 versions for size <= 16 bytes and up to 1X faster for or size > 16 bytes on Haswell. Select AVX2 version on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast. NB: It uses TZCNT instead of BSF since TZCNT produces the same result as BSF for non-zero input. TZCNT is faster than BSF and is executed as BSF if machine doesn't support TZCNT. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add strlen-sse2, strnlen-sse2, strlen-avx2, strnlen-avx2, wcslen-sse2, wcslen-avx2 and wcsnlen-avx2. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add tests for __strlen_avx2, __strlen_sse2, __strnlen_avx2, __strnlen_sse2, __wcslen_avx2, __wcslen_sse2 and __wcsnlen_avx2. * sysdeps/x86_64/multiarch/strlen-avx2.S: New file. * sysdeps/x86_64/multiarch/strlen-sse2.S: Likewise. * sysdeps/x86_64/multiarch/strlen.c: Likewise. * sysdeps/x86_64/multiarch/strnlen-avx2.S: Likewise. * sysdeps/x86_64/multiarch/strnlen-sse2.S: Likewise. * sysdeps/x86_64/multiarch/strnlen.c: Likewise. * sysdeps/x86_64/multiarch/wcslen-avx2.S: Likewise. * sysdeps/x86_64/multiarch/wcslen-sse2.S: Likewise. * sysdeps/x86_64/multiarch/wcslen.c: Likewise. * sysdeps/x86_64/multiarch/wcsnlen-avx2.S: Likewise. * sysdeps/x86_64/multiarch/wcsnlen.c (OPTIMIZE (avx2)): New. (IFUNC_SELECTOR): Return OPTIMIZE (avx2) on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast.
2017-06-09x86-64: Optimize memchr/rawmemchr/wmemchr with SSE2/AVX2H.J. Lu
SSE2 memchr is extended to support wmemchr. AVX2 memchr/rawmemchr/wmemchr are added to search 32 bytes with a single vector compare instruction. AVX2 memchr/rawmemchr/wmemchr are as fast as SSE2 memchr/rawmemchr/wmemchr for small sizes and up to 1.5X faster for larger sizes on Haswell and Skylake. Select AVX2 memchr/rawmemchr/wmemchr on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast. NB: It uses TZCNT instead of BSF since TZCNT produces the same result as BSF for non-zero input. TZCNT is faster than BSF and is executed as BSF if machine doesn't support TZCNT. * sysdeps/x86_64/memchr.S (MEMCHR): New. Depending on if USE_AS_WMEMCHR is defined. (PCMPEQ): Likewise. (memchr): Renamed to ... (MEMCHR): This. Support wmemchr if USE_AS_WMEMCHR is defined. Replace pcmpeqb with PCMPEQ. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add memchr-sse2, rawmemchr-sse2, memchr-avx2, rawmemchr-avx2, wmemchr-sse4_1, wmemchr-avx2 and wmemchr-c. * sysdeps/x86_64/multiarch/ifunc-avx2.h: New file. * sysdeps/x86_64/multiarch/memchr-avx2.S: Likewise. * sysdeps/x86_64/multiarch/memchr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/memchr.c: Likewise. * sysdeps/x86_64/multiarch/rawmemchr-avx2.S: Likewise. * sysdeps/x86_64/multiarch/rawmemchr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/rawmemchr.c: Likewise. * sysdeps/x86_64/multiarch/wmemchr-avx2.S: Likewise. * sysdeps/x86_64/multiarch/wmemchr-sse2.S: Likewise. * sysdeps/x86_64/multiarch/wmemchr.c: Likewise. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Test __memchr_avx2, __memchr_sse2, __rawmemchr_avx2, __rawmemchr_sse2, __wmemchr_avx2 and __wmemchr_sse2.
2017-06-07x86-64: Rename wmemset.h to ifunc-wmemset.hH.J. Lu
No code changes. * sysdeps/x86_64/multiarch/wmemset.c: Include ifunc-wmemset.h instead of wmemset.h. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/multiarch/wmemset.h: Renamed to ... * sysdeps/x86_64/multiarch/ifunc-wmemset.h: This.
2017-06-07x86-64: Fold ifunc-sse4_1.h into wcsnlen.cH.J. Lu
Since ifunc-sse4_1.h is included only by wcsnlen.c, we can fold it into wcsnlen.c. No code changes in wcsnlen.o. 2017-06-07 H.J. Lu <hongjiu.lu@intel.com> * sysdeps/x86_64/multiarch/ifunc-sse4_1.h: Removed and folded into ... * sysdeps/x86_64/multiarch/wcsnlen.c: Here. Don't include ifunc-sse4_1.h.
2017-06-06x86-64: Move wcsnlen.S to multiarch/wcsnlen-sse4_1.SH.J. Lu
Since wcsnlen.S uses pminud which is the part of SSE4.1, move wcsnlen.S to multiarch/wcsnlen-sse4_1.S. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wcsnlen-sse4_1 and wcsnlen-c. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Test __wcsnlen_sse4_1 and __wcsnlen_sse2. * sysdeps/x86_64/multiarch/ifunc-sse4_1.h: New file. * sysdeps/x86_64/multiarch/wcsnlen-c.c: Likewise. * sysdeps/x86_64/multiarch/wcsnlen-sse4_1.S: Likewise. * sysdeps/x86_64/multiarch/wcsnlen.c: Likewise. * sysdeps/x86_64/wcsnlen.S: Removed.
2017-06-06Optimize generic spinlock code and use C11 like atomic macros.Stefan Liebler
This patch optimizes the generic spinlock code. The type pthread_spinlock_t is a typedef to volatile int on all archs. Passing a volatile pointer to the atomic macros which are not mapped to the C11 atomic builtins can lead to extra stores and loads to stack if such a macro creates a temporary variable by using "__typeof (*(mem)) tmp;". Thus, those macros which are used by spinlock code - atomic_exchange_acquire, atomic_load_relaxed, atomic_compare_exchange_weak - have to be adjusted. According to the comment from Szabolcs Nagy, the type of a cast expression is unqualified (see http://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_423.htm): __typeof ((__typeof (*(mem)) *(mem)) tmp; Thus from spinlock perspective the variable tmp is of type int instead of type volatile int. This patch adjusts those macros in include/atomic.h. With this construct GCC >= 5 omits the extra stores and loads. The atomic macros are replaced by the C11 like atomic macros and thus the code is aligned to it. The pthread_spin_unlock implementation is now using release memory order instead of sequentially consistent memory order. The issue with passed volatile int pointers applies to the C11 like atomic macros as well as the ones used before. I've added a glibc_likely hint to the first atomic exchange in pthread_spin_lock in order to return immediately to the caller if the lock is free. Without the hint, there is an additional jump if the lock is free. I've added the atomic_spin_nop macro within the loop of plain reads. The plain reads are also realized by C11 like atomic_load_relaxed macro. The new define ATOMIC_EXCHANGE_USES_CAS determines if the first try to acquire the spinlock in pthread_spin_lock or pthread_spin_trylock is an exchange or a CAS. This is defined in atomic-machine.h for all architectures. The define SPIN_LOCK_READS_BETWEEN_CMPXCHG is now removed. There is no technical reason for throwing in a CAS every now and then, and so far we have no evidence that it can improve performance. If that would be the case, we have to adjust other spin-waiting loops elsewhere, too! Using a CAS loop without plain reads is not a good idea on many targets and wasn't used by one. Thus there is now no option to do so. Architectures are now using the generic spinlock automatically if they do not provide an own implementation. Thus the pthread_spin_lock.c files in sysdeps folder are deleted. ChangeLog: * NEWS: Mention new spinlock implementation. * include/atomic.h: (__atomic_val_bysize): Cast type to omit volatile qualifier. (atomic_exchange_acq): Likewise. (atomic_load_relaxed): Likewise. (ATOMIC_EXCHANGE_USES_CAS): Check definition. * nptl/pthread_spin_init.c (pthread_spin_init): Use atomic_store_relaxed. * nptl/pthread_spin_lock.c (pthread_spin_lock): Use C11-like atomic macros. * nptl/pthread_spin_trylock.c (pthread_spin_trylock): Likewise. * nptl/pthread_spin_unlock.c (pthread_spin_unlock): Use atomic_store_release. * sysdeps/aarch64/nptl/pthread_spin_lock.c: Delete File. * sysdeps/arm/nptl/pthread_spin_lock.c: Likewise. * sysdeps/hppa/nptl/pthread_spin_lock.c: Likewise. * sysdeps/m68k/nptl/pthread_spin_lock.c: Likewise. * sysdeps/microblaze/nptl/pthread_spin_lock.c: Likewise. * sysdeps/mips/nptl/pthread_spin_lock.c: Likewise. * sysdeps/nios2/nptl/pthread_spin_lock.c: Likewise. * sysdeps/aarch64/atomic-machine.h (ATOMIC_EXCHANGE_USES_CAS): Define. * sysdeps/alpha/atomic-machine.h: Likewise. * sysdeps/arm/atomic-machine.h: Likewise. * sysdeps/i386/atomic-machine.h: Likewise. * sysdeps/ia64/atomic-machine.h: Likewise. * sysdeps/m68k/coldfire/atomic-machine.h: Likewise. * sysdeps/m68k/m680x0/m68020/atomic-machine.h: Likewise. * sysdeps/microblaze/atomic-machine.h: Likewise. * sysdeps/mips/atomic-machine.h: Likewise. * sysdeps/powerpc/powerpc32/atomic-machine.h: Likewise. * sysdeps/powerpc/powerpc64/atomic-machine.h: Likewise. * sysdeps/s390/atomic-machine.h: Likewise. * sysdeps/sparc/sparc32/atomic-machine.h: Likewise. * sysdeps/sparc/sparc32/sparcv9/atomic-machine.h: Likewise. * sysdeps/sparc/sparc64/atomic-machine.h: Likewise. * sysdeps/tile/tilegx/atomic-machine.h: Likewise. * sysdeps/tile/tilepro/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/hppa/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/m68k/coldfire/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/nios2/atomic-machine.h: Likewise. * sysdeps/unix/sysv/linux/sh/atomic-machine.h: Likewise. * sysdeps/x86_64/atomic-machine.h: Likewise.
2017-06-05x86-64: Optimize memcmp/wmemcmp with AVX2 and MOVBEH.J. Lu
Optimize x86-64 memcmp/wmemcmp with AVX2. It uses vector compare as much as possible. It is as fast as SSE4 memcmp for size <= 16 bytes and up to 2X faster for size > 16 bytes on Haswell and Skylake. Select AVX2 memcmp/wmemcmp on AVX2 machines where vzeroupper is preferred and AVX unaligned load is fast. NB: It uses TZCNT instead of BSF since TZCNT produces the same result as BSF for non-zero input. TZCNT is faster than BSF and is executed as BSF if machine doesn't support TZCNT. Key features: 1. For size from 2 to 7 bytes, load as big endian with movbe and bswap to avoid branches. 2. Use overlapping compare to avoid branch. 3. Use vector compare when size >= 4 bytes for memcmp or size >= 8 bytes for wmemcmp. 4. If size is 8 * VEC_SIZE or less, unroll the loop. 5. Compare 4 * VEC_SIZE at a time with the aligned first memory area. 6. Use 2 vector compares when size is 2 * VEC_SIZE or less. 7. Use 4 vector compares when size is 4 * VEC_SIZE or less. 8. Use 8 vector compares when size is 8 * VEC_SIZE or less. * sysdeps/x86/cpu-features.h (index_cpu_MOVBE): New. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add memcmp-avx2 and wmemcmp-avx2. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Test __memcmp_avx2 and __wmemcmp_avx2. * sysdeps/x86_64/multiarch/memcmp-avx2.S: New file. * sysdeps/x86_64/multiarch/wmemcmp-avx2.S: Likewise. * sysdeps/x86_64/multiarch/memcmp.S: Use __memcmp_avx2 on AVX 2 machines if AVX unaligned load is fast and vzeroupper is preferred. * sysdeps/x86_64/multiarch/wmemcmp.S: Use __wmemcmp_avx2 on AVX 2 machines if AVX unaligned load is fast and vzeroupper is preferred.
2017-06-05x86-64: Optimize wmemset with SSE2/AVX2/AVX512H.J. Lu
The difference between memset and wmemset is byte vs int. Add stubs to SSE2/AVX2/AVX512 memset for wmemset with updated constant and size: SSE2 wmemset: shl $0x2,%rdx movd %esi,%xmm0 mov %rdi,%rax pshufd $0x0,%xmm0,%xmm0 jmp entry_from_wmemset SSE2 memset: movd %esi,%xmm0 mov %rdi,%rax punpcklbw %xmm0,%xmm0 punpcklwd %xmm0,%xmm0 pshufd $0x0,%xmm0,%xmm0 entry_from_wmemset: Since the ERMS versions of wmemset requires "rep stosl" instead of "rep stosb", only the vector store stubs of SSE2/AVX2/AVX512 wmemset are added. The SSE2 wmemset is about 3X faster and the AVX2 wmemset is about 6X faster on Haswell. * include/wchar.h (__wmemset_chk): New. * sysdeps/x86_64/memset.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to MEMSET_VDUP_TO_VEC0_AND_SET_RETURN. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_CHK_SYMBOL): Likewise. (WMEMSET_SYMBOL): Likewise. (__wmemset): Add hidden definition. (wmemset): Add weak hidden definition. * sysdeps/x86_64/multiarch/Makefile (sysdep_routines): Add wmemset_chk-nonshared. * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list): Add __wmemset_sse2_unaligned, __wmemset_avx2_unaligned, __wmemset_avx512_unaligned, __wmemset_chk_sse2_unaligned, __wmemset_chk_avx2_unaligned and __wmemset_chk_avx512_unaligned. * sysdeps/x86_64/multiarch/memset-avx2-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S (VDUP_TO_VEC0_AND_SET_RETURN): Renamed to ... (MEMSET_VDUP_TO_VEC0_AND_SET_RETURN): This. (WMEMSET_VDUP_TO_VEC0_AND_SET_RETURN): New. (WMEMSET_SYMBOL): Likewise. * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: Updated. (WMEMSET_CHK_SYMBOL): New. (WMEMSET_CHK_SYMBOL (__wmemset_chk, unaligned)): Likewise. (WMEMSET_SYMBOL (__wmemset, unaligned)): Likewise. * sysdeps/x86_64/multiarch/memset.S (WMEMSET_SYMBOL): New. (libc_hidden_builtin_def): Also define __GI_wmemset and __GI___wmemset. (weak_alias): New. * sysdeps/x86_64/multiarch/wmemset.c: New file. * sysdeps/x86_64/multiarch/wmemset.h: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk-nonshared.S: Likewise. * sysdeps/x86_64/multiarch/wmemset_chk.c: Likewise. * sysdeps/x86_64/wmemset.c: Likewise. * sysdeps/x86_64/wmemset_chk.c: Likewise.
2017-06-05x86-64: Update strlen.S to support wcslen/wcsnlenH.J. Lu
The difference between strlen and wcslen is byte vs int. We can replace pminub and pcmpeqb with pminud and pcmpeqd to turn strlen into wcslen. * sysdeps/x86_64/strlen.S (PMINU): New. (PCMPEQ): Likewise. (SHIFT_RETURN): Likewise. (FIND_ZERO): Replace pcmpeqb with PCMPEQ. (strlen): Add SHIFT_RETURN before ret. Replace pcmpeqb and pminub with PCMPEQ and PMINU. * sysdeps/x86_64/wcsnlen.S: New file.
2017-06-05x86_64: Remove redundant REX bytes from memrchr.SH.J. Lu
By x86-64 specification, 32-bit destination registers are zero-extended to 64 bits. There is no need to use 64-bit registers when only the lower 32 bits are non-zero. Also 2 instructions in: mov %rdi, %rcx and $15, %rcx jz L(length_less16_offset0) mov %rdi, %rcx <<< redundant and $15, %rcx <<< redundant are redundant. * sysdeps/x86_64/memrchr.S (__memrchr): Use 32-bit registers for the lower 32 bits. Remove redundant instructions.
2017-05-30x86_64: Remove redundant REX bytes from memchr.SH.J. Lu
By x86-64 specification, 32-bit destination registers are zero-extended to 64 bits. There is no need to use 64-bit registers when only the lower 32 bits are non-zero. * sysdeps/x86_64/memchr.S (MEMCHR): Use 32-bit registers for the lower 32 bits.
2017-05-20x86_64: Remove L(return_null) from rawmemchr.SH.J. Lu
L(return_null) is unused. * sysdeps/x86_64/rawmemchr.S (L(return_null)): Removed.
2017-05-19x86: Optimize SSE2 memchr overflow calculationH.J. Lu
SSE2 memchr computes "edx + ecx - 16" where ecx is less than 16. Use "edx - (16 - ecx)", instead of satured math, to avoid possible addition overflow. This replaces add %ecx, %edx sbb %eax, %eax or %eax, %edx sub $16, %edx with neg %ecx add $16, %ecx sub %ecx, %edx It is the same for x86_64, except for rcx/rdx, instead of ecx/edx. * sysdeps/i386/i686/multiarch/memchr-sse2.S (MEMCHR): Use "edx + ecx - 16" to avoid possible addition overflow. * sysdeps/x86_64/memchr.S (memchr): Likewise.
2017-05-18Correct comments in x86_64/multiarch/memcmp.SH.J. Lu
* sysdeps/x86_64/multiarch/memcmp.S (__GI_memcmp): Correct comments.
2017-05-11Suppress internal declarations for most of the testsuite.Zack Weinberg
This patch adds a new build module called 'testsuite'. IS_IN (testsuite) implies _ISOMAC, as do IS_IN_build and __cplusplus (which means several ad-hoc tests for __cplusplus can go away). libc-symbols.h now suppresses almost all of *itself* when _ISOMAC is defined; in particular, _ISOMAC mode does not get config.h automatically anymore. There are still quite a few tests that need to see internal gunk of one variety or another. For them, we now have 'tests-internal' and 'test-internal-extras'; files in this category will still be compiled with MODULE_NAME=nonlib, and everything proceeds as it always has. The bulk of this patch is moving tests from 'tests' to 'tests-internal'. There is also 'tests-static-internal', which has the same effect on files in 'tests-static', and 'modules-names-tests', which has the *inverse* effect on files in 'modules-names' (it's inverted because most of the things in modules-names are *not* tests). For both of these, the file must appear in *both* the new variable and the old one. There is also now a special case for when libc-symbols.h is included without MODULE_NAME being defined at all. (This happens during the creation of libc-modules.h, and also when preprocessing Versions files.) When this happens, IS_IN is set to be always false and _ISOMAC is *not* defined, which was the status quo, but now it's explicit. The remaining changes to C source files in this patch seemed likely to cause problems in the absence of the main change. They should be relatively self-explanatory. In a few cases I duplicated a definition from an internal header rather than move the test to tests-internal; this was a judgement call each time and I'm happy to change those however reviewers feel is more appropriate. * Makerules: New subdir configuration variables 'tests-internal' and 'test-internal-extras'. Test files in these categories will still be compiled with MODULE_NAME=nonlib. Test files in the existing categories (tests, xtests, test-srcs, test-extras) are now compiled with MODULE_NAME=testsuite. New subdir configuration variable 'modules-names-tests'. Files which are in both 'modules-names' and 'modules-names-tests' will be compiled with MODULE_NAME=testsuite instead of MODULE_NAME=extramodules. (gen-as-const-headers): Move to tests-internal. (do-tests-clean, common-mostlyclean): Support tests-internal. * Makeconfig (built-modules): Add testsuite. * Makefile: Change libof-check-installed-headers-c and libof-check-installed-headers-cxx to 'testsuite'. * Rules: Likewise. Support tests-internal. * benchtests/strcoll-inputs/filelist#en_US.UTF-8: Remove extra-modules.mk. * config.h.in: Don't check for __OPTIMIZE__ or __FAST_MATH__ here. * include/libc-symbols.h: Move definitions of _GNU_SOURCE, PASTE_NAME, PASTE_NAME1, IN_MODULE, IS_IN, and IS_IN_LIB to the very top of the file and rationalize their order. If MODULE_NAME is not defined at all, define IS_IN to always be false, and don't define _ISOMAC. If any of IS_IN (testsuite), IS_IN_build, or __cplusplus are true, define _ISOMAC and suppress everything else in this file, starting with the inclusion of config.h. Do check for inappropriate definitions of __OPTIMIZE__ and __FAST_MATH__ here, but only if _ISOMAC is not defined. Correct some out-of-date commentary. * include/math.h: If _ISOMAC is defined, undefine NO_LONG_DOUBLE and _Mlong_double_ before including math.h. * include/string.h: If _ISOMAC is defined, don't expose _STRING_ARCH_unaligned. Move a comment to a more appropriate location. * include/errno.h, include/stdio.h, include/stdlib.h, include/string.h * include/time.h, include/unistd.h, include/wchar.h: No need to check __cplusplus nor use __BEGIN_DECLS/__END_DECLS. * misc/sys/cdefs.h (__NTHNL): New macro. * sysdeps/m68k/m680x0/fpu/bits/mathinline.h (__m81_defun): Use __NTHNL to avoid errors with GCC 6. * elf/tst-env-setuid-tunables.c: Include config.h with _LIBC defined, for HAVE_TUNABLES. * inet/tst-checks-posix.c: No need to define _ISOMAC. * intl/tst-gettext2.c: Provide own definition of N_. * math/test-signgam-finite-c99.c: No need to define _ISOMAC. * math/test-signgam-main.c: No need to define _ISOMAC. * stdlib/tst-strtod.c: Convert to test-driver. Split locale_test to... * stdlib/tst-strtod1i.c: ...this new file. * stdlib/tst-strtod5.c: Convert to test-driver and add copyright notice. Split tests of __strtod_internal to... * stdlib/tst-strtod5i.c: ...this new file. * string/test-string.h: Include stdint.h. Duplicate definition of inhibit_loop_to_libcall here (from libc-symbols.h). * string/test-strstr.c: Provide dummy definition of libc_hidden_builtin_def when including strstr.c. * sysdeps/ia64/fpu/libm-symbols.h: Suppress entire file in _ISOMAC mode; no need to test __STRICT_ANSI__ nor __cplusplus as well. * sysdeps/x86_64/fpu/math-tests-arch.h: Include cpu-features.h. Don't include init-arch.h. * sysdeps/x86_64/multiarch/test-multiarch.h: Include cpu-features.h. Don't include init-arch.h. * elf/Makefile: Move tst-ptrguard1-static, tst-stackguard1-static, tst-tls1-static, tst-tls2-static, tst-tls3-static, loadtest, unload, unload2, circleload1, neededtest, neededtest2, neededtest3, neededtest4, tst-tls1, tst-tls2, tst-tls3, tst-tls6, tst-tls7, tst-tls8, tst-dlmopen2, tst-ptrguard1, tst-stackguard1, tst-_dl_addr_inside_object, and all of the ifunc tests to tests-internal. Don't add $(modules-names) to test-extras. * inet/Makefile: Move tst-inet6_scopeid_pton to tests-internal. Add tst-deadline to tests-static-internal. * malloc/Makefile: Move tst-mallocstate and tst-scratch_buffer to tests-internal. * misc/Makefile: Move tst-atomic and tst-atomic-long to tests-internal. * nptl/Makefile: Move tst-typesizes, tst-rwlock19, tst-sem11, tst-sem12, tst-sem13, tst-barrier5, tst-signal7, tst-tls3, tst-tls3-malloc, tst-tls5, tst-stackguard1, tst-sem11-static, tst-sem12-static, and tst-stackguard1-static to tests-internal. Link tests-internal with libpthread also. Don't add $(modules-names) to test-extras. * nss/Makefile: Move tst-field to tests-internal. * posix/Makefile: Move bug-regex5, bug-regex20, bug-regex33, tst-rfc3484, tst-rfc3484-2, and tst-rfc3484-3 to tests-internal. * stdlib/Makefile: Move tst-strtod1i, tst-strtod3, tst-strtod4, tst-strtod5i, tst-tls-atexit, and tst-tls-atexit-nodelete to tests-internal. * sunrpc/Makefile: Move tst-svc_register to tests-internal. * sysdeps/powerpc/Makefile: Move test-get_hwcap and test-get_hwcap-static to tests-internal. * sysdeps/unix/sysv/linux/Makefile: Move tst-setgetname to tests-internal. * sysdeps/x86_64/fpu/Makefile: Add all libmvec test modules to modules-names-tests.
2017-05-03x86: Set dl_platform and dl_hwcap from CPU features [BZ #21391]H.J. Lu
dl_platform and dl_hwcap are set from AT_PLATFORM and AT_HWCAP very early during startup. They are used by dynamic linker to determine platform and build an array of hardware capability names, which are added to search path when loading shared object. dl_platform and dl_hwcap are unused on x86-64. On i386, i386, i486, i586 and i686 platforms were supported and only SSE2 capability was used. On x86, usage of AT_PLATFORM and AT_HWCAP to determine platform and processor capabilities is obsolete since all information is available in dl_x86_cpu_features. This patch sets dl_platform and dl_hwcap from dl_x86_cpu_features in dynamic linker. On i386, the available plaforms are changed to i586 and i686 since i386 has been deprecated. On x86-64, the available plaforms are haswell, which is for Haswell class processors with BMI1, BMI2, LZCNT, MOVBE, POPCNT, AVX2 and FMA, and xeon_phi, which is for Xeon Phi class processors with AVX512F, AVX512CD, AVX512ER and AVX512PF. A capability, avx512_1, is also added to x86-64 for AVX512 ISAs: AVX512F, AVX512CD, AVX512BW, AVX512DQ and AVX512VL. [BZ #21391] * sysdeps/i386/dl-machine.h (dl_platform_init) [IS_IN (rtld)]: Only call init_cpu_features. [!IS_IN (rtld)]: Only set GLRO(dl_platform) to NULL if needed. * sysdeps/x86_64/dl-machine.h (dl_platform_init): Likewise. * sysdeps/i386/dl-procinfo.h: Removed. * sysdeps/unix/sysv/linux/i386/dl-procinfo.h: Don't include <sysdeps/i386/dl-procinfo.h> nor <ldsodefs.h>. Include <sysdeps/x86/dl-procinfo.h>. (_dl_procinfo): Replace _DL_HWCAP_COUNT with 32. * sysdeps/unix/sysv/linux/x86_64/dl-procinfo.h [!IS_IN (ldconfig)]: Include <sysdeps/x86/dl-procinfo.h> instead of <sysdeps/generic/dl-procinfo.h>. * sysdeps/x86/cpu-features.c: Include <dl-hwcap.h>. (init_cpu_features): Set dl_platform, dl_hwcap and dl_hwcap_mask. * sysdeps/x86/cpu-features.h (bit_cpu_LZCNT): New. (bit_cpu_MOVBE): Likewise. (bit_cpu_BMI1): Likewise. (bit_cpu_BMI2): Likewise. (index_cpu_BMI1): Likewise. (index_cpu_BMI2): Likewise. (index_cpu_LZCNT): Likewise. (index_cpu_MOVBE): Likewise. (index_cpu_POPCNT): Likewise. (reg_BMI1): Likewise. (reg_BMI2): Likewise. (reg_LZCNT): Likewise. (reg_MOVBE): Likewise. (reg_POPCNT): Likewise. * sysdeps/x86/dl-hwcap.h: New file. * sysdeps/x86/dl-procinfo.h: Likewise. * sysdeps/x86/dl-procinfo.c (_dl_x86_hwcap_flags): New. (_dl_x86_platforms): Likewise.
2017-04-18x86: Use AVX2 memcpy/memset on Skylake server [BZ #21396]H.J. Lu
On Skylake server, AVX512 load/store instructions in memcpy/memset may lead to lower CPU turbo frequency in certain situations. Use of AVX2 in memcpy/memset has been observed to have improved overall performance in many workloads due to the higher frequency. Since AVX512ER is unique to Xeon Phi, this patch sets Prefer_No_AVX512 if AVX512ER isn't available so that AVX2 versions of memcpy/memset are used on Skylake server. [BZ #21396] * sysdeps/x86/cpu-features.c (init_cpu_features): Set Prefer_No_AVX512 if AVX512ER isn't available. * sysdeps/x86/cpu-features.h (bit_arch_Prefer_No_AVX512): New. (index_arch_Prefer_No_AVX512): Likewise. * sysdeps/x86_64/multiarch/memcpy.S (__new_memcpy): Don't use AVX512 version if Prefer_No_AVX512 is set. * sysdeps/x86_64/multiarch/memcpy_chk.S (__memcpy_chk): Likewise. * sysdeps/x86_64/multiarch/memmove.S (__libc_memmove): Likewise. * sysdeps/x86_64/multiarch/memmove_chk.S (__memmove_chk): Likewise. * sysdeps/x86_64/multiarch/mempcpy.S (__mempcpy): Likewise. * sysdeps/x86_64/multiarch/mempcpy_chk.S (__mempcpy_chk): Likewise. * sysdeps/x86_64/multiarch/memset.S (memset): Likewise. * sysdeps/x86_64/multiarch/memset_chk.S (__memset_chk): Likewise.
2017-04-10Add sysdeps/x86/dl-procinfo.cH.J. Lu
Add sysdeps/x86/dl-procinfo.c for x86 version of processor capability information to reduce duplication between i386 and x86_64 dl-procinfo.c. * sysdeps/i386/dl-procinfo.c: Include <sysdeps/x86/dl-procinfo.c>. * sysdeps/x86_64/dl-procinfo.c: Likewise. * sysdeps/x86/dl-procinfo.c: New file.
2017-04-04Remove CALL_THREAD_FCT macroAdhemerval Zanella
This patch removes CALL_THREAD_FCT macro usage and its defition for x86. For 32 bits it usage is only for force 16 stack alignment, however stack is already explicit aligned in clone syscall. For 64 bits and x32 it just a function call and there is no need to code it with inline assembly. Checked on i686-linux-gnu, x86_64-linux-gnu, and x86_64-linux-gnu-x32. * nptl/pthread_create.c (START_THREAD_DEFN): Remove CALL_THREAD_FCT macro usage. * sysdeps/i386/nptl/tls.h (CALL_THREAD_FCT): Remove definition. * sysdeps/x86_64/nptl/tls.h (CALL_THREAD_FCT): Likewise. * sysdeps/x86_64/32/nptl/tls.h: Remove file.
2017-03-21x86-64: Improve branch predication in _dl_runtime_resolve_avx512_opt [BZ #21258]H.J. Lu
On Skylake server, _dl_runtime_resolve_avx512_opt is used to preserve the first 8 vector registers. The code layout is if only %xmm0 - %xmm7 registers are used preserve %xmm0 - %xmm7 registers if only %ymm0 - %ymm7 registers are used preserve %ymm0 - %ymm7 registers preserve %zmm0 - %zmm7 registers Branch predication always executes the fallthrough code path to preserve %zmm0 - %zmm7 registers speculatively, even though only %xmm0 - %xmm7 registers are used. This leads to lower CPU frequency on Skylake server. This patch changes the fallthrough code path to preserve %xmm0 - %xmm7 registers instead: if whole %zmm0 - %zmm7 registers are used preserve %zmm0 - %zmm7 registers if only %ymm0 - %ymm7 registers are used preserve %ymm0 - %ymm7 registers preserve %xmm0 - %xmm7 registers Tested on Skylake server. [BZ #21258] * sysdeps/x86_64/dl-trampoline.S (_dl_runtime_resolve_opt): Define only if _dl_runtime_resolve is defined to _dl_runtime_resolve_sse_vex. * sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_opt): Fallthrough to _dl_runtime_resolve_sse_vex.
2017-03-15x86_64: fix static build of __mempcpy_chk for compilers defaulting to PIC/PIEMike Frysinger
When glibc is compiled with gcc 6.2 that has been configured with to default to PIC/PIE, the static version of __mempcpy_chk is not built, as the test is done on PIC instead of SHARED. Fix the test to check for SHARED, like it is done for similar functions like __memcpy_chk. 2017-03-12 Mike Frysinger <vapier@gentoo.org> * sysdeps/x86_64/mempcpy_chk.S (__mempcpy_chk): Check for SHARED instead of PIC.
2017-03-02Document and fix --enable-bind-now [BZ #21015]Florian Weimer
2017-03-01Narrowing the visibility of libc-internal.h even further.Zack Weinberg
posix/wordexp-test.c used libc-internal.h for PTR_ALIGN_DOWN; similar to what was done with libc-diag.h, I have split the definitions of cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and PTR_ALIGN_DOWN to a new header, libc-pointer-arith.h. It then occurred to me that the remaining declarations in libc-internal.h are mostly to do with early initialization, and probably most of the files including it, even in the core code, don't need it anymore. Indeed, only 19 files actually need what remains of libc-internal.h. 23 others need libc-diag.h instead, and 12 need libc-pointer-arith.h instead. No file needs more than one of them, and 16 don't need any of them! So, with this patch, libc-internal.h stops including libc-diag.h as well as losing the pointer arithmetic macros, and all including files are adjusted. * include/libc-pointer-arith.h: New file. Define cast_to_integer, ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and PTR_ALIGN_DOWN here. * include/libc-internal.h: Definitions of above macros moved from here. Don't include libc-diag.h anymore either. * posix/wordexp-test.c: Include stdint.h and libc-pointer-arith.h. Don't include libc-internal.h. * debug/pcprofile.c, elf/dl-tunables.c, elf/soinit.c, io/openat.c * io/openat64.c, misc/ptrace.c, nptl/pthread_clock_gettime.c * nptl/pthread_clock_settime.c, nptl/pthread_cond_common.c * string/strcoll_l.c, sysdeps/nacl/brk.c * sysdeps/unix/clock_settime.c * sysdeps/unix/sysv/linux/i386/get_clockfreq.c * sysdeps/unix/sysv/linux/ia64/get_clockfreq.c * sysdeps/unix/sysv/linux/powerpc/get_clockfreq.c * sysdeps/unix/sysv/linux/sparc/sparc64/get_clockfreq.c: Don't include libc-internal.h. * elf/get-dynamic-info.h, iconv/loop.c * iconvdata/iso-2022-cn-ext.c, locale/weight.h, locale/weightwc.h * misc/reboot.c, nis/nis_table.c, nptl_db/thread_dbP.h * nscd/connections.c, resolv/res_send.c, soft-fp/fmadf4.c * soft-fp/fmasf4.c, soft-fp/fmatf4.c, stdio-common/vfscanf.c * sysdeps/ieee754/dbl-64/e_lgamma_r.c * sysdeps/ieee754/dbl-64/k_rem_pio2.c * sysdeps/ieee754/flt-32/e_lgammaf_r.c * sysdeps/ieee754/flt-32/k_rem_pio2f.c * sysdeps/ieee754/ldbl-128/k_tanl.c * sysdeps/ieee754/ldbl-128ibm/k_tanl.c * sysdeps/ieee754/ldbl-96/e_lgammal_r.c * sysdeps/ieee754/ldbl-96/k_tanl.c, sysdeps/nptl/futex-internal.h: Include libc-diag.h instead of libc-internal.h. * elf/dl-load.c, elf/dl-reloc.c, locale/programs/locarchive.c * nptl/nptl-init.c, string/strcspn.c, string/strspn.c * malloc/malloc.c, sysdeps/i386/nptl/tls.h * sysdeps/nacl/dl-map-segments.h, sysdeps/x86_64/atomic-machine.h * sysdeps/unix/sysv/linux/spawni.c * sysdeps/x86_64/nptl/tls.h: Include libc-pointer-arith.h instead of libc-internal.h. * elf/get-dynamic-info.h, sysdeps/nacl/dl-map-segments.h * sysdeps/x86_64/atomic-machine.h: Add multiple include guard.
2017-02-25Allow direct use of math_ldbl.h in testsuite.Zack Weinberg
A few 'long double'-related tests include math_private.h just for their variety of math_ldbl.h, which contains macros for assembling and disassembling the binary representation of 'long double'. math_ldbl.h insists on being included from math_private.h, but if we relax this restriction (and fix some portability sloppiness) we can use it directly and not have to expose all of math_private.h to the testsuite. * sysdeps/generic/math_private.h: Use __BIG_ENDIAN and __LITTLE_ENDIAN, not BIG_ENDIAN and LITTLE_ENDIAN. * sysdeps/generic/math_ldbl.h * sysdeps/ia64/fpu/math_ldbl.h * sysdeps/ieee754/ldbl-128/math_ldbl.h * sysdeps/ieee754/ldbl-128ibm/math_ldbl.h * sysdeps/ieee754/ldbl-96/math_ldbl.h * sysdeps/powerpc/fpu/math_ldbl.h * sysdeps/x86_64/fpu/math_ldbl.h: Allow direct inclusion. Use uintNN_t instead of u_intNN_t. Use __BIG_ENDIAN and __LITTLE_ENDIAN, not BIG_ENDIAN and LITTLE_ENDIAN. Include endian.h and/or stdint.h if necessary. Add copyright notices. * sysdeps/ieee754/ldbl-128ibm/math_ldbl.h (ldbl_canonicalize_int): Don't use EXTRACT_WORDS64. * sysdeps/ieee754/ldbl-96/test-canonical-ldbl-96.c * sysdeps/ieee754/ldbl-96/test-totalorderl-ldbl-96.c * sysdeps/ieee754/ldbl-128ibm/test-canonical-ldbl-128ibm.c * sysdeps/ieee754/ldbl-128ibm/test-totalorderl-ldbl-128ibm.c: Include math_ldbl.h, not math_private.h.
2017-02-24Run libm tests separately for each function.Joseph Myers
At present, libm tests for each function get built into a single executable (for each floating point type, for each of normal / inline / finite-math-only functions, plus vector variants) and run together, resulting in a single PASS or FAIL (for each of those nine variants plus vector variants). Building this executable involves reading over 50 MB of libm-test-*.c sources. This patch arranges for tests of each function to be run separately from the makefiles instead. There are 121 functions being tested for each (type, variant pair) (actually 126, but run as 121 from the Makefile because each of the pairs (exp10, pow10), (isfinite, finite), (lgamma, gamma), (remainder, drem), (scalbn, ldexp), shares a table of test results and so is run together), so 1089 separate tests run from the Makefile, plus 48 vector tests on x86_64 (six functions for eight vector variants). Each test only involves a libm-test-<func>.c file of no more than about 4 MB, rather than all such files taking about 50 MB. With tests run separately, test summaries will indicate which functions actually have problems (of course, those problems may just be out-of-date libm-test-ulps files if the file hasn't been updated for the architecture in question recently). All the .c files for the 1089+48 tests are generated automatically from the Makefiles. Various checked-in boilerplate .c files are removed as no longer needed. CFLAGS definitions for the different kinds of tests are generated using makefile iterators to apply target-specific variable settings. libm-have-vector-test.h is no longer needed; the list of functions to test for each vector type is now in the sysdeps Makefile. This should reduce the amount of boilerplate needed for float128 testing support; test-float128.h will still be needed, but not various .c files or Makefile CFLAGS definitions. The logic for creating dependencies on libm-test-support-*.o files should also render <https://sourceware.org/ml/libc-alpha/2017-02/msg00279.html> unnecessary. Tested for x86_64 and x86. * math/Makefile (libm-tests-generated): Remove variable. (libm-tests-base-normal): New variable. (libm-tests-base-finite): Likewise. (libm-tests-base-inline): Likewise. (libm-tests-base): Likewise. (libm-tests-normal): Likewise. (libm-tests-finite): Likewise. (libm-tests-inline): Likewise. (libm-tests-vector): Likewise. (libm-tests): Define in terms of these new variables. (libm-tests-for-type): New variable. (libm-tests.o): Move definition. (tests): Move addition of $(libm-tests). (generated): Update for new and removed libm test files. ($(objpfx)libm-test.c): Remove target. ($(objpfx)libm-have-vector-test.h): Likewise. (CFLAGS-test-double-vlen2.c): Remove variable. (CFLAGS-test-double-vlen4.c): Likewise. (CFLAGS-test-double-vlen8.c): Likewise. (CFLAGS-test-float-vlen4.c): Likewise. (CFLAGS-test-float-vlen8.c): Likewise. (CFLAGS-test-float-vlen16.c): Likewise. (CFLAGS-test-float.c): Likewise. (CFLAGS-test-float-finite.c): Likewise. (CFLAGS-libm-test-support-float.c): Likewise. (CFLAGS-test-double.c): Likewise. (CFLAGS-test-double-finite.c): Likewise. (CFLAGS-libm-test-support-double.c): Likewise. (CFLAGS-test-ldouble.c): Likewise. (CFLAGS-test-ldouble-finite.c): Likewise. (CFLAGS-libm-test-support-ldouble.c): Likewise. (libm-test-inline-cflags): New variable. (CFLAGS-test-ifloat.c): Remove variable. (CFLAGS-test-idouble.c): Likewise. (CFLAGS-test-ildouble.c): Likewise. ($(addprefix $(objpfx), $(libm-tests.o))): Move target and update dependencies. ($(foreach t,$(libm-tests-normal),$(objpfx)$(t).c)): New rule. ($(foreach t,$(libm-tests-finite),$(objpfx)$(t).c)): Likewise. ($(foreach t,$(libm-tests-inline),$(objpfx)$(t).c)): Likewise. ($(foreach t,$(libm-tests-vector),$(objpfx)$(t).c)): Likewise. ($(foreach t,$(types),$(objpfx)libm-test-support-$(t).c)): Likewise. (dependencies on libm-test-support-*.o): Remove. ($(foreach f,$(libm-test-funcs-all),$(objpfx)$(o)-$(f).o)): New rules using iterators. ($(addprefix $(objpfx),$(call libm-tests-for-type,$(o)))): Likewise. ($(objpfx)libm-test-support-$(o).o): Likewise. ($(addprefix $(objpfx),$(filter-out $(tests-static) $(libm-vec-tests),$(tests)))): Filter out $(libm-tests-vector) instead. ($(addprefix $(objpfx), $(libm-vec-tests))): Use iterator to define rule instead. * math/README.libm-test: Update. * math/libm-test-acos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-acosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-asin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-asinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-atan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-atan2.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-atanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cabs.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cacos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cacosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-canonicalize.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-carg.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-casin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-casinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-catan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-catanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cbrt.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ccos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ccosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ceil.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cexp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cimag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-clog.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-clog10.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-conj.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-copysign.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cosh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cpow.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-cproj.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-creal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-csin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-csinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-csqrt.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ctan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ctanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-erf.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-erfc.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-exp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-exp10.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-exp2.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-expm1.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fabs.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fdim.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-floor.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fma.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmax.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmaxmag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fminmag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fmod.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fpclassify.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-frexp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fromfp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-fromfpx.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-getpayload.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-hypot.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ilogb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-iscanonical.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-iseqsig.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isfinite.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isgreater.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isgreaterequal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isinf.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isless.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-islessequal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-islessgreater.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isnan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isnormal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-issignaling.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-issubnormal.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-isunordered.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-iszero.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-j0.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-j1.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-jn.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-lgamma.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-llogb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-llrint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-llround.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log10.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log1p.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-log2.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-logb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-lrint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-lround.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-modf.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nearbyint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nextafter.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nextdown.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nexttoward.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-nextup.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-pow.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-remainder.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-remquo.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-rint.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-round.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-roundeven.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-scalb.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-scalbln.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-scalbn.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-setpayload.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-setpayloadsig.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-signbit.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-significand.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sin.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sincos.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sinh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-sqrt.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-tan.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-tanh.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-tgamma.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-totalorder.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-totalordermag.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-trunc.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ufromfp.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-ufromfpx.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-y0.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-y1.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-yn.inc: Include libm-test-driver.c. (do_test): New function. * math/libm-test-driver.c: Do not include libm-have-vector-test.h. (HAVE_VECTOR): Remove macro. (START): Do not call HAVE_VECTOR. * math/test-double-vlen2.h (FUNC_TEST): Remove macro. * math/test-double-vlen4.h (FUNC_TEST): Remove macro. * math/test-double-vlen8.h (FUNC_TEST): Remove macro. * math/test-float-vlen16.h (FUNC_TEST): Remove macro. * math/test-float-vlen4.h (FUNC_TEST): Remove macro. * math/test-float-vlen8.h (FUNC_TEST): Remove macro. * math/test-math-vector.h (FUNC_TEST): New macro. (WRAPPER_DECL): Rename to WRAPPER_DECL_f. * sysdeps/x86_64/fpu/Makefile (double-vlen2-funcs): New variable. (double-vlen4-funcs): Likewise. (double-vlen4-avx2-funcs): Likewise. (double-vlen8-funcs): Likewise. (float-vlen4-funcs): Likewise. (float-vlen8-funcs): Likewise. (float-vlen8-avx2-funcs): Likewise. (float-vlen16-funcs): Likewise. (CFLAGS-test-double-vlen4-avx2.c): Remove variable. (CFLAGS-test-float-vlen8-avx2.c): Likewise. * sysdeps/x86_64/fpu/test-double-vlen4.h (TEST_VECTOR_cos): Remove macro. (TEST_VECTOR_sin): Likewise. (TEST_VECTOR_sincos): Likewise. (TEST_VECTOR_log): Likewise. (TEST_VECTOR_exp): Likewise. (TEST_VECTOR_pow): Likewise. * sysdeps/x86_64/fpu/test-double-vlen8.h (TEST_VECTOR_cos): Likewise. (TEST_VECTOR_sin): Likewise. (TEST_VECTOR_sincos): Likewise. (TEST_VECTOR_log): Likewise. (TEST_VECTOR_exp): Likewise. (TEST_VECTOR_pow): Likewise. * sysdeps/x86_64/fpu/test-float-vlen16.h (TEST_VECTOR_cosf): Likewise. (TEST_VECTOR_sinf): Likewise. (TEST_VECTOR_sincosf): Likewise. (TEST_VECTOR_logf): Likewise. (TEST_VECTOR_expf): Likewise. (TEST_VECTOR_powf): Likewise. * sysdeps/x86_64/fpu/test-float-vlen8.h (TEST_VECTOR_cosf): Likewise. (TEST_VECTOR_sinf): Likewise. (TEST_VECTOR_sincosf): Likewise. (TEST_VECTOR_logf): Likewise. (TEST_VECTOR_expf): Likewise. (TEST_VECTOR_powf): Likewise. * math/gen-libm-have-vector-test.sh: Remove file. * math/libm-test.inc: Likewise. * math/libm-test-support-double.c: Likewise. * math/libm-test-support-float.c: Likewise. * math/libm-test-support-ldouble.c: Likewise. * math/test-double-finite.c: Likewise.: Likewise. * math/test-double.c: Likewise. * math/test-float-finite.c: Likewise. * math/test-float.c: Likewise. * math/test-idouble.c: Likewise. * math/test-ifloat.c: Likewise. * math/test-ildouble.c: Likewise. * math/test-ldouble-finite.c: Likewise. * math/test-ldouble.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen2.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen2.h: Likewise. * sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen4.c: Likewise. * sysdeps/x86_64/fpu/test-double-vlen8.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen16.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen4.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen4.h: Likewise. * sysdeps/x86_64/fpu/test-float-vlen8-avx2.c: Likewise. * sysdeps/x86_64/fpu/test-float-vlen8.c: Likewise.
2017-02-17Move tests of catan, catanh to auto-libm-test-*.Joseph Myers
This patch moves tests of catan and catanh with finite inputs (other than the divide-by-zero cases producing an exact infinity) to using the auto-libm-test machinery. Each of auto-libm-test-out-catan and auto-libm-test-out-catanh takes about three seconds to generate on my system (so in fact it wasn't necessary after all to defer the move to auto-libm-test-* until the output files were split up by function). Tested for x86_64 and x86 and ulps updated accordingly. * math/auto-libm-test-in: Add tests of catan and catanh. * math/auto-libm-test-out-catan: New generated file. * math/auto-libm-test-out-catanh: Likewise. * math/libm-test-catan.inc (catan_test_data): Use AUTO_TESTS_c_c. Move tests with finite inputs, except divide-by-zero cases, to auto-libm-test-in. * math/libm-test-catanh.inc (catanh_test_data): Likewise. * math/Makefile (libm-test-funcs-auto): Add catan and catanh. (libm-test-funcs-noauto): Remove catan and catanh. * sysdeps/i386/fpu/libm-test-ulps: Update. * sysdeps/i386/i686/fpu/multiarch/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise.
2017-02-17Move tests of casin, casinh to auto-libm-test-*.Joseph Myers
This patch moves tests of casin and casinh with finite inputs to using the auto-libm-test machinery. Each of auto-libm-test-out-casin and auto-libm-test-out-casinh takes about 38 minutes to generate on my system because of MPC slowness on special cases that appear in the tests (with MPC 1.0.3; I don't know to what extent current MPC master might speed it up). Tested for x86_64 and x86 and ulps updated accordingly. * math/auto-libm-test-in: Add tests of casin and casinh. * math/auto-libm-test-out-casin: New generated file. * math/auto-libm-test-out-casinh: Likewise. * math/libm-test-casin.inc (casin_test_data): Use AUTO_TESTS_c_c. Move tests with finite inputs to auto-libm-test-in. * math/libm-test-casinh.inc (casinh_test_data): Likewise. * math/Makefile (libm-test-funcs-auto): Add casin and casinh. (libm-test-funcs-noauto): Remove casin and casinh. * sysdeps/i386/fpu/libm-test-ulps: Update. * sysdeps/i386/i686/fpu/multiarch/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise.
2017-02-17Move tests of cacos, cacosh to auto-libm-test-*.Joseph Myers
This patch moves tests of cacos and cacosh with finite inputs to using the auto-libm-test machinery. Each of auto-libm-test-out-cacos and auto-libm-test-out-cacosh takes about 80 minutes to generate on my system because of MPC slowness on special cases that appear in the tests (with MPC 1.0.3; I don't know to what extent current MPC master might speed it up). Tested for x86_64 and x86 and ulps updated accordingly. * math/auto-libm-test-in: Add tests of cacos and cacosh. * math/auto-libm-test-out-cacos: New generated file. * math/auto-libm-test-out-cacosh: Likewise. * math/libm-test-cacos.inc (cacos_test_data): Use AUTO_TESTS_c_c. Move tests with finite inputs to auto-libm-test-in. * math/libm-test-cacosh.inc (cacosh_test_data): Likewise. * math/Makefile (libm-test-funcs-auto): Add cacos and cacosh. (libm-test-funcs-noauto): Remove cacos and cacosh. * sysdeps/i386/fpu/libm-test-ulps: Update. * sysdeps/i386/i686/fpu/multiarch/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise.
2017-02-17Revert header inclusion changes that break math/ testing on x86_64.Joseph Myers
Revert: 2017-02-16 Zack Weinberg <zackw@panix.com> * sysdeps/x86_64/fpu/math-tests-arch.h: Include cpu-features.h. Don't include init-arch.h. * sysdeps/x86_64/multiarch/test-multiarch.h: Include cpu-features.h. Don't include init-arch.h.
2017-02-16Add missing header files throughout the testsuite.Zack Weinberg
* crypt/md5.h: Test _LIBC with #if defined, not #if. * dirent/opendir-tst1.c: Include sys/stat.h. * dirent/tst-fdopendir.c: Include sys/stat.h. * dirent/tst-fdopendir2.c: Include stdlib.h. * dirent/tst-scandir.c: Include stdbool.h. * elf/tst-auditmod1.c: Include link.h and stddef.h. * elf/tst-tls15.c: Include stdlib.h. * elf/tst-tls16.c: Include stdlib.h. * elf/tst-tls17.c: Include stdlib.h. * elf/tst-tls18.c: Include stdlib.h. * iconv/tst-iconv6.c: Include endian.h. * iconvdata/bug-iconv11.c: Include limits.h. * io/test-utime.c: Include stdint.h. * io/tst-faccessat.c: Include sys/stat.h. * io/tst-fchmodat.c: Include sys/stat.h. * io/tst-fchownat.c: Include sys/stat.h. * io/tst-fstatat.c: Include sys/stat.h. * io/tst-futimesat.c: Include sys/stat.h. * io/tst-linkat.c: Include sys/stat.h. * io/tst-mkdirat.c: Include sys/stat.h and stdbool.h. * io/tst-mkfifoat.c: Include sys/stat.h and stdbool.h. * io/tst-mknodat.c: Include sys/stat.h and stdbool.h. * io/tst-openat.c: Include stdbool.h. * io/tst-readlinkat.c: Include sys/stat.h. * io/tst-renameat.c: Include sys/stat.h. * io/tst-symlinkat.c: Include sys/stat.h. * io/tst-unlinkat.c: Include stdbool.h. * libio/bug-memstream1.c: Include stdlib.h. * libio/bug-wmemstream1.c: Include stdlib.h. * libio/tst-fwrite-error.c: Include stdlib.h. * libio/tst-memstream1.c: Include stdlib.h. * libio/tst-memstream2.c: Include stdlib.h. * libio/tst-memstream3.c: Include stdlib.h. * malloc/tst-interpose-aux.c: Include stdint.h. * misc/tst-preadvwritev-common.c: Include sys/stat.h. * nptl/tst-basic7.c: Include limits.h. * nptl/tst-cancel25.c: Include pthread.h, not pthreadP.h. * nptl/tst-cancel4.c: Include stddef.h, limits.h, and sys/stat.h. * nptl/tst-cancel4_1.c: Include stddef.h. * nptl/tst-cancel4_2.c: Include stddef.h. * nptl/tst-cond16.c: Include limits.h. Use sysconf(_SC_PAGESIZE) instead of __getpagesize. * nptl/tst-cond18.c: Include limits.h. Use sysconf(_SC_PAGESIZE) instead of __getpagesize. * nptl/tst-cond4.c: Include stdint.h. * nptl/tst-cond6.c: Include stdint.h. * nptl/tst-stack2.c: Include limits.h. * nptl/tst-stackguard1.c: Include stddef.h. * nptl/tst-tls4.c: Include stdint.h. Don't include tls.h. * nptl/tst-tls4moda.c: Include stddef.h. Don't include stdio.h, unistd.h, or tls.h. * nptl/tst-tls4modb.c: Include stddef.h. Don't include stdio.h, unistd.h, or tls.h. * nptl/tst-tls5.h: Include stddef.h. Don't include stdlib.h or tls.h. * posix/tst-getaddrinfo2.c: Include stdio.h. * posix/tst-getaddrinfo5.c: Include stdio.h. * posix/tst-pathconf.c: Include sys/stat.h. * posix/tst-posix_fadvise-common.c: Include stdint.h. * posix/tst-preadwrite-common.c: Include sys/stat.h. * posix/tst-regex.c: Include stdint.h. Don't include spawn.h or spawn_int.h. * posix/tst-regexloc.c: Don't include spawn.h or spawn_int.h. * posix/tst-vfork3.c: Include sys/stat.h. * resolv/tst-bug18665-tcp.c: Include stdlib.h. * resolv/tst-res_hconf_reorder.c: Include stdlib.h. * resolv/tst-resolv-search.c: Include stdlib.h. * stdio-common/tst-fmemopen2.c: Include stdint.h. * stdio-common/tst-vfprintf-width-prec.c: Include stdlib.h. * stdlib/test-canon.c: Include sys/stat.h. * stdlib/tst-tls-atexit.c: Include stdbool.h. * string/test-memchr.c: Include stdint.h. * string/tst-cmp.c: Include stdint.h. * sysdeps/pthread/tst-timer.c: Include stdint.h. * sysdeps/unix/sysv/linux/tst-sync_file_range.c: Include stdint.h. * sysdeps/wordsize-64/tst-writev.c: Include limits.h and stdint.h. * sysdeps/x86_64/fpu/math-tests-arch.h: Include cpu-features.h. Don't include init-arch.h. * sysdeps/x86_64/multiarch/test-multiarch.h: Include cpu-features.h. Don't include init-arch.h. * sysdeps/x86_64/tst-auditmod10b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod3b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod4b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod5b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod6b.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod6c.c: Include link.h and stddef.h. * sysdeps/x86_64/tst-auditmod7b.c: Include link.h and stddef.h. * time/clocktest.c: Include stdint.h. * time/tst-posixtz.c: Include stdint.h. * timezone/tst-timezone.c: Include stdint.h.
2017-02-15Move most libmvec test contents from .c to .h files.Joseph Myers
The libmvec tests put substantive, architecture-specific contents in .c files such as test-double-vlen4.c, so making those files architecture-specific and causing issues for generating such files automatically when splitting up tests by function. This patch moves all the substantive contents to .h files, so the .c files only include the .h file and then libm-test.c. This allows for automatic generation of per-function .c files in future. The .h files in turn #include or #include_next the architecture-independent file and add the architecture-specific definitions to that. (Splitting by function should in fact allow the TEST_VECTOR_* macros to be replaced by sysdeps makefile information on which functions to test in each case, removing the need for gen-libm-have-vector-test.sh as well as removing the need for some of the architecture-specific headers.) Tested for x86_64. * sysdeps/x86_64/fpu/test-double-vlen2.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen2.h: ... here. New file. * sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen4-avx2.h: ... here. New file. * sysdeps/x86_64/fpu/test-double-vlen4.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen4.h: ... here. New file. * sysdeps/x86_64/fpu/test-double-vlen8.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-double-vlen8.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen16.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen16.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen4.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen4.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen8-avx2.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen8-avx2.h: ... here. New file. * sysdeps/x86_64/fpu/test-float-vlen8.c: Move most contents to, and include ... * sysdeps/x86_64/fpu/test-float-vlen8.h: ... here. New file.
2017-02-09x86-64: Verify that _dl_runtime_resolve preserves vector registersH.J. Lu
On x86-64, _dl_runtime_resolve must preserve the first 8 vector registers. Add 3 _dl_runtime_resolve tests to verify that SSE, AVX and AVX512 registers are preserved. * sysdeps/x86_64/Makefile (tests): Add tst-sse, tst-avx and tst-avx512. (test-extras): Add tst-avx-aux and tst-avx512-aux. (extra-test-objs): Add tst-avx-aux.o and tst-avx512-aux.o. (modules-names): Add tst-ssemod, tst-avxmod and tst-avx512mod. ($(objpfx)tst-sse): New rule. ($(objpfx)tst-avx): Likewise. ($(objpfx)tst-avx512): Likewise. (CFLAGS-tst-avx-aux.c): New. (CFLAGS-tst-avxmod.c): Likewise. (CFLAGS-tst-avx512-aux.c): Likewise. (CFLAGS-tst-avx512mod.c): Likewise. * sysdeps/x86_64/tst-avx-aux.c: New file. * sysdeps/x86_64/tst-avx.c: Likewise. * sysdeps/x86_64/tst-avx512-aux.c: Likewise. * sysdeps/x86_64/tst-avx512.c: Likewise. * sysdeps/x86_64/tst-avx512mod.c: Likewise. * sysdeps/x86_64/tst-avxmod.c: Likewise. * sysdeps/x86_64/tst-sse.c: Likewise. * sysdeps/x86_64/tst-ssemod.c: Likewise.
2017-02-06Remove i686, x86_64, and powerpc strtok implementationsAdhemerval Zanella
Based on comments on previous attempt to address BZ#16640 [1], the idea is not support invalid use of strtok (the original bug report proposal). This leader to a new strtok optimized strtok implementation [2]. The idea of this patch is to fix BZ#16640 to align all the implementations to a same contract. However, with newer strtok code it is better to get remove the old assembly ones instead of fix them. For x86 is a gain in all cases since the new implementation can potentially use sse2/sse42 implementation for strspn and strcspn. This shows a better performance on both i686 and x86_64 using the string benchtests. On powerpc64 the gains are mixed, where only for larger inputs or keys some gains are showns (based on benchtest it seems that it shows some gains for keys larger than 10 and inputs larger than 32). I would prefer to remove the optimized implementation based on first code simplicity and second because some more gain could be optimized using a better optimized strcspn/strspn code (as for x86). However if powerpc arch maintainers prefer I can send a v2 with the assembly code adjusted instead. Checked on x86_64-linux-gnu, i686-linux-gnu, and powerpc64le-linux-gnu. [BZ #16640] * sysdeps/i386/i686/strtok.S: Remove file. * sysdeps/i386/i686/strtok_r.S: Likewise. * sysdeps/i386/strtok.S: Likewise. * sysdeps/i386/strtok_r.S: Likewise. * sysdeps/powerpc/powerpc64/strtok.S: Likewise. * sysdeps/powerpc/powerpc64/strtok_r.S: Likewise. * sysdeps/x86_64/strtok.S: Likewise. * sysdeps/x86_64/strtok_r.S: Likewise. [1] https://sourceware.org/ml/libc-alpha/2016-10/msg00411.html [2] https://sourceware.org/ml/libc-alpha/2016-12/msg00461.html
2017-02-02Allow IFUNC relocation against unrelocated shared libraryH.J. Lu
IFUNC relocation against definition in unrelocated shared library will lead to segfault when the IFUNC function is called. This patch allows such IFUNC relocations with a warning. This isn't a real fix for https://sourceware.org/bugzilla/show_bug.cgi?id=21041 It simply allows the program to load. The program will segfault when longjmp is called. * sysdeps/i386/dl-machine.h (elf_machine_rel): Replace _dl_fatal_printf with _dl_error_printf for IFUNC relocation against unrelocated shared library. * sysdeps/x86_64/dl-machine.h (elf_machine_rela): Likewise.
2017-01-30Add VZEROUPPER to memset-vec-unaligned-erms.S [BZ #21081]H.J. Lu
Since memset-vec-unaligned-erms.S has VDUP_TO_VEC0_AND_SET_RETURN at function entry, memset optimized for AVX2 and AVX512 will always use ymm/zmm register. VZEROUPPER should be placed before ret in L(stosb): movq %rdx, %rcx movzbl %sil, %eax movq %rdi, %rdx rep stosb movq %rdx, %rax ret since it can be reached from L(stosb_more_2x_vec): cmpq $REP_STOSB_THRESHOLD, %rdx ja L(stosb) [BZ #21081] * sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S (L(stosb)): Add VZEROUPPER before ret.
2017-01-03Fix x86 strncat optimized implementation for large sizesAdhemerval Zanella
Similar to BZ#19387, BZ#21014, and BZ#20971, both x86 sse2 strncat optimized assembly implementations do not handle the size overflow correctly. The x86_64 one is in fact an issue with strcpy-sse2-unaligned, but that is triggered also with strncat optimized implementation. This patch uses a similar strategy used on 3daef2c8ee4df2, where saturared math is used for overflow case. Checked on x86_64-linux-gnu and i686-linux-gnu. It fixes BZ #19390. [BZ #19390] * string/test-strncat.c (test_main): Add tests with SIZE_MAX as maximum string size. * sysdeps/i386/i686/multiarch/strcat-sse2.S (STRCAT): Avoid overflow in pointer addition. * sysdeps/x86_64/multiarch/strcpy-sse2-unaligned.S (STRCPY): Likewise.
2017-01-01Update copyright dates with scripts/update-copyrights.Joseph Myers
2016-12-27Fix x86_64 memchr for large input sizesAdhemerval Zanella
Current optimized memchr for x86_64 does for input arguments pointers module 64 in range of [49,63] if there is no searchr char in the rest of 64-byte block a pointer addition which might overflow: * sysdeps/x86_64/memchr.S 77 .p2align 4 78 L(unaligned_no_match): 79 add %rcx, %rdx Add (uintptr_t)s % 16 to n in %rdx. 80 sub $16, %rdx 81 jbe L(return_null) This patch fixes by adding a saturated math that sets a maximum pointer value if it overflows (UINTPTR_MAX). Checked on x86_64-linux-gnu and powerpc64-linux-gnu. [BZ# 19387] * sysdeps/x86_64/memchr.S (memchr): Avoid overflow in pointer addition. * string/test-memchr.c (do_test): Remove alignment limitation. (test_main): Add test that trigger BZ# 19387.
2016-12-26Do not stack-protect ifunc resolvers [BZ #7065]Nick Alcock
When dynamically linking, ifunc resolvers are called before TLS is initialized, so they cannot be safely stack-protected. We avoid disabling stack-protection on large numbers of files by using __attribute__ ((__optimize__ ("-fno-stack-protector"))) to turn it off just for the resolvers themselves. (We provide the attribute even when statically linking, because we will later use it elsewhere too.)
2016-12-21x86_64: tst-quad1pie, tst-quad2pie: compile with -fPIE [BZ #7065]Nick Alcock
With stack protection enabled, these files have external symbol references for the first time, so the fact that they are not compiled with -fPIE and are then linked into a -pie binary starts to hurt.
2016-12-15Fix x86, x86_64 fmax, fmin sNaN handling, add tests (bug 20947).Joseph Myers
Various fmax and fmin function implementations mishandle sNaN arguments: (a) When both arguments are NaNs, the return value should be a qNaN, but sometimes it is an sNaN if at least one argument is an sNaN. (b) Under TS 18661-1 semantics, if either argument is an sNaN then the result should be a qNaN (whereas if one argument is a qNaN and the other is not a NaN, the result should be the non-NaN argument). Various implementations treat sNaNs like qNaNs here. This patch fixes the x86 and x86_64 versions (ignoring float and double for 32-bit x86 given the inability to reliably avoid the sNaN turning into a qNaN before it gets to the called function). Tests of sNaN inputs to these functions are added. Note on architecture versions I haven't changed for this issue: AArch64 already gets this right (it uses a hardware instruction with the correct semantics for both quiet and signaling NaNs) and does not need changes. It's possible Alpha, IA64, SPARC might need changes (this would be shown by the testsuite if so). Tested for x86_64 and x86 (both i686 and i586 builds, to cover the different x86 implementations). [BZ #20947] * sysdeps/i386/fpu/s_fmaxl.S (__fmaxl): Add the arguments when either is a signaling NaN. * sysdeps/i386/fpu/s_fminl.S (__fminl): Likewise. Make code follow fmaxl more closely. * sysdeps/i386/i686/fpu/s_fmaxl.S (__fmaxl): Add the arguments when either is a signaling NaN. * sysdeps/i386/i686/fpu/s_fminl.S (__fminl): Likewise. * sysdeps/x86_64/fpu/s_fmax.S (__fmax): Likewise. * sysdeps/x86_64/fpu/s_fmaxf.S (__fmaxf): Likewise. * sysdeps/x86_64/fpu/s_fmaxl.S (__fmaxl): Likewise. * sysdeps/x86_64/fpu/s_fmin.S (__fmin): Likewise. * sysdeps/x86_64/fpu/s_fminf.S (__fminf): Likewise. * sysdeps/x86_64/fpu/s_fminl.S (__fminl): Likewise. * math/libm-test.inc (fmax_test_data): Add tests of sNaN inputs. (fmin_test_data): Likewise.
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-17 14:57:18 +0000