|
Message-ID: <20181115195148.GA5044@pi3.com.pl> Date: Thu, 15 Nov 2018 20:51:48 +0100 From: Adam Zabrocki <pi3@....com.pl> To: lkrg-users@...ts.openwall.com, Jiří Moravec <jim.lkml@...il.com> Subject: Re: p_lkrg failed to start with error: 'KMOD error! Can't initialize global modules variable' Hi Jiří, Can you verify if bitbucket repo version of LKRG also works for you? I've set-up a new gentoo VMs and don't see any problems (please see mailthread below), but would like to get your input too. Thanks, Adam On Wed, Nov 14, 2018 at 06:16:00PM +0100, Adam Zabrocki wrote: > I've verified kernel 4.19.1 and didn't have any problems on Fedora, even > without changing wrapper code which is somehow necessary for the gcc which > Gentoo uses. I've refactor wrapper and set-up Gentoo environment and I was > able to run LKRG on 4.19.1 on Gentoo without any problems as well: > > === Fedora - kernel 4.19.1 === > root@...-vm:~/zzz# git clone > https://Adam_pi3@...bucket.org/Adam_pi3/lkrg-main.git > Cloning into 'lkrg-main'... > remote: Counting objects: 1119, done. > remote: Compressing objects: 100% (1028/1028), done. > remote: Total 1119 (delta 608), reused 179 (delta 83) > Receiving objects: 100% (1119/1119), 209.73 KiB | 2.83 MiB/s, done. > Resolving deltas: 100% (608/608), done. > -root@...-vm:~/zzz# ls -al > total 12 > drwxr-xr-x 3 root root 4096 Nov 13 23:54 . > drwx------ 15 root root 4096 Nov 13 23:53 .. > drwxr-xr-x 4 root root 4096 Nov 13 23:54 lkrg-main > root@...-vm:~/zzz# cd lkrg-main/ > root@...-vm:~/zzz/lkrg-main# make clean; make > make -C /lib/modules/4.19.1-041901-generic/build M=/root/zzz/lkrg-main clean > make[1]: Entering directory '/usr/src/linux-headers-4.19.1-041901-generic' > make[1]: Leaving directory '/usr/src/linux-headers-4.19.1-041901-generic' > rm -f Module.markers modules.order > rm -f /root/zzz/lkrg-main/src/modules/kmod/client/kmod/Module.markers > rm -f /root/zzz/lkrg-main/src/modules/kmod/client/kmod/modules.order > rm -f -rf output > make -C /lib/modules/4.19.1-041901-generic/build M=/root/zzz/lkrg-main modules > make[1]: Entering directory '/usr/src/linux-headers-4.19.1-041901-generic' > Makefile:960: "Cannot use CONFIG_STACK_VALIDATION=y, please install libelf-dev, > libelf-devel or elfutils-libelf-devel" > CC [M] /root/zzz/lkrg-main/src/modules/ksyms/p_resolve_ksym.o > CC [M] /root/zzz/lkrg-main/src/modules/hashing/p_lkrg_fast_hash.o > CC [M] /root/zzz/lkrg-main/src/modules/comm_channel/p_comm_channel.o > CC [M] /root/zzz/lkrg-main/src/modules/wrap/p_struct_wrap.o > CC [M] /root/zzz/lkrg-main/src/modules/integrity_timer/p_integrity_timer.o > CC [M] /root/zzz/lkrg-main/src/modules/kmod/p_kmod.o > CC [M] /root/zzz/lkrg-main/src/modules/database/CPU.o > CC [M] /root/zzz/lkrg-main/src/modules/database/arch/x86/IDT_MSR_CRx.o > CC [M] > /root/zzz/lkrg-main/src/modules/database/arch/x86/p_switch_idt/p_switch_idt.o > CC [M] /root/zzz/lkrg-main/src/modules/database/arch/p_arch_metadata.o > CC [M] /root/zzz/lkrg-main/src/modules/database/p_database.o > CC [M] /root/zzz/lkrg-main/src/modules/notifiers/p_notifiers.o > CC [M] /root/zzz/lkrg-main/src/modules/self-defense/hiding/p_hiding.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/p_rb_ed_trees/p_rb_ed_pids/p_rb_ed_pids_tree.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_execve/p_sys_execve.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_execveat/p_sys_execveat.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_call_usermodehelper/p_call_usermodehelper.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_do_exit/p_do_exit.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_do_fork/p_do_fork.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setuid/p_sys_setuid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setreuid/p_sys_setreuid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setresuid/p_sys_setresuid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setfsuid/p_sys_setfsuid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setgid/p_sys_setgid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setregid/p_sys_setregid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setresgid/p_sys_setresgid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setfsgid/p_sys_setfsgid.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_set_current_groups/p_set_current_groups.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_do_init_module/p_do_init_module.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_finit_module/p_sys_finit_module.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_delete_module/p_sys_delete_module.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_generic_permission/p_generic_permission.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sel_write_enforce/p_sel_write_enforce.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_seccomp/p_seccomp.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_unshare/p_sys_unshare.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_userns_install/p_userns_install.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/caps/p_sys_capset/p_sys_capset.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/caps/p_cap_task_prctl/p_cap_task_prctl.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_key_change_session_keyring/p_key_change_session_keyring.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_sys_add_key/p_sys_add_key.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_sys_request_key/p_sys_request_key.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_sys_keyctl/p_sys_keyctl.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_ptrace/p_sys_ptrace.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_execve/p_compat_sys_execve.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_execveat/p_compat_sys_execveat.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_keyctl/p_compat_sys_keyctl.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_ptrace/p_compat_sys_ptrace.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_delete_module/p_compat_sys_delete_module.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_capset/p_compat_sys_capset.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_add_key/p_compat_sys_add_key.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_request_key/p_compat_sys_request_key.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_execve/p_x32_sys_execve.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_execveat/p_x32_sys_execveat.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_keyctl/p_x32_sys_keyctl.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_ptrace/p_x32_sys_ptrace.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/override/p_override_creds/p_override_creds.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/syscalls/override/p_revert_creds/p_revert_creds.o > CC [M] > /root/zzz/lkrg-main/src/modules/exploit_detection/p_exploit_detection.o > CC [M] /root/zzz/lkrg-main/src/p_lkrg_main.o > LD [M] /root/zzz/lkrg-main/p_lkrg.o > Building modules, stage 2. > MODPOST 1 modules > CC /root/zzz/lkrg-main/p_lkrg.mod.o > LD [M] /root/zzz/lkrg-main/p_lkrg.ko > make[1]: Leaving directory '/usr/src/linux-headers-4.19.1-041901-generic' > mkdir -p output > cp /root/zzz/lkrg-main/p_lkrg.ko output > root@...-vm:~/zzz/lkrg-main# insmod output/p_lkrg.ko p_init_log_level=0 > root@...-vm:~/zzz/lkrg-main# dmesg |tail > [ 54.169366] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 133.479974] new mount options do not match the existing superblock, will be > ignored > [ 354.224236] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 556.507388] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready > [ 556.507417] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready > [ 668.016417] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 728.438762] p_lkrg: loading out-of-tree module taints kernel. > [ 728.438826] p_lkrg: module verification failed: signature and/or required > key missing - tainting kernel > [ 728.440743] [p_lkrg] Loading LKRG... > [ 728.976751] [p_lkrg] LKRG initialized successfully! > root@...-vm:~/zzz/lkrg-main# uname -a > Linux pi3-vm 4.19.1-041901-generic #201811041431 SMP Sun Nov 4 14:33:06 UTC > 2018 x86_64 x86_64 x86_64 GNU/Linux > root@...-vm:~/zzz/lkrg-main# dmesg |tail > [ 54.169366] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 133.479974] new mount options do not match the existing superblock, will be > ignored > [ 354.224236] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 556.507388] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready > [ 556.507417] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready > [ 668.016417] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 728.438762] p_lkrg: loading out-of-tree module taints kernel. > [ 728.438826] p_lkrg: module verification failed: signature and/or required > key missing - tainting kernel > [ 728.440743] [p_lkrg] Loading LKRG... > [ 728.976751] [p_lkrg] LKRG initialized successfully! > root@...-vm:~/zzz/lkrg-main# sysctl lkrg.log_level=1 > lkrg.log_level = 1 > root@...-vm:~/zzz/lkrg-main# sysctl lkrg.force_run=1 > lkrg.force_run = 1 > root@...-vm:~/zzz/lkrg-main# dmesg |tail > [ 133.479974] new mount options do not match the existing superblock, will be > ignored > [ 354.224236] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 556.507388] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready > [ 556.507417] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready > [ 668.016417] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 728.438762] p_lkrg: loading out-of-tree module taints kernel. > [ 728.438826] p_lkrg: module verification failed: signature and/or required > key missing - tainting kernel > [ 728.440743] [p_lkrg] Loading LKRG... > [ 728.976751] [p_lkrg] LKRG initialized successfully! > [ 826.969011] [p_lkrg] System is clean! > === End === > > For some reasons gcc on Gentoo are more strict and requires to refactor wrapper > code which I've made and push the changes to the official repo here: > > https://bitbucket.org/Adam_pi3/lkrg-main/commits/d17deeb974bf69494c72d29996d17840ac2b85ca > > === Gentoo - kernel 4.19.1 === > pi3-gentoo ~/lkrg-main # cat /etc/gentoo-release > Gentoo Base System release 2.6 > pi3-gentoo ~/lkrg-main # uname -a > Linux pi3-gentoo 4.19.1-gentoo #1 SMP Tue Nov 13 11:23:06 EST 2018 x86_64 > Intel(R) Core(TM) i5-7300U CPU @ 2.60GHz GenuineIntel GNU/Linux > pi3-gentoo ~/lkrg-main # make clean; make > make -C /lib/modules/4.19.1-gentoo/build M=/root/lkrg-main clean > make[1]: Entering directory '/usr/src/linux-4.19.1-gentoo' > make[1]: Leaving directory '/usr/src/linux-4.19.1-gentoo' > rm -f Module.markers modules.order > rm -f /root/lkrg-main/src/modules/kmod/client/kmod/Module.markers > rm -f /root/lkrg-main/src/modules/kmod/client/kmod/modules.order > rm -f -rf output > make -C /lib/modules/4.19.1-gentoo/build M=/root/lkrg-main modules > make[1]: Entering directory '/usr/src/linux-4.19.1-gentoo' > CC [M] /root/lkrg-main/src/modules/ksyms/p_resolve_ksym.o > CC [M] /root/lkrg-main/src/modules/hashing/p_lkrg_fast_hash.o > CC [M] /root/lkrg-main/src/modules/comm_channel/p_comm_channel.o > CC [M] /root/lkrg-main/src/modules/integrity_timer/p_integrity_timer.o > CC [M] /root/lkrg-main/src/modules/kmod/p_kmod.o > CC [M] /root/lkrg-main/src/modules/database/CPU.o > CC [M] /root/lkrg-main/src/modules/database/arch/x86/IDT_MSR_CRx.o > CC [M] > /root/lkrg-main/src/modules/database/arch/x86/p_switch_idt/p_switch_idt.o > CC [M] /root/lkrg-main/src/modules/database/arch/p_arch_metadata.o > CC [M] /root/lkrg-main/src/modules/database/p_database.o > CC [M] /root/lkrg-main/src/modules/notifiers/p_notifiers.o > CC [M] /root/lkrg-main/src/modules/self-defense/hiding/p_hiding.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/p_rb_ed_trees/p_rb_ed_pids/p_rb_ed_pids_tree.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_execve/p_sys_execve.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_execveat/p_sys_execveat.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_call_usermodehelper/p_call_usermodehelper.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_do_exit/p_do_exit.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_do_fork/p_do_fork.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setuid/p_sys_setuid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setreuid/p_sys_setreuid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setresuid/p_sys_setresuid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setfsuid/p_sys_setfsuid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setgid/p_sys_setgid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setregid/p_sys_setregid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setresgid/p_sys_setresgid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_setfsgid/p_sys_setfsgid.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_set_current_groups/p_set_current_groups.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_do_init_module/p_do_init_module.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_finit_module/p_sys_finit_module.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_delete_module/p_sys_delete_module.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_generic_permission/p_generic_permission.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sel_write_enforce/p_sel_write_enforce.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_seccomp/p_seccomp.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_unshare/p_sys_unshare.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_userns_install/p_userns_install.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/caps/p_sys_capset/p_sys_capset.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/caps/p_cap_task_prctl/p_cap_task_prctl.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_key_change_session_keyring/p_key_change_session_keyring.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_sys_add_key/p_sys_add_key.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_sys_request_key/p_sys_request_key.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/keyring/p_sys_keyctl/p_sys_keyctl.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/p_sys_ptrace/p_sys_ptrace.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_execve/p_compat_sys_execve.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_execveat/p_compat_sys_execveat.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_keyctl/p_compat_sys_keyctl.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_ptrace/p_compat_sys_ptrace.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_delete_module/p_compat_sys_delete_module.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_capset/p_compat_sys_capset.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_add_key/p_compat_sys_add_key.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/compat/p_compat_sys_request_key/p_compat_sys_request_key.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_execve/p_x32_sys_execve.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_execveat/p_x32_sys_execveat.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_keyctl/p_x32_sys_keyctl.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/__x32/p_x32_sys_ptrace/p_x32_sys_ptrace.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/override/p_override_creds/p_override_creds.o > CC [M] > /root/lkrg-main/src/modules/exploit_detection/syscalls/override/p_revert_creds/p_revert_creds.o > CC [M] /root/lkrg-main/src/modules/exploit_detection/p_exploit_detection.o > CC [M] /root/lkrg-main/src/p_lkrg_main.o > LD [M] /root/lkrg-main/p_lkrg.o > Building modules, stage 2. > MODPOST 1 modules > CC /root/lkrg-main/p_lkrg.mod.o > LD [M] /root/lkrg-main/p_lkrg.ko > make[1]: Leaving directory '/usr/src/linux-4.19.1-gentoo' > mkdir -p output > cp /root/lkrg-main/p_lkrg.ko output > pi3-gentoo ~/lkrg-main # sysctl lkrg.force_run=1 > lkrg.force_run = 1 > pi3-gentoo ~/lkrg-main # dmesg > ... > ... > [ 6158.208071] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 6458.219218] hv_balloon: Balloon request will be partially fulfilled. Balloon > floor reached. > [ 6699.625164] [p_lkrg] Loading LKRG... > [ 6699.861386] [p_lkrg] LKRG initialized successfully! > [ 6707.946765] [p_lkrg] System is clean! > pi3-gentoo ~/lkrg-main # > === End === > > Thanks, > Adam > > On Mon, Nov 12, 2018 at 08:02:28PM +0100, Solar Designer wrote: > > On Mon, Nov 12, 2018 at 06:40:16PM +0100, Adam Zabrocki wrote: > > > Thanks for details. I didn't have a chance to run LKRG on kernel 4.19+. The > > > latest kernel version which I've tested is 4.18.7. I will try to set-up testing > > > environment and take a look at the reported problem. > > > > Ouch. I think we should adopt a policy to always test on latest > > released mainline kernel before releasing a new version of LKRG. > > > > Alexander > > -- > pi3 (pi3ki31ny) - pi3 (at) itsec pl > http://pi3.com.pl -- pi3 (pi3ki31ny) - pi3 (at) itsec pl http://pi3.com.pl
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.